Why can't I run multiple cuda tasks ?
A single task uses less than 1GB inbuilt, but a second task fails because of insufficient memory.
Please check - either the error or the message seems incorrect.

In earlier requests Pianoman postet, that the tasks need up to 2 GB mamory. Not always, but from time to time.
I tried it on my GTX1060 / 3GB as well, 80% of the wu's fail.
My 2 pc' with gpu's 4GB or more work fine with 2 wu's.

@alex
Thanx for the info.
I tried looking for some relevant post, but didn't find that of Pianoman.
My cards have only 2GB, so maybe I'm lucky that none of my 'singletons' have crashed !

You can find it in in the issue discussions, wu's fail with err. message out of memory

The reply was:
The error indicates the system ran out of GPU RAM.

Each WU takes on the order of 1.6GB-1.9GB of GPU memory when computing. And we developed the cuda app on a system with a 1650 with only 4GB of ram, so your 1060 6GB should have plenty of headroom with memory.
Are you were running anything else graphics intensive at the time? maybe a game? Or are you trying to run multiple WUs at the same time on a GPU? if so you could easily run out of GPU memory in total.

Hope that helps, and thanks for crunching!

Sadly, 2GB is really pushing it. PyTorch is very aggressive with memory usage, and will preallocate and pin memory wherever it can. It;s also hard to get a real profile of the amount used since some tools include both GPU memory and system memory.

We could and should be better about pinning this down and minimizing this. Its just not high on my priority list unfortunately.