So we finally got around to testing dataset 3 WUs on GPUs, and we are seeing a little bit of speedup. On an AMD ROCm system (linux, vega56) I'm seeing runtimes reduced about 40%, and that's without any real optimizations. 26s/epoch GPU, vs 44s/epoch CPU on the same machine.

That means we should prioritize releasing some GPU accelerated clients soon. Linux ROCm and CUDA will be first. Windows CUDA will follow. ROCm will require some pretty specific requirements and probably require advanced editing of config files on the client (to add "rocm" as an "app plan" in BOINC terminology).

Don't get too excited yet, just letting you all know there's been some progress and some actual benefit.

Thanks for the updates!
I'm not quite sure about the AMD Vega56 GPU, but I had expected at least an order of magnitude faster that those 40% you mentioned.
Maybe PyTorch doesn't make as good use of the AMD ROCm as it does with CUDA?
It should be very interesting to hear from you when you have a CUDA test app ready. I have two machines with usable GPUs (GTX960 and GTX750Ti) and I will gladly help with the testing. I'll set them up on MLC as soon as there is a CUDA app! :-)
Nice weekend!!!
//Gunnar

That was fast! Great to see the GPU porting process moving along. While I think a 40% runtime increase is already impressive, I must side with Gunnar's expectation here.

I am not sure what factor of improvement can usually be expected, but I had rather thought this ballpark number to be in the range of 3-5x . Converting this number to a CPU equivalence, a rather powerful GPU could only be as efficient as 1.66, so not even 2 CPU threads. A mere 6-core could thus outpace a GPU by a factor of 3.6 x. I would definitely also give the GPU app version a go with my 750Ti, but I expect that this old GPU probably wouldn't even achieve this 40% decrease in runtime, rendering it rather inefficient. While some people with more powerful hardware might also jump in to support beta testing the GPU version, should the numbers roughly meet these expectations, I'd rather run the GPU on GPU Grid and/or F@H at the moment.

I too would expect a MUCH larger reduction than a mere 40%. Using my MLC CPU averages across my 350 threads of ~ 2 hours, I would expect run times of <20 min on a GTX 1080Ti and <10 min on a RTX 2080Ti and half of that on the latest GTX/RTX 30x0 (for those who can afford the ridiculous price).

Unfortunately I have only 3 x GTX 970's on Linux (Ubuntu) machines and will run them when you are ready to test. The 15 x GTX10x0 & RTX2080Ti & the one wimpy GTX 750Ti are Win10 but will add them too when you are ready.

Best of luck!

---

Every problem is different. I'm frankly ecstatic we're seeing benefit at all! Blanket statements like "GPUs are always X times faster" really need to come with an asterisk.

Our particular networks so far have had a relatively low number of parameters, and our inputs are small compared to, say image processing.

Using a GPU has overhead, and the computation needs to be big enough to overcome that overhead. The bigger the network, the more parallel computation to be exploited on a GPU. For Dataset 1 and Dataset 2, the size of the networks is tiny (~4100 parameters). That fits well within CPU caches, and as such is very fast to compute on CPUs already, and the overhead of moving and formatting the data to the GPU and back dwarfs the small benefit from parallel computation on the GPU. Dataset 3's WUs are networks with approximately 136,000 parameters. Bigger, but still small/low-end-of-medium by ML standards (a relatively pedestrian LeNet CNN/image classification network is well over 1m parameters). So we're seeing some benefit with Dataset 3 (~1.8x faster), but it's still not large enough to have the out-sized impact you might be expecting. If it did, then using multi-threading on the CPU would probably also be a win (for the record, the current client app actually supports multhreading now, and I do benchmark this, but the speedup is only ~1.3-1.4x on CPU for Dataset 3. and a net slowdown on Datasets 1 and 2.). Other tricks like using less precision (fp16/int8) would certainly speed things up, but we'd lose the very precision this project is trying to measure/capture.

Now, future datasets on this project won't have such small WUs. Dataset 4 (dev in progress) will be LeNet based. then we might finally see some of the speedups you all are expecting versus CPU, and future versions will be CNNs targeting CIFAR-10/CIFAR-100 benchmarks. So it's useful to get GPU support out and working now that we're seeing some benefit.

I hope that answers some of your questions (any why GPU support hasn't been a priority until now).

Preliminary CUDA benchmarks coming soon, but probably not by tomorrow.

Thanks for the thorough and understandable explanation!!
Now I'm beginning to understand it a bit better. :-)
We will wait eagerly. Just issue a new test app if you want to get help with testing the CUDA version!
Nice crunchy weekend!!!
//Gunnar

You may be interested in the following information.
I didn’t go deep into ML methods, but the abbreviations seemed familiar to me. And I found them in the list of my "BOINC trophies"
The now virtually dead project Citizen Science Grid in 2017 launched different versions of image processing applications
(starting with manually searching and identifying birds in photos)
The names had similar abbreviations - CIFAR-10, MNIST, CNN, Scaled FMP, etc.

Thanks as well from me. Sounds like this is really a necessary step going forward, especially with more complex experiments to come.

I am also starting to think about how much work in a network training setup can be parallelised in general. So f.ex. imagining a fully trained model and then applying its learnt structure for classification purposes on various subsets of unseen data or learning a model at the same time parallel with various sets of hyperparameters, I see the potential for parallelisation. But what about the case here where we train a model only once on a given set of constraints/hyperparameters? IMO training is of rather successive nature, in a way of the prior epoch directly backpropagating to adjust the next epoch's neural weights. I don't see any major phase in training that could be parallelised as you cannot simply train the network with weights that have not yet been computed. Or do I miss some major point? Just thinking out loud here...
The "marginal" benefits you seem to describe (30-40%) by doubling the thread count via multithreading with dataset 3 WUs, seems to only support my thoughts here, that at least for now there is a cap on the benefits of parallelisation of the WU of any kind due to the very nature of the network training process.

As you mentioned however, the benefits especially for the more complex use cases could be far larger. I suspect as well that any kind of WU that would do dataset manipulations rather than only training, doing classification with a learnt network (f.ex. CNN image classification, ...) or any kind of recurrent network would benefit way more by parallelisation.

And thanks for letting us know about some of your "behind the scenes" testing about code/app optimisation. Great to read about the many ways you already actively tested for us to gain speed and efficiency with the current WU. Thanks as well for the peak review to dataset 4/5 targeting CIFAR! ... The time for a rig upgrade has arrived :)

Every problem is different. I'm frankly ecstatic we're seeing benefit at all! Blanket statements like "GPUs are always X times faster" really need to come with an asterisk.

Yes. I compare the efficiency (meaning energy per work unit) between a CPU and a GPU, not the speed. On Einstein, the Gravity Wave efficiency increases by less than a factor of two (about 1.4 the last time I checked).

On Asteroids, it is even worse. There is a loss of efficiency in going to a GPU.
http://asteroidsathome.net/boinc/forum_thread.php?id=807&postid=6738#6738

I would love to see a factor of 10 or more in all cases, but it does not work that way.

EDIT: It depends on how a project measures the output too. On Folding, there used to be a big (factor of 10) increase in going to a GPU. But they have a "quick return bonus" that favors increases in speed. As the CPUs have gotten more cores, they are gaining faster than the GPUs. On a Ryzen 3950X (32 virtual cores) I can get about 650 k PPD for around 110 watts. That is not far behind a GPU for the OpenCl work units, though the new CUDA work is helping the GPUs now.

EDIT 2: While we are on the subject, I checked the Quarantine@Home work units too, since they run Autodock on both CPUs and GPUs. The new Open Pandemics on WCG is using Autodock too, though a later version, which is CPU only now as they try to get the GPU version working.

At any rate, the Quarantine ones ran only 16 times faster on a GTX 1060 as on a single core of an i7-4771. If you ran them on 16 cores of a Ryzen 3000 series, you could do the same amount of work on less power on a CPU. I was slightly disappointed and a bit surprised.
https://www.worldcommunitygrid.org/forums/wcg/viewthread_thread,42470_offset,100#637691
https://www.worldcommunitygrid.org/forums/wcg/viewthread_thread,42470_offset,80#637684

At any rate, the Quarantine ones ran only 16 times faster on a GTX 1060 as on a single core of an i7-4771. If you ran them on 16 cores of a Ryzen 3000 series, you could do the same amount of work on less power on a CPU. I was slightly disappointed and a bit surprised.

You can use both cpu and gpu and you double your performance with a single host...

Is there any plan for adding GPU/CUDA support to the arm-unknown-linux-gnu platform?

While some may argue that GPU on ARM devices are typically not as powerful as those on desktop/workstations, note that Nvidia is acquiring Arm Holdings. In fact, they just released a $59 Jetson Nano with a CUDA-compatible GPU. Since we already have UW for arm-unknown-linux-gnu, and GPU support is currently under active development, I think it makes sense to put a little more effort into the combination of ARM and CUDA.

Is there any plan for adding GPU/CUDA support to the arm-unknown-linux-gnu platform?

While some may argue that GPU on ARM devices are typically not as powerful as those on desktop/workstations, note that Nvidia is acquiring Arm Holdings. In fact, they just released a $59 Jetson Nano with a CUDA-compatible GPU. Since we already have UW for arm-unknown-linux-gnu, and GPU support is currently under active development, I think it makes sense to put a little more effort into the combination of ARM and CUDA.

I saw that $60 nano announcement, looks fun!

Maybe at some point, but ARM-CUDA is pretty low on the priority list for now. Mainly because a) as of today, not many people have ARM+nvidia, and b) we don't have any ARM+nvidia hardware at the moment to test on. The latter can be solved with money, time, and effort; the former? we'll have to wait and see. I'd put it around the same priority as ppc64le CPU support. Small marketshare but potentially powerful. But at least with ppc64le, we have access to some hardware today.

How is testing coming along? I always happen to see several WUs on the server status page listed as test application and wonder whether internal testing with your system has already made it to beta testing here?

I have some nvidia hardware coming early next week to test. ROCm packaging is under way. Also OSX support, but that's another thread.

Been a busy week with non-MLC stuff, so I'm just now catching up.

Hi everyone,
I recently installed Boinc on a raspberry pi4 and was pleased to see it was contributing spare cycles to projects. Then I saw a Nvidia Jetson TK1 new on ebay and thought that might be a useful device to Boinc with..given its parallel processing capabilities. However I see the arm architecture isnt recognised and errors are reported. I'm writing this post on the system now, though Im certain Cuda isn't active yet, I would be grateful if anyone knows of any active forum or links relative to enabling this type of embedded board for project use.
I am reading through https://boinc.berkeley.edu/wiki/Client_configuration#Options
just looking for additional help.
Many Thanks, Risque.

Apologies, I did search for "Jetson TK1" before posting a new thread. Too specific. However Im glad its being discussed. I have 192 Kepler cores idle ..
Risque

Just to let you know there is progress. This is with a rand_automata/DS3 datafile. Want to get it into testing tonight or tomorrow.

PS C:\build\git\mlds\build\Release> .\mlds.exe -a LSTM -s 32 -w 64 --lr 0.001 -b 2
Machine Learning Dataset Generator v9.70 (Windows/x64) (libTorch: release/1.6 GPU: GeForce GTX 1650)
[2020-10-24 19:50:24                    main:402]       :       INFO    :       Set logging level to 1
[2020-10-24 19:50:24                    main:406]       :       INFO    :       Running in BOINC Standalone mode
[2020-10-24 19:50:24                    main:411]       :       INFO    :       Resolving all filenames
[2020-10-24 19:50:24                    main:419]       :       INFO    :       Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2020-10-24 19:50:24                    main:419]       :       INFO    :       Resolved: model.cfg => model.cfg (exists = 0)
[2020-10-24 19:50:24                    main:419]       :       INFO    :       Resolved: model-final.pt => model-final.pt (exists = 0)
[2020-10-24 19:50:24                    main:419]       :       INFO    :       Resolved: model-input.pt => model-input.pt (exists = 0)
[2020-10-24 19:50:24                    main:419]       :       INFO    :       Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2020-10-24 19:50:24                    main:433]       :       INFO    :       Dataset filename: dataset.hdf5
[2020-10-24 19:50:24                    main:435]       :       INFO    :       Configuration:
[2020-10-24 19:50:24                    main:436]       :       INFO    :           Model type: LSTM
[2020-10-24 19:50:24                    main:437]       :       INFO    :           Validation Loss Threshold: 0.0001
[2020-10-24 19:50:24                    main:438]       :       INFO    :           Max Epochs: 100
[2020-10-24 19:50:24                    main:439]       :       INFO    :           Batch Size: 32
[2020-10-24 19:50:24                    main:440]       :       INFO    :           Learning Rate: 0.001
[2020-10-24 19:50:24                    main:441]       :       INFO    :           Patience: 10
[2020-10-24 19:50:24                    main:442]       :       INFO    :           Hidden Width: 64
[2020-10-24 19:50:24                    main:443]       :       INFO    :           # Recurrent Layers: 4
[2020-10-24 19:50:24                    main:444]       :       INFO    :           # Backend Layers: 2
[2020-10-24 19:50:24                    main:445]       :       INFO    :           # Threads: 1
[2020-10-24 19:50:24                    main:447]       :       INFO    :       Preparing Dataset
[2020-10-24 19:50:24    load_hdf5_ds_into_tensor:28]    :       INFO    :       Loading Dataset /Xt from dataset.hdf5 into memory
[2020-10-24 19:50:24    load_hdf5_ds_into_tensor:28]    :       INFO    :       Loading Dataset /Yt from dataset.hdf5 into memory
[2020-10-24 19:50:27                    load:109]       :       INFO    :       Successfully loaded dataset of 4096 examples into memory.
[2020-10-24 19:50:27    load_hdf5_ds_into_tensor:28]    :       INFO    :       Loading Dataset /Xv from dataset.hdf5 into memory
[2020-10-24 19:50:27    load_hdf5_ds_into_tensor:28]    :       INFO    :       Loading Dataset /Yv from dataset.hdf5 into memory
[2020-10-24 19:50:27                    load:109]       :       INFO    :       Successfully loaded dataset of 512 examples into memory.
[2020-10-24 19:50:27                    main:455]       :       INFO    :       Creating Model
[2020-10-24 19:50:27                    main:468]       :       INFO    :       Preparing config file
[2020-10-24 19:50:27                    main:480]       :       INFO    :       Creating new config file
[2020-10-24 19:50:29                    main:520]       :       INFO    :       Loading DataLoader into Memory
[2020-10-24 19:50:29                    main:523]       :       INFO    :       Starting Training
[2020-10-24 19:50:43                    main:535]       :       INFO    :       Epoch 1 | loss: 0.0660976 | val_loss: 0.0658724 | Time: 13927.1 ms
[2020-10-24 19:50:56                    main:535]       :       INFO    :       Epoch 2 | loss: 0.0646902 | val_loss: 0.0627847 | Time: 13075.8 ms
[2020-10-24 19:51:09                    main:535]       :       INFO    :       Epoch 3 | loss: 0.0610156 | val_loss: 0.0593617 | Time: 12932.5 ms
[2020-10-24 19:51:22                    main:535]       :       INFO    :       Epoch 4 | loss: 0.0580832 | val_loss: 0.0569896 | Time: 13173.9 ms
[2020-10-24 19:51:35                    main:535]       :       INFO    :       Epoch 5 | loss: 0.0561381 | val_loss: 0.055336 | Time: 12685.3 ms
[2020-10-24 19:51:48                    main:535]       :       INFO    :       Epoch 6 | loss: 0.0546499 | val_loss: 0.0539673 | Time: 13349.8 ms
[2020-10-24 19:52:01                    main:535]       :       INFO    :       Epoch 7 | loss: 0.0533704 | val_loss: 0.0527103 | Time: 13045.8 ms
.....

[/code]

For tiny computations, please add Intel GPUs first.
If Vega 54 is getting a very small speed boost, it's probably not worth investing in on big GPUs, like the RTX 3000 series.
There are plenty of Linux and Windows PCs that only have The Collatz (and maybe Einstein) on Intel IGPs.
Intel IGPs should be the right GPU for anything of 60% improvement or less.
If you're talking about comparing CPU vs GPU, a 2080 is about 200x faster than most quad core CPUs, but those aren't the ones you should focus on.
Intel 11th gen CPUs have pretty powerful IGPs (1Tflops and more).
But for most, like Celerons, they can handle 100% extra load (200Gflops CPU and 200Gflops IGP).

For big GPUs they usually run multiple WUs in parallel.
It'll depend how much Double Precision calculations you have to do.
Sometimes hogging a powerful GPU, because MLC maxes out a GPU's DP, isn't the best solution.
I always found 1 CPU + GPU core (CPU for 32 and 64 bit computations, GPU for 32bit and lower) is optimal. Especially for small WUs on a small IGP.

Hi everyone,
I recently installed Boinc on a raspberry pi4 and was pleased to see it was contributing spare cycles to projects. Then I saw a Nvidia Jetson TK1 new on ebay and thought that might be a useful device to Boinc with..given its parallel processing capabilities. However I see the arm architecture isnt recognised and errors are reported. I'm writing this post on the system now, though Im certain Cuda isn't active yet, I would be grateful if anyone knows of any active forum or links relative to enabling this type of embedded board for project use.
I am reading through https://boinc.berkeley.edu/wiki/Client_configuration#Options
just looking for additional help.
Many Thanks, Risque.

Nvidia developerboards are nothing but low power ARM cores (A55, A53, A52 or lower) with a low power GPU that sits somewhere around a GT730.
They're not the right boards to do compute calculations with.
Especially their CPU compute numbers are very, very slow!
They're expensive, and for the $99 a jetson nano costs, you'll probably rather buy a GT1030 (that's about twice to trice as fast).

The Pi4B uses A72 cores, they're slightly better for compute loads.

Just to let you know there is progress. This is with a rand_automata/DS3 datafile. Want to get it into testing tonight or tomorrow.

Awesome! My GPUs are eager to test.

---

Reno, NV
Team: SETI.USA