Questions and Answers :
Issue Discussion :
Out of work for CPU
Message board moderation
| Author | Message |
|---|---|
|
Send message Joined: 8 Jun 21 Posts: 4 Credit: 1,655,207 RAC: 0 |
As the title reads. Just checked the server status and it seems that we are out of work for the CPU app so my CPU's currently idling. Will a new dataset be posted soon? |
|
Send message Joined: 9 Jul 20 Posts: 142 Credit: 11,536,204 RAC: 3 |
The last news (TMIM Notes June 8, 2021) did read that "DS1/DS2 continues along as a slow pace. It will continue in the background until we have 10,000 samples of each." As we are now nearing completion of the 5,000 samples mark, I guess we'll soon see the jump back to 50% completion as we'll move on to train the second half of networks to get to 10,000 samples. This website is also a great place to occasionally check on the progress of the various experiments. I could be wrong though... |
|
Send message Joined: 8 Jun 21 Posts: 4 Credit: 1,655,207 RAC: 0 |
Actually you can check the tasks remaining for each application from this page: https://www.mlcathome.org/mlcathome/server_status.php At the bottom of the page, there's a section that shows tasks by application. The column "Unsent" is the tasks that is left for distribution. Currently the CPU unsent tasks are 0 so we're out of work for CPU for the time being. Guess I'll just put my CPU to other projects in the meantime. |
|
Send message Joined: 9 Jul 20 Posts: 142 Credit: 11,536,204 RAC: 3 |
Yeah, I know that. But obtained trained network samples is not the same as the number of work units AFAIK. That said, the admin's latest news post stated the intention of moving forward with the DS1/2 experiments to reach the next defined milestone of 10k samples. That's all. I am just stating what might be in the near future while you were referring to the present as represented by the server stats. Guess I wasn't clear about that. Cheers |
|
Send message Joined: 8 Jun 21 Posts: 4 Credit: 1,655,207 RAC: 0 |
Ah gotcha. No worries man. Yeah was referring to the present because I thought it might be a system glitch but we actually had run out of work for CPU (good thing btw!) and wanted to let the admin know about it. Maybe there might be more work to upload or something. |
|
Send message Joined: 30 Aug 20 Posts: 25 Credit: 47,025,926 RAC: 0 |
From what I understand DS3 is what is currently running on CPU; DS1 and 2 are currently GPU tasks. The latest news article mentions DS4 will be ready any day now. I am simulating some black holes in the meantime. |
|
Send message Joined: 9 Jul 20 Posts: 142 Credit: 11,536,204 RAC: 3 |
What about the rand WU. Are these networks from DS3? Anyway, hopefully we’ll see the launch of DS4 Witze the new client software soon. |
|
Send message Joined: 30 Jun 20 Posts: 462 Credit: 21,406,548 RAC: 0 |
Yep, we're run out of initial DS3 WUs the CPU queue. This is great news! Unfortunately, DS4 isn't quite ready yet, although I think we're only a day or two away. In the mean time, later today, I'll send out some DS1/DS2 WUs from the GPU queue to the CPU one. Expect more to start flowing within the next 24 hours. Sorry for the gap, this is completely on me. |
|
Send message Joined: 8 Jun 21 Posts: 4 Credit: 1,655,207 RAC: 0 |
Forgot to check back on the thread, I let my CPU simulate some proteins while waiting in the meantime but it seems that the CPU queues are now filled up again so it's back to machine learning again! Thanks for the hard work @pianoman! Don't apologise for not filling up the queue, I'm just glad there's something my CPU is good for! |
|
Send message Joined: 23 Sep 20 Posts: 24 Credit: 15,318,198 RAC: 1,992 |
Yep, we're run out of initial DS3 WUs the CPU queue. This is great news! Can you do this again as the CPU queue is dry? |
|
Send message Joined: 30 Jun 20 Posts: 462 Credit: 21,406,548 RAC: 0 |
pumped out another 8K WUs out last night, and they're all consumed again. I'm going to push out some more, and look into a way to automate this until DS4 is ready (or rather, until the windows CPU client that supports DS4 is ready). |
|
Send message Joined: 23 Sep 20 Posts: 24 Credit: 15,318,198 RAC: 1,992 |
pumped out another 8K WUs out last night, and they're all consumed again. I'm going to push out some more, and look into a way to automate this until DS4 is ready (or rather, until the windows CPU client that supports DS4 is ready). Aaaaand…the well has run dry already |
|
Send message Joined: 7 Jul 20 Posts: 23 Credit: 39,708,780 RAC: 358 |
Appears to be another 29000 ready to send, up from 3000 or so last night. Very nicely done. |
|
Send message Joined: 17 Aug 21 Posts: 1 Credit: 544,180 RAC: 0 |
Just checked the server status and it seems that we are out of work for the CPU app |
|
Send message Joined: 30 Jun 20 Posts: 462 Credit: 21,406,548 RAC: 0 |
Before anyone complains, yes, we're out of work for the CPU queue at the moment, but there's good reason this time! We're updating to a new app tonight (what's been in testing) that is incompatible with the current WUs in-flight. So tongiht we'll be aborting all the outstanding WUs on the CPU queue, and re-issuing them to be compatible with the new client. It should all be resolved by tomorrow, jst be aware you may get some aborted WUs and that's unavoidable during this transition. Note the GPU clients aren't ready yet for the new WU type (they may or may not take long, the linux side especially changed the way it links the code, so it may take a bit to iron that out w/ cuda and rocm), but for now the GPUs will continue to crunch with the old WUs to finish up the work there. Thanks for your patience and understanding |
|
Send message Joined: 9 Jul 20 Posts: 142 Credit: 11,536,204 RAC: 3 |
Thanks for the quick update. Was wondering already why I couldn't receive any CPU work on my Win machine. I'll start running my test WUs :) |
|
Send message Joined: 23 Sep 20 Posts: 24 Credit: 15,318,198 RAC: 1,992 |
I hope the latest issues have been satisfactorily resolved, the CPU queue has run dry. |
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)