Out of work for CPU

Questions and Answers : Issue Discussion : Out of work for CPU
Message board moderation

To post messages, you must log in.

AuthorMessage
Geralt

Send message
Joined: 8 Jun 21
Posts: 4
Credit: 1,655,207
RAC: 0
Message 1230 - Posted: 15 Jun 2021, 12:30:38 UTC

As the title reads. Just checked the server status and it seems that we are out of work for the CPU app so my CPU's currently idling. Will a new dataset be posted soon?
ID: 1230 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
bozz4science

Send message
Joined: 9 Jul 20
Posts: 142
Credit: 11,536,204
RAC: 3
Message 1231 - Posted: 15 Jun 2021, 13:32:54 UTC - in response to Message 1230.  
Last modified: 15 Jun 2021, 13:33:45 UTC

The last news (TMIM Notes June 8, 2021) did read that
"DS1/DS2 continues along as a slow pace. It will continue in the background until we have 10,000 samples of each."

As we are now nearing completion of the 5,000 samples mark, I guess we'll soon see the jump back to 50% completion as we'll move on to train the second half of networks to get to 10,000 samples. This website is also a great place to occasionally check on the progress of the various experiments. I could be wrong though...
ID: 1231 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Geralt

Send message
Joined: 8 Jun 21
Posts: 4
Credit: 1,655,207
RAC: 0
Message 1232 - Posted: 15 Jun 2021, 14:13:28 UTC

Actually you can check the tasks remaining for each application from this page: https://www.mlcathome.org/mlcathome/server_status.php

At the bottom of the page, there's a section that shows tasks by application. The column "Unsent" is the tasks that is left for distribution. Currently the CPU unsent tasks are 0 so we're out of work for CPU for the time being. Guess I'll just put my CPU to other projects in the meantime.
ID: 1232 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
bozz4science

Send message
Joined: 9 Jul 20
Posts: 142
Credit: 11,536,204
RAC: 3
Message 1233 - Posted: 15 Jun 2021, 15:02:53 UTC - in response to Message 1232.  

Yeah, I know that. But obtained trained network samples is not the same as the number of work units AFAIK. That said, the admin's latest news post stated the intention of moving forward with the DS1/2 experiments to reach the next defined milestone of 10k samples. That's all. I am just stating what might be in the near future while you were referring to the present as represented by the server stats. Guess I wasn't clear about that. Cheers
ID: 1233 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Geralt

Send message
Joined: 8 Jun 21
Posts: 4
Credit: 1,655,207
RAC: 0
Message 1234 - Posted: 15 Jun 2021, 23:48:07 UTC

Ah gotcha. No worries man. Yeah was referring to the present because I thought it might be a system glitch but we actually had run out of work for CPU (good thing btw!) and wanted to let the admin know about it. Maybe there might be more work to upload or something.
ID: 1234 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
W8n4Singularity
Avatar

Send message
Joined: 30 Aug 20
Posts: 25
Credit: 47,025,926
RAC: 0
Message 1235 - Posted: 16 Jun 2021, 6:53:55 UTC

From what I understand DS3 is what is currently running on CPU; DS1 and 2 are currently GPU tasks. The latest news article mentions DS4 will be ready any day now. I am simulating some black holes in the meantime.
ID: 1235 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
bozz4science

Send message
Joined: 9 Jul 20
Posts: 142
Credit: 11,536,204
RAC: 3
Message 1236 - Posted: 16 Jun 2021, 7:00:41 UTC - in response to Message 1235.  

What about the rand WU. Are these networks from DS3?

Anyway, hopefully we’ll see the launch of DS4 Witze the new client software soon.
ID: 1236 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pianoman [MLC@Home Admin]
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Jun 20
Posts: 462
Credit: 21,406,548
RAC: 0
Message 1237 - Posted: 16 Jun 2021, 17:57:06 UTC

Yep, we're run out of initial DS3 WUs the CPU queue. This is great news!

Unfortunately, DS4 isn't quite ready yet, although I think we're only a day or two away. In the mean time, later today, I'll send out some DS1/DS2 WUs from the GPU queue to the CPU one. Expect more to start flowing within the next 24 hours.

Sorry for the gap, this is completely on me.
ID: 1237 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Geralt

Send message
Joined: 8 Jun 21
Posts: 4
Credit: 1,655,207
RAC: 0
Message 1238 - Posted: 18 Jun 2021, 18:16:18 UTC

Forgot to check back on the thread, I let my CPU simulate some proteins while waiting in the meantime but it seems that the CPU queues are now filled up again so it's back to machine learning again! Thanks for the hard work @pianoman! Don't apologise for not filling up the queue, I'm just glad there's something my CPU is good for!
ID: 1238 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
swiftmallard
Avatar

Send message
Joined: 23 Sep 20
Posts: 24
Credit: 15,318,198
RAC: 1,992
Message 1293 - Posted: 28 Jul 2021, 22:26:18 UTC - in response to Message 1237.  

Yep, we're run out of initial DS3 WUs the CPU queue. This is great news!

Unfortunately, DS4 isn't quite ready yet, although I think we're only a day or two away. In the mean time, later today, I'll send out some DS1/DS2 WUs from the GPU queue to the CPU one. Expect more to start flowing within the next 24 hours.

Sorry for the gap, this is completely on me.


Can you do this again as the CPU queue is dry?
ID: 1293 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pianoman [MLC@Home Admin]
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Jun 20
Posts: 462
Credit: 21,406,548
RAC: 0
Message 1299 - Posted: 29 Jul 2021, 21:25:13 UTC - in response to Message 1293.  

pumped out another 8K WUs out last night, and they're all consumed again. I'm going to push out some more, and look into a way to automate this until DS4 is ready (or rather, until the windows CPU client that supports DS4 is ready).
ID: 1299 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
swiftmallard
Avatar

Send message
Joined: 23 Sep 20
Posts: 24
Credit: 15,318,198
RAC: 1,992
Message 1304 - Posted: 2 Aug 2021, 22:32:27 UTC - in response to Message 1299.  

pumped out another 8K WUs out last night, and they're all consumed again. I'm going to push out some more, and look into a way to automate this until DS4 is ready (or rather, until the windows CPU client that supports DS4 is ready).

Aaaaand…the well has run dry already
ID: 1304 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
wolfman1360

Send message
Joined: 7 Jul 20
Posts: 23
Credit: 39,708,780
RAC: 358
Message 1305 - Posted: 3 Aug 2021, 19:04:23 UTC

Appears to be another 29000 ready to send, up from 3000 or so last night.
Very nicely done.
ID: 1305 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rymorea

Send message
Joined: 17 Aug 21
Posts: 1
Credit: 544,180
RAC: 0
Message 1325 - Posted: 17 Aug 2021, 21:16:14 UTC

Just checked the server status and it seems that we are out of work for the CPU app
ID: 1325 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pianoman [MLC@Home Admin]
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Jun 20
Posts: 462
Credit: 21,406,548
RAC: 0
Message 1328 - Posted: 24 Aug 2021, 20:17:46 UTC

Before anyone complains, yes, we're out of work for the CPU queue at the moment, but there's good reason this time!

We're updating to a new app tonight (what's been in testing) that is incompatible with the current WUs in-flight. So tongiht we'll be aborting all the outstanding WUs on the CPU queue, and re-issuing them to be compatible with the new client. It should all be resolved by tomorrow, jst be aware you may get some aborted WUs and that's unavoidable during this transition.

Note the GPU clients aren't ready yet for the new WU type (they may or may not take long, the linux side especially changed the way it links the code, so it may take a bit to iron that out w/ cuda and rocm), but for now the GPUs will continue to crunch with the old WUs to finish up the work there.

Thanks for your patience and understanding
ID: 1328 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
bozz4science

Send message
Joined: 9 Jul 20
Posts: 142
Credit: 11,536,204
RAC: 3
Message 1329 - Posted: 25 Aug 2021, 7:59:34 UTC - in response to Message 1328.  

Thanks for the quick update. Was wondering already why I couldn't receive any CPU work on my Win machine. I'll start running my test WUs :)
ID: 1329 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
swiftmallard
Avatar

Send message
Joined: 23 Sep 20
Posts: 24
Credit: 15,318,198
RAC: 1,992
Message 1391 - Posted: 16 Oct 2021, 15:29:39 UTC

I hope the latest issues have been satisfactorily resolved, the CPU queue has run dry.
ID: 1391 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Questions and Answers : Issue Discussion : Out of work for CPU

©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)