Current WU issues, working on a fix

Message boards : News : Current WU issues, working on a fix
Message board moderation

To post messages, you must log in.

AuthorMessage
pianoman [MLC@Home Admin]
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Jun 20
Posts: 462
Credit: 21,406,548
RAC: 0
Message 1388 - Posted: 14 Oct 2021, 20:18:19 UTC

A short note that we're aware of the issue with WUs coming from the CPU and TEST work queues (the GPU queue appears fine at the moment). This is due to a server-side issue related to some cleanup and upgrades I've been doing behind the scenes that appears to have gone haywire, and since it initially seemed to be working I didn't catch it immediately, leading to compounding the issues.

This is unacceptable and I apologize. While this had been tested, this failure mode was unforeseen. You rely on us to keep things running smoothly, and I failed you.

Over the next 24 hours we'll be sending out cancellations for the corrupted WUs, and may stop/start the service a few times while we try to clean things up. Please bear with us and thanks for your patience.

I stress : no data was lost, and the nature of the failure is to fail-fast on the client, so there is little to no wasted computer cycles.

Thanks again, and we'll do better in the future.
ID: 1388 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
pianoman [MLC@Home Admin]
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Jun 20
Posts: 462
Credit: 21,406,548
RAC: 0
Message 1390 - Posted: 15 Oct 2021, 15:13:21 UTC

New, corrected WUs are starting to trickle out again on the CPU queue.. we're being cautios and monitoring them closely before dumping another big batch, but it looks like we're back to normal.
ID: 1390 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Albert Argilaga, Ph.D.
Avatar

Send message
Joined: 4 Oct 20
Posts: 6
Credit: 386,575
RAC: 0
Message 1393 - Posted: 19 Oct 2021, 0:47:36 UTC - in response to Message 1388.  
Last modified: 19 Oct 2021, 0:48:36 UTC

No worries man. You can even create a badge for people who did try to run a corrupted WU
ID: 1393 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Eindjy

Send message
Joined: 23 Oct 21
Posts: 1
Credit: 5,460
RAC: 0
Message 1396 - Posted: 23 Oct 2021, 13:53:11 UTC

is it fine now , coz i run 3 tasks
ID: 1396 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : News : Current WU issues, working on a fix

©2024 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)