In case some of you have noticed, we're aware the validation queue is backing up a bit. This is a good problem to have, and we're aware of it. The short answer is that Dataset 3 WUs take 10 times as long to validate than those for Dataset 1 and 2, and even after adding a second validator process, the server is straining to keep up with the extra load from how quickly all of you are all ripping through Dataset 3 WUs. As I said, a good problem to have!

Long term, this is why we need a new server, which should be arriving at the university next week, which has 3x the number of cores, twice the RAM, and twice the speed. So this whole issue should go away soon.

Short term there are still some things on our end we can do to alleviate the backlog. There are definitely some inefficiencies that can be addressed in the validation process, because they weren't an issue until dataset 3. Also, we're interleaving more dataset 1 and 2 WUs back into the work dispatcher, since they validate much quicker, that should also relieve some of the pressure.

So, keep crunching, we're handling it on our side. 33% through the first milestone dataset 3 in less than 5 days. Wow. Thanks all!

Backlog cleared. Several optimizations put in to cut down on validation time and we seem to be back to normal.