This Week in [email protected]
Notes for Oct 12 2020
A weekly summary of news and notes for [email protected]

Summary
Happy 100 days, [email protected]! On October 8th, [email protected] passed 100 days old as a project, and early adopter badges were added to everyone with credit at that point. Thanks again for all your support.

Beyond that, we transitioned to our new server, which is quite a step up from the old one, and puts us in good shape for Dataset 3+4 validation and beyond. Read on for more information.

News:

• Dataset 3 WUs processing going fantastically, but a few stragglers remain before we hit the first milestone (100x100). New WUs have been released towards the next milestone (100x1000). We'll continue to trickle them out about 500 at a time, and update the scoreboard to keep track.
  
• We studied the the runtimes of dataset 3 WUs, and decided to increase the credit awarded for all new WUs. There are still old WUs in the pipeline that have the old credit value, and will take a while for those time finish, but new ones created will have more credit.
  
• We're testing a new, longer set of Dataset1+2 WUs in the mldstest application, with a longer runtime and a corresponding change in credit awarded. Datasets 1+2 continue also to make progress in parallel with Dataset 3.
  
• New server up and running. Thanks for your patience. There may need to be another quick downtime (15 minutes) to swap a disk around insice the machine, but otherwise its been running fine since Saturday afternoon.
  
• Old credit should be counted towards badges now!
  
• This doesn't effect anyone here, but the FLOPS estimate for dataset 3 is grossly underspecified (which is why the "GFLOPS" estimate below is so much lower than usual). We'll fix that in future WUs. It doesn't matter for users, but it helps convince other researchers if I can quote a (more) accurate GFLOPS estimate..
  
• Next up (in vague priority order): GPU support, Dataset 4, Dataset Release Paper writing, OSX support.
  

Project status snapshot:
(note these numbers are approximations)

Tasks
Tasks ready to send 41238
Tasks in progress 21350
Users
With credit 910
Registered in past 24 hours 47
Hosts
With recent credit 2029
Registered in past 24 hours 58
Current GigaFLOPS 24544.93

Dataset 1 and 2 progress:

SingleDirectMachine      10002/10004
EightBitMachine          10001/10006
SingleInvertMachine      10001/10003
SimpleXORMachine         10000/10002
ParityMachine              804/10005

ParityModified             218/10005
EightBitModified          6221/10006
SimpleXORModified        10005/10005
SingleDirectModified     10004/10004
SingleInvertModified     10002/10002 

Dataset 3 progress:

Overall (so far): 17197/30112
Milestone 1, 100x100:  9975/10000
Milestone 2, 100x1000: 17197/100000
Milestone 3: 100x10000: 17197/1000000

Last week's TWIM Notes: Oct 5 2020

Thanks again to all our volunteers!

-- The [email protected] Admins

Thanks for the update! Great that the first big milestone on dataset will soon be reached.

Will those longer ds 1/2 WUs just be the same network but trained for more epochs to assure convergence to a stable network structure?

As always, love that you are so transparent with us and the glimpse on the future pipeline as well! Keep up the good work :)

Thank you for the update.

Updates like this are the only thing that make me feel sane anymore, when i can delve into this stuff, i feel at peace, somewhat whole.

It's a pleasant distraction from 2020.

Thank You again, as bozz stated, the transparency is something I also love!

Take Luck and Care!

Been a loooong week, so apologies for the delay in response.

The current length (in epochs) of DS 1/2 WUs was a compromise to work around a BOINC limitation. BOINC wants WUs to be a fixed amount of "effort", and some machines converge really quickly, while Parity and EightBit WUs take hundreds or thousands of epochs to finish. Currently, we run for 128 epochs, which wass a good compromise. However, all the "converge quickly" networks are finished. Meaning each currently running DS 1+2 WU is only making 128 epochs at a time.. then waiting for multiple results, and then waiting to be downloaded again.

If we up the minimum number of epochs for the WU, it'll cut down on the time the WUs are waiting on the server, and more time computing on your machines... since we no longer have to worry about wasted epochs for the easier WUs. That's the theory anyway. I released a few longer WUs on mldstest, but haven't been able to look at the results to see if they worked (and by worked, I mean finished, gave the correct credit, and were properly validated.. I don't expect them to converge yet!).

In other good news, dataset 3 milestone 1 (100x100) is complete! Now time to collect the dataset, write up some preliminary results, and make the dataset available. I know everyone wants a timeline for that, but it'll probably be a few weeks to prepare. In the meantime, results continue towards milestone 2 (1000x100)! (I'll update the frontpage scoreboard this weekend).