What is MLC@Home?

MLC@Home is a distributed computing project dedicated to understanding and interpreting complex machine learning models, with an emphasis on neural networks. It uses the BOINC distributed computing platform. You can find more information on our main website here: https://www.mlcathome.org.

Neural Networks have fuelled a machine learning revolution over the past decade that has led to machines accomplishing amazingly complex tasks. However, these models are largly black boxes: we know they work, but they are so complex (up to hundreds of millions of parameters!) that we struggle to understand the limits of such systems. Yet understanding networks becomes extremely important as networks are deployed in safety critical fields, like medicine and autonomous vehicles.

MLC@Home provides an open, collaborative platform for researchers studying machine learning comprehension. It allows us to train thousands of networks in parallel, with tightly controlled inputs, hyperparameters, and network structures. We use this to gain insights into these complex models.

We ask for volunteers to donate some of their background computing time to help us continue our research. We use the time-tested BOINC distributed computing infrastructure — the same infrastructure that powers SETI@home's search for alien life, and Rosetta@home's search for effective medications. BOINC is fun — you get credit for each bit of compute that you do, with leaderboards and milestones. All while helping further open research. Please follow the link below to join, and happy crunching!

Join MLC@Home

Already joined? Log in.


Current WU issues, working on a fix
A short note that we're aware of the issue with WUs coming from the CPU and TEST work queues (the GPU queue appears fine at the moment). This is due to a server-side issue related to some cleanup and upgrades I've been doing behind the scenes that appears to have gone haywire, and since it initially seemed to be working I didn't catch it immediately, leading to compounding the issues.

This is unacceptable and I apologize. While this had been tested, this failure mode was unforeseen. You rely on us to keep things running smoothly, and I failed you.

Over the next 24 hours we'll be sending out cancellations for the corrupted WUs, and may stop/start the service a few times while we try to clean things up. Please bear with us and thanks for your patience.

I stress : no data was lost, and the nature of the failure is to fail-fast on the client, so there is little to no wasted computer cycles.

Thanks again, and we'll do better in the future.
14 Oct 2021, 20:18:19 UTC · Discuss

Updated CPU client 9.9x release and issues
Earlier this week, we released the latest v9.90 CPU client after almost 3 weeks
of testing. While it initially seemed to be working fine, a number of errors started
accumulating over the last 24 hours. We've identified a server configuration issue
and believe it is now fixed as of 6AM UTC today. The server was generating
invalid WUs for the MLDS queue. We've cancelled all of the problematic WUs
and are adding new ones to the main queue. The GPU clients and MLDSTEST
queue remained unaffected.

v9.90 is an important release for MLDS, as it contains support for CNNs and Dense
feed forward network types needed for DS4. Highlights include:

    - Statically linked binary for Linux (no more AppImage)
    - DS4 support! (CNN and Dense networks)
    - Better NaN handling
    - Update to libTorch 1.9
    - Wrapper instead of BOINC native API

Our apologies for the bumpy rollout, but if you've had issues with computation errors
earlier this week, please retry at your earliest convenience, and post any issues
to our forums or our Discord server.
28 Aug 2021, 6:13:16 UTC · Discuss

[TMIM Notes] Aug 6 2021
This Month in MLC@Home
Notes for Aug 6 2021
A monthly summary of news and notes for MLC@Home

Another month of good progress on MLC! First, this past month saw the completion of DS3! You have trained over 1,000,000 neural networks for DS3, which is a huge accomplishment. We're continuing to bundle and evaluate the dataset, so look for a complete public release shortly.

We also spend some time on the backend preparing for DS4. We've updated the website to show DS4 progress, but haven't sent any DS4 WUs yet. Instead we spent the bulk of the month trying to get the new client to work under Windows, which hasn't been going well. We spent a good 2.5 weeks trying to get pytorch and the client to compile (and run) statically on windows. Even though it now compiles, the client crashes when running. So last week we switched back to linking dynamically, and want to get an updated windows client out this weekend. The Linux/CPU version of the new client appears to be performing fantastically, so thanks to everyone who ran WUs from the "mldstest" queue tested!

DS4 WUs are incompatible with the older client, so we'll only release DS4 WUs as the new (v9.9x) client become available for each platform. This means CPUs first. GPUs will continue to work on finishing up DS1 and DS2.

Speaking of DS1/DS2, we're approaching the end of DS1 with only a few more weeks to go. and when we complete those networks we'll switch to DS2 to finish those up as well.

So, lots of movement this month behind the scenes, and great progress on the existing datasets. If *any* windows developers would like to help us out getting the new windows CPU client out the door, please contact us directly, we could use the help.

Other News

  • GPU and CPU queues are stacked with DS1/DS2 WUs until those completes and/or until DS4 is ready. The CPU queue will transition to DS4 first, as we assume GPU builds will be even more of a headache than the CPU ones have been.
  • Because existing WUs are incompatible with the new client, we've been keeping the CPU and GPU queues a little less full than we have in the past, because when the new client comes out, we don't want to have to cancel a bunch of existing WUs to be replaced with new ones. Unfortunately, since the updated windows CPU client is taking a longer than planned, we've run out of WUs a few times in the past month. We're trying to stay on top of this, and automating the process to keep them from running completely dry in the future.
  • We have a new developer who has joined the team, who is working on more graceful handling of NaN errors, which has been an issue for a long time. If that's ready before the windows CPU client is ready, then that fix will be in the next release. You can see more in the #devel channel on Discord, or the issue on gitlab.
  • Reminder: the MLC client is open source, and has an issues list at gitlab. If you're a programmer or data scientist and want to help, feel free to look over the issues and submit a pull request.

Project status snapshot:
(note these numbers are approximations)

Last month's TMIM Notes: Jul 1 2021

Thanks again to all our volunteers!

-- The MLC@Home Admins(s)
Homepage: https://www.mlcathome.org/
Discord invite: https://discord.gg/BdE4PGpX2y
Twitter: @MLCHome2
6 Aug 2021, 20:48:04 UTC · Discuss

[TMIM Notes] July 1 2021 --- Celebrating 1 year of MLC@Home!
This Month in MLC@Home
Notes for July 1 2021
A monthly summary of news and notes for MLC@Home

Happy first birthday to MLC@Home! This project went live on July 1, 2020, and caught on pretty quickly in the BOINC community. We've remained focused on our goal, which is breaking open the black box of neural networks to explain why they make the choices they do. This is so important as machine learning permeates more and more of our everyday life; from autonomous cars, to banking decisions, and medical diagnoses. We need research to understand how to keep bias out of these systems.

We are also the first, and to date only, public machine learning focused BOINC project. This means that while we could leverage the BOINC framework for job management, we have to build most of the ML client infrastructure from the ground up. This hasn't always been smooth, but we've accomplished so much in the past year regardless.

In the past year, we have:

  • Received contributions from over 2500+ volunteers and 9200+ hosts
  • Processed over 3.4 million BOINC workunits
  • Trained over 1.1 million neural networks for analysis over 3 different datasets, the largest datasets of their kind
  • Generated over 4.3TB of data for analysis
  • Published one academic paper (more coming..)
  • Presented at the 2021 BOINC Workshop
  • Released 47 client versions targeting 3 different CPU architectures, 2 GPU architectures, and multiple versions of Windows and Liunx.
  • Outgrew the initial server within the first few months!

I'm overwhelmed by our community and what we've accomplished together. We've already shown that networks trained with the same data cluster together in weight space, despite the randomness associated with neural network training. We've also shown we can use this clustering to detect networks trained with poisoned data versus clean data, a significant finding in the field.

But there's still soo much more to do! So while we want to acknowledge and celebrate what we've jointly accomplished so far, let's also look forward and set some loose goals for the next year of MLC@Home:

  • MLDS will continue near term!
    DS4 is (almost) ready and expands the dataset to include CNN network types as well as RNNs used in DS1-3. DS5 will likely vary the shape and size of each network slightly to see if clustering still happens when shape is varies. Future MLDS work beyond DS5 is TBD, but we expect there to be plenty DS4/DS5 WUs for many months to come. We expect to update the paper with the latest runs over the next month.

  • We'd like to expand beyond MLDS!
    We are the first project to do ML on a BOINC-sized scale. We would like to expand to supporting other areas of research, and want to commit to bringing at least one other ML project online within the next year. Please contact us if you are a researcher who is interested in working with the platform!

  • We need to improve the technical side of the project
    From the client supporting AMD GPUs and OSX to optimizing utilization of graphics cards to a better validation process for WUs, there's a laundry list of technical issues we'd like to address, and have not done so effectively in the past three months. We're also hitting some corner-cases of the BOINC software stack that are tricky to work around. If you are a developer and want to help, we'd welcome the support.

  • We'd like to improve outreach
    To get more people involved, we'd like to produce a few short videos about the project, what we've found and how others can help. These should be short, easily accessible, and easy to share. We'd like to produce at least one of these within the next 6 months.

These are loose goals but should give you an idea where we're concentrating our efforts for the next year. If you have further insights, please share them below or on Discord.

Thanks again for supporting MLC@Home, and here to many more years of successful, important research in an important field.

Other News

  • DS3 is all but complete (just a last few 130+ trickling in!). I consider DS3 to be the most important dataset and can't wait to run our analysis on the whole thing!
  • From now on we'll be blasting DS1 (then DS2) WUs into both the GPU and CPU queues until that completes and/or until DS4 is ready. We'll try to get those over the hump ASAP.
  • Some fun news! MLC Discord user Tankbuster has updated our banner graphic! See the updated banner on project and home pages!
  • Even more exciting, Tankbuster built a prototype graphics app for MLC@Home! You can see mockups and videos and follow the discussion at the MLC Discord server (link at the bottom). Screenshot:
  • Reminder: the MLC client is open source, and has an issues list at gitlab. If you're a programmer or data scientist and want to help, feel free to look over the issues and submit a pull request.

Project status snapshot:
(note these numbers are approximations)

Last month's TMIM Notes: Jun 8 2021

Thanks again to all our volunteers!

-- The MLC@Home Admins(s)
Homepage: https://www.mlcathome.org/
Discord invite: https://discord.gg/BdE4PGpX2y
Twitter: @MLCHome2
2 Jul 2021, 1:40:09 UTC · Discuss

[TMIM Notes] Jun 8 2021 posted
MLC@Home has posted the Jun 8 2021 edition of its monthly "This Month In MLC@Home" newsletter!
A monthly update including the new client with DS4 support, a note on disk space on the server, and a mvoe to monthly updates instead of weekly ones moving forward.

Read the update and join the discussion here.
9 Jun 2021, 4:22:49 UTC · Discuss

... more

News is available as an RSS feed   RSS

©2021 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)