Hi,
i have 6 invalid wu's on different PC's. The common thing is: all have all or some entries in stderr of loss or val_loss of nan.
One example is https://www.mlcathome.org/mlcathome/result.php?resultid=3189474
Is the value 'nan' the reason for the invalid wu? If yes, why does it make sense to finish calculation? Would it be possible to simply restart the wu?

nan is probably the output code for "not a number" such as the value resulting from a square root of negative number. I don't think it is a problem with your machines, but more likely a problem with the simulation parameters causing it. Restarting the WU would more than likely just result in repeating the error.

The thing is, that these wu's validate on other PC's. So a reinit could propably help.
The other idea is: if the first nan occurs, does it make sense to complete the wu wasting GPU-time? If a reinit is no option, the wu could be stopped with an errorcode other than 0 (as my failed wu's do).
If this is a more common problem it might be worth to look into the code. But i did not find similar posts. A windows-only problem?

The number of failed wu's (Bestätigungsfehler) is now up to 9. At least this wu also failed on another PC, a windows server. https://www.mlcathome.org/mlcathome/show_host_detail.php?hostid=5082
This one validated on another win10 pc https://www.mlcathome.org/mlcathome/workunit.php?wuid=1470598

NaNs mean that the floating point value is either too small or too large to be represented by the floating point machinery, or don't exist (such as division by zero). Getting NaNs is sadly not all that unusual when training neural networks, especially ones that are difficult to learn like ParityMachine. Training a neural network means searching a multi-dimensional hyperplane for a global minima (the point with the minimal error given the training set). The search is stochastic in nature.. sometimes you just get unlucky and get stuck in a search path that spirals out of control. That's a vastly simplified description of a complex topic but I hope it at least makes some sense.

However, the presence of NaNs in your result should not cause a validation failure. it's not your fault the algorithm happened to search down an un-fruitful path in the error plane, so you should still get credit for the computation. Instead, we simply don't generate a follow-on WU to continue searching down that path. I'll need to check the DB later tonight to see why this particular WU failed validation, but the nan isn't the reason.

Ideally, we would stop computation at the first sign of a NaN, but due to a quirk of the way boinc works, ending WUs early causes more harm than good (it's a long story).

NaNs are relatively rare overall, so its not worth the headache to stop early. However, it may be worth re-evaluating that policy now since a) We're sending a lot of extra Parity WUs to try and get the number of completed networks up quickly, and b) the new GPU-based WUs are running 8x the number of epochs as the CPU ones.. so there's a potential if a NaN is encountered early there's a bit more of a penalty to the science.

Thanks for reporting this and let us think about it for a bit and see if we can tweak the algorithm going forward.

Thank you for that explanation.
It's a young project and failurs are an option, otherwise progress would be too slow. I accept that. Happy that you are aware of these facts. I have seen projects where the admins were not aware (for at least half a year) that the formula they used was wrong.

As a follow-up here, I have noticed some evidence in the log that NaN's may be causing a small subset of WUs to be marked as invalid. Not explicitly, but because some code to translate the results to json are causing the json parser to fail to parse the file (it expects a number, non NaNs) is causing a failure in another part of the validation process. I'll work on fixing that over the next few days.

Murphy said: If something can go wrong, it will. Even if there is only a little chance.

Thank you for keeping us informed!

As a follow-up here, I have noticed some evidence in the log that NaN's may be causing a small subset of WUs to be marked as invalid. Not explicitly, but because some code to translate the results to json are causing the json parser to fail to parse the file (it expects a number, non NaNs) is causing a failure in another part of the validation process. I'll work on fixing that over the next few days.

Is this line can help?

terminate called after throwing an instance of 'nlohmann::detail::type_error'
  what():  [json.exception.type_error.302] type must be number, but is null