Task 6641428
| Name | ParityMachine-1627979002-26639-2_0 |
| Workunit | 4391598 |
| Created | 15 Aug 2021, 7:05:37 UTC |
| Sent | 21 Aug 2021, 6:56:19 UTC |
| Report deadline | 28 Aug 2021, 6:56:19 UTC |
| Received | 21 Aug 2021, 19:06:02 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Aborted by user |
| Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
| Computer ID | 14471 |
| Run time | 2 hours 48 min 31 sec |
| CPU time | 2 hours 45 min 37 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 18,425.66 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 2.81 GB |
| Peak swap size | 4.70 GB |
| Peak disk usage | 1.53 GB |
Stderr output
<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3090)
[2021-08-21 10:15:36 main:435] : INFO : Set logging level to 1
[2021-08-21 10:15:36 main:441] : INFO : Running in BOINC Client mode
[2021-08-21 10:15:36 main:444] : INFO : Resolving all filenames
[2021-08-21 10:15:36 main:452] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 (exists = 1)
[2021-08-21 10:15:36 main:452] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1627979002-26639-2_0_r726961398_1 (exists = 0)
[2021-08-21 10:15:36 main:452] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1627979002-26639-2_0_r726961398_0 (exists = 0)
[2021-08-21 10:15:36 main:452] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1627979002-26639-2 (exists = 1)
[2021-08-21 10:15:36 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2021-08-21 10:15:36 main:472] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5
[2021-08-21 10:15:36 main:474] : INFO : Configuration:
[2021-08-21 10:15:36 main:475] : INFO : Model type: GRU
[2021-08-21 10:15:36 main:476] : INFO : Validation Loss Threshold: 0.0001
[2021-08-21 10:15:36 main:477] : INFO : Max Epochs: 2048
[2021-08-21 10:15:36 main:478] : INFO : Batch Size: 128
[2021-08-21 10:15:36 main:479] : INFO : Learning Rate: 0.01
[2021-08-21 10:15:36 main:480] : INFO : Patience: 10
[2021-08-21 10:15:36 main:481] : INFO : Hidden Width: 12
[2021-08-21 10:15:36 main:482] : INFO : # Recurrent Layers: 4
[2021-08-21 10:15:36 main:483] : INFO : # Backend Layers: 4
[2021-08-21 10:15:36 main:484] : INFO : # Threads: 1
[2021-08-21 10:15:36 main:486] : INFO : Preparing Dataset
[2021-08-21 10:15:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-08-21 10:15:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-08-21 10:39:55 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2021-08-21 10:39:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-08-21 10:39:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-08-21 10:39:55 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2021-08-21 10:39:55 main:494] : INFO : Creating Model
[2021-08-21 10:39:55 main:507] : INFO : Preparing config file
[2021-08-21 10:39:55 main:519] : INFO : Creating new config file
[2021-08-21 10:39:55 main:538] : INFO : This is a continuation WU, loading previous network
[2021-08-21 10:58:50 main:559] : INFO : Loading DataLoader into Memory
[2021-08-21 10:58:51 main:562] : INFO : Starting Training
[2021-08-21 11:00:24 main:574] : INFO : Epoch 1 | loss: 0.031177 | val_loss: 0.0311642 | Time: 93416.4 ms
[2021-08-21 11:00:29 main:574] : INFO : Epoch 2 | loss: 0.0311486 | val_loss: 0.0311609 | Time: 4791.1 ms
[2021-08-21 11:00:34 main:574] : INFO : Epoch 3 | loss: 0.0311482 | val_loss: 0.0311624 | Time: 4776.36 ms
[2021-08-21 11:00:38 main:574] : INFO : Epoch 4 | loss: 0.0311455 | val_loss: 0.0311643 | Time: 4784.42 ms
[2021-08-21 11:00:43 main:574] : INFO : Epoch 5 | loss: 0.0311436 | val_loss: 0.0311612 | Time: 4773.72 ms
[2021-08-21 11:00:48 main:574] : INFO : Epoch 6 | loss: 0.0311434 | val_loss: 0.0311573 | Time: 4824.85 ms
[2021-08-21 11:00:53 main:574] : INFO : Epoch 7 | loss: 0.0311448 | val_loss: 0.0311591 | Time: 4819.28 ms
[2021-08-21 11:00:58 main:574] : INFO : Epoch 8 | loss: 0.0311434 | val_loss: 0.031163 | Time: 4804.05 ms
[2021-08-21 11:01:03 main:574] : INFO : Epoch 9 | loss: 0.0311424 | val_loss: 0.0311571 | Time: 4795.08 ms
[2021-08-21 11:01:07 main:574] : INFO : Epoch 10 | loss: 0.0311421 | val_loss: 0.0311586 | Time: 4820.84 ms
[2021-08-21 11:01:12 main:574] : INFO : Epoch 11 | loss: 0.031142 | val_loss: 0.0311575 | Time: 4824.1 ms
[2021-08-21 11:01:17 main:574] : INFO : Epoch 12 | loss: 0.0311427 | val_loss: 0.0311556 | Time: 4788.73 ms
[2021-08-21 11:01:22 main:574] : INFO : Epoch 13 | loss: 0.0311455 | val_loss: 0.0311552 | Time: 4809.57 ms
[2021-08-21 11:01:27 main:574] : INFO : Epoch 14 | loss: 0.0311455 | val_loss: 0.0311636 | Time: 4784.72 ms
[2021-08-21 11:01:32 main:574] : INFO : Epoch 15 | loss: 0.0311452 | val_loss: 0.0311616 | Time: 4813.12 ms
[2021-08-21 11:01:37 main:574] : INFO : Epoch 16 | loss: 0.0311463 | val_loss: 0.0311598 | Time: 4810.92 ms
[2021-08-21 11:01:41 main:574] : INFO : Epoch 17 | loss: 0.0311435 | val_loss: 0.031159 | Time: 4795.8 ms
[2021-08-21 11:01:46 main:574] : INFO : Epoch 18 | loss: 0.0311408 | val_loss: 0.0311594 | Time: 4801.16 ms
[2021-08-21 11:01:51 main:574] : INFO : Epoch 19 | loss: 0.0311408 | val_loss: 0.0311584 | Time: 4799.24 ms
[2021-08-21 11:01:56 main:574] : INFO : Epoch 20 | loss: 0.0311425 | val_loss: 0.031157 | Time: 4771.44 ms
[2021-08-21 11:02:01 main:574] : INFO : Epoch 21 | loss: 0.0311399 | val_loss: 0.0311554 | Time: 4806.48 ms
[2021-08-21 11:02:06 main:574] : INFO : Epoch 22 | loss: 0.0311402 | val_loss: 0.0311606 | Time: 4805.64 ms
[2021-08-21 11:02:10 main:574] : INFO : Epoch 23 | loss: 0.031143 | val_loss: 0.0311581 | Time: 4803.33 ms
[2021-08-21 11:02:15 main:574] : INFO : Epoch 24 | loss: 0.0311467 | val_loss: 0.0311547 | Time: 4776.93 ms
[2021-08-21 11:02:20 main:574] : INFO : Epoch 25 | loss: 0.0311484 | val_loss: 0.0311582 | Time: 4790.19 ms
[2021-08-21 11:02:25 main:574] : INFO : Epoch 26 | loss: 0.031142 | val_loss: 0.0311535 | Time: 4762.29 ms
[2021-08-21 11:02:30 main:574] : INFO : Epoch 27 | loss: 0.0311405 | val_loss: 0.031157 | Time: 4801.85 ms
[2021-08-21 11:02:35 main:574] : INFO : Epoch 28 | loss: 0.0311382 | val_loss: 0.0311593 | Time: 4802.48 ms
[2021-08-21 11:02:39 main:574] : INFO : Epoch 29 | loss: 0.0311388 | val_loss: 0.0311576 | Time: 4801.43 ms
[2021-08-21 11:02:44 main:574] : INFO : Epoch 30 | loss: 0.0311372 | val_loss: 0.0311594 | Time: 4769.31 ms
[2021-08-21 11:02:49 main:574] : INFO : Epoch 31 | loss: 0.0311374 | val_loss: 0.0311563 | Time: 4757.04 ms
[2021-08-21 11:02:54 main:574] : INFO : Epoch 32 | loss: 0.0311383 | val_loss: 0.0311589 | Time: 4828 ms
[2021-08-21 11:02:59 main:574] : INFO : Epoch 33 | loss: 0.0311379 | val_loss: 0.0311601 | Time: 4795.91 ms
[2021-08-21 11:03:04 main:574] : INFO : Epoch 34 | loss: 0.0311385 | val_loss: 0.031161 | Time: 4791.66 ms
[2021-08-21 11:03:08 main:574] : INFO : Epoch 35 | loss: 0.0311402 | val_loss: 0.0311619 | Time: 4774.55 ms
[2021-08-21 11:03:13 main:574] : INFO : Epoch 36 | loss: 0.0311412 | val_loss: 0.0311574 | Time: 4776.02 ms
[2021-08-21 11:03:18 main:574] : INFO : Epoch 37 | loss: 0.0311397 | val_loss: 0.0311565 | Time: 4844.42 ms
[2021-08-21 11:03:23 main:574] : INFO : Epoch 38 | loss: 0.0311394 | val_loss: 0.0311568 | Time: 4816.36 ms
[2021-08-21 11:03:28 main:574] : INFO : Epoch 39 | loss: 0.0311383 | val_loss: 0.0311527 | Time: 4810.41 ms
[2021-08-21 11:03:33 main:574] : INFO : Epoch 40 | loss: 0.0311376 | val_loss: 0.0311567 | Time: 4783.17 ms
[2021-08-21 11:03:38 main:574] : INFO : Epoch 41 | loss: 0.0311389 | val_loss: 0.0311546 | Time: 4999.96 ms
[2021-08-21 11:03:42 main:574] : INFO : Epoch 42 | loss: 0.0311377 | val_loss: 0.0311554 | Time: 4834.7 ms
[2021-08-21 11:03:47 main:574] : INFO : Epoch 43 | loss: 0.0311392 | val_loss: 0.0311553 | Time: 4856.4 ms
[2021-08-21 11:03:52 main:574] : INFO : Epoch 44 | loss: 0.0311392 | val_loss: 0.0311589 | Time: 4848.54 ms
[2021-08-21 11:03:57 main:574] : INFO : Epoch 45 | loss: 0.0311379 | val_loss: 0.0311559 | Time: 4845.16 ms
[2021-08-21 11:04:02 main:574] : INFO : Epoch 46 | loss: 0.0311361 | val_loss: 0.0311543 | Time: 4846.17 ms
[2021-08-21 11:04:07 main:574] : INFO : Epoch 47 | loss: 0.0311393 | val_loss: 0.0311551 | Time: 4883.92 ms
[2021-08-21 11:04:12 main:574] : INFO : Epoch 48 | loss: 0.0311382 | val_loss: 0.0311569 | Time: 4890.32 ms
[2021-08-21 11:04:17 main:574] : INFO : Epoch 49 | loss: 0.0311381 | val_loss: 0.0311575 | Time: 4844.07 ms
[2021-08-21 11:04:22 main:574] : INFO : Epoch 50 | loss: 0.0311357 | val_loss: 0.0311514 | Time: 4850.77 ms
[2021-08-21 11:04:27 main:574] : INFO : Epoch 51 | loss: 0.0311367 | val_loss: 0.0311573 | Time: 4840.37 ms
[2021-08-21 11:04:31 main:574] : INFO : Epoch 52 | loss: 0.0311345 | val_loss: 0.0311529 | Time: 4848.94 ms
[2021-08-21 11:04:36 main:574] : INFO : Epoch 53 | loss: 0.0311378 | val_loss: 0.0311544 | Time: 4837.97 ms
[2021-08-21 11:04:41 main:574] : INFO : Epoch 54 | loss: 0.0311399 | val_loss: 0.031157 | Time: 4819.95 ms
[2021-08-21 11:04:46 main:574] : INFO : Epoch 55 | loss: 0.031139 | val_loss: 0.0311536 | Time: 4801.34 ms
[2021-08-21 11:04:51 main:574] : INFO : Epoch 56 | loss: 0.0311426 | val_loss: 0.0311574 | Time: 4916.78 ms
[2021-08-21 11:04:56 main:574] : INFO : Epoch 57 | loss: 0.0311502 | val_loss: 0.0311581 | Time: 4817.74 ms
[2021-08-21 11:05:01 main:574] : INFO : Epoch 58 | loss: 0.0311503 | val_loss: 0.0311573 | Time: 4836.68 ms
[2021-08-21 11:05:06 main:574] : INFO : Epoch 59 | loss: 0.0311462 | val_loss: 0.0311579 | Time: 4819.6 ms
[2021-08-21 11:05:10 main:574] : INFO : Epoch 60 | loss: 0.0311466 | val_loss: 0.0311536 | Time: 4777.24 ms
[2021-08-21 11:05:15 main:574] : INFO : Epoch 61 | loss: 0.0311446 | val_loss: 0.0311532 | Time: 4763.64 ms
[2021-08-21 11:05:20 main:574] : INFO : Epoch 62 | loss: 0.0311448 | val_loss: 0.0311648 | Time: 4925.37 ms
[2021-08-21 11:05:25 main:574] : INFO : Epoch 63 | loss: 0.0311432 | val_loss: 0.0311571 | Time: 4838.97 ms
[2021-08-21 11:05:30 main:574] : INFO : Epoch 64 | loss: 0.0311439 | val_loss: 0.0311612 | Time: 4792.59 ms
[2021-08-21 11:05:35 main:574] : INFO : Epoch 65 | loss: 0.0311425 | val_loss: 0.0311624 | Time: 4839.02 ms
[2021-08-21 11:05:40 main:574] : INFO : Epoch 66 | loss: 0.0311398 | val_loss: 0.0311576 | Time: 4814.85 ms
[2021-08-21 11:05:44 main:574] : INFO : Epoch 67 | loss: 0.0311393 | val_loss: 0.0311632 | Time: 4792.17 ms
[2021-08-21 11:05:49 main:574] : INFO : Epoch 68 | loss: 0.0311389 | val_loss: 0.0311569 | Time: 4756.8 ms
[2021-08-21 11:05:54 main:574] : INFO : Epoch 69 | loss: 0.0311401 | val_loss: 0.0311657 | Time: 4780.04 ms
[2021-08-21 11:05:59 main:574] : INFO : Epoch 70 | loss: 0.03114 | val_loss: 0.0311553 | Time: 4785.77 ms
[2021-08-21 11:06:04 main:574] : INFO : Epoch 71 | loss: 0.0311383 | val_loss: 0.0311586 | Time: 4817.94 ms
[2021-08-21 11:06:09 main:574] : INFO : Epoch 72 | loss: 0.0311386 | val_loss: 0.0311565 | Time: 4815.22 ms
[2021-08-21 11:06:13 main:574] : INFO : Epoch 73 | loss: 0.031139 | val_loss: 0.031157 | Time: 4807.76 ms
[2021-08-21 11:06:18 main:574] : INFO : Epoch 74 | loss: 0.0311373 | val_loss: 0.031157 | Time: 4917.16 ms
[2021-08-21 11:06:23 main:574] : INFO : Epoch 75 | loss: 0.0311377 | val_loss: 0.0311591 | Time: 4794.32 ms
[2021-08-21 11:06:28 main:574] : INFO : Epoch 76 | loss: 0.0311374 | val_loss: 0.0311575 | Time: 4796.91 ms
[2021-08-21 11:06:33 main:574] : INFO : Epoch 77 | loss: 0.0311353 | val_loss: 0.0311568 | Time: 4800.64 ms
[2021-08-21 11:06:38 main:574] : INFO : Epoch 78 | loss: 0.0311373 | val_loss: 0.0311522 | Time: 4857.93 ms
[2021-08-21 11:06:43 main:574] : INFO : Epoch 79 | loss: 0.0311375 | val_loss: 0.031154 | Time: 4818.14 ms
[2021-08-21 11:06:48 main:574] : INFO : Epoch 80 | loss: 0.0311373 | val_loss: 0.0311584 | Time: 4956.83 ms
[2021-08-21 11:06:52 main:574] : INFO : Epoch 81 | loss: 0.0311352 | val_loss: 0.0311559 | Time: 4807.64 ms
[2021-08-21 11:06:57 main:574] : INFO : Epoch 82 | loss: 0.0311432 | val_loss: 0.0311603 | Time: 4843.8 ms
[2021-08-21 11:07:02 main:574] : INFO : Epoch 83 | loss: 0.0311463 | val_loss: 0.031167 | Time: 4876.76 ms
[2021-08-21 11:07:07 main:574] : INFO : Epoch 84 | loss: 0.031146 | val_loss: 0.0311659 | Time: 4774.18 ms
[2021-08-21 11:07:12 main:574] : INFO : Epoch 85 | loss: 0.0311484 | val_loss: 0.0311619 | Time: 4810.68 ms
[2021-08-21 11:07:17 main:574] : INFO : Epoch 86 | loss: 0.0311459 | val_loss: 0.0311615 | Time: 4793.71 ms
[2021-08-21 11:07:22 main:574] : INFO : Epoch 87 | loss: 0.0311464 | val_loss: 0.0311646 | Time: 4818.62 ms
[2021-08-21 11:07:26 main:574] : INFO : Epoch 88 | loss: 0.0311443 | val_loss: 0.0311643 | Time: 4801.59 ms
[2021-08-21 11:07:31 main:574] : INFO : Epoch 89 | loss: 0.031142 | val_loss: 0.0311566 | Time: 4795.5 ms
[2021-08-21 11:07:36 main:574] : INFO : Epoch 90 | loss: 0.0311406 | val_loss: 0.0311565 | Time: 4796.63 ms
[2021-08-21 11:07:41 main:574] : INFO : Epoch 91 | loss: 0.0311391 | val_loss: 0.0311581 | Time: 4789.21 ms
[2021-08-21 11:07:46 main:574] : INFO : Epoch 92 | loss: 0.0311373 | val_loss: 0.0311602 | Time: 4783.34 ms
[2021-08-21 11:07:51 main:574] : INFO : Epoch 93 | loss: 0.0311359 | val_loss: 0.0311587 | Time: 4764.63 ms
[2021-08-21 11:07:55 main:574] : INFO : Epoch 94 | loss: 0.0311371 | val_loss: 0.0311559 | Time: 4795.59 ms
[2021-08-21 11:08:00 main:574] : INFO : Epoch 95 | loss: 0.0311372 | val_loss: 0.0311546 | Time: 4855.97 ms
[2021-08-21 11:08:05 main:574] : INFO : Epoch 96 | loss: 0.0311406 | val_loss: 0.0311628 | Time: 4789.72 ms
[2021-08-21 11:08:10 main:574] : INFO : Epoch 97 | loss: 0.0311409 | val_loss: 0.0311544 | Time: 4745.02 ms
[2021-08-21 11:08:15 main:574] : INFO : Epoch 98 | loss: 0.0311356 | val_loss: 0.0311599 | Time: 4788.76 ms
[2021-08-21 11:08:20 main:574] : INFO : Epoch 99 | loss: 0.0311367 | val_loss: 0.0311587 | Time: 4785.58 ms
[2021-08-21 11:08:24 main:574] : INFO : Epoch 100 | loss: 0.0311348 | val_loss: 0.03116 | Time: 4789.86 ms
[2021-08-21 11:08:29 main:574] : INFO : Epoch 101 | loss: 0.0311346 | val_loss: 0.031159 | Time: 4822.39 ms
[2021-08-21 11:08:34 main:574] : INFO : Epoch 102 | loss: 0.0311338 | val_loss: 0.0311575 | Time: 4792.81 ms
[2021-08-21 11:08:39 main:574] : INFO : Epoch 103 | loss: 0.0311344 | val_loss: 0.0311581 | Time: 4798.9 ms
[2021-08-21 11:08:44 main:574] : INFO : Epoch 104 | loss: 0.0311332 | val_loss: 0.0311633 | Time: 4763.97 ms
[2021-08-21 11:08:49 main:574] : INFO : Epoch 105 | loss: 0.0311338 | val_loss: 0.0311588 | Time: 4786.09 ms
[2021-08-21 11:08:53 main:574] : INFO : Epoch 106 | loss: 0.0311319 | val_loss: 0.031157 | Time: 4782.2 ms
[2021-08-21 11:08:58 main:574] : INFO : Epoch 107 | loss: 0.0311309 | val_loss: 0.0311638 | Time: 4779.34 ms
[2021-08-21 11:09:03 main:574] : INFO : Epoch 108 | loss: 0.0311334 | val_loss: 0.0311709 | Time: 4792.33 ms
[2021-08-21 11:09:08 main:574] : INFO : Epoch 109 | loss: 0.0311365 | val_loss: 0.0311574 | Time: 4771.62 ms
[2021-08-21 11:09:13 main:574] : INFO : Epoch 110 | loss: 0.0311361 | val_loss: 0.031161 | Time: 4801.1 ms
[2021-08-21 11:09:18 main:574] : INFO : Epoch 111 | loss: 0.0311406 | val_loss: 0.0311558 | Time: 4891.64 ms
[2021-08-21 11:09:33 main:574] : INFO : Epoch 112 | loss: 0.0311412 | val_loss: 0.0311421 | Time: 15365.6 ms
[2021-08-21 11:10:01 main:574] : INFO : Epoch 113 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 28101.6 ms
[2021-08-21 11:10:41 main:574] : INFO : Epoch 114 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 39740.7 ms
[2021-08-21 11:11:32 main:574] : INFO : Epoch 115 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 51170.9 ms
[2021-08-21 11:12:35 main:574] : INFO : Epoch 116 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 62609.5 ms
[2021-08-21 11:13:49 main:574] : INFO : Epoch 117 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 74151.1 ms
[2021-08-21 11:15:16 main:574] : INFO : Epoch 118 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 86365.5 ms
[2021-08-21 11:16:53 main:574] : INFO : Epoch 119 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 97539.3 ms
[2021-08-21 11:18:42 main:574] : INFO : Epoch 120 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 108692 ms
[2021-08-21 11:20:42 main:574] : INFO : Epoch 121 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 120075 ms
[2021-08-21 11:22:55 main:574] : INFO : Epoch 122 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 133007 ms
[2021-08-21 11:25:20 main:574] : INFO : Epoch 123 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 144647 ms
[2021-08-21 11:27:56 main:574] : INFO : Epoch 124 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 156001 ms
[2021-08-21 11:30:42 main:574] : INFO : Epoch 125 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 166267 ms
[2021-08-21 11:33:43 main:574] : INFO : Epoch 126 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 180917 ms
[2021-08-21 11:36:55 main:574] : INFO : Epoch 127 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 192013 ms
[2021-08-21 11:40:17 main:574] : INFO : Epoch 128 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 201754 ms
[2021-08-21 11:43:50 main:574] : INFO : Epoch 129 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 212583 ms
[2021-08-21 11:47:34 main:574] : INFO : Epoch 130 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 224136 ms
[2021-08-21 11:51:33 main:574] : INFO : Epoch 131 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 239066 ms
[2021-08-21 11:55:42 main:574] : INFO : Epoch 132 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 249261 ms
[2021-08-21 12:00:03 main:574] : INFO : Epoch 133 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 260861 ms
[2021-08-21 12:04:35 main:574] : INFO : Epoch 134 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 271505 ms
[2021-08-21 12:09:24 main:574] : INFO : Epoch 135 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 289358 ms
[2021-08-21 12:14:19 main:574] : INFO : Epoch 136 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 295331 ms
[2021-08-21 12:19:27 main:574] : INFO : Epoch 137 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 307657 ms
[2021-08-21 12:24:49 main:574] : INFO : Epoch 138 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 321521 ms
[2021-08-21 12:30:21 main:574] : INFO : Epoch 139 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 332192 ms
[2021-08-21 12:36:02 main:574] : INFO : Epoch 140 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 340746 ms
[2021-08-21 12:41:54 main:574] : INFO : Epoch 141 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 352635 ms
[2021-08-21 12:47:59 main:574] : INFO : Epoch 142 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 364546 ms
[2021-08-21 12:54:17 main:574] : INFO : Epoch 143 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 378469 ms
[2021-08-21 13:00:45 main:574] : INFO : Epoch 144 | loss: 0.0311421 | val_loss: 0.0311421 | Time: 387572 ms
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFA1B009A92
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.17.0
Dump Timestamp : 08/21/21 13:04:04
Install Directory : C:\Program Files\BOINC\
Data Directory : D:\B_DATA
Project Symstore :
LoadLibraryA( D:\B_DATA\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( D:\B_DATA\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\B_DATA\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\B_DATA\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: D:\B_DATA\slots\6;D:\B_DATA\projects\www.mlcathome.org_mlcathome
ModLoad: 0000000009f70000 00000000003df000 D:\B_DATA\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000001d2b0000 00000000001f5000 C:\Windows\SYSTEM32\ntdll.dll (6.2.19041.1110) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.19041.1023 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1023
ModLoad: 000000001c580000 00000000000bd000 C:\Windows\System32\KERNEL32.DLL (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 000000001af40000 00000000002c9000 C:\Windows\System32\KERNELBASE.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 000000001cad0000 00000000001a0000 C:\Windows\System32\USER32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001ad40000 0000000000022000 C:\Windows\System32\win32u.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 000000001c8b0000 000000000002a000 C:\Windows\System32\GDI32.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.19041.746 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.746
ModLoad: 000000001ae00000 000000000010b000 C:\Windows\System32\gdi32full.dll (6.2.19041.1110) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.19041.1110 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1110
ModLoad: 000000001aca0000 000000000009d000 C:\Windows\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 000000001a9e0000 0000000000100000 C:\Windows\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 000000001c7f0000 00000000000ac000 C:\Windows\System32\ADVAPI32.dll (6.2.19041.1052) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001b510000 000000000009e000 C:\Windows\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.19041.546
ModLoad: 000000001b970000 000000000009b000 C:\Windows\System32\sechost.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001c390000 000000000012a000 C:\Windows\System32\RPCRT4.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000003b60000 0000000000058000 D:\B_DATA\slots\6\c10.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000004970000 0000000000092000 C:\Windows\SYSTEM32\MSVCP140.dll (14.28.29325.2) (-exported- Symbols Loaded)
Linked PDB Filename : D:\agent\_work\49\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
File Version : 14.28.29325.2 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29325.2
ModLoad: 0000000004a10000 00000000001e4000 C:\Windows\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.19041.867 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.867
ModLoad: 0000000004950000 0000000000019000 C:\Windows\SYSTEM32\VCRUNTIME140.dll (14.28.29325.2) (-exported- Symbols Loaded)
Linked PDB Filename : D:\agent\_work\49\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
File Version : 14.28.29325.2 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29325.2
ModLoad: 000000000cba0000 000000000000c000 C:\Windows\SYSTEM32\VCRUNTIME140_1.dll (14.28.29325.2) (-exported- Symbols Loaded)
Linked PDB Filename : D:\agent\_work\49\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
File Version : 14.28.29325.2 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29325.2
ModLoad: 00000000d3180000 000000000abe7000 D:\B_DATA\slots\6\torch_cpu.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000d2ea0000 00000000002db000 D:\B_DATA\slots\6\fbgemm.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000003b00000 0000000000040000 D:\B_DATA\slots\6\asmjit.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000d2c90000 000000000020d000 D:\B_DATA\slots\6\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
Linked PDB Filename : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
File Version : 20200205
Company Name : Intel Corporation
Product Name : Intel(R) OpenMP* Runtime Library
Product Version : 5.0
ModLoad: 00000000e6280000 0000000024b49000 D:\B_DATA\slots\6\torch_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000fde00000 0000000000045000 D:\B_DATA\slots\6\c10_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000cf7d0000 00000000034b2000 D:\B_DATA\slots\6\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA RAND Library
Product Version : 6,14,11,1012
ModLoad: 0000000016840000 0000000000010000 D:\B_DATA\slots\6\nvToolsExt64_1.dll (-exported- Symbols Loaded)
Linked PDB Filename : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb
ModLoad: 00000000cc720000 00000000030a5000 D:\B_DATA\slots\6\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1031
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA SPARSE BLAS Library
Product Version : 6,14,11,1031
ModLoad: 00000000c89c0000 0000000003d5b000 D:\B_DATA\slots\6\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 00000000bf820000 000000000919f000 D:\B_DATA\slots\6\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA FFT Library
Product Version : 6,14,11,1012
ModLoad: 00000000bd950000 0000000001eca000 D:\B_DATA\slots\6\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 000000004d4d0000 000000001b3a4000 D:\B_DATA\slots\6\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,10020
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.86 CUDNN Library
Product Version : 6,14,11,10020
ModLoad: 000000001c550000 0000000000030000 C:\Windows\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000188d0000 0000000000012000 C:\Windows\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 000000001cd20000 0000000000472000 C:\Windows\System32\Setupapi.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : setupapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001b2c0000 000000000004e000 C:\Windows\System32\cfgmgr32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : cfgmgr32.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 000000001af10000 0000000000027000 C:\Windows\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001a7a0000 0000000000034000 C:\Windows\SYSTEM32\DEVOBJ.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : devobj.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 000000001aae0000 0000000000060000 C:\Windows\System32\WINTRUST.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : wintrust.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 000000001ab40000 000000000015f000 C:\Windows\System32\CRYPT32.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : crypt32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001a5c0000 0000000000012000 C:\Windows\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msasn1.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 000000001bc50000 000000000073f000 C:\Windows\System32\Shell32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : shell32.pdb
File Version : 10.0.19041.964 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.964
ModLoad: 0000000018ad0000 0000000000790000 C:\Windows\SYSTEM32\windows.storage.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : Windows.Storage.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001b610000 0000000000355000 C:\Windows\System32\combase.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : combase.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001a360000 000000000002c000 C:\Windows\SYSTEM32\Wldp.dll (6.2.19041.662) (-exported- Symbols Loaded)
Linked PDB Filename : WLDP.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001b330000 00000000000ae000 C:\Windows\System32\SHCORE.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shcore.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001c950000 0000000000055000 C:\Windows\System32\shlwapi.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shlwapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001d280000 00000000002cc000 C:\Windows\system32\nvcuda.dll (30.0.14.7168) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r471_59\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
File Version : 30.0.14.7168
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.4.112 driver
Product Version : 30.0.14.7168
ModLoad: 0000000012fd0000 000000000000a000 C:\Windows\SYSTEM32\VERSION.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 000000001baa0000 00000000017d7000 C:\Windows\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_253e24b785ea60ca\nvcuda64.dll (30.0.14.7168) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r471_59\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
File Version : 30.0.14.7168
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.4.112 driver
Product Version : 30.0.14.7168
ModLoad: 0000000012f70000 0000000000031000 C:\Windows\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : cryptnet.pdb
File Version : 10.0.19041.906 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.906
ModLoad: 0000000012c90000 000000000014a000 C:\Windows\SYSTEM32\drvstore.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : drvstore.pdb
File Version : 10.0.19041.1081 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1081
ModLoad: 000000001a2d0000 000000000000c000 C:\Windows\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cryptbase.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000018370000 000000000009e000 C:\Windows\system32\uxtheme.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : UxTheme.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000009ec0000 0000000000719000 C:\Windows\SYSTEM32\nvapi64.dll (30.0.14.7168) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r471_59\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
File Version : 30.0.14.7168
Company Name : NVIDIA Corporation
Product Name : NVIDIA Windows drivers
Product Version : 30.0.14.7168
ModLoad: 000000001b3e0000 000000000012a000 C:\Windows\System32\ole32.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : ole32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001b220000 000000000087b000 C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_253e24b785ea60ca\nvptxJitCompiler64.dll (30.0.14.7168) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 30.0.14.7168
Company Name : NVIDIA Corporation
Product Name : NVIDIA Windows PTX JIT Compiler
Product Version : 30.0.14.7168
ModLoad: 00000000157f0000 0000000000009000 D:\B_DATA\slots\6\caffe2_nvrtc.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000077a10000 0000000000fca000 D:\B_DATA\slots\6\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6.14.11.9000
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.89 NVRTC Library
Product Version : 6.14.11.9000
ModLoad: 000000001ad70000 0000000000083000 C:\Windows\System32\bcryptPrimitives.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.19041.1023 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1023
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 5296, Write: 4649, Other 19810
- I/O Transfers Counters -
Read: 42426213, Write: -1954535248, Other 57280
- Paged Pool Usage -
QuotaPagedPoolUsage: 3574392, QuotaPeakPagedPoolUsage: 3574616
QuotaNonPagedPoolUsage: 64944, QuotaPeakNonPagedPoolUsage: 78272
- Virtual Memory Usage -
VirtualSize: 566435840, PeakVirtualSize: 2060259328
- Pagefile Usage -
PagefileUsage: 566435840, PeakPagefileUsage: 780558336
- Working Set Size -
WorkingSetSize: -1937494016, PeakWorkingSetSize: -1263992832, PageFaultCount: 43280075
*** Dump of thread ID 17040 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2504492.000000
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFA1B009A92
- Registers -
rax=0000000000000000 rbx=0000000000000001 rcx=000000000a228c40 rdx=00000000d63fe960 rsi=0000000000000000 rdi=0000000000000000
r8=00000000d63fe960 r9=000000000a228c30 r10=0000000000000fff r11=0000000000000ff0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=000000001b009a92 rsp=00000000d63fe938 rbp=0000000000000000
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
d63fe930 09fa9100 00000001 d63fe960 d63fe960 0a228c30 KERNELBASE!DebugBreak+0x0
d63fed70 09fa9ce4 9ae00000 d63ff0d4 00000000 00000374 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
d63ffcd0 09faa084 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
d63ffd00 1c597034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
d63ffd30 1d302651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
d63ffdb0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 13052 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 3438749952.000000, User Time: 57242345472.000000, Wait Time: 2503907.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=0000000082ca4aa8 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000082ca4aa8
r8=0000000004800000 r9=00000000d5ee1700 r10=000000000bf80bd0 r11=00000000d5ee1758 r12=0000000000000000 r13=0000000000000000
r14=00000000d5ee55a0 r15=0000000082ca4a58 rip=000000001d350814 rsp=00000000d5ee5578 rbp=00000000d5ee55d0
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
d5ee5570 1d314021 00000000 d874ea63 0ad495d0 d5ee55b0 ntdll!NtWaitForAlertByThreadId+0x0
d5ee55f0 1aface29 82ca4aa0 82ca4a98 82ca4a50 d5ee5210 ntdll!RtlSleepConditionVariableSRW+0x0
d5ee5630 049abf3d 00000000 d5ee5210 fde02e0d 03b82433 KERNELBASE!SleepConditionVariableSRW+0x0
d5ee5670 04982798 00000000 d8750231 03b99c50 00000000 MSVCP140!__crtSetThreadpoolWait+0x0
d5ee56a0 04982a4a b48a80d1 82ca4a50 6f903090 82ca4a98 MSVCP140!std::_Winerror_message+0x0
d5ee56d0 da36064b 82ca4a50 d5ee57d0 82ca4a50 ffffffff MSVCP140!_Cnd_wait+0x0
d5ee5710 da3625b1 09eb2ba0 6f903080 6f903080 00000000 torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0
d5ee5a00 da35f15d d5ee5ce0 d5ee5ce0 d5ee5ce0 09eb2ba0 torch_cpu!torch::autograd::Engine::thread_main+0x0
d5ee5c70 da35eea0 8e4bfe50 dd886d10 fffffffe 00000000 torch_cpu!torch::autograd::Engine::execute_with_graph_task+0x0
d5ee5da0 da3556f9 64edf1f0 64edf1f0 d5ee5d90 d5ee5e98 torch_cpu!torch::autograd::Engine::execute+0x0
d5ee62c0 da355086 00000000 d5ee6728 64edf000 d5ee6390 torch_cpu!torch::autograd::Engine::make_anomaly_metadata+0x0
d5ee6360 da6509be 835165b0 64edf1f8 00000000 d5ee6300 torch_cpu!torch::autograd::backward+0x0
d5ee6440 da081cd0 0bc22d30 00000000 00000000 00000000 torch_cpu!torch::autograd::VariableType::allCUDATypes+0x0
d5ee6470 d8d20a06 00000000 00000000 00010001 00000001 torch_cpu!torch::autograd::GraphRoot::apply+0x0
d5ee66a0 09f87166 d5ee6770 d5ee67b0 d5ee6770 d5ee6d10 torch_cpu!at::Tensor::backward+0x0
d5ee6ba0 09f8b950 09fbd95f 09fbd95f 00000000 19bfd030 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0
d5eff940 0a1cc6f8 0bcc2150 00000000 00000000 0bd9fcf0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
d5eff980 1c597034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
d5eff9b0 1d302651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
d5effa30 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 10340 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2504489.000000
- Registers -
rax=000000000000005b rbx=0000000000000007 rcx=0000000000000007 rdx=00000000d64ff620 rsi=0000000000000000 rdi=0000000000000007
r8=00000000d64ff2c8 r9=000000000bec4110 r10=0000000000000000 r11=0000000000000246 r12=0000000000000064 r13=00000000d64ff620
r14=00000000d64ff320 r15=0000000000000000 rip=000000001d34d974 rsp=00000000d64ff2c8 rbp=000000000bec4110
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
d64ff2c0 1af8cab0 00000000 d5d11000 1d41a4d0 d5d11000 ntdll!ZwWaitForMultipleObjects+0x0
d64ff5b0 1af8c9ae 09eb5750 006e0069 00000007 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0
d64ff5f0 1bc71624 0bec6db0 00000000 00000000 00000000 KERNELBASE!WaitForMultipleObjects+0x0
d64ff830 1bd1831d 00000000 0bec4110 00000000 00000000 nvcuda64!+0x0
d64ff8e0 1bc70ff3 1d058eaa 990c911d 0c4ac360 00000000 nvcuda64!cuProfilerStop+0x0
d64ff910 1bfa6df4 0c4ac360 00000000 00000000 00000000 nvcuda64!+0x0
d64ff940 1c597034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
d64ff970 1d302651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
d64ff9f0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 10596 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2504481.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=000000000bf83ac0 rdx=00000000d65ff7e8 rsi=00000000d65ff750 rdi=0000000000000000
r8=0000000000000001 r9=0000000000002964 r10=000000000c1588c0 r11=00000000f6e8f64b r12=00000000d65ff7e8 r13=0000000000000000
r14=000000000bf83ac0 r15=000000000c1588c0 rip=000000001d350814 rsp=00000000d65ff728 rbp=00000000d65ff780
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
d65ff720 1d314295 00000102 00000102 00000000 d65ff750 ntdll!NtWaitForAlertByThreadId+0x0
d65ff790 1afacd79 0c1560a0 00000000 00000000 00000000 ntdll!RtlSleepConditionVariableCS+0x0
d65ff7c0 1bbdf2cc 0c1560a0 0c1560a0 0c1560a0 ffb3b4c0 KERNELBASE!SleepConditionVariableCS+0x0
d65ff820 1bc70ff3 00000001 1ccf6312 372e7c9c 00000000 nvcuda64!+0x0
d65ff850 1bfa6df4 0c4ab9d0 00000000 00000000 00000000 nvcuda64!+0x0
d65ff880 1c597034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
d65ff8b0 1d302651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
d65ff930 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 6824 (state: Running): ***
- Information -
Status: Base Priority: Unknown, Priority: Unknown, , Kernel Time: 2629843712.000000, User Time: 36074532864.000000, Wait Time: 2504487.000000
- Registers -
rax=0000000000000000 rbx=0000000000000000 rcx=000000001bac28e7 rdx=0000000000000000 rsi=000000000c1560a0 rdi=0000000000000006
r8=000000001baa0000 r9=00000000d5ffafe0 r10=0000000004893390 r11=00000000d5ffb038 r12=0000000000000008 r13=000000001baa0000
r14=000000000000000a r15=0000000000000001 rip=000000001bac28e5 rsp=00000000d5ffaf98 rbp=0000000000000017
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000206
- Callstack -
ChildEBP RetAddr Args to Child
d5ffaf90 1bea9cf6 40000280 00000007 00000000 00000000 nvcuda64!+0x0
d5ffb030 1be95202 0c1560a0 0c1560a0 00000000 0bea7690 nvcuda64!cuProfilerStop+0x0
d5ffb060 1bbde00a 00000008 0c9ad2a0 00000001 0c1560a0 nvcuda64!cuProfilerStop+0x0
d5ffb0a0 1bd5299f 00000001 00000000 00000001 00000001 nvcuda64!+0x0
d5ffb0f0 1bc52b06 d5ffb130 3a638840 00000000 00000000 nvcuda64!cuProfilerStop+0x0
d5ffb120 1bc53345 00000002 0c1560a0 0beaa480 0c1560a0 nvcuda64!+0x0
d5ffb1a0 1bc533a3 00000000 00000000 0beaa480 00000000 nvcuda64!+0x0
d5ffb1d0 1bca2863 0c1560a0 593230e0 d5ffb269 5931c840 nvcuda64!+0x0
d5ffb2c0 4e2731fa 0a8fffd0 000003e7 80efd190 593230e0 nvcuda64!cuStreamSynchronize_ptsz+0x0
d5ffb2f0 4e270fbe 0a560860 41daaaab 00000001 ffffff9d cudnn64_7!cudnnSetDropoutDescriptor+0x0
d5ffb3b0 4e2262a3 00000000 00000000 d5ffb490 00000000 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d5ffb3f0 4e21c77c d5ffb5e0 d5ffb500 d5960a30 00000400 cudnn64_7!cudnnRNNBackwardDataEx+0x0
d5ffb530 4e226ea4 00000400 00000400 d5960a30 6410ebe0 cudnn64_7!cudnnRNNBackwardDataEx+0x0
d5ffd690 e8ffc701 8274abc0 40700000 8274abc0 00000000 cudnn64_7!cudnnRNNBackwardWeights+0x0
d5ffdb90 e8ff9db5 00000010 d5ffdca0 d5ffe520 d5ffe508 torch_cuda!at::native::_cudnn_rnn_backward+0x0
d5ffdd80 e9052808 d5ffdf70 d5ffec58 d5ffde60 00000004 torch_cuda!at::native::_cudnn_rnn_backward+0x0
d5ffde90 e906103d d5ffe520 d5ffe500 d5ffec38 00000000 torch_cuda!at::native::set_storage_cuda_+0x0
d5ffdfa0 d8bcc201 d5ffe520 d874ea63 835144b0 d5ffe668 torch_cuda!at::native::set_storage_cuda_+0x0
d5ffe130 d8c1bfea d5ffebf0 00000090 00000007 dbc6f880 torch_cpu!at::native::mkldnn_sigmoid_+0x0
d5ffe260 d8c034da ddaa4668 d5ffe4a0 d5ffec58 d5ffe340 torch_cpu!at::bucketize_out+0x0
d5ffe350 d9efe0a9 d5ffe4a0 d5ffebf0 d5ffe490 00000000 torch_cpu!at::_cudnn_rnn_backward+0x0
d5ffe630 d9f0a94d 83513ab0 da37794a 00000000 00000004 torch_cpu!torch::autograd::GraphRoot::apply+0x0
d5ffe740 d8bcc201 d5ffec48 835137b0 00000000 d5ffed20 torch_cpu!torch::autograd::GraphRoot::apply+0x0
d5ffe8d0 d8c1bfea d5ffebf0 00000000 00000000 00000000 torch_cpu!at::native::mkldnn_sigmoid_+0x0
d5ffea00 d8c034da ddaa4668 d5ffed70 d5ffec58 d5ffeae0 torch_cpu!at::bucketize_out+0x0
d5ffeaf0 d9e0994d d5ffed70 fcfed3d0 d5ffec30 053721c8 torch_cpu!at::_cudnn_rnn_backward+0x0
d5ffef90 d9df56b1 6f903090 d5fff298 00000000 03b63bd6 torch_cpu!torch::autograd::generated::CudnnRnnBackward::apply+0x0
d5fff1a0 da35d1ea d5fff2b0 6f903090 d5fff2b0 6f903090 torch_cpu!torch::autograd::Node::operator()+0x0
d5fff380 da35dbdd 00000001 00000001 00000001 00000001 torch_cpu!torch::autograd::Engine::add_thread_pool_task+0x0
d5fff860 da362812 00000000 00000000 6f903080 ffffffff torch_cpu!torch::autograd::Engine::evaluate_function+0x0
d5fffb50 da362471 dd886d10 d5fffb90 dd886d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0
d5fffba0 da359744 00080001 554d44b0 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0
d5fffbd0 1aa01bb2 554d44b0 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0
d5fffc00 1c597034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0
d5fffc30 1d302651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
d5fffcb0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)