Task 4346521
| Name | ParityModified-1612597748-8468-3_0 |
| Workunit | 2417230 |
| Created | 7 Mar 2021, 21:31:20 UTC |
| Sent | 19 Mar 2021, 8:29:20 UTC |
| Report deadline | 26 Mar 2021, 8:29:20 UTC |
| Received | 19 Mar 2021, 16:59:57 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | -529697949 (0xE06D7363) Unknown error code |
| Computer ID | 1545 |
| Run time | 6 min 11 sec |
| CPU time | 5 min 41 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 4,174.72 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.55 GB |
| Peak swap size | 3.49 GB |
| Peak disk usage | 1.53 GB |
Stderr output
<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 3765269347 (0xe06d7363)</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce GTX 970)
[2021-03-19 18:30:44 main:435] : INFO : Set logging level to 1
[2021-03-19 18:30:44 main:441] : INFO : Running in BOINC Client mode
[2021-03-19 18:30:44 main:444] : INFO : Resolving all filenames
[2021-03-19 18:30:44 main:452] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 (exists = 1)
[2021-03-19 18:30:44 main:452] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1612597748-8468-3_0_r765073786_1 (exists = 0)
[2021-03-19 18:30:44 main:452] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1612597748-8468-3_0_r765073786_0 (exists = 0)
[2021-03-19 18:30:44 main:452] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1612597748-8468-3 (exists = 1)
[2021-03-19 18:30:44 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2021-03-19 18:30:44 main:472] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5
[2021-03-19 18:30:44 main:474] : INFO : Configuration:
[2021-03-19 18:30:44 main:475] : INFO : Model type: GRU
[2021-03-19 18:30:44 main:476] : INFO : Validation Loss Threshold: 0.0001
[2021-03-19 18:30:44 main:477] : INFO : Max Epochs: 2048
[2021-03-19 18:30:44 main:478] : INFO : Batch Size: 128
[2021-03-19 18:30:44 main:479] : INFO : Learning Rate: 0.01
[2021-03-19 18:30:44 main:480] : INFO : Patience: 10
[2021-03-19 18:30:44 main:481] : INFO : Hidden Width: 12
[2021-03-19 18:30:44 main:482] : INFO : # Recurrent Layers: 4
[2021-03-19 18:30:45 main:483] : INFO : # Backend Layers: 4
[2021-03-19 18:30:45 main:484] : INFO : # Threads: 1
[2021-03-19 18:30:45 main:486] : INFO : Preparing Dataset
[2021-03-19 18:30:45 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory
[2021-03-19 18:30:45 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory
[2021-03-19 18:30:47 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2021-03-19 18:30:47 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory
[2021-03-19 18:30:47 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory
[2021-03-19 18:30:47 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2021-03-19 18:30:47 main:494] : INFO : Creating Model
[2021-03-19 18:30:47 main:507] : INFO : Preparing config file
[2021-03-19 18:30:47 main:519] : INFO : Creating new config file
[2021-03-19 18:30:47 main:538] : INFO : This is a continuation WU, loading previous network
[2021-03-19 18:30:48 main:559] : INFO : Loading DataLoader into Memory
[2021-03-19 18:30:48 main:562] : INFO : Starting Training
[2021-03-19 18:30:50 main:574] : INFO : Epoch 1 | loss: 0.0312048 | val_loss: 0.0311778 | Time: 1965.26 ms
[2021-03-19 18:30:52 main:574] : INFO : Epoch 2 | loss: 0.0311723 | val_loss: 0.031175 | Time: 1694.21 ms
[2021-03-19 18:30:53 main:574] : INFO : Epoch 3 | loss: 0.0311702 | val_loss: 0.0311758 | Time: 1701.41 ms
[2021-03-19 18:30:55 main:574] : INFO : Epoch 4 | loss: 0.031168 | val_loss: 0.0311771 | Time: 1694.48 ms
[2021-03-19 18:30:57 main:574] : INFO : Epoch 5 | loss: 0.031168 | val_loss: 0.0311748 | Time: 1700.46 ms
[2021-03-19 18:30:59 main:574] : INFO : Epoch 6 | loss: 0.0311704 | val_loss: 0.0311771 | Time: 1709.75 ms
[2021-03-19 18:31:00 main:574] : INFO : Epoch 7 | loss: 0.0311709 | val_loss: 0.031176 | Time: 1702.61 ms
[2021-03-19 18:31:02 main:574] : INFO : Epoch 8 | loss: 0.031169 | val_loss: 0.0311746 | Time: 1698.05 ms
[2021-03-19 18:31:04 main:574] : INFO : Epoch 9 | loss: 0.0311692 | val_loss: 0.0311759 | Time: 1688.96 ms
[2021-03-19 18:31:06 main:574] : INFO : Epoch 10 | loss: 0.0311709 | val_loss: 0.0311725 | Time: 1700.15 ms
[2021-03-19 18:31:07 main:574] : INFO : Epoch 11 | loss: 0.0311685 | val_loss: 0.031174 | Time: 1708.94 ms
[2021-03-19 18:31:09 main:574] : INFO : Epoch 12 | loss: 0.0311742 | val_loss: 0.0311776 | Time: 1711.08 ms
[2021-03-19 18:31:11 main:574] : INFO : Epoch 13 | loss: 0.0311721 | val_loss: 0.0311774 | Time: 1703.28 ms
[2021-03-19 18:31:13 main:574] : INFO : Epoch 14 | loss: 0.0311704 | val_loss: 0.0311748 | Time: 1722 ms
[2021-03-19 18:31:14 main:574] : INFO : Epoch 15 | loss: 0.031169 | val_loss: 0.0311767 | Time: 1666.46 ms
[2021-03-19 18:31:16 main:574] : INFO : Epoch 16 | loss: 0.0311687 | val_loss: 0.0311744 | Time: 1702.68 ms
[2021-03-19 18:31:18 main:574] : INFO : Epoch 17 | loss: 0.0311666 | val_loss: 0.0311733 | Time: 1717.59 ms
[2021-03-19 18:31:20 main:574] : INFO : Epoch 18 | loss: 0.0311676 | val_loss: 0.0311725 | Time: 1706.48 ms
[2021-03-19 18:31:21 main:574] : INFO : Epoch 19 | loss: 0.0311699 | val_loss: 0.0311729 | Time: 1712.7 ms
[2021-03-19 18:31:23 main:574] : INFO : Epoch 20 | loss: 0.0311676 | val_loss: 0.0311724 | Time: 1705.27 ms
[2021-03-19 18:31:25 main:574] : INFO : Epoch 21 | loss: 0.0311693 | val_loss: 0.0311734 | Time: 1674.22 ms
[2021-03-19 18:31:27 main:574] : INFO : Epoch 22 | loss: 0.0311686 | val_loss: 0.0311733 | Time: 1717.41 ms
[2021-03-19 18:31:28 main:574] : INFO : Epoch 23 | loss: 0.0311676 | val_loss: 0.0311705 | Time: 1713.15 ms
[2021-03-19 18:31:30 main:574] : INFO : Epoch 24 | loss: 0.0311674 | val_loss: 0.0311723 | Time: 1698.87 ms
[2021-03-19 18:31:32 main:574] : INFO : Epoch 25 | loss: 0.0311663 | val_loss: 0.0311735 | Time: 1730.63 ms
[2021-03-19 18:31:34 main:574] : INFO : Epoch 26 | loss: 0.031167 | val_loss: 0.031174 | Time: 1699.77 ms
[2021-03-19 18:31:35 main:574] : INFO : Epoch 27 | loss: 0.0311695 | val_loss: 0.0311731 | Time: 1668.16 ms
[2021-03-19 18:31:37 main:574] : INFO : Epoch 28 | loss: 0.0311695 | val_loss: 0.0311718 | Time: 1717.51 ms
[2021-03-19 18:31:39 main:574] : INFO : Epoch 29 | loss: 0.0311688 | val_loss: 0.0311697 | Time: 1721.27 ms
[2021-03-19 18:31:41 main:574] : INFO : Epoch 30 | loss: 0.0311684 | val_loss: 0.0311715 | Time: 1713.68 ms
[2021-03-19 18:31:42 main:574] : INFO : Epoch 31 | loss: 0.0311659 | val_loss: 0.0311691 | Time: 1734.25 ms
[2021-03-19 18:31:44 main:574] : INFO : Epoch 32 | loss: 0.0311653 | val_loss: 0.0311718 | Time: 1672.38 ms
[2021-03-19 18:31:46 main:574] : INFO : Epoch 33 | loss: 0.0311642 | val_loss: 0.0311708 | Time: 1716.77 ms
[2021-03-19 18:31:48 main:574] : INFO : Epoch 34 | loss: 0.0311662 | val_loss: 0.0311716 | Time: 1703.61 ms
[2021-03-19 18:31:49 main:574] : INFO : Epoch 35 | loss: 0.031166 | val_loss: 0.0311723 | Time: 1703.97 ms
[2021-03-19 18:31:51 main:574] : INFO : Epoch 36 | loss: 0.0311677 | val_loss: 0.0311717 | Time: 1706.28 ms
[2021-03-19 18:31:53 main:574] : INFO : Epoch 37 | loss: 0.031166 | val_loss: 0.0311733 | Time: 1704.72 ms
[2021-03-19 18:31:55 main:574] : INFO : Epoch 38 | loss: 0.0311663 | val_loss: 0.0311738 | Time: 1672.21 ms
[2021-03-19 18:31:56 main:574] : INFO : Epoch 39 | loss: 0.0311647 | val_loss: 0.0311715 | Time: 1702.13 ms
[2021-03-19 18:31:58 main:574] : INFO : Epoch 40 | loss: 0.031164 | val_loss: 0.0311705 | Time: 1697.71 ms
[2021-03-19 18:32:00 main:574] : INFO : Epoch 41 | loss: 0.0311638 | val_loss: 0.0311707 | Time: 1707.55 ms
[2021-03-19 18:32:02 main:574] : INFO : Epoch 42 | loss: 0.0311662 | val_loss: 0.0311705 | Time: 1717.48 ms
[2021-03-19 18:32:03 main:574] : INFO : Epoch 43 | loss: 0.0311624 | val_loss: 0.0311704 | Time: 1713.36 ms
[2021-03-19 18:32:05 main:574] : INFO : Epoch 44 | loss: 0.0311619 | val_loss: 0.031174 | Time: 1687.21 ms
[2021-03-19 18:32:07 main:574] : INFO : Epoch 45 | loss: 0.0311687 | val_loss: 0.031177 | Time: 1699.06 ms
[2021-03-19 18:32:09 main:574] : INFO : Epoch 46 | loss: 0.0311722 | val_loss: 0.0311703 | Time: 1696.51 ms
[2021-03-19 18:32:10 main:574] : INFO : Epoch 47 | loss: 0.0311658 | val_loss: 0.0311725 | Time: 1726.01 ms
[2021-03-19 18:32:12 main:574] : INFO : Epoch 48 | loss: 0.0311653 | val_loss: 0.0311683 | Time: 1728.14 ms
[2021-03-19 18:32:14 main:574] : INFO : Epoch 49 | loss: 0.0311621 | val_loss: 0.031172 | Time: 1693.9 ms
[2021-03-19 18:32:16 main:574] : INFO : Epoch 50 | loss: 0.0311617 | val_loss: 0.0311694 | Time: 1687.2 ms
[2021-03-19 18:32:17 main:574] : INFO : Epoch 51 | loss: 0.0311594 | val_loss: 0.0311684 | Time: 1713.6 ms
[2021-03-19 18:32:19 main:574] : INFO : Epoch 52 | loss: 0.0311593 | val_loss: 0.0311741 | Time: 1717.83 ms
[2021-03-19 18:32:21 main:574] : INFO : Epoch 53 | loss: 0.0311624 | val_loss: 0.0311698 | Time: 1713.6 ms
[2021-03-19 18:32:23 main:574] : INFO : Epoch 54 | loss: 0.0311605 | val_loss: 0.0311695 | Time: 1698.26 ms
[2021-03-19 18:32:24 main:574] : INFO : Epoch 55 | loss: 0.0311597 | val_loss: 0.0311681 | Time: 1668.96 ms
[2021-03-19 18:32:26 main:574] : INFO : Epoch 56 | loss: 0.0311586 | val_loss: 0.0311703 | Time: 1694.37 ms
[2021-03-19 18:32:28 main:574] : INFO : Epoch 57 | loss: 0.031161 | val_loss: 0.0311718 | Time: 1714.25 ms
[2021-03-19 18:32:30 main:574] : INFO : Epoch 58 | loss: 0.0311602 | val_loss: 0.0311707 | Time: 1713.17 ms
[2021-03-19 18:32:31 main:574] : INFO : Epoch 59 | loss: 0.0311604 | val_loss: 0.0311696 | Time: 1712.55 ms
[2021-03-19 18:32:33 main:574] : INFO : Epoch 60 | loss: 0.0311602 | val_loss: 0.031171 | Time: 1711.91 ms
[2021-03-19 18:32:35 main:574] : INFO : Epoch 61 | loss: 0.0311614 | val_loss: 0.031178 | Time: 1673.26 ms
[2021-03-19 18:32:37 main:574] : INFO : Epoch 62 | loss: 0.0311625 | val_loss: 0.0311722 | Time: 1701.06 ms
[2021-03-19 18:32:38 main:574] : INFO : Epoch 63 | loss: 0.0311638 | val_loss: 0.0311716 | Time: 1703.65 ms
[2021-03-19 18:32:40 main:574] : INFO : Epoch 64 | loss: 0.0311604 | val_loss: 0.03117 | Time: 1703.51 ms
[2021-03-19 18:32:42 main:574] : INFO : Epoch 65 | loss: 0.0311602 | val_loss: 0.0311723 | Time: 1704.42 ms
[2021-03-19 18:32:44 main:574] : INFO : Epoch 66 | loss: 0.0311608 | val_loss: 0.0311692 | Time: 1686.77 ms
[2021-03-19 18:32:45 main:574] : INFO : Epoch 67 | loss: 0.0311613 | val_loss: 0.0311738 | Time: 1689.38 ms
[2021-03-19 18:32:47 main:574] : INFO : Epoch 68 | loss: 0.0311591 | val_loss: 0.0311697 | Time: 1734.35 ms
[2021-03-19 18:32:49 main:574] : INFO : Epoch 69 | loss: 0.0311594 | val_loss: 0.0311701 | Time: 1728.42 ms
[2021-03-19 18:32:51 main:574] : INFO : Epoch 70 | loss: 0.0311596 | val_loss: 0.0311688 | Time: 1702.87 ms
[2021-03-19 18:32:52 main:574] : INFO : Epoch 71 | loss: 0.0311583 | val_loss: 0.0311694 | Time: 1713.78 ms
[2021-03-19 18:32:54 main:574] : INFO : Epoch 72 | loss: 0.0311585 | val_loss: 0.0311721 | Time: 1701.81 ms
[2021-03-19 18:32:56 main:574] : INFO : Epoch 73 | loss: 0.0311582 | val_loss: 0.0311705 | Time: 1706.32 ms
[2021-03-19 18:32:58 main:574] : INFO : Epoch 74 | loss: 0.0311593 | val_loss: 0.0311673 | Time: 1716.74 ms
[2021-03-19 18:32:59 main:574] : INFO : Epoch 75 | loss: 0.0311568 | val_loss: 0.0311716 | Time: 1716.9 ms
[2021-03-19 18:33:01 main:574] : INFO : Epoch 76 | loss: 0.0311587 | val_loss: 0.0311708 | Time: 1709.77 ms
[2021-03-19 18:33:03 main:574] : INFO : Epoch 77 | loss: 0.0311588 | val_loss: 0.0311704 | Time: 1698.83 ms
[2021-03-19 18:33:05 main:574] : INFO : Epoch 78 | loss: 0.0311548 | val_loss: 0.0311687 | Time: 1662.72 ms
[2021-03-19 18:33:06 main:574] : INFO : Epoch 79 | loss: 0.0311574 | val_loss: 0.0311732 | Time: 1734.07 ms
[2021-03-19 18:33:08 main:574] : INFO : Epoch 80 | loss: 0.0311616 | val_loss: 0.0311689 | Time: 1712.66 ms
[2021-03-19 18:33:10 main:574] : INFO : Epoch 81 | loss: 0.0311602 | val_loss: 0.0311702 | Time: 1734.86 ms
[2021-03-19 18:33:12 main:574] : INFO : Epoch 82 | loss: 0.0311573 | val_loss: 0.0311691 | Time: 1712.29 ms
[2021-03-19 18:33:13 main:574] : INFO : Epoch 83 | loss: 0.0311579 | val_loss: 0.0311669 | Time: 1733.46 ms
[2021-03-19 18:33:15 main:574] : INFO : Epoch 84 | loss: 0.0311582 | val_loss: 0.0311684 | Time: 1675.35 ms
[2021-03-19 18:33:17 main:574] : INFO : Epoch 85 | loss: 0.0311557 | val_loss: 0.0311701 | Time: 1711.34 ms
[2021-03-19 18:33:19 main:574] : INFO : Epoch 86 | loss: 0.0311554 | val_loss: 0.0311691 | Time: 1714 ms
[2021-03-19 18:33:20 main:574] : INFO : Epoch 87 | loss: 0.0311565 | val_loss: 0.0311698 | Time: 1726.53 ms
[2021-03-19 18:33:22 main:574] : INFO : Epoch 88 | loss: 0.0311577 | val_loss: 0.0311702 | Time: 1716.46 ms
[2021-03-19 18:33:24 main:574] : INFO : Epoch 89 | loss: 0.0311567 | val_loss: 0.0311715 | Time: 1684.69 ms
[2021-03-19 18:33:26 main:574] : INFO : Epoch 90 | loss: 0.0311614 | val_loss: 0.031172 | Time: 1691.03 ms
[2021-03-19 18:33:27 main:574] : INFO : Epoch 91 | loss: 0.0311582 | val_loss: 0.0311691 | Time: 1711.42 ms
[2021-03-19 18:33:29 main:574] : INFO : Epoch 92 | loss: 0.0311559 | val_loss: 0.0311701 | Time: 1696.73 ms
[2021-03-19 18:33:31 main:574] : INFO : Epoch 93 | loss: 0.0311554 | val_loss: 0.0311688 | Time: 1714.62 ms
[2021-03-19 18:33:33 main:574] : INFO : Epoch 94 | loss: 0.0311553 | val_loss: 0.0311709 | Time: 1741.28 ms
[2021-03-19 18:33:34 main:574] : INFO : Epoch 95 | loss: 0.0311538 | val_loss: 0.0311677 | Time: 1676.18 ms
[2021-03-19 18:33:36 main:574] : INFO : Epoch 96 | loss: 0.0311549 | val_loss: 0.0311675 | Time: 1697.28 ms
[2021-03-19 18:33:38 main:574] : INFO : Epoch 97 | loss: 0.0311552 | val_loss: 0.0311703 | Time: 1712.46 ms
[2021-03-19 18:33:40 main:574] : INFO : Epoch 98 | loss: 0.0311543 | val_loss: 0.0311746 | Time: 1701.31 ms
[2021-03-19 18:33:41 main:574] : INFO : Epoch 99 | loss: 0.0311569 | val_loss: 0.0311681 | Time: 1714.41 ms
[2021-03-19 18:33:43 main:574] : INFO : Epoch 100 | loss: 0.031156 | val_loss: 0.0311744 | Time: 1707.75 ms
[2021-03-19 18:33:45 main:574] : INFO : Epoch 101 | loss: 0.0311544 | val_loss: 0.0311659 | Time: 1693.86 ms
[2021-03-19 18:33:47 main:574] : INFO : Epoch 102 | loss: 0.0311534 | val_loss: 0.0311694 | Time: 1714.77 ms
[2021-03-19 18:33:48 main:574] : INFO : Epoch 103 | loss: 0.0311568 | val_loss: 0.0311708 | Time: 1694.02 ms
[2021-03-19 18:33:50 main:574] : INFO : Epoch 104 | loss: 0.03116 | val_loss: 0.0311696 | Time: 1721.35 ms
[2021-03-19 18:33:52 main:574] : INFO : Epoch 105 | loss: 0.0311632 | val_loss: 0.0311675 | Time: 1716.45 ms
[2021-03-19 18:33:54 main:574] : INFO : Epoch 106 | loss: 0.0311594 | val_loss: 0.031167 | Time: 1713.6 ms
[2021-03-19 18:33:55 main:574] : INFO : Epoch 107 | loss: 0.0311587 | val_loss: 0.0311699 | Time: 1692.79 ms
[2021-03-19 18:33:57 main:574] : INFO : Epoch 108 | loss: 0.0311593 | val_loss: 0.0311724 | Time: 1706.21 ms
[2021-03-19 18:33:59 main:574] : INFO : Epoch 109 | loss: 0.03116 | val_loss: 0.031171 | Time: 1704.77 ms
[2021-03-19 18:34:01 main:574] : INFO : Epoch 110 | loss: 0.0311616 | val_loss: 0.0311711 | Time: 1699.79 ms
[2021-03-19 18:34:02 main:574] : INFO : Epoch 111 | loss: 0.0311775 | val_loss: 0.0311871 | Time: 1726.55 ms
[2021-03-19 18:34:04 main:574] : INFO : Epoch 112 | loss: 0.0311829 | val_loss: 0.0311775 | Time: 1688.65 ms
[2021-03-19 18:34:06 main:574] : INFO : Epoch 113 | loss: 0.0311761 | val_loss: 0.0311777 | Time: 1702.62 ms
[2021-03-19 18:34:08 main:574] : INFO : Epoch 114 | loss: 0.0311762 | val_loss: 0.0311749 | Time: 1686.32 ms
[2021-03-19 18:34:09 main:574] : INFO : Epoch 115 | loss: 0.031175 | val_loss: 0.031173 | Time: 1736.4 ms
[2021-03-19 18:34:11 main:574] : INFO : Epoch 116 | loss: 0.0311729 | val_loss: 0.0311711 | Time: 1721.09 ms
[2021-03-19 18:34:13 main:574] : INFO : Epoch 117 | loss: 0.0311706 | val_loss: 0.0311712 | Time: 1698.85 ms
[2021-03-19 18:34:15 main:574] : INFO : Epoch 118 | loss: 0.0311688 | val_loss: 0.031169 | Time: 1672.42 ms
[2021-03-19 18:34:16 main:574] : INFO : Epoch 119 | loss: 0.0311683 | val_loss: 0.0311718 | Time: 1708.02 ms
[2021-03-19 18:34:18 main:574] : INFO : Epoch 120 | loss: 0.0311706 | val_loss: 0.0311727 | Time: 1723.99 ms
[2021-03-19 18:34:20 main:574] : INFO : Epoch 121 | loss: 0.0311684 | val_loss: 0.0311716 | Time: 1713.35 ms
[2021-03-19 18:34:22 main:574] : INFO : Epoch 122 | loss: 0.0311652 | val_loss: 0.0311697 | Time: 1713.81 ms
[2021-03-19 18:34:24 main:574] : INFO : Epoch 123 | loss: 0.0311643 | val_loss: 0.0311695 | Time: 1704.6 ms
[2021-03-19 18:34:25 main:574] : INFO : Epoch 124 | loss: 0.0311635 | val_loss: 0.0311693 | Time: 1664.81 ms
[2021-03-19 18:34:27 main:574] : INFO : Epoch 125 | loss: 0.031164 | val_loss: 0.0311722 | Time: 1719.74 ms
[2021-03-19 18:34:29 main:574] : INFO : Epoch 126 | loss: 0.0311643 | val_loss: 0.0311671 | Time: 1726.3 ms
[2021-03-19 18:34:31 main:574] : INFO : Epoch 127 | loss: 0.0311611 | val_loss: 0.0311694 | Time: 1702.22 ms
[2021-03-19 18:34:32 main:574] : INFO : Epoch 128 | loss: 0.0311639 | val_loss: 0.0311695 | Time: 1726.42 ms
[2021-03-19 18:34:34 main:574] : INFO : Epoch 129 | loss: 0.0311688 | val_loss: 0.0311671 | Time: 1684.97 ms
[2021-03-19 18:34:36 main:574] : INFO : Epoch 130 | loss: 0.0311694 | val_loss: 0.0311666 | Time: 1671.32 ms
[2021-03-19 18:34:37 main:574] : INFO : Epoch 131 | loss: 0.0311666 | val_loss: 0.0311711 | Time: 1706.78 ms
[2021-03-19 18:34:39 main:574] : INFO : Epoch 132 | loss: 0.0311659 | val_loss: 0.0311673 | Time: 1711.88 ms
[2021-03-19 18:34:41 main:574] : INFO : Epoch 133 | loss: 0.0311647 | val_loss: 0.0311678 | Time: 1712.69 ms
[2021-03-19 18:34:43 main:574] : INFO : Epoch 134 | loss: 0.0311643 | val_loss: 0.0311691 | Time: 1718.53 ms
[2021-03-19 18:34:44 main:574] : INFO : Epoch 135 | loss: 0.0311631 | val_loss: 0.0311649 | Time: 1668.63 ms
[2021-03-19 18:34:46 main:574] : INFO : Epoch 136 | loss: 0.0311637 | val_loss: 0.0311661 | Time: 1717.5 ms
[2021-03-19 18:34:48 main:574] : INFO : Epoch 137 | loss: 0.0311617 | val_loss: 0.0311639 | Time: 1707.19 ms
[2021-03-19 18:34:50 main:574] : INFO : Epoch 138 | loss: 0.0311637 | val_loss: 0.0311688 | Time: 1718.68 ms
[2021-03-19 18:34:51 main:574] : INFO : Epoch 139 | loss: 0.0311667 | val_loss: 0.031166 | Time: 1706.18 ms
[2021-03-19 18:34:53 main:574] : INFO : Epoch 140 | loss: 0.0311638 | val_loss: 0.031165 | Time: 1704.86 ms
[2021-03-19 18:34:55 main:574] : INFO : Epoch 141 | loss: 0.031163 | val_loss: 0.0311648 | Time: 1667.49 ms
[2021-03-19 18:34:57 main:574] : INFO : Epoch 142 | loss: 0.0311626 | val_loss: 0.031164 | Time: 1689.06 ms
[2021-03-19 18:34:58 main:574] : INFO : Epoch 143 | loss: 0.031161 | val_loss: 0.0311777 | Time: 1712.18 ms
[2021-03-19 18:35:00 main:574] : INFO : Epoch 144 | loss: 0.0311603 | val_loss: 0.0311666 | Time: 1726.81 ms
[2021-03-19 18:35:02 main:574] : INFO : Epoch 145 | loss: 0.0311578 | val_loss: 0.0311656 | Time: 1725.85 ms
[2021-03-19 18:35:04 main:574] : INFO : Epoch 146 | loss: 0.0311586 | val_loss: 0.0311652 | Time: 1704.26 ms
[2021-03-19 18:35:05 main:574] : INFO : Epoch 147 | loss: 0.0311574 | val_loss: 0.0311657 | Time: 1682.54 ms
[2021-03-19 18:35:07 main:574] : INFO : Epoch 148 | loss: 0.0311569 | val_loss: 0.0311645 | Time: 1704.44 ms
[2021-03-19 18:35:09 main:574] : INFO : Epoch 149 | loss: 0.0311563 | val_loss: 0.0311659 | Time: 1715.24 ms
[2021-03-19 18:35:11 main:574] : INFO : Epoch 150 | loss: 0.0311555 | val_loss: 0.0311638 | Time: 1706.42 ms
[2021-03-19 18:35:12 main:574] : INFO : Epoch 151 | loss: 0.0311558 | val_loss: 0.031165 | Time: 1711.64 ms
[2021-03-19 18:35:14 main:574] : INFO : Epoch 152 | loss: 0.0311559 | val_loss: 0.0311734 | Time: 1683.2 ms
[2021-03-19 18:35:16 main:574] : INFO : Epoch 153 | loss: 0.0311574 | val_loss: 0.0311697 | Time: 1707.79 ms
[2021-03-19 18:35:18 main:574] : INFO : Epoch 154 | loss: 0.0311565 | val_loss: 0.0311675 | Time: 1707.54 ms
[2021-03-19 18:35:19 main:574] : INFO : Epoch 155 | loss: 0.0311569 | val_loss: 0.0311663 | Time: 1726.15 ms
[2021-03-19 18:35:21 main:574] : INFO : Epoch 156 | loss: 0.0311571 | val_loss: 0.0311676 | Time: 1709.23 ms
[2021-03-19 18:35:23 main:574] : INFO : Epoch 157 | loss: 0.0311552 | val_loss: 0.031165 | Time: 1704.58 ms
[2021-03-19 18:35:25 main:574] : INFO : Epoch 158 | loss: 0.0311535 | val_loss: 0.0311712 | Time: 1675.19 ms
[2021-03-19 18:35:26 main:574] : INFO : Epoch 159 | loss: 0.0311558 | val_loss: 0.0311706 | Time: 1703.61 ms
[2021-03-19 18:35:28 main:574] : INFO : Epoch 160 | loss: 0.031159 | val_loss: 0.0311708 | Time: 1718.95 ms
[2021-03-19 18:35:30 main:574] : INFO : Epoch 161 | loss: 0.03116 | val_loss: 0.0311667 | Time: 1729.46 ms
[2021-03-19 18:35:32 main:574] : INFO : Epoch 162 | loss: 0.0311579 | val_loss: 0.0311727 | Time: 1704.37 ms
[2021-03-19 18:35:34 main:574] : INFO : Epoch 163 | loss: 0.031159 | val_loss: 0.0311714 | Time: 1726.53 ms
[2021-03-19 18:35:35 main:574] : INFO : Epoch 164 | loss: 0.0311683 | val_loss: 0.0311814 | Time: 1669.14 ms
[2021-03-19 18:35:37 main:574] : INFO : Epoch 165 | loss: 0.0311587 | val_loss: 0.0311666 | Time: 1709.59 ms
[2021-03-19 18:35:39 main:574] : INFO : Epoch 166 | loss: 0.0311556 | val_loss: 0.0311651 | Time: 1711.58 ms
[2021-03-19 18:35:40 main:574] : INFO : Epoch 167 | loss: 0.031154 | val_loss: 0.0311628 | Time: 1709.95 ms
[2021-03-19 18:35:42 main:574] : INFO : Epoch 168 | loss: 0.0311533 | val_loss: 0.0311637 | Time: 1704.55 ms
[2021-03-19 18:35:44 main:574] : INFO : Epoch 169 | loss: 0.031152 | val_loss: 0.0311652 | Time: 1706.65 ms
[2021-03-19 18:35:46 main:574] : INFO : Epoch 170 | loss: 0.0311514 | val_loss: 0.0311637 | Time: 1682.11 ms
[2021-03-19 18:35:47 main:574] : INFO : Epoch 171 | loss: 0.0311518 | val_loss: 0.0311652 | Time: 1706.45 ms
[2021-03-19 18:35:49 main:574] : INFO : Epoch 172 | loss: 0.0311508 | val_loss: 0.031166 | Time: 1726.19 ms
[2021-03-19 18:35:51 main:574] : INFO : Epoch 173 | loss: 0.0311515 | val_loss: 0.0311648 | Time: 1722.41 ms
[2021-03-19 18:35:53 main:574] : INFO : Epoch 174 | loss: 0.03115 | val_loss: 0.0311664 | Time: 1720.09 ms
[2021-03-19 18:35:55 main:574] : INFO : Epoch 175 | loss: 0.0311504 | val_loss: 0.0311651 | Time: 1677.99 ms
[2021-03-19 18:35:56 main:574] : INFO : Epoch 176 | loss: 0.0311513 | val_loss: 0.0311646 | Time: 1689.22 ms
[2021-03-19 18:35:58 main:574] : INFO : Epoch 177 | loss: 0.0311514 | val_loss: 0.0311637 | Time: 1744.66 ms
[2021-03-19 18:36:00 main:574] : INFO : Epoch 178 | loss: 0.0311498 | val_loss: 0.031169 | Time: 1709.27 ms
[2021-03-19 18:36:02 main:574] : INFO : Epoch 179 | loss: 0.0311535 | val_loss: 0.0311682 | Time: 1729.67 ms
[2021-03-19 18:36:03 main:574] : INFO : Epoch 180 | loss: 0.0311507 | val_loss: 0.0311631 | Time: 1696.5 ms
[2021-03-19 18:36:05 main:574] : INFO : Epoch 181 | loss: 0.0311486 | val_loss: 0.0311653 | Time: 1668.07 ms
[2021-03-19 18:36:07 main:574] : INFO : Epoch 182 | loss: 0.0311487 | val_loss: 0.0311676 | Time: 1733.1 ms
[2021-03-19 18:36:09 main:574] : INFO : Epoch 183 | loss: 0.031149 | val_loss: 0.031166 | Time: 1728.43 ms
[2021-03-19 18:36:10 main:574] : INFO : Epoch 184 | loss: 0.0311488 | val_loss: 0.031164 | Time: 1695.82 ms
[2021-03-19 18:36:12 main:574] : INFO : Epoch 185 | loss: 0.0311481 | val_loss: 0.0311634 | Time: 1746.9 ms
[2021-03-19 18:36:14 main:574] : INFO : Epoch 186 | loss: 0.0311475 | val_loss: 0.0311638 | Time: 1691.98 ms
[2021-03-19 18:36:16 main:574] : INFO : Epoch 187 | loss: 0.0311465 | val_loss: 0.0311648 | Time: 1721 ms
[2021-03-19 18:36:17 main:574] : INFO : Epoch 188 | loss: 0.0311472 | val_loss: 0.031163 | Time: 1700.51 ms
[2021-03-19 18:36:19 main:574] : INFO : Epoch 189 | loss: 0.0311471 | val_loss: 0.0311631 | Time: 1701.15 ms
[2021-03-19 18:36:21 main:574] : INFO : Epoch 190 | loss: 0.0311481 | val_loss: 0.0311622 | Time: 1721.54 ms
[2021-03-19 18:36:23 main:574] : INFO : Epoch 191 | loss: 0.0311459 | val_loss: 0.0311639 | Time: 1724.56 ms
[2021-03-19 18:36:24 main:574] : INFO : Epoch 192 | loss: 0.0311461 | val_loss: 0.0311637 | Time: 1674.65 ms
[2021-03-19 18:36:26 main:574] : INFO : Epoch 193 | loss: 0.0311495 | val_loss: 0.0311737 | Time: 1699.85 ms
[2021-03-19 18:36:28 main:574] : INFO : Epoch 194 | loss: 0.0311489 | val_loss: 0.0311696 | Time: 1714.66 ms
[2021-03-19 18:36:30 main:574] : INFO : Epoch 195 | loss: 0.031146 | val_loss: 0.0311626 | Time: 1699.66 ms
[2021-03-19 18:36:31 main:574] : INFO : Epoch 196 | loss: 0.0311459 | val_loss: 0.0311692 | Time: 1706.79 ms
[2021-03-19 18:36:33 main:574] : INFO : Epoch 197 | loss: 0.0311456 | val_loss: 0.0311651 | Time: 1699.48 ms
[2021-03-19 18:36:35 main:574] : INFO : Epoch 198 | loss: 0.0311458 | val_loss: 0.0311658 | Time: 1672.55 ms
[2021-03-19 18:36:37 main:574] : INFO : Epoch 199 | loss: 0.0311445 | val_loss: 0.0311645 | Time: 1729.01 ms
THCudaCheck FAIL file=..\..\aten\src\THC\THCCachingHostAllocator.cpp line=278 error=719 : unspecified launch failure
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFBD952D759
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.17.0
Dump Timestamp : 03/19/21 18:36:37
Install Directory : D:\Program Files\BOINC\
Data Directory : D:\ProgramData\BOINC
Project Symstore :
LoadLibraryA( D:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( D:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: D:\ProgramData\BOINC\slots\7;D:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome
ModLoad: 000000009c890000 00000000003df000 D:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000dbc30000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000db710000 00000000000bd000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.804) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.19041.804 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.804
ModLoad: 00000000d9500000 00000000002c9000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.804) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.19041.804 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.804
ModLoad: 00000000da5b0000 00000000001a0000 C:\WINDOWS\System32\USER32.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d93d0000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.867) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.19041.867 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.867
ModLoad: 00000000dac00000 000000000002a000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.19041.746 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.746
ModLoad: 00000000d9960000 000000000010b000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.19041.746 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.746
ModLoad: 00000000d9b40000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 00000000d9400000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 00000000dad00000 00000000000ac000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.610) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000daf20000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.19041.546
ModLoad: 00000000dbb50000 000000000009c000 C:\WINDOWS\System32\sechost.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000db870000 000000000012b000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d4130000 0000000000058000 D:\ProgramData\BOINC\slots\7\c10.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000c95e0000 0000000000091000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.27.29114.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\A01\_work\62\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
File Version : 14.27.29114.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.27.29114.0
ModLoad: 00000000d3750000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.19041.867 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.867
ModLoad: 00000000cab00000 0000000000019000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.27.29114.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\A01\_work\62\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
File Version : 14.27.29114.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.27.29114.0
ModLoad: 00000000d4120000 000000000000c000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.27.29114.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\A01\_work\62\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
File Version : 14.27.29114.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.27.29114.0
ModLoad: 0000000078750000 000000000abe7000 D:\ProgramData\BOINC\slots\7\torch_cpu.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000c4b60000 000000000020d000 D:\ProgramData\BOINC\slots\7\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
Linked PDB Filename : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
File Version : 20200205
Company Name : Intel Corporation
Product Name : Intel(R) OpenMP* Runtime Library
Product Version : 5.0
ModLoad: 00000000a8120000 00000000002db000 D:\ProgramData\BOINC\slots\7\fbgemm.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000d1320000 0000000000040000 D:\ProgramData\BOINC\slots\7\asmjit.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000edf00000 0000000024b49000 D:\ProgramData\BOINC\slots\7\torch_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000c9800000 0000000000045000 D:\ProgramData\BOINC\slots\7\c10_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000003faf0000 00000000030a5000 D:\ProgramData\BOINC\slots\7\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1031
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA SPARSE BLAS Library
Product Version : 6,14,11,1031
ModLoad: 00000000d41a0000 0000000000010000 D:\ProgramData\BOINC\slots\7\nvToolsExt64_1.dll (-exported- Symbols Loaded)
Linked PDB Filename : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb
ModLoad: 000000003c630000 00000000034b2000 D:\ProgramData\BOINC\slots\7\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA RAND Library
Product Version : 6,14,11,1012
ModLoad: 00000000388d0000 0000000003d5b000 D:\ProgramData\BOINC\slots\7\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 0000000036a00000 0000000001eca000 D:\ProgramData\BOINC\slots\7\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 000000002d860000 000000000919f000 D:\ProgramData\BOINC\slots\7\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA FFT Library
Product Version : 6,14,11,1012
ModLoad: 00000000bb6e0000 000000001b3a4000 D:\ProgramData\BOINC\slots\7\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,10020
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.86 CUDNN Library
Product Version : 6,14,11,10020
ModLoad: 00000000daef0000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000d72f0000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000d9cf0000 0000000000469000 C:\WINDOWS\System32\Setupapi.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : setupapi.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d9af0000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cfgmgr32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000d9930000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.662) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d9120000 000000000002c000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : devobj.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000d9370000 0000000000060000 C:\WINDOWS\System32\WINTRUST.dll (6.2.19041.804) (-exported- Symbols Loaded)
Linked PDB Filename : wintrust.pdb
File Version : 10.0.19041.804 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.804
ModLoad: 00000000d97d0000 000000000015f000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : crypt32.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d8f60000 0000000000012000 C:\WINDOWS\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msasn1.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000dafc0000 0000000000742000 C:\WINDOWS\System32\Shell32.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : shell32.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d74f0000 0000000000790000 C:\WINDOWS\SYSTEM32\windows.storage.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : Windows.Storage.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000da160000 0000000000355000 C:\WINDOWS\System32\combase.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : combase.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d8de0000 000000000002c000 C:\WINDOWS\SYSTEM32\Wldp.dll (6.2.19041.662) (-exported- Symbols Loaded)
Linked PDB Filename : WLDP.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000daac0000 00000000000ae000 C:\WINDOWS\System32\SHCORE.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : shcore.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000da7b0000 0000000000055000 C:\WINDOWS\System32\shlwapi.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : shlwapi.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000a7eb0000 0000000000267000 C:\WINDOWS\system32\nvcuda.dll (27.21.14.5671) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r456_61\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
File Version : 27.21.14.5671
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.1.96 driver
Product Version : 27.21.14.5671
ModLoad: 00000000d3ee0000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000020970000 000000000152a000 C:\WINDOWS\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_60daf66a00f2e0b6\nvcuda64.dll (27.21.14.5671) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r456_61\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
File Version : 27.21.14.5671
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.1.96 driver
Product Version : 27.21.14.5671
ModLoad: 00000000dac90000 000000000006b000 C:\WINDOWS\System32\WS2_32.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : ws2_32.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d3e40000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cryptnet.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000d8d50000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cryptbase.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000d6e30000 000000000009e000 C:\WINDOWS\system32\uxtheme.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : UxTheme.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d0bd0000 00000000006d5000 C:\WINDOWS\SYSTEM32\nvapi64.dll (27.21.14.5671) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r456_61\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
File Version : 27.21.14.5671
Company Name : NVIDIA Corporation
Product Name : NVIDIA Windows drivers
Product Version : 27.21.14.5671
ModLoad: 00000000da8e0000 000000000012a000 C:\WINDOWS\System32\OLE32.DLL (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : ole32.pdb
File Version : 10.0.19041.320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.320
ModLoad: 00000000d4110000 0000000000009000 D:\ProgramData\BOINC\slots\7\caffe2_nvrtc.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000001f9a0000 0000000000fca000 D:\ProgramData\BOINC\slots\7\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6.14.11.9000
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.89 NVRTC Library
Product Version : 6.14.11.9000
ModLoad: 00000000d9a70000 0000000000080000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.662) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.19041.662 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.662
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 3823, Write: 1040, Other 2518
- I/O Transfers Counters -
Read: 6519352, Write: 130499, Other 37326
- Paged Pool Usage -
QuotaPagedPoolUsage: 3545752, QuotaPeakPagedPoolUsage: 3545752
QuotaNonPagedPoolUsage: 49496, QuotaPeakNonPagedPoolUsage: 49904
- Virtual Memory Usage -
VirtualSize: -545746944, PeakVirtualSize: -29483008
- Pagefile Usage -
PagefileUsage: -545746944, PeakPagefileUsage: -545304576
- Working Set Size -
WorkingSetSize: 1667870720, PeakWorkingSetSize: 1670770688, PageFaultCount: 575441
*** Dump of thread ID 14604 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 475312512.000000, User Time: 1629062528.000000, Wait Time: 53621592.000000
- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFBD952D759
- Registers -
rax=0000000000000000 rbx=0000000082bc62f0 rcx=0000000000000000 rdx=0000000000000000 rsi=000000006d4e3e60 rdi=0000000019930520
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000006000 r13=0000000000000000
r14=000000006d4e4ba0 r15=000000006d4e4b20 rip=00000000d952d759 rsp=000000006d4e3d00 rbp=000000006d4e4708
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
- Callstack -
ChildEBP RetAddr Args to Child
6d4e3dd0 cab06220 ffd48bb3 00000000 6d4e4ba0 7dd1a014 KERNELBASE!RaiseException+0x0
6d4e3e30 7e315272 ffd48bd3 00000000 000007cb 00000116 VCRUNTIME140!_CxxThrowException+0x0
6d4e3e80 7e3156d4 00000000 00000000 8133d290 f1765f68 torch_cpu!at::Tensor::zero_+0x0
6d4e46e0 f0d29ebe 6d4e4910 f1766850 f1766880 000002cf torch_cpu!_THError+0x0
6d4e4720 f0d25412 6d4e4910 6d4e4870 9b488040 f1b9a720 torch_cuda!__THCudaCheck+0x0
6d4e4760 7dfc59ff 6d4e4b00 6d4e4910 6d4e4de0 d41697b0 torch_cuda!at::native::set_storage_cuda_+0x0
6d4e4af0 7e16726f f139f630 7dd1ea63 00000001 d414ef8f torch_cpu!at::native::empty_cpu+0x0
6d4e4b30 7e120615 6d4e0000 6d4e4ba8 6d4e4da0 d4137ce6 torch_cpu!at::native::mkldnn_sigmoid_+0x0
6d4e4b70 7e12204a 6d4e0000 6d4e4c00 9b469500 f0066ae5 torch_cpu!at::native::mkldnn_sigmoid_+0x0
6d4e4c00 7e120615 00000000 7dd1ea63 6d4e4ce0 6d4e4de0 torch_cpu!at::native::mkldnn_sigmoid_+0x0
6d4e4c40 7e1f1eb0 6d4e0000 00000000 6d4e4dc8 00000001 torch_cpu!at::native::mkldnn_sigmoid_+0x0
6d4e4ce0 7e202eec 39226d40 d4169c50 6d4e4e40 d4133bd6 torch_cpu!at::bucketize_out+0x0
6d4e4d30 f042f160 6d4e0000 00000000 6d4e4e40 00000000 torch_cpu!at::empty+0x0
6d4e50d0 f0432de8 d4169c50 7dd20528 9c3700d0 d4169c50 torch_cuda!at::native::cummin_helper_cuda+0x0
6d4e52b0 f0434379 6d4e53c0 14662a30 6d4e57c0 14662a48 torch_cuda!at::native::cummin_helper_cuda+0x0
6d4e54f0 f0433d64 6d4e0000 6d4e57c0 6d4e5600 6d4e57c0 torch_cuda!at::native::cat_out_cuda+0x0
6d4e56f0 f0cd1dd2 6d4e5748 00000001 6d4e5748 6d4e57c0 torch_cuda!at::native::cat_cuda+0x0
6d4e5770 f0ce4179 00000001 6d4e57b0 9c36e868 04000000 torch_cuda!at::native::set_storage_cuda_+0x0
6d4e57b0 7e1f1d7d 997411a0 6d4e5b20 6d4e5af0 00000000 torch_cuda!at::native::set_storage_cuda_+0x0
6d4e5830 7e1d0d81 00000000 d4153994 6d4e6000 14661bb0 torch_cpu!at::bucketize_out+0x0
6d4e5870 7dfdb4a5 6d4e58b0 9c36fcd0 6d4e5980 6d4e59f0 torch_cpu!at::_cat+0x0
6d4e5940 7e2c27bd 6d4e5a10 00000101 6d4e5998 d4181ea8 torch_cpu!at::native::cat+0x0
6d4e59c0 7dd829ae 146627d0 146627e0 146627e8 6d4e5978 torch_cpu!at::zeros_out+0x0
6d4e5a00 7e1f1d7d 6d4e5a40 6d4e5ab0 00000000 dbc547b1 torch_cpu!torch::nn::functional::BatchNormFuncOptions::~BatchNormFuncOptions+0x0
6d4e5a80 7e1f75d1 6d4e6020 83070640 6d4e6000 14661bb0 torch_cpu!at::bucketize_out+0x0
6d4e5ac0 7dfe7d4c 6d4e5d50 00000000 6d4e5d50 00000000 torch_cpu!at::cat+0x0
6d4e5c80 7e2dc38d 6d4e5cd8 00000001 6d4e5cd8 6d4e5d50 torch_cpu!at::native::stack+0x0
6d4e5d00 7dd829ae 6d4e6000 14661bb0 00000000 00000000 torch_cpu!at::zeros_out+0x0
6d4e5d40 7e1f1d7d 6d4e5d80 6d4e5df0 00000000 80f91c8f torch_cpu!torch::nn::functional::BatchNormFuncOptions::~BatchNormFuncOptions+0x0
6d4e5dc0 7e2402e1 6d4e5ef0 d4153720 6d4e6020 83070640 torch_cpu!at::bucketize_out+0x0
6d4e5e00 7f5d3839 6d4e5e80 9c36bfd0 9c36bfd0 6d4e5f10 torch_cpu!at::stack+0x0
6d4e5fd0 7dd829ae 00000000 6d4e6120 00000000 6d4e60c0 torch_cpu!torch::autograd::GraphRoot::apply+0x0
6d4e6010 7e1f1d7d 6d4e6050 6d4e60c0 9c371520 00000000 torch_cpu!torch::nn::functional::BatchNormFuncOptions::~BatchNormFuncOptions+0x0
6d4e6090 7e2402e1 6d4e6200 7dd19cac 0f8e82d0 14661bb0 torch_cpu!at::bucketize_out+0x0
6d4e60d0 9c89ff7e 6d4e6198 99b90320 99b90320 6d4e62e0 torch_cpu!at::stack+0x0
6d4e6170 9c8a2ef8 9c89fe30 6d4e61b0 6d4e61b0 6d4e6200 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::addCallback+0x0
6d4e61d0 9c8a3ea4 f65916b0 dbc547b1 6d4e6370 9a920000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Object::getSlot+0x0
6d4e6310 9c89d3d2 9cf1c6e0 1467c110 6d4e6370 dbc55ba1 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::markCompleted+0x0
6d4e6340 9c8a4541 00000000 14681dc0 10354860 dbc55ba1 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Object::operator=+0x0
6d4e64f0 9c8a743a 6d4e65c0 00000000 6d4e7000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::markCompleted+0x0
6d4e69f0 9c8ab950 6d4e6af0 6d4e6af0 00000000 c2aaafb0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0
6d4ff790 9caec6f8 9c3e0610 00000000 00000000 9c42aa80 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
6d4ff7d0 db727034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
6d4ff800 dbc82651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
6d4ff880 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 3644 (state: Waiting): ***
- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 53621592.000000
- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=000000006d9ff850 rsi=0000000000000000 rdi=0000000000000064
r8=000000006d9ff828 r9=0000000000000000 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000dbccd294 rsp=000000006d9ff828 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
6d9ff820 d954d84e 6d9ff8e8 00000000 00000000 00000000 ntdll!ZwDelayExecution+0x0
6d9ff8c0 9c8ca07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0
6d9ff8f0 db727034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
6d9ff920 dbc82651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
6d9ff9a0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 10036 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 53621592.000000
- Registers -
rax=000000000000005b rbx=000000000000000b rcx=000000000000000b rdx=000000006daffa90 rsi=0000000000000000 rdi=000000000000000b
r8=000000009cabe9c0 r9=000000009c5f7a30 r10=0000000000000000 r11=000000006daffa00 r12=0000000000000064 r13=000000006daffa90
r14=000000006daff790 r15=0000000000000000 rip=00000000dbccd764 rsp=000000006daff738 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
6daff730 d9550d40 00000000 6d38f000 dbd9a4d0 6d38f000 ntdll!ZwWaitForMultipleObjects+0x0
6daffa20 d9550c3e 9a932d80 004e0049 0000000b 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0
6daffa60 20b1ba44 9c5faa90 00000000 00000000 00000000 KERNELBASE!WaitForMultipleObjects+0x0
6daffca0 20bb08dd 00000000 00000004 00000000 00000000 nvcuda64!+0x0
6daffd50 20b1b413 be5deea5 be23e243 9caf4fb0 00000000 nvcuda64!cuProfilerStop+0x0
6daffd80 20e29200 9caf4fb0 00000000 00000000 00000000 nvcuda64!+0x0
6daffdb0 db727034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
6daffde0 dbc82651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
6daffe60 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 13080 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 53599068.000000
- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=00000000000002c4 rdx=0000000000000000 rsi=0000000000000000 rdi=00000000000002c4
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=000000009c5e0170 r13=0000000000000000
r14=00000000000002c4 r15=0000000000000000 rip=00000000dbcccc94 rsp=000000006dbffd38 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
6dbffd30 d951a34e 00000000 00000000 00000000 00000000 ntdll!NtWaitForSingleObject+0x0
6dbffdd0 20d2a9f8 00000000 d9522cb5 00000000 000002c4 KERNELBASE!WaitForSingleObjectEx+0x0
6dbffe40 20b1b413 00000001 9d0837d0 9caf57f0 00000000 nvcuda64!cuProfilerStop+0x0
6dbffe70 20e29200 9caf57f0 00000000 00000000 00000000 nvcuda64!+0x0
6dbffea0 db727034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
6dbffed0 dbc82651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
6dbfff50 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 13396 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 53621572.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=000000009c862258 rdx=000000006dcff908 rsi=000000006dcff870 rdi=0000000000000000
r8=0000000000000001 r9=0000000000003454 r10=000000009c8669d8 r11=00000000f4041465 r12=000000006dcff908 r13=0000000000000000
r14=000000009c862258 r15=000000009c8669d8 rip=00000000dbcd0604 rsp=000000006dcff848 rbp=000000006dcff8a0
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
6dcff840 dbc94295 00000102 00000102 00000000 6dcff870 ntdll!ZwWaitForAlertByThreadId+0x0
6dcff8b0 d956b9d9 9c8641c0 00000000 00000000 00000000 ntdll!RtlSleepConditionVariableCS+0x0
6dcff8e0 20aa21da 9c8641c0 dbc807b0 9c8641c0 ffb3b4c0 KERNELBASE!SleepConditionVariableCS+0x0
6dcff940 20b1b413 00000001 be431219 00000000 00000000 nvcuda64!+0x0
6dcff970 20e29200 9caf50a0 00000000 00000000 00000000 nvcuda64!+0x0
6dcff9a0 db727034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
6dcff9d0 dbc82651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
6dcffa50 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 11072 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 88125000.000000, User Time: 1219843712.000000, Wait Time: 53621592.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=0000000095c92898 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000095c92898
r8=0000000081227a38 r9=000000009b1e7b98 r10=0000000000000038 r11=000000006ddfc230 r12=0000000000000000 r13=0000000000000000
r14=000000006ddff530 r15=0000000095c92848 rip=00000000dbcd0604 rsp=000000006ddff508 rbp=000000006ddff560
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
6ddff500 dbc94021 99b63a78 99b63a78 6ddff640 99b63a78 ntdll!ZwWaitForAlertByThreadId+0x0
6ddff580 d956ba89 95c92840 95c92888 00000000 00000000 ntdll!RtlSleepConditionVariableSRW+0x0
6ddff5c0 c961ba8d 97c00010 97c00010 97c00010 00000000 KERNELBASE!SleepConditionVariableSRW+0x0
6ddff600 c95f2848 997e3130 00000000 00000000 7f9305b4 MSVCP140!__crtSetThreadpoolWait+0x0
6ddff630 c95f2710 00000000 c95f2f17 00000000 997e3130 MSVCP140!std::_Winerror_message+0x0
6ddff6a0 7f93064b 997e3110 000001c0 95c92840 95c92840 MSVCP140!std::_Winerror_message+0x0
6ddff6e0 7f9325b1 1452a1c0 ffffffff 997e3110 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0
6ddff9d0 7f932471 82e56d10 6ddffa10 82e56d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0
6ddffa20 7f929744 00080001 12195920 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0
6ddffa50 d9421bb2 12195920 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0
6ddffa80 db727034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0
6ddffab0 dbc82651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
6ddffb30 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)