Task 10954113
| Name | ParityModified-1639957496-17006-2-0_0 |
| Workunit | 8247535 |
| Created | 22 Jan 2022, 12:09:40 UTC |
| Sent | 6 Feb 2022, 13:50:53 UTC |
| Report deadline | 14 Feb 2022, 13:50:53 UTC |
| Received | 2 Mar 2022, 8:07:16 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | -529697949 (0xE06D7363) Unknown error code |
| Computer ID | 11317 |
| Run time | 14 min 34 sec |
| CPU time | 11 min 58 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 6,856.70 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.62 GB |
| Peak swap size | 3.63 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.20</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 3765269347 (0xe06d7363)</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1070)
[2022-02-06 21:32:21 main:435] : INFO : Set logging level to 1
[2022-02-06 21:32:21 main:441] : INFO : Running in BOINC Client mode
[2022-02-06 21:32:21 main:444] : INFO : Resolving all filenames
[2022-02-06 21:32:21 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-06 21:32:21 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-02-06 21:32:22 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-06 21:32:22 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-02-06 21:32:22 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-02-06 21:32:22 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-02-06 21:32:22 main:474] : INFO : Configuration:
[2022-02-06 21:32:22 main:475] : INFO : Model type: GRU
[2022-02-06 21:32:22 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-02-06 21:32:22 main:477] : INFO : Max Epochs: 2048
[2022-02-06 21:32:22 main:478] : INFO : Batch Size: 128
[2022-02-06 21:32:22 main:479] : INFO : Learning Rate: 0.01
[2022-02-06 21:32:22 main:480] : INFO : Patience: 10
[2022-02-06 21:32:22 main:481] : INFO : Hidden Width: 12
[2022-02-06 21:32:22 main:482] : INFO : # Recurrent Layers: 4
[2022-02-06 21:32:22 main:483] : INFO : # Backend Layers: 4
[2022-02-06 21:32:22 main:484] : INFO : # Threads: 1
[2022-02-06 21:32:22 main:486] : INFO : Preparing Dataset
[2022-02-06 21:32:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-06 21:32:23 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-06 21:32:24 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-02-06 21:32:24 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-06 21:32:24 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-06 21:32:25 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-02-06 21:32:25 main:494] : INFO : Creating Model
[2022-02-06 21:32:25 main:507] : INFO : Preparing config file
[2022-02-06 21:32:25 main:519] : INFO : Creating new config file
[2022-02-06 21:32:25 main:538] : INFO : This is a continuation WU, loading previous network
[2022-02-06 21:32:26 main:559] : INFO : Loading DataLoader into Memory
[2022-02-06 21:32:26 main:562] : INFO : Starting Training
[2022-02-06 21:32:31 main:574] : INFO : Epoch 1 | loss: 0.0312639 | val_loss: 0.0312513 | Time: 5509.85 ms
[2022-02-06 21:32:37 main:574] : INFO : Epoch 2 | loss: 0.0312467 | val_loss: 0.0312408 | Time: 5550.59 ms
[2022-02-06 21:32:43 main:574] : INFO : Epoch 3 | loss: 0.0312404 | val_loss: 0.0312342 | Time: 5630.61 ms
[2022-02-06 21:32:48 main:574] : INFO : Epoch 4 | loss: 0.0312318 | val_loss: 0.0312324 | Time: 5563.29 ms
[2022-02-06 21:32:54 main:574] : INFO : Epoch 5 | loss: 0.0312293 | val_loss: 0.0312306 | Time: 5493.37 ms
[2022-02-06 21:32:59 main:574] : INFO : Epoch 6 | loss: 0.0312246 | val_loss: 0.0312239 | Time: 5327.66 ms
[2022-02-06 21:33:05 main:574] : INFO : Epoch 7 | loss: 0.0312218 | val_loss: 0.0312202 | Time: 5335.35 ms
[2022-02-06 21:33:10 main:574] : INFO : Epoch 8 | loss: 0.0312183 | val_loss: 0.0312183 | Time: 5616.43 ms
[2022-02-06 21:33:16 main:574] : INFO : Epoch 9 | loss: 0.0312165 | val_loss: 0.0312137 | Time: 5361.32 ms
[2022-02-06 21:33:21 main:574] : INFO : Epoch 10 | loss: 0.0312142 | val_loss: 0.0312119 | Time: 5453.89 ms
[2022-02-06 21:33:27 main:574] : INFO : Epoch 11 | loss: 0.0312128 | val_loss: 0.0312171 | Time: 5598.18 ms
[2022-02-06 21:33:32 main:574] : INFO : Epoch 12 | loss: 0.0312129 | val_loss: 0.0312117 | Time: 5404.77 ms
[2022-02-06 21:33:38 main:574] : INFO : Epoch 13 | loss: 0.0312143 | val_loss: 0.0312104 | Time: 5470.88 ms
[2022-02-06 21:33:43 main:574] : INFO : Epoch 14 | loss: 0.0312099 | val_loss: 0.0312092 | Time: 5583.07 ms
[2022-02-06 21:33:49 main:574] : INFO : Epoch 15 | loss: 0.0312083 | val_loss: 0.0312075 | Time: 5476.61 ms
[2022-02-06 21:33:54 main:574] : INFO : Epoch 16 | loss: 0.0312088 | val_loss: 0.031214 | Time: 5390.67 ms
[2022-02-06 21:34:00 main:574] : INFO : Epoch 17 | loss: 0.0312097 | val_loss: 0.0312084 | Time: 5676.28 ms
[2022-02-06 21:34:06 main:574] : INFO : Epoch 18 | loss: 0.031207 | val_loss: 0.0312051 | Time: 5523.22 ms
[2022-02-06 21:34:11 main:574] : INFO : Epoch 19 | loss: 0.0312067 | val_loss: 0.0312093 | Time: 5433.78 ms
[2022-02-06 21:34:17 main:574] : INFO : Epoch 20 | loss: 0.0312112 | val_loss: 0.0312043 | Time: 5607.25 ms
[2022-02-06 21:34:22 main:574] : INFO : Epoch 21 | loss: 0.0312072 | val_loss: 0.0312154 | Time: 5421.74 ms
[2022-02-06 21:34:28 main:574] : INFO : Epoch 22 | loss: 0.031209 | val_loss: 0.0312056 | Time: 5518.17 ms
[2022-02-06 21:34:34 main:574] : INFO : Epoch 23 | loss: 0.031205 | val_loss: 0.0312049 | Time: 5771.72 ms
[2022-02-06 21:34:39 main:574] : INFO : Epoch 24 | loss: 0.0312042 | val_loss: 0.0312035 | Time: 5421.64 ms
[2022-02-06 21:34:45 main:574] : INFO : Epoch 25 | loss: 0.0312035 | val_loss: 0.0312061 | Time: 5504.77 ms
[2022-02-06 21:34:50 main:574] : INFO : Epoch 26 | loss: 0.0312055 | val_loss: 0.0312038 | Time: 5354.72 ms
[2022-02-06 21:34:56 main:574] : INFO : Epoch 27 | loss: 0.0312033 | val_loss: 0.0312031 | Time: 5650.28 ms
[2022-02-06 21:35:01 main:574] : INFO : Epoch 28 | loss: 0.0312043 | val_loss: 0.0312072 | Time: 5465.63 ms
[2022-02-06 21:35:07 main:574] : INFO : Epoch 29 | loss: 0.0312035 | val_loss: 0.0312055 | Time: 5500.87 ms
[2022-02-06 21:35:13 main:574] : INFO : Epoch 30 | loss: 0.0312047 | val_loss: 0.0312109 | Time: 5556.04 ms
[2022-02-06 21:35:18 main:574] : INFO : Epoch 31 | loss: 0.0312042 | val_loss: 0.0312089 | Time: 5449.65 ms
[2022-02-06 21:35:24 main:574] : INFO : Epoch 32 | loss: 0.031205 | val_loss: 0.0312032 | Time: 5481.47 ms
[2022-02-06 21:35:29 main:574] : INFO : Epoch 33 | loss: 0.0312022 | val_loss: 0.0312035 | Time: 5682.45 ms
[2022-02-06 21:35:35 main:574] : INFO : Epoch 34 | loss: 0.031203 | val_loss: 0.0312041 | Time: 5368.5 ms
[2022-02-06 21:35:41 main:574] : INFO : Epoch 35 | loss: 0.0312027 | val_loss: 0.0312032 | Time: 5621.93 ms
[2022-02-06 21:35:46 main:574] : INFO : Epoch 36 | loss: 0.0312027 | val_loss: 0.0312029 | Time: 5589.59 ms
[2022-02-06 21:35:52 main:574] : INFO : Epoch 37 | loss: 0.0312027 | val_loss: 0.0312109 | Time: 5410.27 ms
[2022-02-06 21:35:57 main:574] : INFO : Epoch 38 | loss: 0.0312062 | val_loss: 0.0312061 | Time: 5584.79 ms
[2022-02-06 21:36:03 main:574] : INFO : Epoch 39 | loss: 0.0312029 | val_loss: 0.0312049 | Time: 5631.59 ms
[2022-02-06 21:36:09 main:574] : INFO : Epoch 40 | loss: 0.0312013 | val_loss: 0.0312045 | Time: 5505.28 ms
[2022-02-06 21:36:14 main:574] : INFO : Epoch 41 | loss: 0.0312046 | val_loss: 0.0312081 | Time: 5467.67 ms
[2022-02-06 21:36:20 main:574] : INFO : Epoch 42 | loss: 0.0312051 | val_loss: 0.0312127 | Time: 5393.23 ms
[2022-02-06 21:36:25 main:574] : INFO : Epoch 43 | loss: 0.0312055 | val_loss: 0.031203 | Time: 5636.41 ms
[2022-02-06 21:36:31 main:574] : INFO : Epoch 44 | loss: 0.0312015 | val_loss: 0.0312057 | Time: 5362.52 ms
[2022-02-06 21:36:36 main:574] : INFO : Epoch 45 | loss: 0.0312042 | val_loss: 0.0312046 | Time: 5505.94 ms
[2022-02-06 21:36:42 main:574] : INFO : Epoch 46 | loss: 0.0312022 | val_loss: 0.0312062 | Time: 5509.94 ms
[2022-02-06 21:36:48 main:574] : INFO : Epoch 47 | loss: 0.0312005 | val_loss: 0.0312026 | Time: 5649.35 ms
[2022-02-06 21:36:53 main:574] : INFO : Epoch 48 | loss: 0.0312011 | val_loss: 0.031204 | Time: 5343.72 ms
[2022-02-06 21:36:59 main:574] : INFO : Epoch 49 | loss: 0.0312012 | val_loss: 0.0312034 | Time: 5547.95 ms
[2022-02-06 21:37:04 main:574] : INFO : Epoch 50 | loss: 0.0312011 | val_loss: 0.0312049 | Time: 5699.04 ms
[2022-02-06 21:37:10 main:574] : INFO : Epoch 51 | loss: 0.0312023 | val_loss: 0.0312054 | Time: 5326.94 ms
[2022-02-06 21:37:15 main:574] : INFO : Epoch 52 | loss: 0.031203 | val_loss: 0.0312054 | Time: 5384.84 ms
[2022-02-06 21:37:21 main:574] : INFO : Epoch 53 | loss: 0.0311998 | val_loss: 0.0312027 | Time: 5640.99 ms
[2022-02-06 21:37:27 main:574] : INFO : Epoch 54 | loss: 0.0312007 | val_loss: 0.0312051 | Time: 5563.29 ms
[2022-02-06 21:37:32 main:574] : INFO : Epoch 55 | loss: 0.0312018 | val_loss: 0.0312026 | Time: 5556.74 ms
[2022-02-06 21:37:38 main:574] : INFO : Epoch 56 | loss: 0.0312031 | val_loss: 0.0312071 | Time: 5582.71 ms
[2022-02-06 21:37:43 main:574] : INFO : Epoch 57 | loss: 0.0312015 | val_loss: 0.031213 | Time: 5523.33 ms
[2022-02-06 21:37:49 main:574] : INFO : Epoch 58 | loss: 0.0312028 | val_loss: 0.031203 | Time: 5570.67 ms
[2022-02-06 21:37:55 main:574] : INFO : Epoch 59 | loss: 0.0311997 | val_loss: 0.031203 | Time: 5552.58 ms
[2022-02-06 21:38:00 main:574] : INFO : Epoch 60 | loss: 0.0311988 | val_loss: 0.0312018 | Time: 5465.33 ms
[2022-02-06 21:38:06 main:574] : INFO : Epoch 61 | loss: 0.0311984 | val_loss: 0.0312015 | Time: 5655.27 ms
[2022-02-06 21:38:11 main:574] : INFO : Epoch 62 | loss: 0.0311989 | val_loss: 0.031203 | Time: 5469.76 ms
[2022-02-06 21:38:17 main:574] : INFO : Epoch 63 | loss: 0.0311988 | val_loss: 0.0312058 | Time: 5519.8 ms
[2022-02-06 21:38:23 main:574] : INFO : Epoch 64 | loss: 0.031199 | val_loss: 0.0312023 | Time: 5627.8 ms
[2022-02-06 21:38:28 main:574] : INFO : Epoch 65 | loss: 0.0311984 | val_loss: 0.0312035 | Time: 5504.67 ms
[2022-02-06 21:38:34 main:574] : INFO : Epoch 66 | loss: 0.0311982 | val_loss: 0.0312028 | Time: 5376.23 ms
[2022-02-06 21:38:39 main:574] : INFO : Epoch 67 | loss: 0.0311989 | val_loss: 0.0312027 | Time: 5397.63 ms
[2022-02-06 21:38:45 main:574] : INFO : Epoch 68 | loss: 0.0311996 | val_loss: 0.0312042 | Time: 5631 ms
[2022-02-06 21:38:50 main:574] : INFO : Epoch 69 | loss: 0.0312006 | val_loss: 0.031206 | Time: 5281.8 ms
[2022-02-06 21:38:56 main:574] : INFO : Epoch 70 | loss: 0.0311997 | val_loss: 0.0312066 | Time: 5553.15 ms
[2022-02-06 21:39:01 main:574] : INFO : Epoch 71 | loss: 0.0311995 | val_loss: 0.031203 | Time: 5546.78 ms
[2022-02-06 21:39:07 main:574] : INFO : Epoch 72 | loss: 0.0311987 | val_loss: 0.031204 | Time: 5595.34 ms
[2022-02-06 21:39:13 main:574] : INFO : Epoch 73 | loss: 0.0311974 | val_loss: 0.0312045 | Time: 5499.68 ms
[2022-02-06 21:39:18 main:574] : INFO : Epoch 74 | loss: 0.0312005 | val_loss: 0.0312122 | Time: 5409.87 ms
[2022-02-06 21:39:24 main:574] : INFO : Epoch 75 | loss: 0.031198 | val_loss: 0.0312054 | Time: 5508.87 ms
[2022-02-06 21:39:29 main:574] : INFO : Epoch 76 | loss: 0.0311981 | val_loss: 0.0312038 | Time: 5589.05 ms
[2022-02-06 21:39:35 main:574] : INFO : Epoch 77 | loss: 0.0311983 | val_loss: 0.031204 | Time: 5293.82 ms
[2022-02-06 21:39:40 main:574] : INFO : Epoch 78 | loss: 0.0311979 | val_loss: 0.0312027 | Time: 5540.2 ms
[2022-02-06 21:39:46 main:574] : INFO : Epoch 79 | loss: 0.031197 | val_loss: 0.0312028 | Time: 5721.47 ms
[2022-02-06 21:39:51 main:574] : INFO : Epoch 80 | loss: 0.0311978 | val_loss: 0.0312047 | Time: 5366.33 ms
[2022-02-06 21:39:57 main:574] : INFO : Epoch 81 | loss: 0.031198 | val_loss: 0.0312034 | Time: 5480.9 ms
[2022-02-06 21:40:03 main:574] : INFO : Epoch 82 | loss: 0.0312018 | val_loss: 0.0312135 | Time: 5624.23 ms
[2022-02-06 21:40:08 main:574] : INFO : Epoch 83 | loss: 0.0312086 | val_loss: 0.031216 | Time: 5520.85 ms
[2022-02-06 21:40:14 main:574] : INFO : Epoch 84 | loss: 0.0312023 | val_loss: 0.0312082 | Time: 5352.03 ms
[2022-02-06 21:40:19 main:574] : INFO : Epoch 85 | loss: 0.0311985 | val_loss: 0.0312031 | Time: 5501.55 ms
[2022-02-06 21:40:25 main:574] : INFO : Epoch 86 | loss: 0.0311973 | val_loss: 0.0312038 | Time: 5667.15 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1070)
[2022-02-06 21:48:56 main:435] : INFO : Set logging level to 1
[2022-02-06 21:48:56 main:441] : INFO : Running in BOINC Client mode
[2022-02-06 21:48:56 main:444] : INFO : Resolving all filenames
[2022-02-06 21:48:57 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-06 21:48:57 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1)
[2022-02-06 21:48:57 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-06 21:48:57 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-02-06 21:48:57 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-02-06 21:48:57 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-02-06 21:48:57 main:474] : INFO : Configuration:
[2022-02-06 21:48:57 main:475] : INFO : Model type: GRU
[2022-02-06 21:48:57 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-02-06 21:48:57 main:477] : INFO : Max Epochs: 2048
[2022-02-06 21:48:57 main:478] : INFO : Batch Size: 128
[2022-02-06 21:48:57 main:479] : INFO : Learning Rate: 0.01
[2022-02-06 21:48:57 main:480] : INFO : Patience: 10
[2022-02-06 21:48:57 main:481] : INFO : Hidden Width: 12
[2022-02-06 21:48:57 main:482] : INFO : # Recurrent Layers: 4
[2022-02-06 21:48:57 main:483] : INFO : # Backend Layers: 4
[2022-02-06 21:48:57 main:484] : INFO : # Threads: 1
[2022-02-06 21:48:57 main:486] : INFO : Preparing Dataset
[2022-02-06 21:48:57 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-06 21:48:58 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-06 21:48:59 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-02-06 21:48:59 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-06 21:49:00 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-06 21:49:00 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-02-06 21:49:00 main:494] : INFO : Creating Model
[2022-02-06 21:49:00 main:507] : INFO : Preparing config file
[2022-02-06 21:49:00 main:511] : INFO : Found checkpoint, attempting to load...
[2022-02-06 21:49:00 main:512] : INFO : Loading config
[2022-02-06 21:49:00 main:514] : INFO : Loading state
[2022-02-06 21:49:01 main:559] : INFO : Loading DataLoader into Memory
[2022-02-06 21:49:01 main:562] : INFO : Starting Training
[2022-02-06 21:49:08 main:574] : INFO : Epoch 73 | loss: 0.0312806 | val_loss: 0.0312261 | Time: 6531.63 ms
[2022-02-06 21:49:14 main:574] : INFO : Epoch 74 | loss: 0.0312106 | val_loss: 0.0312048 | Time: 5973.35 ms
[2022-02-06 21:49:20 main:574] : INFO : Epoch 75 | loss: 0.0312032 | val_loss: 0.0311981 | Time: 5974.92 ms
[2022-02-06 21:49:26 main:574] : INFO : Epoch 76 | loss: 0.0312013 | val_loss: 0.031198 | Time: 6198.9 ms
[2022-02-06 21:49:32 main:574] : INFO : Epoch 77 | loss: 0.0312005 | val_loss: 0.0311993 | Time: 6020.98 ms
[2022-02-06 21:49:39 main:574] : INFO : Epoch 78 | loss: 0.0312023 | val_loss: 0.0311975 | Time: 6224.58 ms
[2022-02-06 21:49:45 main:574] : INFO : Epoch 79 | loss: 0.0312019 | val_loss: 0.031198 | Time: 5947.5 ms
[2022-02-06 21:49:51 main:574] : INFO : Epoch 80 | loss: 0.0311983 | val_loss: 0.0311973 | Time: 6024.4 ms
[2022-02-06 21:49:57 main:574] : INFO : Epoch 81 | loss: 0.0311994 | val_loss: 0.0311998 | Time: 6356.88 ms
[2022-02-06 21:50:03 main:574] : INFO : Epoch 82 | loss: 0.0312 | val_loss: 0.0311998 | Time: 6079.66 ms
[2022-02-06 21:50:10 main:574] : INFO : Epoch 83 | loss: 0.0312011 | val_loss: 0.0311982 | Time: 6071.06 ms
[2022-02-06 21:50:16 main:574] : INFO : Epoch 84 | loss: 0.0311998 | val_loss: 0.0311996 | Time: 6116.4 ms
[2022-02-06 21:50:22 main:574] : INFO : Epoch 85 | loss: 0.0312001 | val_loss: 0.0311999 | Time: 5994.68 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1070)
[2022-02-23 16:54:00 main:435] : INFO : Set logging level to 1
[2022-02-23 16:54:00 main:441] : INFO : Running in BOINC Client mode
[2022-02-23 16:54:00 main:444] : INFO : Resolving all filenames
[2022-02-23 16:54:00 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-23 16:54:00 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1)
[2022-02-23 16:54:00 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-23 16:54:00 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-02-23 16:54:01 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-02-23 16:54:01 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-02-23 16:54:01 main:474] : INFO : Configuration:
[2022-02-23 16:54:01 main:475] : INFO : Model type: GRU
[2022-02-23 16:54:01 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-02-23 16:54:01 main:477] : INFO : Max Epochs: 2048
[2022-02-23 16:54:01 main:478] : INFO : Batch Size: 128
[2022-02-23 16:54:01 main:479] : INFO : Learning Rate: 0.01
[2022-02-23 16:54:01 main:480] : INFO : Patience: 10
[2022-02-23 16:54:01 main:481] : INFO : Hidden Width: 12
[2022-02-23 16:54:01 main:482] : INFO : # Recurrent Layers: 4
[2022-02-23 16:54:01 main:483] : INFO : # Backend Layers: 4
[2022-02-23 16:54:02 main:484] : INFO : # Threads: 1
[2022-02-23 16:54:02 main:486] : INFO : Preparing Dataset
[2022-02-23 16:54:02 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-23 16:54:02 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-23 16:54:11 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-02-23 16:54:12 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-23 16:54:12 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-23 16:54:12 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-02-23 16:54:12 main:494] : INFO : Creating Model
[2022-02-23 16:54:13 main:507] : INFO : Preparing config file
[2022-02-23 16:54:13 main:511] : INFO : Found checkpoint, attempting to load...
[2022-02-23 16:54:13 main:512] : INFO : Loading config
[2022-02-23 16:54:13 main:514] : INFO : Loading state
[2022-02-23 16:54:27 main:559] : INFO : Loading DataLoader into Memory
[2022-02-23 16:54:28 main:562] : INFO : Starting Training
[2022-02-23 16:54:40 main:574] : INFO : Epoch 73 | loss: 0.0312597 | val_loss: 0.0312132 | Time: 11847.5 ms
[2022-02-23 16:54:45 main:574] : INFO : Epoch 74 | loss: 0.0312113 | val_loss: 0.0312055 | Time: 5787.74 ms
[2022-02-23 16:55:01 main:574] : INFO : Epoch 75 | loss: 0.0312059 | val_loss: 0.0312018 | Time: 15130 ms
[2022-02-23 17:15:52 main:574] : INFO : Epoch 76 | loss: 0.0312075 | val_loss: 0.0312082 | Time: 1.25075e+06 ms
[2022-02-23 17:15:58 main:574] : INFO : Epoch 77 | loss: 0.0312032 | val_loss: 0.0311985 | Time: 5905.95 ms
[2022-02-23 17:16:04 main:574] : INFO : Epoch 78 | loss: 0.0312014 | val_loss: 0.0312013 | Time: 5798.84 ms
[2022-02-23 17:16:10 main:574] : INFO : Epoch 79 | loss: 0.0312026 | val_loss: 0.0312045 | Time: 5997.63 ms
[2022-02-23 17:16:16 main:574] : INFO : Epoch 80 | loss: 0.0312098 | val_loss: 0.0312164 | Time: 5802.3 ms
[2022-02-23 17:16:21 main:574] : INFO : Epoch 81 | loss: 0.0312198 | val_loss: 0.0312216 | Time: 5902.32 ms
[2022-02-23 17:16:28 main:574] : INFO : Epoch 82 | loss: 0.0312228 | val_loss: 0.0312237 | Time: 6007.39 ms
[2022-02-23 17:16:34 main:574] : INFO : Epoch 83 | loss: 0.0312218 | val_loss: 0.0312252 | Time: 5930.7 ms
[2022-02-23 17:16:40 main:574] : INFO : Epoch 84 | loss: 0.0312227 | val_loss: 0.0312223 | Time: 5974.68 ms
[2022-02-23 17:16:46 main:574] : INFO : Epoch 85 | loss: 0.0312245 | val_loss: 0.0312277 | Time: 6378.76 ms
[2022-02-23 17:16:52 main:574] : INFO : Epoch 86 | loss: 0.0312222 | val_loss: 0.0312206 | Time: 6142.26 ms
[2022-02-23 17:16:59 main:574] : INFO : Epoch 87 | loss: 0.0312194 | val_loss: 0.0312193 | Time: 6218.87 ms
[2022-02-23 17:17:05 main:574] : INFO : Epoch 88 | loss: 0.031221 | val_loss: 0.0312221 | Time: 6057.42 ms
[2022-02-23 17:17:11 main:574] : INFO : Epoch 89 | loss: 0.0312207 | val_loss: 0.0312189 | Time: 6223.58 ms
[2022-02-23 17:17:17 main:574] : INFO : Epoch 90 | loss: 0.0312189 | val_loss: 0.0312189 | Time: 6223.36 ms
[2022-02-23 17:17:23 main:574] : INFO : Epoch 91 | loss: 0.0312191 | val_loss: 0.0312244 | Time: 6161.81 ms
[2022-02-23 17:17:30 main:574] : INFO : Epoch 92 | loss: 0.0312217 | val_loss: 0.0312195 | Time: 6242.07 ms
[2022-02-23 17:17:36 main:574] : INFO : Epoch 93 | loss: 0.0312203 | val_loss: 0.0312181 | Time: 6137.46 ms
[2022-02-23 17:17:42 main:574] : INFO : Epoch 94 | loss: 0.0312185 | val_loss: 0.0312181 | Time: 6409.15 ms
[2022-02-23 17:17:49 main:574] : INFO : Epoch 95 | loss: 0.0312176 | val_loss: 0.0312177 | Time: 6170.71 ms
[2022-02-23 17:17:55 main:574] : INFO : Epoch 96 | loss: 0.0312171 | val_loss: 0.0312177 | Time: 6236 ms
[2022-02-23 17:18:01 main:574] : INFO : Epoch 97 | loss: 0.031218 | val_loss: 0.0312203 | Time: 6073.15 ms
[2022-02-23 17:18:07 main:574] : INFO : Epoch 98 | loss: 0.0312185 | val_loss: 0.0312206 | Time: 6089.43 ms
[2022-02-23 17:18:14 main:574] : INFO : Epoch 99 | loss: 0.0312168 | val_loss: 0.0312171 | Time: 6197.79 ms
[2022-02-23 17:18:20 main:574] : INFO : Epoch 100 | loss: 0.0312176 | val_loss: 0.031217 | Time: 6016.09 ms
[2022-02-23 17:18:26 main:574] : INFO : Epoch 101 | loss: 0.0312183 | val_loss: 0.0312211 | Time: 6149.63 ms
[2022-02-23 17:18:32 main:574] : INFO : Epoch 102 | loss: 0.0312173 | val_loss: 0.0312172 | Time: 6021.49 ms
[2022-02-23 17:18:38 main:574] : INFO : Epoch 103 | loss: 0.0312166 | val_loss: 0.0312188 | Time: 6201.66 ms
[2022-02-23 17:18:44 main:574] : INFO : Epoch 104 | loss: 0.031217 | val_loss: 0.0312219 | Time: 6044.88 ms
[2022-02-23 17:18:51 main:574] : INFO : Epoch 105 | loss: 0.0312175 | val_loss: 0.0312204 | Time: 6002.86 ms
[2022-02-23 17:18:57 main:574] : INFO : Epoch 106 | loss: 0.0312163 | val_loss: 0.0312175 | Time: 6216.43 ms
[2022-02-23 17:19:03 main:574] : INFO : Epoch 107 | loss: 0.0312153 | val_loss: 0.031218 | Time: 6036.74 ms
[2022-02-23 17:19:09 main:574] : INFO : Epoch 108 | loss: 0.0312169 | val_loss: 0.0312223 | Time: 6126.15 ms
[2022-02-23 17:19:15 main:574] : INFO : Epoch 109 | loss: 0.0312185 | val_loss: 0.0312193 | Time: 6019.27 ms
[2022-02-23 17:19:21 main:574] : INFO : Epoch 110 | loss: 0.0312159 | val_loss: 0.0312187 | Time: 6007.91 ms
[2022-02-23 17:19:27 main:574] : INFO : Epoch 111 | loss: 0.0312161 | val_loss: 0.0312179 | Time: 6050.78 ms
[2022-02-23 17:19:33 main:574] : INFO : Epoch 112 | loss: 0.0312163 | val_loss: 0.0312217 | Time: 5894.54 ms
[2022-02-23 17:19:40 main:574] : INFO : Epoch 113 | loss: 0.0312163 | val_loss: 0.0312188 | Time: 6012.58 ms
[2022-02-23 17:19:46 main:574] : INFO : Epoch 114 | loss: 0.0312153 | val_loss: 0.0312181 | Time: 5982.48 ms
[2022-02-23 17:19:52 main:574] : INFO : Epoch 115 | loss: 0.0312158 | val_loss: 0.0312183 | Time: 6022.38 ms
[2022-02-23 17:19:58 main:574] : INFO : Epoch 116 | loss: 0.0312152 | val_loss: 0.0312237 | Time: 6130.25 ms
[2022-02-23 17:20:04 main:574] : INFO : Epoch 117 | loss: 0.0312157 | val_loss: 0.0312202 | Time: 5961.58 ms
[2022-02-23 17:20:10 main:574] : INFO : Epoch 118 | loss: 0.031216 | val_loss: 0.0312179 | Time: 6009.34 ms
[2022-02-23 17:20:16 main:574] : INFO : Epoch 119 | loss: 0.031215 | val_loss: 0.0312177 | Time: 5785.52 ms
[2022-02-23 17:20:22 main:574] : INFO : Epoch 120 | loss: 0.0312154 | val_loss: 0.0312187 | Time: 5773.58 ms
[2022-02-23 17:20:28 main:574] : INFO : Epoch 121 | loss: 0.0312152 | val_loss: 0.0312192 | Time: 5961.59 ms
[2022-02-23 17:20:34 main:574] : INFO : Epoch 122 | loss: 0.031217 | val_loss: 0.0312225 | Time: 5793.69 ms
[2022-02-23 17:20:40 main:574] : INFO : Epoch 123 | loss: 0.0312157 | val_loss: 0.0312188 | Time: 5997.24 ms
[2022-02-23 17:20:46 main:574] : INFO : Epoch 124 | loss: 0.0312172 | val_loss: 0.0312194 | Time: 5768.2 ms
[2022-02-23 17:20:51 main:574] : INFO : Epoch 125 | loss: 0.0312184 | val_loss: 0.0312274 | Time: 5738.95 ms
[2022-02-23 17:20:57 main:574] : INFO : Epoch 126 | loss: 0.0312152 | val_loss: 0.0312176 | Time: 5851.45 ms
[2022-02-23 17:21:03 main:574] : INFO : Epoch 127 | loss: 0.0312136 | val_loss: 0.031219 | Time: 5864.44 ms
[2022-02-23 17:21:09 main:574] : INFO : Epoch 128 | loss: 0.0312141 | val_loss: 0.0312233 | Time: 5879.37 ms
[2022-02-23 17:21:15 main:574] : INFO : Epoch 129 | loss: 0.031215 | val_loss: 0.0312185 | Time: 5956.64 ms
[2022-02-23 17:21:22 main:574] : INFO : Epoch 130 | loss: 0.031214 | val_loss: 0.0312242 | Time: 6491.85 ms
[2022-02-23 17:21:39 main:574] : INFO : Epoch 131 | loss: 0.0312161 | val_loss: 0.0312199 | Time: 16940.9 ms
[2022-02-23 17:22:47 main:574] : INFO : Epoch 132 | loss: 0.0312135 | val_loss: 0.031222 | Time: 68035.5 ms
[2022-02-23 17:22:55 main:574] : INFO : Epoch 133 | loss: 0.0312143 | val_loss: 0.0312186 | Time: 7176.27 ms
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFF276F4F69
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.17.0
Dump Timestamp : 02/23/22 17:38:44
Install Directory : C:\Program Files\BOINC\
Data Directory : D:\BOINC
Project Symstore :
LoadLibraryA( D:\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( D:\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\BOINC\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: D:\BOINC\slots\0;D:\BOINC\projects\www.mlcathome.org_mlcathome
ModLoad: 00000000c1bf0000 00000000003df000 D:\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000029c90000 00000000001f5000 C:\Windows\SYSTEM32\ntdll.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.19041.1466 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1466
ModLoad: 0000000029b90000 00000000000be000 C:\Windows\System32\KERNEL32.DLL (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.19041.1503 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1503
ModLoad: 00000000276c0000 00000000002c8000 C:\Windows\System32\KERNELBASE.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.19041.1503 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1503
ModLoad: 0000000004d20000 0000000000111000 C:\Program Files\Bitdefender\Bitdefender Security\atcuf\dlls_265693163949223832\atcuf64.dll (1.45.321.0) (-exported- Symbols Loaded)
Linked PDB Filename : E:\builds\ATC-DEFAULT-SOURCES\bin\x64\Release\atcuf64\atcuf64.pdb
File Version : 1.45.321.0 #0xc1c07b8bc
Company Name : Bitdefender S.R.L. Bucharest, ROMANIA
Product Name : Bitdefender® ATC
Product Version : 4
ModLoad: 0000000028f40000 00000000001a0000 C:\Windows\System32\USER32.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000027a10000 0000000000022000 C:\Windows\System32\win32u.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.19041.1466 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1466
ModLoad: 00000000285e0000 000000000002b000 C:\Windows\System32\GDI32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.19041.1202 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1202
ModLoad: 00000000273a0000 000000000010d000 C:\Windows\System32\gdi32full.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.19041.1466 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1466
ModLoad: 0000000027a40000 000000000009d000 C:\Windows\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 0000000027be0000 0000000000100000 C:\Windows\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 0000000027e00000 00000000000ae000 C:\Windows\System32\ADVAPI32.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000284c0000 000000000009e000 C:\Windows\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.19041.546
ModLoad: 0000000028420000 000000000009c000 C:\Windows\System32\sechost.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000029840000 0000000000125000 C:\Windows\System32\RPCRT4.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000f3240000 0000000000058000 D:\BOINC\slots\0\c10.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000795e0000 0000000024b49000 D:\BOINC\slots\0\torch_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000006e9f0000 000000000abe7000 D:\BOINC\slots\0\torch_cpu.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000016040000 000000000008d000 C:\Windows\SYSTEM32\MSVCP140.dll (14.29.30133.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
File Version : 14.29.30133.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.29.30133.0
ModLoad: 0000000020e90000 00000000001e4000 C:\Windows\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.19041.867 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.867
ModLoad: 0000000016f80000 000000000001b000 C:\Windows\SYSTEM32\VCRUNTIME140.dll (14.29.30133.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
File Version : 14.29.30133.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.29.30133.0
ModLoad: 0000000016f70000 000000000000c000 C:\Windows\SYSTEM32\VCRUNTIME140_1.dll (14.29.30133.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
File Version : 14.29.30133.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.29.30133.0
ModLoad: 0000000003df0000 0000000000045000 D:\BOINC\slots\0\c10_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000a4810000 00000000030a5000 D:\BOINC\slots\0\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1031
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA SPARSE BLAS Library
Product Version : 6,14,11,1031
ModLoad: 00000000a1350000 00000000034b2000 D:\BOINC\slots\0\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA RAND Library
Product Version : 6,14,11,1012
ModLoad: 0000000053640000 000000001b3a4000 D:\BOINC\slots\0\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,10020
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.86 CUDNN Library
Product Version : 6,14,11,10020
ModLoad: 00000000221a0000 0000000000010000 D:\BOINC\slots\0\nvToolsExt64_1.dll (-exported- Symbols Loaded)
Linked PDB Filename : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb
ModLoad: 000000004a4a0000 000000000919f000 D:\BOINC\slots\0\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA FFT Library
Product Version : 6,14,11,1012
ModLoad: 0000000046740000 0000000003d5b000 D:\BOINC\slots\0\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 00000000c4030000 000000000020d000 D:\BOINC\slots\0\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
Linked PDB Filename : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
File Version : 20200205
Company Name : Intel Corporation
Product Name : Intel(R) OpenMP* Runtime Library
Product Version : 5.0
ModLoad: 00000000bf100000 00000000002db000 D:\BOINC\slots\0\fbgemm.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000009f480000 0000000001eca000 D:\BOINC\slots\0\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 0000000003d40000 0000000000040000 D:\BOINC\slots\0\asmjit.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000001f610000 000000000002c000 C:\Windows\SYSTEM32\dbgcore.DLL (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : dbgcore.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 00000000299a0000 0000000000030000 C:\Windows\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000252b0000 0000000000012000 C:\Windows\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000028ac0000 0000000000472000 C:\Windows\System32\Setupapi.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : setupapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000027990000 000000000004e000 C:\Windows\System32\cfgmgr32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : cfgmgr32.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 00000000279e0000 0000000000027000 C:\Windows\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000027140000 0000000000034000 C:\Windows\SYSTEM32\DEVOBJ.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : devobj.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 0000000027b70000 0000000000069000 C:\Windows\System32\WINTRUST.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : wintrust.pdb
File Version : 10.0.19041.1503 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1503
ModLoad: 00000000274b0000 0000000000156000 C:\Windows\System32\CRYPT32.dll (6.2.19041.1320) (-exported- Symbols Loaded)
Linked PDB Filename : crypt32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000026f80000 0000000000012000 C:\Windows\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msasn1.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000290e0000 0000000000744000 C:\Windows\System32\Shell32.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : shell32.pdb
File Version : 10.0.19041.964 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.964
ModLoad: 00000000254b0000 0000000000794000 C:\Windows\SYSTEM32\windows.storage.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : Windows.Storage.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000280c0000 0000000000355000 C:\Windows\System32\combase.dll (6.2.19041.1348) (-exported- Symbols Loaded)
Linked PDB Filename : combase.pdb
File Version : 10.0.19041.1320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1320
ModLoad: 0000000026df0000 000000000002e000 C:\Windows\SYSTEM32\Wldp.dll (6.2.19041.1320) (-exported- Symbols Loaded)
Linked PDB Filename : WLDP.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000029ae0000 00000000000ad000 C:\Windows\System32\SHCORE.dll (6.2.19041.1387) (-exported- Symbols Loaded)
Linked PDB Filename : shcore.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000029a80000 0000000000055000 C:\Windows\System32\shlwapi.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shlwapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000b9f10000 00000000002cd000 C:\Windows\system32\nvcuda.dll (30.0.15.1123) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_04\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
File Version : 30.0.15.1123
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.6.58 driver
Product Version : 30.0.15.1123
ModLoad: 00000000450a0000 00000000016a0000 C:\Windows\system32\DriverStore\FileRepository\nvgbdi.inf_amd64_68f94e3d52ac5935\nvcuda64.dll (30.0.15.1123) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_04\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
File Version : 30.0.15.1123
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.6.58 driver
Product Version : 30.0.15.1123
ModLoad: 0000000021e20000 000000000000a000 C:\Windows\SYSTEM32\VERSION.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000022450000 0000000000031000 C:\Windows\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : cryptnet.pdb
File Version : 10.0.19041.906 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.906
ModLoad: 000000001e670000 000000000014a000 C:\Windows\SYSTEM32\drvstore.dll (6.2.19041.1320) (-exported- Symbols Loaded)
Linked PDB Filename : drvstore.pdb
File Version : 10.0.19041.1320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1320
ModLoad: 0000000026d60000 000000000000c000 C:\Windows\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cryptbase.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 000000001d610000 0000000000767000 C:\Windows\SYSTEM32\nvapi64.dll (30.0.15.1123) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_04\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
File Version : 30.0.15.1123
Company Name : NVIDIA Corporation
Product Name : NVIDIA Windows drivers
Product Version : 30.0.15.1123
ModLoad: 0000000022180000 0000000000009000 D:\BOINC\slots\0\caffe2_nvrtc.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000440d0000 0000000000fca000 D:\BOINC\slots\0\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6.14.11.9000
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.89 NVRTC Library
Product Version : 6.14.11.9000
ModLoad: 0000000027ae0000 0000000000082000 C:\Windows\System32\bcryptPrimitives.dll (6.2.19041.1415) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.19041.1415 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1415
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 3701, Write: 566, Other 1516
- I/O Transfers Counters -
Read: 6972962, Write: 128578, Other 40639
- Paged Pool Usage -
QuotaPagedPoolUsage: 3497136, QuotaPeakPagedPoolUsage: 3513520
QuotaNonPagedPoolUsage: 55008, QuotaPeakNonPagedPoolUsage: 55144
- Virtual Memory Usage -
VirtualSize: -384417792, PeakVirtualSize: 225185792
- Pagefile Usage -
PagefileUsage: -384417792, PeakPagefileUsage: -383926272
- Working Set Size -
WorkingSetSize: 381616128, PeakWorkingSetSize: 1741127680, PageFaultCount: 623134
*** Dump of thread ID 12392 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 1246093696.000000, User Time: 807968768.000000, Wait Time: 125211768.000000
- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFF276F4F69
- Registers -
rax=0000000000000008 rbx=000000007d05af00 rcx=0000000070d70000 rdx=00000000e8903db9 rsi=00000000e8904670 rdi=0000000019930520
r8=0000000000000000 r9=0000000000000010 r10=0000000000000000 r11=0000000071620b00 r12=00000000e89051c0 r13=000000006fb81c60
r14=0000000000000002 r15=0000000000000144 rip=00000000276f4f69 rsp=00000000e8904420 rbp=00000000e8904660
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
- Callstack -
ChildEBP RetAddr Args to Child
e89044f0 16f86480 e8904648 6554c9f0 00000144 00000b39 KERNELBASE!RaiseException+0x0
e8904550 7b335380 7c983020 00000000 e8904608 00000144 VCRUNTIME140!_CxxThrowException+0x0
e8904760 7b311fcc 00000003 e8905680 00000040 00000001 torch_cuda!at::native::_triangular_solve_helper_cuda+0x0
e8904e60 7b307026 00000003 e8905680 00000003 e89051c0 torch_cuda!at::native::_triangular_solve_helper_cuda+0x0
e8904fd0 7b316649 a0452150 00000000 e8905680 00000000 torch_cuda!at::native::_triangular_solve_helper_cuda+0x0
e8905170 7b348bba a0451e10 74271344 7b2c10d0 e8905680 torch_cuda!at::native::_triangular_solve_helper_cuda+0x0
e8905370 7b34c8f8 e89058c8 79311380 a1075330 00000002 torch_cuda!at::native::_triangular_solve_helper_cuda+0x0
e89053b0 740cff3a e89053d8 e89053e8 e8905400 00000000 torch_cuda!at::native::_triangular_solve_helper_cuda+0x0
e89055a0 740d2f37 e8905b40 a1075330 e8905b40 e8906018 torch_cpu!at::native::triangular_solve_out+0x0
e8905b10 740d2df7 79319608 7b2a88fd 00000001 74595dc2 torch_cpu!at::native::add_out+0x0
e8905b60 7c3b5f28 e8905ba0 00000001 e8905ba0 00000001 torch_cpu!at::native::add_+0x0
e8905be0 7c3c5f3f 79310640 70e23378 70d8d330 00000001 torch_cuda!at::native::set_storage_cuda_+0x0
e8905c30 744956f4 79310640 79310640 e8905bf0 774aefa8 torch_cuda!at::native::set_storage_cuda_+0x0
e8905cf0 7458dbb5 5803b0b4 00000000 a1075330 f3263720 torch_cpu!at::bucketize_out+0x0
e8905d40 757bd169 79310640 ffffffff bd32f852 00000000 torch_cpu!at::Tensor::add_+0x0
e8905e70 7440680f 72444ec8 73fdbfac e89060e0 79310640 torch_cpu!torch::autograd::GraphRoot::apply+0x0
e8905ec0 744956f4 79310640 00000000 a1075330 7449676e torch_cpu!at::native::mkldnn_sigmoid_+0x0
e8905f80 7458dbb5 00000000 00000000 cccccccd 00000000 torch_cpu!at::bucketize_out+0x0
e8905fd0 75fb1b30 a1075320 0078003d 0078003d 00000000 torch_cpu!at::Tensor::add_+0x0
e89069d0 c1c07190 e8906aa0 e8906aa0 e89074e0 00000000 torch_cpu!torch::optim::Adam::step+0x0
e8906ed0 c1c0b950 e8906fd0 e8906fd0 00000000 37968050 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0
e891fc70 c1e4c6f8 7301d080 00000000 00000000 730d6750 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
e891fcb0 29ba7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
e891fce0 29ce2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
e891fd60 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 7208 (state: Waiting): ***
- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 156250.000000, Wait Time: 125211760.000000
- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=00000000e8dff690 rsi=0000000000000000 rdi=0000000000000064
r8=00000000e8dfe6d0 r9=0000000000000020 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=0000000029d2d3f4 rsp=00000000e8dff668 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
e8dff660 2770962e e8dff728 00000000 00000000 00000000 ntdll!NtDelayExecution+0x0
e8dff700 c1c2a07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0
e8dff730 29ba7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
e8dff760 29ce2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
e8dff7e0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 11796 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 312500.000000, Wait Time: 125211768.000000
- Registers -
rax=000000000000005b rbx=0000000000000006 rcx=0000000000000006 rdx=00000000e8eff8f0 rsi=0000000000000000 rdi=0000000000000006
r8=00000000e8eff598 r9=0000000073228a30 r10=0000000000000000 r11=0000000000000246 r12=0000000000000064 r13=00000000e8eff8f0
r14=00000000e8eff5f0 r15=0000000000000000 rip=0000000029d2d8c4 rsp=00000000e8eff598 rbp=0000000073228a30
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
e8eff590 2770cb20 00000000 04d246dc 29dfa4d0 e8b19000 ntdll!ZwWaitForMultipleObjects+0x0
e8eff880 2770ca1e 00000206 276ea395 00000006 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0
e8eff8c0 4527d384 7322c0f0 00000000 04e1e850 04d34357 KERNELBASE!WaitForMultipleObjects+0x0
e8effb00 453256fd 00000000 73228a30 00000000 00000000 nvcuda64!+0x0
e8effbb0 4527cd53 3066795b f1d255a5 73c17580 00000000 nvcuda64!cuProfilerStop+0x0
e8effbe0 455b4064 73c17580 00000000 00000000 00000000 nvcuda64!+0x0
e8effc10 29ba7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
e8effc40 29ce2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
e8effcc0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 13300 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 921875008.000000, User Time: 530468736.000000, Wait Time: 125151120.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=0000000083bf82c8 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000083bf82c8
r8=00000000e8ffeb10 r9=0000000000000000 r10=0000000000000038 r11=00000000e8ffea50 r12=0000000000000000 r13=0000000000000000
r14=00000000e8fff290 r15=0000000083bf8278 rip=0000000029d30764 rsp=00000000e8fff268 rbp=00000000e8fff2c0
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
e8fff260 29cf4021 e8fff560 00000000 e8fff360 00000000 ntdll!NtWaitForAlertByThreadId+0x0
e8fff2e0 2772ce89 83bf82c0 83bf82b8 83bf8270 75bd05b4 ntdll!RtlSleepConditionVariableSRW+0x0
e8fff320 16052959 00000000 8eaccbb0 00000000 00000000 KERNELBASE!SleepConditionVariableSRW+0x0
e8fff350 16052bea baaa2584 00000000 e8fff560 83bf82b8 MSVCP140!std::_Winerror_message+0x0
e8fff380 75bd064b 83bf8270 e8fff480 83bf8270 8eaccb90 MSVCP140!_Cnd_wait+0x0
e8fff3c0 75bd25b1 a101aa60 ffffffff 8eaccb90 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0
e8fff6b0 75bd2471 790f6d10 e8fff6f0 790f6d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0
e8fff700 75bc9744 00080001 a070d420 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0
e8fff730 27c01bb2 a070d420 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0
e8fff760 29ba7034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0
e8fff790 29ce2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
e8fff810 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)