Task 14179870
| Name | ParityModified-1647047563-29761-3-0_0 |
| Workunit | 11241918 |
| Created | 1 Apr 2022, 6:22:10 UTC |
| Sent | 5 Apr 2022, 17:23:22 UTC |
| Report deadline | 13 Apr 2022, 17:23:22 UTC |
| Received | 6 Apr 2022, 12:03:31 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | -529697949 (0xE06D7363) Unknown error code |
| Computer ID | 4503 |
| Run time | 4 hours 24 min 4 sec |
| CPU time | 7 min 27 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 10,073.24 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 2.08 GB |
| Peak swap size | 4.38 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 3765269347 (0xe06d7363)</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 2080)
[2022-04-06 01:07:18 main:435] : INFO : Set logging level to 1
[2022-04-06 01:07:18 main:441] : INFO : Running in BOINC Client mode
[2022-04-06 01:07:18 main:444] : INFO : Resolving all filenames
[2022-04-06 01:07:18 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-06 01:07:18 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-04-06 01:07:18 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-06 01:07:18 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-06 01:07:18 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-04-06 01:07:18 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-04-06 01:07:18 main:474] : INFO : Configuration:
[2022-04-06 01:07:18 main:475] : INFO : Model type: GRU
[2022-04-06 01:07:18 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-04-06 01:07:18 main:477] : INFO : Max Epochs: 2048
[2022-04-06 01:07:18 main:478] : INFO : Batch Size: 128
[2022-04-06 01:07:18 main:479] : INFO : Learning Rate: 0.01
[2022-04-06 01:07:18 main:480] : INFO : Patience: 10
[2022-04-06 01:07:18 main:481] : INFO : Hidden Width: 12
[2022-04-06 01:07:18 main:482] : INFO : # Recurrent Layers: 4
[2022-04-06 01:07:18 main:483] : INFO : # Backend Layers: 4
[2022-04-06 01:07:18 main:484] : INFO : # Threads: 1
[2022-04-06 01:07:18 main:486] : INFO : Preparing Dataset
[2022-04-06 01:07:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-06 01:07:19 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-06 01:07:20 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-04-06 01:07:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-06 01:07:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-06 01:07:20 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-04-06 01:07:20 main:494] : INFO : Creating Model
[2022-04-06 01:07:21 main:507] : INFO : Preparing config file
[2022-04-06 01:07:21 main:519] : INFO : Creating new config file
[2022-04-06 01:07:21 main:538] : INFO : This is a continuation WU, loading previous network
[2022-04-06 01:07:25 main:559] : INFO : Loading DataLoader into Memory
[2022-04-06 01:07:48 main:562] : INFO : Starting Training
[2022-04-06 01:08:00 main:574] : INFO : Epoch 1 | loss: 0.0312513 | val_loss: 0.0311845 | Time: 11830.5 ms
[2022-04-06 01:08:05 main:574] : INFO : Epoch 2 | loss: 0.0311414 | val_loss: 0.0311636 | Time: 5529.6 ms
[2022-04-06 01:09:19 main:574] : INFO : Epoch 3 | loss: 0.0311309 | val_loss: 0.0311657 | Time: 72383.4 ms
[2022-04-06 01:09:24 main:574] : INFO : Epoch 4 | loss: 0.0311265 | val_loss: 0.0311575 | Time: 4875.25 ms
[2022-04-06 01:09:54 main:574] : INFO : Epoch 5 | loss: 0.0311256 | val_loss: 0.031158 | Time: 29897.7 ms
[2022-04-06 01:10:34 main:574] : INFO : Epoch 6 | loss: 0.0311241 | val_loss: 0.0311587 | Time: 39227.9 ms
[2022-04-06 01:10:39 main:574] : INFO : Epoch 7 | loss: 0.0311263 | val_loss: 0.0311645 | Time: 5286.63 ms
[2022-04-06 01:12:51 main:574] : INFO : Epoch 8 | loss: 0.0311271 | val_loss: 0.0311666 | Time: 132207 ms
[2022-04-06 01:15:26 main:574] : INFO : Epoch 9 | loss: 0.0311301 | val_loss: 0.0311605 | Time: 149612 ms
[2022-04-06 01:18:37 main:574] : INFO : Epoch 10 | loss: 0.0311309 | val_loss: 0.0311566 | Time: 186799 ms
[2022-04-06 01:18:43 main:574] : INFO : Epoch 11 | loss: 0.0311288 | val_loss: 0.0311569 | Time: 4521.56 ms
[2022-04-06 01:18:49 main:574] : INFO : Epoch 12 | loss: 0.0311235 | val_loss: 0.0311609 | Time: 4865.1 ms
[2022-04-06 01:18:54 main:574] : INFO : Epoch 13 | loss: 0.0311257 | val_loss: 0.0311569 | Time: 4473.64 ms
[2022-04-06 01:19:21 main:574] : INFO : Epoch 14 | loss: 0.0311223 | val_loss: 0.0311658 | Time: 27246.7 ms
[2022-04-06 01:19:26 main:574] : INFO : Epoch 15 | loss: 0.0311209 | val_loss: 0.0311623 | Time: 4678.13 ms
[2022-04-06 01:20:20 main:574] : INFO : Epoch 16 | loss: 0.0311215 | val_loss: 0.031156 | Time: 52829.2 ms
[2022-04-06 01:22:45 main:574] : INFO : Epoch 17 | loss: 0.0311217 | val_loss: 0.0311571 | Time: 135390 ms
[2022-04-06 01:24:47 main:574] : INFO : Epoch 18 | loss: 0.0311223 | val_loss: 0.0311581 | Time: 109877 ms
[2022-04-06 01:27:11 main:574] : INFO : Epoch 19 | loss: 0.0311197 | val_loss: 0.0311618 | Time: 144469 ms
[2022-04-06 01:29:42 main:574] : INFO : Epoch 20 | loss: 0.0311207 | val_loss: 0.031157 | Time: 146030 ms
[2022-04-06 01:32:41 main:574] : INFO : Epoch 21 | loss: 0.0311195 | val_loss: 0.0311591 | Time: 172196 ms
[2022-04-06 01:34:53 main:574] : INFO : Epoch 22 | loss: 0.0311204 | val_loss: 0.0311746 | Time: 132383 ms
[2022-04-06 01:35:59 main:574] : INFO : Epoch 23 | loss: 0.0311216 | val_loss: 0.0311615 | Time: 65800.8 ms
[2022-04-06 01:38:14 main:574] : INFO : Epoch 24 | loss: 0.0311189 | val_loss: 0.0311584 | Time: 135464 ms
[2022-04-06 01:40:02 main:574] : INFO : Epoch 25 | loss: 0.0311203 | val_loss: 0.0311633 | Time: 107905 ms
[2022-04-06 01:43:37 main:574] : INFO : Epoch 26 | loss: 0.0311224 | val_loss: 0.0311552 | Time: 215077 ms
[2022-04-06 01:46:41 main:574] : INFO : Epoch 27 | loss: 0.0311207 | val_loss: 0.031156 | Time: 179333 ms
[2022-04-06 01:49:25 main:574] : INFO : Epoch 28 | loss: 0.0311191 | val_loss: 0.0311536 | Time: 159136 ms
[2022-04-06 01:50:52 main:574] : INFO : Epoch 29 | loss: 0.0311212 | val_loss: 0.0311539 | Time: 82108 ms
[2022-04-06 01:53:04 main:574] : INFO : Epoch 30 | loss: 0.0311212 | val_loss: 0.0311636 | Time: 131408 ms
[2022-04-06 01:55:28 main:574] : INFO : Epoch 31 | loss: 0.0311188 | val_loss: 0.0311599 | Time: 144483 ms
[2022-04-06 01:58:25 main:574] : INFO : Epoch 32 | loss: 0.031118 | val_loss: 0.0311564 | Time: 172204 ms
[2022-04-06 02:01:06 main:574] : INFO : Epoch 33 | loss: 0.0311181 | val_loss: 0.0311593 | Time: 147886 ms
[2022-04-06 02:03:42 main:574] : INFO : Epoch 34 | loss: 0.031118 | val_loss: 0.0311599 | Time: 150825 ms
[2022-04-06 02:05:55 main:574] : INFO : Epoch 35 | loss: 0.0311209 | val_loss: 0.031159 | Time: 127780 ms
[2022-04-06 02:07:57 main:574] : INFO : Epoch 36 | loss: 0.0311271 | val_loss: 0.0311569 | Time: 122015 ms
[2022-04-06 02:10:08 main:574] : INFO : Epoch 37 | loss: 0.0311271 | val_loss: 0.031153 | Time: 126496 ms
[2022-04-06 02:12:32 main:574] : INFO : Epoch 38 | loss: 0.0311256 | val_loss: 0.0311546 | Time: 138963 ms
[2022-04-06 02:15:20 main:574] : INFO : Epoch 39 | loss: 0.0311245 | val_loss: 0.0311581 | Time: 163897 ms
[2022-04-06 02:18:21 main:574] : INFO : Epoch 40 | loss: 0.0311227 | val_loss: 0.0311559 | Time: 175764 ms
[2022-04-06 02:20:58 main:574] : INFO : Epoch 41 | loss: 0.0311184 | val_loss: 0.0311565 | Time: 157309 ms
[2022-04-06 02:22:22 main:574] : INFO : Epoch 42 | loss: 0.0311181 | val_loss: 0.0311629 | Time: 83721.9 ms
[2022-04-06 02:24:31 main:574] : INFO : Epoch 43 | loss: 0.0311196 | val_loss: 0.0311513 | Time: 128798 ms
[2022-04-06 02:27:25 main:574] : INFO : Epoch 44 | loss: 0.0311191 | val_loss: 0.0311552 | Time: 174586 ms
[2022-04-06 02:30:25 main:574] : INFO : Epoch 45 | loss: 0.0311199 | val_loss: 0.0311546 | Time: 174588 ms
[2022-04-06 02:32:59 main:574] : INFO : Epoch 46 | loss: 0.0311197 | val_loss: 0.0311525 | Time: 149374 ms
[2022-04-06 02:34:25 main:574] : INFO : Epoch 47 | loss: 0.031118 | val_loss: 0.0311548 | Time: 86600.1 ms
[2022-04-06 02:35:52 main:574] : INFO : Epoch 48 | loss: 0.031116 | val_loss: 0.0311562 | Time: 81166.9 ms
[2022-04-06 02:38:42 main:574] : INFO : Epoch 49 | loss: 0.0311191 | val_loss: 0.0311521 | Time: 165071 ms
[2022-04-06 02:39:56 main:574] : INFO : Epoch 50 | loss: 0.0311175 | val_loss: 0.0311545 | Time: 74392.1 ms
[2022-04-06 02:42:52 main:574] : INFO : Epoch 51 | loss: 0.0311189 | val_loss: 0.0311598 | Time: 170986 ms
[2022-04-06 02:45:59 main:574] : INFO : Epoch 52 | loss: 0.0311193 | val_loss: 0.0311551 | Time: 186434 ms
[2022-04-06 02:48:19 main:574] : INFO : Epoch 53 | loss: 0.0311193 | val_loss: 0.0311635 | Time: 140661 ms
[2022-04-06 02:50:54 main:574] : INFO : Epoch 54 | loss: 0.0311229 | val_loss: 0.0311491 | Time: 145765 ms
[2022-04-06 02:51:30 main:574] : INFO : Epoch 55 | loss: 0.0311241 | val_loss: 0.0311523 | Time: 35641.4 ms
[2022-04-06 02:51:36 main:574] : INFO : Epoch 56 | loss: 0.0311278 | val_loss: 0.0311626 | Time: 6187.17 ms
[2022-04-06 02:52:20 main:574] : INFO : Epoch 57 | loss: 0.031145 | val_loss: 0.0311585 | Time: 42571 ms
[2022-04-06 02:52:56 main:574] : INFO : Epoch 58 | loss: 0.0311504 | val_loss: 0.0311566 | Time: 36410.9 ms
[2022-04-06 02:55:08 main:574] : INFO : Epoch 59 | loss: 0.0311383 | val_loss: 0.0311534 | Time: 131466 ms
[2022-04-06 02:57:47 main:574] : INFO : Epoch 60 | loss: 0.0311352 | val_loss: 0.0311534 | Time: 154205 ms
[2022-04-06 03:00:07 main:574] : INFO : Epoch 61 | loss: 0.0311332 | val_loss: 0.0311531 | Time: 135422 ms
[2022-04-06 03:02:55 main:574] : INFO : Epoch 62 | loss: 0.0311292 | val_loss: 0.0311584 | Time: 163113 ms
[2022-04-06 03:04:45 main:574] : INFO : Epoch 63 | loss: 0.0311326 | val_loss: 0.0311537 | Time: 110020 ms
[2022-04-06 03:05:58 main:574] : INFO : Epoch 64 | loss: 0.0311314 | val_loss: 0.0311578 | Time: 68645.3 ms
[2022-04-06 03:08:14 main:574] : INFO : Epoch 65 | loss: 0.0311262 | val_loss: 0.031153 | Time: 130140 ms
[2022-04-06 03:10:01 main:574] : INFO : Epoch 66 | loss: 0.0311232 | val_loss: 0.0311494 | Time: 97486.2 ms
[2022-04-06 03:11:49 main:574] : INFO : Epoch 67 | loss: 0.0311199 | val_loss: 0.0311528 | Time: 107683 ms
[2022-04-06 03:12:38 main:574] : INFO : Epoch 68 | loss: 0.0311217 | val_loss: 0.0311463 | Time: 39955.5 ms
[2022-04-06 03:12:43 main:574] : INFO : Epoch 69 | loss: 0.0311208 | val_loss: 0.031149 | Time: 5381.28 ms
[2022-04-06 03:14:28 main:574] : INFO : Epoch 70 | loss: 0.0311237 | val_loss: 0.0311519 | Time: 103829 ms
[2022-04-06 03:14:33 main:574] : INFO : Epoch 71 | loss: 0.031119 | val_loss: 0.0311487 | Time: 5760.43 ms
[2022-04-06 03:16:47 main:574] : INFO : Epoch 72 | loss: 0.0311226 | val_loss: 0.0311549 | Time: 132668 ms
[2022-04-06 03:16:53 main:574] : INFO : Epoch 73 | loss: 0.0311201 | val_loss: 0.0311519 | Time: 5110.93 ms
[2022-04-06 03:19:44 main:574] : INFO : Epoch 74 | loss: 0.031121 | val_loss: 0.0311567 | Time: 168867 ms
[2022-04-06 03:20:52 main:574] : INFO : Epoch 75 | loss: 0.031119 | val_loss: 0.0311463 | Time: 62677.1 ms
[2022-04-06 03:23:03 main:574] : INFO : Epoch 76 | loss: 0.0311205 | val_loss: 0.0311484 | Time: 131505 ms
[2022-04-06 03:24:13 main:574] : INFO : Epoch 77 | loss: 0.0311247 | val_loss: 0.0311531 | Time: 69233.9 ms
[2022-04-06 03:24:17 main:574] : INFO : Epoch 78 | loss: 0.0311262 | val_loss: 0.0311531 | Time: 4790.9 ms
[2022-04-06 03:25:35 main:574] : INFO : Epoch 79 | loss: 0.0311219 | val_loss: 0.0311557 | Time: 76569.3 ms
[2022-04-06 03:28:22 main:574] : INFO : Epoch 80 | loss: 0.0311223 | val_loss: 0.0311586 | Time: 161822 ms
[2022-04-06 03:31:18 main:574] : INFO : Epoch 81 | loss: 0.0311194 | val_loss: 0.0311549 | Time: 166250 ms
[2022-04-06 03:33:58 main:574] : INFO : Epoch 82 | loss: 0.031119 | val_loss: 0.031149 | Time: 159832 ms
[2022-04-06 03:35:55 main:574] : INFO : Epoch 83 | loss: 0.0311184 | val_loss: 0.0311507 | Time: 111619 ms
[2022-04-06 03:37:51 main:574] : INFO : Epoch 84 | loss: 0.0311155 | val_loss: 0.0311527 | Time: 115959 ms
[2022-04-06 03:39:58 main:574] : INFO : Epoch 85 | loss: 0.0311228 | val_loss: 0.0311578 | Time: 121942 ms
[2022-04-06 03:41:38 main:574] : INFO : Epoch 86 | loss: 0.0311345 | val_loss: 0.0311562 | Time: 99361.4 ms
[2022-04-06 03:44:18 main:574] : INFO : Epoch 87 | loss: 0.0311328 | val_loss: 0.03116 | Time: 154397 ms
[2022-04-06 03:47:26 main:574] : INFO : Epoch 88 | loss: 0.031135 | val_loss: 0.0311515 | Time: 184058 ms
[2022-04-06 03:50:11 main:574] : INFO : Epoch 89 | loss: 0.0311289 | val_loss: 0.0311582 | Time: 165056 ms
[2022-04-06 03:51:48 main:574] : INFO : Epoch 90 | loss: 0.0311275 | val_loss: 0.0311538 | Time: 96178.8 ms
[2022-04-06 03:53:01 main:574] : INFO : Epoch 91 | loss: 0.0311235 | val_loss: 0.0311537 | Time: 68381.7 ms
[2022-04-06 03:55:25 main:574] : INFO : Epoch 92 | loss: 0.0311222 | val_loss: 0.0311542 | Time: 134096 ms
[2022-04-06 03:55:31 main:574] : INFO : Epoch 93 | loss: 0.0311225 | val_loss: 0.0311556 | Time: 5612.62 ms
[2022-04-06 03:58:01 main:574] : INFO : Epoch 94 | loss: 0.0311201 | val_loss: 0.0311535 | Time: 149760 ms
[2022-04-06 04:01:03 main:574] : INFO : Epoch 95 | loss: 0.0311225 | val_loss: 0.0311566 | Time: 181700 ms
[2022-04-06 04:02:15 main:574] : INFO : Epoch 96 | loss: 0.03112 | val_loss: 0.0311542 | Time: 67082.7 ms
[2022-04-06 04:02:21 main:574] : INFO : Epoch 97 | loss: 0.0311229 | val_loss: 0.0311534 | Time: 6531.25 ms
[2022-04-06 04:03:00 main:574] : INFO : Epoch 98 | loss: 0.0311191 | val_loss: 0.0311568 | Time: 38344.5 ms
[2022-04-06 04:03:13 main:574] : INFO : Epoch 99 | loss: 0.0311162 | val_loss: 0.0311508 | Time: 13662.7 ms
[2022-04-06 04:03:23 main:574] : INFO : Epoch 100 | loss: 0.0311178 | val_loss: 0.0311492 | Time: 9667.74 ms
[2022-04-06 04:04:06 main:574] : INFO : Epoch 101 | loss: 0.0311189 | val_loss: 0.0311548 | Time: 37628.6 ms
[2022-04-06 04:05:35 main:574] : INFO : Epoch 102 | loss: 0.0311254 | val_loss: 0.0311598 | Time: 89309.4 ms
[2022-04-06 04:08:58 main:574] : INFO : Epoch 103 | loss: 0.031127 | val_loss: 0.0311577 | Time: 185485 ms
[2022-04-06 04:11:04 main:574] : INFO : Epoch 104 | loss: 0.031129 | val_loss: 0.0311637 | Time: 120643 ms
[2022-04-06 04:11:09 main:574] : INFO : Epoch 105 | loss: 0.0311268 | val_loss: 0.0311551 | Time: 5435.97 ms
[2022-04-06 04:12:24 main:574] : INFO : Epoch 106 | loss: 0.031123 | val_loss: 0.0311549 | Time: 74530.4 ms
[2022-04-06 04:15:24 main:574] : INFO : Epoch 107 | loss: 0.03112 | val_loss: 0.0311532 | Time: 174581 ms
[2022-04-06 04:18:50 main:574] : INFO : Epoch 108 | loss: 0.0311192 | val_loss: 0.0311517 | Time: 201870 ms
[2022-04-06 04:20:57 main:574] : INFO : Epoch 109 | loss: 0.0311185 | val_loss: 0.0311525 | Time: 126724 ms
[2022-04-06 04:23:09 main:574] : INFO : Epoch 110 | loss: 0.0311181 | val_loss: 0.0311515 | Time: 132016 ms
[2022-04-06 04:25:27 main:574] : INFO : Epoch 111 | loss: 0.0311255 | val_loss: 0.0311695 | Time: 137853 ms
[2022-04-06 04:28:33 main:574] : INFO : Epoch 112 | loss: 0.0311373 | val_loss: 0.0311557 | Time: 186421 ms
[2022-04-06 04:31:51 main:574] : INFO : Epoch 113 | loss: 0.0311366 | val_loss: 0.031157 | Time: 197615 ms
[2022-04-06 04:32:58 main:574] : INFO : Epoch 114 | loss: 0.0311317 | val_loss: 0.0311562 | Time: 61709 ms
[2022-04-06 04:35:50 main:574] : INFO : Epoch 115 | loss: 0.0311308 | val_loss: 0.0311582 | Time: 172193 ms
[2022-04-06 04:37:27 main:574] : INFO : Epoch 116 | loss: 0.0311286 | val_loss: 0.0311558 | Time: 96803.1 ms
[2022-04-06 04:40:04 main:574] : INFO : Epoch 117 | loss: 0.0311332 | val_loss: 0.0311542 | Time: 156876 ms
[2022-04-06 04:42:02 main:574] : INFO : Epoch 118 | loss: 0.0311319 | val_loss: 0.0311599 | Time: 118177 ms
[2022-04-06 04:44:57 main:574] : INFO : Epoch 119 | loss: 0.0311275 | val_loss: 0.0311551 | Time: 170206 ms
[2022-04-06 04:48:06 main:574] : INFO : Epoch 120 | loss: 0.0311248 | val_loss: 0.031151 | Time: 184090 ms
[2022-04-06 04:50:54 main:574] : INFO : Epoch 121 | loss: 0.0311246 | val_loss: 0.0311532 | Time: 162852 ms
[2022-04-06 04:51:27 main:574] : INFO : Epoch 122 | loss: 0.0311253 | val_loss: 0.0311528 | Time: 33272.3 ms
[2022-04-06 04:54:33 main:574] : INFO : Epoch 123 | loss: 0.0311231 | val_loss: 0.0311548 | Time: 180512 ms
[2022-04-06 04:57:03 main:574] : INFO : Epoch 124 | loss: 0.0311221 | val_loss: 0.0311544 | Time: 149851 ms
[2022-04-06 05:00:26 main:574] : INFO : Epoch 125 | loss: 0.0311227 | val_loss: 0.0311528 | Time: 197097 ms
[2022-04-06 05:02:56 main:574] : INFO : Epoch 126 | loss: 0.031124 | val_loss: 0.0311547 | Time: 144971 ms
[2022-04-06 05:03:18 main:574] : INFO : Epoch 127 | loss: 0.0311288 | val_loss: 0.0311581 | Time: 22241.2 ms
[2022-04-06 05:05:58 main:574] : INFO : Epoch 128 | loss: 0.0311296 | val_loss: 0.0311526 | Time: 160058 ms
[2022-04-06 05:07:28 main:574] : INFO : Epoch 129 | loss: 0.0311279 | val_loss: 0.0311511 | Time: 89901.4 ms
[2022-04-06 05:10:02 main:574] : INFO : Epoch 130 | loss: 0.0311268 | val_loss: 0.0311495 | Time: 153877 ms
[2022-04-06 05:11:22 main:574] : INFO : Epoch 131 | loss: 0.0311248 | val_loss: 0.0311524 | Time: 79751.3 ms
[2022-04-06 05:14:39 main:574] : INFO : Epoch 132 | loss: 0.0311292 | val_loss: 0.031153 | Time: 192395 ms
[2022-04-06 05:17:13 main:574] : INFO : Epoch 133 | loss: 0.0311315 | val_loss: 0.0311547 | Time: 149492 ms
[2022-04-06 05:17:19 main:574] : INFO : Epoch 134 | loss: 0.0311283 | val_loss: 0.0311557 | Time: 5346.67 ms
[2022-04-06 05:19:38 main:574] : INFO : Epoch 135 | loss: 0.0311368 | val_loss: 0.0311548 | Time: 138528 ms
[2022-04-06 05:22:37 main:574] : INFO : Epoch 136 | loss: 0.0311413 | val_loss: 0.0311572 | Time: 178122 ms
[2022-04-06 05:24:11 main:574] : INFO : Epoch 137 | loss: 0.0311376 | val_loss: 0.031152 | Time: 88556.4 ms
[2022-04-06 05:27:41 main:574] : INFO : Epoch 138 | loss: 0.0311317 | val_loss: 0.0311543 | Time: 204255 ms
[2022-04-06 05:30:45 main:574] : INFO : Epoch 139 | loss: 0.0311326 | val_loss: 0.0311489 | Time: 184051 ms
[2022-04-06 05:30:59 main:574] : INFO : Epoch 140 | loss: 0.0311296 | val_loss: 0.0311473 | Time: 13503.3 ms
[2022-04-06 05:33:03 main:574] : INFO : Epoch 141 | loss: 0.0311259 | val_loss: 0.0311485 | Time: 124531 ms
[2022-04-06 05:34:43 main:574] : INFO : Epoch 142 | loss: 0.0311237 | val_loss: 0.0311503 | Time: 99662.5 ms
[2022-04-06 05:35:26 main:574] : INFO : Epoch 143 | loss: 0.0311232 | val_loss: 0.0311522 | Time: 38315.3 ms
[2022-04-06 05:35:32 main:574] : INFO : Epoch 144 | loss: 0.0311249 | val_loss: 0.0311475 | Time: 5189.67 ms
[2022-04-06 05:35:39 main:574] : INFO : Epoch 145 | loss: 0.031124 | val_loss: 0.0311498 | Time: 6108.47 ms
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFF27294F69
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.17.0
Dump Timestamp : 04/06/22 05:39:09
Install Directory : C:\Program Files\BOINC\
Data Directory : C:\ProgramData\BOINC
Project Symstore :
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\16;C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome
ModLoad: 000000002c280000 00000000003df000 C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000294d0000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000028900000 00000000000be000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000027260000 00000000002c8000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 00000000283d0000 00000000001a0000 C:\WINDOWS\System32\USER32.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000026c90000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.1586) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.19041.1586 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1586
ModLoad: 0000000027740000 000000000002b000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.19041.1202 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1202
ModLoad: 0000000026f90000 000000000010b000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000026bf0000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 00000000270a0000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 0000000028850000 00000000000ae000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000293e0000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.19041.546
ModLoad: 0000000027770000 000000000009c000 C:\WINDOWS\System32\sechost.dll (6.2.19041.1586) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000027870000 0000000000125000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000006140000 0000000000058000 C:\ProgramData\BOINC\slots\16\c10.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000007c40000 0000000000092000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.28.29334.0) (-exported- Symbols Loaded)
Linked PDB Filename : D:\a01\_work\6\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
File Version : 14.28.29334.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29334.0
ModLoad: 000000001a230000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.19041.867 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.867
ModLoad: 000000000cd10000 0000000000019000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.28.29334.0) (-exported- Symbols Loaded)
Linked PDB Filename : D:\a01\_work\6\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
File Version : 14.28.29334.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29334.0
ModLoad: 000000000d570000 000000000000c000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.28.29334.0) (-exported- Symbols Loaded)
Linked PDB Filename : D:\a01\_work\6\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
File Version : 14.28.29334.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29334.0
ModLoad: 00000000aad10000 000000000abe7000 C:\ProgramData\BOINC\slots\16\torch_cpu.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000c0080000 00000000002db000 C:\ProgramData\BOINC\slots\16\fbgemm.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000e4610000 000000000020d000 C:\ProgramData\BOINC\slots\16\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
Linked PDB Filename : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
File Version : 20200205
Company Name : Intel Corporation
Product Name : Intel(R) OpenMP* Runtime Library
Product Version : 5.0
ModLoad: 00000000f9160000 0000000000040000 C:\ProgramData\BOINC\slots\16\asmjit.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000007af60000 0000000024b49000 C:\ProgramData\BOINC\slots\16\torch_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000f6390000 0000000000045000 C:\ProgramData\BOINC\slots\16\c10_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000bcbc0000 00000000034b2000 C:\ProgramData\BOINC\slots\16\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA RAND Library
Product Version : 6,14,11,1012
ModLoad: 0000000007ea0000 0000000000010000 C:\ProgramData\BOINC\slots\16\nvToolsExt64_1.dll (-exported- Symbols Loaded)
Linked PDB Filename : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb
ModLoad: 00000000b9b10000 00000000030a5000 C:\ProgramData\BOINC\slots\16\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1031
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA SPARSE BLAS Library
Product Version : 6,14,11,1031
ModLoad: 000000005fbb0000 000000001b3a4000 C:\ProgramData\BOINC\slots\16\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,10020
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.86 CUDNN Library
Product Version : 6,14,11,10020
ModLoad: 0000000056a10000 000000000919f000 C:\ProgramData\BOINC\slots\16\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA FFT Library
Product Version : 6,14,11,1012
ModLoad: 0000000052cb0000 0000000003d5b000 C:\ProgramData\BOINC\slots\16\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 00000000b6aa0000 0000000001eca000 C:\ProgramData\BOINC\slots\16\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 0000000028710000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000024a40000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000289c0000 0000000000470000 C:\WINDOWS\System32\Setupapi.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : setupapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000271a0000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : cfgmgr32.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 0000000026cc0000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000269b0000 0000000000033000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : devobj.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 00000000271f0000 0000000000068000 C:\WINDOWS\System32\WINTRUST.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : wintrust.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000026d80000 0000000000156000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.1320) (-exported- Symbols Loaded)
Linked PDB Filename : crypt32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000267d0000 0000000000012000 C:\WINDOWS\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msasn1.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000027a50000 0000000000744000 C:\WINDOWS\System32\Shell32.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : shell32.pdb
File Version : 10.0.19041.964 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.964
ModLoad: 0000000024d20000 0000000000794000 C:\WINDOWS\SYSTEM32\windows.storage.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : Windows.Storage.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000028e30000 0000000000354000 C:\WINDOWS\System32\combase.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : combase.pdb
File Version : 10.0.19041.1320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1320
ModLoad: 0000000026650000 000000000002c000 C:\WINDOWS\SYSTEM32\Wldp.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : WLDP.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000027690000 00000000000ad000 C:\WINDOWS\System32\SHCORE.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : shcore.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000275a0000 0000000000055000 C:\WINDOWS\System32\shlwapi.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shlwapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000b9840000 00000000002cc000 C:\WINDOWS\system32\nvcuda.dll (30.0.14.7208) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r470_00\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
File Version : 30.0.14.7208
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.4.136 driver
Product Version : 30.0.14.7208
ModLoad: 00000000216e0000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000a9b40000 00000000011c5000 C:\WINDOWS\system32\DriverStore\FileRepository\nvhdc.inf_amd64_312d953c81546f4f\nvcuda64.dll (30.0.14.7208) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r470_00\drivers\gpgpu\_out\wddm2_ms_shipping_wddm2_amd64_release\nvcuda.pdb
File Version : 30.0.14.7208
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.4.136 driver
Product Version : 30.0.14.7208
ModLoad: 000000001dc70000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : cryptnet.pdb
File Version : 10.0.19041.906 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.906
ModLoad: 0000000018c60000 0000000000148000 C:\WINDOWS\SYSTEM32\drvstore.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : drvstore.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 00000000265c0000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cryptbase.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000024640000 000000000009e000 C:\WINDOWS\system32\uxtheme.dll (6.2.19041.1266) (-exported- Symbols Loaded)
Linked PDB Filename : UxTheme.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000005070000 0000000000719000 C:\WINDOWS\SYSTEM32\nvapi64.dll (30.0.14.7208) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r470_00\drivers\nvapi\gpu\_out\wddm2_ms_shipping_wddm2_amd64_release\nvapi64.pdb
File Version : 30.0.14.7208
Company Name : NVIDIA Corporation
Product Name : NVIDIA Windows drivers
Product Version : 30.0.14.7208
ModLoad: 00000000285e0000 000000000012a000 C:\WINDOWS\System32\ole32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
Linked PDB Filename : ole32.pdb
File Version : 10.0.19041.1320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1320
ModLoad: 00000000219e0000 0000000000009000 C:\ProgramData\BOINC\slots\16\caffe2_nvrtc.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000501c0000 0000000000fca000 C:\ProgramData\BOINC\slots\16\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6.14.11.9000
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.89 NVRTC Library
Product Version : 6.14.11.9000
ModLoad: 0000000026cf0000 0000000000082000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.1415) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.19041.1415 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1415
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 3766, Write: 1897, Other 5632
- I/O Transfers Counters -
Read: 7190780, Write: 3815244, Other 48198
- Paged Pool Usage -
QuotaPagedPoolUsage: 3536200, QuotaPeakPagedPoolUsage: 3554120
QuotaNonPagedPoolUsage: 59288, QuotaPeakNonPagedPoolUsage: 59424
- Virtual Memory Usage -
VirtualSize: 416968704, PeakVirtualSize: 1032597504
- Pagefile Usage -
PagefileUsage: 416968704, PeakPagefileUsage: 416968704
- Working Set Size -
WorkingSetSize: -2130034688, PeakWorkingSetSize: -2056146944, PageFaultCount: 901395
*** Dump of thread ID 4328 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 119843752.000000, User Time: 2433125120.000000, Wait Time: 154588816.000000
- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFF27294F69
- Registers -
rax=000000008a2e4038 rbx=000000007e9db320 rcx=0000000058380860 rdx=00000000f6391999 rsi=000000008a2e4bc0 rdi=0000000019930520
r8=00000000bc566b20 r9=00000000f63c9008 r10=000000008a2e4ba0 r11=0000000000000002 r12=000000002681e200 r13=00000000d103bac0
r14=0000000026814800 r15=0000000000633450 rip=0000000027294f69 rsp=000000008a2e46e0 rbp=000000008a2e4950
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
- Callstack -
ChildEBP RetAddr Args to Child
8a2e47b0 0cd16220 7e7b1fc8 7e7b17f0 00000321 00000000 KERNELBASE!RaiseException+0x0
8a2e4810 7dcd8f53 7e7adbb8 3c000000 8a2e49b8 3b900000 VCRUNTIME140!_CxxThrowException+0x0
8a2e4cf0 7dd325ea 8a2e4db8 00000001 8a2e4db8 00000001 torch_cuda!at::native::_cudnn_rnn+0x0
8a2e4dd0 7dd40cbf 8a2e54c0 8a2e6cb8 8a2e5e10 00000001 torch_cuda!at::native::set_storage_cuda_+0x0
8a2e4eb0 b075b9fc 8a2e54c0 8a2e5ba0 00000000 8a2e4fc8 torch_cuda!at::native::set_storage_cuda_+0x0
8a2e5030 b07931a2 8a2e53c0 00000001 00000000 00000000 torch_cpu!at::native::mkldnn_sigmoid_+0x0
8a2e5310 b1a42613 00000000 8a2e5e20 8a2e5e20 8a2e5420 torch_cpu!at::_cudnn_rnn+0x0
8a2e5810 b1a5080b da1a9d70 d101d460 8a2e5940 00000004 torch_cpu!torch::autograd::NotImplemented::~NotImplemented+0x0
8a2e58f0 b075b9fc 8a2e5e70 b1b75878 8a2e5a50 a9b66167 torch_cpu!torch::autograd::GraphRoot::apply+0x0
8a2e5a70 b07931a2 ce4e0ed0 00000000 da1a92e0 7ebfa3f0 torch_cpu!at::native::mkldnn_sigmoid_+0x0
8a2e5d50 7dcd69a1 8a2e0001 00000001 00000004 00000004 torch_cpu!at::_cudnn_rnn+0x0
8a2e5f20 7dce0117 00000000 8a2e6cb8 f6392e0d 8a2e6040 torch_cuda!at::native::_use_cudnn_ctc_loss+0x0
8a2e5fb0 b04e107b b5631ac0 8a2e62d0 597dfe79 a9d2abc3 torch_cuda!at::native::_cudnn_rnn_flatten_weight+0x0
8a2e6220 b0513b16 00000001 8a2e6cb8 00000080 d103bdf8 torch_cpu!at::native::fbgemm_pack_quantized_matrix+0x0
8a2e63b0 b088ae89 8a2e6428 00000001 8a2e6428 00000001 torch_cpu!at::native::gru+0x0
8a2e6440 b1b79fa3 8a2e6540 00000001 8a2e66c0 8a2e6cb8 torch_cpu!at::zeros_out+0x0
8a2e64d0 b087dd35 06191ea8 596aa4f0 00000000 06191ea8 torch_cpu!torch::autograd::GraphRoot::apply+0x0
8a2e6550 b07588ec b56375c8 1e6c7170 cd1c9090 8a2e6770 torch_cpu!at::zeros_out+0x0
8a2e6650 b07cc1bb 0000000c 00000080 00000004 d101d800 torch_cpu!at::native::mkldnn_sigmoid_+0x0
8a2e66d0 b22b7294 8a2e67f0 1e6c7170 8a2e67e0 8a2e6988 torch_cpu!at::gru+0x0
8a2e6900 b22b62ab 8a2e6c98 b06e030e 8a2e6c98 8a2e6cb8 torch_cpu!torch::nn::GRUImpl::forward_helper+0x0
8a2e69f0 2c292621 00000002 d15e5e00 00000000 06191ea8 torch_cpu!torch::nn::GRUImpl::forward+0x0
8a2e6bf0 2c297e98 8a2e6c48 8a2e0000 d15e0601 d15e5e00 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::error+0x0
8a2e7090 2c29b964 8a2e7190 8a2e7190 00000000 32cd25b0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Object::unsafeRemoveSlot+0x0
8a2ffe30 2c4dc6f8 59c4bdc0 00000000 00000000 5988d8e0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
8a2ffe70 28917034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
8a2ffea0 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
8a2fff20 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 4808 (state: Waiting): ***
- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 154588816.000000
- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=000000008a7ff9e0 rsi=0000000000000000 rdi=0000000000000064
r8=000000008a7ff9b8 r9=0000000000000000 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=000000002956d3f4 rsp=000000008a7ff9b8 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
8a7ff9b0 272a962e 8a7ffa78 00000000 00000000 00000000 ntdll!NtDelayExecution+0x0
8a7ffa50 2c2ba07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0
8a7ffa80 28917034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
8a7ffab0 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
8a7ffb30 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 12552 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 154588816.000000
- Registers -
rax=000000000000005b rbx=000000000000000e rcx=000000000000000e rdx=000000008a8ffb10 rsi=0000000000000000 rdi=000000000000000e
r8=0000000000000001 r9=0000000000000000 r10=0000000000000000 r11=000000008a8ffa80 r12=0000000000000064 r13=000000008a8ffb10
r14=000000008a8ff810 r15=0000000000000000 rip=000000002956d8c4 rsp=000000008a8ff7b8 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
8a8ff7b0 272acb20 00000000 8a094000 2963a4d0 8a094000 ntdll!ZwWaitForMultipleObjects+0x0
8a8ffaa0 272aca1e 58115ef0 004e0049 0000000e 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0
8a8ffae0 a9d11224 59d74ef0 00000000 00000000 00000000 KERNELBASE!WaitForMultipleObjects+0x0
8a8ffd20 a9db7fcd 00000000 59d71d10 00000000 00000000 nvcuda64!+0x0
8a8ffdd0 a9d10bf3 e8f0c8d8 dc5a12c8 5a306ba0 00000000 nvcuda64!cuProfilerStop+0x0
8a8ffe00 aa0458a4 5a306ba0 00000000 00000000 00000000 nvcuda64!+0x0
8a8ffe30 28917034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
8a8ffe60 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
8a8ffee0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 1540 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 153544912.000000
- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=00000000000002d4 rdx=0000000000000000 rsi=0000000000000000 rdi=00000000000002d4
r8=000000008a9ff6d8 r9=0000000000000000 r10=0000000000000000 r11=000000008a9ff738 r12=0000000000000000 r13=0000000000000000
r14=00000000000002d4 r15=0000000000000000 rip=000000002956cdf4 rsp=000000008a9ff438 rbp=0000000059d37330
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
8a9ff430 27281a5e 00000000 29522630 0000027f 00000000 ntdll!ZwWaitForSingleObject+0x0
8a9ff4d0 a9f3d50d 5a306990 00000000 00000000 000002d4 KERNELBASE!WaitForSingleObjectEx+0x0
8a9ff730 a9d10bf3 00000000 5a306990 a9b40000 59d8faf0 nvcuda64!cuProfilerStop+0x0
8a9ff760 aa0458a4 5a306990 00000000 00000000 00000000 nvcuda64!+0x0
8a9ff790 28917034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
8a9ff7c0 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
8a9ff840 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 17540 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 70156248.000000, User Time: 1851718784.000000, Wait Time: 154585296.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=00000000af2eec88 rdx=0000000000000000 rsi=0000000000000000 rdi=00000000af2eec88
r8=0000000000000004 r9=0000000000000000 r10=0000000000000440 r11=00000000d103cde0 r12=0000000000000000 r13=0000000000000000
r14=000000008abff560 r15=00000000af2eec38 rip=0000000029570764 rsp=000000008abff538 rbp=000000008abff590
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
8abff530 29534021 bc91fa88 07c52f97 cc12ec60 cc12ec50 ntdll!NtWaitForAlertByThreadId+0x0
8abff5b0 272ccfa9 af2eec80 af2eec78 af2eec30 00000000 ntdll!RtlSleepConditionVariableSRW+0x0
8abff5f0 07c7bf3d cc12ec80 00000000 00000000 b1ef05b4 KERNELBASE!SleepConditionVariableSRW+0x0
8abff630 07c52798 00000000 cc12ec80 00000000 00000000 MSVCP140!__crtSetThreadpoolWait+0x0
8abff660 07c52a4a 2e17b7a3 00000000 8abff870 af2eec78 MSVCP140!std::_Winerror_message+0x0
8abff690 b1ef064b af2eec30 8abff790 af2eec30 cc12ec60 MSVCP140!_Cnd_wait+0x0
8abff6d0 b1ef25b1 d1e5a7f0 ffffffff cc12ec60 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0
8abff9c0 b1ef2471 b5416d10 8abffa00 b5416d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0
8abffa10 b1ee9744 00080001 cead8e20 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0
8abffa40 270c1bb2 cead8e20 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0
8abffa70 28917034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0
8abffaa0 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
8abffb20 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 19896 (state: Waiting): ***
- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 154587504.000000
- Registers -
rax=00000000000001d3 rbx=0000000024791b00 rcx=0000000000000070 rdx=0000000024791780 rsi=00000000581033a0 rdi=0000000024791b00
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000295707c4 rsp=000000008a3ff888 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
8a3ff880 29522dc7 00000000 00000000 581033a0 00000000 ntdll!ZwWaitForWorkViaWorkerFactory+0x0
8a3ffb80 28917034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0
8a3ffbb0 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
8a3ffc30 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)