Task 5547634

Name ParityMachine-1625098352-17862-0_0
Workunit 3396460
Created 1 Jul 2021, 0:12:34 UTC
Sent 3 Jul 2021, 21:18:45 UTC
Report deadline 11 Jul 2021, 21:18:45 UTC
Received 3 Jul 2021, 22:45:30 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -529697949 (0xE06D7363) Unknown error code
Computer ID 7258
Run time 26 min 56 sec
CPU time 26 min 56 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 1,132.78 GFLOPS
Application version Machine Learning Dataset Generator (GPU) v9.75 (cuda10200)
windows_x86_64
Peak working set size 2.07 GB
Peak swap size 3.57 GB
Peak disk usage 1.53 GB

Stderr output

<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 3765269347 (0xe06d7363)</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GT 1030)
[2021-07-04 00:17:17	                main:435]	:	INFO	:	Set logging level to 1
[2021-07-04 00:17:17	                main:441]	:	INFO	:	Running in BOINC Client mode
[2021-07-04 00:17:17	                main:444]	:	INFO	:	Resolving all filenames
[2021-07-04 00:17:17	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 (exists = 1)
[2021-07-04 00:17:17	                main:452]	:	INFO	:	Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1625098352-17862-0_0_r271966573_1 (exists = 0)
[2021-07-04 00:17:17	                main:452]	:	INFO	:	Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1625098352-17862-0_0_r271966573_0 (exists = 0)
[2021-07-04 00:17:17	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 0)
[2021-07-04 00:17:17	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2021-07-04 00:17:17	                main:472]	:	INFO	:	Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5
[2021-07-04 00:17:17	                main:474]	:	INFO	:	Configuration: 
[2021-07-04 00:17:17	                main:475]	:	INFO	:	    Model type: GRU
[2021-07-04 00:17:17	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2021-07-04 00:17:17	                main:477]	:	INFO	:	    Max Epochs: 2048
[2021-07-04 00:17:17	                main:478]	:	INFO	:	    Batch Size: 128
[2021-07-04 00:17:17	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2021-07-04 00:17:17	                main:480]	:	INFO	:	    Patience: 10
[2021-07-04 00:17:17	                main:481]	:	INFO	:	    Hidden Width: 12
[2021-07-04 00:17:17	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2021-07-04 00:17:17	                main:483]	:	INFO	:	    # Backend Layers: 4
[2021-07-04 00:17:17	                main:484]	:	INFO	:	    # Threads: 1
[2021-07-04 00:17:17	                main:486]	:	INFO	:	Preparing Dataset
[2021-07-04 00:17:17	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-04 00:17:18	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-04 00:17:24	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2021-07-04 00:17:24	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-04 00:17:24	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-04 00:17:25	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2021-07-04 00:17:25	                main:494]	:	INFO	:	Creating Model
[2021-07-04 00:17:25	                main:507]	:	INFO	:	Preparing config file
[2021-07-04 00:17:25	                main:519]	:	INFO	:	Creating new config file
[2021-07-04 00:17:28	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2021-07-04 00:17:28	                main:562]	:	INFO	:	Starting Training
[2021-07-04 00:17:37	                main:574]	:	INFO	:	Epoch 1 | loss: 0.0440539 | val_loss: 0.0320175 | Time: 9279.71 ms
[2021-07-04 00:17:46	                main:574]	:	INFO	:	Epoch 2 | loss: 0.0316276 | val_loss: 0.0313583 | Time: 8817.72 ms
[2021-07-04 00:17:54	                main:574]	:	INFO	:	Epoch 3 | loss: 0.0313139 | val_loss: 0.0312837 | Time: 8600.97 ms
[2021-07-04 00:18:02	                main:574]	:	INFO	:	Epoch 4 | loss: 0.0312677 | val_loss: 0.0312564 | Time: 7422.26 ms
[2021-07-04 00:18:10	                main:574]	:	INFO	:	Epoch 5 | loss: 0.0312533 | val_loss: 0.0312513 | Time: 8561.17 ms
[2021-07-04 00:18:19	                main:574]	:	INFO	:	Epoch 6 | loss: 0.0312508 | val_loss: 0.031251 | Time: 8557.81 ms
[2021-07-04 00:18:27	                main:574]	:	INFO	:	Epoch 7 | loss: 0.0312507 | val_loss: 0.0312511 | Time: 8565.86 ms
[2021-07-04 00:18:36	                main:574]	:	INFO	:	Epoch 8 | loss: 0.0312506 | val_loss: 0.0312509 | Time: 8556.73 ms
[2021-07-04 00:18:45	                main:574]	:	INFO	:	Epoch 9 | loss: 0.0312504 | val_loss: 0.0312509 | Time: 8563.66 ms
[2021-07-04 00:18:52	                main:574]	:	INFO	:	Epoch 10 | loss: 0.0312507 | val_loss: 0.0312505 | Time: 7378.74 ms
[2021-07-04 00:19:00	                main:574]	:	INFO	:	Epoch 11 | loss: 0.0312505 | val_loss: 0.031252 | Time: 8505.89 ms
[2021-07-04 00:19:09	                main:574]	:	INFO	:	Epoch 12 | loss: 0.0312508 | val_loss: 0.0312506 | Time: 8561.95 ms
[2021-07-04 00:19:18	                main:574]	:	INFO	:	Epoch 13 | loss: 0.0312503 | val_loss: 0.0312502 | Time: 8603.97 ms
[2021-07-04 00:19:26	                main:574]	:	INFO	:	Epoch 14 | loss: 0.0312503 | val_loss: 0.0312508 | Time: 8614.05 ms
[2021-07-04 00:19:35	                main:574]	:	INFO	:	Epoch 15 | loss: 0.0312505 | val_loss: 0.0312509 | Time: 8533.71 ms
[2021-07-04 00:19:42	                main:574]	:	INFO	:	Epoch 16 | loss: 0.0312506 | val_loss: 0.0312512 | Time: 7378.06 ms
[2021-07-04 00:19:51	                main:574]	:	INFO	:	Epoch 17 | loss: 0.0312506 | val_loss: 0.0312516 | Time: 8534.9 ms
[2021-07-04 00:19:59	                main:574]	:	INFO	:	Epoch 18 | loss: 0.0312504 | val_loss: 0.0312508 | Time: 8596.03 ms
[2021-07-04 00:20:08	                main:574]	:	INFO	:	Epoch 19 | loss: 0.0312519 | val_loss: 0.0312517 | Time: 8550.83 ms
[2021-07-04 00:20:16	                main:574]	:	INFO	:	Epoch 20 | loss: 0.0312511 | val_loss: 0.0312507 | Time: 8526.58 ms
[2021-07-04 00:20:25	                main:574]	:	INFO	:	Epoch 21 | loss: 0.0312503 | val_loss: 0.03125 | Time: 8506.38 ms
[2021-07-04 00:20:33	                main:574]	:	INFO	:	Epoch 22 | loss: 0.03125 | val_loss: 0.0312524 | Time: 8575.8 ms
[2021-07-04 00:20:41	                main:574]	:	INFO	:	Epoch 23 | loss: 0.0312541 | val_loss: 0.0312557 | Time: 7400.28 ms
[2021-07-04 00:20:49	                main:574]	:	INFO	:	Epoch 24 | loss: 0.0312529 | val_loss: 0.031255 | Time: 8507.17 ms
[2021-07-04 00:20:58	                main:574]	:	INFO	:	Epoch 25 | loss: 0.0312507 | val_loss: 0.0312499 | Time: 8572.8 ms
[2021-07-04 00:21:07	                main:574]	:	INFO	:	Epoch 26 | loss: 0.0312509 | val_loss: 0.0312494 | Time: 8581.03 ms
[2021-07-04 00:21:15	                main:574]	:	INFO	:	Epoch 27 | loss: 0.0312499 | val_loss: 0.0312496 | Time: 8526.57 ms
[2021-07-04 00:21:24	                main:574]	:	INFO	:	Epoch 28 | loss: 0.0312495 | val_loss: 0.0312502 | Time: 8599.2 ms
[2021-07-04 00:21:31	                main:574]	:	INFO	:	Epoch 29 | loss: 0.0312495 | val_loss: 0.0312505 | Time: 7363.45 ms
[2021-07-04 00:21:40	                main:574]	:	INFO	:	Epoch 30 | loss: 0.0312509 | val_loss: 0.0312539 | Time: 8532.81 ms
[2021-07-04 00:21:48	                main:574]	:	INFO	:	Epoch 31 | loss: 0.0312497 | val_loss: 0.0312473 | Time: 8556.66 ms
[2021-07-04 00:21:57	                main:574]	:	INFO	:	Epoch 32 | loss: 0.0312469 | val_loss: 0.0312456 | Time: 8597.76 ms
[2021-07-04 00:22:05	                main:574]	:	INFO	:	Epoch 33 | loss: 0.0312453 | val_loss: 0.0312457 | Time: 8523.5 ms
[2021-07-04 00:22:14	                main:574]	:	INFO	:	Epoch 34 | loss: 0.0312424 | val_loss: 0.0312452 | Time: 8557.53 ms
[2021-07-04 00:22:22	                main:574]	:	INFO	:	Epoch 35 | loss: 0.0312407 | val_loss: 0.0312368 | Time: 8641.87 ms
[2021-07-04 00:22:30	                main:574]	:	INFO	:	Epoch 36 | loss: 0.0312421 | val_loss: 0.0312464 | Time: 7427.14 ms
[2021-07-04 00:22:38	                main:574]	:	INFO	:	Epoch 37 | loss: 0.0312376 | val_loss: 0.0312294 | Time: 8512.84 ms
[2021-07-04 00:22:47	                main:574]	:	INFO	:	Epoch 38 | loss: 0.0312237 | val_loss: 0.0312228 | Time: 8525.91 ms
[2021-07-04 00:22:55	                main:574]	:	INFO	:	Epoch 39 | loss: 0.0312589 | val_loss: 0.031308 | Time: 8541.35 ms
[2021-07-04 00:23:04	                main:574]	:	INFO	:	Epoch 40 | loss: 0.0312428 | val_loss: 0.031227 | Time: 8502.98 ms
[2021-07-04 00:23:13	                main:574]	:	INFO	:	Epoch 41 | loss: 0.0312147 | val_loss: 0.031208 | Time: 8521.87 ms
[2021-07-04 00:23:20	                main:574]	:	INFO	:	Epoch 42 | loss: 0.0311988 | val_loss: 0.0312091 | Time: 7395.04 ms
[2021-07-04 00:23:28	                main:574]	:	INFO	:	Epoch 43 | loss: 0.0312017 | val_loss: 0.0311818 | Time: 8530.57 ms
[2021-07-04 00:23:36	                main:574]	:	INFO	:	Epoch 44 | loss: 0.0311835 | val_loss: 0.0311776 | Time: 7446.63 ms
[2021-07-04 00:23:40	                main:574]	:	INFO	:	Epoch 45 | loss: 0.0311906 | val_loss: 0.0312199 | Time: 3853.38 ms
[2021-07-04 00:23:47	                main:574]	:	INFO	:	Epoch 46 | loss: 0.0311916 | val_loss: 0.0311919 | Time: 6925.91 ms
[2021-07-04 00:23:55	                main:574]	:	INFO	:	Epoch 47 | loss: 0.0311844 | val_loss: 0.0311756 | Time: 8473.75 ms
[2021-07-04 00:24:04	                main:574]	:	INFO	:	Epoch 48 | loss: 0.0311847 | val_loss: 0.0311749 | Time: 8566.59 ms
[2021-07-04 00:24:11	                main:574]	:	INFO	:	Epoch 49 | loss: 0.0311747 | val_loss: 0.0311711 | Time: 7323.93 ms
[2021-07-04 00:24:20	                main:574]	:	INFO	:	Epoch 50 | loss: 0.0311738 | val_loss: 0.0311885 | Time: 8525.31 ms
[2021-07-04 00:24:28	                main:574]	:	INFO	:	Epoch 51 | loss: 0.0311814 | val_loss: 0.0311865 | Time: 8539.6 ms
[2021-07-04 00:24:37	                main:574]	:	INFO	:	Epoch 52 | loss: 0.0311717 | val_loss: 0.0311792 | Time: 8554.46 ms
[2021-07-04 00:24:45	                main:574]	:	INFO	:	Epoch 53 | loss: 0.0311712 | val_loss: 0.0311694 | Time: 8557.61 ms
[2021-07-04 00:24:54	                main:574]	:	INFO	:	Epoch 54 | loss: 0.0311674 | val_loss: 0.0311713 | Time: 8503.85 ms
[2021-07-04 00:25:01	                main:574]	:	INFO	:	Epoch 55 | loss: 0.0311691 | val_loss: 0.0311699 | Time: 7349.08 ms
[2021-07-04 00:25:10	                main:574]	:	INFO	:	Epoch 56 | loss: 0.0311679 | val_loss: 0.0311741 | Time: 8554.59 ms
[2021-07-04 00:25:18	                main:574]	:	INFO	:	Epoch 57 | loss: 0.0311676 | val_loss: 0.0311707 | Time: 8529.84 ms
[2021-07-04 00:25:27	                main:574]	:	INFO	:	Epoch 58 | loss: 0.0311728 | val_loss: 0.0311761 | Time: 8521.52 ms
[2021-07-04 00:25:35	                main:574]	:	INFO	:	Epoch 59 | loss: 0.0311725 | val_loss: 0.0311843 | Time: 8516.02 ms
[2021-07-04 00:25:43	                main:574]	:	INFO	:	Epoch 60 | loss: 0.0311736 | val_loss: 0.0311684 | Time: 7295.81 ms
[2021-07-04 00:25:51	                main:574]	:	INFO	:	Epoch 61 | loss: 0.031177 | val_loss: 0.0311731 | Time: 8499.05 ms
[2021-07-04 00:26:00	                main:574]	:	INFO	:	Epoch 62 | loss: 0.0311717 | val_loss: 0.031169 | Time: 8585.33 ms
[2021-07-04 00:26:08	                main:574]	:	INFO	:	Epoch 63 | loss: 0.031167 | val_loss: 0.0311686 | Time: 8561.06 ms
[2021-07-04 00:26:17	                main:574]	:	INFO	:	Epoch 64 | loss: 0.0311653 | val_loss: 0.0311711 | Time: 8533.13 ms
[2021-07-04 00:26:25	                main:574]	:	INFO	:	Epoch 65 | loss: 0.0311648 | val_loss: 0.0311695 | Time: 8477.43 ms
[2021-07-04 00:26:33	                main:574]	:	INFO	:	Epoch 66 | loss: 0.031165 | val_loss: 0.0311685 | Time: 7331.78 ms
[2021-07-04 00:26:41	                main:574]	:	INFO	:	Epoch 67 | loss: 0.0311671 | val_loss: 0.0311713 | Time: 8529.53 ms
[2021-07-04 00:26:50	                main:574]	:	INFO	:	Epoch 68 | loss: 0.0311683 | val_loss: 0.0311761 | Time: 8613.41 ms
[2021-07-04 00:26:58	                main:574]	:	INFO	:	Epoch 69 | loss: 0.0311737 | val_loss: 0.0311693 | Time: 8536.71 ms
[2021-07-04 00:27:07	                main:574]	:	INFO	:	Epoch 70 | loss: 0.0311714 | val_loss: 0.0312081 | Time: 8513.23 ms
[2021-07-04 00:27:15	                main:574]	:	INFO	:	Epoch 71 | loss: 0.0311876 | val_loss: 0.0311698 | Time: 8640.04 ms
[2021-07-04 00:27:23	                main:574]	:	INFO	:	Epoch 72 | loss: 0.0311727 | val_loss: 0.0311701 | Time: 7415.78 ms
[2021-07-04 00:27:31	                main:574]	:	INFO	:	Epoch 73 | loss: 0.0311699 | val_loss: 0.0311702 | Time: 8493.75 ms
[2021-07-04 00:27:40	                main:574]	:	INFO	:	Epoch 74 | loss: 0.0311681 | val_loss: 0.0311728 | Time: 8581.98 ms
[2021-07-04 00:27:48	                main:574]	:	INFO	:	Epoch 75 | loss: 0.0311703 | val_loss: 0.0311723 | Time: 8578.46 ms
[2021-07-04 00:27:57	                main:574]	:	INFO	:	Epoch 76 | loss: 0.0311628 | val_loss: 0.0311688 | Time: 8470.02 ms
[2021-07-04 00:28:06	                main:574]	:	INFO	:	Epoch 77 | loss: 0.0311613 | val_loss: 0.0311744 | Time: 8585.2 ms
[2021-07-04 00:28:14	                main:574]	:	INFO	:	Epoch 78 | loss: 0.0311607 | val_loss: 0.0311684 | Time: 8614.06 ms
[2021-07-04 00:28:22	                main:574]	:	INFO	:	Epoch 79 | loss: 0.0311614 | val_loss: 0.0311734 | Time: 7377.44 ms
[2021-07-04 00:28:30	                main:574]	:	INFO	:	Epoch 80 | loss: 0.031161 | val_loss: 0.03117 | Time: 8573.45 ms
[2021-07-04 00:28:39	                main:574]	:	INFO	:	Epoch 81 | loss: 0.0311647 | val_loss: 0.0311696 | Time: 8578.98 ms
[2021-07-04 00:28:47	                main:574]	:	INFO	:	Epoch 82 | loss: 0.0311615 | val_loss: 0.031168 | Time: 8552.84 ms
[2021-07-04 00:28:56	                main:574]	:	INFO	:	Epoch 83 | loss: 0.0311609 | val_loss: 0.0311737 | Time: 8557.68 ms
[2021-07-04 00:29:04	                main:574]	:	INFO	:	Epoch 84 | loss: 0.0311718 | val_loss: 0.0311724 | Time: 8543.3 ms
[2021-07-04 00:29:12	                main:574]	:	INFO	:	Epoch 85 | loss: 0.0311641 | val_loss: 0.0311682 | Time: 7365.77 ms
[2021-07-04 00:29:20	                main:574]	:	INFO	:	Epoch 86 | loss: 0.0311581 | val_loss: 0.0311679 | Time: 8546.58 ms
[2021-07-04 00:29:29	                main:574]	:	INFO	:	Epoch 87 | loss: 0.0311614 | val_loss: 0.0311676 | Time: 8467.51 ms
[2021-07-04 00:29:38	                main:574]	:	INFO	:	Epoch 88 | loss: 0.0311584 | val_loss: 0.0311686 | Time: 8229.57 ms
[2021-07-04 00:29:46	                main:574]	:	INFO	:	Epoch 89 | loss: 0.0311568 | val_loss: 0.0311666 | Time: 8558.41 ms
[2021-07-04 00:29:55	                main:574]	:	INFO	:	Epoch 90 | loss: 0.0311612 | val_loss: 0.0311684 | Time: 8567.66 ms
[2021-07-04 00:30:04	                main:574]	:	INFO	:	Epoch 91 | loss: 0.031165 | val_loss: 0.0311694 | Time: 8555.52 ms
[2021-07-04 00:30:12	                main:574]	:	INFO	:	Epoch 92 | loss: 0.0311585 | val_loss: 0.0311764 | Time: 8519.94 ms
[2021-07-04 00:30:21	                main:574]	:	INFO	:	Epoch 93 | loss: 0.0311582 | val_loss: 0.0311663 | Time: 8576.66 ms
[2021-07-04 00:30:28	                main:574]	:	INFO	:	Epoch 94 | loss: 0.0311542 | val_loss: 0.0311667 | Time: 7394.72 ms
[2021-07-04 00:30:37	                main:574]	:	INFO	:	Epoch 95 | loss: 0.0311536 | val_loss: 0.0311657 | Time: 8567.43 ms
[2021-07-04 00:30:45	                main:574]	:	INFO	:	Epoch 96 | loss: 0.0311538 | val_loss: 0.0311707 | Time: 8547.18 ms
[2021-07-04 00:30:54	                main:574]	:	INFO	:	Epoch 97 | loss: 0.0311526 | val_loss: 0.0311651 | Time: 8557.27 ms
[2021-07-04 00:31:02	                main:574]	:	INFO	:	Epoch 98 | loss: 0.03115 | val_loss: 0.031165 | Time: 8569.54 ms
[2021-07-04 00:31:11	                main:574]	:	INFO	:	Epoch 99 | loss: 0.0311498 | val_loss: 0.0311638 | Time: 8504.03 ms
[2021-07-04 00:31:19	                main:574]	:	INFO	:	Epoch 100 | loss: 0.03115 | val_loss: 0.0311679 | Time: 8618.54 ms
[2021-07-04 00:31:27	                main:574]	:	INFO	:	Epoch 101 | loss: 0.031148 | val_loss: 0.0311638 | Time: 7397.44 ms
[2021-07-04 00:31:35	                main:574]	:	INFO	:	Epoch 102 | loss: 0.0311483 | val_loss: 0.031164 | Time: 8540.6 ms
[2021-07-04 00:31:44	                main:574]	:	INFO	:	Epoch 103 | loss: 0.0311471 | val_loss: 0.0311655 | Time: 8540.57 ms
[2021-07-04 00:31:52	                main:574]	:	INFO	:	Epoch 104 | loss: 0.0311489 | val_loss: 0.031164 | Time: 8557.16 ms
[2021-07-04 00:32:01	                main:574]	:	INFO	:	Epoch 105 | loss: 0.0311462 | val_loss: 0.0311629 | Time: 8525.54 ms
[2021-07-04 00:32:10	                main:574]	:	INFO	:	Epoch 106 | loss: 0.0311468 | val_loss: 0.0311642 | Time: 8613.82 ms
[2021-07-04 00:32:18	                main:574]	:	INFO	:	Epoch 107 | loss: 0.0311435 | val_loss: 0.0311622 | Time: 8525.61 ms
[2021-07-04 00:32:25	                main:574]	:	INFO	:	Epoch 108 | loss: 0.0311431 | val_loss: 0.0311608 | Time: 7339.95 ms
[2021-07-04 00:32:34	                main:574]	:	INFO	:	Epoch 109 | loss: 0.0311433 | val_loss: 0.0311674 | Time: 8608.39 ms
[2021-07-04 00:32:43	                main:574]	:	INFO	:	Epoch 110 | loss: 0.0311426 | val_loss: 0.0311602 | Time: 8544.22 ms
[2021-07-04 00:32:51	                main:574]	:	INFO	:	Epoch 111 | loss: 0.0311389 | val_loss: 0.0311651 | Time: 8529.94 ms
[2021-07-04 00:33:00	                main:574]	:	INFO	:	Epoch 112 | loss: 0.0311404 | val_loss: 0.0311637 | Time: 8480 ms
[2021-07-04 00:33:07	                main:574]	:	INFO	:	Epoch 113 | loss: 0.0311449 | val_loss: 0.0311598 | Time: 7391.12 ms
[2021-07-04 00:33:16	                main:574]	:	INFO	:	Epoch 114 | loss: 0.031138 | val_loss: 0.0311662 | Time: 8577.55 ms
[2021-07-04 00:33:24	                main:574]	:	INFO	:	Epoch 115 | loss: 0.0311369 | val_loss: 0.0311636 | Time: 8580.75 ms
[2021-07-04 00:33:33	                main:574]	:	INFO	:	Epoch 116 | loss: 0.031135 | val_loss: 0.0311626 | Time: 8549.96 ms
[2021-07-04 00:33:41	                main:574]	:	INFO	:	Epoch 117 | loss: 0.0311346 | val_loss: 0.031172 | Time: 8573.42 ms
[2021-07-04 00:33:50	                main:574]	:	INFO	:	Epoch 118 | loss: 0.031134 | val_loss: 0.0311619 | Time: 8550.39 ms
[2021-07-04 00:33:58	                main:574]	:	INFO	:	Epoch 119 | loss: 0.0311338 | val_loss: 0.0311613 | Time: 8567.34 ms
[2021-07-04 00:34:06	                main:574]	:	INFO	:	Epoch 120 | loss: 0.0311333 | val_loss: 0.0311691 | Time: 7397.36 ms
[2021-07-04 00:34:14	                main:574]	:	INFO	:	Epoch 121 | loss: 0.0311349 | val_loss: 0.0311589 | Time: 8534.91 ms
[2021-07-04 00:34:23	                main:574]	:	INFO	:	Epoch 122 | loss: 0.03113 | val_loss: 0.0311667 | Time: 8566.72 ms
[2021-07-04 00:34:32	                main:574]	:	INFO	:	Epoch 123 | loss: 0.0311302 | val_loss: 0.0311647 | Time: 8592.59 ms
[2021-07-04 00:34:40	                main:574]	:	INFO	:	Epoch 124 | loss: 0.0311418 | val_loss: 0.031178 | Time: 8624.88 ms
[2021-07-04 00:34:49	                main:574]	:	INFO	:	Epoch 125 | loss: 0.0311304 | val_loss: 0.0311703 | Time: 8489.71 ms
[2021-07-04 00:34:56	                main:574]	:	INFO	:	Epoch 126 | loss: 0.0311267 | val_loss: 0.0311646 | Time: 7436.15 ms
[2021-07-04 00:35:05	                main:574]	:	INFO	:	Epoch 127 | loss: 0.0311246 | val_loss: 0.0311627 | Time: 8536.03 ms
[2021-07-04 00:35:13	                main:574]	:	INFO	:	Epoch 128 | loss: 0.0311238 | val_loss: 0.0311626 | Time: 8540.41 ms
[2021-07-04 00:35:22	                main:574]	:	INFO	:	Epoch 129 | loss: 0.0311259 | val_loss: 0.0311791 | Time: 8585.2 ms
[2021-07-04 00:35:30	                main:574]	:	INFO	:	Epoch 130 | loss: 0.0311229 | val_loss: 0.031166 | Time: 8515.28 ms
[2021-07-04 00:35:39	                main:574]	:	INFO	:	Epoch 131 | loss: 0.0311195 | val_loss: 0.0311742 | Time: 8568.35 ms
[2021-07-04 00:35:48	                main:574]	:	INFO	:	Epoch 132 | loss: 0.0311225 | val_loss: 0.0311702 | Time: 8581.76 ms
[2021-07-04 00:35:55	                main:574]	:	INFO	:	Epoch 133 | loss: 0.0311163 | val_loss: 0.0311686 | Time: 7371.74 ms
[2021-07-04 00:36:03	                main:574]	:	INFO	:	Epoch 134 | loss: 0.0311137 | val_loss: 0.0311842 | Time: 8528.06 ms
[2021-07-04 00:36:12	                main:574]	:	INFO	:	Epoch 135 | loss: 0.031113 | val_loss: 0.0311735 | Time: 8574.94 ms
[2021-07-04 00:36:21	                main:574]	:	INFO	:	Epoch 136 | loss: 0.0311164 | val_loss: 0.0311757 | Time: 8520.22 ms
[2021-07-04 00:36:29	                main:574]	:	INFO	:	Epoch 137 | loss: 0.0311129 | val_loss: 0.0311872 | Time: 8533.65 ms
[2021-07-04 00:36:38	                main:574]	:	INFO	:	Epoch 138 | loss: 0.031115 | val_loss: 0.0311709 | Time: 8582.48 ms
[2021-07-04 00:36:45	                main:574]	:	INFO	:	Epoch 139 | loss: 0.031115 | val_loss: 0.0311769 | Time: 7320.24 ms
[2021-07-04 00:36:54	                main:574]	:	INFO	:	Epoch 140 | loss: 0.0311061 | val_loss: 0.0311759 | Time: 8549.11 ms
[2021-07-04 00:37:02	                main:574]	:	INFO	:	Epoch 141 | loss: 0.0311061 | val_loss: 0.0311786 | Time: 8544.49 ms
[2021-07-04 00:37:11	                main:574]	:	INFO	:	Epoch 142 | loss: 0.0311064 | val_loss: 0.0311629 | Time: 8554.12 ms
[2021-07-04 00:37:19	                main:574]	:	INFO	:	Epoch 143 | loss: 0.0311113 | val_loss: 0.0311871 | Time: 8500.9 ms
[2021-07-04 00:37:28	                main:574]	:	INFO	:	Epoch 144 | loss: 0.031105 | val_loss: 0.0312018 | Time: 8556.86 ms
[2021-07-04 00:37:35	                main:574]	:	INFO	:	Epoch 145 | loss: 0.0311064 | val_loss: 0.0311785 | Time: 7361.07 ms
[2021-07-04 00:37:44	                main:574]	:	INFO	:	Epoch 146 | loss: 0.0311029 | val_loss: 0.0311827 | Time: 8537.28 ms
[2021-07-04 00:37:52	                main:574]	:	INFO	:	Epoch 147 | loss: 0.0311026 | val_loss: 0.0311746 | Time: 8547.56 ms
[2021-07-04 00:38:01	                main:574]	:	INFO	:	Epoch 148 | loss: 0.0311037 | val_loss: 0.0311913 | Time: 8521.37 ms
[2021-07-04 00:38:09	                main:574]	:	INFO	:	Epoch 149 | loss: 0.0311012 | val_loss: 0.0311843 | Time: 8520.33 ms
[2021-07-04 00:38:18	                main:574]	:	INFO	:	Epoch 150 | loss: 0.0311047 | val_loss: 0.0311922 | Time: 8590.97 ms
[2021-07-04 00:38:25	                main:574]	:	INFO	:	Epoch 151 | loss: 0.031103 | val_loss: 0.031184 | Time: 7349.39 ms
[2021-07-04 00:38:34	                main:574]	:	INFO	:	Epoch 152 | loss: 0.031108 | val_loss: 0.0311701 | Time: 8514.21 ms
[2021-07-04 00:38:42	                main:574]	:	INFO	:	Epoch 153 | loss: 0.0311009 | val_loss: 0.0312093 | Time: 8573.2 ms
[2021-07-04 00:38:51	                main:574]	:	INFO	:	Epoch 154 | loss: 0.0311018 | val_loss: 0.0311893 | Time: 8579.75 ms
[2021-07-04 00:38:59	                main:574]	:	INFO	:	Epoch 155 | loss: 0.0311096 | val_loss: 0.0311782 | Time: 8572.7 ms
[2021-07-04 00:39:08	                main:574]	:	INFO	:	Epoch 156 | loss: 0.031111 | val_loss: 0.0311994 | Time: 8579.87 ms
[2021-07-04 00:39:15	                main:574]	:	INFO	:	Epoch 157 | loss: 0.0310953 | val_loss: 0.0311876 | Time: 7347.91 ms
[2021-07-04 00:39:24	                main:574]	:	INFO	:	Epoch 158 | loss: 0.0310899 | val_loss: 0.0311858 | Time: 8573.23 ms
[2021-07-04 00:39:32	                main:574]	:	INFO	:	Epoch 159 | loss: 0.0310911 | val_loss: 0.031191 | Time: 8550.91 ms
[2021-07-04 00:39:41	                main:574]	:	INFO	:	Epoch 160 | loss: 0.0310992 | val_loss: 0.0311899 | Time: 8578.02 ms
[2021-07-04 00:39:50	                main:574]	:	INFO	:	Epoch 161 | loss: 0.0310925 | val_loss: 0.0311857 | Time: 8541.57 ms
[2021-07-04 00:39:58	                main:574]	:	INFO	:	Epoch 162 | loss: 0.0310891 | val_loss: 0.031193 | Time: 8599.95 ms
[2021-07-04 00:40:07	                main:574]	:	INFO	:	Epoch 163 | loss: 0.0310977 | val_loss: 0.0311951 | Time: 8548.91 ms
[2021-07-04 00:40:14	                main:574]	:	INFO	:	Epoch 164 | loss: 0.0310956 | val_loss: 0.0311837 | Time: 7345.79 ms
[2021-07-04 00:40:23	                main:574]	:	INFO	:	Epoch 165 | loss: 0.0310843 | val_loss: 0.0312084 | Time: 8511.99 ms
[2021-07-04 00:40:31	                main:574]	:	INFO	:	Epoch 166 | loss: 0.0310866 | val_loss: 0.0311957 | Time: 8570.62 ms
[2021-07-04 00:40:40	                main:574]	:	INFO	:	Epoch 167 | loss: 0.0310922 | val_loss: 0.0311852 | Time: 8504.48 ms
[2021-07-04 00:40:48	                main:574]	:	INFO	:	Epoch 168 | loss: 0.0310935 | val_loss: 0.0312011 | Time: 8577.01 ms
[2021-07-04 00:40:57	                main:574]	:	INFO	:	Epoch 169 | loss: 0.0310895 | val_loss: 0.0311971 | Time: 8551.32 ms
[2021-07-04 00:41:04	                main:574]	:	INFO	:	Epoch 170 | loss: 0.0310811 | val_loss: 0.0311761 | Time: 7381.39 ms
[2021-07-04 00:41:13	                main:574]	:	INFO	:	Epoch 171 | loss: 0.0310808 | val_loss: 0.0312167 | Time: 8570.49 ms
[2021-07-04 00:41:21	                main:574]	:	INFO	:	Epoch 172 | loss: 0.0310788 | val_loss: 0.0311913 | Time: 8547.36 ms
[2021-07-04 00:41:30	                main:574]	:	INFO	:	Epoch 173 | loss: 0.0310823 | val_loss: 0.0311707 | Time: 8531.76 ms
[2021-07-04 00:41:38	                main:574]	:	INFO	:	Epoch 174 | loss: 0.031094 | val_loss: 0.0312166 | Time: 8530.29 ms
[2021-07-04 00:41:46	                main:574]	:	INFO	:	Epoch 175 | loss: 0.0310826 | val_loss: 0.0311981 | Time: 7351.46 ms
[2021-07-04 00:41:54	                main:574]	:	INFO	:	Epoch 176 | loss: 0.0310798 | val_loss: 0.0311992 | Time: 8559.64 ms
[2021-07-04 00:42:03	                main:574]	:	INFO	:	Epoch 177 | loss: 0.0310762 | val_loss: 0.031195 | Time: 8523.8 ms
[2021-07-04 00:42:11	                main:574]	:	INFO	:	Epoch 178 | loss: 0.0310821 | val_loss: 0.0311931 | Time: 8552.18 ms
[2021-07-04 00:42:20	                main:574]	:	INFO	:	Epoch 179 | loss: 0.0310817 | val_loss: 0.0311936 | Time: 8604.21 ms
[2021-07-04 00:42:28	                main:574]	:	INFO	:	Epoch 180 | loss: 0.0310768 | val_loss: 0.0312244 | Time: 8502.1 ms
[2021-07-04 00:42:36	                main:574]	:	INFO	:	Epoch 181 | loss: 0.0310778 | val_loss: 0.0312172 | Time: 7348.62 ms
[2021-07-04 00:42:44	                main:574]	:	INFO	:	Epoch 182 | loss: 0.0310742 | val_loss: 0.0311968 | Time: 8555.29 ms
[2021-07-04 00:42:53	                main:574]	:	INFO	:	Epoch 183 | loss: 0.0310737 | val_loss: 0.0312078 | Time: 8521.29 ms
[2021-07-04 00:43:01	                main:574]	:	INFO	:	Epoch 184 | loss: 0.0310767 | val_loss: 0.031181 | Time: 8547.8 ms
[2021-07-04 00:43:10	                main:574]	:	INFO	:	Epoch 185 | loss: 0.0310776 | val_loss: 0.0312116 | Time: 8510.9 ms
[2021-07-04 00:43:19	                main:574]	:	INFO	:	Epoch 186 | loss: 0.0310751 | val_loss: 0.0312006 | Time: 8546.7 ms
[2021-07-04 00:43:26	                main:574]	:	INFO	:	Epoch 187 | loss: 0.0310738 | val_loss: 0.0312147 | Time: 7381.19 ms
[2021-07-04 00:43:34	                main:574]	:	INFO	:	Epoch 188 | loss: 0.0310744 | val_loss: 0.0312264 | Time: 8563.25 ms
[2021-07-04 00:43:43	                main:574]	:	INFO	:	Epoch 189 | loss: 0.0310755 | val_loss: 0.0311832 | Time: 8569.22 ms
[2021-07-04 00:43:52	                main:574]	:	INFO	:	Epoch 190 | loss: 0.0310752 | val_loss: 0.0312204 | Time: 8553.14 ms
[2021-07-04 00:44:00	                main:574]	:	INFO	:	Epoch 191 | loss: 0.0310726 | val_loss: 0.0311971 | Time: 8515.7 ms
[2021-07-04 00:44:09	                main:574]	:	INFO	:	Epoch 192 | loss: 0.0310762 | val_loss: 0.031237 | Time: 8575.85 ms


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFC23624B89

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.17.0


Dump Timestamp    : 07/04/21 00:44:10
Install Directory : C:\Program Files\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\6;C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome


ModLoad: 00000000f09f0000 00000000003df000 C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000025d10000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 10.0.19041.1023 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1023

ModLoad: 00000000245c0000 00000000000bd000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 10.0.19041.1023 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1023

ModLoad: 00000000235f0000 00000000002c8000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernelbase.pdb
    File Version          : 10.0.19041.1023 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1023

ModLoad: 0000000024850000 00000000001a0000 C:\WINDOWS\System32\USER32.dll (6.2.19041.906) (-exported- Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 10.0.19041.1100 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1100

ModLoad: 0000000023b70000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.1052) (-exported- Symbols Loaded)
    Linked PDB Filename   : win32u.pdb
    File Version          : 10.0.19041.1052 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1052

ModLoad: 00000000244d0000 000000000002a000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 10.0.19041.746 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.746

ModLoad: 0000000023480000 000000000010b000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.928) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32full.pdb
    File Version          : 10.0.19041.928 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.928

ModLoad: 0000000023ba0000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.789

ModLoad: 0000000023a20000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.789

ModLoad: 0000000023f90000 00000000000ac000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.1052) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.19041.320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.320

ModLoad: 0000000025c30000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.19041.546

ModLoad: 0000000024220000 000000000009b000 C:\WINDOWS\System32\sechost.dll (6.2.19041.906) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.19041.320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.320

ModLoad: 0000000025a30000 000000000012b000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.1052) (-exported- Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 10.0.19041.320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.320

ModLoad: 0000000004650000 0000000000058000 C:\ProgramData\BOINC\slots\6\c10.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000000e3f0000 0000000000091000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.25.28508.3) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\8\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
    File Version          : 14.25.28508.3 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.25.28508.3

ModLoad: 000000001c760000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.19041.867 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.867

ModLoad: 0000000015f30000 0000000000019000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.25.28508.3) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\8\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
    File Version          : 14.25.28508.3 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.25.28508.3

ModLoad: 0000000015f20000 000000000000c000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.25.28508.3) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\8\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
    File Version          : 14.25.28508.3 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.25.28508.3

ModLoad: 00000000925d0000 0000000024b49000 C:\ProgramData\BOINC\slots\6\torch_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000008f520000 00000000030a5000 C:\ProgramData\BOINC\slots\6\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1031
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA SPARSE BLAS Library
    Product Version       : 6,14,11,1031

ModLoad: 000000000cd10000 0000000000045000 C:\ProgramData\BOINC\slots\6\c10_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000084930000 000000000abe7000 C:\ProgramData\BOINC\slots\6\torch_cpu.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000001ba40000 0000000000010000 C:\ProgramData\BOINC\slots\6\nvToolsExt64_1.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb

ModLoad: 0000000081470000 00000000034b2000 C:\ProgramData\BOINC\slots\6\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA RAND Library
    Product Version       : 6,14,11,1012

ModLoad: 000000007d710000 0000000003d5b000 C:\ProgramData\BOINC\slots\6\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000d7d10000 000000000020d000 C:\ProgramData\BOINC\slots\6\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
    Linked PDB Filename   : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
    File Version          : 20200205
    Company Name          : Intel Corporation
    Product Name          : Intel(R) OpenMP* Runtime Library
    Product Version       : 5.0

ModLoad: 0000000062360000 000000001b3a4000 C:\ProgramData\BOINC\slots\6\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,10020
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.86 CUDNN Library
    Product Version       : 6,14,11,10020

ModLoad: 0000000060490000 0000000001eca000 C:\ProgramData\BOINC\slots\6\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000d7a30000 00000000002db000 C:\ProgramData\BOINC\slots\6\fbgemm.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000000d6a0000 0000000000040000 C:\ProgramData\BOINC\slots\6\asmjit.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000572f0000 000000000919f000 C:\ProgramData\BOINC\slots\6\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA FFT Library
    Product Version       : 6,14,11,1012

ModLoad: 0000000025980000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000021d40000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000254b0000 0000000000469000 C:\WINDOWS\System32\Setupapi.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 10.0.19041.1100 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1100

ModLoad: 0000000023b20000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000023450000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcrypt.pdb
    File Version          : 10.0.19041.320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.320

ModLoad: 0000000023200000 000000000002c000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000023590000 0000000000060000 C:\WINDOWS\System32\WINTRUST.dll (6.2.19041.1052) (-exported- Symbols Loaded)
    Linked PDB Filename   : wintrust.pdb
    File Version          : 10.0.19041.1052 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1052

ModLoad: 00000000238c0000 000000000015f000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : crypt32.pdb
    File Version          : 10.0.19041.1100 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1100

ModLoad: 0000000023040000 0000000000012000 C:\WINDOWS\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msasn1.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000249f0000 000000000073f000 C:\WINDOWS\System32\Shell32.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : shell32.pdb
    File Version          : 10.0.19041.1100 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1100

ModLoad: 0000000021510000 0000000000790000 C:\WINDOWS\SYSTEM32\windows.storage.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : Windows.Storage.pdb
    File Version          : 10.0.19041.1100 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1100

ModLoad: 0000000025150000 0000000000355000 C:\WINDOWS\System32\combase.dll (6.2.19041.1052) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.19041.320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.320

ModLoad: 0000000022ec0000 000000000002c000 C:\WINDOWS\SYSTEM32\Wldp.dll (6.2.19041.662) (-exported- Symbols Loaded)
    Linked PDB Filename   : WLDP.pdb
    File Version          : 10.0.19041.320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.320

ModLoad: 00000000240c0000 00000000000ae000 C:\WINDOWS\System32\SHCORE.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : shcore.pdb
    File Version          : 10.0.19041.1100 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1100

ModLoad: 0000000024560000 0000000000055000 C:\WINDOWS\System32\shlwapi.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.19041.1100 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1100

ModLoad: 00000000d74a0000 00000000002b4000 C:\WINDOWS\system32\nvcuda.dll (27.21.14.6677) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r465\r466_61\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
    File Version          : 27.21.14.6677
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.3.121 driver
    Product Version       : 27.21.14.6677

ModLoad: 000000001ced0000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000050560000 00000000015db000 C:\WINDOWS\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_b2801df14ec7de03\nvcuda64.dll (27.21.14.6677) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r465\r466_61\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
    File Version          : 27.21.14.6677
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.3.121 driver
    Product Version       : 27.21.14.6677

ModLoad: 000000001cef0000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptnet.pdb
    File Version          : 10.0.19041.906 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.906

ModLoad: 0000000022e20000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000020b10000 000000000009e000 C:\WINDOWS\system32\uxtheme.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : UxTheme.pdb
    File Version          : 10.0.19041.1100 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1100

ModLoad: 00000000057f0000 0000000000708000 C:\WINDOWS\SYSTEM32\nvapi64.dll (27.21.14.6677) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r465\r466_61\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
    File Version          : 27.21.14.6677
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA Windows drivers
    Product Version       : 27.21.14.6677

ModLoad: 0000000024720000 000000000012a000 C:\WINDOWS\System32\OLE32.DLL (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 10.0.19041.320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.320

ModLoad: 000000001a940000 0000000000009000 C:\ProgramData\BOINC\slots\6\caffe2_nvrtc.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000bd920000 0000000000fca000 C:\ProgramData\BOINC\slots\6\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6.14.11.9000
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.89 NVRTC Library
    Product Version       : 6.14.11.9000

ModLoad: 0000000023c40000 0000000000083000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.19041.1023 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1023



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 3839, Write: 1229, Other 3745

- I/O Transfers Counters -
Read: 6822669, Write: 904086, Other 39422

- Paged Pool Usage -
QuotaPagedPoolUsage: 3507616, QuotaPeakPagedPoolUsage: 3524000
QuotaNonPagedPoolUsage: 52624, QuotaPeakNonPagedPoolUsage: 52760

- Virtual Memory Usage -
VirtualSize: -444940288, PeakVirtualSize: 164790272

- Pagefile Usage -
PagefileUsage: -444940288, PeakPagefileUsage: -444833792

- Working Set Size -
WorkingSetSize: 1890799616, PeakWorkingSetSize: -2075471872, PageFaultCount: 1084909

*** Dump of thread ID 16120 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 83906248.000000, User Time: 15006718976.000000, Wait Time: 40513812.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFC23624B89

- Registers -
rax=00000000c99d6c20 rbx=000000009604af00 rcx=0000000000000020 rdx=00000000506f7523 rsi=0000000007955d10 rdi=0000000019930520
r8=00000000000002cf r9=0000000000000000 r10=0000000007955620 r11=0000000007955638 r12=000000001d68f1f0 r13=000000001de17a90
r14=0000000007956ac8 r15=000000001e111920 rip=0000000023624b89 rsp=0000000007955af0 rbp=0000000007955d30
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
07955bc0 15f361e0 07955cb0 1e7253d0 1e111920 000009bc KERNELBASE!RaiseException+0x0 
07955c20 945b094a 95973020 00000000 07955cd0 000002cf VCRUNTIME140!_CxxThrowException+0x0 
07955de0 945b1e02 04689c50 07955ff0 07957550 07956080 torch_cuda!at::native::searchsorted_out_cuda+0x0 
07955fc0 945b205a 00000001 07956ac8 00000001 07956038 torch_cuda!at::native::searchsorted_out_cuda+0x0 
07956000 953a3684 07956080 00000001 07956038 00000001 torch_cuda!at::native::_local_scalar_dense_cuda+0x0 
07956050 953b1107 07956170 89f1bfac 00000001 07956030 torch_cuda!at::native::set_storage_cuda_+0x0 
079560a0 8a3b6bbe 8f250640 00000001 00000000 8be02f07 torch_cuda!at::native::set_storage_cuda_+0x0 
079562b0 8b663c5e 00000000 07956ac8 07956ac8 07956660 torch_cpu!at::_local_scalar_dense+0x0 
079562f0 8a3429c7 07956780 89f1bfac 07956410 079563c0 torch_cpu!torch::autograd::NotImplemented::~NotImplemented+0x0 
07956340 8a3b6bbe 8f250640 07956738 00000000 07957550 torch_cpu!at::native::mkldnn_sigmoid_+0x0 
07956550 8a16dead 079564d8 00000001 079567c0 07956660 torch_cpu!at::_local_scalar_dense+0x0 
07956710 8a4ad1e1 07956748 00000001 079567c0 07956ac8 torch_cpu!at::native::item+0x0 
07956760 8b6c4d4e 079567c0 00000001 07956740 046a1ea8 torch_cpu!at::zeros_out+0x0 
07956790 8a3429c7 1d68ff30 89f1bfac 1d68ff40 046a1ea8 torch_cpu!torch::autograd::GraphRoot::apply+0x0 
079567e0 8a4de69e 8f250640 07956810 8f250640 8a4f129f torch_cpu!at::native::mkldnn_sigmoid_+0x0 
079569f0 8a4ca68e 07956b50 8a1b4e49 07956b10 07956b50 torch_cpu!at::Tensor::item+0x0 
07956a40 f0a0712b 07956b10 07956b10 07956b10 07957550 torch_cpu!at::Tensor::item<double>+0x0 
07956f40 f0a0b950 f0a3d95f f0a3d95f 00000000 9037eda0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0 
0796fce0 f0c4c6f8 bbcc16b0 00000000 00000000 bb5776d0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
0796fd20 245d7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
0796fd50 25d62651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0796fdd0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 21188 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 40513808.000000

- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=00000000080ff6a0 rsi=0000000000000000 rdi=0000000000000064
r8=00000000080ff678 r9=0000000000000000 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=0000000025dad4a4 rsp=00000000080ff678 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
080ff670 236390ae 080ff738 00000000 00000000 00000000 ntdll!NtDelayExecution+0x0 
080ff710 f0a2a07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0 
080ff740 245d7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
080ff770 25d62651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
080ff7f0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 4140 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 40513808.000000

- Registers -
rax=000000000000005b rbx=000000000000000d rcx=000000000000000d rdx=00000000081ffb10 rsi=0000000000000000 rdi=000000000000000d
r8=0000000000000001 r9=0000000000000000 r10=0000000000000000 r11=00000000081ffa80 r12=0000000000000064 r13=00000000081ffb10
r14=00000000081ff810 r15=0000000000000000 rip=0000000025dad974 rsp=00000000081ff7b8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
081ff7b0 2363c5a0 00000000 07b8e000 25e7a4d0 07b8e000 ntdll!ZwWaitForMultipleObjects+0x0 
081ffaa0 2363c49e ba312d30 004e0049 0000000d 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0 
081ffae0 50722634 bbe430d0 00000000 00000000 00000000 KERNELBASE!WaitForMultipleObjects+0x0 
081ffd20 507c465d 00000000 bbe40190 00000000 00000000 nvcuda64!+0x0 
081ffdd0 50722003 e292a07e e255ca57 bc457300 00000000 nvcuda64!cuProfilerStop+0x0 
081ffe00 50a458e4 bc457300 00000000 00000000 00000000 nvcuda64!+0x0 
081ffe30 245d7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
081ffe60 25d62651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
081ffee0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 19648 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 40513808.000000

- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=00000000000001ac rdx=0000000000000000 rsi=00000000082ff9e8 rdi=00000000000001ac
r8=00000000082ff9b8 r9=00000000bbe67ed0 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=00000000000001ac r15=0000000000000001 rip=0000000025dacea4 rsp=00000000082ff9b8 rbp=00000000bbe67ed0
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
082ff9b0 2361168e 0000001a 2486d489 082ffa58 00000000 ntdll!ZwWaitForSingleObject+0x0 
082ffa50 50942d1d bb645500 00000000 00000000 000001ac KERNELBASE!WaitForSingleObjectEx+0x0 
082ffcb0 50722003 00000000 bc457810 50560000 bbe7d500 nvcuda64!cuProfilerStop+0x0 
082ffce0 50a458e4 bc457810 00000000 00000000 00000000 nvcuda64!+0x0 
082ffd10 245d7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
082ffd40 25d62651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
082ffdc0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 23904 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 40513788.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=00000000bbf30278 rdx=00000000083ffa08 rsi=00000000083ff970 rdi=0000000000000000
r8=000000004d8f7204 r9=000000004d8f93c8 r10=00000000bbf34f88 r11=00000000f86aa73f r12=00000000083ffa08 r13=0000000000000000
r14=00000000bbf30278 r15=00000000bbf34f88 rip=0000000025db0814 rsp=00000000083ff948 rbp=00000000083ff9a0
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
083ff940 25d74295 00000102 507fdec8 00000000 083ff970 ntdll!NtWaitForAlertByThreadId+0x0 
083ff9b0 2365c849 bbf32770 00000000 00000000 00000000 ntdll!RtlSleepConditionVariableCS+0x0 
083ff9e0 50699a5c bbf32770 bbf32770 bbf32770 ffb3b4c0 KERNELBASE!SleepConditionVariableCS+0x0 
083ffa40 50722003 00000004 e27d56ef 00000000 00000000 nvcuda64!+0x0 
083ffa70 50a458e4 bc4571e0 00000000 00000000 00000000 nvcuda64!+0x0 
083ffaa0 245d7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
083ffad0 25d62651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
083ffb50 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 24104 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 48437500.000000, User Time: 951249984.000000, Wait Time: 40513764.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=00000000b8423348 rdx=0000000000000000 rsi=0000000000000000 rdi=00000000b8423348
r8=000000001e013740 r9=0000000000000000 r10=0000000000008000 r11=00000000084feed0 r12=0000000000000000 r13=0000000000000000
r14=00000000084ff870 r15=00000000b84232f8 rip=0000000025db0814 rsp=00000000084ff848 rbp=00000000084ff8a0
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
084ff840 25d74021 0c5da788 0c5da788 084ff980 0c5da788 ntdll!NtWaitForAlertByThreadId+0x0 
084ff8c0 2365c8f9 b84232f0 b8423338 00000000 00000000 ntdll!RtlSleepConditionVariableSRW+0x0 
084ff900 0e42bc2d a7833ee0 a7833ee0 a7833ee0 00000000 KERNELBASE!SleepConditionVariableSRW+0x0 
084ff940 0e4028c8 0c21d350 00000000 00000000 8bb105b4 MSVCP140!__crtSetThreadpoolWait+0x0 
084ff970 0e402790 00000000 0e402f97 00000000 0c21d350 MSVCP140!std::_Winerror_message+0x0 
084ff9e0 8bb1064b 0c21d330 000001c0 b84232f0 b84232f0 MSVCP140!std::_Winerror_message+0x0 
084ffa20 8bb125b1 1e4a20e0 ffffffff 0c21d330 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0 
084ffd10 8bb12471 8f036d10 084ffd50 8f036d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0 
084ffd60 8bb09744 00080001 1df05a80 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0 
084ffd90 23a41bb2 1df05a80 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0 
084ffdc0 245d7034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0 
084ffdf0 25d62651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
084ffe70 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 21240 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 40513804.000000

- Registers -
rax=00000000000001d3 rbx=0000000019b1e6d0 rcx=0000000000000070 rdx=0000000019b1e350 rsi=00000000ba3033d0 rdi=0000000019b1e6d0
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=0000000025db0874 rsp=0000000007cff898 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
07cff890 25d62dc7 00000000 00000000 ba3033d0 00000000 ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
07cffb90 245d7034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0 
07cffbc0 25d62651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
07cffc40 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>


©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)