Task 10655846

Name ParityModified-1639946814-29399-1-0_1
Workunit 7559423
Created 11 Jan 2022, 22:33:25 UTC
Sent 11 Jan 2022, 23:52:04 UTC
Report deadline 19 Jan 2022, 23:52:04 UTC
Received 12 Jan 2022, 0:25:10 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -529697949 (0xE06D7363) Unknown error code
Computer ID 8379
Run time 31 min 46 sec
CPU time 31 min 14 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 1,179.45 GFLOPS
Application version Machine Learning Dataset Generator (GPU) v9.75 (cuda10200)
windows_x86_64
Peak working set size 1.65 GB
Peak swap size 3.64 GB
Peak disk usage 1.54 GB

Stderr output

<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 3765269347 (0xe06d7363)</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce MX150)
[2022-01-12 00:52:13	                main:435]	:	INFO	:	Set logging level to 1
[2022-01-12 00:52:13	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-01-12 00:52:13	                main:444]	:	INFO	:	Resolving all filenames
[2022-01-12 00:52:13	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-01-12 00:52:13	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 0)
[2022-01-12 00:52:13	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-01-12 00:52:13	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-01-12 00:52:13	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-01-12 00:52:13	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-01-12 00:52:13	                main:474]	:	INFO	:	Configuration: 
[2022-01-12 00:52:13	                main:475]	:	INFO	:	    Model type: GRU
[2022-01-12 00:52:13	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-01-12 00:52:13	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-01-12 00:52:13	                main:478]	:	INFO	:	    Batch Size: 128
[2022-01-12 00:52:13	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-01-12 00:52:13	                main:480]	:	INFO	:	    Patience: 10
[2022-01-12 00:52:13	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-01-12 00:52:13	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-01-12 00:52:13	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-01-12 00:52:13	                main:484]	:	INFO	:	    # Threads: 1
[2022-01-12 00:52:13	                main:486]	:	INFO	:	Preparing Dataset
[2022-01-12 00:52:13	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-01-12 00:52:14	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-01-12 00:52:20	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-01-12 00:52:20	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-01-12 00:52:20	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-01-12 00:52:20	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-01-12 00:52:20	                main:494]	:	INFO	:	Creating Model
[2022-01-12 00:52:20	                main:507]	:	INFO	:	Preparing config file
[2022-01-12 00:52:20	                main:519]	:	INFO	:	Creating new config file
[2022-01-12 00:52:20	                main:538]	:	INFO	:	This is a continuation WU, loading previous network
[2022-01-12 00:52:23	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-01-12 00:52:23	                main:562]	:	INFO	:	Starting Training
[2022-01-12 00:52:29	                main:574]	:	INFO	:	Epoch 1 | loss: 0.0311695 | val_loss: 0.0312111 | Time: 6537.74 ms
[2022-01-12 00:52:35	                main:574]	:	INFO	:	Epoch 2 | loss: 0.0311149 | val_loss: 0.0312023 | Time: 5741.66 ms
[2022-01-12 00:52:41	                main:574]	:	INFO	:	Epoch 3 | loss: 0.0310987 | val_loss: 0.0312146 | Time: 5675.85 ms
[2022-01-12 00:52:47	                main:574]	:	INFO	:	Epoch 4 | loss: 0.0310998 | val_loss: 0.0312082 | Time: 5635.04 ms
[2022-01-12 00:52:52	                main:574]	:	INFO	:	Epoch 5 | loss: 0.0311048 | val_loss: 0.0312077 | Time: 5575.56 ms
[2022-01-12 00:52:58	                main:574]	:	INFO	:	Epoch 6 | loss: 0.0310953 | val_loss: 0.0312187 | Time: 5681.19 ms
[2022-01-12 00:53:03	                main:574]	:	INFO	:	Epoch 7 | loss: 0.03109 | val_loss: 0.0312086 | Time: 5663.59 ms
[2022-01-12 00:53:09	                main:574]	:	INFO	:	Epoch 8 | loss: 0.0310922 | val_loss: 0.031206 | Time: 5700.57 ms
[2022-01-12 00:53:15	                main:574]	:	INFO	:	Epoch 9 | loss: 0.0310989 | val_loss: 0.0312167 | Time: 5635 ms
[2022-01-12 00:53:20	                main:574]	:	INFO	:	Epoch 10 | loss: 0.0311004 | val_loss: 0.031213 | Time: 5466.82 ms
[2022-01-12 00:53:26	                main:574]	:	INFO	:	Epoch 11 | loss: 0.031088 | val_loss: 0.0312171 | Time: 5717.89 ms
[2022-01-12 00:53:32	                main:574]	:	INFO	:	Epoch 12 | loss: 0.0310868 | val_loss: 0.031215 | Time: 5565.66 ms
[2022-01-12 00:53:37	                main:574]	:	INFO	:	Epoch 13 | loss: 0.0310902 | val_loss: 0.0312171 | Time: 5770.12 ms
[2022-01-12 00:53:43	                main:574]	:	INFO	:	Epoch 14 | loss: 0.0310909 | val_loss: 0.0312085 | Time: 5600.86 ms
[2022-01-12 00:53:49	                main:574]	:	INFO	:	Epoch 15 | loss: 0.0311034 | val_loss: 0.0311948 | Time: 5618.92 ms
[2022-01-12 00:53:54	                main:574]	:	INFO	:	Epoch 16 | loss: 0.0311213 | val_loss: 0.0312023 | Time: 5634.67 ms
[2022-01-12 00:54:00	                main:574]	:	INFO	:	Epoch 17 | loss: 0.0311103 | val_loss: 0.0311981 | Time: 5655.71 ms
[2022-01-12 00:54:06	                main:574]	:	INFO	:	Epoch 18 | loss: 0.0311015 | val_loss: 0.0312143 | Time: 5730.99 ms
[2022-01-12 00:54:11	                main:574]	:	INFO	:	Epoch 19 | loss: 0.0311007 | val_loss: 0.0312036 | Time: 5494.77 ms
[2022-01-12 00:54:17	                main:574]	:	INFO	:	Epoch 20 | loss: 0.0310989 | val_loss: 0.0312159 | Time: 5594.94 ms
[2022-01-12 00:54:22	                main:574]	:	INFO	:	Epoch 21 | loss: 0.0310982 | val_loss: 0.0312186 | Time: 5539.3 ms
[2022-01-12 00:54:28	                main:574]	:	INFO	:	Epoch 22 | loss: 0.0310951 | val_loss: 0.031212 | Time: 5665.35 ms
[2022-01-12 00:54:34	                main:574]	:	INFO	:	Epoch 23 | loss: 0.0311 | val_loss: 0.0312163 | Time: 5624.67 ms
[2022-01-12 00:54:39	                main:574]	:	INFO	:	Epoch 24 | loss: 0.0310896 | val_loss: 0.0312054 | Time: 5675.41 ms
[2022-01-12 00:54:45	                main:574]	:	INFO	:	Epoch 25 | loss: 0.0311368 | val_loss: 0.0312009 | Time: 5710.86 ms
[2022-01-12 00:54:51	                main:574]	:	INFO	:	Epoch 26 | loss: 0.031165 | val_loss: 0.0311797 | Time: 5641.03 ms
[2022-01-12 00:54:55	                main:574]	:	INFO	:	Epoch 27 | loss: 0.0311543 | val_loss: 0.0311812 | Time: 4505.88 ms
[2022-01-12 00:55:01	                main:574]	:	INFO	:	Epoch 28 | loss: 0.0311429 | val_loss: 0.0311842 | Time: 5379.02 ms
[2022-01-12 00:55:05	                main:574]	:	INFO	:	Epoch 29 | loss: 0.0311342 | val_loss: 0.0311881 | Time: 4243.87 ms
[2022-01-12 00:55:09	                main:574]	:	INFO	:	Epoch 30 | loss: 0.0311256 | val_loss: 0.0311869 | Time: 4187.1 ms
[2022-01-12 00:55:14	                main:574]	:	INFO	:	Epoch 31 | loss: 0.0311184 | val_loss: 0.0311924 | Time: 4699.1 ms
[2022-01-12 00:55:18	                main:574]	:	INFO	:	Epoch 32 | loss: 0.0311081 | val_loss: 0.0312024 | Time: 4468.2 ms
[2022-01-12 00:55:23	                main:574]	:	INFO	:	Epoch 33 | loss: 0.0311024 | val_loss: 0.0312058 | Time: 4655.17 ms
[2022-01-12 00:55:29	                main:574]	:	INFO	:	Epoch 34 | loss: 0.0311023 | val_loss: 0.0312162 | Time: 6025.75 ms
[2022-01-12 00:55:35	                main:574]	:	INFO	:	Epoch 35 | loss: 0.0310997 | val_loss: 0.0312064 | Time: 5993.62 ms
[2022-01-12 00:55:41	                main:574]	:	INFO	:	Epoch 36 | loss: 0.0311017 | val_loss: 0.0312095 | Time: 5864.14 ms
[2022-01-12 00:55:47	                main:574]	:	INFO	:	Epoch 37 | loss: 0.0310998 | val_loss: 0.0312045 | Time: 5988.6 ms
[2022-01-12 00:55:53	                main:574]	:	INFO	:	Epoch 38 | loss: 0.0310954 | val_loss: 0.031214 | Time: 6135.59 ms
[2022-01-12 00:55:59	                main:574]	:	INFO	:	Epoch 39 | loss: 0.0311057 | val_loss: 0.0312008 | Time: 6023.83 ms
[2022-01-12 00:56:05	                main:574]	:	INFO	:	Epoch 40 | loss: 0.0311028 | val_loss: 0.0312033 | Time: 6093.74 ms
[2022-01-12 00:56:11	                main:574]	:	INFO	:	Epoch 41 | loss: 0.0310916 | val_loss: 0.031211 | Time: 5979.86 ms
[2022-01-12 00:56:17	                main:574]	:	INFO	:	Epoch 42 | loss: 0.0310889 | val_loss: 0.0312155 | Time: 6161.11 ms
[2022-01-12 00:56:23	                main:574]	:	INFO	:	Epoch 43 | loss: 0.0310894 | val_loss: 0.0312089 | Time: 5932.99 ms
[2022-01-12 00:56:29	                main:574]	:	INFO	:	Epoch 44 | loss: 0.0310886 | val_loss: 0.0312113 | Time: 6057.57 ms
[2022-01-12 00:56:35	                main:574]	:	INFO	:	Epoch 45 | loss: 0.0310939 | val_loss: 0.0312145 | Time: 5981.44 ms
[2022-01-12 00:56:41	                main:574]	:	INFO	:	Epoch 46 | loss: 0.0310881 | val_loss: 0.0312323 | Time: 6017.86 ms
[2022-01-12 00:56:47	                main:574]	:	INFO	:	Epoch 47 | loss: 0.0311032 | val_loss: 0.0312132 | Time: 6067.32 ms
[2022-01-12 00:56:53	                main:574]	:	INFO	:	Epoch 48 | loss: 0.0311087 | val_loss: 0.0312089 | Time: 6116.08 ms
[2022-01-12 00:56:59	                main:574]	:	INFO	:	Epoch 49 | loss: 0.0311019 | val_loss: 0.0312093 | Time: 6079.46 ms
[2022-01-12 00:57:05	                main:574]	:	INFO	:	Epoch 50 | loss: 0.0310964 | val_loss: 0.0312106 | Time: 5986.39 ms
[2022-01-12 00:57:11	                main:574]	:	INFO	:	Epoch 51 | loss: 0.0310963 | val_loss: 0.0312136 | Time: 6055.88 ms
[2022-01-12 00:57:18	                main:574]	:	INFO	:	Epoch 52 | loss: 0.0310953 | val_loss: 0.0312047 | Time: 6115.89 ms
[2022-01-12 00:57:24	                main:574]	:	INFO	:	Epoch 53 | loss: 0.031095 | val_loss: 0.0312102 | Time: 6067.93 ms
[2022-01-12 00:57:30	                main:574]	:	INFO	:	Epoch 54 | loss: 0.031094 | val_loss: 0.0312269 | Time: 5973.63 ms
[2022-01-12 00:57:36	                main:574]	:	INFO	:	Epoch 55 | loss: 0.0310877 | val_loss: 0.0312136 | Time: 5959.3 ms
[2022-01-12 00:57:42	                main:574]	:	INFO	:	Epoch 56 | loss: 0.0310884 | val_loss: 0.0312225 | Time: 6101.99 ms
[2022-01-12 00:57:48	                main:574]	:	INFO	:	Epoch 57 | loss: 0.0310894 | val_loss: 0.0312054 | Time: 6082.76 ms
[2022-01-12 00:57:54	                main:574]	:	INFO	:	Epoch 58 | loss: 0.0310885 | val_loss: 0.0312219 | Time: 6149.87 ms
[2022-01-12 00:58:00	                main:574]	:	INFO	:	Epoch 59 | loss: 0.031093 | val_loss: 0.0312055 | Time: 5942 ms
[2022-01-12 00:58:06	                main:574]	:	INFO	:	Epoch 60 | loss: 0.0311104 | val_loss: 0.0312007 | Time: 5994.53 ms
[2022-01-12 00:58:12	                main:574]	:	INFO	:	Epoch 61 | loss: 0.0311021 | val_loss: 0.0312119 | Time: 6057.88 ms
[2022-01-12 00:58:18	                main:574]	:	INFO	:	Epoch 62 | loss: 0.0310977 | val_loss: 0.0312217 | Time: 5905.35 ms
[2022-01-12 00:58:24	                main:574]	:	INFO	:	Epoch 63 | loss: 0.0310981 | val_loss: 0.0312034 | Time: 5932.83 ms
[2022-01-12 00:58:30	                main:574]	:	INFO	:	Epoch 64 | loss: 0.0310949 | val_loss: 0.0312156 | Time: 6034.16 ms
[2022-01-12 00:58:36	                main:574]	:	INFO	:	Epoch 65 | loss: 0.0310859 | val_loss: 0.0312212 | Time: 6157.85 ms
[2022-01-12 00:58:42	                main:574]	:	INFO	:	Epoch 66 | loss: 0.0310839 | val_loss: 0.0312189 | Time: 6079.17 ms
[2022-01-12 00:58:48	                main:574]	:	INFO	:	Epoch 67 | loss: 0.0310837 | val_loss: 0.0312206 | Time: 6066.85 ms
[2022-01-12 00:58:54	                main:574]	:	INFO	:	Epoch 68 | loss: 0.0310878 | val_loss: 0.0312169 | Time: 6081.58 ms
[2022-01-12 00:59:00	                main:574]	:	INFO	:	Epoch 69 | loss: 0.031087 | val_loss: 0.0312183 | Time: 5973.08 ms
[2022-01-12 00:59:06	                main:574]	:	INFO	:	Epoch 70 | loss: 0.031092 | val_loss: 0.0312148 | Time: 6113.6 ms
[2022-01-12 00:59:12	                main:574]	:	INFO	:	Epoch 71 | loss: 0.0310913 | val_loss: 0.0312118 | Time: 5981.39 ms
[2022-01-12 00:59:18	                main:574]	:	INFO	:	Epoch 72 | loss: 0.0310859 | val_loss: 0.031217 | Time: 5866.6 ms
[2022-01-12 00:59:24	                main:574]	:	INFO	:	Epoch 73 | loss: 0.031083 | val_loss: 0.0312118 | Time: 5947.47 ms
[2022-01-12 00:59:30	                main:574]	:	INFO	:	Epoch 74 | loss: 0.0310932 | val_loss: 0.0312103 | Time: 6088.72 ms
[2022-01-12 00:59:36	                main:574]	:	INFO	:	Epoch 75 | loss: 0.0310906 | val_loss: 0.0312195 | Time: 5936.6 ms
[2022-01-12 00:59:42	                main:574]	:	INFO	:	Epoch 76 | loss: 0.0310923 | val_loss: 0.0312081 | Time: 6061.87 ms
[2022-01-12 00:59:48	                main:574]	:	INFO	:	Epoch 77 | loss: 0.0310928 | val_loss: 0.0312132 | Time: 5943.59 ms
[2022-01-12 00:59:54	                main:574]	:	INFO	:	Epoch 78 | loss: 0.0310952 | val_loss: 0.0312128 | Time: 5995.57 ms
[2022-01-12 01:00:00	                main:574]	:	INFO	:	Epoch 79 | loss: 0.0310936 | val_loss: 0.031209 | Time: 6091 ms
[2022-01-12 01:00:06	                main:574]	:	INFO	:	Epoch 80 | loss: 0.0310966 | val_loss: 0.0312171 | Time: 6063.7 ms
[2022-01-12 01:00:12	                main:574]	:	INFO	:	Epoch 81 | loss: 0.0310877 | val_loss: 0.0312139 | Time: 6088.61 ms
[2022-01-12 01:00:18	                main:574]	:	INFO	:	Epoch 82 | loss: 0.0310883 | val_loss: 0.0312083 | Time: 5985.78 ms
[2022-01-12 01:00:24	                main:574]	:	INFO	:	Epoch 83 | loss: 0.0310883 | val_loss: 0.0312078 | Time: 5974.54 ms
[2022-01-12 01:00:30	                main:574]	:	INFO	:	Epoch 84 | loss: 0.0310952 | val_loss: 0.0312116 | Time: 5970.66 ms
[2022-01-12 01:00:36	                main:574]	:	INFO	:	Epoch 85 | loss: 0.0310904 | val_loss: 0.0312087 | Time: 6091.85 ms
[2022-01-12 01:00:43	                main:574]	:	INFO	:	Epoch 86 | loss: 0.0311 | val_loss: 0.0312064 | Time: 6048.34 ms
[2022-01-12 01:00:49	                main:574]	:	INFO	:	Epoch 87 | loss: 0.0311006 | val_loss: 0.0312133 | Time: 6085.58 ms
[2022-01-12 01:00:55	                main:574]	:	INFO	:	Epoch 88 | loss: 0.0310964 | val_loss: 0.0312202 | Time: 6009.38 ms
[2022-01-12 01:01:01	                main:574]	:	INFO	:	Epoch 89 | loss: 0.0310984 | val_loss: 0.0312153 | Time: 6047.56 ms
[2022-01-12 01:01:07	                main:574]	:	INFO	:	Epoch 90 | loss: 0.0310958 | val_loss: 0.031202 | Time: 5986.35 ms
[2022-01-12 01:01:13	                main:574]	:	INFO	:	Epoch 91 | loss: 0.0310917 | val_loss: 0.0312164 | Time: 5992.51 ms
[2022-01-12 01:01:19	                main:574]	:	INFO	:	Epoch 92 | loss: 0.0310928 | val_loss: 0.0312129 | Time: 5976.7 ms
[2022-01-12 01:01:25	                main:574]	:	INFO	:	Epoch 93 | loss: 0.0310913 | val_loss: 0.0312128 | Time: 6056.18 ms
[2022-01-12 01:01:31	                main:574]	:	INFO	:	Epoch 94 | loss: 0.031087 | val_loss: 0.0312129 | Time: 6086.97 ms
[2022-01-12 01:01:37	                main:574]	:	INFO	:	Epoch 95 | loss: 0.0310955 | val_loss: 0.031214 | Time: 6231.36 ms
[2022-01-12 01:01:43	                main:574]	:	INFO	:	Epoch 96 | loss: 0.0310938 | val_loss: 0.03121 | Time: 5988.52 ms
[2022-01-12 01:01:49	                main:574]	:	INFO	:	Epoch 97 | loss: 0.0310936 | val_loss: 0.031203 | Time: 6063.97 ms
[2022-01-12 01:01:55	                main:574]	:	INFO	:	Epoch 98 | loss: 0.0310878 | val_loss: 0.0312238 | Time: 6134.43 ms
[2022-01-12 01:02:01	                main:574]	:	INFO	:	Epoch 99 | loss: 0.0310867 | val_loss: 0.0312158 | Time: 6102.69 ms
[2022-01-12 01:02:07	                main:574]	:	INFO	:	Epoch 100 | loss: 0.0310923 | val_loss: 0.0312151 | Time: 6027.4 ms
[2022-01-12 01:02:13	                main:574]	:	INFO	:	Epoch 101 | loss: 0.0310912 | val_loss: 0.0312086 | Time: 5946.34 ms
[2022-01-12 01:02:19	                main:574]	:	INFO	:	Epoch 102 | loss: 0.0310884 | val_loss: 0.0312172 | Time: 6050.43 ms
[2022-01-12 01:02:26	                main:574]	:	INFO	:	Epoch 103 | loss: 0.0310847 | val_loss: 0.0312093 | Time: 6146.45 ms
[2022-01-12 01:02:32	                main:574]	:	INFO	:	Epoch 104 | loss: 0.0310857 | val_loss: 0.0312107 | Time: 6049.92 ms
[2022-01-12 01:02:38	                main:574]	:	INFO	:	Epoch 105 | loss: 0.0310867 | val_loss: 0.0312189 | Time: 6116.2 ms
[2022-01-12 01:02:44	                main:574]	:	INFO	:	Epoch 106 | loss: 0.0310868 | val_loss: 0.0312103 | Time: 6083.4 ms
[2022-01-12 01:02:50	                main:574]	:	INFO	:	Epoch 107 | loss: 0.0310921 | val_loss: 0.0312079 | Time: 5957.28 ms
[2022-01-12 01:02:56	                main:574]	:	INFO	:	Epoch 108 | loss: 0.0310936 | val_loss: 0.0312127 | Time: 6073.7 ms
[2022-01-12 01:03:02	                main:574]	:	INFO	:	Epoch 109 | loss: 0.0310883 | val_loss: 0.0312126 | Time: 5967.76 ms
[2022-01-12 01:03:08	                main:574]	:	INFO	:	Epoch 110 | loss: 0.0310948 | val_loss: 0.0312122 | Time: 6193.14 ms
[2022-01-12 01:03:14	                main:574]	:	INFO	:	Epoch 111 | loss: 0.0310956 | val_loss: 0.0312153 | Time: 6029.99 ms
[2022-01-12 01:03:20	                main:574]	:	INFO	:	Epoch 112 | loss: 0.0310954 | val_loss: 0.0312035 | Time: 5919.58 ms
[2022-01-12 01:03:26	                main:574]	:	INFO	:	Epoch 113 | loss: 0.0310928 | val_loss: 0.0312095 | Time: 6058.16 ms
[2022-01-12 01:03:32	                main:574]	:	INFO	:	Epoch 114 | loss: 0.0310934 | val_loss: 0.0311997 | Time: 6032.35 ms
[2022-01-12 01:03:38	                main:574]	:	INFO	:	Epoch 115 | loss: 0.0310885 | val_loss: 0.0312137 | Time: 6081.29 ms
[2022-01-12 01:03:44	                main:574]	:	INFO	:	Epoch 116 | loss: 0.0310865 | val_loss: 0.0312284 | Time: 6139.84 ms


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FF85EAA4F69

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.17.0


Dump Timestamp    : 01/12/22 01:23:56
Install Directory : C:\Program Files\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\4;C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome


ModLoad: 000000001e7f0000 00000000003df000 C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000060ef0000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.1288) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005f1a0000 00000000000be000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.1348) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005ea70000 00000000002c8000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernelbase.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005f260000 00000000001a1000 C:\WINDOWS\System32\USER32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005e600000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : win32u.pdb
    File Version          : 10.0.19041.1387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1387

ModLoad: 0000000060e80000 000000000002b000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 10.0.19041.1202 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1202

ModLoad: 000000005ed90000 000000000010d000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32full.pdb
    File Version          : 10.0.19041.1387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1387

ModLoad: 000000005e630000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.789

ModLoad: 000000005e700000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.789

ModLoad: 000000004bd90000 0000000000058000 C:\ProgramData\BOINC\slots\4\c10.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000005f8a0000 00000000000ac000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.1052) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 0000000060d30000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.19041.546

ModLoad: 000000005ff20000 000000000009b000 C:\WINDOWS\System32\sechost.dll (6.2.19041.1415) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.19041.320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.320

ModLoad: 000000005fdf0000 000000000012a000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.1288) (-exported- Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 0000000045690000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.19041.867 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.867

ModLoad: 0000000045430000 0000000000099000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.23.27820.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\2\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
    File Version          : 14.23.27820.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174; 2017
    Product Version       : 14.23.27820.0

ModLoad: 00000000207a0000 0000000000016000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.23.27820.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\2\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
    File Version          : 14.23.27820.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174; 2017
    Product Version       : 14.23.27820.0

ModLoad: 0000000059df0000 000000000000b000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.23.27820.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\2\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
    File Version          : 14.23.27820.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174; 2017
    Product Version       : 14.23.27820.0

ModLoad: 00000000f5400000 000000000abe7000 C:\ProgramData\BOINC\slots\4\torch_cpu.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000022360000 00000000002db000 C:\ProgramData\BOINC\slots\4\fbgemm.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000035c20000 000000000020d000 C:\ProgramData\BOINC\slots\4\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
    Linked PDB Filename   : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
    File Version          : 20200205
    Company Name          : Intel Corporation
    Product Name          : Intel(R) OpenMP* Runtime Library
    Product Version       : 5.0

ModLoad: 00000000511b0000 0000000000040000 C:\ProgramData\BOINC\slots\4\asmjit.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000d08b0000 0000000024b49000 C:\ProgramData\BOINC\slots\4\torch_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000046720000 0000000000045000 C:\ProgramData\BOINC\slots\4\c10_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000005f50000 00000000030a5000 C:\ProgramData\BOINC\slots\4\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1031
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA SPARSE BLAS Library
    Product Version       : 6,14,11,1031

ModLoad: 0000000002a90000 00000000034b2000 C:\ProgramData\BOINC\slots\4\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA RAND Library
    Product Version       : 6,14,11,1012

ModLoad: 0000000059de0000 0000000000010000 C:\ProgramData\BOINC\slots\4\nvToolsExt64_1.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb

ModLoad: 00000000a8600000 0000000003d5b000 C:\ProgramData\BOINC\slots\4\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000c7710000 000000000919f000 C:\ProgramData\BOINC\slots\4\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA FFT Library
    Product Version       : 6,14,11,1012

ModLoad: 00000000a6730000 0000000001eca000 C:\ProgramData\BOINC\slots\4\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000ac360000 000000001b3a4000 C:\ProgramData\BOINC\slots\4\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,10020
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.86 CUDNN Library
    Product Version       : 6,14,11,10020

ModLoad: 000000005fbc0000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000005ceb0000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000005f420000 0000000000472000 C:\WINDOWS\System32\Setupapi.dll (6.2.19041.1237) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005ed40000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 10.0.19041.1151 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1151

ModLoad: 000000005e6d0000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcrypt.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005e3a0000 0000000000034000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.19041.1151) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 10.0.19041.1151 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1151

ModLoad: 000000005e800000 0000000000060000 C:\WINDOWS\System32\WINTRUST.dll (6.2.19041.1266) (-exported- Symbols Loaded)
    Linked PDB Filename   : wintrust.pdb
    File Version          : 10.0.19041.1266 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1266

ModLoad: 000000005e910000 0000000000156000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.1320) (-exported- Symbols Loaded)
    Linked PDB Filename   : crypt32.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005e1e0000 0000000000012000 C:\WINDOWS\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msasn1.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000060560000 0000000000744000 C:\WINDOWS\System32\Shell32.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : shell32.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005c6d0000 0000000000794000 C:\WINDOWS\SYSTEM32\windows.storage.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : Windows.Storage.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005ffc0000 0000000000355000 C:\WINDOWS\System32\combase.dll (6.2.19041.1348) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000005e050000 000000000002e000 C:\WINDOWS\SYSTEM32\Wldp.dll (6.2.19041.1320) (-exported- Symbols Loaded)
    Linked PDB Filename   : WLDP.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 0000000060390000 00000000000ad000 C:\WINDOWS\System32\SHCORE.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : shcore.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 0000000060330000 0000000000055000 C:\WINDOWS\System32\shlwapi.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 0000000035e30000 0000000000267000 C:\WINDOWS\system32\nvcuda.dll (27.21.14.5730) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r457_22\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
    File Version          : 27.21.14.5730
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.1.114 driver
    Product Version       : 27.21.14.5730

ModLoad: 0000000058400000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000018ad0000 000000000152c000 C:\WINDOWS\system32\DriverStore\FileRepository\nvmii.inf_amd64_323ec550858ed703\nvcuda64.dll (27.21.14.5730) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r457_22\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
    File Version          : 27.21.14.5730
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.1.114 driver
    Product Version       : 27.21.14.5730

ModLoad: 000000005fc50000 000000000006b000 C:\WINDOWS\System32\WS2_32.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : ws2_32.pdb
    File Version          : 10.0.19041.1081 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1081

ModLoad: 00000000583c0000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptnet.pdb
    File Version          : 10.0.19041.906 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.906

ModLoad: 000000005dfa0000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000005bca0000 000000000009e000 C:\WINDOWS\system32\uxtheme.dll (6.2.19041.1266) (-exported- Symbols Loaded)
    Linked PDB Filename   : UxTheme.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 000000004f9a0000 00000000006d6000 C:\WINDOWS\SYSTEM32\nvapi64.dll (27.21.14.5730) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r457_22\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
    File Version          : 27.21.14.5730
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA Windows drivers
    Product Version       : 27.21.14.5730

ModLoad: 000000005fcc0000 000000000012a000 C:\WINDOWS\System32\OLE32.DLL (6.2.19041.1202) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operativsystem
    Product Version       : 10.0.19041.1415

ModLoad: 0000000059db0000 0000000000009000 C:\ProgramData\BOINC\slots\4\caffe2_nvrtc.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000017b00000 0000000000fca000 C:\ProgramData\BOINC\slots\4\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6.14.11.9000
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.89 NVRTC Library
    Product Version       : 6.14.11.9000

ModLoad: 000000005eea0000 0000000000082000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.1415) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1415



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 3739, Write: 754, Other 3122

- I/O Transfers Counters -
Read: 7086635, Write: 104383, Other 39006

- Paged Pool Usage -
QuotaPagedPoolUsage: 3505584, QuotaPeakPagedPoolUsage: 3521968
QuotaNonPagedPoolUsage: 52488, QuotaPeakNonPagedPoolUsage: 52624

- Virtual Memory Usage -
VirtualSize: -428548096, PeakVirtualSize: 184627200

- Pagefile Usage -
PagefileUsage: -428548096, PeakPagefileUsage: -428101632

- Working Set Size -
WorkingSetSize: 1737322496, PeakWorkingSetSize: 1763610624, PageFaultCount: 637531

*** Dump of thread ID 13556 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 11312187392.000000, User Time: 5677031424.000000, Wait Time: 140911696.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FF85EAA4F69

- Registers -
rax=0000000000000002 rbx=00000000d432af00 rcx=0000000000000000 rdx=00000000ffffffff rsi=0000000000335ad0 rdi=0000000019930520
r8=0000000000000010 r9=0000000000335250 r10=00000000d2d64df7 r11=0000000000000000 r12=000000001c835c10 r13=000000001cfc5890
r14=0000000000336888 r15=000000001dba41b0 rip=000000005eaa4f69 rsp=00000000003358b0 rbp=0000000000335af0
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00335980 207a4890 00335a70 2081c710 1dba41b0 000009bc KERNELBASE!RaiseException+0x0 
003359e0 d289094a d3c53020 00000000 00335a90 000003e7 VCRUNTIME140!_CxxThrowException+0x0 
00335ba0 d2891e02 4bdc9c50 00335db0 00337310 00335e40 torch_cuda!at::native::searchsorted_out_cuda+0x0 
00335d80 d289205a 00000001 00336888 00000001 00335df8 torch_cuda!at::native::searchsorted_out_cuda+0x0 
00335dc0 d3683684 00335e40 00000001 00335df8 00000001 torch_cuda!at::native::_local_scalar_dense_cuda+0x0 
00335e10 d3691107 00335f30 fa9ebfac 00000001 00335df0 torch_cuda!at::native::set_storage_cuda_+0x0 
00335e60 fae86bbe ffd20640 00000001 00000000 fc8d2f07 torch_cuda!at::native::set_storage_cuda_+0x0 
00336070 fc133c5e 00000000 00336888 00336888 00336420 torch_cpu!at::_local_scalar_dense+0x0 
003360b0 fae129c7 00336540 fa9ebfac 003361d0 00336180 torch_cpu!torch::autograd::NotImplemented::~NotImplemented+0x0 
00336100 fae86bbe ffd20640 003364f8 00000000 00337310 torch_cpu!at::native::mkldnn_sigmoid_+0x0 
00336310 fac3dead 00336298 00000001 00336580 00336420 torch_cpu!at::_local_scalar_dense+0x0 
003364d0 faf7d1e1 00336508 00000001 00336580 00336888 torch_cpu!at::native::item+0x0 
00336520 fc194d4e 00336580 00000001 00336500 4bde1ea8 torch_cpu!at::zeros_out+0x0 
00336550 fae129c7 1c8362d0 fa9ebfac 1c8362e0 4bde1ea8 torch_cpu!torch::autograd::GraphRoot::apply+0x0 
003365a0 fafae69e ffd20640 003365d0 ffd20640 fafc129f torch_cpu!at::native::mkldnn_sigmoid_+0x0 
003367b0 faf9a68e 00336910 fac84e49 003368d0 00336910 torch_cpu!at::Tensor::item+0x0 
00336800 1e80712b 003368d0 003368d0 003368d0 00337310 torch_cpu!at::Tensor::item<double>+0x0 
00336d00 1e80b950 00336e00 00336e00 00000000 8a954220 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0 
0034faa0 1ea4c6f8 c0d333f0 00000000 00000000 c0d98a30 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
0034fae0 5f1b7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
0034fb10 60f42651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0034fb90 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 1488 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 140911696.000000

- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=0000000000affbc0 rsi=0000000000000000 rdi=0000000000000064
r8=000000000059a000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=0000000060f8d3f4 rsp=0000000000affb98 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
00affb90 5eab962e 00affc58 00000000 00000000 00000000 ntdll!NtDelayExecution+0x0 
00affc30 1e82a07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0 
00affc60 5f1b7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
00affc90 60f42651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00affd10 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 3860 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 140911696.000000

- Registers -
rax=000000000000005b rbx=000000000000000d rcx=000000000000000d rdx=0000000000bff7a0 rsi=0000000000000000 rdi=000000000000000d
r8=00000000c12fc020 r9=0000000000000000 r10=0000000000000000 r11=0000000000bff710 r12=0000000000000064 r13=0000000000bff7a0
r14=0000000000bff4a0 r15=0000000000000000 rip=0000000060f8d8c4 rsp=0000000000bff448 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
00bff440 5eabcb20 00000000 0058f000 6105a4d0 0058f000 ntdll!ZwWaitForMultipleObjects+0x0 
00bff730 5eabca1e bf353480 004e0049 0000000d 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0 
00bff770 18c7bb54 c0f2a1a0 00000000 00000000 00000000 KERNELBASE!WaitForMultipleObjects+0x0 
00bff9b0 18d109ed 00000000 00000004 00000000 00000000 nvcuda64!+0x0 
00bffa60 18c7b523 57242479 56030dc4 c17950c0 00000000 nvcuda64!cuProfilerStop+0x0 
00bffa90 18f8a750 c17950c0 00000000 00000000 00000000 nvcuda64!+0x0 
00bffac0 5f1b7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
00bffaf0 60f42651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00bffb70 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 6824 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 140789984.000000

- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=00000000000002b8 rdx=0000000000000000 rsi=0000000000000000 rdi=00000000000002b8
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=00000000c0f498b0 r13=0000000000000000
r14=00000000000002b8 r15=0000000000000000 rip=0000000060f8cdf4 rsp=0000000000cff618 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
00cff610 5ea91a5e 00000000 00000000 00000000 00000000 ntdll!ZwWaitForSingleObject+0x0 
00cff6b0 18e8bf48 00000000 5ea9a395 00000000 000002b8 KERNELBASE!WaitForSingleObjectEx+0x0 
00cff720 18c7b523 00000001 c157ba90 c0cfcbb0 00000000 nvcuda64!cuProfilerStop+0x0 
00cff750 18f8a750 c0cfcbb0 00000000 00000000 00000000 nvcuda64!+0x0 
00cff780 5f1b7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
00cff7b0 60f42651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00cff830 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 11864 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 140911680.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=00000000c11b64e8 rdx=0000000000dffba8 rsi=0000000000dffb10 rdi=0000000000000000
r8=0000000000dffae8 r9=0000000000dffb40 r10=0000000000000000 r11=0000000000000246 r12=0000000000dffba8 r13=0000000000000000
r14=00000000c11b64e8 r15=00000000c11b9858 rip=0000000060f90764 rsp=0000000000dffae8 rbp=0000000000dffb40
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
00dffae0 60f54295 00000102 18d4964e 00000000 00dffb10 ntdll!NtWaitForAlertByThreadId+0x0 
00dffb50 5eadcdd9 c11b7040 00000000 00000000 00000000 ntdll!RtlSleepConditionVariableCS+0x0 
00dffb80 18c0223a c11b7040 c11b7040 c11b7040 ffb3b4c0 KERNELBASE!SleepConditionVariableCS+0x0 
00dffbe0 18c7b523 00000003 57052640 00000000 00000000 nvcuda64!+0x0 
00dffc10 18f8a750 c0cfcd90 00000000 00000000 00000000 nvcuda64!+0x0 
00dffc40 5f1b7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
00dffc70 60f42651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00dffcf0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 4196 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 116406248.000000, User Time: 1627187456.000000, Wait Time: 140834448.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=0000000005247218 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000005247218
r8=0000000000000000 r9=0000000000000000 r10=0000000000008000 r11=0000000000efb450 r12=0000000000000000 r13=0000000000000000
r14=0000000000eff760 r15=00000000052471c8 rip=0000000060f90764 rsp=0000000000eff738 rbp=0000000000eff790
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
00eff730 60f54021 0b943a38 0b943a38 00eff870 0b943a38 ntdll!NtWaitForAlertByThreadId+0x0 
00eff7b0 5eadce89 052471c0 05247208 00000000 00000000 ntdll!RtlSleepConditionVariableSRW+0x0 
00eff7f0 4546567d 0ba5c540 0ba5c540 0ba5c540 00000000 KERNELBASE!SleepConditionVariableSRW+0x0 
00eff830 454420b8 1d0073d0 00000000 00000000 fc5e05b4 MSVCP140!__crtSetThreadpoolWait+0x0 
00eff860 45441f80 00000000 45442787 00000000 1d0073d0 MSVCP140!std::_Winerror_message+0x0 
00eff8d0 fc5e064b 1d0073b0 000001c0 052471c0 052471c0 MSVCP140!std::_Winerror_message+0x0 
00eff910 fc5e25b1 1dd3eef0 ffffffff 1d0073b0 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0 
00effc00 fc5e2471 ffb06d10 00effc40 ffb06d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0 
00effc50 fc5d9744 00080001 1dd0f610 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0 
00effc80 5e721bb2 1dd0f610 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0 
00effcb0 5f1b7034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0 
00effce0 60f42651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00effd60 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 18768 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 140911648.000000

- Registers -
rax=00000000000001d3 rbx=000000000be3f540 rcx=0000000000000070 rdx=000000000be3f1c0 rsi=00000000bf343540 rdi=000000000be3f540
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=0000000060f907c4 rsp=00000000006ff928 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
006ff920 60f42dc7 00000000 00000000 bf343540 00000000 ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
006ffc20 5f1b7034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0 
006ffc50 60f42651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
006ffcd0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>


©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)