Task 5638050
| Name | ParityMachine-1624423002-31433-2_0 |
| Workunit | 3478983 |
| Created | 5 Jul 2021, 6:30:13 UTC |
| Sent | 10 Jul 2021, 22:45:29 UTC |
| Report deadline | 17 Jul 2021, 22:45:29 UTC |
| Received | 11 Jul 2021, 4:46:50 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED |
| Computer ID | 9215 |
| Run time | 3 hours 43 min 55 sec |
| CPU time | 3 hours 32 min 50 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 15,279.60 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 2.08 GB |
| Peak swap size | 4.41 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
exceeded elapsed time limit 13433.76 (4000000.00G/297.76G)</message>
<stderr_txt>
Time: 9711.61 ms
[2021-07-10 22:59:25 main:574] : INFO : Epoch 1104 | loss: 0.0311692 | val_loss: 0.0311828 | Time: 9470.51 ms
[2021-07-10 22:59:35 main:574] : INFO : Epoch 1105 | loss: 0.0311714 | val_loss: 0.0311853 | Time: 9717.57 ms
[2021-07-10 22:59:44 main:574] : INFO : Epoch 1106 | loss: 0.0311682 | val_loss: 0.0311826 | Time: 9557.67 ms
[2021-07-10 22:59:54 main:574] : INFO : Epoch 1107 | loss: 0.0311669 | val_loss: 0.031182 | Time: 9297.95 ms
[2021-07-10 23:00:03 main:574] : INFO : Epoch 1108 | loss: 0.0311705 | val_loss: 0.0311838 | Time: 9672.33 ms
[2021-07-10 23:00:13 main:574] : INFO : Epoch 1109 | loss: 0.0311697 | val_loss: 0.0311822 | Time: 9339.99 ms
[2021-07-10 23:00:22 main:574] : INFO : Epoch 1110 | loss: 0.0311708 | val_loss: 0.0311827 | Time: 9213.5 ms
[2021-07-10 23:00:31 main:574] : INFO : Epoch 1111 | loss: 0.0311706 | val_loss: 0.0311831 | Time: 9016.7 ms
[2021-07-10 23:00:40 main:574] : INFO : Epoch 1112 | loss: 0.0311739 | val_loss: 0.03118 | Time: 9127.3 ms
[2021-07-10 23:00:50 main:574] : INFO : Epoch 1113 | loss: 0.031171 | val_loss: 0.0311822 | Time: 9460.85 ms
[2021-07-10 23:00:59 main:574] : INFO : Epoch 1114 | loss: 0.0311708 | val_loss: 0.0311826 | Time: 9262.84 ms
[2021-07-10 23:01:08 main:574] : INFO : Epoch 1115 | loss: 0.0311703 | val_loss: 0.0311808 | Time: 9106.95 ms
[2021-07-10 23:01:17 main:574] : INFO : Epoch 1116 | loss: 0.0311697 | val_loss: 0.0311799 | Time: 9318.05 ms
[2021-07-10 23:01:27 main:574] : INFO : Epoch 1117 | loss: 0.0311691 | val_loss: 0.0311828 | Time: 9554.55 ms
[2021-07-10 23:01:36 main:574] : INFO : Epoch 1118 | loss: 0.0311721 | val_loss: 0.0311816 | Time: 9400.16 ms
[2021-07-10 23:01:46 main:574] : INFO : Epoch 1119 | loss: 0.0311689 | val_loss: 0.0311857 | Time: 9586.04 ms
[2021-07-10 23:01:55 main:574] : INFO : Epoch 1120 | loss: 0.0311733 | val_loss: 0.0311888 | Time: 9084.16 ms
[2021-07-10 23:02:05 main:574] : INFO : Epoch 1121 | loss: 0.0311718 | val_loss: 0.0311802 | Time: 9614.06 ms
[2021-07-10 23:02:14 main:574] : INFO : Epoch 1122 | loss: 0.0311697 | val_loss: 0.0311839 | Time: 9547.89 ms
[2021-07-10 23:02:24 main:574] : INFO : Epoch 1123 | loss: 0.0311698 | val_loss: 0.0311825 | Time: 9216.15 ms
[2021-07-10 23:02:33 main:574] : INFO : Epoch 1124 | loss: 0.0311692 | val_loss: 0.0311796 | Time: 9794.35 ms
[2021-07-10 23:02:43 main:574] : INFO : Epoch 1125 | loss: 0.03117 | val_loss: 0.0311828 | Time: 9630.17 ms
[2021-07-10 23:02:52 main:574] : INFO : Epoch 1126 | loss: 0.0311681 | val_loss: 0.0311809 | Time: 9340.4 ms
[2021-07-10 23:03:01 main:574] : INFO : Epoch 1127 | loss: 0.0311684 | val_loss: 0.0311817 | Time: 8943.6 ms
[2021-07-10 23:03:11 main:574] : INFO : Epoch 1128 | loss: 0.0311699 | val_loss: 0.0311854 | Time: 9571.53 ms
[2021-07-10 23:03:21 main:574] : INFO : Epoch 1129 | loss: 0.0311728 | val_loss: 0.0311866 | Time: 9758.69 ms
[2021-07-10 23:03:30 main:574] : INFO : Epoch 1130 | loss: 0.03117 | val_loss: 0.0311825 | Time: 9562.73 ms
[2021-07-10 23:03:39 main:574] : INFO : Epoch 1131 | loss: 0.0311693 | val_loss: 0.0311836 | Time: 9214.37 ms
[2021-07-10 23:03:49 main:574] : INFO : Epoch 1132 | loss: 0.0311699 | val_loss: 0.0311893 | Time: 9498.95 ms
[2021-07-10 23:03:58 main:574] : INFO : Epoch 1133 | loss: 0.031169 | val_loss: 0.0311828 | Time: 9403.13 ms
[2021-07-10 23:04:08 main:574] : INFO : Epoch 1134 | loss: 0.0311674 | val_loss: 0.0311813 | Time: 9415.72 ms
[2021-07-10 23:04:18 main:574] : INFO : Epoch 1135 | loss: 0.0311673 | val_loss: 0.0311845 | Time: 9641.77 ms
[2021-07-10 23:04:27 main:574] : INFO : Epoch 1136 | loss: 0.0311667 | val_loss: 0.0311977 | Time: 9441.65 ms
[2021-07-10 23:04:37 main:574] : INFO : Epoch 1137 | loss: 0.0311711 | val_loss: 0.0311811 | Time: 9652.47 ms
[2021-07-10 23:04:46 main:574] : INFO : Epoch 1138 | loss: 0.0311681 | val_loss: 0.0311835 | Time: 9261.53 ms
[2021-07-10 23:04:55 main:574] : INFO : Epoch 1139 | loss: 0.0311678 | val_loss: 0.0311821 | Time: 9386.21 ms
[2021-07-10 23:05:05 main:574] : INFO : Epoch 1140 | loss: 0.0311702 | val_loss: 0.031187 | Time: 9645.67 ms
[2021-07-10 23:05:15 main:574] : INFO : Epoch 1141 | loss: 0.0311682 | val_loss: 0.0311841 | Time: 9585.63 ms
[2021-07-10 23:05:24 main:574] : INFO : Epoch 1142 | loss: 0.0311657 | val_loss: 0.0311879 | Time: 9280.27 ms
[2021-07-10 23:05:34 main:574] : INFO : Epoch 1143 | loss: 0.031167 | val_loss: 0.0311841 | Time: 9823.2 ms
[2021-07-10 23:05:43 main:574] : INFO : Epoch 1144 | loss: 0.0311662 | val_loss: 0.0311813 | Time: 9519.69 ms
[2021-07-10 23:05:53 main:574] : INFO : Epoch 1145 | loss: 0.0311655 | val_loss: 0.0311843 | Time: 9406.83 ms
[2021-07-10 23:06:02 main:574] : INFO : Epoch 1146 | loss: 0.031166 | val_loss: 0.0311808 | Time: 9493.03 ms
[2021-07-10 23:06:12 main:574] : INFO : Epoch 1147 | loss: 0.0311657 | val_loss: 0.0311838 | Time: 9546.93 ms
[2021-07-10 23:06:21 main:574] : INFO : Epoch 1148 | loss: 0.0311692 | val_loss: 0.0311832 | Time: 9249.91 ms
[2021-07-10 23:06:30 main:574] : INFO : Epoch 1149 | loss: 0.0311654 | val_loss: 0.0311833 | Time: 9200.45 ms
[2021-07-10 23:06:40 main:574] : INFO : Epoch 1150 | loss: 0.0311674 | val_loss: 0.031186 | Time: 9629.06 ms
[2021-07-10 23:06:50 main:574] : INFO : Epoch 1151 | loss: 0.031169 | val_loss: 0.0311831 | Time: 9731.28 ms
[2021-07-10 23:06:59 main:574] : INFO : Epoch 1152 | loss: 0.0311679 | val_loss: 0.031193 | Time: 9386.23 ms
[2021-07-10 23:07:09 main:574] : INFO : Epoch 1153 | loss: 0.0311688 | val_loss: 0.0311849 | Time: 9750.86 ms
[2021-07-10 23:07:19 main:574] : INFO : Epoch 1154 | loss: 0.0311655 | val_loss: 0.031182 | Time: 9810.08 ms
[2021-07-10 23:07:28 main:574] : INFO : Epoch 1155 | loss: 0.0311672 | val_loss: 0.0311836 | Time: 9117.64 ms
[2021-07-10 23:07:37 main:574] : INFO : Epoch 1156 | loss: 0.0311659 | val_loss: 0.0311842 | Time: 9413.67 ms
[2021-07-10 23:07:47 main:574] : INFO : Epoch 1157 | loss: 0.031173 | val_loss: 0.0312018 | Time: 9346.99 ms
[2021-07-10 23:07:56 main:574] : INFO : Epoch 1158 | loss: 0.0359103 | val_loss: 0.0323255 | Time: 9465.79 ms
[2021-07-10 23:08:06 main:574] : INFO : Epoch 1159 | loss: 0.0344184 | val_loss: 0.0321473 | Time: 9747.16 ms
[2021-07-10 23:08:15 main:574] : INFO : Epoch 1160 | loss: 0.0316073 | val_loss: 0.0312655 | Time: 9477.36 ms
[2021-07-10 23:08:25 main:574] : INFO : Epoch 1161 | loss: 0.0312695 | val_loss: 0.0312253 | Time: 9534.45 ms
[2021-07-10 23:08:34 main:574] : INFO : Epoch 1162 | loss: 0.031207 | val_loss: 0.0312003 | Time: 9196.79 ms
[2021-07-10 23:08:43 main:574] : INFO : Epoch 1163 | loss: 0.0311891 | val_loss: 0.0311888 | Time: 9217.9 ms
[2021-07-10 23:08:53 main:574] : INFO : Epoch 1164 | loss: 0.0311808 | val_loss: 0.0311864 | Time: 9447.57 ms
[2021-07-10 23:09:02 main:574] : INFO : Epoch 1165 | loss: 0.031177 | val_loss: 0.031184 | Time: 9329.98 ms
[2021-07-10 23:09:11 main:574] : INFO : Epoch 1166 | loss: 0.031174 | val_loss: 0.0311831 | Time: 9136.84 ms
[2021-07-10 23:09:21 main:574] : INFO : Epoch 1167 | loss: 0.0311726 | val_loss: 0.0311839 | Time: 9540.26 ms
[2021-07-10 23:09:30 main:574] : INFO : Epoch 1168 | loss: 0.031173 | val_loss: 0.0311823 | Time: 9269.07 ms
[2021-07-10 23:09:40 main:574] : INFO : Epoch 1169 | loss: 0.0311713 | val_loss: 0.0311827 | Time: 9476.53 ms
[2021-07-10 23:09:49 main:574] : INFO : Epoch 1170 | loss: 0.0311707 | val_loss: 0.0311826 | Time: 9587.05 ms
[2021-07-10 23:09:59 main:574] : INFO : Epoch 1171 | loss: 0.0311708 | val_loss: 0.0311819 | Time: 9450.47 ms
[2021-07-10 23:10:08 main:574] : INFO : Epoch 1172 | loss: 0.0311702 | val_loss: 0.0311806 | Time: 9426.79 ms
[2021-07-10 23:10:17 main:574] : INFO : Epoch 1173 | loss: 0.031171 | val_loss: 0.0311813 | Time: 9192.35 ms
[2021-07-10 23:10:27 main:574] : INFO : Epoch 1174 | loss: 0.0311711 | val_loss: 0.0311818 | Time: 9527.95 ms
[2021-07-10 23:10:36 main:574] : INFO : Epoch 1175 | loss: 0.03117 | val_loss: 0.0311824 | Time: 9469.5 ms
[2021-07-10 23:10:46 main:574] : INFO : Epoch 1176 | loss: 0.0311698 | val_loss: 0.0311816 | Time: 9457.85 ms
[2021-07-10 23:10:55 main:574] : INFO : Epoch 1177 | loss: 0.0311698 | val_loss: 0.0311815 | Time: 9145.71 ms
[2021-07-10 23:11:05 main:574] : INFO : Epoch 1178 | loss: 0.0311688 | val_loss: 0.031182 | Time: 9436.71 ms
[2021-07-10 23:11:14 main:574] : INFO : Epoch 1179 | loss: 0.0311696 | val_loss: 0.0311824 | Time: 9692.67 ms
[2021-07-10 23:11:24 main:574] : INFO : Epoch 1180 | loss: 0.0311696 | val_loss: 0.0311816 | Time: 9294.16 ms
[2021-07-10 23:11:33 main:574] : INFO : Epoch 1181 | loss: 0.0311702 | val_loss: 0.0311814 | Time: 9424.97 ms
[2021-07-10 23:11:42 main:574] : INFO : Epoch 1182 | loss: 0.0311699 | val_loss: 0.0311811 | Time: 9248.99 ms
[2021-07-10 23:11:52 main:574] : INFO : Epoch 1183 | loss: 0.0311699 | val_loss: 0.031182 | Time: 9349.17 ms
[2021-07-10 23:12:01 main:574] : INFO : Epoch 1184 | loss: 0.0311697 | val_loss: 0.0311811 | Time: 9568.32 ms
[2021-07-10 23:12:11 main:574] : INFO : Epoch 1185 | loss: 0.0311686 | val_loss: 0.0311806 | Time: 9263.6 ms
[2021-07-10 23:12:20 main:574] : INFO : Epoch 1186 | loss: 0.0311687 | val_loss: 0.0311795 | Time: 9330.85 ms
[2021-07-10 23:12:29 main:574] : INFO : Epoch 1187 | loss: 0.0311691 | val_loss: 0.0311815 | Time: 9456.32 ms
[2021-07-10 23:12:39 main:574] : INFO : Epoch 1188 | loss: 0.0311688 | val_loss: 0.0311836 | Time: 9567.88 ms
[2021-07-10 23:12:48 main:574] : INFO : Epoch 1189 | loss: 0.0311694 | val_loss: 0.0311819 | Time: 9317.59 ms
[2021-07-10 23:12:58 main:574] : INFO : Epoch 1190 | loss: 0.031169 | val_loss: 0.0311821 | Time: 9648.26 ms
[2021-07-10 23:13:07 main:574] : INFO : Epoch 1191 | loss: 0.0311668 | val_loss: 0.0311817 | Time: 9114.39 ms
[2021-07-10 23:13:16 main:574] : INFO : Epoch 1192 | loss: 0.0311674 | val_loss: 0.0311818 | Time: 9185.96 ms
[2021-07-10 23:13:26 main:574] : INFO : Epoch 1193 | loss: 0.0311677 | val_loss: 0.0311822 | Time: 9323.86 ms
[2021-07-10 23:13:35 main:574] : INFO : Epoch 1194 | loss: 0.0311671 | val_loss: 0.0311811 | Time: 9572.08 ms
[2021-07-10 23:13:45 main:574] : INFO : Epoch 1195 | loss: 0.0311661 | val_loss: 0.0311809 | Time: 9605.75 ms
[2021-07-10 23:13:54 main:574] : INFO : Epoch 1196 | loss: 0.0311664 | val_loss: 0.0311818 | Time: 9373.8 ms
[2021-07-10 23:14:04 main:574] : INFO : Epoch 1197 | loss: 0.0311658 | val_loss: 0.0311817 | Time: 9475.97 ms
[2021-07-10 23:14:13 main:574] : INFO : Epoch 1198 | loss: 0.0311662 | val_loss: 0.0311806 | Time: 9701.72 ms
[2021-07-10 23:14:23 main:574] : INFO : Epoch 1199 | loss: 0.0311669 | val_loss: 0.0311811 | Time: 9296.69 ms
[2021-07-10 23:14:32 main:574] : INFO : Epoch 1200 | loss: 0.0311657 | val_loss: 0.0311816 | Time: 9404.39 ms
[2021-07-10 23:14:41 main:574] : INFO : Epoch 1201 | loss: 0.0311668 | val_loss: 0.0311817 | Time: 9178.28 ms
[2021-07-10 23:14:51 main:574] : INFO : Epoch 1202 | loss: 0.0311678 | val_loss: 0.031184 | Time: 9271.02 ms
[2021-07-10 23:15:00 main:574] : INFO : Epoch 1203 | loss: 0.0311658 | val_loss: 0.0311858 | Time: 9401.51 ms
[2021-07-10 23:15:10 main:574] : INFO : Epoch 1204 | loss: 0.0311666 | val_loss: 0.0311852 | Time: 9586.79 ms
[2021-07-10 23:15:19 main:574] : INFO : Epoch 1205 | loss: 0.0311657 | val_loss: 0.0311825 | Time: 9296.54 ms
[2021-07-10 23:15:29 main:574] : INFO : Epoch 1206 | loss: 0.0311655 | val_loss: 0.0311826 | Time: 9578.39 ms
[2021-07-10 23:15:38 main:574] : INFO : Epoch 1207 | loss: 0.0311665 | val_loss: 0.0311834 | Time: 9444.43 ms
[2021-07-10 23:15:47 main:574] : INFO : Epoch 1208 | loss: 0.0311687 | val_loss: 0.0311844 | Time: 9273.27 ms
[2021-07-10 23:15:57 main:574] : INFO : Epoch 1209 | loss: 0.0311662 | val_loss: 0.0311836 | Time: 9439.05 ms
[2021-07-10 23:16:06 main:574] : INFO : Epoch 1210 | loss: 0.0311666 | val_loss: 0.0311836 | Time: 9564.82 ms
[2021-07-10 23:16:16 main:574] : INFO : Epoch 1211 | loss: 0.0311672 | val_loss: 0.0311831 | Time: 9184.77 ms
[2021-07-10 23:16:25 main:574] : INFO : Epoch 1212 | loss: 0.0311672 | val_loss: 0.031186 | Time: 9647.22 ms
[2021-07-10 23:16:35 main:574] : INFO : Epoch 1213 | loss: 0.0311672 | val_loss: 0.0311834 | Time: 9502.75 ms
[2021-07-10 23:16:44 main:574] : INFO : Epoch 1214 | loss: 0.0311657 | val_loss: 0.0311813 | Time: 9305.61 ms
[2021-07-10 23:16:54 main:574] : INFO : Epoch 1215 | loss: 0.031166 | val_loss: 0.0311806 | Time: 9439.58 ms
[2021-07-10 23:17:03 main:574] : INFO : Epoch 1216 | loss: 0.0311669 | val_loss: 0.0311816 | Time: 9610.43 ms
[2021-07-10 23:17:13 main:574] : INFO : Epoch 1217 | loss: 0.0311658 | val_loss: 0.031182 | Time: 9519.88 ms
[2021-07-10 23:17:22 main:574] : INFO : Epoch 1218 | loss: 0.0311655 | val_loss: 0.0311803 | Time: 9484.02 ms
[2021-07-10 23:17:32 main:574] : INFO : Epoch 1219 | loss: 0.0311666 | val_loss: 0.0311835 | Time: 9440.07 ms
[2021-07-10 23:17:41 main:574] : INFO : Epoch 1220 | loss: 0.0311682 | val_loss: 0.0311794 | Time: 9399.76 ms
[2021-07-10 23:17:51 main:574] : INFO : Epoch 1221 | loss: 0.0311671 | val_loss: 0.0311811 | Time: 9590.68 ms
[2021-07-10 23:18:00 main:574] : INFO : Epoch 1222 | loss: 0.0311674 | val_loss: 0.0311798 | Time: 9724.33 ms
[2021-07-10 23:18:10 main:574] : INFO : Epoch 1223 | loss: 0.0311675 | val_loss: 0.0311808 | Time: 9653.32 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 2080 Ti)
[2021-07-10 23:33:04 main:435] : INFO : Set logging level to 1
[2021-07-10 23:33:04 main:441] : INFO : Running in BOINC Client mode
[2021-07-10 23:33:04 main:444] : INFO : Resolving all filenames
[2021-07-10 23:33:04 main:452] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 (exists = 1)
[2021-07-10 23:33:04 main:452] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1624423002-31433-2_0_r1949237289_1 (exists = 1)
[2021-07-10 23:33:04 main:452] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1624423002-31433-2_0_r1949237289_0 (exists = 0)
[2021-07-10 23:33:04 main:452] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1624423002-31433-2 (exists = 1)
[2021-07-10 23:33:04 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2021-07-10 23:33:04 main:472] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5
[2021-07-10 23:33:04 main:474] : INFO : Configuration:
[2021-07-10 23:33:04 main:475] : INFO : Model type: GRU
[2021-07-10 23:33:04 main:476] : INFO : Validation Loss Threshold: 0.0001
[2021-07-10 23:33:04 main:477] : INFO : Max Epochs: 2048
[2021-07-10 23:33:04 main:478] : INFO : Batch Size: 128
[2021-07-10 23:33:04 main:479] : INFO : Learning Rate: 0.01
[2021-07-10 23:33:04 main:480] : INFO : Patience: 10
[2021-07-10 23:33:04 main:481] : INFO : Hidden Width: 12
[2021-07-10 23:33:04 main:482] : INFO : # Recurrent Layers: 4
[2021-07-10 23:33:04 main:483] : INFO : # Backend Layers: 4
[2021-07-10 23:33:04 main:484] : INFO : # Threads: 1
[2021-07-10 23:33:04 main:486] : INFO : Preparing Dataset
[2021-07-10 23:33:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-10 23:33:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-10 23:33:07 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2021-07-10 23:33:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-10 23:33:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-10 23:33:07 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2021-07-10 23:33:07 main:494] : INFO : Creating Model
[2021-07-10 23:33:07 main:507] : INFO : Preparing config file
[2021-07-10 23:33:07 main:511] : INFO : Found checkpoint, attempting to load...
[2021-07-10 23:33:07 main:512] : INFO : Loading config
[2021-07-10 23:33:07 main:514] : INFO : Loading state
[2021-07-10 23:33:09 main:559] : INFO : Loading DataLoader into Memory
[2021-07-10 23:33:09 main:562] : INFO : Starting Training
[2021-07-10 23:33:20 main:574] : INFO : Epoch 1224 | loss: 0.0316366 | val_loss: 0.0312721 | Time: 10147.5 ms
[2021-07-10 23:33:29 main:574] : INFO : Epoch 1225 | loss: 0.0312405 | val_loss: 0.0312058 | Time: 9463.19 ms
[2021-07-10 23:33:39 main:574] : INFO : Epoch 1226 | loss: 0.0311828 | val_loss: 0.0311903 | Time: 9622.28 ms
[2021-07-10 23:33:49 main:574] : INFO : Epoch 1227 | loss: 0.0311731 | val_loss: 0.0311811 | Time: 9778.5 ms
[2021-07-10 23:33:58 main:574] : INFO : Epoch 1228 | loss: 0.03117 | val_loss: 0.031181 | Time: 9447.2 ms
[2021-07-10 23:34:07 main:574] : INFO : Epoch 1229 | loss: 0.0311706 | val_loss: 0.0311832 | Time: 9388.61 ms
[2021-07-10 23:34:17 main:574] : INFO : Epoch 1230 | loss: 0.0311712 | val_loss: 0.0311803 | Time: 9295.4 ms
[2021-07-10 23:34:26 main:574] : INFO : Epoch 1231 | loss: 0.0311739 | val_loss: 0.0311795 | Time: 9559.93 ms
[2021-07-10 23:34:36 main:574] : INFO : Epoch 1232 | loss: 0.0311704 | val_loss: 0.0311787 | Time: 9485.72 ms
[2021-07-10 23:34:45 main:574] : INFO : Epoch 1233 | loss: 0.0311703 | val_loss: 0.0311787 | Time: 9349.37 ms
[2021-07-10 23:34:54 main:574] : INFO : Epoch 1234 | loss: 0.0311711 | val_loss: 0.0311786 | Time: 9304.28 ms
[2021-07-10 23:35:04 main:574] : INFO : Epoch 1235 | loss: 0.0311695 | val_loss: 0.0311777 | Time: 9487.75 ms
[2021-07-10 23:35:13 main:574] : INFO : Epoch 1236 | loss: 0.0311689 | val_loss: 0.0311784 | Time: 9482.35 ms
[2021-07-10 23:35:23 main:574] : INFO : Epoch 1237 | loss: 0.0311679 | val_loss: 0.0311753 | Time: 9425.95 ms
[2021-07-10 23:35:32 main:574] : INFO : Epoch 1238 | loss: 0.0311654 | val_loss: 0.0311785 | Time: 9320.14 ms
[2021-07-10 23:35:42 main:574] : INFO : Epoch 1239 | loss: 0.0311668 | val_loss: 0.0311751 | Time: 9268.26 ms
[2021-07-10 23:35:51 main:574] : INFO : Epoch 1240 | loss: 0.0311662 | val_loss: 0.0311774 | Time: 9346.57 ms
[2021-07-10 23:36:00 main:574] : INFO : Epoch 1241 | loss: 0.0311687 | val_loss: 0.0311796 | Time: 9253.15 ms
[2021-07-10 23:36:09 main:574] : INFO : Epoch 1242 | loss: 0.031166 | val_loss: 0.0311737 | Time: 9176.12 ms
[2021-07-10 23:36:19 main:574] : INFO : Epoch 1243 | loss: 0.0311652 | val_loss: 0.0311784 | Time: 9171.87 ms
[2021-07-10 23:36:28 main:574] : INFO : Epoch 1244 | loss: 0.0311653 | val_loss: 0.0311756 | Time: 9362.88 ms
[2021-07-10 23:36:37 main:574] : INFO : Epoch 1245 | loss: 0.0311666 | val_loss: 0.0311736 | Time: 9262.34 ms
[2021-07-10 23:36:47 main:574] : INFO : Epoch 1246 | loss: 0.0311643 | val_loss: 0.0311717 | Time: 9531.87 ms
[2021-07-10 23:36:56 main:574] : INFO : Epoch 1247 | loss: 0.0311657 | val_loss: 0.0311774 | Time: 9161.89 ms
[2021-07-10 23:37:05 main:574] : INFO : Epoch 1248 | loss: 0.0311646 | val_loss: 0.0311763 | Time: 9345.94 ms
[2021-07-10 23:37:15 main:574] : INFO : Epoch 1249 | loss: 0.0311633 | val_loss: 0.0311711 | Time: 9307.5 ms
[2021-07-10 23:37:24 main:574] : INFO : Epoch 1250 | loss: 0.0311628 | val_loss: 0.0311705 | Time: 9492.55 ms
[2021-07-10 23:37:34 main:574] : INFO : Epoch 1251 | loss: 0.0311664 | val_loss: 0.0311733 | Time: 9559.88 ms
[2021-07-10 23:37:43 main:574] : INFO : Epoch 1252 | loss: 0.0311607 | val_loss: 0.0311708 | Time: 9687.39 ms
[2021-07-10 23:37:53 main:574] : INFO : Epoch 1253 | loss: 0.0311596 | val_loss: 0.0311718 | Time: 9309.02 ms
[2021-07-10 23:38:02 main:574] : INFO : Epoch 1254 | loss: 0.0311591 | val_loss: 0.0311719 | Time: 9584.46 ms
[2021-07-10 23:38:12 main:574] : INFO : Epoch 1255 | loss: 0.0311658 | val_loss: 0.031186 | Time: 9287.87 ms
[2021-07-10 23:38:21 main:574] : INFO : Epoch 1256 | loss: 0.0311695 | val_loss: 0.0311743 | Time: 9189.41 ms
[2021-07-10 23:38:30 main:574] : INFO : Epoch 1257 | loss: 0.0311654 | val_loss: 0.031172 | Time: 9244.15 ms
[2021-07-10 23:38:39 main:574] : INFO : Epoch 1258 | loss: 0.0311618 | val_loss: 0.0311703 | Time: 9229.67 ms
[2021-07-10 23:38:49 main:574] : INFO : Epoch 1259 | loss: 0.031161 | val_loss: 0.0311704 | Time: 9361.15 ms
[2021-07-10 23:38:58 main:574] : INFO : Epoch 1260 | loss: 0.0311586 | val_loss: 0.0311714 | Time: 9259.99 ms
[2021-07-10 23:39:07 main:574] : INFO : Epoch 1261 | loss: 0.0311599 | val_loss: 0.0311723 | Time: 9222.87 ms
[2021-07-10 23:39:16 main:574] : INFO : Epoch 1262 | loss: 0.0311586 | val_loss: 0.0311689 | Time: 9102.9 ms
[2021-07-10 23:39:26 main:574] : INFO : Epoch 1263 | loss: 0.0311578 | val_loss: 0.0311684 | Time: 9354.02 ms
[2021-07-10 23:39:35 main:574] : INFO : Epoch 1264 | loss: 0.0311585 | val_loss: 0.0311692 | Time: 9505.76 ms
[2021-07-10 23:39:45 main:574] : INFO : Epoch 1265 | loss: 0.0311567 | val_loss: 0.0311691 | Time: 9330.07 ms
[2021-07-10 23:39:54 main:574] : INFO : Epoch 1266 | loss: 0.0311559 | val_loss: 0.0311687 | Time: 9423.24 ms
[2021-07-10 23:40:03 main:574] : INFO : Epoch 1267 | loss: 0.031156 | val_loss: 0.0311787 | Time: 9225.64 ms
[2021-07-10 23:40:13 main:574] : INFO : Epoch 1268 | loss: 0.031158 | val_loss: 0.0311695 | Time: 9388.72 ms
[2021-07-10 23:40:22 main:574] : INFO : Epoch 1269 | loss: 0.0311592 | val_loss: 0.0311688 | Time: 9206.5 ms
[2021-07-10 23:40:31 main:574] : INFO : Epoch 1270 | loss: 0.0311588 | val_loss: 0.031172 | Time: 9256.78 ms
[2021-07-10 23:40:41 main:574] : INFO : Epoch 1271 | loss: 0.0311629 | val_loss: 0.0311738 | Time: 9665.02 ms
[2021-07-10 23:40:50 main:574] : INFO : Epoch 1272 | loss: 0.0311636 | val_loss: 0.0311749 | Time: 9241.13 ms
[2021-07-10 23:40:59 main:574] : INFO : Epoch 1273 | loss: 0.0311649 | val_loss: 0.0311771 | Time: 9240.91 ms
[2021-07-10 23:41:09 main:574] : INFO : Epoch 1274 | loss: 0.031164 | val_loss: 0.0311737 | Time: 9188.57 ms
[2021-07-10 23:41:18 main:574] : INFO : Epoch 1275 | loss: 0.0311627 | val_loss: 0.0311745 | Time: 9279.04 ms
[2021-07-10 23:41:27 main:574] : INFO : Epoch 1276 | loss: 0.0311636 | val_loss: 0.0311732 | Time: 9240.26 ms
[2021-07-10 23:41:36 main:574] : INFO : Epoch 1277 | loss: 0.0311629 | val_loss: 0.0311709 | Time: 9095.12 ms
[2021-07-10 23:41:45 main:574] : INFO : Epoch 1278 | loss: 0.0311609 | val_loss: 0.0311724 | Time: 9092.59 ms
[2021-07-10 23:41:55 main:574] : INFO : Epoch 1279 | loss: 0.0311596 | val_loss: 0.0311724 | Time: 9134.65 ms
[2021-07-10 23:42:04 main:574] : INFO : Epoch 1280 | loss: 0.0311572 | val_loss: 0.0311739 | Time: 9292.11 ms
[2021-07-10 23:42:13 main:574] : INFO : Epoch 1281 | loss: 0.0311592 | val_loss: 0.031176 | Time: 9214.35 ms
[2021-07-10 23:42:22 main:574] : INFO : Epoch 1282 | loss: 0.0311591 | val_loss: 0.0311744 | Time: 9263.42 ms
[2021-07-10 23:42:31 main:574] : INFO : Epoch 1283 | loss: 0.0311566 | val_loss: 0.0311717 | Time: 9013.44 ms
[2021-07-10 23:42:41 main:574] : INFO : Epoch 1284 | loss: 0.0311596 | val_loss: 0.0311721 | Time: 9442.8 ms
[2021-07-10 23:42:50 main:574] : INFO : Epoch 1285 | loss: 0.0311597 | val_loss: 0.0311741 | Time: 9463.25 ms
[2021-07-10 23:43:00 main:574] : INFO : Epoch 1286 | loss: 0.0311575 | val_loss: 0.0311756 | Time: 9187.47 ms
[2021-07-10 23:43:09 main:574] : INFO : Epoch 1287 | loss: 0.0311557 | val_loss: 0.0311728 | Time: 9404.17 ms
[2021-07-10 23:43:18 main:574] : INFO : Epoch 1288 | loss: 0.0311556 | val_loss: 0.0311719 | Time: 9232.8 ms
[2021-07-10 23:43:28 main:574] : INFO : Epoch 1289 | loss: 0.031155 | val_loss: 0.0311722 | Time: 9423.05 ms
[2021-07-10 23:43:37 main:574] : INFO : Epoch 1290 | loss: 0.031155 | val_loss: 0.0311718 | Time: 9266.49 ms
[2021-07-10 23:43:47 main:574] : INFO : Epoch 1291 | loss: 0.0311538 | val_loss: 0.0311703 | Time: 9589.48 ms
[2021-07-10 23:43:56 main:574] : INFO : Epoch 1292 | loss: 0.0311548 | val_loss: 0.0311702 | Time: 9146.23 ms
[2021-07-10 23:44:05 main:574] : INFO : Epoch 1293 | loss: 0.0311529 | val_loss: 0.0311704 | Time: 9005.64 ms
[2021-07-10 23:44:14 main:574] : INFO : Epoch 1294 | loss: 0.0311633 | val_loss: 0.0311729 | Time: 9176.8 ms
[2021-07-10 23:44:23 main:574] : INFO : Epoch 1295 | loss: 0.0311675 | val_loss: 0.0311712 | Time: 9258.12 ms
[2021-07-10 23:44:33 main:574] : INFO : Epoch 1296 | loss: 0.0311675 | val_loss: 0.0311709 | Time: 9286.7 ms
[2021-07-10 23:44:42 main:574] : INFO : Epoch 1297 | loss: 0.0311669 | val_loss: 0.0311704 | Time: 9290.2 ms
[2021-07-10 23:44:51 main:574] : INFO : Epoch 1298 | loss: 0.0311639 | val_loss: 0.0311693 | Time: 9165.13 ms
[2021-07-10 23:45:00 main:574] : INFO : Epoch 1299 | loss: 0.0311647 | val_loss: 0.0311743 | Time: 9302.24 ms
[2021-07-10 23:45:10 main:574] : INFO : Epoch 1300 | loss: 0.0311641 | val_loss: 0.0311683 | Time: 9247.48 ms
[2021-07-10 23:45:19 main:574] : INFO : Epoch 1301 | loss: 0.0311618 | val_loss: 0.031171 | Time: 9433.54 ms
[2021-07-10 23:45:28 main:574] : INFO : Epoch 1302 | loss: 0.0311614 | val_loss: 0.031168 | Time: 9291.83 ms
[2021-07-10 23:45:37 main:574] : INFO : Epoch 1303 | loss: 0.0311634 | val_loss: 0.0311732 | Time: 9028.38 ms
[2021-07-10 23:45:47 main:574] : INFO : Epoch 1304 | loss: 0.0311635 | val_loss: 0.0311696 | Time: 9152.78 ms
[2021-07-10 23:45:56 main:574] : INFO : Epoch 1305 | loss: 0.0311608 | val_loss: 0.0311676 | Time: 9007.35 ms
[2021-07-10 23:46:05 main:574] : INFO : Epoch 1306 | loss: 0.0311602 | val_loss: 0.0311681 | Time: 9393.29 ms
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFC36BC9A92
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.17.0
Dump Timestamp : 07/10/21 23:46:07
Install Directory : C:\Program Files\BOINC\
Data Directory : C:\ProgramData\BOINC
Project Symstore :
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\62;C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome
ModLoad: 0000000091440000 00000000003df000 C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000038ff0000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.19041.1023 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1023
ModLoad: 00000000389b0000 00000000000bd000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.19041.1023 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1023
ModLoad: 0000000036b00000 00000000002c9000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.19041.1023 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1023
ModLoad: 0000000037060000 00000000001a0000 C:\WINDOWS\System32\USER32.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000036f30000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.19041.1081 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1081
ModLoad: 0000000038c20000 000000000002a000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.19041.746 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.746
ModLoad: 00000000369f0000 000000000010b000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.928) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.19041.928 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.928
ModLoad: 0000000036f60000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 0000000036840000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 0000000004780000 0000000000058000 C:\ProgramData\BOINC\slots\62\c10.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000038cd0000 00000000000ac000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.1052) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000038b80000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.19041.546
ModLoad: 0000000037200000 000000000009b000 C:\WINDOWS\System32\sechost.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000037a30000 000000000012a000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000001c370000 0000000000092000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.28.29325.2) (-exported- Symbols Loaded)
Linked PDB Filename : D:\agent\_work\49\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
File Version : 14.28.29325.2 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29325.2
ModLoad: 0000000029370000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.19041.867 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.867
ModLoad: 000000001c410000 0000000000019000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.28.29325.2) (-exported- Symbols Loaded)
Linked PDB Filename : D:\agent\_work\49\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
File Version : 14.28.29325.2 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29325.2
ModLoad: 000000001c360000 000000000000c000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.28.29325.2) (-exported- Symbols Loaded)
Linked PDB Filename : D:\agent\_work\49\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
File Version : 14.28.29325.2 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29325.2
ModLoad: 000000009e200000 000000000abe7000 C:\ProgramData\BOINC\slots\62\torch_cpu.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000b2120000 00000000002db000 C:\ProgramData\BOINC\slots\62\fbgemm.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000b7080000 000000000020d000 C:\ProgramData\BOINC\slots\62\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
Linked PDB Filename : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
File Version : 20200205
Company Name : Intel Corporation
Product Name : Intel(R) OpenMP* Runtime Library
Product Version : 5.0
ModLoad: 0000000006190000 0000000000040000 C:\ProgramData\BOINC\slots\62\asmjit.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000796b0000 0000000024b49000 C:\ProgramData\BOINC\slots\62\torch_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000f71a0000 0000000000045000 C:\ProgramData\BOINC\slots\62\c10_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000076600000 00000000030a5000 C:\ProgramData\BOINC\slots\62\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1031
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA SPARSE BLAS Library
Product Version : 6,14,11,1031
ModLoad: 0000000073140000 00000000034b2000 C:\ProgramData\BOINC\slots\62\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA RAND Library
Product Version : 6,14,11,1012
ModLoad: 000000000fc50000 0000000000010000 C:\ProgramData\BOINC\slots\62\nvToolsExt64_1.dll (-exported- Symbols Loaded)
Linked PDB Filename : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb
ModLoad: 000000004ae90000 0000000003d5b000 C:\ProgramData\BOINC\slots\62\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 000000004ebf0000 000000000919f000 C:\ProgramData\BOINC\slots\62\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA FFT Library
Product Version : 6,14,11,1012
ModLoad: 0000000048fc0000 0000000001eca000 C:\ProgramData\BOINC\slots\62\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 0000000057d90000 000000001b3a4000 C:\ProgramData\BOINC\slots\62\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,10020
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.86 CUDNN Library
Product Version : 6,14,11,10020
ModLoad: 0000000037880000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000034680000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000372a0000 0000000000469000 C:\WINDOWS\System32\Setupapi.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : setupapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000367c0000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cfgmgr32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000036810000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000364e0000 000000000002c000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : devobj.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000037000000 0000000000060000 C:\WINDOWS\System32\WINTRUST.dll (6.2.19041.1052) (-exported- Symbols Loaded)
Linked PDB Filename : wintrust.pdb
File Version : 10.0.19041.1052 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1052
ModLoad: 0000000036dd0000 000000000015f000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.844) (-exported- Symbols Loaded)
Linked PDB Filename : crypt32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000036320000 0000000000012000 C:\WINDOWS\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msasn1.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000037ed0000 000000000073f000 C:\WINDOWS\System32\Shell32.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shell32.pdb
File Version : 10.0.19041.964 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.964
ModLoad: 0000000034880000 0000000000791000 C:\WINDOWS\SYSTEM32\windows.storage.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : Windows.Storage.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000037b60000 0000000000355000 C:\WINDOWS\System32\combase.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : combase.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000361a0000 000000000002c000 C:\WINDOWS\SYSTEM32\Wldp.dll (6.2.19041.662) (-exported- Symbols Loaded)
Linked PDB Filename : WLDP.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000038a70000 00000000000ae000 C:\WINDOWS\System32\SHCORE.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shcore.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000038620000 0000000000055000 C:\WINDOWS\System32\shlwapi.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shlwapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000e4650000 00000000002cc000 C:\WINDOWS\system32\nvcuda.dll (30.0.14.7111) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r470_96\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
File Version : 30.0.14.7111
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.4.56 driver
Product Version : 30.0.14.7111
ModLoad: 0000000030250000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000acb50000 000000000161a000 C:\WINDOWS\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_4a746d937e6a7240\nvcuda64.dll (30.0.14.7111) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r470_96\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
File Version : 30.0.14.7111
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.4.56 driver
Product Version : 30.0.14.7111
ModLoad: 000000002fa20000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : cryptnet.pdb
File Version : 10.0.19041.906 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.906
ModLoad: 0000000036100000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cryptbase.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000034150000 000000000009e000 C:\WINDOWS\system32\uxtheme.dll (6.2.19041.1081) (-exported- Symbols Loaded)
Linked PDB Filename : UxTheme.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000002cc30000 0000000000718000 C:\WINDOWS\SYSTEM32\nvapi64.dll (30.0.14.7111) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r470\r470_96\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
File Version : 30.0.14.7111
Company Name : NVIDIA Corporation
Product Name : NVIDIA Windows drivers
Product Version : 30.0.14.7111
ModLoad: 0000000038880000 000000000012a000 C:\WINDOWS\System32\ole32.dll (6.2.19041.746) (-exported- Symbols Loaded)
Linked PDB Filename : ole32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000000c4b0000 0000000000009000 C:\ProgramData\BOINC\slots\62\caffe2_nvrtc.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000047ff0000 0000000000fca000 C:\ProgramData\BOINC\slots\62\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6.14.11.9000
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.89 NVRTC Library
Product Version : 6.14.11.9000
ModLoad: 0000000036730000 0000000000083000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.19041.1023 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1023
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 3727, Write: 2853, Other 3934
- I/O Transfers Counters -
Read: 6617198, Write: 8885283, Other 56772
- Paged Pool Usage -
QuotaPagedPoolUsage: 3537520, QuotaPeakPagedPoolUsage: 3537792
QuotaNonPagedPoolUsage: 61544, QuotaPeakNonPagedPoolUsage: 61808
- Virtual Memory Usage -
VirtualSize: 436899840, PeakVirtualSize: 1069645824
- Pagefile Usage -
PagefileUsage: 436899840, PeakPagefileUsage: 436916224
- Working Set Size -
WorkingSetSize: -2063298560, PeakWorkingSetSize: -2057846784, PageFaultCount: 846915
*** Dump of thread ID 17220 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 12599441.000000
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFC36BC9A92
- Registers -
rax=0000000000000000 rbx=0000000000000001 rcx=00000000916f8c40 rdx=00000000b23fe550 rsi=0000000000000000 rdi=0000000000000000
r8=00000000b23fe550 r9=00000000916f8c30 r10=0000000000000fff r11=0000000000000ff0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=0000000036bc9a92 rsp=00000000b23fe528 rbp=0000000000000000
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
b23fe520 91479100 00000001 b23fe550 b23fe550 916f8c30 KERNELBASE!DebugBreak+0x0
b23fe960 91479ce4 02600000 b23fecc4 00000000 00000300 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
b23ff8c0 9147a084 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
b23ff8f0 389c7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
b23ff920 39042651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
b23ff9a0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 12296 (state: Running): ***
- Information -
Status: Base Priority: Unknown, Priority: Unknown, , Kernel Time: 2670937600.000000, User Time: 1802968704.000000, Wait Time: 12599417.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=000000003f2ccb08 rdx=0000000000000000 rsi=0000000000000000 rdi=000000003f2ccb08
r8=0000000000000010 r9=000000001c410000 r10=000000003eefc9f8 r11=00000000b1ee6188 r12=0000000000000000 r13=0000000000000000
r14=00000000b1ee5760 r15=000000003f2ccab8 rip=0000000039090814 rsp=00000000b1ee5738 rbp=00000000b1ee5790
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
b1ee5730 39054021 00000000 a37cea63 b03073e0 b1ee5770 ntdll!NtWaitForAlertByThreadId+0x0
b1ee57b0 36b6ce29 3f2ccb00 3f2ccaf8 3f2ccab0 b1ee53d0 ntdll!RtlSleepConditionVariableSRW+0x0
b1ee57f0 1c3abf3d 00000000 b1ee53d0 f71a2e0d 047a2433 KERNELBASE!SleepConditionVariableSRW+0x0
b1ee5830 1c382798 00000000 a37d0231 047b9c50 00000000 MSVCP140!__crtSetThreadpoolWait+0x0
b1ee5860 1c382a4a f79d5988 3f2ccab0 4d19af80 3f2ccaf8 MSVCP140!std::_Winerror_message+0x0
b1ee5890 a53e064b 3f2ccab0 b1ee5990 3f2ccab0 ffffffff MSVCP140!_Cnd_wait+0x0
b1ee58d0 a53e25b1 af731910 4d19af70 4d19af70 00000000 torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0
b1ee5bc0 a53df15d b1ee5ea0 b1ee5ea0 b1ee5ea0 af731910 torch_cpu!torch::autograd::Engine::thread_main+0x0
b1ee5e30 a53deea0 38e3f8e0 a8906d10 fffffffe 00000000 torch_cpu!torch::autograd::Engine::execute_with_graph_task+0x0
b1ee5f60 a53d56f9 523050f0 523050f0 b1ee5f50 b1ee6058 torch_cpu!torch::autograd::Engine::execute+0x0
b1ee6480 a53d5086 00000000 b1ee68e8 52305000 b1ee6550 torch_cpu!torch::autograd::Engine::make_anomaly_metadata+0x0
b1ee6520 a56d09be 3eef10a0 523050f8 00000000 b1ee64c0 torch_cpu!torch::autograd::backward+0x0
b1ee6600 a5101cd0 b1356350 00000000 00000000 00000000 torch_cpu!torch::autograd::VariableType::allCUDATypes+0x0
b1ee6630 a3da0a06 00000000 00000000 00010001 00000001 torch_cpu!torch::autograd::GraphRoot::apply+0x0
b1ee6860 91457166 b1ee6930 b1ee6970 b1ee6930 b1ee6ed0 torch_cpu!at::Tensor::backward+0x0
b1ee6d60 9145b950 9148d95f 9148d95f 00000000 982b1c90 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0
b1effb00 9169c6f8 b12689c0 00000000 00000000 af725d40 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
b1effb40 389c7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
b1effb70 39042651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
b1effbf0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 13184 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 12599437.000000
- Registers -
rax=000000000000005b rbx=0000000000000007 rcx=0000000000000007 rdx=00000000b24ff8a0 rsi=0000000000000000 rdi=0000000000000007
r8=00000000b1739810 r9=00000000b14edf00 r10=0000000000000000 r11=00000000b24ff810 r12=0000000000000064 r13=00000000b24ff8a0
r14=00000000b24ff5a0 r15=0000000000000000 rip=000000003908d974 rsp=00000000b24ff548 rbp=00000000b14edf20
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
b24ff540 36b4cab0 00000000 b1df2000 3915a4d0 b1df2000 ntdll!ZwWaitForMultipleObjects+0x0
b24ff830 36b4c9ae af7355b0 004e0049 00000007 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0
b24ff870 acd211f4 b14f0980 00000000 00000000 00000000 KERNELBASE!WaitForMultipleObjects+0x0
b24ffab0 acdc7eed 00000000 b14edf20 00000000 00000000 nvcuda64!+0x0
b24ffb60 acd20bc3 5e009923 8b984bba b1cd2b60 00000000 nvcuda64!cuProfilerStop+0x0
b24ffb90 ad056294 b1cd2b60 00000000 00000000 00000000 nvcuda64!+0x0
b24ffbc0 389c7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
b24ffbf0 39042651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
b24ffc70 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 30988 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 12599414.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=00000000b1738c90 rdx=00000000b25ffcf8 rsi=00000000b25ffc60 rdi=0000000000000000
r8=00000000494398d1 r9=000000004943bf4b r10=00000000b173d9e0 r11=00000000f5a5859a r12=00000000b25ffcf8 r13=0000000000000000
r14=00000000b1738c90 r15=00000000b173d9e0 rip=0000000039090814 rsp=00000000b25ffc38 rbp=00000000b25ffc90
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
b25ffc30 39054295 00000102 00000102 00000000 b25ffc60 ntdll!NtWaitForAlertByThreadId+0x0
b25ffca0 36b6cd79 b173b1c0 00000000 00000000 00000000 ntdll!RtlSleepConditionVariableCS+0x0
b25ffcd0 acc8ef6c b173b1c0 390407b0 b173b1c0 ffb3b4c0 KERNELBASE!SleepConditionVariableCS+0x0
b25ffd30 acd20bc3 00000002 5ded5a16 00000000 00000000 nvcuda64!+0x0
b25ffd60 ad056294 b1cd3820 00000000 00000000 00000000 nvcuda64!+0x0
b25ffd90 389c7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
b25ffdc0 39042651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
b25ffe40 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 24344 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 1982031232.000000, User Time: 1341874944.000000, Wait Time: 12599433.000000
- Registers -
rax=0000000000001256 rbx=0000000000000001 rcx=00000000b26f7458 rdx=0000000000001e96 rsi=00000000deadbeef rdi=0000000000000000
r8=0000000000000504 r9=0000000059b94f70 r10=0000000000000008 r11=0000000080000000 r12=0000000000000400 r13=00000000b1f82630
r14=00000000bf37ac20 r15=0000000000000001 rip=0000000036f35b04 rsp=00000000b26f7408 rbp=00000000b26f7510
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
b26f7400 acf4a42b 00000000 00000000 00000000 00000000 win32u!NtGdiDdDDISubmitCommandToHwQueue+0x0
b26f76f0 acd9cace b147d420 b147d420 b1a30b58 00000000 nvcuda64!cuProfilerStop+0x0
b26f9dd0 acbc269d b26fa700 b147d420 b173b1c0 b1738c90 nvcuda64!cuProfilerStop+0x0
b26f9e40 acbc23ed 52541200 bf37ac20 bf37ac20 b19ce450 nvcuda64!+0x0
b26f9eb0 aceb1f4d 3eddb000 c73bc060 525412c0 b1a0f2b0 nvcuda64!+0x0
b26f9ef0 accd677f b173b1c0 c73bc044 b26f9f50 b173b1c0 nvcuda64!cuProfilerStop+0x0
b26fa520 ace89b42 00000000 00000000 00000000 00000000 nvcuda64!+0x0
b26fa640 acb8a75d 0e2ed2b0 b26fa7d0 0e2ed2b0 00000000 nvcuda64!cuProfilerStop+0x0
b26fa690 acb8ac52 00000003 525412c0 b173b1c0 00000006 nvcuda64!+0x0
b26fa960 acd7f477 00000000 00000000 00000000 b26faa70 nvcuda64!+0x0
b26faac0 58b329b6 00000000 00000006 00000100 58abe350 nvcuda64!cuEventRecordWithFlags+0x0
b26fabd0 58b2fdf4 58abe350 aff2ffd0 0e2ed2b0 b26fadf0 cudnn64_7!cudnnSetDropoutDescriptor+0x0
b26fad50 58ac04e1 58abe350 50226300 58abe350 000000c6 cudnn64_7!cudnnSetDropoutDescriptor+0x0
b26fae60 58ac5374 0000000c 00000080 4f880000 4f8a1000 cudnn64_7!cudnnRNNForwardTrainingEx+0x0
b26fb4a0 58abeafa 00000004 50226300 00000004 00000000 cudnn64_7!cudnnRNNForwardTrainingEx+0x0
b26fb660 58ad9738 50226300 00000400 52272300 3edde560 cudnn64_7!cudnnRNNForwardTrainingEx+0x0
b26fd870 7c42af5a b26fee60 4f880000 b26fee60 3edde560 cudnn64_7!cudnnRNNBackwardData+0x0
b26fde00 7c429c90 00000010 00000000 b26fe790 b26fe778 torch_cuda!at::native::_cudnn_rnn_backward+0x0
b26fdff0 7c482808 b26fe1e0 b26feec8 b26fe0d0 00000004 torch_cuda!at::native::_cudnn_rnn_backward+0x0
b26fe100 7c49103d b26fe790 b26fe770 b26feea8 00000000 torch_cuda!at::native::set_storage_cuda_+0x0
b26fe210 a3c4c201 b26fe790 a37cea63 3eef3fa0 b26fe8d8 torch_cuda!at::native::set_storage_cuda_+0x0
b26fe3a0 a3c9bfea b26fee60 00000090 00000007 a6cef880 torch_cpu!at::native::mkldnn_sigmoid_+0x0
b26fe4d0 a3c834da a8b24668 b26fe710 b26feec8 b26fe5b0 torch_cpu!at::bucketize_out+0x0
b26fe5c0 a4f7e0a9 b26fe710 b26fee60 b26fe700 00000000 torch_cpu!at::_cudnn_rnn_backward+0x0
b26fe8a0 a4f8a94d 3eef08a0 a53f794a 00000000 00000004 torch_cpu!torch::autograd::GraphRoot::apply+0x0
b26fe9b0 a3c4c201 b26feeb8 3eef22a0 00000000 b26fef90 torch_cpu!torch::autograd::GraphRoot::apply+0x0
b26feb40 a3c9bfea b26fee60 00000000 00000000 00000000 torch_cpu!at::native::mkldnn_sigmoid_+0x0
b26fec70 a3c834da a8b24668 b26fefe0 b26feec8 b26fed50 torch_cpu!at::bucketize_out+0x0
b26fed60 a4e8994d b26fefe0 13dcd790 b26feea0 526ad8f8 torch_cpu!at::_cudnn_rnn_backward+0x0
b26ff200 a4e756b1 4d19af80 b26ff508 00000000 04783bd6 torch_cpu!torch::autograd::generated::CudnnRnnBackward::apply+0x0
b26ff410 a53dd1ea b26ff520 4d19af80 b26ff520 4d19af80 torch_cpu!torch::autograd::Node::operator()+0x0
b26ff5f0 a53ddbdd 00000001 00000001 00000001 00000001 torch_cpu!torch::autograd::Engine::add_thread_pool_task+0x0
b26ffad0 a53e2812 00000000 00000000 4d19af70 ffffffff torch_cpu!torch::autograd::Engine::evaluate_function+0x0
b26ffdc0 a53e2471 a8906d10 b26ffe00 a8906d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0
b26ffe10 a53d9744 00080001 5180b790 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0
b26ffe40 36861bb2 5180b790 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0
b26ffe70 389c7034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0
b26ffea0 39042651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
b26fff20 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 34304 (state: Waiting): ***
- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 12557009.000000
- Registers -
rax=00000000000001d3 rbx=000000002b63fd80 rcx=0000000000000014 rdx=000000002b63fa00 rsi=00000000af720b50 rdi=000000002b63fd80
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000002 r15=0000000000000000 rip=0000000039090874 rsp=00000000b1fffba8 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
b1fffba0 39042dc7 af720af8 af720af8 af720b50 2b63fd80 ntdll!ZwWaitForWorkViaWorkerFactory+0x0
b1fffea0 389c7034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0
b1fffed0 39042651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
b1ffff50 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)