Task 5789244
| Name | ParityMachine-1624823822-21588-1_0 |
| Workunit | 3618194 |
| Created | 11 Jul 2021, 21:05:36 UTC |
| Sent | 16 Jul 2021, 18:44:58 UTC |
| Report deadline | 23 Jul 2021, 18:44:58 UTC |
| Received | 18 Jul 2021, 12:49:00 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 193 (0x000000C1) EXIT_SIGNAL |
| Computer ID | 13546 |
| Run time | 1 hours 0 min 57 sec |
| CPU time | 1 hours 0 min 49 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 15,412.34 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.80 (cuda10200) x86_64-pc-linux-gnu |
| Peak working set size | 2.33 GB |
| Peak swap size | 10.56 GB |
| Peak disk usage | 2.98 GB |
Stderr output
<core_client_version>7.14.2</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63)</message> <stderr_txt> main:581] : INFO : Epoch 1125 | loss: nan | val_loss: nan | Time: 3051.7 ms [2021-07-18 13:36:27 main:581] : INFO : Epoch 1126 | loss: nan | val_loss: nan | Time: 2666.86 ms [2021-07-18 13:36:30 main:581] : INFO : Epoch 1127 | loss: nan | val_loss: nan | Time: 2966.29 ms [2021-07-18 13:36:32 main:581] : INFO : Epoch 1128 | loss: nan | val_loss: nan | Time: 2037.42 ms [2021-07-18 13:36:34 main:581] : INFO : Epoch 1129 | loss: nan | val_loss: nan | Time: 2327.51 ms [2021-07-18 13:36:36 main:581] : INFO : Epoch 1130 | loss: nan | val_loss: nan | Time: 2145.86 ms [2021-07-18 13:36:39 main:581] : INFO : Epoch 1131 | loss: nan | val_loss: nan | Time: 3039.95 ms [2021-07-18 13:36:43 main:581] : INFO : Epoch 1132 | loss: nan | val_loss: nan | Time: 3347.65 ms [2021-07-18 13:36:45 main:581] : INFO : Epoch 1133 | loss: nan | val_loss: nan | Time: 2649.61 ms [2021-07-18 13:36:48 main:581] : INFO : Epoch 1134 | loss: nan | val_loss: nan | Time: 2326.99 ms [2021-07-18 13:36:50 main:581] : INFO : Epoch 1135 | loss: nan | val_loss: nan | Time: 2666.83 ms [2021-07-18 13:36:53 main:581] : INFO : Epoch 1136 | loss: nan | val_loss: nan | Time: 2398.79 ms [2021-07-18 13:36:55 main:581] : INFO : Epoch 1137 | loss: nan | val_loss: nan | Time: 1916.04 ms [2021-07-18 13:36:56 main:581] : INFO : Epoch 1138 | loss: nan | val_loss: nan | Time: 1704.52 ms [2021-07-18 13:36:58 main:581] : INFO : Epoch 1139 | loss: nan | val_loss: nan | Time: 1768.2 ms [2021-07-18 13:37:00 main:581] : INFO : Epoch 1140 | loss: nan | val_loss: nan | Time: 2058.31 ms [2021-07-18 13:37:02 main:581] : INFO : Epoch 1141 | loss: nan | val_loss: nan | Time: 2181.03 ms [2021-07-18 13:37:06 main:581] : INFO : Epoch 1142 | loss: nan | val_loss: nan | Time: 3196.56 ms [2021-07-18 13:37:08 main:581] : INFO : Epoch 1143 | loss: nan | val_loss: nan | Time: 2538.17 ms [2021-07-18 13:37:10 main:581] : INFO : Epoch 1144 | loss: nan | val_loss: nan | Time: 2095.06 ms [2021-07-18 13:37:12 main:581] : INFO : Epoch 1145 | loss: nan | val_loss: nan | Time: 2176.48 ms [2021-07-18 13:37:15 main:581] : INFO : Epoch 1146 | loss: nan | val_loss: nan | Time: 2404.63 ms [2021-07-18 13:37:17 main:581] : INFO : Epoch 1147 | loss: nan | val_loss: nan | Time: 2639.45 ms [2021-07-18 13:37:20 main:581] : INFO : Epoch 1148 | loss: nan | val_loss: nan | Time: 2841.73 ms [2021-07-18 13:37:23 main:581] : INFO : Epoch 1149 | loss: nan | val_loss: nan | Time: 2716.63 ms [2021-07-18 13:37:26 main:581] : INFO : Epoch 1150 | loss: nan | val_loss: nan | Time: 2580.92 ms [2021-07-18 13:37:27 main:581] : INFO : Epoch 1151 | loss: nan | val_loss: nan | Time: 1801.83 ms [2021-07-18 13:37:29 main:581] : INFO : Epoch 1152 | loss: nan | val_loss: nan | Time: 1941.13 ms [2021-07-18 13:37:31 main:581] : INFO : Epoch 1153 | loss: nan | val_loss: nan | Time: 1998.44 ms [2021-07-18 13:37:33 main:581] : INFO : Epoch 1154 | loss: nan | val_loss: nan | Time: 1856.2 ms [2021-07-18 13:37:35 main:581] : INFO : Epoch 1155 | loss: nan | val_loss: nan | Time: 1701.99 ms [2021-07-18 13:37:37 main:581] : INFO : Epoch 1156 | loss: nan | val_loss: nan | Time: 1802.04 ms [2021-07-18 13:37:38 main:581] : INFO : Epoch 1157 | loss: nan | val_loss: nan | Time: 1689.55 ms [2021-07-18 13:37:41 main:581] : INFO : Epoch 1158 | loss: nan | val_loss: nan | Time: 2482.77 ms [2021-07-18 13:37:44 main:581] : INFO : Epoch 1159 | loss: nan | val_loss: nan | Time: 3143.45 ms [2021-07-18 13:37:47 main:581] : INFO : Epoch 1160 | loss: nan | val_loss: nan | Time: 3509.5 ms [2021-07-18 13:37:51 main:581] : INFO : Epoch 1161 | loss: nan | val_loss: nan | Time: 3515.13 ms [2021-07-18 13:37:53 main:581] : INFO : Epoch 1162 | loss: nan | val_loss: nan | Time: 2491.84 ms [2021-07-18 13:37:56 main:581] : INFO : Epoch 1163 | loss: nan | val_loss: nan | Time: 2699.51 ms [2021-07-18 13:37:59 main:581] : INFO : Epoch 1164 | loss: nan | val_loss: nan | Time: 2382.2 ms [2021-07-18 13:38:01 main:581] : INFO : Epoch 1165 | loss: nan | val_loss: nan | Time: 2489.38 ms [2021-07-18 13:38:03 main:581] : INFO : Epoch 1166 | loss: nan | val_loss: nan | Time: 2242.75 ms [2021-07-18 13:38:06 main:581] : INFO : Epoch 1167 | loss: nan | val_loss: nan | Time: 2302.2 ms [2021-07-18 13:38:08 main:581] : INFO : Epoch 1168 | loss: nan | val_loss: nan | Time: 2253.34 ms [2021-07-18 13:38:11 main:581] : INFO : Epoch 1169 | loss: nan | val_loss: nan | Time: 3608.71 ms [2021-07-18 13:38:14 main:581] : INFO : Epoch 1170 | loss: nan | val_loss: nan | Time: 2901.75 ms [2021-07-18 13:38:17 main:581] : INFO : Epoch 1171 | loss: nan | val_loss: nan | Time: 2638.44 ms [2021-07-18 13:38:19 main:581] : INFO : Epoch 1172 | loss: nan | val_loss: nan | Time: 2509.35 ms [2021-07-18 13:38:22 main:581] : INFO : Epoch 1173 | loss: nan | val_loss: nan | Time: 2548.26 ms [2021-07-18 13:38:24 main:581] : INFO : Epoch 1174 | loss: nan | val_loss: nan | Time: 2305.82 ms [2021-07-18 13:38:27 main:581] : INFO : Epoch 1175 | loss: nan | val_loss: nan | Time: 2292.94 ms [2021-07-18 13:38:29 main:581] : INFO : Epoch 1176 | loss: nan | val_loss: nan | Time: 2422.51 ms [2021-07-18 13:38:31 main:581] : INFO : Epoch 1177 | loss: nan | val_loss: nan | Time: 2337.33 ms [2021-07-18 13:38:34 main:581] : INFO : Epoch 1178 | loss: nan | val_loss: nan | Time: 2361.39 ms [2021-07-18 13:38:36 main:581] : INFO : Epoch 1179 | loss: nan | val_loss: nan | Time: 2375.9 ms [2021-07-18 13:38:39 main:581] : INFO : Epoch 1180 | loss: nan | val_loss: nan | Time: 2968.94 ms [2021-07-18 13:38:42 main:581] : INFO : Epoch 1181 | loss: nan | val_loss: nan | Time: 2493.04 ms [2021-07-18 13:38:44 main:581] : INFO : Epoch 1182 | loss: nan | val_loss: nan | Time: 2348.67 ms [2021-07-18 13:38:46 main:581] : INFO : Epoch 1183 | loss: nan | val_loss: nan | Time: 1933.21 ms [2021-07-18 13:38:48 main:581] : INFO : Epoch 1184 | loss: nan | val_loss: nan | Time: 2290.66 ms [2021-07-18 13:38:50 main:581] : INFO : Epoch 1185 | loss: nan | val_loss: nan | Time: 2221.96 ms [2021-07-18 13:38:53 main:581] : INFO : Epoch 1186 | loss: nan | val_loss: nan | Time: 2491.96 ms [2021-07-18 13:38:55 main:581] : INFO : Epoch 1187 | loss: nan | val_loss: nan | Time: 2600.83 ms [2021-07-18 13:38:58 main:581] : INFO : Epoch 1188 | loss: nan | val_loss: nan | Time: 2548.53 ms [2021-07-18 13:39:01 main:581] : INFO : Epoch 1189 | loss: nan | val_loss: nan | Time: 2980.23 ms [2021-07-18 13:39:04 main:581] : INFO : Epoch 1190 | loss: nan | val_loss: nan | Time: 3088.72 ms [2021-07-18 13:39:07 main:581] : INFO : Epoch 1191 | loss: nan | val_loss: nan | Time: 3066.17 ms [2021-07-18 13:39:10 main:581] : INFO : Epoch 1192 | loss: nan | val_loss: nan | Time: 3027.32 ms [2021-07-18 13:39:13 main:581] : INFO : Epoch 1193 | loss: nan | val_loss: nan | Time: 2450.89 ms [2021-07-18 13:39:15 main:581] : INFO : Epoch 1194 | loss: nan | val_loss: nan | Time: 2314.11 ms [2021-07-18 13:39:18 main:581] : INFO : Epoch 1195 | loss: nan | val_loss: nan | Time: 2784.69 ms [2021-07-18 13:39:21 main:581] : INFO : Epoch 1196 | loss: nan | val_loss: nan | Time: 2984.29 ms [2021-07-18 13:39:23 main:581] : INFO : Epoch 1197 | loss: nan | val_loss: nan | Time: 2373.58 ms [2021-07-18 13:39:25 main:581] : INFO : Epoch 1198 | loss: nan | val_loss: nan | Time: 2167.65 ms [2021-07-18 13:39:27 main:581] : INFO : Epoch 1199 | loss: nan | val_loss: nan | Time: 1868.64 ms [2021-07-18 13:39:29 main:581] : INFO : Epoch 1200 | loss: nan | val_loss: nan | Time: 2257.82 ms [2021-07-18 13:39:32 main:581] : INFO : Epoch 1201 | loss: nan | val_loss: nan | Time: 3004.73 ms [2021-07-18 13:39:35 main:581] : INFO : Epoch 1202 | loss: nan | val_loss: nan | Time: 2630.79 ms [2021-07-18 13:39:37 main:581] : INFO : Epoch 1203 | loss: nan | val_loss: nan | Time: 2261.76 ms [2021-07-18 13:39:39 main:581] : INFO : Epoch 1204 | loss: nan | val_loss: nan | Time: 2139.19 ms [2021-07-18 13:39:42 main:581] : INFO : Epoch 1205 | loss: nan | val_loss: nan | Time: 2881.19 ms [2021-07-18 13:39:46 main:581] : INFO : Epoch 1206 | loss: nan | val_loss: nan | Time: 3949.63 ms [2021-07-18 13:39:49 main:581] : INFO : Epoch 1207 | loss: nan | val_loss: nan | Time: 3166.5 ms [2021-07-18 13:39:52 main:581] : INFO : Epoch 1208 | loss: nan | val_loss: nan | Time: 2811.33 ms [2021-07-18 13:39:55 main:581] : INFO : Epoch 1209 | loss: nan | val_loss: nan | Time: 2961.59 ms [2021-07-18 13:39:58 main:581] : INFO : Epoch 1210 | loss: nan | val_loss: nan | Time: 2395.25 ms [2021-07-18 13:40:00 main:581] : INFO : Epoch 1211 | loss: nan | val_loss: nan | Time: 2386.57 ms [2021-07-18 13:40:02 main:581] : INFO : Epoch 1212 | loss: nan | val_loss: nan | Time: 2432.91 ms [2021-07-18 13:40:05 main:581] : INFO : Epoch 1213 | loss: nan | val_loss: nan | Time: 2096.93 ms [2021-07-18 13:40:07 main:581] : INFO : Epoch 1214 | loss: nan | val_loss: nan | Time: 2634.68 ms [2021-07-18 13:40:10 main:581] : INFO : Epoch 1215 | loss: nan | val_loss: nan | Time: 2716.07 ms [2021-07-18 13:40:13 main:581] : INFO : Epoch 1216 | loss: nan | val_loss: nan | Time: 3546.16 ms [2021-07-18 13:40:16 main:581] : INFO : Epoch 1217 | loss: nan | val_loss: nan | Time: 3040.04 ms [2021-07-18 13:40:19 main:581] : INFO : Epoch 1218 | loss: nan | val_loss: nan | Time: 2455.04 ms [2021-07-18 13:40:22 main:581] : INFO : Epoch 1219 | loss: nan | val_loss: nan | Time: 2627.21 ms [2021-07-18 13:40:24 main:581] : INFO : Epoch 1220 | loss: nan | val_loss: nan | Time: 2362.18 ms [2021-07-18 13:40:26 main:581] : INFO : Epoch 1221 | loss: nan | val_loss: nan | Time: 2551.75 ms [2021-07-18 13:40:29 main:581] : INFO : Epoch 1222 | loss: nan | val_loss: nan | Time: 2417.34 ms [2021-07-18 13:40:32 main:581] : INFO : Epoch 1223 | loss: nan | val_loss: nan | Time: 2742.15 ms [2021-07-18 13:40:34 main:581] : INFO : Epoch 1224 | loss: nan | val_loss: nan | Time: 2735.26 ms [2021-07-18 13:40:37 main:581] : INFO : Epoch 1225 | loss: nan | val_loss: nan | Time: 2403.99 ms [2021-07-18 13:40:40 main:581] : INFO : Epoch 1226 | loss: nan | val_loss: nan | Time: 2859.01 ms [2021-07-18 13:40:42 main:581] : INFO : Epoch 1227 | loss: nan | val_loss: nan | Time: 2280.27 ms [2021-07-18 13:40:44 main:581] : INFO : Epoch 1228 | loss: nan | val_loss: nan | Time: 2415.58 ms [2021-07-18 13:40:47 main:581] : INFO : Epoch 1229 | loss: nan | val_loss: nan | Time: 2641.89 ms [2021-07-18 13:40:50 main:581] : INFO : Epoch 1230 | loss: nan | val_loss: nan | Time: 3047.62 ms [2021-07-18 13:40:52 main:581] : INFO : Epoch 1231 | loss: nan | val_loss: nan | Time: 2417.01 ms [2021-07-18 13:40:55 main:581] : INFO : Epoch 1232 | loss: nan | val_loss: nan | Time: 2631.13 ms [2021-07-18 13:40:58 main:581] : INFO : Epoch 1233 | loss: nan | val_loss: nan | Time: 2901.01 ms [2021-07-18 13:41:00 main:581] : INFO : Epoch 1234 | loss: nan | val_loss: nan | Time: 2250.08 ms [2021-07-18 13:41:02 main:581] : INFO : Epoch 1235 | loss: nan | val_loss: nan | Time: 2032.91 ms [2021-07-18 13:41:05 main:581] : INFO : Epoch 1236 | loss: nan | val_loss: nan | Time: 2614.78 ms [2021-07-18 13:41:07 main:581] : INFO : Epoch 1237 | loss: nan | val_loss: nan | Time: 2256.73 ms [2021-07-18 13:41:10 main:581] : INFO : Epoch 1238 | loss: nan | val_loss: nan | Time: 2410.75 ms [2021-07-18 13:41:12 main:581] : INFO : Epoch 1239 | loss: nan | val_loss: nan | Time: 2606.25 ms [2021-07-18 13:41:15 main:581] : INFO : Epoch 1240 | loss: nan | val_loss: nan | Time: 3038.37 ms [2021-07-18 13:41:18 main:581] : INFO : Epoch 1241 | loss: nan | val_loss: nan | Time: 2665.25 ms [2021-07-18 13:41:20 main:581] : INFO : Epoch 1242 | loss: nan | val_loss: nan | Time: 2555.23 ms [2021-07-18 13:41:23 main:581] : INFO : Epoch 1243 | loss: nan | val_loss: nan | Time: 2844.62 ms [2021-07-18 13:41:26 main:581] : INFO : Epoch 1244 | loss: nan | val_loss: nan | Time: 2765.89 ms [2021-07-18 13:41:29 main:581] : INFO : Epoch 1245 | loss: nan | val_loss: nan | Time: 2571.13 ms [2021-07-18 13:41:31 main:581] : INFO : Epoch 1246 | loss: nan | val_loss: nan | Time: 2539.25 ms [2021-07-18 13:41:33 main:581] : INFO : Epoch 1247 | loss: nan | val_loss: nan | Time: 1996.96 ms [2021-07-18 13:41:35 main:581] : INFO : Epoch 1248 | loss: nan | val_loss: nan | Time: 1786.64 ms [2021-07-18 13:41:37 main:581] : INFO : Epoch 1249 | loss: nan | val_loss: nan | Time: 2180.23 ms [2021-07-18 13:41:40 main:581] : INFO : Epoch 1250 | loss: nan | val_loss: nan | Time: 2545.85 ms [2021-07-18 13:41:42 main:581] : INFO : Epoch 1251 | loss: nan | val_loss: nan | Time: 2482.97 ms [2021-07-18 13:41:44 main:581] : INFO : Epoch 1252 | loss: nan | val_loss: nan | Time: 2362.98 ms [2021-07-18 13:41:47 main:581] : INFO : Epoch 1253 | loss: nan | val_loss: nan | Time: 2089.33 ms [2021-07-18 13:41:49 main:581] : INFO : Epoch 1254 | loss: nan | val_loss: nan | Time: 2467.7 ms [2021-07-18 13:41:52 main:581] : INFO : Epoch 1255 | loss: nan | val_loss: nan | Time: 2497.03 ms [2021-07-18 13:41:54 main:581] : INFO : Epoch 1256 | loss: nan | val_loss: nan | Time: 2513.03 ms [2021-07-18 13:41:57 main:581] : INFO : Epoch 1257 | loss: nan | val_loss: nan | Time: 2443.41 ms [2021-07-18 13:41:59 main:581] : INFO : Epoch 1258 | loss: nan | val_loss: nan | Time: 2443.61 ms [2021-07-18 13:42:01 main:581] : INFO : Epoch 1259 | loss: nan | val_loss: nan | Time: 2153.98 ms [2021-07-18 13:42:03 main:581] : INFO : Epoch 1260 | loss: nan | val_loss: nan | Time: 2341.03 ms [2021-07-18 13:42:06 main:581] : INFO : Epoch 1261 | loss: nan | val_loss: nan | Time: 2476.89 ms [2021-07-18 13:42:08 main:581] : INFO : Epoch 1262 | loss: nan | val_loss: nan | Time: 2517.82 ms [2021-07-18 13:42:11 main:581] : INFO : Epoch 1263 | loss: nan | val_loss: nan | Time: 2572.38 ms [2021-07-18 13:42:13 main:581] : INFO : Epoch 1264 | loss: nan | val_loss: nan | Time: 2440.5 ms [2021-07-18 13:42:16 main:581] : INFO : Epoch 1265 | loss: nan | val_loss: nan | Time: 2376 ms [2021-07-18 13:42:18 main:581] : INFO : Epoch 1266 | loss: nan | val_loss: nan | Time: 2485.85 ms [2021-07-18 13:42:21 main:581] : INFO : Epoch 1267 | loss: nan | val_loss: nan | Time: 2486.27 ms [2021-07-18 13:42:23 main:581] : INFO : Epoch 1268 | loss: nan | val_loss: nan | Time: 2372.68 ms [2021-07-18 13:42:25 main:581] : INFO : Epoch 1269 | loss: nan | val_loss: nan | Time: 2266.14 ms [2021-07-18 13:42:28 main:581] : INFO : Epoch 1270 | loss: nan | val_loss: nan | Time: 2246.45 ms [2021-07-18 13:42:30 main:581] : INFO : Epoch 1271 | loss: nan | val_loss: nan | Time: 2398.32 ms [2021-07-18 13:42:33 main:581] : INFO : Epoch 1272 | loss: nan | val_loss: nan | Time: 2449.85 ms [2021-07-18 13:42:35 main:581] : INFO : Epoch 1273 | loss: nan | val_loss: nan | Time: 2610.53 ms [2021-07-18 13:42:38 main:581] : INFO : Epoch 1274 | loss: nan | val_loss: nan | Time: 3054.2 ms [2021-07-18 13:42:41 main:581] : INFO : Epoch 1275 | loss: nan | val_loss: nan | Time: 2532.54 ms [2021-07-18 13:42:44 main:581] : INFO : Epoch 1276 | loss: nan | val_loss: nan | Time: 3011.03 ms [2021-07-18 13:42:47 main:581] : INFO : Epoch 1277 | loss: nan | val_loss: nan | Time: 3603.73 ms [2021-07-18 13:42:51 main:581] : INFO : Epoch 1278 | loss: nan | val_loss: nan | Time: 3595.36 ms [2021-07-18 13:42:54 main:581] : INFO : Epoch 1279 | loss: nan | val_loss: nan | Time: 3168.33 ms [2021-07-18 13:42:56 main:581] : INFO : Epoch 1280 | loss: nan | val_loss: nan | Time: 2340.56 ms [2021-07-18 13:42:59 main:581] : INFO : Epoch 1281 | loss: nan | val_loss: nan | Time: 2455.08 ms [2021-07-18 13:43:02 main:581] : INFO : Epoch 1282 | loss: nan | val_loss: nan | Time: 2632.34 ms [2021-07-18 13:43:05 main:581] : INFO : Epoch 1283 | loss: nan | val_loss: nan | Time: 3111.45 ms [2021-07-18 13:43:07 main:581] : INFO : Epoch 1284 | loss: nan | val_loss: nan | Time: 2719.06 ms [2021-07-18 13:43:10 main:581] : INFO : Epoch 1285 | loss: nan | val_loss: nan | Time: 2732.33 ms [2021-07-18 13:43:14 main:581] : INFO : Epoch 1286 | loss: nan | val_loss: nan | Time: 3498.9 ms [2021-07-18 13:43:16 main:581] : INFO : Epoch 1287 | loss: nan | val_loss: nan | Time: 2777.82 ms [2021-07-18 13:43:20 main:581] : INFO : Epoch 1288 | loss: nan | val_loss: nan | Time: 3392.62 ms [2021-07-18 13:43:23 main:581] : INFO : Epoch 1289 | loss: nan | val_loss: nan | Time: 2847.64 ms [2021-07-18 13:43:25 main:581] : INFO : Epoch 1290 | loss: nan | val_loss: nan | Time: 2533.74 ms [2021-07-18 13:43:28 main:581] : INFO : Epoch 1291 | loss: nan | val_loss: nan | Time: 2550.93 ms [2021-07-18 13:43:30 main:581] : INFO : Epoch 1292 | loss: nan | val_loss: nan | Time: 2738.2 ms [2021-07-18 13:43:33 main:581] : INFO : Epoch 1293 | loss: nan | val_loss: nan | Time: 2700.63 ms [2021-07-18 13:43:36 main:581] : INFO : Epoch 1294 | loss: nan | val_loss: nan | Time: 2522.13 ms [2021-07-18 13:43:38 main:581] : INFO : Epoch 1295 | loss: nan | val_loss: nan | Time: 2485.17 ms [2021-07-18 13:43:42 main:581] : INFO : Epoch 1296 | loss: nan | val_loss: nan | Time: 3622.58 ms [2021-07-18 13:43:46 main:581] : INFO : Epoch 1297 | loss: nan | val_loss: nan | Time: 4257.53 ms [2021-07-18 13:43:50 main:581] : INFO : Epoch 1298 | loss: nan | val_loss: nan | Time: 3482.1 ms [2021-07-18 13:43:52 main:581] : INFO : Epoch 1299 | loss: nan | val_loss: nan | Time: 2341.33 ms [2021-07-18 13:43:54 main:581] : INFO : Epoch 1300 | loss: nan | val_loss: nan | Time: 2506.39 ms [2021-07-18 13:43:57 main:581] : INFO : Epoch 1301 | loss: nan | val_loss: nan | Time: 2858.86 ms [2021-07-18 13:44:01 main:581] : INFO : Epoch 1302 | loss: nan | val_loss: nan | Time: 3667.05 ms [2021-07-18 13:44:04 main:581] : INFO : Epoch 1303 | loss: nan | val_loss: nan | Time: 2631.42 ms [2021-07-18 13:44:06 main:581] : INFO : Epoch 1304 | loss: nan | val_loss: nan | Time: 2566.24 ms [2021-07-18 13:44:08 main:581] : INFO : Epoch 1305 | loss: nan | val_loss: nan | Time: 2362.75 ms [2021-07-18 13:44:12 main:581] : INFO : Epoch 1306 | loss: nan | val_loss: nan | Time: 3187.41 ms [2021-07-18 13:44:15 main:581] : INFO : Epoch 1307 | loss: nan | val_loss: nan | Time: 3228.29 ms [2021-07-18 13:44:18 main:581] : INFO : Epoch 1308 | loss: nan | val_loss: nan | Time: 3133.12 ms [2021-07-18 13:44:21 main:581] : INFO : Epoch 1309 | loss: nan | val_loss: nan | Time: 2658.72 ms [2021-07-18 13:44:23 main:581] : INFO : Epoch 1310 | loss: nan | val_loss: nan | Time: 2142.68 ms [2021-07-18 13:44:26 main:581] : INFO : Epoch 1311 | loss: nan | val_loss: nan | Time: 3061.67 ms [2021-07-18 13:44:29 main:581] : INFO : Epoch 1312 | loss: nan | val_loss: nan | Time: 3198.91 ms [2021-07-18 13:44:32 main:581] : INFO : Epoch 1313 | loss: nan | val_loss: nan | Time: 2527.94 ms [2021-07-18 13:44:35 main:581] : INFO : Epoch 1314 | loss: nan | val_loss: nan | Time: 3693.16 ms [2021-07-18 13:44:39 main:581] : INFO : Epoch 1315 | loss: nan | val_loss: nan | Time: 3828.72 ms [2021-07-18 13:44:42 main:581] : INFO : Epoch 1316 | loss: nan | val_loss: nan | Time: 3235.13 ms [2021-07-18 13:44:45 main:581] : INFO : Epoch 1317 | loss: nan | val_loss: nan | Time: 2319.57 ms [2021-07-18 13:44:47 main:581] : INFO : Epoch 1318 | loss: nan | val_loss: nan | Time: 1835.84 ms [2021-07-18 13:44:48 main:581] : INFO : Epoch 1319 | loss: nan | val_loss: nan | Time: 1776.97 ms [2021-07-18 13:44:50 main:581] : INFO : Epoch 1320 | loss: nan | val_loss: nan | Time: 1705.4 ms [2021-07-18 13:44:52 main:581] : INFO : Epoch 1321 | loss: nan | val_loss: nan | Time: 2155.79 ms [2021-07-18 13:44:54 main:581] : INFO : Epoch 1322 | loss: nan | val_loss: nan | Time: 2297.11 ms [2021-07-18 13:44:57 main:581] : INFO : Epoch 1323 | loss: nan | val_loss: nan | Time: 2657.41 ms [2021-07-18 13:44:59 main:581] : INFO : Epoch 1324 | loss: nan | val_loss: nan | Time: 2363.76 ms [2021-07-18 13:45:02 main:581] : INFO : Epoch 1325 | loss: nan | val_loss: nan | Time: 2347.51 ms [2021-07-18 13:45:04 main:581] : INFO : Epoch 1326 | loss: nan | val_loss: nan | Time: 2376.39 ms [2021-07-18 13:45:07 main:581] : INFO : Epoch 1327 | loss: nan | val_loss: nan | Time: 2321.88 ms [2021-07-18 13:45:09 main:581] : INFO : Epoch 1328 | loss: nan | val_loss: nan | Time: 2175.85 ms [2021-07-18 13:45:11 main:581] : INFO : Epoch 1329 | loss: nan | val_loss: nan | Time: 2268.5 ms [2021-07-18 13:45:13 main:581] : INFO : Epoch 1330 | loss: nan | val_loss: nan | Time: 2385.89 ms [2021-07-18 13:45:16 main:581] : INFO : Epoch 1331 | loss: nan | val_loss: nan | Time: 2206 ms [2021-07-18 13:45:18 main:581] : INFO : Epoch 1332 | loss: nan | val_loss: nan | Time: 2325.41 ms [2021-07-18 13:45:20 main:581] : INFO : Epoch 1333 | loss: nan | val_loss: nan | Time: 2230.48 ms [2021-07-18 13:45:22 main:581] : INFO : Epoch 1334 | loss: nan | val_loss: nan | Time: 2083.44 ms [2021-07-18 13:45:25 main:581] : INFO : Epoch 1335 | loss: nan | val_loss: nan | Time: 2721.58 ms [2021-07-18 13:45:28 main:581] : INFO : Epoch 1336 | loss: nan | val_loss: nan | Time: 3420.67 ms [2021-07-18 13:45:32 main:581] : INFO : Epoch 1337 | loss: nan | val_loss: nan | Time: 3189.13 ms [2021-07-18 13:45:35 main:581] : INFO : Epoch 1338 | loss: nan | val_loss: nan | Time: 3208.81 ms [2021-07-18 13:45:38 main:581] : INFO : Epoch 1339 | loss: nan | val_loss: nan | Time: 3470.66 ms [2021-07-18 13:45:41 main:581] : INFO : Epoch 1340 | loss: nan | val_loss: nan | Time: 2422.73 ms [2021-07-18 13:45:43 main:581] : INFO : Epoch 1341 | loss: nan | val_loss: nan | Time: 2717.79 ms [2021-07-18 13:45:46 main:581] : INFO : Epoch 1342 | loss: nan | val_loss: nan | Time: 2603.78 ms [2021-07-18 13:45:48 main:581] : INFO : Epoch 1343 | loss: nan | val_loss: nan | Time: 2442.06 ms [2021-07-18 13:45:51 main:581] : INFO : Epoch 1344 | loss: nan | val_loss: nan | Time: 2388.03 ms [2021-07-18 13:45:54 main:581] : INFO : Epoch 1345 | loss: nan | val_loss: nan | Time: 2953.95 ms [2021-07-18 13:45:57 main:581] : INFO : Epoch 1346 | loss: nan | val_loss: nan | Time: 3549.2 ms [2021-07-18 13:46:00 main:581] : INFO : Epoch 1347 | loss: nan | val_loss: nan | Time: 2856.19 ms [2021-07-18 13:46:04 main:581] : INFO : Epoch 1348 | loss: nan | val_loss: nan | Time: 4081.94 ms [2021-07-18 13:46:08 main:581] : INFO : Epoch 1349 | loss: nan | val_loss: nan | Time: 3728.89 ms [2021-07-18 13:46:11 main:581] : INFO : Epoch 1350 | loss: nan | val_loss: nan | Time: 2893.43 ms [2021-07-18 13:46:13 main:581] : INFO : Epoch 1351 | loss: nan | val_loss: nan | Time: 2580.97 ms [2021-07-18 13:46:16 main:581] : INFO : Epoch 1352 | loss: nan | val_loss: nan | Time: 2401.49 ms [2021-07-18 13:46:18 main:581] : INFO : Epoch 1353 | loss: nan | val_loss: nan | Time: 2279.43 ms [2021-07-18 13:46:21 main:581] : INFO : Epoch 1354 | loss: nan | val_loss: nan | Time: 2559.38 ms [2021-07-18 13:46:23 main:581] : INFO : Epoch 1355 | loss: nan | val_loss: nan | Time: 2326.65 ms [2021-07-18 13:46:26 main:581] : INFO : Epoch 1356 | loss: nan | val_loss: nan | Time: 3388.86 ms [2021-07-18 13:46:31 main:581] : INFO : Epoch 1357 | loss: nan | val_loss: nan | Time: 4095.21 ms [2021-07-18 13:46:33 main:581] : INFO : Epoch 1358 | loss: nan | val_loss: nan | Time: 2733.56 ms [2021-07-18 13:46:36 main:581] : INFO : Epoch 1359 | loss: nan | val_loss: nan | Time: 2488.65 ms [2021-07-18 13:46:39 main:581] : INFO : Epoch 1360 | loss: nan | val_loss: nan | Time: 2814.38 ms [2021-07-18 13:46:41 main:581] : INFO : Epoch 1361 | loss: nan | val_loss: nan | Time: 2809.22 ms [2021-07-18 13:46:44 main:581] : INFO : Epoch 1362 | loss: nan | val_loss: nan | Time: 2868.39 ms [2021-07-18 13:46:47 main:581] : INFO : Epoch 1363 | loss: nan | val_loss: nan | Time: 2449.89 ms [2021-07-18 13:46:49 main:581] : INFO : Epoch 1364 | loss: nan | val_loss: nan | Time: 2674.8 ms [2021-07-18 13:46:52 main:581] : INFO : Epoch 1365 | loss: nan | val_loss: nan | Time: 3004.98 ms [2021-07-18 13:46:55 main:581] : INFO : Epoch 1366 | loss: nan | val_loss: nan | Time: 2593.8 ms [2021-07-18 13:46:57 main:581] : INFO : Epoch 1367 | loss: nan | val_loss: nan | Time: 2136.07 ms [2021-07-18 13:47:00 main:581] : INFO : Epoch 1368 | loss: nan | val_loss: nan | Time: 2818.72 ms [2021-07-18 13:47:03 main:581] : INFO : Epoch 1369 | loss: nan | val_loss: nan | Time: 3400.24 ms [2021-07-18 13:47:06 main:581] : INFO : Epoch 1370 | loss: nan | val_loss: nan | Time: 2740.53 ms [2021-07-18 13:47:09 main:581] : INFO : Epoch 1371 | loss: nan | val_loss: nan | Time: 2564.94 ms [2021-07-18 13:47:11 main:581] : INFO : Epoch 1372 | loss: nan | val_loss: nan | Time: 2202.84 ms [2021-07-18 13:47:13 main:581] : INFO : Epoch 1373 | loss: nan | val_loss: nan | Time: 2350.15 ms [2021-07-18 13:47:16 main:581] : INFO : Epoch 1374 | loss: nan | val_loss: nan | Time: 2357.99 ms [2021-07-18 13:47:18 main:581] : INFO : Epoch 1375 | loss: nan | val_loss: nan | Time: 2366.73 ms [2021-07-18 13:47:20 main:581] : INFO : Epoch 1376 | loss: nan | val_loss: nan | Time: 2393.1 ms [2021-07-18 13:47:23 main:581] : INFO : Epoch 1377 | loss: nan | val_loss: nan | Time: 2323.98 ms [2021-07-18 13:47:25 main:581] : INFO : Epoch 1378 | loss: nan | val_loss: nan | Time: 2361.2 ms [2021-07-18 13:47:27 main:581] : INFO : Epoch 1379 | loss: nan | val_loss: nan | Time: 2226.5 ms [2021-07-18 13:47:30 main:581] : INFO : Epoch 1380 | loss: nan | val_loss: nan | Time: 2449.33 ms [2021-07-18 13:47:32 main:581] : INFO : Epoch 1381 | loss: nan | val_loss: nan | Time: 2453.49 ms [2021-07-18 13:47:35 main:581] : INFO : Epoch 1382 | loss: nan | val_loss: nan | Time: 2373.59 ms [2021-07-18 13:47:37 main:581] : INFO : Epoch 1383 | loss: nan | val_loss: nan | Time: 2262.81 ms [2021-07-18 13:47:39 main:581] : INFO : Epoch 1384 | loss: nan | val_loss: nan | Time: 2333.7 ms [2021-07-18 13:47:41 main:581] : INFO : Epoch 1385 | loss: nan | val_loss: nan | Time: 2392.98 ms [2021-07-18 13:47:44 main:581] : INFO : Epoch 1386 | loss: nan | val_loss: nan | Time: 2323.43 ms [2021-07-18 13:47:46 main:581] : INFO : Epoch 1387 | loss: nan | val_loss: nan | Time: 2352.84 ms [2021-07-18 13:47:49 main:581] : INFO : Epoch 1388 | loss: nan | val_loss: nan | Time: 2386.47 ms [2021-07-18 13:47:51 main:581] : INFO : Epoch 1389 | loss: nan | val_loss: nan | Time: 2413.63 ms [2021-07-18 13:47:53 main:581] : INFO : Epoch 1390 | loss: nan | val_loss: nan | Time: 2404.45 ms [2021-07-18 13:47:56 main:581] : INFO : Epoch 1391 | loss: nan | val_loss: nan | Time: 2228.54 ms [2021-07-18 13:47:58 main:581] : INFO : Epoch 1392 | loss: nan | val_loss: nan | Time: 2532.94 ms [2021-07-18 13:48:01 main:581] : INFO : Epoch 1393 | loss: nan | val_loss: nan | Time: 2368.71 ms [2021-07-18 13:48:03 main:581] : INFO : Epoch 1394 | loss: nan | val_loss: nan | Time: 2666.97 ms [2021-07-18 13:48:06 main:581] : INFO : Epoch 1395 | loss: nan | val_loss: nan | Time: 2922.09 ms [2021-07-18 13:48:08 main:581] : INFO : Epoch 1396 | loss: nan | val_loss: nan | Time: 1986.55 ms [2021-07-18 13:48:10 main:581] : INFO : Epoch 1397 | loss: nan | val_loss: nan | Time: 2273 ms [2021-07-18 13:48:14 main:581] : INFO : Epoch 1398 | loss: nan | val_loss: nan | Time: 3773.76 ms [2021-07-18 13:48:17 main:581] : INFO : Epoch 1399 | loss: nan | val_loss: nan | Time: 2863.11 ms [2021-07-18 13:48:19 main:581] : INFO : Epoch 1400 | loss: nan | val_loss: nan | Time: 2335.7 ms [2021-07-18 13:48:22 main:581] : INFO : Epoch 1401 | loss: nan | val_loss: nan | Time: 2412.78 ms [2021-07-18 13:48:24 main:581] : INFO : Epoch 1402 | loss: nan | val_loss: nan | Time: 2604.75 ms [2021-07-18 13:48:28 main:581] : INFO : Epoch 1403 | loss: nan | val_loss: nan | Time: 3242.9 ms [2021-07-18 13:48:31 main:581] : INFO : Epoch 1404 | loss: nan | val_loss: nan | Time: 3208.94 ms [2021-07-18 13:48:34 main:581] : INFO : Epoch 1405 | loss: nan | val_loss: nan | Time: 3077.49 ms [2021-07-18 13:48:37 main:581] : INFO : Epoch 1406 | loss: nan | val_loss: nan | Time: 2914.56 ms [2021-07-18 13:48:41 main:581] : INFO : Epoch 1407 | loss: nan | val_loss: nan | Time: 3831.09 ms [2021-07-18 13:48:43 main:581] : INFO : Epoch 1408 | loss: nan | val_loss: nan | Time: 2660.15 ms [2021-07-18 13:48:46 main:581] : INFO : Epoch 1409 | loss: nan | val_loss: nan | Time: 3020.65 ms [2021-07-18 13:48:49 main:581] : INFO : Epoch 1410 | loss: nan | val_loss: nan | Time: 3085.32 ms [2021-07-18 13:48:53 main:581] : INFO : Epoch 1411 | loss: nan | val_loss: nan | Time: 3116.05 ms [2021-07-18 13:48:55 main:581] : INFO : Epoch 1412 | loss: nan | val_loss: nan | Time: 2911.58 ms [2021-07-18 13:48:58 main:581] : INFO : Epoch 1413 | loss: nan | val_loss: nan | Time: 2959.89 ms [2021-07-18 13:49:02 main:581] : INFO : Epoch 1414 | loss: nan | val_loss: nan | Time: 3227.51 ms [2021-07-18 13:49:05 main:581] : INFO : Epoch 1415 | loss: nan | val_loss: nan | Time: 3253.33 ms [2021-07-18 13:49:08 main:581] : INFO : Epoch 1416 | loss: nan | val_loss: nan | Time: 3419.75 ms [2021-07-18 13:49:11 main:581] : INFO : Epoch 1417 | loss: nan | val_loss: nan | Time: 2882.43 ms [2021-07-18 13:49:14 main:581] : INFO : Epoch 1418 | loss: nan | val_loss: nan | Time: 3046.53 ms [2021-07-18 13:49:17 main:581] : INFO : Epoch 1419 | loss: nan | val_loss: nan | Time: 2933.5 ms [2021-07-18 13:49:21 main:581] : INFO : Epoch 1420 | loss: nan | val_loss: nan | Time: 3553.45 ms [2021-07-18 13:49:24 main:581] : INFO : Epoch 1421 | loss: nan | val_loss: nan | Time: 3452.09 ms [2021-07-18 13:49:27 main:581] : INFO : Epoch 1422 | loss: nan | val_loss: nan | Time: 3137.09 ms [2021-07-18 13:49:30 main:581] : INFO : Epoch 1423 | loss: nan | val_loss: nan | Time: 3025.39 ms [2021-07-18 13:49:34 main:581] : INFO : Epoch 1424 | loss: nan | val_loss: nan | Time: 3289.79 ms [2021-07-18 13:49:36 main:581] : INFO : Epoch 1425 | loss: nan | val_loss: nan | Time: 2881.9 ms [2021-07-18 13:49:39 main:581] : INFO : Epoch 1426 | loss: nan | val_loss: nan | Time: 2881.4 ms [2021-07-18 13:49:42 main:581] : INFO : Epoch 1427 | loss: nan | val_loss: nan | Time: 2824.32 ms [2021-07-18 13:49:46 main:581] : INFO : Epoch 1428 | loss: nan | val_loss: nan | Time: 3602.58 ms [2021-07-18 13:49:50 main:581] : INFO : Epoch 1429 | loss: nan | val_loss: nan | Time: 3894.7 ms [2021-07-18 13:49:52 main:581] : INFO : Epoch 1430 | loss: nan | val_loss: nan | Time: 2160.53 ms [2021-07-18 13:49:54 main:581] : INFO : Epoch 1431 | loss: nan | val_loss: nan | Time: 1835.41 ms [2021-07-18 13:49:56 main:581] : INFO : Epoch 1432 | loss: nan | val_loss: nan | Time: 1954.29 ms [2021-07-18 13:49:58 main:581] : INFO : Epoch 1433 | loss: nan | val_loss: nan | Time: 1929.54 ms [2021-07-18 13:50:00 main:581] : INFO : Epoch 1434 | loss: nan | val_loss: nan | Time: 2019.91 ms [2021-07-18 13:50:01 main:581] : INFO : Epoch 1435 | loss: nan | val_loss: nan | Time: 1888.27 ms [2021-07-18 13:50:04 main:581] : INFO : Epoch 1436 | loss: nan | val_loss: nan | Time: 2771.41 ms [2021-07-18 13:50:07 main:581] : INFO : Epoch 1437 | loss: nan | val_loss: nan | Time: 2739.24 ms [2021-07-18 13:50:10 main:581] : INFO : Epoch 1438 | loss: nan | val_loss: nan | Time: 2508.78 ms [2021-07-18 13:50:12 main:581] : INFO : Epoch 1439 | loss: nan | val_loss: nan | Time: 2896.93 ms [2021-07-18 13:50:15 main:581] : INFO : Epoch 1440 | loss: nan | val_loss: nan | Time: 2939.75 ms [2021-07-18 13:50:18 main:581] : INFO : Epoch 1441 | loss: nan | val_loss: nan | Time: 2496.26 ms [2021-07-18 13:50:20 main:581] : INFO : Epoch 1442 | loss: nan | val_loss: nan | Time: 2415.75 ms [2021-07-18 13:50:24 main:581] : INFO : Epoch 1443 | loss: nan | val_loss: nan | Time: 3247.95 ms [2021-07-18 13:50:27 main:581] : INFO : Epoch 1444 | loss: nan | val_loss: nan | Time: 3100.89 ms [2021-07-18 13:50:29 main:581] : INFO : Epoch 1445 | loss: nan | val_loss: nan | Time: 2412.11 ms [2021-07-18 13:50:31 main:581] : INFO : Epoch 1446 | loss: nan | val_loss: nan | Time: 1966.15 ms [2021-07-18 13:50:34 main:581] : INFO : Epoch 1447 | loss: nan | val_loss: nan | Time: 2924.38 ms [2021-07-18 13:50:36 main:581] : INFO : Epoch 1448 | loss: nan | val_loss: nan | Time: 2107.2 ms [2021-07-18 13:50:38 main:581] : INFO : Epoch 1449 | loss: nan | val_loss: nan | Time: 1988.22 ms [2021-07-18 13:50:40 main:581] : INFO : Epoch 1450 | loss: nan | val_loss: nan | Time: 1897.7 ms [2021-07-18 13:50:42 main:581] : INFO : Epoch 1451 | loss: nan | val_loss: nan | Time: 1864.04 ms [2021-07-18 13:50:44 main:581] : INFO : Epoch 1452 | loss: nan | val_loss: nan | Time: 1911.47 ms [2021-07-18 13:50:46 main:581] : INFO : Epoch 1453 | loss: nan | val_loss: nan | Time: 2043.02 ms [2021-07-18 13:50:48 main:581] : INFO : Epoch 1454 | loss: nan | val_loss: nan | Time: 2079.25 ms [2021-07-18 13:50:50 main:581] : INFO : Epoch 1455 | loss: nan | val_loss: nan | Time: 2065.17 ms [2021-07-18 13:50:52 main:581] : INFO : Epoch 1456 | loss: nan | val_loss: nan | Time: 2026.63 ms [2021-07-18 13:50:54 main:581] : INFO : Epoch 1457 | loss: nan | val_loss: nan | Time: 2081.24 ms [2021-07-18 13:50:56 main:581] : INFO : Epoch 1458 | loss: nan | val_loss: nan | Time: 2211.09 ms [2021-07-18 13:50:59 main:581] : INFO : Epoch 1459 | loss: nan | val_loss: nan | Time: 2879.92 ms [2021-07-18 13:51:01 main:581] : INFO : Epoch 1460 | loss: nan | val_loss: nan | Time: 1914.34 ms [2021-07-18 13:51:03 main:581] : INFO : Epoch 1461 | loss: nan | val_loss: nan | Time: 1986.88 ms [2021-07-18 13:51:06 main:581] : INFO : Epoch 1462 | loss: nan | val_loss: nan | Time: 3269.89 ms [2021-07-18 13:51:09 main:581] : INFO : Epoch 1463 | loss: nan | val_loss: nan | Time: 2486.24 ms [2021-07-18 13:51:11 main:581] : INFO : Epoch 1464 | loss: nan | val_loss: nan | Time: 2167.69 ms [2021-07-18 13:51:13 main:581] : INFO : Epoch 1465 | loss: nan | val_loss: nan | Time: 1857.19 ms [2021-07-18 13:51:15 main:581] : INFO : Epoch 1466 | loss: nan | val_loss: nan | Time: 1854.77 ms [2021-07-18 13:51:16 main:581] : INFO : Epoch 1467 | loss: nan | val_loss: nan | Time: 1872.39 ms [2021-07-18 13:51:19 main:581] : INFO : Epoch 1468 | loss: nan | val_loss: nan | Time: 2010.88 ms [2021-07-18 13:51:20 main:581] : INFO : Epoch 1469 | loss: nan | val_loss: nan | Time: 1988.85 ms [2021-07-18 13:51:23 main:581] : INFO : Epoch 1470 | loss: nan | val_loss: nan | Time: 2058.63 ms [2021-07-18 13:51:25 main:581] : INFO : Epoch 1471 | loss: nan | val_loss: nan | Time: 2093.76 ms [2021-07-18 13:51:27 main:581] : INFO : Epoch 1472 | loss: nan | val_loss: nan | Time: 2395.11 ms [2021-07-18 13:51:29 main:581] : INFO : Epoch 1473 | loss: nan | val_loss: nan | Time: 2361.8 ms [2021-07-18 13:51:32 main:581] : INFO : Epoch 1474 | loss: nan | val_loss: nan | Time: 2286.46 ms [2021-07-18 13:51:34 main:581] : INFO : Epoch 1475 | loss: nan | val_loss: nan | Time: 2177.49 ms [2021-07-18 13:51:36 main:581] : INFO : Epoch 1476 | loss: nan | val_loss: nan | Time: 2340.48 ms [2021-07-18 13:51:38 main:581] : INFO : Epoch 1477 | loss: nan | val_loss: nan | Time: 2251.41 ms [2021-07-18 13:51:41 main:581] : INFO : Epoch 1478 | loss: nan | val_loss: nan | Time: 2250.75 ms [2021-07-18 13:51:43 main:581] : INFO : Epoch 1479 | loss: nan | val_loss: nan | Time: 2266.98 ms [2021-07-18 13:51:45 main:581] : INFO : Epoch 1480 | loss: nan | val_loss: nan | Time: 2301.69 ms [2021-07-18 13:51:48 main:581] : INFO : Epoch 1481 | loss: nan | val_loss: nan | Time: 2398.77 ms [2021-07-18 13:51:50 main:581] : INFO : Epoch 1482 | loss: nan | val_loss: nan | Time: 2322.46 ms [2021-07-18 13:51:52 main:581] : INFO : Epoch 1483 | loss: nan | val_loss: nan | Time: 1980.85 ms [2021-07-18 13:51:54 main:581] : INFO : Epoch 1484 | loss: nan | val_loss: nan | Time: 2372.15 ms [2021-07-18 13:51:57 main:581] : INFO : Epoch 1485 | loss: nan | val_loss: nan | Time: 2847.7 ms [2021-07-18 13:52:01 main:581] : INFO : Epoch 1486 | loss: nan | val_loss: nan | Time: 3396.62 ms [2021-07-18 13:52:03 main:581] : INFO : Epoch 1487 | loss: nan | val_loss: nan | Time: 2397.84 ms [2021-07-18 13:52:06 main:581] : INFO : Epoch 1488 | loss: nan | val_loss: nan | Time: 2905.51 ms [2021-07-18 13:52:09 main:581] : INFO : Epoch 1489 | loss: nan | val_loss: nan | Time: 3032.23 ms [2021-07-18 13:52:11 main:581] : INFO : Epoch 1490 | loss: nan | val_loss: nan | Time: 2332.94 ms [2021-07-18 13:52:14 main:581] : INFO : Epoch 1491 | loss: nan | val_loss: nan | Time: 2400.56 ms [2021-07-18 13:52:17 main:581] : INFO : Epoch 1492 | loss: nan | val_loss: nan | Time: 2860.96 ms [2021-07-18 13:52:19 main:581] : INFO : Epoch 1493 | loss: nan | val_loss: nan | Time: 2464 ms [2021-07-18 13:52:22 main:581] : INFO : Epoch 1494 | loss: nan | val_loss: nan | Time: 3115.62 ms [2021-07-18 13:52:25 main:581] : INFO : Epoch 1495 | loss: nan | val_loss: nan | Time: 2771.8 ms [2021-07-18 13:52:28 main:581] : INFO : Epoch 1496 | loss: nan | val_loss: nan | Time: 3226.62 ms [2021-07-18 13:52:31 main:581] : INFO : Epoch 1497 | loss: nan | val_loss: nan | Time: 2891.7 ms [2021-07-18 13:52:34 main:581] : INFO : Epoch 1498 | loss: nan | val_loss: nan | Time: 2833.88 ms [2021-07-18 13:52:36 main:581] : INFO : Epoch 1499 | loss: nan | val_loss: nan | Time: 2336.79 ms [2021-07-18 13:52:39 main:581] : INFO : Epoch 1500 | loss: nan | val_loss: nan | Time: 2388.63 ms [2021-07-18 13:52:41 main:581] : INFO : Epoch 1501 | loss: nan | val_loss: nan | Time: 2818.87 ms [2021-07-18 13:52:44 main:581] : INFO : Epoch 1502 | loss: nan | val_loss: nan | Time: 2879.87 ms [2021-07-18 13:52:47 main:581] : INFO : Epoch 1503 | loss: nan | val_loss: nan | Time: 2498.75 ms [2021-07-18 13:52:50 main:581] : INFO : Epoch 1504 | loss: nan | val_loss: nan | Time: 3600.22 ms [2021-07-18 13:52:53 main:581] : INFO : Epoch 1505 | loss: nan | val_loss: nan | Time: 2491.51 ms [2021-07-18 13:52:55 main:581] : INFO : Epoch 1506 | loss: nan | val_loss: nan | Time: 2535.25 ms [2021-07-18 13:52:58 main:581] : INFO : Epoch 1507 | loss: nan | val_loss: nan | Time: 3026.49 ms [2021-07-18 13:53:01 main:581] : INFO : Epoch 1508 | loss: nan | val_loss: nan | Time: 2450.82 ms [2021-07-18 13:53:03 main:581] : INFO : Epoch 1509 | loss: nan | val_loss: nan | Time: 2388.98 ms [2021-07-18 13:53:06 main:581] : INFO : Epoch 1510 | loss: nan | val_loss: nan | Time: 2498.96 ms [2021-07-18 13:53:08 main:581] : INFO : Epoch 1511 | loss: nan | val_loss: nan | Time: 2400.79 ms [2021-07-18 13:53:11 main:581] : INFO : Epoch 1512 | loss: nan | val_loss: nan | Time: 3096.23 ms [2021-07-18 13:53:13 main:581] : INFO : Epoch 1513 | loss: nan | val_loss: nan | Time: 1882.38 ms [2021-07-18 13:53:15 main:581] : INFO : Epoch 1514 | loss: nan | val_loss: nan | Time: 1760.06 ms [2021-07-18 13:53:17 main:581] : INFO : Epoch 1515 | loss: nan | val_loss: nan | Time: 1869.98 ms [2021-07-18 13:53:19 main:581] : INFO : Epoch 1516 | loss: nan | val_loss: nan | Time: 1864.27 ms [2021-07-18 13:53:20 main:581] : INFO : Epoch 1517 | loss: nan | val_loss: nan | Time: 1784.6 ms [2021-07-18 13:53:22 main:581] : INFO : Epoch 1518 | loss: nan | val_loss: nan | Time: 1787.26 ms [2021-07-18 13:53:24 main:581] : INFO : Epoch 1519 | loss: nan | val_loss: nan | Time: 1913.73 ms [2021-07-18 13:53:26 main:581] : INFO : Epoch 1520 | loss: nan | val_loss: nan | Time: 1892.82 ms [2021-07-18 13:53:28 main:581] : INFO : Epoch 1521 | loss: nan | val_loss: nan | Time: 1893.43 ms [2021-07-18 13:53:31 main:581] : INFO : Epoch 1522 | loss: nan | val_loss: nan | Time: 2846.01 ms [2021-07-18 13:53:33 main:581] : INFO : Epoch 1523 | loss: nan | val_loss: nan | Time: 2315.49 ms [2021-07-18 13:53:35 main:581] : INFO : Epoch 1524 | loss: nan | val_loss: nan | Time: 2190.17 ms [2021-07-18 13:53:38 main:581] : INFO : Epoch 1525 | loss: nan | val_loss: nan | Time: 2432.32 ms [2021-07-18 13:53:40 main:581] : INFO : Epoch 1526 | loss: nan | val_loss: nan | Time: 2364.09 ms [2021-07-18 13:53:42 main:581] : INFO : Epoch 1527 | loss: nan | val_loss: nan | Time: 2385.12 ms [2021-07-18 13:53:45 main:581] : INFO : Epoch 1528 | loss: nan | val_loss: nan | Time: 2322.24 ms [2021-07-18 13:53:47 main:581] : INFO : Epoch 1529 | loss: nan | val_loss: nan | Time: 2475.88 ms [2021-07-18 13:53:50 main:581] : INFO : Epoch 1530 | loss: nan | val_loss: nan | Time: 2562.15 ms [2021-07-18 13:53:52 main:581] : INFO : Epoch 1531 | loss: nan | val_loss: nan | Time: 2181.99 ms [2021-07-18 13:53:54 main:581] : INFO : Epoch 1532 | loss: nan | val_loss: nan | Time: 2432.59 ms [2021-07-18 13:53:57 main:581] : INFO : Epoch 1533 | loss: nan | val_loss: nan | Time: 2157.22 ms [2021-07-18 13:53:59 main:581] : INFO : Epoch 1534 | loss: nan | val_loss: nan | Time: 2258.74 ms [2021-07-18 13:54:01 main:581] : INFO : Epoch 1535 | loss: nan | val_loss: nan | Time: 2573.3 ms [2021-07-18 13:54:04 main:581] : INFO : Epoch 1536 | loss: nan | val_loss: nan | Time: 2618.65 ms [2021-07-18 13:54:07 main:581] : INFO : Epoch 1537 | loss: nan | val_loss: nan | Time: 2465.22 ms [2021-07-18 13:54:09 main:581] : INFO : Epoch 1538 | loss: nan | val_loss: nan | Time: 2457.61 ms [2021-07-18 13:54:12 main:581] : INFO : Epoch 1539 | loss: nan | val_loss: nan | Time: 2546.56 ms [2021-07-18 13:54:14 main:581] : INFO : Epoch 1540 | loss: nan | val_loss: nan | Time: 2775.48 ms [2021-07-18 13:54:17 main:581] : INFO : Epoch 1541 | loss: nan | val_loss: nan | Time: 2500.77 ms [2021-07-18 13:54:20 main:581] : INFO : Epoch 1542 | loss: nan | val_loss: nan | Time: 2829.45 ms [2021-07-18 13:54:22 main:581] : INFO : Epoch 1543 | loss: nan | val_loss: nan | Time: 2293.63 ms [2021-07-18 13:54:24 main:581] : INFO : Epoch 1544 | loss: nan | val_loss: nan | Time: 1840.67 ms [2021-07-18 13:54:26 main:581] : INFO : Epoch 1545 | loss: nan | val_loss: nan | Time: 1983.24 ms [2021-07-18 13:54:28 main:581] : INFO : Epoch 1546 | loss: nan | val_loss: nan | Time: 2003.68 ms [2021-07-18 13:54:30 main:581] : INFO : Epoch 1547 | loss: nan | val_loss: nan | Time: 1955.83 ms [2021-07-18 13:54:32 main:581] : INFO : Epoch 1548 | loss: nan | val_loss: nan | Time: 2472.4 ms [2021-07-18 13:54:35 main:581] : INFO : Epoch 1549 | loss: nan | val_loss: nan | Time: 2864.36 ms [2021-07-18 13:54:38 main:581] : INFO : Epoch 1550 | loss: nan | val_loss: nan | Time: 2904.1 ms [2021-07-18 13:54:40 main:581] : INFO : Epoch 1551 | loss: nan | val_loss: nan | Time: 2393.28 ms [2021-07-18 13:54:43 main:581] : INFO : Epoch 1552 | loss: nan | val_loss: nan | Time: 2692.59 ms [2021-07-18 13:54:45 main:581] : INFO : Epoch 1553 | loss: nan | val_loss: nan | Time: 2369.9 ms [2021-07-18 13:54:49 main:581] : INFO : Epoch 1554 | loss: nan | val_loss: nan | Time: 3841.1 ms [2021-07-18 13:54:53 main:581] : INFO : Epoch 1555 | loss: nan | val_loss: nan | Time: 3553.54 ms [2021-07-18 13:54:55 main:581] : INFO : Epoch 1556 | loss: nan | val_loss: nan | Time: 2412.62 ms [2021-07-18 13:54:58 main:581] : INFO : Epoch 1557 | loss: nan | val_loss: nan | Time: 2409.5 ms [2021-07-18 13:55:00 main:581] : INFO : Epoch 1558 | loss: nan | val_loss: nan | Time: 2278.89 ms [2021-07-18 13:55:02 main:581] : INFO : Epoch 1559 | loss: nan | val_loss: nan | Time: 2161.68 ms [2021-07-18 13:55:05 main:581] : INFO : Epoch 1560 | loss: nan | val_loss: nan | Time: 2670.87 ms [2021-07-18 13:55:07 main:581] : INFO : Epoch 1561 | loss: nan | val_loss: nan | Time: 2466.39 ms [2021-07-18 13:55:10 main:581] : INFO : Epoch 1562 | loss: nan | val_loss: nan | Time: 2749.51 ms [2021-07-18 13:55:14 main:581] : INFO : Epoch 1563 | loss: nan | val_loss: nan | Time: 3748.06 ms [2021-07-18 13:55:18 main:581] : INFO : Epoch 1564 | loss: nan | val_loss: nan | Time: 4399.86 ms [2021-07-18 13:55:21 main:581] : INFO : Epoch 1565 | loss: nan | val_loss: nan | Time: 2973.12 ms [2021-07-18 13:55:23 main:581] : INFO : Epoch 1566 | loss: nan | val_loss: nan | Time: 2104.79 ms [2021-07-18 13:55:26 main:581] : INFO : Epoch 1567 | loss: nan | val_loss: nan | Time: 2540.33 ms [2021-07-18 13:55:28 main:581] : INFO : Epoch 1568 | loss: nan | val_loss: nan | Time: 2335.56 ms [2021-07-18 13:55:31 main:581] : INFO : Epoch 1569 | loss: nan | val_loss: nan | Time: 2627.98 ms [2021-07-18 13:55:33 main:581] : INFO : Epoch 1570 | loss: nan | val_loss: nan | Time: 2420.88 ms [2021-07-18 13:55:37 main:581] : INFO : Epoch 1571 | loss: nan | val_loss: nan | Time: 3680.17 ms [2021-07-18 13:55:40 main:581] : INFO : Epoch 1572 | loss: nan | val_loss: nan | Time: 3106.05 ms [2021-07-18 13:55:43 main:581] : INFO : Epoch 1573 | loss: nan | val_loss: nan | Time: 3527.21 ms [2021-07-18 13:55:46 main:581] : INFO : Epoch 1574 | loss: nan | val_loss: nan | Time: 2688.99 ms [2021-07-18 13:55:49 main:581] : INFO : Epoch 1575 | loss: nan | val_loss: nan | Time: 2584.49 ms [2021-07-18 13:55:51 main:581] : INFO : Epoch 1576 | loss: nan | val_loss: nan | Time: 2077.33 ms [2021-07-18 13:55:53 main:581] : INFO : Epoch 1577 | loss: nan | val_loss: nan | Time: 2435.54 ms [2021-07-18 13:55:56 main:581] : INFO : Epoch 1578 | loss: nan | val_loss: nan | Time: 2481 ms [2021-07-18 13:55:58 main:581] : INFO : Epoch 1579 | loss: nan | val_loss: nan | Time: 2692.37 ms [2021-07-18 13:56:02 main:581] : INFO : Epoch 1580 | loss: nan | val_loss: nan | Time: 3780.07 ms [2021-07-18 13:56:05 main:581] : INFO : Epoch 1581 | loss: nan | val_loss: nan | Time: 2882.09 ms [2021-07-18 13:56:08 main:581] : INFO : Epoch 1582 | loss: nan | val_loss: nan | Time: 2822.25 ms [2021-07-18 13:56:11 main:581] : INFO : Epoch 1583 | loss: nan | val_loss: nan | Time: 2940.3 ms [2021-07-18 13:56:13 main:581] : INFO : Epoch 1584 | loss: nan | val_loss: nan | Time: 2386.43 ms [2021-07-18 13:56:16 main:581] : INFO : Epoch 1585 | loss: nan | val_loss: nan | Time: 2310.76 ms [2021-07-18 13:56:18 main:581] : INFO : Epoch 1586 | loss: nan | val_loss: nan | Time: 2522.14 ms [2021-07-18 13:56:20 main:581] : INFO : Epoch 1587 | loss: nan | val_loss: nan | Time: 2440.25 ms [2021-07-18 13:56:23 main:581] : INFO : Epoch 1588 | loss: nan | val_loss: nan | Time: 2471.54 ms [2021-07-18 13:56:25 main:581] : INFO : Epoch 1589 | loss: nan | val_loss: nan | Time: 2296.77 ms [2021-07-18 13:56:27 main:581] : INFO : Epoch 1590 | loss: nan | val_loss: nan | Time: 2024.2 ms [2021-07-18 13:56:30 main:581] : INFO : Epoch 1591 | loss: nan | val_loss: nan | Time: 2290.83 ms [2021-07-18 13:56:32 main:581] : INFO : Epoch 1592 | loss: nan | val_loss: nan | Time: 2267.6 ms [2021-07-18 13:56:34 main:581] : INFO : Epoch 1593 | loss: nan | val_loss: nan | Time: 2343.27 ms [2021-07-18 13:56:36 main:581] : INFO : Epoch 1594 | loss: nan | val_loss: nan | Time: 2320.7 ms [2021-07-18 13:56:39 main:581] : INFO : Epoch 1595 | loss: nan | val_loss: nan | Time: 2281.51 ms [2021-07-18 13:56:41 main:581] : INFO : Epoch 1596 | loss: nan | val_loss: nan | Time: 2220.99 ms [2021-07-18 13:56:43 main:581] : INFO : Epoch 1597 | loss: nan | val_loss: nan | Time: 2325.9 ms [2021-07-18 13:56:46 main:581] : INFO : Epoch 1598 | loss: nan | val_loss: nan | Time: 2240.79 ms [2021-07-18 13:56:48 main:581] : INFO : Epoch 1599 | loss: nan | val_loss: nan | Time: 2341.5 ms [2021-07-18 13:56:50 main:581] : INFO : Epoch 1600 | loss: nan | val_loss: nan | Time: 2291.43 ms [2021-07-18 13:56:53 main:581] : INFO : Epoch 1601 | loss: nan | val_loss: nan | Time: 2322.18 ms [2021-07-18 13:56:55 main:581] : INFO : Epoch 1602 | loss: nan | val_loss: nan | Time: 2329.76 ms [2021-07-18 13:56:57 main:581] : INFO : Epoch 1603 | loss: nan | val_loss: nan | Time: 2202.56 ms [2021-07-18 13:56:59 main:581] : INFO : Epoch 1604 | loss: nan | val_loss: nan | Time: 2260.96 ms [2021-07-18 13:57:02 main:581] : INFO : Epoch 1605 | loss: nan | val_loss: nan | Time: 2291.88 ms SIGSEGV: segmentation violation Stack trace (40 frames): ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x56322e88df9c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f05f6989730] /lib/x86_64-linux-gnu/libc.so.6(+0x39d57)[0x7f05782a3d57] /lib/x86_64-linux-gnu/libc.so.6(+0x39eba)[0x7f05782a3eba] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379b25)[0x56322e889b25] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379bee)[0x56322e889bee] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f05f6989730] ./libcudnn_ops_infer.so.8(+0x9a6377)[0x7f04f0219377] ./libcudnn_ops_infer.so.8(+0x9aacaa)[0x7f04f021dcaa] ./libcudnn_ops_infer.so.8(+0x6abd9a)[0x7f04eff1ed9a] ./libcudnn_ops_infer.so.8(+0x6accf3)[0x7f04eff1fcf3] ./libcudnn_ops_infer.so.8(+0x68e125)[0x7f04eff01125] ./libcudnn_ops_infer.so.8(+0x68f8ea)[0x7f04eff028ea] ./libcudnn_ops_infer.so.8(_ZN5cudnn12cublasGemmExEP13cublasContext17cublasOperation_tS2_iiiPKvS4_14cudaDataType_tiS4_S5_iS4_PvS5_iS5_16cublasGemmAlgo_t+0x12c)[0x7f04efa9659c] ./libcudnn_adv_infer.so.8(+0x777ec)[0x7f04db4877ec] ./libcudnn_adv_infer.so.8(+0xe89f)[0x7f04db41e89f] ./libcudnn_adv_infer.so.8(_Z18RNN_forwardGenericP12cudnnContextP14cudnnRNNStructiPKP17cudnnTensorStructPKvS4_S8_S4_S8_P17cudnnFilterStructS8_S6_PvS4_SB_S4_SB_SB_mSB_m18cudnnForwardMode_t+0x5d7)[0x7f04db4628c7] ./libcudnn_adv_train.so.8(cudnnRNNForwardTraining+0x20e)[0x7f04c661661e] ./libtorch_cuda.so(_ZN2at6native10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS1_EElS3_S3_S3_lllbdbbNS5_IlEES3_+0x2214)[0x7f0579071a64] ./libtorch_cuda.so(+0x85face)[0x7f05790bbace] ./libtorch_cuda.so(+0x895988)[0x7f05790f1988] ./libtorch_cpu.so(_ZN2at10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS0_EElS2_S2_S2_lllbdbbNS4_IlEES2_+0x2bc)[0x7f05f162a51c] ./libtorch_cpu.so(+0x21cf2d4)[0x7f05f29f42d4] ./libtorch_cpu.so(+0x21edb28)[0x7f05f2a12b28] ./libtorch_cpu.so(_ZN2at10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS0_EElS2_S2_S2_lllbdbbNS4_IlEES2_+0x2bc)[0x7f05f162a51c] ./libtorch_cuda.so(+0x80f0fd)[0x7f057906b0fd] ./libtorch_cuda.so(+0x80fb0a)[0x7f057906bb0a] ./libtorch_cpu.so(+0x97fe93)[0x7f05f11a4e93] ./libtorch_cpu.so(_ZN2at6native3gruERKNS_6TensorES3_N3c108ArrayRefIS1_EEbldbbb+0xea)[0x7f05f11889ba] ./libtorch_cpu.so(+0xeb0a96)[0x7f05f16d5a96] ./libtorch_cpu.so(+0x22e3c92)[0x7f05f2b08c92] ./libtorch_cpu.so(+0xedd579)[0x7f05f1702579] ./libtorch_cpu.so(_ZN2at3gruERKNS_6TensorES2_N3c108ArrayRefIS0_EEbldbbb+0x1b1)[0x7f05f1607ee1] ./libtorch_cpu.so(_ZN5torch2nn7GRUImpl14forward_helperERKN2at6TensorES5_S5_lS3_+0x43a)[0x7f05f35714ca] ./libtorch_cpu.so(_ZN5torch2nn7GRUImpl7forwardERKN2at6TensorES3_+0xb0)[0x7f05f3571630] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x92fa8)[0x56322e5a2fa8] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x9a0e6)[0x56322e5aa0e6] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8ad10)[0x56322e59ad10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f057828e09b] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x56322e59675a] Exiting... DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048 nthreads: 1 gpudev: 0 Re-exec()-ing to set environment correctly Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: GeForce RTX 2080 Ti) [2021-07-18 14:47:33 main:442] : INFO : Set logging level to 1 [2021-07-18 14:47:33 main:448] : INFO : Running in BOINC Client mode [2021-07-18 14:47:33 main:451] : INFO : Resolving all filenames [2021-07-18 14:47:33 main:459] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 (exists = 1) [2021-07-18 14:47:33 main:459] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1624823822-21588-1_0_r824845378_1 (exists = 1) [2021-07-18 14:47:33 main:459] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1624823822-21588-1_0_r824845378_0 (exists = 0) [2021-07-18 14:47:33 main:459] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1624823822-21588-1 (exists = 1) [2021-07-18 14:47:33 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2021-07-18 14:47:33 main:479] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 [2021-07-18 14:47:33 main:481] : INFO : Configuration: [2021-07-18 14:47:33 main:482] : INFO : Model type: GRU [2021-07-18 14:47:33 main:483] : INFO : Validation Loss Threshold: 0.0001 [2021-07-18 14:47:33 main:484] : INFO : Max Epochs: 2048 [2021-07-18 14:47:33 main:485] : INFO : Batch Size: 128 [2021-07-18 14:47:33 main:486] : INFO : Learning Rate: 0.01 [2021-07-18 14:47:33 main:487] : INFO : Patience: 10 [2021-07-18 14:47:33 main:488] : INFO : Hidden Width: 12 [2021-07-18 14:47:33 main:489] : INFO : # Recurrent Layers: 4 [2021-07-18 14:47:33 main:490] : INFO : # Backend Layers: 4 [2021-07-18 14:47:33 main:491] : INFO : # Threads: 1 [2021-07-18 14:47:33 main:493] : INFO : Preparing Dataset [2021-07-18 14:47:33 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory [2021-07-18 14:47:33 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory [2021-07-18 14:47:36 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2021-07-18 14:47:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory [2021-07-18 14:47:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory [2021-07-18 14:47:36 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2021-07-18 14:47:36 main:501] : INFO : Creating Model [2021-07-18 14:47:36 main:514] : INFO : Preparing config file [2021-07-18 14:47:36 main:518] : INFO : Found checkpoint, attempting to load... [2021-07-18 14:47:36 main:519] : INFO : Loading config terminate called after throwing an instance of 'nlohmann::detail::type_error' what(): [json.exception.type_error.302] type must be number, but is null SIGABRT: abort called Stack trace (24 frames): ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x55fc2af23f9c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f45a4054730] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f452596c7bb] /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f4525957535] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x135)[0x55fc2afd57f5] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398846)[0x55fc2af3e846] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398891)[0x55fc2af3e891] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x3968c4)[0x55fc2af3c8c4] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe4af0)[0x55fc2ac8aaf0] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe1682)[0x55fc2ac87682] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd9a2b)[0x55fc2ac7fa2b] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd5f7f)[0x55fc2ac7bf7f] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcf7f4)[0x55fc2ac757f4] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe16bc)[0x55fc2ac876bc] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe4c96)[0x55fc2ac8ac96] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe17ca)[0x55fc2ac877ca] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd9b28)[0x55fc2ac7fb28] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd5faf)[0x55fc2ac7bfaf] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcf848)[0x55fc2ac75848] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcc66f)[0x55fc2ac7266f] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xc8d8c)[0x55fc2ac6ed8c] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8a2d1)[0x55fc2ac302d1] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f452595909b] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x55fc2ac2c75a] Exiting... </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)