Task 5747679
| Name | ParityMachine-1625432385-20358-1_0 |
| Workunit | 3579376 |
| Created | 10 Jul 2021, 1:08:19 UTC |
| Sent | 16 Jul 2021, 18:44:58 UTC |
| Report deadline | 23 Jul 2021, 18:44:58 UTC |
| Received | 16 Jul 2021, 22:15:56 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 193 (0x000000C1) EXIT_SIGNAL |
| Computer ID | 13546 |
| Run time | 1 hours 0 min 56 sec |
| CPU time | 1 hours 0 min 45 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 15,412.34 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.80 (cuda10200) x86_64-pc-linux-gnu |
| Peak working set size | 2.33 GB |
| Peak swap size | 10.56 GB |
| Peak disk usage | 2.98 GB |
Stderr output
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)</message>
<stderr_txt>
main:581] : INFO : Epoch 1233 | loss: 0.0311734 | val_loss: 0.0311846 | Time: 2457.45 ms
[2021-07-16 22:30:16 main:581] : INFO : Epoch 1234 | loss: 0.0311723 | val_loss: 0.0311848 | Time: 2721.91 ms
[2021-07-16 22:30:19 main:581] : INFO : Epoch 1235 | loss: 0.0311744 | val_loss: 0.0311881 | Time: 2968.56 ms
[2021-07-16 22:30:22 main:581] : INFO : Epoch 1236 | loss: 0.0311756 | val_loss: 0.0311867 | Time: 2561.61 ms
[2021-07-16 22:30:24 main:581] : INFO : Epoch 1237 | loss: 0.0311733 | val_loss: 0.0311853 | Time: 2514.14 ms
[2021-07-16 22:30:27 main:581] : INFO : Epoch 1238 | loss: 0.0311726 | val_loss: 0.0311848 | Time: 2339.62 ms
[2021-07-16 22:30:29 main:581] : INFO : Epoch 1239 | loss: 0.0311723 | val_loss: 0.0311838 | Time: 2261.89 ms
[2021-07-16 22:30:32 main:581] : INFO : Epoch 1240 | loss: 0.0311717 | val_loss: 0.0311846 | Time: 2708.82 ms
[2021-07-16 22:30:34 main:581] : INFO : Epoch 1241 | loss: 0.0311721 | val_loss: 0.0311841 | Time: 2454.68 ms
[2021-07-16 22:30:37 main:581] : INFO : Epoch 1242 | loss: 0.0311719 | val_loss: 0.0311843 | Time: 2365.24 ms
[2021-07-16 22:30:40 main:581] : INFO : Epoch 1243 | loss: 0.0311725 | val_loss: 0.0311832 | Time: 3153.23 ms
[2021-07-16 22:30:43 main:581] : INFO : Epoch 1244 | loss: 0.0311731 | val_loss: 0.0311847 | Time: 3254.56 ms
[2021-07-16 22:30:45 main:581] : INFO : Epoch 1245 | loss: 0.0311724 | val_loss: 0.0311839 | Time: 2340.38 ms
[2021-07-16 22:30:48 main:581] : INFO : Epoch 1246 | loss: 0.0311717 | val_loss: 0.0311829 | Time: 2318.24 ms
[2021-07-16 22:30:50 main:581] : INFO : Epoch 1247 | loss: 0.031172 | val_loss: 0.031183 | Time: 2333.35 ms
[2021-07-16 22:30:53 main:581] : INFO : Epoch 1248 | loss: 0.0311713 | val_loss: 0.0311864 | Time: 2813.97 ms
[2021-07-16 22:30:56 main:581] : INFO : Epoch 1249 | loss: 0.0311724 | val_loss: 0.0311843 | Time: 3031.89 ms
[2021-07-16 22:30:58 main:581] : INFO : Epoch 1250 | loss: 0.0311712 | val_loss: 0.0311834 | Time: 2510.09 ms
[2021-07-16 22:31:01 main:581] : INFO : Epoch 1251 | loss: 0.0311714 | val_loss: 0.0311828 | Time: 2982.68 ms
[2021-07-16 22:31:04 main:581] : INFO : Epoch 1252 | loss: 0.0311714 | val_loss: 0.0311829 | Time: 2230.59 ms
[2021-07-16 22:31:06 main:581] : INFO : Epoch 1253 | loss: 0.0311711 | val_loss: 0.0311833 | Time: 2289.88 ms
[2021-07-16 22:31:08 main:581] : INFO : Epoch 1254 | loss: 0.0311707 | val_loss: 0.0311836 | Time: 2430.47 ms
[2021-07-16 22:31:11 main:581] : INFO : Epoch 1255 | loss: 0.0311708 | val_loss: 0.0311869 | Time: 2335.32 ms
[2021-07-16 22:31:13 main:581] : INFO : Epoch 1256 | loss: 0.0311722 | val_loss: 0.031184 | Time: 2153.61 ms
[2021-07-16 22:31:16 main:581] : INFO : Epoch 1257 | loss: 0.031172 | val_loss: 0.0311869 | Time: 2828.76 ms
[2021-07-16 22:31:19 main:581] : INFO : Epoch 1258 | loss: 0.031171 | val_loss: 0.0311836 | Time: 3283.25 ms
[2021-07-16 22:31:21 main:581] : INFO : Epoch 1259 | loss: 0.0311707 | val_loss: 0.0311842 | Time: 2466.54 ms
[2021-07-16 22:31:24 main:581] : INFO : Epoch 1260 | loss: 0.0311706 | val_loss: 0.0311835 | Time: 3039.38 ms
[2021-07-16 22:31:27 main:581] : INFO : Epoch 1261 | loss: 0.0311705 | val_loss: 0.0311853 | Time: 2284.1 ms
[2021-07-16 22:31:29 main:581] : INFO : Epoch 1262 | loss: 0.0311723 | val_loss: 0.0311832 | Time: 2407.72 ms
[2021-07-16 22:31:31 main:581] : INFO : Epoch 1263 | loss: 0.0311704 | val_loss: 0.031184 | Time: 2254.34 ms
[2021-07-16 22:31:34 main:581] : INFO : Epoch 1264 | loss: 0.0311706 | val_loss: 0.0311837 | Time: 2466.78 ms
[2021-07-16 22:31:36 main:581] : INFO : Epoch 1265 | loss: 0.03117 | val_loss: 0.0311843 | Time: 2573.28 ms
[2021-07-16 22:31:39 main:581] : INFO : Epoch 1266 | loss: 0.0311703 | val_loss: 0.0311837 | Time: 2869.16 ms
[2021-07-16 22:31:42 main:581] : INFO : Epoch 1267 | loss: 0.0311706 | val_loss: 0.0311846 | Time: 2796.92 ms
[2021-07-16 22:31:44 main:581] : INFO : Epoch 1268 | loss: 0.0311709 | val_loss: 0.0311844 | Time: 2425.49 ms
[2021-07-16 22:31:47 main:581] : INFO : Epoch 1269 | loss: 0.0311712 | val_loss: 0.0311835 | Time: 2573.52 ms
[2021-07-16 22:31:49 main:581] : INFO : Epoch 1270 | loss: 0.031171 | val_loss: 0.0311839 | Time: 2373.31 ms
[2021-07-16 22:31:52 main:581] : INFO : Epoch 1271 | loss: 0.0311719 | val_loss: 0.0311866 | Time: 2248.52 ms
[2021-07-16 22:31:53 main:581] : INFO : Epoch 1272 | loss: 0.0311717 | val_loss: 0.0311865 | Time: 1787.17 ms
[2021-07-16 22:31:55 main:581] : INFO : Epoch 1273 | loss: 0.031173 | val_loss: 0.0311837 | Time: 2028.32 ms
[2021-07-16 22:31:57 main:581] : INFO : Epoch 1274 | loss: 0.0311706 | val_loss: 0.0311839 | Time: 1942.32 ms
[2021-07-16 22:31:59 main:581] : INFO : Epoch 1275 | loss: 0.0311702 | val_loss: 0.0311829 | Time: 1861.71 ms
[2021-07-16 22:32:02 main:581] : INFO : Epoch 1276 | loss: 0.0311714 | val_loss: 0.0311832 | Time: 2461.78 ms
[2021-07-16 22:32:05 main:581] : INFO : Epoch 1277 | loss: 0.0311698 | val_loss: 0.0311838 | Time: 2789.88 ms
[2021-07-16 22:32:07 main:581] : INFO : Epoch 1278 | loss: 0.0311701 | val_loss: 0.0311845 | Time: 2445.51 ms
[2021-07-16 22:32:09 main:581] : INFO : Epoch 1279 | loss: 0.0311713 | val_loss: 0.0311837 | Time: 2180.66 ms
[2021-07-16 22:32:11 main:581] : INFO : Epoch 1280 | loss: 0.0311717 | val_loss: 0.0311841 | Time: 2322.6 ms
[2021-07-16 22:32:14 main:581] : INFO : Epoch 1281 | loss: 0.0311696 | val_loss: 0.0311859 | Time: 2828.3 ms
[2021-07-16 22:32:17 main:581] : INFO : Epoch 1282 | loss: 0.0311703 | val_loss: 0.0311842 | Time: 2640.39 ms
[2021-07-16 22:32:19 main:581] : INFO : Epoch 1283 | loss: 0.0311703 | val_loss: 0.0311868 | Time: 2499.79 ms
[2021-07-16 22:32:22 main:581] : INFO : Epoch 1284 | loss: 0.0311709 | val_loss: 0.0311831 | Time: 2781.36 ms
[2021-07-16 22:32:25 main:581] : INFO : Epoch 1285 | loss: 0.0311731 | val_loss: 0.0311874 | Time: 2461.28 ms
[2021-07-16 22:32:27 main:581] : INFO : Epoch 1286 | loss: 0.0311746 | val_loss: 0.0311888 | Time: 2204.47 ms
[2021-07-16 22:32:29 main:581] : INFO : Epoch 1287 | loss: 0.0311708 | val_loss: 0.0311843 | Time: 2339.42 ms
[2021-07-16 22:32:32 main:581] : INFO : Epoch 1288 | loss: 0.0311704 | val_loss: 0.0311845 | Time: 2529.8 ms
[2021-07-16 22:32:34 main:581] : INFO : Epoch 1289 | loss: 0.0311701 | val_loss: 0.0311871 | Time: 2432.06 ms
[2021-07-16 22:32:37 main:581] : INFO : Epoch 1290 | loss: 0.0311711 | val_loss: 0.0311841 | Time: 3068.32 ms
[2021-07-16 22:32:40 main:581] : INFO : Epoch 1291 | loss: 0.0311703 | val_loss: 0.0311849 | Time: 2561.12 ms
[2021-07-16 22:32:42 main:581] : INFO : Epoch 1292 | loss: 0.0311721 | val_loss: 0.0311839 | Time: 2164.75 ms
[2021-07-16 22:32:45 main:581] : INFO : Epoch 1293 | loss: 0.0311706 | val_loss: 0.0311849 | Time: 2588.83 ms
[2021-07-16 22:32:48 main:581] : INFO : Epoch 1294 | loss: 0.0311707 | val_loss: 0.0311837 | Time: 3027.06 ms
[2021-07-16 22:32:50 main:581] : INFO : Epoch 1295 | loss: 0.0311699 | val_loss: 0.0311844 | Time: 2415.52 ms
[2021-07-16 22:32:52 main:581] : INFO : Epoch 1296 | loss: 0.0311696 | val_loss: 0.0311845 | Time: 2417.46 ms
[2021-07-16 22:32:55 main:581] : INFO : Epoch 1297 | loss: 0.0311693 | val_loss: 0.0311862 | Time: 2394.46 ms
[2021-07-16 22:32:57 main:581] : INFO : Epoch 1298 | loss: 0.0311702 | val_loss: 0.0311869 | Time: 2444.8 ms
[2021-07-16 22:33:00 main:581] : INFO : Epoch 1299 | loss: 0.0311718 | val_loss: 0.0311833 | Time: 2730.11 ms
[2021-07-16 22:33:02 main:581] : INFO : Epoch 1300 | loss: 0.0311701 | val_loss: 0.0311827 | Time: 2413.43 ms
[2021-07-16 22:33:05 main:581] : INFO : Epoch 1301 | loss: 0.0311703 | val_loss: 0.0311837 | Time: 2370.9 ms
[2021-07-16 22:33:07 main:581] : INFO : Epoch 1302 | loss: 0.0311709 | val_loss: 0.0311844 | Time: 2038.51 ms
[2021-07-16 22:33:10 main:581] : INFO : Epoch 1303 | loss: 0.0311701 | val_loss: 0.0311867 | Time: 2797.79 ms
[2021-07-16 22:33:12 main:581] : INFO : Epoch 1304 | loss: 0.0311697 | val_loss: 0.0311857 | Time: 1945.6 ms
[2021-07-16 22:33:14 main:581] : INFO : Epoch 1305 | loss: 0.0311695 | val_loss: 0.0311845 | Time: 2574.37 ms
[2021-07-16 22:33:17 main:581] : INFO : Epoch 1306 | loss: 0.0311695 | val_loss: 0.0311848 | Time: 2672.89 ms
[2021-07-16 22:33:20 main:581] : INFO : Epoch 1307 | loss: 0.0311699 | val_loss: 0.0311841 | Time: 2752.99 ms
[2021-07-16 22:33:22 main:581] : INFO : Epoch 1308 | loss: 0.0311695 | val_loss: 0.0311846 | Time: 2567.3 ms
[2021-07-16 22:33:24 main:581] : INFO : Epoch 1309 | loss: 0.0311699 | val_loss: 0.0311837 | Time: 1809.33 ms
[2021-07-16 22:33:26 main:581] : INFO : Epoch 1310 | loss: 0.0311703 | val_loss: 0.0311842 | Time: 1667.73 ms
[2021-07-16 22:33:27 main:581] : INFO : Epoch 1311 | loss: 0.0311704 | val_loss: 0.0311839 | Time: 1705.49 ms
[2021-07-16 22:33:29 main:581] : INFO : Epoch 1312 | loss: 0.0311703 | val_loss: 0.0311842 | Time: 1874.82 ms
[2021-07-16 22:33:31 main:581] : INFO : Epoch 1313 | loss: 0.0311701 | val_loss: 0.0311853 | Time: 1731.02 ms
[2021-07-16 22:33:33 main:581] : INFO : Epoch 1314 | loss: 0.0311699 | val_loss: 0.0311835 | Time: 1779.89 ms
[2021-07-16 22:33:35 main:581] : INFO : Epoch 1315 | loss: 0.0311694 | val_loss: 0.031184 | Time: 1972.65 ms
[2021-07-16 22:33:37 main:581] : INFO : Epoch 1316 | loss: 0.0311694 | val_loss: 0.0311878 | Time: 1953.69 ms
[2021-07-16 22:33:39 main:581] : INFO : Epoch 1317 | loss: 0.0311701 | val_loss: 0.0311839 | Time: 1949.07 ms
[2021-07-16 22:33:41 main:581] : INFO : Epoch 1318 | loss: 0.0311694 | val_loss: 0.0311847 | Time: 2351.78 ms
[2021-07-16 22:33:43 main:581] : INFO : Epoch 1319 | loss: 0.0311689 | val_loss: 0.0311838 | Time: 2247.53 ms
[2021-07-16 22:33:45 main:581] : INFO : Epoch 1320 | loss: 0.0311695 | val_loss: 0.0311865 | Time: 2243.23 ms
[2021-07-16 22:33:48 main:581] : INFO : Epoch 1321 | loss: 0.0311697 | val_loss: 0.031185 | Time: 2532.43 ms
[2021-07-16 22:33:50 main:581] : INFO : Epoch 1322 | loss: 0.0311697 | val_loss: 0.0311886 | Time: 2397.85 ms
[2021-07-16 22:33:53 main:581] : INFO : Epoch 1323 | loss: 0.0311694 | val_loss: 0.0311842 | Time: 2318.91 ms
[2021-07-16 22:33:55 main:581] : INFO : Epoch 1324 | loss: 0.0311693 | val_loss: 0.0311847 | Time: 2386.92 ms
[2021-07-16 22:33:57 main:581] : INFO : Epoch 1325 | loss: 0.0311698 | val_loss: 0.0311849 | Time: 2312.17 ms
[2021-07-16 22:34:00 main:581] : INFO : Epoch 1326 | loss: 0.0311701 | val_loss: 0.0311836 | Time: 2357.08 ms
[2021-07-16 22:34:02 main:581] : INFO : Epoch 1327 | loss: 0.0311695 | val_loss: 0.0311845 | Time: 2413.96 ms
[2021-07-16 22:34:04 main:581] : INFO : Epoch 1328 | loss: 0.031169 | val_loss: 0.0311839 | Time: 2253.29 ms
[2021-07-16 22:34:07 main:581] : INFO : Epoch 1329 | loss: 0.0311712 | val_loss: 0.0311837 | Time: 2189.21 ms
[2021-07-16 22:34:09 main:581] : INFO : Epoch 1330 | loss: 0.0311704 | val_loss: 0.0311878 | Time: 2049.99 ms
[2021-07-16 22:34:11 main:581] : INFO : Epoch 1331 | loss: 0.0311705 | val_loss: 0.0311851 | Time: 2157.08 ms
[2021-07-16 22:34:13 main:581] : INFO : Epoch 1332 | loss: 0.0311691 | val_loss: 0.0311842 | Time: 2239.35 ms
[2021-07-16 22:34:16 main:581] : INFO : Epoch 1333 | loss: 0.031169 | val_loss: 0.0311846 | Time: 2431.3 ms
[2021-07-16 22:34:18 main:581] : INFO : Epoch 1334 | loss: 0.0311695 | val_loss: 0.0311836 | Time: 2421 ms
[2021-07-16 22:34:20 main:581] : INFO : Epoch 1335 | loss: 0.0311694 | val_loss: 0.0311837 | Time: 2418.52 ms
[2021-07-16 22:34:23 main:581] : INFO : Epoch 1336 | loss: 0.0311696 | val_loss: 0.0311851 | Time: 2483.22 ms
[2021-07-16 22:34:25 main:581] : INFO : Epoch 1337 | loss: 0.0311689 | val_loss: 0.0311838 | Time: 2405.58 ms
[2021-07-16 22:34:27 main:581] : INFO : Epoch 1338 | loss: 0.0311691 | val_loss: 0.0311832 | Time: 2176.64 ms
[2021-07-16 22:34:30 main:581] : INFO : Epoch 1339 | loss: 0.0311688 | val_loss: 0.0311862 | Time: 2196.43 ms
[2021-07-16 22:34:32 main:581] : INFO : Epoch 1340 | loss: 0.0311694 | val_loss: 0.0311842 | Time: 2110.92 ms
[2021-07-16 22:34:34 main:581] : INFO : Epoch 1341 | loss: 0.0311687 | val_loss: 0.0311864 | Time: 2353.18 ms
[2021-07-16 22:34:36 main:581] : INFO : Epoch 1342 | loss: 0.0311689 | val_loss: 0.0311843 | Time: 2311.26 ms
[2021-07-16 22:34:39 main:581] : INFO : Epoch 1343 | loss: 0.0311691 | val_loss: 0.0311846 | Time: 2355.14 ms
[2021-07-16 22:34:41 main:581] : INFO : Epoch 1344 | loss: 0.0311695 | val_loss: 0.0311861 | Time: 2437.46 ms
[2021-07-16 22:34:44 main:581] : INFO : Epoch 1345 | loss: 0.0311695 | val_loss: 0.0311887 | Time: 2380.16 ms
[2021-07-16 22:34:46 main:581] : INFO : Epoch 1346 | loss: 0.0311733 | val_loss: 0.0311874 | Time: 2349.82 ms
[2021-07-16 22:34:48 main:581] : INFO : Epoch 1347 | loss: 0.031172 | val_loss: 0.0311831 | Time: 2333.71 ms
[2021-07-16 22:34:51 main:581] : INFO : Epoch 1348 | loss: 0.0311701 | val_loss: 0.0311831 | Time: 2352.58 ms
[2021-07-16 22:34:53 main:581] : INFO : Epoch 1349 | loss: 0.0311692 | val_loss: 0.0311837 | Time: 2381.7 ms
[2021-07-16 22:34:55 main:581] : INFO : Epoch 1350 | loss: 0.03117 | val_loss: 0.0311844 | Time: 2249.49 ms
[2021-07-16 22:34:57 main:581] : INFO : Epoch 1351 | loss: 0.0311699 | val_loss: 0.031184 | Time: 2199.1 ms
[2021-07-16 22:34:59 main:581] : INFO : Epoch 1352 | loss: 0.0311699 | val_loss: 0.0311838 | Time: 1717.38 ms
[2021-07-16 22:35:01 main:581] : INFO : Epoch 1353 | loss: 0.031169 | val_loss: 0.0311855 | Time: 1790.07 ms
[2021-07-16 22:35:04 main:581] : INFO : Epoch 1354 | loss: 0.0311692 | val_loss: 0.0311848 | Time: 2880.33 ms
[2021-07-16 22:35:07 main:581] : INFO : Epoch 1355 | loss: 0.0311711 | val_loss: 0.0311846 | Time: 3266.7 ms
[2021-07-16 22:35:10 main:581] : INFO : Epoch 1356 | loss: 0.0311738 | val_loss: 0.0311846 | Time: 3133.36 ms
[2021-07-16 22:35:13 main:581] : INFO : Epoch 1357 | loss: 0.0311722 | val_loss: 0.0311838 | Time: 2347.14 ms
[2021-07-16 22:35:15 main:581] : INFO : Epoch 1358 | loss: 0.0311702 | val_loss: 0.0311833 | Time: 2241.51 ms
[2021-07-16 22:35:17 main:581] : INFO : Epoch 1359 | loss: 0.0311696 | val_loss: 0.0311847 | Time: 2219.35 ms
[2021-07-16 22:35:19 main:581] : INFO : Epoch 1360 | loss: 0.0311692 | val_loss: 0.0311867 | Time: 2441.76 ms
[2021-07-16 22:35:22 main:581] : INFO : Epoch 1361 | loss: 0.0311694 | val_loss: 0.0311859 | Time: 2410.45 ms
[2021-07-16 22:35:24 main:581] : INFO : Epoch 1362 | loss: 0.0311689 | val_loss: 0.0311851 | Time: 2286.69 ms
[2021-07-16 22:35:27 main:581] : INFO : Epoch 1363 | loss: 0.0311696 | val_loss: 0.0311848 | Time: 2445.38 ms
[2021-07-16 22:35:29 main:581] : INFO : Epoch 1364 | loss: 0.0311689 | val_loss: 0.0311848 | Time: 2451.79 ms
[2021-07-16 22:35:32 main:581] : INFO : Epoch 1365 | loss: 0.0311689 | val_loss: 0.0311835 | Time: 2431.38 ms
[2021-07-16 22:35:34 main:581] : INFO : Epoch 1366 | loss: 0.031169 | val_loss: 0.0311855 | Time: 2170.14 ms
[2021-07-16 22:35:36 main:581] : INFO : Epoch 1367 | loss: 0.0311688 | val_loss: 0.0311834 | Time: 2248.9 ms
[2021-07-16 22:35:38 main:581] : INFO : Epoch 1368 | loss: 0.0311694 | val_loss: 0.0311836 | Time: 2339.7 ms
[2021-07-16 22:35:41 main:581] : INFO : Epoch 1369 | loss: 0.03117 | val_loss: 0.0311844 | Time: 2372.44 ms
[2021-07-16 22:35:43 main:581] : INFO : Epoch 1370 | loss: 0.0311691 | val_loss: 0.0311838 | Time: 2481.87 ms
[2021-07-16 22:35:46 main:581] : INFO : Epoch 1371 | loss: 0.0311697 | val_loss: 0.0311836 | Time: 2439.16 ms
[2021-07-16 22:35:48 main:581] : INFO : Epoch 1372 | loss: 0.0311723 | val_loss: 0.0311848 | Time: 2322.91 ms
[2021-07-16 22:35:50 main:581] : INFO : Epoch 1373 | loss: 0.031169 | val_loss: 0.0311844 | Time: 2093.01 ms
[2021-07-16 22:35:52 main:581] : INFO : Epoch 1374 | loss: 0.0311702 | val_loss: 0.0311838 | Time: 2301.17 ms
[2021-07-16 22:35:55 main:581] : INFO : Epoch 1375 | loss: 0.0311707 | val_loss: 0.0311873 | Time: 2481.97 ms
[2021-07-16 22:35:57 main:581] : INFO : Epoch 1376 | loss: 0.0311703 | val_loss: 0.0311846 | Time: 2312.52 ms
[2021-07-16 22:35:59 main:581] : INFO : Epoch 1377 | loss: 0.031172 | val_loss: 0.0311857 | Time: 2306.97 ms
[2021-07-16 22:36:02 main:581] : INFO : Epoch 1378 | loss: 0.0311717 | val_loss: 0.0311897 | Time: 2251.57 ms
[2021-07-16 22:36:04 main:581] : INFO : Epoch 1379 | loss: 0.0311711 | val_loss: 0.0311836 | Time: 2357.89 ms
[2021-07-16 22:36:06 main:581] : INFO : Epoch 1380 | loss: 0.031171 | val_loss: 0.0311822 | Time: 2293.17 ms
[2021-07-16 22:36:09 main:581] : INFO : Epoch 1381 | loss: 0.0311709 | val_loss: 0.0311869 | Time: 2344.19 ms
[2021-07-16 22:36:11 main:581] : INFO : Epoch 1382 | loss: 0.0311716 | val_loss: 0.0311851 | Time: 2425.6 ms
[2021-07-16 22:36:13 main:581] : INFO : Epoch 1383 | loss: 0.0311751 | val_loss: 0.0311853 | Time: 2196 ms
[2021-07-16 22:36:15 main:581] : INFO : Epoch 1384 | loss: 0.0311752 | val_loss: 0.0311843 | Time: 2218.56 ms
[2021-07-16 22:36:18 main:581] : INFO : Epoch 1385 | loss: 0.0311737 | val_loss: 0.0311835 | Time: 2369.03 ms
[2021-07-16 22:36:20 main:581] : INFO : Epoch 1386 | loss: 0.0311715 | val_loss: 0.0311838 | Time: 2377.18 ms
[2021-07-16 22:36:23 main:581] : INFO : Epoch 1387 | loss: 0.0311708 | val_loss: 0.0311835 | Time: 2327.8 ms
[2021-07-16 22:36:25 main:581] : INFO : Epoch 1388 | loss: 0.0311699 | val_loss: 0.0311835 | Time: 2391.81 ms
[2021-07-16 22:36:27 main:581] : INFO : Epoch 1389 | loss: 0.0311698 | val_loss: 0.0311842 | Time: 2395.79 ms
[2021-07-16 22:36:30 main:581] : INFO : Epoch 1390 | loss: 0.0311696 | val_loss: 0.031189 | Time: 2389.03 ms
[2021-07-16 22:36:32 main:581] : INFO : Epoch 1391 | loss: 0.0311725 | val_loss: 0.0311851 | Time: 2281.65 ms
[2021-07-16 22:36:34 main:581] : INFO : Epoch 1392 | loss: 0.0311697 | val_loss: 0.0311843 | Time: 2244.98 ms
[2021-07-16 22:36:36 main:581] : INFO : Epoch 1393 | loss: 0.0311701 | val_loss: 0.0311855 | Time: 2165.59 ms
[2021-07-16 22:36:39 main:581] : INFO : Epoch 1394 | loss: 0.0311711 | val_loss: 0.0311902 | Time: 2318.79 ms
[2021-07-16 22:36:41 main:581] : INFO : Epoch 1395 | loss: 0.0311705 | val_loss: 0.0311847 | Time: 2499.09 ms
[2021-07-16 22:36:44 main:581] : INFO : Epoch 1396 | loss: 0.0311695 | val_loss: 0.031184 | Time: 2389.12 ms
[2021-07-16 22:36:46 main:581] : INFO : Epoch 1397 | loss: 0.0311698 | val_loss: 0.0311854 | Time: 2399.21 ms
[2021-07-16 22:36:48 main:581] : INFO : Epoch 1398 | loss: 0.0311697 | val_loss: 0.0311858 | Time: 2312.83 ms
[2021-07-16 22:36:51 main:581] : INFO : Epoch 1399 | loss: 0.0311698 | val_loss: 0.031184 | Time: 2340.45 ms
[2021-07-16 22:36:53 main:581] : INFO : Epoch 1400 | loss: 0.0311689 | val_loss: 0.0311831 | Time: 2409.09 ms
[2021-07-16 22:36:55 main:581] : INFO : Epoch 1401 | loss: 0.0311693 | val_loss: 0.031184 | Time: 2298.79 ms
[2021-07-16 22:36:58 main:581] : INFO : Epoch 1402 | loss: 0.0311704 | val_loss: 0.0311829 | Time: 2261.71 ms
[2021-07-16 22:37:00 main:581] : INFO : Epoch 1403 | loss: 0.0311703 | val_loss: 0.0311825 | Time: 2220.36 ms
[2021-07-16 22:37:02 main:581] : INFO : Epoch 1404 | loss: 0.0311692 | val_loss: 0.031185 | Time: 2365.16 ms
[2021-07-16 22:37:05 main:581] : INFO : Epoch 1405 | loss: 0.0311695 | val_loss: 0.0311843 | Time: 2318.82 ms
[2021-07-16 22:37:07 main:581] : INFO : Epoch 1406 | loss: 0.0311718 | val_loss: 0.0311869 | Time: 2227.74 ms
[2021-07-16 22:37:09 main:581] : INFO : Epoch 1407 | loss: 0.031169 | val_loss: 0.0311839 | Time: 2394.35 ms
[2021-07-16 22:37:12 main:581] : INFO : Epoch 1408 | loss: 0.0311694 | val_loss: 0.0311856 | Time: 2400.44 ms
[2021-07-16 22:37:14 main:581] : INFO : Epoch 1409 | loss: 0.0311701 | val_loss: 0.0311841 | Time: 2445.27 ms
[2021-07-16 22:37:16 main:581] : INFO : Epoch 1410 | loss: 0.0311698 | val_loss: 0.0311837 | Time: 2466.37 ms
[2021-07-16 22:37:19 main:581] : INFO : Epoch 1411 | loss: 0.0311693 | val_loss: 0.0311836 | Time: 2193.77 ms
[2021-07-16 22:37:21 main:581] : INFO : Epoch 1412 | loss: 0.0311684 | val_loss: 0.0311833 | Time: 2169.61 ms
[2021-07-16 22:37:23 main:581] : INFO : Epoch 1413 | loss: 0.0311697 | val_loss: 0.0311858 | Time: 2278.07 ms
[2021-07-16 22:37:26 main:581] : INFO : Epoch 1414 | loss: 0.0311699 | val_loss: 0.0311832 | Time: 2379.91 ms
[2021-07-16 22:37:27 main:581] : INFO : Epoch 1415 | loss: 0.0311691 | val_loss: 0.0311824 | Time: 1882.1 ms
[2021-07-16 22:37:29 main:581] : INFO : Epoch 1416 | loss: 0.0311696 | val_loss: 0.0311848 | Time: 1752 ms
[2021-07-16 22:37:31 main:581] : INFO : Epoch 1417 | loss: 0.0311687 | val_loss: 0.0311857 | Time: 1809.77 ms
[2021-07-16 22:37:33 main:581] : INFO : Epoch 1418 | loss: 0.0311684 | val_loss: 0.031184 | Time: 2012.57 ms
[2021-07-16 22:37:35 main:581] : INFO : Epoch 1419 | loss: 0.0311688 | val_loss: 0.0311884 | Time: 2265.55 ms
[2021-07-16 22:37:37 main:581] : INFO : Epoch 1420 | loss: 0.0311693 | val_loss: 0.0311838 | Time: 2103.5 ms
[2021-07-16 22:37:39 main:581] : INFO : Epoch 1421 | loss: 0.0311705 | val_loss: 0.0311836 | Time: 2129.08 ms
[2021-07-16 22:37:42 main:581] : INFO : Epoch 1422 | loss: 0.0311694 | val_loss: 0.0311827 | Time: 2204.18 ms
[2021-07-16 22:37:44 main:581] : INFO : Epoch 1423 | loss: 0.0311694 | val_loss: 0.0311894 | Time: 2198.26 ms
[2021-07-16 22:37:46 main:581] : INFO : Epoch 1424 | loss: 0.0311714 | val_loss: 0.0311926 | Time: 2234.49 ms
[2021-07-16 22:37:48 main:581] : INFO : Epoch 1425 | loss: 0.0311721 | val_loss: 0.0311837 | Time: 2301.38 ms
[2021-07-16 22:37:50 main:581] : INFO : Epoch 1426 | loss: 0.0311711 | val_loss: 0.0311861 | Time: 2045.12 ms
[2021-07-16 22:37:53 main:581] : INFO : Epoch 1427 | loss: 0.0311712 | val_loss: 0.0311844 | Time: 2138.53 ms
[2021-07-16 22:37:55 main:581] : INFO : Epoch 1428 | loss: 0.0311711 | val_loss: 0.0311836 | Time: 2198.54 ms
[2021-07-16 22:37:57 main:581] : INFO : Epoch 1429 | loss: 0.0311705 | val_loss: 0.0311836 | Time: 2253.17 ms
[2021-07-16 22:37:59 main:581] : INFO : Epoch 1430 | loss: 0.0311711 | val_loss: 0.0311852 | Time: 2064.25 ms
[2021-07-16 22:38:01 main:581] : INFO : Epoch 1431 | loss: 0.0311719 | val_loss: 0.0311842 | Time: 2167.03 ms
[2021-07-16 22:38:04 main:581] : INFO : Epoch 1432 | loss: 0.0311714 | val_loss: 0.0311846 | Time: 2328.17 ms
[2021-07-16 22:38:06 main:581] : INFO : Epoch 1433 | loss: 0.0311713 | val_loss: 0.0311855 | Time: 2281.66 ms
[2021-07-16 22:38:08 main:581] : INFO : Epoch 1434 | loss: 0.031171 | val_loss: 0.0311838 | Time: 2027.14 ms
[2021-07-16 22:38:10 main:581] : INFO : Epoch 1435 | loss: 0.0311708 | val_loss: 0.0311856 | Time: 2153.83 ms
[2021-07-16 22:38:12 main:581] : INFO : Epoch 1436 | loss: 0.0311709 | val_loss: 0.0311855 | Time: 2190.46 ms
[2021-07-16 22:38:14 main:581] : INFO : Epoch 1437 | loss: 0.0311708 | val_loss: 0.0311852 | Time: 2012.59 ms
[2021-07-16 22:38:16 main:581] : INFO : Epoch 1438 | loss: 0.0311697 | val_loss: 0.0311847 | Time: 1920.23 ms
[2021-07-16 22:38:18 main:581] : INFO : Epoch 1439 | loss: 0.0311694 | val_loss: 0.0311851 | Time: 2027.07 ms
[2021-07-16 22:38:20 main:581] : INFO : Epoch 1440 | loss: 0.0311692 | val_loss: 0.0311858 | Time: 2005.48 ms
[2021-07-16 22:38:22 main:581] : INFO : Epoch 1441 | loss: 0.0311687 | val_loss: 0.0311858 | Time: 2090.3 ms
[2021-07-16 22:38:25 main:581] : INFO : Epoch 1442 | loss: 0.0311701 | val_loss: 0.0311848 | Time: 2196.44 ms
[2021-07-16 22:38:27 main:581] : INFO : Epoch 1443 | loss: 0.0311689 | val_loss: 0.0311852 | Time: 2356.95 ms
[2021-07-16 22:38:29 main:581] : INFO : Epoch 1444 | loss: 0.0311743 | val_loss: 0.031187 | Time: 2317.29 ms
[2021-07-16 22:38:31 main:581] : INFO : Epoch 1445 | loss: 0.0311766 | val_loss: 0.0311918 | Time: 2206.99 ms
[2021-07-16 22:38:34 main:581] : INFO : Epoch 1446 | loss: 0.0311754 | val_loss: 0.0311853 | Time: 2205.94 ms
[2021-07-16 22:38:36 main:581] : INFO : Epoch 1447 | loss: 0.0311733 | val_loss: 0.0311846 | Time: 2007.1 ms
[2021-07-16 22:38:38 main:581] : INFO : Epoch 1448 | loss: 0.0311734 | val_loss: 0.0311846 | Time: 2099.6 ms
[2021-07-16 22:38:40 main:581] : INFO : Epoch 1449 | loss: 0.0311732 | val_loss: 0.0311852 | Time: 2174.77 ms
[2021-07-16 22:38:42 main:581] : INFO : Epoch 1450 | loss: 0.031172 | val_loss: 0.0311844 | Time: 2109.98 ms
[2021-07-16 22:38:44 main:581] : INFO : Epoch 1451 | loss: 0.0311717 | val_loss: 0.0311844 | Time: 2113.43 ms
[2021-07-16 22:38:46 main:581] : INFO : Epoch 1452 | loss: 0.0311751 | val_loss: 0.0311957 | Time: 2075.27 ms
[2021-07-16 22:38:48 main:581] : INFO : Epoch 1453 | loss: 0.0311754 | val_loss: 0.0311875 | Time: 1982.24 ms
[2021-07-16 22:38:50 main:581] : INFO : Epoch 1454 | loss: 0.0311731 | val_loss: 0.0311898 | Time: 2189.86 ms
[2021-07-16 22:38:53 main:581] : INFO : Epoch 1455 | loss: 0.0311713 | val_loss: 0.031187 | Time: 2208.49 ms
[2021-07-16 22:38:55 main:581] : INFO : Epoch 1456 | loss: 0.0311692 | val_loss: 0.0311851 | Time: 2305.13 ms
[2021-07-16 22:38:57 main:581] : INFO : Epoch 1457 | loss: 0.0311699 | val_loss: 0.0311862 | Time: 2263.06 ms
[2021-07-16 22:38:59 main:581] : INFO : Epoch 1458 | loss: 0.031171 | val_loss: 0.0311859 | Time: 2176.86 ms
[2021-07-16 22:39:02 main:581] : INFO : Epoch 1459 | loss: 0.0311706 | val_loss: 0.0311849 | Time: 2234.96 ms
[2021-07-16 22:39:04 main:581] : INFO : Epoch 1460 | loss: 0.0311699 | val_loss: 0.0311852 | Time: 2023.5 ms
[2021-07-16 22:39:06 main:581] : INFO : Epoch 1461 | loss: 0.0311704 | val_loss: 0.0311861 | Time: 2031.23 ms
[2021-07-16 22:39:08 main:581] : INFO : Epoch 1462 | loss: 0.0311711 | val_loss: 0.0311867 | Time: 2010.27 ms
[2021-07-16 22:39:10 main:581] : INFO : Epoch 1463 | loss: 0.0311696 | val_loss: 0.0311865 | Time: 2029.43 ms
[2021-07-16 22:39:12 main:581] : INFO : Epoch 1464 | loss: 0.0311687 | val_loss: 0.0311851 | Time: 2000.72 ms
[2021-07-16 22:39:14 main:581] : INFO : Epoch 1465 | loss: 0.0311686 | val_loss: 0.031185 | Time: 2325.01 ms
[2021-07-16 22:39:16 main:581] : INFO : Epoch 1466 | loss: 0.0311695 | val_loss: 0.031185 | Time: 2258.73 ms
[2021-07-16 22:39:18 main:581] : INFO : Epoch 1467 | loss: 0.0311692 | val_loss: 0.0311854 | Time: 2128.79 ms
[2021-07-16 22:39:21 main:581] : INFO : Epoch 1468 | loss: 0.0311688 | val_loss: 0.0311859 | Time: 2188.14 ms
[2021-07-16 22:39:23 main:581] : INFO : Epoch 1469 | loss: 0.0311687 | val_loss: 0.0311868 | Time: 2074.23 ms
[2021-07-16 22:39:25 main:581] : INFO : Epoch 1470 | loss: 0.0311706 | val_loss: 0.0311888 | Time: 1930.97 ms
[2021-07-16 22:39:26 main:581] : INFO : Epoch 1471 | loss: 0.0311696 | val_loss: 0.0311877 | Time: 1886.55 ms
[2021-07-16 22:39:28 main:581] : INFO : Epoch 1472 | loss: 0.0311709 | val_loss: 0.0311857 | Time: 1925.1 ms
[2021-07-16 22:39:30 main:581] : INFO : Epoch 1473 | loss: 0.0311697 | val_loss: 0.0311857 | Time: 2011.53 ms
[2021-07-16 22:39:33 main:581] : INFO : Epoch 1474 | loss: 0.0311707 | val_loss: 0.0311863 | Time: 2198.36 ms
[2021-07-16 22:39:35 main:581] : INFO : Epoch 1475 | loss: 0.03117 | val_loss: 0.0311897 | Time: 2158.69 ms
[2021-07-16 22:39:37 main:581] : INFO : Epoch 1476 | loss: 0.0311763 | val_loss: 0.0311891 | Time: 2011.87 ms
[2021-07-16 22:39:39 main:581] : INFO : Epoch 1477 | loss: 0.0311748 | val_loss: 0.0311849 | Time: 2081.02 ms
[2021-07-16 22:39:41 main:581] : INFO : Epoch 1478 | loss: 0.0311718 | val_loss: 0.0311827 | Time: 2212.87 ms
[2021-07-16 22:39:43 main:581] : INFO : Epoch 1479 | loss: 0.0311695 | val_loss: 0.0311839 | Time: 1933.36 ms
[2021-07-16 22:39:45 main:581] : INFO : Epoch 1480 | loss: 0.0311693 | val_loss: 0.0311838 | Time: 1906.67 ms
[2021-07-16 22:39:47 main:581] : INFO : Epoch 1481 | loss: 0.031169 | val_loss: 0.0311853 | Time: 1885.51 ms
[2021-07-16 22:39:49 main:581] : INFO : Epoch 1482 | loss: 0.0311705 | val_loss: 0.0311862 | Time: 1971.53 ms
[2021-07-16 22:39:51 main:581] : INFO : Epoch 1483 | loss: 0.0311694 | val_loss: 0.0311872 | Time: 2272.14 ms
[2021-07-16 22:39:53 main:581] : INFO : Epoch 1484 | loss: 0.0311697 | val_loss: 0.0311867 | Time: 2285.52 ms
[2021-07-16 22:39:55 main:581] : INFO : Epoch 1485 | loss: 0.031172 | val_loss: 0.0311899 | Time: 2090.67 ms
[2021-07-16 22:39:57 main:581] : INFO : Epoch 1486 | loss: 0.0311708 | val_loss: 0.0311833 | Time: 2018.87 ms
[2021-07-16 22:39:59 main:581] : INFO : Epoch 1487 | loss: 0.0311693 | val_loss: 0.0311846 | Time: 2042.71 ms
[2021-07-16 22:40:01 main:581] : INFO : Epoch 1488 | loss: 0.0311697 | val_loss: 0.0311862 | Time: 2006.83 ms
[2021-07-16 22:40:03 main:581] : INFO : Epoch 1489 | loss: 0.0311695 | val_loss: 0.0311842 | Time: 1984.1 ms
[2021-07-16 22:40:06 main:581] : INFO : Epoch 1490 | loss: 0.0311724 | val_loss: 0.031185 | Time: 2055.84 ms
[2021-07-16 22:40:08 main:581] : INFO : Epoch 1491 | loss: 0.0311697 | val_loss: 0.0311853 | Time: 2076.69 ms
[2021-07-16 22:40:10 main:581] : INFO : Epoch 1492 | loss: 0.0311691 | val_loss: 0.0311852 | Time: 2160.25 ms
[2021-07-16 22:40:12 main:581] : INFO : Epoch 1493 | loss: 0.0311689 | val_loss: 0.0311862 | Time: 2318.9 ms
[2021-07-16 22:40:14 main:581] : INFO : Epoch 1494 | loss: 0.03117 | val_loss: 0.0311852 | Time: 2205.4 ms
[2021-07-16 22:40:16 main:581] : INFO : Epoch 1495 | loss: 0.0311702 | val_loss: 0.0311873 | Time: 2039.71 ms
[2021-07-16 22:40:18 main:581] : INFO : Epoch 1496 | loss: 0.0311683 | val_loss: 0.0311848 | Time: 1958.23 ms
[2021-07-16 22:40:20 main:581] : INFO : Epoch 1497 | loss: 0.0311697 | val_loss: 0.0311859 | Time: 2118.15 ms
[2021-07-16 22:40:22 main:581] : INFO : Epoch 1498 | loss: 0.0311686 | val_loss: 0.0311837 | Time: 1925.22 ms
[2021-07-16 22:40:24 main:581] : INFO : Epoch 1499 | loss: 0.0311689 | val_loss: 0.0311849 | Time: 2018.36 ms
[2021-07-16 22:40:26 main:581] : INFO : Epoch 1500 | loss: 0.031169 | val_loss: 0.0311851 | Time: 2054.44 ms
[2021-07-16 22:40:29 main:581] : INFO : Epoch 1501 | loss: 0.0311695 | val_loss: 0.0311847 | Time: 2246.85 ms
[2021-07-16 22:40:31 main:581] : INFO : Epoch 1502 | loss: 0.0311741 | val_loss: 0.0311842 | Time: 2231.95 ms
[2021-07-16 22:40:33 main:581] : INFO : Epoch 1503 | loss: 0.0311725 | val_loss: 0.0311853 | Time: 2204.74 ms
[2021-07-16 22:40:35 main:581] : INFO : Epoch 1504 | loss: 0.0311704 | val_loss: 0.0311854 | Time: 2198.29 ms
[2021-07-16 22:40:37 main:581] : INFO : Epoch 1505 | loss: 0.0311699 | val_loss: 0.0311841 | Time: 2047.41 ms
[2021-07-16 22:40:39 main:581] : INFO : Epoch 1506 | loss: 0.0311689 | val_loss: 0.0311859 | Time: 1974.55 ms
[2021-07-16 22:40:41 main:581] : INFO : Epoch 1507 | loss: 0.0311696 | val_loss: 0.0311846 | Time: 2072.24 ms
[2021-07-16 22:40:43 main:581] : INFO : Epoch 1508 | loss: 0.0311702 | val_loss: 0.0311884 | Time: 2016.99 ms
[2021-07-16 22:40:45 main:581] : INFO : Epoch 1509 | loss: 0.0311709 | val_loss: 0.0311856 | Time: 2001.29 ms
[2021-07-16 22:40:47 main:581] : INFO : Epoch 1510 | loss: 0.0311702 | val_loss: 0.0311852 | Time: 2001.89 ms
[2021-07-16 22:40:50 main:581] : INFO : Epoch 1511 | loss: 0.0311682 | val_loss: 0.0311849 | Time: 2163.05 ms
[2021-07-16 22:40:52 main:581] : INFO : Epoch 1512 | loss: 0.0311683 | val_loss: 0.0311864 | Time: 2058.15 ms
[2021-07-16 22:40:54 main:581] : INFO : Epoch 1513 | loss: 0.0311698 | val_loss: 0.0311852 | Time: 2044.28 ms
[2021-07-16 22:40:56 main:581] : INFO : Epoch 1514 | loss: 0.0311701 | val_loss: 0.0311877 | Time: 2110.52 ms
[2021-07-16 22:40:58 main:581] : INFO : Epoch 1515 | loss: 0.0311707 | val_loss: 0.0311852 | Time: 2126.59 ms
[2021-07-16 22:41:00 main:581] : INFO : Epoch 1516 | loss: 0.0311686 | val_loss: 0.0311854 | Time: 2074.5 ms
[2021-07-16 22:41:02 main:581] : INFO : Epoch 1517 | loss: 0.0311679 | val_loss: 0.0311857 | Time: 2047.5 ms
[2021-07-16 22:41:04 main:581] : INFO : Epoch 1518 | loss: 0.0311683 | val_loss: 0.0311855 | Time: 2147.08 ms
[2021-07-16 22:41:06 main:581] : INFO : Epoch 1519 | loss: 0.0311673 | val_loss: 0.0311859 | Time: 2009.97 ms
[2021-07-16 22:41:08 main:581] : INFO : Epoch 1520 | loss: 0.0311677 | val_loss: 0.0311883 | Time: 1921.83 ms
[2021-07-16 22:41:10 main:581] : INFO : Epoch 1521 | loss: 0.0311688 | val_loss: 0.0311853 | Time: 2141.08 ms
[2021-07-16 22:41:12 main:581] : INFO : Epoch 1522 | loss: 0.0311687 | val_loss: 0.0311863 | Time: 2127.52 ms
[2021-07-16 22:41:15 main:581] : INFO : Epoch 1523 | loss: 0.0311674 | val_loss: 0.0311848 | Time: 2156.38 ms
[2021-07-16 22:41:17 main:581] : INFO : Epoch 1524 | loss: 0.0311679 | val_loss: 0.0311845 | Time: 2034.27 ms
[2021-07-16 22:41:19 main:581] : INFO : Epoch 1525 | loss: 0.0311682 | val_loss: 0.0311847 | Time: 2095.52 ms
[2021-07-16 22:41:21 main:581] : INFO : Epoch 1526 | loss: 0.0311674 | val_loss: 0.0311856 | Time: 2010.87 ms
[2021-07-16 22:41:23 main:581] : INFO : Epoch 1527 | loss: 0.0311681 | val_loss: 0.031185 | Time: 2181.44 ms
[2021-07-16 22:41:25 main:581] : INFO : Epoch 1528 | loss: 0.0311676 | val_loss: 0.0311853 | Time: 2163.35 ms
[2021-07-16 22:41:27 main:581] : INFO : Epoch 1529 | loss: 0.0311685 | val_loss: 0.0311869 | Time: 1995.23 ms
[2021-07-16 22:41:29 main:581] : INFO : Epoch 1530 | loss: 0.0311679 | val_loss: 0.0311836 | Time: 2033.07 ms
[2021-07-16 22:41:31 main:581] : INFO : Epoch 1531 | loss: 0.0311697 | val_loss: 0.0311876 | Time: 2078.01 ms
[2021-07-16 22:41:33 main:581] : INFO : Epoch 1532 | loss: 0.0311692 | val_loss: 0.0311854 | Time: 2131.8 ms
[2021-07-16 22:41:35 main:581] : INFO : Epoch 1533 | loss: 0.0311684 | val_loss: 0.0311832 | Time: 2094.6 ms
[2021-07-16 22:41:37 main:581] : INFO : Epoch 1534 | loss: 0.0311685 | val_loss: 0.0311854 | Time: 2119.72 ms
[2021-07-16 22:41:40 main:581] : INFO : Epoch 1535 | loss: 0.03117 | val_loss: 0.0311857 | Time: 2031.6 ms
[2021-07-16 22:41:41 main:581] : INFO : Epoch 1536 | loss: 0.0311692 | val_loss: 0.0311895 | Time: 1983.62 ms
[2021-07-16 22:41:44 main:581] : INFO : Epoch 1537 | loss: 0.0311706 | val_loss: 0.0311836 | Time: 2095.86 ms
[2021-07-16 22:41:46 main:581] : INFO : Epoch 1538 | loss: 0.0311688 | val_loss: 0.0311847 | Time: 2174.39 ms
[2021-07-16 22:41:48 main:581] : INFO : Epoch 1539 | loss: 0.0311673 | val_loss: 0.031185 | Time: 2008.69 ms
[2021-07-16 22:41:50 main:581] : INFO : Epoch 1540 | loss: 0.0311675 | val_loss: 0.0311854 | Time: 2072.72 ms
[2021-07-16 22:41:52 main:581] : INFO : Epoch 1541 | loss: 0.03117 | val_loss: 0.0311851 | Time: 2146.31 ms
[2021-07-16 22:41:54 main:581] : INFO : Epoch 1542 | loss: 0.0311702 | val_loss: 0.0311866 | Time: 2106.61 ms
[2021-07-16 22:41:56 main:581] : INFO : Epoch 1543 | loss: 0.0311688 | val_loss: 0.0311857 | Time: 1997.18 ms
[2021-07-16 22:41:58 main:581] : INFO : Epoch 1544 | loss: 0.0311676 | val_loss: 0.0311851 | Time: 2150.11 ms
[2021-07-16 22:42:00 main:581] : INFO : Epoch 1545 | loss: 0.0311685 | val_loss: 0.031186 | Time: 2184.34 ms
[2021-07-16 22:42:03 main:581] : INFO : Epoch 1546 | loss: 0.031168 | val_loss: 0.031185 | Time: 2068.39 ms
[2021-07-16 22:42:04 main:581] : INFO : Epoch 1547 | loss: 0.0311688 | val_loss: 0.0311851 | Time: 1992.68 ms
[2021-07-16 22:42:07 main:581] : INFO : Epoch 1548 | loss: 0.0311704 | val_loss: 0.0311868 | Time: 2098.96 ms
[2021-07-16 22:42:09 main:581] : INFO : Epoch 1549 | loss: 0.0311691 | val_loss: 0.03119 | Time: 2178.47 ms
[2021-07-16 22:42:11 main:581] : INFO : Epoch 1550 | loss: 0.0311691 | val_loss: 0.0311851 | Time: 2540.7 ms
[2021-07-16 22:42:13 main:581] : INFO : Epoch 1551 | loss: 0.0311683 | val_loss: 0.0311851 | Time: 2161.37 ms
[2021-07-16 22:42:16 main:581] : INFO : Epoch 1552 | loss: 0.031173 | val_loss: 0.0311917 | Time: 2092.33 ms
[2021-07-16 22:42:18 main:581] : INFO : Epoch 1553 | loss: 0.0311704 | val_loss: 0.0311846 | Time: 2080.38 ms
[2021-07-16 22:42:20 main:581] : INFO : Epoch 1554 | loss: 0.031169 | val_loss: 0.0311883 | Time: 2292.19 ms
[2021-07-16 22:42:22 main:581] : INFO : Epoch 1555 | loss: 0.0311726 | val_loss: 0.0311934 | Time: 2433.34 ms
[2021-07-16 22:42:25 main:581] : INFO : Epoch 1556 | loss: 0.0311721 | val_loss: 0.0311879 | Time: 2319.54 ms
[2021-07-16 22:42:27 main:581] : INFO : Epoch 1557 | loss: 0.0311676 | val_loss: 0.0311855 | Time: 2135.07 ms
[2021-07-16 22:42:29 main:581] : INFO : Epoch 1558 | loss: 0.0311689 | val_loss: 0.0311862 | Time: 2116.84 ms
[2021-07-16 22:42:31 main:581] : INFO : Epoch 1559 | loss: 0.0311676 | val_loss: 0.0311863 | Time: 2344.71 ms
[2021-07-16 22:42:33 main:581] : INFO : Epoch 1560 | loss: 0.0311679 | val_loss: 0.0311938 | Time: 2168.01 ms
[2021-07-16 22:42:36 main:581] : INFO : Epoch 1561 | loss: 0.031169 | val_loss: 0.0311856 | Time: 2130.24 ms
[2021-07-16 22:42:38 main:581] : INFO : Epoch 1562 | loss: 0.031168 | val_loss: 0.0311861 | Time: 2042.13 ms
[2021-07-16 22:42:40 main:581] : INFO : Epoch 1563 | loss: 0.0311679 | val_loss: 0.0311845 | Time: 2307.72 ms
[2021-07-16 22:42:42 main:581] : INFO : Epoch 1564 | loss: 0.0311715 | val_loss: 0.0311852 | Time: 2374.04 ms
[2021-07-16 22:42:45 main:581] : INFO : Epoch 1565 | loss: 0.0311714 | val_loss: 0.0311875 | Time: 2399.22 ms
[2021-07-16 22:42:47 main:581] : INFO : Epoch 1566 | loss: 0.031173 | val_loss: 0.0311835 | Time: 2330.57 ms
[2021-07-16 22:42:49 main:581] : INFO : Epoch 1567 | loss: 0.0311715 | val_loss: 0.0311844 | Time: 2020.27 ms
[2021-07-16 22:42:51 main:581] : INFO : Epoch 1568 | loss: 0.031171 | val_loss: 0.0311852 | Time: 2080.16 ms
[2021-07-16 22:42:53 main:581] : INFO : Epoch 1569 | loss: 0.0311685 | val_loss: 0.0311841 | Time: 2330.12 ms
[2021-07-16 22:42:56 main:581] : INFO : Epoch 1570 | loss: 0.0311695 | val_loss: 0.0311845 | Time: 2255.28 ms
[2021-07-16 22:42:58 main:581] : INFO : Epoch 1571 | loss: 0.0311684 | val_loss: 0.0311845 | Time: 2103.63 ms
[2021-07-16 22:43:00 main:581] : INFO : Epoch 1572 | loss: 0.0311697 | val_loss: 0.0311875 | Time: 2312.6 ms
[2021-07-16 22:43:03 main:581] : INFO : Epoch 1573 | loss: 0.0311705 | val_loss: 0.031184 | Time: 2347.58 ms
[2021-07-16 22:43:05 main:581] : INFO : Epoch 1574 | loss: 0.031168 | val_loss: 0.0311839 | Time: 2323.74 ms
[2021-07-16 22:43:07 main:581] : INFO : Epoch 1575 | loss: 0.0311677 | val_loss: 0.0311846 | Time: 2205.35 ms
[2021-07-16 22:43:09 main:581] : INFO : Epoch 1576 | loss: 0.0311686 | val_loss: 0.0311896 | Time: 2167.9 ms
[2021-07-16 22:43:11 main:581] : INFO : Epoch 1577 | loss: 0.0311694 | val_loss: 0.031185 | Time: 2083.07 ms
[2021-07-16 22:43:14 main:581] : INFO : Epoch 1578 | loss: 0.0311683 | val_loss: 0.0311859 | Time: 2313.87 ms
[2021-07-16 22:43:16 main:581] : INFO : Epoch 1579 | loss: 0.0311682 | val_loss: 0.0311855 | Time: 2213.47 ms
[2021-07-16 22:43:18 main:581] : INFO : Epoch 1580 | loss: 0.0311685 | val_loss: 0.0311877 | Time: 2244.38 ms
[2021-07-16 22:43:20 main:581] : INFO : Epoch 1581 | loss: 0.0311693 | val_loss: 0.0311904 | Time: 2252.59 ms
[2021-07-16 22:43:22 main:581] : INFO : Epoch 1582 | loss: 0.031171 | val_loss: 0.0311901 | Time: 2187.92 ms
[2021-07-16 22:43:25 main:581] : INFO : Epoch 1583 | loss: 0.031169 | val_loss: 0.0311853 | Time: 2090.88 ms
[2021-07-16 22:43:27 main:581] : INFO : Epoch 1584 | loss: 0.0311731 | val_loss: 0.0311886 | Time: 2213.36 ms
[2021-07-16 22:43:29 main:581] : INFO : Epoch 1585 | loss: 0.0311707 | val_loss: 0.0311876 | Time: 2205.44 ms
[2021-07-16 22:43:31 main:581] : INFO : Epoch 1586 | loss: 0.0311677 | val_loss: 0.0311858 | Time: 2195.85 ms
[2021-07-16 22:43:33 main:581] : INFO : Epoch 1587 | loss: 0.0311676 | val_loss: 0.0311914 | Time: 2172.18 ms
[2021-07-16 22:43:36 main:581] : INFO : Epoch 1588 | loss: 0.0311683 | val_loss: 0.0311859 | Time: 2225.89 ms
[2021-07-16 22:43:38 main:581] : INFO : Epoch 1589 | loss: 0.0311683 | val_loss: 0.0311887 | Time: 2085.97 ms
[2021-07-16 22:43:40 main:581] : INFO : Epoch 1590 | loss: 0.0311696 | val_loss: 0.0311859 | Time: 2234.94 ms
[2021-07-16 22:43:42 main:581] : INFO : Epoch 1591 | loss: 0.0311732 | val_loss: 0.0311852 | Time: 2223.34 ms
[2021-07-16 22:43:44 main:581] : INFO : Epoch 1592 | loss: nan | val_loss: nan | Time: 2099.23 ms
[2021-07-16 22:43:46 main:581] : INFO : Epoch 1593 | loss: nan | val_loss: nan | Time: 2164.95 ms
[2021-07-16 22:43:49 main:581] : INFO : Epoch 1594 | loss: nan | val_loss: nan | Time: 2176.71 ms
[2021-07-16 22:43:51 main:581] : INFO : Epoch 1595 | loss: nan | val_loss: nan | Time: 2239.7 ms
[2021-07-16 22:43:53 main:581] : INFO : Epoch 1596 | loss: nan | val_loss: nan | Time: 2156.83 ms
[2021-07-16 22:43:55 main:581] : INFO : Epoch 1597 | loss: nan | val_loss: nan | Time: 2289.8 ms
[2021-07-16 22:43:57 main:581] : INFO : Epoch 1598 | loss: nan | val_loss: nan | Time: 2190.98 ms
[2021-07-16 22:44:00 main:581] : INFO : Epoch 1599 | loss: nan | val_loss: nan | Time: 2249.93 ms
[2021-07-16 22:44:02 main:581] : INFO : Epoch 1600 | loss: nan | val_loss: nan | Time: 2087.1 ms
[2021-07-16 22:44:04 main:581] : INFO : Epoch 1601 | loss: nan | val_loss: nan | Time: 2137.45 ms
[2021-07-16 22:44:06 main:581] : INFO : Epoch 1602 | loss: nan | val_loss: nan | Time: 2097.3 ms
[2021-07-16 22:44:08 main:581] : INFO : Epoch 1603 | loss: nan | val_loss: nan | Time: 2250.13 ms
[2021-07-16 22:44:11 main:581] : INFO : Epoch 1604 | loss: nan | val_loss: nan | Time: 2270.52 ms
[2021-07-16 22:44:13 main:581] : INFO : Epoch 1605 | loss: nan | val_loss: nan | Time: 2307.74 ms
[2021-07-16 22:44:15 main:581] : INFO : Epoch 1606 | loss: nan | val_loss: nan | Time: 2234.03 ms
[2021-07-16 22:44:17 main:581] : INFO : Epoch 1607 | loss: nan | val_loss: nan | Time: 2147.1 ms
[2021-07-16 22:44:19 main:581] : INFO : Epoch 1608 | loss: nan | val_loss: nan | Time: 2169.9 ms
[2021-07-16 22:44:22 main:581] : INFO : Epoch 1609 | loss: nan | val_loss: nan | Time: 2086.32 ms
[2021-07-16 22:44:24 main:581] : INFO : Epoch 1610 | loss: nan | val_loss: nan | Time: 2251.94 ms
[2021-07-16 22:44:26 main:581] : INFO : Epoch 1611 | loss: nan | val_loss: nan | Time: 2079.4 ms
[2021-07-16 22:44:28 main:581] : INFO : Epoch 1612 | loss: nan | val_loss: nan | Time: 2150.08 ms
[2021-07-16 22:44:30 main:581] : INFO : Epoch 1613 | loss: nan | val_loss: nan | Time: 2228.18 ms
[2021-07-16 22:44:32 main:581] : INFO : Epoch 1614 | loss: nan | val_loss: nan | Time: 2267.75 ms
[2021-07-16 22:44:35 main:581] : INFO : Epoch 1615 | loss: nan | val_loss: nan | Time: 2158.09 ms
[2021-07-16 22:44:37 main:581] : INFO : Epoch 1616 | loss: nan | val_loss: nan | Time: 2235.09 ms
[2021-07-16 22:44:39 main:581] : INFO : Epoch 1617 | loss: nan | val_loss: nan | Time: 2372.39 ms
[2021-07-16 22:44:41 main:581] : INFO : Epoch 1618 | loss: nan | val_loss: nan | Time: 2148.68 ms
[2021-07-16 22:44:44 main:581] : INFO : Epoch 1619 | loss: nan | val_loss: nan | Time: 2104.34 ms
[2021-07-16 22:44:46 main:581] : INFO : Epoch 1620 | loss: nan | val_loss: nan | Time: 2105.61 ms
[2021-07-16 22:44:48 main:581] : INFO : Epoch 1621 | loss: nan | val_loss: nan | Time: 2078.09 ms
[2021-07-16 22:44:50 main:581] : INFO : Epoch 1622 | loss: nan | val_loss: nan | Time: 2306.79 ms
[2021-07-16 22:44:52 main:581] : INFO : Epoch 1623 | loss: nan | val_loss: nan | Time: 2315.57 ms
[2021-07-16 22:44:54 main:581] : INFO : Epoch 1624 | loss: nan | val_loss: nan | Time: 2178.21 ms
[2021-07-16 22:44:57 main:581] : INFO : Epoch 1625 | loss: nan | val_loss: nan | Time: 2176.21 ms
[2021-07-16 22:44:59 main:581] : INFO : Epoch 1626 | loss: nan | val_loss: nan | Time: 2110.67 ms
[2021-07-16 22:45:01 main:581] : INFO : Epoch 1627 | loss: nan | val_loss: nan | Time: 2220.97 ms
[2021-07-16 22:45:03 main:581] : INFO : Epoch 1628 | loss: nan | val_loss: nan | Time: 2223.42 ms
[2021-07-16 22:45:05 main:581] : INFO : Epoch 1629 | loss: nan | val_loss: nan | Time: 2132.21 ms
[2021-07-16 22:45:07 main:581] : INFO : Epoch 1630 | loss: nan | val_loss: nan | Time: 2027.93 ms
[2021-07-16 22:45:10 main:581] : INFO : Epoch 1631 | loss: nan | val_loss: nan | Time: 2335 ms
[2021-07-16 22:45:12 main:581] : INFO : Epoch 1632 | loss: nan | val_loss: nan | Time: 2387.36 ms
[2021-07-16 22:45:14 main:581] : INFO : Epoch 1633 | loss: nan | val_loss: nan | Time: 2350.46 ms
[2021-07-16 22:45:17 main:581] : INFO : Epoch 1634 | loss: nan | val_loss: nan | Time: 2321.37 ms
[2021-07-16 22:45:19 main:581] : INFO : Epoch 1635 | loss: nan | val_loss: nan | Time: 2057.91 ms
[2021-07-16 22:45:21 main:581] : INFO : Epoch 1636 | loss: nan | val_loss: nan | Time: 2055.75 ms
[2021-07-16 22:45:23 main:581] : INFO : Epoch 1637 | loss: nan | val_loss: nan | Time: 2172.87 ms
[2021-07-16 22:45:25 main:581] : INFO : Epoch 1638 | loss: nan | val_loss: nan | Time: 2062.1 ms
[2021-07-16 22:45:27 main:581] : INFO : Epoch 1639 | loss: nan | val_loss: nan | Time: 2080.28 ms
[2021-07-16 22:45:29 main:581] : INFO : Epoch 1640 | loss: nan | val_loss: nan | Time: 2176.91 ms
[2021-07-16 22:45:32 main:581] : INFO : Epoch 1641 | loss: nan | val_loss: nan | Time: 2324.08 ms
[2021-07-16 22:45:34 main:581] : INFO : Epoch 1642 | loss: nan | val_loss: nan | Time: 2331.96 ms
[2021-07-16 22:45:36 main:581] : INFO : Epoch 1643 | loss: nan | val_loss: nan | Time: 2175.79 ms
[2021-07-16 22:45:38 main:581] : INFO : Epoch 1644 | loss: nan | val_loss: nan | Time: 2127.05 ms
[2021-07-16 22:45:40 main:581] : INFO : Epoch 1645 | loss: nan | val_loss: nan | Time: 2056.49 ms
[2021-07-16 22:45:43 main:581] : INFO : Epoch 1646 | loss: nan | val_loss: nan | Time: 2256.23 ms
[2021-07-16 22:45:45 main:581] : INFO : Epoch 1647 | loss: nan | val_loss: nan | Time: 2088.92 ms
[2021-07-16 22:45:47 main:581] : INFO : Epoch 1648 | loss: nan | val_loss: nan | Time: 2210.77 ms
[2021-07-16 22:45:49 main:581] : INFO : Epoch 1649 | loss: nan | val_loss: nan | Time: 2288.51 ms
[2021-07-16 22:45:51 main:581] : INFO : Epoch 1650 | loss: nan | val_loss: nan | Time: 2188.35 ms
[2021-07-16 22:45:54 main:581] : INFO : Epoch 1651 | loss: nan | val_loss: nan | Time: 2212.94 ms
[2021-07-16 22:45:56 main:581] : INFO : Epoch 1652 | loss: nan | val_loss: nan | Time: 2191.48 ms
[2021-07-16 22:45:58 main:581] : INFO : Epoch 1653 | loss: nan | val_loss: nan | Time: 2076.99 ms
[2021-07-16 22:46:00 main:581] : INFO : Epoch 1654 | loss: nan | val_loss: nan | Time: 2156.85 ms
[2021-07-16 22:46:02 main:581] : INFO : Epoch 1655 | loss: nan | val_loss: nan | Time: 2251.56 ms
[2021-07-16 22:46:04 main:581] : INFO : Epoch 1656 | loss: nan | val_loss: nan | Time: 2090.08 ms
[2021-07-16 22:46:07 main:581] : INFO : Epoch 1657 | loss: nan | val_loss: nan | Time: 2151.29 ms
[2021-07-16 22:46:09 main:581] : INFO : Epoch 1658 | loss: nan | val_loss: nan | Time: 2187.16 ms
[2021-07-16 22:46:11 main:581] : INFO : Epoch 1659 | loss: nan | val_loss: nan | Time: 2161.22 ms
[2021-07-16 22:46:13 main:581] : INFO : Epoch 1660 | loss: nan | val_loss: nan | Time: 2259.03 ms
[2021-07-16 22:46:15 main:581] : INFO : Epoch 1661 | loss: nan | val_loss: nan | Time: 2136.81 ms
[2021-07-16 22:46:17 main:581] : INFO : Epoch 1662 | loss: nan | val_loss: nan | Time: 2100.5 ms
[2021-07-16 22:46:20 main:581] : INFO : Epoch 1663 | loss: nan | val_loss: nan | Time: 2154.53 ms
[2021-07-16 22:46:22 main:581] : INFO : Epoch 1664 | loss: nan | val_loss: nan | Time: 2133.45 ms
[2021-07-16 22:46:24 main:581] : INFO : Epoch 1665 | loss: nan | val_loss: nan | Time: 2137.86 ms
[2021-07-16 22:46:26 main:581] : INFO : Epoch 1666 | loss: nan | val_loss: nan | Time: 2283.59 ms
[2021-07-16 22:46:28 main:581] : INFO : Epoch 1667 | loss: nan | val_loss: nan | Time: 2327.49 ms
[2021-07-16 22:46:31 main:581] : INFO : Epoch 1668 | loss: nan | val_loss: nan | Time: 2083.93 ms
[2021-07-16 22:46:33 main:581] : INFO : Epoch 1669 | loss: nan | val_loss: nan | Time: 2180.31 ms
[2021-07-16 22:46:35 main:581] : INFO : Epoch 1670 | loss: nan | val_loss: nan | Time: 2212.31 ms
[2021-07-16 22:46:37 main:581] : INFO : Epoch 1671 | loss: nan | val_loss: nan | Time: 2100.9 ms
[2021-07-16 22:46:39 main:581] : INFO : Epoch 1672 | loss: nan | val_loss: nan | Time: 2255.38 ms
[2021-07-16 22:46:41 main:581] : INFO : Epoch 1673 | loss: nan | val_loss: nan | Time: 2058.51 ms
[2021-07-16 22:46:43 main:581] : INFO : Epoch 1674 | loss: nan | val_loss: nan | Time: 2105.94 ms
[2021-07-16 22:46:46 main:581] : INFO : Epoch 1675 | loss: nan | val_loss: nan | Time: 2233.3 ms
[2021-07-16 22:46:48 main:581] : INFO : Epoch 1676 | loss: nan | val_loss: nan | Time: 2257.03 ms
[2021-07-16 22:46:50 main:581] : INFO : Epoch 1677 | loss: nan | val_loss: nan | Time: 2240.96 ms
[2021-07-16 22:46:52 main:581] : INFO : Epoch 1678 | loss: nan | val_loss: nan | Time: 2250.79 ms
[2021-07-16 22:46:55 main:581] : INFO : Epoch 1679 | loss: nan | val_loss: nan | Time: 2150.06 ms
[2021-07-16 22:46:57 main:581] : INFO : Epoch 1680 | loss: nan | val_loss: nan | Time: 2120.95 ms
[2021-07-16 22:46:59 main:581] : INFO : Epoch 1681 | loss: nan | val_loss: nan | Time: 2074.1 ms
[2021-07-16 22:47:01 main:581] : INFO : Epoch 1682 | loss: nan | val_loss: nan | Time: 2158.46 ms
[2021-07-16 22:47:03 main:581] : INFO : Epoch 1683 | loss: nan | val_loss: nan | Time: 2170.19 ms
[2021-07-16 22:47:05 main:581] : INFO : Epoch 1684 | loss: nan | val_loss: nan | Time: 2076.31 ms
[2021-07-16 22:47:07 main:581] : INFO : Epoch 1685 | loss: nan | val_loss: nan | Time: 2111.72 ms
[2021-07-16 22:47:10 main:581] : INFO : Epoch 1686 | loss: nan | val_loss: nan | Time: 2222.87 ms
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x55e5225baf9c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f4d16944730]
/lib/x86_64-linux-gnu/libc.so.6(+0x39d57)[0x7f4c9825ed57]
/lib/x86_64-linux-gnu/libc.so.6(+0x39eba)[0x7f4c9825eeba]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379b25)[0x55e5225b6b25]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379bee)[0x55e5225b6bee]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f4d16944730]
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_wait+0x1fa)[0x7f4d1694000a]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(_ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE+0xc)[0x55e522651abc]
./libtorch_cpu.so(+0x275d0c3)[0x7f4d12f3d0c3]
./libtorch_cpu.so(_ZN5torch8autograd6Engine11thread_mainERKSt10shared_ptrINS0_9GraphTaskEE+0xfc)[0x7f4d12f41c8c]
./libtorch_cpu.so(_ZN5torch8autograd6Engine23execute_with_graph_taskERKSt10shared_ptrINS0_9GraphTaskEES2_INS0_4NodeEE+0x490)[0x7f4d12f3f7c0]
./libtorch_cpu.so(_ZN5torch8autograd6Engine7executeERKSt6vectorINS0_4EdgeESaIS3_EERKS2_IN2at6TensorESaIS9_EEbbS7_+0x82f)[0x7f4d12f3e9af]
./libtorch_cpu.so(+0x274c935)[0x7f4d12f2c935]
./libtorch_cpu.so(_ZN5torch8autograd8backwardERKSt6vectorIN2at6TensorESaIS3_EES7_N3c108optionalIbEEb+0x87)[0x7f4d12f2d8b7]
./libtorch_cpu.so(+0x2c15b10)[0x7f4d133f5b10]
./libtorch_cpu.so(+0x2407072)[0x7f4d12be7072]
./libtorch_cpu.so(_ZNK2at6Tensor8backwardERKS0_N3c108optionalIbEEb+0x155)[0x7f4d11743065]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x9a27f)[0x55e5222d727f]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8ad10)[0x55e5222c7d10]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f4c9824909b]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x55e5222c375a]
Exiting...
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: GeForce RTX 2080 Ti)
[2021-07-17 00:14:22 main:442] : INFO : Set logging level to 1
[2021-07-17 00:14:22 main:448] : INFO : Running in BOINC Client mode
[2021-07-17 00:14:22 main:451] : INFO : Resolving all filenames
[2021-07-17 00:14:22 main:459] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 (exists = 1)
[2021-07-17 00:14:22 main:459] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1625432385-20358-1_0_r700848754_1 (exists = 1)
[2021-07-17 00:14:22 main:459] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1625432385-20358-1_0_r700848754_0 (exists = 0)
[2021-07-17 00:14:22 main:459] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1625432385-20358-1 (exists = 1)
[2021-07-17 00:14:22 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2021-07-17 00:14:22 main:479] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5
[2021-07-17 00:14:22 main:481] : INFO : Configuration:
[2021-07-17 00:14:22 main:482] : INFO : Model type: GRU
[2021-07-17 00:14:22 main:483] : INFO : Validation Loss Threshold: 0.0001
[2021-07-17 00:14:22 main:484] : INFO : Max Epochs: 2048
[2021-07-17 00:14:22 main:485] : INFO : Batch Size: 128
[2021-07-17 00:14:22 main:486] : INFO : Learning Rate: 0.01
[2021-07-17 00:14:22 main:487] : INFO : Patience: 10
[2021-07-17 00:14:22 main:488] : INFO : Hidden Width: 12
[2021-07-17 00:14:22 main:489] : INFO : # Recurrent Layers: 4
[2021-07-17 00:14:22 main:490] : INFO : # Backend Layers: 4
[2021-07-17 00:14:22 main:491] : INFO : # Threads: 1
[2021-07-17 00:14:22 main:493] : INFO : Preparing Dataset
[2021-07-17 00:14:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-17 00:14:23 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-17 00:14:28 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2021-07-17 00:14:28 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-17 00:14:28 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-17 00:14:28 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2021-07-17 00:14:28 main:501] : INFO : Creating Model
[2021-07-17 00:14:28 main:514] : INFO : Preparing config file
[2021-07-17 00:14:28 main:518] : INFO : Found checkpoint, attempting to load...
[2021-07-17 00:14:28 main:519] : INFO : Loading config
terminate called after throwing an instance of 'nlohmann::detail::type_error'
what(): [json.exception.type_error.302] type must be number, but is null
SIGABRT: abort called
Stack trace (24 frames):
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x563f9b49ef9c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7ff9a4808730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7ff9261207bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7ff92610b535]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x135)[0x563f9b5507f5]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398846)[0x563f9b4b9846]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398891)[0x563f9b4b9891]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x3968c4)[0x563f9b4b78c4]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe4af0)[0x563f9b205af0]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe1682)[0x563f9b202682]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd9a2b)[0x563f9b1faa2b]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd5f7f)[0x563f9b1f6f7f]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcf7f4)[0x563f9b1f07f4]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe16bc)[0x563f9b2026bc]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe4c96)[0x563f9b205c96]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe17ca)[0x563f9b2027ca]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd9b28)[0x563f9b1fab28]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd5faf)[0x563f9b1f6faf]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcf848)[0x563f9b1f0848]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcc66f)[0x563f9b1ed66f]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xc8d8c)[0x563f9b1e9d8c]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8a2d1)[0x563f9b1ab2d1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7ff92610d09b]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x563f9b1a775a]
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)