Task 5789233
| Name | ParityMachine-1625099757-30493-1_0 |
| Workunit | 3618183 |
| Created | 11 Jul 2021, 21:04:04 UTC |
| Sent | 16 Jul 2021, 18:44:58 UTC |
| Report deadline | 23 Jul 2021, 18:44:58 UTC |
| Received | 18 Jul 2021, 2:44:07 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 193 (0x000000C1) EXIT_SIGNAL |
| Computer ID | 13546 |
| Run time | 1 hours 1 min 1 sec |
| CPU time | 1 hours 0 min 46 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 15,412.34 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.80 (cuda10200) x86_64-pc-linux-gnu |
| Peak working set size | 2.33 GB |
| Peak swap size | 10.57 GB |
| Peak disk usage | 2.98 GB |
Stderr output
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)</message>
<stderr_txt>
main:581] : INFO : Epoch 1110 | loss: nan | val_loss: nan | Time: 2436.89 ms
[2021-07-18 03:32:59 main:581] : INFO : Epoch 1111 | loss: nan | val_loss: nan | Time: 2378.34 ms
[2021-07-18 03:33:01 main:581] : INFO : Epoch 1112 | loss: nan | val_loss: nan | Time: 2305.15 ms
[2021-07-18 03:33:05 main:581] : INFO : Epoch 1113 | loss: nan | val_loss: nan | Time: 3455.72 ms
[2021-07-18 03:33:08 main:581] : INFO : Epoch 1114 | loss: nan | val_loss: nan | Time: 3078.06 ms
[2021-07-18 03:33:12 main:581] : INFO : Epoch 1115 | loss: nan | val_loss: nan | Time: 3932.14 ms
[2021-07-18 03:33:15 main:581] : INFO : Epoch 1116 | loss: nan | val_loss: nan | Time: 3096.91 ms
[2021-07-18 03:33:17 main:581] : INFO : Epoch 1117 | loss: nan | val_loss: nan | Time: 2500.81 ms
[2021-07-18 03:33:20 main:581] : INFO : Epoch 1118 | loss: nan | val_loss: nan | Time: 2606.13 ms
[2021-07-18 03:33:23 main:581] : INFO : Epoch 1119 | loss: nan | val_loss: nan | Time: 2761.44 ms
[2021-07-18 03:33:25 main:581] : INFO : Epoch 1120 | loss: nan | val_loss: nan | Time: 2261.99 ms
[2021-07-18 03:33:28 main:581] : INFO : Epoch 1121 | loss: nan | val_loss: nan | Time: 3045.82 ms
[2021-07-18 03:33:32 main:581] : INFO : Epoch 1122 | loss: nan | val_loss: nan | Time: 3447.03 ms
[2021-07-18 03:33:34 main:581] : INFO : Epoch 1123 | loss: nan | val_loss: nan | Time: 2921.52 ms
[2021-07-18 03:33:37 main:581] : INFO : Epoch 1124 | loss: nan | val_loss: nan | Time: 2364.48 ms
[2021-07-18 03:33:39 main:581] : INFO : Epoch 1125 | loss: nan | val_loss: nan | Time: 2134.78 ms
[2021-07-18 03:33:41 main:581] : INFO : Epoch 1126 | loss: nan | val_loss: nan | Time: 2289.16 ms
[2021-07-18 03:33:43 main:581] : INFO : Epoch 1127 | loss: nan | val_loss: nan | Time: 2251.95 ms
[2021-07-18 03:33:46 main:581] : INFO : Epoch 1128 | loss: nan | val_loss: nan | Time: 2279.13 ms
[2021-07-18 03:33:48 main:581] : INFO : Epoch 1129 | loss: nan | val_loss: nan | Time: 2320.7 ms
[2021-07-18 03:33:50 main:581] : INFO : Epoch 1130 | loss: nan | val_loss: nan | Time: 2052.48 ms
[2021-07-18 03:33:52 main:581] : INFO : Epoch 1131 | loss: nan | val_loss: nan | Time: 2153.48 ms
[2021-07-18 03:33:55 main:581] : INFO : Epoch 1132 | loss: nan | val_loss: nan | Time: 2354.55 ms
[2021-07-18 03:33:57 main:581] : INFO : Epoch 1133 | loss: nan | val_loss: nan | Time: 2634.3 ms
[2021-07-18 03:34:01 main:581] : INFO : Epoch 1134 | loss: nan | val_loss: nan | Time: 3439.02 ms
[2021-07-18 03:34:04 main:581] : INFO : Epoch 1135 | loss: nan | val_loss: nan | Time: 3359.56 ms
[2021-07-18 03:34:08 main:581] : INFO : Epoch 1136 | loss: nan | val_loss: nan | Time: 3804.15 ms
[2021-07-18 03:34:11 main:581] : INFO : Epoch 1137 | loss: nan | val_loss: nan | Time: 2718.44 ms
[2021-07-18 03:34:13 main:581] : INFO : Epoch 1138 | loss: nan | val_loss: nan | Time: 2814.13 ms
[2021-07-18 03:34:16 main:581] : INFO : Epoch 1139 | loss: nan | val_loss: nan | Time: 2292.27 ms
[2021-07-18 03:34:18 main:581] : INFO : Epoch 1140 | loss: nan | val_loss: nan | Time: 2536.2 ms
[2021-07-18 03:34:21 main:581] : INFO : Epoch 1141 | loss: nan | val_loss: nan | Time: 2392.02 ms
[2021-07-18 03:34:23 main:581] : INFO : Epoch 1142 | loss: nan | val_loss: nan | Time: 2782.39 ms
[2021-07-18 03:34:26 main:581] : INFO : Epoch 1143 | loss: nan | val_loss: nan | Time: 2551.27 ms
[2021-07-18 03:34:30 main:581] : INFO : Epoch 1144 | loss: nan | val_loss: nan | Time: 3817.45 ms
[2021-07-18 03:34:33 main:581] : INFO : Epoch 1145 | loss: nan | val_loss: nan | Time: 3488.65 ms
[2021-07-18 03:34:35 main:581] : INFO : Epoch 1146 | loss: nan | val_loss: nan | Time: 1968.37 ms
[2021-07-18 03:34:37 main:581] : INFO : Epoch 1147 | loss: nan | val_loss: nan | Time: 1960.25 ms
[2021-07-18 03:34:39 main:581] : INFO : Epoch 1148 | loss: nan | val_loss: nan | Time: 2138.34 ms
[2021-07-18 03:34:43 main:581] : INFO : Epoch 1149 | loss: nan | val_loss: nan | Time: 3412.95 ms
[2021-07-18 03:34:46 main:581] : INFO : Epoch 1150 | loss: nan | val_loss: nan | Time: 3660.98 ms
[2021-07-18 03:34:49 main:581] : INFO : Epoch 1151 | loss: nan | val_loss: nan | Time: 2849.46 ms
[2021-07-18 03:34:52 main:581] : INFO : Epoch 1152 | loss: nan | val_loss: nan | Time: 2640.75 ms
[2021-07-18 03:34:55 main:581] : INFO : Epoch 1153 | loss: nan | val_loss: nan | Time: 2890.3 ms
[2021-07-18 03:34:58 main:581] : INFO : Epoch 1154 | loss: nan | val_loss: nan | Time: 2730.17 ms
[2021-07-18 03:35:01 main:581] : INFO : Epoch 1155 | loss: nan | val_loss: nan | Time: 3095.99 ms
[2021-07-18 03:35:04 main:581] : INFO : Epoch 1156 | loss: nan | val_loss: nan | Time: 2901.41 ms
[2021-07-18 03:35:07 main:581] : INFO : Epoch 1157 | loss: nan | val_loss: nan | Time: 2997.11 ms
[2021-07-18 03:35:10 main:581] : INFO : Epoch 1158 | loss: nan | val_loss: nan | Time: 3907.2 ms
[2021-07-18 03:35:14 main:581] : INFO : Epoch 1159 | loss: nan | val_loss: nan | Time: 3264.94 ms
[2021-07-18 03:35:16 main:581] : INFO : Epoch 1160 | loss: nan | val_loss: nan | Time: 2010.26 ms
[2021-07-18 03:35:18 main:581] : INFO : Epoch 1161 | loss: nan | val_loss: nan | Time: 1969.28 ms
[2021-07-18 03:35:20 main:581] : INFO : Epoch 1162 | loss: nan | val_loss: nan | Time: 1822.57 ms
[2021-07-18 03:35:21 main:581] : INFO : Epoch 1163 | loss: nan | val_loss: nan | Time: 1924.69 ms
[2021-07-18 03:35:23 main:581] : INFO : Epoch 1164 | loss: nan | val_loss: nan | Time: 1965.38 ms
[2021-07-18 03:35:26 main:581] : INFO : Epoch 1165 | loss: nan | val_loss: nan | Time: 2311.09 ms
[2021-07-18 03:35:29 main:581] : INFO : Epoch 1166 | loss: nan | val_loss: nan | Time: 2974.05 ms
[2021-07-18 03:35:31 main:581] : INFO : Epoch 1167 | loss: nan | val_loss: nan | Time: 2169.35 ms
[2021-07-18 03:35:33 main:581] : INFO : Epoch 1168 | loss: nan | val_loss: nan | Time: 2046.43 ms
[2021-07-18 03:35:35 main:581] : INFO : Epoch 1169 | loss: nan | val_loss: nan | Time: 2020.63 ms
[2021-07-18 03:35:37 main:581] : INFO : Epoch 1170 | loss: nan | val_loss: nan | Time: 1989.76 ms
[2021-07-18 03:35:39 main:581] : INFO : Epoch 1171 | loss: nan | val_loss: nan | Time: 1944.41 ms
[2021-07-18 03:35:41 main:581] : INFO : Epoch 1172 | loss: nan | val_loss: nan | Time: 1888.4 ms
[2021-07-18 03:35:43 main:581] : INFO : Epoch 1173 | loss: nan | val_loss: nan | Time: 1910.07 ms
[2021-07-18 03:35:45 main:581] : INFO : Epoch 1174 | loss: nan | val_loss: nan | Time: 1966.14 ms
[2021-07-18 03:35:47 main:581] : INFO : Epoch 1175 | loss: nan | val_loss: nan | Time: 2042.8 ms
[2021-07-18 03:35:49 main:581] : INFO : Epoch 1176 | loss: nan | val_loss: nan | Time: 2027.87 ms
[2021-07-18 03:35:51 main:581] : INFO : Epoch 1177 | loss: nan | val_loss: nan | Time: 1815.28 ms
[2021-07-18 03:35:52 main:581] : INFO : Epoch 1178 | loss: nan | val_loss: nan | Time: 1756.74 ms
[2021-07-18 03:35:54 main:581] : INFO : Epoch 1179 | loss: nan | val_loss: nan | Time: 1803.53 ms
[2021-07-18 03:35:56 main:581] : INFO : Epoch 1180 | loss: nan | val_loss: nan | Time: 1618.74 ms
[2021-07-18 03:35:58 main:581] : INFO : Epoch 1181 | loss: nan | val_loss: nan | Time: 1965.22 ms
[2021-07-18 03:36:00 main:581] : INFO : Epoch 1182 | loss: nan | val_loss: nan | Time: 2004.39 ms
[2021-07-18 03:36:02 main:581] : INFO : Epoch 1183 | loss: nan | val_loss: nan | Time: 2040.29 ms
[2021-07-18 03:36:04 main:581] : INFO : Epoch 1184 | loss: nan | val_loss: nan | Time: 2015.51 ms
[2021-07-18 03:36:06 main:581] : INFO : Epoch 1185 | loss: nan | val_loss: nan | Time: 2355.16 ms
[2021-07-18 03:36:09 main:581] : INFO : Epoch 1186 | loss: nan | val_loss: nan | Time: 3355.12 ms
[2021-07-18 03:36:12 main:581] : INFO : Epoch 1187 | loss: nan | val_loss: nan | Time: 2758.32 ms
[2021-07-18 03:36:14 main:581] : INFO : Epoch 1188 | loss: nan | val_loss: nan | Time: 1977.09 ms
[2021-07-18 03:36:16 main:581] : INFO : Epoch 1189 | loss: nan | val_loss: nan | Time: 1869.08 ms
[2021-07-18 03:36:18 main:581] : INFO : Epoch 1190 | loss: nan | val_loss: nan | Time: 1870.4 ms
[2021-07-18 03:36:20 main:581] : INFO : Epoch 1191 | loss: nan | val_loss: nan | Time: 1587.98 ms
[2021-07-18 03:36:21 main:581] : INFO : Epoch 1192 | loss: nan | val_loss: nan | Time: 1653.8 ms
[2021-07-18 03:36:23 main:581] : INFO : Epoch 1193 | loss: nan | val_loss: nan | Time: 1956.54 ms
[2021-07-18 03:36:25 main:581] : INFO : Epoch 1194 | loss: nan | val_loss: nan | Time: 2108.26 ms
[2021-07-18 03:36:29 main:581] : INFO : Epoch 1195 | loss: nan | val_loss: nan | Time: 4176.08 ms
[2021-07-18 03:36:32 main:581] : INFO : Epoch 1196 | loss: nan | val_loss: nan | Time: 2439.87 ms
[2021-07-18 03:36:34 main:581] : INFO : Epoch 1197 | loss: nan | val_loss: nan | Time: 1909.27 ms
[2021-07-18 03:36:36 main:581] : INFO : Epoch 1198 | loss: nan | val_loss: nan | Time: 1921.74 ms
[2021-07-18 03:36:38 main:581] : INFO : Epoch 1199 | loss: nan | val_loss: nan | Time: 2037.71 ms
[2021-07-18 03:36:40 main:581] : INFO : Epoch 1200 | loss: nan | val_loss: nan | Time: 2384.13 ms
[2021-07-18 03:36:42 main:581] : INFO : Epoch 1201 | loss: nan | val_loss: nan | Time: 2340.54 ms
[2021-07-18 03:36:44 main:581] : INFO : Epoch 1202 | loss: nan | val_loss: nan | Time: 1935.24 ms
[2021-07-18 03:36:46 main:581] : INFO : Epoch 1203 | loss: nan | val_loss: nan | Time: 1966.9 ms
[2021-07-18 03:36:48 main:581] : INFO : Epoch 1204 | loss: nan | val_loss: nan | Time: 2040.37 ms
[2021-07-18 03:36:50 main:581] : INFO : Epoch 1205 | loss: nan | val_loss: nan | Time: 2018.53 ms
[2021-07-18 03:36:52 main:581] : INFO : Epoch 1206 | loss: nan | val_loss: nan | Time: 1978.21 ms
[2021-07-18 03:36:54 main:581] : INFO : Epoch 1207 | loss: nan | val_loss: nan | Time: 1855.59 ms
[2021-07-18 03:36:56 main:581] : INFO : Epoch 1208 | loss: nan | val_loss: nan | Time: 2117.1 ms
[2021-07-18 03:36:59 main:581] : INFO : Epoch 1209 | loss: nan | val_loss: nan | Time: 2657.83 ms
[2021-07-18 03:37:02 main:581] : INFO : Epoch 1210 | loss: nan | val_loss: nan | Time: 3378.49 ms
[2021-07-18 03:37:05 main:581] : INFO : Epoch 1211 | loss: nan | val_loss: nan | Time: 2611.56 ms
[2021-07-18 03:37:07 main:581] : INFO : Epoch 1212 | loss: nan | val_loss: nan | Time: 2182.6 ms
[2021-07-18 03:37:10 main:581] : INFO : Epoch 1213 | loss: nan | val_loss: nan | Time: 2760.2 ms
[2021-07-18 03:37:13 main:581] : INFO : Epoch 1214 | loss: nan | val_loss: nan | Time: 2589.29 ms
[2021-07-18 03:37:16 main:581] : INFO : Epoch 1215 | loss: nan | val_loss: nan | Time: 3250.7 ms
[2021-07-18 03:37:19 main:581] : INFO : Epoch 1216 | loss: nan | val_loss: nan | Time: 3515.08 ms
[2021-07-18 03:37:22 main:581] : INFO : Epoch 1217 | loss: nan | val_loss: nan | Time: 2332.26 ms
[2021-07-18 03:37:24 main:581] : INFO : Epoch 1218 | loss: nan | val_loss: nan | Time: 2548.84 ms
[2021-07-18 03:37:27 main:581] : INFO : Epoch 1219 | loss: nan | val_loss: nan | Time: 2797.94 ms
[2021-07-18 03:37:29 main:581] : INFO : Epoch 1220 | loss: nan | val_loss: nan | Time: 2274.44 ms
[2021-07-18 03:37:31 main:581] : INFO : Epoch 1221 | loss: nan | val_loss: nan | Time: 1789.55 ms
[2021-07-18 03:37:33 main:581] : INFO : Epoch 1222 | loss: nan | val_loss: nan | Time: 1858.64 ms
[2021-07-18 03:37:35 main:581] : INFO : Epoch 1223 | loss: nan | val_loss: nan | Time: 1897.98 ms
[2021-07-18 03:37:37 main:581] : INFO : Epoch 1224 | loss: nan | val_loss: nan | Time: 2039.09 ms
[2021-07-18 03:37:39 main:581] : INFO : Epoch 1225 | loss: nan | val_loss: nan | Time: 2037.46 ms
[2021-07-18 03:37:42 main:581] : INFO : Epoch 1226 | loss: nan | val_loss: nan | Time: 3326.16 ms
[2021-07-18 03:37:46 main:581] : INFO : Epoch 1227 | loss: nan | val_loss: nan | Time: 3356.22 ms
[2021-07-18 03:37:48 main:581] : INFO : Epoch 1228 | loss: nan | val_loss: nan | Time: 2773.32 ms
[2021-07-18 03:37:51 main:581] : INFO : Epoch 1229 | loss: nan | val_loss: nan | Time: 2375.55 ms
[2021-07-18 03:37:54 main:581] : INFO : Epoch 1230 | loss: nan | val_loss: nan | Time: 3239.86 ms
[2021-07-18 03:37:56 main:581] : INFO : Epoch 1231 | loss: nan | val_loss: nan | Time: 2221.53 ms
[2021-07-18 03:37:59 main:581] : INFO : Epoch 1232 | loss: nan | val_loss: nan | Time: 2350.7 ms
[2021-07-18 03:38:01 main:581] : INFO : Epoch 1233 | loss: nan | val_loss: nan | Time: 2747.32 ms
[2021-07-18 03:38:05 main:581] : INFO : Epoch 1234 | loss: nan | val_loss: nan | Time: 3355.74 ms
[2021-07-18 03:38:08 main:581] : INFO : Epoch 1235 | loss: nan | val_loss: nan | Time: 3629.52 ms
[2021-07-18 03:38:11 main:581] : INFO : Epoch 1236 | loss: nan | val_loss: nan | Time: 2754.23 ms
[2021-07-18 03:38:14 main:581] : INFO : Epoch 1237 | loss: nan | val_loss: nan | Time: 2975.61 ms
[2021-07-18 03:38:17 main:581] : INFO : Epoch 1238 | loss: nan | val_loss: nan | Time: 3453.31 ms
[2021-07-18 03:38:20 main:581] : INFO : Epoch 1239 | loss: nan | val_loss: nan | Time: 2426.97 ms
[2021-07-18 03:38:22 main:581] : INFO : Epoch 1240 | loss: nan | val_loss: nan | Time: 2401.97 ms
[2021-07-18 03:38:24 main:581] : INFO : Epoch 1241 | loss: nan | val_loss: nan | Time: 2007.88 ms
[2021-07-18 03:38:27 main:581] : INFO : Epoch 1242 | loss: nan | val_loss: nan | Time: 2667.23 ms
[2021-07-18 03:38:30 main:581] : INFO : Epoch 1243 | loss: nan | val_loss: nan | Time: 3151.86 ms
[2021-07-18 03:38:34 main:581] : INFO : Epoch 1244 | loss: nan | val_loss: nan | Time: 3383.53 ms
[2021-07-18 03:38:38 main:581] : INFO : Epoch 1245 | loss: nan | val_loss: nan | Time: 4080.61 ms
[2021-07-18 03:38:41 main:581] : INFO : Epoch 1246 | loss: nan | val_loss: nan | Time: 2980.4 ms
[2021-07-18 03:38:43 main:581] : INFO : Epoch 1247 | loss: nan | val_loss: nan | Time: 2731.17 ms
[2021-07-18 03:38:46 main:581] : INFO : Epoch 1248 | loss: nan | val_loss: nan | Time: 2432.46 ms
[2021-07-18 03:38:48 main:581] : INFO : Epoch 1249 | loss: nan | val_loss: nan | Time: 2529.4 ms
[2021-07-18 03:38:51 main:581] : INFO : Epoch 1250 | loss: nan | val_loss: nan | Time: 2456.19 ms
[2021-07-18 03:38:53 main:581] : INFO : Epoch 1251 | loss: nan | val_loss: nan | Time: 2538.42 ms
[2021-07-18 03:38:56 main:581] : INFO : Epoch 1252 | loss: nan | val_loss: nan | Time: 2931.43 ms
[2021-07-18 03:38:59 main:581] : INFO : Epoch 1253 | loss: nan | val_loss: nan | Time: 2770.11 ms
[2021-07-18 03:39:02 main:581] : INFO : Epoch 1254 | loss: nan | val_loss: nan | Time: 3170.22 ms
[2021-07-18 03:39:05 main:581] : INFO : Epoch 1255 | loss: nan | val_loss: nan | Time: 2829.17 ms
[2021-07-18 03:39:08 main:581] : INFO : Epoch 1256 | loss: nan | val_loss: nan | Time: 2758.64 ms
[2021-07-18 03:39:10 main:581] : INFO : Epoch 1257 | loss: nan | val_loss: nan | Time: 2507.46 ms
[2021-07-18 03:39:13 main:581] : INFO : Epoch 1258 | loss: nan | val_loss: nan | Time: 2764.74 ms
[2021-07-18 03:39:15 main:581] : INFO : Epoch 1259 | loss: nan | val_loss: nan | Time: 2489 ms
[2021-07-18 03:39:18 main:581] : INFO : Epoch 1260 | loss: nan | val_loss: nan | Time: 2775.75 ms
[2021-07-18 03:39:21 main:581] : INFO : Epoch 1261 | loss: nan | val_loss: nan | Time: 2722.86 ms
[2021-07-18 03:39:24 main:581] : INFO : Epoch 1262 | loss: nan | val_loss: nan | Time: 3116.64 ms
[2021-07-18 03:39:27 main:581] : INFO : Epoch 1263 | loss: nan | val_loss: nan | Time: 2888.51 ms
[2021-07-18 03:39:30 main:581] : INFO : Epoch 1264 | loss: nan | val_loss: nan | Time: 2857.08 ms
[2021-07-18 03:39:32 main:581] : INFO : Epoch 1265 | loss: nan | val_loss: nan | Time: 2305.24 ms
[2021-07-18 03:39:35 main:581] : INFO : Epoch 1266 | loss: nan | val_loss: nan | Time: 2495.86 ms
[2021-07-18 03:39:38 main:581] : INFO : Epoch 1267 | loss: nan | val_loss: nan | Time: 3068.04 ms
[2021-07-18 03:39:42 main:581] : INFO : Epoch 1268 | loss: nan | val_loss: nan | Time: 4038.16 ms
[2021-07-18 03:39:44 main:581] : INFO : Epoch 1269 | loss: nan | val_loss: nan | Time: 2332.12 ms
[2021-07-18 03:39:47 main:581] : INFO : Epoch 1270 | loss: nan | val_loss: nan | Time: 2509.12 ms
[2021-07-18 03:39:49 main:581] : INFO : Epoch 1271 | loss: nan | val_loss: nan | Time: 2269.81 ms
[2021-07-18 03:39:51 main:581] : INFO : Epoch 1272 | loss: nan | val_loss: nan | Time: 1793.68 ms
[2021-07-18 03:39:53 main:581] : INFO : Epoch 1273 | loss: nan | val_loss: nan | Time: 1895.56 ms
[2021-07-18 03:39:55 main:581] : INFO : Epoch 1274 | loss: nan | val_loss: nan | Time: 2022.74 ms
[2021-07-18 03:39:57 main:581] : INFO : Epoch 1275 | loss: nan | val_loss: nan | Time: 2391.75 ms
[2021-07-18 03:40:00 main:581] : INFO : Epoch 1276 | loss: nan | val_loss: nan | Time: 3390.26 ms
[2021-07-18 03:40:04 main:581] : INFO : Epoch 1277 | loss: nan | val_loss: nan | Time: 3466.81 ms
[2021-07-18 03:40:07 main:581] : INFO : Epoch 1278 | loss: nan | val_loss: nan | Time: 2764.2 ms
[2021-07-18 03:40:09 main:581] : INFO : Epoch 1279 | loss: nan | val_loss: nan | Time: 2482.13 ms
[2021-07-18 03:40:12 main:581] : INFO : Epoch 1280 | loss: nan | val_loss: nan | Time: 2973.45 ms
[2021-07-18 03:40:15 main:581] : INFO : Epoch 1281 | loss: nan | val_loss: nan | Time: 2663.95 ms
[2021-07-18 03:40:17 main:581] : INFO : Epoch 1282 | loss: nan | val_loss: nan | Time: 2375.73 ms
[2021-07-18 03:40:19 main:581] : INFO : Epoch 1283 | loss: nan | val_loss: nan | Time: 2362.67 ms
[2021-07-18 03:40:23 main:581] : INFO : Epoch 1284 | loss: nan | val_loss: nan | Time: 3497.39 ms
[2021-07-18 03:40:27 main:581] : INFO : Epoch 1285 | loss: nan | val_loss: nan | Time: 3859.45 ms
[2021-07-18 03:40:31 main:581] : INFO : Epoch 1286 | loss: nan | val_loss: nan | Time: 4415.17 ms
[2021-07-18 03:40:34 main:581] : INFO : Epoch 1287 | loss: nan | val_loss: nan | Time: 2816.37 ms
[2021-07-18 03:40:36 main:581] : INFO : Epoch 1288 | loss: nan | val_loss: nan | Time: 2373.9 ms
[2021-07-18 03:40:39 main:581] : INFO : Epoch 1289 | loss: nan | val_loss: nan | Time: 2495.12 ms
[2021-07-18 03:40:41 main:581] : INFO : Epoch 1290 | loss: nan | val_loss: nan | Time: 2067.39 ms
[2021-07-18 03:40:43 main:581] : INFO : Epoch 1291 | loss: nan | val_loss: nan | Time: 2402.87 ms
[2021-07-18 03:40:46 main:581] : INFO : Epoch 1292 | loss: nan | val_loss: nan | Time: 3001.22 ms
[2021-07-18 03:40:51 main:581] : INFO : Epoch 1293 | loss: nan | val_loss: nan | Time: 4651.01 ms
[2021-07-18 03:40:55 main:581] : INFO : Epoch 1294 | loss: nan | val_loss: nan | Time: 3580.61 ms
[2021-07-18 03:40:57 main:581] : INFO : Epoch 1295 | loss: nan | val_loss: nan | Time: 2242.75 ms
[2021-07-18 03:40:59 main:581] : INFO : Epoch 1296 | loss: nan | val_loss: nan | Time: 2402.14 ms
[2021-07-18 03:41:02 main:581] : INFO : Epoch 1297 | loss: nan | val_loss: nan | Time: 2316.75 ms
[2021-07-18 03:41:04 main:581] : INFO : Epoch 1298 | loss: nan | val_loss: nan | Time: 2314.18 ms
[2021-07-18 03:41:06 main:581] : INFO : Epoch 1299 | loss: nan | val_loss: nan | Time: 2136.42 ms
[2021-07-18 03:41:08 main:581] : INFO : Epoch 1300 | loss: nan | val_loss: nan | Time: 2233.9 ms
[2021-07-18 03:41:11 main:581] : INFO : Epoch 1301 | loss: nan | val_loss: nan | Time: 2398.68 ms
[2021-07-18 03:41:14 main:581] : INFO : Epoch 1302 | loss: nan | val_loss: nan | Time: 3813.9 ms
[2021-07-18 03:41:19 main:581] : INFO : Epoch 1303 | loss: nan | val_loss: nan | Time: 4296.31 ms
[2021-07-18 03:41:22 main:581] : INFO : Epoch 1304 | loss: nan | val_loss: nan | Time: 3128.15 ms
[2021-07-18 03:41:24 main:581] : INFO : Epoch 1305 | loss: nan | val_loss: nan | Time: 2071.34 ms
[2021-07-18 03:41:26 main:581] : INFO : Epoch 1306 | loss: nan | val_loss: nan | Time: 1976.12 ms
[2021-07-18 03:41:29 main:581] : INFO : Epoch 1307 | loss: nan | val_loss: nan | Time: 2701.88 ms
[2021-07-18 03:41:31 main:581] : INFO : Epoch 1308 | loss: nan | val_loss: nan | Time: 1957.95 ms
[2021-07-18 03:41:33 main:581] : INFO : Epoch 1309 | loss: nan | val_loss: nan | Time: 1993.03 ms
[2021-07-18 03:41:35 main:581] : INFO : Epoch 1310 | loss: nan | val_loss: nan | Time: 1952.29 ms
[2021-07-18 03:41:36 main:581] : INFO : Epoch 1311 | loss: nan | val_loss: nan | Time: 1593.42 ms
[2021-07-18 03:41:38 main:581] : INFO : Epoch 1312 | loss: nan | val_loss: nan | Time: 2234.96 ms
[2021-07-18 03:41:41 main:581] : INFO : Epoch 1313 | loss: nan | val_loss: nan | Time: 2472.7 ms
[2021-07-18 03:41:43 main:581] : INFO : Epoch 1314 | loss: nan | val_loss: nan | Time: 2056.29 ms
[2021-07-18 03:41:45 main:581] : INFO : Epoch 1315 | loss: nan | val_loss: nan | Time: 1983.06 ms
[2021-07-18 03:41:47 main:581] : INFO : Epoch 1316 | loss: nan | val_loss: nan | Time: 1988.27 ms
[2021-07-18 03:41:49 main:581] : INFO : Epoch 1317 | loss: nan | val_loss: nan | Time: 2090.81 ms
[2021-07-18 03:41:51 main:581] : INFO : Epoch 1318 | loss: nan | val_loss: nan | Time: 2172.74 ms
[2021-07-18 03:41:53 main:581] : INFO : Epoch 1319 | loss: nan | val_loss: nan | Time: 1893.54 ms
[2021-07-18 03:41:55 main:581] : INFO : Epoch 1320 | loss: nan | val_loss: nan | Time: 2002.48 ms
[2021-07-18 03:41:57 main:581] : INFO : Epoch 1321 | loss: nan | val_loss: nan | Time: 1951.77 ms
[2021-07-18 03:41:59 main:581] : INFO : Epoch 1322 | loss: nan | val_loss: nan | Time: 2160.39 ms
[2021-07-18 03:42:02 main:581] : INFO : Epoch 1323 | loss: nan | val_loss: nan | Time: 2932.4 ms
[2021-07-18 03:42:04 main:581] : INFO : Epoch 1324 | loss: nan | val_loss: nan | Time: 2282.79 ms
[2021-07-18 03:42:08 main:581] : INFO : Epoch 1325 | loss: nan | val_loss: nan | Time: 3534.99 ms
[2021-07-18 03:42:11 main:581] : INFO : Epoch 1326 | loss: nan | val_loss: nan | Time: 3207.81 ms
[2021-07-18 03:42:13 main:581] : INFO : Epoch 1327 | loss: nan | val_loss: nan | Time: 2029.46 ms
[2021-07-18 03:42:16 main:581] : INFO : Epoch 1328 | loss: nan | val_loss: nan | Time: 2637.74 ms
[2021-07-18 03:42:19 main:581] : INFO : Epoch 1329 | loss: nan | val_loss: nan | Time: 3105.95 ms
[2021-07-18 03:42:22 main:581] : INFO : Epoch 1330 | loss: nan | val_loss: nan | Time: 2721.63 ms
[2021-07-18 03:42:24 main:581] : INFO : Epoch 1331 | loss: nan | val_loss: nan | Time: 2378.5 ms
[2021-07-18 03:42:26 main:581] : INFO : Epoch 1332 | loss: nan | val_loss: nan | Time: 2414.69 ms
[2021-07-18 03:42:29 main:581] : INFO : Epoch 1333 | loss: nan | val_loss: nan | Time: 3027.75 ms
[2021-07-18 03:42:33 main:581] : INFO : Epoch 1334 | loss: nan | val_loss: nan | Time: 3308.64 ms
[2021-07-18 03:42:36 main:581] : INFO : Epoch 1335 | loss: nan | val_loss: nan | Time: 2872.19 ms
[2021-07-18 03:42:38 main:581] : INFO : Epoch 1336 | loss: nan | val_loss: nan | Time: 2450.27 ms
[2021-07-18 03:42:41 main:581] : INFO : Epoch 1337 | loss: nan | val_loss: nan | Time: 3377.22 ms
[2021-07-18 03:42:44 main:581] : INFO : Epoch 1338 | loss: nan | val_loss: nan | Time: 2393.15 ms
[2021-07-18 03:42:46 main:581] : INFO : Epoch 1339 | loss: nan | val_loss: nan | Time: 2488.75 ms
[2021-07-18 03:42:49 main:581] : INFO : Epoch 1340 | loss: nan | val_loss: nan | Time: 2414.82 ms
[2021-07-18 03:42:51 main:581] : INFO : Epoch 1341 | loss: nan | val_loss: nan | Time: 2538.12 ms
[2021-07-18 03:42:55 main:581] : INFO : Epoch 1342 | loss: nan | val_loss: nan | Time: 3655.63 ms
[2021-07-18 03:42:58 main:581] : INFO : Epoch 1343 | loss: nan | val_loss: nan | Time: 3072.37 ms
[2021-07-18 03:43:03 main:581] : INFO : Epoch 1344 | loss: nan | val_loss: nan | Time: 4837.29 ms
[2021-07-18 03:43:06 main:581] : INFO : Epoch 1345 | loss: nan | val_loss: nan | Time: 2927.6 ms
[2021-07-18 03:43:08 main:581] : INFO : Epoch 1346 | loss: nan | val_loss: nan | Time: 2368.42 ms
[2021-07-18 03:43:10 main:581] : INFO : Epoch 1347 | loss: nan | val_loss: nan | Time: 1998.51 ms
[2021-07-18 03:43:13 main:581] : INFO : Epoch 1348 | loss: nan | val_loss: nan | Time: 3312.01 ms
[2021-07-18 03:43:16 main:581] : INFO : Epoch 1349 | loss: nan | val_loss: nan | Time: 2296.33 ms
[2021-07-18 03:43:19 main:581] : INFO : Epoch 1350 | loss: nan | val_loss: nan | Time: 3225.1 ms
[2021-07-18 03:43:22 main:581] : INFO : Epoch 1351 | loss: nan | val_loss: nan | Time: 2854.05 ms
[2021-07-18 03:43:25 main:581] : INFO : Epoch 1352 | loss: nan | val_loss: nan | Time: 3385.64 ms
[2021-07-18 03:43:28 main:581] : INFO : Epoch 1353 | loss: nan | val_loss: nan | Time: 2629.45 ms
[2021-07-18 03:43:31 main:581] : INFO : Epoch 1354 | loss: nan | val_loss: nan | Time: 2884.18 ms
[2021-07-18 03:43:35 main:581] : INFO : Epoch 1355 | loss: nan | val_loss: nan | Time: 3869.04 ms
[2021-07-18 03:43:38 main:581] : INFO : Epoch 1356 | loss: nan | val_loss: nan | Time: 3623.13 ms
[2021-07-18 03:43:41 main:581] : INFO : Epoch 1357 | loss: nan | val_loss: nan | Time: 2693.71 ms
[2021-07-18 03:43:44 main:581] : INFO : Epoch 1358 | loss: nan | val_loss: nan | Time: 2815.77 ms
[2021-07-18 03:43:46 main:581] : INFO : Epoch 1359 | loss: nan | val_loss: nan | Time: 2226.47 ms
[2021-07-18 03:43:48 main:581] : INFO : Epoch 1360 | loss: nan | val_loss: nan | Time: 2265.4 ms
[2021-07-18 03:43:52 main:581] : INFO : Epoch 1361 | loss: nan | val_loss: nan | Time: 3540.55 ms
[2021-07-18 03:43:56 main:581] : INFO : Epoch 1362 | loss: nan | val_loss: nan | Time: 3907.25 ms
[2021-07-18 03:43:59 main:581] : INFO : Epoch 1363 | loss: nan | val_loss: nan | Time: 3755.71 ms
[2021-07-18 03:44:02 main:581] : INFO : Epoch 1364 | loss: nan | val_loss: nan | Time: 2924.52 ms
[2021-07-18 03:44:05 main:581] : INFO : Epoch 1365 | loss: nan | val_loss: nan | Time: 2783.16 ms
[2021-07-18 03:44:08 main:581] : INFO : Epoch 1366 | loss: nan | val_loss: nan | Time: 3063.44 ms
[2021-07-18 03:44:11 main:581] : INFO : Epoch 1367 | loss: nan | val_loss: nan | Time: 3244.84 ms
[2021-07-18 03:44:14 main:581] : INFO : Epoch 1368 | loss: nan | val_loss: nan | Time: 2527.47 ms
[2021-07-18 03:44:17 main:581] : INFO : Epoch 1369 | loss: nan | val_loss: nan | Time: 2798.42 ms
[2021-07-18 03:44:19 main:581] : INFO : Epoch 1370 | loss: nan | val_loss: nan | Time: 2280.12 ms
[2021-07-18 03:44:22 main:581] : INFO : Epoch 1371 | loss: nan | val_loss: nan | Time: 3039.85 ms
[2021-07-18 03:44:25 main:581] : INFO : Epoch 1372 | loss: nan | val_loss: nan | Time: 2935.81 ms
[2021-07-18 03:44:29 main:581] : INFO : Epoch 1373 | loss: nan | val_loss: nan | Time: 3716.49 ms
[2021-07-18 03:44:33 main:581] : INFO : Epoch 1374 | loss: nan | val_loss: nan | Time: 4020.39 ms
[2021-07-18 03:44:35 main:581] : INFO : Epoch 1375 | loss: nan | val_loss: nan | Time: 2393.05 ms
[2021-07-18 03:44:38 main:581] : INFO : Epoch 1376 | loss: nan | val_loss: nan | Time: 2437.98 ms
[2021-07-18 03:44:40 main:581] : INFO : Epoch 1377 | loss: nan | val_loss: nan | Time: 2627.82 ms
[2021-07-18 03:44:43 main:581] : INFO : Epoch 1378 | loss: nan | val_loss: nan | Time: 2878.98 ms
[2021-07-18 03:44:46 main:581] : INFO : Epoch 1379 | loss: nan | val_loss: nan | Time: 2446.05 ms
[2021-07-18 03:44:48 main:581] : INFO : Epoch 1380 | loss: nan | val_loss: nan | Time: 2427.35 ms
[2021-07-18 03:44:51 main:581] : INFO : Epoch 1381 | loss: nan | val_loss: nan | Time: 2477.24 ms
[2021-07-18 03:44:53 main:581] : INFO : Epoch 1382 | loss: nan | val_loss: nan | Time: 2684.52 ms
[2021-07-18 03:44:56 main:581] : INFO : Epoch 1383 | loss: nan | val_loss: nan | Time: 2472.77 ms
[2021-07-18 03:44:58 main:581] : INFO : Epoch 1384 | loss: nan | val_loss: nan | Time: 2573.28 ms
[2021-07-18 03:45:01 main:581] : INFO : Epoch 1385 | loss: nan | val_loss: nan | Time: 2363.2 ms
[2021-07-18 03:45:04 main:581] : INFO : Epoch 1386 | loss: nan | val_loss: nan | Time: 3207.97 ms
[2021-07-18 03:45:08 main:581] : INFO : Epoch 1387 | loss: nan | val_loss: nan | Time: 3704.91 ms
[2021-07-18 03:45:10 main:581] : INFO : Epoch 1388 | loss: nan | val_loss: nan | Time: 2312.32 ms
[2021-07-18 03:45:13 main:581] : INFO : Epoch 1389 | loss: nan | val_loss: nan | Time: 2741.55 ms
[2021-07-18 03:45:17 main:581] : INFO : Epoch 1390 | loss: nan | val_loss: nan | Time: 4260.81 ms
[2021-07-18 03:45:19 main:581] : INFO : Epoch 1391 | loss: nan | val_loss: nan | Time: 2641.96 ms
[2021-07-18 03:45:22 main:581] : INFO : Epoch 1392 | loss: nan | val_loss: nan | Time: 2435.04 ms
[2021-07-18 03:45:25 main:581] : INFO : Epoch 1393 | loss: nan | val_loss: nan | Time: 2792.9 ms
[2021-07-18 03:45:27 main:581] : INFO : Epoch 1394 | loss: nan | val_loss: nan | Time: 2610.76 ms
[2021-07-18 03:45:30 main:581] : INFO : Epoch 1395 | loss: nan | val_loss: nan | Time: 2983.87 ms
[2021-07-18 03:45:33 main:581] : INFO : Epoch 1396 | loss: nan | val_loss: nan | Time: 2957.69 ms
[2021-07-18 03:45:36 main:581] : INFO : Epoch 1397 | loss: nan | val_loss: nan | Time: 2610.85 ms
[2021-07-18 03:45:39 main:581] : INFO : Epoch 1398 | loss: nan | val_loss: nan | Time: 2942.48 ms
[2021-07-18 03:45:41 main:581] : INFO : Epoch 1399 | loss: nan | val_loss: nan | Time: 2620.34 ms
[2021-07-18 03:45:44 main:581] : INFO : Epoch 1400 | loss: nan | val_loss: nan | Time: 2966.19 ms
[2021-07-18 03:45:48 main:581] : INFO : Epoch 1401 | loss: nan | val_loss: nan | Time: 3304.71 ms
[2021-07-18 03:45:51 main:581] : INFO : Epoch 1402 | loss: nan | val_loss: nan | Time: 3627.92 ms
[2021-07-18 03:45:54 main:581] : INFO : Epoch 1403 | loss: nan | val_loss: nan | Time: 2476.63 ms
[2021-07-18 03:45:56 main:581] : INFO : Epoch 1404 | loss: nan | val_loss: nan | Time: 2671.03 ms
[2021-07-18 03:45:59 main:581] : INFO : Epoch 1405 | loss: nan | val_loss: nan | Time: 2367.53 ms
[2021-07-18 03:46:02 main:581] : INFO : Epoch 1406 | loss: nan | val_loss: nan | Time: 3107.21 ms
[2021-07-18 03:46:04 main:581] : INFO : Epoch 1407 | loss: nan | val_loss: nan | Time: 2217.21 ms
[2021-07-18 03:46:07 main:581] : INFO : Epoch 1408 | loss: nan | val_loss: nan | Time: 3187.59 ms
[2021-07-18 03:46:11 main:581] : INFO : Epoch 1409 | loss: nan | val_loss: nan | Time: 3319.82 ms
[2021-07-18 03:46:13 main:581] : INFO : Epoch 1410 | loss: nan | val_loss: nan | Time: 2436.45 ms
[2021-07-18 03:46:16 main:581] : INFO : Epoch 1411 | loss: nan | val_loss: nan | Time: 2502.59 ms
[2021-07-18 03:46:18 main:581] : INFO : Epoch 1412 | loss: nan | val_loss: nan | Time: 2536.28 ms
[2021-07-18 03:46:21 main:581] : INFO : Epoch 1413 | loss: nan | val_loss: nan | Time: 2503.27 ms
[2021-07-18 03:46:23 main:581] : INFO : Epoch 1414 | loss: nan | val_loss: nan | Time: 2502.73 ms
[2021-07-18 03:46:25 main:581] : INFO : Epoch 1415 | loss: nan | val_loss: nan | Time: 2307.55 ms
[2021-07-18 03:46:28 main:581] : INFO : Epoch 1416 | loss: nan | val_loss: nan | Time: 2382.92 ms
[2021-07-18 03:46:30 main:581] : INFO : Epoch 1417 | loss: nan | val_loss: nan | Time: 2289.32 ms
[2021-07-18 03:46:33 main:581] : INFO : Epoch 1418 | loss: nan | val_loss: nan | Time: 2453.69 ms
[2021-07-18 03:46:35 main:581] : INFO : Epoch 1419 | loss: nan | val_loss: nan | Time: 2485.13 ms
[2021-07-18 03:46:37 main:581] : INFO : Epoch 1420 | loss: nan | val_loss: nan | Time: 2242.07 ms
[2021-07-18 03:46:40 main:581] : INFO : Epoch 1421 | loss: nan | val_loss: nan | Time: 2270.55 ms
[2021-07-18 03:46:42 main:581] : INFO : Epoch 1422 | loss: nan | val_loss: nan | Time: 2384.25 ms
[2021-07-18 03:46:44 main:581] : INFO : Epoch 1423 | loss: nan | val_loss: nan | Time: 2470.77 ms
[2021-07-18 03:46:47 main:581] : INFO : Epoch 1424 | loss: nan | val_loss: nan | Time: 2436.36 ms
[2021-07-18 03:46:49 main:581] : INFO : Epoch 1425 | loss: nan | val_loss: nan | Time: 2514.66 ms
[2021-07-18 03:46:52 main:581] : INFO : Epoch 1426 | loss: nan | val_loss: nan | Time: 2499.88 ms
[2021-07-18 03:46:54 main:581] : INFO : Epoch 1427 | loss: nan | val_loss: nan | Time: 2429.31 ms
[2021-07-18 03:46:57 main:581] : INFO : Epoch 1428 | loss: nan | val_loss: nan | Time: 2490.63 ms
[2021-07-18 03:46:59 main:581] : INFO : Epoch 1429 | loss: nan | val_loss: nan | Time: 2385.94 ms
[2021-07-18 03:47:02 main:581] : INFO : Epoch 1430 | loss: nan | val_loss: nan | Time: 2423.52 ms
[2021-07-18 03:47:04 main:581] : INFO : Epoch 1431 | loss: nan | val_loss: nan | Time: 2475.86 ms
[2021-07-18 03:47:07 main:581] : INFO : Epoch 1432 | loss: nan | val_loss: nan | Time: 2401.54 ms
[2021-07-18 03:47:09 main:581] : INFO : Epoch 1433 | loss: nan | val_loss: nan | Time: 2244.42 ms
[2021-07-18 03:47:11 main:581] : INFO : Epoch 1434 | loss: nan | val_loss: nan | Time: 2340.4 ms
[2021-07-18 03:47:13 main:581] : INFO : Epoch 1435 | loss: nan | val_loss: nan | Time: 2325.59 ms
[2021-07-18 03:47:16 main:581] : INFO : Epoch 1436 | loss: nan | val_loss: nan | Time: 2428.54 ms
[2021-07-18 03:47:18 main:581] : INFO : Epoch 1437 | loss: nan | val_loss: nan | Time: 2485.54 ms
[2021-07-18 03:47:21 main:581] : INFO : Epoch 1438 | loss: nan | val_loss: nan | Time: 2472.82 ms
[2021-07-18 03:47:23 main:581] : INFO : Epoch 1439 | loss: nan | val_loss: nan | Time: 2494.2 ms
[2021-07-18 03:47:26 main:581] : INFO : Epoch 1440 | loss: nan | val_loss: nan | Time: 2417.81 ms
[2021-07-18 03:47:28 main:581] : INFO : Epoch 1441 | loss: nan | val_loss: nan | Time: 2441.14 ms
[2021-07-18 03:47:31 main:581] : INFO : Epoch 1442 | loss: nan | val_loss: nan | Time: 2487.72 ms
[2021-07-18 03:47:33 main:581] : INFO : Epoch 1443 | loss: nan | val_loss: nan | Time: 2406.81 ms
[2021-07-18 03:47:36 main:581] : INFO : Epoch 1444 | loss: nan | val_loss: nan | Time: 2456.35 ms
[2021-07-18 03:47:38 main:581] : INFO : Epoch 1445 | loss: nan | val_loss: nan | Time: 2420.36 ms
[2021-07-18 03:47:40 main:581] : INFO : Epoch 1446 | loss: nan | val_loss: nan | Time: 2375.49 ms
[2021-07-18 03:47:43 main:581] : INFO : Epoch 1447 | loss: nan | val_loss: nan | Time: 2484.91 ms
[2021-07-18 03:47:45 main:581] : INFO : Epoch 1448 | loss: nan | val_loss: nan | Time: 2296.49 ms
[2021-07-18 03:47:47 main:581] : INFO : Epoch 1449 | loss: nan | val_loss: nan | Time: 2287.59 ms
[2021-07-18 03:47:50 main:581] : INFO : Epoch 1450 | loss: nan | val_loss: nan | Time: 2531.01 ms
[2021-07-18 03:47:53 main:581] : INFO : Epoch 1451 | loss: nan | val_loss: nan | Time: 2566.81 ms
[2021-07-18 03:47:55 main:581] : INFO : Epoch 1452 | loss: nan | val_loss: nan | Time: 2507.1 ms
[2021-07-18 03:47:58 main:581] : INFO : Epoch 1453 | loss: nan | val_loss: nan | Time: 2494.42 ms
[2021-07-18 03:48:00 main:581] : INFO : Epoch 1454 | loss: nan | val_loss: nan | Time: 2453.28 ms
[2021-07-18 03:48:02 main:581] : INFO : Epoch 1455 | loss: nan | val_loss: nan | Time: 2367.74 ms
[2021-07-18 03:48:04 main:581] : INFO : Epoch 1456 | loss: nan | val_loss: nan | Time: 2168.87 ms
[2021-07-18 03:48:07 main:581] : INFO : Epoch 1457 | loss: nan | val_loss: nan | Time: 2194.26 ms
[2021-07-18 03:48:09 main:581] : INFO : Epoch 1458 | loss: nan | val_loss: nan | Time: 2456.83 ms
[2021-07-18 03:48:12 main:581] : INFO : Epoch 1459 | loss: nan | val_loss: nan | Time: 2457.46 ms
[2021-07-18 03:48:14 main:581] : INFO : Epoch 1460 | loss: nan | val_loss: nan | Time: 2539.58 ms
[2021-07-18 03:48:17 main:581] : INFO : Epoch 1461 | loss: nan | val_loss: nan | Time: 2524.71 ms
[2021-07-18 03:48:19 main:581] : INFO : Epoch 1462 | loss: nan | val_loss: nan | Time: 2535.37 ms
[2021-07-18 03:48:22 main:581] : INFO : Epoch 1463 | loss: nan | val_loss: nan | Time: 2485.1 ms
[2021-07-18 03:48:24 main:581] : INFO : Epoch 1464 | loss: nan | val_loss: nan | Time: 2345.16 ms
[2021-07-18 03:48:26 main:581] : INFO : Epoch 1465 | loss: nan | val_loss: nan | Time: 2340.06 ms
[2021-07-18 03:48:29 main:581] : INFO : Epoch 1466 | loss: nan | val_loss: nan | Time: 2449.44 ms
[2021-07-18 03:48:31 main:581] : INFO : Epoch 1467 | loss: nan | val_loss: nan | Time: 2413.63 ms
[2021-07-18 03:48:34 main:581] : INFO : Epoch 1468 | loss: nan | val_loss: nan | Time: 2403.24 ms
[2021-07-18 03:48:36 main:581] : INFO : Epoch 1469 | loss: nan | val_loss: nan | Time: 2376.37 ms
[2021-07-18 03:48:39 main:581] : INFO : Epoch 1470 | loss: nan | val_loss: nan | Time: 2524.92 ms
[2021-07-18 03:48:41 main:581] : INFO : Epoch 1471 | loss: nan | val_loss: nan | Time: 2488.57 ms
[2021-07-18 03:48:43 main:581] : INFO : Epoch 1472 | loss: nan | val_loss: nan | Time: 2363.78 ms
[2021-07-18 03:48:46 main:581] : INFO : Epoch 1473 | loss: nan | val_loss: nan | Time: 2480.77 ms
[2021-07-18 03:48:48 main:581] : INFO : Epoch 1474 | loss: nan | val_loss: nan | Time: 2501.63 ms
[2021-07-18 03:48:51 main:581] : INFO : Epoch 1475 | loss: nan | val_loss: nan | Time: 2521.78 ms
[2021-07-18 03:48:53 main:581] : INFO : Epoch 1476 | loss: nan | val_loss: nan | Time: 2467.2 ms
[2021-07-18 03:48:56 main:581] : INFO : Epoch 1477 | loss: nan | val_loss: nan | Time: 2415.58 ms
[2021-07-18 03:48:58 main:581] : INFO : Epoch 1478 | loss: nan | val_loss: nan | Time: 2468.84 ms
[2021-07-18 03:49:01 main:581] : INFO : Epoch 1479 | loss: nan | val_loss: nan | Time: 2505.44 ms
[2021-07-18 03:49:03 main:581] : INFO : Epoch 1480 | loss: nan | val_loss: nan | Time: 2458.81 ms
[2021-07-18 03:49:06 main:581] : INFO : Epoch 1481 | loss: nan | val_loss: nan | Time: 2465.58 ms
[2021-07-18 03:49:08 main:581] : INFO : Epoch 1482 | loss: nan | val_loss: nan | Time: 2465.68 ms
[2021-07-18 03:49:10 main:581] : INFO : Epoch 1483 | loss: nan | val_loss: nan | Time: 2332.83 ms
[2021-07-18 03:49:13 main:581] : INFO : Epoch 1484 | loss: nan | val_loss: nan | Time: 2464.56 ms
[2021-07-18 03:49:15 main:581] : INFO : Epoch 1485 | loss: nan | val_loss: nan | Time: 2466.72 ms
[2021-07-18 03:49:18 main:581] : INFO : Epoch 1486 | loss: nan | val_loss: nan | Time: 2367.18 ms
[2021-07-18 03:49:20 main:581] : INFO : Epoch 1487 | loss: nan | val_loss: nan | Time: 2532.67 ms
[2021-07-18 03:49:23 main:581] : INFO : Epoch 1488 | loss: nan | val_loss: nan | Time: 2533.02 ms
[2021-07-18 03:49:25 main:581] : INFO : Epoch 1489 | loss: nan | val_loss: nan | Time: 2499.13 ms
[2021-07-18 03:49:28 main:581] : INFO : Epoch 1490 | loss: nan | val_loss: nan | Time: 2317.08 ms
[2021-07-18 03:49:30 main:581] : INFO : Epoch 1491 | loss: nan | val_loss: nan | Time: 2372.51 ms
[2021-07-18 03:49:33 main:581] : INFO : Epoch 1492 | loss: nan | val_loss: nan | Time: 2491.55 ms
[2021-07-18 03:49:35 main:581] : INFO : Epoch 1493 | loss: nan | val_loss: nan | Time: 2143.02 ms
[2021-07-18 03:49:39 main:581] : INFO : Epoch 1494 | loss: nan | val_loss: nan | Time: 3958.68 ms
[2021-07-18 03:49:41 main:581] : INFO : Epoch 1495 | loss: nan | val_loss: nan | Time: 2560.44 ms
[2021-07-18 03:49:44 main:581] : INFO : Epoch 1496 | loss: nan | val_loss: nan | Time: 2563.93 ms
[2021-07-18 03:49:47 main:581] : INFO : Epoch 1497 | loss: nan | val_loss: nan | Time: 3073.75 ms
[2021-07-18 03:49:50 main:581] : INFO : Epoch 1498 | loss: nan | val_loss: nan | Time: 3622.85 ms
[2021-07-18 03:49:54 main:581] : INFO : Epoch 1499 | loss: nan | val_loss: nan | Time: 3416.36 ms
[2021-07-18 03:49:58 main:581] : INFO : Epoch 1500 | loss: nan | val_loss: nan | Time: 3959.03 ms
[2021-07-18 03:50:01 main:581] : INFO : Epoch 1501 | loss: nan | val_loss: nan | Time: 3499.78 ms
[2021-07-18 03:50:04 main:581] : INFO : Epoch 1502 | loss: nan | val_loss: nan | Time: 2679.1 ms
[2021-07-18 03:50:07 main:581] : INFO : Epoch 1503 | loss: nan | val_loss: nan | Time: 2547.98 ms
[2021-07-18 03:50:09 main:581] : INFO : Epoch 1504 | loss: nan | val_loss: nan | Time: 2456.59 ms
[2021-07-18 03:50:12 main:581] : INFO : Epoch 1505 | loss: nan | val_loss: nan | Time: 3258.66 ms
[2021-07-18 03:50:14 main:581] : INFO : Epoch 1506 | loss: nan | val_loss: nan | Time: 2165.69 ms
[2021-07-18 03:50:18 main:581] : INFO : Epoch 1507 | loss: nan | val_loss: nan | Time: 3787.13 ms
[2021-07-18 03:50:23 main:581] : INFO : Epoch 1508 | loss: nan | val_loss: nan | Time: 4349.84 ms
[2021-07-18 03:50:26 main:581] : INFO : Epoch 1509 | loss: nan | val_loss: nan | Time: 3744.61 ms
[2021-07-18 03:50:28 main:581] : INFO : Epoch 1510 | loss: nan | val_loss: nan | Time: 2046.09 ms
[2021-07-18 03:50:30 main:581] : INFO : Epoch 1511 | loss: nan | val_loss: nan | Time: 1960.2 ms
[2021-07-18 03:50:33 main:581] : INFO : Epoch 1512 | loss: nan | val_loss: nan | Time: 2403.06 ms
[2021-07-18 03:50:35 main:581] : INFO : Epoch 1513 | loss: nan | val_loss: nan | Time: 2430.17 ms
[2021-07-18 03:50:38 main:581] : INFO : Epoch 1514 | loss: nan | val_loss: nan | Time: 2510.46 ms
[2021-07-18 03:50:40 main:581] : INFO : Epoch 1515 | loss: nan | val_loss: nan | Time: 2236.66 ms
[2021-07-18 03:50:43 main:581] : INFO : Epoch 1516 | loss: nan | val_loss: nan | Time: 2607.7 ms
[2021-07-18 03:50:46 main:581] : INFO : Epoch 1517 | loss: nan | val_loss: nan | Time: 3295.58 ms
[2021-07-18 03:50:49 main:581] : INFO : Epoch 1518 | loss: nan | val_loss: nan | Time: 3549.88 ms
[2021-07-18 03:50:53 main:581] : INFO : Epoch 1519 | loss: nan | val_loss: nan | Time: 3643.88 ms
[2021-07-18 03:50:56 main:581] : INFO : Epoch 1520 | loss: nan | val_loss: nan | Time: 3050.21 ms
[2021-07-18 03:50:59 main:581] : INFO : Epoch 1521 | loss: nan | val_loss: nan | Time: 2700.85 ms
[2021-07-18 03:51:01 main:581] : INFO : Epoch 1522 | loss: nan | val_loss: nan | Time: 2489.03 ms
[2021-07-18 03:51:04 main:581] : INFO : Epoch 1523 | loss: nan | val_loss: nan | Time: 2488.08 ms
[2021-07-18 03:51:06 main:581] : INFO : Epoch 1524 | loss: nan | val_loss: nan | Time: 2141.89 ms
[2021-07-18 03:51:08 main:581] : INFO : Epoch 1525 | loss: nan | val_loss: nan | Time: 2152.89 ms
[2021-07-18 03:51:10 main:581] : INFO : Epoch 1526 | loss: nan | val_loss: nan | Time: 2310.38 ms
[2021-07-18 03:51:13 main:581] : INFO : Epoch 1527 | loss: nan | val_loss: nan | Time: 2534.06 ms
[2021-07-18 03:51:17 main:581] : INFO : Epoch 1528 | loss: nan | val_loss: nan | Time: 3601.43 ms
[2021-07-18 03:51:19 main:581] : INFO : Epoch 1529 | loss: nan | val_loss: nan | Time: 2937.54 ms
[2021-07-18 03:51:22 main:581] : INFO : Epoch 1530 | loss: nan | val_loss: nan | Time: 2810.93 ms
[2021-07-18 03:51:25 main:581] : INFO : Epoch 1531 | loss: nan | val_loss: nan | Time: 2476.76 ms
[2021-07-18 03:51:27 main:581] : INFO : Epoch 1532 | loss: nan | val_loss: nan | Time: 2636.35 ms
[2021-07-18 03:51:29 main:581] : INFO : Epoch 1533 | loss: nan | val_loss: nan | Time: 1743.98 ms
[2021-07-18 03:51:31 main:581] : INFO : Epoch 1534 | loss: nan | val_loss: nan | Time: 1514.05 ms
[2021-07-18 03:51:32 main:581] : INFO : Epoch 1535 | loss: nan | val_loss: nan | Time: 1784.45 ms
[2021-07-18 03:51:34 main:581] : INFO : Epoch 1536 | loss: nan | val_loss: nan | Time: 1978.74 ms
[2021-07-18 03:51:36 main:581] : INFO : Epoch 1537 | loss: nan | val_loss: nan | Time: 1944.66 ms
[2021-07-18 03:51:38 main:581] : INFO : Epoch 1538 | loss: nan | val_loss: nan | Time: 2020.97 ms
[2021-07-18 03:51:40 main:581] : INFO : Epoch 1539 | loss: nan | val_loss: nan | Time: 1988.27 ms
[2021-07-18 03:51:42 main:581] : INFO : Epoch 1540 | loss: nan | val_loss: nan | Time: 2003.72 ms
[2021-07-18 03:51:44 main:581] : INFO : Epoch 1541 | loss: nan | val_loss: nan | Time: 1944.73 ms
[2021-07-18 03:51:46 main:581] : INFO : Epoch 1542 | loss: nan | val_loss: nan | Time: 1952.56 ms
[2021-07-18 03:51:49 main:581] : INFO : Epoch 1543 | loss: nan | val_loss: nan | Time: 3037.25 ms
[2021-07-18 03:51:53 main:581] : INFO : Epoch 1544 | loss: nan | val_loss: nan | Time: 3309.38 ms
[2021-07-18 03:51:56 main:581] : INFO : Epoch 1545 | loss: nan | val_loss: nan | Time: 3289.62 ms
[2021-07-18 03:51:58 main:581] : INFO : Epoch 1546 | loss: nan | val_loss: nan | Time: 2438.47 ms
[2021-07-18 03:52:01 main:581] : INFO : Epoch 1547 | loss: nan | val_loss: nan | Time: 2486.94 ms
[2021-07-18 03:52:04 main:581] : INFO : Epoch 1548 | loss: nan | val_loss: nan | Time: 2686.84 ms
[2021-07-18 03:52:05 main:581] : INFO : Epoch 1549 | loss: nan | val_loss: nan | Time: 1948.45 ms
[2021-07-18 03:52:07 main:581] : INFO : Epoch 1550 | loss: nan | val_loss: nan | Time: 1982.54 ms
[2021-07-18 03:52:09 main:581] : INFO : Epoch 1551 | loss: nan | val_loss: nan | Time: 1967.07 ms
[2021-07-18 03:52:12 main:581] : INFO : Epoch 1552 | loss: nan | val_loss: nan | Time: 2558.13 ms
[2021-07-18 03:52:14 main:581] : INFO : Epoch 1553 | loss: nan | val_loss: nan | Time: 2476.26 ms
[2021-07-18 03:52:17 main:581] : INFO : Epoch 1554 | loss: nan | val_loss: nan | Time: 2856.89 ms
[2021-07-18 03:52:19 main:581] : INFO : Epoch 1555 | loss: nan | val_loss: nan | Time: 1812.95 ms
[2021-07-18 03:52:21 main:581] : INFO : Epoch 1556 | loss: nan | val_loss: nan | Time: 1972.73 ms
[2021-07-18 03:52:23 main:581] : INFO : Epoch 1557 | loss: nan | val_loss: nan | Time: 1849.75 ms
[2021-07-18 03:52:25 main:581] : INFO : Epoch 1558 | loss: nan | val_loss: nan | Time: 1745.39 ms
[2021-07-18 03:52:27 main:581] : INFO : Epoch 1559 | loss: nan | val_loss: nan | Time: 1862.02 ms
[2021-07-18 03:52:29 main:581] : INFO : Epoch 1560 | loss: nan | val_loss: nan | Time: 1982.4 ms
[2021-07-18 03:52:32 main:581] : INFO : Epoch 1561 | loss: nan | val_loss: nan | Time: 3600.9 ms
[2021-07-18 03:52:36 main:581] : INFO : Epoch 1562 | loss: nan | val_loss: nan | Time: 3745.39 ms
[2021-07-18 03:52:38 main:581] : INFO : Epoch 1563 | loss: nan | val_loss: nan | Time: 1877.97 ms
[2021-07-18 03:52:41 main:581] : INFO : Epoch 1564 | loss: nan | val_loss: nan | Time: 2914.49 ms
[2021-07-18 03:52:44 main:581] : INFO : Epoch 1565 | loss: nan | val_loss: nan | Time: 3194.89 ms
[2021-07-18 03:52:47 main:581] : INFO : Epoch 1566 | loss: nan | val_loss: nan | Time: 3338.57 ms
[2021-07-18 03:52:50 main:581] : INFO : Epoch 1567 | loss: nan | val_loss: nan | Time: 3031.76 ms
[2021-07-18 03:52:53 main:581] : INFO : Epoch 1568 | loss: nan | val_loss: nan | Time: 2565.31 ms
[2021-07-18 03:52:56 main:581] : INFO : Epoch 1569 | loss: nan | val_loss: nan | Time: 2926.3 ms
[2021-07-18 03:52:59 main:581] : INFO : Epoch 1570 | loss: nan | val_loss: nan | Time: 3394.63 ms
[2021-07-18 03:53:03 main:581] : INFO : Epoch 1571 | loss: nan | val_loss: nan | Time: 3450.23 ms
[2021-07-18 03:53:06 main:581] : INFO : Epoch 1572 | loss: nan | val_loss: nan | Time: 3350.1 ms
[2021-07-18 03:53:09 main:581] : INFO : Epoch 1573 | loss: nan | val_loss: nan | Time: 3294 ms
[2021-07-18 03:53:13 main:581] : INFO : Epoch 1574 | loss: nan | val_loss: nan | Time: 3467.72 ms
[2021-07-18 03:53:16 main:581] : INFO : Epoch 1575 | loss: nan | val_loss: nan | Time: 3313.47 ms
[2021-07-18 03:53:19 main:581] : INFO : Epoch 1576 | loss: nan | val_loss: nan | Time: 2546.11 ms
[2021-07-18 03:53:21 main:581] : INFO : Epoch 1577 | loss: nan | val_loss: nan | Time: 2510.06 ms
[2021-07-18 03:53:24 main:581] : INFO : Epoch 1578 | loss: nan | val_loss: nan | Time: 2776.33 ms
[2021-07-18 03:53:26 main:581] : INFO : Epoch 1579 | loss: nan | val_loss: nan | Time: 2441.69 ms
[2021-07-18 03:53:29 main:581] : INFO : Epoch 1580 | loss: nan | val_loss: nan | Time: 2729.94 ms
[2021-07-18 03:53:31 main:581] : INFO : Epoch 1581 | loss: nan | val_loss: nan | Time: 2228.71 ms
[2021-07-18 03:53:33 main:581] : INFO : Epoch 1582 | loss: nan | val_loss: nan | Time: 2130.77 ms
[2021-07-18 03:53:36 main:581] : INFO : Epoch 1583 | loss: nan | val_loss: nan | Time: 3096.36 ms
[2021-07-18 03:53:40 main:581] : INFO : Epoch 1584 | loss: nan | val_loss: nan | Time: 3870.35 ms
[2021-07-18 03:53:44 main:581] : INFO : Epoch 1585 | loss: nan | val_loss: nan | Time: 3837.32 ms
[2021-07-18 03:53:47 main:581] : INFO : Epoch 1586 | loss: nan | val_loss: nan | Time: 2931.48 ms
[2021-07-18 03:53:49 main:581] : INFO : Epoch 1587 | loss: nan | val_loss: nan | Time: 2206.83 ms
free(): invalid pointer
SIGABRT: abort called
Stack trace (45 frames):
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x55aade5eff9c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f99ce4e2730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f994fdfa7bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f994fde5535]
/lib/x86_64-linux-gnu/libc.so.6(+0x79508)[0x7f994fe3c508]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fc1a)[0x7f994fe42c1a]
/lib/x86_64-linux-gnu/libc.so.6(+0x8142c)[0x7f994fe4442c]
/lib/x86_64-linux-gnu/libc.so.6(+0x39de4)[0x7f994fdfcde4]
/lib/x86_64-linux-gnu/libc.so.6(+0x39eba)[0x7f994fdfceba]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379b25)[0x55aade5ebb25]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379bee)[0x55aade5ebbee]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f99ce4e2730]
./libcudnn_ops_infer.so.8(+0xbfd374)[0x7f98c4470374]
./libcudnn_ops_infer.so.8(+0x9ab975)[0x7f98c421e975]
./libcudnn_ops_infer.so.8(+0x9a958f)[0x7f98c421c58f]
./libcudnn_ops_infer.so.8(+0x6ac0db)[0x7f98c3f1f0db]
./libcudnn_ops_infer.so.8(+0x68e125)[0x7f98c3f01125]
./libcudnn_ops_infer.so.8(+0x68f8ea)[0x7f98c3f028ea]
./libcudnn_ops_infer.so.8(_ZN5cudnn12cublasGemmExEP13cublasContext17cublasOperation_tS2_iiiPKvS4_14cudaDataType_tiS4_S5_iS4_PvS5_iS5_16cublasGemmAlgo_t+0x12c)[0x7f98c3a9659c]
./libcudnn_adv_infer.so.8(+0x77e31)[0x7f98af487e31]
./libcudnn_adv_infer.so.8(+0xe89f)[0x7f98af41e89f]
./libcudnn_adv_infer.so.8(_Z18RNN_forwardGenericP12cudnnContextP14cudnnRNNStructiPKP17cudnnTensorStructPKvS4_S8_S4_S8_P17cudnnFilterStructS8_S6_PvS4_SB_S4_SB_SB_mSB_m18cudnnForwardMode_t+0x5d7)[0x7f98af4628c7]
./libcudnn_adv_train.so.8(cudnnRNNForwardTraining+0x20e)[0x7f989a61661e]
./libtorch_cuda.so(_ZN2at6native10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS1_EElS3_S3_S3_lllbdbbNS5_IlEES3_+0x2214)[0x7f9950bcaa64]
./libtorch_cuda.so(+0x85face)[0x7f9950c14ace]
./libtorch_cuda.so(+0x895988)[0x7f9950c4a988]
./libtorch_cpu.so(_ZN2at10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS0_EElS2_S2_S2_lllbdbbNS4_IlEES2_+0x2bc)[0x7f99c918351c]
./libtorch_cpu.so(+0x21cf2d4)[0x7f99ca54d2d4]
./libtorch_cpu.so(+0x21edb28)[0x7f99ca56bb28]
./libtorch_cpu.so(_ZN2at10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS0_EElS2_S2_S2_lllbdbbNS4_IlEES2_+0x2bc)[0x7f99c918351c]
./libtorch_cuda.so(+0x80f0fd)[0x7f9950bc40fd]
./libtorch_cuda.so(+0x80fb0a)[0x7f9950bc4b0a]
./libtorch_cpu.so(+0x97fe93)[0x7f99c8cfde93]
./libtorch_cpu.so(_ZN2at6native3gruERKNS_6TensorES3_N3c108ArrayRefIS1_EEbldbbb+0xea)[0x7f99c8ce19ba]
./libtorch_cpu.so(+0xeb0a96)[0x7f99c922ea96]
./libtorch_cpu.so(+0x22e3c92)[0x7f99ca661c92]
./libtorch_cpu.so(+0xedd579)[0x7f99c925b579]
./libtorch_cpu.so(_ZN2at3gruERKNS_6TensorES2_N3c108ArrayRefIS0_EEbldbbb+0x1b1)[0x7f99c9160ee1]
./libtorch_cpu.so(_ZN5torch2nn7GRUImpl14forward_helperERKN2at6TensorES5_S5_lS3_+0x43a)[0x7f99cb0ca4ca]
./libtorch_cpu.so(_ZN5torch2nn7GRUImpl7forwardERKN2at6TensorES3_+0xb0)[0x7f99cb0ca630]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x92fa8)[0x55aade304fa8]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x9a0e6)[0x55aade30c0e6]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8ad10)[0x55aade2fcd10]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f994fde709b]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x55aade2f875a]
Exiting...
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: GeForce RTX 2080 Ti)
[2021-07-18 04:42:35 main:442] : INFO : Set logging level to 1
[2021-07-18 04:42:35 main:448] : INFO : Running in BOINC Client mode
[2021-07-18 04:42:35 main:451] : INFO : Resolving all filenames
[2021-07-18 04:42:35 main:459] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 (exists = 1)
[2021-07-18 04:42:35 main:459] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1625099757-30493-1_0_r1314395814_1 (exists = 1)
[2021-07-18 04:42:35 main:459] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1625099757-30493-1_0_r1314395814_0 (exists = 0)
[2021-07-18 04:42:35 main:459] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1625099757-30493-1 (exists = 1)
[2021-07-18 04:42:35 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2021-07-18 04:42:35 main:479] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5
[2021-07-18 04:42:35 main:481] : INFO : Configuration:
[2021-07-18 04:42:35 main:482] : INFO : Model type: GRU
[2021-07-18 04:42:35 main:483] : INFO : Validation Loss Threshold: 0.0001
[2021-07-18 04:42:35 main:484] : INFO : Max Epochs: 2048
[2021-07-18 04:42:35 main:485] : INFO : Batch Size: 128
[2021-07-18 04:42:35 main:486] : INFO : Learning Rate: 0.01
[2021-07-18 04:42:35 main:487] : INFO : Patience: 10
[2021-07-18 04:42:35 main:488] : INFO : Hidden Width: 12
[2021-07-18 04:42:35 main:489] : INFO : # Recurrent Layers: 4
[2021-07-18 04:42:35 main:490] : INFO : # Backend Layers: 4
[2021-07-18 04:42:35 main:491] : INFO : # Threads: 1
[2021-07-18 04:42:35 main:493] : INFO : Preparing Dataset
[2021-07-18 04:42:35 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-18 04:42:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-18 04:42:44 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2021-07-18 04:42:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-18 04:42:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-07-18 04:42:44 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2021-07-18 04:42:44 main:501] : INFO : Creating Model
[2021-07-18 04:42:44 main:514] : INFO : Preparing config file
[2021-07-18 04:42:44 main:518] : INFO : Found checkpoint, attempting to load...
[2021-07-18 04:42:44 main:519] : INFO : Loading config
terminate called after throwing an instance of 'nlohmann::detail::type_error'
what(): [json.exception.type_error.302] type must be number, but is null
SIGABRT: abort called
Stack trace (24 frames):
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x559cb4309f9c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f070cd59730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f068e6717bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f068e65c535]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x135)[0x559cb43bb7f5]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398846)[0x559cb4324846]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398891)[0x559cb4324891]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x3968c4)[0x559cb43228c4]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe4af0)[0x559cb4070af0]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe1682)[0x559cb406d682]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd9a2b)[0x559cb4065a2b]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd5f7f)[0x559cb4061f7f]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcf7f4)[0x559cb405b7f4]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe16bc)[0x559cb406d6bc]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe4c96)[0x559cb4070c96]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe17ca)[0x559cb406d7ca]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd9b28)[0x559cb4065b28]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd5faf)[0x559cb4061faf]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcf848)[0x559cb405b848]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcc66f)[0x559cb405866f]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xc8d8c)[0x559cb4054d8c]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8a2d1)[0x559cb40162d1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f068e65e09b]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x559cb401275a]
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)