Task 14624022
| Name | ParityModified-1647047553-29267-3-0_4 |
| Workunit | 11618266 |
| Created | 18 Apr 2022, 15:59:11 UTC |
| Sent | 18 Apr 2022, 16:00:02 UTC |
| Report deadline | 26 Apr 2022, 16:00:02 UTC |
| Received | 22 Apr 2022, 8:51:05 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 193 (0x000000C1) EXIT_SIGNAL |
| Computer ID | 12007 |
| Run time | 5 min 19 sec |
| CPU time | 5 min 10 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 1,884.93 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.80 (cuda10200) x86_64-pc-linux-gnu |
| Peak working set size | 1.82 GB |
| Peak swap size | 13.38 GB |
| Peak disk usage | 2.99 GB |
Stderr output
<core_client_version>7.16.6</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63)</message> <stderr_txt> DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048 nthreads: 1 gpudev: 0 Re-exec()-ing to set environment correctly Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce GTX 1050) [2022-04-18 18:00:20 main:442] : INFO : Set logging level to 1 [2022-04-18 18:00:20 main:448] : INFO : Running in BOINC Client mode [2022-04-18 18:00:20 main:451] : INFO : Resolving all filenames [2022-04-18 18:00:20 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-18 18:00:20 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0) [2022-04-18 18:00:20 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-18 18:00:20 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-18 18:00:20 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0) [2022-04-18 18:00:20 main:479] : INFO : Dataset filename: dataset.hdf5 [2022-04-18 18:00:20 main:481] : INFO : Configuration: [2022-04-18 18:00:20 main:482] : INFO : Model type: GRU [2022-04-18 18:00:20 main:483] : INFO : Validation Loss Threshold: 0.0001 [2022-04-18 18:00:20 main:484] : INFO : Max Epochs: 2048 [2022-04-18 18:00:20 main:485] : INFO : Batch Size: 128 [2022-04-18 18:00:20 main:486] : INFO : Learning Rate: 0.01 [2022-04-18 18:00:20 main:487] : INFO : Patience: 10 [2022-04-18 18:00:20 main:488] : INFO : Hidden Width: 12 [2022-04-18 18:00:20 main:489] : INFO : # Recurrent Layers: 4 [2022-04-18 18:00:20 main:490] : INFO : # Backend Layers: 4 [2022-04-18 18:00:20 main:491] : INFO : # Threads: 1 [2022-04-18 18:00:20 main:493] : INFO : Preparing Dataset [2022-04-18 18:00:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-18 18:00:21 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-18 18:00:22 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-18 18:00:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-18 18:00:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-18 18:00:22 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-18 18:00:22 main:501] : INFO : Creating Model [2022-04-18 18:00:22 main:514] : INFO : Preparing config file [2022-04-18 18:00:22 main:526] : INFO : Creating new config file [2022-04-18 18:00:22 main:545] : INFO : This is a continuation WU, loading previous network [2022-04-18 18:00:23 main:566] : INFO : Loading DataLoader into Memory [2022-04-18 18:00:23 main:569] : INFO : Starting Training [2022-04-18 18:00:25 main:581] : INFO : Epoch 1 | loss: nan | val_loss: nan | Time: 2765.4 ms [2022-04-18 18:00:28 main:581] : INFO : Epoch 2 | loss: nan | val_loss: nan | Time: 2367.71 ms [2022-04-18 18:00:30 main:581] : INFO : Epoch 3 | loss: nan | val_loss: nan | Time: 2401.96 ms [2022-04-18 18:00:33 main:581] : INFO : Epoch 4 | loss: nan | val_loss: nan | Time: 2395.05 ms [2022-04-18 18:00:35 main:581] : INFO : Epoch 5 | loss: nan | val_loss: nan | Time: 2339.56 ms [2022-04-18 18:00:37 main:581] : INFO : Epoch 6 | loss: nan | val_loss: nan | Time: 2385.87 ms [2022-04-18 18:00:40 main:581] : INFO : Epoch 7 | loss: nan | val_loss: nan | Time: 2361.47 ms [2022-04-18 18:00:42 main:581] : INFO : Epoch 8 | loss: nan | val_loss: nan | Time: 2382.15 ms [2022-04-18 18:00:44 main:581] : INFO : Epoch 9 | loss: nan | val_loss: nan | Time: 2332.88 ms [2022-04-18 18:00:47 main:581] : INFO : Epoch 10 | loss: nan | val_loss: nan | Time: 2358.42 ms [2022-04-18 18:00:49 main:581] : INFO : Epoch 11 | loss: nan | val_loss: nan | Time: 2346.99 ms [2022-04-18 18:00:51 main:581] : INFO : Epoch 12 | loss: nan | val_loss: nan | Time: 2346.47 ms [2022-04-18 18:00:54 main:581] : INFO : Epoch 13 | loss: nan | val_loss: nan | Time: 2399.26 ms [2022-04-18 18:00:56 main:581] : INFO : Epoch 14 | loss: nan | val_loss: nan | Time: 2382.96 ms [2022-04-18 18:00:59 main:581] : INFO : Epoch 15 | loss: nan | val_loss: nan | Time: 2377.6 ms [2022-04-18 18:01:01 main:581] : INFO : Epoch 16 | loss: nan | val_loss: nan | Time: 2375.93 ms [2022-04-18 18:01:03 main:581] : INFO : Epoch 17 | loss: nan | val_loss: nan | Time: 2399.96 ms [2022-04-18 18:01:06 main:581] : INFO : Epoch 18 | loss: nan | val_loss: nan | Time: 2372.58 ms [2022-04-18 18:01:08 main:581] : INFO : Epoch 19 | loss: nan | val_loss: nan | Time: 2370.24 ms [2022-04-18 18:01:10 main:581] : INFO : Epoch 20 | loss: nan | val_loss: nan | Time: 2332.22 ms [2022-04-18 18:01:13 main:581] : INFO : Epoch 21 | loss: nan | val_loss: nan | Time: 2392.09 ms [2022-04-18 18:01:15 main:581] : INFO : Epoch 22 | loss: nan | val_loss: nan | Time: 2357.78 ms [2022-04-18 18:01:18 main:581] : INFO : Epoch 23 | loss: nan | val_loss: nan | Time: 2384.86 ms [2022-04-18 18:01:20 main:581] : INFO : Epoch 24 | loss: nan | val_loss: nan | Time: 2372.58 ms [2022-04-18 18:01:22 main:581] : INFO : Epoch 25 | loss: nan | val_loss: nan | Time: 2368.87 ms [2022-04-18 18:01:25 main:581] : INFO : Epoch 26 | loss: nan | val_loss: nan | Time: 2385.34 ms [2022-04-18 18:01:27 main:581] : INFO : Epoch 27 | loss: nan | val_loss: nan | Time: 2358.44 ms [2022-04-18 18:01:29 main:581] : INFO : Epoch 28 | loss: nan | val_loss: nan | Time: 2406.87 ms [2022-04-18 18:01:32 main:581] : INFO : Epoch 29 | loss: nan | val_loss: nan | Time: 2370.65 ms [2022-04-18 18:01:34 main:581] : INFO : Epoch 30 | loss: nan | val_loss: nan | Time: 2389.12 ms [2022-04-18 18:01:37 main:581] : INFO : Epoch 31 | loss: nan | val_loss: nan | Time: 2391.95 ms [2022-04-18 18:01:39 main:581] : INFO : Epoch 32 | loss: nan | val_loss: nan | Time: 2362.33 ms [2022-04-18 18:01:41 main:581] : INFO : Epoch 33 | loss: nan | val_loss: nan | Time: 2353.4 ms [2022-04-18 18:01:44 main:581] : INFO : Epoch 34 | loss: nan | val_loss: nan | Time: 2375.33 ms [2022-04-18 18:01:46 main:581] : INFO : Epoch 35 | loss: nan | val_loss: nan | Time: 2407.31 ms [2022-04-18 18:01:49 main:581] : INFO : Epoch 36 | loss: nan | val_loss: nan | Time: 2413.27 ms [2022-04-18 18:01:51 main:581] : INFO : Epoch 37 | loss: nan | val_loss: nan | Time: 2357.96 ms [2022-04-18 18:01:53 main:581] : INFO : Epoch 38 | loss: nan | val_loss: nan | Time: 2394.21 ms [2022-04-18 18:01:56 main:581] : INFO : Epoch 39 | loss: nan | val_loss: nan | Time: 2384.82 ms [2022-04-18 18:01:58 main:581] : INFO : Epoch 40 | loss: nan | val_loss: nan | Time: 2413.34 ms [2022-04-18 18:02:00 main:581] : INFO : Epoch 41 | loss: nan | val_loss: nan | Time: 2417.26 ms [2022-04-18 18:02:03 main:581] : INFO : Epoch 42 | loss: nan | val_loss: nan | Time: 2404.96 ms [2022-04-18 18:02:05 main:581] : INFO : Epoch 43 | loss: nan | val_loss: nan | Time: 2411.72 ms [2022-04-18 18:02:08 main:581] : INFO : Epoch 44 | loss: nan | val_loss: nan | Time: 2382.03 ms [2022-04-18 18:02:10 main:581] : INFO : Epoch 45 | loss: nan | val_loss: nan | Time: 2362.38 ms [2022-04-18 18:02:12 main:581] : INFO : Epoch 46 | loss: nan | val_loss: nan | Time: 2403.67 ms [2022-04-18 18:02:15 main:581] : INFO : Epoch 47 | loss: nan | val_loss: nan | Time: 2383.01 ms [2022-04-18 18:02:17 main:581] : INFO : Epoch 48 | loss: nan | val_loss: nan | Time: 2375.1 ms [2022-04-18 18:02:20 main:581] : INFO : Epoch 49 | loss: nan | val_loss: nan | Time: 2372.17 ms [2022-04-18 18:02:22 main:581] : INFO : Epoch 50 | loss: nan | val_loss: nan | Time: 2374.84 ms [2022-04-18 18:02:24 main:581] : INFO : Epoch 51 | loss: nan | val_loss: nan | Time: 2393.77 ms [2022-04-18 18:02:27 main:581] : INFO : Epoch 52 | loss: nan | val_loss: nan | Time: 2387.84 ms [2022-04-18 18:02:29 main:581] : INFO : Epoch 53 | loss: nan | val_loss: nan | Time: 2375.64 ms [2022-04-18 18:02:31 main:581] : INFO : Epoch 54 | loss: nan | val_loss: nan | Time: 2376.57 ms [2022-04-18 18:02:34 main:581] : INFO : Epoch 55 | loss: nan | val_loss: nan | Time: 2365.11 ms [2022-04-18 18:02:36 main:581] : INFO : Epoch 56 | loss: nan | val_loss: nan | Time: 2350.02 ms [2022-04-18 18:02:39 main:581] : INFO : Epoch 57 | loss: nan | val_loss: nan | Time: 2347.16 ms [2022-04-18 18:02:41 main:581] : INFO : Epoch 58 | loss: nan | val_loss: nan | Time: 2354.37 ms [2022-04-18 18:02:43 main:581] : INFO : Epoch 59 | loss: nan | val_loss: nan | Time: 2380.82 ms [2022-04-18 18:02:46 main:581] : INFO : Epoch 60 | loss: nan | val_loss: nan | Time: 2367.37 ms [2022-04-18 18:02:48 main:581] : INFO : Epoch 61 | loss: nan | val_loss: nan | Time: 2405.25 ms [2022-04-18 18:02:50 main:581] : INFO : Epoch 62 | loss: nan | val_loss: nan | Time: 2380.95 ms [2022-04-18 18:02:53 main:581] : INFO : Epoch 63 | loss: nan | val_loss: nan | Time: 2436.31 ms [2022-04-18 18:02:55 main:581] : INFO : Epoch 64 | loss: nan | val_loss: nan | Time: 2397.96 ms [2022-04-18 18:02:58 main:581] : INFO : Epoch 65 | loss: nan | val_loss: nan | Time: 2413.21 ms [2022-04-18 18:03:00 main:581] : INFO : Epoch 66 | loss: nan | val_loss: nan | Time: 2336.09 ms [2022-04-18 18:03:02 main:581] : INFO : Epoch 67 | loss: nan | val_loss: nan | Time: 2334.88 ms [2022-04-18 18:03:05 main:581] : INFO : Epoch 68 | loss: nan | val_loss: nan | Time: 2354.33 ms [2022-04-18 18:03:07 main:581] : INFO : Epoch 69 | loss: nan | val_loss: nan | Time: 2371.29 ms [2022-04-18 18:03:09 main:581] : INFO : Epoch 70 | loss: nan | val_loss: nan | Time: 2272.48 ms [2022-04-18 18:03:12 main:581] : INFO : Epoch 71 | loss: nan | val_loss: nan | Time: 2356.87 ms [2022-04-18 18:03:14 main:581] : INFO : Epoch 72 | loss: nan | val_loss: nan | Time: 2378.58 ms [2022-04-18 18:03:16 main:581] : INFO : Epoch 73 | loss: nan | val_loss: nan | Time: 2342.58 ms [2022-04-18 18:03:19 main:581] : INFO : Epoch 74 | loss: nan | val_loss: nan | Time: 2370.6 ms [2022-04-18 18:03:21 main:581] : INFO : Epoch 75 | loss: nan | val_loss: nan | Time: 2356.19 ms [2022-04-18 18:03:24 main:581] : INFO : Epoch 76 | loss: nan | val_loss: nan | Time: 2323.61 ms [2022-04-18 18:03:26 main:581] : INFO : Epoch 77 | loss: nan | val_loss: nan | Time: 2368.13 ms [2022-04-18 18:03:28 main:581] : INFO : Epoch 78 | loss: nan | val_loss: nan | Time: 2296.22 ms [2022-04-18 18:03:31 main:581] : INFO : Epoch 79 | loss: nan | val_loss: nan | Time: 2382.52 ms [2022-04-18 18:03:33 main:581] : INFO : Epoch 80 | loss: nan | val_loss: nan | Time: 2374.02 ms [2022-04-18 18:03:35 main:581] : INFO : Epoch 81 | loss: nan | val_loss: nan | Time: 2378.73 ms [2022-04-18 18:03:38 main:581] : INFO : Epoch 82 | loss: nan | val_loss: nan | Time: 2369.5 ms [2022-04-18 18:03:40 main:581] : INFO : Epoch 83 | loss: nan | val_loss: nan | Time: 2342.41 ms [2022-04-18 18:03:42 main:581] : INFO : Epoch 84 | loss: nan | val_loss: nan | Time: 2376.27 ms [2022-04-18 18:03:45 main:581] : INFO : Epoch 85 | loss: nan | val_loss: nan | Time: 2345.47 ms [2022-04-18 18:03:47 main:581] : INFO : Epoch 86 | loss: nan | val_loss: nan | Time: 2349.05 ms [2022-04-18 18:03:49 main:581] : INFO : Epoch 87 | loss: nan | val_loss: nan | Time: 2342.03 ms [2022-04-18 18:03:52 main:581] : INFO : Epoch 88 | loss: nan | val_loss: nan | Time: 2348.38 ms [2022-04-18 18:03:54 main:581] : INFO : Epoch 89 | loss: nan | val_loss: nan | Time: 2336.43 ms [2022-04-18 18:03:56 main:581] : INFO : Epoch 90 | loss: nan | val_loss: nan | Time: 2377.6 ms [2022-04-18 18:03:59 main:581] : INFO : Epoch 91 | loss: nan | val_loss: nan | Time: 2344.24 ms [2022-04-18 18:04:01 main:581] : INFO : Epoch 92 | loss: nan | val_loss: nan | Time: 2355.91 ms [2022-04-18 18:04:04 main:581] : INFO : Epoch 93 | loss: nan | val_loss: nan | Time: 2359.67 ms [2022-04-18 18:04:06 main:581] : INFO : Epoch 94 | loss: nan | val_loss: nan | Time: 2339.11 ms [2022-04-18 18:04:08 main:581] : INFO : Epoch 95 | loss: nan | val_loss: nan | Time: 2358.28 ms [2022-04-18 18:04:11 main:581] : INFO : Epoch 96 | loss: nan | val_loss: nan | Time: 2347.53 ms [2022-04-18 18:04:13 main:581] : INFO : Epoch 97 | loss: nan | val_loss: nan | Time: 2384.33 ms [2022-04-18 18:04:15 main:581] : INFO : Epoch 98 | loss: nan | val_loss: nan | Time: 2379.99 ms [2022-04-18 18:04:18 main:581] : INFO : Epoch 99 | loss: nan | val_loss: nan | Time: 2365.94 ms [2022-04-18 18:04:20 main:581] : INFO : Epoch 100 | loss: nan | val_loss: nan | Time: 2404.73 ms [2022-04-18 18:04:22 main:581] : INFO : Epoch 101 | loss: nan | val_loss: nan | Time: 2341.52 ms [2022-04-18 18:04:25 main:581] : INFO : Epoch 102 | loss: nan | val_loss: nan | Time: 2327.76 ms [2022-04-18 18:04:27 main:581] : INFO : Epoch 103 | loss: nan | val_loss: nan | Time: 2390.64 ms [2022-04-18 18:04:30 main:581] : INFO : Epoch 104 | loss: nan | val_loss: nan | Time: 2336.16 ms [2022-04-18 18:04:32 main:581] : INFO : Epoch 105 | loss: nan | val_loss: nan | Time: 2365.2 ms [2022-04-18 18:04:34 main:581] : INFO : Epoch 106 | loss: nan | val_loss: nan | Time: 2352.61 ms [2022-04-18 18:04:37 main:581] : INFO : Epoch 107 | loss: nan | val_loss: nan | Time: 2394.02 ms [2022-04-18 18:04:39 main:581] : INFO : Epoch 108 | loss: nan | val_loss: nan | Time: 2349.79 ms [2022-04-18 18:04:41 main:581] : INFO : Epoch 109 | loss: nan | val_loss: nan | Time: 2341.26 ms [2022-04-18 18:04:44 main:581] : INFO : Epoch 110 | loss: nan | val_loss: nan | Time: 2350.88 ms [2022-04-18 18:04:46 main:581] : INFO : Epoch 111 | loss: nan | val_loss: nan | Time: 2365.06 ms [2022-04-18 18:04:48 main:581] : INFO : Epoch 112 | loss: nan | val_loss: nan | Time: 2370.87 ms [2022-04-18 18:04:51 main:581] : INFO : Epoch 113 | loss: nan | val_loss: nan | Time: 2384.06 ms [2022-04-18 18:04:53 main:581] : INFO : Epoch 114 | loss: nan | val_loss: nan | Time: 2337.23 ms [2022-04-18 18:04:55 main:581] : INFO : Epoch 115 | loss: nan | val_loss: nan | Time: 2343.93 ms [2022-04-18 18:04:58 main:581] : INFO : Epoch 116 | loss: nan | val_loss: nan | Time: 2343.82 ms [2022-04-18 18:05:00 main:581] : INFO : Epoch 117 | loss: nan | val_loss: nan | Time: 2344.69 ms [2022-04-18 18:05:03 main:581] : INFO : Epoch 118 | loss: nan | val_loss: nan | Time: 2378.34 ms [2022-04-18 18:05:05 main:581] : INFO : Epoch 119 | loss: nan | val_loss: nan | Time: 2354.39 ms [2022-04-18 18:05:07 main:581] : INFO : Epoch 120 | loss: nan | val_loss: nan | Time: 2298.72 ms [2022-04-18 18:05:10 main:581] : INFO : Epoch 121 | loss: nan | val_loss: nan | Time: 2330.74 ms [2022-04-18 18:05:12 main:581] : INFO : Epoch 122 | loss: nan | val_loss: nan | Time: 2367.16 ms [2022-04-18 18:05:14 main:581] : INFO : Epoch 123 | loss: nan | val_loss: nan | Time: 2349.89 ms [2022-04-18 18:05:17 main:581] : INFO : Epoch 124 | loss: nan | val_loss: nan | Time: 2321.82 ms [2022-04-18 18:05:19 main:581] : INFO : Epoch 125 | loss: nan | val_loss: nan | Time: 2347.24 ms [2022-04-18 18:05:21 main:581] : INFO : Epoch 126 | loss: nan | val_loss: nan | Time: 2301.53 ms [2022-04-18 18:05:24 main:581] : INFO : Epoch 127 | loss: nan | val_loss: nan | Time: 2278.44 ms [2022-04-18 18:05:26 main:581] : INFO : Epoch 128 | loss: nan | val_loss: nan | Time: 2357.78 ms [2022-04-18 18:05:28 main:581] : INFO : Epoch 129 | loss: nan | val_loss: nan | Time: 2338.38 ms [2022-04-18 18:05:31 main:581] : INFO : Epoch 130 | loss: nan | val_loss: nan | Time: 2313.48 ms [2022-04-18 18:05:33 main:581] : INFO : Epoch 131 | loss: nan | val_loss: nan | Time: 2298.07 ms [2022-04-18 18:05:35 main:581] : INFO : Epoch 132 | loss: nan | val_loss: nan | Time: 2323.82 ms [2022-04-18 18:05:37 main:581] : INFO : Epoch 133 | loss: nan | val_loss: nan | Time: 2288.79 ms [2022-04-18 18:05:40 main:581] : INFO : Epoch 134 | loss: nan | val_loss: nan | Time: 2319.59 ms [2022-04-18 18:05:42 main:581] : INFO : Epoch 135 | loss: nan | val_loss: nan | Time: 2342.08 ms [2022-04-18 18:05:44 main:581] : INFO : Epoch 136 | loss: nan | val_loss: nan | Time: 2358.76 ms [2022-04-18 18:05:47 main:581] : INFO : Epoch 137 | loss: nan | val_loss: nan | Time: 2323.43 ms [2022-04-18 18:05:49 main:581] : INFO : Epoch 138 | loss: nan | val_loss: nan | Time: 2338.04 ms [2022-04-18 18:05:51 main:581] : INFO : Epoch 139 | loss: nan | val_loss: nan | Time: 2343.99 ms [2022-04-18 18:05:54 main:581] : INFO : Epoch 140 | loss: nan | val_loss: nan | Time: 2330.02 ms [2022-04-18 18:05:56 main:581] : INFO : Epoch 141 | loss: nan | val_loss: nan | Time: 2320.48 ms [2022-04-18 18:05:58 main:581] : INFO : Epoch 142 | loss: nan | val_loss: nan | Time: 2323.81 ms [2022-04-18 18:06:01 main:581] : INFO : Epoch 143 | loss: nan | val_loss: nan | Time: 2311.14 ms [2022-04-18 18:06:03 main:581] : INFO : Epoch 144 | loss: nan | val_loss: nan | Time: 2339.62 ms [2022-04-18 18:06:05 main:581] : INFO : Epoch 145 | loss: nan | val_loss: nan | Time: 2283.58 ms [2022-04-18 18:06:08 main:581] : INFO : Epoch 146 | loss: nan | val_loss: nan | Time: 2302.62 ms [2022-04-18 18:06:10 main:581] : INFO : Epoch 147 | loss: nan | val_loss: nan | Time: 2316.52 ms [2022-04-18 18:06:12 main:581] : INFO : Epoch 148 | loss: nan | val_loss: nan | Time: 2369.68 ms [2022-04-18 18:06:15 main:581] : INFO : Epoch 149 | loss: nan | val_loss: nan | Time: 2351.77 ms [2022-04-18 18:06:17 main:581] : INFO : Epoch 150 | loss: nan | val_loss: nan | Time: 2378.66 ms [2022-04-18 18:06:19 main:581] : INFO : Epoch 151 | loss: nan | val_loss: nan | Time: 2350.08 ms [2022-04-18 18:06:22 main:581] : INFO : Epoch 152 | loss: nan | val_loss: nan | Time: 2341.46 ms [2022-04-18 18:06:24 main:581] : INFO : Epoch 153 | loss: nan | val_loss: nan | Time: 2329.52 ms [2022-04-18 18:06:26 main:581] : INFO : Epoch 154 | loss: nan | val_loss: nan | Time: 2338.71 ms [2022-04-18 18:06:29 main:581] : INFO : Epoch 155 | loss: nan | val_loss: nan | Time: 2385.47 ms [2022-04-18 18:06:31 main:581] : INFO : Epoch 156 | loss: nan | val_loss: nan | Time: 2389.34 ms [2022-04-18 18:06:34 main:581] : INFO : Epoch 157 | loss: nan | val_loss: nan | Time: 2343.96 ms [2022-04-18 18:06:36 main:581] : INFO : Epoch 158 | loss: nan | val_loss: nan | Time: 2329.92 ms [2022-04-18 18:06:38 main:581] : INFO : Epoch 159 | loss: nan | val_loss: nan | Time: 2342.21 ms [2022-04-18 18:06:41 main:581] : INFO : Epoch 160 | loss: nan | val_loss: nan | Time: 2343.13 ms [2022-04-18 18:06:43 main:581] : INFO : Epoch 161 | loss: nan | val_loss: nan | Time: 2341.92 ms [2022-04-18 18:06:45 main:581] : INFO : Epoch 162 | loss: nan | val_loss: nan | Time: 2344.91 ms [2022-04-18 18:06:48 main:581] : INFO : Epoch 163 | loss: nan | val_loss: nan | Time: 2341.8 ms [2022-04-18 18:06:50 main:581] : INFO : Epoch 164 | loss: nan | val_loss: nan | Time: 2370.22 ms [2022-04-18 18:06:52 main:581] : INFO : Epoch 165 | loss: nan | val_loss: nan | Time: 2345.01 ms [2022-04-18 18:06:55 main:581] : INFO : Epoch 166 | loss: nan | val_loss: nan | Time: 2334.73 ms [2022-04-18 18:06:57 main:581] : INFO : Epoch 167 | loss: nan | val_loss: nan | Time: 2338.7 ms [2022-04-18 18:06:59 main:581] : INFO : Epoch 168 | loss: nan | val_loss: nan | Time: 2353.8 ms [2022-04-18 18:07:02 main:581] : INFO : Epoch 169 | loss: nan | val_loss: nan | Time: 2346.57 ms [2022-04-18 18:07:04 main:581] : INFO : Epoch 170 | loss: nan | val_loss: nan | Time: 2346.1 ms [2022-04-18 18:07:06 main:581] : INFO : Epoch 171 | loss: nan | val_loss: nan | Time: 2333.18 ms [2022-04-18 18:07:09 main:581] : INFO : Epoch 172 | loss: nan | val_loss: nan | Time: 2349.16 ms [2022-04-18 18:07:11 main:581] : INFO : Epoch 173 | loss: nan | val_loss: nan | Time: 2325.62 ms [2022-04-18 18:07:13 main:581] : INFO : Epoch 174 | loss: nan | val_loss: nan | Time: 2317.49 ms [2022-04-18 18:07:16 main:581] : INFO : Epoch 175 | loss: nan | val_loss: nan | Time: 2374.12 ms [2022-04-18 18:07:18 main:581] : INFO : Epoch 176 | loss: nan | val_loss: nan | Time: 2546.71 ms [2022-04-18 18:07:21 main:581] : INFO : Epoch 177 | loss: nan | val_loss: nan | Time: 2348.48 ms [2022-04-18 18:07:23 main:581] : INFO : Epoch 178 | loss: nan | val_loss: nan | Time: 2371.22 ms [2022-04-18 18:07:25 main:581] : INFO : Epoch 179 | loss: nan | val_loss: nan | Time: 2440.49 ms [2022-04-18 18:07:28 main:581] : INFO : Epoch 180 | loss: nan | val_loss: nan | Time: 2344.54 ms [2022-04-18 18:07:30 main:581] : INFO : Epoch 181 | loss: nan | val_loss: nan | Time: 2345.14 ms [2022-04-18 18:07:33 main:581] : INFO : Epoch 182 | loss: nan | val_loss: nan | Time: 2379.59 ms [2022-04-18 18:07:35 main:581] : INFO : Epoch 183 | loss: nan | val_loss: nan | Time: 2360.57 ms [2022-04-18 18:07:37 main:581] : INFO : Epoch 184 | loss: nan | val_loss: nan | Time: 2338.48 ms [2022-04-18 18:07:40 main:581] : INFO : Epoch 185 | loss: nan | val_loss: nan | Time: 2371.12 ms [2022-04-18 18:07:42 main:581] : INFO : Epoch 186 | loss: nan | val_loss: nan | Time: 2391.39 ms [2022-04-18 18:07:44 main:581] : INFO : Epoch 187 | loss: nan | val_loss: nan | Time: 2366 ms [2022-04-18 18:07:47 main:581] : INFO : Epoch 188 | loss: nan | val_loss: nan | Time: 2346.02 ms [2022-04-18 18:07:49 main:581] : INFO : Epoch 189 | loss: nan | val_loss: nan | Time: 2364.56 ms [2022-04-18 18:07:51 main:581] : INFO : Epoch 190 | loss: nan | val_loss: nan | Time: 2389.07 ms [2022-04-18 18:07:54 main:581] : INFO : Epoch 191 | loss: nan | val_loss: nan | Time: 2337.57 ms [2022-04-18 18:07:56 main:581] : INFO : Epoch 192 | loss: nan | val_loss: nan | Time: 2331.18 ms [2022-04-18 18:07:58 main:581] : INFO : Epoch 193 | loss: nan | val_loss: nan | Time: 2367.54 ms [2022-04-18 18:08:01 main:581] : INFO : Epoch 194 | loss: nan | val_loss: nan | Time: 2364.59 ms [2022-04-18 18:08:03 main:581] : INFO : Epoch 195 | loss: nan | val_loss: nan | Time: 2361.2 ms DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048 nthreads: 1 gpudev: 0 Re-exec()-ing to set environment correctly Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce GTX 1050) [2022-04-22 09:50:53 main:442] : INFO : Set logging level to 1 [2022-04-22 09:50:53 main:448] : INFO : Running in BOINC Client mode [2022-04-22 09:50:53 main:451] : INFO : Resolving all filenames [2022-04-22 09:50:53 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-22 09:50:53 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-22 09:50:53 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-22 09:50:53 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-22 09:50:53 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-22 09:50:53 main:479] : INFO : Dataset filename: dataset.hdf5 [2022-04-22 09:50:53 main:481] : INFO : Configuration: [2022-04-22 09:50:53 main:482] : INFO : Model type: GRU [2022-04-22 09:50:53 main:483] : INFO : Validation Loss Threshold: 0.0001 [2022-04-22 09:50:53 main:484] : INFO : Max Epochs: 2048 [2022-04-22 09:50:53 main:485] : INFO : Batch Size: 128 [2022-04-22 09:50:53 main:486] : INFO : Learning Rate: 0.01 [2022-04-22 09:50:53 main:487] : INFO : Patience: 10 [2022-04-22 09:50:53 main:488] : INFO : Hidden Width: 12 [2022-04-22 09:50:53 main:489] : INFO : # Recurrent Layers: 4 [2022-04-22 09:50:53 main:490] : INFO : # Backend Layers: 4 [2022-04-22 09:50:53 main:491] : INFO : # Threads: 1 [2022-04-22 09:50:53 main:493] : INFO : Preparing Dataset [2022-04-22 09:50:53 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-22 09:50:54 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-22 09:51:03 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-22 09:51:03 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-22 09:51:03 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-22 09:51:03 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-22 09:51:03 main:501] : INFO : Creating Model [2022-04-22 09:51:03 main:514] : INFO : Preparing config file [2022-04-22 09:51:03 main:518] : INFO : Found checkpoint, attempting to load... [2022-04-22 09:51:03 main:519] : INFO : Loading config terminate called after throwing an instance of 'nlohmann::detail::type_error' what(): [json.exception.type_error.302] type must be number, but is null SIGABRT: abort called Stack trace (24 frames): ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x56420757df9c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0)[0x7ff79e6b73c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7ff71ffdb03b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7ff71ffba859] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x135)[0x56420762f7f5] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398846)[0x564207598846] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398891)[0x564207598891] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x3968c4)[0x5642075968c4] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe4af0)[0x5642072e4af0] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe1682)[0x5642072e1682] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd9a2b)[0x5642072d9a2b] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd5f7f)[0x5642072d5f7f] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcf7f4)[0x5642072cf7f4] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe16bc)[0x5642072e16bc] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe4c96)[0x5642072e4c96] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xe17ca)[0x5642072e17ca] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd9b28)[0x5642072d9b28] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xd5faf)[0x5642072d5faf] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcf848)[0x5642072cf848] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xcc66f)[0x5642072cc66f] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xc8d8c)[0x5642072c8d8c] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8a2d1)[0x56420728a2d1] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7ff71ffbc0b3] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x56420728675a] Exiting... </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)