Task 10019514
| Name | ParityModified-1639958619-28255-0_0 |
| Workunit | 7398243 |
| Created | 20 Dec 2021, 0:03:43 UTC |
| Sent | 2 Jan 2022, 19:38:30 UTC |
| Report deadline | 10 Jan 2022, 19:38:30 UTC |
| Received | 11 Jan 2022, 1:24:49 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Aborted by user |
| Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
| Computer ID | 157 |
| Run time | 16 hours 26 min 5 sec |
| CPU time | 5 hours 51 min 37 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 4,474.06 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.63 GB |
| Peak swap size | 3.61 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.20</core_client_version> <![CDATA[ <message> aborted by user</message> <stderr_txt> [2022-01-10 11:11:54 main:478] : INFO : Batch Size: 128 [2022-01-10 11:11:55 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 11:11:55 main:480] : INFO : Patience: 10 [2022-01-10 11:11:55 main:481] : INFO : Hidden Width: 12 [2022-01-10 11:11:55 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 11:11:55 main:483] : INFO : # Backend Layers: 4 [2022-01-10 11:11:55 main:484] : INFO : # Threads: 1 [2022-01-10 11:11:55 main:486] : INFO : Preparing Dataset [2022-01-10 11:11:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 11:14:00 main:435] : INFO : Set logging level to 1 [2022-01-10 11:14:00 main:441] : INFO : Running in BOINC Client mode [2022-01-10 11:14:00 main:444] : INFO : Resolving all filenames [2022-01-10 11:14:00 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 11:14:00 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 11:14:00 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 11:14:00 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 11:14:00 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 11:14:00 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 11:14:01 main:474] : INFO : Configuration: [2022-01-10 11:14:06 main:475] : INFO : Model type: GRU [2022-01-10 11:14:07 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 11:14:07 main:477] : INFO : Max Epochs: 2048 [2022-01-10 11:14:07 main:478] : INFO : Batch Size: 128 [2022-01-10 11:14:07 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 11:14:07 main:480] : INFO : Patience: 10 [2022-01-10 11:14:07 main:481] : INFO : Hidden Width: 12 [2022-01-10 11:14:07 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 11:14:07 main:483] : INFO : # Backend Layers: 4 [2022-01-10 11:14:07 main:484] : INFO : # Threads: 1 [2022-01-10 11:14:07 main:486] : INFO : Preparing Dataset [2022-01-10 11:14:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 11:14:41 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 11:14:44 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 11:14:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 11:14:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 11:14:44 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 11:14:44 main:494] : INFO : Creating Model [2022-01-10 11:14:44 main:507] : INFO : Preparing config file [2022-01-10 11:14:44 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 11:14:44 main:512] : INFO : Loading config [2022-01-10 11:14:44 main:514] : INFO : Loading state Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 11:26:40 main:435] : INFO : Set logging level to 1 [2022-01-10 11:26:41 main:441] : INFO : Running in BOINC Client mode [2022-01-10 11:26:41 main:444] : INFO : Resolving all filenames [2022-01-10 11:26:41 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 11:26:41 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 11:26:41 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 11:26:41 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 11:26:41 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 11:26:41 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 11:26:41 main:474] : INFO : Configuration: [2022-01-10 11:26:41 main:475] : INFO : Model type: GRU [2022-01-10 11:26:41 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 11:26:41 main:477] : INFO : Max Epochs: 2048 [2022-01-10 11:26:41 main:478] : INFO : Batch Size: 128 [2022-01-10 11:26:47 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 11:26:48 main:480] : INFO : Patience: 10 [2022-01-10 11:26:48 main:481] : INFO : Hidden Width: 12 [2022-01-10 11:26:48 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 11:26:48 main:483] : INFO : # Backend Layers: 4 [2022-01-10 11:26:48 main:484] : INFO : # Threads: 1 [2022-01-10 11:26:48 main:486] : INFO : Preparing Dataset [2022-01-10 11:26:48 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 11:27:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 11:27:26 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 11:27:26 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 11:27:26 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 11:27:26 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 11:27:26 main:494] : INFO : Creating Model [2022-01-10 11:27:26 main:507] : INFO : Preparing config file [2022-01-10 11:27:26 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 11:27:26 main:512] : INFO : Loading config [2022-01-10 11:27:26 main:514] : INFO : Loading state [2022-01-10 11:27:28 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 11:27:28 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 11:31:48 main:435] : INFO : Set logging level to 1 [2022-01-10 11:31:48 main:441] : INFO : Running in BOINC Client mode [2022-01-10 11:31:48 main:444] : INFO : Resolving all filenames [2022-01-10 11:31:48 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 11:31:48 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 11:31:48 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 11:31:48 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 11:31:48 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 11:31:48 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 11:31:48 main:474] : INFO : Configuration: [2022-01-10 11:31:48 main:475] : INFO : Model type: GRU [2022-01-10 11:31:48 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 11:31:48 main:477] : INFO : Max Epochs: 2048 [2022-01-10 11:31:54 main:478] : INFO : Batch Size: 128 [2022-01-10 11:31:54 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 11:31:54 main:480] : INFO : Patience: 10 [2022-01-10 11:31:54 main:481] : INFO : Hidden Width: 12 [2022-01-10 11:31:54 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 11:31:54 main:483] : INFO : # Backend Layers: 4 [2022-01-10 11:31:54 main:484] : INFO : # Threads: 1 [2022-01-10 11:31:54 main:486] : INFO : Preparing Dataset [2022-01-10 11:31:54 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 11:32:30 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 11:32:37 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 11:32:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 11:32:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 11:32:52 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 11:32:52 main:494] : INFO : Creating Model [2022-01-10 11:32:52 main:507] : INFO : Preparing config file [2022-01-10 11:32:52 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 11:32:52 main:512] : INFO : Loading config [2022-01-10 11:32:53 main:514] : INFO : Loading state [2022-01-10 11:32:59 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 11:32:59 main:562] : INFO : Starting Training [2022-01-10 11:43:51 main:574] : INFO : Epoch 1801 | loss: 0.0311445 | val_loss: 0.0312432 | Time: 652041 ms [2022-01-10 11:53:12 main:574] : INFO : Epoch 1802 | loss: 0.0310652 | val_loss: 0.0312971 | Time: 560593 ms [2022-01-10 11:58:46 main:574] : INFO : Epoch 1803 | loss: 0.0310252 | val_loss: 0.0312934 | Time: 334415 ms [2022-01-10 12:11:37 main:574] : INFO : Epoch 1804 | loss: 0.0310251 | val_loss: 0.031322 | Time: 770630 ms [2022-01-10 12:17:58 main:574] : INFO : Epoch 1805 | loss: 0.0310227 | val_loss: 0.0313104 | Time: 381065 ms [2022-01-10 12:23:38 main:574] : INFO : Epoch 1806 | loss: 0.0310209 | val_loss: 0.0313357 | Time: 339560 ms [2022-01-10 12:37:38 main:574] : INFO : Epoch 1807 | loss: 0.0310144 | val_loss: 0.0312965 | Time: 839488 ms [2022-01-10 12:45:36 main:574] : INFO : Epoch 1808 | loss: 0.0310095 | val_loss: 0.0312897 | Time: 472436 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 13:03:23 main:435] : INFO : Set logging level to 1 [2022-01-10 13:03:23 main:441] : INFO : Running in BOINC Client mode [2022-01-10 13:03:23 main:444] : INFO : Resolving all filenames [2022-01-10 13:03:24 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 13:03:24 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 13:03:30 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 13:03:30 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 13:03:30 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 13:03:30 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 13:03:30 main:474] : INFO : Configuration: [2022-01-10 13:03:30 main:475] : INFO : Model type: GRU [2022-01-10 13:03:30 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 13:03:30 main:477] : INFO : Max Epochs: 2048 [2022-01-10 13:03:30 main:478] : INFO : Batch Size: 128 [2022-01-10 13:03:30 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 13:03:30 main:480] : INFO : Patience: 10 [2022-01-10 13:03:30 main:481] : INFO : Hidden Width: 12 [2022-01-10 13:03:30 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 13:03:31 main:483] : INFO : # Backend Layers: 4 [2022-01-10 13:03:31 main:484] : INFO : # Threads: 1 [2022-01-10 13:03:31 main:486] : INFO : Preparing Dataset [2022-01-10 13:03:31 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 13:03:47 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 13:04:10 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 13:04:10 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 13:04:10 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 13:04:16 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 13:04:16 main:494] : INFO : Creating Model [2022-01-10 13:04:16 main:507] : INFO : Preparing config file [2022-01-10 13:04:16 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 13:04:16 main:512] : INFO : Loading config [2022-01-10 13:04:16 main:514] : INFO : Loading state [2022-01-10 13:04:17 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 13:04:17 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 13:08:31 main:435] : INFO : Set logging level to 1 [2022-01-10 13:08:31 main:441] : INFO : Running in BOINC Client mode [2022-01-10 13:08:31 main:444] : INFO : Resolving all filenames [2022-01-10 13:08:32 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 13:08:32 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 13:08:32 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 13:08:32 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 13:08:32 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 13:08:32 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 13:08:32 main:474] : INFO : Configuration: [2022-01-10 13:08:32 main:475] : INFO : Model type: GRU [2022-01-10 13:08:32 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 13:08:32 main:477] : INFO : Max Epochs: 2048 [2022-01-10 13:08:32 main:478] : INFO : Batch Size: 128 [2022-01-10 13:08:32 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 13:08:32 main:480] : INFO : Patience: 10 [2022-01-10 13:08:33 main:481] : INFO : Hidden Width: 12 [2022-01-10 13:08:33 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 13:08:33 main:483] : INFO : # Backend Layers: 4 [2022-01-10 13:08:33 main:484] : INFO : # Threads: 1 [2022-01-10 13:08:33 main:486] : INFO : Preparing Dataset [2022-01-10 13:08:33 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 13:09:26 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 13:09:53 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 13:09:54 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 13:09:54 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 13:10:09 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 13:10:09 main:494] : INFO : Creating Model [2022-01-10 13:10:09 main:507] : INFO : Preparing config file [2022-01-10 13:10:09 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 13:10:09 main:512] : INFO : Loading config [2022-01-10 13:10:09 main:514] : INFO : Loading state [2022-01-10 13:10:10 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 13:10:10 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 13:14:13 main:435] : INFO : Set logging level to 1 [2022-01-10 13:14:13 main:441] : INFO : Running in BOINC Client mode [2022-01-10 13:14:13 main:444] : INFO : Resolving all filenames [2022-01-10 13:14:13 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 13:14:13 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 13:14:14 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 13:14:14 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 13:14:14 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 13:14:19 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 13:14:19 main:474] : INFO : Configuration: [2022-01-10 13:14:19 main:475] : INFO : Model type: GRU [2022-01-10 13:14:19 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 13:14:19 main:477] : INFO : Max Epochs: 2048 [2022-01-10 13:14:19 main:478] : INFO : Batch Size: 128 [2022-01-10 13:14:20 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 13:14:20 main:480] : INFO : Patience: 10 [2022-01-10 13:14:20 main:481] : INFO : Hidden Width: 12 [2022-01-10 13:14:20 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 13:14:20 main:483] : INFO : # Backend Layers: 4 [2022-01-10 13:14:20 main:484] : INFO : # Threads: 1 [2022-01-10 13:14:20 main:486] : INFO : Preparing Dataset [2022-01-10 13:14:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 13:28:09 main:435] : INFO : Set logging level to 1 [2022-01-10 13:28:09 main:441] : INFO : Running in BOINC Client mode [2022-01-10 13:28:09 main:444] : INFO : Resolving all filenames [2022-01-10 13:28:09 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 13:28:15 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 13:28:15 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 13:28:15 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 13:28:15 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 13:28:16 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 13:28:16 main:474] : INFO : Configuration: [2022-01-10 13:28:16 main:475] : INFO : Model type: GRU [2022-01-10 13:28:16 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 13:28:16 main:477] : INFO : Max Epochs: 2048 [2022-01-10 13:28:16 main:478] : INFO : Batch Size: 128 [2022-01-10 13:28:16 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 13:28:16 main:480] : INFO : Patience: 10 [2022-01-10 13:28:16 main:481] : INFO : Hidden Width: 12 [2022-01-10 13:28:16 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 13:28:16 main:483] : INFO : # Backend Layers: 4 [2022-01-10 13:28:16 main:484] : INFO : # Threads: 1 [2022-01-10 13:28:22 main:486] : INFO : Preparing Dataset [2022-01-10 13:28:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 13:28:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 13:29:14 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 13:29:14 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 13:29:14 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 13:29:19 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 13:29:19 main:494] : INFO : Creating Model [2022-01-10 13:29:19 main:507] : INFO : Preparing config file [2022-01-10 13:29:19 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 13:29:20 main:512] : INFO : Loading config [2022-01-10 13:29:20 main:514] : INFO : Loading state [2022-01-10 13:29:21 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 13:29:21 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 13:33:17 main:435] : INFO : Set logging level to 1 [2022-01-10 13:33:17 main:441] : INFO : Running in BOINC Client mode [2022-01-10 13:33:17 main:444] : INFO : Resolving all filenames [2022-01-10 13:33:17 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 13:33:17 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 13:33:17 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 13:33:17 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 13:33:17 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 13:33:17 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 13:33:23 main:474] : INFO : Configuration: [2022-01-10 13:33:23 main:475] : INFO : Model type: GRU [2022-01-10 13:33:23 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 13:33:23 main:477] : INFO : Max Epochs: 2048 [2022-01-10 13:33:23 main:478] : INFO : Batch Size: 128 [2022-01-10 13:33:23 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 13:33:23 main:480] : INFO : Patience: 10 [2022-01-10 13:33:23 main:481] : INFO : Hidden Width: 12 [2022-01-10 13:33:23 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 13:33:23 main:483] : INFO : # Backend Layers: 4 [2022-01-10 13:33:23 main:484] : INFO : # Threads: 1 [2022-01-10 13:33:24 main:486] : INFO : Preparing Dataset [2022-01-10 13:33:24 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 13:34:28 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 13:34:53 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 13:34:53 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 13:34:53 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 13:34:59 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 13:34:59 main:494] : INFO : Creating Model [2022-01-10 13:34:59 main:507] : INFO : Preparing config file [2022-01-10 13:34:59 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 13:34:59 main:512] : INFO : Loading config [2022-01-10 13:34:59 main:514] : INFO : Loading state [2022-01-10 13:35:00 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 13:35:00 main:562] : INFO : Starting Training [2022-01-10 13:43:09 main:574] : INFO : Epoch 1809 | loss: 0.031212 | val_loss: 0.0311848 | Time: 488928 ms [2022-01-10 13:49:00 main:574] : INFO : Epoch 1810 | loss: 0.0311531 | val_loss: 0.0311863 | Time: 350203 ms [2022-01-10 14:02:56 main:574] : INFO : Epoch 1811 | loss: 0.0311358 | val_loss: 0.0311844 | Time: 835931 ms [2022-01-10 14:08:45 main:574] : INFO : Epoch 1812 | loss: 0.031122 | val_loss: 0.0311924 | Time: 349136 ms [2022-01-10 14:15:32 main:574] : INFO : Epoch 1813 | loss: 0.0311094 | val_loss: 0.0312012 | Time: 406997 ms [2022-01-10 14:29:57 main:574] : INFO : Epoch 1814 | loss: 0.03109 | val_loss: 0.0312558 | Time: 864443 ms [2022-01-10 14:34:13 main:574] : INFO : Epoch 1815 | loss: 0.0310514 | val_loss: 0.0312829 | Time: 256082 ms [2022-01-10 14:37:27 main:574] : INFO : Epoch 1816 | loss: 0.0310345 | val_loss: 0.0312634 | Time: 193950 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 14:59:11 main:435] : INFO : Set logging level to 1 [2022-01-10 14:59:11 main:441] : INFO : Running in BOINC Client mode [2022-01-10 14:59:11 main:444] : INFO : Resolving all filenames [2022-01-10 14:59:11 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 14:59:11 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 14:59:11 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 14:59:11 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 14:59:11 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 14:59:11 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 14:59:11 main:474] : INFO : Configuration: [2022-01-10 14:59:12 main:475] : INFO : Model type: GRU [2022-01-10 14:59:22 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 14:59:22 main:477] : INFO : Max Epochs: 2048 [2022-01-10 14:59:23 main:478] : INFO : Batch Size: 128 [2022-01-10 14:59:23 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 14:59:23 main:480] : INFO : Patience: 10 [2022-01-10 14:59:23 main:481] : INFO : Hidden Width: 12 [2022-01-10 14:59:23 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 14:59:23 main:483] : INFO : # Backend Layers: 4 [2022-01-10 14:59:23 main:484] : INFO : # Threads: 1 [2022-01-10 14:59:23 main:486] : INFO : Preparing Dataset [2022-01-10 14:59:23 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 15:04:30 main:435] : INFO : Set logging level to 1 [2022-01-10 15:04:30 main:441] : INFO : Running in BOINC Client mode [2022-01-10 15:04:30 main:444] : INFO : Resolving all filenames [2022-01-10 15:04:30 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 15:04:30 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 15:04:30 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 15:04:30 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 15:04:36 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 15:04:36 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 15:04:36 main:474] : INFO : Configuration: [2022-01-10 15:04:36 main:475] : INFO : Model type: GRU [2022-01-10 15:04:36 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 15:04:36 main:477] : INFO : Max Epochs: 2048 [2022-01-10 15:04:36 main:478] : INFO : Batch Size: 128 [2022-01-10 15:04:36 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 15:04:36 main:480] : INFO : Patience: 10 [2022-01-10 15:04:37 main:481] : INFO : Hidden Width: 12 [2022-01-10 15:04:37 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 15:04:37 main:483] : INFO : # Backend Layers: 4 [2022-01-10 15:04:42 main:484] : INFO : # Threads: 1 [2022-01-10 15:04:42 main:486] : INFO : Preparing Dataset [2022-01-10 15:04:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 16:04:36 main:435] : INFO : Set logging level to 1 [2022-01-10 16:04:36 main:441] : INFO : Running in BOINC Client mode [2022-01-10 16:04:36 main:444] : INFO : Resolving all filenames [2022-01-10 16:04:36 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 16:04:36 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 16:04:41 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 16:04:41 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 16:04:41 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 16:04:41 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 16:04:41 main:474] : INFO : Configuration: [2022-01-10 16:04:41 main:475] : INFO : Model type: GRU [2022-01-10 16:04:42 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 16:04:42 main:477] : INFO : Max Epochs: 2048 [2022-01-10 16:04:42 main:478] : INFO : Batch Size: 128 [2022-01-10 16:04:42 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 16:04:42 main:480] : INFO : Patience: 10 [2022-01-10 16:04:42 main:481] : INFO : Hidden Width: 12 [2022-01-10 16:04:42 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 16:04:42 main:483] : INFO : # Backend Layers: 4 [2022-01-10 16:04:42 main:484] : INFO : # Threads: 1 [2022-01-10 16:04:42 main:486] : INFO : Preparing Dataset [2022-01-10 16:04:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 16:05:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 16:06:07 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 16:06:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 16:06:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 16:06:22 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 16:06:22 main:494] : INFO : Creating Model [2022-01-10 16:06:22 main:507] : INFO : Preparing config file [2022-01-10 16:06:22 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 16:06:22 main:512] : INFO : Loading config [2022-01-10 16:06:22 main:514] : INFO : Loading state Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 16:24:24 main:435] : INFO : Set logging level to 1 [2022-01-10 16:24:24 main:441] : INFO : Running in BOINC Client mode [2022-01-10 16:24:24 main:444] : INFO : Resolving all filenames [2022-01-10 16:24:24 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 16:24:24 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 16:24:24 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 16:24:30 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 16:24:30 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 16:24:30 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 16:24:30 main:474] : INFO : Configuration: [2022-01-10 16:24:30 main:475] : INFO : Model type: GRU [2022-01-10 16:24:30 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 16:24:30 main:477] : INFO : Max Epochs: 2048 [2022-01-10 16:24:30 main:478] : INFO : Batch Size: 128 [2022-01-10 16:24:30 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 16:24:31 main:480] : INFO : Patience: 10 [2022-01-10 16:24:31 main:481] : INFO : Hidden Width: 12 [2022-01-10 16:24:31 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 16:24:31 main:483] : INFO : # Backend Layers: 4 [2022-01-10 16:24:36 main:484] : INFO : # Threads: 1 [2022-01-10 16:24:36 main:486] : INFO : Preparing Dataset [2022-01-10 16:24:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 16:25:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 16:25:15 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 16:25:15 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 16:25:34 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 16:25:34 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 16:25:34 main:494] : INFO : Creating Model [2022-01-10 16:25:34 main:507] : INFO : Preparing config file [2022-01-10 16:25:34 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 16:25:34 main:512] : INFO : Loading config [2022-01-10 16:25:35 main:514] : INFO : Loading state [2022-01-10 16:25:41 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 16:25:41 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 16:28:57 main:435] : INFO : Set logging level to 1 [2022-01-10 16:28:57 main:441] : INFO : Running in BOINC Client mode [2022-01-10 16:28:57 main:444] : INFO : Resolving all filenames [2022-01-10 16:28:57 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 16:28:57 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 16:28:58 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 16:28:58 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 16:28:58 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 16:28:58 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 16:28:58 main:474] : INFO : Configuration: [2022-01-10 16:28:58 main:475] : INFO : Model type: GRU [2022-01-10 16:29:03 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 16:29:03 main:477] : INFO : Max Epochs: 2048 [2022-01-10 16:29:03 main:478] : INFO : Batch Size: 128 [2022-01-10 16:29:03 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 16:29:03 main:480] : INFO : Patience: 10 [2022-01-10 16:29:03 main:481] : INFO : Hidden Width: 12 [2022-01-10 16:29:03 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 16:29:03 main:483] : INFO : # Backend Layers: 4 [2022-01-10 16:29:04 main:484] : INFO : # Threads: 1 [2022-01-10 16:29:04 main:486] : INFO : Preparing Dataset [2022-01-10 16:29:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 16:29:41 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 16:31:56 main:435] : INFO : Set logging level to 1 [2022-01-10 16:31:56 main:441] : INFO : Running in BOINC Client mode [2022-01-10 16:31:56 main:444] : INFO : Resolving all filenames [2022-01-10 16:31:56 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 16:31:56 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 16:31:56 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 16:31:56 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 16:31:56 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 16:32:02 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 16:32:02 main:474] : INFO : Configuration: [2022-01-10 16:32:02 main:475] : INFO : Model type: GRU [2022-01-10 16:32:03 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 16:32:03 main:477] : INFO : Max Epochs: 2048 [2022-01-10 16:32:03 main:478] : INFO : Batch Size: 128 [2022-01-10 16:32:03 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 16:32:03 main:480] : INFO : Patience: 10 [2022-01-10 16:32:03 main:481] : INFO : Hidden Width: 12 [2022-01-10 16:32:03 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 16:32:03 main:483] : INFO : # Backend Layers: 4 [2022-01-10 16:32:03 main:484] : INFO : # Threads: 1 [2022-01-10 16:32:03 main:486] : INFO : Preparing Dataset [2022-01-10 16:32:03 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 16:33:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 16:33:27 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 16:33:27 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 16:33:47 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 16:33:47 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 16:33:47 main:494] : INFO : Creating Model [2022-01-10 16:33:47 main:507] : INFO : Preparing config file [2022-01-10 16:33:47 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 16:33:47 main:512] : INFO : Loading config [2022-01-10 16:33:47 main:514] : INFO : Loading state [2022-01-10 16:33:57 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 16:33:58 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 16:37:08 main:435] : INFO : Set logging level to 1 [2022-01-10 16:37:08 main:441] : INFO : Running in BOINC Client mode [2022-01-10 16:37:08 main:444] : INFO : Resolving all filenames [2022-01-10 16:37:08 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 16:37:08 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 16:37:08 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 16:37:08 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 16:37:08 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 16:37:08 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 16:37:08 main:474] : INFO : Configuration: [2022-01-10 16:37:08 main:475] : INFO : Model type: GRU [2022-01-10 16:37:08 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 16:37:08 main:477] : INFO : Max Epochs: 2048 [2022-01-10 16:37:08 main:478] : INFO : Batch Size: 128 [2022-01-10 16:37:09 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 16:37:09 main:480] : INFO : Patience: 10 [2022-01-10 16:37:09 main:481] : INFO : Hidden Width: 12 [2022-01-10 16:37:09 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 16:37:09 main:483] : INFO : # Backend Layers: 4 [2022-01-10 16:37:09 main:484] : INFO : # Threads: 1 [2022-01-10 16:37:09 main:486] : INFO : Preparing Dataset [2022-01-10 16:37:15 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 16:37:40 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 16:37:46 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 16:37:46 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 16:37:46 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 16:37:47 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 16:37:47 main:494] : INFO : Creating Model [2022-01-10 16:37:47 main:507] : INFO : Preparing config file [2022-01-10 16:37:47 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 16:37:47 main:512] : INFO : Loading config [2022-01-10 16:37:47 main:514] : INFO : Loading state [2022-01-10 16:37:49 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 16:37:49 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 16:54:04 main:435] : INFO : Set logging level to 1 [2022-01-10 16:54:04 main:441] : INFO : Running in BOINC Client mode [2022-01-10 16:54:04 main:444] : INFO : Resolving all filenames [2022-01-10 16:54:04 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 16:54:04 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 16:54:04 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 16:54:04 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 16:54:05 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 16:54:05 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 16:54:11 main:474] : INFO : Configuration: [2022-01-10 16:54:11 main:475] : INFO : Model type: GRU [2022-01-10 16:54:11 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 16:54:11 main:477] : INFO : Max Epochs: 2048 [2022-01-10 16:54:11 main:478] : INFO : Batch Size: 128 [2022-01-10 16:54:11 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 16:54:11 main:480] : INFO : Patience: 10 [2022-01-10 16:54:12 main:481] : INFO : Hidden Width: 12 [2022-01-10 16:54:12 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 16:54:12 main:483] : INFO : # Backend Layers: 4 [2022-01-10 16:54:12 main:484] : INFO : # Threads: 1 [2022-01-10 16:54:12 main:486] : INFO : Preparing Dataset [2022-01-10 16:54:12 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 16:54:46 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 16:54:48 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 16:54:48 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 16:54:49 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 16:54:49 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 16:54:49 main:494] : INFO : Creating Model [2022-01-10 16:54:49 main:507] : INFO : Preparing config file [2022-01-10 16:54:49 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 16:54:50 main:512] : INFO : Loading config [2022-01-10 16:54:50 main:514] : INFO : Loading state [2022-01-10 16:54:57 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 16:54:57 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 16:59:11 main:435] : INFO : Set logging level to 1 [2022-01-10 16:59:11 main:441] : INFO : Running in BOINC Client mode [2022-01-10 16:59:12 main:444] : INFO : Resolving all filenames [2022-01-10 16:59:17 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 16:59:18 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 16:59:18 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 16:59:18 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 16:59:18 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 16:59:18 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 16:59:18 main:474] : INFO : Configuration: [2022-01-10 16:59:18 main:475] : INFO : Model type: GRU [2022-01-10 16:59:18 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 16:59:18 main:477] : INFO : Max Epochs: 2048 [2022-01-10 16:59:18 main:478] : INFO : Batch Size: 128 [2022-01-10 16:59:18 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 16:59:23 main:480] : INFO : Patience: 10 [2022-01-10 16:59:23 main:481] : INFO : Hidden Width: 12 [2022-01-10 16:59:24 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 16:59:24 main:483] : INFO : # Backend Layers: 4 [2022-01-10 16:59:24 main:484] : INFO : # Threads: 1 [2022-01-10 16:59:24 main:486] : INFO : Preparing Dataset [2022-01-10 16:59:24 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 16:59:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 16:59:55 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 16:59:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 16:59:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 16:59:56 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 16:59:56 main:494] : INFO : Creating Model [2022-01-10 16:59:56 main:507] : INFO : Preparing config file [2022-01-10 16:59:56 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 16:59:57 main:512] : INFO : Loading config [2022-01-10 16:59:57 main:514] : INFO : Loading state [2022-01-10 17:00:04 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 17:00:04 main:562] : INFO : Starting Training [2022-01-10 17:13:54 main:574] : INFO : Epoch 1813 | loss: 0.0312183 | val_loss: 0.03121 | Time: 830379 ms [2022-01-10 17:21:52 main:574] : INFO : Epoch 1814 | loss: 0.0311222 | val_loss: 0.0312121 | Time: 478119 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 17:28:20 main:435] : INFO : Set logging level to 1 [2022-01-10 17:28:21 main:441] : INFO : Running in BOINC Client mode [2022-01-10 17:28:21 main:444] : INFO : Resolving all filenames [2022-01-10 17:28:21 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 17:28:21 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 17:28:21 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 17:28:21 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 17:28:21 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 17:28:21 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 17:28:21 main:474] : INFO : Configuration: [2022-01-10 17:28:21 main:475] : INFO : Model type: GRU [2022-01-10 17:28:21 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 17:28:21 main:477] : INFO : Max Epochs: 2048 [2022-01-10 17:28:21 main:478] : INFO : Batch Size: 128 [2022-01-10 17:28:22 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 17:28:22 main:480] : INFO : Patience: 10 [2022-01-10 17:28:22 main:481] : INFO : Hidden Width: 12 [2022-01-10 17:28:22 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 17:28:22 main:483] : INFO : # Backend Layers: 4 [2022-01-10 17:28:22 main:484] : INFO : # Threads: 1 [2022-01-10 17:28:22 main:486] : INFO : Preparing Dataset [2022-01-10 17:28:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 17:28:23 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 17:28:25 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 17:28:25 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 17:28:26 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 17:28:26 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 17:28:26 main:494] : INFO : Creating Model [2022-01-10 17:28:27 main:507] : INFO : Preparing config file [2022-01-10 17:28:27 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 17:28:27 main:512] : INFO : Loading config [2022-01-10 17:28:27 main:514] : INFO : Loading state [2022-01-10 17:28:34 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 17:28:34 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 17:33:33 main:435] : INFO : Set logging level to 1 [2022-01-10 17:33:33 main:441] : INFO : Running in BOINC Client mode [2022-01-10 17:33:33 main:444] : INFO : Resolving all filenames [2022-01-10 17:33:33 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 17:33:33 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 17:33:33 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 17:33:33 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 17:33:33 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 17:33:33 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 17:33:33 main:474] : INFO : Configuration: [2022-01-10 17:33:34 main:475] : INFO : Model type: GRU [2022-01-10 17:33:34 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 17:33:34 main:477] : INFO : Max Epochs: 2048 [2022-01-10 17:33:34 main:478] : INFO : Batch Size: 128 [2022-01-10 17:33:34 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 17:33:34 main:480] : INFO : Patience: 10 [2022-01-10 17:33:34 main:481] : INFO : Hidden Width: 12 [2022-01-10 17:33:34 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 17:33:34 main:483] : INFO : # Backend Layers: 4 [2022-01-10 17:33:34 main:484] : INFO : # Threads: 1 [2022-01-10 17:33:34 main:486] : INFO : Preparing Dataset [2022-01-10 17:33:34 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 17:34:15 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 17:34:37 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 17:34:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 17:34:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 17:35:02 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 17:35:02 main:494] : INFO : Creating Model [2022-01-10 17:35:02 main:507] : INFO : Preparing config file [2022-01-10 17:35:02 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 17:35:02 main:512] : INFO : Loading config [2022-01-10 17:35:02 main:514] : INFO : Loading state [2022-01-10 17:35:03 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 17:35:04 main:562] : INFO : Starting Training [2022-01-10 17:48:45 main:574] : INFO : Epoch 1813 | loss: 0.0312078 | val_loss: 0.0312206 | Time: 821478 ms [2022-01-10 17:55:20 main:574] : INFO : Epoch 1814 | loss: 0.0311014 | val_loss: 0.0312296 | Time: 395030 ms [2022-01-10 18:01:29 main:574] : INFO : Epoch 1815 | loss: 0.0310662 | val_loss: 0.0312559 | Time: 368457 ms [2022-01-10 18:16:24 main:574] : INFO : Epoch 1816 | loss: 0.0310436 | val_loss: 0.031323 | Time: 895316 ms [2022-01-10 18:21:23 main:574] : INFO : Epoch 1817 | loss: 0.0310378 | val_loss: 0.0312755 | Time: 299224 ms [2022-01-10 18:32:27 main:574] : INFO : Epoch 1818 | loss: 0.0310239 | val_loss: 0.0312771 | Time: 663140 ms [2022-01-10 18:43:36 main:574] : INFO : Epoch 1819 | loss: 0.0310233 | val_loss: 0.0312651 | Time: 663820 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 18:51:43 main:435] : INFO : Set logging level to 1 [2022-01-10 18:51:43 main:441] : INFO : Running in BOINC Client mode [2022-01-10 18:51:43 main:444] : INFO : Resolving all filenames [2022-01-10 18:51:43 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 18:51:48 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 18:51:48 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 18:51:48 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 18:51:49 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 18:51:49 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 18:51:49 main:474] : INFO : Configuration: [2022-01-10 18:51:49 main:475] : INFO : Model type: GRU [2022-01-10 18:51:49 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 18:51:49 main:477] : INFO : Max Epochs: 2048 [2022-01-10 18:51:49 main:478] : INFO : Batch Size: 128 [2022-01-10 18:51:49 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 18:51:49 main:480] : INFO : Patience: 10 [2022-01-10 18:51:49 main:481] : INFO : Hidden Width: 12 [2022-01-10 18:51:49 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 18:51:49 main:483] : INFO : # Backend Layers: 4 [2022-01-10 18:51:49 main:484] : INFO : # Threads: 1 [2022-01-10 18:51:49 main:486] : INFO : Preparing Dataset [2022-01-10 18:51:49 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2022-01-10 19:09:34 main:435] : INFO : Set logging level to 1 [2022-01-10 19:09:34 main:441] : INFO : Running in BOINC Client mode [2022-01-10 19:09:34 main:444] : INFO : Resolving all filenames [2022-01-10 19:09:34 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-01-10 19:09:34 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-01-10 19:09:34 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-01-10 19:09:34 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-01-10 19:09:34 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-01-10 19:09:34 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-01-10 19:09:34 main:474] : INFO : Configuration: [2022-01-10 19:09:34 main:475] : INFO : Model type: GRU [2022-01-10 19:09:34 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-01-10 19:09:35 main:477] : INFO : Max Epochs: 2048 [2022-01-10 19:09:35 main:478] : INFO : Batch Size: 128 [2022-01-10 19:09:35 main:479] : INFO : Learning Rate: 0.01 [2022-01-10 19:09:35 main:480] : INFO : Patience: 10 [2022-01-10 19:09:35 main:481] : INFO : Hidden Width: 12 [2022-01-10 19:09:35 main:482] : INFO : # Recurrent Layers: 4 [2022-01-10 19:09:35 main:483] : INFO : # Backend Layers: 4 [2022-01-10 19:09:35 main:484] : INFO : # Threads: 1 [2022-01-10 19:09:35 main:486] : INFO : Preparing Dataset [2022-01-10 19:09:35 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-01-10 19:09:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-01-10 19:09:46 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-01-10 19:09:46 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-01-10 19:09:46 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-01-10 19:09:46 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-01-10 19:09:46 main:494] : INFO : Creating Model [2022-01-10 19:09:47 main:507] : INFO : Preparing config file [2022-01-10 19:09:47 main:511] : INFO : Found checkpoint, attempting to load... [2022-01-10 19:09:47 main:512] : INFO : Loading config [2022-01-10 19:09:47 main:514] : INFO : Loading state [2022-01-10 19:09:53 main:559] : INFO : Loading DataLoader into Memory [2022-01-10 19:09:53 main:562] : INFO : Starting Training </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)