Task 14450403
| Name | ParityModified-1647042746-16459-3-0_0 |
| Workunit | 11493285 |
| Created | 6 Apr 2022, 4:59:55 UTC |
| Sent | 11 Apr 2022, 5:38:50 UTC |
| Report deadline | 19 Apr 2022, 5:38:50 UTC |
| Received | 20 Apr 2022, 9:14:08 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 19089 |
| Run time | 9 hours 57 min 32 sec |
| CPU time | 9 hours 48 min 52 sec |
| Validate state | Task was reported too late to validate |
| Credit | 0.00 |
| Device peak FLOPS | 5,476.98 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 2.70 GB |
| Peak swap size | 4.42 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.20</core_client_version> <![CDATA[ <stderr_txt> ience: 10 [2022-04-14 09:17:23 main:481] : INFO : Hidden Width: 12 [2022-04-14 09:17:23 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 09:17:24 main:483] : INFO : # Backend Layers: 4 [2022-04-14 09:17:24 main:484] : INFO : # Threads: 1 [2022-04-14 09:17:24 main:486] : INFO : Preparing Dataset [2022-04-14 09:17:24 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 09:17:24 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 09:21:47 main:435] : INFO : Set logging level to 1 [2022-04-14 09:21:47 main:441] : INFO : Running in BOINC Client mode [2022-04-14 09:21:47 main:444] : INFO : Resolving all filenames [2022-04-14 09:21:47 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 09:21:47 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 09:21:47 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 09:21:47 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 09:21:47 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 09:21:47 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 09:21:47 main:474] : INFO : Configuration: [2022-04-14 09:21:47 main:475] : INFO : Model type: GRU [2022-04-14 09:21:47 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 09:21:47 main:477] : INFO : Max Epochs: 2048 [2022-04-14 09:21:47 main:478] : INFO : Batch Size: 128 [2022-04-14 09:21:47 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 09:21:47 main:480] : INFO : Patience: 10 [2022-04-14 09:21:47 main:481] : INFO : Hidden Width: 12 [2022-04-14 09:21:47 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 09:21:47 main:483] : INFO : # Backend Layers: 4 [2022-04-14 09:21:47 main:484] : INFO : # Threads: 1 [2022-04-14 09:21:47 main:486] : INFO : Preparing Dataset [2022-04-14 09:21:47 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 09:21:47 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 09:47:02 main:435] : INFO : Set logging level to 1 [2022-04-14 09:47:02 main:441] : INFO : Running in BOINC Client mode [2022-04-14 09:47:02 main:444] : INFO : Resolving all filenames [2022-04-14 09:47:02 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 09:47:02 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 09:47:02 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 09:47:02 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 09:47:02 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 09:47:02 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 09:47:02 main:474] : INFO : Configuration: [2022-04-14 09:47:02 main:475] : INFO : Model type: GRU [2022-04-14 09:47:02 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 09:47:02 main:477] : INFO : Max Epochs: 2048 [2022-04-14 09:47:02 main:478] : INFO : Batch Size: 128 [2022-04-14 09:47:02 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 09:47:02 main:480] : INFO : Patience: 10 [2022-04-14 09:47:02 main:481] : INFO : Hidden Width: 12 [2022-04-14 09:47:02 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 09:47:02 main:483] : INFO : # Backend Layers: 4 [2022-04-14 09:47:02 main:484] : INFO : # Threads: 1 [2022-04-14 09:47:02 main:486] : INFO : Preparing Dataset [2022-04-14 09:47:02 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 09:47:03 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 10:00:41 main:435] : INFO : Set logging level to 1 [2022-04-14 10:00:41 main:441] : INFO : Running in BOINC Client mode [2022-04-14 10:00:41 main:444] : INFO : Resolving all filenames [2022-04-14 10:00:41 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 10:00:41 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 10:00:41 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 10:00:41 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 10:00:41 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 10:00:41 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 10:00:41 main:474] : INFO : Configuration: [2022-04-14 10:00:41 main:475] : INFO : Model type: GRU [2022-04-14 10:00:41 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 10:00:41 main:477] : INFO : Max Epochs: 2048 [2022-04-14 10:00:41 main:478] : INFO : Batch Size: 128 [2022-04-14 10:00:41 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 10:00:41 main:480] : INFO : Patience: 10 [2022-04-14 10:00:41 main:481] : INFO : Hidden Width: 12 [2022-04-14 10:00:41 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 10:00:41 main:483] : INFO : # Backend Layers: 4 [2022-04-14 10:00:41 main:484] : INFO : # Threads: 1 [2022-04-14 10:00:41 main:486] : INFO : Preparing Dataset [2022-04-14 10:00:41 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 10:00:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 10:06:15 main:435] : INFO : Set logging level to 1 [2022-04-14 10:06:15 main:441] : INFO : Running in BOINC Client mode [2022-04-14 10:06:15 main:444] : INFO : Resolving all filenames [2022-04-14 10:06:15 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 10:06:15 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 10:06:15 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 10:06:15 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 10:06:15 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 10:06:15 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 10:06:15 main:474] : INFO : Configuration: [2022-04-14 10:06:15 main:475] : INFO : Model type: GRU [2022-04-14 10:06:15 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 10:06:15 main:477] : INFO : Max Epochs: 2048 [2022-04-14 10:06:15 main:478] : INFO : Batch Size: 128 [2022-04-14 10:06:15 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 10:06:15 main:480] : INFO : Patience: 10 [2022-04-14 10:06:15 main:481] : INFO : Hidden Width: 12 [2022-04-14 10:06:15 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 10:06:15 main:483] : INFO : # Backend Layers: 4 [2022-04-14 10:06:15 main:484] : INFO : # Threads: 1 [2022-04-14 10:06:15 main:486] : INFO : Preparing Dataset [2022-04-14 10:06:15 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 10:06:15 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 10:11:28 main:435] : INFO : Set logging level to 1 [2022-04-14 10:11:28 main:441] : INFO : Running in BOINC Client mode [2022-04-14 10:11:28 main:444] : INFO : Resolving all filenames [2022-04-14 10:11:28 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 10:11:29 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 10:11:29 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 10:11:29 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 10:11:29 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 10:11:29 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 10:11:29 main:474] : INFO : Configuration: [2022-04-14 10:11:29 main:475] : INFO : Model type: GRU [2022-04-14 10:11:29 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 10:11:29 main:477] : INFO : Max Epochs: 2048 [2022-04-14 10:11:29 main:478] : INFO : Batch Size: 128 [2022-04-14 10:11:29 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 10:11:29 main:480] : INFO : Patience: 10 [2022-04-14 10:11:29 main:481] : INFO : Hidden Width: 12 [2022-04-14 10:11:29 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 10:11:29 main:483] : INFO : # Backend Layers: 4 [2022-04-14 10:11:29 main:484] : INFO : # Threads: 1 [2022-04-14 10:11:29 main:486] : INFO : Preparing Dataset [2022-04-14 10:11:29 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 10:11:29 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 10:16:59 main:435] : INFO : Set logging level to 1 [2022-04-14 10:16:59 main:441] : INFO : Running in BOINC Client mode [2022-04-14 10:16:59 main:444] : INFO : Resolving all filenames [2022-04-14 10:16:59 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 10:16:59 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 10:16:59 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 10:16:59 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 10:16:59 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 10:16:59 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 10:16:59 main:474] : INFO : Configuration: [2022-04-14 10:16:59 main:475] : INFO : Model type: GRU [2022-04-14 10:16:59 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 10:16:59 main:477] : INFO : Max Epochs: 2048 [2022-04-14 10:16:59 main:478] : INFO : Batch Size: 128 [2022-04-14 10:16:59 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 10:16:59 main:480] : INFO : Patience: 10 [2022-04-14 10:16:59 main:481] : INFO : Hidden Width: 12 [2022-04-14 10:16:59 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 10:16:59 main:483] : INFO : # Backend Layers: 4 [2022-04-14 10:16:59 main:484] : INFO : # Threads: 1 [2022-04-14 10:16:59 main:486] : INFO : Preparing Dataset [2022-04-14 10:16:59 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 10:16:59 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 11:00:41 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 11:00:41 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 11:00:41 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 11:00:42 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 11:00:42 main:494] : INFO : Creating Model [2022-04-14 11:00:42 main:507] : INFO : Preparing config file [2022-04-14 11:00:42 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 11:00:42 main:512] : INFO : Loading config [2022-04-14 11:00:42 main:514] : INFO : Loading state [2022-04-14 11:33:56 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 11:33:56 main:562] : INFO : Starting Training Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 11:49:51 main:435] : INFO : Set logging level to 1 [2022-04-14 11:49:51 main:441] : INFO : Running in BOINC Client mode [2022-04-14 11:49:51 main:444] : INFO : Resolving all filenames [2022-04-14 11:49:51 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 11:49:51 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 11:49:51 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 11:49:51 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 11:49:51 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 11:49:51 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 11:49:51 main:474] : INFO : Configuration: [2022-04-14 11:49:51 main:475] : INFO : Model type: GRU [2022-04-14 11:49:51 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 11:49:51 main:477] : INFO : Max Epochs: 2048 [2022-04-14 11:49:51 main:478] : INFO : Batch Size: 128 [2022-04-14 11:49:51 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 11:49:51 main:480] : INFO : Patience: 10 [2022-04-14 11:49:51 main:481] : INFO : Hidden Width: 12 [2022-04-14 11:49:51 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 11:49:51 main:483] : INFO : # Backend Layers: 4 [2022-04-14 11:49:51 main:484] : INFO : # Threads: 1 [2022-04-14 11:49:51 main:486] : INFO : Preparing Dataset [2022-04-14 11:49:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 11:49:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 11:53:48 main:435] : INFO : Set logging level to 1 [2022-04-14 11:53:48 main:441] : INFO : Running in BOINC Client mode [2022-04-14 11:53:48 main:444] : INFO : Resolving all filenames [2022-04-14 11:53:48 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 11:53:48 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 11:53:48 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 11:53:48 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 11:53:48 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 11:53:48 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 11:53:48 main:474] : INFO : Configuration: [2022-04-14 11:53:48 main:475] : INFO : Model type: GRU [2022-04-14 11:53:48 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 11:53:48 main:477] : INFO : Max Epochs: 2048 [2022-04-14 11:53:48 main:478] : INFO : Batch Size: 128 [2022-04-14 11:53:48 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 11:53:48 main:480] : INFO : Patience: 10 [2022-04-14 11:53:48 main:481] : INFO : Hidden Width: 12 [2022-04-14 11:53:48 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 11:53:48 main:483] : INFO : # Backend Layers: 4 [2022-04-14 11:53:48 main:484] : INFO : # Threads: 1 [2022-04-14 11:53:48 main:486] : INFO : Preparing Dataset [2022-04-14 11:53:48 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 11:53:48 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-14 12:01:22 main:435] : INFO : Set logging level to 1 [2022-04-14 12:01:22 main:441] : INFO : Running in BOINC Client mode [2022-04-14 12:01:22 main:444] : INFO : Resolving all filenames [2022-04-14 12:01:22 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 12:01:22 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 12:01:22 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 12:01:22 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 12:01:22 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 12:01:22 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 12:01:22 main:474] : INFO : Configuration: [2022-04-14 12:01:22 main:475] : INFO : Model type: GRU [2022-04-14 12:01:22 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 12:01:22 main:477] : INFO : Max Epochs: 2048 [2022-04-14 12:01:22 main:478] : INFO : Batch Size: 128 [2022-04-14 12:01:22 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 12:01:22 main:480] : INFO : Patience: 10 [2022-04-14 12:01:22 main:481] : INFO : Hidden Width: 12 [2022-04-14 12:01:22 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 12:01:22 main:483] : INFO : # Backend Layers: 4 [2022-04-14 12:01:22 main:484] : INFO : # Threads: 1 [2022-04-14 12:01:22 main:486] : INFO : Preparing Dataset [2022-04-14 12:01:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 12:01:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 12:43:35 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 12:43:35 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 12:43:35 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 12:43:35 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 12:43:35 main:494] : INFO : Creating Model [2022-04-14 12:43:35 main:507] : INFO : Preparing config file [2022-04-14 12:43:35 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 12:43:35 main:512] : INFO : Loading config [2022-04-14 12:43:35 main:514] : INFO : Loading state [2022-04-14 13:24:38 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 13:24:38 main:562] : INFO : Starting Training [2022-04-14 13:26:44 main:574] : INFO : Epoch 1960 | loss: 0.0315834 | val_loss: 0.0313349 | Time: 125694 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-16 12:23:18 main:435] : INFO : Set logging level to 1 [2022-04-16 12:23:18 main:441] : INFO : Running in BOINC Client mode [2022-04-16 12:23:18 main:444] : INFO : Resolving all filenames [2022-04-16 12:23:18 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-16 12:23:18 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-16 12:23:18 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-16 12:23:18 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-16 12:23:18 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-16 12:23:18 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-16 12:23:18 main:474] : INFO : Configuration: [2022-04-16 12:23:18 main:475] : INFO : Model type: GRU [2022-04-16 12:23:18 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-16 12:23:18 main:477] : INFO : Max Epochs: 2048 [2022-04-16 12:23:18 main:478] : INFO : Batch Size: 128 [2022-04-16 12:23:18 main:479] : INFO : Learning Rate: 0.01 [2022-04-16 12:23:18 main:480] : INFO : Patience: 10 [2022-04-16 12:23:18 main:481] : INFO : Hidden Width: 12 [2022-04-16 12:23:18 main:482] : INFO : # Recurrent Layers: 4 [2022-04-16 12:23:18 main:483] : INFO : # Backend Layers: 4 [2022-04-16 12:23:18 main:484] : INFO : # Threads: 1 [2022-04-16 12:23:18 main:486] : INFO : Preparing Dataset [2022-04-16 12:23:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-16 12:23:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 08:36:31 main:435] : INFO : Set logging level to 1 [2022-04-19 08:36:31 main:441] : INFO : Running in BOINC Client mode [2022-04-19 08:36:31 main:444] : INFO : Resolving all filenames [2022-04-19 08:36:31 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 08:36:31 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 08:36:31 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 08:36:31 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 08:36:31 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 08:36:31 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 08:36:31 main:474] : INFO : Configuration: [2022-04-19 08:36:31 main:475] : INFO : Model type: GRU [2022-04-19 08:36:31 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 08:36:31 main:477] : INFO : Max Epochs: 2048 [2022-04-19 08:36:31 main:478] : INFO : Batch Size: 128 [2022-04-19 08:36:31 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 08:36:31 main:480] : INFO : Patience: 10 [2022-04-19 08:36:31 main:481] : INFO : Hidden Width: 12 [2022-04-19 08:36:31 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 08:36:31 main:483] : INFO : # Backend Layers: 4 [2022-04-19 08:36:31 main:484] : INFO : # Threads: 1 [2022-04-19 08:36:31 main:486] : INFO : Preparing Dataset [2022-04-19 08:36:31 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 08:36:31 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 11:14:19 main:435] : INFO : Set logging level to 1 [2022-04-19 11:14:19 main:441] : INFO : Running in BOINC Client mode [2022-04-19 11:14:19 main:444] : INFO : Resolving all filenames [2022-04-19 11:14:19 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 11:14:19 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 11:14:19 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 11:14:19 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 11:14:19 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 11:14:19 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 11:14:19 main:474] : INFO : Configuration: [2022-04-19 11:14:19 main:475] : INFO : Model type: GRU [2022-04-19 11:14:19 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 11:14:19 main:477] : INFO : Max Epochs: 2048 [2022-04-19 11:14:19 main:478] : INFO : Batch Size: 128 [2022-04-19 11:14:19 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 11:14:19 main:480] : INFO : Patience: 10 [2022-04-19 11:14:19 main:481] : INFO : Hidden Width: 12 [2022-04-19 11:14:19 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 11:14:19 main:483] : INFO : # Backend Layers: 4 [2022-04-19 11:14:19 main:484] : INFO : # Threads: 1 [2022-04-19 11:14:19 main:486] : INFO : Preparing Dataset [2022-04-19 11:14:19 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 11:14:19 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 12:06:32 main:435] : INFO : Set logging level to 1 [2022-04-19 12:06:32 main:441] : INFO : Running in BOINC Client mode [2022-04-19 12:06:32 main:444] : INFO : Resolving all filenames [2022-04-19 12:06:32 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 12:06:32 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 12:06:32 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 12:06:32 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 12:06:32 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 12:06:32 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 12:06:32 main:474] : INFO : Configuration: [2022-04-19 12:06:32 main:475] : INFO : Model type: GRU [2022-04-19 12:06:32 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 12:06:32 main:477] : INFO : Max Epochs: 2048 [2022-04-19 12:06:32 main:478] : INFO : Batch Size: 128 [2022-04-19 12:06:32 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 12:06:32 main:480] : INFO : Patience: 10 [2022-04-19 12:06:32 main:481] : INFO : Hidden Width: 12 [2022-04-19 12:06:32 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 12:06:32 main:483] : INFO : # Backend Layers: 4 [2022-04-19 12:06:32 main:484] : INFO : # Threads: 1 [2022-04-19 12:06:32 main:486] : INFO : Preparing Dataset [2022-04-19 12:06:32 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 12:06:32 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 12:12:08 main:435] : INFO : Set logging level to 1 [2022-04-19 12:12:08 main:441] : INFO : Running in BOINC Client mode [2022-04-19 12:12:08 main:444] : INFO : Resolving all filenames [2022-04-19 12:12:08 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 12:12:08 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 12:12:08 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 12:12:08 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 12:12:08 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 12:12:08 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 12:12:08 main:474] : INFO : Configuration: [2022-04-19 12:12:08 main:475] : INFO : Model type: GRU [2022-04-19 12:12:08 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 12:12:08 main:477] : INFO : Max Epochs: 2048 [2022-04-19 12:12:08 main:478] : INFO : Batch Size: 128 [2022-04-19 12:12:08 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 12:12:08 main:480] : INFO : Patience: 10 [2022-04-19 12:12:08 main:481] : INFO : Hidden Width: 12 [2022-04-19 12:12:08 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 12:12:08 main:483] : INFO : # Backend Layers: 4 [2022-04-19 12:12:08 main:484] : INFO : # Threads: 1 [2022-04-19 12:12:08 main:486] : INFO : Preparing Dataset [2022-04-19 12:12:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 12:12:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-19 13:03:32 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-19 13:03:32 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-19 13:03:32 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-19 13:03:32 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-19 13:03:32 main:494] : INFO : Creating Model [2022-04-19 13:03:32 main:507] : INFO : Preparing config file [2022-04-19 13:03:32 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-19 13:03:32 main:512] : INFO : Loading config [2022-04-19 13:03:32 main:514] : INFO : Loading state [2022-04-19 13:27:18 main:559] : INFO : Loading DataLoader into Memory [2022-04-19 13:27:18 main:562] : INFO : Starting Training [2022-04-19 13:29:17 main:574] : INFO : Epoch 1961 | loss: 0.0316756 | val_loss: 0.0313175 | Time: 119134 ms [2022-04-19 13:29:23 main:574] : INFO : Epoch 1962 | loss: 0.031275 | val_loss: 0.031249 | Time: 6004.09 ms [2022-04-19 13:29:29 main:574] : INFO : Epoch 1963 | loss: 0.0312293 | val_loss: 0.0312072 | Time: 5921.04 ms [2022-04-19 13:29:35 main:574] : INFO : Epoch 1964 | loss: 0.0312098 | val_loss: 0.0312024 | Time: 5926.65 ms [2022-04-19 13:29:41 main:574] : INFO : Epoch 1965 | loss: 0.031201 | val_loss: 0.0311922 | Time: 5966.84 ms [2022-04-19 13:29:47 main:574] : INFO : Epoch 1966 | loss: 0.0311972 | val_loss: 0.0311941 | Time: 5969.04 ms [2022-04-19 13:29:53 main:574] : INFO : Epoch 1967 | loss: 0.0311968 | val_loss: 0.0311918 | Time: 5972.33 ms [2022-04-19 13:29:59 main:574] : INFO : Epoch 1968 | loss: 0.0311949 | val_loss: 0.0311936 | Time: 5902.88 ms [2022-04-19 13:30:05 main:574] : INFO : Epoch 1969 | loss: 0.0311972 | val_loss: 0.031196 | Time: 6174.97 ms [2022-04-19 13:30:11 main:574] : INFO : Epoch 1970 | loss: 0.0311961 | val_loss: 0.0311907 | Time: 5933.48 ms [2022-04-19 13:30:17 main:574] : INFO : Epoch 1971 | loss: 0.0311936 | val_loss: 0.0311906 | Time: 6216.87 ms [2022-04-19 13:30:23 main:574] : INFO : Epoch 1972 | loss: 0.0311939 | val_loss: 0.0311911 | Time: 6311.04 ms [2022-04-19 13:30:30 main:574] : INFO : Epoch 1973 | loss: 0.0311941 | val_loss: 0.0311914 | Time: 6427.68 ms [2022-04-19 13:30:36 main:574] : INFO : Epoch 1974 | loss: 0.0311969 | val_loss: 0.0311928 | Time: 6153.98 ms [2022-04-19 13:30:42 main:574] : INFO : Epoch 1975 | loss: 0.0311946 | val_loss: 0.0312006 | Time: 6359.33 ms [2022-04-19 13:30:48 main:574] : INFO : Epoch 1976 | loss: 0.0311971 | val_loss: 0.0311912 | Time: 6036.12 ms [2022-04-19 13:30:55 main:574] : INFO : Epoch 1977 | loss: 0.0311946 | val_loss: 0.0311952 | Time: 6223.74 ms [2022-04-19 13:31:01 main:574] : INFO : Epoch 1978 | loss: 0.0311944 | val_loss: 0.0311904 | Time: 6349.76 ms [2022-04-19 13:31:07 main:574] : INFO : Epoch 1979 | loss: 0.0311941 | val_loss: 0.0311939 | Time: 6357.88 ms [2022-04-19 13:31:13 main:574] : INFO : Epoch 1980 | loss: 0.0311958 | val_loss: 0.03119 | Time: 6149.42 ms [2022-04-19 13:31:20 main:574] : INFO : Epoch 1981 | loss: 0.0311954 | val_loss: 0.0311957 | Time: 6566.97 ms [2022-04-19 13:31:26 main:574] : INFO : Epoch 1982 | loss: 0.0311954 | val_loss: 0.0311907 | Time: 6423.75 ms [2022-04-19 13:31:33 main:574] : INFO : Epoch 1983 | loss: 0.0311946 | val_loss: 0.0311922 | Time: 6369.38 ms [2022-04-19 13:31:39 main:574] : INFO : Epoch 1984 | loss: 0.0311945 | val_loss: 0.0311904 | Time: 6318.45 ms [2022-04-19 13:31:46 main:574] : INFO : Epoch 1985 | loss: 0.0311954 | val_loss: 0.0311903 | Time: 6422.53 ms [2022-04-19 13:31:52 main:574] : INFO : Epoch 1986 | loss: 0.0311969 | val_loss: 0.0311899 | Time: 6476.06 ms [2022-04-19 13:31:58 main:574] : INFO : Epoch 1987 | loss: 0.0311976 | val_loss: 0.0311965 | Time: 6283.25 ms [2022-04-19 13:32:05 main:574] : INFO : Epoch 1988 | loss: 0.0311976 | val_loss: 0.031195 | Time: 6418.66 ms [2022-04-19 13:32:11 main:574] : INFO : Epoch 1989 | loss: 0.0311937 | val_loss: 0.0311897 | Time: 6080.51 ms [2022-04-19 13:32:17 main:574] : INFO : Epoch 1990 | loss: 0.0311932 | val_loss: 0.0311905 | Time: 6272.57 ms [2022-04-19 13:32:23 main:574] : INFO : Epoch 1991 | loss: 0.0311926 | val_loss: 0.0311906 | Time: 6231.72 ms [2022-04-19 13:32:30 main:574] : INFO : Epoch 1992 | loss: 0.0311929 | val_loss: 0.03119 | Time: 6703.15 ms [2022-04-19 13:32:37 main:574] : INFO : Epoch 1993 | loss: 0.0311946 | val_loss: 0.0311942 | Time: 6434.16 ms [2022-04-19 13:32:43 main:574] : INFO : Epoch 1994 | loss: 0.0311985 | val_loss: 0.0311902 | Time: 6672.44 ms [2022-04-19 13:32:50 main:574] : INFO : Epoch 1995 | loss: 0.031196 | val_loss: 0.0311905 | Time: 6443.87 ms [2022-04-19 13:32:56 main:574] : INFO : Epoch 1996 | loss: 0.0311947 | val_loss: 0.0311927 | Time: 6584.33 ms [2022-04-19 13:33:03 main:574] : INFO : Epoch 1997 | loss: 0.0311946 | val_loss: 0.0311933 | Time: 6684.92 ms [2022-04-19 13:33:10 main:574] : INFO : Epoch 1998 | loss: 0.0311952 | val_loss: 0.031193 | Time: 6661.72 ms [2022-04-19 13:33:16 main:574] : INFO : Epoch 1999 | loss: 0.0311943 | val_loss: 0.0311922 | Time: 6382.27 ms [2022-04-19 13:33:23 main:574] : INFO : Epoch 2000 | loss: 0.031194 | val_loss: 0.0311909 | Time: 6636.67 ms [2022-04-19 13:33:29 main:574] : INFO : Epoch 2001 | loss: 0.0311931 | val_loss: 0.0311904 | Time: 6561.74 ms [2022-04-19 13:33:36 main:574] : INFO : Epoch 2002 | loss: 0.0311932 | val_loss: 0.0311897 | Time: 6525.57 ms [2022-04-19 13:33:42 main:574] : INFO : Epoch 2003 | loss: 0.0311934 | val_loss: 0.0311896 | Time: 6538.28 ms [2022-04-19 13:33:49 main:574] : INFO : Epoch 2004 | loss: 0.0311928 | val_loss: 0.0311899 | Time: 6625.94 ms [2022-04-19 13:33:55 main:574] : INFO : Epoch 2005 | loss: 0.0311928 | val_loss: 0.0311897 | Time: 6339.59 ms [2022-04-19 13:34:02 main:574] : INFO : Epoch 2006 | loss: 0.0311923 | val_loss: 0.0311894 | Time: 6409.66 ms [2022-04-19 13:34:08 main:574] : INFO : Epoch 2007 | loss: 0.0311923 | val_loss: 0.031191 | Time: 6645.46 ms [2022-04-19 13:34:15 main:574] : INFO : Epoch 2008 | loss: 0.0311924 | val_loss: 0.03119 | Time: 6617.36 ms [2022-04-19 13:34:22 main:574] : INFO : Epoch 2009 | loss: 0.031194 | val_loss: 0.0311909 | Time: 6770.04 ms [2022-04-19 13:34:29 main:574] : INFO : Epoch 2010 | loss: 0.0311929 | val_loss: 0.0311896 | Time: 6702.8 ms [2022-04-19 13:34:35 main:574] : INFO : Epoch 2011 | loss: 0.031192 | val_loss: 0.0311887 | Time: 6626.54 ms [2022-04-19 13:34:42 main:574] : INFO : Epoch 2012 | loss: 0.0311917 | val_loss: 0.0311888 | Time: 6677.55 ms [2022-04-19 13:34:48 main:574] : INFO : Epoch 2013 | loss: 0.0311921 | val_loss: 0.0311902 | Time: 6512.01 ms [2022-04-19 13:34:55 main:574] : INFO : Epoch 2014 | loss: 0.0311936 | val_loss: 0.0311914 | Time: 6899.6 ms [2022-04-19 13:35:02 main:574] : INFO : Epoch 2015 | loss: 0.031192 | val_loss: 0.03119 | Time: 6700.87 ms [2022-04-19 13:35:10 main:574] : INFO : Epoch 2016 | loss: 0.0311919 | val_loss: 0.0311916 | Time: 7994.3 ms [2022-04-19 13:35:19 main:574] : INFO : Epoch 2017 | loss: 0.0311917 | val_loss: 0.0311894 | Time: 8700.37 ms [2022-04-19 13:35:27 main:574] : INFO : Epoch 2018 | loss: 0.0311935 | val_loss: 0.0311918 | Time: 8760.34 ms [2022-04-19 13:35:36 main:574] : INFO : Epoch 2019 | loss: 0.0311955 | val_loss: 0.031191 | Time: 8700.47 ms [2022-04-19 13:35:45 main:574] : INFO : Epoch 2020 | loss: 0.0311928 | val_loss: 0.0311895 | Time: 8906.65 ms [2022-04-19 13:35:54 main:574] : INFO : Epoch 2021 | loss: 0.0311912 | val_loss: 0.0311894 | Time: 8482.5 ms [2022-04-19 13:36:02 main:574] : INFO : Epoch 2022 | loss: 0.0311909 | val_loss: 0.031189 | Time: 8789.74 ms [2022-04-19 13:36:11 main:574] : INFO : Epoch 2023 | loss: 0.031192 | val_loss: 0.0311897 | Time: 8486.34 ms [2022-04-19 13:36:19 main:574] : INFO : Epoch 2024 | loss: 0.0311929 | val_loss: 0.031194 | Time: 8325.61 ms [2022-04-19 13:36:28 main:574] : INFO : Epoch 2025 | loss: 0.0311924 | val_loss: 0.0311904 | Time: 8343.2 ms [2022-04-19 13:36:36 main:574] : INFO : Epoch 2026 | loss: 0.031193 | val_loss: 0.0311934 | Time: 8596.22 ms [2022-04-19 13:36:45 main:574] : INFO : Epoch 2027 | loss: 0.0311964 | val_loss: 0.0311913 | Time: 8601.5 ms [2022-04-19 13:36:53 main:574] : INFO : Epoch 2028 | loss: 0.0311928 | val_loss: 0.0311887 | Time: 8376.57 ms [2022-04-19 13:37:02 main:574] : INFO : Epoch 2029 | loss: 0.0311938 | val_loss: 0.0311885 | Time: 8789.39 ms [2022-04-19 13:37:10 main:574] : INFO : Epoch 2030 | loss: 0.0311936 | val_loss: 0.0311906 | Time: 8050.49 ms [2022-04-19 13:37:18 main:574] : INFO : Epoch 2031 | loss: 0.0311905 | val_loss: 0.0311873 | Time: 8300.94 ms [2022-04-19 13:37:27 main:574] : INFO : Epoch 2032 | loss: 0.0311905 | val_loss: 0.0311877 | Time: 8261.81 ms [2022-04-19 13:37:35 main:574] : INFO : Epoch 2033 | loss: 0.0311926 | val_loss: 0.0311983 | Time: 8335.02 ms [2022-04-19 13:37:43 main:574] : INFO : Epoch 2034 | loss: 0.0311941 | val_loss: 0.0311926 | Time: 8318.91 ms [2022-04-19 13:37:52 main:574] : INFO : Epoch 2035 | loss: 0.0311921 | val_loss: 0.0311899 | Time: 8160.84 ms [2022-04-19 13:38:00 main:574] : INFO : Epoch 2036 | loss: 0.031193 | val_loss: 0.0311894 | Time: 8219.35 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 13:57:45 main:435] : INFO : Set logging level to 1 [2022-04-19 13:57:45 main:441] : INFO : Running in BOINC Client mode [2022-04-19 13:57:45 main:444] : INFO : Resolving all filenames [2022-04-19 13:57:45 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 13:57:45 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 13:57:45 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 13:57:45 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 13:57:45 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 13:57:45 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 13:57:45 main:474] : INFO : Configuration: [2022-04-19 13:57:45 main:475] : INFO : Model type: GRU [2022-04-19 13:57:45 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 13:57:45 main:477] : INFO : Max Epochs: 2048 [2022-04-19 13:57:45 main:478] : INFO : Batch Size: 128 [2022-04-19 13:57:45 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 13:57:45 main:480] : INFO : Patience: 10 [2022-04-19 13:57:45 main:481] : INFO : Hidden Width: 12 [2022-04-19 13:57:45 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 13:57:45 main:483] : INFO : # Backend Layers: 4 [2022-04-19 13:57:45 main:484] : INFO : # Threads: 1 [2022-04-19 13:57:45 main:486] : INFO : Preparing Dataset [2022-04-19 13:57:45 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 13:57:45 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 15:16:03 main:435] : INFO : Set logging level to 1 [2022-04-19 15:16:03 main:441] : INFO : Running in BOINC Client mode [2022-04-19 15:16:03 main:444] : INFO : Resolving all filenames [2022-04-19 15:16:03 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 15:16:03 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 15:16:03 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 15:16:03 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 15:16:03 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 15:16:03 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 15:16:03 main:474] : INFO : Configuration: [2022-04-19 15:16:03 main:475] : INFO : Model type: GRU [2022-04-19 15:16:03 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 15:16:03 main:477] : INFO : Max Epochs: 2048 [2022-04-19 15:16:03 main:478] : INFO : Batch Size: 128 [2022-04-19 15:16:03 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 15:16:03 main:480] : INFO : Patience: 10 [2022-04-19 15:16:03 main:481] : INFO : Hidden Width: 12 [2022-04-19 15:16:03 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 15:16:03 main:483] : INFO : # Backend Layers: 4 [2022-04-19 15:16:03 main:484] : INFO : # Threads: 1 [2022-04-19 15:16:03 main:486] : INFO : Preparing Dataset [2022-04-19 15:16:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 15:16:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 16:07:32 main:435] : INFO : Set logging level to 1 [2022-04-19 16:07:32 main:441] : INFO : Running in BOINC Client mode [2022-04-19 16:07:32 main:444] : INFO : Resolving all filenames [2022-04-19 16:07:32 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 16:07:32 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 16:07:32 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 16:07:32 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 16:07:32 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 16:07:32 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 16:07:32 main:474] : INFO : Configuration: [2022-04-19 16:07:32 main:475] : INFO : Model type: GRU [2022-04-19 16:07:32 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 16:07:32 main:477] : INFO : Max Epochs: 2048 [2022-04-19 16:07:32 main:478] : INFO : Batch Size: 128 [2022-04-19 16:07:32 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 16:07:32 main:480] : INFO : Patience: 10 [2022-04-19 16:07:32 main:481] : INFO : Hidden Width: 12 [2022-04-19 16:07:32 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 16:07:32 main:483] : INFO : # Backend Layers: 4 [2022-04-19 16:07:32 main:484] : INFO : # Threads: 1 [2022-04-19 16:07:32 main:486] : INFO : Preparing Dataset [2022-04-19 16:07:32 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 16:07:32 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 16:11:43 main:435] : INFO : Set logging level to 1 [2022-04-19 16:11:43 main:441] : INFO : Running in BOINC Client mode [2022-04-19 16:11:43 main:444] : INFO : Resolving all filenames [2022-04-19 16:11:43 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 16:11:43 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 16:11:43 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 16:11:43 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 16:11:43 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 16:11:43 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 16:11:43 main:474] : INFO : Configuration: [2022-04-19 16:11:43 main:475] : INFO : Model type: GRU [2022-04-19 16:11:43 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 16:11:43 main:477] : INFO : Max Epochs: 2048 [2022-04-19 16:11:43 main:478] : INFO : Batch Size: 128 [2022-04-19 16:11:43 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 16:11:43 main:480] : INFO : Patience: 10 [2022-04-19 16:11:43 main:481] : INFO : Hidden Width: 12 [2022-04-19 16:11:43 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 16:11:43 main:483] : INFO : # Backend Layers: 4 [2022-04-19 16:11:43 main:484] : INFO : # Threads: 1 [2022-04-19 16:11:43 main:486] : INFO : Preparing Dataset [2022-04-19 16:11:43 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 16:11:43 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-19 16:59:27 main:435] : INFO : Set logging level to 1 [2022-04-19 16:59:27 main:441] : INFO : Running in BOINC Client mode [2022-04-19 16:59:27 main:444] : INFO : Resolving all filenames [2022-04-19 16:59:27 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-19 16:59:27 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-19 16:59:27 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-19 16:59:27 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-19 16:59:27 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-19 16:59:27 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-19 16:59:27 main:474] : INFO : Configuration: [2022-04-19 16:59:27 main:475] : INFO : Model type: GRU [2022-04-19 16:59:27 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-19 16:59:27 main:477] : INFO : Max Epochs: 2048 [2022-04-19 16:59:27 main:478] : INFO : Batch Size: 128 [2022-04-19 16:59:27 main:479] : INFO : Learning Rate: 0.01 [2022-04-19 16:59:27 main:480] : INFO : Patience: 10 [2022-04-19 16:59:27 main:481] : INFO : Hidden Width: 12 [2022-04-19 16:59:27 main:482] : INFO : # Recurrent Layers: 4 [2022-04-19 16:59:27 main:483] : INFO : # Backend Layers: 4 [2022-04-19 16:59:27 main:484] : INFO : # Threads: 1 [2022-04-19 16:59:27 main:486] : INFO : Preparing Dataset [2022-04-19 16:59:27 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-19 16:59:27 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-20 09:11:37 main:435] : INFO : Set logging level to 1 [2022-04-20 09:11:37 main:441] : INFO : Running in BOINC Client mode [2022-04-20 09:11:37 main:444] : INFO : Resolving all filenames [2022-04-20 09:11:37 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-20 09:11:37 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-20 09:11:37 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-20 09:11:37 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-20 09:11:37 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-20 09:11:37 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-20 09:11:37 main:474] : INFO : Configuration: [2022-04-20 09:11:37 main:475] : INFO : Model type: GRU [2022-04-20 09:11:37 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-20 09:11:37 main:477] : INFO : Max Epochs: 2048 [2022-04-20 09:11:37 main:478] : INFO : Batch Size: 128 [2022-04-20 09:11:37 main:479] : INFO : Learning Rate: 0.01 [2022-04-20 09:11:37 main:480] : INFO : Patience: 10 [2022-04-20 09:11:37 main:481] : INFO : Hidden Width: 12 [2022-04-20 09:11:37 main:482] : INFO : # Recurrent Layers: 4 [2022-04-20 09:11:37 main:483] : INFO : # Backend Layers: 4 [2022-04-20 09:11:37 main:484] : INFO : # Threads: 1 [2022-04-20 09:11:37 main:486] : INFO : Preparing Dataset [2022-04-20 09:11:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-20 09:11:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-20 09:49:57 main:435] : INFO : Set logging level to 1 [2022-04-20 09:49:57 main:441] : INFO : Running in BOINC Client mode [2022-04-20 09:49:57 main:444] : INFO : Resolving all filenames [2022-04-20 09:49:57 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-20 09:49:57 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-20 09:49:57 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-20 09:49:57 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-20 09:49:57 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-20 09:49:57 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-20 09:49:57 main:474] : INFO : Configuration: [2022-04-20 09:49:57 main:475] : INFO : Model type: GRU [2022-04-20 09:49:57 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-20 09:49:57 main:477] : INFO : Max Epochs: 2048 [2022-04-20 09:49:57 main:478] : INFO : Batch Size: 128 [2022-04-20 09:49:57 main:479] : INFO : Learning Rate: 0.01 [2022-04-20 09:49:57 main:480] : INFO : Patience: 10 [2022-04-20 09:49:57 main:481] : INFO : Hidden Width: 12 [2022-04-20 09:49:57 main:482] : INFO : # Recurrent Layers: 4 [2022-04-20 09:49:57 main:483] : INFO : # Backend Layers: 4 [2022-04-20 09:49:57 main:484] : INFO : # Threads: 1 [2022-04-20 09:49:57 main:486] : INFO : Preparing Dataset [2022-04-20 09:49:57 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-20 09:49:58 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 3060 Laptop GPU) [2022-04-20 10:01:18 main:435] : INFO : Set logging level to 1 [2022-04-20 10:01:18 main:441] : INFO : Running in BOINC Client mode [2022-04-20 10:01:18 main:444] : INFO : Resolving all filenames [2022-04-20 10:01:18 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-20 10:01:18 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-20 10:01:18 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-20 10:01:18 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-20 10:01:18 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-20 10:01:18 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-20 10:01:18 main:474] : INFO : Configuration: [2022-04-20 10:01:18 main:475] : INFO : Model type: GRU [2022-04-20 10:01:18 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-20 10:01:18 main:477] : INFO : Max Epochs: 2048 [2022-04-20 10:01:18 main:478] : INFO : Batch Size: 128 [2022-04-20 10:01:18 main:479] : INFO : Learning Rate: 0.01 [2022-04-20 10:01:18 main:480] : INFO : Patience: 10 [2022-04-20 10:01:18 main:481] : INFO : Hidden Width: 12 [2022-04-20 10:01:18 main:482] : INFO : # Recurrent Layers: 4 [2022-04-20 10:01:18 main:483] : INFO : # Backend Layers: 4 [2022-04-20 10:01:18 main:484] : INFO : # Threads: 1 [2022-04-20 10:01:18 main:486] : INFO : Preparing Dataset [2022-04-20 10:01:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-20 10:01:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-20 10:40:52 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-20 10:40:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-20 10:40:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-20 10:40:52 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-20 10:40:52 main:494] : INFO : Creating Model [2022-04-20 10:40:52 main:507] : INFO : Preparing config file [2022-04-20 10:40:52 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-20 10:40:52 main:512] : INFO : Loading config [2022-04-20 10:40:52 main:514] : INFO : Loading state [2022-04-20 11:09:23 main:559] : INFO : Loading DataLoader into Memory [2022-04-20 11:09:23 main:562] : INFO : Starting Training [2022-04-20 11:11:50 main:574] : INFO : Epoch 2032 | loss: 0.0313925 | val_loss: 0.0312728 | Time: 146512 ms [2022-04-20 11:11:57 main:574] : INFO : Epoch 2033 | loss: 0.0312312 | val_loss: 0.0312079 | Time: 7501.58 ms [2022-04-20 11:12:05 main:574] : INFO : Epoch 2034 | loss: 0.0311996 | val_loss: 0.0311946 | Time: 7726.54 ms [2022-04-20 11:12:13 main:574] : INFO : Epoch 2035 | loss: 0.0311967 | val_loss: 0.0311921 | Time: 7819.4 ms [2022-04-20 11:12:21 main:574] : INFO : Epoch 2036 | loss: 0.0311921 | val_loss: 0.0311889 | Time: 7790.32 ms [2022-04-20 11:12:28 main:574] : INFO : Epoch 2037 | loss: 0.0311908 | val_loss: 0.0311886 | Time: 7651.47 ms [2022-04-20 11:12:36 main:574] : INFO : Epoch 2038 | loss: 0.031192 | val_loss: 0.0311948 | Time: 7757.89 ms [2022-04-20 11:12:44 main:574] : INFO : Epoch 2039 | loss: 0.0311944 | val_loss: 0.0311906 | Time: 7685.01 ms [2022-04-20 11:12:52 main:574] : INFO : Epoch 2040 | loss: 0.0311927 | val_loss: 0.0311882 | Time: 7981.36 ms [2022-04-20 11:12:59 main:574] : INFO : Epoch 2041 | loss: 0.0311908 | val_loss: 0.0311911 | Time: 7709.13 ms [2022-04-20 11:13:07 main:574] : INFO : Epoch 2042 | loss: 0.0311925 | val_loss: 0.0311897 | Time: 7889 ms [2022-04-20 11:13:15 main:574] : INFO : Epoch 2043 | loss: 0.0311907 | val_loss: 0.0311879 | Time: 7892.5 ms [2022-04-20 11:13:23 main:574] : INFO : Epoch 2044 | loss: 0.0311923 | val_loss: 0.0311955 | Time: 8017.85 ms [2022-04-20 11:13:31 main:574] : INFO : Epoch 2045 | loss: 0.0311942 | val_loss: 0.0311901 | Time: 8018.59 ms [2022-04-20 11:13:39 main:574] : INFO : Epoch 2046 | loss: 0.0311909 | val_loss: 0.0311881 | Time: 7853.82 ms [2022-04-20 11:13:47 main:574] : INFO : Epoch 2047 | loss: 0.0311944 | val_loss: 0.031191 | Time: 8176.25 ms [2022-04-20 11:13:55 main:574] : INFO : Epoch 2048 | loss: 0.0311931 | val_loss: 0.0311898 | Time: 8068.16 ms [2022-04-20 11:13:55 main:597] : INFO : Saving trained model to model-final.pt, val_loss 0.0311898 [2022-04-20 11:13:55 main:603] : INFO : Saving end state to config to file [2022-04-20 11:13:55 main:608] : INFO : Success, exiting.. 11:13:55 (44308): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)