Task 9265483
| Name | ParityModified-1636178297-7363-2-0_0 |
| Workunit | 6694800 |
| Created | 29 Nov 2021, 19:09:12 UTC |
| Sent | 5 Dec 2021, 22:35:20 UTC |
| Report deadline | 13 Dec 2021, 22:35:20 UTC |
| Received | 21 Dec 2021, 3:59:45 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 10909 |
| Run time | 3 hours 30 min 1 sec |
| CPU time | 3 hours 19 min 52 sec |
| Validate state | Task was reported too late to validate |
| Credit | 0.00 |
| Device peak FLOPS | 4,378.72 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.62 GB |
| Peak swap size | 3.61 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version> <![CDATA[ <stderr_txt> val_loss: 9.19017e-05 | Time: 8314.23 ms [2021-12-14 23:21:19 main:574] : INFO : Epoch 1731 | loss: 8.7296e-05 | val_loss: 9.20946e-05 | Time: 8103.16 ms [2021-12-14 23:21:27 main:574] : INFO : Epoch 1732 | loss: 9.62122e-05 | val_loss: 0.000109384 | Time: 8550.41 ms [2021-12-14 23:21:36 main:574] : INFO : Epoch 1733 | loss: 9.23233e-05 | val_loss: 9.41698e-05 | Time: 8608.7 ms [2021-12-14 23:21:44 main:574] : INFO : Epoch 1734 | loss: 8.69378e-05 | val_loss: 9.46611e-05 | Time: 8159.54 ms [2021-12-14 23:21:52 main:574] : INFO : Epoch 1735 | loss: 8.59519e-05 | val_loss: 9.08007e-05 | Time: 8146.92 ms [2021-12-14 23:22:01 main:574] : INFO : Epoch 1736 | loss: 8.45386e-05 | val_loss: 9.27314e-05 | Time: 8267.09 ms [2021-12-14 23:22:09 main:574] : INFO : Epoch 1737 | loss: 8.77087e-05 | val_loss: 0.000128503 | Time: 8468.11 ms [2021-12-14 23:22:18 main:574] : INFO : Epoch 1738 | loss: 0.000104973 | val_loss: 9.17898e-05 | Time: 8756.88 ms [2021-12-14 23:22:41 main:574] : INFO : Epoch 1739 | loss: 8.51224e-05 | val_loss: 9.0969e-05 | Time: 22545.4 ms [2021-12-14 23:23:00 main:574] : INFO : Epoch 1740 | loss: 8.13083e-05 | val_loss: 8.61784e-05 | Time: 19215.9 ms [2021-12-14 23:23:10 main:574] : INFO : Epoch 1741 | loss: 7.90497e-05 | val_loss: 8.42023e-05 | Time: 9653.95 ms [2021-12-14 23:23:29 main:574] : INFO : Epoch 1742 | loss: 7.80675e-05 | val_loss: 8.47692e-05 | Time: 19567.9 ms [2021-12-14 23:23:37 main:574] : INFO : Epoch 1743 | loss: 7.64125e-05 | val_loss: 8.38027e-05 | Time: 7536.15 ms [2021-12-14 23:23:45 main:574] : INFO : Epoch 1744 | loss: 0.000107588 | val_loss: 9.04038e-05 | Time: 8402.35 ms [2021-12-14 23:23:54 main:574] : INFO : Epoch 1745 | loss: 8.44212e-05 | val_loss: 9.22783e-05 | Time: 8338.06 ms [2021-12-14 23:24:02 main:574] : INFO : Epoch 1746 | loss: 7.53384e-05 | val_loss: 8.33332e-05 | Time: 7448.67 ms [2021-12-14 23:24:09 main:574] : INFO : Epoch 1747 | loss: 7.04724e-05 | val_loss: 7.97198e-05 | Time: 7355.65 ms [2021-12-14 23:24:17 main:574] : INFO : Epoch 1748 | loss: 6.73727e-05 | val_loss: 7.95539e-05 | Time: 7744.04 ms [2021-12-14 23:24:25 main:574] : INFO : Epoch 1749 | loss: 7.52925e-05 | val_loss: 8.4712e-05 | Time: 7653.17 ms [2021-12-14 23:24:32 main:574] : INFO : Epoch 1750 | loss: 6.81572e-05 | val_loss: 7.64894e-05 | Time: 7397.35 ms [2021-12-14 23:24:39 main:574] : INFO : Epoch 1751 | loss: 6.45971e-05 | val_loss: 7.95447e-05 | Time: 6914.58 ms [2021-12-14 23:24:46 main:574] : INFO : Epoch 1752 | loss: 6.46565e-05 | val_loss: 7.1408e-05 | Time: 6862.89 ms [2021-12-14 23:24:53 main:574] : INFO : Epoch 1753 | loss: 7.23736e-05 | val_loss: 9.192e-05 | Time: 6923.73 ms [2021-12-14 23:25:00 main:574] : INFO : Epoch 1754 | loss: 6.69748e-05 | val_loss: 7.55737e-05 | Time: 7084.81 ms [2021-12-14 23:25:08 main:574] : INFO : Epoch 1755 | loss: 6.18936e-05 | val_loss: 7.11762e-05 | Time: 6896.76 ms [2021-12-14 23:25:14 main:574] : INFO : Epoch 1756 | loss: 6.04676e-05 | val_loss: 6.92956e-05 | Time: 6772.01 ms [2021-12-14 23:25:21 main:574] : INFO : Epoch 1757 | loss: 5.98363e-05 | val_loss: 6.87256e-05 | Time: 6666.52 ms [2021-12-14 23:25:28 main:574] : INFO : Epoch 1758 | loss: 5.97967e-05 | val_loss: 6.75276e-05 | Time: 6716.17 ms [2021-12-14 23:25:35 main:574] : INFO : Epoch 1759 | loss: 6.23904e-05 | val_loss: 0.000125854 | Time: 6909.48 ms [2021-12-14 23:25:42 main:574] : INFO : Epoch 1760 | loss: 7.77047e-05 | val_loss: 7.19894e-05 | Time: 6864.64 ms [2021-12-14 23:25:49 main:574] : INFO : Epoch 1761 | loss: 5.86248e-05 | val_loss: 6.82633e-05 | Time: 6961.63 ms [2021-12-14 23:25:56 main:574] : INFO : Epoch 1762 | loss: 5.65019e-05 | val_loss: 6.60822e-05 | Time: 6826.73 ms [2021-12-14 23:26:03 main:574] : INFO : Epoch 1763 | loss: 5.62639e-05 | val_loss: 6.56153e-05 | Time: 6733.65 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2021-12-17 22:09:45 main:435] : INFO : Set logging level to 1 [2021-12-17 22:09:46 main:441] : INFO : Running in BOINC Client mode [2021-12-17 22:09:46 main:444] : INFO : Resolving all filenames [2021-12-17 22:09:50 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2021-12-17 22:09:50 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2021-12-18 16:41:14 main:435] : INFO : Set logging level to 1 [2021-12-18 16:41:15 main:441] : INFO : Running in BOINC Client mode [2021-12-18 16:41:15 main:444] : INFO : Resolving all filenames [2021-12-18 16:41:15 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2021-12-18 16:41:15 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2021-12-18 16:41:15 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2021-12-18 16:41:15 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2021-12-18 16:41:15 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2021-12-18 16:41:15 main:472] : INFO : Dataset filename: dataset.hdf5 [2021-12-18 16:41:15 main:474] : INFO : Configuration: [2021-12-18 16:41:16 main:475] : INFO : Model type: GRU [2021-12-18 16:41:16 main:476] : INFO : Validation Loss Threshold: 0.0001 [2021-12-18 16:41:16 main:477] : INFO : Max Epochs: 2048 [2021-12-18 16:41:16 main:478] : INFO : Batch Size: 128 [2021-12-18 16:41:16 main:479] : INFO : Learning Rate: 0.01 [2021-12-18 16:41:16 main:480] : INFO : Patience: 10 [2021-12-18 16:41:16 main:481] : INFO : Hidden Width: 12 [2021-12-18 16:41:16 main:482] : INFO : # Recurrent Layers: 4 [2021-12-18 16:41:16 main:483] : INFO : # Backend Layers: 4 [2021-12-18 16:41:16 main:484] : INFO : # Threads: 1 [2021-12-18 16:41:16 main:486] : INFO : Preparing Dataset [2021-12-18 16:41:16 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2021-12-18 16:41:16 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2021-12-18 16:41:39 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2021-12-18 16:41:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2021-12-18 16:41:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2021-12-18 16:41:40 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2021-12-18 16:41:40 main:494] : INFO : Creating Model [2021-12-18 16:41:41 main:507] : INFO : Preparing config file [2021-12-18 16:41:41 main:511] : INFO : Found checkpoint, attempting to load... [2021-12-18 16:41:41 main:512] : INFO : Loading config [2021-12-18 16:41:41 main:514] : INFO : Loading state [2021-12-18 16:41:49 main:559] : INFO : Loading DataLoader into Memory [2021-12-18 16:41:49 main:562] : INFO : Starting Training [2021-12-18 16:42:20 main:574] : INFO : Epoch 1730 | loss: 0.00215984 | val_loss: 0.000684756 | Time: 30931.7 ms [2021-12-18 16:42:29 main:574] : INFO : Epoch 1731 | loss: 0.000378173 | val_loss: 0.00018275 | Time: 8078.52 ms [2021-12-18 16:42:37 main:574] : INFO : Epoch 1732 | loss: 0.000167166 | val_loss: 0.000133949 | Time: 7990.89 ms [2021-12-18 16:42:45 main:574] : INFO : Epoch 1733 | loss: 0.000123696 | val_loss: 0.000112362 | Time: 8088.03 ms [2021-12-18 16:42:53 main:574] : INFO : Epoch 1734 | loss: 0.000108982 | val_loss: 0.000105098 | Time: 7574.46 ms [2021-12-18 16:43:01 main:574] : INFO : Epoch 1735 | loss: 0.000102193 | val_loss: 0.000100911 | Time: 7793.35 ms [2021-12-18 16:43:09 main:574] : INFO : Epoch 1736 | loss: 9.90991e-05 | val_loss: 9.93261e-05 | Time: 8081.77 ms [2021-12-18 16:43:17 main:574] : INFO : Epoch 1737 | loss: 9.67761e-05 | val_loss: 9.80333e-05 | Time: 8054.25 ms [2021-12-18 16:43:25 main:574] : INFO : Epoch 1738 | loss: 9.47296e-05 | val_loss: 9.66226e-05 | Time: 7830.3 ms [2021-12-18 16:43:33 main:574] : INFO : Epoch 1739 | loss: 9.33145e-05 | val_loss: 9.59843e-05 | Time: 7861.58 ms [2021-12-18 16:43:41 main:574] : INFO : Epoch 1740 | loss: 9.21157e-05 | val_loss: 9.43645e-05 | Time: 7850.62 ms [2021-12-18 16:43:49 main:574] : INFO : Epoch 1741 | loss: 9.07026e-05 | val_loss: 9.35219e-05 | Time: 7441.29 ms [2021-12-18 16:43:57 main:574] : INFO : Epoch 1742 | loss: 8.95078e-05 | val_loss: 9.30731e-05 | Time: 7672.16 ms [2021-12-18 16:44:05 main:574] : INFO : Epoch 1743 | loss: 8.8311e-05 | val_loss: 9.2176e-05 | Time: 7925.12 ms [2021-12-18 16:44:14 main:574] : INFO : Epoch 1744 | loss: 8.74478e-05 | val_loss: 9.14292e-05 | Time: 9055.88 ms [2021-12-18 16:44:22 main:574] : INFO : Epoch 1745 | loss: 8.65061e-05 | val_loss: 9.01375e-05 | Time: 8264.22 ms [2021-12-18 16:44:30 main:574] : INFO : Epoch 1746 | loss: 8.56311e-05 | val_loss: 9.03939e-05 | Time: 8096.14 ms [2021-12-18 16:44:38 main:574] : INFO : Epoch 1747 | loss: 8.52721e-05 | val_loss: 8.90228e-05 | Time: 7353.18 ms [2021-12-18 16:44:45 main:574] : INFO : Epoch 1748 | loss: 8.45033e-05 | val_loss: 8.89248e-05 | Time: 7313.66 ms [2021-12-18 16:44:53 main:574] : INFO : Epoch 1749 | loss: 8.3835e-05 | val_loss: 8.92445e-05 | Time: 7282.03 ms [2021-12-18 16:45:00 main:574] : INFO : Epoch 1750 | loss: 8.31327e-05 | val_loss: 8.8637e-05 | Time: 7607.34 ms [2021-12-18 16:45:08 main:574] : INFO : Epoch 1751 | loss: 8.22403e-05 | val_loss: 8.73021e-05 | Time: 7316.34 ms [2021-12-18 16:45:15 main:574] : INFO : Epoch 1752 | loss: 8.08846e-05 | val_loss: 8.6342e-05 | Time: 7549.51 ms [2021-12-18 16:45:22 main:574] : INFO : Epoch 1753 | loss: 7.97848e-05 | val_loss: 8.73064e-05 | Time: 7124.93 ms [2021-12-18 16:45:30 main:574] : INFO : Epoch 1754 | loss: 8.28328e-05 | val_loss: 8.5696e-05 | Time: 7713.38 ms [2021-12-18 16:45:50 main:574] : INFO : Epoch 1755 | loss: 7.7939e-05 | val_loss: 8.34761e-05 | Time: 19973.3 ms [2021-12-18 16:46:00 main:574] : INFO : Epoch 1756 | loss: 7.69635e-05 | val_loss: 8.24275e-05 | Time: 9539.26 ms [2021-12-18 16:46:09 main:574] : INFO : Epoch 1757 | loss: 8.36942e-05 | val_loss: 8.40849e-05 | Time: 8941.67 ms [2021-12-18 16:46:17 main:574] : INFO : Epoch 1758 | loss: 7.72742e-05 | val_loss: 8.2293e-05 | Time: 8715.91 ms [2021-12-18 16:46:25 main:574] : INFO : Epoch 1759 | loss: 7.4469e-05 | val_loss: 8.08298e-05 | Time: 7581.65 ms [2021-12-18 16:46:33 main:574] : INFO : Epoch 1760 | loss: 7.32621e-05 | val_loss: 8.05625e-05 | Time: 7361.85 ms [2021-12-18 16:46:41 main:574] : INFO : Epoch 1761 | loss: 8.12741e-05 | val_loss: 9.1234e-05 | Time: 7835.37 ms [2021-12-18 16:46:48 main:574] : INFO : Epoch 1762 | loss: 7.78516e-05 | val_loss: 8.06253e-05 | Time: 7477.61 ms [2021-12-18 16:46:55 main:574] : INFO : Epoch 1763 | loss: 7.19366e-05 | val_loss: 8.12772e-05 | Time: 7403.61 ms [2021-12-18 16:47:03 main:574] : INFO : Epoch 1764 | loss: 7.08049e-05 | val_loss: 7.83237e-05 | Time: 7342.05 ms [2021-12-18 16:47:10 main:574] : INFO : Epoch 1765 | loss: 6.92489e-05 | val_loss: 7.75055e-05 | Time: 7276.18 ms [2021-12-18 16:47:18 main:574] : INFO : Epoch 1766 | loss: 7.79478e-05 | val_loss: 8.08021e-05 | Time: 7689.48 ms [2021-12-18 16:47:25 main:574] : INFO : Epoch 1767 | loss: 7.79881e-05 | val_loss: 7.9577e-05 | Time: 7320.87 ms [2021-12-18 16:47:33 main:574] : INFO : Epoch 1768 | loss: 7.20222e-05 | val_loss: 7.99739e-05 | Time: 7343.84 ms [2021-12-18 16:47:40 main:574] : INFO : Epoch 1769 | loss: 6.92694e-05 | val_loss: 7.64871e-05 | Time: 7687.97 ms [2021-12-18 16:47:48 main:574] : INFO : Epoch 1770 | loss: 6.69601e-05 | val_loss: 7.55957e-05 | Time: 7272.87 ms [2021-12-18 16:47:55 main:574] : INFO : Epoch 1771 | loss: 6.65982e-05 | val_loss: 7.48695e-05 | Time: 7552.58 ms [2021-12-18 16:48:03 main:574] : INFO : Epoch 1772 | loss: 6.66811e-05 | val_loss: 7.66427e-05 | Time: 7473.65 ms [2021-12-18 16:48:10 main:574] : INFO : Epoch 1773 | loss: 6.6602e-05 | val_loss: 7.38993e-05 | Time: 7244.75 ms [2021-12-18 16:48:18 main:574] : INFO : Epoch 1774 | loss: 6.55968e-05 | val_loss: 7.48378e-05 | Time: 7613.22 ms [2021-12-18 16:48:25 main:574] : INFO : Epoch 1775 | loss: 6.59003e-05 | val_loss: 7.53824e-05 | Time: 7470.27 ms [2021-12-18 16:48:33 main:574] : INFO : Epoch 1776 | loss: 6.58133e-05 | val_loss: 7.56981e-05 | Time: 7445.76 ms [2021-12-18 16:48:40 main:574] : INFO : Epoch 1777 | loss: 6.45062e-05 | val_loss: 7.53797e-05 | Time: 7514.01 ms [2021-12-18 16:48:48 main:574] : INFO : Epoch 1778 | loss: 6.45372e-05 | val_loss: 7.38149e-05 | Time: 7294.02 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2021-12-18 17:03:17 main:435] : INFO : Set logging level to 1 [2021-12-18 17:03:17 main:441] : INFO : Running in BOINC Client mode [2021-12-18 17:03:17 main:444] : INFO : Resolving all filenames [2021-12-18 17:03:17 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2021-12-18 17:03:17 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2021-12-18 17:03:17 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2021-12-18 17:03:17 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2021-12-18 17:03:17 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2021-12-18 17:03:18 main:472] : INFO : Dataset filename: dataset.hdf5 [2021-12-18 17:03:18 main:474] : INFO : Configuration: [2021-12-18 17:03:18 main:475] : INFO : Model type: GRU [2021-12-18 17:03:18 main:476] : INFO : Validation Loss Threshold: 0.0001 [2021-12-18 17:03:18 main:477] : INFO : Max Epochs: 2048 [2021-12-18 17:03:18 main:478] : INFO : Batch Size: 128 [2021-12-18 17:03:18 main:479] : INFO : Learning Rate: 0.01 [2021-12-18 17:03:18 main:480] : INFO : Patience: 10 [2021-12-18 17:03:18 main:481] : INFO : Hidden Width: 12 [2021-12-18 17:03:18 main:482] : INFO : # Recurrent Layers: 4 [2021-12-18 17:03:18 main:483] : INFO : # Backend Layers: 4 [2021-12-18 17:03:18 main:484] : INFO : # Threads: 1 [2021-12-18 17:03:18 main:486] : INFO : Preparing Dataset [2021-12-18 17:03:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2021-12-18 17:03:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2021-12-18 17:03:20 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2021-12-18 17:03:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2021-12-18 17:03:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2021-12-18 17:03:20 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2021-12-18 17:03:20 main:494] : INFO : Creating Model [2021-12-18 17:03:20 main:507] : INFO : Preparing config file [2021-12-18 17:03:20 main:511] : INFO : Found checkpoint, attempting to load... [2021-12-18 17:03:20 main:512] : INFO : Loading config [2021-12-18 17:03:20 main:514] : INFO : Loading state [2021-12-18 17:03:21 main:559] : INFO : Loading DataLoader into Memory [2021-12-18 17:03:21 main:562] : INFO : Starting Training [2021-12-18 17:03:25 main:574] : INFO : Epoch 1760 | loss: 0.00286382 | val_loss: 0.000714148 | Time: 3942.76 ms [2021-12-18 17:03:31 main:574] : INFO : Epoch 1761 | loss: 0.000459089 | val_loss: 0.000299308 | Time: 6097.05 ms [2021-12-18 17:03:39 main:574] : INFO : Epoch 1762 | loss: 0.000195248 | val_loss: 0.000137234 | Time: 7552.21 ms [2021-12-18 17:03:46 main:574] : INFO : Epoch 1763 | loss: 0.000125127 | val_loss: 0.00011438 | Time: 7372.03 ms [2021-12-18 17:03:54 main:574] : INFO : Epoch 1764 | loss: 0.000104605 | val_loss: 9.96578e-05 | Time: 7252.17 ms [2021-12-18 17:04:01 main:574] : INFO : Epoch 1765 | loss: 9.44048e-05 | val_loss: 9.40146e-05 | Time: 7219.13 ms [2021-12-18 17:04:08 main:574] : INFO : Epoch 1766 | loss: 8.88056e-05 | val_loss: 9.04256e-05 | Time: 7211.96 ms [2021-12-18 17:04:16 main:574] : INFO : Epoch 1767 | loss: 8.43609e-05 | val_loss: 8.80085e-05 | Time: 7295.01 ms [2021-12-18 17:04:23 main:574] : INFO : Epoch 1768 | loss: 8.11527e-05 | val_loss: 8.57009e-05 | Time: 7346.57 ms [2021-12-18 17:04:30 main:574] : INFO : Epoch 1769 | loss: 8.39171e-05 | val_loss: 9.9124e-05 | Time: 7323.76 ms [2021-12-18 17:04:38 main:574] : INFO : Epoch 1770 | loss: 8.93528e-05 | val_loss: 8.94014e-05 | Time: 7781.53 ms [2021-12-18 17:04:46 main:574] : INFO : Epoch 1771 | loss: 7.95336e-05 | val_loss: 8.26616e-05 | Time: 8256.6 ms [2021-12-18 17:04:54 main:574] : INFO : Epoch 1772 | loss: 7.56596e-05 | val_loss: 8.15723e-05 | Time: 7670.53 ms [2021-12-18 17:05:02 main:574] : INFO : Epoch 1773 | loss: 7.38437e-05 | val_loss: 8.06078e-05 | Time: 7592.96 ms [2021-12-18 17:05:09 main:574] : INFO : Epoch 1774 | loss: 7.23623e-05 | val_loss: 7.94442e-05 | Time: 7347.74 ms [2021-12-18 17:05:16 main:574] : INFO : Epoch 1775 | loss: 7.13191e-05 | val_loss: 7.86887e-05 | Time: 7285.17 ms [2021-12-18 17:05:24 main:574] : INFO : Epoch 1776 | loss: 7.02765e-05 | val_loss: 7.78604e-05 | Time: 7571.49 ms [2021-12-18 17:05:32 main:574] : INFO : Epoch 1777 | loss: 6.92766e-05 | val_loss: 7.71892e-05 | Time: 7722.52 ms [2021-12-18 17:05:39 main:574] : INFO : Epoch 1778 | loss: 6.85838e-05 | val_loss: 7.73119e-05 | Time: 7313 ms [2021-12-18 17:05:47 main:574] : INFO : Epoch 1779 | loss: 6.79176e-05 | val_loss: 7.68534e-05 | Time: 7365.34 ms [2021-12-18 17:05:54 main:574] : INFO : Epoch 1780 | loss: 6.73461e-05 | val_loss: 7.61906e-05 | Time: 7159.5 ms [2021-12-18 17:06:01 main:574] : INFO : Epoch 1781 | loss: 6.67272e-05 | val_loss: 7.58443e-05 | Time: 7427.52 ms [2021-12-18 17:06:09 main:574] : INFO : Epoch 1782 | loss: 6.6516e-05 | val_loss: 7.53491e-05 | Time: 7448.12 ms [2021-12-18 17:06:16 main:574] : INFO : Epoch 1783 | loss: 6.61272e-05 | val_loss: 7.54877e-05 | Time: 7590.89 ms [2021-12-18 17:06:24 main:574] : INFO : Epoch 1784 | loss: 6.58407e-05 | val_loss: 7.52986e-05 | Time: 7378.01 ms [2021-12-18 17:06:31 main:574] : INFO : Epoch 1785 | loss: 6.59068e-05 | val_loss: 7.49169e-05 | Time: 7275.12 ms [2021-12-18 17:06:39 main:574] : INFO : Epoch 1786 | loss: 6.54649e-05 | val_loss: 7.44472e-05 | Time: 7259.34 ms [2021-12-18 17:06:46 main:574] : INFO : Epoch 1787 | loss: 6.50758e-05 | val_loss: 7.4395e-05 | Time: 7568.42 ms [2021-12-18 17:06:54 main:574] : INFO : Epoch 1788 | loss: 6.51797e-05 | val_loss: 7.59567e-05 | Time: 7350.57 ms [2021-12-18 17:07:01 main:574] : INFO : Epoch 1789 | loss: 6.55955e-05 | val_loss: 7.41504e-05 | Time: 7474.8 ms [2021-12-18 17:07:09 main:574] : INFO : Epoch 1790 | loss: 6.5184e-05 | val_loss: 7.4762e-05 | Time: 7250.57 ms [2021-12-18 17:07:17 main:574] : INFO : Epoch 1791 | loss: 6.44357e-05 | val_loss: 7.34871e-05 | Time: 8162.21 ms [2021-12-18 17:07:24 main:574] : INFO : Epoch 1792 | loss: 6.48714e-05 | val_loss: 8.23364e-05 | Time: 7311.35 ms [2021-12-18 17:07:32 main:574] : INFO : Epoch 1793 | loss: 6.95084e-05 | val_loss: 7.41748e-05 | Time: 7414.93 ms [2021-12-18 17:07:39 main:574] : INFO : Epoch 1794 | loss: 6.4953e-05 | val_loss: 7.35576e-05 | Time: 7464.72 ms [2021-12-18 17:07:46 main:574] : INFO : Epoch 1795 | loss: 6.38183e-05 | val_loss: 7.38575e-05 | Time: 7271.88 ms [2021-12-18 17:07:54 main:574] : INFO : Epoch 1796 | loss: 6.35736e-05 | val_loss: 7.31042e-05 | Time: 7050.38 ms [2021-12-18 17:08:01 main:574] : INFO : Epoch 1797 | loss: 6.32302e-05 | val_loss: 7.3099e-05 | Time: 7298.06 ms [2021-12-18 17:08:08 main:574] : INFO : Epoch 1798 | loss: 6.28527e-05 | val_loss: 7.26612e-05 | Time: 7378.78 ms [2021-12-18 17:08:16 main:574] : INFO : Epoch 1799 | loss: 6.31041e-05 | val_loss: 7.27806e-05 | Time: 7556.23 ms [2021-12-18 17:08:23 main:574] : INFO : Epoch 1800 | loss: 6.31717e-05 | val_loss: 7.27049e-05 | Time: 7409.57 ms [2021-12-18 17:08:31 main:574] : INFO : Epoch 1801 | loss: 6.30993e-05 | val_loss: 7.39646e-05 | Time: 7312.83 ms [2021-12-18 17:08:38 main:574] : INFO : Epoch 1802 | loss: 6.29588e-05 | val_loss: 7.32504e-05 | Time: 7315.2 ms [2021-12-18 17:08:46 main:574] : INFO : Epoch 1803 | loss: 6.35651e-05 | val_loss: 7.38436e-05 | Time: 7358.33 ms [2021-12-18 17:08:53 main:574] : INFO : Epoch 1804 | loss: 6.21389e-05 | val_loss: 7.21309e-05 | Time: 7249.86 ms [2021-12-18 17:09:00 main:574] : INFO : Epoch 1805 | loss: 6.21378e-05 | val_loss: 7.30361e-05 | Time: 7262.4 ms [2021-12-18 17:09:08 main:574] : INFO : Epoch 1806 | loss: 7.01098e-05 | val_loss: 7.30871e-05 | Time: 7401.8 ms [2021-12-18 17:09:15 main:574] : INFO : Epoch 1807 | loss: 6.2888e-05 | val_loss: 7.1906e-05 | Time: 7362.47 ms [2021-12-18 17:09:22 main:574] : INFO : Epoch 1808 | loss: 6.18327e-05 | val_loss: 7.19888e-05 | Time: 7171.76 ms [2021-12-18 17:09:30 main:574] : INFO : Epoch 1809 | loss: 6.18697e-05 | val_loss: 7.17211e-05 | Time: 7313.92 ms [2021-12-18 17:09:37 main:574] : INFO : Epoch 1810 | loss: 6.12883e-05 | val_loss: 7.22429e-05 | Time: 7509.55 ms [2021-12-18 17:09:45 main:574] : INFO : Epoch 1811 | loss: 6.11654e-05 | val_loss: 7.17554e-05 | Time: 7504.25 ms [2021-12-18 17:09:52 main:574] : INFO : Epoch 1812 | loss: 6.13242e-05 | val_loss: 7.15695e-05 | Time: 7328.58 ms [2021-12-18 17:10:00 main:574] : INFO : Epoch 1813 | loss: 6.14572e-05 | val_loss: 7.23647e-05 | Time: 7375.66 ms [2021-12-18 17:10:07 main:574] : INFO : Epoch 1814 | loss: 6.10352e-05 | val_loss: 7.20591e-05 | Time: 7518.97 ms [2021-12-18 17:10:14 main:574] : INFO : Epoch 1815 | loss: 6.0718e-05 | val_loss: 7.09918e-05 | Time: 7263.44 ms [2021-12-18 17:10:22 main:574] : INFO : Epoch 1816 | loss: 6.20689e-05 | val_loss: 7.76224e-05 | Time: 7941.28 ms [2021-12-18 17:10:30 main:574] : INFO : Epoch 1817 | loss: 6.12468e-05 | val_loss: 7.13115e-05 | Time: 7354.03 ms [2021-12-18 17:10:37 main:574] : INFO : Epoch 1818 | loss: 5.88922e-05 | val_loss: 7.16654e-05 | Time: 7319.2 ms [2021-12-18 17:10:45 main:574] : INFO : Epoch 1819 | loss: 5.8463e-05 | val_loss: 7.13356e-05 | Time: 7306 ms [2021-12-18 17:10:52 main:574] : INFO : Epoch 1820 | loss: 6.13443e-05 | val_loss: 7.3055e-05 | Time: 7580.55 ms [2021-12-18 17:11:00 main:574] : INFO : Epoch 1821 | loss: 5.90504e-05 | val_loss: 7.33317e-05 | Time: 7318.54 ms [2021-12-18 17:11:07 main:574] : INFO : Epoch 1822 | loss: 5.79183e-05 | val_loss: 6.94276e-05 | Time: 7577.53 ms [2021-12-18 17:11:15 main:574] : INFO : Epoch 1823 | loss: 5.66683e-05 | val_loss: 7.00219e-05 | Time: 7575.43 ms [2021-12-18 17:11:22 main:574] : INFO : Epoch 1824 | loss: 5.71114e-05 | val_loss: 7.64601e-05 | Time: 7491.17 ms [2021-12-18 17:11:30 main:574] : INFO : Epoch 1825 | loss: 6.11423e-05 | val_loss: 7.05225e-05 | Time: 7223.13 ms [2021-12-18 17:11:37 main:574] : INFO : Epoch 1826 | loss: 5.74965e-05 | val_loss: 7.02137e-05 | Time: 7420.79 ms [2021-12-18 17:11:44 main:574] : INFO : Epoch 1827 | loss: 5.65379e-05 | val_loss: 6.81058e-05 | Time: 7358.11 ms [2021-12-18 17:11:52 main:574] : INFO : Epoch 1828 | loss: 5.51407e-05 | val_loss: 6.75113e-05 | Time: 7516.55 ms [2021-12-18 17:11:59 main:574] : INFO : Epoch 1829 | loss: 5.56405e-05 | val_loss: 6.83599e-05 | Time: 7314.36 ms [2021-12-18 17:12:07 main:574] : INFO : Epoch 1830 | loss: 5.62929e-05 | val_loss: 7.28089e-05 | Time: 7413.65 ms [2021-12-18 17:12:14 main:574] : INFO : Epoch 1831 | loss: 5.81903e-05 | val_loss: 6.65189e-05 | Time: 7330.94 ms [2021-12-18 17:12:22 main:574] : INFO : Epoch 1832 | loss: 5.51069e-05 | val_loss: 6.74461e-05 | Time: 7561.34 ms [2021-12-18 17:12:29 main:574] : INFO : Epoch 1833 | loss: 5.49423e-05 | val_loss: 6.63556e-05 | Time: 7271.16 ms [2021-12-18 17:12:37 main:574] : INFO : Epoch 1834 | loss: 5.44985e-05 | val_loss: 6.74913e-05 | Time: 7504.68 ms [2021-12-18 17:12:44 main:574] : INFO : Epoch 1835 | loss: 5.40081e-05 | val_loss: 6.56586e-05 | Time: 7330.87 ms [2021-12-18 17:12:52 main:574] : INFO : Epoch 1836 | loss: 5.85451e-05 | val_loss: 6.55804e-05 | Time: 7518.67 ms [2021-12-18 17:12:59 main:574] : INFO : Epoch 1837 | loss: 5.34756e-05 | val_loss: 6.54773e-05 | Time: 7390.37 ms [2021-12-18 17:13:07 main:574] : INFO : Epoch 1838 | loss: 5.38096e-05 | val_loss: 6.50082e-05 | Time: 7461.02 ms [2021-12-18 17:13:14 main:574] : INFO : Epoch 1839 | loss: 5.24575e-05 | val_loss: 6.34694e-05 | Time: 7217.15 ms [2021-12-18 17:13:21 main:574] : INFO : Epoch 1840 | loss: 5.28574e-05 | val_loss: 6.5114e-05 | Time: 7450.18 ms [2021-12-18 17:13:29 main:574] : INFO : Epoch 1841 | loss: 5.58147e-05 | val_loss: 6.28222e-05 | Time: 7199.78 ms [2021-12-18 17:13:36 main:574] : INFO : Epoch 1842 | loss: 5.43763e-05 | val_loss: 6.63078e-05 | Time: 7475.43 ms [2021-12-18 17:13:44 main:574] : INFO : Epoch 1843 | loss: 5.30974e-05 | val_loss: 6.32189e-05 | Time: 7419.08 ms [2021-12-18 17:13:51 main:574] : INFO : Epoch 1844 | loss: 5.26311e-05 | val_loss: 6.56331e-05 | Time: 7464.51 ms [2021-12-18 17:13:58 main:574] : INFO : Epoch 1845 | loss: 5.36728e-05 | val_loss: 6.30916e-05 | Time: 7101.76 ms [2021-12-18 17:14:06 main:574] : INFO : Epoch 1846 | loss: 5.20605e-05 | val_loss: 6.13289e-05 | Time: 7564.82 ms [2021-12-18 17:14:13 main:574] : INFO : Epoch 1847 | loss: 5.27578e-05 | val_loss: 6.11313e-05 | Time: 7207.95 ms [2021-12-18 17:14:21 main:574] : INFO : Epoch 1848 | loss: 5.30111e-05 | val_loss: 6.1509e-05 | Time: 7415.95 ms [2021-12-18 17:14:28 main:574] : INFO : Epoch 1849 | loss: 5.12362e-05 | val_loss: 6.44071e-05 | Time: 7459.24 ms [2021-12-18 17:14:36 main:574] : INFO : Epoch 1850 | loss: 5.18365e-05 | val_loss: 6.41441e-05 | Time: 7329.75 ms [2021-12-18 17:14:43 main:574] : INFO : Epoch 1851 | loss: 5.15518e-05 | val_loss: 6.1727e-05 | Time: 7458.15 ms [2021-12-18 17:14:51 main:574] : INFO : Epoch 1852 | loss: 5.18553e-05 | val_loss: 6.21812e-05 | Time: 7717.08 ms [2021-12-18 17:14:59 main:574] : INFO : Epoch 1853 | loss: 5.08173e-05 | val_loss: 6.08634e-05 | Time: 8029.4 ms [2021-12-18 17:15:07 main:574] : INFO : Epoch 1854 | loss: 4.98907e-05 | val_loss: 6.52547e-05 | Time: 7911.35 ms [2021-12-18 17:15:14 main:574] : INFO : Epoch 1855 | loss: 5.35691e-05 | val_loss: 5.92799e-05 | Time: 7349.98 ms [2021-12-18 17:15:22 main:574] : INFO : Epoch 1856 | loss: 4.8948e-05 | val_loss: 5.91051e-05 | Time: 7326.58 ms [2021-12-18 17:15:29 main:574] : INFO : Epoch 1857 | loss: 4.93101e-05 | val_loss: 5.78925e-05 | Time: 7116.59 ms [2021-12-18 17:15:36 main:574] : INFO : Epoch 1858 | loss: 5.02234e-05 | val_loss: 6.15213e-05 | Time: 7334.79 ms [2021-12-18 17:15:43 main:574] : INFO : Epoch 1859 | loss: 5.83878e-05 | val_loss: 6.06522e-05 | Time: 7268.04 ms [2021-12-18 17:15:51 main:574] : INFO : Epoch 1860 | loss: 5.02704e-05 | val_loss: 5.8346e-05 | Time: 7481.05 ms [2021-12-18 17:15:58 main:574] : INFO : Epoch 1861 | loss: 4.7732e-05 | val_loss: 5.77283e-05 | Time: 7282.82 ms [2021-12-18 17:16:06 main:574] : INFO : Epoch 1862 | loss: 4.86486e-05 | val_loss: 5.9085e-05 | Time: 7339.8 ms [2021-12-18 17:16:13 main:574] : INFO : Epoch 1863 | loss: 5.00371e-05 | val_loss: 5.60604e-05 | Time: 7444.89 ms [2021-12-18 17:16:21 main:574] : INFO : Epoch 1864 | loss: 4.70829e-05 | val_loss: 5.62631e-05 | Time: 7289.47 ms [2021-12-18 17:16:28 main:574] : INFO : Epoch 1865 | loss: 4.68269e-05 | val_loss: 5.56342e-05 | Time: 7267.54 ms [2021-12-18 17:16:35 main:574] : INFO : Epoch 1866 | loss: 4.83913e-05 | val_loss: 6.21801e-05 | Time: 7491.29 ms [2021-12-18 17:16:43 main:574] : INFO : Epoch 1867 | loss: 4.75566e-05 | val_loss: 5.73852e-05 | Time: 7413.93 ms [2021-12-18 17:16:50 main:574] : INFO : Epoch 1868 | loss: 4.71329e-05 | val_loss: 5.58979e-05 | Time: 7588.16 ms [2021-12-18 17:16:58 main:574] : INFO : Epoch 1869 | loss: 5.0741e-05 | val_loss: 5.96654e-05 | Time: 7206.4 ms [2021-12-18 17:17:05 main:574] : INFO : Epoch 1870 | loss: 4.95232e-05 | val_loss: 6.05684e-05 | Time: 7701.62 ms [2021-12-18 17:17:13 main:574] : INFO : Epoch 1871 | loss: 4.78892e-05 | val_loss: 5.59608e-05 | Time: 7372.38 ms [2021-12-18 17:17:20 main:574] : INFO : Epoch 1872 | loss: 4.60292e-05 | val_loss: 5.44036e-05 | Time: 7468.53 ms [2021-12-18 17:17:28 main:574] : INFO : Epoch 1873 | loss: 4.60941e-05 | val_loss: 5.53819e-05 | Time: 7165.39 ms [2021-12-18 17:17:35 main:574] : INFO : Epoch 1874 | loss: 4.49896e-05 | val_loss: 5.43029e-05 | Time: 7431.7 ms [2021-12-18 17:17:43 main:574] : INFO : Epoch 1875 | loss: 5.79175e-05 | val_loss: 6.04527e-05 | Time: 7415.27 ms [2021-12-18 17:17:50 main:574] : INFO : Epoch 1876 | loss: 4.87879e-05 | val_loss: 5.48281e-05 | Time: 7320.06 ms [2021-12-18 17:17:57 main:574] : INFO : Epoch 1877 | loss: 4.47394e-05 | val_loss: 5.41464e-05 | Time: 7163.65 ms [2021-12-18 17:18:05 main:574] : INFO : Epoch 1878 | loss: 4.43003e-05 | val_loss: 5.54434e-05 | Time: 7616.37 ms [2021-12-18 17:18:13 main:574] : INFO : Epoch 1879 | loss: 4.68631e-05 | val_loss: 6.38199e-05 | Time: 7901.19 ms [2021-12-18 17:18:20 main:574] : INFO : Epoch 1880 | loss: 5.107e-05 | val_loss: 6.0232e-05 | Time: 7580.41 ms [2021-12-18 17:18:28 main:574] : INFO : Epoch 1881 | loss: 4.7984e-05 | val_loss: 5.56309e-05 | Time: 7368.08 ms [2021-12-18 17:18:35 main:574] : INFO : Epoch 1882 | loss: 4.59998e-05 | val_loss: 5.86222e-05 | Time: 7722.25 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2021-12-20 18:00:33 main:435] : INFO : Set logging level to 1 [2021-12-20 18:00:33 main:441] : INFO : Running in BOINC Client mode [2021-12-20 18:00:33 main:444] : INFO : Resolving all filenames [2021-12-20 18:00:33 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2021-12-20 18:00:33 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2021-12-20 18:00:33 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2021-12-20 18:00:33 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2021-12-20 18:00:33 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2021-12-20 18:00:33 main:472] : INFO : Dataset filename: dataset.hdf5 [2021-12-20 18:00:33 main:474] : INFO : Configuration: [2021-12-20 18:00:34 main:475] : INFO : Model type: GRU [2021-12-20 18:00:34 main:476] : INFO : Validation Loss Threshold: 0.0001 [2021-12-20 18:00:34 main:477] : INFO : Max Epochs: 2048 [2021-12-20 18:00:34 main:478] : INFO : Batch Size: 128 [2021-12-20 18:00:34 main:479] : INFO : Learning Rate: 0.01 [2021-12-20 18:00:34 main:480] : INFO : Patience: 10 [2021-12-20 18:00:34 main:481] : INFO : Hidden Width: 12 [2021-12-20 18:00:34 main:482] : INFO : # Recurrent Layers: 4 [2021-12-20 18:00:35 main:483] : INFO : # Backend Layers: 4 [2021-12-20 18:00:35 main:484] : INFO : # Threads: 1 [2021-12-20 18:00:35 main:486] : INFO : Preparing Dataset [2021-12-20 18:00:35 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2021-12-20 19:30:20 main:435] : INFO : Set logging level to 1 [2021-12-20 19:30:21 main:441] : INFO : Running in BOINC Client mode [2021-12-20 19:30:21 main:444] : INFO : Resolving all filenames [2021-12-20 19:30:21 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2021-12-20 19:30:21 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2021-12-20 19:30:21 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2021-12-20 19:30:21 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2021-12-20 19:30:21 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2021-12-20 19:30:21 main:472] : INFO : Dataset filename: dataset.hdf5 [2021-12-20 19:30:21 main:474] : INFO : Configuration: [2021-12-20 19:30:21 main:475] : INFO : Model type: GRU [2021-12-20 19:30:21 main:476] : INFO : Validation Loss Threshold: 0.0001 [2021-12-20 19:30:21 main:477] : INFO : Max Epochs: 2048 [2021-12-20 19:30:21 main:478] : INFO : Batch Size: 128 [2021-12-20 19:30:22 main:479] : INFO : Learning Rate: 0.01 [2021-12-20 19:30:22 main:480] : INFO : Patience: 10 [2021-12-20 19:30:22 main:481] : INFO : Hidden Width: 12 [2021-12-20 19:30:22 main:482] : INFO : # Recurrent Layers: 4 [2021-12-20 19:30:22 main:483] : INFO : # Backend Layers: 4 [2021-12-20 19:30:22 main:484] : INFO : # Threads: 1 [2021-12-20 19:30:22 main:486] : INFO : Preparing Dataset [2021-12-20 19:30:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2021-12-20 19:30:22 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2021-12-20 19:30:35 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2021-12-20 19:30:35 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2021-12-20 19:30:35 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2021-12-20 19:30:36 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2021-12-20 19:30:36 main:494] : INFO : Creating Model [2021-12-20 19:30:37 main:507] : INFO : Preparing config file [2021-12-20 19:30:37 main:511] : INFO : Found checkpoint, attempting to load... [2021-12-20 19:30:37 main:512] : INFO : Loading config [2021-12-20 19:30:37 main:514] : INFO : Loading state [2021-12-20 19:31:43 main:559] : INFO : Loading DataLoader into Memory [2021-12-20 19:31:43 main:562] : INFO : Starting Training [2021-12-20 19:32:17 main:574] : INFO : Epoch 1883 | loss: 0.00370456 | val_loss: 0.0010275 | Time: 33994.2 ms [2021-12-20 19:32:37 main:574] : INFO : Epoch 1884 | loss: 0.000578514 | val_loss: 0.00024515 | Time: 19596.8 ms [2021-12-20 19:32:46 main:574] : INFO : Epoch 1885 | loss: 0.000180245 | val_loss: 0.000122171 | Time: 8608.65 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB) [2021-12-20 19:38:00 main:435] : INFO : Set logging level to 1 [2021-12-20 19:38:00 main:441] : INFO : Running in BOINC Client mode [2021-12-20 19:38:00 main:444] : INFO : Resolving all filenames [2021-12-20 19:38:00 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2021-12-20 19:38:00 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2021-12-20 19:38:00 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2021-12-20 19:38:00 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2021-12-20 19:38:01 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2021-12-20 19:38:01 main:472] : INFO : Dataset filename: dataset.hdf5 [2021-12-20 19:38:01 main:474] : INFO : Configuration: [2021-12-20 19:38:01 main:475] : INFO : Model type: GRU [2021-12-20 19:38:01 main:476] : INFO : Validation Loss Threshold: 0.0001 [2021-12-20 19:38:01 main:477] : INFO : Max Epochs: 2048 [2021-12-20 19:38:01 main:478] : INFO : Batch Size: 128 [2021-12-20 19:38:01 main:479] : INFO : Learning Rate: 0.01 [2021-12-20 19:38:01 main:480] : INFO : Patience: 10 [2021-12-20 19:38:01 main:481] : INFO : Hidden Width: 12 [2021-12-20 19:38:01 main:482] : INFO : # Recurrent Layers: 4 [2021-12-20 19:38:01 main:483] : INFO : # Backend Layers: 4 [2021-12-20 19:38:01 main:484] : INFO : # Threads: 1 [2021-12-20 19:38:01 main:486] : INFO : Preparing Dataset [2021-12-20 19:38:02 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2021-12-20 19:38:02 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2021-12-20 19:38:08 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2021-12-20 19:38:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2021-12-20 19:38:09 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2021-12-20 19:38:09 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2021-12-20 19:38:09 main:494] : INFO : Creating Model [2021-12-20 19:38:09 main:507] : INFO : Preparing config file [2021-12-20 19:38:09 main:511] : INFO : Found checkpoint, attempting to load... [2021-12-20 19:38:09 main:512] : INFO : Loading config [2021-12-20 19:38:09 main:514] : INFO : Loading state [2021-12-20 19:38:17 main:559] : INFO : Loading DataLoader into Memory [2021-12-20 19:38:18 main:562] : INFO : Starting Training [2021-12-20 19:38:31 main:574] : INFO : Epoch 1883 | loss: 0.00347752 | val_loss: 0.00115406 | Time: 13078.1 ms [2021-12-20 19:40:18 main:574] : INFO : Epoch 1884 | loss: 0.000554904 | val_loss: 0.000229782 | Time: 107586 ms [2021-12-20 19:40:40 main:574] : INFO : Epoch 1885 | loss: 0.000178639 | val_loss: 0.000122431 | Time: 21293.6 ms [2021-12-20 19:41:01 main:574] : INFO : Epoch 1886 | loss: 0.000100363 | val_loss: 8.85433e-05 | Time: 20565.2 ms [2021-12-20 19:41:19 main:574] : INFO : Epoch 1887 | loss: 7.68944e-05 | val_loss: 7.51836e-05 | Time: 18790 ms [2021-12-20 19:41:27 main:574] : INFO : Epoch 1888 | loss: 6.65593e-05 | val_loss: 6.89865e-05 | Time: 7281.19 ms [2021-12-20 19:41:34 main:574] : INFO : Epoch 1889 | loss: 6.06762e-05 | val_loss: 6.58873e-05 | Time: 7358.63 ms [2021-12-20 19:41:42 main:574] : INFO : Epoch 1890 | loss: 5.72664e-05 | val_loss: 6.37612e-05 | Time: 7448.84 ms [2021-12-20 19:41:50 main:574] : INFO : Epoch 1891 | loss: 5.47695e-05 | val_loss: 6.15791e-05 | Time: 7870.56 ms [2021-12-20 19:41:57 main:574] : INFO : Epoch 1892 | loss: 5.29339e-05 | val_loss: 6.02271e-05 | Time: 7665.79 ms [2021-12-20 19:42:05 main:574] : INFO : Epoch 1893 | loss: 5.18184e-05 | val_loss: 5.7684e-05 | Time: 7876.79 ms [2021-12-20 19:42:13 main:574] : INFO : Epoch 1894 | loss: 5.05244e-05 | val_loss: 5.70091e-05 | Time: 7594.36 ms [2021-12-20 19:42:21 main:574] : INFO : Epoch 1895 | loss: 4.98353e-05 | val_loss: 5.54313e-05 | Time: 7854.11 ms [2021-12-20 19:42:29 main:574] : INFO : Epoch 1896 | loss: 4.89594e-05 | val_loss: 5.50601e-05 | Time: 7960.12 ms [2021-12-20 19:42:36 main:574] : INFO : Epoch 1897 | loss: 4.78333e-05 | val_loss: 5.39252e-05 | Time: 7563.42 ms [2021-12-20 19:42:44 main:574] : INFO : Epoch 1898 | loss: 4.74277e-05 | val_loss: 5.32134e-05 | Time: 7210.53 ms [2021-12-20 19:42:51 main:574] : INFO : Epoch 1899 | loss: 4.70597e-05 | val_loss: 5.27657e-05 | Time: 7015.32 ms [2021-12-20 19:42:58 main:574] : INFO : Epoch 1900 | loss: 4.61277e-05 | val_loss: 5.25081e-05 | Time: 7109.76 ms [2021-12-20 19:43:05 main:574] : INFO : Epoch 1901 | loss: 4.58813e-05 | val_loss: 5.20963e-05 | Time: 7531.28 ms [2021-12-20 19:43:13 main:574] : INFO : Epoch 1902 | loss: 4.53649e-05 | val_loss: 5.1865e-05 | Time: 7047.39 ms [2021-12-20 19:43:20 main:574] : INFO : Epoch 1903 | loss: 4.50575e-05 | val_loss: 5.20481e-05 | Time: 7381.4 ms [2021-12-20 19:43:27 main:574] : INFO : Epoch 1904 | loss: 4.48769e-05 | val_loss: 5.12073e-05 | Time: 7269.2 ms [2021-12-20 19:43:35 main:574] : INFO : Epoch 1905 | loss: 4.48654e-05 | val_loss: 5.15402e-05 | Time: 7386.88 ms [2021-12-20 19:43:42 main:574] : INFO : Epoch 1906 | loss: 4.43315e-05 | val_loss: 5.03001e-05 | Time: 7210.1 ms [2021-12-20 19:43:50 main:574] : INFO : Epoch 1907 | loss: 4.37531e-05 | val_loss: 5.04849e-05 | Time: 7383.97 ms [2021-12-20 19:43:57 main:574] : INFO : Epoch 1908 | loss: 4.36121e-05 | val_loss: 4.9951e-05 | Time: 7021.51 ms [2021-12-20 19:44:04 main:574] : INFO : Epoch 1909 | loss: 4.44569e-05 | val_loss: 5.51594e-05 | Time: 7416.72 ms [2021-12-20 19:44:11 main:574] : INFO : Epoch 1910 | loss: 4.45702e-05 | val_loss: 5.03456e-05 | Time: 7155.7 ms [2021-12-20 19:44:19 main:574] : INFO : Epoch 1911 | loss: 4.33288e-05 | val_loss: 4.95892e-05 | Time: 7247.04 ms [2021-12-20 19:44:26 main:574] : INFO : Epoch 1912 | loss: 4.30917e-05 | val_loss: 4.99793e-05 | Time: 7303.01 ms [2021-12-20 19:44:34 main:574] : INFO : Epoch 1913 | loss: 4.254e-05 | val_loss: 4.93875e-05 | Time: 7597.14 ms [2021-12-20 19:44:41 main:574] : INFO : Epoch 1914 | loss: 4.26012e-05 | val_loss: 4.88779e-05 | Time: 7161.68 ms [2021-12-20 19:44:48 main:574] : INFO : Epoch 1915 | loss: 4.23672e-05 | val_loss: 4.84901e-05 | Time: 7001.89 ms [2021-12-20 19:44:55 main:574] : INFO : Epoch 1916 | loss: 4.23522e-05 | val_loss: 5.00685e-05 | Time: 7123.24 ms [2021-12-20 19:45:02 main:574] : INFO : Epoch 1917 | loss: 4.38859e-05 | val_loss: 5.36828e-05 | Time: 7131.31 ms [2021-12-20 19:45:09 main:574] : INFO : Epoch 1918 | loss: 4.40635e-05 | val_loss: 4.92504e-05 | Time: 7155 ms [2021-12-20 19:45:17 main:574] : INFO : Epoch 1919 | loss: 4.24972e-05 | val_loss: 4.90743e-05 | Time: 7029.92 ms [2021-12-20 19:45:24 main:574] : INFO : Epoch 1920 | loss: 4.2237e-05 | val_loss: 5.18331e-05 | Time: 6961.91 ms [2021-12-20 19:45:31 main:574] : INFO : Epoch 1921 | loss: 4.24262e-05 | val_loss: 4.89328e-05 | Time: 7274.57 ms [2021-12-20 19:45:38 main:574] : INFO : Epoch 1922 | loss: 4.16347e-05 | val_loss: 4.82871e-05 | Time: 6908.77 ms [2021-12-20 19:45:45 main:574] : INFO : Epoch 1923 | loss: 4.1658e-05 | val_loss: 4.91245e-05 | Time: 7405.49 ms [2021-12-20 19:45:52 main:574] : INFO : Epoch 1924 | loss: 4.16618e-05 | val_loss: 4.85163e-05 | Time: 7148.06 ms [2021-12-20 19:46:00 main:574] : INFO : Epoch 1925 | loss: 4.14023e-05 | val_loss: 4.92524e-05 | Time: 7417.24 ms [2021-12-20 19:46:07 main:574] : INFO : Epoch 1926 | loss: 4.15842e-05 | val_loss: 4.78808e-05 | Time: 7418.37 ms [2021-12-20 19:46:15 main:574] : INFO : Epoch 1927 | loss: 4.1505e-05 | val_loss: 4.98137e-05 | Time: 7227.99 ms [2021-12-20 19:46:22 main:574] : INFO : Epoch 1928 | loss: 4.54326e-05 | val_loss: 5.2999e-05 | Time: 6816.85 ms [2021-12-20 19:46:29 main:574] : INFO : Epoch 1929 | loss: 4.27798e-05 | val_loss: 5.03591e-05 | Time: 7178.37 ms [2021-12-20 19:46:36 main:574] : INFO : Epoch 1930 | loss: 4.14071e-05 | val_loss: 4.85018e-05 | Time: 6987.2 ms [2021-12-20 19:46:43 main:574] : INFO : Epoch 1931 | loss: 4.13411e-05 | val_loss: 4.91126e-05 | Time: 6914.36 ms [2021-12-20 19:46:50 main:574] : INFO : Epoch 1932 | loss: 4.11588e-05 | val_loss: 4.81349e-05 | Time: 7297.34 ms [2021-12-20 19:46:57 main:574] : INFO : Epoch 1933 | loss: 4.11795e-05 | val_loss: 4.97572e-05 | Time: 7086.62 ms [2021-12-20 19:47:04 main:574] : INFO : Epoch 1934 | loss: 4.09691e-05 | val_loss: 4.88179e-05 | Time: 7241.33 ms [2021-12-20 19:47:12 main:574] : INFO : Epoch 1935 | loss: 4.03898e-05 | val_loss: 4.7728e-05 | Time: 7196.16 ms [2021-12-20 19:47:19 main:574] : INFO : Epoch 1936 | loss: 4.04481e-05 | val_loss: 4.80269e-05 | Time: 7175.71 ms [2021-12-20 19:47:26 main:574] : INFO : Epoch 1937 | loss: 4.01944e-05 | val_loss: 4.79411e-05 | Time: 7200.92 ms [2021-12-20 19:47:33 main:574] : INFO : Epoch 1938 | loss: 4.02095e-05 | val_loss: 4.8654e-05 | Time: 7010.15 ms [2021-12-20 19:47:41 main:574] : INFO : Epoch 1939 | loss: 4.12683e-05 | val_loss: 5.23507e-05 | Time: 7466.49 ms [2021-12-20 19:47:48 main:574] : INFO : Epoch 1940 | loss: 4.18544e-05 | val_loss: 4.82981e-05 | Time: 7200.44 ms [2021-12-20 19:47:55 main:574] : INFO : Epoch 1941 | loss: 4.31857e-05 | val_loss: 4.93082e-05 | Time: 7188.75 ms [2021-12-20 19:48:02 main:574] : INFO : Epoch 1942 | loss: 4.09263e-05 | val_loss: 4.84568e-05 | Time: 7137.22 ms [2021-12-20 19:48:10 main:574] : INFO : Epoch 1943 | loss: 4.05773e-05 | val_loss: 5.15637e-05 | Time: 7240.59 ms [2021-12-20 19:48:17 main:574] : INFO : Epoch 1944 | loss: 4.09301e-05 | val_loss: 4.76616e-05 | Time: 6939.89 ms [2021-12-20 19:48:24 main:574] : INFO : Epoch 1945 | loss: 3.94023e-05 | val_loss: 4.8227e-05 | Time: 7312.49 ms [2021-12-20 19:48:32 main:574] : INFO : Epoch 1946 | loss: 3.90599e-05 | val_loss: 4.73311e-05 | Time: 7429.93 ms [2021-12-20 19:48:39 main:574] : INFO : Epoch 1947 | loss: 3.99287e-05 | val_loss: 4.69361e-05 | Time: 7445.8 ms [2021-12-20 19:48:46 main:574] : INFO : Epoch 1948 | loss: 4.36497e-05 | val_loss: 5.48211e-05 | Time: 7213.09 ms [2021-12-20 19:48:54 main:574] : INFO : Epoch 1949 | loss: 4.41079e-05 | val_loss: 4.77012e-05 | Time: 7315.63 ms [2021-12-20 19:49:01 main:574] : INFO : Epoch 1950 | loss: 4.18295e-05 | val_loss: 4.68405e-05 | Time: 7212.04 ms [2021-12-20 19:49:08 main:574] : INFO : Epoch 1951 | loss: 4.14782e-05 | val_loss: 4.84308e-05 | Time: 7389.61 ms [2021-12-20 19:49:16 main:574] : INFO : Epoch 1952 | loss: 3.90579e-05 | val_loss: 4.71159e-05 | Time: 7331.52 ms [2021-12-20 19:49:23 main:574] : INFO : Epoch 1953 | loss: 3.80176e-05 | val_loss: 4.60318e-05 | Time: 7323.81 ms [2021-12-20 19:49:31 main:574] : INFO : Epoch 1954 | loss: 3.82463e-05 | val_loss: 4.56237e-05 | Time: 7497.51 ms [2021-12-20 19:49:38 main:574] : INFO : Epoch 1955 | loss: 3.83077e-05 | val_loss: 4.71069e-05 | Time: 7455.23 ms [2021-12-20 19:49:46 main:574] : INFO : Epoch 1956 | loss: 3.84084e-05 | val_loss: 4.46952e-05 | Time: 7509.68 ms [2021-12-20 19:49:53 main:574] : INFO : Epoch 1957 | loss: 3.84998e-05 | val_loss: 4.59551e-05 | Time: 7204.27 ms [2021-12-20 19:50:00 main:574] : INFO : Epoch 1958 | loss: 3.76828e-05 | val_loss: 4.33153e-05 | Time: 7565.67 ms [2021-12-20 19:50:08 main:574] : INFO : Epoch 1959 | loss: 3.76877e-05 | val_loss: 4.40785e-05 | Time: 7454.29 ms [2021-12-20 19:50:15 main:574] : INFO : Epoch 1960 | loss: 4.05509e-05 | val_loss: 4.47598e-05 | Time: 7212.45 ms [2021-12-20 19:50:22 main:574] : INFO : Epoch 1961 | loss: 4.17776e-05 | val_loss: 4.64969e-05 | Time: 6895.84 ms [2021-12-20 19:50:30 main:574] : INFO : Epoch 1962 | loss: 3.82565e-05 | val_loss: 4.37786e-05 | Time: 7345.37 ms [2021-12-20 19:50:37 main:574] : INFO : Epoch 1963 | loss: 3.76431e-05 | val_loss: 4.3395e-05 | Time: 7228.05 ms [2021-12-20 19:50:44 main:574] : INFO : Epoch 1964 | loss: 3.58661e-05 | val_loss: 4.17294e-05 | Time: 7217.82 ms [2021-12-20 19:50:51 main:574] : INFO : Epoch 1965 | loss: 3.61952e-05 | val_loss: 4.12151e-05 | Time: 7311.57 ms [2021-12-20 19:50:58 main:574] : INFO : Epoch 1966 | loss: 4.80769e-05 | val_loss: 5.1188e-05 | Time: 7010.12 ms [2021-12-20 19:51:06 main:574] : INFO : Epoch 1967 | loss: 0.00120093 | val_loss: 0.00633324 | Time: 7311.18 ms [2021-12-20 19:51:13 main:574] : INFO : Epoch 1968 | loss: 0.00480884 | val_loss: 0.00218813 | Time: 7411.49 ms [2021-12-20 19:51:21 main:574] : INFO : Epoch 1969 | loss: 0.000955653 | val_loss: 0.0003402 | Time: 7683.47 ms [2021-12-20 19:51:29 main:574] : INFO : Epoch 1970 | loss: 0.000231183 | val_loss: 0.000162647 | Time: 7361.35 ms [2021-12-20 19:51:36 main:574] : INFO : Epoch 1971 | loss: 0.000133893 | val_loss: 0.000113588 | Time: 7419.86 ms [2021-12-20 19:51:43 main:574] : INFO : Epoch 1972 | loss: 0.00010181 | val_loss: 9.65343e-05 | Time: 7393.68 ms [2021-12-20 19:51:51 main:574] : INFO : Epoch 1973 | loss: 9.04133e-05 | val_loss: 8.83433e-05 | Time: 7313.68 ms [2021-12-20 19:51:58 main:574] : INFO : Epoch 1974 | loss: 8.41072e-05 | val_loss: 8.555e-05 | Time: 7224.12 ms [2021-12-20 19:52:05 main:574] : INFO : Epoch 1975 | loss: 8.03536e-05 | val_loss: 8.07396e-05 | Time: 7144.83 ms [2021-12-20 19:52:13 main:574] : INFO : Epoch 1976 | loss: 7.67384e-05 | val_loss: 7.76763e-05 | Time: 7046.17 ms [2021-12-20 19:52:20 main:574] : INFO : Epoch 1977 | loss: 7.39277e-05 | val_loss: 7.49289e-05 | Time: 7108.82 ms [2021-12-20 19:52:27 main:574] : INFO : Epoch 1978 | loss: 7.18396e-05 | val_loss: 7.30291e-05 | Time: 7460.52 ms [2021-12-20 19:52:35 main:574] : INFO : Epoch 1979 | loss: 7.01692e-05 | val_loss: 7.06924e-05 | Time: 8197.97 ms [2021-12-20 19:52:44 main:574] : INFO : Epoch 1980 | loss: 6.89619e-05 | val_loss: 7.06209e-05 | Time: 8285.53 ms [2021-12-20 19:52:51 main:574] : INFO : Epoch 1981 | loss: 6.78573e-05 | val_loss: 6.88906e-05 | Time: 7431.25 ms [2021-12-20 19:52:59 main:574] : INFO : Epoch 1982 | loss: 6.69789e-05 | val_loss: 6.80057e-05 | Time: 7263 ms [2021-12-20 19:53:06 main:574] : INFO : Epoch 1983 | loss: 6.61208e-05 | val_loss: 6.68756e-05 | Time: 7480.63 ms [2021-12-20 19:53:13 main:574] : INFO : Epoch 1984 | loss: 6.526e-05 | val_loss: 6.58489e-05 | Time: 7246.91 ms [2021-12-20 19:53:21 main:574] : INFO : Epoch 1985 | loss: 6.50698e-05 | val_loss: 6.59206e-05 | Time: 7187.16 ms [2021-12-20 19:53:28 main:574] : INFO : Epoch 1986 | loss: 6.46794e-05 | val_loss: 6.5528e-05 | Time: 7099.92 ms [2021-12-20 19:53:35 main:574] : INFO : Epoch 1987 | loss: 6.41928e-05 | val_loss: 6.44584e-05 | Time: 7298.24 ms [2021-12-20 19:53:42 main:574] : INFO : Epoch 1988 | loss: 6.35635e-05 | val_loss: 6.38854e-05 | Time: 6998.87 ms [2021-12-20 19:53:49 main:574] : INFO : Epoch 1989 | loss: 6.33022e-05 | val_loss: 6.38873e-05 | Time: 7241.3 ms [2021-12-20 19:53:57 main:574] : INFO : Epoch 1990 | loss: 6.26986e-05 | val_loss: 6.24353e-05 | Time: 7285.43 ms [2021-12-20 19:54:04 main:574] : INFO : Epoch 1991 | loss: 6.18948e-05 | val_loss: 6.13783e-05 | Time: 7524.82 ms [2021-12-20 19:54:11 main:574] : INFO : Epoch 1992 | loss: 6.15421e-05 | val_loss: 6.1237e-05 | Time: 7011.54 ms [2021-12-20 19:54:19 main:574] : INFO : Epoch 1993 | loss: 6.16163e-05 | val_loss: 6.22692e-05 | Time: 7322.23 ms [2021-12-20 19:54:26 main:574] : INFO : Epoch 1994 | loss: 6.12005e-05 | val_loss: 6.09989e-05 | Time: 7107.91 ms [2021-12-20 19:54:33 main:574] : INFO : Epoch 1995 | loss: 6.07931e-05 | val_loss: 6.1411e-05 | Time: 7120.13 ms [2021-12-20 19:54:41 main:574] : INFO : Epoch 1996 | loss: 6.05961e-05 | val_loss: 5.96325e-05 | Time: 7388.86 ms [2021-12-20 19:54:48 main:574] : INFO : Epoch 1997 | loss: 5.99634e-05 | val_loss: 5.99213e-05 | Time: 7004.54 ms [2021-12-20 19:54:55 main:574] : INFO : Epoch 1998 | loss: 5.98736e-05 | val_loss: 5.94863e-05 | Time: 7122.8 ms [2021-12-20 19:55:02 main:574] : INFO : Epoch 1999 | loss: 5.94888e-05 | val_loss: 5.86166e-05 | Time: 7463.35 ms [2021-12-20 19:55:08 main:574] : INFO : Epoch 2000 | loss: 5.92995e-05 | val_loss: 5.89177e-05 | Time: 6025.32 ms [2021-12-20 19:55:12 main:574] : INFO : Epoch 2001 | loss: 5.90974e-05 | val_loss: 5.84131e-05 | Time: 3266.14 ms [2021-12-20 19:55:15 main:574] : INFO : Epoch 2002 | loss: 5.8622e-05 | val_loss: 5.82767e-05 | Time: 3128.11 ms [2021-12-20 19:55:18 main:574] : INFO : Epoch 2003 | loss: 5.88608e-05 | val_loss: 5.77712e-05 | Time: 3295.39 ms [2021-12-20 19:55:21 main:574] : INFO : Epoch 2004 | loss: 5.80169e-05 | val_loss: 5.71947e-05 | Time: 2986.52 ms [2021-12-20 19:55:24 main:574] : INFO : Epoch 2005 | loss: 5.78136e-05 | val_loss: 5.72051e-05 | Time: 3118.53 ms [2021-12-20 19:55:27 main:574] : INFO : Epoch 2006 | loss: 5.76667e-05 | val_loss: 5.70388e-05 | Time: 3079.2 ms [2021-12-20 19:55:31 main:574] : INFO : Epoch 2007 | loss: 5.75571e-05 | val_loss: 5.71509e-05 | Time: 3002.88 ms [2021-12-20 19:55:34 main:574] : INFO : Epoch 2008 | loss: 5.70629e-05 | val_loss: 5.70455e-05 | Time: 3050.16 ms [2021-12-20 19:55:37 main:574] : INFO : Epoch 2009 | loss: 5.7255e-05 | val_loss: 5.60287e-05 | Time: 2982.78 ms [2021-12-20 19:55:40 main:574] : INFO : Epoch 2010 | loss: 5.66402e-05 | val_loss: 5.67396e-05 | Time: 3030.13 ms [2021-12-20 19:55:43 main:574] : INFO : Epoch 2011 | loss: 5.65572e-05 | val_loss: 5.64084e-05 | Time: 2951.13 ms [2021-12-20 19:55:46 main:574] : INFO : Epoch 2012 | loss: 5.64205e-05 | val_loss: 5.63892e-05 | Time: 3044.94 ms [2021-12-20 19:55:49 main:574] : INFO : Epoch 2013 | loss: 5.60267e-05 | val_loss: 5.67335e-05 | Time: 3081.93 ms [2021-12-20 19:55:52 main:574] : INFO : Epoch 2014 | loss: 5.62571e-05 | val_loss: 5.51391e-05 | Time: 3015.04 ms [2021-12-20 19:55:56 main:574] : INFO : Epoch 2015 | loss: 5.57535e-05 | val_loss: 5.4897e-05 | Time: 3127.98 ms [2021-12-20 19:55:59 main:574] : INFO : Epoch 2016 | loss: 5.55105e-05 | val_loss: 5.45899e-05 | Time: 2917.04 ms [2021-12-20 19:56:02 main:574] : INFO : Epoch 2017 | loss: 5.53527e-05 | val_loss: 5.48852e-05 | Time: 3111.06 ms [2021-12-20 19:56:05 main:574] : INFO : Epoch 2018 | loss: 5.5254e-05 | val_loss: 5.56229e-05 | Time: 3143.05 ms [2021-12-20 19:56:09 main:574] : INFO : Epoch 2019 | loss: 5.53835e-05 | val_loss: 5.53368e-05 | Time: 3296.37 ms [2021-12-20 19:56:12 main:574] : INFO : Epoch 2020 | loss: 5.50508e-05 | val_loss: 5.48712e-05 | Time: 3145.55 ms [2021-12-20 19:56:15 main:574] : INFO : Epoch 2021 | loss: 5.51034e-05 | val_loss: 5.49034e-05 | Time: 3213.52 ms [2021-12-20 19:56:22 main:574] : INFO : Epoch 2022 | loss: 5.47648e-05 | val_loss: 5.45203e-05 | Time: 6710.21 ms [2021-12-20 19:56:30 main:574] : INFO : Epoch 2023 | loss: 5.44674e-05 | val_loss: 5.39896e-05 | Time: 7359.76 ms [2021-12-20 19:56:37 main:574] : INFO : Epoch 2024 | loss: 5.43701e-05 | val_loss: 5.40611e-05 | Time: 7355.51 ms [2021-12-20 19:56:44 main:574] : INFO : Epoch 2025 | loss: 5.43645e-05 | val_loss: 5.39765e-05 | Time: 6926.26 ms [2021-12-20 19:56:51 main:574] : INFO : Epoch 2026 | loss: 5.43317e-05 | val_loss: 5.36273e-05 | Time: 7131.22 ms [2021-12-20 19:56:58 main:574] : INFO : Epoch 2027 | loss: 5.38056e-05 | val_loss: 5.35963e-05 | Time: 7013.6 ms [2021-12-20 19:57:05 main:574] : INFO : Epoch 2028 | loss: 5.36603e-05 | val_loss: 5.35664e-05 | Time: 7036.75 ms [2021-12-20 19:57:13 main:574] : INFO : Epoch 2029 | loss: 5.34448e-05 | val_loss: 5.39608e-05 | Time: 7142.19 ms [2021-12-20 19:57:19 main:574] : INFO : Epoch 2030 | loss: 5.35561e-05 | val_loss: 5.2912e-05 | Time: 6784.76 ms [2021-12-20 19:57:27 main:574] : INFO : Epoch 2031 | loss: 5.34255e-05 | val_loss: 5.33631e-05 | Time: 6959.99 ms [2021-12-20 19:57:34 main:574] : INFO : Epoch 2032 | loss: 5.38315e-05 | val_loss: 5.29631e-05 | Time: 7021.07 ms [2021-12-20 19:57:41 main:574] : INFO : Epoch 2033 | loss: 5.36083e-05 | val_loss: 5.31963e-05 | Time: 7307.57 ms [2021-12-20 19:57:48 main:574] : INFO : Epoch 2034 | loss: 5.33379e-05 | val_loss: 5.28851e-05 | Time: 6843.94 ms [2021-12-20 19:57:55 main:574] : INFO : Epoch 2035 | loss: 5.30847e-05 | val_loss: 5.27336e-05 | Time: 7160.56 ms [2021-12-20 19:58:02 main:574] : INFO : Epoch 2036 | loss: 5.28849e-05 | val_loss: 5.36377e-05 | Time: 7387.33 ms [2021-12-20 19:58:10 main:574] : INFO : Epoch 2037 | loss: 5.43453e-05 | val_loss: 5.33856e-05 | Time: 7163.05 ms [2021-12-20 19:58:16 main:574] : INFO : Epoch 2038 | loss: 5.34927e-05 | val_loss: 5.25156e-05 | Time: 6568.16 ms [2021-12-20 19:58:23 main:574] : INFO : Epoch 2039 | loss: 5.26753e-05 | val_loss: 5.25712e-05 | Time: 6977.12 ms [2021-12-20 19:58:31 main:574] : INFO : Epoch 2040 | loss: 5.26237e-05 | val_loss: 5.24395e-05 | Time: 7160.08 ms [2021-12-20 19:58:38 main:574] : INFO : Epoch 2041 | loss: 5.26847e-05 | val_loss: 5.19498e-05 | Time: 7085.37 ms [2021-12-20 19:58:45 main:574] : INFO : Epoch 2042 | loss: 5.25115e-05 | val_loss: 5.18232e-05 | Time: 7011.72 ms [2021-12-20 19:58:52 main:574] : INFO : Epoch 2043 | loss: 5.36272e-05 | val_loss: 5.43331e-05 | Time: 7213.66 ms [2021-12-20 19:58:59 main:574] : INFO : Epoch 2044 | loss: 5.39852e-05 | val_loss: 5.19002e-05 | Time: 7260.56 ms [2021-12-20 19:59:07 main:574] : INFO : Epoch 2045 | loss: 5.2857e-05 | val_loss: 5.14399e-05 | Time: 7265.79 ms [2021-12-20 19:59:14 main:574] : INFO : Epoch 2046 | loss: 5.24575e-05 | val_loss: 5.12698e-05 | Time: 6941.47 ms [2021-12-20 19:59:21 main:574] : INFO : Epoch 2047 | loss: 5.21945e-05 | val_loss: 5.12694e-05 | Time: 6938.91 ms [2021-12-20 19:59:28 main:574] : INFO : Epoch 2048 | loss: 5.18942e-05 | val_loss: 5.17471e-05 | Time: 6924.64 ms [2021-12-20 19:59:28 main:597] : INFO : Saving trained model to model-final.pt, val_loss 5.17471e-05 [2021-12-20 19:59:28 main:603] : INFO : Saving end state to config to file [2021-12-20 19:59:28 main:608] : INFO : Success, exiting.. 19:59:28 (7580): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)