Task 14622566
| Name | ParityModified-1647046272-24354-3-0_1 |
| Workunit | 11516871 |
| Created | 17 Apr 2022, 5:53:43 UTC |
| Sent | 17 Apr 2022, 5:54:26 UTC |
| Report deadline | 25 Apr 2022, 5:54:26 UTC |
| Received | 17 Apr 2022, 13:53:54 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 2381 |
| Run time | 3 hours 30 min 8 sec |
| CPU time | 2 hours 39 min 37 sec |
| Validate state | Valid |
| Credit | 4,160.00 |
| Device peak FLOPS | 11,156.59 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 2.08 GB |
| Peak swap size | 4.38 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version>
<![CDATA[
<stderr_txt>
main:514] : INFO : Loading state
[2022-04-17 08:12:54 main:559] : INFO : Loading DataLoader into Memory
[2022-04-17 08:12:54 main:562] : INFO : Starting Training
[2022-04-17 08:12:58 main:574] : INFO : Epoch 1726 | loss: 0.0194198 | val_loss: 0.0187088 | Time: 4358.16 ms
[2022-04-17 08:13:02 main:574] : INFO : Epoch 1727 | loss: 0.0184215 | val_loss: 0.0182609 | Time: 3734.67 ms
[2022-04-17 08:13:06 main:574] : INFO : Epoch 1728 | loss: 0.0182591 | val_loss: 0.0182248 | Time: 3849.33 ms
[2022-04-17 08:13:10 main:574] : INFO : Epoch 1729 | loss: 0.0181468 | val_loss: 0.0180748 | Time: 3881.31 ms
[2022-04-17 08:13:14 main:574] : INFO : Epoch 1730 | loss: 0.0180787 | val_loss: 0.0180587 | Time: 4105.23 ms
[2022-04-17 08:13:18 main:574] : INFO : Epoch 1731 | loss: 0.018069 | val_loss: 0.0180534 | Time: 3977.39 ms
[2022-04-17 08:13:22 main:574] : INFO : Epoch 1732 | loss: 0.0180635 | val_loss: 0.0180515 | Time: 4228.16 ms
[2022-04-17 08:13:26 main:574] : INFO : Epoch 1733 | loss: 0.0180606 | val_loss: 0.0180463 | Time: 4047.75 ms
[2022-04-17 08:13:30 main:574] : INFO : Epoch 1734 | loss: 0.0180576 | val_loss: 0.018052 | Time: 4102.58 ms
[2022-04-17 08:13:34 main:574] : INFO : Epoch 1735 | loss: 0.0180584 | val_loss: 0.018047 | Time: 3903.19 ms
[2022-04-17 08:13:37 main:574] : INFO : Epoch 1736 | loss: 0.0180565 | val_loss: 0.0180442 | Time: 3249.08 ms
[2022-04-17 08:13:42 main:574] : INFO : Epoch 1737 | loss: 0.0180564 | val_loss: 0.0180438 | Time: 4248.81 ms
[2022-04-17 08:13:46 main:574] : INFO : Epoch 1738 | loss: 0.0180526 | val_loss: 0.0180394 | Time: 4020.76 ms
[2022-04-17 08:13:50 main:574] : INFO : Epoch 1739 | loss: 0.018049 | val_loss: 0.0180337 | Time: 3970.72 ms
[2022-04-17 08:13:54 main:574] : INFO : Epoch 1740 | loss: 0.0180446 | val_loss: 0.0180311 | Time: 4037.83 ms
[2022-04-17 08:13:58 main:574] : INFO : Epoch 1741 | loss: 0.0180382 | val_loss: 0.0180234 | Time: 3890.02 ms
[2022-04-17 08:14:02 main:574] : INFO : Epoch 1742 | loss: 0.0180346 | val_loss: 0.0180298 | Time: 4066.57 ms
[2022-04-17 08:14:05 main:574] : INFO : Epoch 1743 | loss: 0.0180332 | val_loss: 0.0180233 | Time: 3371.04 ms
[2022-04-17 08:14:09 main:574] : INFO : Epoch 1744 | loss: 0.0180297 | val_loss: 0.0180153 | Time: 4102.46 ms
[2022-04-17 08:14:13 main:574] : INFO : Epoch 1745 | loss: 0.0180196 | val_loss: 0.018011 | Time: 3990.75 ms
[2022-04-17 08:14:17 main:574] : INFO : Epoch 1746 | loss: 0.0180081 | val_loss: 0.0179716 | Time: 3994.7 ms
[2022-04-17 08:14:21 main:574] : INFO : Epoch 1747 | loss: 0.0179736 | val_loss: 0.0179516 | Time: 3888.85 ms
[2022-04-17 08:14:25 main:574] : INFO : Epoch 1748 | loss: 0.0179571 | val_loss: 0.0179289 | Time: 3817.64 ms
[2022-04-17 08:14:29 main:574] : INFO : Epoch 1749 | loss: 0.017931 | val_loss: 0.0178693 | Time: 3883.33 ms
[2022-04-17 08:14:32 main:574] : INFO : Epoch 1750 | loss: 0.0184493 | val_loss: 0.0182807 | Time: 3363.83 ms
[2022-04-17 08:14:36 main:574] : INFO : Epoch 1751 | loss: 0.0181196 | val_loss: 0.017841 | Time: 4043.66 ms
[2022-04-17 08:14:40 main:574] : INFO : Epoch 1752 | loss: 0.0177335 | val_loss: 0.0176077 | Time: 4054.22 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 2080 SUPER)
[2022-04-17 08:17:42 main:435] : INFO : Set logging level to 1
[2022-04-17 08:17:42 main:441] : INFO : Running in BOINC Client mode
[2022-04-17 08:17:42 main:444] : INFO : Resolving all filenames
[2022-04-17 08:17:42 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-17 08:17:42 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1)
[2022-04-17 08:17:42 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-17 08:17:42 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-17 08:17:42 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-04-17 08:17:42 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-04-17 08:17:42 main:474] : INFO : Configuration:
[2022-04-17 08:17:42 main:475] : INFO : Model type: GRU
[2022-04-17 08:17:42 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-04-17 08:17:42 main:477] : INFO : Max Epochs: 2048
[2022-04-17 08:17:42 main:478] : INFO : Batch Size: 128
[2022-04-17 08:17:42 main:479] : INFO : Learning Rate: 0.01
[2022-04-17 08:17:42 main:480] : INFO : Patience: 10
[2022-04-17 08:17:42 main:481] : INFO : Hidden Width: 12
[2022-04-17 08:17:42 main:482] : INFO : # Recurrent Layers: 4
[2022-04-17 08:17:42 main:483] : INFO : # Backend Layers: 4
[2022-04-17 08:17:42 main:484] : INFO : # Threads: 1
[2022-04-17 08:17:42 main:486] : INFO : Preparing Dataset
[2022-04-17 08:17:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-17 08:17:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-17 08:17:44 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-04-17 08:17:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-17 08:17:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-17 08:17:44 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-04-17 08:17:44 main:494] : INFO : Creating Model
[2022-04-17 08:17:44 main:507] : INFO : Preparing config file
[2022-04-17 08:17:44 main:511] : INFO : Found checkpoint, attempting to load...
[2022-04-17 08:17:44 main:512] : INFO : Loading config
[2022-04-17 08:17:44 main:514] : INFO : Loading state
[2022-04-17 08:17:45 main:559] : INFO : Loading DataLoader into Memory
[2022-04-17 08:17:45 main:562] : INFO : Starting Training
[2022-04-17 08:17:49 main:574] : INFO : Epoch 1741 | loss: 0.0188058 | val_loss: 0.0182173 | Time: 4136.59 ms
[2022-04-17 08:17:53 main:574] : INFO : Epoch 1742 | loss: 0.0181293 | val_loss: 0.0180804 | Time: 4178.26 ms
[2022-04-17 08:17:57 main:574] : INFO : Epoch 1743 | loss: 0.0180562 | val_loss: 0.0180301 | Time: 4081.55 ms
[2022-04-17 08:18:01 main:574] : INFO : Epoch 1744 | loss: 0.0180361 | val_loss: 0.0180215 | Time: 4121.88 ms
[2022-04-17 08:18:05 main:574] : INFO : Epoch 1745 | loss: 0.0180281 | val_loss: 0.0180147 | Time: 4075.47 ms
[2022-04-17 08:18:09 main:574] : INFO : Epoch 1746 | loss: 0.018023 | val_loss: 0.0180107 | Time: 3852.3 ms
[2022-04-17 08:18:12 main:574] : INFO : Epoch 1747 | loss: 0.0180183 | val_loss: 0.0180068 | Time: 3194.06 ms
[2022-04-17 08:18:16 main:574] : INFO : Epoch 1748 | loss: 0.0180149 | val_loss: 0.0179973 | Time: 4018.95 ms
[2022-04-17 08:18:21 main:574] : INFO : Epoch 1749 | loss: 0.0179754 | val_loss: 0.0179534 | Time: 4068.52 ms
[2022-04-17 08:18:25 main:574] : INFO : Epoch 1750 | loss: 0.0179567 | val_loss: 0.0179387 | Time: 4089.04 ms
[2022-04-17 08:18:29 main:574] : INFO : Epoch 1751 | loss: 0.0179308 | val_loss: 0.0178837 | Time: 4154.45 ms
[2022-04-17 08:18:33 main:574] : INFO : Epoch 1752 | loss: 0.0178696 | val_loss: 0.0178317 | Time: 3816.49 ms
[2022-04-17 08:18:36 main:574] : INFO : Epoch 1753 | loss: 0.0178002 | val_loss: 0.0177215 | Time: 3803.3 ms
[2022-04-17 08:18:40 main:574] : INFO : Epoch 1754 | loss: 0.0177378 | val_loss: 0.0177495 | Time: 3429.59 ms
[2022-04-17 08:18:44 main:574] : INFO : Epoch 1755 | loss: 0.0177759 | val_loss: 0.0177306 | Time: 4107.47 ms
[2022-04-17 08:18:48 main:574] : INFO : Epoch 1756 | loss: 0.0175933 | val_loss: 0.0174877 | Time: 3903.77 ms
[2022-04-17 08:18:52 main:574] : INFO : Epoch 1757 | loss: 0.0173842 | val_loss: 0.0171754 | Time: 4098.91 ms
[2022-04-17 08:18:56 main:574] : INFO : Epoch 1758 | loss: 0.0173135 | val_loss: 0.0168348 | Time: 3920.52 ms
[2022-04-17 08:19:00 main:574] : INFO : Epoch 1759 | loss: 0.0164639 | val_loss: 0.015221 | Time: 3938.84 ms
[2022-04-17 08:19:04 main:574] : INFO : Epoch 1760 | loss: 0.0137297 | val_loss: 0.0133284 | Time: 3885.58 ms
[2022-04-17 08:19:08 main:574] : INFO : Epoch 1761 | loss: 0.0134897 | val_loss: 0.0119972 | Time: 3847.52 ms
[2022-04-17 08:19:11 main:574] : INFO : Epoch 1762 | loss: 0.00906443 | val_loss: 0.00429198 | Time: 3860.32 ms
[2022-04-17 08:19:15 main:574] : INFO : Epoch 1763 | loss: 0.00299264 | val_loss: 0.00116582 | Time: 3927.86 ms
[2022-04-17 08:19:19 main:574] : INFO : Epoch 1764 | loss: 0.000530237 | val_loss: 0.000244556 | Time: 3891.43 ms
[2022-04-17 08:19:23 main:574] : INFO : Epoch 1765 | loss: 0.000187583 | val_loss: 0.000142986 | Time: 3839.72 ms
[2022-04-17 08:19:27 main:574] : INFO : Epoch 1766 | loss: 0.000142296 | val_loss: 0.000123413 | Time: 3815.11 ms
[2022-04-17 08:19:31 main:574] : INFO : Epoch 1767 | loss: 0.000128203 | val_loss: 0.000112798 | Time: 3845.46 ms
[2022-04-17 08:19:35 main:574] : INFO : Epoch 1768 | loss: 0.000121821 | val_loss: 0.000108097 | Time: 3862.24 ms
[2022-04-17 08:19:39 main:574] : INFO : Epoch 1769 | loss: 0.000117362 | val_loss: 0.000104935 | Time: 3894.97 ms
[2022-04-17 08:19:42 main:574] : INFO : Epoch 1770 | loss: 0.000118405 | val_loss: 0.000106101 | Time: 3837.57 ms
[2022-04-17 08:19:46 main:574] : INFO : Epoch 1771 | loss: 0.000115527 | val_loss: 0.000101748 | Time: 3891.97 ms
[2022-04-17 08:19:50 main:574] : INFO : Epoch 1772 | loss: 0.000112353 | val_loss: 0.000102273 | Time: 3906.27 ms
[2022-04-17 08:19:54 main:574] : INFO : Epoch 1773 | loss: 0.000109675 | val_loss: 9.78067e-05 | Time: 3869.92 ms
[2022-04-17 08:19:58 main:574] : INFO : Epoch 1774 | loss: 0.000103963 | val_loss: 9.65614e-05 | Time: 3985.93 ms
[2022-04-17 08:20:02 main:574] : INFO : Epoch 1775 | loss: 9.91085e-05 | val_loss: 9.21195e-05 | Time: 4058.73 ms
[2022-04-17 08:20:06 main:574] : INFO : Epoch 1776 | loss: 9.60442e-05 | val_loss: 8.95314e-05 | Time: 3859.24 ms
[2022-04-17 08:20:10 main:574] : INFO : Epoch 1777 | loss: 9.59265e-05 | val_loss: 0.000101397 | Time: 3882.41 ms
[2022-04-17 08:20:14 main:574] : INFO : Epoch 1778 | loss: 0.00010197 | val_loss: 8.69947e-05 | Time: 3914.39 ms
[2022-04-17 08:20:18 main:574] : INFO : Epoch 1779 | loss: 9.34273e-05 | val_loss: 8.61229e-05 | Time: 3845.8 ms
[2022-04-17 08:20:21 main:574] : INFO : Epoch 1780 | loss: 9.10901e-05 | val_loss: 8.48612e-05 | Time: 3839.45 ms
[2022-04-17 08:20:25 main:574] : INFO : Epoch 1781 | loss: 9.02943e-05 | val_loss: 8.61901e-05 | Time: 3822.56 ms
[2022-04-17 08:20:29 main:574] : INFO : Epoch 1782 | loss: 8.94513e-05 | val_loss: 8.69435e-05 | Time: 3885.98 ms
[2022-04-17 08:20:33 main:574] : INFO : Epoch 1783 | loss: 8.93154e-05 | val_loss: 8.50403e-05 | Time: 3820.16 ms
[2022-04-17 08:20:37 main:574] : INFO : Epoch 1784 | loss: 8.84497e-05 | val_loss: 8.93141e-05 | Time: 3812.98 ms
[2022-04-17 08:20:41 main:574] : INFO : Epoch 1785 | loss: 8.70263e-05 | val_loss: 8.15636e-05 | Time: 3868.89 ms
[2022-04-17 08:20:45 main:574] : INFO : Epoch 1786 | loss: 9.15488e-05 | val_loss: 8.2334e-05 | Time: 3882.21 ms
[2022-04-17 08:20:48 main:574] : INFO : Epoch 1787 | loss: 8.721e-05 | val_loss: 8.2671e-05 | Time: 3887.47 ms
[2022-04-17 08:20:53 main:574] : INFO : Epoch 1788 | loss: 8.45765e-05 | val_loss: 8.09651e-05 | Time: 4117.08 ms
[2022-04-17 08:20:56 main:574] : INFO : Epoch 1789 | loss: 8.42118e-05 | val_loss: 8.02812e-05 | Time: 3835.25 ms
[2022-04-17 08:21:00 main:574] : INFO : Epoch 1790 | loss: 8.31563e-05 | val_loss: 7.94537e-05 | Time: 4005.16 ms
[2022-04-17 08:21:04 main:574] : INFO : Epoch 1791 | loss: 8.30558e-05 | val_loss: 8.00641e-05 | Time: 3979.07 ms
[2022-04-17 08:21:08 main:574] : INFO : Epoch 1792 | loss: 8.87565e-05 | val_loss: 8.29643e-05 | Time: 3958.77 ms
[2022-04-17 08:21:12 main:574] : INFO : Epoch 1793 | loss: 8.38518e-05 | val_loss: 7.89577e-05 | Time: 3936.67 ms
[2022-04-17 08:21:16 main:574] : INFO : Epoch 1794 | loss: 8.27431e-05 | val_loss: 7.90426e-05 | Time: 3707.73 ms
[2022-04-17 08:21:20 main:574] : INFO : Epoch 1795 | loss: 8.30712e-05 | val_loss: 7.88647e-05 | Time: 4035.57 ms
[2022-04-17 08:21:24 main:574] : INFO : Epoch 1796 | loss: 8.5126e-05 | val_loss: 8.53152e-05 | Time: 3921.09 ms
[2022-04-17 08:21:28 main:574] : INFO : Epoch 1797 | loss: 8.36515e-05 | val_loss: 7.91337e-05 | Time: 3866.16 ms
[2022-04-17 08:21:32 main:574] : INFO : Epoch 1798 | loss: 8.18735e-05 | val_loss: 7.80813e-05 | Time: 3875.37 ms
[2022-04-17 08:21:36 main:574] : INFO : Epoch 1799 | loss: 8.17362e-05 | val_loss: 7.80293e-05 | Time: 3878.79 ms
[2022-04-17 08:21:40 main:574] : INFO : Epoch 1800 | loss: 8.38938e-05 | val_loss: 9.20119e-05 | Time: 3889.46 ms
[2022-04-17 08:21:44 main:574] : INFO : Epoch 1801 | loss: 8.61583e-05 | val_loss: 7.95262e-05 | Time: 3955.3 ms
[2022-04-17 08:21:47 main:574] : INFO : Epoch 1802 | loss: 8.26318e-05 | val_loss: 8.09805e-05 | Time: 3891.41 ms
[2022-04-17 08:21:51 main:574] : INFO : Epoch 1803 | loss: 8.26155e-05 | val_loss: 7.8372e-05 | Time: 3899.7 ms
[2022-04-17 08:21:55 main:574] : INFO : Epoch 1804 | loss: 8.19575e-05 | val_loss: 7.71457e-05 | Time: 3888.18 ms
[2022-04-17 08:21:59 main:574] : INFO : Epoch 1805 | loss: 8.28152e-05 | val_loss: 8.22982e-05 | Time: 4028.58 ms
[2022-04-17 08:22:03 main:574] : INFO : Epoch 1806 | loss: 8.9011e-05 | val_loss: 9.24952e-05 | Time: 3976.52 ms
[2022-04-17 08:22:07 main:574] : INFO : Epoch 1807 | loss: 8.49184e-05 | val_loss: 7.87032e-05 | Time: 3772.56 ms
[2022-04-17 08:22:11 main:574] : INFO : Epoch 1808 | loss: 8.11999e-05 | val_loss: 7.72141e-05 | Time: 3856.23 ms
[2022-04-17 08:22:15 main:574] : INFO : Epoch 1809 | loss: 8.02909e-05 | val_loss: 7.65129e-05 | Time: 3913.38 ms
[2022-04-17 08:22:19 main:574] : INFO : Epoch 1810 | loss: 7.99314e-05 | val_loss: 7.82828e-05 | Time: 3868.8 ms
[2022-04-17 08:22:23 main:574] : INFO : Epoch 1811 | loss: 8.3014e-05 | val_loss: 7.81399e-05 | Time: 3949.12 ms
[2022-04-17 08:22:27 main:574] : INFO : Epoch 1812 | loss: 8.19093e-05 | val_loss: 7.86702e-05 | Time: 3911.57 ms
[2022-04-17 08:22:30 main:574] : INFO : Epoch 1813 | loss: 8.02267e-05 | val_loss: 7.55861e-05 | Time: 3934.06 ms
[2022-04-17 08:22:34 main:574] : INFO : Epoch 1814 | loss: 7.53308e-05 | val_loss: 5.73719e-05 | Time: 3880.71 ms
[2022-04-17 08:22:38 main:574] : INFO : Epoch 1815 | loss: 4.06049e-05 | val_loss: 3.62183e-05 | Time: 3863.34 ms
[2022-04-17 08:22:42 main:574] : INFO : Epoch 1816 | loss: 3.35378e-05 | val_loss: 3.51179e-05 | Time: 3866.15 ms
[2022-04-17 08:22:46 main:574] : INFO : Epoch 1817 | loss: 3.28003e-05 | val_loss: 3.50441e-05 | Time: 3818.76 ms
[2022-04-17 08:22:50 main:574] : INFO : Epoch 1818 | loss: 3.71078e-05 | val_loss: 5.04835e-05 | Time: 3957.87 ms
[2022-04-17 08:22:54 main:574] : INFO : Epoch 1819 | loss: 3.57694e-05 | val_loss: 3.42412e-05 | Time: 3861.44 ms
[2022-04-17 08:22:58 main:574] : INFO : Epoch 1820 | loss: 3.15839e-05 | val_loss: 3.3368e-05 | Time: 3846.14 ms
[2022-04-17 08:23:02 main:574] : INFO : Epoch 1821 | loss: 3.11973e-05 | val_loss: 3.35787e-05 | Time: 3969.31 ms
[2022-04-17 08:23:05 main:574] : INFO : Epoch 1822 | loss: 3.19057e-05 | val_loss: 3.29106e-05 | Time: 3200.36 ms
[2022-04-17 08:23:08 main:574] : INFO : Epoch 1823 | loss: 3.02357e-05 | val_loss: 3.22031e-05 | Time: 3269.95 ms
[2022-04-17 08:23:12 main:574] : INFO : Epoch 1824 | loss: 2.93376e-05 | val_loss: 3.14704e-05 | Time: 3939.15 ms
[2022-04-17 08:23:16 main:574] : INFO : Epoch 1825 | loss: 2.90443e-05 | val_loss: 3.29929e-05 | Time: 4059.53 ms
[2022-04-17 08:23:20 main:574] : INFO : Epoch 1826 | loss: 4.24263e-05 | val_loss: 4.5273e-05 | Time: 4079.19 ms
[2022-04-17 08:23:24 main:574] : INFO : Epoch 1827 | loss: 3.27635e-05 | val_loss: 3.38997e-05 | Time: 4152.78 ms
[2022-04-17 08:23:28 main:574] : INFO : Epoch 1828 | loss: 2.94203e-05 | val_loss: 3.20968e-05 | Time: 4001.27 ms
[2022-04-17 08:23:32 main:574] : INFO : Epoch 1829 | loss: 2.94132e-05 | val_loss: 3.35563e-05 | Time: 4069.51 ms
[2022-04-17 08:23:36 main:574] : INFO : Epoch 1830 | loss: 2.91606e-05 | val_loss: 3.14396e-05 | Time: 4075.88 ms
[2022-04-17 08:23:41 main:574] : INFO : Epoch 1831 | loss: 2.84293e-05 | val_loss: 3.09901e-05 | Time: 4160.56 ms
[2022-04-17 08:23:45 main:574] : INFO : Epoch 1832 | loss: 2.83628e-05 | val_loss: 3.10624e-05 | Time: 4328.56 ms
[2022-04-17 08:23:49 main:574] : INFO : Epoch 1833 | loss: 2.8228e-05 | val_loss: 3.10604e-05 | Time: 4454.04 ms
[2022-04-17 08:23:54 main:574] : INFO : Epoch 1834 | loss: 2.95231e-05 | val_loss: 3.2947e-05 | Time: 4187.32 ms
[2022-04-17 08:23:58 main:574] : INFO : Epoch 1835 | loss: 3.25133e-05 | val_loss: 3.8866e-05 | Time: 4096.77 ms
[2022-04-17 08:24:02 main:574] : INFO : Epoch 1836 | loss: 3.07304e-05 | val_loss: 3.11214e-05 | Time: 4206.99 ms
[2022-04-17 08:24:06 main:574] : INFO : Epoch 1837 | loss: 2.91915e-05 | val_loss: 3.3379e-05 | Time: 4051.68 ms
[2022-04-17 08:24:10 main:574] : INFO : Epoch 1838 | loss: 2.84263e-05 | val_loss: 3.08009e-05 | Time: 4130.63 ms
[2022-04-17 08:24:14 main:574] : INFO : Epoch 1839 | loss: 2.86955e-05 | val_loss: 3.34899e-05 | Time: 4200.75 ms
[2022-04-17 08:24:18 main:574] : INFO : Epoch 1840 | loss: 3.17427e-05 | val_loss: 3.33507e-05 | Time: 4026.45 ms
[2022-04-17 08:24:23 main:574] : INFO : Epoch 1841 | loss: 2.98575e-05 | val_loss: 3.36662e-05 | Time: 4237.98 ms
[2022-04-17 08:24:27 main:574] : INFO : Epoch 1842 | loss: 2.90341e-05 | val_loss: 3.25099e-05 | Time: 4143.99 ms
[2022-04-17 08:24:31 main:574] : INFO : Epoch 1843 | loss: 3.09565e-05 | val_loss: 3.15826e-05 | Time: 4228.16 ms
[2022-04-17 08:24:35 main:574] : INFO : Epoch 1844 | loss: 2.86667e-05 | val_loss: 3.48057e-05 | Time: 4165.82 ms
[2022-04-17 08:24:39 main:574] : INFO : Epoch 1845 | loss: 3.15223e-05 | val_loss: 3.19982e-05 | Time: 4334.24 ms
[2022-04-17 08:24:44 main:574] : INFO : Epoch 1846 | loss: 2.85572e-05 | val_loss: 3.14201e-05 | Time: 4450.6 ms
[2022-04-17 08:24:48 main:574] : INFO : Epoch 1847 | loss: 2.84288e-05 | val_loss: 3.28813e-05 | Time: 4450.25 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 2080 SUPER)
[2022-04-17 08:27:50 main:435] : INFO : Set logging level to 1
[2022-04-17 08:27:50 main:441] : INFO : Running in BOINC Client mode
[2022-04-17 08:27:50 main:444] : INFO : Resolving all filenames
[2022-04-17 08:27:50 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-17 08:27:50 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1)
[2022-04-17 08:27:50 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-17 08:27:50 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-17 08:27:50 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-04-17 08:27:50 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-04-17 08:27:50 main:474] : INFO : Configuration:
[2022-04-17 08:27:50 main:475] : INFO : Model type: GRU
[2022-04-17 08:27:50 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-04-17 08:27:50 main:477] : INFO : Max Epochs: 2048
[2022-04-17 08:27:50 main:478] : INFO : Batch Size: 128
[2022-04-17 08:27:50 main:479] : INFO : Learning Rate: 0.01
[2022-04-17 08:27:50 main:480] : INFO : Patience: 10
[2022-04-17 08:27:50 main:481] : INFO : Hidden Width: 12
[2022-04-17 08:27:50 main:482] : INFO : # Recurrent Layers: 4
[2022-04-17 08:27:50 main:483] : INFO : # Backend Layers: 4
[2022-04-17 08:27:50 main:484] : INFO : # Threads: 1
[2022-04-17 08:27:50 main:486] : INFO : Preparing Dataset
[2022-04-17 08:27:50 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-17 08:27:50 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-17 08:27:52 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-04-17 08:27:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-17 08:27:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-17 08:27:52 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-04-17 08:27:52 main:494] : INFO : Creating Model
[2022-04-17 08:27:52 main:507] : INFO : Preparing config file
[2022-04-17 08:27:52 main:511] : INFO : Found checkpoint, attempting to load...
[2022-04-17 08:27:52 main:512] : INFO : Loading config
[2022-04-17 08:27:52 main:514] : INFO : Loading state
[2022-04-17 08:27:53 main:559] : INFO : Loading DataLoader into Memory
[2022-04-17 08:27:53 main:562] : INFO : Starting Training
[2022-04-17 08:27:57 main:574] : INFO : Epoch 1836 | loss: 0.00194724 | val_loss: 0.000718137 | Time: 4115.08 ms
[2022-04-17 08:28:01 main:574] : INFO : Epoch 1837 | loss: 0.000293139 | val_loss: 7.82372e-05 | Time: 3996.58 ms
[2022-04-17 08:28:05 main:574] : INFO : Epoch 1838 | loss: 8.03353e-05 | val_loss: 5.06032e-05 | Time: 4115.34 ms
[2022-04-17 08:28:09 main:574] : INFO : Epoch 1839 | loss: 3.99479e-05 | val_loss: 3.66347e-05 | Time: 4045.41 ms
[2022-04-17 08:28:13 main:574] : INFO : Epoch 1840 | loss: 3.18147e-05 | val_loss: 3.2981e-05 | Time: 3997.33 ms
[2022-04-17 08:28:17 main:574] : INFO : Epoch 1841 | loss: 2.98785e-05 | val_loss: 3.22667e-05 | Time: 3827.62 ms
[2022-04-17 08:28:21 main:574] : INFO : Epoch 1842 | loss: 2.93646e-05 | val_loss: 3.1892e-05 | Time: 3833.89 ms
[2022-04-17 08:28:24 main:574] : INFO : Epoch 1843 | loss: 2.90753e-05 | val_loss: 3.16749e-05 | Time: 3890.12 ms
[2022-04-17 08:28:28 main:574] : INFO : Epoch 1844 | loss: 2.88897e-05 | val_loss: 3.15402e-05 | Time: 3888.35 ms
[2022-04-17 08:28:32 main:574] : INFO : Epoch 1845 | loss: 2.87329e-05 | val_loss: 3.13752e-05 | Time: 3875.78 ms
[2022-04-17 08:28:36 main:574] : INFO : Epoch 1846 | loss: 2.86116e-05 | val_loss: 3.12596e-05 | Time: 3819.78 ms
[2022-04-17 08:28:40 main:574] : INFO : Epoch 1847 | loss: 2.85058e-05 | val_loss: 3.11792e-05 | Time: 3824.49 ms
[2022-04-17 08:28:44 main:574] : INFO : Epoch 1848 | loss: 2.84134e-05 | val_loss: 3.10806e-05 | Time: 3869.82 ms
[2022-04-17 08:28:48 main:574] : INFO : Epoch 1849 | loss: 2.83326e-05 | val_loss: 3.10092e-05 | Time: 3905.85 ms
[2022-04-17 08:28:52 main:574] : INFO : Epoch 1850 | loss: 2.82312e-05 | val_loss: 3.09192e-05 | Time: 3916.02 ms
[2022-04-17 08:28:55 main:574] : INFO : Epoch 1851 | loss: 2.81702e-05 | val_loss: 3.08933e-05 | Time: 3878.44 ms
[2022-04-17 08:28:59 main:574] : INFO : Epoch 1852 | loss: 2.81273e-05 | val_loss: 3.07892e-05 | Time: 3861.87 ms
[2022-04-17 08:29:03 main:574] : INFO : Epoch 1853 | loss: 2.81118e-05 | val_loss: 3.08523e-05 | Time: 3943.23 ms
[2022-04-17 08:29:07 main:574] : INFO : Epoch 1854 | loss: 2.80972e-05 | val_loss: 3.06347e-05 | Time: 3749.26 ms
[2022-04-17 08:29:11 main:574] : INFO : Epoch 1855 | loss: 2.80615e-05 | val_loss: 3.06983e-05 | Time: 4085.89 ms
[2022-04-17 08:29:15 main:574] : INFO : Epoch 1856 | loss: 2.79627e-05 | val_loss: 3.05727e-05 | Time: 4052.78 ms
[2022-04-17 08:29:19 main:574] : INFO : Epoch 1857 | loss: 2.79128e-05 | val_loss: 3.04993e-05 | Time: 3950.69 ms
[2022-04-17 08:29:23 main:574] : INFO : Epoch 1858 | loss: 2.78592e-05 | val_loss: 3.04814e-05 | Time: 4012.56 ms
[2022-04-17 08:29:27 main:574] : INFO : Epoch 1859 | loss: 2.78422e-05 | val_loss: 3.04524e-05 | Time: 3831.58 ms
[2022-04-17 08:29:31 main:574] : INFO : Epoch 1860 | loss: 2.77918e-05 | val_loss: 3.04272e-05 | Time: 3746.09 ms
[2022-04-17 08:29:35 main:574] : INFO : Epoch 1861 | loss: 2.77721e-05 | val_loss: 3.04115e-05 | Time: 3908.22 ms
[2022-04-17 08:29:38 main:574] : INFO : Epoch 1862 | loss: 2.77182e-05 | val_loss: 3.03274e-05 | Time: 3605.77 ms
[2022-04-17 08:29:42 main:574] : INFO : Epoch 1863 | loss: 2.76782e-05 | val_loss: 3.0313e-05 | Time: 4165.74 ms
[2022-04-17 08:29:46 main:574] : INFO : Epoch 1864 | loss: 2.76498e-05 | val_loss: 3.03057e-05 | Time: 3858.31 ms
[2022-04-17 08:29:50 main:574] : INFO : Epoch 1865 | loss: 2.76369e-05 | val_loss: 3.02914e-05 | Time: 3749.71 ms
[2022-04-17 08:29:54 main:574] : INFO : Epoch 1866 | loss: 2.76155e-05 | val_loss: 3.0239e-05 | Time: 3902.6 ms
[2022-04-17 08:29:58 main:574] : INFO : Epoch 1867 | loss: 2.7643e-05 | val_loss: 3.025e-05 | Time: 3812.63 ms
[2022-04-17 08:30:02 main:574] : INFO : Epoch 1868 | loss: 2.76197e-05 | val_loss: 3.01593e-05 | Time: 4191.97 ms
[2022-04-17 08:30:06 main:574] : INFO : Epoch 1869 | loss: 2.75961e-05 | val_loss: 3.02865e-05 | Time: 3973.87 ms
[2022-04-17 08:30:10 main:574] : INFO : Epoch 1870 | loss: 2.75269e-05 | val_loss: 3.0103e-05 | Time: 4003.32 ms
[2022-04-17 08:30:14 main:574] : INFO : Epoch 1871 | loss: 2.74869e-05 | val_loss: 3.00827e-05 | Time: 4089.6 ms
[2022-04-17 08:30:18 main:574] : INFO : Epoch 1872 | loss: 2.74617e-05 | val_loss: 3.00062e-05 | Time: 4057.98 ms
[2022-04-17 08:30:22 main:574] : INFO : Epoch 1873 | loss: 2.74328e-05 | val_loss: 2.9968e-05 | Time: 4283.73 ms
[2022-04-17 08:30:26 main:574] : INFO : Epoch 1874 | loss: 2.73988e-05 | val_loss: 2.99234e-05 | Time: 3536.93 ms
[2022-04-17 08:30:30 main:574] : INFO : Epoch 1875 | loss: 2.73233e-05 | val_loss: 2.98589e-05 | Time: 3909.75 ms
[2022-04-17 08:30:34 main:574] : INFO : Epoch 1876 | loss: 2.73045e-05 | val_loss: 2.98283e-05 | Time: 3952.26 ms
[2022-04-17 08:30:38 main:574] : INFO : Epoch 1877 | loss: 2.72452e-05 | val_loss: 2.97158e-05 | Time: 3947.83 ms
[2022-04-17 08:30:42 main:574] : INFO : Epoch 1878 | loss: 2.72139e-05 | val_loss: 2.97607e-05 | Time: 4015.95 ms
[2022-04-17 08:30:46 main:574] : INFO : Epoch 1879 | loss: 2.71985e-05 | val_loss: 2.96994e-05 | Time: 4134.06 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 2080 SUPER)
[2022-04-17 08:34:49 main:435] : INFO : Set logging level to 1
[2022-04-17 08:34:49 main:441] : INFO : Running in BOINC Client mode
[2022-04-17 08:34:49 main:444] : INFO : Resolving all filenames
[2022-04-17 08:34:49 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-17 08:34:49 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1)
[2022-04-17 08:34:49 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-17 08:34:49 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-17 08:34:49 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-04-17 08:34:49 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-04-17 08:34:49 main:474] : INFO : Configuration:
[2022-04-17 08:34:49 main:475] : INFO : Model type: GRU
[2022-04-17 08:34:49 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-04-17 08:34:49 main:477] : INFO : Max Epochs: 2048
[2022-04-17 08:34:49 main:478] : INFO : Batch Size: 128
[2022-04-17 08:34:49 main:479] : INFO : Learning Rate: 0.01
[2022-04-17 08:34:49 main:480] : INFO : Patience: 10
[2022-04-17 08:34:49 main:481] : INFO : Hidden Width: 12
[2022-04-17 08:34:49 main:482] : INFO : # Recurrent Layers: 4
[2022-04-17 08:34:49 main:483] : INFO : # Backend Layers: 4
[2022-04-17 08:34:49 main:484] : INFO : # Threads: 1
[2022-04-17 08:34:49 main:486] : INFO : Preparing Dataset
[2022-04-17 08:34:49 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-17 08:34:50 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-17 08:34:51 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-04-17 08:34:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-17 08:34:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-17 08:34:51 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-04-17 08:34:51 main:494] : INFO : Creating Model
[2022-04-17 08:34:51 main:507] : INFO : Preparing config file
[2022-04-17 08:34:51 main:511] : INFO : Found checkpoint, attempting to load...
[2022-04-17 08:34:51 main:512] : INFO : Loading config
[2022-04-17 08:34:51 main:514] : INFO : Loading state
[2022-04-17 08:34:52 main:559] : INFO : Loading DataLoader into Memory
[2022-04-17 08:34:52 main:562] : INFO : Starting Training
[2022-04-17 08:34:56 main:574] : INFO : Epoch 1867 | loss: 0.00124984 | val_loss: 0.000274957 | Time: 4330.15 ms
[2022-04-17 08:35:01 main:574] : INFO : Epoch 1868 | loss: 0.000200974 | val_loss: 0.000112525 | Time: 4198.09 ms
[2022-04-17 08:35:05 main:574] : INFO : Epoch 1869 | loss: 5.29714e-05 | val_loss: 2.6697e-05 | Time: 4039.09 ms
[2022-04-17 08:35:09 main:574] : INFO : Epoch 1870 | loss: 3.02832e-05 | val_loss: 5.3299e-05 | Time: 4177.34 ms
[2022-04-17 08:35:13 main:574] : INFO : Epoch 1871 | loss: 0.000106048 | val_loss: 2.54689e-05 | Time: 4093.94 ms
[2022-04-17 08:35:17 main:574] : INFO : Epoch 1872 | loss: 2.28291e-05 | val_loss: 2.12697e-05 | Time: 4061.89 ms
[2022-04-17 08:35:21 main:574] : INFO : Epoch 1873 | loss: 2.14743e-05 | val_loss: 2.13014e-05 | Time: 4101.11 ms
[2022-04-17 08:35:25 main:574] : INFO : Epoch 1874 | loss: 2.12419e-05 | val_loss: 2.13619e-05 | Time: 4085.91 ms
[2022-04-17 08:35:29 main:574] : INFO : Epoch 1875 | loss: 2.1108e-05 | val_loss: 2.13459e-05 | Time: 4132.92 ms
[2022-04-17 08:35:34 main:574] : INFO : Epoch 1876 | loss: 2.1156e-05 | val_loss: 2.1205e-05 | Time: 4117.36 ms
[2022-04-17 08:35:38 main:574] : INFO : Epoch 1877 | loss: 2.13163e-05 | val_loss: 2.13713e-05 | Time: 4118.08 ms
[2022-04-17 08:35:42 main:574] : INFO : Epoch 1878 | loss: 2.11621e-05 | val_loss: 2.10173e-05 | Time: 4098.85 ms
[2022-04-17 08:35:46 main:574] : INFO : Epoch 1879 | loss: 2.09838e-05 | val_loss: 2.05956e-05 | Time: 4234.53 ms
[2022-04-17 08:35:50 main:574] : INFO : Epoch 1880 | loss: 2.07754e-05 | val_loss: 2.02983e-05 | Time: 4287.63 ms
[2022-04-17 08:35:55 main:574] : INFO : Epoch 1881 | loss: 2.06884e-05 | val_loss: 2.02001e-05 | Time: 4432.2 ms
[2022-04-17 08:35:59 main:574] : INFO : Epoch 1882 | loss: 2.06114e-05 | val_loss: 2.01382e-05 | Time: 4377.49 ms
[2022-04-17 08:36:03 main:574] : INFO : Epoch 1883 | loss: 2.05853e-05 | val_loss: 2.01024e-05 | Time: 4391.99 ms
[2022-04-17 08:36:08 main:574] : INFO : Epoch 1884 | loss: 2.05531e-05 | val_loss: 2.01109e-05 | Time: 4272.49 ms
[2022-04-17 08:36:12 main:574] : INFO : Epoch 1885 | loss: 2.0644e-05 | val_loss: 2.00805e-05 | Time: 4601.56 ms
[2022-04-17 08:36:16 main:574] : INFO : Epoch 1886 | loss: 2.05572e-05 | val_loss: 2.01261e-05 | Time: 3771.77 ms
[2022-04-17 08:36:20 main:574] : INFO : Epoch 1887 | loss: 2.04789e-05 | val_loss: 1.9994e-05 | Time: 4111.75 ms
[2022-04-17 08:36:24 main:574] : INFO : Epoch 1888 | loss: 2.04292e-05 | val_loss: 1.99518e-05 | Time: 4209.4 ms
[2022-04-17 08:36:29 main:574] : INFO : Epoch 1889 | loss: 2.04104e-05 | val_loss: 1.98557e-05 | Time: 4959.56 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 2080 SUPER)
[2022-04-17 08:39:34 main:435] : INFO : Set logging level to 1
[2022-04-17 08:39:34 main:441] : INFO : Running in BOINC Client mode
[2022-04-17 08:39:34 main:444] : INFO : Resolving all filenames
[2022-04-17 08:39:34 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-17 08:39:34 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1)
[2022-04-17 08:39:34 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-17 08:39:34 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-17 08:39:34 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-04-17 08:39:34 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-04-17 08:39:34 main:474] : INFO : Configuration:
[2022-04-17 08:39:34 main:475] : INFO : Model type: GRU
[2022-04-17 08:39:34 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-04-17 08:39:34 main:477] : INFO : Max Epochs: 2048
[2022-04-17 08:39:34 main:478] : INFO : Batch Size: 128
[2022-04-17 08:39:34 main:479] : INFO : Learning Rate: 0.01
[2022-04-17 08:39:34 main:480] : INFO : Patience: 10
[2022-04-17 08:39:34 main:481] : INFO : Hidden Width: 12
[2022-04-17 08:39:34 main:482] : INFO : # Recurrent Layers: 4
[2022-04-17 08:39:34 main:483] : INFO : # Backend Layers: 4
[2022-04-17 08:39:34 main:484] : INFO : # Threads: 1
[2022-04-17 08:39:34 main:486] : INFO : Preparing Dataset
[2022-04-17 08:39:34 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-17 08:39:34 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-17 08:39:36 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-04-17 08:39:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-17 08:39:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-17 08:39:36 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-04-17 08:39:36 main:494] : INFO : Creating Model
[2022-04-17 08:39:36 main:507] : INFO : Preparing config file
[2022-04-17 08:39:36 main:511] : INFO : Found checkpoint, attempting to load...
[2022-04-17 08:39:36 main:512] : INFO : Loading config
[2022-04-17 08:39:36 main:514] : INFO : Loading state
[2022-04-17 08:39:37 main:559] : INFO : Loading DataLoader into Memory
[2022-04-17 08:39:37 main:562] : INFO : Starting Training
[2022-04-17 08:39:41 main:574] : INFO : Epoch 1881 | loss: 0.0010413 | val_loss: 0.000338821 | Time: 3814.49 ms
[2022-04-17 08:39:45 main:574] : INFO : Epoch 1882 | loss: 0.000172564 | val_loss: 7.89564e-05 | Time: 4052.81 ms
[2022-04-17 08:39:49 main:574] : INFO : Epoch 1883 | loss: 5.4813e-05 | val_loss: 4.54804e-05 | Time: 4101.61 ms
[2022-04-17 08:39:53 main:574] : INFO : Epoch 1884 | loss: 3.4579e-05 | val_loss: 3.39149e-05 | Time: 4189.3 ms
[2022-04-17 08:39:57 main:574] : INFO : Epoch 1885 | loss: 2.93992e-05 | val_loss: 3.13605e-05 | Time: 4254.07 ms
[2022-04-17 08:40:01 main:574] : INFO : Epoch 1886 | loss: 2.80731e-05 | val_loss: 2.90501e-05 | Time: 3799.13 ms
[2022-04-17 08:40:05 main:574] : INFO : Epoch 1887 | loss: 2.34616e-05 | val_loss: 2.17146e-05 | Time: 4127.8 ms
[2022-04-17 08:40:09 main:574] : INFO : Epoch 1888 | loss: 5.76809e-05 | val_loss: 2.4508e-05 | Time: 4057.17 ms
[2022-04-17 08:40:13 main:574] : INFO : Epoch 1889 | loss: 2.42945e-05 | val_loss: 2.40436e-05 | Time: 4153.83 ms
[2022-04-17 08:40:17 main:574] : INFO : Epoch 1890 | loss: 2.3857e-05 | val_loss: 2.35586e-05 | Time: 3763.62 ms
[2022-04-17 08:40:21 main:574] : INFO : Epoch 1891 | loss: 2.35356e-05 | val_loss: 2.30454e-05 | Time: 3995.45 ms
[2022-04-17 08:40:25 main:574] : INFO : Epoch 1892 | loss: 2.3077e-05 | val_loss: 2.24075e-05 | Time: 4033.46 ms
[2022-04-17 08:40:29 main:574] : INFO : Epoch 1893 | loss: 2.1933e-05 | val_loss: 2.08652e-05 | Time: 3928.11 ms
[2022-04-17 08:40:33 main:574] : INFO : Epoch 1894 | loss: 2.14951e-05 | val_loss: 2.07932e-05 | Time: 3847.66 ms
[2022-04-17 08:40:37 main:574] : INFO : Epoch 1895 | loss: 2.14687e-05 | val_loss: 2.07421e-05 | Time: 3987.26 ms
[2022-04-17 08:40:41 main:574] : INFO : Epoch 1896 | loss: 2.13658e-05 | val_loss: 2.07515e-05 | Time: 4039.47 ms
[2022-04-17 08:40:45 main:574] : INFO : Epoch 1897 | loss: 2.13555e-05 | val_loss: 2.07023e-05 | Time: 3897.87 ms
[2022-04-17 08:40:49 main:574] : INFO : Epoch 1898 | loss: 2.13108e-05 | val_loss: 2.06705e-05 | Time: 3881.03 ms
[2022-04-17 08:40:53 main:574] : INFO : Epoch 1899 | loss: 2.13298e-05 | val_loss: 2.05769e-05 | Time: 3773.05 ms
[2022-04-17 08:40:56 main:574] : INFO : Epoch 1900 | loss: 2.12939e-05 | val_loss: 2.05912e-05 | Time: 3807.23 ms
[2022-04-17 08:41:00 main:574] : INFO : Epoch 1901 | loss: 2.12666e-05 | val_loss: 2.05605e-05 | Time: 3765.03 ms
[2022-04-17 08:41:04 main:574] : INFO : Epoch 1902 | loss: 2.13326e-05 | val_loss: 2.05054e-05 | Time: 3948.68 ms
[2022-04-17 08:41:08 main:574] : INFO : Epoch 1903 | loss: 2.1262e-05 | val_loss: 2.06136e-05 | Time: 3829.4 ms
[2022-04-17 08:41:11 main:574] : INFO : Epoch 1904 | loss: 2.12323e-05 | val_loss: 2.05024e-05 | Time: 3519.1 ms
[2022-04-17 08:41:15 main:574] : INFO : Epoch 1905 | loss: 2.1185e-05 | val_loss: 2.05131e-05 | Time: 3835.31 ms
[2022-04-17 08:41:19 main:574] : INFO : Epoch 1906 | loss: 2.11611e-05 | val_loss: 2.0464e-05 | Time: 4029.15 ms
[2022-04-17 08:41:23 main:574] : INFO : Epoch 1907 | loss: 2.12192e-05 | val_loss: 2.05721e-05 | Time: 3721.35 ms
[2022-04-17 08:41:26 main:574] : INFO : Epoch 1908 | loss: 2.11943e-05 | val_loss: 2.03765e-05 | Time: 3210.45 ms
[2022-04-17 08:41:30 main:574] : INFO : Epoch 1909 | loss: 2.11562e-05 | val_loss: 2.05683e-05 | Time: 4054.72 ms
[2022-04-17 08:41:34 main:574] : INFO : Epoch 1910 | loss: 2.11459e-05 | val_loss: 2.04126e-05 | Time: 4105.68 ms
[2022-04-17 08:41:39 main:574] : INFO : Epoch 1911 | loss: 2.10498e-05 | val_loss: 2.01079e-05 | Time: 4342.71 ms
[2022-04-17 08:41:42 main:574] : INFO : Epoch 1912 | loss: 1.97764e-05 | val_loss: 1.64511e-05 | Time: 3383.98 ms
[2022-04-17 08:41:46 main:574] : INFO : Epoch 1913 | loss: 1.70181e-05 | val_loss: 1.60989e-05 | Time: 3865.46 ms
[2022-04-17 08:41:50 main:574] : INFO : Epoch 1914 | loss: 1.6953e-05 | val_loss: 1.60419e-05 | Time: 4065.49 ms
[2022-04-17 08:41:54 main:574] : INFO : Epoch 1915 | loss: 1.68188e-05 | val_loss: 1.59421e-05 | Time: 3678.55 ms
[2022-04-17 08:41:57 main:574] : INFO : Epoch 1916 | loss: 1.67957e-05 | val_loss: 1.5961e-05 | Time: 3724.54 ms
[2022-04-17 08:42:01 main:574] : INFO : Epoch 1917 | loss: 1.67623e-05 | val_loss: 1.59017e-05 | Time: 3953.39 ms
[2022-04-17 08:42:05 main:574] : INFO : Epoch 1918 | loss: 1.67407e-05 | val_loss: 1.58478e-05 | Time: 3886.68 ms
[2022-04-17 08:42:09 main:574] : INFO : Epoch 1919 | loss: 1.67328e-05 | val_loss: 1.59526e-05 | Time: 3790.14 ms
[2022-04-17 08:42:13 main:574] : INFO : Epoch 1920 | loss: 1.67174e-05 | val_loss: 1.58309e-05 | Time: 3770.02 ms
[2022-04-17 08:42:17 main:574] : INFO : Epoch 1921 | loss: 1.67583e-05 | val_loss: 1.5879e-05 | Time: 3673.39 ms
[2022-04-17 08:42:20 main:574] : INFO : Epoch 1922 | loss: 1.67019e-05 | val_loss: 1.58083e-05 | Time: 3688.98 ms
[2022-04-17 08:42:24 main:574] : INFO : Epoch 1923 | loss: 1.66699e-05 | val_loss: 1.58799e-05 | Time: 3617.26 ms
[2022-04-17 08:42:27 main:574] : INFO : Epoch 1924 | loss: 1.67296e-05 | val_loss: 1.58898e-05 | Time: 3595.69 ms
[2022-04-17 08:42:31 main:574] : INFO : Epoch 1925 | loss: 1.67179e-05 | val_loss: 1.59334e-05 | Time: 3387.64 ms
[2022-04-17 08:42:35 main:574] : INFO : Epoch 1926 | loss: 1.66829e-05 | val_loss: 1.58716e-05 | Time: 4074.74 ms
[2022-04-17 08:42:39 main:574] : INFO : Epoch 1927 | loss: 1.66731e-05 | val_loss: 1.57866e-05 | Time: 3991.89 ms
[2022-04-17 08:42:43 main:574] : INFO : Epoch 1928 | loss: 1.665e-05 | val_loss: 1.58139e-05 | Time: 3998.59 ms
[2022-04-17 08:42:46 main:574] : INFO : Epoch 1929 | loss: 1.67059e-05 | val_loss: 1.57473e-05 | Time: 3282.82 ms
[2022-04-17 08:42:50 main:574] : INFO : Epoch 1930 | loss: 1.66254e-05 | val_loss: 1.58489e-05 | Time: 4134.29 ms
[2022-04-17 08:42:54 main:574] : INFO : Epoch 1931 | loss: 1.65724e-05 | val_loss: 1.57553e-05 | Time: 4023.18 ms
[2022-04-17 08:42:58 main:574] : INFO : Epoch 1932 | loss: 1.65961e-05 | val_loss: 1.58082e-05 | Time: 3872.14 ms
[2022-04-17 08:43:03 main:574] : INFO : Epoch 1933 | loss: 1.6572e-05 | val_loss: 1.56669e-05 | Time: 4579.12 ms
[2022-04-17 08:43:06 main:574] : INFO : Epoch 1934 | loss: 1.65807e-05 | val_loss: 1.58048e-05 | Time: 3383.31 ms
[2022-04-17 08:43:10 main:574] : INFO : Epoch 1935 | loss: 1.66183e-05 | val_loss: 1.58075e-05 | Time: 3796.17 ms
[2022-04-17 08:43:14 main:574] : INFO : Epoch 1936 | loss: 1.66074e-05 | val_loss: 1.58447e-05 | Time: 3831.66 ms
[2022-04-17 08:43:18 main:574] : INFO : Epoch 1937 | loss: 1.66071e-05 | val_loss: 1.57073e-05 | Time: 3838.51 ms
[2022-04-17 08:43:21 main:574] : INFO : Epoch 1938 | loss: 1.6622e-05 | val_loss: 1.59467e-05 | Time: 3503.19 ms
[2022-04-17 08:43:25 main:574] : INFO : Epoch 1939 | loss: 1.67381e-05 | val_loss: 1.59676e-05 | Time: 3851.75 ms
[2022-04-17 08:43:29 main:574] : INFO : Epoch 1940 | loss: 1.6626e-05 | val_loss: 1.57191e-05 | Time: 3791.84 ms
[2022-04-17 08:43:33 main:574] : INFO : Epoch 1941 | loss: 1.6618e-05 | val_loss: 1.57814e-05 | Time: 3842.4 ms
[2022-04-17 08:43:37 main:574] : INFO : Epoch 1942 | loss: 1.65571e-05 | val_loss: 1.57016e-05 | Time: 3997.85 ms
[2022-04-17 08:43:40 main:574] : INFO : Epoch 1943 | loss: 1.65481e-05 | val_loss: 1.56832e-05 | Time: 3487.89 ms
[2022-04-17 08:43:44 main:574] : INFO : Epoch 1944 | loss: 1.66011e-05 | val_loss: 1.57516e-05 | Time: 3823.85 ms
[2022-04-17 08:43:48 main:574] : INFO : Epoch 1945 | loss: 1.66625e-05 | val_loss: 1.58551e-05 | Time: 3782.89 ms
[2022-04-17 08:43:52 main:574] : INFO : Epoch 1946 | loss: 1.6597e-05 | val_loss: 1.57455e-05 | Time: 3808.1 ms
[2022-04-17 08:43:55 main:574] : INFO : Epoch 1947 | loss: 1.63857e-05 | val_loss: 1.57578e-05 | Time: 3662.04 ms
[2022-04-17 08:43:59 main:574] : INFO : Epoch 1948 | loss: 1.63471e-05 | val_loss: 1.56669e-05 | Time: 3441.94 ms
[2022-04-17 08:44:03 main:574] : INFO : Epoch 1949 | loss: 1.63264e-05 | val_loss: 1.58605e-05 | Time: 4234.04 ms
[2022-04-17 08:44:07 main:574] : INFO : Epoch 1950 | loss: 1.64287e-05 | val_loss: 1.57248e-05 | Time: 3809.59 ms
[2022-04-17 08:44:11 main:574] : INFO : Epoch 1951 | loss: 1.63829e-05 | val_loss: 1.57971e-05 | Time: 3717.91 ms
[2022-04-17 08:44:14 main:574] : INFO : Epoch 1952 | loss: 1.65038e-05 | val_loss: 1.5938e-05 | Time: 3325.54 ms
[2022-04-17 08:44:18 main:574] : INFO : Epoch 1953 | loss: 1.64744e-05 | val_loss: 1.60257e-05 | Time: 3995.45 ms
[2022-04-17 08:44:22 main:574] : INFO : Epoch 1954 | loss: 1.65014e-05 | val_loss: 1.59596e-05 | Time: 4075.94 ms
[2022-04-17 08:44:26 main:574] : INFO : Epoch 1955 | loss: 1.66945e-05 | val_loss: 1.60022e-05 | Time: 3791.8 ms
[2022-04-17 08:44:29 main:574] : INFO : Epoch 1956 | loss: 1.64176e-05 | val_loss: 1.57712e-05 | Time: 3354.91 ms
[2022-04-17 08:44:33 main:574] : INFO : Epoch 1957 | loss: 1.63942e-05 | val_loss: 1.5707e-05 | Time: 3886.5 ms
[2022-04-17 08:44:37 main:574] : INFO : Epoch 1958 | loss: 1.63435e-05 | val_loss: 1.56466e-05 | Time: 3889.66 ms
[2022-04-17 08:44:41 main:574] : INFO : Epoch 1959 | loss: 1.63202e-05 | val_loss: 1.56792e-05 | Time: 3978.66 ms
[2022-04-17 08:44:45 main:574] : INFO : Epoch 1960 | loss: 1.63222e-05 | val_loss: 1.5711e-05 | Time: 3684.21 ms
[2022-04-17 08:44:48 main:574] : INFO : Epoch 1961 | loss: 1.64664e-05 | val_loss: 1.58029e-05 | Time: 3906.73 ms
[2022-04-17 08:44:52 main:574] : INFO : Epoch 1962 | loss: 1.63556e-05 | val_loss: 1.57105e-05 | Time: 3895.47 ms
[2022-04-17 08:44:56 main:574] : INFO : Epoch 1963 | loss: 1.63914e-05 | val_loss: 1.59502e-05 | Time: 3910.4 ms
[2022-04-17 08:45:00 main:574] : INFO : Epoch 1964 | loss: 1.67071e-05 | val_loss: 1.75761e-05 | Time: 3924.47 ms
[2022-04-17 08:45:04 main:574] : INFO : Epoch 1965 | loss: 1.80641e-05 | val_loss: 1.59834e-05 | Time: 4025.03 ms
[2022-04-17 08:45:08 main:574] : INFO : Epoch 1966 | loss: 1.6503e-05 | val_loss: 1.57439e-05 | Time: 3994.96 ms
[2022-04-17 08:45:12 main:574] : INFO : Epoch 1967 | loss: 1.6672e-05 | val_loss: 1.56531e-05 | Time: 3745.38 ms
[2022-04-17 08:45:16 main:574] : INFO : Epoch 1968 | loss: 1.65717e-05 | val_loss: 1.6518e-05 | Time: 3778.41 ms
[2022-04-17 08:45:20 main:574] : INFO : Epoch 1969 | loss: 1.67648e-05 | val_loss: 1.57855e-05 | Time: 3731.65 ms
[2022-04-17 08:45:23 main:574] : INFO : Epoch 1970 | loss: 1.64187e-05 | val_loss: 1.63631e-05 | Time: 3637.25 ms
[2022-04-17 08:45:27 main:574] : INFO : Epoch 1971 | loss: 1.99819e-05 | val_loss: 1.58224e-05 | Time: 3759.3 ms
[2022-04-17 08:45:31 main:574] : INFO : Epoch 1972 | loss: 1.69387e-05 | val_loss: 1.57379e-05 | Time: 3810.45 ms
[2022-04-17 08:45:35 main:574] : INFO : Epoch 1973 | loss: 1.65952e-05 | val_loss: 1.5832e-05 | Time: 3813.83 ms
[2022-04-17 08:45:38 main:574] : INFO : Epoch 1974 | loss: 1.64292e-05 | val_loss: 1.56332e-05 | Time: 3123.56 ms
[2022-04-17 08:45:41 main:574] : INFO : Epoch 1975 | loss: 1.64016e-05 | val_loss: 1.57626e-05 | Time: 3669.88 ms
[2022-04-17 08:45:45 main:574] : INFO : Epoch 1976 | loss: 1.64989e-05 | val_loss: 1.58635e-05 | Time: 3691.91 ms
[2022-04-17 08:45:49 main:574] : INFO : Epoch 1977 | loss: 1.67339e-05 | val_loss: 1.66266e-05 | Time: 3939.78 ms
[2022-04-17 08:45:53 main:574] : INFO : Epoch 1978 | loss: 1.6571e-05 | val_loss: 1.57261e-05 | Time: 3726.02 ms
[2022-04-17 08:45:56 main:574] : INFO : Epoch 1979 | loss: 1.71708e-05 | val_loss: 2.31799e-05 | Time: 3689.06 ms
[2022-04-17 08:46:00 main:574] : INFO : Epoch 1980 | loss: 2.93571e-05 | val_loss: 2.10137e-05 | Time: 4075.92 ms
[2022-04-17 08:46:04 main:574] : INFO : Epoch 1981 | loss: 1.82018e-05 | val_loss: 1.59288e-05 | Time: 3962.49 ms
[2022-04-17 08:46:08 main:574] : INFO : Epoch 1982 | loss: 1.65988e-05 | val_loss: 1.58304e-05 | Time: 3803.88 ms
[2022-04-17 08:46:12 main:574] : INFO : Epoch 1983 | loss: 1.64088e-05 | val_loss: 1.57198e-05 | Time: 3849.29 ms
[2022-04-17 08:46:16 main:574] : INFO : Epoch 1984 | loss: 1.63221e-05 | val_loss: 1.56619e-05 | Time: 3833.23 ms
[2022-04-17 08:46:20 main:574] : INFO : Epoch 1985 | loss: 1.62476e-05 | val_loss: 1.56617e-05 | Time: 3746.62 ms
[2022-04-17 08:46:23 main:574] : INFO : Epoch 1986 | loss: 1.62901e-05 | val_loss: 1.57369e-05 | Time: 3729.73 ms
[2022-04-17 08:46:27 main:574] : INFO : Epoch 1987 | loss: 1.62991e-05 | val_loss: 1.59783e-05 | Time: 3764.67 ms
[2022-04-17 08:46:31 main:574] : INFO : Epoch 1988 | loss: 1.68013e-05 | val_loss: 1.58673e-05 | Time: 3834.3 ms
[2022-04-17 08:46:35 main:574] : INFO : Epoch 1989 | loss: 1.6487e-05 | val_loss: 1.58495e-05 | Time: 3938.33 ms
[2022-04-17 08:46:39 main:574] : INFO : Epoch 1990 | loss: 1.63772e-05 | val_loss: 1.58174e-05 | Time: 3916.24 ms
[2022-04-17 08:46:43 main:574] : INFO : Epoch 1991 | loss: 1.68442e-05 | val_loss: 1.64775e-05 | Time: 3785.53 ms
[2022-04-17 08:46:47 main:574] : INFO : Epoch 1992 | loss: 1.70711e-05 | val_loss: 1.65816e-05 | Time: 3831.78 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce RTX 2080 SUPER)
[2022-04-17 08:49:47 main:435] : INFO : Set logging level to 1
[2022-04-17 08:49:47 main:441] : INFO : Running in BOINC Client mode
[2022-04-17 08:49:47 main:444] : INFO : Resolving all filenames
[2022-04-17 08:49:47 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-17 08:49:47 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1)
[2022-04-17 08:49:47 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-17 08:49:47 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-17 08:49:47 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-04-17 08:49:47 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-04-17 08:49:47 main:474] : INFO : Configuration:
[2022-04-17 08:49:47 main:475] : INFO : Model type: GRU
[2022-04-17 08:49:47 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-04-17 08:49:47 main:477] : INFO : Max Epochs: 2048
[2022-04-17 08:49:47 main:478] : INFO : Batch Size: 128
[2022-04-17 08:49:47 main:479] : INFO : Learning Rate: 0.01
[2022-04-17 08:49:47 main:480] : INFO : Patience: 10
[2022-04-17 08:49:47 main:481] : INFO : Hidden Width: 12
[2022-04-17 08:49:47 main:482] : INFO : # Recurrent Layers: 4
[2022-04-17 08:49:47 main:483] : INFO : # Backend Layers: 4
[2022-04-17 08:49:47 main:484] : INFO : # Threads: 1
[2022-04-17 08:49:47 main:486] : INFO : Preparing Dataset
[2022-04-17 08:49:47 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-17 08:49:48 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-17 08:49:49 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-04-17 08:49:49 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-17 08:49:49 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-17 08:49:49 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-04-17 08:49:49 main:494] : INFO : Creating Model
[2022-04-17 08:49:49 main:507] : INFO : Preparing config file
[2022-04-17 08:49:49 main:511] : INFO : Found checkpoint, attempting to load...
[2022-04-17 08:49:49 main:512] : INFO : Loading config
[2022-04-17 08:49:49 main:514] : INFO : Loading state
[2022-04-17 08:49:50 main:559] : INFO : Loading DataLoader into Memory
[2022-04-17 08:49:50 main:562] : INFO : Starting Training
[2022-04-17 08:49:54 main:574] : INFO : Epoch 1992 | loss: 0.000389066 | val_loss: 0.000130022 | Time: 4060.2 ms
[2022-04-17 08:49:58 main:574] : INFO : Epoch 1993 | loss: 6.83769e-05 | val_loss: 3.03204e-05 | Time: 3835.73 ms
[2022-04-17 08:50:02 main:574] : INFO : Epoch 1994 | loss: 2.61199e-05 | val_loss: 1.82999e-05 | Time: 3895.35 ms
[2022-04-17 08:50:06 main:574] : INFO : Epoch 1995 | loss: 1.85192e-05 | val_loss: 1.67126e-05 | Time: 4103.46 ms
[2022-04-17 08:50:10 main:574] : INFO : Epoch 1996 | loss: 1.68431e-05 | val_loss: 1.58748e-05 | Time: 4038.47 ms
[2022-04-17 08:50:14 main:574] : INFO : Epoch 1997 | loss: 1.63698e-05 | val_loss: 1.53983e-05 | Time: 3957.05 ms
[2022-04-17 08:50:18 main:574] : INFO : Epoch 1998 | loss: 1.62238e-05 | val_loss: 1.52625e-05 | Time: 4062.8 ms
[2022-04-17 08:50:22 main:574] : INFO : Epoch 1999 | loss: 1.61065e-05 | val_loss: 1.52327e-05 | Time: 3932.22 ms
[2022-04-17 08:50:25 main:574] : INFO : Epoch 2000 | loss: 1.60933e-05 | val_loss: 1.51411e-05 | Time: 3268.89 ms
[2022-04-17 08:50:29 main:574] : INFO : Epoch 2001 | loss: 1.60689e-05 | val_loss: 1.50653e-05 | Time: 4004.37 ms
[2022-04-17 08:50:33 main:574] : INFO : Epoch 2002 | loss: 1.6038e-05 | val_loss: 1.50569e-05 | Time: 3993.39 ms
[2022-04-17 08:50:37 main:574] : INFO : Epoch 2003 | loss: 1.59937e-05 | val_loss: 1.51249e-05 | Time: 4118.99 ms
[2022-04-17 08:50:41 main:574] : INFO : Epoch 2004 | loss: 1.59671e-05 | val_loss: 1.50063e-05 | Time: 4052.06 ms
[2022-04-17 08:50:45 main:574] : INFO : Epoch 2005 | loss: 1.59452e-05 | val_loss: 1.50015e-05 | Time: 3858.24 ms
[2022-04-17 08:50:49 main:574] : INFO : Epoch 2006 | loss: 1.59101e-05 | val_loss: 1.51455e-05 | Time: 3922.17 ms
[2022-04-17 08:50:53 main:574] : INFO : Epoch 2007 | loss: 1.58909e-05 | val_loss: 1.49637e-05 | Time: 3851.52 ms
[2022-04-17 08:50:57 main:574] : INFO : Epoch 2008 | loss: 1.58218e-05 | val_loss: 1.49815e-05 | Time: 3855.64 ms
[2022-04-17 08:51:01 main:574] : INFO : Epoch 2009 | loss: 1.58158e-05 | val_loss: 1.49654e-05 | Time: 3980.62 ms
[2022-04-17 08:51:05 main:574] : INFO : Epoch 2010 | loss: 1.58163e-05 | val_loss: 1.49176e-05 | Time: 3892.87 ms
[2022-04-17 08:51:09 main:574] : INFO : Epoch 2011 | loss: 1.57852e-05 | val_loss: 1.48811e-05 | Time: 3861.36 ms
[2022-04-17 08:51:13 main:574] : INFO : Epoch 2012 | loss: 1.58569e-05 | val_loss: 1.51234e-05 | Time: 3894.5 ms
[2022-04-17 08:51:16 main:574] : INFO : Epoch 2013 | loss: 1.57769e-05 | val_loss: 1.48679e-05 | Time: 3827.93 ms
[2022-04-17 08:51:20 main:574] : INFO : Epoch 2014 | loss: 1.5796e-05 | val_loss: 1.48999e-05 | Time: 3914.59 ms
[2022-04-17 08:51:24 main:574] : INFO : Epoch 2015 | loss: 1.5781e-05 | val_loss: 1.49093e-05 | Time: 3827.7 ms
[2022-04-17 08:51:28 main:574] : INFO : Epoch 2016 | loss: 1.58137e-05 | val_loss: 1.49257e-05 | Time: 3851.05 ms
[2022-04-17 08:51:32 main:574] : INFO : Epoch 2017 | loss: 1.57691e-05 | val_loss: 1.49295e-05 | Time: 3816.26 ms
[2022-04-17 08:51:36 main:574] : INFO : Epoch 2018 | loss: 1.5758e-05 | val_loss: 1.49789e-05 | Time: 3863.31 ms
[2022-04-17 08:51:40 main:574] : INFO : Epoch 2019 | loss: 1.57502e-05 | val_loss: 1.49098e-05 | Time: 3859.32 ms
[2022-04-17 08:51:43 main:574] : INFO : Epoch 2020 | loss: 1.578e-05 | val_loss: 1.48712e-05 | Time: 3828.16 ms
[2022-04-17 08:51:47 main:574] : INFO : Epoch 2021 | loss: 1.57717e-05 | val_loss: 1.49255e-05 | Time: 3839.76 ms
[2022-04-17 08:51:51 main:574] : INFO : Epoch 2022 | loss: 1.57547e-05 | val_loss: 1.49809e-05 | Time: 3970.19 ms
[2022-04-17 08:51:55 main:574] : INFO : Epoch 2023 | loss: 1.58946e-05 | val_loss: 1.481e-05 | Time: 3686.1 ms
[2022-04-17 08:51:59 main:574] : INFO : Epoch 2024 | loss: 1.57695e-05 | val_loss: 1.48715e-05 | Time: 3859.65 ms
[2022-04-17 08:52:03 main:574] : INFO : Epoch 2025 | loss: 1.57999e-05 | val_loss: 1.49808e-05 | Time: 3906.36 ms
[2022-04-17 08:52:07 main:574] : INFO : Epoch 2026 | loss: 1.5915e-05 | val_loss: 1.51304e-05 | Time: 3873.68 ms
[2022-04-17 08:52:10 main:574] : INFO : Epoch 2027 | loss: 1.63793e-05 | val_loss: 1.70928e-05 | Time: 3902.73 ms
[2022-04-17 08:52:14 main:574] : INFO : Epoch 2028 | loss: 1.85501e-05 | val_loss: 1.58517e-05 | Time: 3846.29 ms
[2022-04-17 08:52:18 main:574] : INFO : Epoch 2029 | loss: 1.60788e-05 | val_loss: 1.50194e-05 | Time: 3833.89 ms
[2022-04-17 08:52:22 main:574] : INFO : Epoch 2030 | loss: 1.5827e-05 | val_loss: 1.52991e-05 | Time: 3991.75 ms
[2022-04-17 08:52:26 main:574] : INFO : Epoch 2031 | loss: 1.67342e-05 | val_loss: 1.86048e-05 | Time: 4118.23 ms
[2022-04-17 08:52:30 main:574] : INFO : Epoch 2032 | loss: 1.84785e-05 | val_loss: 1.64602e-05 | Time: 4048.18 ms
[2022-04-17 08:52:34 main:574] : INFO : Epoch 2033 | loss: 1.64699e-05 | val_loss: 1.52287e-05 | Time: 3980.6 ms
[2022-04-17 08:52:38 main:574] : INFO : Epoch 2034 | loss: 1.58917e-05 | val_loss: 1.50834e-05 | Time: 4090.35 ms
[2022-04-17 08:52:42 main:574] : INFO : Epoch 2035 | loss: 1.65409e-05 | val_loss: 1.89146e-05 | Time: 3832.39 ms
[2022-04-17 08:52:46 main:574] : INFO : Epoch 2036 | loss: 1.96102e-05 | val_loss: 1.70981e-05 | Time: 3462.65 ms
[2022-04-17 08:52:50 main:574] : INFO : Epoch 2037 | loss: 1.6563e-05 | val_loss: 1.50223e-05 | Time: 4137.26 ms
[2022-04-17 08:52:54 main:574] : INFO : Epoch 2038 | loss: 1.60073e-05 | val_loss: 1.49632e-05 | Time: 4115.38 ms
[2022-04-17 08:52:58 main:574] : INFO : Epoch 2039 | loss: 1.57033e-05 | val_loss: 1.5165e-05 | Time: 3931.34 ms
[2022-04-17 08:53:02 main:574] : INFO : Epoch 2040 | loss: 1.89936e-05 | val_loss: 3.15104e-05 | Time: 4027.8 ms
[2022-04-17 08:53:06 main:574] : INFO : Epoch 2041 | loss: 2.15424e-05 | val_loss: 1.82736e-05 | Time: 3866.82 ms
[2022-04-17 08:53:10 main:574] : INFO : Epoch 2042 | loss: 1.67582e-05 | val_loss: 1.49854e-05 | Time: 3867.33 ms
[2022-04-17 08:53:14 main:574] : INFO : Epoch 2043 | loss: 1.62278e-05 | val_loss: 1.56384e-05 | Time: 3841.29 ms
[2022-04-17 08:53:17 main:574] : INFO : Epoch 2044 | loss: 1.60183e-05 | val_loss: 1.50739e-05 | Time: 3837.36 ms
[2022-04-17 08:53:21 main:574] : INFO : Epoch 2045 | loss: 1.6001e-05 | val_loss: 1.54724e-05 | Time: 3857.43 ms
[2022-04-17 08:53:25 main:574] : INFO : Epoch 2046 | loss: 1.6279e-05 | val_loss: 1.66582e-05 | Time: 3948.23 ms
[2022-04-17 08:53:29 main:574] : INFO : Epoch 2047 | loss: 1.84508e-05 | val_loss: 2.09054e-05 | Time: 3867.32 ms
[2022-04-17 08:53:33 main:574] : INFO : Epoch 2048 | loss: 1.8256e-05 | val_loss: 1.62727e-05 | Time: 3912.17 ms
[2022-04-17 08:53:33 main:597] : INFO : Saving trained model to model-final.pt, val_loss 1.62727e-05
[2022-04-17 08:53:33 main:603] : INFO : Saving end state to config to file
[2022-04-17 08:53:33 main:608] : INFO : Success, exiting..
08:53:33 (2880): called boinc_finish(0)
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)