Task 11200517

Name ParityModified-1639966065-27873-2-0_0
Workunit 8474376
Created 31 Jan 2022, 15:02:18 UTC
Sent 16 Feb 2022, 9:52:15 UTC
Report deadline 24 Feb 2022, 9:52:15 UTC
Received 28 Feb 2022, 18:57:37 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 11321
Run time 2 hours 5 min 34 sec
CPU time 2 hours 0 min 58 sec
Validate state Task was reported too late to validate
Credit 0.00
Device peak FLOPS 2,461.93 GFLOPS
Application version Machine Learning Dataset Generator (GPU) v9.75 (cuda10200)
windows_x86_64
Peak working set size 2.74 GB
Peak swap size 4.30 GB
Peak disk usage 1.54 GB

Stderr output

<core_client_version>7.16.20</core_client_version>
<![CDATA[
<stderr_txt>
och 1644 | loss: 0.0310327 | val_loss: 0.0310792 | Time: 3833.24 ms
[2022-02-27 21:25:54	                main:574]	:	INFO	:	Epoch 1645 | loss: 0.0310271 | val_loss: 0.0310817 | Time: 3835.83 ms
[2022-02-27 21:25:58	                main:574]	:	INFO	:	Epoch 1646 | loss: 0.0310299 | val_loss: 0.0310895 | Time: 3815.61 ms
[2022-02-27 21:26:12	                main:574]	:	INFO	:	Epoch 1647 | loss: 0.0310271 | val_loss: 0.0310879 | Time: 13928.7 ms
[2022-02-27 21:26:16	                main:574]	:	INFO	:	Epoch 1648 | loss: 0.0310254 | val_loss: 0.0310824 | Time: 4127.01 ms
[2022-02-27 21:26:20	                main:574]	:	INFO	:	Epoch 1649 | loss: 0.031028 | val_loss: 0.0310768 | Time: 3850.16 ms
[2022-02-27 21:26:23	                main:574]	:	INFO	:	Epoch 1650 | loss: 0.0310276 | val_loss: 0.0310745 | Time: 3868.59 ms
[2022-02-27 21:26:27	                main:574]	:	INFO	:	Epoch 1651 | loss: 0.0310222 | val_loss: 0.0310922 | Time: 3824.65 ms
[2022-02-27 21:26:31	                main:574]	:	INFO	:	Epoch 1652 | loss: 0.0310247 | val_loss: 0.0310799 | Time: 3854.47 ms
[2022-02-27 21:26:35	                main:574]	:	INFO	:	Epoch 1653 | loss: 0.0310198 | val_loss: 0.0310746 | Time: 3854.37 ms
[2022-02-27 21:26:39	                main:574]	:	INFO	:	Epoch 1654 | loss: 0.0310161 | val_loss: 0.0310828 | Time: 3912.65 ms
[2022-02-27 21:26:43	                main:574]	:	INFO	:	Epoch 1655 | loss: 0.0310166 | val_loss: 0.0310839 | Time: 4019.87 ms
[2022-02-27 21:26:47	                main:574]	:	INFO	:	Epoch 1656 | loss: 0.0310163 | val_loss: 0.0310752 | Time: 4137.9 ms
[2022-02-27 21:26:51	                main:574]	:	INFO	:	Epoch 1657 | loss: 0.0310152 | val_loss: 0.0310747 | Time: 3984.04 ms
[2022-02-27 21:26:55	                main:574]	:	INFO	:	Epoch 1658 | loss: 0.0310197 | val_loss: 0.0310731 | Time: 3931.53 ms
[2022-02-27 21:26:59	                main:574]	:	INFO	:	Epoch 1659 | loss: 0.0310204 | val_loss: 0.0310715 | Time: 4249.47 ms
[2022-02-27 21:27:14	                main:574]	:	INFO	:	Epoch 1660 | loss: 0.0310175 | val_loss: 0.0310743 | Time: 14764.6 ms
[2022-02-27 21:27:18	                main:574]	:	INFO	:	Epoch 1661 | loss: 0.0310227 | val_loss: 0.0310791 | Time: 3834.9 ms
[2022-02-27 21:27:22	                main:574]	:	INFO	:	Epoch 1662 | loss: 0.0310219 | val_loss: 0.031081 | Time: 3881.92 ms
[2022-02-27 21:27:26	                main:574]	:	INFO	:	Epoch 1663 | loss: 0.0310279 | val_loss: 0.0310847 | Time: 3852.69 ms
[2022-02-27 21:27:29	                main:574]	:	INFO	:	Epoch 1664 | loss: 0.0310382 | val_loss: 0.0310854 | Time: 3822.76 ms
[2022-02-27 21:27:33	                main:574]	:	INFO	:	Epoch 1665 | loss: 0.0310495 | val_loss: 0.0310846 | Time: 3847.17 ms
[2022-02-27 21:27:37	                main:574]	:	INFO	:	Epoch 1666 | loss: 0.0310419 | val_loss: 0.0310789 | Time: 3821.04 ms
[2022-02-27 21:27:41	                main:574]	:	INFO	:	Epoch 1667 | loss: 0.0310344 | val_loss: 0.0310888 | Time: 4193.67 ms
[2022-02-27 21:27:45	                main:574]	:	INFO	:	Epoch 1668 | loss: 0.0310359 | val_loss: 0.0310988 | Time: 4016.73 ms
[2022-02-27 21:27:49	                main:574]	:	INFO	:	Epoch 1669 | loss: 0.0310282 | val_loss: 0.0310934 | Time: 4031.94 ms
[2022-02-27 21:27:53	                main:574]	:	INFO	:	Epoch 1670 | loss: 0.0310228 | val_loss: 0.0310806 | Time: 3931.63 ms
[2022-02-27 21:27:57	                main:574]	:	INFO	:	Epoch 1671 | loss: 0.0310241 | val_loss: 0.0310933 | Time: 3908.5 ms
[2022-02-27 21:28:02	                main:574]	:	INFO	:	Epoch 1672 | loss: 0.0310218 | val_loss: 0.03109 | Time: 4236.3 ms
[2022-02-27 21:28:16	                main:574]	:	INFO	:	Epoch 1673 | loss: 0.0310178 | val_loss: 0.0310837 | Time: 14349.3 ms
[2022-02-27 21:28:20	                main:574]	:	INFO	:	Epoch 1674 | loss: 0.0310199 | val_loss: 0.0310719 | Time: 3842.35 ms
[2022-02-27 21:28:24	                main:574]	:	INFO	:	Epoch 1675 | loss: 0.0310224 | val_loss: 0.0310763 | Time: 3884.4 ms
[2022-02-27 21:28:28	                main:574]	:	INFO	:	Epoch 1676 | loss: 0.0310217 | val_loss: 0.0310757 | Time: 3867.77 ms
[2022-02-27 21:28:31	                main:574]	:	INFO	:	Epoch 1677 | loss: 0.0310173 | val_loss: 0.0310838 | Time: 3890.84 ms
[2022-02-27 21:28:35	                main:574]	:	INFO	:	Epoch 1678 | loss: 0.0310143 | val_loss: 0.0310846 | Time: 3794.88 ms
[2022-02-27 21:28:39	                main:574]	:	INFO	:	Epoch 1679 | loss: 0.0310137 | val_loss: 0.0310895 | Time: 3834.41 ms
[2022-02-27 21:28:43	                main:574]	:	INFO	:	Epoch 1680 | loss: 0.0310169 | val_loss: 0.0310799 | Time: 4019.61 ms
[2022-02-27 21:28:47	                main:574]	:	INFO	:	Epoch 1681 | loss: 0.0310192 | val_loss: 0.0310825 | Time: 3986.14 ms
[2022-02-27 21:28:51	                main:574]	:	INFO	:	Epoch 1682 | loss: 0.0310169 | val_loss: 0.0310766 | Time: 3986.24 ms
[2022-02-27 21:28:55	                main:574]	:	INFO	:	Epoch 1683 | loss: 0.031015 | val_loss: 0.0310787 | Time: 3946.95 ms
[2022-02-27 21:28:59	                main:574]	:	INFO	:	Epoch 1684 | loss: 0.0310175 | val_loss: 0.0310795 | Time: 4095.93 ms
[2022-02-27 21:29:13	                main:574]	:	INFO	:	Epoch 1685 | loss: 0.0310168 | val_loss: 0.0310791 | Time: 14155.5 ms
[2022-02-27 21:29:17	                main:574]	:	INFO	:	Epoch 1686 | loss: 0.0310129 | val_loss: 0.0310733 | Time: 3852.06 ms
[2022-02-27 21:29:21	                main:574]	:	INFO	:	Epoch 1687 | loss: 0.0310209 | val_loss: 0.0310694 | Time: 3834.05 ms
[2022-02-27 21:29:25	                main:574]	:	INFO	:	Epoch 1688 | loss: 0.0310198 | val_loss: 0.0310763 | Time: 3885.61 ms
[2022-02-27 21:29:29	                main:574]	:	INFO	:	Epoch 1689 | loss: 0.0310212 | val_loss: 0.0310743 | Time: 3848.45 ms
[2022-02-27 21:29:33	                main:574]	:	INFO	:	Epoch 1690 | loss: 0.0310231 | val_loss: 0.0310781 | Time: 4529.65 ms
[2022-02-27 21:29:38	                main:574]	:	INFO	:	Epoch 1691 | loss: 0.0310176 | val_loss: 0.0310793 | Time: 4503.77 ms
[2022-02-27 21:29:42	                main:574]	:	INFO	:	Epoch 1692 | loss: 0.0310163 | val_loss: 0.0310773 | Time: 4451.06 ms
[2022-02-27 21:29:47	                main:574]	:	INFO	:	Epoch 1693 | loss: 0.0310184 | val_loss: 0.031072 | Time: 4629.34 ms
[2022-02-27 21:29:51	                main:574]	:	INFO	:	Epoch 1694 | loss: 0.031021 | val_loss: 0.0310723 | Time: 4520.38 ms
[2022-02-27 21:29:56	                main:574]	:	INFO	:	Epoch 1695 | loss: 0.0310221 | val_loss: 0.0310656 | Time: 4450.38 ms
[2022-02-27 21:30:00	                main:574]	:	INFO	:	Epoch 1696 | loss: 0.0310165 | val_loss: 0.0310701 | Time: 4604.73 ms
[2022-02-28 17:35:05	                main:574]	:	INFO	:	Epoch 1697 | loss: 0.0310166 | val_loss: 0.0310811 | Time: 7.23125e+07 ms
[2022-02-28 17:35:19	                main:574]	:	INFO	:	Epoch 1698 | loss: 0.0310227 | val_loss: 0.0310805 | Time: 14181.7 ms
[2022-02-28 17:35:23	                main:574]	:	INFO	:	Epoch 1699 | loss: 0.0310208 | val_loss: 0.0310691 | Time: 4063.4 ms
[2022-02-28 17:35:38	                main:574]	:	INFO	:	Epoch 1700 | loss: 0.0310167 | val_loss: 0.0310671 | Time: 14826.7 ms
[2022-02-28 17:35:42	                main:574]	:	INFO	:	Epoch 1701 | loss: 0.031025 | val_loss: 0.0310686 | Time: 3781.14 ms
[2022-02-28 17:35:46	                main:574]	:	INFO	:	Epoch 1702 | loss: 0.0310323 | val_loss: 0.0310733 | Time: 3742.32 ms
[2022-02-28 17:35:49	                main:574]	:	INFO	:	Epoch 1703 | loss: 0.0310308 | val_loss: 0.0310735 | Time: 3824.75 ms
[2022-02-28 17:35:53	                main:574]	:	INFO	:	Epoch 1704 | loss: 0.0310314 | val_loss: 0.0310728 | Time: 4002.51 ms
[2022-02-28 17:36:08	                main:574]	:	INFO	:	Epoch 1705 | loss: 0.0310261 | val_loss: 0.0310711 | Time: 14850.9 ms
[2022-02-28 17:36:12	                main:574]	:	INFO	:	Epoch 1706 | loss: 0.0310195 | val_loss: 0.0310688 | Time: 3803.98 ms
[2022-02-28 17:36:16	                main:574]	:	INFO	:	Epoch 1707 | loss: 0.031022 | val_loss: 0.0310692 | Time: 3804.16 ms
[2022-02-28 17:36:20	                main:574]	:	INFO	:	Epoch 1708 | loss: 0.0310185 | val_loss: 0.0310666 | Time: 3771.39 ms
[2022-02-28 17:36:23	                main:574]	:	INFO	:	Epoch 1709 | loss: 0.031016 | val_loss: 0.0310665 | Time: 3799.17 ms
[2022-02-28 17:36:27	                main:574]	:	INFO	:	Epoch 1710 | loss: 0.0310119 | val_loss: 0.031062 | Time: 3854.62 ms
[2022-02-28 17:36:32	                main:574]	:	INFO	:	Epoch 1711 | loss: 0.031012 | val_loss: 0.0310644 | Time: 4193.5 ms
[2022-02-28 17:36:36	                main:574]	:	INFO	:	Epoch 1712 | loss: 0.0310138 | val_loss: 0.0310679 | Time: 4342.21 ms
[2022-02-28 17:36:40	                main:574]	:	INFO	:	Epoch 1713 | loss: 0.0310124 | val_loss: 0.03106 | Time: 4149.52 ms
[2022-02-28 17:36:44	                main:574]	:	INFO	:	Epoch 1714 | loss: 0.03101 | val_loss: 0.0310659 | Time: 4152.19 ms
[2022-02-28 17:36:48	                main:574]	:	INFO	:	Epoch 1715 | loss: 0.0310154 | val_loss: 0.0310963 | Time: 4005.4 ms
[2022-02-28 17:36:53	                main:574]	:	INFO	:	Epoch 1716 | loss: 0.0310392 | val_loss: 0.0310786 | Time: 4332.05 ms
[2022-02-28 17:37:07	                main:574]	:	INFO	:	Epoch 1717 | loss: 0.0310376 | val_loss: 0.03107 | Time: 14403.5 ms
[2022-02-28 17:37:11	                main:574]	:	INFO	:	Epoch 1718 | loss: 0.0310294 | val_loss: 0.0310665 | Time: 4194.08 ms
[2022-02-28 17:37:15	                main:574]	:	INFO	:	Epoch 1719 | loss: 0.0310245 | val_loss: 0.0310683 | Time: 3862.28 ms
[2022-02-28 17:37:19	                main:574]	:	INFO	:	Epoch 1720 | loss: 0.0310195 | val_loss: 0.0310655 | Time: 3804.04 ms
[2022-02-28 17:37:23	                main:574]	:	INFO	:	Epoch 1721 | loss: 0.0310161 | val_loss: 0.0310627 | Time: 3798.03 ms
[2022-02-28 17:37:27	                main:574]	:	INFO	:	Epoch 1722 | loss: 0.0310148 | val_loss: 0.0310581 | Time: 3840.2 ms
[2022-02-28 17:37:30	                main:574]	:	INFO	:	Epoch 1723 | loss: 0.0310109 | val_loss: 0.0310641 | Time: 3829.83 ms
[2022-02-28 17:37:34	                main:574]	:	INFO	:	Epoch 1724 | loss: 0.03101 | val_loss: 0.0310737 | Time: 3841.72 ms
[2022-02-28 17:37:38	                main:574]	:	INFO	:	Epoch 1725 | loss: 0.0310254 | val_loss: 0.0310654 | Time: 3880.81 ms
[2022-02-28 17:37:42	                main:574]	:	INFO	:	Epoch 1726 | loss: 0.0310214 | val_loss: 0.0310696 | Time: 3864.62 ms
[2022-02-28 17:37:46	                main:574]	:	INFO	:	Epoch 1727 | loss: 0.0310172 | val_loss: 0.0310715 | Time: 3836.56 ms
[2022-02-28 17:37:50	                main:574]	:	INFO	:	Epoch 1728 | loss: 0.0310134 | val_loss: 0.0310714 | Time: 3904.7 ms
[2022-02-28 17:37:54	                main:574]	:	INFO	:	Epoch 1729 | loss: 0.031015 | val_loss: 0.0310733 | Time: 3882.7 ms
[2022-02-28 17:38:09	                main:574]	:	INFO	:	Epoch 1730 | loss: 0.0310169 | val_loss: 0.0310758 | Time: 15072.8 ms
[2022-02-28 17:38:13	                main:574]	:	INFO	:	Epoch 1731 | loss: 0.0310125 | val_loss: 0.0310797 | Time: 3851.8 ms
[2022-02-28 17:38:16	                main:574]	:	INFO	:	Epoch 1732 | loss: 0.0310087 | val_loss: 0.0310697 | Time: 3834.86 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650 Ti with Max-Q Design)
[2022-02-28 17:49:43	                main:435]	:	INFO	:	Set logging level to 1
[2022-02-28 17:49:43	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-02-28 17:49:43	                main:444]	:	INFO	:	Resolving all filenames
[2022-02-28 17:49:43	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-28 17:49:43	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-02-28 17:49:43	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-28 17:49:43	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-02-28 17:49:43	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-02-28 17:49:43	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-02-28 17:49:43	                main:474]	:	INFO	:	Configuration: 
[2022-02-28 17:49:43	                main:475]	:	INFO	:	    Model type: GRU
[2022-02-28 17:49:43	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-02-28 17:49:43	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-02-28 17:49:43	                main:478]	:	INFO	:	    Batch Size: 128
[2022-02-28 17:49:43	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-02-28 17:49:43	                main:480]	:	INFO	:	    Patience: 10
[2022-02-28 17:49:43	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-02-28 17:49:43	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-02-28 17:49:43	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-02-28 17:49:43	                main:484]	:	INFO	:	    # Threads: 1
[2022-02-28 17:49:43	                main:486]	:	INFO	:	Preparing Dataset
[2022-02-28 17:49:43	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-28 17:49:44	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-28 17:49:46	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-02-28 17:49:46	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-28 17:49:46	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-28 17:49:47	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-02-28 17:49:47	                main:494]	:	INFO	:	Creating Model
[2022-02-28 17:49:47	                main:507]	:	INFO	:	Preparing config file
[2022-02-28 17:49:47	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-02-28 17:49:47	                main:512]	:	INFO	:	Loading config
[2022-02-28 17:49:47	                main:514]	:	INFO	:	Loading state
[2022-02-28 17:49:48	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-02-28 17:49:48	                main:562]	:	INFO	:	Starting Training
[2022-02-28 17:49:52	                main:574]	:	INFO	:	Epoch 1731 | loss: 0.0311278 | val_loss: 0.0310905 | Time: 3860.13 ms
[2022-02-28 17:49:55	                main:574]	:	INFO	:	Epoch 1732 | loss: 0.0310272 | val_loss: 0.0310726 | Time: 3138.36 ms
[2022-02-28 17:50:09	                main:574]	:	INFO	:	Epoch 1733 | loss: 0.0310173 | val_loss: 0.0310717 | Time: 14118.2 ms
[2022-02-28 17:50:13	                main:574]	:	INFO	:	Epoch 1734 | loss: 0.0310109 | val_loss: 0.0310592 | Time: 3855.94 ms
[2022-02-28 17:50:17	                main:574]	:	INFO	:	Epoch 1735 | loss: 0.03101 | val_loss: 0.0310711 | Time: 3848.12 ms
[2022-02-28 17:50:21	                main:574]	:	INFO	:	Epoch 1736 | loss: 0.0310147 | val_loss: 0.0310812 | Time: 3857.61 ms
[2022-02-28 17:50:25	                main:574]	:	INFO	:	Epoch 1737 | loss: 0.031017 | val_loss: 0.0310882 | Time: 3845.65 ms
[2022-02-28 17:50:29	                main:574]	:	INFO	:	Epoch 1738 | loss: 0.0310177 | val_loss: 0.0310755 | Time: 3855.1 ms
[2022-02-28 17:50:33	                main:574]	:	INFO	:	Epoch 1739 | loss: 0.0310226 | val_loss: 0.0310717 | Time: 3823.45 ms
[2022-02-28 17:50:36	                main:574]	:	INFO	:	Epoch 1740 | loss: 0.0310172 | val_loss: 0.0310817 | Time: 3852.5 ms
[2022-02-28 17:50:40	                main:574]	:	INFO	:	Epoch 1741 | loss: 0.0310155 | val_loss: 0.0310684 | Time: 3873.87 ms
[2022-02-28 17:50:47	                main:574]	:	INFO	:	Epoch 1742 | loss: 0.0310077 | val_loss: 0.0310705 | Time: 6705.05 ms
[2022-02-28 17:51:22	                main:574]	:	INFO	:	Epoch 1743 | loss: 0.0310044 | val_loss: 0.031067 | Time: 33581.8 ms
[2022-02-28 17:51:26	                main:574]	:	INFO	:	Epoch 1744 | loss: 0.0310077 | val_loss: 0.0310703 | Time: 3805.82 ms
[2022-02-28 17:51:29	                main:574]	:	INFO	:	Epoch 1745 | loss: 0.0310064 | val_loss: 0.031066 | Time: 3859.06 ms
[2022-02-28 17:51:33	                main:574]	:	INFO	:	Epoch 1746 | loss: 0.0310043 | val_loss: 0.031078 | Time: 3847.46 ms
[2022-02-28 17:51:37	                main:574]	:	INFO	:	Epoch 1747 | loss: 0.031004 | val_loss: 0.0310698 | Time: 3941.31 ms
[2022-02-28 17:51:52	                main:574]	:	INFO	:	Epoch 1748 | loss: 0.0310024 | val_loss: 0.0310644 | Time: 14432 ms
[2022-02-28 17:51:56	                main:574]	:	INFO	:	Epoch 1749 | loss: 0.0310007 | val_loss: 0.0310591 | Time: 3826.84 ms
[2022-02-28 17:52:09	                main:574]	:	INFO	:	Epoch 1750 | loss: 0.0310015 | val_loss: 0.0310537 | Time: 13940.2 ms
[2022-02-28 17:52:13	                main:574]	:	INFO	:	Epoch 1751 | loss: 0.0310048 | val_loss: 0.0310691 | Time: 3905.84 ms
[2022-02-28 17:52:17	                main:574]	:	INFO	:	Epoch 1752 | loss: 0.031004 | val_loss: 0.0310612 | Time: 3838 ms
[2022-02-28 17:52:21	                main:574]	:	INFO	:	Epoch 1753 | loss: 0.031001 | val_loss: 0.0310644 | Time: 3842.6 ms
[2022-02-28 17:52:25	                main:574]	:	INFO	:	Epoch 1754 | loss: 0.030997 | val_loss: 0.0310658 | Time: 3895.84 ms
[2022-02-28 17:52:29	                main:574]	:	INFO	:	Epoch 1755 | loss: 0.0309991 | val_loss: 0.0310706 | Time: 3837.11 ms
[2022-02-28 17:52:33	                main:574]	:	INFO	:	Epoch 1756 | loss: 0.0310007 | val_loss: 0.0310636 | Time: 3842.41 ms
[2022-02-28 17:52:37	                main:574]	:	INFO	:	Epoch 1757 | loss: 0.0309977 | val_loss: 0.0310675 | Time: 3886.19 ms
[2022-02-28 17:52:41	                main:574]	:	INFO	:	Epoch 1758 | loss: 0.0309997 | val_loss: 0.0310648 | Time: 3966.01 ms
[2022-02-28 17:52:45	                main:574]	:	INFO	:	Epoch 1759 | loss: 0.0310016 | val_loss: 0.031067 | Time: 4115.34 ms
[2022-02-28 17:52:49	                main:574]	:	INFO	:	Epoch 1760 | loss: 0.0310031 | val_loss: 0.0310607 | Time: 3999.13 ms
[2022-02-28 17:52:53	                main:574]	:	INFO	:	Epoch 1761 | loss: 0.0310021 | val_loss: 0.0310695 | Time: 4140.85 ms
[2022-02-28 17:52:57	                main:574]	:	INFO	:	Epoch 1762 | loss: 0.0310013 | val_loss: 0.0310705 | Time: 4089.63 ms
[2022-02-28 17:53:11	                main:574]	:	INFO	:	Epoch 1763 | loss: 0.0309996 | val_loss: 0.0310701 | Time: 14356.6 ms
[2022-02-28 17:53:15	                main:574]	:	INFO	:	Epoch 1764 | loss: 0.0310046 | val_loss: 0.0310665 | Time: 3818.64 ms
[2022-02-28 17:53:19	                main:574]	:	INFO	:	Epoch 1765 | loss: 0.0310028 | val_loss: 0.0310681 | Time: 3858.47 ms
[2022-02-28 17:53:23	                main:574]	:	INFO	:	Epoch 1766 | loss: 0.0309954 | val_loss: 0.0310687 | Time: 3820.47 ms
[2022-02-28 17:53:27	                main:574]	:	INFO	:	Epoch 1767 | loss: 0.0309959 | val_loss: 0.0310581 | Time: 3884.63 ms
[2022-02-28 17:53:31	                main:574]	:	INFO	:	Epoch 1768 | loss: 0.0309984 | val_loss: 0.0310667 | Time: 3841.65 ms
[2022-02-28 17:53:35	                main:574]	:	INFO	:	Epoch 1769 | loss: 0.0309993 | val_loss: 0.0310582 | Time: 3929.42 ms
[2022-02-28 17:53:39	                main:574]	:	INFO	:	Epoch 1770 | loss: 0.0309994 | val_loss: 0.0310701 | Time: 3961.78 ms
[2022-02-28 17:53:42	                main:574]	:	INFO	:	Epoch 1771 | loss: 0.030994 | val_loss: 0.0310704 | Time: 3949.94 ms
[2022-02-28 17:53:47	                main:574]	:	INFO	:	Epoch 1772 | loss: 0.0309951 | val_loss: 0.0310698 | Time: 4087.71 ms
[2022-02-28 17:53:51	                main:574]	:	INFO	:	Epoch 1773 | loss: 0.030998 | val_loss: 0.0310679 | Time: 4134.01 ms
[2022-02-28 17:53:55	                main:574]	:	INFO	:	Epoch 1774 | loss: 0.0309988 | val_loss: 0.0310662 | Time: 4138.9 ms
[2022-02-28 17:54:09	                main:574]	:	INFO	:	Epoch 1775 | loss: 0.0310136 | val_loss: 0.0310678 | Time: 14255.6 ms
[2022-02-28 17:54:13	                main:574]	:	INFO	:	Epoch 1776 | loss: 0.0310105 | val_loss: 0.0310647 | Time: 3869.52 ms
[2022-02-28 17:54:17	                main:574]	:	INFO	:	Epoch 1777 | loss: 0.0310086 | val_loss: 0.0310664 | Time: 3936.7 ms
[2022-02-28 17:54:31	                main:574]	:	INFO	:	Epoch 1778 | loss: 0.0310157 | val_loss: 0.0310712 | Time: 14054.7 ms
[2022-02-28 17:54:35	                main:574]	:	INFO	:	Epoch 1779 | loss: 0.0310229 | val_loss: 0.0310579 | Time: 3838.09 ms
[2022-02-28 17:54:39	                main:574]	:	INFO	:	Epoch 1780 | loss: 0.0310138 | val_loss: 0.031071 | Time: 4430.44 ms
[2022-02-28 17:54:45	                main:574]	:	INFO	:	Epoch 1781 | loss: 0.0310133 | val_loss: 0.0310628 | Time: 5774.18 ms
[2022-02-28 17:54:51	                main:574]	:	INFO	:	Epoch 1782 | loss: 0.0310131 | val_loss: 0.0310714 | Time: 5850.06 ms
[2022-02-28 17:54:56	                main:574]	:	INFO	:	Epoch 1783 | loss: 0.0310058 | val_loss: 0.0310663 | Time: 5509.49 ms
[2022-02-28 17:55:12	                main:574]	:	INFO	:	Epoch 1784 | loss: 0.0310018 | val_loss: 0.0310668 | Time: 15206.2 ms
[2022-02-28 17:55:16	                main:574]	:	INFO	:	Epoch 1785 | loss: 0.0309979 | val_loss: 0.0310669 | Time: 3874.99 ms
[2022-02-28 17:55:21	                main:574]	:	INFO	:	Epoch 1786 | loss: 0.0309983 | val_loss: 0.0310703 | Time: 5060.21 ms
[2022-02-28 17:55:27	                main:574]	:	INFO	:	Epoch 1787 | loss: 0.0310201 | val_loss: 0.0310692 | Time: 6276.22 ms
[2022-02-28 17:55:33	                main:574]	:	INFO	:	Epoch 1788 | loss: 0.0310186 | val_loss: 0.0310592 | Time: 6316.37 ms
[2022-02-28 17:55:40	                main:574]	:	INFO	:	Epoch 1789 | loss: 0.0310108 | val_loss: 0.0310633 | Time: 6417.31 ms
[2022-02-28 17:55:46	                main:574]	:	INFO	:	Epoch 1790 | loss: 0.0310092 | val_loss: 0.0310613 | Time: 6345.82 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650 Ti with Max-Q Design)
[2022-02-28 18:00:43	                main:435]	:	INFO	:	Set logging level to 1
[2022-02-28 18:00:43	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-02-28 18:00:43	                main:444]	:	INFO	:	Resolving all filenames
[2022-02-28 18:00:43	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-28 18:00:43	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-02-28 18:00:43	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-28 18:00:43	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-02-28 18:00:43	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-02-28 18:00:43	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-02-28 18:00:43	                main:474]	:	INFO	:	Configuration: 
[2022-02-28 18:00:43	                main:475]	:	INFO	:	    Model type: GRU
[2022-02-28 18:00:43	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-02-28 18:00:43	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-02-28 18:00:43	                main:478]	:	INFO	:	    Batch Size: 128
[2022-02-28 18:00:43	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-02-28 18:00:43	                main:480]	:	INFO	:	    Patience: 10
[2022-02-28 18:00:43	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-02-28 18:00:43	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-02-28 18:00:43	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-02-28 18:00:43	                main:484]	:	INFO	:	    # Threads: 1
[2022-02-28 18:00:43	                main:486]	:	INFO	:	Preparing Dataset
[2022-02-28 18:00:43	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-28 18:00:43	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-28 18:00:45	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-02-28 18:00:45	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-28 18:00:45	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-28 18:00:45	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-02-28 18:00:45	                main:494]	:	INFO	:	Creating Model
[2022-02-28 18:00:45	                main:507]	:	INFO	:	Preparing config file
[2022-02-28 18:00:45	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-02-28 18:00:45	                main:512]	:	INFO	:	Loading config
[2022-02-28 18:00:45	                main:514]	:	INFO	:	Loading state
[2022-02-28 18:00:47	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-02-28 18:00:47	                main:562]	:	INFO	:	Starting Training
[2022-02-28 18:00:51	                main:574]	:	INFO	:	Epoch 1791 | loss: 0.0311295 | val_loss: 0.0310744 | Time: 3741.47 ms
[2022-02-28 18:00:54	                main:574]	:	INFO	:	Epoch 1792 | loss: 0.0310193 | val_loss: 0.0310699 | Time: 3132.15 ms
[2022-02-28 18:00:57	                main:574]	:	INFO	:	Epoch 1793 | loss: 0.0310058 | val_loss: 0.0310503 | Time: 3557.98 ms
[2022-02-28 18:01:11	                main:574]	:	INFO	:	Epoch 1794 | loss: 0.031009 | val_loss: 0.0310531 | Time: 14048.8 ms
[2022-02-28 18:01:15	                main:574]	:	INFO	:	Epoch 1795 | loss: 0.0310055 | val_loss: 0.0310642 | Time: 3814.95 ms
[2022-02-28 18:01:19	                main:574]	:	INFO	:	Epoch 1796 | loss: 0.0310019 | val_loss: 0.0310616 | Time: 3829.74 ms
[2022-02-28 18:01:23	                main:574]	:	INFO	:	Epoch 1797 | loss: 0.0310006 | val_loss: 0.0310561 | Time: 3836.58 ms
[2022-02-28 18:01:27	                main:574]	:	INFO	:	Epoch 1798 | loss: 0.0309967 | val_loss: 0.0310601 | Time: 3829.25 ms
[2022-02-28 18:01:30	                main:574]	:	INFO	:	Epoch 1799 | loss: 0.0309966 | val_loss: 0.0310556 | Time: 3808.01 ms
[2022-02-28 18:01:34	                main:574]	:	INFO	:	Epoch 1800 | loss: 0.030996 | val_loss: 0.0310538 | Time: 3898.54 ms
[2022-02-28 18:01:38	                main:574]	:	INFO	:	Epoch 1801 | loss: 0.030999 | val_loss: 0.0310638 | Time: 3809.5 ms
[2022-02-28 18:01:42	                main:574]	:	INFO	:	Epoch 1802 | loss: 0.0310031 | val_loss: 0.0310622 | Time: 4038.45 ms
[2022-02-28 18:01:46	                main:574]	:	INFO	:	Epoch 1803 | loss: 0.0309952 | val_loss: 0.0310652 | Time: 3916.52 ms
[2022-02-28 18:01:50	                main:574]	:	INFO	:	Epoch 1804 | loss: 0.0309969 | val_loss: 0.0310602 | Time: 3824.12 ms
[2022-02-28 18:01:54	                main:574]	:	INFO	:	Epoch 1805 | loss: 0.0309905 | val_loss: 0.031064 | Time: 3855.3 ms
[2022-02-28 18:01:58	                main:574]	:	INFO	:	Epoch 1806 | loss: 0.0309914 | val_loss: 0.031063 | Time: 4081.9 ms
[2022-02-28 18:02:12	                main:574]	:	INFO	:	Epoch 1807 | loss: 0.0310124 | val_loss: 0.0310782 | Time: 14129 ms
[2022-02-28 18:02:16	                main:574]	:	INFO	:	Epoch 1808 | loss: 0.0310149 | val_loss: 0.0310671 | Time: 3860.02 ms
[2022-02-28 18:02:20	                main:574]	:	INFO	:	Epoch 1809 | loss: 0.0310159 | val_loss: 0.0310687 | Time: 3866.41 ms
[2022-02-28 18:02:24	                main:574]	:	INFO	:	Epoch 1810 | loss: 0.031025 | val_loss: 0.0310813 | Time: 3886.17 ms
[2022-02-28 18:02:28	                main:574]	:	INFO	:	Epoch 1811 | loss: 0.031025 | val_loss: 0.031067 | Time: 3848.11 ms
[2022-02-28 18:02:31	                main:574]	:	INFO	:	Epoch 1812 | loss: 0.0310204 | val_loss: 0.031061 | Time: 3875.51 ms
[2022-02-28 18:02:35	                main:574]	:	INFO	:	Epoch 1813 | loss: 0.0310114 | val_loss: 0.031061 | Time: 3880 ms
[2022-02-28 18:02:39	                main:574]	:	INFO	:	Epoch 1814 | loss: 0.0310068 | val_loss: 0.0310687 | Time: 4085.1 ms
[2022-02-28 18:02:43	                main:574]	:	INFO	:	Epoch 1815 | loss: 0.0310072 | val_loss: 0.0310548 | Time: 3979.86 ms
[2022-02-28 18:02:47	                main:574]	:	INFO	:	Epoch 1816 | loss: 0.0310062 | val_loss: 0.0310591 | Time: 3972.02 ms
[2022-02-28 18:02:51	                main:574]	:	INFO	:	Epoch 1817 | loss: 0.0310069 | val_loss: 0.0310598 | Time: 4097.37 ms
[2022-02-28 18:02:55	                main:574]	:	INFO	:	Epoch 1818 | loss: 0.0310051 | val_loss: 0.0310611 | Time: 3978.92 ms
[2022-02-28 18:02:59	                main:574]	:	INFO	:	Epoch 1819 | loss: 0.0310036 | val_loss: 0.031055 | Time: 4036.71 ms
[2022-02-28 18:03:14	                main:574]	:	INFO	:	Epoch 1820 | loss: 0.0310004 | val_loss: 0.0310564 | Time: 14443 ms
[2022-02-28 18:03:18	                main:574]	:	INFO	:	Epoch 1821 | loss: 0.0309987 | val_loss: 0.0310549 | Time: 4035.96 ms
[2022-02-28 18:03:25	                main:574]	:	INFO	:	Epoch 1822 | loss: 0.0310044 | val_loss: 0.03105 | Time: 6483.36 ms
[2022-02-28 18:03:31	                main:574]	:	INFO	:	Epoch 1823 | loss: 0.0310001 | val_loss: 0.0310566 | Time: 6488.26 ms
[2022-02-28 18:03:37	                main:574]	:	INFO	:	Epoch 1824 | loss: 0.0310004 | val_loss: 0.0310481 | Time: 6448.82 ms
[2022-02-28 18:03:44	                main:574]	:	INFO	:	Epoch 1825 | loss: 0.0309987 | val_loss: 0.0310524 | Time: 6346.67 ms
[2022-02-28 18:03:50	                main:574]	:	INFO	:	Epoch 1826 | loss: 0.0309953 | val_loss: 0.0310584 | Time: 6472.08 ms
[2022-02-28 18:03:57	                main:574]	:	INFO	:	Epoch 1827 | loss: 0.0310029 | val_loss: 0.0310587 | Time: 6532.24 ms
[2022-02-28 18:04:12	                main:574]	:	INFO	:	Epoch 1828 | loss: 0.0310053 | val_loss: 0.0310581 | Time: 15333 ms
[2022-02-28 18:04:16	                main:574]	:	INFO	:	Epoch 1829 | loss: 0.0310053 | val_loss: 0.0310605 | Time: 3894.09 ms
[2022-02-28 18:04:32	                main:574]	:	INFO	:	Epoch 1830 | loss: 0.0310012 | val_loss: 0.0310527 | Time: 15787.1 ms
[2022-02-28 18:04:36	                main:574]	:	INFO	:	Epoch 1831 | loss: 0.0309966 | val_loss: 0.0310504 | Time: 3820.61 ms
[2022-02-28 18:04:42	                main:574]	:	INFO	:	Epoch 1832 | loss: 0.0309959 | val_loss: 0.031052 | Time: 6490.41 ms
[2022-02-28 18:04:49	                main:574]	:	INFO	:	Epoch 1833 | loss: 0.0310019 | val_loss: 0.03106 | Time: 6451.4 ms
[2022-02-28 18:04:55	                main:574]	:	INFO	:	Epoch 1834 | loss: 0.0310035 | val_loss: 0.031061 | Time: 6257.22 ms
[2022-02-28 18:05:12	                main:574]	:	INFO	:	Epoch 1835 | loss: 0.0310048 | val_loss: 0.0310664 | Time: 16889.9 ms
[2022-02-28 18:05:16	                main:574]	:	INFO	:	Epoch 1836 | loss: 0.0310192 | val_loss: 0.0310545 | Time: 3989.22 ms
[2022-02-28 18:05:22	                main:574]	:	INFO	:	Epoch 1837 | loss: 0.03101 | val_loss: 0.0310488 | Time: 6532.12 ms
[2022-02-28 18:05:29	                main:574]	:	INFO	:	Epoch 1838 | loss: 0.0310128 | val_loss: 0.0310463 | Time: 6513.02 ms
[2022-02-28 18:05:35	                main:574]	:	INFO	:	Epoch 1839 | loss: 0.0310214 | val_loss: 0.0310627 | Time: 6324.68 ms
[2022-02-28 18:05:41	                main:574]	:	INFO	:	Epoch 1840 | loss: 0.0310218 | val_loss: 0.0310715 | Time: 6070.97 ms
[2022-02-28 18:05:46	                main:574]	:	INFO	:	Epoch 1841 | loss: 0.0310303 | val_loss: 0.0310714 | Time: 4361.4 ms
[2022-02-28 18:05:50	                main:574]	:	INFO	:	Epoch 1842 | loss: 0.0310315 | val_loss: 0.0310602 | Time: 4205.02 ms
[2022-02-28 18:05:55	                main:574]	:	INFO	:	Epoch 1843 | loss: 0.0310295 | val_loss: 0.0310599 | Time: 4633.19 ms
[2022-02-28 18:05:59	                main:574]	:	INFO	:	Epoch 1844 | loss: 0.0310246 | val_loss: 0.0310604 | Time: 4459.21 ms
[2022-02-28 18:06:14	                main:574]	:	INFO	:	Epoch 1845 | loss: 0.0310154 | val_loss: 0.0310551 | Time: 14687.4 ms
[2022-02-28 18:06:18	                main:574]	:	INFO	:	Epoch 1846 | loss: 0.0310116 | val_loss: 0.0310695 | Time: 3939.01 ms
[2022-02-28 18:06:22	                main:574]	:	INFO	:	Epoch 1847 | loss: 0.0310145 | val_loss: 0.0310578 | Time: 3867.27 ms
[2022-02-28 18:06:26	                main:574]	:	INFO	:	Epoch 1848 | loss: 0.0310067 | val_loss: 0.0310544 | Time: 3895.71 ms
[2022-02-28 18:06:29	                main:574]	:	INFO	:	Epoch 1849 | loss: 0.0310027 | val_loss: 0.0310595 | Time: 3860.79 ms
[2022-02-28 18:06:34	                main:574]	:	INFO	:	Epoch 1850 | loss: 0.0310015 | val_loss: 0.0310555 | Time: 4235.96 ms
[2022-02-28 18:06:38	                main:574]	:	INFO	:	Epoch 1851 | loss: 0.0309994 | val_loss: 0.0310523 | Time: 4365.22 ms
[2022-02-28 18:06:42	                main:574]	:	INFO	:	Epoch 1852 | loss: 0.0310185 | val_loss: 0.0310604 | Time: 4262.28 ms
[2022-02-28 18:06:47	                main:574]	:	INFO	:	Epoch 1853 | loss: 0.0310112 | val_loss: 0.0310546 | Time: 4170.49 ms
[2022-02-28 18:07:01	                main:574]	:	INFO	:	Epoch 1854 | loss: 0.0310107 | val_loss: 0.0310619 | Time: 14578 ms
[2022-02-28 18:07:05	                main:574]	:	INFO	:	Epoch 1855 | loss: 0.0310134 | val_loss: 0.0310569 | Time: 3902.58 ms
[2022-02-28 18:07:09	                main:574]	:	INFO	:	Epoch 1856 | loss: 0.0310225 | val_loss: 0.0310623 | Time: 3850.29 ms
[2022-02-28 18:07:13	                main:574]	:	INFO	:	Epoch 1857 | loss: 0.0310182 | val_loss: 0.0310637 | Time: 3860.44 ms
[2022-02-28 18:07:17	                main:574]	:	INFO	:	Epoch 1858 | loss: 0.0310149 | val_loss: 0.0310662 | Time: 3885.92 ms
[2022-02-28 18:07:21	                main:574]	:	INFO	:	Epoch 1859 | loss: 0.0310091 | val_loss: 0.031065 | Time: 3863.16 ms
[2022-02-28 18:07:25	                main:574]	:	INFO	:	Epoch 1860 | loss: 0.0310122 | val_loss: 0.0310606 | Time: 3892.67 ms
[2022-02-28 18:07:29	                main:574]	:	INFO	:	Epoch 1861 | loss: 0.0310069 | val_loss: 0.031056 | Time: 4032.38 ms
[2022-02-28 18:07:33	                main:574]	:	INFO	:	Epoch 1862 | loss: 0.0310042 | val_loss: 0.0310612 | Time: 3979.8 ms
[2022-02-28 18:07:37	                main:574]	:	INFO	:	Epoch 1863 | loss: 0.0310071 | val_loss: 0.0310581 | Time: 3985.12 ms
[2022-02-28 18:07:40	                main:574]	:	INFO	:	Epoch 1864 | loss: 0.0310104 | val_loss: 0.0310573 | Time: 3755.55 ms
[2022-02-28 18:07:45	                main:574]	:	INFO	:	Epoch 1865 | loss: 0.0310039 | val_loss: 0.0310581 | Time: 5052.91 ms
[2022-02-28 18:07:49	                main:574]	:	INFO	:	Epoch 1866 | loss: 0.0310038 | val_loss: 0.0310577 | Time: 4040.34 ms
[2022-02-28 18:07:53	                main:574]	:	INFO	:	Epoch 1867 | loss: 0.0310006 | val_loss: 0.0310647 | Time: 4041.26 ms
[2022-02-28 18:07:58	                main:574]	:	INFO	:	Epoch 1868 | loss: 0.031005 | val_loss: 0.0310618 | Time: 4037.46 ms
[2022-02-28 18:08:02	                main:574]	:	INFO	:	Epoch 1869 | loss: 0.031004 | val_loss: 0.0310553 | Time: 4064.22 ms
[2022-02-28 18:08:06	                main:574]	:	INFO	:	Epoch 1870 | loss: 0.0310054 | val_loss: 0.0310601 | Time: 4312.95 ms
[2022-02-28 18:08:10	                main:574]	:	INFO	:	Epoch 1871 | loss: 0.0310156 | val_loss: 0.0310579 | Time: 4399.07 ms
[2022-02-28 18:08:25	                main:574]	:	INFO	:	Epoch 1872 | loss: 0.0310105 | val_loss: 0.0310691 | Time: 14667.5 ms
[2022-02-28 18:08:29	                main:574]	:	INFO	:	Epoch 1873 | loss: 0.0310432 | val_loss: 0.0310818 | Time: 4244.87 ms
[2022-02-28 18:09:24	                main:574]	:	INFO	:	Epoch 1874 | loss: 0.0310285 | val_loss: 0.0310745 | Time: 54454.3 ms
[2022-02-28 18:09:28	                main:574]	:	INFO	:	Epoch 1875 | loss: 0.0310176 | val_loss: 0.0310561 | Time: 3826.57 ms
[2022-02-28 18:09:31	                main:574]	:	INFO	:	Epoch 1876 | loss: 0.0310109 | val_loss: 0.0310572 | Time: 3864.94 ms
[2022-02-28 18:09:35	                main:574]	:	INFO	:	Epoch 1877 | loss: 0.0310104 | val_loss: 0.0310518 | Time: 3867.33 ms
[2022-02-28 18:09:39	                main:574]	:	INFO	:	Epoch 1878 | loss: 0.0310069 | val_loss: 0.0310509 | Time: 3877.13 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650 Ti with Max-Q Design)
[2022-02-28 18:38:41	                main:435]	:	INFO	:	Set logging level to 1
[2022-02-28 18:38:41	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-02-28 18:38:41	                main:444]	:	INFO	:	Resolving all filenames
[2022-02-28 18:38:41	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-28 18:38:41	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-02-28 18:38:41	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-28 18:38:41	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-02-28 18:38:41	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-02-28 18:38:41	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-02-28 18:38:41	                main:474]	:	INFO	:	Configuration: 
[2022-02-28 18:38:41	                main:475]	:	INFO	:	    Model type: GRU
[2022-02-28 18:38:41	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-02-28 18:38:41	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-02-28 18:38:41	                main:478]	:	INFO	:	    Batch Size: 128
[2022-02-28 18:38:41	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-02-28 18:38:41	                main:480]	:	INFO	:	    Patience: 10
[2022-02-28 18:38:41	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-02-28 18:38:41	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-02-28 18:38:41	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-02-28 18:38:41	                main:484]	:	INFO	:	    # Threads: 1
[2022-02-28 18:38:41	                main:486]	:	INFO	:	Preparing Dataset
[2022-02-28 18:38:41	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-28 18:38:41	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-28 18:38:43	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-02-28 18:38:43	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-28 18:38:43	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-28 18:38:43	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-02-28 18:38:43	                main:494]	:	INFO	:	Creating Model
[2022-02-28 18:38:43	                main:507]	:	INFO	:	Preparing config file
[2022-02-28 18:38:43	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-02-28 18:38:43	                main:512]	:	INFO	:	Loading config
[2022-02-28 18:38:43	                main:514]	:	INFO	:	Loading state
[2022-02-28 18:39:26	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-02-28 18:39:26	                main:562]	:	INFO	:	Starting Training
[2022-02-28 18:39:30	                main:574]	:	INFO	:	Epoch 1854 | loss: 0.0310963 | val_loss: 0.0311042 | Time: 4046.76 ms
[2022-02-28 18:39:33	                main:574]	:	INFO	:	Epoch 1855 | loss: 0.0310216 | val_loss: 0.0310521 | Time: 3169.74 ms
[2022-02-28 18:39:37	                main:574]	:	INFO	:	Epoch 1856 | loss: 0.0310085 | val_loss: 0.0310499 | Time: 3833.74 ms
[2022-02-28 18:39:41	                main:574]	:	INFO	:	Epoch 1857 | loss: 0.0310043 | val_loss: 0.0310602 | Time: 3864.46 ms
[2022-02-28 18:39:45	                main:574]	:	INFO	:	Epoch 1858 | loss: 0.0310037 | val_loss: 0.0310459 | Time: 3839.7 ms
[2022-02-28 18:39:48	                main:574]	:	INFO	:	Epoch 1859 | loss: 0.0309994 | val_loss: 0.0310635 | Time: 3882.88 ms
[2022-02-28 18:39:52	                main:574]	:	INFO	:	Epoch 1860 | loss: 0.0309976 | val_loss: 0.0310485 | Time: 3956.73 ms
[2022-02-28 18:40:07	                main:574]	:	INFO	:	Epoch 1861 | loss: 0.0309961 | val_loss: 0.0310532 | Time: 14161.2 ms
[2022-02-28 18:40:10	                main:574]	:	INFO	:	Epoch 1862 | loss: 0.0309949 | val_loss: 0.0310448 | Time: 3856.94 ms
[2022-02-28 18:40:14	                main:574]	:	INFO	:	Epoch 1863 | loss: 0.0309979 | val_loss: 0.0310645 | Time: 3821.91 ms
[2022-02-28 18:40:18	                main:574]	:	INFO	:	Epoch 1864 | loss: 0.0310009 | val_loss: 0.0310553 | Time: 3862 ms
[2022-02-28 18:40:22	                main:574]	:	INFO	:	Epoch 1865 | loss: 0.0310052 | val_loss: 0.031066 | Time: 3880.04 ms
[2022-02-28 18:40:26	                main:574]	:	INFO	:	Epoch 1866 | loss: 0.0310065 | val_loss: 0.0310499 | Time: 3884.75 ms
[2022-02-28 18:40:30	                main:574]	:	INFO	:	Epoch 1867 | loss: 0.0310073 | val_loss: 0.0310493 | Time: 3884.66 ms
[2022-02-28 18:40:34	                main:574]	:	INFO	:	Epoch 1868 | loss: 0.0310002 | val_loss: 0.0310461 | Time: 4061.89 ms
[2022-02-28 18:40:38	                main:574]	:	INFO	:	Epoch 1869 | loss: 0.0309965 | val_loss: 0.0310513 | Time: 4101.85 ms
[2022-02-28 18:40:42	                main:574]	:	INFO	:	Epoch 1870 | loss: 0.0309967 | val_loss: 0.0310547 | Time: 4004.69 ms
[2022-02-28 18:40:46	                main:574]	:	INFO	:	Epoch 1871 | loss: 0.0309915 | val_loss: 0.0310489 | Time: 4152.25 ms
[2022-02-28 18:40:50	                main:574]	:	INFO	:	Epoch 1872 | loss: 0.030993 | val_loss: 0.0310572 | Time: 4025.35 ms
[2022-02-28 18:41:05	                main:574]	:	INFO	:	Epoch 1873 | loss: 0.0309919 | val_loss: 0.0310518 | Time: 14519.9 ms
[2022-02-28 18:41:09	                main:574]	:	INFO	:	Epoch 1874 | loss: 0.0309905 | val_loss: 0.0310497 | Time: 3863.1 ms
[2022-02-28 18:41:12	                main:574]	:	INFO	:	Epoch 1875 | loss: 0.0309931 | val_loss: 0.0310541 | Time: 3829.61 ms
[2022-02-28 18:41:16	                main:574]	:	INFO	:	Epoch 1876 | loss: 0.0310019 | val_loss: 0.031057 | Time: 3836.17 ms
[2022-02-28 18:41:20	                main:574]	:	INFO	:	Epoch 1877 | loss: 0.0309974 | val_loss: 0.0310526 | Time: 3836.4 ms
[2022-02-28 18:41:24	                main:574]	:	INFO	:	Epoch 1878 | loss: 0.0309974 | val_loss: 0.0310609 | Time: 3821.93 ms
[2022-02-28 18:41:28	                main:574]	:	INFO	:	Epoch 1879 | loss: 0.0309911 | val_loss: 0.031053 | Time: 4047.18 ms
[2022-02-28 18:41:32	                main:574]	:	INFO	:	Epoch 1880 | loss: 0.0309974 | val_loss: 0.0310604 | Time: 4082.38 ms
[2022-02-28 18:41:36	                main:574]	:	INFO	:	Epoch 1881 | loss: 0.0310054 | val_loss: 0.0310579 | Time: 4184.42 ms
[2022-02-28 18:41:41	                main:574]	:	INFO	:	Epoch 1882 | loss: 0.0310104 | val_loss: 0.0310706 | Time: 4223.83 ms
[2022-02-28 18:41:45	                main:574]	:	INFO	:	Epoch 1883 | loss: 0.0310138 | val_loss: 0.0310668 | Time: 3977.78 ms
[2022-02-28 18:41:49	                main:574]	:	INFO	:	Epoch 1884 | loss: 0.0310228 | val_loss: 0.0310752 | Time: 4422.81 ms
[2022-02-28 18:41:53	                main:574]	:	INFO	:	Epoch 1885 | loss: 0.0310386 | val_loss: 0.0310782 | Time: 4176.93 ms
[2022-02-28 18:42:07	                main:574]	:	INFO	:	Epoch 1886 | loss: 0.0310173 | val_loss: 0.0310818 | Time: 14042 ms
[2022-02-28 18:42:11	                main:574]	:	INFO	:	Epoch 1887 | loss: 0.0310222 | val_loss: 0.0310681 | Time: 3852.61 ms
[2022-02-28 18:42:15	                main:574]	:	INFO	:	Epoch 1888 | loss: 0.0310132 | val_loss: 0.0310696 | Time: 3830.29 ms
[2022-02-28 18:42:19	                main:574]	:	INFO	:	Epoch 1889 | loss: 0.0310023 | val_loss: 0.0310608 | Time: 3845.58 ms
[2022-02-28 18:42:23	                main:574]	:	INFO	:	Epoch 1890 | loss: 0.0310009 | val_loss: 0.0310655 | Time: 3785.41 ms
[2022-02-28 18:42:26	                main:574]	:	INFO	:	Epoch 1891 | loss: 0.0309993 | val_loss: 0.03106 | Time: 3849.01 ms
[2022-02-28 18:42:31	                main:574]	:	INFO	:	Epoch 1892 | loss: 0.0309959 | val_loss: 0.0310561 | Time: 4309.95 ms
[2022-02-28 18:42:35	                main:574]	:	INFO	:	Epoch 1893 | loss: 0.0309985 | val_loss: 0.0310556 | Time: 4051.35 ms
[2022-02-28 18:42:39	                main:574]	:	INFO	:	Epoch 1894 | loss: 0.0309969 | val_loss: 0.031063 | Time: 4114.69 ms
[2022-02-28 18:42:43	                main:574]	:	INFO	:	Epoch 1895 | loss: 0.0309939 | val_loss: 0.0310643 | Time: 3974.63 ms
[2022-02-28 18:42:47	                main:574]	:	INFO	:	Epoch 1896 | loss: 0.0309984 | val_loss: 0.0310526 | Time: 4032.71 ms
[2022-02-28 18:42:51	                main:574]	:	INFO	:	Epoch 1897 | loss: 0.0309955 | val_loss: 0.0310607 | Time: 4153.25 ms
[2022-02-28 18:43:06	                main:574]	:	INFO	:	Epoch 1898 | loss: 0.0309957 | val_loss: 0.0310555 | Time: 14708.4 ms
[2022-02-28 18:43:10	                main:574]	:	INFO	:	Epoch 1899 | loss: 0.0309961 | val_loss: 0.0310611 | Time: 3833.49 ms
[2022-02-28 18:43:14	                main:574]	:	INFO	:	Epoch 1900 | loss: 0.0309948 | val_loss: 0.0310552 | Time: 3863.11 ms
[2022-02-28 18:43:19	                main:574]	:	INFO	:	Epoch 1901 | loss: 0.0309898 | val_loss: 0.0310601 | Time: 4997.97 ms
[2022-02-28 18:43:24	                main:574]	:	INFO	:	Epoch 1902 | loss: 0.030992 | val_loss: 0.0310651 | Time: 5089.35 ms
[2022-02-28 18:43:29	                main:574]	:	INFO	:	Epoch 1903 | loss: 0.0309958 | val_loss: 0.0310535 | Time: 5322.87 ms
[2022-02-28 18:43:34	                main:574]	:	INFO	:	Epoch 1904 | loss: 0.0309943 | val_loss: 0.0310577 | Time: 4944.84 ms
[2022-02-28 18:43:39	                main:574]	:	INFO	:	Epoch 1905 | loss: 0.030992 | val_loss: 0.0310487 | Time: 5112.91 ms
[2022-02-28 18:43:45	                main:574]	:	INFO	:	Epoch 1906 | loss: 0.0309892 | val_loss: 0.0310526 | Time: 5494.15 ms
[2022-02-28 18:43:50	                main:574]	:	INFO	:	Epoch 1907 | loss: 0.0309867 | val_loss: 0.0310559 | Time: 5212.73 ms
[2022-02-28 18:44:05	                main:574]	:	INFO	:	Epoch 1908 | loss: 0.0309845 | val_loss: 0.0310561 | Time: 15343.2 ms
[2022-02-28 18:44:09	                main:574]	:	INFO	:	Epoch 1909 | loss: 0.0309837 | val_loss: 0.0310531 | Time: 3829.13 ms
[2022-02-28 18:44:13	                main:574]	:	INFO	:	Epoch 1910 | loss: 0.0309795 | val_loss: 0.0310529 | Time: 3854.42 ms
[2022-02-28 18:44:18	                main:574]	:	INFO	:	Epoch 1911 | loss: 0.030979 | val_loss: 0.031058 | Time: 5325.46 ms
[2022-02-28 18:44:24	                main:574]	:	INFO	:	Epoch 1912 | loss: 0.0309797 | val_loss: 0.0310555 | Time: 5557.02 ms
[2022-02-28 18:44:30	                main:574]	:	INFO	:	Epoch 1913 | loss: 0.0309826 | val_loss: 0.0310722 | Time: 6297.22 ms
[2022-02-28 18:44:36	                main:574]	:	INFO	:	Epoch 1914 | loss: 0.0309878 | val_loss: 0.0310643 | Time: 5899.74 ms
[2022-02-28 18:44:41	                main:574]	:	INFO	:	Epoch 1915 | loss: 0.0309844 | val_loss: 0.0310607 | Time: 5547.61 ms
[2022-02-28 18:44:48	                main:574]	:	INFO	:	Epoch 1916 | loss: 0.0309819 | val_loss: 0.0310611 | Time: 6031.71 ms
[2022-02-28 18:44:53	                main:574]	:	INFO	:	Epoch 1917 | loss: 0.0309908 | val_loss: 0.0310729 | Time: 5772.99 ms
[2022-02-28 18:45:08	                main:574]	:	INFO	:	Epoch 1918 | loss: 0.0309965 | val_loss: 0.0310677 | Time: 14274.7 ms
[2022-02-28 18:45:13	                main:574]	:	INFO	:	Epoch 1919 | loss: 0.0309952 | val_loss: 0.031072 | Time: 4872.13 ms
[2022-02-28 18:45:18	                main:574]	:	INFO	:	Epoch 1920 | loss: 0.0309972 | val_loss: 0.0310663 | Time: 5516.63 ms
[2022-02-28 18:45:24	                main:574]	:	INFO	:	Epoch 1921 | loss: 0.0309961 | val_loss: 0.0310614 | Time: 5733.14 ms
[2022-02-28 18:45:29	                main:574]	:	INFO	:	Epoch 1922 | loss: 0.0309962 | val_loss: 0.0310674 | Time: 5458.35 ms
[2022-02-28 18:45:36	                main:574]	:	INFO	:	Epoch 1923 | loss: 0.0310028 | val_loss: 0.0310592 | Time: 6296.99 ms
[2022-02-28 18:45:40	                main:574]	:	INFO	:	Epoch 1924 | loss: 0.0310068 | val_loss: 0.0310539 | Time: 4902.35 ms
[2022-02-28 18:45:45	                main:574]	:	INFO	:	Epoch 1925 | loss: 0.031 | val_loss: 0.0310531 | Time: 4350.92 ms
[2022-02-28 18:45:49	                main:574]	:	INFO	:	Epoch 1926 | loss: 0.0309963 | val_loss: 0.0310509 | Time: 4436.25 ms
[2022-02-28 18:45:54	                main:574]	:	INFO	:	Epoch 1927 | loss: 0.0310195 | val_loss: 0.0310935 | Time: 4518 ms
[2022-02-28 18:46:08	                main:574]	:	INFO	:	Epoch 1928 | loss: 0.0310869 | val_loss: 0.0311117 | Time: 14044.8 ms
[2022-02-28 18:46:12	                main:574]	:	INFO	:	Epoch 1929 | loss: 0.0310857 | val_loss: 0.0310989 | Time: 3853 ms
[2022-02-28 18:46:16	                main:574]	:	INFO	:	Epoch 1930 | loss: 0.0310701 | val_loss: 0.0310926 | Time: 3857.49 ms
[2022-02-28 18:46:22	                main:574]	:	INFO	:	Epoch 1931 | loss: 0.0310593 | val_loss: 0.0310917 | Time: 6210.89 ms
[2022-02-28 18:46:26	                main:574]	:	INFO	:	Epoch 1932 | loss: 0.0310539 | val_loss: 0.0310888 | Time: 4382.87 ms
[2022-02-28 18:46:31	                main:574]	:	INFO	:	Epoch 1933 | loss: 0.0310484 | val_loss: 0.0310801 | Time: 4343.67 ms
[2022-02-28 18:46:35	                main:574]	:	INFO	:	Epoch 1934 | loss: 0.0310403 | val_loss: 0.0310958 | Time: 4391.74 ms
[2022-02-28 18:46:40	                main:574]	:	INFO	:	Epoch 1935 | loss: 0.0310635 | val_loss: 0.0310867 | Time: 4476.67 ms
[2022-02-28 18:46:44	                main:574]	:	INFO	:	Epoch 1936 | loss: 0.0310546 | val_loss: 0.0310795 | Time: 4371.04 ms
[2022-02-28 18:46:48	                main:574]	:	INFO	:	Epoch 1937 | loss: 0.0310393 | val_loss: 0.0310687 | Time: 4231.57 ms
[2022-02-28 18:46:53	                main:574]	:	INFO	:	Epoch 1938 | loss: 0.0310315 | val_loss: 0.0310717 | Time: 4587.93 ms
[2022-02-28 18:47:07	                main:574]	:	INFO	:	Epoch 1939 | loss: 0.0310283 | val_loss: 0.0310689 | Time: 14658.4 ms
[2022-02-28 18:47:11	                main:574]	:	INFO	:	Epoch 1940 | loss: 0.0310255 | val_loss: 0.031063 | Time: 3834.23 ms
[2022-02-28 18:47:15	                main:574]	:	INFO	:	Epoch 1941 | loss: 0.0310233 | val_loss: 0.031075 | Time: 3841.13 ms
[2022-02-28 18:47:19	                main:574]	:	INFO	:	Epoch 1942 | loss: 0.0310206 | val_loss: 0.0310759 | Time: 3841.81 ms
[2022-02-28 18:47:23	                main:574]	:	INFO	:	Epoch 1943 | loss: 0.0310155 | val_loss: 0.0310713 | Time: 3802.88 ms
[2022-02-28 18:47:27	                main:574]	:	INFO	:	Epoch 1944 | loss: 0.0310156 | val_loss: 0.0310637 | Time: 3945.14 ms
[2022-02-28 18:47:31	                main:574]	:	INFO	:	Epoch 1945 | loss: 0.0310175 | val_loss: 0.0310665 | Time: 3947.77 ms
[2022-02-28 18:47:35	                main:574]	:	INFO	:	Epoch 1946 | loss: 0.0310185 | val_loss: 0.0310697 | Time: 4201.88 ms
[2022-02-28 18:47:39	                main:574]	:	INFO	:	Epoch 1947 | loss: 0.0310325 | val_loss: 0.0310674 | Time: 4091.32 ms
[2022-02-28 18:47:43	                main:574]	:	INFO	:	Epoch 1948 | loss: 0.0310293 | val_loss: 0.0310569 | Time: 3937.71 ms
[2022-02-28 18:47:47	                main:574]	:	INFO	:	Epoch 1949 | loss: 0.0310234 | val_loss: 0.0310476 | Time: 4161.83 ms
[2022-02-28 18:47:51	                main:574]	:	INFO	:	Epoch 1950 | loss: 0.0310237 | val_loss: 0.0310563 | Time: 4239.04 ms
[2022-02-28 18:48:06	                main:574]	:	INFO	:	Epoch 1951 | loss: 0.0310379 | val_loss: 0.0310723 | Time: 14270.3 ms
[2022-02-28 18:48:10	                main:574]	:	INFO	:	Epoch 1952 | loss: 0.0310569 | val_loss: 0.0310845 | Time: 3914.1 ms
[2022-02-28 18:48:13	                main:574]	:	INFO	:	Epoch 1953 | loss: 0.0310494 | val_loss: 0.031067 | Time: 3840.03 ms
[2022-02-28 18:48:17	                main:574]	:	INFO	:	Epoch 1954 | loss: 0.0310359 | val_loss: 0.0310669 | Time: 3848 ms
[2022-02-28 18:48:22	                main:574]	:	INFO	:	Epoch 1955 | loss: 0.0310301 | val_loss: 0.0310736 | Time: 4246.23 ms
[2022-02-28 18:48:27	                main:574]	:	INFO	:	Epoch 1956 | loss: 0.0310253 | val_loss: 0.0310654 | Time: 5342.38 ms
[2022-02-28 18:48:32	                main:574]	:	INFO	:	Epoch 1957 | loss: 0.031021 | val_loss: 0.0310585 | Time: 5087.73 ms
[2022-02-28 18:48:37	                main:574]	:	INFO	:	Epoch 1958 | loss: 0.0310173 | val_loss: 0.0310578 | Time: 5334.47 ms
[2022-02-28 18:48:43	                main:574]	:	INFO	:	Epoch 1959 | loss: 0.0310175 | val_loss: 0.0310584 | Time: 5211.45 ms
[2022-02-28 18:48:48	                main:574]	:	INFO	:	Epoch 1960 | loss: 0.0310145 | val_loss: 0.0310639 | Time: 5028.92 ms
[2022-02-28 18:48:53	                main:574]	:	INFO	:	Epoch 1961 | loss: 0.0310242 | val_loss: 0.0310717 | Time: 5239.19 ms
[2022-02-28 18:49:08	                main:574]	:	INFO	:	Epoch 1962 | loss: 0.0310215 | val_loss: 0.0310767 | Time: 14728.7 ms
[2022-02-28 18:49:11	                main:574]	:	INFO	:	Epoch 1963 | loss: 0.0310174 | val_loss: 0.0310615 | Time: 3872.34 ms
[2022-02-28 18:49:16	                main:574]	:	INFO	:	Epoch 1964 | loss: 0.0310135 | val_loss: 0.031071 | Time: 4738.86 ms
[2022-02-28 18:49:21	                main:574]	:	INFO	:	Epoch 1965 | loss: 0.0310136 | val_loss: 0.0310642 | Time: 5145.17 ms
[2022-02-28 18:49:27	                main:574]	:	INFO	:	Epoch 1966 | loss: 0.0310104 | val_loss: 0.0310642 | Time: 5745.04 ms
[2022-02-28 18:49:33	                main:574]	:	INFO	:	Epoch 1967 | loss: 0.0310126 | val_loss: 0.0310542 | Time: 5340 ms
[2022-02-28 18:49:38	                main:574]	:	INFO	:	Epoch 1968 | loss: 0.0310119 | val_loss: 0.0310648 | Time: 5443.88 ms
[2022-02-28 18:49:44	                main:574]	:	INFO	:	Epoch 1969 | loss: 0.0310133 | val_loss: 0.0310672 | Time: 5732.24 ms
[2022-02-28 18:49:49	                main:574]	:	INFO	:	Epoch 1970 | loss: 0.0310147 | val_loss: 0.0310617 | Time: 5688.26 ms
[2022-02-28 18:50:06	                main:574]	:	INFO	:	Epoch 1971 | loss: 0.0310095 | val_loss: 0.0310578 | Time: 16837.8 ms
[2022-02-28 18:50:10	                main:574]	:	INFO	:	Epoch 1972 | loss: 0.0310046 | val_loss: 0.0310523 | Time: 4180.05 ms
[2022-02-28 18:50:16	                main:574]	:	INFO	:	Epoch 1973 | loss: 0.0310079 | val_loss: 0.0310559 | Time: 5770.72 ms
[2022-02-28 18:50:22	                main:574]	:	INFO	:	Epoch 1974 | loss: 0.031004 | val_loss: 0.031056 | Time: 5556.31 ms
[2022-02-28 18:50:28	                main:574]	:	INFO	:	Epoch 1975 | loss: 0.0309984 | val_loss: 0.0310532 | Time: 5861.62 ms
[2022-02-28 18:50:33	                main:574]	:	INFO	:	Epoch 1976 | loss: 0.0310409 | val_loss: 0.0310946 | Time: 5778.3 ms
[2022-02-28 18:50:39	                main:574]	:	INFO	:	Epoch 1977 | loss: 0.031059 | val_loss: 0.0310809 | Time: 5496.95 ms
[2022-02-28 18:50:45	                main:574]	:	INFO	:	Epoch 1978 | loss: 0.0310496 | val_loss: 0.031075 | Time: 5940.51 ms
[2022-02-28 18:50:52	                main:574]	:	INFO	:	Epoch 1979 | loss: 0.0310383 | val_loss: 0.0310681 | Time: 6673.54 ms
[2022-02-28 18:51:17	                main:574]	:	INFO	:	Epoch 1980 | loss: 0.0310327 | val_loss: 0.0310745 | Time: 25247.7 ms
[2022-02-28 18:51:21	                main:574]	:	INFO	:	Epoch 1981 | loss: 0.0310302 | val_loss: 0.031075 | Time: 3829.24 ms
[2022-02-28 18:51:25	                main:574]	:	INFO	:	Epoch 1982 | loss: 0.031024 | val_loss: 0.0310662 | Time: 3862.25 ms
[2022-02-28 18:51:28	                main:574]	:	INFO	:	Epoch 1983 | loss: 0.0310176 | val_loss: 0.0310687 | Time: 3861.47 ms
[2022-02-28 18:51:32	                main:574]	:	INFO	:	Epoch 1984 | loss: 0.0310141 | val_loss: 0.0310723 | Time: 3809.81 ms
[2022-02-28 18:51:36	                main:574]	:	INFO	:	Epoch 1985 | loss: 0.0310124 | val_loss: 0.0310656 | Time: 3877.99 ms
[2022-02-28 18:51:40	                main:574]	:	INFO	:	Epoch 1986 | loss: 0.0310185 | val_loss: 0.031078 | Time: 4053.91 ms
[2022-02-28 18:51:45	                main:574]	:	INFO	:	Epoch 1987 | loss: 0.031032 | val_loss: 0.0310618 | Time: 4333.13 ms
[2022-02-28 18:51:49	                main:574]	:	INFO	:	Epoch 1988 | loss: 0.0310245 | val_loss: 0.0310577 | Time: 4651.17 ms
[2022-02-28 18:51:54	                main:574]	:	INFO	:	Epoch 1989 | loss: 0.0310148 | val_loss: 0.0310652 | Time: 5012.19 ms
[2022-02-28 18:52:09	                main:574]	:	INFO	:	Epoch 1990 | loss: 0.0310114 | val_loss: 0.031062 | Time: 14752.2 ms
[2022-02-28 18:52:13	                main:574]	:	INFO	:	Epoch 1991 | loss: 0.0310086 | val_loss: 0.0310585 | Time: 3849.81 ms
[2022-02-28 18:52:17	                main:574]	:	INFO	:	Epoch 1992 | loss: 0.0310054 | val_loss: 0.0310636 | Time: 3874.4 ms
[2022-02-28 18:52:21	                main:574]	:	INFO	:	Epoch 1993 | loss: 0.0310057 | val_loss: 0.0310647 | Time: 3825.45 ms
[2022-02-28 18:52:24	                main:574]	:	INFO	:	Epoch 1994 | loss: 0.0310081 | val_loss: 0.031069 | Time: 3828.13 ms
[2022-02-28 18:52:28	                main:574]	:	INFO	:	Epoch 1995 | loss: 0.0310234 | val_loss: 0.0310677 | Time: 3833.36 ms
[2022-02-28 18:52:32	                main:574]	:	INFO	:	Epoch 1996 | loss: 0.0365422 | val_loss: 0.0320424 | Time: 4085.44 ms
[2022-02-28 18:52:36	                main:574]	:	INFO	:	Epoch 1997 | loss: 0.0400677 | val_loss: 0.0320367 | Time: 4097.09 ms
[2022-02-28 18:52:41	                main:574]	:	INFO	:	Epoch 1998 | loss: 0.032086 | val_loss: 0.0314853 | Time: 4122.59 ms
[2022-02-28 18:52:45	                main:574]	:	INFO	:	Epoch 1999 | loss: 0.0313239 | val_loss: 0.0312214 | Time: 4061.28 ms
[2022-02-28 18:52:49	                main:574]	:	INFO	:	Epoch 2000 | loss: 0.0312033 | val_loss: 0.0311797 | Time: 4127.68 ms
[2022-02-28 18:52:53	                main:574]	:	INFO	:	Epoch 2001 | loss: 0.0311677 | val_loss: 0.0311528 | Time: 4147.68 ms
[2022-02-28 18:53:07	                main:574]	:	INFO	:	Epoch 2002 | loss: 0.0311503 | val_loss: 0.031146 | Time: 14168.4 ms
[2022-02-28 18:53:11	                main:574]	:	INFO	:	Epoch 2003 | loss: 0.0311431 | val_loss: 0.0311366 | Time: 3835.42 ms
[2022-02-28 18:53:15	                main:574]	:	INFO	:	Epoch 2004 | loss: 0.031133 | val_loss: 0.0311252 | Time: 3801.2 ms
[2022-02-28 18:53:19	                main:574]	:	INFO	:	Epoch 2005 | loss: 0.0311212 | val_loss: 0.0311192 | Time: 3854.09 ms
[2022-02-28 18:53:22	                main:574]	:	INFO	:	Epoch 2006 | loss: 0.0311133 | val_loss: 0.0311158 | Time: 3834.32 ms
[2022-02-28 18:53:26	                main:574]	:	INFO	:	Epoch 2007 | loss: 0.0311088 | val_loss: 0.0311073 | Time: 3854.18 ms
[2022-02-28 18:53:31	                main:574]	:	INFO	:	Epoch 2008 | loss: 0.031102 | val_loss: 0.0311042 | Time: 4917.01 ms
[2022-02-28 18:53:37	                main:574]	:	INFO	:	Epoch 2009 | loss: 0.0310988 | val_loss: 0.0311043 | Time: 5347.34 ms
[2022-02-28 18:53:42	                main:574]	:	INFO	:	Epoch 2010 | loss: 0.0310972 | val_loss: 0.0311028 | Time: 5115.55 ms
[2022-02-28 18:53:47	                main:574]	:	INFO	:	Epoch 2011 | loss: 0.0310966 | val_loss: 0.0311018 | Time: 4944.32 ms
[2022-02-28 18:53:52	                main:574]	:	INFO	:	Epoch 2012 | loss: 0.0310951 | val_loss: 0.0311012 | Time: 5084.86 ms
[2022-02-28 18:54:08	                main:574]	:	INFO	:	Epoch 2013 | loss: 0.0310943 | val_loss: 0.0311065 | Time: 16066.6 ms
[2022-02-28 18:54:12	                main:574]	:	INFO	:	Epoch 2014 | loss: 0.0310901 | val_loss: 0.0310975 | Time: 3814.02 ms
[2022-02-28 18:54:16	                main:574]	:	INFO	:	Epoch 2015 | loss: 0.031089 | val_loss: 0.0311019 | Time: 4296.97 ms
[2022-02-28 18:54:21	                main:574]	:	INFO	:	Epoch 2016 | loss: 0.031089 | val_loss: 0.0310904 | Time: 5199.35 ms
[2022-02-28 18:54:26	                main:574]	:	INFO	:	Epoch 2017 | loss: 0.0310763 | val_loss: 0.031084 | Time: 5168.95 ms
[2022-02-28 18:54:32	                main:574]	:	INFO	:	Epoch 2018 | loss: 0.0310676 | val_loss: 0.0310837 | Time: 5862.53 ms
[2022-02-28 18:54:37	                main:574]	:	INFO	:	Epoch 2019 | loss: 0.0310639 | val_loss: 0.0310758 | Time: 5217.35 ms
[2022-02-28 18:54:43	                main:574]	:	INFO	:	Epoch 2020 | loss: 0.0310568 | val_loss: 0.0310727 | Time: 5033.01 ms
[2022-02-28 18:54:48	                main:574]	:	INFO	:	Epoch 2021 | loss: 0.0310528 | val_loss: 0.0310701 | Time: 5362.03 ms
[2022-02-28 18:54:54	                main:574]	:	INFO	:	Epoch 2022 | loss: 0.0310509 | val_loss: 0.0310683 | Time: 5826.45 ms
[2022-02-28 18:55:09	                main:574]	:	INFO	:	Epoch 2023 | loss: 0.0310515 | val_loss: 0.0310696 | Time: 14896.6 ms
[2022-02-28 18:55:13	                main:574]	:	INFO	:	Epoch 2024 | loss: 0.0310549 | val_loss: 0.0310725 | Time: 4506.66 ms
[2022-02-28 18:55:19	                main:574]	:	INFO	:	Epoch 2025 | loss: 0.0310531 | val_loss: 0.0310705 | Time: 5892.61 ms
[2022-02-28 18:55:25	                main:574]	:	INFO	:	Epoch 2026 | loss: 0.0310523 | val_loss: 0.0310692 | Time: 6026.15 ms
[2022-02-28 18:55:31	                main:574]	:	INFO	:	Epoch 2027 | loss: 0.0310482 | val_loss: 0.0310681 | Time: 5839.63 ms
[2022-02-28 18:55:37	                main:574]	:	INFO	:	Epoch 2028 | loss: 0.0310492 | val_loss: 0.0310706 | Time: 5818.64 ms
[2022-02-28 18:55:42	                main:574]	:	INFO	:	Epoch 2029 | loss: 0.0310468 | val_loss: 0.0310711 | Time: 5448.04 ms
[2022-02-28 18:55:48	                main:574]	:	INFO	:	Epoch 2030 | loss: 0.0310437 | val_loss: 0.0310696 | Time: 5362.35 ms
[2022-02-28 18:55:53	                main:574]	:	INFO	:	Epoch 2031 | loss: 0.031042 | val_loss: 0.0310749 | Time: 5886.96 ms
[2022-02-28 18:56:09	                main:574]	:	INFO	:	Epoch 2032 | loss: 0.0310426 | val_loss: 0.0310739 | Time: 15309.9 ms
[2022-02-28 18:56:13	                main:574]	:	INFO	:	Epoch 2033 | loss: 0.0310416 | val_loss: 0.0310725 | Time: 3843.1 ms
[2022-02-28 18:56:17	                main:574]	:	INFO	:	Epoch 2034 | loss: 0.0310408 | val_loss: 0.0310763 | Time: 3864.55 ms
[2022-02-28 18:56:20	                main:574]	:	INFO	:	Epoch 2035 | loss: 0.0310368 | val_loss: 0.0310694 | Time: 3858.61 ms
[2022-02-28 18:56:25	                main:574]	:	INFO	:	Epoch 2036 | loss: 0.0310373 | val_loss: 0.0310677 | Time: 4437.44 ms
[2022-02-28 18:56:29	                main:574]	:	INFO	:	Epoch 2037 | loss: 0.0310367 | val_loss: 0.0310666 | Time: 4508.72 ms
[2022-02-28 18:56:34	                main:574]	:	INFO	:	Epoch 2038 | loss: 0.0310332 | val_loss: 0.0310677 | Time: 4341.2 ms
[2022-02-28 18:56:39	                main:574]	:	INFO	:	Epoch 2039 | loss: 0.0310294 | val_loss: 0.0310671 | Time: 4845.48 ms
[2022-02-28 18:56:43	                main:574]	:	INFO	:	Epoch 2040 | loss: 0.0310257 | val_loss: 0.0310702 | Time: 4339.65 ms
[2022-02-28 18:56:47	                main:574]	:	INFO	:	Epoch 2041 | loss: 0.0310242 | val_loss: 0.0310689 | Time: 4340.62 ms
[2022-02-28 18:56:52	                main:574]	:	INFO	:	Epoch 2042 | loss: 0.0310274 | val_loss: 0.0310653 | Time: 4625.95 ms
[2022-02-28 18:56:56	                main:574]	:	INFO	:	Epoch 2043 | loss: 0.0310271 | val_loss: 0.0310621 | Time: 4374.36 ms
[2022-02-28 18:57:10	                main:574]	:	INFO	:	Epoch 2044 | loss: 0.0310233 | val_loss: 0.0310631 | Time: 14003.7 ms
[2022-02-28 18:57:14	                main:574]	:	INFO	:	Epoch 2045 | loss: 0.0310271 | val_loss: 0.0310639 | Time: 3834.98 ms
[2022-02-28 18:57:18	                main:574]	:	INFO	:	Epoch 2046 | loss: 0.0310237 | val_loss: 0.0310655 | Time: 3859.29 ms
[2022-02-28 18:57:22	                main:574]	:	INFO	:	Epoch 2047 | loss: 0.0310192 | val_loss: 0.0310652 | Time: 3826.66 ms
[2022-02-28 18:57:26	                main:574]	:	INFO	:	Epoch 2048 | loss: 0.0310227 | val_loss: 0.0310627 | Time: 3882.7 ms
[2022-02-28 18:57:26	                main:597]	:	INFO	:	Saving trained model to model-final.pt, val_loss 0.0310627
[2022-02-28 18:57:26	                main:603]	:	INFO	:	Saving end state to config to file
[2022-02-28 18:57:26	                main:608]	:	INFO	:	Success, exiting..
18:57:26 (27408): called boinc_finish(0)

</stderr_txt>
]]>


©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)