Task 13341866

Name ParityModified-1645994450-5533-3-0_0
Workunit 10453235
Created 17 Mar 2022, 21:17:17 UTC
Sent 23 Mar 2022, 8:52:15 UTC
Report deadline 31 Mar 2022, 8:52:15 UTC
Received 24 May 2022, 2:12:38 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 10909
Run time 4 hours 41 min 17 sec
CPU time 3 hours 52 min 47 sec
Validate state Task was reported too late to validate
Credit 0.00
Device peak FLOPS 4,378.82 GFLOPS
Application version Machine Learning Dataset Generator (GPU) v9.75 (cuda10200)
windows_x86_64
Peak working set size 2.34 GB
Peak swap size 3.61 GB
Peak disk usage 1.54 GB

Stderr output

<core_client_version>7.16.11</core_client_version>
<![CDATA[
<stderr_txt>
       main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-13 20:09:14	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-13 20:09:14	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-13 20:09:14	                main:480]	:	INFO	:	    Patience: 10
[2022-05-13 20:09:14	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-13 20:09:15	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-13 20:09:15	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-13 20:09:15	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-13 20:09:15	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-13 20:09:15	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-13 20:09:15	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-13 20:10:49	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-13 20:10:49	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-13 20:10:49	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-13 20:10:49	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-13 20:10:49	                main:494]	:	INFO	:	Creating Model
[2022-05-13 20:10:49	                main:507]	:	INFO	:	Preparing config file
[2022-05-13 20:10:49	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-13 20:10:49	                main:512]	:	INFO	:	Loading config
[2022-05-13 20:10:50	                main:514]	:	INFO	:	Loading state
[2022-05-13 20:11:08	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-05-13 20:11:09	                main:562]	:	INFO	:	Starting Training
[2022-05-13 20:11:35	                main:574]	:	INFO	:	Epoch 1858 | loss: 0.0313392 | val_loss: 0.031208 | Time: 26758.6 ms
[2022-05-13 20:12:08	                main:574]	:	INFO	:	Epoch 1859 | loss: 0.0311174 | val_loss: 0.0311653 | Time: 32657.6 ms
[2022-05-13 20:12:21	                main:574]	:	INFO	:	Epoch 1860 | loss: 0.0310713 | val_loss: 0.0311865 | Time: 12620.4 ms
[2022-05-13 20:12:31	                main:574]	:	INFO	:	Epoch 1861 | loss: 0.0310611 | val_loss: 0.0312115 | Time: 10722.7 ms
[2022-05-13 20:12:42	                main:574]	:	INFO	:	Epoch 1862 | loss: 0.0310491 | val_loss: 0.0312219 | Time: 10541.7 ms
[2022-05-13 20:12:52	                main:574]	:	INFO	:	Epoch 1863 | loss: 0.0310402 | val_loss: 0.0312101 | Time: 10369 ms
[2022-05-13 20:13:03	                main:574]	:	INFO	:	Epoch 1864 | loss: 0.0310386 | val_loss: 0.0312081 | Time: 10631.7 ms
[2022-05-13 20:13:14	                main:574]	:	INFO	:	Epoch 1865 | loss: 0.0310303 | val_loss: 0.0312109 | Time: 10561.1 ms
[2022-05-13 20:13:35	                main:574]	:	INFO	:	Epoch 1866 | loss: 0.0310302 | val_loss: 0.0312076 | Time: 21088.8 ms
[2022-05-13 20:13:55	                main:574]	:	INFO	:	Epoch 1867 | loss: 0.0310281 | val_loss: 0.0312047 | Time: 20555.3 ms
[2022-05-13 20:14:16	                main:574]	:	INFO	:	Epoch 1868 | loss: 0.0310294 | val_loss: 0.0312018 | Time: 20456.1 ms
[2022-05-13 20:14:26	                main:574]	:	INFO	:	Epoch 1869 | loss: 0.0310226 | val_loss: 0.031202 | Time: 9822.37 ms
[2022-05-13 20:14:37	                main:574]	:	INFO	:	Epoch 1870 | loss: 0.0310175 | val_loss: 0.0312164 | Time: 10723.5 ms
[2022-05-13 20:15:05	                main:574]	:	INFO	:	Epoch 1871 | loss: 0.0310216 | val_loss: 0.0312223 | Time: 28062.2 ms
[2022-05-13 20:15:26	                main:574]	:	INFO	:	Epoch 1872 | loss: 0.0310255 | val_loss: 0.0312004 | Time: 21035.6 ms
[2022-05-13 20:15:37	                main:574]	:	INFO	:	Epoch 1873 | loss: 0.0310215 | val_loss: 0.031218 | Time: 11028.5 ms
[2022-05-13 20:15:59	                main:574]	:	INFO	:	Epoch 1874 | loss: 0.0310137 | val_loss: 0.0312201 | Time: 21765.4 ms
[2022-05-13 20:16:09	                main:574]	:	INFO	:	Epoch 1875 | loss: 0.0310152 | val_loss: 0.0312035 | Time: 10652.8 ms
[2022-05-13 20:16:19	                main:574]	:	INFO	:	Epoch 1876 | loss: 0.03102 | val_loss: 0.0312255 | Time: 10003.4 ms
[2022-05-13 20:16:30	                main:574]	:	INFO	:	Epoch 1877 | loss: 0.0310328 | val_loss: 0.0311897 | Time: 9963.65 ms
[2022-05-13 20:16:40	                main:574]	:	INFO	:	Epoch 1878 | loss: 0.0310335 | val_loss: 0.0312308 | Time: 10216.3 ms
[2022-05-13 20:16:50	                main:574]	:	INFO	:	Epoch 1879 | loss: 0.031022 | val_loss: 0.0312062 | Time: 10449.7 ms
[2022-05-13 20:17:01	                main:574]	:	INFO	:	Epoch 1880 | loss: 0.0310227 | val_loss: 0.0312253 | Time: 10250.4 ms
[2022-05-13 20:17:11	                main:574]	:	INFO	:	Epoch 1881 | loss: 0.0310127 | val_loss: 0.0312122 | Time: 10250.9 ms
[2022-05-13 20:17:21	                main:574]	:	INFO	:	Epoch 1882 | loss: 0.0310268 | val_loss: 0.0312134 | Time: 10067.5 ms
[2022-05-13 20:17:31	                main:574]	:	INFO	:	Epoch 1883 | loss: 0.0310424 | val_loss: 0.0311912 | Time: 9850.66 ms
[2022-05-13 20:17:41	                main:574]	:	INFO	:	Epoch 1884 | loss: 0.0310275 | val_loss: 0.0312064 | Time: 10057.2 ms
[2022-05-13 20:17:51	                main:574]	:	INFO	:	Epoch 1885 | loss: 0.0310166 | val_loss: 0.0312238 | Time: 10079.7 ms
[2022-05-13 20:18:01	                main:574]	:	INFO	:	Epoch 1886 | loss: 0.0310464 | val_loss: 0.031182 | Time: 10086.4 ms
[2022-05-13 20:18:11	                main:574]	:	INFO	:	Epoch 1887 | loss: 0.0310947 | val_loss: 0.0311733 | Time: 10021.8 ms
[2022-05-13 20:18:21	                main:574]	:	INFO	:	Epoch 1888 | loss: 0.0310837 | val_loss: 0.0311923 | Time: 9479.72 ms
[2022-05-13 20:18:31	                main:574]	:	INFO	:	Epoch 1889 | loss: 0.0310781 | val_loss: 0.0311967 | Time: 10095.4 ms
[2022-05-13 20:18:41	                main:574]	:	INFO	:	Epoch 1890 | loss: 0.0310636 | val_loss: 0.0311804 | Time: 9965.56 ms
[2022-05-13 20:18:53	                main:574]	:	INFO	:	Epoch 1891 | loss: 0.0310584 | val_loss: 0.0311919 | Time: 11517.6 ms
[2022-05-13 20:19:14	                main:574]	:	INFO	:	Epoch 1892 | loss: 0.0310509 | val_loss: 0.0312041 | Time: 21012.5 ms
[2022-05-13 20:19:24	                main:574]	:	INFO	:	Epoch 1893 | loss: 0.0310391 | val_loss: 0.0312182 | Time: 10022.6 ms
[2022-05-13 20:19:34	                main:574]	:	INFO	:	Epoch 1894 | loss: 0.031034 | val_loss: 0.0311838 | Time: 9866.24 ms
[2022-05-13 20:19:43	                main:574]	:	INFO	:	Epoch 1895 | loss: 0.0310282 | val_loss: 0.031201 | Time: 9613.65 ms
[2022-05-13 20:19:55	                main:574]	:	INFO	:	Epoch 1896 | loss: 0.0310247 | val_loss: 0.0312157 | Time: 11335.8 ms
[2022-05-13 20:20:04	                main:574]	:	INFO	:	Epoch 1897 | loss: 0.0310227 | val_loss: 0.0312139 | Time: 9448.1 ms
[2022-05-13 20:20:14	                main:574]	:	INFO	:	Epoch 1898 | loss: 0.0310209 | val_loss: 0.0312324 | Time: 9842.81 ms
[2022-05-13 20:20:23	                main:574]	:	INFO	:	Epoch 1899 | loss: 0.0310178 | val_loss: 0.0312134 | Time: 9282.64 ms
[2022-05-13 20:20:33	                main:574]	:	INFO	:	Epoch 1900 | loss: 0.0310198 | val_loss: 0.0312264 | Time: 9484.2 ms
[2022-05-13 20:20:43	                main:574]	:	INFO	:	Epoch 1901 | loss: 0.0310134 | val_loss: 0.031215 | Time: 9652.44 ms
[2022-05-13 20:20:52	                main:574]	:	INFO	:	Epoch 1902 | loss: 0.031023 | val_loss: 0.0312066 | Time: 9851.09 ms
[2022-05-13 20:21:02	                main:574]	:	INFO	:	Epoch 1903 | loss: 0.0310351 | val_loss: 0.0311966 | Time: 9935.75 ms
[2022-05-13 20:21:12	                main:574]	:	INFO	:	Epoch 1904 | loss: 0.0310353 | val_loss: 0.0312184 | Time: 9597.24 ms
[2022-05-13 20:21:22	                main:574]	:	INFO	:	Epoch 1905 | loss: 0.0310312 | val_loss: 0.0312074 | Time: 9778.68 ms
[2022-05-13 20:21:32	                main:574]	:	INFO	:	Epoch 1906 | loss: 0.0310279 | val_loss: 0.0312163 | Time: 10026.6 ms
[2022-05-13 20:21:42	                main:574]	:	INFO	:	Epoch 1907 | loss: 0.0310212 | val_loss: 0.0312297 | Time: 9780.67 ms
[2022-05-13 20:21:51	                main:574]	:	INFO	:	Epoch 1908 | loss: 0.0310122 | val_loss: 0.0312057 | Time: 9265.59 ms
[2022-05-13 20:22:01	                main:574]	:	INFO	:	Epoch 1909 | loss: 0.0310162 | val_loss: 0.031222 | Time: 10080.7 ms
[2022-05-13 20:22:11	                main:574]	:	INFO	:	Epoch 1910 | loss: 0.0310102 | val_loss: 0.031221 | Time: 9653.19 ms
[2022-05-13 20:22:21	                main:574]	:	INFO	:	Epoch 1911 | loss: 0.0310084 | val_loss: 0.0312322 | Time: 10015.2 ms
[2022-05-13 20:22:31	                main:574]	:	INFO	:	Epoch 1912 | loss: 0.031013 | val_loss: 0.0312195 | Time: 9835.36 ms
[2022-05-13 20:22:41	                main:574]	:	INFO	:	Epoch 1913 | loss: 0.031016 | val_loss: 0.031224 | Time: 10192.5 ms
[2022-05-13 20:22:51	                main:574]	:	INFO	:	Epoch 1914 | loss: 0.0310353 | val_loss: 0.0312223 | Time: 9565.47 ms
[2022-05-13 20:23:00	                main:574]	:	INFO	:	Epoch 1915 | loss: 0.0310379 | val_loss: 0.0312057 | Time: 9425.19 ms
[2022-05-13 20:23:10	                main:574]	:	INFO	:	Epoch 1916 | loss: 0.0310311 | val_loss: 0.0312088 | Time: 9870.6 ms
[2022-05-13 20:23:21	                main:574]	:	INFO	:	Epoch 1917 | loss: 0.0310222 | val_loss: 0.0312146 | Time: 10574.8 ms
[2022-05-13 20:23:31	                main:574]	:	INFO	:	Epoch 1918 | loss: 0.0310222 | val_loss: 0.0312031 | Time: 10147.6 ms
[2022-05-13 20:23:40	                main:574]	:	INFO	:	Epoch 1919 | loss: 0.0310176 | val_loss: 0.0312069 | Time: 9419.55 ms
[2022-05-13 20:23:49	                main:574]	:	INFO	:	Epoch 1920 | loss: 0.0310237 | val_loss: 0.0312393 | Time: 8718.87 ms
[2022-05-13 20:23:58	                main:574]	:	INFO	:	Epoch 1921 | loss: 0.0310259 | val_loss: 0.0311957 | Time: 8869.07 ms
[2022-05-13 20:24:07	                main:574]	:	INFO	:	Epoch 1922 | loss: 0.0310216 | val_loss: 0.0312175 | Time: 8771.1 ms
[2022-05-13 20:24:15	                main:574]	:	INFO	:	Epoch 1923 | loss: 0.0310176 | val_loss: 0.0312021 | Time: 8680.12 ms
[2022-05-13 20:24:24	                main:574]	:	INFO	:	Epoch 1924 | loss: 0.0310167 | val_loss: 0.0312163 | Time: 8871.37 ms
[2022-05-13 20:24:35	                main:574]	:	INFO	:	Epoch 1925 | loss: 0.0310136 | val_loss: 0.0312102 | Time: 10116.7 ms
Machine Learning Dataset Generator vMachine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-16 17:02:42	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-16 17:02:42	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-16 17:02:42	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-16 17:02:42	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-16 17:02:42	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-16 17:02:42	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-16 17:02:42	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-16 17:02:42	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-16 17:02:42	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-16 17:02:42	                main:474]	:	INFO	:	Configuration: 
[2022-05-16 17:02:42	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-16 17:02:42	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-16 17:02:42	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-16 17:02:42	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-16 17:02:42	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-16 17:02:42	                main:480]	:	INFO	:	    Patience: 10
[2022-05-16 17:02:42	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-16 17:02:42	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-16 17:02:42	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-16 17:02:42	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-16 17:02:42	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-16 17:02:43	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-16 17:11:00	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-16 17:11:00	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-16 17:11:00	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-16 17:11:00	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-16 17:11:00	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-16 17:11:00	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-16 17:11:01	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-16 17:11:01	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-16 17:11:01	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-16 17:11:01	                main:474]	:	INFO	:	Configuration: 
[2022-05-16 17:11:01	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-16 17:11:01	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-16 17:11:01	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-16 17:11:01	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-16 17:11:01	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-16 17:11:02	                main:480]	:	INFO	:	    Patience: 10
[2022-05-16 17:11:02	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-16 17:11:02	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-16 17:11:02	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-16 17:11:02	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-16 17:11:02	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-16 17:11:02	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-16 17:11:04	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-16 17:12:25	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-16 17:12:25	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-16 17:12:26	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-16 17:12:26	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-16 17:12:27	                main:494]	:	INFO	:	Creating Model
[2022-05-16 17:12:27	                main:507]	:	INFO	:	Preparing config file
[2022-05-16 17:12:27	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-16 17:12:27	                main:512]	:	INFO	:	Loading config
[2022-05-16 17:12:28	                main:514]	:	INFO	:	Loading state
[2022-05-16 17:12:58	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-05-16 17:13:00	                main:562]	:	INFO	:	Starting Training
[2022-05-16 17:13:26	                main:574]	:	INFO	:	Epoch 1926 | loss: 0.0311444 | val_loss: 0.0312224 | Time: 26509.3 ms
[2022-05-16 17:13:37	                main:574]	:	INFO	:	Epoch 1927 | loss: 0.0310588 | val_loss: 0.0312147 | Time: 9926.35 ms
[2022-05-16 17:13:46	                main:574]	:	INFO	:	Epoch 1928 | loss: 0.0310404 | val_loss: 0.0312105 | Time: 9282.37 ms
[2022-05-16 17:13:55	                main:574]	:	INFO	:	Epoch 1929 | loss: 0.0310335 | val_loss: 0.0311826 | Time: 9127.72 ms
[2022-05-16 17:14:05	                main:574]	:	INFO	:	Epoch 1930 | loss: 0.0310273 | val_loss: 0.0312096 | Time: 9459.76 ms
[2022-05-16 17:14:14	                main:574]	:	INFO	:	Epoch 1931 | loss: 0.0310178 | val_loss: 0.0312089 | Time: 9610.63 ms
[2022-05-16 17:14:35	                main:574]	:	INFO	:	Epoch 1932 | loss: 0.0310189 | val_loss: 0.031195 | Time: 20072.4 ms
[2022-05-16 17:14:55	                main:574]	:	INFO	:	Epoch 1933 | loss: 0.0310204 | val_loss: 0.0312498 | Time: 20041.5 ms
[2022-05-16 17:15:05	                main:574]	:	INFO	:	Epoch 1934 | loss: 0.0310345 | val_loss: 0.0312152 | Time: 9915.03 ms
[2022-05-16 17:15:26	                main:574]	:	INFO	:	Epoch 1935 | loss: 0.0310312 | val_loss: 0.0311993 | Time: 20536.7 ms
[2022-05-16 17:15:36	                main:574]	:	INFO	:	Epoch 1936 | loss: 0.0310253 | val_loss: 0.0312087 | Time: 9274.23 ms
[2022-05-16 17:15:46	                main:574]	:	INFO	:	Epoch 1937 | loss: 0.031021 | val_loss: 0.0312215 | Time: 9766.1 ms
[2022-05-16 17:15:55	                main:574]	:	INFO	:	Epoch 1938 | loss: 0.0310109 | val_loss: 0.0312225 | Time: 8794.42 ms
[2022-05-16 17:16:04	                main:574]	:	INFO	:	Epoch 1939 | loss: 0.0310068 | val_loss: 0.0312265 | Time: 8785.38 ms
[2022-05-16 17:16:13	                main:574]	:	INFO	:	Epoch 1940 | loss: 0.0310096 | val_loss: 0.0311923 | Time: 8410.92 ms
[2022-05-16 17:16:24	                main:574]	:	INFO	:	Epoch 1941 | loss: 0.0310108 | val_loss: 0.031198 | Time: 9398.85 ms
[2022-05-16 17:16:34	                main:574]	:	INFO	:	Epoch 1942 | loss: 0.0310129 | val_loss: 0.0312068 | Time: 9320.87 ms
[2022-05-16 17:16:43	                main:574]	:	INFO	:	Epoch 1943 | loss: 0.0310178 | val_loss: 0.0312187 | Time: 9071.74 ms
[2022-05-16 17:16:53	                main:574]	:	INFO	:	Epoch 1944 | loss: 0.0310217 | val_loss: 0.0312117 | Time: 8670.82 ms
[2022-05-16 17:17:02	                main:574]	:	INFO	:	Epoch 1945 | loss: 0.0310263 | val_loss: 0.0312133 | Time: 9036.01 ms
[2022-05-16 17:17:12	                main:574]	:	INFO	:	Epoch 1946 | loss: 0.0310165 | val_loss: 0.0312317 | Time: 8699.68 ms
[2022-05-16 17:17:22	                main:574]	:	INFO	:	Epoch 1947 | loss: 0.0310094 | val_loss: 0.0312211 | Time: 9116.56 ms
[2022-05-16 17:17:31	                main:574]	:	INFO	:	Epoch 1948 | loss: 0.0310137 | val_loss: 0.0312103 | Time: 8529.14 ms
[2022-05-16 17:17:41	                main:574]	:	INFO	:	Epoch 1949 | loss: 0.0310158 | val_loss: 0.0312118 | Time: 8238.46 ms
[2022-05-16 17:17:50	                main:574]	:	INFO	:	Epoch 1950 | loss: 0.0310169 | val_loss: 0.0312039 | Time: 8419.2 ms
[2022-05-16 17:17:59	                main:574]	:	INFO	:	Epoch 1951 | loss: 0.0310116 | val_loss: 0.031241 | Time: 8363.03 ms
[2022-05-16 17:18:09	                main:574]	:	INFO	:	Epoch 1952 | loss: 0.031014 | val_loss: 0.0312202 | Time: 9287.14 ms
[2022-05-16 17:18:19	                main:574]	:	INFO	:	Epoch 1953 | loss: 0.0310097 | val_loss: 0.0312079 | Time: 9257 ms
[2022-05-16 17:18:29	                main:574]	:	INFO	:	Epoch 1954 | loss: 0.031019 | val_loss: 0.0312428 | Time: 8907.84 ms
[2022-05-16 17:18:38	                main:574]	:	INFO	:	Epoch 1955 | loss: 0.0310074 | val_loss: 0.031213 | Time: 8683.06 ms
[2022-05-16 17:18:47	                main:574]	:	INFO	:	Epoch 1956 | loss: 0.0310138 | val_loss: 0.0312331 | Time: 8233.89 ms
[2022-05-16 17:18:56	                main:574]	:	INFO	:	Epoch 1957 | loss: 0.0310181 | val_loss: 0.0312193 | Time: 8222.23 ms
[2022-05-16 17:19:05	                main:574]	:	INFO	:	Epoch 1958 | loss: 0.0310159 | val_loss: 0.0312363 | Time: 8555.9 ms
[2022-05-16 17:19:14	                main:574]	:	INFO	:	Epoch 1959 | loss: 0.0310135 | val_loss: 0.0312138 | Time: 8170.54 ms
[2022-05-16 17:19:22	                main:574]	:	INFO	:	Epoch 1960 | loss: 0.0310083 | val_loss: 0.031208 | Time: 8270.53 ms
[2022-05-16 17:19:31	                main:574]	:	INFO	:	Epoch 1961 | loss: 0.0310088 | val_loss: 0.0312164 | Time: 8216.15 ms
[2022-05-16 17:19:40	                main:574]	:	INFO	:	Epoch 1962 | loss: 0.0310036 | val_loss: 0.0312167 | Time: 8129.56 ms
[2022-05-16 17:19:49	                main:574]	:	INFO	:	Epoch 1963 | loss: 0.0310037 | val_loss: 0.0312378 | Time: 8570.91 ms
[2022-05-16 17:19:58	                main:574]	:	INFO	:	Epoch 1964 | loss: 0.0310144 | val_loss: 0.0311842 | Time: 8166.6 ms
[2022-05-16 17:20:07	                main:574]	:	INFO	:	Epoch 1965 | loss: 0.0310187 | val_loss: 0.0312198 | Time: 8326.52 ms
[2022-05-16 17:20:16	                main:574]	:	INFO	:	Epoch 1966 | loss: 0.0310157 | val_loss: 0.0312282 | Time: 8332.3 ms
[2022-05-16 17:20:25	                main:574]	:	INFO	:	Epoch 1967 | loss: 0.0310242 | val_loss: 0.0312078 | Time: 8154.21 ms
[2022-05-16 17:20:34	                main:574]	:	INFO	:	Epoch 1968 | loss: 0.0310362 | val_loss: 0.0312118 | Time: 8128.1 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-16 18:47:55	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-16 18:47:56	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-16 18:47:56	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-16 18:47:56	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-16 18:47:57	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-16 18:47:57	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-16 18:47:57	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-16 18:47:57	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-16 18:47:58	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-16 18:47:58	                main:474]	:	INFO	:	Configuration: 
[2022-05-16 18:47:58	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-16 18:47:58	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-16 18:47:58	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-16 18:47:59	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-16 18:47:59	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-16 18:47:59	                main:480]	:	INFO	:	    Patience: 10
[2022-05-16 18:47:59	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-16 18:47:59	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-16 18:47:59	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-16 18:48:00	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-16 18:48:00	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-16 18:48:00	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-16 18:48:15	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-16 18:50:17	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-16 18:50:17	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-16 18:50:17	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-16 18:50:17	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-16 18:50:17	                main:494]	:	INFO	:	Creating Model
[2022-05-16 18:50:19	                main:507]	:	INFO	:	Preparing config file
[2022-05-16 18:50:19	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-16 18:50:19	                main:512]	:	INFO	:	Loading config
[2022-05-16 18:50:19	                main:514]	:	INFO	:	Loading state
[2022-05-16 18:51:53	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-05-16 18:51:54	                main:562]	:	INFO	:	Starting Training
[2022-05-16 18:53:12	                main:574]	:	INFO	:	Epoch 1940 | loss: 0.0311353 | val_loss: 0.0312206 | Time: 77805.4 ms
[2022-05-16 18:53:26	                main:574]	:	INFO	:	Epoch 1941 | loss: 0.0310426 | val_loss: 0.0312263 | Time: 9409.64 ms
[2022-05-16 18:53:35	                main:574]	:	INFO	:	Epoch 1942 | loss: 0.0310185 | val_loss: 0.0311979 | Time: 9189.89 ms
[2022-05-16 18:53:45	                main:574]	:	INFO	:	Epoch 1943 | loss: 0.0310185 | val_loss: 0.0312497 | Time: 9493.8 ms
[2022-05-16 18:53:55	                main:574]	:	INFO	:	Epoch 1944 | loss: 0.0310182 | val_loss: 0.0312322 | Time: 9907.83 ms
[2022-05-16 18:54:07	                main:574]	:	INFO	:	Epoch 1945 | loss: 0.0310086 | val_loss: 0.0312244 | Time: 10978 ms
[2022-05-16 18:54:17	                main:574]	:	INFO	:	Epoch 1946 | loss: 0.0310115 | val_loss: 0.0312373 | Time: 9726.66 ms
[2022-05-16 18:54:28	                main:574]	:	INFO	:	Epoch 1947 | loss: 0.0310097 | val_loss: 0.0312231 | Time: 10800.8 ms
[2022-05-16 18:54:38	                main:574]	:	INFO	:	Epoch 1948 | loss: 0.0310246 | val_loss: 0.03121 | Time: 9890.41 ms
[2022-05-16 18:54:49	                main:574]	:	INFO	:	Epoch 1949 | loss: 0.0310415 | val_loss: 0.0312094 | Time: 9894.6 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-16 19:36:12	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-16 19:36:12	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-16 19:36:12	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-16 19:36:12	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-16 19:36:12	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-16 19:36:12	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-16 19:36:12	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-16 19:36:12	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-16 19:36:13	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-16 19:36:13	                main:474]	:	INFO	:	Configuration: 
[2022-05-16 19:36:13	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-16 19:36:13	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-16 19:36:13	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-16 19:36:13	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-16 19:36:13	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-16 19:36:13	                main:480]	:	INFO	:	    Patience: 10
[2022-05-16 19:36:13	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-16 19:36:13	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-16 19:36:13	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-16 19:36:13	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-16 19:36:13	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-16 19:36:13	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-16 19:36:14	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-16 19:37:57	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-16 19:37:57	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-16 19:37:59	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-16 19:38:03	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-16 19:38:03	                main:494]	:	INFO	:	Creating Model
[2022-05-16 19:38:05	                main:507]	:	INFO	:	Preparing config file
[2022-05-16 19:38:05	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-16 19:38:06	                main:512]	:	INFO	:	Loading config
[2022-05-16 19:38:06	                main:514]	:	INFO	:	Loading state
[2022-05-16 19:39:39	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-05-16 19:39:41	                main:562]	:	INFO	:	Starting Training
[2022-05-16 19:40:53	                main:574]	:	INFO	:	Epoch 1941 | loss: 0.0311069 | val_loss: 0.031196 | Time: 71583.3 ms
[2022-05-16 19:41:04	                main:574]	:	INFO	:	Epoch 1942 | loss: 0.031044 | val_loss: 0.0312096 | Time: 10733.4 ms
[2022-05-16 19:41:15	                main:574]	:	INFO	:	Epoch 1943 | loss: 0.031027 | val_loss: 0.0312129 | Time: 10023.2 ms
[2022-05-16 19:41:27	                main:574]	:	INFO	:	Epoch 1944 | loss: 0.031038 | val_loss: 0.0312039 | Time: 9141.94 ms
[2022-05-16 19:41:37	                main:574]	:	INFO	:	Epoch 1945 | loss: 0.0310437 | val_loss: 0.0311922 | Time: 9211.38 ms
[2022-05-16 19:41:47	                main:574]	:	INFO	:	Epoch 1946 | loss: 0.031032 | val_loss: 0.0312132 | Time: 9313.03 ms
[2022-05-16 19:41:57	                main:574]	:	INFO	:	Epoch 1947 | loss: 0.0310214 | val_loss: 0.0312278 | Time: 9110.52 ms
[2022-05-16 19:42:08	                main:574]	:	INFO	:	Epoch 1948 | loss: 0.0310131 | val_loss: 0.031191 | Time: 10153.4 ms
[2022-05-16 19:42:46	                main:574]	:	INFO	:	Epoch 1949 | loss: 0.0310114 | val_loss: 0.0312032 | Time: 38378.8 ms
[2022-05-16 19:43:18	                main:574]	:	INFO	:	Epoch 1950 | loss: 0.0310132 | val_loss: 0.0312023 | Time: 30618.1 ms
[2022-05-16 19:43:59	                main:574]	:	INFO	:	Epoch 1951 | loss: 0.0310179 | val_loss: 0.0312048 | Time: 40546.6 ms
[2022-05-16 19:44:21	                main:574]	:	INFO	:	Epoch 1952 | loss: 0.031034 | val_loss: 0.0312071 | Time: 21699.6 ms
[2022-05-16 19:44:53	                main:574]	:	INFO	:	Epoch 1953 | loss: 0.0310522 | val_loss: 0.0311797 | Time: 31222.9 ms
[2022-05-16 19:45:25	                main:574]	:	INFO	:	Epoch 1954 | loss: 0.0310424 | val_loss: 0.0312178 | Time: 31908.5 ms
[2022-05-16 19:45:47	                main:574]	:	INFO	:	Epoch 1955 | loss: 0.0310216 | val_loss: 0.0312115 | Time: 21101.1 ms
[2022-05-16 19:46:20	                main:574]	:	INFO	:	Epoch 1956 | loss: 0.0310154 | val_loss: 0.0312123 | Time: 32282.9 ms
[2022-05-16 19:46:52	                main:574]	:	INFO	:	Epoch 1957 | loss: 0.0310234 | val_loss: 0.0312169 | Time: 31206 ms
[2022-05-16 19:47:44	                main:574]	:	INFO	:	Epoch 1958 | loss: 0.031018 | val_loss: 0.0312206 | Time: 51173.1 ms
[2022-05-16 19:48:24	                main:574]	:	INFO	:	Epoch 1959 | loss: 0.0310104 | val_loss: 0.0312031 | Time: 39530.9 ms
[2022-05-16 19:48:45	                main:574]	:	INFO	:	Epoch 1960 | loss: 0.0310146 | val_loss: 0.0312128 | Time: 21119.4 ms
[2022-05-16 19:49:16	                main:574]	:	INFO	:	Epoch 1961 | loss: 0.0310076 | val_loss: 0.0312288 | Time: 30621.5 ms
[2022-05-16 19:49:37	                main:574]	:	INFO	:	Epoch 1962 | loss: 0.0310089 | val_loss: 0.0312226 | Time: 20960.7 ms
[2022-05-16 19:49:54	                main:574]	:	INFO	:	Epoch 1963 | loss: 0.0310119 | val_loss: 0.0312207 | Time: 17503.8 ms
[2022-05-16 19:50:06	                main:574]	:	INFO	:	Epoch 1964 | loss: 0.0310122 | val_loss: 0.0311939 | Time: 11347.2 ms
[2022-05-16 19:50:16	                main:574]	:	INFO	:	Epoch 1965 | loss: 0.0310037 | val_loss: 0.0312147 | Time: 10366.6 ms
[2022-05-16 19:50:26	                main:574]	:	INFO	:	Epoch 1966 | loss: 0.0310047 | val_loss: 0.0312367 | Time: 9879.51 ms
[2022-05-16 19:50:37	                main:574]	:	INFO	:	Epoch 1967 | loss: 0.0310123 | val_loss: 0.0312114 | Time: 10350 ms
[2022-05-16 19:50:48	                main:574]	:	INFO	:	Epoch 1968 | loss: 0.0310071 | val_loss: 0.0312111 | Time: 11059.6 ms
[2022-05-16 19:51:00	                main:574]	:	INFO	:	Epoch 1969 | loss: 0.0310046 | val_loss: 0.0312039 | Time: 10914.1 ms
[2022-05-16 19:51:12	                main:574]	:	INFO	:	Epoch 1970 | loss: 0.0310021 | val_loss: 0.0312048 | Time: 12148.8 ms
[2022-05-16 19:51:23	                main:574]	:	INFO	:	Epoch 1971 | loss: 0.0310046 | val_loss: 0.0312259 | Time: 10886.7 ms
[2022-05-16 19:51:36	                main:574]	:	INFO	:	Epoch 1972 | loss: 0.0310132 | val_loss: 0.0312324 | Time: 12513.3 ms
[2022-05-16 19:51:47	                main:574]	:	INFO	:	Epoch 1973 | loss: 0.0310045 | val_loss: 0.0312233 | Time: 11330.6 ms
[2022-05-16 19:52:01	                main:574]	:	INFO	:	Epoch 1974 | loss: 0.0310029 | val_loss: 0.0312333 | Time: 13021.5 ms
[2022-05-16 19:52:23	                main:574]	:	INFO	:	Epoch 1975 | loss: 0.0310086 | val_loss: 0.0312303 | Time: 22438.4 ms
[2022-05-16 19:52:34	                main:574]	:	INFO	:	Epoch 1976 | loss: 0.0310081 | val_loss: 0.0312436 | Time: 10583.9 ms
[2022-05-16 19:52:47	                main:574]	:	INFO	:	Epoch 1977 | loss: 0.0310519 | val_loss: 0.0311924 | Time: 12450.7 ms
[2022-05-16 19:53:00	                main:574]	:	INFO	:	Epoch 1978 | loss: 0.0311855 | val_loss: 0.0311821 | Time: 12910.8 ms
[2022-05-16 19:53:11	                main:574]	:	INFO	:	Epoch 1979 | loss: 0.0311726 | val_loss: 0.0311659 | Time: 11376.4 ms
[2022-05-16 19:53:22	                main:574]	:	INFO	:	Epoch 1980 | loss: 0.0311603 | val_loss: 0.0311606 | Time: 10787.4 ms
[2022-05-16 19:53:33	                main:574]	:	INFO	:	Epoch 1981 | loss: 0.0311504 | val_loss: 0.0311587 | Time: 11169.3 ms
[2022-05-16 19:53:44	                main:574]	:	INFO	:	Epoch 1982 | loss: 0.0311429 | val_loss: 0.0311559 | Time: 10896.9 ms
[2022-05-16 19:53:55	                main:574]	:	INFO	:	Epoch 1983 | loss: 0.0311379 | val_loss: 0.0311541 | Time: 11193.9 ms
[2022-05-16 19:54:06	                main:574]	:	INFO	:	Epoch 1984 | loss: 0.0311341 | val_loss: 0.0311557 | Time: 11286.4 ms
[2022-05-16 19:54:18	                main:574]	:	INFO	:	Epoch 1985 | loss: 0.0311293 | val_loss: 0.0311567 | Time: 11734.5 ms
[2022-05-16 19:54:29	                main:574]	:	INFO	:	Epoch 1986 | loss: 0.0311258 | val_loss: 0.0311587 | Time: 10842.4 ms
[2022-05-16 19:54:41	                main:574]	:	INFO	:	Epoch 1987 | loss: 0.0311246 | val_loss: 0.0311579 | Time: 11718.6 ms
[2022-05-16 19:54:53	                main:574]	:	INFO	:	Epoch 1988 | loss: 0.0311224 | val_loss: 0.0311571 | Time: 12235 ms
[2022-05-16 19:55:06	                main:574]	:	INFO	:	Epoch 1989 | loss: 0.0311219 | val_loss: 0.0311551 | Time: 12397.7 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-17 18:42:07	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-17 18:42:07	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-17 18:42:08	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-17 18:42:08	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-17 18:42:08	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-17 18:42:08	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-17 18:42:08	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-17 18:42:08	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-17 18:42:08	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-17 18:42:08	                main:474]	:	INFO	:	Configuration: 
[2022-05-17 18:42:08	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-17 18:42:08	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-17 18:42:08	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-17 18:42:08	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-17 18:42:08	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-17 18:42:08	                main:480]	:	INFO	:	    Patience: 10
[2022-05-17 18:42:08	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-17 18:42:08	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-17 18:42:08	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-17 18:42:08	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-17 18:42:08	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-17 18:42:09	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-17 18:42:09	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-17 18:42:18	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-17 18:42:18	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-17 18:42:18	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-17 18:42:18	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-17 18:42:18	                main:494]	:	INFO	:	Creating Model
[2022-05-17 18:42:19	                main:507]	:	INFO	:	Preparing config file
[2022-05-17 18:42:19	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-17 18:42:19	                main:512]	:	INFO	:	Loading config
[2022-05-17 18:42:19	                main:514]	:	INFO	:	Loading state
[2022-05-17 18:42:38	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-05-17 18:42:39	                main:562]	:	INFO	:	Starting Training
[2022-05-17 18:43:40	                main:574]	:	INFO	:	Epoch 1972 | loss: 0.0310737 | val_loss: 0.0312174 | Time: 61382.5 ms
[2022-05-17 18:44:21	                main:574]	:	INFO	:	Epoch 1973 | loss: 0.0310244 | val_loss: 0.0312209 | Time: 41071.4 ms
[2022-05-17 18:45:01	                main:574]	:	INFO	:	Epoch 1974 | loss: 0.0310096 | val_loss: 0.0312004 | Time: 38968.1 ms
[2022-05-17 18:45:38	                main:574]	:	INFO	:	Epoch 1975 | loss: 0.0310082 | val_loss: 0.031214 | Time: 37283.5 ms
[2022-05-17 18:46:04	                main:574]	:	INFO	:	Epoch 1976 | loss: 0.0310072 | val_loss: 0.0312258 | Time: 26479.8 ms
[2022-05-17 18:46:43	                main:574]	:	INFO	:	Epoch 1977 | loss: 0.0310049 | val_loss: 0.031224 | Time: 38156.8 ms
[2022-05-17 18:47:19	                main:574]	:	INFO	:	Epoch 1978 | loss: 0.0310063 | val_loss: 0.0312381 | Time: 35923 ms
[2022-05-17 18:47:46	                main:574]	:	INFO	:	Epoch 1979 | loss: 0.031006 | val_loss: 0.0312166 | Time: 27480.9 ms
[2022-05-17 18:48:24	                main:574]	:	INFO	:	Epoch 1980 | loss: 0.031013 | val_loss: 0.0312041 | Time: 37030.5 ms
[2022-05-17 18:49:01	                main:574]	:	INFO	:	Epoch 1981 | loss: 0.0310153 | val_loss: 0.0312159 | Time: 37458.8 ms
[2022-05-17 18:49:40	                main:574]	:	INFO	:	Epoch 1982 | loss: 0.0310081 | val_loss: 0.0312197 | Time: 38498.5 ms
[2022-05-17 18:50:07	                main:574]	:	INFO	:	Epoch 1983 | loss: 0.0310129 | val_loss: 0.0312078 | Time: 27640.2 ms
[2022-05-17 18:50:44	                main:574]	:	INFO	:	Epoch 1984 | loss: 0.0310196 | val_loss: 0.031201 | Time: 36223.7 ms
[2022-05-17 18:51:21	                main:574]	:	INFO	:	Epoch 1985 | loss: 0.0310304 | val_loss: 0.0312119 | Time: 37836 ms
[2022-05-17 18:52:00	                main:574]	:	INFO	:	Epoch 1986 | loss: 0.0310215 | val_loss: 0.0312121 | Time: 38118 ms
[2022-05-17 18:52:27	                main:574]	:	INFO	:	Epoch 1987 | loss: 0.0310107 | val_loss: 0.0311966 | Time: 27169.5 ms
[2022-05-17 18:53:05	                main:574]	:	INFO	:	Epoch 1988 | loss: 0.0310154 | val_loss: 0.031228 | Time: 38130.4 ms
[2022-05-17 18:53:41	                main:574]	:	INFO	:	Epoch 1989 | loss: 0.0310246 | val_loss: 0.0311971 | Time: 35345.6 ms
[2022-05-17 18:54:09	                main:574]	:	INFO	:	Epoch 1990 | loss: 0.0310465 | val_loss: 0.0311822 | Time: 28422.3 ms
[2022-05-17 18:54:46	                main:574]	:	INFO	:	Epoch 1991 | loss: 0.0310344 | val_loss: 0.0312078 | Time: 37043.7 ms
[2022-05-17 18:55:24	                main:574]	:	INFO	:	Epoch 1992 | loss: 0.0310268 | val_loss: 0.0312062 | Time: 37560.2 ms
[2022-05-17 18:56:00	                main:574]	:	INFO	:	Epoch 1993 | loss: 0.0310168 | val_loss: 0.0312323 | Time: 36537.1 ms
[2022-05-17 18:56:29	                main:574]	:	INFO	:	Epoch 1994 | loss: 0.0310159 | val_loss: 0.0312066 | Time: 28314 ms
[2022-05-17 18:57:05	                main:574]	:	INFO	:	Epoch 1995 | loss: 0.0310219 | val_loss: 0.0311991 | Time: 36622.1 ms
[2022-05-17 18:57:42	                main:574]	:	INFO	:	Epoch 1996 | loss: 0.0310112 | val_loss: 0.0312091 | Time: 36419.6 ms
[2022-05-17 18:58:09	                main:574]	:	INFO	:	Epoch 1997 | loss: 0.031012 | val_loss: 0.0312061 | Time: 27298.4 ms
[2022-05-17 18:58:46	                main:574]	:	INFO	:	Epoch 1998 | loss: 0.0310121 | val_loss: 0.0312211 | Time: 36442.8 ms
[2022-05-17 18:59:28	                main:574]	:	INFO	:	Epoch 1999 | loss: 0.0310167 | val_loss: 0.0312209 | Time: 41810 ms
[2022-05-17 19:00:05	                main:574]	:	INFO	:	Epoch 2000 | loss: 0.0310119 | val_loss: 0.0312256 | Time: 37199.4 ms
[2022-05-17 19:00:42	                main:574]	:	INFO	:	Epoch 2001 | loss: 0.0310032 | val_loss: 0.0312104 | Time: 37074.7 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-17 19:06:20	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-17 19:06:20	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-17 19:06:20	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-17 19:06:20	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-17 19:06:20	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-17 19:06:20	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-17 19:06:20	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-17 19:06:20	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-17 19:06:20	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-17 19:06:20	                main:474]	:	INFO	:	Configuration: 
[2022-05-17 19:06:20	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-17 19:06:20	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-17 19:06:20	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-17 19:06:20	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-17 19:06:20	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-17 19:06:20	                main:480]	:	INFO	:	    Patience: 10
[2022-05-17 19:06:20	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-17 19:06:20	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-17 19:06:20	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-17 19:06:20	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-17 19:06:21	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-17 19:06:21	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-17 19:06:21	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-17 19:06:23	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-17 19:06:23	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-17 19:06:23	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-17 19:06:23	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-17 19:06:23	                main:494]	:	INFO	:	Creating Model
[2022-05-17 19:06:23	                main:507]	:	INFO	:	Preparing config file
[2022-05-17 19:06:23	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-17 19:06:24	                main:512]	:	INFO	:	Loading config
[2022-05-17 19:06:24	                main:514]	:	INFO	:	Loading state
[2022-05-17 19:06:25	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-05-17 19:06:25	                main:562]	:	INFO	:	Starting Training
[2022-05-17 19:06:30	                main:574]	:	INFO	:	Epoch 1997 | loss: 0.0310675 | val_loss: 0.0311961 | Time: 4975.54 ms
[2022-05-17 19:06:54	                main:574]	:	INFO	:	Epoch 1998 | loss: 0.0310334 | val_loss: 0.0312165 | Time: 24562.2 ms
[2022-05-17 19:07:32	                main:574]	:	INFO	:	Epoch 1999 | loss: 0.0310236 | val_loss: 0.0312335 | Time: 37796.6 ms
[2022-05-17 19:08:13	                main:574]	:	INFO	:	Epoch 2000 | loss: 0.0310181 | val_loss: 0.0312147 | Time: 40459.1 ms
[2022-05-17 19:08:51	                main:574]	:	INFO	:	Epoch 2001 | loss: 0.0310096 | val_loss: 0.0312049 | Time: 38284.4 ms
[2022-05-17 19:09:27	                main:574]	:	INFO	:	Epoch 2002 | loss: 0.0310091 | val_loss: 0.0312274 | Time: 36316.2 ms
[2022-05-17 19:09:54	                main:574]	:	INFO	:	Epoch 2003 | loss: 0.0310083 | val_loss: 0.0312127 | Time: 26024.7 ms
[2022-05-17 19:10:39	                main:574]	:	INFO	:	Epoch 2004 | loss: 0.031008 | val_loss: 0.0312147 | Time: 45797.8 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-17 19:18:29	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-17 19:18:29	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-17 19:18:29	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-17 19:18:29	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-17 19:18:29	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-17 19:18:29	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-17 19:18:29	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-17 19:18:29	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-17 19:18:29	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-17 19:18:29	                main:474]	:	INFO	:	Configuration: 
[2022-05-17 19:18:29	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-17 19:18:30	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-17 19:18:30	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-17 19:18:30	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-17 19:18:30	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-17 19:18:30	                main:480]	:	INFO	:	    Patience: 10
[2022-05-17 19:18:30	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-17 19:18:30	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-17 19:18:30	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-17 19:18:30	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-17 19:18:30	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-17 19:18:30	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-17 19:18:30	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-17 19:18:32	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-17 19:18:32	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-17 19:18:32	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-17 19:18:32	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-17 19:18:32	                main:494]	:	INFO	:	Creating Model
[2022-05-17 19:18:32	                main:507]	:	INFO	:	Preparing config file
[2022-05-17 19:18:32	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-17 19:18:33	                main:512]	:	INFO	:	Loading config
[2022-05-17 19:18:33	                main:514]	:	INFO	:	Loading state
[2022-05-17 19:18:33	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-05-17 19:18:33	                main:562]	:	INFO	:	Starting Training
[2022-05-17 19:18:39	                main:574]	:	INFO	:	Epoch 1997 | loss: 0.031076 | val_loss: 0.0312453 | Time: 5212.35 ms
[2022-05-17 19:18:59	                main:574]	:	INFO	:	Epoch 1998 | loss: 0.0310306 | val_loss: 0.0311921 | Time: 20447.1 ms
[2022-05-17 19:19:34	                main:574]	:	INFO	:	Epoch 1999 | loss: 0.0310154 | val_loss: 0.0312218 | Time: 34436.4 ms
[2022-05-17 19:19:58	                main:574]	:	INFO	:	Epoch 2000 | loss: 0.031008 | val_loss: 0.0312263 | Time: 23948.5 ms
[2022-05-17 19:20:22	                main:574]	:	INFO	:	Epoch 2001 | loss: 0.0310202 | val_loss: 0.0312106 | Time: 24210.7 ms
[2022-05-17 19:20:55	                main:574]	:	INFO	:	Epoch 2002 | loss: 0.031035 | val_loss: 0.0312318 | Time: 33298 ms
[2022-05-17 19:21:20	                main:574]	:	INFO	:	Epoch 2003 | loss: 0.0310212 | val_loss: 0.031228 | Time: 24685.8 ms
[2022-05-17 19:21:54	                main:574]	:	INFO	:	Epoch 2004 | loss: 0.0310285 | val_loss: 0.0312372 | Time: 34093.3 ms
[2022-05-17 19:22:19	                main:574]	:	INFO	:	Epoch 2005 | loss: 0.031027 | val_loss: 0.0312112 | Time: 24527.7 ms
[2022-05-17 19:22:43	                main:574]	:	INFO	:	Epoch 2006 | loss: 0.0310175 | val_loss: 0.031201 | Time: 23852.5 ms
[2022-05-17 19:23:22	                main:574]	:	INFO	:	Epoch 2007 | loss: 0.0310182 | val_loss: 0.0312459 | Time: 39081.4 ms
[2022-05-17 19:24:16	                main:574]	:	INFO	:	Epoch 2008 | loss: 0.0310223 | val_loss: 0.0311841 | Time: 53970.2 ms
[2022-05-17 19:24:42	                main:574]	:	INFO	:	Epoch 2009 | loss: 0.0310198 | val_loss: 0.0312375 | Time: 25852.8 ms
[2022-05-17 19:25:17	                main:574]	:	INFO	:	Epoch 2010 | loss: 0.0310165 | val_loss: 0.0312135 | Time: 34860.2 ms
[2022-05-17 19:25:52	                main:574]	:	INFO	:	Epoch 2011 | loss: 0.0310106 | val_loss: 0.0311971 | Time: 35532.3 ms
[2022-05-17 19:26:29	                main:574]	:	INFO	:	Epoch 2012 | loss: 0.0310083 | val_loss: 0.0312159 | Time: 36242.9 ms
[2022-05-17 19:26:54	                main:574]	:	INFO	:	Epoch 2013 | loss: 0.0310199 | val_loss: 0.0312273 | Time: 25053.2 ms
[2022-05-17 19:27:28	                main:574]	:	INFO	:	Epoch 2014 | loss: 0.031015 | val_loss: 0.0312078 | Time: 34343 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-18 16:59:01	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-18 16:59:01	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-18 16:59:01	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-18 16:59:01	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-18 16:59:01	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-18 16:59:01	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-18 16:59:01	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-18 16:59:01	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-18 16:59:01	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-18 16:59:01	                main:474]	:	INFO	:	Configuration: 
[2022-05-18 16:59:01	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-18 16:59:01	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-18 16:59:01	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-18 16:59:01	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-18 16:59:02	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-18 16:59:02	                main:480]	:	INFO	:	    Patience: 10
[2022-05-18 16:59:02	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-18 16:59:02	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-18 16:59:02	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-18 16:59:02	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-18 16:59:02	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-18 16:59:02	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-18 16:59:02	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-18 16:59:26	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-18 16:59:26	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-18 16:59:27	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-18 16:59:27	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-18 16:59:27	                main:494]	:	INFO	:	Creating Model
[2022-05-18 16:59:29	                main:507]	:	INFO	:	Preparing config file
[2022-05-18 16:59:29	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-18 16:59:29	                main:512]	:	INFO	:	Loading config
[2022-05-18 16:59:29	                main:514]	:	INFO	:	Loading state
Machine Learning Dataset Generator vMachine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-20 23:05:54	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-20 23:05:54	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-20 23:05:55	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-20 23:05:55	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-20 23:05:55	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-20 23:05:55	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-20 23:05:55	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-20 23:05:55	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-20 23:05:55	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-20 23:05:55	                main:474]	:	INFO	:	Configuration: 
[2022-05-20 23:05:55	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-20 23:05:55	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1060 6GB)
[2022-05-22 20:59:16	                main:435]	:	INFO	:	Set logging level to 1
[2022-05-22 20:59:16	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-05-22 20:59:16	                main:444]	:	INFO	:	Resolving all filenames
[2022-05-22 20:59:17	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-05-22 20:59:17	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-05-22 20:59:17	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-05-22 20:59:17	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-05-22 20:59:18	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-05-22 20:59:18	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-05-22 20:59:18	                main:474]	:	INFO	:	Configuration: 
[2022-05-22 20:59:19	                main:475]	:	INFO	:	    Model type: GRU
[2022-05-22 20:59:19	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-05-22 20:59:19	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-05-22 20:59:19	                main:478]	:	INFO	:	    Batch Size: 128
[2022-05-22 20:59:20	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-05-22 20:59:20	                main:480]	:	INFO	:	    Patience: 10
[2022-05-22 20:59:20	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-05-22 20:59:20	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-05-22 20:59:21	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-05-22 20:59:21	                main:484]	:	INFO	:	    # Threads: 1
[2022-05-22 20:59:21	                main:486]	:	INFO	:	Preparing Dataset
[2022-05-22 20:59:21	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-05-22 20:59:29	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-05-22 21:02:36	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-05-22 21:02:37	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-05-22 21:02:38	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-05-22 21:02:39	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-05-22 21:02:39	                main:494]	:	INFO	:	Creating Model
[2022-05-22 21:02:42	                main:507]	:	INFO	:	Preparing config file
[2022-05-22 21:02:43	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-05-22 21:02:43	                main:512]	:	INFO	:	Loading config
[2022-05-22 21:02:43	                main:514]	:	INFO	:	Loading state
[2022-05-22 21:05:00	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-05-22 21:05:01	                main:562]	:	INFO	:	Starting Training
[2022-05-22 21:06:04	                main:574]	:	INFO	:	Epoch 2009 | loss: 0.0310955 | val_loss: 0.0312266 | Time: 62411.3 ms
[2022-05-22 21:06:15	                main:574]	:	INFO	:	Epoch 2010 | loss: 0.0310388 | val_loss: 0.0312022 | Time: 9413.4 ms
[2022-05-22 21:06:25	                main:574]	:	INFO	:	Epoch 2011 | loss: 0.0310208 | val_loss: 0.0312211 | Time: 9398.29 ms
[2022-05-22 21:06:35	                main:574]	:	INFO	:	Epoch 2012 | loss: 0.0310102 | val_loss: 0.0311953 | Time: 9435.22 ms
[2022-05-22 21:06:44	                main:574]	:	INFO	:	Epoch 2013 | loss: 0.0310091 | val_loss: 0.0312037 | Time: 9312.13 ms
[2022-05-22 21:06:55	                main:574]	:	INFO	:	Epoch 2014 | loss: 0.0310124 | val_loss: 0.0312294 | Time: 9940.73 ms
[2022-05-22 21:07:05	                main:574]	:	INFO	:	Epoch 2015 | loss: 0.0310056 | val_loss: 0.0312042 | Time: 9548.45 ms
[2022-05-22 21:07:15	                main:574]	:	INFO	:	Epoch 2016 | loss: 0.0310019 | val_loss: 0.0312126 | Time: 9863.8 ms
[2022-05-22 21:07:26	                main:574]	:	INFO	:	Epoch 2017 | loss: 0.0310165 | val_loss: 0.0312039 | Time: 10135.9 ms
[2022-05-22 21:07:48	                main:574]	:	INFO	:	Epoch 2018 | loss: 0.0310289 | val_loss: 0.0312249 | Time: 21075.2 ms
[2022-05-22 21:07:57	                main:574]	:	INFO	:	Epoch 2019 | loss: 0.0310253 | val_loss: 0.0312079 | Time: 9192.93 ms
[2022-05-22 21:08:07	                main:574]	:	INFO	:	Epoch 2020 | loss: 0.0310148 | val_loss: 0.0312236 | Time: 9166.95 ms
[2022-05-22 21:08:16	                main:574]	:	INFO	:	Epoch 2021 | loss: 0.0310108 | val_loss: 0.0312192 | Time: 9422.09 ms
[2022-05-22 21:08:26	                main:574]	:	INFO	:	Epoch 2022 | loss: 0.0310065 | val_loss: 0.0312108 | Time: 8960.28 ms
[2022-05-22 21:08:36	                main:574]	:	INFO	:	Epoch 2023 | loss: 0.031007 | val_loss: 0.031216 | Time: 9835.61 ms
[2022-05-22 21:08:45	                main:574]	:	INFO	:	Epoch 2024 | loss: 0.0310001 | val_loss: 0.0312357 | Time: 8795.41 ms
[2022-05-22 21:08:54	                main:574]	:	INFO	:	Epoch 2025 | loss: 0.0310012 | val_loss: 0.0312307 | Time: 8908.94 ms
[2022-05-22 21:09:04	                main:574]	:	INFO	:	Epoch 2026 | loss: 0.0310006 | val_loss: 0.0312355 | Time: 8794.24 ms
[2022-05-22 21:09:13	                main:574]	:	INFO	:	Epoch 2027 | loss: 0.0310099 | val_loss: 0.0312038 | Time: 8845.37 ms
[2022-05-22 21:09:23	                main:574]	:	INFO	:	Epoch 2028 | loss: 0.0310138 | val_loss: 0.0312216 | Time: 9288.38 ms
[2022-05-22 21:09:32	                main:574]	:	INFO	:	Epoch 2029 | loss: 0.0310155 | val_loss: 0.0312121 | Time: 8531.8 ms
[2022-05-22 21:09:41	                main:574]	:	INFO	:	Epoch 2030 | loss: 0.0310087 | val_loss: 0.0312082 | Time: 8930.19 ms
[2022-05-22 21:09:51	                main:574]	:	INFO	:	Epoch 2031 | loss: 0.0310059 | val_loss: 0.0312265 | Time: 8876.92 ms
[2022-05-22 21:09:59	                main:574]	:	INFO	:	Epoch 2032 | loss: 0.0310019 | val_loss: 0.031198 | Time: 8288.74 ms
[2022-05-22 21:10:09	                main:574]	:	INFO	:	Epoch 2033 | loss: 0.0310043 | val_loss: 0.0312206 | Time: 8624.2 ms
[2022-05-22 21:10:18	                main:574]	:	INFO	:	Epoch 2034 | loss: 0.0310005 | val_loss: 0.0312101 | Time: 8612.14 ms
[2022-05-22 21:10:27	                main:574]	:	INFO	:	Epoch 2035 | loss: 0.0310079 | val_loss: 0.0312377 | Time: 8750.21 ms
[2022-05-22 21:10:36	                main:574]	:	INFO	:	Epoch 2036 | loss: 0.0310054 | val_loss: 0.0312386 | Time: 8046.88 ms
[2022-05-22 21:10:45	                main:574]	:	INFO	:	Epoch 2037 | loss: 0.0310033 | val_loss: 0.0312063 | Time: 8536.34 ms
[2022-05-22 21:10:54	                main:574]	:	INFO	:	Epoch 2038 | loss: 0.0310153 | val_loss: 0.0312136 | Time: 8416.24 ms
[2022-05-22 21:11:02	                main:574]	:	INFO	:	Epoch 2039 | loss: 0.031017 | val_loss: 0.0312151 | Time: 8282.45 ms
[2022-05-22 21:11:11	                main:574]	:	INFO	:	Epoch 2040 | loss: 0.0310137 | val_loss: 0.0312017 | Time: 7905.06 ms
[2022-05-22 21:11:19	                main:574]	:	INFO	:	Epoch 2041 | loss: 0.0310147 | val_loss: 0.0312506 | Time: 8232.7 ms
[2022-05-22 21:11:28	                main:574]	:	INFO	:	Epoch 2042 | loss: 0.0310135 | val_loss: 0.0312104 | Time: 8044.02 ms
[2022-05-22 21:11:37	                main:574]	:	INFO	:	Epoch 2043 | loss: 0.0310162 | val_loss: 0.03121 | Time: 8122.53 ms
[2022-05-22 21:11:45	                main:574]	:	INFO	:	Epoch 2044 | loss: 0.03101 | val_loss: 0.0312124 | Time: 7721.23 ms
[2022-05-22 21:11:54	                main:574]	:	INFO	:	Epoch 2045 | loss: 0.0310074 | val_loss: 0.0312429 | Time: 8175.9 ms
[2022-05-22 21:12:02	                main:574]	:	INFO	:	Epoch 2046 | loss: 0.031013 | val_loss: 0.0312362 | Time: 8114.44 ms
[2022-05-22 21:12:11	                main:574]	:	INFO	:	Epoch 2047 | loss: 0.0310133 | val_loss: 0.0312127 | Time: 8078.19 ms
[2022-05-22 21:12:19	                main:574]	:	INFO	:	Epoch 2048 | loss: 0.0310174 | val_loss: 0.0311993 | Time: 7604.66 ms
[2022-05-22 21:12:20	                main:597]	:	INFO	:	Saving trained model to model-final.pt, val_loss 0.0311993
[2022-05-22 21:12:20	                main:603]	:	INFO	:	Saving end state to config to file
[2022-05-22 21:12:20	                main:608]	:	INFO	:	Success, exiting..
21:12:20 (5000): called boinc_finish(0)

</stderr_txt>
]]>


©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)