Task 12863575
| Name | ParityModified-1645997006-4934-2-0_0 |
| Workunit | 10003115 |
| Created | 10 Mar 2022, 7:05:17 UTC |
| Sent | 10 Mar 2022, 11:27:31 UTC |
| Report deadline | 18 Mar 2022, 11:27:31 UTC |
| Received | 15 Apr 2022, 6:00:46 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 17083 |
| Run time | 5 hours 6 min 28 sec |
| CPU time | 4 hours 52 min 13 sec |
| Validate state | Task was reported too late to validate |
| Credit | 0.00 |
| Device peak FLOPS | 3,197.72 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 2.08 GB |
| Peak swap size | 4.30 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.20</core_client_version> <![CDATA[ <stderr_txt> 283 | val_loss: 0.0311582 | Time: 6382.17 ms [2022-04-13 22:57:07 main:574] : INFO : Epoch 1772 | loss: 0.0311286 | val_loss: 0.0311562 | Time: 6207.86 ms [2022-04-13 22:57:14 main:574] : INFO : Epoch 1773 | loss: 0.0311273 | val_loss: 0.0311536 | Time: 6324.8 ms [2022-04-13 22:57:20 main:574] : INFO : Epoch 1774 | loss: 0.0311283 | val_loss: 0.0311552 | Time: 6456.84 ms [2022-04-13 22:57:27 main:574] : INFO : Epoch 1775 | loss: 0.0311275 | val_loss: 0.0311553 | Time: 6360.58 ms [2022-04-13 22:57:33 main:574] : INFO : Epoch 1776 | loss: 0.0311292 | val_loss: 0.0311534 | Time: 6241.34 ms [2022-04-13 22:57:39 main:574] : INFO : Epoch 1777 | loss: 0.0311305 | val_loss: 0.0311532 | Time: 6033 ms [2022-04-13 22:57:45 main:574] : INFO : Epoch 1778 | loss: 0.0311286 | val_loss: 0.031156 | Time: 6547.81 ms [2022-04-13 22:57:52 main:574] : INFO : Epoch 1779 | loss: 0.0311263 | val_loss: 0.031155 | Time: 6273.97 ms [2022-04-13 22:57:58 main:574] : INFO : Epoch 1780 | loss: 0.0311263 | val_loss: 0.0311567 | Time: 6149.73 ms [2022-04-13 22:58:04 main:574] : INFO : Epoch 1781 | loss: 0.0311279 | val_loss: 0.0311596 | Time: 6214.47 ms [2022-04-13 22:58:10 main:574] : INFO : Epoch 1782 | loss: 0.0311294 | val_loss: 0.03116 | Time: 6095.43 ms [2022-04-13 22:58:17 main:574] : INFO : Epoch 1783 | loss: 0.0311281 | val_loss: 0.0311611 | Time: 6389.73 ms [2022-04-13 22:58:23 main:574] : INFO : Epoch 1784 | loss: 0.031127 | val_loss: 0.0311577 | Time: 6509.94 ms [2022-04-13 22:58:30 main:574] : INFO : Epoch 1785 | loss: 0.0311239 | val_loss: 0.0311615 | Time: 6435.55 ms [2022-04-13 22:58:36 main:574] : INFO : Epoch 1786 | loss: 0.0311245 | val_loss: 0.0311589 | Time: 6194.25 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650) [2022-04-14 00:29:39 main:435] : INFO : Set logging level to 1 [2022-04-14 00:29:39 main:441] : INFO : Running in BOINC Client mode [2022-04-14 00:29:39 main:444] : INFO : Resolving all filenames [2022-04-14 00:29:39 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 00:29:39 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 00:29:39 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 00:29:39 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 00:29:39 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 00:29:39 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 00:29:39 main:474] : INFO : Configuration: [2022-04-14 00:29:39 main:475] : INFO : Model type: GRU [2022-04-14 00:29:39 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 00:29:39 main:477] : INFO : Max Epochs: 2048 [2022-04-14 00:29:39 main:478] : INFO : Batch Size: 128 [2022-04-14 00:29:39 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 00:29:39 main:480] : INFO : Patience: 10 [2022-04-14 00:29:39 main:481] : INFO : Hidden Width: 12 [2022-04-14 00:29:39 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 00:29:39 main:483] : INFO : # Backend Layers: 4 [2022-04-14 00:29:39 main:484] : INFO : # Threads: 1 [2022-04-14 00:29:39 main:486] : INFO : Preparing Dataset [2022-04-14 00:29:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 00:29:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 00:29:41 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 00:29:41 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 00:29:41 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 00:29:41 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 00:29:41 main:494] : INFO : Creating Model [2022-04-14 00:29:41 main:507] : INFO : Preparing config file [2022-04-14 00:29:41 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 00:29:41 main:512] : INFO : Loading config [2022-04-14 00:29:41 main:514] : INFO : Loading state [2022-04-14 00:29:43 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 00:29:43 main:562] : INFO : Starting Training [2022-04-14 00:29:48 main:574] : INFO : Epoch 1782 | loss: 0.0311615 | val_loss: 0.0311597 | Time: 4833.15 ms [2022-04-14 00:29:52 main:574] : INFO : Epoch 1783 | loss: 0.0311302 | val_loss: 0.0311569 | Time: 4432.02 ms [2022-04-14 00:29:57 main:574] : INFO : Epoch 1784 | loss: 0.0311282 | val_loss: 0.0311533 | Time: 4542.86 ms [2022-04-14 00:30:01 main:574] : INFO : Epoch 1785 | loss: 0.0311287 | val_loss: 0.0311537 | Time: 4849.12 ms [2022-04-14 00:30:07 main:574] : INFO : Epoch 1786 | loss: 0.0311288 | val_loss: 0.031152 | Time: 5029 ms [2022-04-14 00:30:11 main:574] : INFO : Epoch 1787 | loss: 0.0311252 | val_loss: 0.0311533 | Time: 4925.73 ms [2022-04-14 00:30:17 main:574] : INFO : Epoch 1788 | loss: 0.0311238 | val_loss: 0.0311528 | Time: 5075.49 ms [2022-04-14 00:30:21 main:574] : INFO : Epoch 1789 | loss: 0.0311242 | val_loss: 0.0311541 | Time: 4931.45 ms [2022-04-14 00:30:26 main:574] : INFO : Epoch 1790 | loss: 0.0311251 | val_loss: 0.0311544 | Time: 4822.85 ms [2022-04-14 00:30:31 main:574] : INFO : Epoch 1791 | loss: 0.0311235 | val_loss: 0.0311524 | Time: 5104.12 ms [2022-04-14 00:30:36 main:574] : INFO : Epoch 1792 | loss: 0.031125 | val_loss: 0.0311555 | Time: 4875.63 ms [2022-04-14 00:30:41 main:574] : INFO : Epoch 1793 | loss: 0.0311259 | val_loss: 0.0311546 | Time: 5122.1 ms [2022-04-14 00:30:46 main:574] : INFO : Epoch 1794 | loss: 0.0311323 | val_loss: 0.0311591 | Time: 4966.7 ms [2022-04-14 00:30:52 main:574] : INFO : Epoch 1795 | loss: 0.0311339 | val_loss: 0.0311578 | Time: 5172.76 ms [2022-04-14 00:30:57 main:574] : INFO : Epoch 1796 | loss: 0.0311342 | val_loss: 0.0311548 | Time: 4985.54 ms [2022-04-14 00:31:02 main:574] : INFO : Epoch 1797 | loss: 0.0311325 | val_loss: 0.0311525 | Time: 5084.67 ms [2022-04-14 00:31:07 main:574] : INFO : Epoch 1798 | loss: 0.0311292 | val_loss: 0.0311506 | Time: 5037.28 ms [2022-04-14 00:31:12 main:574] : INFO : Epoch 1799 | loss: 0.0311287 | val_loss: 0.0311529 | Time: 5206.06 ms [2022-04-14 00:31:17 main:574] : INFO : Epoch 1800 | loss: 0.0311317 | val_loss: 0.0311627 | Time: 5148.7 ms [2022-04-14 00:31:23 main:574] : INFO : Epoch 1801 | loss: 0.0311304 | val_loss: 0.031153 | Time: 5657.62 ms [2022-04-14 00:31:30 main:574] : INFO : Epoch 1802 | loss: 0.0311256 | val_loss: 0.0311535 | Time: 7180.41 ms [2022-04-14 00:31:37 main:574] : INFO : Epoch 1803 | loss: 0.0311242 | val_loss: 0.0311557 | Time: 7214.24 ms [2022-04-14 00:31:44 main:574] : INFO : Epoch 1804 | loss: 0.0311235 | val_loss: 0.0311566 | Time: 7092 ms [2022-04-14 00:31:51 main:574] : INFO : Epoch 1805 | loss: 0.031125 | val_loss: 0.0311518 | Time: 6819.94 ms [2022-04-14 00:31:58 main:574] : INFO : Epoch 1806 | loss: 0.0311244 | val_loss: 0.0311538 | Time: 7127.55 ms [2022-04-14 00:32:05 main:574] : INFO : Epoch 1807 | loss: 0.0311258 | val_loss: 0.0311544 | Time: 7056.33 ms [2022-04-14 00:32:12 main:574] : INFO : Epoch 1808 | loss: 0.0311226 | val_loss: 0.031155 | Time: 7065.35 ms [2022-04-14 00:32:19 main:574] : INFO : Epoch 1809 | loss: 0.0311221 | val_loss: 0.0311547 | Time: 6779.73 ms [2022-04-14 00:32:26 main:574] : INFO : Epoch 1810 | loss: 0.0311226 | val_loss: 0.0311518 | Time: 6798.98 ms [2022-04-14 00:32:33 main:574] : INFO : Epoch 1811 | loss: 0.0311253 | val_loss: 0.031154 | Time: 7264.8 ms [2022-04-14 00:32:40 main:574] : INFO : Epoch 1812 | loss: 0.0311245 | val_loss: 0.0311517 | Time: 6786.05 ms [2022-04-14 00:32:46 main:574] : INFO : Epoch 1813 | loss: 0.0311271 | val_loss: 0.0311533 | Time: 6320.63 ms [2022-04-14 00:32:52 main:574] : INFO : Epoch 1814 | loss: 0.0311258 | val_loss: 0.0311527 | Time: 5916.17 ms [2022-04-14 00:32:58 main:574] : INFO : Epoch 1815 | loss: 0.031124 | val_loss: 0.0311536 | Time: 5813.9 ms [2022-04-14 00:33:04 main:574] : INFO : Epoch 1816 | loss: 0.0311268 | val_loss: 0.0311545 | Time: 6030.56 ms [2022-04-14 00:33:10 main:574] : INFO : Epoch 1817 | loss: 0.0311285 | val_loss: 0.0311574 | Time: 5869.09 ms [2022-04-14 00:33:16 main:574] : INFO : Epoch 1818 | loss: 0.0311303 | val_loss: 0.0311549 | Time: 6195.3 ms [2022-04-14 00:33:22 main:574] : INFO : Epoch 1819 | loss: 0.031129 | val_loss: 0.0311545 | Time: 5765.57 ms [2022-04-14 00:33:28 main:574] : INFO : Epoch 1820 | loss: 0.0311283 | val_loss: 0.0311566 | Time: 6187.42 ms [2022-04-14 00:33:35 main:574] : INFO : Epoch 1821 | loss: 0.0311287 | val_loss: 0.031156 | Time: 5922.51 ms [2022-04-14 00:33:41 main:574] : INFO : Epoch 1822 | loss: 0.0311279 | val_loss: 0.0311529 | Time: 5973.79 ms [2022-04-14 00:33:47 main:574] : INFO : Epoch 1823 | loss: 0.0311274 | val_loss: 0.0311509 | Time: 5905.25 ms [2022-04-14 00:33:53 main:574] : INFO : Epoch 1824 | loss: 0.031126 | val_loss: 0.0311596 | Time: 5978 ms [2022-04-14 00:33:59 main:574] : INFO : Epoch 1825 | loss: 0.0311495 | val_loss: 0.0311664 | Time: 5850.01 ms [2022-04-14 00:34:06 main:574] : INFO : Epoch 1826 | loss: 0.0311593 | val_loss: 0.0311712 | Time: 5726.02 ms [2022-04-14 00:34:11 main:574] : INFO : Epoch 1827 | loss: 0.0311589 | val_loss: 0.0311664 | Time: 5818.29 ms [2022-04-14 00:34:17 main:574] : INFO : Epoch 1828 | loss: 0.0311573 | val_loss: 0.0311677 | Time: 5692.88 ms [2022-04-14 00:34:23 main:574] : INFO : Epoch 1829 | loss: 0.0311564 | val_loss: 0.0311605 | Time: 5814.44 ms [2022-04-14 00:34:29 main:574] : INFO : Epoch 1830 | loss: 0.0311512 | val_loss: 0.0311597 | Time: 6123 ms [2022-04-14 00:34:35 main:574] : INFO : Epoch 1831 | loss: 0.0311492 | val_loss: 0.0311591 | Time: 5950.03 ms [2022-04-14 00:34:41 main:574] : INFO : Epoch 1832 | loss: 0.0311469 | val_loss: 0.0311553 | Time: 5896.68 ms [2022-04-14 00:34:47 main:574] : INFO : Epoch 1833 | loss: 0.0311462 | val_loss: 0.0311572 | Time: 5691.01 ms [2022-04-14 00:34:53 main:574] : INFO : Epoch 1834 | loss: 0.0311451 | val_loss: 0.0311536 | Time: 6000.57 ms [2022-04-14 00:34:59 main:574] : INFO : Epoch 1835 | loss: 0.0311494 | val_loss: 0.0311555 | Time: 5901.83 ms [2022-04-14 00:35:05 main:574] : INFO : Epoch 1836 | loss: 0.0311482 | val_loss: 0.0311544 | Time: 5904.86 ms [2022-04-14 00:35:10 main:574] : INFO : Epoch 1837 | loss: 0.0311452 | val_loss: 0.0311514 | Time: 5987.13 ms [2022-04-14 00:35:17 main:574] : INFO : Epoch 1838 | loss: 0.0311476 | val_loss: 0.0311652 | Time: 6016.25 ms [2022-04-14 00:35:23 main:574] : INFO : Epoch 1839 | loss: 0.0311507 | val_loss: 0.031161 | Time: 6087.72 ms [2022-04-14 00:35:29 main:574] : INFO : Epoch 1840 | loss: 0.0311484 | val_loss: 0.0311576 | Time: 5914.5 ms [2022-04-14 00:35:35 main:574] : INFO : Epoch 1841 | loss: 0.0311475 | val_loss: 0.0311544 | Time: 6113.12 ms [2022-04-14 00:35:41 main:574] : INFO : Epoch 1842 | loss: 0.0311445 | val_loss: 0.0311519 | Time: 6246.77 ms [2022-04-14 00:35:47 main:574] : INFO : Epoch 1843 | loss: 0.0311433 | val_loss: 0.0311515 | Time: 5898.36 ms [2022-04-14 00:35:53 main:574] : INFO : Epoch 1844 | loss: 0.0311432 | val_loss: 0.0311492 | Time: 6167.19 ms [2022-04-14 00:35:59 main:574] : INFO : Epoch 1845 | loss: 0.0311423 | val_loss: 0.0311546 | Time: 6217.01 ms [2022-04-14 00:36:05 main:574] : INFO : Epoch 1846 | loss: 0.0311444 | val_loss: 0.0311553 | Time: 6189.04 ms [2022-04-14 00:36:12 main:574] : INFO : Epoch 1847 | loss: 0.0311421 | val_loss: 0.0311526 | Time: 6054.11 ms [2022-04-14 00:36:18 main:574] : INFO : Epoch 1848 | loss: 0.0311435 | val_loss: 0.031162 | Time: 6009.14 ms [2022-04-14 00:36:24 main:574] : INFO : Epoch 1849 | loss: 0.0311406 | val_loss: 0.0311531 | Time: 6230.71 ms [2022-04-14 00:36:30 main:574] : INFO : Epoch 1850 | loss: 0.0311382 | val_loss: 0.0311507 | Time: 5898.03 ms [2022-04-14 00:36:36 main:574] : INFO : Epoch 1851 | loss: 0.0311377 | val_loss: 0.0311514 | Time: 6185 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650) [2022-04-14 02:16:20 main:435] : INFO : Set logging level to 1 [2022-04-14 02:16:20 main:441] : INFO : Running in BOINC Client mode [2022-04-14 02:16:20 main:444] : INFO : Resolving all filenames [2022-04-14 02:16:20 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 02:16:20 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 02:16:20 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 02:16:20 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 02:16:20 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 02:16:20 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 02:16:20 main:474] : INFO : Configuration: [2022-04-14 02:16:20 main:475] : INFO : Model type: GRU [2022-04-14 02:16:20 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 02:16:20 main:477] : INFO : Max Epochs: 2048 [2022-04-14 02:16:20 main:478] : INFO : Batch Size: 128 [2022-04-14 02:16:20 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 02:16:20 main:480] : INFO : Patience: 10 [2022-04-14 02:16:20 main:481] : INFO : Hidden Width: 12 [2022-04-14 02:16:20 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 02:16:20 main:483] : INFO : # Backend Layers: 4 [2022-04-14 02:16:20 main:484] : INFO : # Threads: 1 [2022-04-14 02:16:20 main:486] : INFO : Preparing Dataset [2022-04-14 02:16:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 02:16:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 02:16:23 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 02:16:23 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 02:16:23 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 02:16:23 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 02:16:23 main:494] : INFO : Creating Model [2022-04-14 02:16:23 main:507] : INFO : Preparing config file [2022-04-14 02:16:23 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 02:16:23 main:512] : INFO : Loading config [2022-04-14 02:16:23 main:514] : INFO : Loading state [2022-04-14 02:16:24 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 02:16:24 main:562] : INFO : Starting Training [2022-04-14 02:16:29 main:574] : INFO : Epoch 1849 | loss: 0.031164 | val_loss: 0.0311535 | Time: 4918.17 ms [2022-04-14 02:16:34 main:574] : INFO : Epoch 1850 | loss: 0.0311416 | val_loss: 0.0311491 | Time: 4885.02 ms [2022-04-14 02:16:39 main:574] : INFO : Epoch 1851 | loss: 0.0311369 | val_loss: 0.031148 | Time: 5006.05 ms [2022-04-14 02:16:44 main:574] : INFO : Epoch 1852 | loss: 0.0311354 | val_loss: 0.0311468 | Time: 5019.83 ms [2022-04-14 02:16:49 main:574] : INFO : Epoch 1853 | loss: 0.0311334 | val_loss: 0.031149 | Time: 4966.87 ms [2022-04-14 02:16:54 main:574] : INFO : Epoch 1854 | loss: 0.0311347 | val_loss: 0.0311503 | Time: 5064.08 ms [2022-04-14 02:16:59 main:574] : INFO : Epoch 1855 | loss: 0.0311347 | val_loss: 0.0311482 | Time: 4950.03 ms [2022-04-14 02:17:04 main:574] : INFO : Epoch 1856 | loss: 0.0311361 | val_loss: 0.0311515 | Time: 5150.29 ms [2022-04-14 02:17:09 main:574] : INFO : Epoch 1857 | loss: 0.0311367 | val_loss: 0.0311511 | Time: 5140.02 ms [2022-04-14 02:17:15 main:574] : INFO : Epoch 1858 | loss: 0.0311362 | val_loss: 0.031152 | Time: 5893.67 ms [2022-04-14 02:17:21 main:574] : INFO : Epoch 1859 | loss: 0.0311346 | val_loss: 0.0311454 | Time: 6241.37 ms [2022-04-14 02:17:28 main:574] : INFO : Epoch 1860 | loss: 0.0311328 | val_loss: 0.0311476 | Time: 6506.52 ms [2022-04-14 02:17:34 main:574] : INFO : Epoch 1861 | loss: 0.0311332 | val_loss: 0.0311495 | Time: 6590.71 ms [2022-04-14 02:17:41 main:574] : INFO : Epoch 1862 | loss: 0.0311344 | val_loss: 0.0311483 | Time: 6594.27 ms [2022-04-14 02:17:48 main:574] : INFO : Epoch 1863 | loss: 0.0311305 | val_loss: 0.0311487 | Time: 6625.36 ms [2022-04-14 02:17:54 main:574] : INFO : Epoch 1864 | loss: 0.0311282 | val_loss: 0.0311489 | Time: 6840.84 ms [2022-04-14 02:18:01 main:574] : INFO : Epoch 1865 | loss: 0.0311283 | val_loss: 0.0311472 | Time: 6702.21 ms [2022-04-14 02:18:08 main:574] : INFO : Epoch 1866 | loss: 0.031128 | val_loss: 0.0311477 | Time: 6819.03 ms [2022-04-14 02:18:15 main:574] : INFO : Epoch 1867 | loss: 0.0311269 | val_loss: 0.031148 | Time: 6658.62 ms [2022-04-14 02:18:21 main:574] : INFO : Epoch 1868 | loss: 0.0311258 | val_loss: 0.0311481 | Time: 6722.47 ms [2022-04-14 02:18:28 main:574] : INFO : Epoch 1869 | loss: 0.031125 | val_loss: 0.0311481 | Time: 6763.76 ms [2022-04-14 02:18:35 main:574] : INFO : Epoch 1870 | loss: 0.0311256 | val_loss: 0.0311468 | Time: 6972.46 ms [2022-04-14 02:18:42 main:574] : INFO : Epoch 1871 | loss: 0.0311268 | val_loss: 0.0311511 | Time: 6779.66 ms [2022-04-14 02:18:48 main:574] : INFO : Epoch 1872 | loss: 0.0311312 | val_loss: 0.0311526 | Time: 6597.82 ms [2022-04-14 02:18:55 main:574] : INFO : Epoch 1873 | loss: 0.0311379 | val_loss: 0.0311503 | Time: 6685.81 ms [2022-04-14 02:19:02 main:574] : INFO : Epoch 1874 | loss: 0.0311387 | val_loss: 0.0311522 | Time: 6794.19 ms [2022-04-14 02:19:09 main:574] : INFO : Epoch 1875 | loss: 0.0311387 | val_loss: 0.0311528 | Time: 6661.39 ms [2022-04-14 02:19:15 main:574] : INFO : Epoch 1876 | loss: 0.0311361 | val_loss: 0.0311551 | Time: 6188.88 ms [2022-04-14 02:19:21 main:574] : INFO : Epoch 1877 | loss: 0.0311381 | val_loss: 0.0311535 | Time: 6650.08 ms [2022-04-14 02:19:28 main:574] : INFO : Epoch 1878 | loss: 0.0311351 | val_loss: 0.0311547 | Time: 6468.38 ms [2022-04-14 02:19:36 main:574] : INFO : Epoch 1879 | loss: 0.0311339 | val_loss: 0.0311471 | Time: 8213.06 ms [2022-04-14 02:19:43 main:574] : INFO : Epoch 1880 | loss: 0.0311314 | val_loss: 0.031148 | Time: 6542.64 ms [2022-04-14 02:19:50 main:574] : INFO : Epoch 1881 | loss: 0.0311321 | val_loss: 0.0311507 | Time: 6911.49 ms [2022-04-14 02:19:56 main:574] : INFO : Epoch 1882 | loss: 0.0311325 | val_loss: 0.0311485 | Time: 6095.75 ms [2022-04-14 02:20:02 main:574] : INFO : Epoch 1883 | loss: 0.0311338 | val_loss: 0.0311563 | Time: 6696.48 ms [2022-04-14 02:20:09 main:574] : INFO : Epoch 1884 | loss: 0.0311329 | val_loss: 0.0311502 | Time: 6365.79 ms [2022-04-14 02:20:16 main:574] : INFO : Epoch 1885 | loss: 0.0311328 | val_loss: 0.031149 | Time: 6923.91 ms [2022-04-14 02:20:22 main:574] : INFO : Epoch 1886 | loss: 0.0311328 | val_loss: 0.0311508 | Time: 6498.1 ms [2022-04-14 02:20:29 main:574] : INFO : Epoch 1887 | loss: 0.0311365 | val_loss: 0.0311538 | Time: 6736.81 ms [2022-04-14 02:20:36 main:574] : INFO : Epoch 1888 | loss: 0.0311394 | val_loss: 0.0311611 | Time: 6624.22 ms [2022-04-14 02:20:42 main:574] : INFO : Epoch 1889 | loss: 0.0311577 | val_loss: 0.0311671 | Time: 6470.57 ms [2022-04-14 02:20:49 main:574] : INFO : Epoch 1890 | loss: 0.0311599 | val_loss: 0.0311658 | Time: 6950.26 ms [2022-04-14 02:20:56 main:574] : INFO : Epoch 1891 | loss: 0.031164 | val_loss: 0.0311713 | Time: 6720.64 ms [2022-04-14 02:21:02 main:574] : INFO : Epoch 1892 | loss: 0.0311668 | val_loss: 0.0311662 | Time: 6709.2 ms [2022-04-14 02:21:09 main:574] : INFO : Epoch 1893 | loss: 0.031163 | val_loss: 0.0311651 | Time: 6645.4 ms [2022-04-14 02:21:16 main:574] : INFO : Epoch 1894 | loss: 0.0311593 | val_loss: 0.0311653 | Time: 7044.54 ms [2022-04-14 02:21:23 main:574] : INFO : Epoch 1895 | loss: 0.0311588 | val_loss: 0.0311624 | Time: 6442.98 ms [2022-04-14 02:21:29 main:574] : INFO : Epoch 1896 | loss: 0.0311575 | val_loss: 0.0311646 | Time: 6726.43 ms [2022-04-14 02:21:36 main:574] : INFO : Epoch 1897 | loss: 0.0311566 | val_loss: 0.0311641 | Time: 6703.23 ms [2022-04-14 02:21:43 main:574] : INFO : Epoch 1898 | loss: 0.0311564 | val_loss: 0.0311643 | Time: 6746.26 ms [2022-04-14 02:21:50 main:574] : INFO : Epoch 1899 | loss: 0.0311552 | val_loss: 0.0311629 | Time: 6776.6 ms [2022-04-14 02:21:56 main:574] : INFO : Epoch 1900 | loss: 0.0311532 | val_loss: 0.0311623 | Time: 6504.33 ms [2022-04-14 02:22:03 main:574] : INFO : Epoch 1901 | loss: 0.0311539 | val_loss: 0.0311625 | Time: 6526.8 ms [2022-04-14 02:22:09 main:574] : INFO : Epoch 1902 | loss: 0.0311545 | val_loss: 0.0311637 | Time: 6562.05 ms [2022-04-14 02:22:16 main:574] : INFO : Epoch 1903 | loss: 0.0311534 | val_loss: 0.0311643 | Time: 6759.27 ms [2022-04-14 02:22:23 main:574] : INFO : Epoch 1904 | loss: 0.0311517 | val_loss: 0.031163 | Time: 6656.94 ms [2022-04-14 02:22:29 main:574] : INFO : Epoch 1905 | loss: 0.0311513 | val_loss: 0.0311614 | Time: 6746.64 ms [2022-04-14 02:22:36 main:574] : INFO : Epoch 1906 | loss: 0.0311525 | val_loss: 0.0311648 | Time: 6387.63 ms [2022-04-14 02:22:42 main:574] : INFO : Epoch 1907 | loss: 0.0311535 | val_loss: 0.031162 | Time: 6712.34 ms [2022-04-14 02:22:49 main:574] : INFO : Epoch 1908 | loss: 0.0311526 | val_loss: 0.031162 | Time: 6829.35 ms [2022-04-14 02:22:56 main:574] : INFO : Epoch 1909 | loss: 0.0311504 | val_loss: 0.03116 | Time: 7056.03 ms [2022-04-14 02:23:03 main:574] : INFO : Epoch 1910 | loss: 0.0311497 | val_loss: 0.0311608 | Time: 6680.38 ms [2022-04-14 02:23:10 main:574] : INFO : Epoch 1911 | loss: 0.0311495 | val_loss: 0.0311605 | Time: 6552.07 ms [2022-04-14 02:23:16 main:574] : INFO : Epoch 1912 | loss: 0.0311477 | val_loss: 0.0311628 | Time: 6490.39 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650) [2022-04-14 21:47:03 main:435] : INFO : Set logging level to 1 [2022-04-14 21:47:03 main:441] : INFO : Running in BOINC Client mode [2022-04-14 21:47:03 main:444] : INFO : Resolving all filenames [2022-04-14 21:47:03 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 21:47:03 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 21:47:03 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 21:47:03 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 21:47:03 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 21:47:03 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 21:47:03 main:474] : INFO : Configuration: [2022-04-14 21:47:03 main:475] : INFO : Model type: GRU [2022-04-14 21:47:03 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 21:47:03 main:477] : INFO : Max Epochs: 2048 [2022-04-14 21:47:03 main:478] : INFO : Batch Size: 128 [2022-04-14 21:47:03 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 21:47:03 main:480] : INFO : Patience: 10 [2022-04-14 21:47:03 main:481] : INFO : Hidden Width: 12 [2022-04-14 21:47:03 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 21:47:03 main:483] : INFO : # Backend Layers: 4 [2022-04-14 21:47:03 main:484] : INFO : # Threads: 1 [2022-04-14 21:47:03 main:486] : INFO : Preparing Dataset [2022-04-14 21:47:03 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 21:47:03 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 21:47:05 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 21:47:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 21:47:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 21:47:06 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 21:47:06 main:494] : INFO : Creating Model [2022-04-14 21:47:06 main:507] : INFO : Preparing config file [2022-04-14 21:47:06 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 21:47:06 main:512] : INFO : Loading config [2022-04-14 21:47:06 main:514] : INFO : Loading state [2022-04-14 21:47:07 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 21:47:07 main:562] : INFO : Starting Training [2022-04-14 21:47:12 main:574] : INFO : Epoch 1911 | loss: 0.0311799 | val_loss: 0.0311621 | Time: 4881.27 ms [2022-04-14 21:47:18 main:574] : INFO : Epoch 1912 | loss: 0.0311555 | val_loss: 0.0311608 | Time: 6347.76 ms [2022-04-14 21:47:23 main:574] : INFO : Epoch 1913 | loss: 0.0311517 | val_loss: 0.0311605 | Time: 4917.49 ms [2022-04-14 21:47:28 main:574] : INFO : Epoch 1914 | loss: 0.0311504 | val_loss: 0.0311588 | Time: 5102.89 ms [2022-04-14 21:47:33 main:574] : INFO : Epoch 1915 | loss: 0.0311507 | val_loss: 0.031161 | Time: 4942.64 ms [2022-04-14 21:47:39 main:574] : INFO : Epoch 1916 | loss: 0.0311527 | val_loss: 0.0311631 | Time: 5111.5 ms [2022-04-14 21:47:44 main:574] : INFO : Epoch 1917 | loss: 0.0311526 | val_loss: 0.0311619 | Time: 5240.39 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650) [2022-04-14 22:18:31 main:435] : INFO : Set logging level to 1 [2022-04-14 22:18:31 main:441] : INFO : Running in BOINC Client mode [2022-04-14 22:18:31 main:444] : INFO : Resolving all filenames [2022-04-14 22:18:31 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 22:18:31 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 22:18:31 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 22:18:31 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 22:18:31 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 22:18:31 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 22:18:31 main:474] : INFO : Configuration: [2022-04-14 22:18:31 main:475] : INFO : Model type: GRU [2022-04-14 22:18:31 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 22:18:31 main:477] : INFO : Max Epochs: 2048 [2022-04-14 22:18:31 main:478] : INFO : Batch Size: 128 [2022-04-14 22:18:31 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 22:18:31 main:480] : INFO : Patience: 10 [2022-04-14 22:18:31 main:481] : INFO : Hidden Width: 12 [2022-04-14 22:18:31 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 22:18:31 main:483] : INFO : # Backend Layers: 4 [2022-04-14 22:18:31 main:484] : INFO : # Threads: 1 [2022-04-14 22:18:31 main:486] : INFO : Preparing Dataset [2022-04-14 22:18:31 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 22:18:31 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 22:18:33 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 22:18:33 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 22:18:33 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 22:18:33 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 22:18:33 main:494] : INFO : Creating Model [2022-04-14 22:18:33 main:507] : INFO : Preparing config file [2022-04-14 22:18:33 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 22:18:33 main:512] : INFO : Loading config [2022-04-14 22:18:33 main:514] : INFO : Loading state [2022-04-14 22:18:34 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 22:18:34 main:562] : INFO : Starting Training [2022-04-14 22:18:39 main:574] : INFO : Epoch 1911 | loss: 0.0311804 | val_loss: 0.0311658 | Time: 4995.89 ms [2022-04-14 22:18:44 main:574] : INFO : Epoch 1912 | loss: 0.0311557 | val_loss: 0.0311603 | Time: 4492.36 ms [2022-04-14 22:18:48 main:574] : INFO : Epoch 1913 | loss: 0.0311569 | val_loss: 0.0311635 | Time: 4473.27 ms [2022-04-14 22:18:53 main:574] : INFO : Epoch 1914 | loss: 0.0311549 | val_loss: 0.0311571 | Time: 4668.7 ms [2022-04-14 22:18:58 main:574] : INFO : Epoch 1915 | loss: 0.0311548 | val_loss: 0.0311587 | Time: 4935.87 ms [2022-04-14 22:19:03 main:574] : INFO : Epoch 1916 | loss: 0.0311551 | val_loss: 0.0311618 | Time: 4808.03 ms [2022-04-14 22:19:07 main:574] : INFO : Epoch 1917 | loss: 0.0311569 | val_loss: 0.0311585 | Time: 4799.38 ms [2022-04-14 22:19:12 main:574] : INFO : Epoch 1918 | loss: 0.0311553 | val_loss: 0.0311601 | Time: 4962.18 ms [2022-04-14 22:19:17 main:574] : INFO : Epoch 1919 | loss: 0.0311539 | val_loss: 0.0311597 | Time: 4914.53 ms [2022-04-14 22:19:22 main:574] : INFO : Epoch 1920 | loss: 0.0311538 | val_loss: 0.0311575 | Time: 5117.08 ms [2022-04-14 22:19:28 main:574] : INFO : Epoch 1921 | loss: 0.0311528 | val_loss: 0.0311574 | Time: 5243.45 ms [2022-04-14 22:19:33 main:574] : INFO : Epoch 1922 | loss: 0.0311539 | val_loss: 0.0311577 | Time: 5263.62 ms [2022-04-14 22:19:38 main:574] : INFO : Epoch 1923 | loss: 0.0311566 | val_loss: 0.0311585 | Time: 5216.5 ms [2022-04-14 22:19:44 main:574] : INFO : Epoch 1924 | loss: 0.0311553 | val_loss: 0.0311556 | Time: 5513.55 ms [2022-04-14 22:19:50 main:574] : INFO : Epoch 1925 | loss: 0.0311548 | val_loss: 0.031156 | Time: 6194.68 ms [2022-04-14 22:19:57 main:574] : INFO : Epoch 1926 | loss: 0.0311526 | val_loss: 0.0311583 | Time: 6852.09 ms [2022-04-14 22:20:04 main:574] : INFO : Epoch 1927 | loss: 0.0311542 | val_loss: 0.0311571 | Time: 7079.55 ms [2022-04-14 22:20:11 main:574] : INFO : Epoch 1928 | loss: 0.0311554 | val_loss: 0.0311562 | Time: 7328.75 ms [2022-04-14 22:20:18 main:574] : INFO : Epoch 1929 | loss: 0.0311553 | val_loss: 0.0311614 | Time: 6815.6 ms [2022-04-14 22:20:25 main:574] : INFO : Epoch 1930 | loss: 0.0311529 | val_loss: 0.0311595 | Time: 7156.94 ms [2022-04-14 22:20:32 main:574] : INFO : Epoch 1931 | loss: 0.0311513 | val_loss: 0.0311579 | Time: 7174.72 ms [2022-04-14 22:20:39 main:574] : INFO : Epoch 1932 | loss: 0.0311521 | val_loss: 0.0311613 | Time: 6937.25 ms [2022-04-14 22:20:46 main:574] : INFO : Epoch 1933 | loss: 0.031153 | val_loss: 0.0311591 | Time: 7171 ms [2022-04-14 22:20:54 main:574] : INFO : Epoch 1934 | loss: 0.0311576 | val_loss: 0.0311676 | Time: 7091.77 ms [2022-04-14 22:21:00 main:574] : INFO : Epoch 1935 | loss: 0.0311585 | val_loss: 0.0311619 | Time: 6631.66 ms [2022-04-14 22:21:06 main:574] : INFO : Epoch 1936 | loss: 0.0311575 | val_loss: 0.0311612 | Time: 6176.75 ms [2022-04-14 22:21:13 main:574] : INFO : Epoch 1937 | loss: 0.0311607 | val_loss: 0.0311636 | Time: 6180.15 ms [2022-04-14 22:21:19 main:574] : INFO : Epoch 1938 | loss: 0.0311632 | val_loss: 0.0311668 | Time: 6040.87 ms [2022-04-14 22:21:25 main:574] : INFO : Epoch 1939 | loss: 0.0311612 | val_loss: 0.0311643 | Time: 6197.52 ms [2022-04-14 22:21:32 main:574] : INFO : Epoch 1940 | loss: 0.0311599 | val_loss: 0.0311633 | Time: 6772.8 ms [2022-04-14 22:21:38 main:574] : INFO : Epoch 1941 | loss: 0.0311584 | val_loss: 0.0311637 | Time: 6669.35 ms [2022-04-14 22:21:44 main:574] : INFO : Epoch 1942 | loss: 0.0311571 | val_loss: 0.0311615 | Time: 6090.91 ms [2022-04-14 22:21:51 main:574] : INFO : Epoch 1943 | loss: 0.0311578 | val_loss: 0.0311621 | Time: 6251.9 ms [2022-04-14 22:21:57 main:574] : INFO : Epoch 1944 | loss: 0.0311576 | val_loss: 0.0311632 | Time: 6030.12 ms [2022-04-14 22:22:03 main:574] : INFO : Epoch 1945 | loss: 0.0311576 | val_loss: 0.0311636 | Time: 6148.44 ms [2022-04-14 22:22:09 main:574] : INFO : Epoch 1946 | loss: 0.0311579 | val_loss: 0.0311704 | Time: 6265.49 ms [2022-04-14 22:22:15 main:574] : INFO : Epoch 1947 | loss: 0.0311646 | val_loss: 0.031168 | Time: 6208.27 ms [2022-04-14 22:22:22 main:574] : INFO : Epoch 1948 | loss: 0.0311618 | val_loss: 0.031166 | Time: 6575.03 ms [2022-04-14 22:22:28 main:574] : INFO : Epoch 1949 | loss: 0.0311583 | val_loss: 0.0311636 | Time: 6087.84 ms [2022-04-14 22:22:35 main:574] : INFO : Epoch 1950 | loss: 0.0311571 | val_loss: 0.0311638 | Time: 6604.08 ms [2022-04-14 22:22:41 main:574] : INFO : Epoch 1951 | loss: 0.0311565 | val_loss: 0.0311645 | Time: 6617.11 ms [2022-04-14 22:22:48 main:574] : INFO : Epoch 1952 | loss: 0.0311578 | val_loss: 0.0311624 | Time: 6644.42 ms [2022-04-14 22:22:54 main:574] : INFO : Epoch 1953 | loss: 0.0311562 | val_loss: 0.0311644 | Time: 6551.14 ms [2022-04-14 22:23:01 main:574] : INFO : Epoch 1954 | loss: 0.0311552 | val_loss: 0.0311621 | Time: 6704.38 ms [2022-04-14 22:23:08 main:574] : INFO : Epoch 1955 | loss: 0.031154 | val_loss: 0.0311642 | Time: 6600.18 ms [2022-04-14 22:23:14 main:574] : INFO : Epoch 1956 | loss: 0.0311539 | val_loss: 0.0311605 | Time: 6626.55 ms [2022-04-14 22:23:21 main:574] : INFO : Epoch 1957 | loss: 0.0311535 | val_loss: 0.031161 | Time: 6746.64 ms [2022-04-14 22:23:28 main:574] : INFO : Epoch 1958 | loss: 0.0311528 | val_loss: 0.0311606 | Time: 6895.66 ms [2022-04-14 22:23:35 main:574] : INFO : Epoch 1959 | loss: 0.0311528 | val_loss: 0.0311633 | Time: 6785.78 ms [2022-04-14 22:23:42 main:574] : INFO : Epoch 1960 | loss: 0.0311519 | val_loss: 0.0311617 | Time: 6802.64 ms [2022-04-14 22:23:48 main:574] : INFO : Epoch 1961 | loss: 0.0311513 | val_loss: 0.0311612 | Time: 6331.01 ms [2022-04-14 22:23:55 main:574] : INFO : Epoch 1962 | loss: 0.0311515 | val_loss: 0.0311612 | Time: 6503.33 ms [2022-04-14 22:24:01 main:574] : INFO : Epoch 1963 | loss: 0.0311516 | val_loss: 0.0311619 | Time: 6247.93 ms [2022-04-14 22:24:08 main:574] : INFO : Epoch 1964 | loss: 0.0311521 | val_loss: 0.0311599 | Time: 6080.4 ms [2022-04-14 22:24:15 main:574] : INFO : Epoch 1965 | loss: 0.0311521 | val_loss: 0.0311605 | Time: 6702.54 ms [2022-04-14 22:24:21 main:574] : INFO : Epoch 1966 | loss: 0.031154 | val_loss: 0.0311694 | Time: 6421.06 ms [2022-04-14 22:24:28 main:574] : INFO : Epoch 1967 | loss: 0.0311716 | val_loss: 0.031179 | Time: 6859.62 ms [2022-04-14 22:24:35 main:574] : INFO : Epoch 1968 | loss: 0.0311775 | val_loss: 0.0311764 | Time: 6806.2 ms [2022-04-14 22:24:42 main:574] : INFO : Epoch 1969 | loss: 0.0311776 | val_loss: 0.0311789 | Time: 7110.16 ms [2022-04-14 22:24:49 main:574] : INFO : Epoch 1970 | loss: 0.0311773 | val_loss: 0.0311747 | Time: 6926.75 ms [2022-04-14 22:24:55 main:574] : INFO : Epoch 1971 | loss: 0.0311746 | val_loss: 0.0311757 | Time: 6656.72 ms [2022-04-14 22:25:02 main:574] : INFO : Epoch 1972 | loss: 0.0311735 | val_loss: 0.0311752 | Time: 6892.14 ms [2022-04-14 22:25:10 main:574] : INFO : Epoch 1973 | loss: 0.0311729 | val_loss: 0.0311754 | Time: 7321.59 ms [2022-04-14 22:25:16 main:574] : INFO : Epoch 1974 | loss: 0.0311725 | val_loss: 0.0311753 | Time: 6755.99 ms [2022-04-14 22:25:23 main:574] : INFO : Epoch 1975 | loss: 0.0311725 | val_loss: 0.0311774 | Time: 6434.29 ms [2022-04-14 22:25:30 main:574] : INFO : Epoch 1976 | loss: 0.0311733 | val_loss: 0.0311757 | Time: 6955.65 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650) [2022-04-14 22:36:05 main:435] : INFO : Set logging level to 1 [2022-04-14 22:36:05 main:441] : INFO : Running in BOINC Client mode [2022-04-14 22:36:05 main:444] : INFO : Resolving all filenames [2022-04-14 22:36:05 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 22:36:05 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 22:36:05 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 22:36:05 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 22:36:05 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 22:36:05 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 22:36:05 main:474] : INFO : Configuration: [2022-04-14 22:36:05 main:475] : INFO : Model type: GRU [2022-04-14 22:36:05 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 22:36:05 main:477] : INFO : Max Epochs: 2048 [2022-04-14 22:36:05 main:478] : INFO : Batch Size: 128 [2022-04-14 22:36:05 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 22:36:05 main:480] : INFO : Patience: 10 [2022-04-14 22:36:05 main:481] : INFO : Hidden Width: 12 [2022-04-14 22:36:05 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 22:36:05 main:483] : INFO : # Backend Layers: 4 [2022-04-14 22:36:05 main:484] : INFO : # Threads: 1 [2022-04-14 22:36:05 main:486] : INFO : Preparing Dataset [2022-04-14 22:36:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 22:36:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 22:36:07 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 22:36:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 22:36:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 22:36:07 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 22:36:07 main:494] : INFO : Creating Model [2022-04-14 22:36:07 main:507] : INFO : Preparing config file [2022-04-14 22:36:07 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 22:36:07 main:512] : INFO : Loading config [2022-04-14 22:36:07 main:514] : INFO : Loading state [2022-04-14 22:36:08 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 22:36:08 main:562] : INFO : Starting Training [2022-04-14 22:36:13 main:574] : INFO : Epoch 1975 | loss: 0.0312018 | val_loss: 0.0311833 | Time: 4784.77 ms [2022-04-14 22:36:17 main:574] : INFO : Epoch 1976 | loss: 0.03117 | val_loss: 0.031173 | Time: 4356.49 ms [2022-04-14 22:36:22 main:574] : INFO : Epoch 1977 | loss: 0.031163 | val_loss: 0.0311769 | Time: 4565.52 ms [2022-04-14 22:36:27 main:574] : INFO : Epoch 1978 | loss: 0.031161 | val_loss: 0.0311716 | Time: 4587.69 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650) [2022-04-14 22:39:35 main:435] : INFO : Set logging level to 1 [2022-04-14 22:39:35 main:441] : INFO : Running in BOINC Client mode [2022-04-14 22:39:35 main:444] : INFO : Resolving all filenames [2022-04-14 22:39:35 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 22:39:35 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 22:39:35 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 22:39:35 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 22:39:35 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 22:39:35 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 22:39:35 main:474] : INFO : Configuration: [2022-04-14 22:39:35 main:475] : INFO : Model type: GRU [2022-04-14 22:39:35 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 22:39:35 main:477] : INFO : Max Epochs: 2048 [2022-04-14 22:39:35 main:478] : INFO : Batch Size: 128 [2022-04-14 22:39:35 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 22:39:35 main:480] : INFO : Patience: 10 [2022-04-14 22:39:35 main:481] : INFO : Hidden Width: 12 [2022-04-14 22:39:35 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 22:39:35 main:483] : INFO : # Backend Layers: 4 [2022-04-14 22:39:35 main:484] : INFO : # Threads: 1 [2022-04-14 22:39:35 main:486] : INFO : Preparing Dataset [2022-04-14 22:39:35 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 22:39:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 22:39:37 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 22:39:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 22:39:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 22:39:38 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 22:39:38 main:494] : INFO : Creating Model [2022-04-14 22:39:38 main:507] : INFO : Preparing config file [2022-04-14 22:39:38 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 22:39:38 main:512] : INFO : Loading config [2022-04-14 22:39:38 main:514] : INFO : Loading state [2022-04-14 22:39:39 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 22:39:39 main:562] : INFO : Starting Training [2022-04-14 22:39:44 main:574] : INFO : Epoch 1975 | loss: 0.031205 | val_loss: 0.0311799 | Time: 5076.28 ms [2022-04-14 22:39:49 main:574] : INFO : Epoch 1976 | loss: 0.0311673 | val_loss: 0.0311742 | Time: 4686.2 ms [2022-04-14 22:39:54 main:574] : INFO : Epoch 1977 | loss: 0.0311628 | val_loss: 0.031175 | Time: 4823.96 ms [2022-04-14 22:39:59 main:574] : INFO : Epoch 1978 | loss: 0.0311613 | val_loss: 0.0311696 | Time: 5037.9 ms [2022-04-14 22:40:04 main:574] : INFO : Epoch 1979 | loss: 0.0311591 | val_loss: 0.031169 | Time: 5068.16 ms [2022-04-14 22:40:09 main:574] : INFO : Epoch 1980 | loss: 0.031159 | val_loss: 0.03117 | Time: 5181.6 ms [2022-04-14 22:40:14 main:574] : INFO : Epoch 1981 | loss: 0.0311582 | val_loss: 0.0311712 | Time: 5309.4 ms [2022-04-14 22:40:19 main:574] : INFO : Epoch 1982 | loss: 0.0311582 | val_loss: 0.0311695 | Time: 5004.86 ms [2022-04-14 22:40:25 main:574] : INFO : Epoch 1983 | loss: 0.0311549 | val_loss: 0.0311661 | Time: 5366.04 ms [2022-04-14 22:40:30 main:574] : INFO : Epoch 1984 | loss: 0.031153 | val_loss: 0.0311659 | Time: 5363.09 ms [2022-04-14 22:40:35 main:574] : INFO : Epoch 1985 | loss: 0.0311525 | val_loss: 0.0311657 | Time: 5326.14 ms [2022-04-14 22:40:41 main:574] : INFO : Epoch 1986 | loss: 0.0311547 | val_loss: 0.0311639 | Time: 6011.99 ms [2022-04-14 22:40:48 main:574] : INFO : Epoch 1987 | loss: 0.0311523 | val_loss: 0.0311681 | Time: 7256.24 ms [2022-04-14 22:40:56 main:574] : INFO : Epoch 1988 | loss: 0.0311537 | val_loss: 0.0311665 | Time: 7416.31 ms [2022-04-14 22:41:03 main:574] : INFO : Epoch 1989 | loss: 0.031152 | val_loss: 0.0311667 | Time: 6867.05 ms [2022-04-14 22:41:10 main:574] : INFO : Epoch 1990 | loss: 0.0311537 | val_loss: 0.0311675 | Time: 6977.51 ms [2022-04-14 22:41:17 main:574] : INFO : Epoch 1991 | loss: 0.031154 | val_loss: 0.0311666 | Time: 6884.08 ms [2022-04-14 22:41:23 main:574] : INFO : Epoch 1992 | loss: 0.0311566 | val_loss: 0.0311669 | Time: 6791.37 ms [2022-04-14 22:41:30 main:574] : INFO : Epoch 1993 | loss: 0.0311536 | val_loss: 0.0311655 | Time: 6995.22 ms [2022-04-14 22:41:38 main:574] : INFO : Epoch 1994 | loss: 0.0311501 | val_loss: 0.0311646 | Time: 7143.63 ms [2022-04-14 22:41:45 main:574] : INFO : Epoch 1995 | loss: 0.0311491 | val_loss: 0.0311638 | Time: 7028.8 ms [2022-04-14 22:41:52 main:574] : INFO : Epoch 1996 | loss: 0.0311493 | val_loss: 0.0311671 | Time: 7064.02 ms [2022-04-14 22:41:59 main:574] : INFO : Epoch 1997 | loss: 0.0311475 | val_loss: 0.0311615 | Time: 7040.73 ms [2022-04-14 22:42:06 main:574] : INFO : Epoch 1998 | loss: 0.0311487 | val_loss: 0.0311641 | Time: 7146.98 ms [2022-04-14 22:42:13 main:574] : INFO : Epoch 1999 | loss: 0.0311499 | val_loss: 0.0311643 | Time: 7020.26 ms [2022-04-14 22:42:20 main:574] : INFO : Epoch 2000 | loss: 0.0311488 | val_loss: 0.0311627 | Time: 7399.54 ms [2022-04-14 22:42:28 main:574] : INFO : Epoch 2001 | loss: 0.0311551 | val_loss: 0.0311672 | Time: 7665.41 ms [2022-04-14 22:42:35 main:574] : INFO : Epoch 2002 | loss: 0.0311554 | val_loss: 0.0311636 | Time: 7490 ms [2022-04-14 22:42:43 main:574] : INFO : Epoch 2003 | loss: 0.0311558 | val_loss: 0.0311675 | Time: 7485.6 ms [2022-04-14 22:42:50 main:574] : INFO : Epoch 2004 | loss: 0.0311549 | val_loss: 0.0311643 | Time: 7256.31 ms [2022-04-14 22:42:57 main:574] : INFO : Epoch 2005 | loss: 0.0311525 | val_loss: 0.0311641 | Time: 6945.62 ms [2022-04-14 22:43:04 main:574] : INFO : Epoch 2006 | loss: 0.0311509 | val_loss: 0.0311641 | Time: 6559.21 ms [2022-04-14 22:43:11 main:574] : INFO : Epoch 2007 | loss: 0.0311526 | val_loss: 0.0311625 | Time: 7146.4 ms [2022-04-14 22:43:18 main:574] : INFO : Epoch 2008 | loss: 0.0311508 | val_loss: 0.0311651 | Time: 6757.7 ms [2022-04-14 22:43:25 main:574] : INFO : Epoch 2009 | loss: 0.0311507 | val_loss: 0.0311623 | Time: 6936.29 ms [2022-04-14 22:43:32 main:574] : INFO : Epoch 2010 | loss: 0.0311493 | val_loss: 0.0311628 | Time: 7006.04 ms [2022-04-14 22:43:39 main:574] : INFO : Epoch 2011 | loss: 0.0311478 | val_loss: 0.0311638 | Time: 7221.82 ms [2022-04-14 22:43:46 main:574] : INFO : Epoch 2012 | loss: 0.0311471 | val_loss: 0.0311606 | Time: 6942.23 ms [2022-04-14 22:43:53 main:574] : INFO : Epoch 2013 | loss: 0.0311466 | val_loss: 0.0311643 | Time: 7202.33 ms [2022-04-14 22:44:00 main:574] : INFO : Epoch 2014 | loss: 0.0311451 | val_loss: 0.0311597 | Time: 7214.1 ms [2022-04-14 22:44:07 main:574] : INFO : Epoch 2015 | loss: 0.0311434 | val_loss: 0.0311575 | Time: 7088.96 ms [2022-04-14 22:44:14 main:574] : INFO : Epoch 2016 | loss: 0.0311435 | val_loss: 0.0311623 | Time: 6712.76 ms [2022-04-14 22:44:21 main:574] : INFO : Epoch 2017 | loss: 0.0311429 | val_loss: 0.0311596 | Time: 6522.55 ms [2022-04-14 22:44:27 main:574] : INFO : Epoch 2018 | loss: 0.0311424 | val_loss: 0.0311587 | Time: 6430.93 ms [2022-04-14 22:44:34 main:574] : INFO : Epoch 2019 | loss: 0.031143 | val_loss: 0.031162 | Time: 6577.71 ms [2022-04-14 22:44:40 main:574] : INFO : Epoch 2020 | loss: 0.0311437 | val_loss: 0.0311616 | Time: 6809.93 ms [2022-04-14 22:44:47 main:574] : INFO : Epoch 2021 | loss: 0.0311472 | val_loss: 0.0311599 | Time: 6850.09 ms [2022-04-14 22:44:54 main:574] : INFO : Epoch 2022 | loss: 0.0311469 | val_loss: 0.0311616 | Time: 6874.45 ms [2022-04-14 22:45:01 main:574] : INFO : Epoch 2023 | loss: 0.0311488 | val_loss: 0.0311609 | Time: 6697.59 ms [2022-04-14 22:45:08 main:574] : INFO : Epoch 2024 | loss: 0.031148 | val_loss: 0.0311622 | Time: 6805.35 ms [2022-04-14 22:45:14 main:574] : INFO : Epoch 2025 | loss: 0.0311457 | val_loss: 0.0311588 | Time: 6737.15 ms [2022-04-14 22:45:21 main:574] : INFO : Epoch 2026 | loss: 0.0311448 | val_loss: 0.031161 | Time: 6321.97 ms [2022-04-14 22:45:27 main:574] : INFO : Epoch 2027 | loss: 0.0311452 | val_loss: 0.0311657 | Time: 6657.53 ms [2022-04-14 22:45:34 main:574] : INFO : Epoch 2028 | loss: 0.0311427 | val_loss: 0.0311607 | Time: 6646.05 ms [2022-04-14 22:45:41 main:574] : INFO : Epoch 2029 | loss: 0.0311422 | val_loss: 0.0311623 | Time: 6514.9 ms [2022-04-14 22:45:47 main:574] : INFO : Epoch 2030 | loss: 0.0311424 | val_loss: 0.0311607 | Time: 6654.95 ms [2022-04-14 22:45:54 main:574] : INFO : Epoch 2031 | loss: 0.031142 | val_loss: 0.0311584 | Time: 6809.92 ms [2022-04-14 22:46:01 main:574] : INFO : Epoch 2032 | loss: 0.0311419 | val_loss: 0.0311547 | Time: 7024.88 ms [2022-04-14 22:46:08 main:574] : INFO : Epoch 2033 | loss: 0.0311413 | val_loss: 0.0311603 | Time: 7386.66 ms [2022-04-14 22:46:15 main:574] : INFO : Epoch 2034 | loss: 0.0311399 | val_loss: 0.0311562 | Time: 6845.86 ms [2022-04-14 22:46:22 main:574] : INFO : Epoch 2035 | loss: 0.0311434 | val_loss: 0.0311569 | Time: 6610.21 ms [2022-04-14 22:46:29 main:574] : INFO : Epoch 2036 | loss: 0.0311417 | val_loss: 0.0311605 | Time: 7270.07 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650) [2022-04-14 22:55:49 main:435] : INFO : Set logging level to 1 [2022-04-14 22:55:49 main:441] : INFO : Running in BOINC Client mode [2022-04-14 22:55:49 main:444] : INFO : Resolving all filenames [2022-04-14 22:55:49 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 22:55:49 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 22:55:49 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 22:55:49 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 22:55:49 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 22:55:49 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 22:55:49 main:474] : INFO : Configuration: [2022-04-14 22:55:49 main:475] : INFO : Model type: GRU [2022-04-14 22:55:49 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 22:55:49 main:477] : INFO : Max Epochs: 2048 [2022-04-14 22:55:49 main:478] : INFO : Batch Size: 128 [2022-04-14 22:55:49 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 22:55:49 main:480] : INFO : Patience: 10 [2022-04-14 22:55:49 main:481] : INFO : Hidden Width: 12 [2022-04-14 22:55:49 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 22:55:49 main:483] : INFO : # Backend Layers: 4 [2022-04-14 22:55:49 main:484] : INFO : # Threads: 1 [2022-04-14 22:55:49 main:486] : INFO : Preparing Dataset [2022-04-14 22:55:49 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 22:55:49 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 22:55:51 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 22:55:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 22:55:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 22:55:51 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 22:55:51 main:494] : INFO : Creating Model [2022-04-14 22:55:51 main:507] : INFO : Preparing config file [2022-04-14 22:55:51 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 22:55:51 main:512] : INFO : Loading config [2022-04-14 22:55:51 main:514] : INFO : Loading state [2022-04-14 22:55:53 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 22:55:53 main:562] : INFO : Starting Training [2022-04-14 22:55:58 main:574] : INFO : Epoch 2036 | loss: 0.0311723 | val_loss: 0.0311634 | Time: 5147.26 ms [2022-04-14 22:56:02 main:574] : INFO : Epoch 2037 | loss: 0.031144 | val_loss: 0.0311601 | Time: 4721.46 ms [2022-04-14 22:56:07 main:574] : INFO : Epoch 2038 | loss: 0.0311425 | val_loss: 0.0311596 | Time: 4752.8 ms [2022-04-14 22:56:12 main:574] : INFO : Epoch 2039 | loss: 0.0311412 | val_loss: 0.0311616 | Time: 4804.34 ms [2022-04-14 22:56:17 main:574] : INFO : Epoch 2040 | loss: 0.0311406 | val_loss: 0.0311599 | Time: 5087.08 ms [2022-04-14 22:56:22 main:574] : INFO : Epoch 2041 | loss: 0.03114 | val_loss: 0.0311614 | Time: 5015.78 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1650) [2022-04-14 22:59:25 main:435] : INFO : Set logging level to 1 [2022-04-14 22:59:25 main:441] : INFO : Running in BOINC Client mode [2022-04-14 22:59:25 main:444] : INFO : Resolving all filenames [2022-04-14 22:59:25 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-04-14 22:59:25 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-04-14 22:59:25 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-04-14 22:59:25 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-04-14 22:59:25 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-04-14 22:59:25 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-04-14 22:59:25 main:474] : INFO : Configuration: [2022-04-14 22:59:25 main:475] : INFO : Model type: GRU [2022-04-14 22:59:25 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-04-14 22:59:25 main:477] : INFO : Max Epochs: 2048 [2022-04-14 22:59:25 main:478] : INFO : Batch Size: 128 [2022-04-14 22:59:25 main:479] : INFO : Learning Rate: 0.01 [2022-04-14 22:59:25 main:480] : INFO : Patience: 10 [2022-04-14 22:59:25 main:481] : INFO : Hidden Width: 12 [2022-04-14 22:59:25 main:482] : INFO : # Recurrent Layers: 4 [2022-04-14 22:59:25 main:483] : INFO : # Backend Layers: 4 [2022-04-14 22:59:25 main:484] : INFO : # Threads: 1 [2022-04-14 22:59:25 main:486] : INFO : Preparing Dataset [2022-04-14 22:59:25 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-04-14 22:59:26 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-04-14 22:59:27 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-04-14 22:59:27 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-04-14 22:59:28 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-04-14 22:59:28 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-04-14 22:59:28 main:494] : INFO : Creating Model [2022-04-14 22:59:28 main:507] : INFO : Preparing config file [2022-04-14 22:59:28 main:511] : INFO : Found checkpoint, attempting to load... [2022-04-14 22:59:28 main:512] : INFO : Loading config [2022-04-14 22:59:28 main:514] : INFO : Loading state [2022-04-14 22:59:29 main:559] : INFO : Loading DataLoader into Memory [2022-04-14 22:59:29 main:562] : INFO : Starting Training [2022-04-14 22:59:34 main:574] : INFO : Epoch 2036 | loss: 0.0311719 | val_loss: 0.0311633 | Time: 5128.77 ms [2022-04-14 22:59:39 main:574] : INFO : Epoch 2037 | loss: 0.0311478 | val_loss: 0.0311648 | Time: 4719.78 ms [2022-04-14 22:59:43 main:574] : INFO : Epoch 2038 | loss: 0.0311456 | val_loss: 0.0311631 | Time: 4577.13 ms [2022-04-14 22:59:48 main:574] : INFO : Epoch 2039 | loss: 0.0311421 | val_loss: 0.0311646 | Time: 4597.06 ms [2022-04-14 22:59:53 main:574] : INFO : Epoch 2040 | loss: 0.0311418 | val_loss: 0.0311621 | Time: 4721.21 ms [2022-04-14 22:59:58 main:574] : INFO : Epoch 2041 | loss: 0.0311404 | val_loss: 0.0311573 | Time: 4864.21 ms [2022-04-14 23:00:02 main:574] : INFO : Epoch 2042 | loss: 0.0311396 | val_loss: 0.0311607 | Time: 4949.73 ms [2022-04-14 23:00:07 main:574] : INFO : Epoch 2043 | loss: 0.0311378 | val_loss: 0.0311584 | Time: 4790.39 ms [2022-04-14 23:00:12 main:574] : INFO : Epoch 2044 | loss: 0.031138 | val_loss: 0.0311595 | Time: 4849.78 ms [2022-04-14 23:00:17 main:574] : INFO : Epoch 2045 | loss: 0.0311394 | val_loss: 0.0311618 | Time: 4898.13 ms [2022-04-14 23:00:22 main:574] : INFO : Epoch 2046 | loss: 0.0311381 | val_loss: 0.0311612 | Time: 5100.43 ms [2022-04-14 23:00:27 main:574] : INFO : Epoch 2047 | loss: 0.0311371 | val_loss: 0.0311565 | Time: 5103.19 ms [2022-04-14 23:00:33 main:574] : INFO : Epoch 2048 | loss: 0.0311386 | val_loss: 0.0311584 | Time: 6233.06 ms [2022-04-14 23:00:33 main:597] : INFO : Saving trained model to model-final.pt, val_loss 0.0311584 [2022-04-14 23:00:33 main:603] : INFO : Saving end state to config to file [2022-04-14 23:00:33 main:608] : INFO : Success, exiting.. 23:00:33 (13556): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)