Task 12953340
| Name | ParityModified-1647040725-30006-0_0 |
| Workunit | 10085673 |
| Created | 11 Mar 2022, 23:18:46 UTC |
| Sent | 13 Mar 2022, 9:45:35 UTC |
| Report deadline | 21 Mar 2022, 9:45:35 UTC |
| Received | 14 Mar 2022, 8:32:23 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 1150 |
| Run time | 57 min 5 sec |
| CPU time | 55 min 32 sec |
| Validate state | Valid |
| Credit | 4,160.00 |
| Device peak FLOPS | 4,174.91 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.54 GB |
| Peak swap size | 3.48 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.20</core_client_version> <![CDATA[ <stderr_txt> e: 1656.67 ms [2022-03-14 03:17:54 main:574] : INFO : Epoch 1685 | loss: 0.031015 | val_loss: 0.031276 | Time: 1645.83 ms [2022-03-14 03:17:56 main:574] : INFO : Epoch 1686 | loss: 0.0310149 | val_loss: 0.0313338 | Time: 1640.03 ms [2022-03-14 03:17:58 main:574] : INFO : Epoch 1687 | loss: 0.0310029 | val_loss: 0.0313062 | Time: 1636.82 ms [2022-03-14 03:17:59 main:574] : INFO : Epoch 1688 | loss: 0.0310015 | val_loss: 0.0313185 | Time: 1638.98 ms [2022-03-14 03:18:01 main:574] : INFO : Epoch 1689 | loss: 0.0310012 | val_loss: 0.0313034 | Time: 1662.27 ms [2022-03-14 03:18:03 main:574] : INFO : Epoch 1690 | loss: 0.031007 | val_loss: 0.0313044 | Time: 1656.14 ms [2022-03-14 03:18:04 main:574] : INFO : Epoch 1691 | loss: 0.0310071 | val_loss: 0.0313095 | Time: 1645.13 ms [2022-03-14 03:18:06 main:574] : INFO : Epoch 1692 | loss: 0.0310068 | val_loss: 0.0313094 | Time: 1646.47 ms [2022-03-14 03:18:08 main:574] : INFO : Epoch 1693 | loss: 0.0310073 | val_loss: 0.031282 | Time: 1642.49 ms [2022-03-14 03:18:09 main:574] : INFO : Epoch 1694 | loss: 0.0310029 | val_loss: 0.0313104 | Time: 1640.05 ms [2022-03-14 03:18:11 main:574] : INFO : Epoch 1695 | loss: 0.0309996 | val_loss: 0.031327 | Time: 1648.21 ms [2022-03-14 03:18:13 main:574] : INFO : Epoch 1696 | loss: 0.031003 | val_loss: 0.0313105 | Time: 1653.66 ms [2022-03-14 03:18:14 main:574] : INFO : Epoch 1697 | loss: 0.031012 | val_loss: 0.0312693 | Time: 1640.71 ms [2022-03-14 03:18:16 main:574] : INFO : Epoch 1698 | loss: 0.031023 | val_loss: 0.0312936 | Time: 1635.63 ms [2022-03-14 03:18:18 main:574] : INFO : Epoch 1699 | loss: 0.0310161 | val_loss: 0.031284 | Time: 1641.19 ms [2022-03-14 03:18:19 main:574] : INFO : Epoch 1700 | loss: 0.03101 | val_loss: 0.0313058 | Time: 1643.5 ms [2022-03-14 03:18:21 main:574] : INFO : Epoch 1701 | loss: 0.0310037 | val_loss: 0.0312967 | Time: 1646.27 ms [2022-03-14 03:18:22 main:574] : INFO : Epoch 1702 | loss: 0.0310136 | val_loss: 0.0313076 | Time: 1649.98 ms [2022-03-14 03:18:24 main:574] : INFO : Epoch 1703 | loss: 0.0310143 | val_loss: 0.0312982 | Time: 1656.59 ms [2022-03-14 03:18:26 main:574] : INFO : Epoch 1704 | loss: 0.0310078 | val_loss: 0.0312812 | Time: 1642.65 ms [2022-03-14 03:18:27 main:574] : INFO : Epoch 1705 | loss: 0.031003 | val_loss: 0.0312826 | Time: 1652.4 ms [2022-03-14 03:18:29 main:574] : INFO : Epoch 1706 | loss: 0.0310076 | val_loss: 0.0312995 | Time: 1640.5 ms [2022-03-14 03:18:31 main:574] : INFO : Epoch 1707 | loss: 0.0310054 | val_loss: 0.0312997 | Time: 1649.01 ms [2022-03-14 03:18:32 main:574] : INFO : Epoch 1708 | loss: 0.0310042 | val_loss: 0.0313066 | Time: 1645.79 ms [2022-03-14 03:18:34 main:574] : INFO : Epoch 1709 | loss: 0.0310092 | val_loss: 0.0313223 | Time: 1655.33 ms [2022-03-14 03:18:36 main:574] : INFO : Epoch 1710 | loss: 0.0310148 | val_loss: 0.0313102 | Time: 1636.74 ms [2022-03-14 03:18:37 main:574] : INFO : Epoch 1711 | loss: 0.0310251 | val_loss: 0.0312966 | Time: 1650.4 ms [2022-03-14 03:18:39 main:574] : INFO : Epoch 1712 | loss: 0.0310186 | val_loss: 0.0313156 | Time: 1638.91 ms [2022-03-14 03:18:41 main:574] : INFO : Epoch 1713 | loss: 0.0310101 | val_loss: 0.0312903 | Time: 1652.49 ms [2022-03-14 03:18:42 main:574] : INFO : Epoch 1714 | loss: 0.0310194 | val_loss: 0.0312904 | Time: 1653.47 ms [2022-03-14 03:18:44 main:574] : INFO : Epoch 1715 | loss: 0.0310085 | val_loss: 0.0312815 | Time: 1649.92 ms [2022-03-14 03:18:46 main:574] : INFO : Epoch 1716 | loss: 0.0310086 | val_loss: 0.031321 | Time: 1649.19 ms [2022-03-14 03:18:47 main:574] : INFO : Epoch 1717 | loss: 0.0310061 | val_loss: 0.0313332 | Time: 1648.1 ms [2022-03-14 03:18:49 main:574] : INFO : Epoch 1718 | loss: 0.0310048 | val_loss: 0.0313028 | Time: 1637.48 ms [2022-03-14 03:18:51 main:574] : INFO : Epoch 1719 | loss: 0.0310124 | val_loss: 0.0313113 | Time: 1650.05 ms [2022-03-14 03:18:52 main:574] : INFO : Epoch 1720 | loss: 0.0310049 | val_loss: 0.0312994 | Time: 1654.15 ms [2022-03-14 03:18:54 main:574] : INFO : Epoch 1721 | loss: 0.0310074 | val_loss: 0.0312869 | Time: 1659.67 ms [2022-03-14 03:18:56 main:574] : INFO : Epoch 1722 | loss: 0.0310157 | val_loss: 0.0313104 | Time: 1651.02 ms [2022-03-14 03:18:57 main:574] : INFO : Epoch 1723 | loss: 0.0310141 | val_loss: 0.0313178 | Time: 1647.1 ms [2022-03-14 03:18:59 main:574] : INFO : Epoch 1724 | loss: 0.0310056 | val_loss: 0.0313013 | Time: 1639.08 ms [2022-03-14 03:19:01 main:574] : INFO : Epoch 1725 | loss: 0.0310075 | val_loss: 0.0312788 | Time: 1654.5 ms [2022-03-14 03:19:02 main:574] : INFO : Epoch 1726 | loss: 0.031011 | val_loss: 0.0312875 | Time: 1651.34 ms [2022-03-14 03:19:04 main:574] : INFO : Epoch 1727 | loss: 0.0310077 | val_loss: 0.0312928 | Time: 1651.13 ms [2022-03-14 03:19:06 main:574] : INFO : Epoch 1728 | loss: 0.0310068 | val_loss: 0.0312896 | Time: 1644.48 ms [2022-03-14 03:19:07 main:574] : INFO : Epoch 1729 | loss: 0.031011 | val_loss: 0.0312984 | Time: 1648.78 ms [2022-03-14 03:19:09 main:574] : INFO : Epoch 1730 | loss: 0.0310147 | val_loss: 0.0313178 | Time: 1650.02 ms [2022-03-14 03:19:10 main:574] : INFO : Epoch 1731 | loss: 0.0310076 | val_loss: 0.0312903 | Time: 1653.4 ms [2022-03-14 03:19:12 main:574] : INFO : Epoch 1732 | loss: 0.0309987 | val_loss: 0.0313203 | Time: 1663.25 ms [2022-03-14 03:19:14 main:574] : INFO : Epoch 1733 | loss: 0.0310004 | val_loss: 0.031297 | Time: 1653.97 ms [2022-03-14 03:19:15 main:574] : INFO : Epoch 1734 | loss: 0.0310134 | val_loss: 0.0312957 | Time: 1640.59 ms [2022-03-14 03:19:17 main:574] : INFO : Epoch 1735 | loss: 0.0310088 | val_loss: 0.0312991 | Time: 1649.8 ms [2022-03-14 03:19:19 main:574] : INFO : Epoch 1736 | loss: 0.0310081 | val_loss: 0.0313155 | Time: 1645.39 ms [2022-03-14 03:19:20 main:574] : INFO : Epoch 1737 | loss: 0.0310065 | val_loss: 0.0312981 | Time: 1648.74 ms [2022-03-14 03:19:22 main:574] : INFO : Epoch 1738 | loss: 0.0310008 | val_loss: 0.0313048 | Time: 1647.92 ms [2022-03-14 03:19:24 main:574] : INFO : Epoch 1739 | loss: 0.031004 | val_loss: 0.0313073 | Time: 1660.51 ms [2022-03-14 03:19:25 main:574] : INFO : Epoch 1740 | loss: 0.0310013 | val_loss: 0.0312947 | Time: 1643.68 ms [2022-03-14 03:19:27 main:574] : INFO : Epoch 1741 | loss: 0.0310086 | val_loss: 0.0313168 | Time: 1647.32 ms [2022-03-14 03:19:29 main:574] : INFO : Epoch 1742 | loss: 0.0310136 | val_loss: 0.0313198 | Time: 1646.16 ms [2022-03-14 03:19:30 main:574] : INFO : Epoch 1743 | loss: 0.0310175 | val_loss: 0.0312987 | Time: 1651.63 ms [2022-03-14 03:19:32 main:574] : INFO : Epoch 1744 | loss: 0.031009 | val_loss: 0.0313165 | Time: 1640.02 ms [2022-03-14 03:19:34 main:574] : INFO : Epoch 1745 | loss: 0.0310081 | val_loss: 0.0313117 | Time: 1660.2 ms [2022-03-14 03:19:35 main:574] : INFO : Epoch 1746 | loss: 0.0310069 | val_loss: 0.0313037 | Time: 1634.58 ms [2022-03-14 03:19:37 main:574] : INFO : Epoch 1747 | loss: 0.0310081 | val_loss: 0.0313219 | Time: 1643.76 ms [2022-03-14 03:19:39 main:574] : INFO : Epoch 1748 | loss: 0.0310024 | val_loss: 0.0313222 | Time: 1639.9 ms [2022-03-14 03:19:40 main:574] : INFO : Epoch 1749 | loss: 0.0310019 | val_loss: 0.0313147 | Time: 1652.52 ms [2022-03-14 03:19:42 main:574] : INFO : Epoch 1750 | loss: 0.0310027 | val_loss: 0.0313216 | Time: 1648.65 ms [2022-03-14 03:19:44 main:574] : INFO : Epoch 1751 | loss: 0.0310114 | val_loss: 0.0313 | Time: 1653.75 ms [2022-03-14 03:19:45 main:574] : INFO : Epoch 1752 | loss: 0.0310025 | val_loss: 0.0313168 | Time: 1642.25 ms [2022-03-14 03:19:47 main:574] : INFO : Epoch 1753 | loss: 0.0310023 | val_loss: 0.0312967 | Time: 1645.88 ms [2022-03-14 03:19:49 main:574] : INFO : Epoch 1754 | loss: 0.0310053 | val_loss: 0.0313194 | Time: 1649.14 ms [2022-03-14 03:19:50 main:574] : INFO : Epoch 1755 | loss: 0.0310056 | val_loss: 0.0313171 | Time: 1652.55 ms [2022-03-14 03:19:52 main:574] : INFO : Epoch 1756 | loss: 0.0310194 | val_loss: 0.0312994 | Time: 1642.57 ms [2022-03-14 03:19:53 main:574] : INFO : Epoch 1757 | loss: 0.03101 | val_loss: 0.0313189 | Time: 1646.2 ms [2022-03-14 03:19:55 main:574] : INFO : Epoch 1758 | loss: 0.0310104 | val_loss: 0.0313052 | Time: 1641.37 ms [2022-03-14 03:19:57 main:574] : INFO : Epoch 1759 | loss: 0.031005 | val_loss: 0.0313076 | Time: 1637.55 ms [2022-03-14 03:19:58 main:574] : INFO : Epoch 1760 | loss: 0.0310055 | val_loss: 0.0313317 | Time: 1651.38 ms [2022-03-14 03:20:00 main:574] : INFO : Epoch 1761 | loss: 0.0310052 | val_loss: 0.0312974 | Time: 1646.55 ms [2022-03-14 03:20:02 main:574] : INFO : Epoch 1762 | loss: 0.0310126 | val_loss: 0.0312722 | Time: 1651.5 ms [2022-03-14 03:20:03 main:574] : INFO : Epoch 1763 | loss: 0.0310085 | val_loss: 0.0312924 | Time: 1654.82 ms [2022-03-14 03:20:05 main:574] : INFO : Epoch 1764 | loss: 0.0310125 | val_loss: 0.0312965 | Time: 1642.3 ms [2022-03-14 03:20:07 main:574] : INFO : Epoch 1765 | loss: 0.0310114 | val_loss: 0.0313028 | Time: 1639.57 ms [2022-03-14 03:20:08 main:574] : INFO : Epoch 1766 | loss: 0.0310122 | val_loss: 0.0312885 | Time: 1645.76 ms [2022-03-14 03:20:10 main:574] : INFO : Epoch 1767 | loss: 0.0310145 | val_loss: 0.0313037 | Time: 1651.5 ms [2022-03-14 03:20:12 main:574] : INFO : Epoch 1768 | loss: 0.0310133 | val_loss: 0.0313279 | Time: 1653.42 ms [2022-03-14 03:20:13 main:574] : INFO : Epoch 1769 | loss: 0.0310018 | val_loss: 0.0313072 | Time: 1655.48 ms [2022-03-14 03:20:15 main:574] : INFO : Epoch 1770 | loss: 0.0310036 | val_loss: 0.0313162 | Time: 1641.88 ms [2022-03-14 03:20:17 main:574] : INFO : Epoch 1771 | loss: 0.0310043 | val_loss: 0.0313021 | Time: 1633.87 ms [2022-03-14 03:20:18 main:574] : INFO : Epoch 1772 | loss: 0.0310053 | val_loss: 0.0312987 | Time: 1638.18 ms [2022-03-14 03:20:20 main:574] : INFO : Epoch 1773 | loss: 0.0310085 | val_loss: 0.0313097 | Time: 1642.87 ms [2022-03-14 03:20:22 main:574] : INFO : Epoch 1774 | loss: 0.0310054 | val_loss: 0.0312961 | Time: 1642.83 ms [2022-03-14 03:20:23 main:574] : INFO : Epoch 1775 | loss: 0.0310069 | val_loss: 0.0312999 | Time: 1683.49 ms [2022-03-14 03:20:25 main:574] : INFO : Epoch 1776 | loss: 0.0310029 | val_loss: 0.0313036 | Time: 1650.69 ms [2022-03-14 03:20:27 main:574] : INFO : Epoch 1777 | loss: 0.0310038 | val_loss: 0.0313322 | Time: 1638.38 ms [2022-03-14 03:20:28 main:574] : INFO : Epoch 1778 | loss: 0.0310058 | val_loss: 0.0312808 | Time: 1638.07 ms [2022-03-14 03:20:30 main:574] : INFO : Epoch 1779 | loss: 0.0310073 | val_loss: 0.0313386 | Time: 1637.89 ms [2022-03-14 03:20:32 main:574] : INFO : Epoch 1780 | loss: 0.0310084 | val_loss: 0.0313036 | Time: 1648.95 ms [2022-03-14 03:20:33 main:574] : INFO : Epoch 1781 | loss: 0.0310093 | val_loss: 0.0312903 | Time: 1654.78 ms [2022-03-14 03:20:35 main:574] : INFO : Epoch 1782 | loss: 0.0310097 | val_loss: 0.0312875 | Time: 1659.77 ms [2022-03-14 03:20:36 main:574] : INFO : Epoch 1783 | loss: 0.0310053 | val_loss: 0.031334 | Time: 1639.92 ms [2022-03-14 03:20:38 main:574] : INFO : Epoch 1784 | loss: 0.0310021 | val_loss: 0.0313048 | Time: 1645.43 ms [2022-03-14 03:20:40 main:574] : INFO : Epoch 1785 | loss: 0.0309998 | val_loss: 0.031306 | Time: 1649.62 ms [2022-03-14 03:20:41 main:574] : INFO : Epoch 1786 | loss: 0.0310097 | val_loss: 0.0312983 | Time: 1663.53 ms [2022-03-14 03:20:43 main:574] : INFO : Epoch 1787 | loss: 0.0310068 | val_loss: 0.0313166 | Time: 1690.76 ms [2022-03-14 03:20:45 main:574] : INFO : Epoch 1788 | loss: 0.0310042 | val_loss: 0.0313354 | Time: 1645.95 ms [2022-03-14 03:20:46 main:574] : INFO : Epoch 1789 | loss: 0.0310014 | val_loss: 0.0313251 | Time: 1639.8 ms [2022-03-14 03:20:48 main:574] : INFO : Epoch 1790 | loss: 0.0310004 | val_loss: 0.0313099 | Time: 1647.96 ms [2022-03-14 03:20:50 main:574] : INFO : Epoch 1791 | loss: 0.0310062 | val_loss: 0.0313249 | Time: 1637.2 ms [2022-03-14 03:20:51 main:574] : INFO : Epoch 1792 | loss: 0.0310046 | val_loss: 0.0313025 | Time: 1654.25 ms [2022-03-14 03:20:53 main:574] : INFO : Epoch 1793 | loss: 0.0310023 | val_loss: 0.0312996 | Time: 1654.64 ms [2022-03-14 03:20:55 main:574] : INFO : Epoch 1794 | loss: 0.0310068 | val_loss: 0.0313219 | Time: 1646.82 ms [2022-03-14 03:20:56 main:574] : INFO : Epoch 1795 | loss: 0.031003 | val_loss: 0.0312859 | Time: 1636.18 ms [2022-03-14 03:20:58 main:574] : INFO : Epoch 1796 | loss: 0.0310002 | val_loss: 0.0313079 | Time: 1650.94 ms [2022-03-14 03:21:00 main:574] : INFO : Epoch 1797 | loss: 0.0310076 | val_loss: 0.0313432 | Time: 1634.64 ms [2022-03-14 03:21:01 main:574] : INFO : Epoch 1798 | loss: 0.031 | val_loss: 0.031332 | Time: 1655.84 ms [2022-03-14 03:21:03 main:574] : INFO : Epoch 1799 | loss: 0.0309995 | val_loss: 0.0313045 | Time: 1654.76 ms [2022-03-14 03:21:05 main:574] : INFO : Epoch 1800 | loss: 0.0310114 | val_loss: 0.0313079 | Time: 1642.69 ms [2022-03-14 03:21:06 main:574] : INFO : Epoch 1801 | loss: 0.0310086 | val_loss: 0.0313158 | Time: 1641.93 ms [2022-03-14 03:21:08 main:574] : INFO : Epoch 1802 | loss: 0.031011 | val_loss: 0.0312907 | Time: 1642.74 ms [2022-03-14 03:21:10 main:574] : INFO : Epoch 1803 | loss: 0.0310122 | val_loss: 0.0312927 | Time: 1631.57 ms [2022-03-14 03:21:11 main:574] : INFO : Epoch 1804 | loss: 0.0310043 | val_loss: 0.0313234 | Time: 1653.78 ms [2022-03-14 03:21:13 main:574] : INFO : Epoch 1805 | loss: 0.0310046 | val_loss: 0.0313196 | Time: 1656.32 ms [2022-03-14 03:21:15 main:574] : INFO : Epoch 1806 | loss: 0.0310126 | val_loss: 0.0313266 | Time: 1647.34 ms [2022-03-14 03:21:16 main:574] : INFO : Epoch 1807 | loss: 0.0310046 | val_loss: 0.0313014 | Time: 1634.3 ms [2022-03-14 03:21:18 main:574] : INFO : Epoch 1808 | loss: 0.0310034 | val_loss: 0.031349 | Time: 1644.49 ms [2022-03-14 03:21:19 main:574] : INFO : Epoch 1809 | loss: 0.031006 | val_loss: 0.0313239 | Time: 1638.99 ms [2022-03-14 03:21:21 main:574] : INFO : Epoch 1810 | loss: 0.0310042 | val_loss: 0.0313031 | Time: 1647.33 ms [2022-03-14 03:21:23 main:574] : INFO : Epoch 1811 | loss: 0.031015 | val_loss: 0.0312861 | Time: 1652.15 ms [2022-03-14 03:21:24 main:574] : INFO : Epoch 1812 | loss: 0.0310098 | val_loss: 0.0313229 | Time: 1659.98 ms [2022-03-14 03:21:26 main:574] : INFO : Epoch 1813 | loss: 0.0310092 | val_loss: 0.0312865 | Time: 1637.32 ms [2022-03-14 03:21:28 main:574] : INFO : Epoch 1814 | loss: 0.0310128 | val_loss: 0.0312955 | Time: 1642.57 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 970) [2022-03-14 03:22:13 main:435] : INFO : Set logging level to 1 [2022-03-14 03:22:13 main:441] : INFO : Running in BOINC Client mode [2022-03-14 03:22:13 main:444] : INFO : Resolving all filenames [2022-03-14 03:22:13 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-14 03:22:13 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-14 03:22:13 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-14 03:22:13 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-03-14 03:22:13 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-14 03:22:13 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-14 03:22:13 main:474] : INFO : Configuration: [2022-03-14 03:22:13 main:475] : INFO : Model type: GRU [2022-03-14 03:22:13 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-14 03:22:13 main:477] : INFO : Max Epochs: 2048 [2022-03-14 03:22:13 main:478] : INFO : Batch Size: 128 [2022-03-14 03:22:13 main:479] : INFO : Learning Rate: 0.01 [2022-03-14 03:22:13 main:480] : INFO : Patience: 10 [2022-03-14 03:22:13 main:481] : INFO : Hidden Width: 12 [2022-03-14 03:22:13 main:482] : INFO : # Recurrent Layers: 4 [2022-03-14 03:22:13 main:483] : INFO : # Backend Layers: 4 [2022-03-14 03:22:13 main:484] : INFO : # Threads: 1 [2022-03-14 03:22:13 main:486] : INFO : Preparing Dataset [2022-03-14 03:22:13 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-14 03:22:14 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-14 03:22:15 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-14 03:22:15 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-14 03:22:15 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-14 03:22:15 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-14 03:22:15 main:494] : INFO : Creating Model [2022-03-14 03:22:15 main:507] : INFO : Preparing config file [2022-03-14 03:22:15 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-14 03:22:15 main:512] : INFO : Loading config [2022-03-14 03:22:15 main:514] : INFO : Loading state [2022-03-14 03:22:15 main:559] : INFO : Loading DataLoader into Memory [2022-03-14 03:22:16 main:562] : INFO : Starting Training [2022-03-14 03:22:17 main:574] : INFO : Epoch 1794 | loss: 0.0310863 | val_loss: 0.0312851 | Time: 1839.87 ms [2022-03-14 03:22:19 main:574] : INFO : Epoch 1795 | loss: 0.0310324 | val_loss: 0.0312897 | Time: 1621.89 ms [2022-03-14 03:22:21 main:574] : INFO : Epoch 1796 | loss: 0.0310214 | val_loss: 0.0312956 | Time: 1631.51 ms [2022-03-14 03:22:22 main:574] : INFO : Epoch 1797 | loss: 0.0310094 | val_loss: 0.0313087 | Time: 1630.97 ms [2022-03-14 03:22:24 main:574] : INFO : Epoch 1798 | loss: 0.0310076 | val_loss: 0.0313153 | Time: 1647.73 ms [2022-03-14 03:22:26 main:574] : INFO : Epoch 1799 | loss: 0.0310295 | val_loss: 0.0312902 | Time: 1645 ms [2022-03-14 03:22:27 main:574] : INFO : Epoch 1800 | loss: 0.0310139 | val_loss: 0.0313035 | Time: 1631.43 ms [2022-03-14 03:22:29 main:574] : INFO : Epoch 1801 | loss: 0.0310161 | val_loss: 0.0312953 | Time: 1638.6 ms [2022-03-14 03:22:31 main:574] : INFO : Epoch 1802 | loss: 0.0310078 | val_loss: 0.0312958 | Time: 1691.72 ms [2022-03-14 03:22:32 main:574] : INFO : Epoch 1803 | loss: 0.0310051 | val_loss: 0.0313074 | Time: 1649.9 ms [2022-03-14 03:22:34 main:574] : INFO : Epoch 1804 | loss: 0.0310069 | val_loss: 0.0312635 | Time: 1645.09 ms [2022-03-14 03:22:35 main:574] : INFO : Epoch 1805 | loss: 0.0310143 | val_loss: 0.0312725 | Time: 1639.45 ms [2022-03-14 03:22:37 main:574] : INFO : Epoch 1806 | loss: 0.0310101 | val_loss: 0.031312 | Time: 1638.14 ms [2022-03-14 03:22:39 main:574] : INFO : Epoch 1807 | loss: 0.0310053 | val_loss: 0.0313021 | Time: 1643.41 ms [2022-03-14 03:22:40 main:574] : INFO : Epoch 1808 | loss: 0.0310066 | val_loss: 0.031325 | Time: 1642.68 ms [2022-03-14 03:22:42 main:574] : INFO : Epoch 1809 | loss: 0.0310037 | val_loss: 0.0313084 | Time: 1647.06 ms [2022-03-14 03:22:44 main:574] : INFO : Epoch 1810 | loss: 0.0310063 | val_loss: 0.0312883 | Time: 1642.4 ms [2022-03-14 03:22:45 main:574] : INFO : Epoch 1811 | loss: 0.0310003 | val_loss: 0.0313093 | Time: 1640.29 ms [2022-03-14 03:22:47 main:574] : INFO : Epoch 1812 | loss: 0.0310008 | val_loss: 0.0312956 | Time: 1634.26 ms [2022-03-14 03:22:49 main:574] : INFO : Epoch 1813 | loss: 0.031022 | val_loss: 0.0312599 | Time: 1642.46 ms [2022-03-14 03:22:50 main:574] : INFO : Epoch 1814 | loss: 0.0310138 | val_loss: 0.0312952 | Time: 1645.41 ms [2022-03-14 03:22:52 main:574] : INFO : Epoch 1815 | loss: 0.0310055 | val_loss: 0.0313011 | Time: 1651.72 ms [2022-03-14 03:22:54 main:574] : INFO : Epoch 1816 | loss: 0.0310116 | val_loss: 0.0313161 | Time: 1661.32 ms [2022-03-14 03:22:55 main:574] : INFO : Epoch 1817 | loss: 0.0310159 | val_loss: 0.0313029 | Time: 1659.73 ms [2022-03-14 03:22:57 main:574] : INFO : Epoch 1818 | loss: 0.0310085 | val_loss: 0.0312963 | Time: 1631.69 ms [2022-03-14 03:22:59 main:574] : INFO : Epoch 1819 | loss: 0.0310016 | val_loss: 0.0313058 | Time: 1643.62 ms [2022-03-14 03:23:00 main:574] : INFO : Epoch 1820 | loss: 0.0310057 | val_loss: 0.0312977 | Time: 1641.23 ms [2022-03-14 03:23:02 main:574] : INFO : Epoch 1821 | loss: 0.0310079 | val_loss: 0.0312964 | Time: 1652.69 ms [2022-03-14 03:23:04 main:574] : INFO : Epoch 1822 | loss: 0.0310057 | val_loss: 0.0312987 | Time: 1649.67 ms [2022-03-14 03:23:05 main:574] : INFO : Epoch 1823 | loss: 0.0310074 | val_loss: 0.031299 | Time: 1640.75 ms [2022-03-14 03:23:07 main:574] : INFO : Epoch 1824 | loss: 0.0310049 | val_loss: 0.0313155 | Time: 1634.41 ms [2022-03-14 03:23:08 main:574] : INFO : Epoch 1825 | loss: 0.0310049 | val_loss: 0.0313349 | Time: 1648.52 ms [2022-03-14 03:23:10 main:574] : INFO : Epoch 1826 | loss: 0.0310166 | val_loss: 0.0312946 | Time: 1642.75 ms [2022-03-14 03:23:12 main:574] : INFO : Epoch 1827 | loss: 0.0310087 | val_loss: 0.0312984 | Time: 1649.05 ms [2022-03-14 03:23:13 main:574] : INFO : Epoch 1828 | loss: 0.0310056 | val_loss: 0.0313183 | Time: 1646.19 ms [2022-03-14 03:23:15 main:574] : INFO : Epoch 1829 | loss: 0.0310006 | val_loss: 0.0313245 | Time: 1642.65 ms [2022-03-14 03:23:17 main:574] : INFO : Epoch 1830 | loss: 0.0310162 | val_loss: 0.031269 | Time: 1636.01 ms [2022-03-14 03:23:18 main:574] : INFO : Epoch 1831 | loss: 0.0310135 | val_loss: 0.0313254 | Time: 1644.72 ms [2022-03-14 03:23:20 main:574] : INFO : Epoch 1832 | loss: 0.031005 | val_loss: 0.0312716 | Time: 1638 ms [2022-03-14 03:23:22 main:574] : INFO : Epoch 1833 | loss: 0.0310106 | val_loss: 0.0312827 | Time: 1652.97 ms [2022-03-14 03:23:23 main:574] : INFO : Epoch 1834 | loss: 0.0310081 | val_loss: 0.031305 | Time: 1653.3 ms [2022-03-14 03:23:25 main:574] : INFO : Epoch 1835 | loss: 0.0310062 | val_loss: 0.0313214 | Time: 1653.96 ms [2022-03-14 03:23:27 main:574] : INFO : Epoch 1836 | loss: 0.0310082 | val_loss: 0.031297 | Time: 1649.4 ms [2022-03-14 03:23:28 main:574] : INFO : Epoch 1837 | loss: 0.0310015 | val_loss: 0.0313128 | Time: 1647.65 ms [2022-03-14 03:23:30 main:574] : INFO : Epoch 1838 | loss: 0.0310023 | val_loss: 0.0313163 | Time: 1655.21 ms [2022-03-14 03:23:32 main:574] : INFO : Epoch 1839 | loss: 0.0310052 | val_loss: 0.0313264 | Time: 1654.51 ms [2022-03-14 03:23:33 main:574] : INFO : Epoch 1840 | loss: 0.0310058 | val_loss: 0.0313093 | Time: 1645.15 ms [2022-03-14 03:23:35 main:574] : INFO : Epoch 1841 | loss: 0.0310082 | val_loss: 0.0312989 | Time: 1646.04 ms [2022-03-14 03:23:37 main:574] : INFO : Epoch 1842 | loss: 0.0310192 | val_loss: 0.0313063 | Time: 1632.73 ms [2022-03-14 03:23:38 main:574] : INFO : Epoch 1843 | loss: 0.0310172 | val_loss: 0.0313097 | Time: 1646.63 ms [2022-03-14 03:23:40 main:574] : INFO : Epoch 1844 | loss: 0.0310037 | val_loss: 0.0313046 | Time: 1638.74 ms [2022-03-14 03:23:42 main:574] : INFO : Epoch 1845 | loss: 0.0310029 | val_loss: 0.0313171 | Time: 1656.49 ms [2022-03-14 03:23:43 main:574] : INFO : Epoch 1846 | loss: 0.0310142 | val_loss: 0.0312998 | Time: 1644.88 ms [2022-03-14 03:23:45 main:574] : INFO : Epoch 1847 | loss: 0.0310021 | val_loss: 0.0313073 | Time: 1644.08 ms [2022-03-14 03:23:46 main:574] : INFO : Epoch 1848 | loss: 0.0310057 | val_loss: 0.0313186 | Time: 1639.18 ms [2022-03-14 03:23:48 main:574] : INFO : Epoch 1849 | loss: 0.0310025 | val_loss: 0.0313139 | Time: 1654.56 ms [2022-03-14 03:23:50 main:574] : INFO : Epoch 1850 | loss: 0.0310004 | val_loss: 0.0313145 | Time: 1655.1 ms [2022-03-14 03:23:51 main:574] : INFO : Epoch 1851 | loss: 0.0310035 | val_loss: 0.0313004 | Time: 1657.22 ms [2022-03-14 03:23:53 main:574] : INFO : Epoch 1852 | loss: 0.0310078 | val_loss: 0.031323 | Time: 1658.92 ms [2022-03-14 03:23:55 main:574] : INFO : Epoch 1853 | loss: 0.0310069 | val_loss: 0.0313063 | Time: 1641.96 ms [2022-03-14 03:23:56 main:574] : INFO : Epoch 1854 | loss: 0.0310138 | val_loss: 0.0312951 | Time: 1639.65 ms [2022-03-14 03:23:58 main:574] : INFO : Epoch 1855 | loss: 0.0310125 | val_loss: 0.0313054 | Time: 1650.37 ms [2022-03-14 03:24:00 main:574] : INFO : Epoch 1856 | loss: 0.0310075 | val_loss: 0.0313068 | Time: 1647.85 ms [2022-03-14 03:24:01 main:574] : INFO : Epoch 1857 | loss: 0.0309979 | val_loss: 0.0313007 | Time: 1656.39 ms [2022-03-14 03:24:03 main:574] : INFO : Epoch 1858 | loss: 0.0309968 | val_loss: 0.0313128 | Time: 1647.51 ms [2022-03-14 03:24:05 main:574] : INFO : Epoch 1859 | loss: 0.0310012 | val_loss: 0.0312863 | Time: 1646.56 ms [2022-03-14 03:24:06 main:574] : INFO : Epoch 1860 | loss: 0.0310055 | val_loss: 0.0313062 | Time: 1641.91 ms [2022-03-14 03:24:08 main:574] : INFO : Epoch 1861 | loss: 0.0310023 | val_loss: 0.0313242 | Time: 1649.75 ms [2022-03-14 03:24:10 main:574] : INFO : Epoch 1862 | loss: 0.031002 | val_loss: 0.0313039 | Time: 1639.75 ms [2022-03-14 03:24:11 main:574] : INFO : Epoch 1863 | loss: 0.0310006 | val_loss: 0.0313095 | Time: 1653.6 ms [2022-03-14 03:24:13 main:574] : INFO : Epoch 1864 | loss: 0.0310097 | val_loss: 0.0313242 | Time: 1654.56 ms [2022-03-14 03:24:15 main:574] : INFO : Epoch 1865 | loss: 0.0310089 | val_loss: 0.0312986 | Time: 1654.46 ms [2022-03-14 03:24:16 main:574] : INFO : Epoch 1866 | loss: 0.0310064 | val_loss: 0.0313208 | Time: 1646.48 ms [2022-03-14 03:24:18 main:574] : INFO : Epoch 1867 | loss: 0.0310116 | val_loss: 0.0313026 | Time: 1642.07 ms [2022-03-14 03:24:20 main:574] : INFO : Epoch 1868 | loss: 0.0310103 | val_loss: 0.0313174 | Time: 1649.75 ms [2022-03-14 03:24:21 main:574] : INFO : Epoch 1869 | loss: 0.0310045 | val_loss: 0.0313158 | Time: 1649.98 ms [2022-03-14 03:24:23 main:574] : INFO : Epoch 1870 | loss: 0.0310089 | val_loss: 0.0312772 | Time: 1661.85 ms [2022-03-14 03:24:25 main:574] : INFO : Epoch 1871 | loss: 0.0310086 | val_loss: 0.0313245 | Time: 1642.95 ms [2022-03-14 03:24:26 main:574] : INFO : Epoch 1872 | loss: 0.0310069 | val_loss: 0.0312837 | Time: 1658.47 ms [2022-03-14 03:24:28 main:574] : INFO : Epoch 1873 | loss: 0.0310091 | val_loss: 0.0312881 | Time: 1645.83 ms [2022-03-14 03:24:30 main:574] : INFO : Epoch 1874 | loss: 0.0310003 | val_loss: 0.0313156 | Time: 1647.74 ms [2022-03-14 03:24:31 main:574] : INFO : Epoch 1875 | loss: 0.0310038 | val_loss: 0.0313328 | Time: 1652.18 ms [2022-03-14 03:24:33 main:574] : INFO : Epoch 1876 | loss: 0.0310094 | val_loss: 0.0313167 | Time: 1663.23 ms [2022-03-14 03:24:35 main:574] : INFO : Epoch 1877 | loss: 0.0310062 | val_loss: 0.0312951 | Time: 1651.1 ms [2022-03-14 03:24:36 main:574] : INFO : Epoch 1878 | loss: 0.0310094 | val_loss: 0.031274 | Time: 1645.34 ms [2022-03-14 03:24:38 main:574] : INFO : Epoch 1879 | loss: 0.0310082 | val_loss: 0.0313059 | Time: 1638.94 ms [2022-03-14 03:24:39 main:574] : INFO : Epoch 1880 | loss: 0.0310094 | val_loss: 0.031322 | Time: 1643.5 ms [2022-03-14 03:24:41 main:574] : INFO : Epoch 1881 | loss: 0.0310096 | val_loss: 0.0313049 | Time: 1649.43 ms [2022-03-14 03:24:43 main:574] : INFO : Epoch 1882 | loss: 0.0310216 | val_loss: 0.0313254 | Time: 1658.23 ms [2022-03-14 03:24:44 main:574] : INFO : Epoch 1883 | loss: 0.0310132 | val_loss: 0.031308 | Time: 1642.08 ms [2022-03-14 03:24:46 main:574] : INFO : Epoch 1884 | loss: 0.0310091 | val_loss: 0.0313295 | Time: 1648.82 ms [2022-03-14 03:24:48 main:574] : INFO : Epoch 1885 | loss: 0.0310028 | val_loss: 0.0313026 | Time: 1639.13 ms [2022-03-14 03:24:49 main:574] : INFO : Epoch 1886 | loss: 0.0310028 | val_loss: 0.0313089 | Time: 1644.8 ms [2022-03-14 03:24:51 main:574] : INFO : Epoch 1887 | loss: 0.0309972 | val_loss: 0.0313125 | Time: 1644.23 ms [2022-03-14 03:24:53 main:574] : INFO : Epoch 1888 | loss: 0.0310029 | val_loss: 0.0312841 | Time: 1658.63 ms [2022-03-14 03:24:54 main:574] : INFO : Epoch 1889 | loss: 0.0310111 | val_loss: 0.0313259 | Time: 1639.83 ms [2022-03-14 03:24:56 main:574] : INFO : Epoch 1890 | loss: 0.0310022 | val_loss: 0.0313123 | Time: 1652.37 ms [2022-03-14 03:24:58 main:574] : INFO : Epoch 1891 | loss: 0.0309987 | val_loss: 0.0313168 | Time: 1639.57 ms [2022-03-14 03:24:59 main:574] : INFO : Epoch 1892 | loss: 0.0309991 | val_loss: 0.0313098 | Time: 1651.05 ms [2022-03-14 03:25:01 main:574] : INFO : Epoch 1893 | loss: 0.031004 | val_loss: 0.0313136 | Time: 1642.04 ms [2022-03-14 03:25:03 main:574] : INFO : Epoch 1894 | loss: 0.0309978 | val_loss: 0.0313097 | Time: 1650.33 ms [2022-03-14 03:25:04 main:574] : INFO : Epoch 1895 | loss: 0.031004 | val_loss: 0.0313092 | Time: 1653.52 ms [2022-03-14 03:25:06 main:574] : INFO : Epoch 1896 | loss: 0.0310109 | val_loss: 0.0313057 | Time: 1645.5 ms [2022-03-14 03:25:08 main:574] : INFO : Epoch 1897 | loss: 0.0310136 | val_loss: 0.0313336 | Time: 1642.69 ms [2022-03-14 03:25:09 main:574] : INFO : Epoch 1898 | loss: 0.0310068 | val_loss: 0.0313014 | Time: 1658.79 ms [2022-03-14 03:25:11 main:574] : INFO : Epoch 1899 | loss: 0.0310036 | val_loss: 0.0313103 | Time: 1646.56 ms [2022-03-14 03:25:13 main:574] : INFO : Epoch 1900 | loss: 0.0310049 | val_loss: 0.0313296 | Time: 1667.6 ms [2022-03-14 03:25:14 main:574] : INFO : Epoch 1901 | loss: 0.031012 | val_loss: 0.0313263 | Time: 1634.48 ms [2022-03-14 03:25:16 main:574] : INFO : Epoch 1902 | loss: 0.0310101 | val_loss: 0.0312914 | Time: 1649.75 ms [2022-03-14 03:25:18 main:574] : INFO : Epoch 1903 | loss: 0.0310087 | val_loss: 0.0312951 | Time: 1652.11 ms [2022-03-14 03:25:19 main:574] : INFO : Epoch 1904 | loss: 0.0310012 | val_loss: 0.0313294 | Time: 1659.35 ms [2022-03-14 03:25:21 main:574] : INFO : Epoch 1905 | loss: 0.0310033 | val_loss: 0.03127 | Time: 1645.05 ms [2022-03-14 03:25:23 main:574] : INFO : Epoch 1906 | loss: 0.0310158 | val_loss: 0.0312866 | Time: 1652.29 ms [2022-03-14 03:25:24 main:574] : INFO : Epoch 1907 | loss: 0.0310069 | val_loss: 0.031287 | Time: 1675.85 ms [2022-03-14 03:25:26 main:574] : INFO : Epoch 1908 | loss: 0.0310075 | val_loss: 0.0313004 | Time: 1647.96 ms [2022-03-14 03:25:28 main:574] : INFO : Epoch 1909 | loss: 0.0309988 | val_loss: 0.0313247 | Time: 1643.67 ms [2022-03-14 03:25:29 main:574] : INFO : Epoch 1910 | loss: 0.0310003 | val_loss: 0.0313354 | Time: 1645.42 ms [2022-03-14 03:25:31 main:574] : INFO : Epoch 1911 | loss: 0.0310064 | val_loss: 0.0312872 | Time: 1651.46 ms [2022-03-14 03:25:32 main:574] : INFO : Epoch 1912 | loss: 0.0310072 | val_loss: 0.0313227 | Time: 1648.71 ms [2022-03-14 03:25:34 main:574] : INFO : Epoch 1913 | loss: 0.031012 | val_loss: 0.0312971 | Time: 1649.81 ms [2022-03-14 03:25:36 main:574] : INFO : Epoch 1914 | loss: 0.0310084 | val_loss: 0.0313318 | Time: 1647.4 ms [2022-03-14 03:25:37 main:574] : INFO : Epoch 1915 | loss: 0.0310047 | val_loss: 0.0313145 | Time: 1656.87 ms [2022-03-14 03:25:39 main:574] : INFO : Epoch 1916 | loss: 0.0309989 | val_loss: 0.031322 | Time: 1647.97 ms [2022-03-14 03:25:41 main:574] : INFO : Epoch 1917 | loss: 0.0310024 | val_loss: 0.0313075 | Time: 1654.85 ms [2022-03-14 03:25:42 main:574] : INFO : Epoch 1918 | loss: 0.0310014 | val_loss: 0.0313163 | Time: 1659.23 ms [2022-03-14 03:25:44 main:574] : INFO : Epoch 1919 | loss: 0.0310145 | val_loss: 0.0312884 | Time: 1635.02 ms [2022-03-14 03:25:46 main:574] : INFO : Epoch 1920 | loss: 0.0310209 | val_loss: 0.0312975 | Time: 1648.64 ms [2022-03-14 03:25:47 main:574] : INFO : Epoch 1921 | loss: 0.0310048 | val_loss: 0.0313119 | Time: 1640.09 ms [2022-03-14 03:25:49 main:574] : INFO : Epoch 1922 | loss: 0.0310043 | val_loss: 0.0313196 | Time: 1655.14 ms [2022-03-14 03:25:51 main:574] : INFO : Epoch 1923 | loss: 0.0310055 | val_loss: 0.0312907 | Time: 1644.11 ms [2022-03-14 03:25:52 main:574] : INFO : Epoch 1924 | loss: 0.031017 | val_loss: 0.0312992 | Time: 1650.46 ms [2022-03-14 03:25:54 main:574] : INFO : Epoch 1925 | loss: 0.0310169 | val_loss: 0.0312767 | Time: 1646.56 ms [2022-03-14 03:25:56 main:574] : INFO : Epoch 1926 | loss: 0.0310149 | val_loss: 0.0312964 | Time: 1648.15 ms [2022-03-14 03:25:57 main:574] : INFO : Epoch 1927 | loss: 0.0310137 | val_loss: 0.0313103 | Time: 1644.35 ms [2022-03-14 03:25:59 main:574] : INFO : Epoch 1928 | loss: 0.0310077 | val_loss: 0.0313159 | Time: 1643.52 ms [2022-03-14 03:26:01 main:574] : INFO : Epoch 1929 | loss: 0.0310036 | val_loss: 0.0313299 | Time: 1640.69 ms [2022-03-14 03:26:02 main:574] : INFO : Epoch 1930 | loss: 0.0310092 | val_loss: 0.031279 | Time: 1648.85 ms [2022-03-14 03:26:04 main:574] : INFO : Epoch 1931 | loss: 0.0310108 | val_loss: 0.0313131 | Time: 1648.78 ms [2022-03-14 03:26:06 main:574] : INFO : Epoch 1932 | loss: 0.0310027 | val_loss: 0.0312931 | Time: 1647.83 ms [2022-03-14 03:26:07 main:574] : INFO : Epoch 1933 | loss: 0.0310023 | val_loss: 0.0313092 | Time: 1636.09 ms [2022-03-14 03:26:09 main:574] : INFO : Epoch 1934 | loss: 0.0309999 | val_loss: 0.031319 | Time: 1649.8 ms [2022-03-14 03:26:11 main:574] : INFO : Epoch 1935 | loss: 0.0310009 | val_loss: 0.0313093 | Time: 1649.37 ms [2022-03-14 03:26:12 main:574] : INFO : Epoch 1936 | loss: 0.0310025 | val_loss: 0.0312992 | Time: 1649.15 ms [2022-03-14 03:26:14 main:574] : INFO : Epoch 1937 | loss: 0.031008 | val_loss: 0.0312899 | Time: 1647.85 ms [2022-03-14 03:26:16 main:574] : INFO : Epoch 1938 | loss: 0.0310123 | val_loss: 0.0313064 | Time: 1647.45 ms [2022-03-14 03:26:17 main:574] : INFO : Epoch 1939 | loss: 0.0310072 | val_loss: 0.0312982 | Time: 1642.98 ms [2022-03-14 03:26:19 main:574] : INFO : Epoch 1940 | loss: 0.0309994 | val_loss: 0.0313134 | Time: 1646.13 ms [2022-03-14 03:26:20 main:574] : INFO : Epoch 1941 | loss: 0.0309996 | val_loss: 0.0313001 | Time: 1642.25 ms [2022-03-14 03:26:22 main:574] : INFO : Epoch 1942 | loss: 0.0310075 | val_loss: 0.0313327 | Time: 1648.09 ms [2022-03-14 03:26:24 main:574] : INFO : Epoch 1943 | loss: 0.0310053 | val_loss: 0.0313002 | Time: 1654.02 ms [2022-03-14 03:26:25 main:574] : INFO : Epoch 1944 | loss: 0.0310086 | val_loss: 0.0312988 | Time: 1651.5 ms [2022-03-14 03:26:27 main:574] : INFO : Epoch 1945 | loss: 0.0310098 | val_loss: 0.0312959 | Time: 1641.88 ms [2022-03-14 03:26:29 main:574] : INFO : Epoch 1946 | loss: 0.0310047 | val_loss: 0.0313169 | Time: 1646.77 ms [2022-03-14 03:26:30 main:574] : INFO : Epoch 1947 | loss: 0.0310106 | val_loss: 0.0313163 | Time: 1645.11 ms [2022-03-14 03:26:32 main:574] : INFO : Epoch 1948 | loss: 0.0310077 | val_loss: 0.0313002 | Time: 1655.52 ms [2022-03-14 03:26:34 main:574] : INFO : Epoch 1949 | loss: 0.0309953 | val_loss: 0.0313223 | Time: 1651.24 ms [2022-03-14 03:26:35 main:574] : INFO : Epoch 1950 | loss: 0.0310006 | val_loss: 0.031301 | Time: 1641.23 ms [2022-03-14 03:26:37 main:574] : INFO : Epoch 1951 | loss: 0.0310015 | val_loss: 0.0313021 | Time: 1642.04 ms [2022-03-14 03:26:39 main:574] : INFO : Epoch 1952 | loss: 0.0310016 | val_loss: 0.0313044 | Time: 1646.28 ms [2022-03-14 03:26:40 main:574] : INFO : Epoch 1953 | loss: 0.0310074 | val_loss: 0.0312989 | Time: 1647.47 ms [2022-03-14 03:26:42 main:574] : INFO : Epoch 1954 | loss: 0.0310026 | val_loss: 0.0313172 | Time: 1648.5 ms [2022-03-14 03:26:44 main:574] : INFO : Epoch 1955 | loss: 0.0310071 | val_loss: 0.0312898 | Time: 1651.61 ms [2022-03-14 03:26:45 main:574] : INFO : Epoch 1956 | loss: 0.0310107 | val_loss: 0.0312936 | Time: 1642.37 ms [2022-03-14 03:26:47 main:574] : INFO : Epoch 1957 | loss: 0.0310037 | val_loss: 0.0313178 | Time: 1648.01 ms [2022-03-14 03:26:49 main:574] : INFO : Epoch 1958 | loss: 0.0310093 | val_loss: 0.0313211 | Time: 1648.92 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 970) [2022-03-14 03:27:36 main:435] : INFO : Set logging level to 1 [2022-03-14 03:27:36 main:441] : INFO : Running in BOINC Client mode [2022-03-14 03:27:36 main:444] : INFO : Resolving all filenames [2022-03-14 03:27:36 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-14 03:27:36 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-14 03:27:36 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-14 03:27:36 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-03-14 03:27:36 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-14 03:27:36 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-14 03:27:36 main:474] : INFO : Configuration: [2022-03-14 03:27:36 main:475] : INFO : Model type: GRU [2022-03-14 03:27:36 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-14 03:27:36 main:477] : INFO : Max Epochs: 2048 [2022-03-14 03:27:36 main:478] : INFO : Batch Size: 128 [2022-03-14 03:27:36 main:479] : INFO : Learning Rate: 0.01 [2022-03-14 03:27:36 main:480] : INFO : Patience: 10 [2022-03-14 03:27:36 main:481] : INFO : Hidden Width: 12 [2022-03-14 03:27:36 main:482] : INFO : # Recurrent Layers: 4 [2022-03-14 03:27:36 main:483] : INFO : # Backend Layers: 4 [2022-03-14 03:27:36 main:484] : INFO : # Threads: 1 [2022-03-14 03:27:36 main:486] : INFO : Preparing Dataset [2022-03-14 03:27:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-14 03:27:36 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-14 03:27:37 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-14 03:27:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-14 03:27:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-14 03:27:37 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-14 03:27:37 main:494] : INFO : Creating Model [2022-03-14 03:27:37 main:507] : INFO : Preparing config file [2022-03-14 03:27:37 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-14 03:27:37 main:512] : INFO : Loading config [2022-03-14 03:27:37 main:514] : INFO : Loading state [2022-03-14 03:27:38 main:559] : INFO : Loading DataLoader into Memory [2022-03-14 03:27:38 main:562] : INFO : Starting Training [2022-03-14 03:27:40 main:574] : INFO : Epoch 1953 | loss: 0.0311204 | val_loss: 0.0312587 | Time: 1816 ms [2022-03-14 03:27:41 main:574] : INFO : Epoch 1954 | loss: 0.0310599 | val_loss: 0.0312721 | Time: 1625.1 ms [2022-03-14 03:27:43 main:574] : INFO : Epoch 1955 | loss: 0.0310263 | val_loss: 0.0312687 | Time: 1635.32 ms [2022-03-14 03:27:45 main:574] : INFO : Epoch 1956 | loss: 0.0310117 | val_loss: 0.0312934 | Time: 1634.55 ms [2022-03-14 03:27:46 main:574] : INFO : Epoch 1957 | loss: 0.0310139 | val_loss: 0.031318 | Time: 1634.43 ms [2022-03-14 03:27:48 main:574] : INFO : Epoch 1958 | loss: 0.0310119 | val_loss: 0.0313243 | Time: 1640.08 ms [2022-03-14 03:27:50 main:574] : INFO : Epoch 1959 | loss: 0.031008 | val_loss: 0.031329 | Time: 1639.47 ms [2022-03-14 03:27:51 main:574] : INFO : Epoch 1960 | loss: 0.0310093 | val_loss: 0.031318 | Time: 1647.19 ms [2022-03-14 03:27:53 main:574] : INFO : Epoch 1961 | loss: 0.0310147 | val_loss: 0.0312911 | Time: 1656.91 ms [2022-03-14 03:27:54 main:574] : INFO : Epoch 1962 | loss: 0.0310082 | val_loss: 0.0313056 | Time: 1641.51 ms [2022-03-14 03:27:56 main:574] : INFO : Epoch 1963 | loss: 0.0310061 | val_loss: 0.0313322 | Time: 1635.12 ms [2022-03-14 03:27:58 main:574] : INFO : Epoch 1964 | loss: 0.0310034 | val_loss: 0.0313257 | Time: 1639.88 ms [2022-03-14 03:27:59 main:574] : INFO : Epoch 1965 | loss: 0.0310129 | val_loss: 0.0313041 | Time: 1639.08 ms [2022-03-14 03:28:01 main:574] : INFO : Epoch 1966 | loss: 0.0310107 | val_loss: 0.0312935 | Time: 1647.01 ms [2022-03-14 03:28:03 main:574] : INFO : Epoch 1967 | loss: 0.031004 | val_loss: 0.0313192 | Time: 1641.46 ms [2022-03-14 03:28:04 main:574] : INFO : Epoch 1968 | loss: 0.0310076 | val_loss: 0.0312861 | Time: 1648.86 ms [2022-03-14 03:28:06 main:574] : INFO : Epoch 1969 | loss: 0.0310047 | val_loss: 0.0312844 | Time: 1643.23 ms [2022-03-14 03:28:08 main:574] : INFO : Epoch 1970 | loss: 0.0310061 | val_loss: 0.0312997 | Time: 1643.78 ms [2022-03-14 03:28:09 main:574] : INFO : Epoch 1971 | loss: 0.0310084 | val_loss: 0.0312746 | Time: 1647.23 ms [2022-03-14 03:28:11 main:574] : INFO : Epoch 1972 | loss: 0.0310078 | val_loss: 0.0313146 | Time: 1652.09 ms [2022-03-14 03:28:13 main:574] : INFO : Epoch 1973 | loss: 0.0310017 | val_loss: 0.031306 | Time: 1647.23 ms [2022-03-14 03:28:14 main:574] : INFO : Epoch 1974 | loss: 0.0310083 | val_loss: 0.031313 | Time: 1654.46 ms [2022-03-14 03:28:16 main:574] : INFO : Epoch 1975 | loss: 0.03102 | val_loss: 0.0312847 | Time: 1642.58 ms [2022-03-14 03:28:18 main:574] : INFO : Epoch 1976 | loss: 0.0310124 | val_loss: 0.0313038 | Time: 1638.01 ms [2022-03-14 03:28:19 main:574] : INFO : Epoch 1977 | loss: 0.031005 | val_loss: 0.031319 | Time: 1637.8 ms [2022-03-14 03:28:21 main:574] : INFO : Epoch 1978 | loss: 0.0310038 | val_loss: 0.0313073 | Time: 1648.71 ms [2022-03-14 03:28:23 main:574] : INFO : Epoch 1979 | loss: 0.0310116 | val_loss: 0.0313035 | Time: 1647.49 ms [2022-03-14 03:28:24 main:574] : INFO : Epoch 1980 | loss: 0.0310074 | val_loss: 0.0313058 | Time: 1672.03 ms [2022-03-14 03:28:26 main:574] : INFO : Epoch 1981 | loss: 0.0310132 | val_loss: 0.0312823 | Time: 1642.55 ms [2022-03-14 03:28:28 main:574] : INFO : Epoch 1982 | loss: 0.0310142 | val_loss: 0.0312941 | Time: 1643 ms [2022-03-14 03:28:29 main:574] : INFO : Epoch 1983 | loss: 0.0310058 | val_loss: 0.031297 | Time: 1640.91 ms [2022-03-14 03:28:31 main:574] : INFO : Epoch 1984 | loss: 0.0310016 | val_loss: 0.0312996 | Time: 1647.84 ms [2022-03-14 03:28:32 main:574] : INFO : Epoch 1985 | loss: 0.031001 | val_loss: 0.0313115 | Time: 1655.07 ms [2022-03-14 03:28:34 main:574] : INFO : Epoch 1986 | loss: 0.0310229 | val_loss: 0.031284 | Time: 1661.94 ms [2022-03-14 03:28:36 main:574] : INFO : Epoch 1987 | loss: 0.0310171 | val_loss: 0.0312938 | Time: 1648.9 ms [2022-03-14 03:28:37 main:574] : INFO : Epoch 1988 | loss: 0.0310114 | val_loss: 0.031306 | Time: 1643.82 ms [2022-03-14 03:28:39 main:574] : INFO : Epoch 1989 | loss: 0.0310117 | val_loss: 0.0313066 | Time: 1650.84 ms [2022-03-14 03:28:41 main:574] : INFO : Epoch 1990 | loss: 0.031006 | val_loss: 0.031334 | Time: 1647.72 ms [2022-03-14 03:28:42 main:574] : INFO : Epoch 1991 | loss: 0.0310138 | val_loss: 0.0313107 | Time: 1648.91 ms [2022-03-14 03:28:44 main:574] : INFO : Epoch 1992 | loss: 0.0310051 | val_loss: 0.031292 | Time: 1655.07 ms [2022-03-14 03:28:46 main:574] : INFO : Epoch 1993 | loss: 0.0310027 | val_loss: 0.0313055 | Time: 1635.99 ms [2022-03-14 03:28:47 main:574] : INFO : Epoch 1994 | loss: 0.0310117 | val_loss: 0.0313083 | Time: 1639.2 ms [2022-03-14 03:28:49 main:574] : INFO : Epoch 1995 | loss: 0.0310102 | val_loss: 0.0313177 | Time: 1647.39 ms [2022-03-14 03:28:51 main:574] : INFO : Epoch 1996 | loss: 0.0310116 | val_loss: 0.0312954 | Time: 1650.55 ms [2022-03-14 03:28:52 main:574] : INFO : Epoch 1997 | loss: 0.031009 | val_loss: 0.0312979 | Time: 1643.29 ms [2022-03-14 03:28:54 main:574] : INFO : Epoch 1998 | loss: 0.0310064 | val_loss: 0.0312939 | Time: 1659.12 ms [2022-03-14 03:28:56 main:574] : INFO : Epoch 1999 | loss: 0.031011 | val_loss: 0.0312803 | Time: 1641.54 ms [2022-03-14 03:28:57 main:574] : INFO : Epoch 2000 | loss: 0.0310089 | val_loss: 0.0313192 | Time: 1646.34 ms [2022-03-14 03:28:59 main:574] : INFO : Epoch 2001 | loss: 0.0310125 | val_loss: 0.0312952 | Time: 1636.18 ms [2022-03-14 03:29:01 main:574] : INFO : Epoch 2002 | loss: 0.0310059 | val_loss: 0.0313351 | Time: 1652.67 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 970) [2022-03-14 03:30:07 main:435] : INFO : Set logging level to 1 [2022-03-14 03:30:07 main:441] : INFO : Running in BOINC Client mode [2022-03-14 03:30:07 main:444] : INFO : Resolving all filenames [2022-03-14 03:30:07 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-14 03:30:07 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-14 03:30:07 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-14 03:30:07 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-03-14 03:30:07 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-14 03:30:07 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-14 03:30:07 main:474] : INFO : Configuration: [2022-03-14 03:30:07 main:475] : INFO : Model type: GRU [2022-03-14 03:30:07 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-14 03:30:07 main:477] : INFO : Max Epochs: 2048 [2022-03-14 03:30:07 main:478] : INFO : Batch Size: 128 [2022-03-14 03:30:07 main:479] : INFO : Learning Rate: 0.01 [2022-03-14 03:30:07 main:480] : INFO : Patience: 10 [2022-03-14 03:30:07 main:481] : INFO : Hidden Width: 12 [2022-03-14 03:30:07 main:482] : INFO : # Recurrent Layers: 4 [2022-03-14 03:30:07 main:483] : INFO : # Backend Layers: 4 [2022-03-14 03:30:07 main:484] : INFO : # Threads: 1 [2022-03-14 03:30:07 main:486] : INFO : Preparing Dataset [2022-03-14 03:30:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-14 03:30:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-14 03:30:08 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-14 03:30:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-14 03:30:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-14 03:30:08 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-14 03:30:08 main:494] : INFO : Creating Model [2022-03-14 03:30:08 main:507] : INFO : Preparing config file [2022-03-14 03:30:08 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-14 03:30:08 main:512] : INFO : Loading config [2022-03-14 03:30:08 main:514] : INFO : Loading state [2022-03-14 03:30:09 main:559] : INFO : Loading DataLoader into Memory [2022-03-14 03:30:09 main:562] : INFO : Starting Training [2022-03-14 03:30:11 main:574] : INFO : Epoch 1992 | loss: 0.0310854 | val_loss: 0.0312979 | Time: 1846.19 ms [2022-03-14 03:30:12 main:574] : INFO : Epoch 1993 | loss: 0.0310302 | val_loss: 0.0312825 | Time: 1629.09 ms [2022-03-14 03:30:14 main:574] : INFO : Epoch 1994 | loss: 0.0310106 | val_loss: 0.0312799 | Time: 1634.48 ms [2022-03-14 03:30:16 main:574] : INFO : Epoch 1995 | loss: 0.0310076 | val_loss: 0.0312844 | Time: 1628.35 ms [2022-03-14 03:30:17 main:574] : INFO : Epoch 1996 | loss: 0.0310404 | val_loss: 0.0313066 | Time: 1638.92 ms [2022-03-14 03:30:19 main:574] : INFO : Epoch 1997 | loss: 0.0310236 | val_loss: 0.0312954 | Time: 1635.24 ms [2022-03-14 03:30:21 main:574] : INFO : Epoch 1998 | loss: 0.0310091 | val_loss: 0.0313135 | Time: 1651.24 ms [2022-03-14 03:30:22 main:574] : INFO : Epoch 1999 | loss: 0.0310035 | val_loss: 0.0313032 | Time: 1637.77 ms [2022-03-14 03:30:24 main:574] : INFO : Epoch 2000 | loss: 0.0310037 | val_loss: 0.0313237 | Time: 1649.65 ms [2022-03-14 03:30:25 main:574] : INFO : Epoch 2001 | loss: 0.0310058 | val_loss: 0.0312984 | Time: 1641.53 ms [2022-03-14 03:30:27 main:574] : INFO : Epoch 2002 | loss: 0.0310173 | val_loss: 0.0312763 | Time: 1642.28 ms [2022-03-14 03:30:29 main:574] : INFO : Epoch 2003 | loss: 0.0310104 | val_loss: 0.0313163 | Time: 1635.85 ms [2022-03-14 03:30:30 main:574] : INFO : Epoch 2004 | loss: 0.0309955 | val_loss: 0.0313024 | Time: 1646.67 ms [2022-03-14 03:30:32 main:574] : INFO : Epoch 2005 | loss: 0.0309989 | val_loss: 0.0313017 | Time: 1649.23 ms [2022-03-14 03:30:34 main:574] : INFO : Epoch 2006 | loss: 0.0310091 | val_loss: 0.0312974 | Time: 1666.63 ms [2022-03-14 03:30:35 main:574] : INFO : Epoch 2007 | loss: 0.0310123 | val_loss: 0.0313208 | Time: 1653.6 ms [2022-03-14 03:30:37 main:574] : INFO : Epoch 2008 | loss: 0.0310085 | val_loss: 0.0313115 | Time: 1752.94 ms [2022-03-14 03:30:39 main:574] : INFO : Epoch 2009 | loss: 0.031008 | val_loss: 0.0313022 | Time: 1670.17 ms [2022-03-14 03:30:41 main:574] : INFO : Epoch 2010 | loss: 0.0310056 | val_loss: 0.0313075 | Time: 1826.96 ms [2022-03-14 03:30:42 main:574] : INFO : Epoch 2011 | loss: 0.0310031 | val_loss: 0.031315 | Time: 1673.72 ms [2022-03-14 03:30:44 main:574] : INFO : Epoch 2012 | loss: 0.0310038 | val_loss: 0.0312983 | Time: 1671.03 ms [2022-03-14 03:30:46 main:574] : INFO : Epoch 2013 | loss: 0.0310012 | val_loss: 0.0313091 | Time: 1664.7 ms [2022-03-14 03:30:47 main:574] : INFO : Epoch 2014 | loss: 0.0310026 | val_loss: 0.0313339 | Time: 1652.32 ms [2022-03-14 03:30:49 main:574] : INFO : Epoch 2015 | loss: 0.0310073 | val_loss: 0.0313135 | Time: 1662.5 ms [2022-03-14 03:30:51 main:574] : INFO : Epoch 2016 | loss: 0.0310079 | val_loss: 0.031332 | Time: 1678.94 ms [2022-03-14 03:30:52 main:574] : INFO : Epoch 2017 | loss: 0.0310111 | val_loss: 0.0313007 | Time: 1762.56 ms [2022-03-14 03:30:54 main:574] : INFO : Epoch 2018 | loss: 0.031005 | val_loss: 0.0313145 | Time: 1774.93 ms [2022-03-14 03:30:56 main:574] : INFO : Epoch 2019 | loss: 0.0310031 | val_loss: 0.031316 | Time: 1663.6 ms [2022-03-14 03:30:58 main:574] : INFO : Epoch 2020 | loss: 0.0310051 | val_loss: 0.0313026 | Time: 1657.55 ms [2022-03-14 03:30:59 main:574] : INFO : Epoch 2021 | loss: 0.0310045 | val_loss: 0.031303 | Time: 1660.71 ms [2022-03-14 03:31:01 main:574] : INFO : Epoch 2022 | loss: 0.0310015 | val_loss: 0.0313091 | Time: 1663.3 ms [2022-03-14 03:31:03 main:574] : INFO : Epoch 2023 | loss: 0.0310035 | val_loss: 0.0313378 | Time: 1659.61 ms [2022-03-14 03:31:04 main:574] : INFO : Epoch 2024 | loss: 0.0310077 | val_loss: 0.0313168 | Time: 1666.37 ms [2022-03-14 03:31:06 main:574] : INFO : Epoch 2025 | loss: 0.0310062 | val_loss: 0.0313138 | Time: 1653.95 ms [2022-03-14 03:31:08 main:574] : INFO : Epoch 2026 | loss: 0.031007 | val_loss: 0.0313163 | Time: 1665.01 ms [2022-03-14 03:31:09 main:574] : INFO : Epoch 2027 | loss: 0.0310041 | val_loss: 0.0313006 | Time: 1660.12 ms [2022-03-14 03:31:11 main:574] : INFO : Epoch 2028 | loss: 0.0310156 | val_loss: 0.0313127 | Time: 1661.52 ms [2022-03-14 03:31:13 main:574] : INFO : Epoch 2029 | loss: 0.0310037 | val_loss: 0.0312932 | Time: 1668.37 ms [2022-03-14 03:31:14 main:574] : INFO : Epoch 2030 | loss: 0.0310013 | val_loss: 0.0313137 | Time: 1686.69 ms [2022-03-14 03:31:16 main:574] : INFO : Epoch 2031 | loss: 0.0310005 | val_loss: 0.0313084 | Time: 1655.73 ms [2022-03-14 03:31:18 main:574] : INFO : Epoch 2032 | loss: 0.0310029 | val_loss: 0.031318 | Time: 1663.25 ms [2022-03-14 03:31:19 main:574] : INFO : Epoch 2033 | loss: 0.0310083 | val_loss: 0.0312981 | Time: 1660.12 ms [2022-03-14 03:31:21 main:574] : INFO : Epoch 2034 | loss: 0.0310096 | val_loss: 0.0312911 | Time: 1666.05 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 970) [2022-03-14 03:31:37 main:435] : INFO : Set logging level to 1 [2022-03-14 03:31:37 main:441] : INFO : Running in BOINC Client mode [2022-03-14 03:31:37 main:444] : INFO : Resolving all filenames [2022-03-14 03:31:37 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-14 03:31:37 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-14 03:31:37 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-14 03:31:37 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2022-03-14 03:31:37 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-14 03:31:37 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-14 03:31:37 main:474] : INFO : Configuration: [2022-03-14 03:31:37 main:475] : INFO : Model type: GRU [2022-03-14 03:31:37 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-14 03:31:37 main:477] : INFO : Max Epochs: 2048 [2022-03-14 03:31:37 main:478] : INFO : Batch Size: 128 [2022-03-14 03:31:37 main:479] : INFO : Learning Rate: 0.01 [2022-03-14 03:31:37 main:480] : INFO : Patience: 10 [2022-03-14 03:31:37 main:481] : INFO : Hidden Width: 12 [2022-03-14 03:31:37 main:482] : INFO : # Recurrent Layers: 4 [2022-03-14 03:31:37 main:483] : INFO : # Backend Layers: 4 [2022-03-14 03:31:37 main:484] : INFO : # Threads: 1 [2022-03-14 03:31:37 main:486] : INFO : Preparing Dataset [2022-03-14 03:31:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-14 03:31:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-14 03:31:39 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-14 03:31:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-14 03:31:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-14 03:31:39 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-14 03:31:39 main:494] : INFO : Creating Model [2022-03-14 03:31:39 main:507] : INFO : Preparing config file [2022-03-14 03:31:39 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-14 03:31:39 main:512] : INFO : Loading config [2022-03-14 03:31:39 main:514] : INFO : Loading state [2022-03-14 03:31:40 main:559] : INFO : Loading DataLoader into Memory [2022-03-14 03:31:40 main:562] : INFO : Starting Training [2022-03-14 03:31:41 main:574] : INFO : Epoch 2030 | loss: 0.0310862 | val_loss: 0.0312874 | Time: 1868.52 ms [2022-03-14 03:31:43 main:574] : INFO : Epoch 2031 | loss: 0.0310298 | val_loss: 0.0312814 | Time: 1647.08 ms [2022-03-14 03:31:45 main:574] : INFO : Epoch 2032 | loss: 0.0310147 | val_loss: 0.0312822 | Time: 1657.26 ms [2022-03-14 03:31:46 main:574] : INFO : Epoch 2033 | loss: 0.0310114 | val_loss: 0.0312849 | Time: 1641.16 ms [2022-03-14 03:31:48 main:574] : INFO : Epoch 2034 | loss: 0.0310226 | val_loss: 0.031278 | Time: 1654.74 ms [2022-03-14 03:31:50 main:574] : INFO : Epoch 2035 | loss: 0.0310132 | val_loss: 0.0312981 | Time: 1649.41 ms [2022-03-14 03:31:51 main:574] : INFO : Epoch 2036 | loss: 0.0310025 | val_loss: 0.0313191 | Time: 1653.73 ms [2022-03-14 03:31:53 main:574] : INFO : Epoch 2037 | loss: 0.0310053 | val_loss: 0.0313126 | Time: 1673.11 ms [2022-03-14 03:31:55 main:574] : INFO : Epoch 2038 | loss: 0.031015 | val_loss: 0.0312983 | Time: 1668.93 ms [2022-03-14 03:31:56 main:574] : INFO : Epoch 2039 | loss: 0.0310119 | val_loss: 0.0312798 | Time: 1663.15 ms [2022-03-14 03:31:58 main:574] : INFO : Epoch 2040 | loss: 0.0310136 | val_loss: 0.0312787 | Time: 1663.77 ms [2022-03-14 03:32:00 main:574] : INFO : Epoch 2041 | loss: 0.0310038 | val_loss: 0.0313137 | Time: 1657.85 ms [2022-03-14 03:32:01 main:574] : INFO : Epoch 2042 | loss: 0.0310002 | val_loss: 0.0313062 | Time: 1653.23 ms [2022-03-14 03:32:03 main:574] : INFO : Epoch 2043 | loss: 0.0310019 | val_loss: 0.0313419 | Time: 1708.97 ms [2022-03-14 03:32:05 main:574] : INFO : Epoch 2044 | loss: 0.0310106 | val_loss: 0.031304 | Time: 1666.07 ms [2022-03-14 03:32:06 main:574] : INFO : Epoch 2045 | loss: 0.0310044 | val_loss: 0.0313027 | Time: 1650.93 ms [2022-03-14 03:32:08 main:574] : INFO : Epoch 2046 | loss: 0.0310016 | val_loss: 0.0312934 | Time: 1654.64 ms [2022-03-14 03:32:10 main:574] : INFO : Epoch 2047 | loss: 0.0310073 | val_loss: 0.0313231 | Time: 1670.08 ms [2022-03-14 03:32:11 main:574] : INFO : Epoch 2048 | loss: 0.0310162 | val_loss: 0.0313085 | Time: 1658.15 ms [2022-03-14 03:32:11 main:597] : INFO : Saving trained model to model-final.pt, val_loss 0.0313085 [2022-03-14 03:32:11 main:603] : INFO : Saving end state to config to file [2022-03-14 03:32:11 main:608] : INFO : Success, exiting.. 03:32:11 (332): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)