Task 11862905
| Name | ParityModified-1639963482-30007-3-0_0 |
| Workunit | 9064800 |
| Created | 21 Feb 2022, 14:01:10 UTC |
| Sent | 27 Feb 2022, 12:05:06 UTC |
| Report deadline | 7 Mar 2022, 12:05:06 UTC |
| Received | 27 Feb 2022, 15:13:07 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 1150 |
| Run time | 57 min 9 sec |
| CPU time | 55 min 25 sec |
| Validate state | Valid |
| Credit | 4,160.00 |
| Device peak FLOPS | 4,174.91 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.54 GB |
| Peak swap size | 3.48 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.20</core_client_version> <![CDATA[ <stderr_txt> 22-02-27 07:58:24 main:574] : INFO : Epoch 1674 | loss: 0.0311662 | val_loss: 0.0311844 | Time: 1633.33 ms [2022-02-27 07:58:26 main:574] : INFO : Epoch 1675 | loss: 0.0311676 | val_loss: 0.0311868 | Time: 1630.54 ms [2022-02-27 07:58:28 main:574] : INFO : Epoch 1676 | loss: 0.0311708 | val_loss: 0.0311874 | Time: 1632.6 ms [2022-02-27 07:58:29 main:574] : INFO : Epoch 1677 | loss: 0.0311658 | val_loss: 0.031181 | Time: 1631.11 ms [2022-02-27 07:58:31 main:574] : INFO : Epoch 1678 | loss: 0.031166 | val_loss: 0.0311828 | Time: 1636.24 ms [2022-02-27 07:58:33 main:574] : INFO : Epoch 1679 | loss: 0.031168 | val_loss: 0.0311872 | Time: 1651.76 ms [2022-02-27 07:58:34 main:574] : INFO : Epoch 1680 | loss: 0.0311672 | val_loss: 0.0311862 | Time: 1637.09 ms [2022-02-27 07:58:36 main:574] : INFO : Epoch 1681 | loss: 0.0311652 | val_loss: 0.0311814 | Time: 1627.41 ms [2022-02-27 07:58:38 main:574] : INFO : Epoch 1682 | loss: 0.0311642 | val_loss: 0.0311833 | Time: 1632.6 ms [2022-02-27 07:58:39 main:574] : INFO : Epoch 1683 | loss: 0.0311649 | val_loss: 0.0311843 | Time: 1629.64 ms [2022-02-27 07:58:41 main:574] : INFO : Epoch 1684 | loss: 0.0311646 | val_loss: 0.0311832 | Time: 1641.83 ms [2022-02-27 07:58:42 main:574] : INFO : Epoch 1685 | loss: 0.0311646 | val_loss: 0.0311842 | Time: 1645.78 ms [2022-02-27 07:58:44 main:574] : INFO : Epoch 1686 | loss: 0.031167 | val_loss: 0.0311889 | Time: 1632.09 ms [2022-02-27 07:58:46 main:574] : INFO : Epoch 1687 | loss: 0.0311676 | val_loss: 0.0311818 | Time: 1636.83 ms [2022-02-27 07:58:47 main:574] : INFO : Epoch 1688 | loss: 0.031167 | val_loss: 0.0311847 | Time: 1634.24 ms [2022-02-27 07:58:49 main:574] : INFO : Epoch 1689 | loss: 0.0311647 | val_loss: 0.031184 | Time: 1632.7 ms [2022-02-27 07:58:51 main:574] : INFO : Epoch 1690 | loss: 0.0311644 | val_loss: 0.0311832 | Time: 1638.77 ms [2022-02-27 07:58:52 main:574] : INFO : Epoch 1691 | loss: 0.031165 | val_loss: 0.031185 | Time: 1643.81 ms [2022-02-27 07:58:54 main:574] : INFO : Epoch 1692 | loss: 0.031165 | val_loss: 0.0311821 | Time: 1634.98 ms [2022-02-27 07:58:56 main:574] : INFO : Epoch 1693 | loss: 0.0311661 | val_loss: 0.0311808 | Time: 1629.64 ms [2022-02-27 07:58:57 main:574] : INFO : Epoch 1694 | loss: 0.0311649 | val_loss: 0.0311816 | Time: 1629.09 ms [2022-02-27 07:58:59 main:574] : INFO : Epoch 1695 | loss: 0.031165 | val_loss: 0.0311842 | Time: 1625.3 ms [2022-02-27 07:59:00 main:574] : INFO : Epoch 1696 | loss: 0.0311662 | val_loss: 0.0311853 | Time: 1634.37 ms [2022-02-27 07:59:02 main:574] : INFO : Epoch 1697 | loss: 0.0311668 | val_loss: 0.031182 | Time: 1641.62 ms [2022-02-27 07:59:04 main:574] : INFO : Epoch 1698 | loss: 0.0311672 | val_loss: 0.0311857 | Time: 1641.28 ms [2022-02-27 07:59:05 main:574] : INFO : Epoch 1699 | loss: 0.0311653 | val_loss: 0.0311861 | Time: 1633.76 ms [2022-02-27 07:59:07 main:574] : INFO : Epoch 1700 | loss: 0.0311664 | val_loss: 0.0311841 | Time: 1635.2 ms [2022-02-27 07:59:09 main:574] : INFO : Epoch 1701 | loss: 0.0311675 | val_loss: 0.0311811 | Time: 1634.55 ms [2022-02-27 07:59:10 main:574] : INFO : Epoch 1702 | loss: 0.0311671 | val_loss: 0.0311823 | Time: 1643.23 ms [2022-02-27 07:59:12 main:574] : INFO : Epoch 1703 | loss: 0.0311665 | val_loss: 0.0311812 | Time: 1635.55 ms [2022-02-27 07:59:14 main:574] : INFO : Epoch 1704 | loss: 0.0311672 | val_loss: 0.0311793 | Time: 1640.56 ms [2022-02-27 07:59:15 main:574] : INFO : Epoch 1705 | loss: 0.031169 | val_loss: 0.0311829 | Time: 1627.26 ms [2022-02-27 07:59:17 main:574] : INFO : Epoch 1706 | loss: 0.0311686 | val_loss: 0.0311818 | Time: 1637.5 ms [2022-02-27 07:59:19 main:574] : INFO : Epoch 1707 | loss: 0.0311699 | val_loss: 0.0311842 | Time: 1626.08 ms [2022-02-27 07:59:20 main:574] : INFO : Epoch 1708 | loss: 0.0311678 | val_loss: 0.0311825 | Time: 1644.79 ms [2022-02-27 07:59:22 main:574] : INFO : Epoch 1709 | loss: 0.0311682 | val_loss: 0.0311826 | Time: 1629.96 ms [2022-02-27 07:59:23 main:574] : INFO : Epoch 1710 | loss: 0.0311693 | val_loss: 0.0311814 | Time: 1641.17 ms [2022-02-27 07:59:25 main:574] : INFO : Epoch 1711 | loss: 0.0311674 | val_loss: 0.0311833 | Time: 1629.63 ms [2022-02-27 07:59:27 main:574] : INFO : Epoch 1712 | loss: 0.0311679 | val_loss: 0.0311808 | Time: 1635.26 ms [2022-02-27 07:59:28 main:574] : INFO : Epoch 1713 | loss: 0.0311672 | val_loss: 0.0311824 | Time: 1631.41 ms [2022-02-27 07:59:30 main:574] : INFO : Epoch 1714 | loss: 0.0311678 | val_loss: 0.0311804 | Time: 1639.89 ms [2022-02-27 07:59:32 main:574] : INFO : Epoch 1715 | loss: 0.0311686 | val_loss: 0.0311815 | Time: 1645.54 ms [2022-02-27 07:59:33 main:574] : INFO : Epoch 1716 | loss: 0.0311677 | val_loss: 0.0311801 | Time: 1643.13 ms [2022-02-27 07:59:35 main:574] : INFO : Epoch 1717 | loss: 0.0311674 | val_loss: 0.0311788 | Time: 1630.19 ms [2022-02-27 07:59:37 main:574] : INFO : Epoch 1718 | loss: 0.0311662 | val_loss: 0.0311825 | Time: 1643.73 ms [2022-02-27 07:59:38 main:574] : INFO : Epoch 1719 | loss: 0.0311663 | val_loss: 0.0311824 | Time: 1629.88 ms [2022-02-27 07:59:40 main:574] : INFO : Epoch 1720 | loss: 0.0311669 | val_loss: 0.0311793 | Time: 1642.63 ms [2022-02-27 07:59:42 main:574] : INFO : Epoch 1721 | loss: 0.0311658 | val_loss: 0.0311799 | Time: 1631.94 ms [2022-02-27 07:59:43 main:574] : INFO : Epoch 1722 | loss: 0.031166 | val_loss: 0.0311814 | Time: 1641.24 ms [2022-02-27 07:59:45 main:574] : INFO : Epoch 1723 | loss: 0.0311656 | val_loss: 0.0311831 | Time: 1636.39 ms [2022-02-27 07:59:46 main:574] : INFO : Epoch 1724 | loss: 0.0311687 | val_loss: 0.0311826 | Time: 1634.63 ms [2022-02-27 07:59:48 main:574] : INFO : Epoch 1725 | loss: 0.0311658 | val_loss: 0.03118 | Time: 1631.64 ms [2022-02-27 07:59:50 main:574] : INFO : Epoch 1726 | loss: 0.0311654 | val_loss: 0.0311819 | Time: 1637.46 ms [2022-02-27 07:59:51 main:574] : INFO : Epoch 1727 | loss: 0.0311639 | val_loss: 0.0311818 | Time: 1648.95 ms [2022-02-27 07:59:53 main:574] : INFO : Epoch 1728 | loss: 0.031168 | val_loss: 0.031182 | Time: 1644.57 ms [2022-02-27 07:59:55 main:574] : INFO : Epoch 1729 | loss: 0.031168 | val_loss: 0.0311817 | Time: 1636.16 ms [2022-02-27 07:59:56 main:574] : INFO : Epoch 1730 | loss: 0.0311676 | val_loss: 0.0311798 | Time: 1636.66 ms [2022-02-27 07:59:58 main:574] : INFO : Epoch 1731 | loss: 0.0311688 | val_loss: 0.0311805 | Time: 1638.99 ms [2022-02-27 08:00:00 main:574] : INFO : Epoch 1732 | loss: 0.0311679 | val_loss: 0.0311837 | Time: 1636.15 ms [2022-02-27 08:00:01 main:574] : INFO : Epoch 1733 | loss: 0.0311665 | val_loss: 0.0311838 | Time: 1648.88 ms [2022-02-27 08:00:03 main:574] : INFO : Epoch 1734 | loss: 0.0311678 | val_loss: 0.0311807 | Time: 1647.22 ms [2022-02-27 08:00:05 main:574] : INFO : Epoch 1735 | loss: 0.0311673 | val_loss: 0.0311811 | Time: 1626.02 ms [2022-02-27 08:00:06 main:574] : INFO : Epoch 1736 | loss: 0.0311723 | val_loss: 0.0311913 | Time: 1634.49 ms [2022-02-27 08:00:08 main:574] : INFO : Epoch 1737 | loss: 0.0311694 | val_loss: 0.0311839 | Time: 1635.52 ms [2022-02-27 08:00:10 main:574] : INFO : Epoch 1738 | loss: 0.0311656 | val_loss: 0.0311838 | Time: 1632.09 ms [2022-02-27 08:00:11 main:574] : INFO : Epoch 1739 | loss: 0.0311651 | val_loss: 0.0311841 | Time: 1639.8 ms [2022-02-27 08:00:13 main:574] : INFO : Epoch 1740 | loss: 0.0311645 | val_loss: 0.0311811 | Time: 1642.81 ms [2022-02-27 08:00:14 main:574] : INFO : Epoch 1741 | loss: 0.0311643 | val_loss: 0.0311837 | Time: 1633.28 ms [2022-02-27 08:00:16 main:574] : INFO : Epoch 1742 | loss: 0.0311632 | val_loss: 0.0311814 | Time: 1632.45 ms [2022-02-27 08:00:18 main:574] : INFO : Epoch 1743 | loss: 0.031165 | val_loss: 0.0311811 | Time: 1632.78 ms [2022-02-27 08:00:19 main:574] : INFO : Epoch 1744 | loss: 0.0311653 | val_loss: 0.0311828 | Time: 1639.92 ms [2022-02-27 08:00:21 main:574] : INFO : Epoch 1745 | loss: 0.0311655 | val_loss: 0.0311841 | Time: 1645.65 ms [2022-02-27 08:00:23 main:574] : INFO : Epoch 1746 | loss: 0.0311672 | val_loss: 0.0311882 | Time: 1636.61 ms [2022-02-27 08:00:24 main:574] : INFO : Epoch 1747 | loss: 0.0311662 | val_loss: 0.0311814 | Time: 1632 ms [2022-02-27 08:00:26 main:574] : INFO : Epoch 1748 | loss: 0.0311639 | val_loss: 0.0311812 | Time: 1622.72 ms [2022-02-27 08:00:28 main:574] : INFO : Epoch 1749 | loss: 0.0311641 | val_loss: 0.0311845 | Time: 1640.42 ms [2022-02-27 08:00:29 main:574] : INFO : Epoch 1750 | loss: 0.0311659 | val_loss: 0.0311829 | Time: 1626.47 ms [2022-02-27 08:00:31 main:574] : INFO : Epoch 1751 | loss: 0.0311665 | val_loss: 0.0311849 | Time: 1641.37 ms [2022-02-27 08:00:33 main:574] : INFO : Epoch 1752 | loss: 0.0311659 | val_loss: 0.0311811 | Time: 1640.64 ms [2022-02-27 08:00:34 main:574] : INFO : Epoch 1753 | loss: 0.0311655 | val_loss: 0.0311833 | Time: 1641.07 ms [2022-02-27 08:00:36 main:574] : INFO : Epoch 1754 | loss: 0.0311657 | val_loss: 0.0311813 | Time: 1623.64 ms [2022-02-27 08:00:37 main:574] : INFO : Epoch 1755 | loss: 0.0311647 | val_loss: 0.0311847 | Time: 1638.41 ms [2022-02-27 08:00:39 main:574] : INFO : Epoch 1756 | loss: 0.0311648 | val_loss: 0.0311871 | Time: 1620.99 ms [2022-02-27 08:00:41 main:574] : INFO : Epoch 1757 | loss: 0.0311646 | val_loss: 0.0311829 | Time: 1639.32 ms [2022-02-27 08:00:42 main:574] : INFO : Epoch 1758 | loss: 0.0311646 | val_loss: 0.0311843 | Time: 1638.96 ms [2022-02-27 08:00:44 main:574] : INFO : Epoch 1759 | loss: 0.0311656 | val_loss: 0.0311843 | Time: 1645.69 ms [2022-02-27 08:00:46 main:574] : INFO : Epoch 1760 | loss: 0.0311647 | val_loss: 0.0311828 | Time: 1628.06 ms [2022-02-27 08:00:47 main:574] : INFO : Epoch 1761 | loss: 0.0311643 | val_loss: 0.0311841 | Time: 1653.05 ms [2022-02-27 08:00:49 main:574] : INFO : Epoch 1762 | loss: 0.0311645 | val_loss: 0.0311814 | Time: 1625.27 ms [2022-02-27 08:00:51 main:574] : INFO : Epoch 1763 | loss: 0.0311652 | val_loss: 0.0311816 | Time: 1640.11 ms [2022-02-27 08:00:52 main:574] : INFO : Epoch 1764 | loss: 0.0311646 | val_loss: 0.0311817 | Time: 1640.14 ms [2022-02-27 08:00:54 main:574] : INFO : Epoch 1765 | loss: 0.0311641 | val_loss: 0.0311829 | Time: 1638.91 ms [2022-02-27 08:00:55 main:574] : INFO : Epoch 1766 | loss: 0.0311641 | val_loss: 0.0311853 | Time: 1624.91 ms [2022-02-27 08:00:57 main:574] : INFO : Epoch 1767 | loss: 0.0311659 | val_loss: 0.0311833 | Time: 1633.13 ms [2022-02-27 08:00:59 main:574] : INFO : Epoch 1768 | loss: 0.031166 | val_loss: 0.0311815 | Time: 1627.39 ms [2022-02-27 08:01:00 main:574] : INFO : Epoch 1769 | loss: 0.0311642 | val_loss: 0.031183 | Time: 1639.36 ms [2022-02-27 08:01:02 main:574] : INFO : Epoch 1770 | loss: 0.0311638 | val_loss: 0.0311841 | Time: 1644.66 ms [2022-02-27 08:01:04 main:574] : INFO : Epoch 1771 | loss: 0.0311641 | val_loss: 0.0311863 | Time: 1641.31 ms [2022-02-27 08:01:05 main:574] : INFO : Epoch 1772 | loss: 0.0311652 | val_loss: 0.0311872 | Time: 1628.44 ms [2022-02-27 08:01:07 main:574] : INFO : Epoch 1773 | loss: 0.0311657 | val_loss: 0.0311812 | Time: 1634.31 ms [2022-02-27 08:01:09 main:574] : INFO : Epoch 1774 | loss: 0.0311653 | val_loss: 0.0311826 | Time: 1634.75 ms [2022-02-27 08:01:10 main:574] : INFO : Epoch 1775 | loss: 0.0311674 | val_loss: 0.031182 | Time: 1639.01 ms [2022-02-27 08:01:12 main:574] : INFO : Epoch 1776 | loss: 0.03117 | val_loss: 0.0311845 | Time: 1636.48 ms [2022-02-27 08:01:14 main:574] : INFO : Epoch 1777 | loss: 0.0311706 | val_loss: 0.0311805 | Time: 1643.35 ms [2022-02-27 08:01:15 main:574] : INFO : Epoch 1778 | loss: 0.0311695 | val_loss: 0.0311805 | Time: 1632.38 ms [2022-02-27 08:01:17 main:574] : INFO : Epoch 1779 | loss: 0.031169 | val_loss: 0.031183 | Time: 1637.85 ms [2022-02-27 08:01:18 main:574] : INFO : Epoch 1780 | loss: 0.0311672 | val_loss: 0.0311805 | Time: 1635.49 ms [2022-02-27 08:01:20 main:574] : INFO : Epoch 1781 | loss: 0.0311657 | val_loss: 0.0311811 | Time: 1635.27 ms [2022-02-27 08:01:22 main:574] : INFO : Epoch 1782 | loss: 0.0311668 | val_loss: 0.031179 | Time: 1638.3 ms [2022-02-27 08:01:23 main:574] : INFO : Epoch 1783 | loss: 0.0311663 | val_loss: 0.03118 | Time: 1644.9 ms [2022-02-27 08:01:25 main:574] : INFO : Epoch 1784 | loss: 0.0311662 | val_loss: 0.0311848 | Time: 1628 ms [2022-02-27 08:01:27 main:574] : INFO : Epoch 1785 | loss: 0.0311673 | val_loss: 0.0311784 | Time: 1633.13 ms [2022-02-27 08:01:28 main:574] : INFO : Epoch 1786 | loss: 0.0311651 | val_loss: 0.0311814 | Time: 1635.27 ms [2022-02-27 08:01:30 main:574] : INFO : Epoch 1787 | loss: 0.0311647 | val_loss: 0.0311852 | Time: 1639.08 ms [2022-02-27 08:01:32 main:574] : INFO : Epoch 1788 | loss: 0.0311647 | val_loss: 0.0311802 | Time: 1652.58 ms [2022-02-27 08:01:33 main:574] : INFO : Epoch 1789 | loss: 0.0311644 | val_loss: 0.0311801 | Time: 1639.71 ms [2022-02-27 08:01:35 main:574] : INFO : Epoch 1790 | loss: 0.0311645 | val_loss: 0.0311817 | Time: 1636.14 ms [2022-02-27 08:01:37 main:574] : INFO : Epoch 1791 | loss: 0.0311679 | val_loss: 0.0311799 | Time: 1635.31 ms [2022-02-27 08:01:38 main:574] : INFO : Epoch 1792 | loss: 0.0311683 | val_loss: 0.0311802 | Time: 1636.11 ms [2022-02-27 08:01:40 main:574] : INFO : Epoch 1793 | loss: 0.0311672 | val_loss: 0.0311791 | Time: 1630.99 ms [2022-02-27 08:01:41 main:574] : INFO : Epoch 1794 | loss: 0.0311678 | val_loss: 0.031183 | Time: 1648.14 ms [2022-02-27 08:01:43 main:574] : INFO : Epoch 1795 | loss: 0.0311694 | val_loss: 0.0311798 | Time: 1643.6 ms [2022-02-27 08:01:45 main:574] : INFO : Epoch 1796 | loss: 0.0311673 | val_loss: 0.0311794 | Time: 1641.14 ms [2022-02-27 08:01:46 main:574] : INFO : Epoch 1797 | loss: 0.0311663 | val_loss: 0.031178 | Time: 1634.1 ms [2022-02-27 08:01:48 main:574] : INFO : Epoch 1798 | loss: 0.0311661 | val_loss: 0.0311803 | Time: 1633.83 ms [2022-02-27 08:01:50 main:574] : INFO : Epoch 1799 | loss: 0.0311657 | val_loss: 0.0311796 | Time: 1633.75 ms [2022-02-27 08:01:51 main:574] : INFO : Epoch 1800 | loss: 0.0311658 | val_loss: 0.0311815 | Time: 1643.76 ms [2022-02-27 08:01:53 main:574] : INFO : Epoch 1801 | loss: 0.0311652 | val_loss: 0.0311816 | Time: 1644.63 ms [2022-02-27 08:01:55 main:574] : INFO : Epoch 1802 | loss: 0.0311647 | val_loss: 0.0311832 | Time: 1639.87 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 970) [2022-02-27 08:03:11 main:435] : INFO : Set logging level to 1 [2022-02-27 08:03:11 main:441] : INFO : Running in BOINC Client mode [2022-02-27 08:03:11 main:444] : INFO : Resolving all filenames [2022-02-27 08:03:11 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-02-27 08:03:11 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-02-27 08:03:11 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-02-27 08:03:11 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-02-27 08:03:11 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-02-27 08:03:11 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-02-27 08:03:11 main:474] : INFO : Configuration: [2022-02-27 08:03:11 main:475] : INFO : Model type: GRU [2022-02-27 08:03:11 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-02-27 08:03:11 main:477] : INFO : Max Epochs: 2048 [2022-02-27 08:03:11 main:478] : INFO : Batch Size: 128 [2022-02-27 08:03:11 main:479] : INFO : Learning Rate: 0.01 [2022-02-27 08:03:11 main:480] : INFO : Patience: 10 [2022-02-27 08:03:11 main:481] : INFO : Hidden Width: 12 [2022-02-27 08:03:11 main:482] : INFO : # Recurrent Layers: 4 [2022-02-27 08:03:11 main:483] : INFO : # Backend Layers: 4 [2022-02-27 08:03:11 main:484] : INFO : # Threads: 1 [2022-02-27 08:03:11 main:486] : INFO : Preparing Dataset [2022-02-27 08:03:11 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-02-27 08:03:12 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-02-27 08:03:13 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-02-27 08:03:13 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-02-27 08:03:13 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-02-27 08:03:13 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-02-27 08:03:13 main:494] : INFO : Creating Model [2022-02-27 08:03:13 main:507] : INFO : Preparing config file [2022-02-27 08:03:13 main:511] : INFO : Found checkpoint, attempting to load... [2022-02-27 08:03:13 main:512] : INFO : Loading config [2022-02-27 08:03:13 main:514] : INFO : Loading state [2022-02-27 08:03:13 main:559] : INFO : Loading DataLoader into Memory [2022-02-27 08:03:13 main:562] : INFO : Starting Training [2022-02-27 08:03:15 main:574] : INFO : Epoch 1786 | loss: 0.0311921 | val_loss: 0.0311881 | Time: 1818.91 ms [2022-02-27 08:03:17 main:574] : INFO : Epoch 1787 | loss: 0.0311682 | val_loss: 0.031184 | Time: 1619.58 ms [2022-02-27 08:03:19 main:574] : INFO : Epoch 1788 | loss: 0.0311662 | val_loss: 0.0311798 | Time: 1625.19 ms [2022-02-27 08:03:20 main:574] : INFO : Epoch 1789 | loss: 0.0311643 | val_loss: 0.0311795 | Time: 1626.61 ms [2022-02-27 08:03:22 main:574] : INFO : Epoch 1790 | loss: 0.031163 | val_loss: 0.0311798 | Time: 1631.27 ms [2022-02-27 08:03:23 main:574] : INFO : Epoch 1791 | loss: 0.031164 | val_loss: 0.0311768 | Time: 1636.43 ms [2022-02-27 08:03:25 main:574] : INFO : Epoch 1792 | loss: 0.0311639 | val_loss: 0.031179 | Time: 1624.59 ms [2022-02-27 08:03:27 main:574] : INFO : Epoch 1793 | loss: 0.0311638 | val_loss: 0.0311789 | Time: 1626.51 ms [2022-02-27 08:03:28 main:574] : INFO : Epoch 1794 | loss: 0.0311637 | val_loss: 0.0311802 | Time: 1638.96 ms [2022-02-27 08:03:30 main:574] : INFO : Epoch 1795 | loss: 0.0311638 | val_loss: 0.0311823 | Time: 1623.47 ms [2022-02-27 08:03:32 main:574] : INFO : Epoch 1796 | loss: 0.0311633 | val_loss: 0.0311803 | Time: 1646.44 ms [2022-02-27 08:03:33 main:574] : INFO : Epoch 1797 | loss: 0.031163 | val_loss: 0.0311797 | Time: 1635.76 ms [2022-02-27 08:03:35 main:574] : INFO : Epoch 1798 | loss: 0.031163 | val_loss: 0.0311807 | Time: 1631 ms [2022-02-27 08:03:37 main:574] : INFO : Epoch 1799 | loss: 0.0311632 | val_loss: 0.0311799 | Time: 1629.19 ms [2022-02-27 08:03:38 main:574] : INFO : Epoch 1800 | loss: 0.0311647 | val_loss: 0.0311806 | Time: 1632.88 ms [2022-02-27 08:03:40 main:574] : INFO : Epoch 1801 | loss: 0.0311654 | val_loss: 0.0311823 | Time: 1626.59 ms [2022-02-27 08:03:42 main:574] : INFO : Epoch 1802 | loss: 0.0311636 | val_loss: 0.0311782 | Time: 1642.36 ms [2022-02-27 08:03:43 main:574] : INFO : Epoch 1803 | loss: 0.0311655 | val_loss: 0.0311805 | Time: 1633.25 ms [2022-02-27 08:03:45 main:574] : INFO : Epoch 1804 | loss: 0.0311641 | val_loss: 0.0311785 | Time: 1629.63 ms [2022-02-27 08:03:46 main:574] : INFO : Epoch 1805 | loss: 0.0311636 | val_loss: 0.0311803 | Time: 1628.18 ms [2022-02-27 08:03:48 main:574] : INFO : Epoch 1806 | loss: 0.0311645 | val_loss: 0.0311799 | Time: 1628.39 ms [2022-02-27 08:03:50 main:574] : INFO : Epoch 1807 | loss: 0.0311643 | val_loss: 0.0311792 | Time: 1627.39 ms [2022-02-27 08:03:51 main:574] : INFO : Epoch 1808 | loss: 0.031168 | val_loss: 0.0311807 | Time: 1640.95 ms [2022-02-27 08:03:53 main:574] : INFO : Epoch 1809 | loss: 0.0311691 | val_loss: 0.0311906 | Time: 1630 ms [2022-02-27 08:03:55 main:574] : INFO : Epoch 1810 | loss: 0.031167 | val_loss: 0.0311832 | Time: 1629.03 ms [2022-02-27 08:03:56 main:574] : INFO : Epoch 1811 | loss: 0.0311647 | val_loss: 0.0311799 | Time: 1634.08 ms [2022-02-27 08:03:58 main:574] : INFO : Epoch 1812 | loss: 0.031165 | val_loss: 0.0311808 | Time: 1628.01 ms [2022-02-27 08:04:00 main:574] : INFO : Epoch 1813 | loss: 0.0311644 | val_loss: 0.0311801 | Time: 1629.51 ms [2022-02-27 08:04:01 main:574] : INFO : Epoch 1814 | loss: 0.0311658 | val_loss: 0.0311786 | Time: 1641.03 ms [2022-02-27 08:04:03 main:574] : INFO : Epoch 1815 | loss: 0.0311646 | val_loss: 0.03118 | Time: 1637.48 ms [2022-02-27 08:04:04 main:574] : INFO : Epoch 1816 | loss: 0.0311623 | val_loss: 0.0311826 | Time: 1630.21 ms [2022-02-27 08:04:06 main:574] : INFO : Epoch 1817 | loss: 0.0311628 | val_loss: 0.0311795 | Time: 1628.48 ms [2022-02-27 08:04:08 main:574] : INFO : Epoch 1818 | loss: 0.0311629 | val_loss: 0.0311796 | Time: 1632.38 ms [2022-02-27 08:04:09 main:574] : INFO : Epoch 1819 | loss: 0.0311626 | val_loss: 0.031181 | Time: 1621.71 ms [2022-02-27 08:04:11 main:574] : INFO : Epoch 1820 | loss: 0.0311633 | val_loss: 0.0311799 | Time: 1637.06 ms [2022-02-27 08:04:13 main:574] : INFO : Epoch 1821 | loss: 0.0311635 | val_loss: 0.0311792 | Time: 1646.49 ms [2022-02-27 08:04:14 main:574] : INFO : Epoch 1822 | loss: 0.0311633 | val_loss: 0.0311826 | Time: 1638.99 ms [2022-02-27 08:04:16 main:574] : INFO : Epoch 1823 | loss: 0.0311642 | val_loss: 0.0311791 | Time: 1629.94 ms [2022-02-27 08:04:18 main:574] : INFO : Epoch 1824 | loss: 0.0311634 | val_loss: 0.0311788 | Time: 1633.94 ms [2022-02-27 08:04:19 main:574] : INFO : Epoch 1825 | loss: 0.0311633 | val_loss: 0.0311819 | Time: 1634.18 ms [2022-02-27 08:04:21 main:574] : INFO : Epoch 1826 | loss: 0.0311629 | val_loss: 0.0311805 | Time: 1638.88 ms [2022-02-27 08:04:22 main:574] : INFO : Epoch 1827 | loss: 0.0311627 | val_loss: 0.0311879 | Time: 1632.94 ms [2022-02-27 08:04:24 main:574] : INFO : Epoch 1828 | loss: 0.0311627 | val_loss: 0.031183 | Time: 1634.14 ms [2022-02-27 08:04:26 main:574] : INFO : Epoch 1829 | loss: 0.0311644 | val_loss: 0.031181 | Time: 1634.24 ms [2022-02-27 08:04:27 main:574] : INFO : Epoch 1830 | loss: 0.0311623 | val_loss: 0.0311794 | Time: 1622.86 ms [2022-02-27 08:04:29 main:574] : INFO : Epoch 1831 | loss: 0.0311636 | val_loss: 0.0311796 | Time: 1634.44 ms [2022-02-27 08:04:31 main:574] : INFO : Epoch 1832 | loss: 0.0311618 | val_loss: 0.0311816 | Time: 1630.53 ms [2022-02-27 08:04:32 main:574] : INFO : Epoch 1833 | loss: 0.031161 | val_loss: 0.0311823 | Time: 1649.36 ms [2022-02-27 08:04:34 main:574] : INFO : Epoch 1834 | loss: 0.0311618 | val_loss: 0.0311843 | Time: 1631.86 ms [2022-02-27 08:04:36 main:574] : INFO : Epoch 1835 | loss: 0.0311619 | val_loss: 0.0311803 | Time: 1635.85 ms [2022-02-27 08:04:37 main:574] : INFO : Epoch 1836 | loss: 0.0311613 | val_loss: 0.0311801 | Time: 1632.21 ms [2022-02-27 08:04:39 main:574] : INFO : Epoch 1837 | loss: 0.0311614 | val_loss: 0.0311795 | Time: 1632.13 ms [2022-02-27 08:04:41 main:574] : INFO : Epoch 1838 | loss: 0.0311614 | val_loss: 0.0311816 | Time: 1632.57 ms [2022-02-27 08:04:42 main:574] : INFO : Epoch 1839 | loss: 0.0311615 | val_loss: 0.0311822 | Time: 1634.49 ms [2022-02-27 08:04:44 main:574] : INFO : Epoch 1840 | loss: 0.0311634 | val_loss: 0.0311824 | Time: 1637.79 ms [2022-02-27 08:04:45 main:574] : INFO : Epoch 1841 | loss: 0.0311613 | val_loss: 0.0311881 | Time: 1632.42 ms [2022-02-27 08:04:47 main:574] : INFO : Epoch 1842 | loss: 0.0311602 | val_loss: 0.0311807 | Time: 1635.85 ms [2022-02-27 08:04:49 main:574] : INFO : Epoch 1843 | loss: 0.031159 | val_loss: 0.0311801 | Time: 1635.91 ms [2022-02-27 08:04:50 main:574] : INFO : Epoch 1844 | loss: 0.0311597 | val_loss: 0.0311832 | Time: 1636.07 ms [2022-02-27 08:04:52 main:574] : INFO : Epoch 1845 | loss: 0.0311601 | val_loss: 0.0311836 | Time: 1639.73 ms [2022-02-27 08:04:54 main:574] : INFO : Epoch 1846 | loss: 0.0311597 | val_loss: 0.0311852 | Time: 1633.65 ms [2022-02-27 08:04:55 main:574] : INFO : Epoch 1847 | loss: 0.0311595 | val_loss: 0.0311817 | Time: 1638.76 ms [2022-02-27 08:04:57 main:574] : INFO : Epoch 1848 | loss: 0.0311604 | val_loss: 0.0311818 | Time: 1629.76 ms [2022-02-27 08:04:59 main:574] : INFO : Epoch 1849 | loss: 0.0311625 | val_loss: 0.03118 | Time: 1635.21 ms [2022-02-27 08:05:00 main:574] : INFO : Epoch 1850 | loss: 0.0311622 | val_loss: 0.0311828 | Time: 1633.91 ms [2022-02-27 08:05:02 main:574] : INFO : Epoch 1851 | loss: 0.0311653 | val_loss: 0.0311831 | Time: 1640.79 ms [2022-02-27 08:05:03 main:574] : INFO : Epoch 1852 | loss: 0.0311643 | val_loss: 0.0311815 | Time: 1632.07 ms [2022-02-27 08:05:05 main:574] : INFO : Epoch 1853 | loss: 0.0311638 | val_loss: 0.0311784 | Time: 1631.55 ms [2022-02-27 08:05:07 main:574] : INFO : Epoch 1854 | loss: 0.0311635 | val_loss: 0.0311797 | Time: 1630.8 ms [2022-02-27 08:05:08 main:574] : INFO : Epoch 1855 | loss: 0.0311634 | val_loss: 0.0311795 | Time: 1635.49 ms [2022-02-27 08:05:10 main:574] : INFO : Epoch 1856 | loss: 0.0311645 | val_loss: 0.0311805 | Time: 1634.64 ms [2022-02-27 08:05:12 main:574] : INFO : Epoch 1857 | loss: 0.0311639 | val_loss: 0.0311831 | Time: 1635.88 ms [2022-02-27 08:05:13 main:574] : INFO : Epoch 1858 | loss: 0.031164 | val_loss: 0.0311826 | Time: 1648.71 ms [2022-02-27 08:05:15 main:574] : INFO : Epoch 1859 | loss: 0.031163 | val_loss: 0.0311797 | Time: 1636.13 ms [2022-02-27 08:05:17 main:574] : INFO : Epoch 1860 | loss: 0.0311634 | val_loss: 0.031181 | Time: 1628.59 ms [2022-02-27 08:05:18 main:574] : INFO : Epoch 1861 | loss: 0.0311658 | val_loss: 0.0311808 | Time: 1628.2 ms [2022-02-27 08:05:20 main:574] : INFO : Epoch 1862 | loss: 0.0311652 | val_loss: 0.0311859 | Time: 1628.25 ms [2022-02-27 08:05:22 main:574] : INFO : Epoch 1863 | loss: 0.0311642 | val_loss: 0.0311786 | Time: 1642.89 ms [2022-02-27 08:05:23 main:574] : INFO : Epoch 1864 | loss: 0.0311632 | val_loss: 0.0311818 | Time: 1643.59 ms [2022-02-27 08:05:25 main:574] : INFO : Epoch 1865 | loss: 0.0311629 | val_loss: 0.0311808 | Time: 1625.96 ms [2022-02-27 08:05:26 main:574] : INFO : Epoch 1866 | loss: 0.0311641 | val_loss: 0.0311824 | Time: 1631.61 ms [2022-02-27 08:05:28 main:574] : INFO : Epoch 1867 | loss: 0.0311635 | val_loss: 0.0311819 | Time: 1630.92 ms [2022-02-27 08:05:30 main:574] : INFO : Epoch 1868 | loss: 0.0311617 | val_loss: 0.031184 | Time: 1626.08 ms [2022-02-27 08:05:31 main:574] : INFO : Epoch 1869 | loss: 0.031164 | val_loss: 0.0311819 | Time: 1641.05 ms [2022-02-27 08:05:33 main:574] : INFO : Epoch 1870 | loss: 0.0311654 | val_loss: 0.03118 | Time: 1657.98 ms [2022-02-27 08:05:35 main:574] : INFO : Epoch 1871 | loss: 0.031165 | val_loss: 0.0311805 | Time: 1630.24 ms [2022-02-27 08:05:36 main:574] : INFO : Epoch 1872 | loss: 0.031162 | val_loss: 0.0311813 | Time: 1631.3 ms [2022-02-27 08:05:38 main:574] : INFO : Epoch 1873 | loss: 0.0311623 | val_loss: 0.0311812 | Time: 1634.68 ms [2022-02-27 08:05:40 main:574] : INFO : Epoch 1874 | loss: 0.0311633 | val_loss: 0.0311846 | Time: 1631.71 ms [2022-02-27 08:05:41 main:574] : INFO : Epoch 1875 | loss: 0.0311694 | val_loss: 0.0311806 | Time: 1637.46 ms [2022-02-27 08:05:43 main:574] : INFO : Epoch 1876 | loss: 0.0311687 | val_loss: 0.0311805 | Time: 1638.02 ms [2022-02-27 08:05:45 main:574] : INFO : Epoch 1877 | loss: 0.0311681 | val_loss: 0.031181 | Time: 1633.34 ms [2022-02-27 08:05:46 main:574] : INFO : Epoch 1878 | loss: 0.0311663 | val_loss: 0.0311828 | Time: 1647.93 ms [2022-02-27 08:05:48 main:574] : INFO : Epoch 1879 | loss: 0.031166 | val_loss: 0.0311806 | Time: 1628.48 ms [2022-02-27 08:05:49 main:574] : INFO : Epoch 1880 | loss: 0.0311647 | val_loss: 0.0311795 | Time: 1630.09 ms [2022-02-27 08:05:51 main:574] : INFO : Epoch 1881 | loss: 0.0311667 | val_loss: 0.0311794 | Time: 1633.97 ms [2022-02-27 08:05:53 main:574] : INFO : Epoch 1882 | loss: 0.0311661 | val_loss: 0.0311792 | Time: 1650.94 ms [2022-02-27 08:05:54 main:574] : INFO : Epoch 1883 | loss: 0.0311656 | val_loss: 0.0311799 | Time: 1630.64 ms [2022-02-27 08:05:56 main:574] : INFO : Epoch 1884 | loss: 0.0311643 | val_loss: 0.0311818 | Time: 1627.45 ms [2022-02-27 08:05:58 main:574] : INFO : Epoch 1885 | loss: 0.0311637 | val_loss: 0.0311794 | Time: 1626.1 ms [2022-02-27 08:05:59 main:574] : INFO : Epoch 1886 | loss: 0.0311646 | val_loss: 0.0311796 | Time: 1636.37 ms [2022-02-27 08:06:01 main:574] : INFO : Epoch 1887 | loss: 0.0311629 | val_loss: 0.0311779 | Time: 1636.14 ms [2022-02-27 08:06:03 main:574] : INFO : Epoch 1888 | loss: 0.0311618 | val_loss: 0.0311818 | Time: 1649.88 ms [2022-02-27 08:06:04 main:574] : INFO : Epoch 1889 | loss: 0.0311616 | val_loss: 0.0311806 | Time: 1639.88 ms [2022-02-27 08:06:06 main:574] : INFO : Epoch 1890 | loss: 0.0311622 | val_loss: 0.0311788 | Time: 1634.96 ms [2022-02-27 08:06:07 main:574] : INFO : Epoch 1891 | loss: 0.0311619 | val_loss: 0.0311827 | Time: 1625.42 ms [2022-02-27 08:06:09 main:574] : INFO : Epoch 1892 | loss: 0.0311623 | val_loss: 0.031181 | Time: 1633.59 ms [2022-02-27 08:06:11 main:574] : INFO : Epoch 1893 | loss: 0.0311612 | val_loss: 0.0311823 | Time: 1631.87 ms [2022-02-27 08:06:12 main:574] : INFO : Epoch 1894 | loss: 0.0311617 | val_loss: 0.03118 | Time: 1640.53 ms [2022-02-27 08:06:14 main:574] : INFO : Epoch 1895 | loss: 0.0311623 | val_loss: 0.0311824 | Time: 1650.51 ms [2022-02-27 08:06:16 main:574] : INFO : Epoch 1896 | loss: 0.0311618 | val_loss: 0.0311775 | Time: 1637.2 ms [2022-02-27 08:06:17 main:574] : INFO : Epoch 1897 | loss: 0.0311658 | val_loss: 0.0311844 | Time: 1637.07 ms [2022-02-27 08:06:19 main:574] : INFO : Epoch 1898 | loss: 0.0311662 | val_loss: 0.0311783 | Time: 1633.43 ms [2022-02-27 08:06:21 main:574] : INFO : Epoch 1899 | loss: 0.0311638 | val_loss: 0.031179 | Time: 1635.12 ms [2022-02-27 08:06:22 main:574] : INFO : Epoch 1900 | loss: 0.031163 | val_loss: 0.03118 | Time: 1639.91 ms [2022-02-27 08:06:24 main:574] : INFO : Epoch 1901 | loss: 0.0311618 | val_loss: 0.0311796 | Time: 1645.81 ms [2022-02-27 08:06:26 main:574] : INFO : Epoch 1902 | loss: 0.0311626 | val_loss: 0.0311776 | Time: 1640.74 ms [2022-02-27 08:06:27 main:574] : INFO : Epoch 1903 | loss: 0.0311649 | val_loss: 0.0311812 | Time: 1630.41 ms [2022-02-27 08:06:29 main:574] : INFO : Epoch 1904 | loss: 0.0311662 | val_loss: 0.0311804 | Time: 1629.87 ms [2022-02-27 08:06:31 main:574] : INFO : Epoch 1905 | loss: 0.0311641 | val_loss: 0.0311785 | Time: 1636.74 ms [2022-02-27 08:06:32 main:574] : INFO : Epoch 1906 | loss: 0.0311638 | val_loss: 0.031178 | Time: 1638.12 ms [2022-02-27 08:06:34 main:574] : INFO : Epoch 1907 | loss: 0.0311624 | val_loss: 0.0311788 | Time: 1640.45 ms [2022-02-27 08:06:35 main:574] : INFO : Epoch 1908 | loss: 0.0311668 | val_loss: 0.0311816 | Time: 1634.32 ms [2022-02-27 08:06:37 main:574] : INFO : Epoch 1909 | loss: 0.031167 | val_loss: 0.0311811 | Time: 1626.83 ms [2022-02-27 08:06:39 main:574] : INFO : Epoch 1910 | loss: 0.0311664 | val_loss: 0.031178 | Time: 1627.5 ms [2022-02-27 08:06:40 main:574] : INFO : Epoch 1911 | loss: 0.0311687 | val_loss: 0.0311786 | Time: 1637.99 ms [2022-02-27 08:06:42 main:574] : INFO : Epoch 1912 | loss: 0.0311755 | val_loss: 0.0311812 | Time: 1638.43 ms [2022-02-27 08:06:44 main:574] : INFO : Epoch 1913 | loss: 0.0311744 | val_loss: 0.0311801 | Time: 1637.23 ms [2022-02-27 08:06:45 main:574] : INFO : Epoch 1914 | loss: 0.0311727 | val_loss: 0.0311796 | Time: 1631.59 ms [2022-02-27 08:06:47 main:574] : INFO : Epoch 1915 | loss: 0.0311716 | val_loss: 0.0311784 | Time: 1641.24 ms [2022-02-27 08:06:49 main:574] : INFO : Epoch 1916 | loss: 0.0311708 | val_loss: 0.0311783 | Time: 1632.85 ms [2022-02-27 08:06:50 main:574] : INFO : Epoch 1917 | loss: 0.031169 | val_loss: 0.0311806 | Time: 1636.86 ms [2022-02-27 08:06:52 main:574] : INFO : Epoch 1918 | loss: 0.0311707 | val_loss: 0.0311794 | Time: 1636.38 ms [2022-02-27 08:06:53 main:574] : INFO : Epoch 1919 | loss: 0.0311685 | val_loss: 0.031179 | Time: 1649 ms [2022-02-27 08:06:55 main:574] : INFO : Epoch 1920 | loss: 0.0311679 | val_loss: 0.0311806 | Time: 1635.8 ms [2022-02-27 08:06:57 main:574] : INFO : Epoch 1921 | loss: 0.0311679 | val_loss: 0.0311782 | Time: 1639.77 ms [2022-02-27 08:06:58 main:574] : INFO : Epoch 1922 | loss: 0.0311683 | val_loss: 0.0311797 | Time: 1629.37 ms [2022-02-27 08:07:00 main:574] : INFO : Epoch 1923 | loss: 0.0311679 | val_loss: 0.0311792 | Time: 1647.07 ms [2022-02-27 08:07:02 main:574] : INFO : Epoch 1924 | loss: 0.0311681 | val_loss: 0.0311799 | Time: 1640.12 ms [2022-02-27 08:07:03 main:574] : INFO : Epoch 1925 | loss: 0.0311686 | val_loss: 0.0311866 | Time: 1647.93 ms [2022-02-27 08:07:05 main:574] : INFO : Epoch 1926 | loss: 0.0311703 | val_loss: 0.0311811 | Time: 1630.63 ms [2022-02-27 08:07:07 main:574] : INFO : Epoch 1927 | loss: 0.0314516 | val_loss: 0.0324315 | Time: 1640.86 ms [2022-02-27 08:07:08 main:574] : INFO : Epoch 1928 | loss: 0.0315876 | val_loss: 0.0313134 | Time: 1634.99 ms [2022-02-27 08:07:10 main:574] : INFO : Epoch 1929 | loss: 0.031219 | val_loss: 0.031201 | Time: 1636.79 ms [2022-02-27 08:07:12 main:574] : INFO : Epoch 1930 | loss: 0.0311803 | val_loss: 0.0311825 | Time: 1638.35 ms [2022-02-27 08:07:13 main:574] : INFO : Epoch 1931 | loss: 0.031172 | val_loss: 0.031177 | Time: 1643.51 ms [2022-02-27 08:07:15 main:574] : INFO : Epoch 1932 | loss: 0.0311693 | val_loss: 0.0311791 | Time: 1636.95 ms [2022-02-27 08:07:17 main:574] : INFO : Epoch 1933 | loss: 0.0311688 | val_loss: 0.0311814 | Time: 1644.48 ms [2022-02-27 08:07:18 main:574] : INFO : Epoch 1934 | loss: 0.0311687 | val_loss: 0.031176 | Time: 1629.56 ms [2022-02-27 08:07:20 main:574] : INFO : Epoch 1935 | loss: 0.0311673 | val_loss: 0.0311777 | Time: 1640.82 ms [2022-02-27 08:07:21 main:574] : INFO : Epoch 1936 | loss: 0.0311663 | val_loss: 0.0311773 | Time: 1639.95 ms [2022-02-27 08:07:23 main:574] : INFO : Epoch 1937 | loss: 0.0311656 | val_loss: 0.0311772 | Time: 1656.4 ms [2022-02-27 08:07:25 main:574] : INFO : Epoch 1938 | loss: 0.0311648 | val_loss: 0.031176 | Time: 1635.68 ms [2022-02-27 08:07:26 main:574] : INFO : Epoch 1939 | loss: 0.0311647 | val_loss: 0.0311783 | Time: 1636.55 ms [2022-02-27 08:07:28 main:574] : INFO : Epoch 1940 | loss: 0.0311649 | val_loss: 0.0311777 | Time: 1629.29 ms [2022-02-27 08:07:30 main:574] : INFO : Epoch 1941 | loss: 0.0311646 | val_loss: 0.0311783 | Time: 1636.35 ms [2022-02-27 08:07:31 main:574] : INFO : Epoch 1942 | loss: 0.0311665 | val_loss: 0.0311765 | Time: 1648.74 ms [2022-02-27 08:07:33 main:574] : INFO : Epoch 1943 | loss: 0.0311651 | val_loss: 0.0311799 | Time: 1649.67 ms [2022-02-27 08:07:35 main:574] : INFO : Epoch 1944 | loss: 0.0311648 | val_loss: 0.031176 | Time: 1627.26 ms [2022-02-27 08:07:36 main:574] : INFO : Epoch 1945 | loss: 0.031166 | val_loss: 0.0311847 | Time: 1638.8 ms [2022-02-27 08:07:38 main:574] : INFO : Epoch 1946 | loss: 0.0311668 | val_loss: 0.0311773 | Time: 1630.09 ms [2022-02-27 08:07:40 main:574] : INFO : Epoch 1947 | loss: 0.0311648 | val_loss: 0.0311764 | Time: 1632.38 ms [2022-02-27 08:07:41 main:574] : INFO : Epoch 1948 | loss: 0.0311651 | val_loss: 0.0311762 | Time: 1638.89 ms [2022-02-27 08:07:43 main:574] : INFO : Epoch 1949 | loss: 0.0311646 | val_loss: 0.0311776 | Time: 1660.17 ms [2022-02-27 08:07:45 main:574] : INFO : Epoch 1950 | loss: 0.0311646 | val_loss: 0.0311766 | Time: 1636.43 ms [2022-02-27 08:07:46 main:574] : INFO : Epoch 1951 | loss: 0.0311646 | val_loss: 0.0311787 | Time: 1638.76 ms [2022-02-27 08:07:48 main:574] : INFO : Epoch 1952 | loss: 0.0311648 | val_loss: 0.0311764 | Time: 1631.37 ms [2022-02-27 08:07:49 main:574] : INFO : Epoch 1953 | loss: 0.031165 | val_loss: 0.0311774 | Time: 1635.02 ms [2022-02-27 08:07:51 main:574] : INFO : Epoch 1954 | loss: 0.031164 | val_loss: 0.0311772 | Time: 1633.32 ms [2022-02-27 08:07:53 main:574] : INFO : Epoch 1955 | loss: 0.0311645 | val_loss: 0.0311761 | Time: 1648.3 ms [2022-02-27 08:07:54 main:574] : INFO : Epoch 1956 | loss: 0.0311638 | val_loss: 0.0311769 | Time: 1625.43 ms [2022-02-27 08:07:56 main:574] : INFO : Epoch 1957 | loss: 0.0311639 | val_loss: 0.0311763 | Time: 1635.32 ms [2022-02-27 08:07:58 main:574] : INFO : Epoch 1958 | loss: 0.0311637 | val_loss: 0.0311775 | Time: 1630.55 ms [2022-02-27 08:07:59 main:574] : INFO : Epoch 1959 | loss: 0.0311649 | val_loss: 0.0311792 | Time: 1631.39 ms [2022-02-27 08:08:01 main:574] : INFO : Epoch 1960 | loss: 0.0311643 | val_loss: 0.0311767 | Time: 1642.44 ms [2022-02-27 08:08:03 main:574] : INFO : Epoch 1961 | loss: 0.0311635 | val_loss: 0.0311787 | Time: 1643.97 ms [2022-02-27 08:08:04 main:574] : INFO : Epoch 1962 | loss: 0.0311624 | val_loss: 0.0311795 | Time: 1637.75 ms [2022-02-27 08:08:06 main:574] : INFO : Epoch 1963 | loss: 0.0311628 | val_loss: 0.0311799 | Time: 1676.77 ms [2022-02-27 08:08:08 main:574] : INFO : Epoch 1964 | loss: 0.0311644 | val_loss: 0.0311809 | Time: 1637.76 ms [2022-02-27 08:08:09 main:574] : INFO : Epoch 1965 | loss: 0.0311645 | val_loss: 0.0311776 | Time: 1631.11 ms [2022-02-27 08:08:11 main:574] : INFO : Epoch 1966 | loss: 0.0311645 | val_loss: 0.0311777 | Time: 1635.82 ms [2022-02-27 08:08:12 main:574] : INFO : Epoch 1967 | loss: 0.0311649 | val_loss: 0.0311757 | Time: 1637.77 ms [2022-02-27 08:08:14 main:574] : INFO : Epoch 1968 | loss: 0.0311642 | val_loss: 0.0311766 | Time: 1641.54 ms [2022-02-27 08:08:16 main:574] : INFO : Epoch 1969 | loss: 0.0311643 | val_loss: 0.0311766 | Time: 1640.01 ms [2022-02-27 08:08:17 main:574] : INFO : Epoch 1970 | loss: 0.0311628 | val_loss: 0.0311768 | Time: 1647.37 ms [2022-02-27 08:08:19 main:574] : INFO : Epoch 1971 | loss: 0.0311622 | val_loss: 0.0311756 | Time: 1634.52 ms [2022-02-27 08:08:21 main:574] : INFO : Epoch 1972 | loss: 0.0311621 | val_loss: 0.0311813 | Time: 1643.71 ms [2022-02-27 08:08:22 main:574] : INFO : Epoch 1973 | loss: 0.0311658 | val_loss: 0.0311807 | Time: 1634.84 ms [2022-02-27 08:08:24 main:574] : INFO : Epoch 1974 | loss: 0.031162 | val_loss: 0.0311757 | Time: 1640.45 ms [2022-02-27 08:08:26 main:574] : INFO : Epoch 1975 | loss: 0.0311609 | val_loss: 0.0311772 | Time: 1627.66 ms [2022-02-27 08:08:27 main:574] : INFO : Epoch 1976 | loss: 0.031161 | val_loss: 0.0311771 | Time: 1635.95 ms [2022-02-27 08:08:29 main:574] : INFO : Epoch 1977 | loss: 0.0311611 | val_loss: 0.0311766 | Time: 1632.19 ms [2022-02-27 08:08:31 main:574] : INFO : Epoch 1978 | loss: 0.031162 | val_loss: 0.0311767 | Time: 1649.52 ms [2022-02-27 08:08:32 main:574] : INFO : Epoch 1979 | loss: 0.0311616 | val_loss: 0.031177 | Time: 1646.11 ms [2022-02-27 08:08:34 main:574] : INFO : Epoch 1980 | loss: 0.0311626 | val_loss: 0.0311762 | Time: 1644.98 ms [2022-02-27 08:08:35 main:574] : INFO : Epoch 1981 | loss: 0.0311621 | val_loss: 0.0311766 | Time: 1628.88 ms [2022-02-27 08:08:37 main:574] : INFO : Epoch 1982 | loss: 0.0311619 | val_loss: 0.0311779 | Time: 1632.63 ms [2022-02-27 08:08:39 main:574] : INFO : Epoch 1983 | loss: 0.031161 | val_loss: 0.0311771 | Time: 1625.8 ms [2022-02-27 08:08:40 main:574] : INFO : Epoch 1984 | loss: 0.0311615 | val_loss: 0.0311792 | Time: 1643.88 ms [2022-02-27 08:08:42 main:574] : INFO : Epoch 1985 | loss: 0.0311617 | val_loss: 0.0311787 | Time: 1631.23 ms [2022-02-27 08:08:44 main:574] : INFO : Epoch 1986 | loss: 0.0311618 | val_loss: 0.0311752 | Time: 1650.84 ms [2022-02-27 08:08:45 main:574] : INFO : Epoch 1987 | loss: 0.0311602 | val_loss: 0.0311761 | Time: 1625.94 ms [2022-02-27 08:08:47 main:574] : INFO : Epoch 1988 | loss: 0.0311623 | val_loss: 0.0311755 | Time: 1659.85 ms [2022-02-27 08:08:49 main:574] : INFO : Epoch 1989 | loss: 0.0311627 | val_loss: 0.0311747 | Time: 1640.26 ms [2022-02-27 08:08:50 main:574] : INFO : Epoch 1990 | loss: 0.0311627 | val_loss: 0.031176 | Time: 1642.62 ms [2022-02-27 08:08:52 main:574] : INFO : Epoch 1991 | loss: 0.0311631 | val_loss: 0.0311755 | Time: 1638.51 ms [2022-02-27 08:08:54 main:574] : INFO : Epoch 1992 | loss: 0.0311632 | val_loss: 0.031177 | Time: 1641.29 ms [2022-02-27 08:08:55 main:574] : INFO : Epoch 1993 | loss: 0.0311629 | val_loss: 0.0311775 | Time: 1647.19 ms [2022-02-27 08:08:57 main:574] : INFO : Epoch 1994 | loss: 0.0311614 | val_loss: 0.0311767 | Time: 1635.02 ms [2022-02-27 08:08:59 main:574] : INFO : Epoch 1995 | loss: 0.0311618 | val_loss: 0.0311747 | Time: 1625.99 ms [2022-02-27 08:09:00 main:574] : INFO : Epoch 1996 | loss: 0.0311608 | val_loss: 0.031175 | Time: 1638.12 ms [2022-02-27 08:09:02 main:574] : INFO : Epoch 1997 | loss: 0.031161 | val_loss: 0.0311751 | Time: 1640.12 ms [2022-02-27 08:09:03 main:574] : INFO : Epoch 1998 | loss: 0.0311616 | val_loss: 0.0311763 | Time: 1643.55 ms [2022-02-27 08:09:05 main:574] : INFO : Epoch 1999 | loss: 0.0311606 | val_loss: 0.0311763 | Time: 1628.79 ms [2022-02-27 08:09:07 main:574] : INFO : Epoch 2000 | loss: 0.031161 | val_loss: 0.0311764 | Time: 1635.35 ms [2022-02-27 08:09:08 main:574] : INFO : Epoch 2001 | loss: 0.0311622 | val_loss: 0.0311747 | Time: 1630.2 ms [2022-02-27 08:09:10 main:574] : INFO : Epoch 2002 | loss: 0.0311625 | val_loss: 0.0311766 | Time: 1633.99 ms [2022-02-27 08:09:12 main:574] : INFO : Epoch 2003 | loss: 0.0311635 | val_loss: 0.0311765 | Time: 1638.79 ms [2022-02-27 08:09:13 main:574] : INFO : Epoch 2004 | loss: 0.0311622 | val_loss: 0.0311786 | Time: 1643.63 ms [2022-02-27 08:09:15 main:574] : INFO : Epoch 2005 | loss: 0.0311618 | val_loss: 0.0311778 | Time: 1635.61 ms [2022-02-27 08:09:17 main:574] : INFO : Epoch 2006 | loss: 0.0311617 | val_loss: 0.0311766 | Time: 1639.46 ms [2022-02-27 08:09:18 main:574] : INFO : Epoch 2007 | loss: 0.0311623 | val_loss: 0.0311752 | Time: 1631.6 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 970) [2022-02-27 08:09:55 main:435] : INFO : Set logging level to 1 [2022-02-27 08:09:55 main:441] : INFO : Running in BOINC Client mode [2022-02-27 08:09:55 main:444] : INFO : Resolving all filenames [2022-02-27 08:09:55 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-02-27 08:09:55 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-02-27 08:09:55 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-02-27 08:09:55 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-02-27 08:09:55 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-02-27 08:09:55 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-02-27 08:09:55 main:474] : INFO : Configuration: [2022-02-27 08:09:55 main:475] : INFO : Model type: GRU [2022-02-27 08:09:55 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-02-27 08:09:55 main:477] : INFO : Max Epochs: 2048 [2022-02-27 08:09:55 main:478] : INFO : Batch Size: 128 [2022-02-27 08:09:55 main:479] : INFO : Learning Rate: 0.01 [2022-02-27 08:09:55 main:480] : INFO : Patience: 10 [2022-02-27 08:09:55 main:481] : INFO : Hidden Width: 12 [2022-02-27 08:09:55 main:482] : INFO : # Recurrent Layers: 4 [2022-02-27 08:09:55 main:483] : INFO : # Backend Layers: 4 [2022-02-27 08:09:55 main:484] : INFO : # Threads: 1 [2022-02-27 08:09:55 main:486] : INFO : Preparing Dataset [2022-02-27 08:09:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-02-27 08:09:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-02-27 08:09:56 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-02-27 08:09:56 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-02-27 08:09:56 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-02-27 08:09:56 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-02-27 08:09:56 main:494] : INFO : Creating Model [2022-02-27 08:09:56 main:507] : INFO : Preparing config file [2022-02-27 08:09:56 main:511] : INFO : Found checkpoint, attempting to load... [2022-02-27 08:09:56 main:512] : INFO : Loading config [2022-02-27 08:09:56 main:514] : INFO : Loading state [2022-02-27 08:09:57 main:559] : INFO : Loading DataLoader into Memory [2022-02-27 08:09:57 main:562] : INFO : Starting Training [2022-02-27 08:09:59 main:574] : INFO : Epoch 1985 | loss: 0.0312415 | val_loss: 0.0311921 | Time: 1829.98 ms [2022-02-27 08:10:00 main:574] : INFO : Epoch 1986 | loss: 0.0311714 | val_loss: 0.0311849 | Time: 1621.05 ms [2022-02-27 08:10:02 main:574] : INFO : Epoch 1987 | loss: 0.0311652 | val_loss: 0.0311794 | Time: 1640.07 ms [2022-02-27 08:10:04 main:574] : INFO : Epoch 1988 | loss: 0.0311639 | val_loss: 0.0311766 | Time: 1635.71 ms [2022-02-27 08:10:05 main:574] : INFO : Epoch 1989 | loss: 0.0311638 | val_loss: 0.0311794 | Time: 1632.1 ms [2022-02-27 08:10:07 main:574] : INFO : Epoch 1990 | loss: 0.0311644 | val_loss: 0.0311777 | Time: 1632.72 ms [2022-02-27 08:10:09 main:574] : INFO : Epoch 1991 | loss: 0.0311629 | val_loss: 0.0311762 | Time: 1623.75 ms [2022-02-27 08:10:10 main:574] : INFO : Epoch 1992 | loss: 0.0311627 | val_loss: 0.0311796 | Time: 1639.92 ms [2022-02-27 08:10:12 main:574] : INFO : Epoch 1993 | loss: 0.0311653 | val_loss: 0.0311795 | Time: 1635.01 ms [2022-02-27 08:10:14 main:574] : INFO : Epoch 1994 | loss: 0.0311664 | val_loss: 0.0311764 | Time: 1652.01 ms [2022-02-27 08:10:15 main:574] : INFO : Epoch 1995 | loss: 0.0311658 | val_loss: 0.0311791 | Time: 1630.96 ms [2022-02-27 08:10:17 main:574] : INFO : Epoch 1996 | loss: 0.0311649 | val_loss: 0.0311793 | Time: 1636.31 ms [2022-02-27 08:10:18 main:574] : INFO : Epoch 1997 | loss: 0.0311652 | val_loss: 0.0311816 | Time: 1630.35 ms [2022-02-27 08:10:20 main:574] : INFO : Epoch 1998 | loss: 0.031165 | val_loss: 0.0311825 | Time: 1636.34 ms [2022-02-27 08:10:22 main:574] : INFO : Epoch 1999 | loss: 0.0311714 | val_loss: 0.0311797 | Time: 1642.85 ms [2022-02-27 08:10:23 main:574] : INFO : Epoch 2000 | loss: 0.0311689 | val_loss: 0.0311786 | Time: 1646.48 ms [2022-02-27 08:10:25 main:574] : INFO : Epoch 2001 | loss: 0.0311667 | val_loss: 0.0311798 | Time: 1636.23 ms [2022-02-27 08:10:27 main:574] : INFO : Epoch 2002 | loss: 0.031168 | val_loss: 0.0311808 | Time: 1628.79 ms [2022-02-27 08:10:28 main:574] : INFO : Epoch 2003 | loss: 0.0311672 | val_loss: 0.0311821 | Time: 1631.19 ms [2022-02-27 08:10:30 main:574] : INFO : Epoch 2004 | loss: 0.0311672 | val_loss: 0.0311789 | Time: 1637.85 ms [2022-02-27 08:10:32 main:574] : INFO : Epoch 2005 | loss: 0.0311662 | val_loss: 0.0311803 | Time: 1636.95 ms [2022-02-27 08:10:33 main:574] : INFO : Epoch 2006 | loss: 0.0311663 | val_loss: 0.0311817 | Time: 1642.79 ms [2022-02-27 08:10:35 main:574] : INFO : Epoch 2007 | loss: 0.0311673 | val_loss: 0.0311849 | Time: 1639.98 ms [2022-02-27 08:10:37 main:574] : INFO : Epoch 2008 | loss: 0.0311661 | val_loss: 0.0311848 | Time: 1635.47 ms [2022-02-27 08:10:38 main:574] : INFO : Epoch 2009 | loss: 0.0311682 | val_loss: 0.0311807 | Time: 1633.31 ms [2022-02-27 08:10:40 main:574] : INFO : Epoch 2010 | loss: 0.0311672 | val_loss: 0.0311801 | Time: 1629.78 ms [2022-02-27 08:10:41 main:574] : INFO : Epoch 2011 | loss: 0.0311662 | val_loss: 0.0311815 | Time: 1639.35 ms [2022-02-27 08:10:43 main:574] : INFO : Epoch 2012 | loss: 0.0311663 | val_loss: 0.0311803 | Time: 1645.79 ms [2022-02-27 08:10:45 main:574] : INFO : Epoch 2013 | loss: 0.0311667 | val_loss: 0.0311819 | Time: 1637.23 ms [2022-02-27 08:10:46 main:574] : INFO : Epoch 2014 | loss: 0.0311671 | val_loss: 0.0311809 | Time: 1640.35 ms [2022-02-27 08:10:48 main:574] : INFO : Epoch 2015 | loss: 0.0311666 | val_loss: 0.031183 | Time: 1636.26 ms [2022-02-27 08:10:50 main:574] : INFO : Epoch 2016 | loss: 0.0311674 | val_loss: 0.0311843 | Time: 1628.58 ms [2022-02-27 08:10:51 main:574] : INFO : Epoch 2017 | loss: 0.0311668 | val_loss: 0.0311846 | Time: 1643.52 ms [2022-02-27 08:10:53 main:574] : INFO : Epoch 2018 | loss: 0.0311677 | val_loss: 0.0311821 | Time: 1641.28 ms [2022-02-27 08:10:55 main:574] : INFO : Epoch 2019 | loss: 0.0311669 | val_loss: 0.0311812 | Time: 1649.27 ms [2022-02-27 08:10:56 main:574] : INFO : Epoch 2020 | loss: 0.0311657 | val_loss: 0.0311825 | Time: 1638.91 ms [2022-02-27 08:10:58 main:574] : INFO : Epoch 2021 | loss: 0.0311652 | val_loss: 0.0311815 | Time: 1639.03 ms [2022-02-27 08:11:00 main:574] : INFO : Epoch 2022 | loss: 0.0311662 | val_loss: 0.0311788 | Time: 1625.88 ms [2022-02-27 08:11:01 main:574] : INFO : Epoch 2023 | loss: 0.0311648 | val_loss: 0.0311783 | Time: 1648.1 ms [2022-02-27 08:11:03 main:574] : INFO : Epoch 2024 | loss: 0.0311644 | val_loss: 0.0311783 | Time: 1638.07 ms [2022-02-27 08:11:04 main:574] : INFO : Epoch 2025 | loss: 0.0311643 | val_loss: 0.0311775 | Time: 1648.33 ms [2022-02-27 08:11:06 main:574] : INFO : Epoch 2026 | loss: 0.0311636 | val_loss: 0.0311807 | Time: 1631.22 ms [2022-02-27 08:11:08 main:574] : INFO : Epoch 2027 | loss: 0.031163 | val_loss: 0.0311785 | Time: 1642.28 ms [2022-02-27 08:11:09 main:574] : INFO : Epoch 2028 | loss: 0.031164 | val_loss: 0.0311791 | Time: 1632.74 ms [2022-02-27 08:11:11 main:574] : INFO : Epoch 2029 | loss: 0.0311641 | val_loss: 0.0311818 | Time: 1643.82 ms [2022-02-27 08:11:13 main:574] : INFO : Epoch 2030 | loss: 0.0311623 | val_loss: 0.031179 | Time: 1639.2 ms [2022-02-27 08:11:14 main:574] : INFO : Epoch 2031 | loss: 0.0311625 | val_loss: 0.0311783 | Time: 1636.26 ms [2022-02-27 08:11:16 main:574] : INFO : Epoch 2032 | loss: 0.0311629 | val_loss: 0.0311806 | Time: 1637.86 ms [2022-02-27 08:11:18 main:574] : INFO : Epoch 2033 | loss: 0.0311626 | val_loss: 0.0311802 | Time: 1642.43 ms [2022-02-27 08:11:19 main:574] : INFO : Epoch 2034 | loss: 0.0311676 | val_loss: 0.0311805 | Time: 1631.5 ms [2022-02-27 08:11:21 main:574] : INFO : Epoch 2035 | loss: 0.0311701 | val_loss: 0.0311803 | Time: 1642.18 ms [2022-02-27 08:11:23 main:574] : INFO : Epoch 2036 | loss: 0.0311691 | val_loss: 0.0311772 | Time: 1639.56 ms [2022-02-27 08:11:24 main:574] : INFO : Epoch 2037 | loss: 0.0311681 | val_loss: 0.0311816 | Time: 1646.2 ms [2022-02-27 08:11:26 main:574] : INFO : Epoch 2038 | loss: 0.031168 | val_loss: 0.0311818 | Time: 1629.4 ms [2022-02-27 08:11:28 main:574] : INFO : Epoch 2039 | loss: 0.0311673 | val_loss: 0.0311805 | Time: 1638.08 ms [2022-02-27 08:11:29 main:574] : INFO : Epoch 2040 | loss: 0.0311675 | val_loss: 0.0311819 | Time: 1630.2 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 970) [2022-02-27 08:12:06 main:435] : INFO : Set logging level to 1 [2022-02-27 08:12:06 main:441] : INFO : Running in BOINC Client mode [2022-02-27 08:12:06 main:444] : INFO : Resolving all filenames [2022-02-27 08:12:06 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-02-27 08:12:06 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-02-27 08:12:06 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-02-27 08:12:06 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-02-27 08:12:06 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-02-27 08:12:06 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-02-27 08:12:06 main:474] : INFO : Configuration: [2022-02-27 08:12:06 main:475] : INFO : Model type: GRU [2022-02-27 08:12:06 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-02-27 08:12:06 main:477] : INFO : Max Epochs: 2048 [2022-02-27 08:12:06 main:478] : INFO : Batch Size: 128 [2022-02-27 08:12:06 main:479] : INFO : Learning Rate: 0.01 [2022-02-27 08:12:06 main:480] : INFO : Patience: 10 [2022-02-27 08:12:06 main:481] : INFO : Hidden Width: 12 [2022-02-27 08:12:06 main:482] : INFO : # Recurrent Layers: 4 [2022-02-27 08:12:06 main:483] : INFO : # Backend Layers: 4 [2022-02-27 08:12:06 main:484] : INFO : # Threads: 1 [2022-02-27 08:12:06 main:486] : INFO : Preparing Dataset [2022-02-27 08:12:06 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-02-27 08:12:06 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-02-27 08:12:07 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-02-27 08:12:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-02-27 08:12:08 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-02-27 08:12:08 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-02-27 08:12:08 main:494] : INFO : Creating Model [2022-02-27 08:12:08 main:507] : INFO : Preparing config file [2022-02-27 08:12:08 main:511] : INFO : Found checkpoint, attempting to load... [2022-02-27 08:12:08 main:512] : INFO : Loading config [2022-02-27 08:12:08 main:514] : INFO : Loading state [2022-02-27 08:12:08 main:559] : INFO : Loading DataLoader into Memory [2022-02-27 08:12:08 main:562] : INFO : Starting Training [2022-02-27 08:12:10 main:574] : INFO : Epoch 2024 | loss: 0.0312125 | val_loss: 0.0311873 | Time: 1842.68 ms [2022-02-27 08:12:12 main:574] : INFO : Epoch 2025 | loss: 0.0311714 | val_loss: 0.0311817 | Time: 1626.93 ms [2022-02-27 08:12:13 main:574] : INFO : Epoch 2026 | loss: 0.0311678 | val_loss: 0.0311812 | Time: 1629.64 ms [2022-02-27 08:12:15 main:574] : INFO : Epoch 2027 | loss: 0.0311677 | val_loss: 0.0311794 | Time: 1637.03 ms [2022-02-27 08:12:17 main:574] : INFO : Epoch 2028 | loss: 0.0311665 | val_loss: 0.0311778 | Time: 1634.09 ms [2022-02-27 08:12:18 main:574] : INFO : Epoch 2029 | loss: 0.0311659 | val_loss: 0.0311776 | Time: 1637.9 ms [2022-02-27 08:12:20 main:574] : INFO : Epoch 2030 | loss: 0.0311651 | val_loss: 0.0311769 | Time: 1629.57 ms [2022-02-27 08:12:21 main:574] : INFO : Epoch 2031 | loss: 0.0311645 | val_loss: 0.0311786 | Time: 1641.77 ms [2022-02-27 08:12:23 main:574] : INFO : Epoch 2032 | loss: 0.0311637 | val_loss: 0.0311798 | Time: 1633.83 ms [2022-02-27 08:12:25 main:574] : INFO : Epoch 2033 | loss: 0.0311643 | val_loss: 0.031181 | Time: 1643.99 ms [2022-02-27 08:12:26 main:574] : INFO : Epoch 2034 | loss: 0.0311657 | val_loss: 0.0311803 | Time: 1629.02 ms [2022-02-27 08:12:28 main:574] : INFO : Epoch 2035 | loss: 0.0311667 | val_loss: 0.0311784 | Time: 1631.97 ms [2022-02-27 08:12:30 main:574] : INFO : Epoch 2036 | loss: 0.031166 | val_loss: 0.0311792 | Time: 1630.37 ms [2022-02-27 08:12:31 main:574] : INFO : Epoch 2037 | loss: 0.0311647 | val_loss: 0.0311797 | Time: 1643.06 ms [2022-02-27 08:12:33 main:574] : INFO : Epoch 2038 | loss: 0.0311695 | val_loss: 0.0311842 | Time: 1633.71 ms [2022-02-27 08:12:35 main:574] : INFO : Epoch 2039 | loss: 0.0311723 | val_loss: 0.0311814 | Time: 1646.47 ms [2022-02-27 08:12:36 main:574] : INFO : Epoch 2040 | loss: 0.0311729 | val_loss: 0.0311828 | Time: 1630.94 ms [2022-02-27 08:12:38 main:574] : INFO : Epoch 2041 | loss: 0.0311705 | val_loss: 0.0311795 | Time: 1639.93 ms [2022-02-27 08:12:40 main:574] : INFO : Epoch 2042 | loss: 0.0311685 | val_loss: 0.0311812 | Time: 1626.18 ms [2022-02-27 08:12:41 main:574] : INFO : Epoch 2043 | loss: 0.0311685 | val_loss: 0.0311862 | Time: 1634.41 ms [2022-02-27 08:12:43 main:574] : INFO : Epoch 2044 | loss: 0.0311727 | val_loss: 0.0311814 | Time: 1636.26 ms [2022-02-27 08:12:44 main:574] : INFO : Epoch 2045 | loss: 0.0311682 | val_loss: 0.0311799 | Time: 1639.88 ms [2022-02-27 08:12:46 main:574] : INFO : Epoch 2046 | loss: 0.0311673 | val_loss: 0.0311795 | Time: 1633.65 ms [2022-02-27 08:12:48 main:574] : INFO : Epoch 2047 | loss: 0.0311671 | val_loss: 0.0311786 | Time: 1636.14 ms [2022-02-27 08:12:49 main:574] : INFO : Epoch 2048 | loss: 0.0311675 | val_loss: 0.0311791 | Time: 1632.4 ms [2022-02-27 08:12:49 main:597] : INFO : Saving trained model to model-final.pt, val_loss 0.0311791 [2022-02-27 08:12:49 main:603] : INFO : Saving end state to config to file [2022-02-27 08:12:49 main:608] : INFO : Success, exiting.. 08:12:49 (10264): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)