Task 13208246
| Name | ParityModified-1645998036-13064-3-0_0 |
| Workunit | 10330933 |
| Created | 15 Mar 2022, 2:27:35 UTC |
| Sent | 23 Mar 2022, 2:11:23 UTC |
| Report deadline | 31 Mar 2022, 2:11:23 UTC |
| Received | 25 Mar 2022, 0:19:56 UTC |
| Server state | Over |
| Outcome | Validate error |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 19912 |
| Run time | 3 hours 47 min 48 sec |
| CPU time | 3 hours 34 min 15 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 4,102.12 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.76 GB |
| Peak swap size | 4.31 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.20</core_client_version> <![CDATA[ <stderr_txt> :40 main:574] : INFO : Epoch 1674 | loss: 0.0310923 | val_loss: 0.0311414 | Time: 7632.54 ms [2022-03-23 21:10:48 main:574] : INFO : Epoch 1675 | loss: 0.0310883 | val_loss: 0.0311277 | Time: 8043.05 ms [2022-03-23 21:10:56 main:574] : INFO : Epoch 1676 | loss: 0.0310909 | val_loss: 0.0311286 | Time: 7893.55 ms [2022-03-23 21:11:13 main:574] : INFO : Epoch 1677 | loss: 0.0310855 | val_loss: 0.0311336 | Time: 17343 ms [2022-03-23 21:11:21 main:574] : INFO : Epoch 1678 | loss: 0.0310828 | val_loss: 0.0311308 | Time: 7835.05 ms [2022-03-23 21:11:29 main:574] : INFO : Epoch 1679 | loss: 0.0310853 | val_loss: 0.0311319 | Time: 7589.28 ms [2022-03-23 21:11:37 main:574] : INFO : Epoch 1680 | loss: 0.0310878 | val_loss: 0.0311379 | Time: 8552.81 ms [2022-03-23 21:11:45 main:574] : INFO : Epoch 1681 | loss: 0.0310883 | val_loss: 0.0311322 | Time: 7737.35 ms [2022-03-23 21:11:53 main:574] : INFO : Epoch 1682 | loss: 0.0310914 | val_loss: 0.0311322 | Time: 7685.04 ms [2022-03-23 21:12:01 main:574] : INFO : Epoch 1683 | loss: 0.0310922 | val_loss: 0.031127 | Time: 8448.39 ms [2022-03-23 21:12:09 main:574] : INFO : Epoch 1684 | loss: 0.031087 | val_loss: 0.0311318 | Time: 7594.89 ms [2022-03-23 21:12:17 main:574] : INFO : Epoch 1685 | loss: 0.0310902 | val_loss: 0.0311296 | Time: 8232.4 ms [2022-03-23 21:12:25 main:574] : INFO : Epoch 1686 | loss: 0.0310869 | val_loss: 0.0311338 | Time: 7572.01 ms [2022-03-23 21:12:32 main:574] : INFO : Epoch 1687 | loss: 0.0310849 | val_loss: 0.0311325 | Time: 7689.93 ms [2022-03-23 21:12:41 main:574] : INFO : Epoch 1688 | loss: 0.0310828 | val_loss: 0.0311465 | Time: 8446.04 ms [2022-03-23 21:12:49 main:574] : INFO : Epoch 1689 | loss: 0.0310864 | val_loss: 0.031134 | Time: 7785.87 ms [2022-03-23 21:12:57 main:574] : INFO : Epoch 1690 | loss: 0.0310872 | val_loss: 0.0311333 | Time: 8287.96 ms [2022-03-23 21:13:04 main:574] : INFO : Epoch 1691 | loss: 0.0310883 | val_loss: 0.0311309 | Time: 7638.13 ms [2022-03-23 21:13:13 main:574] : INFO : Epoch 1692 | loss: 0.0310817 | val_loss: 0.031129 | Time: 8472.43 ms [2022-03-23 21:13:20 main:574] : INFO : Epoch 1693 | loss: 0.0310916 | val_loss: 0.031142 | Time: 7491.68 ms [2022-03-23 21:13:29 main:574] : INFO : Epoch 1694 | loss: 0.0310881 | val_loss: 0.0311431 | Time: 8160 ms [2022-03-23 21:13:37 main:574] : INFO : Epoch 1695 | loss: 0.0310891 | val_loss: 0.0311322 | Time: 8761.55 ms [2022-03-23 21:13:55 main:574] : INFO : Epoch 1696 | loss: 0.0310888 | val_loss: 0.0311316 | Time: 17799.7 ms [2022-03-23 21:14:04 main:574] : INFO : Epoch 1697 | loss: 0.0310854 | val_loss: 0.0311365 | Time: 8375.9 ms [2022-03-23 21:14:11 main:574] : INFO : Epoch 1698 | loss: 0.031088 | val_loss: 0.0311331 | Time: 7605.3 ms [2022-03-23 21:14:20 main:574] : INFO : Epoch 1699 | loss: 0.0310887 | val_loss: 0.03113 | Time: 8618.93 ms [2022-03-23 21:14:27 main:574] : INFO : Epoch 1700 | loss: 0.0310894 | val_loss: 0.0311266 | Time: 7457.73 ms [2022-03-23 21:14:35 main:574] : INFO : Epoch 1701 | loss: 0.0310866 | val_loss: 0.0311306 | Time: 7645.04 ms [2022-03-23 21:14:43 main:574] : INFO : Epoch 1702 | loss: 0.0310889 | val_loss: 0.0311378 | Time: 8329.97 ms [2022-03-23 21:14:51 main:574] : INFO : Epoch 1703 | loss: 0.031086 | val_loss: 0.0311334 | Time: 7797.43 ms [2022-03-23 21:15:00 main:574] : INFO : Epoch 1704 | loss: 0.0310903 | val_loss: 0.031129 | Time: 8932.01 ms [2022-03-23 21:15:08 main:574] : INFO : Epoch 1705 | loss: 0.031089 | val_loss: 0.0311404 | Time: 7632.78 ms [2022-03-23 21:15:15 main:574] : INFO : Epoch 1706 | loss: 0.0310873 | val_loss: 0.0311358 | Time: 7830.37 ms [2022-03-23 21:15:24 main:574] : INFO : Epoch 1707 | loss: 0.0310855 | val_loss: 0.031133 | Time: 8243.32 ms [2022-03-23 21:15:31 main:574] : INFO : Epoch 1708 | loss: 0.0310856 | val_loss: 0.0311346 | Time: 7659.56 ms [2022-03-23 21:15:39 main:574] : INFO : Epoch 1709 | loss: 0.0310835 | val_loss: 0.031129 | Time: 7990.08 ms [2022-03-23 21:15:47 main:574] : INFO : Epoch 1710 | loss: 0.0310821 | val_loss: 0.0311279 | Time: 7915.78 ms [2022-03-23 21:15:55 main:574] : INFO : Epoch 1711 | loss: 0.0310826 | val_loss: 0.0311415 | Time: 7756.92 ms [2022-03-23 21:16:04 main:574] : INFO : Epoch 1712 | loss: 0.0310886 | val_loss: 0.0311462 | Time: 8695.16 ms [2022-03-23 21:16:11 main:574] : INFO : Epoch 1713 | loss: 0.0310838 | val_loss: 0.0311292 | Time: 7653.85 ms [2022-03-23 21:16:20 main:574] : INFO : Epoch 1714 | loss: 0.0310862 | val_loss: 0.0311408 | Time: 8246.58 ms [2022-03-23 21:16:27 main:574] : INFO : Epoch 1715 | loss: 0.0310876 | val_loss: 0.0311326 | Time: 7599.83 ms [2022-03-23 21:16:36 main:574] : INFO : Epoch 1716 | loss: 0.031086 | val_loss: 0.0311399 | Time: 8308.26 ms [2022-03-23 21:16:43 main:574] : INFO : Epoch 1717 | loss: 0.0310822 | val_loss: 0.0311261 | Time: 7736.92 ms [2022-03-23 21:16:51 main:574] : INFO : Epoch 1718 | loss: 0.0310859 | val_loss: 0.0311336 | Time: 7655.65 ms [2022-03-23 21:17:10 main:574] : INFO : Epoch 1719 | loss: 0.0310795 | val_loss: 0.031128 | Time: 18711.2 ms [2022-03-23 21:17:17 main:574] : INFO : Epoch 1720 | loss: 0.0310788 | val_loss: 0.0311317 | Time: 7425.01 ms [2022-03-23 21:17:25 main:574] : INFO : Epoch 1721 | loss: 0.0310786 | val_loss: 0.0311325 | Time: 8046.3 ms [2022-03-23 21:17:33 main:574] : INFO : Epoch 1722 | loss: 0.0310807 | val_loss: 0.0311541 | Time: 7323.85 ms [2022-03-23 21:17:40 main:574] : INFO : Epoch 1723 | loss: 0.0310852 | val_loss: 0.0311381 | Time: 7313.49 ms [2022-03-23 21:17:48 main:574] : INFO : Epoch 1724 | loss: 0.0310851 | val_loss: 0.0311347 | Time: 7820.38 ms [2022-03-23 21:17:56 main:574] : INFO : Epoch 1725 | loss: 0.0310862 | val_loss: 0.0311338 | Time: 7975.61 ms [2022-03-23 21:18:04 main:574] : INFO : Epoch 1726 | loss: 0.0310944 | val_loss: 0.0311349 | Time: 8447.52 ms [2022-03-23 21:18:12 main:574] : INFO : Epoch 1727 | loss: 0.0311282 | val_loss: 0.0311365 | Time: 7533.05 ms [2022-03-23 21:18:20 main:574] : INFO : Epoch 1728 | loss: 0.0311203 | val_loss: 0.0311217 | Time: 7888.47 ms [2022-03-23 21:18:28 main:574] : INFO : Epoch 1729 | loss: 0.0311196 | val_loss: 0.0311302 | Time: 8331.85 ms [2022-03-23 21:18:45 main:574] : INFO : Epoch 1730 | loss: 0.0311092 | val_loss: 0.031129 | Time: 17135.7 ms [2022-03-23 21:18:53 main:574] : INFO : Epoch 1731 | loss: 0.0310992 | val_loss: 0.0311263 | Time: 7782.31 ms [2022-03-23 21:19:01 main:574] : INFO : Epoch 1732 | loss: 0.0311009 | val_loss: 0.0311264 | Time: 7745.86 ms [2022-03-23 21:19:19 main:574] : INFO : Epoch 1733 | loss: 0.0311001 | val_loss: 0.0311274 | Time: 18699.8 ms [2022-03-23 21:19:27 main:574] : INFO : Epoch 1734 | loss: 0.031095 | val_loss: 0.0311239 | Time: 7526.79 ms [2022-03-23 21:19:34 main:574] : INFO : Epoch 1735 | loss: 0.0310951 | val_loss: 0.031134 | Time: 7219.66 ms [2022-03-23 21:19:42 main:574] : INFO : Epoch 1736 | loss: 0.0310906 | val_loss: 0.0311268 | Time: 8136.39 ms [2022-03-23 21:19:50 main:574] : INFO : Epoch 1737 | loss: 0.0310887 | val_loss: 0.0311307 | Time: 7461.01 ms [2022-03-23 21:19:58 main:574] : INFO : Epoch 1738 | loss: 0.0310912 | val_loss: 0.0311264 | Time: 8530.07 ms [2022-03-23 21:20:06 main:574] : INFO : Epoch 1739 | loss: 0.0310881 | val_loss: 0.0311371 | Time: 7678.09 ms [2022-03-23 21:20:14 main:574] : INFO : Epoch 1740 | loss: 0.0310924 | val_loss: 0.031127 | Time: 7587.26 ms [2022-03-23 21:20:22 main:574] : INFO : Epoch 1741 | loss: 0.0310951 | val_loss: 0.0311328 | Time: 8214.32 ms [2022-03-23 21:20:30 main:574] : INFO : Epoch 1742 | loss: 0.0310955 | val_loss: 0.0311258 | Time: 7759.34 ms [2022-03-23 21:20:38 main:574] : INFO : Epoch 1743 | loss: 0.0310883 | val_loss: 0.0311264 | Time: 8252.39 ms [2022-03-23 21:20:46 main:574] : INFO : Epoch 1744 | loss: 0.0310833 | val_loss: 0.0311272 | Time: 7810.73 ms [2022-03-23 21:20:54 main:574] : INFO : Epoch 1745 | loss: 0.0310833 | val_loss: 0.0311395 | Time: 7864.84 ms [2022-03-23 21:21:02 main:574] : INFO : Epoch 1746 | loss: 0.0310846 | val_loss: 0.0311261 | Time: 8673.49 ms [2022-03-23 21:21:10 main:574] : INFO : Epoch 1747 | loss: 0.0310845 | val_loss: 0.031126 | Time: 7527.48 ms [2022-03-23 21:21:18 main:574] : INFO : Epoch 1748 | loss: 0.0310847 | val_loss: 0.0311313 | Time: 8303.86 ms [2022-03-23 21:21:26 main:574] : INFO : Epoch 1749 | loss: 0.0310847 | val_loss: 0.0311265 | Time: 8007.45 ms [2022-03-23 21:21:34 main:574] : INFO : Epoch 1750 | loss: 0.0310868 | val_loss: 0.0311303 | Time: 8286.92 ms [2022-03-23 21:21:42 main:574] : INFO : Epoch 1751 | loss: 0.0310872 | val_loss: 0.0311273 | Time: 7700.4 ms [2022-03-23 21:21:50 main:574] : INFO : Epoch 1752 | loss: 0.0310933 | val_loss: 0.0311323 | Time: 7929.09 ms [2022-03-23 21:21:58 main:574] : INFO : Epoch 1753 | loss: 0.0310914 | val_loss: 0.0311392 | Time: 8257.4 ms [2022-03-23 21:22:06 main:574] : INFO : Epoch 1754 | loss: 0.0310851 | val_loss: 0.0311276 | Time: 7787.66 ms [2022-03-23 21:22:15 main:574] : INFO : Epoch 1755 | loss: 0.031084 | val_loss: 0.0311381 | Time: 8538.92 ms [2022-03-23 21:22:32 main:574] : INFO : Epoch 1756 | loss: 0.0310875 | val_loss: 0.0311298 | Time: 17735.7 ms [2022-03-23 21:22:40 main:574] : INFO : Epoch 1757 | loss: 0.0310829 | val_loss: 0.0311292 | Time: 8042.05 ms [2022-03-23 21:22:48 main:574] : INFO : Epoch 1758 | loss: 0.0310818 | val_loss: 0.0311298 | Time: 7491.96 ms [2022-03-23 21:23:00 main:574] : INFO : Epoch 1759 | loss: 0.0310814 | val_loss: 0.0311259 | Time: 12198 ms [2022-03-23 21:23:46 main:574] : INFO : Epoch 1760 | loss: 0.0310809 | val_loss: 0.0311291 | Time: 46216 ms [2022-03-23 21:24:04 main:574] : INFO : Epoch 1761 | loss: 0.031081 | val_loss: 0.0311412 | Time: 17754.1 ms [2022-03-23 21:24:11 main:574] : INFO : Epoch 1762 | loss: 0.0310796 | val_loss: 0.0311353 | Time: 7160.33 ms [2022-03-23 21:24:21 main:574] : INFO : Epoch 1763 | loss: 0.0310782 | val_loss: 0.031132 | Time: 9338.91 ms [2022-03-23 21:24:29 main:574] : INFO : Epoch 1764 | loss: 0.0310823 | val_loss: 0.0311273 | Time: 8262.88 ms [2022-03-23 21:24:37 main:574] : INFO : Epoch 1765 | loss: 0.031082 | val_loss: 0.0311283 | Time: 7949.94 ms [2022-03-23 21:24:45 main:574] : INFO : Epoch 1766 | loss: 0.0310845 | val_loss: 0.0311339 | Time: 8012.43 ms [2022-03-23 21:24:53 main:574] : INFO : Epoch 1767 | loss: 0.0310883 | val_loss: 0.0311346 | Time: 8365.81 ms [2022-03-23 21:25:02 main:574] : INFO : Epoch 1768 | loss: 0.0310823 | val_loss: 0.0311362 | Time: 9097.22 ms [2022-03-23 21:25:10 main:574] : INFO : Epoch 1769 | loss: 0.0310841 | val_loss: 0.0311443 | Time: 7884.61 ms [2022-03-23 21:25:19 main:574] : INFO : Epoch 1770 | loss: 0.03109 | val_loss: 0.03113 | Time: 8403.16 ms [2022-03-23 21:25:27 main:574] : INFO : Epoch 1771 | loss: 0.0310879 | val_loss: 0.0311405 | Time: 8290.64 ms [2022-03-23 21:25:35 main:574] : INFO : Epoch 1772 | loss: 0.0310843 | val_loss: 0.0311322 | Time: 7799.46 ms [2022-03-23 21:25:42 main:574] : INFO : Epoch 1773 | loss: 0.0310817 | val_loss: 0.031131 | Time: 7749.12 ms [2022-03-23 21:25:50 main:574] : INFO : Epoch 1774 | loss: 0.0310813 | val_loss: 0.0311289 | Time: 7825.93 ms [2022-03-23 21:25:59 main:574] : INFO : Epoch 1775 | loss: 0.0310817 | val_loss: 0.0311329 | Time: 8205.61 ms [2022-03-23 21:26:07 main:574] : INFO : Epoch 1776 | loss: 0.0310858 | val_loss: 0.0311307 | Time: 8315.02 ms [2022-03-23 21:26:15 main:574] : INFO : Epoch 1777 | loss: 0.0310856 | val_loss: 0.0311262 | Time: 8169.94 ms [2022-03-23 21:26:23 main:574] : INFO : Epoch 1778 | loss: 0.0310886 | val_loss: 0.031128 | Time: 8042.32 ms [2022-03-23 21:26:31 main:574] : INFO : Epoch 1779 | loss: 0.0310905 | val_loss: 0.0311269 | Time: 8276.77 ms [2022-03-23 21:26:40 main:574] : INFO : Epoch 1780 | loss: 0.0310967 | val_loss: 0.0311274 | Time: 8568.81 ms [2022-03-23 21:26:57 main:574] : INFO : Epoch 1781 | loss: 0.0310966 | val_loss: 0.0311495 | Time: 17101.3 ms [2022-03-23 21:27:05 main:574] : INFO : Epoch 1782 | loss: 0.0311029 | val_loss: 0.0311314 | Time: 7538.61 ms [2022-03-23 21:27:12 main:574] : INFO : Epoch 1783 | loss: 0.0311098 | val_loss: 0.0311326 | Time: 7585.51 ms [2022-03-23 21:27:20 main:574] : INFO : Epoch 1784 | loss: 0.031104 | val_loss: 0.031137 | Time: 7705.61 ms [2022-03-23 21:27:28 main:574] : INFO : Epoch 1785 | loss: 0.0311033 | val_loss: 0.0311341 | Time: 7609.16 ms [2022-03-23 21:27:35 main:574] : INFO : Epoch 1786 | loss: 0.031098 | val_loss: 0.0311291 | Time: 7305.01 ms [2022-03-23 21:27:43 main:574] : INFO : Epoch 1787 | loss: 0.0311002 | val_loss: 0.0311319 | Time: 8323.46 ms [2022-03-23 21:27:51 main:574] : INFO : Epoch 1788 | loss: 0.0310955 | val_loss: 0.031135 | Time: 7844.23 ms [2022-03-23 21:27:59 main:574] : INFO : Epoch 1789 | loss: 0.0310943 | val_loss: 0.0311357 | Time: 7656.36 ms [2022-03-23 21:28:06 main:574] : INFO : Epoch 1790 | loss: 0.0310953 | val_loss: 0.0311265 | Time: 7756.51 ms [2022-03-23 21:28:14 main:574] : INFO : Epoch 1791 | loss: 0.0310904 | val_loss: 0.0311283 | Time: 7757.04 ms [2022-03-23 21:28:22 main:574] : INFO : Epoch 1792 | loss: 0.0310891 | val_loss: 0.0311311 | Time: 7724.95 ms [2022-03-23 21:28:30 main:574] : INFO : Epoch 1793 | loss: 0.0310881 | val_loss: 0.0311268 | Time: 7861.61 ms [2022-03-23 21:28:47 main:574] : INFO : Epoch 1794 | loss: 0.0310864 | val_loss: 0.0311233 | Time: 17373.1 ms [2022-03-23 21:28:55 main:574] : INFO : Epoch 1795 | loss: 0.0310876 | val_loss: 0.0311253 | Time: 7665.79 ms [2022-03-23 21:29:03 main:574] : INFO : Epoch 1796 | loss: 0.031085 | val_loss: 0.031122 | Time: 7837.05 ms [2022-03-23 21:29:11 main:574] : INFO : Epoch 1797 | loss: 0.0310828 | val_loss: 0.0311329 | Time: 7967.26 ms [2022-03-23 21:29:18 main:574] : INFO : Epoch 1798 | loss: 0.0310832 | val_loss: 0.0311328 | Time: 7782.88 ms [2022-03-23 21:29:26 main:574] : INFO : Epoch 1799 | loss: 0.0310842 | val_loss: 0.0311305 | Time: 7586.59 ms [2022-03-23 21:29:34 main:574] : INFO : Epoch 1800 | loss: 0.0310817 | val_loss: 0.031127 | Time: 7532.19 ms [2022-03-23 21:29:41 main:574] : INFO : Epoch 1801 | loss: 0.0310868 | val_loss: 0.0311353 | Time: 7127.15 ms [2022-03-23 21:29:49 main:574] : INFO : Epoch 1802 | loss: 0.0310946 | val_loss: 0.031133 | Time: 8037.25 ms [2022-03-23 21:29:57 main:574] : INFO : Epoch 1803 | loss: 0.0310868 | val_loss: 0.031124 | Time: 8174.95 ms [2022-03-23 21:30:05 main:574] : INFO : Epoch 1804 | loss: 0.0310838 | val_loss: 0.0311258 | Time: 7922.53 ms [2022-03-23 21:30:13 main:574] : INFO : Epoch 1805 | loss: 0.0310827 | val_loss: 0.0311287 | Time: 7624.11 ms [2022-03-23 21:30:20 main:574] : INFO : Epoch 1806 | loss: 0.031083 | val_loss: 0.0311235 | Time: 7225.25 ms [2022-03-23 21:30:28 main:574] : INFO : Epoch 1807 | loss: 0.0310854 | val_loss: 0.0311433 | Time: 7771.74 ms [2022-03-23 21:30:35 main:574] : INFO : Epoch 1808 | loss: 0.0310885 | val_loss: 0.0311354 | Time: 7870.17 ms [2022-03-23 21:30:44 main:574] : INFO : Epoch 1809 | loss: 0.0310877 | val_loss: 0.0311258 | Time: 8147.76 ms [2022-03-23 21:30:52 main:574] : INFO : Epoch 1810 | loss: 0.0310811 | val_loss: 0.0311338 | Time: 8051.47 ms [2022-03-23 21:31:00 main:574] : INFO : Epoch 1811 | loss: 0.0310802 | val_loss: 0.031129 | Time: 8523.6 ms [2022-03-23 21:31:08 main:574] : INFO : Epoch 1812 | loss: 0.0310855 | val_loss: 0.0311346 | Time: 7710.29 ms [2022-03-23 21:31:15 main:574] : INFO : Epoch 1813 | loss: 0.0310808 | val_loss: 0.031128 | Time: 7422.92 ms [2022-03-23 21:31:23 main:574] : INFO : Epoch 1814 | loss: 0.0310812 | val_loss: 0.0311271 | Time: 7935.13 ms [2022-03-23 21:31:31 main:574] : INFO : Epoch 1815 | loss: 0.0310798 | val_loss: 0.031127 | Time: 8244.51 ms [2022-03-23 21:31:39 main:574] : INFO : Epoch 1816 | loss: 0.0310803 | val_loss: 0.03113 | Time: 7758.14 ms [2022-03-23 21:31:47 main:574] : INFO : Epoch 1817 | loss: 0.0310844 | val_loss: 0.0311314 | Time: 7889.32 ms [2022-03-23 21:31:55 main:574] : INFO : Epoch 1818 | loss: 0.031087 | val_loss: 0.0311324 | Time: 7705.87 ms [2022-03-23 21:32:03 main:574] : INFO : Epoch 1819 | loss: 0.03108 | val_loss: 0.0311264 | Time: 7962.39 ms [2022-03-23 21:32:11 main:574] : INFO : Epoch 1820 | loss: 0.031081 | val_loss: 0.0311448 | Time: 7711.23 ms [2022-03-23 21:32:18 main:574] : INFO : Epoch 1821 | loss: 0.0310907 | val_loss: 0.0311248 | Time: 7842.76 ms [2022-03-23 21:32:26 main:574] : INFO : Epoch 1822 | loss: 0.0310855 | val_loss: 0.0311306 | Time: 7624.45 ms [2022-03-23 21:32:34 main:574] : INFO : Epoch 1823 | loss: 0.031085 | val_loss: 0.0311295 | Time: 7678.19 ms [2022-03-23 21:32:41 main:574] : INFO : Epoch 1824 | loss: 0.0310821 | val_loss: 0.0311318 | Time: 7512.07 ms [2022-03-23 21:32:49 main:574] : INFO : Epoch 1825 | loss: 0.0310824 | val_loss: 0.0311356 | Time: 7907.33 ms [2022-03-23 21:32:57 main:574] : INFO : Epoch 1826 | loss: 0.0310816 | val_loss: 0.0311283 | Time: 8173.43 ms [2022-03-23 21:33:05 main:574] : INFO : Epoch 1827 | loss: 0.031089 | val_loss: 0.0311355 | Time: 7859.91 ms [2022-03-23 21:33:13 main:574] : INFO : Epoch 1828 | loss: 0.0310861 | val_loss: 0.0311283 | Time: 8017.78 ms [2022-03-23 21:33:21 main:574] : INFO : Epoch 1829 | loss: 0.0310867 | val_loss: 0.0311377 | Time: 7889.59 ms [2022-03-23 21:33:29 main:574] : INFO : Epoch 1830 | loss: 0.0310872 | val_loss: 0.0311284 | Time: 8089.79 ms [2022-03-23 21:33:37 main:574] : INFO : Epoch 1831 | loss: 0.0310817 | val_loss: 0.0311329 | Time: 7955.33 ms [2022-03-23 21:33:45 main:574] : INFO : Epoch 1832 | loss: 0.0310814 | val_loss: 0.0311302 | Time: 8098.84 ms [2022-03-23 21:33:53 main:574] : INFO : Epoch 1833 | loss: 0.0310842 | val_loss: 0.0311329 | Time: 7816.66 ms [2022-03-23 21:34:01 main:574] : INFO : Epoch 1834 | loss: 0.0310801 | val_loss: 0.0311321 | Time: 8136.91 ms [2022-03-23 21:34:09 main:574] : INFO : Epoch 1835 | loss: 0.0310822 | val_loss: 0.0311319 | Time: 7898.7 ms [2022-03-23 21:34:18 main:574] : INFO : Epoch 1836 | loss: 0.0310849 | val_loss: 0.0311281 | Time: 9093.82 ms [2022-03-23 21:34:37 main:574] : INFO : Epoch 1837 | loss: 0.0310818 | val_loss: 0.0311342 | Time: 18262.5 ms [2022-03-23 21:34:44 main:574] : INFO : Epoch 1838 | loss: 0.0310822 | val_loss: 0.0311326 | Time: 7372.73 ms [2022-03-23 21:34:52 main:574] : INFO : Epoch 1839 | loss: 0.0310881 | val_loss: 0.0311348 | Time: 7651.45 ms [2022-03-23 21:35:00 main:574] : INFO : Epoch 1840 | loss: 0.0310841 | val_loss: 0.0311308 | Time: 8332.24 ms [2022-03-23 21:35:17 main:574] : INFO : Epoch 1841 | loss: 0.0310881 | val_loss: 0.0311372 | Time: 17573.8 ms [2022-03-23 21:35:25 main:574] : INFO : Epoch 1842 | loss: 0.0310832 | val_loss: 0.0311468 | Time: 7464.52 ms [2022-03-23 21:35:33 main:574] : INFO : Epoch 1843 | loss: 0.0310807 | val_loss: 0.0311319 | Time: 7740.76 ms [2022-03-23 21:35:41 main:574] : INFO : Epoch 1844 | loss: 0.0310831 | val_loss: 0.0311495 | Time: 7831.04 ms [2022-03-23 21:35:48 main:574] : INFO : Epoch 1845 | loss: 0.0310891 | val_loss: 0.0311343 | Time: 7634.77 ms [2022-03-23 21:35:56 main:574] : INFO : Epoch 1846 | loss: 0.031083 | val_loss: 0.0311378 | Time: 8075.78 ms [2022-03-23 21:36:04 main:574] : INFO : Epoch 1847 | loss: 0.0310817 | val_loss: 0.0311349 | Time: 8006.46 ms [2022-03-23 21:36:12 main:574] : INFO : Epoch 1848 | loss: 0.0310785 | val_loss: 0.0311331 | Time: 8053.27 ms [2022-03-23 21:36:20 main:574] : INFO : Epoch 1849 | loss: 0.0310788 | val_loss: 0.0311442 | Time: 7840.11 ms [2022-03-23 21:36:28 main:574] : INFO : Epoch 1850 | loss: 0.0310891 | val_loss: 0.0311374 | Time: 7705.58 ms [2022-03-23 21:36:35 main:574] : INFO : Epoch 1851 | loss: 0.0310813 | val_loss: 0.0311335 | Time: 7465.86 ms [2022-03-23 21:36:43 main:574] : INFO : Epoch 1852 | loss: 0.0310815 | val_loss: 0.0311356 | Time: 7920.03 ms [2022-03-23 21:36:51 main:574] : INFO : Epoch 1853 | loss: 0.0310793 | val_loss: 0.0311302 | Time: 7698.94 ms [2022-03-23 21:36:59 main:574] : INFO : Epoch 1854 | loss: 0.0310866 | val_loss: 0.0311327 | Time: 8229.31 ms [2022-03-23 21:37:17 main:574] : INFO : Epoch 1855 | loss: 0.0310834 | val_loss: 0.0311308 | Time: 17748.4 ms [2022-03-23 21:37:24 main:574] : INFO : Epoch 1856 | loss: 0.0310812 | val_loss: 0.0311311 | Time: 7160.1 ms [2022-03-23 21:37:32 main:574] : INFO : Epoch 1857 | loss: 0.0310822 | val_loss: 0.0311321 | Time: 7645.87 ms [2022-03-23 21:37:40 main:574] : INFO : Epoch 1858 | loss: 0.0310836 | val_loss: 0.0311281 | Time: 7673.68 ms [2022-03-23 21:37:48 main:574] : INFO : Epoch 1859 | loss: 0.0310811 | val_loss: 0.0311367 | Time: 8010.85 ms [2022-03-23 21:37:56 main:574] : INFO : Epoch 1860 | loss: 0.0310858 | val_loss: 0.0311312 | Time: 7990.27 ms [2022-03-23 21:38:03 main:574] : INFO : Epoch 1861 | loss: 0.031083 | val_loss: 0.031132 | Time: 7651.45 ms [2022-03-23 21:38:11 main:574] : INFO : Epoch 1862 | loss: 0.0310867 | val_loss: 0.0311322 | Time: 8117.48 ms [2022-03-23 21:38:19 main:574] : INFO : Epoch 1863 | loss: 0.0310844 | val_loss: 0.0311309 | Time: 7910.38 ms [2022-03-23 21:38:28 main:574] : INFO : Epoch 1864 | loss: 0.0310812 | val_loss: 0.0311274 | Time: 8508.98 ms [2022-03-23 21:38:35 main:574] : INFO : Epoch 1865 | loss: 0.0310827 | val_loss: 0.0311291 | Time: 7637.16 ms [2022-03-23 21:38:43 main:574] : INFO : Epoch 1866 | loss: 0.031083 | val_loss: 0.0311265 | Time: 7869.86 ms [2022-03-23 21:38:51 main:574] : INFO : Epoch 1867 | loss: 0.0310792 | val_loss: 0.0311479 | Time: 7486.38 ms [2022-03-23 21:38:59 main:574] : INFO : Epoch 1868 | loss: 0.0310902 | val_loss: 0.0311397 | Time: 8006.46 ms [2022-03-23 21:39:07 main:574] : INFO : Epoch 1869 | loss: 0.0310816 | val_loss: 0.0311334 | Time: 8491.45 ms [2022-03-23 21:39:25 main:574] : INFO : Epoch 1870 | loss: 0.0310771 | val_loss: 0.0311371 | Time: 18146.8 ms [2022-03-23 21:39:33 main:574] : INFO : Epoch 1871 | loss: 0.0310792 | val_loss: 0.0311308 | Time: 7576.94 ms [2022-03-23 21:39:41 main:574] : INFO : Epoch 1872 | loss: 0.0310806 | val_loss: 0.0311287 | Time: 7505.25 ms [2022-03-23 21:39:49 main:574] : INFO : Epoch 1873 | loss: 0.0310786 | val_loss: 0.0311272 | Time: 8043.11 ms [2022-03-23 21:39:57 main:574] : INFO : Epoch 1874 | loss: 0.0310786 | val_loss: 0.031135 | Time: 7885.61 ms [2022-03-23 21:40:04 main:574] : INFO : Epoch 1875 | loss: 0.0310772 | val_loss: 0.0311316 | Time: 7542.96 ms [2022-03-23 21:40:12 main:574] : INFO : Epoch 1876 | loss: 0.0310762 | val_loss: 0.0311289 | Time: 7671.08 ms [2022-03-23 21:40:19 main:574] : INFO : Epoch 1877 | loss: 0.0310803 | val_loss: 0.0311383 | Time: 7412.31 ms [2022-03-23 21:40:27 main:574] : INFO : Epoch 1878 | loss: 0.0310798 | val_loss: 0.0311306 | Time: 8118.71 ms [2022-03-23 21:40:35 main:574] : INFO : Epoch 1879 | loss: 0.0310795 | val_loss: 0.0311362 | Time: 7822.4 ms [2022-03-23 21:40:43 main:574] : INFO : Epoch 1880 | loss: 0.031079 | val_loss: 0.0311304 | Time: 7733.61 ms [2022-03-23 21:40:51 main:574] : INFO : Epoch 1881 | loss: 0.0310882 | val_loss: 0.0311355 | Time: 7875.43 ms [2022-03-23 21:40:59 main:574] : INFO : Epoch 1882 | loss: 0.0310835 | val_loss: 0.0311307 | Time: 8463.81 ms [2022-03-23 21:41:07 main:574] : INFO : Epoch 1883 | loss: 0.0310835 | val_loss: 0.0311296 | Time: 7702.99 ms [2022-03-23 21:41:15 main:574] : INFO : Epoch 1884 | loss: 0.0310815 | val_loss: 0.031133 | Time: 7655.02 ms [2022-03-23 21:41:23 main:574] : INFO : Epoch 1885 | loss: 0.0310853 | val_loss: 0.0311349 | Time: 8138.03 ms [2022-03-23 21:41:31 main:574] : INFO : Epoch 1886 | loss: 0.0310824 | val_loss: 0.0311297 | Time: 7910.82 ms [2022-03-23 21:41:38 main:574] : INFO : Epoch 1887 | loss: 0.0310806 | val_loss: 0.0311296 | Time: 7802.4 ms [2022-03-23 21:41:46 main:574] : INFO : Epoch 1888 | loss: 0.0310871 | val_loss: 0.0311326 | Time: 7447.87 ms [2022-03-23 21:41:54 main:574] : INFO : Epoch 1889 | loss: 0.0310856 | val_loss: 0.0311316 | Time: 7635.53 ms [2022-03-23 21:42:01 main:574] : INFO : Epoch 1890 | loss: 0.0310825 | val_loss: 0.0311333 | Time: 7957.97 ms [2022-03-23 21:42:09 main:574] : INFO : Epoch 1891 | loss: 0.0310804 | val_loss: 0.0311321 | Time: 7681.02 ms [2022-03-23 21:42:17 main:574] : INFO : Epoch 1892 | loss: 0.0310782 | val_loss: 0.0311359 | Time: 7604.57 ms [2022-03-23 21:42:25 main:574] : INFO : Epoch 1893 | loss: 0.0310768 | val_loss: 0.0311297 | Time: 7954.25 ms [2022-03-23 21:42:32 main:574] : INFO : Epoch 1894 | loss: 0.0310796 | val_loss: 0.031135 | Time: 7485.62 ms [2022-03-23 21:42:40 main:574] : INFO : Epoch 1895 | loss: 0.0310822 | val_loss: 0.031128 | Time: 7785.78 ms [2022-03-23 21:42:48 main:574] : INFO : Epoch 1896 | loss: 0.0310841 | val_loss: 0.0311288 | Time: 7793.38 ms [2022-03-23 21:42:56 main:574] : INFO : Epoch 1897 | loss: 0.0310836 | val_loss: 0.0311387 | Time: 7956.57 ms [2022-03-23 21:43:04 main:574] : INFO : Epoch 1898 | loss: 0.0310811 | val_loss: 0.0311316 | Time: 7781.39 ms [2022-03-23 21:43:11 main:574] : INFO : Epoch 1899 | loss: 0.0310814 | val_loss: 0.0311419 | Time: 7350.33 ms [2022-03-23 21:43:19 main:574] : INFO : Epoch 1900 | loss: 0.0310914 | val_loss: 0.0311535 | Time: 7686.95 ms [2022-03-23 21:43:27 main:574] : INFO : Epoch 1901 | loss: 0.0310885 | val_loss: 0.0311375 | Time: 8298.4 ms [2022-03-23 21:43:35 main:574] : INFO : Epoch 1902 | loss: 0.0310928 | val_loss: 0.0311551 | Time: 7724.45 ms [2022-03-23 21:43:42 main:574] : INFO : Epoch 1903 | loss: 0.0310847 | val_loss: 0.0311389 | Time: 7579.21 ms [2022-03-23 21:43:50 main:574] : INFO : Epoch 1904 | loss: 0.0310797 | val_loss: 0.031141 | Time: 7549.38 ms [2022-03-23 21:43:58 main:574] : INFO : Epoch 1905 | loss: 0.0310822 | val_loss: 0.0311312 | Time: 7856.96 ms [2022-03-23 21:44:05 main:574] : INFO : Epoch 1906 | loss: 0.0310802 | val_loss: 0.0311348 | Time: 7690.22 ms [2022-03-23 21:44:13 main:574] : INFO : Epoch 1907 | loss: 0.0310878 | val_loss: 0.0311299 | Time: 7797.99 ms [2022-03-23 21:44:21 main:574] : INFO : Epoch 1908 | loss: 0.0310884 | val_loss: 0.0311277 | Time: 7912.4 ms [2022-03-23 21:44:29 main:574] : INFO : Epoch 1909 | loss: 0.0310877 | val_loss: 0.0311279 | Time: 7509.79 ms [2022-03-23 21:44:36 main:574] : INFO : Epoch 1910 | loss: 0.031086 | val_loss: 0.031131 | Time: 7754.2 ms [2022-03-23 21:44:44 main:574] : INFO : Epoch 1911 | loss: 0.0310831 | val_loss: 0.0311292 | Time: 7690.7 ms [2022-03-23 21:44:52 main:574] : INFO : Epoch 1912 | loss: 0.0310855 | val_loss: 0.0311278 | Time: 7635.67 ms [2022-03-23 21:45:00 main:574] : INFO : Epoch 1913 | loss: 0.0310855 | val_loss: 0.0311286 | Time: 8052.45 ms [2022-03-23 21:45:07 main:574] : INFO : Epoch 1914 | loss: 0.0310835 | val_loss: 0.0311276 | Time: 7515.03 ms [2022-03-23 21:45:15 main:574] : INFO : Epoch 1915 | loss: 0.0310853 | val_loss: 0.0311356 | Time: 8055.73 ms [2022-03-23 21:45:23 main:574] : INFO : Epoch 1916 | loss: 0.0310856 | val_loss: 0.0311315 | Time: 8058.91 ms [2022-03-23 21:45:31 main:574] : INFO : Epoch 1917 | loss: 0.0310869 | val_loss: 0.031133 | Time: 8030.8 ms [2022-03-23 21:45:39 main:574] : INFO : Epoch 1918 | loss: 0.0310816 | val_loss: 0.0311272 | Time: 7638.27 ms [2022-03-23 21:45:46 main:574] : INFO : Epoch 1919 | loss: 0.0310806 | val_loss: 0.0311281 | Time: 7274.89 ms [2022-03-23 21:45:54 main:574] : INFO : Epoch 1920 | loss: 0.0310797 | val_loss: 0.0311299 | Time: 7473.29 ms [2022-03-23 21:46:02 main:574] : INFO : Epoch 1921 | loss: 0.0310817 | val_loss: 0.0311319 | Time: 8274.8 ms [2022-03-23 21:46:10 main:574] : INFO : Epoch 1922 | loss: 0.0310861 | val_loss: 0.0311314 | Time: 7703.35 ms [2022-03-23 21:46:18 main:574] : INFO : Epoch 1923 | loss: 0.0310813 | val_loss: 0.0311333 | Time: 8027.12 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1660 Ti with Max-Q Design) [2022-03-24 00:48:33 main:435] : INFO : Set logging level to 1 [2022-03-24 00:48:33 main:441] : INFO : Running in BOINC Client mode [2022-03-24 00:48:33 main:444] : INFO : Resolving all filenames [2022-03-24 00:48:33 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-24 00:48:33 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-24 00:48:33 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-24 00:48:34 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-03-24 00:48:34 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-24 00:48:34 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-24 00:48:34 main:474] : INFO : Configuration: [2022-03-24 00:48:34 main:475] : INFO : Model type: GRU [2022-03-24 00:48:34 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-24 00:48:34 main:477] : INFO : Max Epochs: 2048 [2022-03-24 00:48:34 main:478] : INFO : Batch Size: 128 [2022-03-24 00:48:34 main:479] : INFO : Learning Rate: 0.01 [2022-03-24 00:48:34 main:480] : INFO : Patience: 10 [2022-03-24 00:48:34 main:481] : INFO : Hidden Width: 12 [2022-03-24 00:48:34 main:482] : INFO : # Recurrent Layers: 4 [2022-03-24 00:48:34 main:483] : INFO : # Backend Layers: 4 [2022-03-24 00:48:34 main:484] : INFO : # Threads: 1 [2022-03-24 00:48:34 main:486] : INFO : Preparing Dataset [2022-03-24 00:48:34 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-24 00:48:34 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-24 00:48:42 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-24 00:48:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-24 00:48:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-24 00:48:42 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-24 00:48:42 main:494] : INFO : Creating Model [2022-03-24 00:48:42 main:507] : INFO : Preparing config file [2022-03-24 00:48:42 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-24 00:48:42 main:512] : INFO : Loading config [2022-03-24 00:48:42 main:514] : INFO : Loading state [2022-03-24 00:48:49 main:559] : INFO : Loading DataLoader into Memory [2022-03-24 00:48:49 main:562] : INFO : Starting Training [2022-03-24 00:49:15 main:574] : INFO : Epoch 1919 | loss: 0.0312918 | val_loss: 0.0311485 | Time: 25298.6 ms [2022-03-24 00:49:23 main:574] : INFO : Epoch 1920 | loss: 0.0311123 | val_loss: 0.031144 | Time: 8318.68 ms [2022-03-24 00:49:31 main:574] : INFO : Epoch 1921 | loss: 0.0310904 | val_loss: 0.0311325 | Time: 8071.23 ms [2022-03-24 00:49:39 main:574] : INFO : Epoch 1922 | loss: 0.0310899 | val_loss: 0.0311251 | Time: 7969.03 ms [2022-03-24 00:49:47 main:574] : INFO : Epoch 1923 | loss: 0.0310838 | val_loss: 0.0311245 | Time: 7960.6 ms [2022-03-24 00:49:54 main:574] : INFO : Epoch 1924 | loss: 0.0310838 | val_loss: 0.0311355 | Time: 7510.44 ms [2022-03-24 00:50:03 main:574] : INFO : Epoch 1925 | loss: 0.0310869 | val_loss: 0.031127 | Time: 8055.56 ms [2022-03-24 00:50:10 main:574] : INFO : Epoch 1926 | loss: 0.0310853 | val_loss: 0.0311296 | Time: 7578.9 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1660 Ti with Max-Q Design) [2022-03-24 03:51:42 main:435] : INFO : Set logging level to 1 [2022-03-24 03:51:42 main:441] : INFO : Running in BOINC Client mode [2022-03-24 03:51:42 main:444] : INFO : Resolving all filenames [2022-03-24 03:51:42 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-24 03:51:42 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-24 03:51:42 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-24 03:51:42 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-03-24 03:51:42 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-24 03:51:42 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-24 03:51:42 main:474] : INFO : Configuration: [2022-03-24 03:51:42 main:475] : INFO : Model type: GRU [2022-03-24 03:51:42 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-24 03:51:42 main:477] : INFO : Max Epochs: 2048 [2022-03-24 03:51:42 main:478] : INFO : Batch Size: 128 [2022-03-24 03:51:42 main:479] : INFO : Learning Rate: 0.01 [2022-03-24 03:51:42 main:480] : INFO : Patience: 10 [2022-03-24 03:51:42 main:481] : INFO : Hidden Width: 12 [2022-03-24 03:51:42 main:482] : INFO : # Recurrent Layers: 4 [2022-03-24 03:51:42 main:483] : INFO : # Backend Layers: 4 [2022-03-24 03:51:42 main:484] : INFO : # Threads: 1 [2022-03-24 03:51:42 main:486] : INFO : Preparing Dataset [2022-03-24 03:51:43 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-24 03:51:46 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-24 03:52:01 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-24 03:52:01 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-24 03:52:01 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-24 03:52:01 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-24 03:52:01 main:494] : INFO : Creating Model [2022-03-24 03:52:01 main:507] : INFO : Preparing config file [2022-03-24 03:52:01 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-24 03:52:01 main:512] : INFO : Loading config [2022-03-24 03:52:01 main:514] : INFO : Loading state [2022-03-24 03:52:05 main:559] : INFO : Loading DataLoader into Memory [2022-03-24 03:52:05 main:562] : INFO : Starting Training [2022-03-24 03:52:16 main:574] : INFO : Epoch 1919 | loss: 0.0313179 | val_loss: 0.0311692 | Time: 10507.6 ms [2022-03-24 03:52:34 main:574] : INFO : Epoch 1920 | loss: 0.0311186 | val_loss: 0.0311419 | Time: 17884.9 ms [2022-03-24 03:52:42 main:574] : INFO : Epoch 1921 | loss: 0.0310902 | val_loss: 0.0311294 | Time: 8307.1 ms [2022-03-24 03:52:50 main:574] : INFO : Epoch 1922 | loss: 0.0310841 | val_loss: 0.031129 | Time: 8124.72 ms [2022-03-24 03:52:58 main:574] : INFO : Epoch 1923 | loss: 0.0310808 | val_loss: 0.0311282 | Time: 8081.15 ms [2022-03-24 03:53:06 main:574] : INFO : Epoch 1924 | loss: 0.0310816 | val_loss: 0.0311274 | Time: 8210.11 ms [2022-03-24 03:53:14 main:574] : INFO : Epoch 1925 | loss: 0.0310853 | val_loss: 0.0311323 | Time: 8062.02 ms [2022-03-24 03:53:23 main:574] : INFO : Epoch 1926 | loss: 0.031087 | val_loss: 0.0311315 | Time: 8577.33 ms [2022-03-24 03:53:41 main:574] : INFO : Epoch 1927 | loss: 0.0310818 | val_loss: 0.0311293 | Time: 17941.7 ms [2022-03-24 03:53:49 main:574] : INFO : Epoch 1928 | loss: 0.0310771 | val_loss: 0.0311281 | Time: 7734.27 ms [2022-03-24 03:53:57 main:574] : INFO : Epoch 1929 | loss: 0.0310791 | val_loss: 0.0311337 | Time: 8324.85 ms [2022-03-24 03:54:05 main:574] : INFO : Epoch 1930 | loss: 0.0310765 | val_loss: 0.0311348 | Time: 7976.09 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1660 Ti with Max-Q Design) [2022-03-24 06:55:39 main:435] : INFO : Set logging level to 1 [2022-03-24 06:55:39 main:441] : INFO : Running in BOINC Client mode [2022-03-24 06:55:39 main:444] : INFO : Resolving all filenames [2022-03-24 06:55:39 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-24 06:55:39 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-24 06:55:39 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-24 06:55:39 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-03-24 06:55:39 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-24 06:55:39 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-24 06:55:39 main:474] : INFO : Configuration: [2022-03-24 06:55:39 main:475] : INFO : Model type: GRU [2022-03-24 06:55:39 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-24 06:55:39 main:477] : INFO : Max Epochs: 2048 [2022-03-24 06:55:39 main:478] : INFO : Batch Size: 128 [2022-03-24 06:55:39 main:479] : INFO : Learning Rate: 0.01 [2022-03-24 06:55:39 main:480] : INFO : Patience: 10 [2022-03-24 06:55:39 main:481] : INFO : Hidden Width: 12 [2022-03-24 06:55:39 main:482] : INFO : # Recurrent Layers: 4 [2022-03-24 06:55:39 main:483] : INFO : # Backend Layers: 4 [2022-03-24 06:55:39 main:484] : INFO : # Threads: 1 [2022-03-24 06:55:39 main:486] : INFO : Preparing Dataset [2022-03-24 06:55:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-24 06:55:40 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-24 06:55:55 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-24 06:55:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-24 06:55:56 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-24 06:55:56 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-24 06:55:56 main:494] : INFO : Creating Model [2022-03-24 06:55:56 main:507] : INFO : Preparing config file [2022-03-24 06:55:56 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-24 06:55:56 main:512] : INFO : Loading config [2022-03-24 06:55:56 main:514] : INFO : Loading state [2022-03-24 06:56:00 main:559] : INFO : Loading DataLoader into Memory [2022-03-24 06:56:00 main:562] : INFO : Starting Training [2022-03-24 06:56:17 main:574] : INFO : Epoch 1919 | loss: 0.0313019 | val_loss: 0.0311544 | Time: 17422.1 ms [2022-03-24 06:56:36 main:574] : INFO : Epoch 1920 | loss: 0.0311194 | val_loss: 0.0311531 | Time: 18277.5 ms [2022-03-24 06:56:43 main:574] : INFO : Epoch 1921 | loss: 0.0310925 | val_loss: 0.0311286 | Time: 7633.17 ms [2022-03-24 06:56:53 main:574] : INFO : Epoch 1922 | loss: 0.0310875 | val_loss: 0.0311334 | Time: 9381.06 ms [2022-03-24 06:57:10 main:574] : INFO : Epoch 1923 | loss: 0.0310832 | val_loss: 0.0311316 | Time: 17368.6 ms [2022-03-24 06:57:19 main:574] : INFO : Epoch 1924 | loss: 0.0310791 | val_loss: 0.0311317 | Time: 8664.56 ms [2022-03-24 06:57:26 main:574] : INFO : Epoch 1925 | loss: 0.0310824 | val_loss: 0.0311332 | Time: 7513.83 ms [2022-03-24 06:57:34 main:574] : INFO : Epoch 1926 | loss: 0.0310808 | val_loss: 0.0311326 | Time: 7939.22 ms [2022-03-24 06:57:42 main:574] : INFO : Epoch 1927 | loss: 0.031084 | val_loss: 0.0311327 | Time: 7471.41 ms [2022-03-24 06:57:49 main:574] : INFO : Epoch 1928 | loss: 0.0310852 | val_loss: 0.0311355 | Time: 7788.71 ms [2022-03-24 06:57:58 main:574] : INFO : Epoch 1929 | loss: 0.0310825 | val_loss: 0.0311318 | Time: 8349.81 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1660 Ti with Max-Q Design) [2022-03-24 19:08:11 main:435] : INFO : Set logging level to 1 [2022-03-24 19:08:11 main:441] : INFO : Running in BOINC Client mode [2022-03-24 19:08:11 main:444] : INFO : Resolving all filenames [2022-03-24 19:08:11 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-24 19:08:11 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-24 19:08:11 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-24 19:08:11 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-03-24 19:08:11 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-24 19:08:11 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-24 19:08:11 main:474] : INFO : Configuration: [2022-03-24 19:08:11 main:475] : INFO : Model type: GRU [2022-03-24 19:08:11 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-24 19:08:11 main:477] : INFO : Max Epochs: 2048 [2022-03-24 19:08:11 main:478] : INFO : Batch Size: 128 [2022-03-24 19:08:11 main:479] : INFO : Learning Rate: 0.01 [2022-03-24 19:08:11 main:480] : INFO : Patience: 10 [2022-03-24 19:08:11 main:481] : INFO : Hidden Width: 12 [2022-03-24 19:08:11 main:482] : INFO : # Recurrent Layers: 4 [2022-03-24 19:08:11 main:483] : INFO : # Backend Layers: 4 [2022-03-24 19:08:11 main:484] : INFO : # Threads: 1 [2022-03-24 19:08:11 main:486] : INFO : Preparing Dataset [2022-03-24 19:08:11 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-24 19:08:11 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-24 19:08:18 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-24 19:08:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-24 19:08:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-24 19:08:19 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-24 19:08:19 main:494] : INFO : Creating Model [2022-03-24 19:08:19 main:507] : INFO : Preparing config file [2022-03-24 19:08:19 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-24 19:08:19 main:512] : INFO : Loading config [2022-03-24 19:08:19 main:514] : INFO : Loading state [2022-03-24 19:08:32 main:559] : INFO : Loading DataLoader into Memory [2022-03-24 19:08:32 main:562] : INFO : Starting Training [2022-03-24 19:08:38 main:574] : INFO : Epoch 1919 | loss: 0.0313087 | val_loss: 0.0311451 | Time: 5909.5 ms [2022-03-24 19:08:43 main:574] : INFO : Epoch 1920 | loss: 0.0311038 | val_loss: 0.0311376 | Time: 5171.55 ms [2022-03-24 19:08:48 main:574] : INFO : Epoch 1921 | loss: 0.0310852 | val_loss: 0.0311239 | Time: 4945.79 ms [2022-03-24 19:08:53 main:574] : INFO : Epoch 1922 | loss: 0.0310807 | val_loss: 0.0311268 | Time: 5080.18 ms [2022-03-24 19:08:58 main:574] : INFO : Epoch 1923 | loss: 0.03108 | val_loss: 0.0311283 | Time: 4962.57 ms [2022-03-24 19:09:04 main:574] : INFO : Epoch 1924 | loss: 0.0310783 | val_loss: 0.0311306 | Time: 5339.32 ms [2022-03-24 19:09:09 main:574] : INFO : Epoch 1925 | loss: 0.0310775 | val_loss: 0.0311283 | Time: 4835.16 ms [2022-03-24 19:09:14 main:574] : INFO : Epoch 1926 | loss: 0.0310755 | val_loss: 0.0311279 | Time: 5079.85 ms [2022-03-24 19:09:18 main:574] : INFO : Epoch 1927 | loss: 0.0310781 | val_loss: 0.0311263 | Time: 4862.65 ms [2022-03-24 19:09:24 main:574] : INFO : Epoch 1928 | loss: 0.0310824 | val_loss: 0.0311299 | Time: 5232.28 ms [2022-03-24 19:09:28 main:574] : INFO : Epoch 1929 | loss: 0.0310797 | val_loss: 0.031134 | Time: 4746.72 ms [2022-03-24 19:09:34 main:574] : INFO : Epoch 1930 | loss: 0.0310814 | val_loss: 0.0311344 | Time: 5114.89 ms [2022-03-24 19:09:38 main:574] : INFO : Epoch 1931 | loss: 0.0310794 | val_loss: 0.0311298 | Time: 4856.9 ms [2022-03-24 19:09:44 main:574] : INFO : Epoch 1932 | loss: 0.0310786 | val_loss: 0.0311373 | Time: 5165.48 ms [2022-03-24 19:09:48 main:574] : INFO : Epoch 1933 | loss: 0.0310819 | val_loss: 0.0311415 | Time: 4828.32 ms [2022-03-24 19:09:54 main:574] : INFO : Epoch 1934 | loss: 0.0310791 | val_loss: 0.0311332 | Time: 5107.35 ms [2022-03-24 19:09:58 main:574] : INFO : Epoch 1935 | loss: 0.0310806 | val_loss: 0.0311331 | Time: 4793.27 ms [2022-03-24 19:10:03 main:574] : INFO : Epoch 1936 | loss: 0.0310819 | val_loss: 0.031138 | Time: 5063.58 ms [2022-03-24 19:10:08 main:574] : INFO : Epoch 1937 | loss: 0.0310804 | val_loss: 0.0311349 | Time: 4750.25 ms [2022-03-24 19:10:13 main:574] : INFO : Epoch 1938 | loss: 0.0310766 | val_loss: 0.0311316 | Time: 4986.38 ms [2022-03-24 19:10:18 main:574] : INFO : Epoch 1939 | loss: 0.0310777 | val_loss: 0.0311317 | Time: 4768.12 ms [2022-03-24 19:10:23 main:574] : INFO : Epoch 1940 | loss: 0.0310807 | val_loss: 0.0311317 | Time: 5385.12 ms [2022-03-24 19:10:28 main:574] : INFO : Epoch 1941 | loss: 0.0310754 | val_loss: 0.0311349 | Time: 4752.15 ms [2022-03-24 19:10:33 main:574] : INFO : Epoch 1942 | loss: 0.0310752 | val_loss: 0.0311302 | Time: 5102.4 ms [2022-03-24 19:10:38 main:574] : INFO : Epoch 1943 | loss: 0.0310738 | val_loss: 0.0311466 | Time: 4824.96 ms [2022-03-24 19:10:43 main:574] : INFO : Epoch 1944 | loss: 0.0310801 | val_loss: 0.0311311 | Time: 5078.28 ms [2022-03-24 19:10:48 main:574] : INFO : Epoch 1945 | loss: 0.0310782 | val_loss: 0.03113 | Time: 4778.97 ms [2022-03-24 19:10:53 main:574] : INFO : Epoch 1946 | loss: 0.0310829 | val_loss: 0.031141 | Time: 5134.93 ms [2022-03-24 19:10:58 main:574] : INFO : Epoch 1947 | loss: 0.0310782 | val_loss: 0.0311252 | Time: 4782 ms [2022-03-24 19:11:03 main:574] : INFO : Epoch 1948 | loss: 0.0310997 | val_loss: 0.0311314 | Time: 5092.03 ms [2022-03-24 19:11:08 main:574] : INFO : Epoch 1949 | loss: 0.0311041 | val_loss: 0.0311361 | Time: 4819.51 ms [2022-03-24 19:11:13 main:574] : INFO : Epoch 1950 | loss: 0.031097 | val_loss: 0.0311296 | Time: 4952.57 ms [2022-03-24 19:11:18 main:574] : INFO : Epoch 1951 | loss: 0.0310907 | val_loss: 0.0311257 | Time: 4827.74 ms [2022-03-24 19:11:23 main:574] : INFO : Epoch 1952 | loss: 0.0310879 | val_loss: 0.0311252 | Time: 5103.14 ms [2022-03-24 19:11:27 main:574] : INFO : Epoch 1953 | loss: 0.0310847 | val_loss: 0.0311239 | Time: 4743.37 ms [2022-03-24 19:11:32 main:574] : INFO : Epoch 1954 | loss: 0.0310814 | val_loss: 0.0311287 | Time: 4937.6 ms [2022-03-24 19:11:37 main:574] : INFO : Epoch 1955 | loss: 0.0310835 | val_loss: 0.0311313 | Time: 4771.15 ms [2022-03-24 19:11:42 main:574] : INFO : Epoch 1956 | loss: 0.031084 | val_loss: 0.0311304 | Time: 5004.05 ms [2022-03-24 19:11:47 main:574] : INFO : Epoch 1957 | loss: 0.0310852 | val_loss: 0.0311331 | Time: 4757.33 ms [2022-03-24 19:11:52 main:574] : INFO : Epoch 1958 | loss: 0.0310915 | val_loss: 0.0311317 | Time: 5111.13 ms [2022-03-24 19:11:57 main:574] : INFO : Epoch 1959 | loss: 0.0310821 | val_loss: 0.0311298 | Time: 4833.21 ms [2022-03-24 19:12:02 main:574] : INFO : Epoch 1960 | loss: 0.0310829 | val_loss: 0.031131 | Time: 5104.11 ms [2022-03-24 19:12:07 main:574] : INFO : Epoch 1961 | loss: 0.0310821 | val_loss: 0.0311261 | Time: 4737.02 ms [2022-03-24 19:12:12 main:574] : INFO : Epoch 1962 | loss: 0.0310797 | val_loss: 0.0311253 | Time: 5096.55 ms [2022-03-24 19:12:17 main:574] : INFO : Epoch 1963 | loss: 0.0310772 | val_loss: 0.031132 | Time: 4849.59 ms [2022-03-24 19:12:22 main:574] : INFO : Epoch 1964 | loss: 0.0310836 | val_loss: 0.0311258 | Time: 5154.76 ms [2022-03-24 19:12:27 main:574] : INFO : Epoch 1965 | loss: 0.0310807 | val_loss: 0.0311272 | Time: 4874.95 ms [2022-03-24 19:12:32 main:574] : INFO : Epoch 1966 | loss: 0.0310782 | val_loss: 0.0311301 | Time: 5103.18 ms [2022-03-24 19:12:37 main:574] : INFO : Epoch 1967 | loss: 0.0310777 | val_loss: 0.0311331 | Time: 4855.54 ms [2022-03-24 19:12:42 main:574] : INFO : Epoch 1968 | loss: 0.0311395 | val_loss: 0.03115 | Time: 5135.78 ms [2022-03-24 19:12:47 main:574] : INFO : Epoch 1969 | loss: 0.0311453 | val_loss: 0.031142 | Time: 4827.9 ms [2022-03-24 19:12:52 main:574] : INFO : Epoch 1970 | loss: 0.0311421 | val_loss: 0.0311439 | Time: 4963.9 ms [2022-03-24 19:12:56 main:574] : INFO : Epoch 1971 | loss: 0.031149 | val_loss: 0.0311416 | Time: 4710.13 ms [2022-03-24 19:13:01 main:574] : INFO : Epoch 1972 | loss: 0.0311369 | val_loss: 0.0311374 | Time: 5132.42 ms [2022-03-24 19:13:06 main:574] : INFO : Epoch 1973 | loss: 0.0311303 | val_loss: 0.0311308 | Time: 4802.23 ms [2022-03-24 19:13:12 main:574] : INFO : Epoch 1974 | loss: 0.0311237 | val_loss: 0.0311311 | Time: 5301.04 ms [2022-03-24 19:13:16 main:574] : INFO : Epoch 1975 | loss: 0.0311233 | val_loss: 0.0311255 | Time: 4695.81 ms [2022-03-24 19:13:21 main:574] : INFO : Epoch 1976 | loss: 0.0311184 | val_loss: 0.0311269 | Time: 5057.38 ms [2022-03-24 19:13:26 main:574] : INFO : Epoch 1977 | loss: 0.0311209 | val_loss: 0.0311326 | Time: 4934.22 ms [2022-03-24 19:13:31 main:574] : INFO : Epoch 1978 | loss: 0.0311215 | val_loss: 0.0311279 | Time: 5053.34 ms [2022-03-24 19:13:36 main:574] : INFO : Epoch 1979 | loss: 0.0311131 | val_loss: 0.0311269 | Time: 4792.09 ms [2022-03-24 19:13:41 main:574] : INFO : Epoch 1980 | loss: 0.0311117 | val_loss: 0.0311266 | Time: 5149.43 ms [2022-03-24 19:13:46 main:574] : INFO : Epoch 1981 | loss: 0.0311106 | val_loss: 0.0311224 | Time: 4829.21 ms [2022-03-24 19:13:51 main:574] : INFO : Epoch 1982 | loss: 0.0311095 | val_loss: 0.0311282 | Time: 5045.81 ms [2022-03-24 19:13:56 main:574] : INFO : Epoch 1983 | loss: 0.0311067 | val_loss: 0.0311362 | Time: 4799.31 ms [2022-03-24 19:14:01 main:574] : INFO : Epoch 1984 | loss: 0.0311063 | val_loss: 0.0311279 | Time: 5272.2 ms [2022-03-24 19:14:06 main:574] : INFO : Epoch 1985 | loss: 0.0311043 | val_loss: 0.0311326 | Time: 4786.29 ms [2022-03-24 19:14:11 main:574] : INFO : Epoch 1986 | loss: 0.0311025 | val_loss: 0.0311377 | Time: 5034.28 ms [2022-03-24 19:14:16 main:574] : INFO : Epoch 1987 | loss: 0.0311023 | val_loss: 0.03113 | Time: 4763.59 ms [2022-03-24 19:14:21 main:574] : INFO : Epoch 1988 | loss: 0.0311002 | val_loss: 0.0311312 | Time: 5024.21 ms [2022-03-24 19:14:26 main:574] : INFO : Epoch 1989 | loss: 0.0311012 | val_loss: 0.0311291 | Time: 5066.46 ms [2022-03-24 19:14:31 main:574] : INFO : Epoch 1990 | loss: 0.0310984 | val_loss: 0.0311291 | Time: 4973.61 ms [2022-03-24 19:14:36 main:574] : INFO : Epoch 1991 | loss: 0.0311042 | val_loss: 0.031131 | Time: 4985.54 ms [2022-03-24 19:14:41 main:574] : INFO : Epoch 1992 | loss: 0.0311034 | val_loss: 0.0311313 | Time: 5393.05 ms [2022-03-24 19:14:46 main:574] : INFO : Epoch 1993 | loss: 0.0310986 | val_loss: 0.0311333 | Time: 4785.28 ms [2022-03-24 19:14:51 main:574] : INFO : Epoch 1994 | loss: 0.0310995 | val_loss: 0.0311339 | Time: 5024.92 ms [2022-03-24 19:14:56 main:574] : INFO : Epoch 1995 | loss: 0.0310993 | val_loss: 0.0311286 | Time: 4858.38 ms [2022-03-24 19:15:01 main:574] : INFO : Epoch 1996 | loss: 0.0310948 | val_loss: 0.0311322 | Time: 5184.84 ms [2022-03-24 19:15:06 main:574] : INFO : Epoch 1997 | loss: 0.0310935 | val_loss: 0.0311253 | Time: 4766.76 ms [2022-03-24 19:15:11 main:574] : INFO : Epoch 1998 | loss: 0.0310906 | val_loss: 0.0311226 | Time: 5085.07 ms [2022-03-24 19:15:16 main:574] : INFO : Epoch 1999 | loss: 0.0310918 | val_loss: 0.0311226 | Time: 4575.91 ms [2022-03-24 19:15:21 main:574] : INFO : Epoch 2000 | loss: 0.0310936 | val_loss: 0.0311268 | Time: 5139.26 ms [2022-03-24 19:15:26 main:574] : INFO : Epoch 2001 | loss: nan | val_loss: nan | Time: 4781.52 ms [2022-03-24 19:15:31 main:574] : INFO : Epoch 2002 | loss: nan | val_loss: nan | Time: 5001.26 ms [2022-03-24 19:15:35 main:574] : INFO : Epoch 2003 | loss: nan | val_loss: nan | Time: 4793.29 ms [2022-03-24 19:15:40 main:574] : INFO : Epoch 2004 | loss: nan | val_loss: nan | Time: 5069.53 ms [2022-03-24 19:15:45 main:574] : INFO : Epoch 2005 | loss: nan | val_loss: nan | Time: 4853.28 ms [2022-03-24 19:15:50 main:574] : INFO : Epoch 2006 | loss: nan | val_loss: nan | Time: 5004.59 ms [2022-03-24 19:15:55 main:574] : INFO : Epoch 2007 | loss: nan | val_loss: nan | Time: 4786.9 ms [2022-03-24 19:16:00 main:574] : INFO : Epoch 2008 | loss: nan | val_loss: nan | Time: 5112.7 ms [2022-03-24 19:16:05 main:574] : INFO : Epoch 2009 | loss: nan | val_loss: nan | Time: 4816.72 ms [2022-03-24 19:16:10 main:574] : INFO : Epoch 2010 | loss: nan | val_loss: nan | Time: 4926.88 ms [2022-03-24 19:16:15 main:574] : INFO : Epoch 2011 | loss: nan | val_loss: nan | Time: 4915.1 ms [2022-03-24 19:16:20 main:574] : INFO : Epoch 2012 | loss: nan | val_loss: nan | Time: 4971.32 ms [2022-03-24 19:16:25 main:574] : INFO : Epoch 2013 | loss: nan | val_loss: nan | Time: 4828.91 ms [2022-03-24 19:16:30 main:574] : INFO : Epoch 2014 | loss: nan | val_loss: nan | Time: 5120.65 ms [2022-03-24 19:16:35 main:574] : INFO : Epoch 2015 | loss: nan | val_loss: nan | Time: 4823.53 ms [2022-03-24 19:16:40 main:574] : INFO : Epoch 2016 | loss: nan | val_loss: nan | Time: 5047.73 ms [2022-03-24 19:16:45 main:574] : INFO : Epoch 2017 | loss: nan | val_loss: nan | Time: 4999.05 ms [2022-03-24 19:16:50 main:574] : INFO : Epoch 2018 | loss: nan | val_loss: nan | Time: 4945.16 ms [2022-03-24 19:16:55 main:574] : INFO : Epoch 2019 | loss: nan | val_loss: nan | Time: 5247.96 ms [2022-03-24 19:17:00 main:574] : INFO : Epoch 2020 | loss: nan | val_loss: nan | Time: 5085.04 ms [2022-03-24 19:17:05 main:574] : INFO : Epoch 2021 | loss: nan | val_loss: nan | Time: 4954.2 ms [2022-03-24 19:17:10 main:574] : INFO : Epoch 2022 | loss: nan | val_loss: nan | Time: 4951.57 ms [2022-03-24 19:17:15 main:574] : INFO : Epoch 2023 | loss: nan | val_loss: nan | Time: 4922.1 ms [2022-03-24 19:17:20 main:574] : INFO : Epoch 2024 | loss: nan | val_loss: nan | Time: 5218.79 ms [2022-03-24 19:17:25 main:574] : INFO : Epoch 2025 | loss: nan | val_loss: nan | Time: 4960.12 ms [2022-03-24 19:17:30 main:574] : INFO : Epoch 2026 | loss: nan | val_loss: nan | Time: 5016.07 ms [2022-03-24 19:17:35 main:574] : INFO : Epoch 2027 | loss: nan | val_loss: nan | Time: 4858.69 ms [2022-03-24 19:17:40 main:574] : INFO : Epoch 2028 | loss: nan | val_loss: nan | Time: 5056.51 ms [2022-03-24 19:17:45 main:574] : INFO : Epoch 2029 | loss: nan | val_loss: nan | Time: 4912.96 ms [2022-03-24 19:17:50 main:574] : INFO : Epoch 2030 | loss: nan | val_loss: nan | Time: 5014.26 ms [2022-03-24 19:17:55 main:574] : INFO : Epoch 2031 | loss: nan | val_loss: nan | Time: 4927.03 ms [2022-03-24 19:18:00 main:574] : INFO : Epoch 2032 | loss: nan | val_loss: nan | Time: 4910.44 ms [2022-03-24 19:18:05 main:574] : INFO : Epoch 2033 | loss: nan | val_loss: nan | Time: 5086.66 ms [2022-03-24 19:18:10 main:574] : INFO : Epoch 2034 | loss: nan | val_loss: nan | Time: 4913.57 ms [2022-03-24 19:18:15 main:574] : INFO : Epoch 2035 | loss: nan | val_loss: nan | Time: 4896.38 ms [2022-03-24 19:18:20 main:574] : INFO : Epoch 2036 | loss: nan | val_loss: nan | Time: 4985.3 ms [2022-03-24 19:18:25 main:574] : INFO : Epoch 2037 | loss: nan | val_loss: nan | Time: 4960.08 ms [2022-03-24 19:18:30 main:574] : INFO : Epoch 2038 | loss: nan | val_loss: nan | Time: 4993.87 ms [2022-03-24 19:18:35 main:574] : INFO : Epoch 2039 | loss: nan | val_loss: nan | Time: 5071.21 ms [2022-03-24 19:18:40 main:574] : INFO : Epoch 2040 | loss: nan | val_loss: nan | Time: 4720.51 ms [2022-03-24 19:18:44 main:574] : INFO : Epoch 2041 | loss: nan | val_loss: nan | Time: 4925.83 ms [2022-03-24 19:18:53 main:574] : INFO : Epoch 2042 | loss: nan | val_loss: nan | Time: 8125.78 ms [2022-03-24 19:19:02 main:574] : INFO : Epoch 2043 | loss: nan | val_loss: nan | Time: 9042.06 ms [2022-03-24 19:19:10 main:574] : INFO : Epoch 2044 | loss: nan | val_loss: nan | Time: 8403.06 ms [2022-03-24 19:19:18 main:574] : INFO : Epoch 2045 | loss: nan | val_loss: nan | Time: 8209.94 ms [2022-03-24 19:19:27 main:574] : INFO : Epoch 2046 | loss: nan | val_loss: nan | Time: 8453.35 ms [2022-03-24 19:19:35 main:574] : INFO : Epoch 2047 | loss: nan | val_loss: nan | Time: 8214.26 ms [2022-03-24 19:19:43 main:574] : INFO : Epoch 2048 | loss: nan | val_loss: nan | Time: 8280.99 ms [2022-03-24 19:19:43 main:597] : INFO : Saving trained model to model-final.pt, val_loss nan [2022-03-24 19:19:43 main:603] : INFO : Saving end state to config to file [2022-03-24 19:19:43 main:608] : INFO : Success, exiting.. 19:19:43 (34676): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)