Task 12943835
| Name | ParityModified-1645997731-23347-3-0_0 |
| Workunit | 10076751 |
| Created | 11 Mar 2022, 19:58:29 UTC |
| Sent | 11 Mar 2022, 20:01:16 UTC |
| Report deadline | 19 Mar 2022, 20:01:16 UTC |
| Received | 13 Mar 2022, 7:21:43 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 13173 |
| Run time | 3 hours 47 min 19 sec |
| CPU time | 3 hours 35 min 20 sec |
| Validate state | Valid |
| Credit | 4,160.00 |
| Device peak FLOPS | 1,168.88 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.55 GB |
| Peak swap size | 3.58 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version> <![CDATA[ <stderr_txt> s [2022-03-13 11:11:10 main:574] : INFO : Epoch 1655 | loss: 0.0311458 | val_loss: 0.0311638 | Time: 10806.2 ms [2022-03-13 11:11:19 main:574] : INFO : Epoch 1656 | loss: 0.0311449 | val_loss: 0.031162 | Time: 7171.84 ms [2022-03-13 11:11:25 main:574] : INFO : Epoch 1657 | loss: 0.0311452 | val_loss: 0.0311633 | Time: 6527.04 ms [2022-03-13 11:11:32 main:574] : INFO : Epoch 1658 | loss: 0.0311438 | val_loss: 0.0311637 | Time: 6539.17 ms [2022-03-13 11:11:38 main:574] : INFO : Epoch 1659 | loss: 0.0311466 | val_loss: 0.0311706 | Time: 6498.49 ms [2022-03-13 11:11:45 main:574] : INFO : Epoch 1660 | loss: 0.0311463 | val_loss: 0.0311642 | Time: 6532.7 ms [2022-03-13 11:11:51 main:574] : INFO : Epoch 1661 | loss: 0.0311441 | val_loss: 0.0311661 | Time: 6519.18 ms [2022-03-13 11:11:58 main:574] : INFO : Epoch 1662 | loss: 0.0311455 | val_loss: 0.0311635 | Time: 6497.64 ms [2022-03-13 11:12:04 main:574] : INFO : Epoch 1663 | loss: 0.0311462 | val_loss: 0.0311645 | Time: 6527.93 ms [2022-03-13 11:12:11 main:574] : INFO : Epoch 1664 | loss: 0.031147 | val_loss: 0.0311663 | Time: 6524 ms [2022-03-13 11:12:17 main:574] : INFO : Epoch 1665 | loss: 0.0311496 | val_loss: 0.0311614 | Time: 6529.49 ms [2022-03-13 11:12:24 main:574] : INFO : Epoch 1666 | loss: 0.0311478 | val_loss: 0.0311647 | Time: 6490.82 ms [2022-03-13 11:12:30 main:574] : INFO : Epoch 1667 | loss: 0.0311459 | val_loss: 0.0311637 | Time: 6502.89 ms [2022-03-13 11:12:37 main:574] : INFO : Epoch 1668 | loss: 0.0311458 | val_loss: 0.0311631 | Time: 6516.54 ms [2022-03-13 11:12:43 main:574] : INFO : Epoch 1669 | loss: 0.0311452 | val_loss: 0.0311626 | Time: 6507.67 ms [2022-03-13 11:12:50 main:574] : INFO : Epoch 1670 | loss: 0.031152 | val_loss: 0.0311689 | Time: 6533.57 ms [2022-03-13 11:12:56 main:574] : INFO : Epoch 1671 | loss: 0.0311513 | val_loss: 0.031163 | Time: 6507.43 ms [2022-03-13 11:13:03 main:574] : INFO : Epoch 1672 | loss: 0.0311493 | val_loss: 0.0311625 | Time: 6516.75 ms [2022-03-13 11:13:09 main:574] : INFO : Epoch 1673 | loss: 0.0311517 | val_loss: 0.0311697 | Time: 6508.82 ms [2022-03-13 11:13:16 main:574] : INFO : Epoch 1674 | loss: 0.0311492 | val_loss: 0.0311686 | Time: 6508.55 ms [2022-03-13 11:13:23 main:574] : INFO : Epoch 1675 | loss: 0.0311511 | val_loss: 0.0311667 | Time: 6521.49 ms [2022-03-13 11:13:29 main:574] : INFO : Epoch 1676 | loss: 0.0311496 | val_loss: 0.0311685 | Time: 6494.7 ms [2022-03-13 11:13:36 main:574] : INFO : Epoch 1677 | loss: 0.0311523 | val_loss: 0.0311718 | Time: 6534.29 ms [2022-03-13 11:13:42 main:574] : INFO : Epoch 1678 | loss: 0.03115 | val_loss: 0.0311651 | Time: 6521 ms [2022-03-13 11:13:49 main:574] : INFO : Epoch 1679 | loss: 0.0311498 | val_loss: 0.031171 | Time: 6497.64 ms [2022-03-13 11:13:55 main:574] : INFO : Epoch 1680 | loss: 0.03115 | val_loss: 0.031166 | Time: 6518.59 ms [2022-03-13 11:14:02 main:574] : INFO : Epoch 1681 | loss: 0.0311491 | val_loss: 0.0311644 | Time: 6526.81 ms [2022-03-13 11:14:08 main:574] : INFO : Epoch 1682 | loss: 0.0311493 | val_loss: 0.0311664 | Time: 6488.23 ms [2022-03-13 11:14:15 main:574] : INFO : Epoch 1683 | loss: 0.0311483 | val_loss: 0.0311727 | Time: 6512.03 ms [2022-03-13 11:14:21 main:574] : INFO : Epoch 1684 | loss: 0.0311495 | val_loss: 0.0311705 | Time: 6505.17 ms [2022-03-13 11:14:28 main:574] : INFO : Epoch 1685 | loss: 0.0311475 | val_loss: 0.0311705 | Time: 6514.05 ms [2022-03-13 11:14:34 main:574] : INFO : Epoch 1686 | loss: 0.0311497 | val_loss: 0.0311696 | Time: 6488.86 ms [2022-03-13 11:14:41 main:574] : INFO : Epoch 1687 | loss: 0.0311503 | val_loss: 0.0311685 | Time: 6521.8 ms [2022-03-13 11:14:47 main:574] : INFO : Epoch 1688 | loss: 0.0311551 | val_loss: 0.0311667 | Time: 6506.33 ms [2022-03-13 11:14:54 main:574] : INFO : Epoch 1689 | loss: 0.0311499 | val_loss: 0.031167 | Time: 6503.22 ms [2022-03-13 11:15:00 main:574] : INFO : Epoch 1690 | loss: 0.0311479 | val_loss: 0.0311625 | Time: 6544.77 ms [2022-03-13 11:15:07 main:574] : INFO : Epoch 1691 | loss: 0.0311491 | val_loss: 0.0311615 | Time: 6495.42 ms [2022-03-13 11:15:13 main:574] : INFO : Epoch 1692 | loss: 0.0311474 | val_loss: 0.0311614 | Time: 6514.67 ms [2022-03-13 11:15:20 main:574] : INFO : Epoch 1693 | loss: 0.0311487 | val_loss: 0.0311664 | Time: 6521.56 ms [2022-03-13 11:15:27 main:574] : INFO : Epoch 1694 | loss: 0.0311488 | val_loss: 0.0311703 | Time: 6501.52 ms [2022-03-13 11:15:33 main:574] : INFO : Epoch 1695 | loss: 0.0311479 | val_loss: 0.0311675 | Time: 6505.62 ms [2022-03-13 11:15:40 main:574] : INFO : Epoch 1696 | loss: 0.0311485 | val_loss: 0.0311645 | Time: 6483.65 ms [2022-03-13 11:15:46 main:574] : INFO : Epoch 1697 | loss: 0.0311469 | val_loss: 0.0311626 | Time: 6526.34 ms [2022-03-13 11:15:53 main:574] : INFO : Epoch 1698 | loss: 0.0311458 | val_loss: 0.031165 | Time: 6505.75 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GT 1030) [2022-03-13 11:22:13 main:435] : INFO : Set logging level to 1 [2022-03-13 11:22:13 main:441] : INFO : Running in BOINC Client mode [2022-03-13 11:22:13 main:444] : INFO : Resolving all filenames [2022-03-13 11:22:13 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-13 11:22:13 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-13 11:22:13 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-13 11:22:13 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-03-13 11:22:13 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-13 11:22:13 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-13 11:22:13 main:474] : INFO : Configuration: [2022-03-13 11:22:13 main:475] : INFO : Model type: GRU [2022-03-13 11:22:13 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-13 11:22:13 main:477] : INFO : Max Epochs: 2048 [2022-03-13 11:22:13 main:478] : INFO : Batch Size: 128 [2022-03-13 11:22:13 main:479] : INFO : Learning Rate: 0.01 [2022-03-13 11:22:13 main:480] : INFO : Patience: 10 [2022-03-13 11:22:13 main:481] : INFO : Hidden Width: 12 [2022-03-13 11:22:13 main:482] : INFO : # Recurrent Layers: 4 [2022-03-13 11:22:13 main:483] : INFO : # Backend Layers: 4 [2022-03-13 11:22:13 main:484] : INFO : # Threads: 1 [2022-03-13 11:22:13 main:486] : INFO : Preparing Dataset [2022-03-13 11:22:13 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-13 11:22:14 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-13 11:22:21 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-13 11:22:21 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-13 11:22:21 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-13 11:22:21 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-13 11:22:21 main:494] : INFO : Creating Model [2022-03-13 11:22:21 main:507] : INFO : Preparing config file [2022-03-13 11:22:21 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-13 11:22:21 main:512] : INFO : Loading config [2022-03-13 11:22:21 main:514] : INFO : Loading state [2022-03-13 11:22:35 main:559] : INFO : Loading DataLoader into Memory [2022-03-13 11:22:35 main:562] : INFO : Starting Training [2022-03-13 11:22:54 main:574] : INFO : Epoch 1697 | loss: 0.0312468 | val_loss: 0.0311943 | Time: 18995.3 ms [2022-03-13 11:23:01 main:574] : INFO : Epoch 1698 | loss: 0.0311582 | val_loss: 0.0311669 | Time: 6761.98 ms [2022-03-13 11:23:08 main:574] : INFO : Epoch 1699 | loss: 0.031151 | val_loss: 0.0311696 | Time: 6888.16 ms [2022-03-13 11:23:16 main:574] : INFO : Epoch 1700 | loss: 0.0311484 | val_loss: 0.0311656 | Time: 8130.08 ms [2022-03-13 11:23:33 main:574] : INFO : Epoch 1701 | loss: 0.0311489 | val_loss: 0.0311638 | Time: 16954.5 ms [2022-03-13 11:23:40 main:574] : INFO : Epoch 1702 | loss: 0.0311488 | val_loss: 0.031166 | Time: 7584.3 ms [2022-03-13 11:23:47 main:574] : INFO : Epoch 1703 | loss: 0.0311483 | val_loss: 0.0311667 | Time: 7043.29 ms [2022-03-13 11:23:54 main:574] : INFO : Epoch 1704 | loss: 0.0311475 | val_loss: 0.0311642 | Time: 6541.23 ms [2022-03-13 11:24:00 main:574] : INFO : Epoch 1705 | loss: 0.0311461 | val_loss: 0.031164 | Time: 6562.24 ms [2022-03-13 11:24:09 main:574] : INFO : Epoch 1706 | loss: 0.0311469 | val_loss: 0.0311654 | Time: 8861.82 ms [2022-03-13 11:24:26 main:574] : INFO : Epoch 1707 | loss: 0.0311474 | val_loss: 0.0311661 | Time: 16829 ms [2022-03-13 11:24:33 main:574] : INFO : Epoch 1708 | loss: 0.0311502 | val_loss: 0.0311691 | Time: 6949.32 ms [2022-03-13 11:24:40 main:574] : INFO : Epoch 1709 | loss: 0.0311466 | val_loss: 0.0311684 | Time: 7117.19 ms [2022-03-13 11:24:47 main:574] : INFO : Epoch 1710 | loss: 0.0311469 | val_loss: 0.0311647 | Time: 7140 ms [2022-03-13 11:24:54 main:574] : INFO : Epoch 1711 | loss: 0.0311463 | val_loss: 0.0311634 | Time: 6961.23 ms [2022-03-13 11:25:01 main:574] : INFO : Epoch 1712 | loss: 0.0311456 | val_loss: 0.0311622 | Time: 7014.64 ms [2022-03-13 11:25:08 main:574] : INFO : Epoch 1713 | loss: 0.0311469 | val_loss: 0.0311625 | Time: 6946.04 ms [2022-03-13 11:25:15 main:574] : INFO : Epoch 1714 | loss: 0.031147 | val_loss: 0.0311638 | Time: 6937.93 ms [2022-03-13 11:25:23 main:574] : INFO : Epoch 1715 | loss: 0.031146 | val_loss: 0.0311666 | Time: 7463.26 ms [2022-03-13 11:25:30 main:574] : INFO : Epoch 1716 | loss: 0.0311489 | val_loss: 0.0311649 | Time: 6964.55 ms [2022-03-13 11:25:37 main:574] : INFO : Epoch 1717 | loss: 0.0311491 | val_loss: 0.0311638 | Time: 6974.34 ms [2022-03-13 11:25:44 main:574] : INFO : Epoch 1718 | loss: 0.0311504 | val_loss: 0.0311636 | Time: 6862.11 ms [2022-03-13 11:25:51 main:574] : INFO : Epoch 1719 | loss: 0.0311466 | val_loss: 0.0311622 | Time: 7006.22 ms [2022-03-13 11:25:58 main:574] : INFO : Epoch 1720 | loss: 0.0311472 | val_loss: 0.0311621 | Time: 7153.62 ms [2022-03-13 11:26:05 main:574] : INFO : Epoch 1721 | loss: 0.0311454 | val_loss: 0.0311655 | Time: 6912.22 ms [2022-03-13 11:26:12 main:574] : INFO : Epoch 1722 | loss: 0.0311461 | val_loss: 0.0311643 | Time: 6930.31 ms [2022-03-13 11:26:20 main:574] : INFO : Epoch 1723 | loss: 0.0311457 | val_loss: 0.0311623 | Time: 7834.78 ms [2022-03-13 11:26:26 main:574] : INFO : Epoch 1724 | loss: 0.0311448 | val_loss: 0.0311647 | Time: 6754.22 ms [2022-03-13 11:26:33 main:574] : INFO : Epoch 1725 | loss: 0.0311459 | val_loss: 0.0311658 | Time: 6531.84 ms [2022-03-13 11:26:39 main:574] : INFO : Epoch 1726 | loss: 0.0311487 | val_loss: 0.0311725 | Time: 6573.94 ms [2022-03-13 11:26:46 main:574] : INFO : Epoch 1727 | loss: 0.031149 | val_loss: 0.0311669 | Time: 6531.74 ms [2022-03-13 11:26:53 main:574] : INFO : Epoch 1728 | loss: 0.031147 | val_loss: 0.0311634 | Time: 6535.16 ms [2022-03-13 11:26:59 main:574] : INFO : Epoch 1729 | loss: 0.0311457 | val_loss: 0.0311631 | Time: 6534.04 ms [2022-03-13 11:27:06 main:574] : INFO : Epoch 1730 | loss: 0.0311456 | val_loss: 0.0311627 | Time: 6509.88 ms [2022-03-13 11:27:12 main:574] : INFO : Epoch 1731 | loss: 0.0311462 | val_loss: 0.0311642 | Time: 6529.45 ms [2022-03-13 11:27:19 main:574] : INFO : Epoch 1732 | loss: 0.0311468 | val_loss: 0.0311606 | Time: 6541.08 ms [2022-03-13 11:27:25 main:574] : INFO : Epoch 1733 | loss: 0.0311476 | val_loss: 0.0311579 | Time: 6515.2 ms [2022-03-13 11:27:32 main:574] : INFO : Epoch 1734 | loss: 0.031147 | val_loss: 0.0311584 | Time: 6534.42 ms [2022-03-13 11:27:38 main:574] : INFO : Epoch 1735 | loss: 0.0311466 | val_loss: 0.0311628 | Time: 6557.86 ms [2022-03-13 11:27:45 main:574] : INFO : Epoch 1736 | loss: 0.0311476 | val_loss: 0.0311625 | Time: 6518.89 ms [2022-03-13 11:27:51 main:574] : INFO : Epoch 1737 | loss: 0.0311473 | val_loss: 0.0311609 | Time: 6523.29 ms [2022-03-13 11:27:58 main:574] : INFO : Epoch 1738 | loss: 0.0311472 | val_loss: 0.0311602 | Time: 6526.95 ms [2022-03-13 11:28:04 main:574] : INFO : Epoch 1739 | loss: 0.031147 | val_loss: 0.0311593 | Time: 6527.82 ms [2022-03-13 11:28:11 main:574] : INFO : Epoch 1740 | loss: 0.0311461 | val_loss: 0.0311599 | Time: 6519.4 ms [2022-03-13 11:28:17 main:574] : INFO : Epoch 1741 | loss: 0.0311486 | val_loss: 0.0311651 | Time: 6571 ms [2022-03-13 11:28:24 main:574] : INFO : Epoch 1742 | loss: 0.0311459 | val_loss: 0.0311596 | Time: 6537.55 ms [2022-03-13 11:28:31 main:574] : INFO : Epoch 1743 | loss: 0.0311436 | val_loss: 0.0311606 | Time: 6507.06 ms [2022-03-13 11:28:37 main:574] : INFO : Epoch 1744 | loss: 0.0311438 | val_loss: 0.0311607 | Time: 6528.17 ms [2022-03-13 11:28:44 main:574] : INFO : Epoch 1745 | loss: 0.0311453 | val_loss: 0.0311609 | Time: 6529.94 ms [2022-03-13 11:28:50 main:574] : INFO : Epoch 1746 | loss: 0.0311458 | val_loss: 0.0311642 | Time: 6515.36 ms [2022-03-13 11:28:57 main:574] : INFO : Epoch 1747 | loss: 0.0311446 | val_loss: 0.0311599 | Time: 6526.46 ms [2022-03-13 11:29:04 main:574] : INFO : Epoch 1748 | loss: 0.0311448 | val_loss: 0.0311604 | Time: 7309.14 ms [2022-03-13 11:29:11 main:574] : INFO : Epoch 1749 | loss: 0.0311453 | val_loss: 0.0311657 | Time: 6626.17 ms [2022-03-13 11:29:18 main:574] : INFO : Epoch 1750 | loss: 0.0311483 | val_loss: 0.0311599 | Time: 6522.85 ms [2022-03-13 11:29:24 main:574] : INFO : Epoch 1751 | loss: 0.031148 | val_loss: 0.0311595 | Time: 6528.43 ms [2022-03-13 11:29:31 main:574] : INFO : Epoch 1752 | loss: 0.0311458 | val_loss: 0.0311584 | Time: 6517.19 ms [2022-03-13 11:29:37 main:574] : INFO : Epoch 1753 | loss: 0.0311442 | val_loss: 0.0311589 | Time: 6655.91 ms [2022-03-13 11:29:44 main:574] : INFO : Epoch 1754 | loss: 0.0311481 | val_loss: 0.0311743 | Time: 7185.63 ms [2022-03-13 11:29:52 main:574] : INFO : Epoch 1755 | loss: 0.0311507 | val_loss: 0.031162 | Time: 7184.13 ms [2022-03-13 11:29:58 main:574] : INFO : Epoch 1756 | loss: 0.0311464 | val_loss: 0.03116 | Time: 6566.65 ms [2022-03-13 11:30:05 main:574] : INFO : Epoch 1757 | loss: 0.0311443 | val_loss: 0.0311602 | Time: 6851.84 ms [2022-03-13 11:30:12 main:574] : INFO : Epoch 1758 | loss: 0.0311448 | val_loss: 0.0311618 | Time: 6509.43 ms [2022-03-13 11:30:18 main:574] : INFO : Epoch 1759 | loss: 0.0311437 | val_loss: 0.0311605 | Time: 6532.56 ms [2022-03-13 11:30:25 main:574] : INFO : Epoch 1760 | loss: 0.031144 | val_loss: 0.0311612 | Time: 6536.18 ms [2022-03-13 11:30:31 main:574] : INFO : Epoch 1761 | loss: 0.0311442 | val_loss: 0.0311627 | Time: 6534.41 ms [2022-03-13 11:30:38 main:574] : INFO : Epoch 1762 | loss: 0.0311463 | val_loss: 0.031161 | Time: 6550.3 ms [2022-03-13 11:30:44 main:574] : INFO : Epoch 1763 | loss: 0.0311444 | val_loss: 0.0311646 | Time: 6536.95 ms [2022-03-13 11:30:51 main:574] : INFO : Epoch 1764 | loss: 0.0311451 | val_loss: 0.0311629 | Time: 6506.98 ms [2022-03-13 11:30:58 main:574] : INFO : Epoch 1765 | loss: 0.0311447 | val_loss: 0.0311591 | Time: 7124.73 ms [2022-03-13 11:31:04 main:574] : INFO : Epoch 1766 | loss: 0.0311457 | val_loss: 0.0311606 | Time: 6536.24 ms [2022-03-13 11:31:11 main:574] : INFO : Epoch 1767 | loss: 0.0311447 | val_loss: 0.0311594 | Time: 6517.15 ms [2022-03-13 11:31:17 main:574] : INFO : Epoch 1768 | loss: 0.0311443 | val_loss: 0.0311635 | Time: 6529.36 ms [2022-03-13 11:31:24 main:574] : INFO : Epoch 1769 | loss: 0.0311451 | val_loss: 0.0311593 | Time: 6543.33 ms [2022-03-13 11:31:31 main:574] : INFO : Epoch 1770 | loss: 0.0311435 | val_loss: 0.0311581 | Time: 6519.44 ms [2022-03-13 11:31:37 main:574] : INFO : Epoch 1771 | loss: 0.0311429 | val_loss: 0.0311602 | Time: 6516.91 ms [2022-03-13 11:31:44 main:574] : INFO : Epoch 1772 | loss: 0.0311454 | val_loss: 0.0311595 | Time: 6532.66 ms [2022-03-13 11:31:50 main:574] : INFO : Epoch 1773 | loss: 0.0311474 | val_loss: 0.0311648 | Time: 6531.52 ms [2022-03-13 11:31:57 main:574] : INFO : Epoch 1774 | loss: 0.0311471 | val_loss: 0.0311642 | Time: 6547.9 ms [2022-03-13 11:32:03 main:574] : INFO : Epoch 1775 | loss: 0.0311432 | val_loss: 0.0311668 | Time: 6542.94 ms [2022-03-13 11:32:10 main:574] : INFO : Epoch 1776 | loss: 0.0311487 | val_loss: 0.0311638 | Time: 6548.13 ms [2022-03-13 11:32:16 main:574] : INFO : Epoch 1777 | loss: 0.0311437 | val_loss: 0.0311618 | Time: 6516.57 ms [2022-03-13 11:32:23 main:574] : INFO : Epoch 1778 | loss: 0.031143 | val_loss: 0.0311601 | Time: 6564.8 ms [2022-03-13 11:32:29 main:574] : INFO : Epoch 1779 | loss: 0.0311447 | val_loss: 0.0311607 | Time: 6531.76 ms [2022-03-13 11:32:36 main:574] : INFO : Epoch 1780 | loss: 0.0311436 | val_loss: 0.0311586 | Time: 6529.68 ms [2022-03-13 11:32:43 main:574] : INFO : Epoch 1781 | loss: 0.0311424 | val_loss: 0.0311592 | Time: 6532.39 ms [2022-03-13 11:32:49 main:574] : INFO : Epoch 1782 | loss: 0.0311428 | val_loss: 0.0311613 | Time: 6527.02 ms [2022-03-13 11:32:56 main:574] : INFO : Epoch 1783 | loss: 0.0311441 | val_loss: 0.0311676 | Time: 6527.5 ms [2022-03-13 11:33:02 main:574] : INFO : Epoch 1784 | loss: 0.0311489 | val_loss: 0.0311603 | Time: 6550.41 ms [2022-03-13 11:33:09 main:574] : INFO : Epoch 1785 | loss: 0.0311485 | val_loss: 0.0311603 | Time: 6542.88 ms [2022-03-13 11:33:15 main:574] : INFO : Epoch 1786 | loss: 0.031151 | val_loss: 0.0311615 | Time: 6529.84 ms [2022-03-13 11:33:22 main:574] : INFO : Epoch 1787 | loss: 0.0311491 | val_loss: 0.0311662 | Time: 6522.2 ms [2022-03-13 11:33:28 main:574] : INFO : Epoch 1788 | loss: 0.0311498 | val_loss: 0.031161 | Time: 6537.01 ms [2022-03-13 11:33:35 main:574] : INFO : Epoch 1789 | loss: 0.0311503 | val_loss: 0.031164 | Time: 6559.93 ms [2022-03-13 11:33:41 main:574] : INFO : Epoch 1790 | loss: 0.0311487 | val_loss: 0.0311639 | Time: 6510.96 ms [2022-03-13 11:33:48 main:574] : INFO : Epoch 1791 | loss: 0.0311494 | val_loss: 0.031164 | Time: 6541.97 ms [2022-03-13 11:33:54 main:574] : INFO : Epoch 1792 | loss: 0.0311486 | val_loss: 0.0311619 | Time: 6533.65 ms [2022-03-13 11:34:01 main:574] : INFO : Epoch 1793 | loss: 0.03115 | val_loss: 0.0311626 | Time: 6532.97 ms [2022-03-13 11:34:08 main:574] : INFO : Epoch 1794 | loss: 0.0311524 | val_loss: 0.0311628 | Time: 6528.01 ms [2022-03-13 11:34:14 main:574] : INFO : Epoch 1795 | loss: 0.0311554 | val_loss: 0.0311625 | Time: 6528.1 ms [2022-03-13 11:34:21 main:574] : INFO : Epoch 1796 | loss: 0.0311547 | val_loss: 0.031164 | Time: 6578.1 ms [2022-03-13 11:34:27 main:574] : INFO : Epoch 1797 | loss: 0.0311529 | val_loss: 0.0311604 | Time: 6539.62 ms [2022-03-13 11:34:34 main:574] : INFO : Epoch 1798 | loss: 0.0311511 | val_loss: 0.0311611 | Time: 6531.81 ms [2022-03-13 11:34:40 main:574] : INFO : Epoch 1799 | loss: 0.0311521 | val_loss: 0.0311621 | Time: 6521.08 ms [2022-03-13 11:34:47 main:574] : INFO : Epoch 1800 | loss: 0.03115 | val_loss: 0.0311627 | Time: 6509.79 ms [2022-03-13 11:34:53 main:574] : INFO : Epoch 1801 | loss: 0.0311496 | val_loss: 0.0311639 | Time: 6543.86 ms [2022-03-13 11:35:00 main:574] : INFO : Epoch 1802 | loss: 0.0311519 | val_loss: 0.0311651 | Time: 6539.95 ms [2022-03-13 11:35:06 main:574] : INFO : Epoch 1803 | loss: 0.0311534 | val_loss: 0.0311622 | Time: 6524.04 ms [2022-03-13 11:35:13 main:574] : INFO : Epoch 1804 | loss: 0.0311496 | val_loss: 0.0311627 | Time: 6534.72 ms [2022-03-13 11:35:19 main:574] : INFO : Epoch 1805 | loss: 0.0311523 | val_loss: 0.0311643 | Time: 6524.88 ms [2022-03-13 11:35:26 main:574] : INFO : Epoch 1806 | loss: 0.0311513 | val_loss: 0.0311674 | Time: 6520.72 ms [2022-03-13 11:35:33 main:574] : INFO : Epoch 1807 | loss: 0.031152 | val_loss: 0.0311658 | Time: 6517.91 ms [2022-03-13 11:35:39 main:574] : INFO : Epoch 1808 | loss: 0.0311524 | val_loss: 0.0311673 | Time: 6536.8 ms [2022-03-13 11:35:46 main:574] : INFO : Epoch 1809 | loss: 0.0311519 | val_loss: 0.0311721 | Time: 6543.01 ms [2022-03-13 11:35:52 main:574] : INFO : Epoch 1810 | loss: 0.0311547 | val_loss: 0.0311644 | Time: 6514.79 ms [2022-03-13 11:35:59 main:574] : INFO : Epoch 1811 | loss: 0.0311507 | val_loss: 0.031163 | Time: 6606.03 ms [2022-03-13 11:36:05 main:574] : INFO : Epoch 1812 | loss: 0.0311527 | val_loss: 0.0311645 | Time: 6540.91 ms [2022-03-13 11:36:12 main:574] : INFO : Epoch 1813 | loss: 0.0311672 | val_loss: 0.0311732 | Time: 6535.21 ms [2022-03-13 11:36:18 main:574] : INFO : Epoch 1814 | loss: 0.0311691 | val_loss: 0.031174 | Time: 6526.36 ms [2022-03-13 11:36:25 main:574] : INFO : Epoch 1815 | loss: 0.0311607 | val_loss: 0.0311705 | Time: 6535.16 ms [2022-03-13 11:36:32 main:574] : INFO : Epoch 1816 | loss: 0.0311543 | val_loss: 0.031164 | Time: 6526.97 ms [2022-03-13 11:36:38 main:574] : INFO : Epoch 1817 | loss: 0.0311533 | val_loss: 0.031163 | Time: 6531.28 ms [2022-03-13 11:36:45 main:574] : INFO : Epoch 1818 | loss: 0.0311509 | val_loss: 0.0311606 | Time: 6530.7 ms [2022-03-13 11:36:51 main:574] : INFO : Epoch 1819 | loss: 0.0311493 | val_loss: 0.031163 | Time: 6534.6 ms [2022-03-13 11:36:58 main:574] : INFO : Epoch 1820 | loss: 0.0311502 | val_loss: 0.0311652 | Time: 6525.27 ms [2022-03-13 11:37:04 main:574] : INFO : Epoch 1821 | loss: 0.0311499 | val_loss: 0.0311629 | Time: 6528.64 ms [2022-03-13 11:37:11 main:574] : INFO : Epoch 1822 | loss: 0.0311492 | val_loss: 0.0311626 | Time: 6525.47 ms [2022-03-13 11:37:17 main:574] : INFO : Epoch 1823 | loss: 0.0311501 | val_loss: 0.0311615 | Time: 6513.4 ms [2022-03-13 11:37:24 main:574] : INFO : Epoch 1824 | loss: 0.0311518 | val_loss: 0.0311669 | Time: 6539.78 ms [2022-03-13 11:37:30 main:574] : INFO : Epoch 1825 | loss: 0.0311506 | val_loss: 0.0311621 | Time: 6549.87 ms [2022-03-13 11:37:37 main:574] : INFO : Epoch 1826 | loss: 0.0311475 | val_loss: 0.0311663 | Time: 6516.82 ms [2022-03-13 11:37:43 main:574] : INFO : Epoch 1827 | loss: 0.0311492 | val_loss: 0.0311653 | Time: 6528.18 ms [2022-03-13 11:37:50 main:574] : INFO : Epoch 1828 | loss: 0.0311478 | val_loss: 0.0311642 | Time: 6530.55 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GT 1030) [2022-03-13 13:22:52 main:435] : INFO : Set logging level to 1 [2022-03-13 13:22:52 main:441] : INFO : Running in BOINC Client mode [2022-03-13 13:22:52 main:444] : INFO : Resolving all filenames [2022-03-13 13:22:52 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-13 13:22:52 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-13 13:22:52 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-13 13:22:52 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-03-13 13:22:52 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-13 13:22:52 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-13 13:22:53 main:474] : INFO : Configuration: [2022-03-13 13:22:53 main:475] : INFO : Model type: GRU [2022-03-13 13:22:53 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-13 13:22:53 main:477] : INFO : Max Epochs: 2048 [2022-03-13 13:22:53 main:478] : INFO : Batch Size: 128 [2022-03-13 13:22:53 main:479] : INFO : Learning Rate: 0.01 [2022-03-13 13:22:53 main:480] : INFO : Patience: 10 [2022-03-13 13:22:53 main:481] : INFO : Hidden Width: 12 [2022-03-13 13:22:53 main:482] : INFO : # Recurrent Layers: 4 [2022-03-13 13:22:53 main:483] : INFO : # Backend Layers: 4 [2022-03-13 13:22:53 main:484] : INFO : # Threads: 1 [2022-03-13 13:22:53 main:486] : INFO : Preparing Dataset [2022-03-13 13:22:53 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-13 13:22:54 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-13 13:23:17 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-13 13:23:17 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-13 13:23:17 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-13 13:23:18 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-13 13:23:18 main:494] : INFO : Creating Model [2022-03-13 13:23:18 main:507] : INFO : Preparing config file [2022-03-13 13:23:18 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-13 13:23:18 main:512] : INFO : Loading config [2022-03-13 13:23:18 main:514] : INFO : Loading state [2022-03-13 13:23:23 main:559] : INFO : Loading DataLoader into Memory [2022-03-13 13:23:23 main:562] : INFO : Starting Training [2022-03-13 13:23:32 main:574] : INFO : Epoch 1811 | loss: 0.0312409 | val_loss: 0.0311895 | Time: 8469.48 ms [2022-03-13 13:23:38 main:574] : INFO : Epoch 1812 | loss: 0.0311593 | val_loss: 0.0311976 | Time: 6540.53 ms [2022-03-13 13:23:45 main:574] : INFO : Epoch 1813 | loss: 0.0311621 | val_loss: 0.0311927 | Time: 6536.28 ms [2022-03-13 13:23:51 main:574] : INFO : Epoch 1814 | loss: 0.0311643 | val_loss: 0.0311791 | Time: 6535.55 ms [2022-03-13 13:23:58 main:574] : INFO : Epoch 1815 | loss: 0.0311523 | val_loss: 0.0311651 | Time: 6532.29 ms [2022-03-13 13:24:05 main:574] : INFO : Epoch 1816 | loss: 0.031154 | val_loss: 0.0311622 | Time: 6552.7 ms [2022-03-13 13:24:11 main:574] : INFO : Epoch 1817 | loss: 0.0311484 | val_loss: 0.0311601 | Time: 6583.03 ms [2022-03-13 13:24:19 main:574] : INFO : Epoch 1818 | loss: 0.0311488 | val_loss: 0.0311635 | Time: 7745.83 ms [2022-03-13 13:24:47 main:574] : INFO : Epoch 1819 | loss: 0.0311491 | val_loss: 0.031163 | Time: 27792.5 ms [2022-03-13 13:24:54 main:574] : INFO : Epoch 1820 | loss: 0.0311509 | val_loss: 0.0311664 | Time: 6677.04 ms [2022-03-13 13:25:00 main:574] : INFO : Epoch 1821 | loss: 0.0311478 | val_loss: 0.0311914 | Time: 6513.43 ms [2022-03-13 13:25:07 main:574] : INFO : Epoch 1822 | loss: 0.031163 | val_loss: 0.0311692 | Time: 6513.22 ms [2022-03-13 13:25:13 main:574] : INFO : Epoch 1823 | loss: 0.0311552 | val_loss: 0.0311615 | Time: 6551.87 ms [2022-03-13 13:25:20 main:574] : INFO : Epoch 1824 | loss: 0.0311502 | val_loss: 0.0311618 | Time: 6548.89 ms [2022-03-13 13:25:26 main:574] : INFO : Epoch 1825 | loss: 0.0311482 | val_loss: 0.0311614 | Time: 6501.07 ms [2022-03-13 13:25:33 main:574] : INFO : Epoch 1826 | loss: 0.0311495 | val_loss: 0.0311619 | Time: 6524.79 ms [2022-03-13 13:25:39 main:574] : INFO : Epoch 1827 | loss: 0.0311505 | val_loss: 0.0311597 | Time: 6549.02 ms [2022-03-13 13:25:46 main:574] : INFO : Epoch 1828 | loss: 0.0311498 | val_loss: 0.0311606 | Time: 6512.35 ms [2022-03-13 13:25:52 main:574] : INFO : Epoch 1829 | loss: 0.0311482 | val_loss: 0.0311582 | Time: 6540.49 ms [2022-03-13 13:25:59 main:574] : INFO : Epoch 1830 | loss: 0.0311473 | val_loss: 0.0311577 | Time: 6755.8 ms [2022-03-13 13:26:06 main:574] : INFO : Epoch 1831 | loss: 0.0311464 | val_loss: 0.0311581 | Time: 6505.42 ms [2022-03-13 13:26:12 main:574] : INFO : Epoch 1832 | loss: 0.0311476 | val_loss: 0.0311597 | Time: 6527.23 ms [2022-03-13 13:26:19 main:574] : INFO : Epoch 1833 | loss: 0.0311474 | val_loss: 0.0311584 | Time: 6534.62 ms [2022-03-13 13:26:25 main:574] : INFO : Epoch 1834 | loss: 0.0311471 | val_loss: 0.0311622 | Time: 6539.31 ms [2022-03-13 13:26:32 main:574] : INFO : Epoch 1835 | loss: 0.0311455 | val_loss: 0.0311582 | Time: 6540.37 ms [2022-03-13 13:26:39 main:574] : INFO : Epoch 1836 | loss: 0.0311439 | val_loss: 0.0311579 | Time: 6542.31 ms [2022-03-13 13:26:45 main:574] : INFO : Epoch 1837 | loss: 0.0311446 | val_loss: 0.0311613 | Time: 6567.65 ms [2022-03-13 13:26:52 main:574] : INFO : Epoch 1838 | loss: 0.0311485 | val_loss: 0.0311572 | Time: 6537.25 ms [2022-03-13 13:26:58 main:574] : INFO : Epoch 1839 | loss: 0.0311455 | val_loss: 0.0311568 | Time: 6539.24 ms [2022-03-13 13:27:05 main:574] : INFO : Epoch 1840 | loss: 0.0311453 | val_loss: 0.0311591 | Time: 6534.73 ms [2022-03-13 13:27:11 main:574] : INFO : Epoch 1841 | loss: 0.0311453 | val_loss: 0.0311591 | Time: 6544.54 ms [2022-03-13 13:27:18 main:574] : INFO : Epoch 1842 | loss: 0.0311469 | val_loss: 0.0311585 | Time: 6544 ms [2022-03-13 13:27:24 main:574] : INFO : Epoch 1843 | loss: 0.031145 | val_loss: 0.0311574 | Time: 6519.22 ms [2022-03-13 13:27:31 main:574] : INFO : Epoch 1844 | loss: 0.0311502 | val_loss: 0.0311638 | Time: 6541 ms [2022-03-13 13:27:37 main:574] : INFO : Epoch 1845 | loss: 0.0311505 | val_loss: 0.0311628 | Time: 6535.03 ms [2022-03-13 13:27:44 main:574] : INFO : Epoch 1846 | loss: 0.0311497 | val_loss: 0.0311639 | Time: 6553.57 ms [2022-03-13 13:27:51 main:574] : INFO : Epoch 1847 | loss: 0.0311494 | val_loss: 0.03116 | Time: 6543.78 ms [2022-03-13 13:27:57 main:574] : INFO : Epoch 1848 | loss: 0.0311482 | val_loss: 0.0311603 | Time: 6531.63 ms [2022-03-13 13:28:04 main:574] : INFO : Epoch 1849 | loss: 0.0311465 | val_loss: 0.0311594 | Time: 6554.37 ms [2022-03-13 13:28:10 main:574] : INFO : Epoch 1850 | loss: 0.0311458 | val_loss: 0.0311586 | Time: 6592.44 ms [2022-03-13 13:28:17 main:574] : INFO : Epoch 1851 | loss: 0.0311453 | val_loss: 0.031159 | Time: 6497.85 ms [2022-03-13 13:28:23 main:574] : INFO : Epoch 1852 | loss: 0.0311461 | val_loss: 0.0311602 | Time: 6540.83 ms [2022-03-13 13:28:30 main:574] : INFO : Epoch 1853 | loss: 0.0311471 | val_loss: 0.0311612 | Time: 6547.22 ms [2022-03-13 13:28:36 main:574] : INFO : Epoch 1854 | loss: 0.0311466 | val_loss: 0.0311669 | Time: 6551.6 ms [2022-03-13 13:28:43 main:574] : INFO : Epoch 1855 | loss: 0.0311477 | val_loss: 0.0311605 | Time: 6544.56 ms [2022-03-13 13:28:49 main:574] : INFO : Epoch 1856 | loss: 0.0311451 | val_loss: 0.0311595 | Time: 6520.52 ms [2022-03-13 13:28:56 main:574] : INFO : Epoch 1857 | loss: 0.0311445 | val_loss: 0.0311601 | Time: 6552.34 ms [2022-03-13 13:29:03 main:574] : INFO : Epoch 1858 | loss: 0.0311455 | val_loss: 0.0311646 | Time: 6574.83 ms [2022-03-13 13:29:09 main:574] : INFO : Epoch 1859 | loss: 0.0311453 | val_loss: 0.031159 | Time: 6547.03 ms [2022-03-13 13:29:16 main:574] : INFO : Epoch 1860 | loss: 0.0311475 | val_loss: 0.0311578 | Time: 6550.83 ms [2022-03-13 13:29:23 main:574] : INFO : Epoch 1861 | loss: 0.0311462 | val_loss: 0.031166 | Time: 7493.16 ms [2022-03-13 13:29:30 main:574] : INFO : Epoch 1862 | loss: 0.0311506 | val_loss: 0.0311606 | Time: 6550.5 ms [2022-03-13 13:29:36 main:574] : INFO : Epoch 1863 | loss: 0.0311451 | val_loss: 0.0311605 | Time: 6517.98 ms [2022-03-13 13:29:43 main:574] : INFO : Epoch 1864 | loss: 0.0311446 | val_loss: 0.0311629 | Time: 6701.82 ms [2022-03-13 13:29:50 main:574] : INFO : Epoch 1865 | loss: 0.0311437 | val_loss: 0.0311597 | Time: 6605.16 ms [2022-03-13 13:29:56 main:574] : INFO : Epoch 1866 | loss: 0.031144 | val_loss: 0.0311635 | Time: 6575.4 ms [2022-03-13 13:30:03 main:574] : INFO : Epoch 1867 | loss: 0.031143 | val_loss: 0.031163 | Time: 6578.66 ms [2022-03-13 13:30:09 main:574] : INFO : Epoch 1868 | loss: 0.031143 | val_loss: 0.0311593 | Time: 6543.77 ms [2022-03-13 13:30:16 main:574] : INFO : Epoch 1869 | loss: 0.0311451 | val_loss: 0.0311598 | Time: 6584.69 ms [2022-03-13 13:30:22 main:574] : INFO : Epoch 1870 | loss: 0.0311448 | val_loss: 0.0311626 | Time: 6529.95 ms [2022-03-13 13:30:29 main:574] : INFO : Epoch 1871 | loss: 0.0311502 | val_loss: 0.0311637 | Time: 6539.62 ms [2022-03-13 13:30:36 main:574] : INFO : Epoch 1872 | loss: 0.0311479 | val_loss: 0.0311618 | Time: 6543.14 ms [2022-03-13 13:30:42 main:574] : INFO : Epoch 1873 | loss: 0.0311477 | val_loss: 0.0311661 | Time: 6563.97 ms [2022-03-13 13:30:49 main:574] : INFO : Epoch 1874 | loss: 0.0311482 | val_loss: 0.031163 | Time: 6568.35 ms [2022-03-13 13:30:55 main:574] : INFO : Epoch 1875 | loss: 0.0311466 | val_loss: 0.0311606 | Time: 6695.98 ms [2022-03-13 13:31:02 main:574] : INFO : Epoch 1876 | loss: 0.031147 | val_loss: 0.0311612 | Time: 6866.1 ms [2022-03-13 13:31:09 main:574] : INFO : Epoch 1877 | loss: 0.0311474 | val_loss: 0.031168 | Time: 6555.66 ms [2022-03-13 13:31:15 main:574] : INFO : Epoch 1878 | loss: 0.0311495 | val_loss: 0.0311612 | Time: 6599.22 ms [2022-03-13 13:31:22 main:574] : INFO : Epoch 1879 | loss: 0.0311447 | val_loss: 0.0311631 | Time: 6526.84 ms [2022-03-13 13:31:29 main:574] : INFO : Epoch 1880 | loss: 0.0311462 | val_loss: 0.0311607 | Time: 6565.75 ms [2022-03-13 13:31:35 main:574] : INFO : Epoch 1881 | loss: 0.0311501 | val_loss: 0.0311652 | Time: 6551.57 ms [2022-03-13 13:31:42 main:574] : INFO : Epoch 1882 | loss: 0.031148 | val_loss: 0.0311634 | Time: 6540.24 ms [2022-03-13 13:31:48 main:574] : INFO : Epoch 1883 | loss: 0.0311473 | val_loss: 0.0311597 | Time: 6543.09 ms [2022-03-13 13:31:55 main:574] : INFO : Epoch 1884 | loss: 0.0311463 | val_loss: 0.0311601 | Time: 6538.51 ms [2022-03-13 13:32:01 main:574] : INFO : Epoch 1885 | loss: 0.0311466 | val_loss: 0.0311597 | Time: 6574.06 ms [2022-03-13 13:32:08 main:574] : INFO : Epoch 1886 | loss: 0.0311454 | val_loss: 0.0311605 | Time: 6544.06 ms [2022-03-13 13:32:14 main:574] : INFO : Epoch 1887 | loss: 0.0311468 | val_loss: 0.0311591 | Time: 6565.11 ms [2022-03-13 13:32:21 main:574] : INFO : Epoch 1888 | loss: 0.0311479 | val_loss: 0.0311589 | Time: 6547.1 ms [2022-03-13 13:32:28 main:574] : INFO : Epoch 1889 | loss: 0.0311493 | val_loss: 0.0311693 | Time: 6546.2 ms [2022-03-13 13:32:34 main:574] : INFO : Epoch 1890 | loss: 0.0311498 | val_loss: 0.0311606 | Time: 6544.13 ms [2022-03-13 13:32:41 main:574] : INFO : Epoch 1891 | loss: 0.0311473 | val_loss: 0.0311606 | Time: 6540.43 ms [2022-03-13 13:32:47 main:574] : INFO : Epoch 1892 | loss: 0.0311464 | val_loss: 0.0311603 | Time: 6556.36 ms [2022-03-13 13:32:54 main:574] : INFO : Epoch 1893 | loss: 0.0311458 | val_loss: 0.0311582 | Time: 6560.6 ms [2022-03-13 13:33:00 main:574] : INFO : Epoch 1894 | loss: 0.0311459 | val_loss: 0.0311594 | Time: 6530.8 ms [2022-03-13 13:33:07 main:574] : INFO : Epoch 1895 | loss: 0.0311508 | val_loss: 0.0311602 | Time: 6599.87 ms [2022-03-13 13:33:14 main:574] : INFO : Epoch 1896 | loss: 0.0311465 | val_loss: 0.0311659 | Time: 6572.33 ms [2022-03-13 13:33:20 main:574] : INFO : Epoch 1897 | loss: 0.0311455 | val_loss: 0.0311644 | Time: 6557 ms [2022-03-13 13:33:27 main:574] : INFO : Epoch 1898 | loss: 0.0311496 | val_loss: 0.0311633 | Time: 6552.17 ms [2022-03-13 13:33:33 main:574] : INFO : Epoch 1899 | loss: 0.0311469 | val_loss: 0.0311592 | Time: 6531.32 ms [2022-03-13 13:33:40 main:574] : INFO : Epoch 1900 | loss: 0.0311491 | val_loss: 0.031159 | Time: 6541.63 ms [2022-03-13 13:33:46 main:574] : INFO : Epoch 1901 | loss: 0.0311488 | val_loss: 0.031161 | Time: 6570.37 ms [2022-03-13 13:33:53 main:574] : INFO : Epoch 1902 | loss: 0.0311467 | val_loss: 0.0311577 | Time: 6527.86 ms [2022-03-13 13:33:59 main:574] : INFO : Epoch 1903 | loss: 0.0311495 | val_loss: 0.0311595 | Time: 6537.07 ms [2022-03-13 13:34:06 main:574] : INFO : Epoch 1904 | loss: 0.0311483 | val_loss: 0.0311592 | Time: 6550.47 ms [2022-03-13 13:34:12 main:574] : INFO : Epoch 1905 | loss: 0.0311476 | val_loss: 0.0311593 | Time: 6553.37 ms [2022-03-13 13:34:19 main:574] : INFO : Epoch 1906 | loss: 0.0311476 | val_loss: 0.0311567 | Time: 6535.43 ms [2022-03-13 13:34:26 main:574] : INFO : Epoch 1907 | loss: 0.0311507 | val_loss: 0.031157 | Time: 6545.93 ms [2022-03-13 13:34:32 main:574] : INFO : Epoch 1908 | loss: 0.0311504 | val_loss: 0.031162 | Time: 6556.85 ms [2022-03-13 13:34:39 main:574] : INFO : Epoch 1909 | loss: 0.0311493 | val_loss: 0.0311599 | Time: 6552.93 ms [2022-03-13 13:34:45 main:574] : INFO : Epoch 1910 | loss: 0.0311455 | val_loss: 0.0311568 | Time: 6532.02 ms [2022-03-13 13:34:52 main:574] : INFO : Epoch 1911 | loss: 0.0311451 | val_loss: 0.0311564 | Time: 6521.42 ms [2022-03-13 13:34:58 main:574] : INFO : Epoch 1912 | loss: 0.0311499 | val_loss: 0.0311662 | Time: 6551.66 ms [2022-03-13 13:35:05 main:574] : INFO : Epoch 1913 | loss: 0.0311461 | val_loss: 0.0311591 | Time: 6575.95 ms [2022-03-13 13:35:11 main:574] : INFO : Epoch 1914 | loss: 0.0311488 | val_loss: 0.0311598 | Time: 6525.9 ms [2022-03-13 13:35:18 main:574] : INFO : Epoch 1915 | loss: 0.0311466 | val_loss: 0.0311605 | Time: 6534.06 ms [2022-03-13 13:35:25 main:574] : INFO : Epoch 1916 | loss: 0.0311483 | val_loss: 0.0311589 | Time: 6574.85 ms [2022-03-13 13:35:31 main:574] : INFO : Epoch 1917 | loss: 0.0311457 | val_loss: 0.0311603 | Time: 6569.46 ms [2022-03-13 13:35:38 main:574] : INFO : Epoch 1918 | loss: 0.0311443 | val_loss: 0.0311619 | Time: 6536.95 ms [2022-03-13 13:35:44 main:574] : INFO : Epoch 1919 | loss: 0.0311461 | val_loss: 0.0311612 | Time: 6537.02 ms [2022-03-13 13:35:51 main:574] : INFO : Epoch 1920 | loss: 0.0311471 | val_loss: 0.0311614 | Time: 6581.24 ms [2022-03-13 13:35:57 main:574] : INFO : Epoch 1921 | loss: 0.0311434 | val_loss: 0.0311603 | Time: 6674.45 ms [2022-03-13 13:36:04 main:574] : INFO : Epoch 1922 | loss: 0.0311453 | val_loss: 0.0311619 | Time: 6536.36 ms [2022-03-13 13:36:11 main:574] : INFO : Epoch 1923 | loss: 0.031144 | val_loss: 0.0311612 | Time: 6761.78 ms [2022-03-13 13:36:17 main:574] : INFO : Epoch 1924 | loss: 0.0311425 | val_loss: 0.03116 | Time: 6532.16 ms [2022-03-13 13:36:24 main:574] : INFO : Epoch 1925 | loss: 0.0311447 | val_loss: 0.0311562 | Time: 6542.54 ms [2022-03-13 13:36:31 main:574] : INFO : Epoch 1926 | loss: 0.031145 | val_loss: 0.0311571 | Time: 6554.56 ms [2022-03-13 13:36:37 main:574] : INFO : Epoch 1927 | loss: 0.0311443 | val_loss: 0.0311571 | Time: 6560.75 ms [2022-03-13 13:36:44 main:574] : INFO : Epoch 1928 | loss: 0.0311436 | val_loss: 0.0311593 | Time: 6523.27 ms [2022-03-13 13:36:50 main:574] : INFO : Epoch 1929 | loss: 0.0311465 | val_loss: 0.0311572 | Time: 6534.18 ms [2022-03-13 13:36:57 main:574] : INFO : Epoch 1930 | loss: 0.0311472 | val_loss: 0.031161 | Time: 6576.48 ms [2022-03-13 13:37:03 main:574] : INFO : Epoch 1931 | loss: 0.0311458 | val_loss: 0.0311602 | Time: 6538.85 ms [2022-03-13 13:37:10 main:574] : INFO : Epoch 1932 | loss: 0.0311455 | val_loss: 0.0311578 | Time: 6537.28 ms [2022-03-13 13:37:16 main:574] : INFO : Epoch 1933 | loss: 0.0311443 | val_loss: 0.0311573 | Time: 6522.76 ms [2022-03-13 13:37:23 main:574] : INFO : Epoch 1934 | loss: 0.0311455 | val_loss: 0.03116 | Time: 6532.43 ms [2022-03-13 13:37:30 main:574] : INFO : Epoch 1935 | loss: 0.0311455 | val_loss: 0.0311598 | Time: 6557.44 ms [2022-03-13 13:37:36 main:574] : INFO : Epoch 1936 | loss: 0.0311438 | val_loss: 0.031157 | Time: 6615.03 ms [2022-03-13 13:37:43 main:574] : INFO : Epoch 1937 | loss: 0.0311444 | val_loss: 0.031161 | Time: 6716.2 ms [2022-03-13 13:37:49 main:574] : INFO : Epoch 1938 | loss: 0.0311473 | val_loss: 0.0311571 | Time: 6598.49 ms [2022-03-13 13:37:56 main:574] : INFO : Epoch 1939 | loss: 0.0311479 | val_loss: 0.0311586 | Time: 6553.16 ms [2022-03-13 13:38:03 main:574] : INFO : Epoch 1940 | loss: 0.0311432 | val_loss: 0.0311584 | Time: 6583.5 ms [2022-03-13 13:38:10 main:574] : INFO : Epoch 1941 | loss: 0.0311434 | val_loss: 0.0311584 | Time: 7434.56 ms [2022-03-13 13:38:18 main:574] : INFO : Epoch 1942 | loss: 0.0311441 | val_loss: 0.0311607 | Time: 7925.45 ms [2022-03-13 13:38:25 main:574] : INFO : Epoch 1943 | loss: 0.0311449 | val_loss: 0.031161 | Time: 6554.01 ms [2022-03-13 13:38:31 main:574] : INFO : Epoch 1944 | loss: 0.0311433 | val_loss: 0.0311614 | Time: 6532.23 ms [2022-03-13 13:38:38 main:574] : INFO : Epoch 1945 | loss: 0.0311457 | val_loss: 0.0311592 | Time: 6546.64 ms [2022-03-13 13:38:44 main:574] : INFO : Epoch 1946 | loss: 0.0311453 | val_loss: 0.0311671 | Time: 6524.99 ms [2022-03-13 13:38:51 main:574] : INFO : Epoch 1947 | loss: 0.031147 | val_loss: 0.0311597 | Time: 6561.09 ms [2022-03-13 13:38:57 main:574] : INFO : Epoch 1948 | loss: 0.0311463 | val_loss: 0.0311668 | Time: 6545.69 ms [2022-03-13 13:39:04 main:574] : INFO : Epoch 1949 | loss: 0.0311461 | val_loss: 0.0311605 | Time: 6553.87 ms [2022-03-13 13:39:10 main:574] : INFO : Epoch 1950 | loss: 0.0311471 | val_loss: 0.0311605 | Time: 6553.51 ms [2022-03-13 13:39:17 main:574] : INFO : Epoch 1951 | loss: 0.0311468 | val_loss: 0.03116 | Time: 6553.05 ms [2022-03-13 13:39:24 main:574] : INFO : Epoch 1952 | loss: 0.0311438 | val_loss: 0.0311585 | Time: 6524.94 ms [2022-03-13 13:39:30 main:574] : INFO : Epoch 1953 | loss: 0.0311439 | val_loss: 0.0311591 | Time: 6545.91 ms [2022-03-13 13:39:37 main:574] : INFO : Epoch 1954 | loss: 0.0311425 | val_loss: 0.0311578 | Time: 6575.65 ms [2022-03-13 13:39:43 main:574] : INFO : Epoch 1955 | loss: 0.0311449 | val_loss: 0.0311607 | Time: 6555.94 ms [2022-03-13 13:39:50 main:574] : INFO : Epoch 1956 | loss: 0.0311444 | val_loss: 0.0311581 | Time: 6547.39 ms [2022-03-13 13:39:56 main:574] : INFO : Epoch 1957 | loss: 0.0311459 | val_loss: 0.0311596 | Time: 6538.29 ms [2022-03-13 13:40:03 main:574] : INFO : Epoch 1958 | loss: 0.0311426 | val_loss: 0.0311588 | Time: 6569.13 ms [2022-03-13 13:40:09 main:574] : INFO : Epoch 1959 | loss: 0.0311419 | val_loss: 0.0311583 | Time: 6589.2 ms [2022-03-13 13:40:16 main:574] : INFO : Epoch 1960 | loss: 0.0311422 | val_loss: 0.0311582 | Time: 6540.86 ms [2022-03-13 13:40:23 main:574] : INFO : Epoch 1961 | loss: 0.0311418 | val_loss: 0.0311626 | Time: 6528.34 ms [2022-03-13 13:40:29 main:574] : INFO : Epoch 1962 | loss: 0.0311426 | val_loss: 0.0311579 | Time: 6544.75 ms [2022-03-13 13:40:36 main:574] : INFO : Epoch 1963 | loss: 0.0311437 | val_loss: 0.0311577 | Time: 6540.51 ms [2022-03-13 13:40:42 main:574] : INFO : Epoch 1964 | loss: 0.0311428 | val_loss: 0.0311584 | Time: 6560.91 ms [2022-03-13 13:40:49 main:574] : INFO : Epoch 1965 | loss: 0.0311443 | val_loss: 0.0311629 | Time: 6556.21 ms [2022-03-13 13:40:56 main:574] : INFO : Epoch 1966 | loss: 0.0311439 | val_loss: 0.0311572 | Time: 6754.66 ms [2022-03-13 13:41:02 main:574] : INFO : Epoch 1967 | loss: 0.0311435 | val_loss: 0.0311658 | Time: 6711.44 ms [2022-03-13 13:41:09 main:574] : INFO : Epoch 1968 | loss: 0.0311426 | val_loss: 0.0311624 | Time: 6536.47 ms [2022-03-13 13:41:15 main:574] : INFO : Epoch 1969 | loss: 0.0311421 | val_loss: 0.0311585 | Time: 6557.71 ms [2022-03-13 13:41:22 main:574] : INFO : Epoch 1970 | loss: 0.0311422 | val_loss: 0.0311572 | Time: 6573.49 ms [2022-03-13 13:41:29 main:574] : INFO : Epoch 1971 | loss: 0.0311439 | val_loss: 0.0311609 | Time: 6535.84 ms [2022-03-13 13:41:35 main:574] : INFO : Epoch 1972 | loss: 0.0311448 | val_loss: 0.0311573 | Time: 6561.7 ms [2022-03-13 13:41:42 main:574] : INFO : Epoch 1973 | loss: 0.0311425 | val_loss: 0.0311565 | Time: 6549.87 ms [2022-03-13 13:41:48 main:574] : INFO : Epoch 1974 | loss: 0.0311403 | val_loss: 0.0311607 | Time: 6551.42 ms [2022-03-13 13:41:55 main:574] : INFO : Epoch 1975 | loss: 0.0311413 | val_loss: 0.0311584 | Time: 6547.15 ms [2022-03-13 13:42:01 main:574] : INFO : Epoch 1976 | loss: 0.0311424 | val_loss: 0.0311595 | Time: 6527.99 ms [2022-03-13 13:42:08 main:574] : INFO : Epoch 1977 | loss: 0.0311444 | val_loss: 0.0311593 | Time: 6598.39 ms [2022-03-13 13:42:14 main:574] : INFO : Epoch 1978 | loss: 0.0311422 | val_loss: 0.0311613 | Time: 6567.68 ms [2022-03-13 13:42:21 main:574] : INFO : Epoch 1979 | loss: 0.031142 | val_loss: 0.0311556 | Time: 6530.54 ms [2022-03-13 13:42:28 main:574] : INFO : Epoch 1980 | loss: 0.0311429 | val_loss: 0.0311632 | Time: 6545.32 ms [2022-03-13 13:42:34 main:574] : INFO : Epoch 1981 | loss: 0.0311463 | val_loss: 0.0311614 | Time: 6522.8 ms [2022-03-13 13:42:41 main:574] : INFO : Epoch 1982 | loss: 0.0311445 | val_loss: 0.0311593 | Time: 6540.53 ms [2022-03-13 13:42:47 main:574] : INFO : Epoch 1983 | loss: 0.0311411 | val_loss: 0.0311577 | Time: 6534.24 ms [2022-03-13 13:42:54 main:574] : INFO : Epoch 1984 | loss: 0.0311407 | val_loss: 0.0311582 | Time: 6542.84 ms [2022-03-13 13:43:00 main:574] : INFO : Epoch 1985 | loss: 0.0311409 | val_loss: 0.0311587 | Time: 6551.49 ms [2022-03-13 13:43:07 main:574] : INFO : Epoch 1986 | loss: 0.0311405 | val_loss: 0.0311595 | Time: 6547.16 ms [2022-03-13 13:43:13 main:574] : INFO : Epoch 1987 | loss: 0.0311415 | val_loss: 0.0311571 | Time: 6540.91 ms [2022-03-13 13:43:20 main:574] : INFO : Epoch 1988 | loss: 0.0311422 | val_loss: 0.0311664 | Time: 6537.86 ms [2022-03-13 13:43:28 main:574] : INFO : Epoch 1989 | loss: 0.0311546 | val_loss: 0.0311781 | Time: 8127.09 ms [2022-03-13 13:43:45 main:574] : INFO : Epoch 1990 | loss: 0.0391202 | val_loss: 0.0317665 | Time: 16945.6 ms [2022-03-13 13:43:52 main:574] : INFO : Epoch 1991 | loss: 0.0366732 | val_loss: 0.0327783 | Time: 6538.94 ms [2022-03-13 13:43:58 main:574] : INFO : Epoch 1992 | loss: 0.0324608 | val_loss: 0.0315334 | Time: 6534.37 ms [2022-03-13 13:44:05 main:574] : INFO : Epoch 1993 | loss: 0.0314119 | val_loss: 0.0313606 | Time: 6543.87 ms [2022-03-13 13:44:11 main:574] : INFO : Epoch 1994 | loss: 0.0312964 | val_loss: 0.0312648 | Time: 6547.3 ms [2022-03-13 13:44:18 main:574] : INFO : Epoch 1995 | loss: 0.0312591 | val_loss: 0.0312543 | Time: 6553.46 ms [2022-03-13 13:44:24 main:574] : INFO : Epoch 1996 | loss: 0.0312491 | val_loss: 0.0312436 | Time: 6550.55 ms [2022-03-13 13:44:31 main:574] : INFO : Epoch 1997 | loss: 0.0312398 | val_loss: 0.031232 | Time: 6598.6 ms [2022-03-13 13:44:38 main:574] : INFO : Epoch 1998 | loss: 0.0312263 | val_loss: 0.0312162 | Time: 6566.87 ms [2022-03-13 13:44:44 main:574] : INFO : Epoch 1999 | loss: 0.0312109 | val_loss: 0.0312024 | Time: 6548.77 ms [2022-03-13 13:44:51 main:574] : INFO : Epoch 2000 | loss: 0.031197 | val_loss: 0.0311951 | Time: 6578.32 ms [2022-03-13 13:44:57 main:574] : INFO : Epoch 2001 | loss: 0.0311914 | val_loss: 0.0311893 | Time: 6551.33 ms [2022-03-13 13:45:04 main:574] : INFO : Epoch 2002 | loss: 0.0311872 | val_loss: 0.0311888 | Time: 6538.95 ms [2022-03-13 13:45:10 main:574] : INFO : Epoch 2003 | loss: 0.0311849 | val_loss: 0.0311858 | Time: 6539.07 ms [2022-03-13 13:45:17 main:574] : INFO : Epoch 2004 | loss: 0.0311838 | val_loss: 0.0311841 | Time: 6582.02 ms [2022-03-13 13:45:23 main:574] : INFO : Epoch 2005 | loss: 0.0311831 | val_loss: 0.0311819 | Time: 6534.19 ms [2022-03-13 13:45:30 main:574] : INFO : Epoch 2006 | loss: 0.0311851 | val_loss: 0.0311821 | Time: 6528.55 ms [2022-03-13 13:45:37 main:574] : INFO : Epoch 2007 | loss: 0.0311807 | val_loss: 0.0311819 | Time: 6535.61 ms [2022-03-13 13:45:43 main:574] : INFO : Epoch 2008 | loss: 0.0311813 | val_loss: 0.031181 | Time: 6541.86 ms [2022-03-13 13:45:50 main:574] : INFO : Epoch 2009 | loss: 0.0311793 | val_loss: 0.0311794 | Time: 6545.33 ms [2022-03-13 13:45:56 main:574] : INFO : Epoch 2010 | loss: 0.0311786 | val_loss: 0.0311783 | Time: 6677.22 ms [2022-03-13 13:46:03 main:574] : INFO : Epoch 2011 | loss: 0.0311789 | val_loss: 0.0311783 | Time: 6575.28 ms [2022-03-13 13:46:10 main:574] : INFO : Epoch 2012 | loss: 0.0311784 | val_loss: 0.0311849 | Time: 6611.48 ms [2022-03-13 13:46:16 main:574] : INFO : Epoch 2013 | loss: 0.0311794 | val_loss: 0.0311775 | Time: 6562.21 ms [2022-03-13 13:46:23 main:574] : INFO : Epoch 2014 | loss: 0.0311787 | val_loss: 0.0311793 | Time: 6564.07 ms [2022-03-13 13:46:29 main:574] : INFO : Epoch 2015 | loss: 0.031178 | val_loss: 0.0311785 | Time: 6539.95 ms [2022-03-13 13:46:36 main:574] : INFO : Epoch 2016 | loss: 0.0311781 | val_loss: 0.0311821 | Time: 6574.94 ms [2022-03-13 13:46:42 main:574] : INFO : Epoch 2017 | loss: 0.0311778 | val_loss: 0.0311796 | Time: 6532.73 ms [2022-03-13 13:46:49 main:574] : INFO : Epoch 2018 | loss: 0.0311767 | val_loss: 0.0311777 | Time: 6558.26 ms [2022-03-13 13:46:55 main:574] : INFO : Epoch 2019 | loss: 0.0311757 | val_loss: 0.0311765 | Time: 6527.85 ms [2022-03-13 13:47:02 main:574] : INFO : Epoch 2020 | loss: 0.031175 | val_loss: 0.0311785 | Time: 6532.71 ms [2022-03-13 13:47:09 main:574] : INFO : Epoch 2021 | loss: 0.0311753 | val_loss: 0.0311765 | Time: 6567.19 ms [2022-03-13 13:47:15 main:574] : INFO : Epoch 2022 | loss: 0.0311756 | val_loss: 0.031177 | Time: 6517.74 ms [2022-03-13 13:47:22 main:574] : INFO : Epoch 2023 | loss: 0.0311754 | val_loss: 0.0311789 | Time: 6543.67 ms [2022-03-13 13:47:28 main:574] : INFO : Epoch 2024 | loss: 0.0311747 | val_loss: 0.0311766 | Time: 6560.67 ms [2022-03-13 13:47:35 main:574] : INFO : Epoch 2025 | loss: 0.0311752 | val_loss: 0.0311766 | Time: 6575.21 ms [2022-03-13 13:47:41 main:574] : INFO : Epoch 2026 | loss: 0.0311752 | val_loss: 0.0311776 | Time: 6526.68 ms [2022-03-13 13:47:48 main:574] : INFO : Epoch 2027 | loss: 0.0311751 | val_loss: 0.0311813 | Time: 6534.69 ms [2022-03-13 13:47:54 main:574] : INFO : Epoch 2028 | loss: 0.0311756 | val_loss: 0.0311767 | Time: 6549.86 ms [2022-03-13 13:48:01 main:574] : INFO : Epoch 2029 | loss: 0.031175 | val_loss: 0.0311764 | Time: 6554.08 ms [2022-03-13 13:48:07 main:574] : INFO : Epoch 2030 | loss: 0.0311738 | val_loss: 0.0311769 | Time: 6539.96 ms [2022-03-13 13:48:14 main:574] : INFO : Epoch 2031 | loss: 0.0311736 | val_loss: 0.0311783 | Time: 6539.17 ms [2022-03-13 13:48:21 main:574] : INFO : Epoch 2032 | loss: 0.0311747 | val_loss: 0.0311756 | Time: 6557.08 ms [2022-03-13 13:48:27 main:574] : INFO : Epoch 2033 | loss: 0.031173 | val_loss: 0.0311773 | Time: 6540.62 ms [2022-03-13 13:48:34 main:574] : INFO : Epoch 2034 | loss: 0.0311728 | val_loss: 0.0311758 | Time: 6552.56 ms [2022-03-13 13:48:40 main:574] : INFO : Epoch 2035 | loss: 0.0311726 | val_loss: 0.0311765 | Time: 6524.03 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GT 1030) [2022-03-13 14:17:44 main:435] : INFO : Set logging level to 1 [2022-03-13 14:17:44 main:441] : INFO : Running in BOINC Client mode [2022-03-13 14:17:44 main:444] : INFO : Resolving all filenames [2022-03-13 14:17:44 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1) [2022-03-13 14:17:44 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 1) [2022-03-13 14:17:44 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0) [2022-03-13 14:17:44 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1) [2022-03-13 14:17:44 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2022-03-13 14:17:44 main:472] : INFO : Dataset filename: dataset.hdf5 [2022-03-13 14:17:44 main:474] : INFO : Configuration: [2022-03-13 14:17:44 main:475] : INFO : Model type: GRU [2022-03-13 14:17:44 main:476] : INFO : Validation Loss Threshold: 0.0001 [2022-03-13 14:17:44 main:477] : INFO : Max Epochs: 2048 [2022-03-13 14:17:44 main:478] : INFO : Batch Size: 128 [2022-03-13 14:17:44 main:479] : INFO : Learning Rate: 0.01 [2022-03-13 14:17:44 main:480] : INFO : Patience: 10 [2022-03-13 14:17:44 main:481] : INFO : Hidden Width: 12 [2022-03-13 14:17:44 main:482] : INFO : # Recurrent Layers: 4 [2022-03-13 14:17:44 main:483] : INFO : # Backend Layers: 4 [2022-03-13 14:17:44 main:484] : INFO : # Threads: 1 [2022-03-13 14:17:44 main:486] : INFO : Preparing Dataset [2022-03-13 14:17:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory [2022-03-13 14:17:45 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory [2022-03-13 14:17:51 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2022-03-13 14:17:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory [2022-03-13 14:17:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory [2022-03-13 14:17:51 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2022-03-13 14:17:51 main:494] : INFO : Creating Model [2022-03-13 14:17:51 main:507] : INFO : Preparing config file [2022-03-13 14:17:51 main:511] : INFO : Found checkpoint, attempting to load... [2022-03-13 14:17:51 main:512] : INFO : Loading config [2022-03-13 14:17:51 main:514] : INFO : Loading state [2022-03-13 14:17:56 main:559] : INFO : Loading DataLoader into Memory [2022-03-13 14:17:56 main:562] : INFO : Starting Training [2022-03-13 14:18:16 main:574] : INFO : Epoch 2018 | loss: 0.0327296 | val_loss: 0.0314527 | Time: 19874.6 ms [2022-03-13 14:18:22 main:574] : INFO : Epoch 2019 | loss: 0.0313901 | val_loss: 0.0312773 | Time: 6529.52 ms [2022-03-13 14:18:29 main:574] : INFO : Epoch 2020 | loss: 0.0312514 | val_loss: 0.0312391 | Time: 6538.48 ms [2022-03-13 14:18:35 main:574] : INFO : Epoch 2021 | loss: 0.0312292 | val_loss: 0.031219 | Time: 6548.31 ms [2022-03-13 14:18:42 main:574] : INFO : Epoch 2022 | loss: 0.0312166 | val_loss: 0.0312105 | Time: 6547.78 ms [2022-03-13 14:18:49 main:574] : INFO : Epoch 2023 | loss: 0.0312543 | val_loss: 0.0312037 | Time: 6536.57 ms [2022-03-13 14:18:55 main:574] : INFO : Epoch 2024 | loss: 0.0312121 | val_loss: 0.031195 | Time: 6532.77 ms [2022-03-13 14:19:02 main:574] : INFO : Epoch 2025 | loss: 0.031189 | val_loss: 0.0311899 | Time: 6540.19 ms [2022-03-13 14:19:08 main:574] : INFO : Epoch 2026 | loss: 0.0311833 | val_loss: 0.0311858 | Time: 6559.35 ms [2022-03-13 14:19:15 main:574] : INFO : Epoch 2027 | loss: 0.0311809 | val_loss: 0.0311882 | Time: 6551.58 ms [2022-03-13 14:19:21 main:574] : INFO : Epoch 2028 | loss: 0.0311878 | val_loss: 0.0312067 | Time: 6540.35 ms [2022-03-13 14:19:28 main:574] : INFO : Epoch 2029 | loss: 0.0311904 | val_loss: 0.0311935 | Time: 6553.23 ms [2022-03-13 14:19:34 main:574] : INFO : Epoch 2030 | loss: 0.0311846 | val_loss: 0.0311853 | Time: 6575.8 ms [2022-03-13 14:19:41 main:574] : INFO : Epoch 2031 | loss: 0.0311843 | val_loss: 0.0311852 | Time: 6556.76 ms [2022-03-13 14:19:48 main:574] : INFO : Epoch 2032 | loss: 0.031186 | val_loss: 0.0311824 | Time: 6549.94 ms [2022-03-13 14:19:54 main:574] : INFO : Epoch 2033 | loss: 0.0311769 | val_loss: 0.0311795 | Time: 6559.44 ms [2022-03-13 14:20:01 main:574] : INFO : Epoch 2034 | loss: 0.0311751 | val_loss: 0.0311779 | Time: 6554.76 ms [2022-03-13 14:20:07 main:574] : INFO : Epoch 2035 | loss: 0.0311654 | val_loss: 0.0311686 | Time: 6574.13 ms [2022-03-13 14:20:14 main:574] : INFO : Epoch 2036 | loss: 0.0311633 | val_loss: 0.0311771 | Time: 6549.97 ms [2022-03-13 14:20:20 main:574] : INFO : Epoch 2037 | loss: 0.0311598 | val_loss: 0.0311697 | Time: 6549.54 ms [2022-03-13 14:20:27 main:574] : INFO : Epoch 2038 | loss: 0.031154 | val_loss: 0.031159 | Time: 6579.56 ms [2022-03-13 14:20:34 main:574] : INFO : Epoch 2039 | loss: 0.0311527 | val_loss: 0.0311598 | Time: 6559.28 ms [2022-03-13 14:20:40 main:574] : INFO : Epoch 2040 | loss: 0.0311505 | val_loss: 0.0311634 | Time: 6602.9 ms [2022-03-13 14:20:47 main:574] : INFO : Epoch 2041 | loss: 0.0311546 | val_loss: 0.0311616 | Time: 6551.66 ms [2022-03-13 14:20:53 main:574] : INFO : Epoch 2042 | loss: 0.0311531 | val_loss: 0.0311604 | Time: 6537.04 ms [2022-03-13 14:21:00 main:574] : INFO : Epoch 2043 | loss: 0.0311517 | val_loss: 0.0311663 | Time: 6813.17 ms [2022-03-13 14:21:07 main:574] : INFO : Epoch 2044 | loss: 0.0311516 | val_loss: 0.0311677 | Time: 6549.8 ms [2022-03-13 14:21:13 main:574] : INFO : Epoch 2045 | loss: 0.031155 | val_loss: 0.0311794 | Time: 6553.25 ms [2022-03-13 14:21:20 main:574] : INFO : Epoch 2046 | loss: 0.0311527 | val_loss: 0.0311645 | Time: 6548.5 ms [2022-03-13 14:21:26 main:574] : INFO : Epoch 2047 | loss: 0.0311528 | val_loss: 0.0311651 | Time: 6572.94 ms [2022-03-13 14:21:33 main:574] : INFO : Epoch 2048 | loss: 0.0311507 | val_loss: 0.0311613 | Time: 6544.09 ms [2022-03-13 14:21:33 main:597] : INFO : Saving trained model to model-final.pt, val_loss 0.0311613 [2022-03-13 14:21:33 main:603] : INFO : Saving end state to config to file [2022-03-13 14:21:33 main:608] : INFO : Success, exiting.. 14:21:33 (11884): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)