Task 2824973
| Name | rand_automata_0006-1604982737-17426-0_3 |
| Workunit | 1263441 |
| Created | 10 Nov 2020, 23:31:35 UTC |
| Sent | 10 Nov 2020, 23:39:08 UTC |
| Report deadline | 11 Nov 2020, 22:09:08 UTC |
| Received | 11 Nov 2020, 3:47:13 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 3241 |
| Run time | 28 min 48 sec |
| CPU time | 26 min 52 sec |
| Validate state | Valid |
| Credit | 959.40 |
| Device peak FLOPS | 9,068.06 GFLOPS |
| Application version | Machine Learning Dataset Generator (test) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.95 GB |
| Peak swap size | 3.92 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version> <![CDATA[ <stderr_txt> Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce RTX 2070 SUPER) [2020-11-10 19:04:16 main:435] : INFO : Set logging level to 1 [2020-11-10 19:04:16 main:441] : INFO : Running in BOINC Client mode [2020-11-10 19:04:16 main:444] : INFO : Resolving all filenames [2020-11-10 19:04:16 main:452] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 (exists = 1) [2020-11-10 19:04:16 main:452] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-1604982737-17426-0_3_r231725764_1 (exists = 0) [2020-11-10 19:04:16 main:452] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-1604982737-17426-0_3_r231725764_0 (exists = 0) [2020-11-10 19:04:16 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2020-11-10 19:04:16 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0) [2020-11-10 19:04:16 main:472] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 [2020-11-10 19:04:16 main:474] : INFO : Configuration: [2020-11-10 19:04:16 main:475] : INFO : Model type: LSTM [2020-11-10 19:04:16 main:476] : INFO : Validation Loss Threshold: 0.0001 [2020-11-10 19:04:16 main:477] : INFO : Max Epochs: 192 [2020-11-10 19:04:16 main:478] : INFO : Batch Size: 32 [2020-11-10 19:04:16 main:479] : INFO : Learning Rate: 0.001 [2020-11-10 19:04:16 main:480] : INFO : Patience: 10 [2020-11-10 19:04:16 main:481] : INFO : Hidden Width: 64 [2020-11-10 19:04:16 main:482] : INFO : # Recurrent Layers: 4 [2020-11-10 19:04:16 main:483] : INFO : # Backend Layers: 2 [2020-11-10 19:04:16 main:484] : INFO : # Threads: 1 [2020-11-10 19:04:16 main:486] : INFO : Preparing Dataset [2020-11-10 19:04:16 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:04:17 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:04:18 load:106] : INFO : Successfully loaded dataset of 4096 examples into memory. [2020-11-10 19:04:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:04:18 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:04:18 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2020-11-10 19:04:18 main:494] : INFO : Creating Model [2020-11-10 19:04:18 main:507] : INFO : Preparing config file [2020-11-10 19:04:18 main:519] : INFO : Creating new config file [2020-11-10 19:04:19 main:559] : INFO : Loading DataLoader into Memory [2020-11-10 19:04:19 main:562] : INFO : Starting Training [2020-11-10 19:04:28 main:574] : INFO : Epoch 1 | loss: 0.0665424 | val_loss: 0.0659839 | Time: 9153.62 ms [2020-11-10 19:04:37 main:574] : INFO : Epoch 2 | loss: 0.0659235 | val_loss: 0.0654512 | Time: 8830.92 ms [2020-11-10 19:04:45 main:574] : INFO : Epoch 3 | loss: 0.0644368 | val_loss: 0.0629748 | Time: 8749.19 ms [2020-11-10 19:04:54 main:574] : INFO : Epoch 4 | loss: 0.0613431 | val_loss: 0.0598065 | Time: 8784.4 ms [2020-11-10 19:05:03 main:574] : INFO : Epoch 5 | loss: 0.0587269 | val_loss: 0.0578159 | Time: 8815.4 ms [2020-11-10 19:05:12 main:574] : INFO : Epoch 6 | loss: 0.057063 | val_loss: 0.0563451 | Time: 8832.88 ms [2020-11-10 19:05:21 main:574] : INFO : Epoch 7 | loss: 0.0556938 | val_loss: 0.0550238 | Time: 8783.16 ms [2020-11-10 19:05:29 main:574] : INFO : Epoch 8 | loss: 0.0544028 | val_loss: 0.0537385 | Time: 8759.75 ms [2020-11-10 19:05:38 main:574] : INFO : Epoch 9 | loss: 0.0529999 | val_loss: 0.0522143 | Time: 8811.09 ms [2020-11-10 19:05:47 main:574] : INFO : Epoch 10 | loss: 0.0515301 | val_loss: 0.0507755 | Time: 8818.18 ms [2020-11-10 19:05:56 main:574] : INFO : Epoch 11 | loss: 0.0498107 | val_loss: 0.0491504 | Time: 8786.86 ms [2020-11-10 19:06:04 main:574] : INFO : Epoch 12 | loss: 0.0478794 | val_loss: 0.0469668 | Time: 8732.22 ms [2020-11-10 19:06:14 main:574] : INFO : Epoch 13 | loss: 0.0456115 | val_loss: 0.0445768 | Time: 9115.77 ms [2020-11-10 19:06:22 main:574] : INFO : Epoch 14 | loss: 0.0431309 | val_loss: 0.0426988 | Time: 8792.48 ms [2020-11-10 19:06:31 main:574] : INFO : Epoch 15 | loss: 0.0404853 | val_loss: 0.0403963 | Time: 8778.51 ms [2020-11-10 19:06:40 main:574] : INFO : Epoch 16 | loss: 0.0379437 | val_loss: 0.0392557 | Time: 8756.25 ms [2020-11-10 19:06:49 main:574] : INFO : Epoch 17 | loss: 0.0347199 | val_loss: 0.0347402 | Time: 8738.67 ms [2020-11-10 19:06:57 main:574] : INFO : Epoch 18 | loss: 0.0325724 | val_loss: 0.0331857 | Time: 8756.24 ms [2020-11-10 19:07:06 main:574] : INFO : Epoch 19 | loss: 0.0312754 | val_loss: 0.03139 | Time: 8757.77 ms [2020-11-10 19:07:15 main:574] : INFO : Epoch 20 | loss: 0.0321422 | val_loss: 0.0277782 | Time: 8774.93 ms [2020-11-10 19:07:24 main:574] : INFO : Epoch 21 | loss: 0.0254984 | val_loss: 0.0273538 | Time: 8843.02 ms [2020-11-10 19:07:33 main:574] : INFO : Epoch 22 | loss: 0.0284958 | val_loss: 0.0313242 | Time: 8754.53 ms [2020-11-10 19:07:42 main:574] : INFO : Epoch 23 | loss: 0.0292286 | val_loss: 0.0304843 | Time: 8866.6 ms [2020-11-10 19:07:50 main:574] : INFO : Epoch 24 | loss: 0.0304473 | val_loss: 0.0234686 | Time: 8947.09 ms [2020-11-10 19:07:59 main:574] : INFO : Epoch 25 | loss: 0.0286827 | val_loss: 0.0320742 | Time: 8748.26 ms [2020-11-10 19:08:08 main:574] : INFO : Epoch 26 | loss: 0.0271466 | val_loss: 0.0269364 | Time: 8804.19 ms [2020-11-10 19:08:17 main:574] : INFO : Epoch 27 | loss: 0.0259868 | val_loss: 0.0231169 | Time: 8852.23 ms [2020-11-10 19:08:26 main:574] : INFO : Epoch 28 | loss: 0.017272 | val_loss: 0.0156902 | Time: 8749.58 ms [2020-11-10 19:08:34 main:574] : INFO : Epoch 29 | loss: 0.0160348 | val_loss: 0.0284488 | Time: 8793.28 ms [2020-11-10 19:08:43 main:574] : INFO : Epoch 30 | loss: 0.0258458 | val_loss: 0.0198575 | Time: 8767.15 ms [2020-11-10 19:08:52 main:574] : INFO : Epoch 31 | loss: 0.0152789 | val_loss: 0.015478 | Time: 8886.07 ms [2020-11-10 19:09:01 main:574] : INFO : Epoch 32 | loss: 0.0147064 | val_loss: 0.012716 | Time: 8764.77 ms [2020-11-10 19:09:10 main:574] : INFO : Epoch 33 | loss: 0.0115314 | val_loss: 0.0119036 | Time: 8926.99 ms [2020-11-10 19:09:19 main:574] : INFO : Epoch 34 | loss: 0.0111836 | val_loss: 0.0124837 | Time: 8933.36 ms [2020-11-10 19:09:28 main:574] : INFO : Epoch 35 | loss: 0.0131674 | val_loss: 0.0112609 | Time: 9024.99 ms [2020-11-10 19:09:37 main:574] : INFO : Epoch 36 | loss: 0.0101534 | val_loss: 0.010566 | Time: 8790.62 ms [2020-11-10 19:09:45 main:574] : INFO : Epoch 37 | loss: 0.0191716 | val_loss: 0.0167325 | Time: 8778.66 ms [2020-11-10 19:09:54 main:574] : INFO : Epoch 38 | loss: 0.0117911 | val_loss: 0.0109697 | Time: 8801.04 ms [2020-11-10 19:10:03 main:574] : INFO : Epoch 39 | loss: 0.0100858 | val_loss: 0.0101677 | Time: 8801.78 ms [2020-11-10 19:10:12 main:574] : INFO : Epoch 40 | loss: 0.00917967 | val_loss: 0.00957559 | Time: 8937.89 ms [2020-11-10 19:10:21 main:574] : INFO : Epoch 41 | loss: 0.00878145 | val_loss: 0.00904134 | Time: 8817.76 ms [2020-11-10 19:10:30 main:574] : INFO : Epoch 42 | loss: 0.00828208 | val_loss: 0.00871625 | Time: 8917.39 ms [2020-11-10 19:10:39 main:574] : INFO : Epoch 43 | loss: 0.0358313 | val_loss: 0.0312988 | Time: 9031.01 ms [2020-11-10 19:10:48 main:574] : INFO : Epoch 44 | loss: 0.0339457 | val_loss: 0.0282608 | Time: 8799.45 ms [2020-11-10 19:10:57 main:574] : INFO : Epoch 45 | loss: 0.0195762 | val_loss: 0.0144349 | Time: 8978.69 ms [2020-11-10 19:11:06 main:574] : INFO : Epoch 46 | loss: 0.0115755 | val_loss: 0.0205397 | Time: 8905.32 ms [2020-11-10 19:11:14 main:574] : INFO : Epoch 47 | loss: 0.0107908 | val_loss: 0.00976483 | Time: 8889.13 ms [2020-11-10 19:11:23 main:574] : INFO : Epoch 48 | loss: 0.00864075 | val_loss: 0.00897597 | Time: 9015.42 ms [2020-11-10 19:11:32 main:574] : INFO : Epoch 49 | loss: 0.0101443 | val_loss: 0.0123166 | Time: 8764.03 ms [2020-11-10 19:11:41 main:574] : INFO : Epoch 50 | loss: 0.00896177 | val_loss: 0.00839482 | Time: 8772.34 ms [2020-11-10 19:11:50 main:574] : INFO : Epoch 51 | loss: 0.00782784 | val_loss: 0.00800005 | Time: 8748.77 ms [2020-11-10 19:11:59 main:574] : INFO : Epoch 52 | loss: 0.00747238 | val_loss: 0.00769966 | Time: 8769.51 ms [2020-11-10 19:12:07 main:574] : INFO : Epoch 53 | loss: 0.00723166 | val_loss: 0.00748451 | Time: 8892.5 ms [2020-11-10 19:12:16 main:574] : INFO : Epoch 54 | loss: 0.00701 | val_loss: 0.0072665 | Time: 8817.44 ms [2020-11-10 19:12:25 main:574] : INFO : Epoch 55 | loss: 0.00681637 | val_loss: 0.0070796 | Time: 8870.89 ms [2020-11-10 19:12:34 main:574] : INFO : Epoch 56 | loss: 0.0066411 | val_loss: 0.00689968 | Time: 8847.7 ms [2020-11-10 19:12:43 main:574] : INFO : Epoch 57 | loss: 0.00647654 | val_loss: 0.00670722 | Time: 8848.11 ms [2020-11-10 19:12:52 main:574] : INFO : Epoch 58 | loss: 0.00631723 | val_loss: 0.00654023 | Time: 8845.05 ms [2020-11-10 19:13:01 main:574] : INFO : Epoch 59 | loss: 0.0061633 | val_loss: 0.00638275 | Time: 8845.74 ms [2020-11-10 19:13:09 main:574] : INFO : Epoch 60 | loss: 0.00601328 | val_loss: 0.00622309 | Time: 8826.98 ms [2020-11-10 19:13:18 main:574] : INFO : Epoch 61 | loss: 0.005861 | val_loss: 0.00606503 | Time: 8873.27 ms [2020-11-10 19:13:27 main:574] : INFO : Epoch 62 | loss: 0.00571276 | val_loss: 0.00586081 | Time: 8884.28 ms [2020-11-10 19:13:36 main:574] : INFO : Epoch 63 | loss: 0.00556902 | val_loss: 0.00570759 | Time: 9162.19 ms [2020-11-10 19:13:46 main:574] : INFO : Epoch 64 | loss: 0.00542495 | val_loss: 0.00555849 | Time: 9514.82 ms [2020-11-10 19:13:55 main:574] : INFO : Epoch 65 | loss: 0.00528189 | val_loss: 0.00541246 | Time: 9154.68 ms [2020-11-10 19:14:04 main:574] : INFO : Epoch 66 | loss: 0.0051412 | val_loss: 0.00526703 | Time: 8913.82 ms [2020-11-10 19:14:13 main:574] : INFO : Epoch 67 | loss: 0.00500044 | val_loss: 0.00512253 | Time: 8812.19 ms [2020-11-10 19:14:22 main:574] : INFO : Epoch 68 | loss: 0.00486156 | val_loss: 0.0049793 | Time: 8847.76 ms [2020-11-10 19:14:30 main:574] : INFO : Epoch 69 | loss: 0.00472376 | val_loss: 0.00484266 | Time: 8805.38 ms [2020-11-10 19:14:39 main:574] : INFO : Epoch 70 | loss: 0.00458714 | val_loss: 0.00469503 | Time: 8825.78 ms [2020-11-10 19:14:48 main:574] : INFO : Epoch 71 | loss: 0.00445143 | val_loss: 0.00451724 | Time: 8810.35 ms [2020-11-10 19:14:57 main:574] : INFO : Epoch 72 | loss: 0.00431835 | val_loss: 0.00436909 | Time: 8813.33 ms [2020-11-10 19:15:06 main:574] : INFO : Epoch 73 | loss: 0.00418593 | val_loss: 0.00423422 | Time: 8830.71 ms [2020-11-10 19:15:15 main:574] : INFO : Epoch 74 | loss: 0.00405538 | val_loss: 0.00409989 | Time: 8836.88 ms [2020-11-10 19:15:24 main:574] : INFO : Epoch 75 | loss: 0.00392692 | val_loss: 0.00396902 | Time: 8841.75 ms [2020-11-10 19:15:32 main:574] : INFO : Epoch 76 | loss: 0.00380069 | val_loss: 0.00383883 | Time: 8871.74 ms [2020-11-10 19:15:41 main:574] : INFO : Epoch 77 | loss: 0.00367649 | val_loss: 0.00371197 | Time: 8875.16 ms [2020-11-10 19:15:50 main:574] : INFO : Epoch 78 | loss: 0.00355478 | val_loss: 0.00358482 | Time: 8850.15 ms [2020-11-10 19:15:59 main:574] : INFO : Epoch 79 | loss: 0.0034349 | val_loss: 0.00346145 | Time: 8871.64 ms [2020-11-10 19:16:08 main:574] : INFO : Epoch 80 | loss: 0.00331668 | val_loss: 0.00328894 | Time: 8879.49 ms [2020-11-10 19:16:17 main:574] : INFO : Epoch 81 | loss: 0.00320198 | val_loss: 0.00317587 | Time: 8886.74 ms [2020-11-10 19:16:26 main:574] : INFO : Epoch 82 | loss: 0.00308789 | val_loss: 0.00305725 | Time: 8834.55 ms [2020-11-10 19:16:34 main:574] : INFO : Epoch 83 | loss: 0.00297725 | val_loss: 0.00294836 | Time: 8793.83 ms [2020-11-10 19:16:43 main:574] : INFO : Epoch 84 | loss: 0.0028692 | val_loss: 0.00283906 | Time: 8848.45 ms [2020-11-10 19:16:52 main:574] : INFO : Epoch 85 | loss: 0.00276444 | val_loss: 0.00273283 | Time: 8834.85 ms [2020-11-10 19:17:01 main:574] : INFO : Epoch 86 | loss: 0.00266109 | val_loss: 0.0026333 | Time: 8809.86 ms [2020-11-10 19:17:10 main:574] : INFO : Epoch 87 | loss: 0.00256039 | val_loss: 0.00253223 | Time: 8798.46 ms [2020-11-10 19:17:19 main:574] : INFO : Epoch 88 | loss: 0.00246283 | val_loss: 0.00243332 | Time: 8767.97 ms [2020-11-10 19:17:27 main:574] : INFO : Epoch 89 | loss: 0.00236818 | val_loss: 0.00233886 | Time: 8796.15 ms [2020-11-10 19:17:37 main:574] : INFO : Epoch 90 | loss: 0.00227608 | val_loss: 0.00224821 | Time: 9187.86 ms [2020-11-10 19:17:46 main:574] : INFO : Epoch 91 | loss: 0.00218636 | val_loss: 0.00215876 | Time: 8995.36 ms [2020-11-10 19:17:55 main:574] : INFO : Epoch 92 | loss: 0.00210017 | val_loss: 0.00207355 | Time: 8890.33 ms [2020-11-10 19:18:03 main:574] : INFO : Epoch 93 | loss: 0.00201546 | val_loss: 0.00198832 | Time: 8879.33 ms [2020-11-10 19:18:12 main:574] : INFO : Epoch 94 | loss: 0.00193443 | val_loss: 0.00191215 | Time: 8855.34 ms [2020-11-10 19:18:21 main:574] : INFO : Epoch 95 | loss: 0.00185576 | val_loss: 0.00183137 | Time: 8881.85 ms [2020-11-10 19:18:30 main:574] : INFO : Epoch 96 | loss: 0.00178004 | val_loss: 0.00175685 | Time: 8933 ms [2020-11-10 19:18:39 main:574] : INFO : Epoch 97 | loss: 0.00170644 | val_loss: 0.00168422 | Time: 8934.76 ms [2020-11-10 19:18:48 main:574] : INFO : Epoch 98 | loss: 0.00163465 | val_loss: 0.00161425 | Time: 8905.31 ms [2020-11-10 19:18:57 main:574] : INFO : Epoch 99 | loss: 0.00156615 | val_loss: 0.00154454 | Time: 8925.73 ms [2020-11-10 19:19:06 main:574] : INFO : Epoch 100 | loss: 0.00149928 | val_loss: 0.00147887 | Time: 8943.76 ms [2020-11-10 19:19:15 main:574] : INFO : Epoch 101 | loss: 0.00143528 | val_loss: 0.00141649 | Time: 8893.46 ms [2020-11-10 19:19:24 main:574] : INFO : Epoch 102 | loss: 0.00137345 | val_loss: 0.0013552 | Time: 8894.4 ms [2020-11-10 19:19:33 main:574] : INFO : Epoch 103 | loss: 0.00131421 | val_loss: 0.00129821 | Time: 8897.34 ms [2020-11-10 19:19:41 main:574] : INFO : Epoch 104 | loss: 0.00125776 | val_loss: 0.00124153 | Time: 8923.42 ms [2020-11-10 19:19:50 main:574] : INFO : Epoch 105 | loss: 0.00120356 | val_loss: 0.00118728 | Time: 8937.08 ms [2020-11-10 19:19:59 main:574] : INFO : Epoch 106 | loss: 0.00115118 | val_loss: 0.00113778 | Time: 8893.37 ms [2020-11-10 19:20:08 main:574] : INFO : Epoch 107 | loss: 0.00110117 | val_loss: 0.00108658 | Time: 8930.11 ms [2020-11-10 19:20:17 main:574] : INFO : Epoch 108 | loss: 0.00105292 | val_loss: 0.00103822 | Time: 8842.31 ms [2020-11-10 19:20:26 main:574] : INFO : Epoch 109 | loss: 0.00100694 | val_loss: 0.000992698 | Time: 8836.87 ms [2020-11-10 19:20:35 main:574] : INFO : Epoch 110 | loss: 0.000963055 | val_loss: 0.000950299 | Time: 8816.66 ms [2020-11-10 19:20:44 main:574] : INFO : Epoch 111 | loss: 0.000920326 | val_loss: 0.000908313 | Time: 8771.45 ms [2020-11-10 19:20:52 main:574] : INFO : Epoch 112 | loss: 0.000879229 | val_loss: 0.00087046 | Time: 8803.14 ms [2020-11-10 19:21:01 main:574] : INFO : Epoch 113 | loss: 0.000840702 | val_loss: 0.00083163 | Time: 8781.86 ms [2020-11-10 19:21:10 main:574] : INFO : Epoch 114 | loss: 0.000803598 | val_loss: 0.000796941 | Time: 8801.46 ms [2020-11-10 19:21:19 main:574] : INFO : Epoch 115 | loss: 0.00076776 | val_loss: 0.000760125 | Time: 8794.12 ms [2020-11-10 19:21:28 main:574] : INFO : Epoch 116 | loss: 0.000733457 | val_loss: 0.000726807 | Time: 8774.2 ms [2020-11-10 19:21:36 main:574] : INFO : Epoch 117 | loss: 0.000701849 | val_loss: 0.0006951 | Time: 8786.84 ms [2020-11-10 19:21:45 main:574] : INFO : Epoch 118 | loss: 0.00067061 | val_loss: 0.000663205 | Time: 8759.68 ms [2020-11-10 19:21:54 main:574] : INFO : Epoch 119 | loss: 0.000641066 | val_loss: 0.000633524 | Time: 8772.03 ms [2020-11-10 19:22:03 main:574] : INFO : Epoch 120 | loss: 0.000612765 | val_loss: 0.000609708 | Time: 8762.69 ms [2020-11-10 19:22:12 main:574] : INFO : Epoch 121 | loss: 0.000586764 | val_loss: 0.000578934 | Time: 8765.37 ms [2020-11-10 19:22:20 main:574] : INFO : Epoch 122 | loss: 0.000561648 | val_loss: 0.000554474 | Time: 8739.06 ms [2020-11-10 19:22:29 main:574] : INFO : Epoch 123 | loss: 0.000536691 | val_loss: 0.000531904 | Time: 8762.61 ms [2020-11-10 19:22:38 main:574] : INFO : Epoch 124 | loss: 0.000514351 | val_loss: 0.000509473 | Time: 8776.18 ms [2020-11-10 19:22:47 main:574] : INFO : Epoch 125 | loss: 0.000491907 | val_loss: 0.000488371 | Time: 8768.29 ms [2020-11-10 19:22:55 main:574] : INFO : Epoch 126 | loss: 0.00047184 | val_loss: 0.000466107 | Time: 8753.83 ms [2020-11-10 19:23:04 main:574] : INFO : Epoch 127 | loss: 0.000451588 | val_loss: 0.000447314 | Time: 8761.1 ms [2020-11-10 19:23:13 main:574] : INFO : Epoch 128 | loss: 0.000433592 | val_loss: 0.000428995 | Time: 8760.36 ms [2020-11-10 19:23:22 main:574] : INFO : Epoch 129 | loss: 0.000415891 | val_loss: 0.000411942 | Time: 8819.23 ms [2020-11-10 19:23:31 main:574] : INFO : Epoch 130 | loss: 0.000398841 | val_loss: 0.000394838 | Time: 8807.79 ms [2020-11-10 19:23:39 main:574] : INFO : Epoch 131 | loss: 0.000383182 | val_loss: 0.000380019 | Time: 8762.37 ms [2020-11-10 19:23:48 main:574] : INFO : Epoch 132 | loss: 0.000368139 | val_loss: 0.00036645 | Time: 8708.24 ms [2020-11-10 19:23:57 main:574] : INFO : Epoch 133 | loss: 0.000353607 | val_loss: 0.000349581 | Time: 8776.98 ms [2020-11-10 19:24:06 main:574] : INFO : Epoch 134 | loss: 0.000340451 | val_loss: 0.000336625 | Time: 8793.63 ms [2020-11-10 19:24:14 main:574] : INFO : Epoch 135 | loss: 0.000327643 | val_loss: 0.000324682 | Time: 8785.59 ms [2020-11-10 19:24:23 main:574] : INFO : Epoch 136 | loss: 0.000314919 | val_loss: 0.000312854 | Time: 8776.66 ms [2020-11-10 19:24:32 main:574] : INFO : Epoch 137 | loss: 0.000303676 | val_loss: 0.00030138 | Time: 8867.7 ms [2020-11-10 19:24:42 main:574] : INFO : Epoch 138 | loss: 0.000292326 | val_loss: 0.000291456 | Time: 9724.07 ms [2020-11-10 19:24:51 main:574] : INFO : Epoch 139 | loss: 0.00028204 | val_loss: 0.000280168 | Time: 8881.68 ms [2020-11-10 19:25:00 main:574] : INFO : Epoch 140 | loss: 0.000272091 | val_loss: 0.000270226 | Time: 8813.8 ms [2020-11-10 19:25:08 main:574] : INFO : Epoch 141 | loss: 0.000262425 | val_loss: 0.000260321 | Time: 8791.76 ms [2020-11-10 19:25:17 main:574] : INFO : Epoch 142 | loss: 0.000253311 | val_loss: 0.00025249 | Time: 8824.43 ms [2020-11-10 19:25:26 main:574] : INFO : Epoch 143 | loss: 0.000245267 | val_loss: 0.000244804 | Time: 8807.79 ms [2020-11-10 19:25:35 main:574] : INFO : Epoch 144 | loss: 0.000236985 | val_loss: 0.000236378 | Time: 8817.26 ms [2020-11-10 19:25:44 main:574] : INFO : Epoch 145 | loss: 0.000229563 | val_loss: 0.000228604 | Time: 8908.58 ms [2020-11-10 19:25:53 main:574] : INFO : Epoch 146 | loss: 0.000221943 | val_loss: 0.000223829 | Time: 8778.66 ms [2020-11-10 19:26:01 main:574] : INFO : Epoch 147 | loss: 0.000215408 | val_loss: 0.000214761 | Time: 8758.54 ms [2020-11-10 19:26:10 main:574] : INFO : Epoch 148 | loss: 0.000208777 | val_loss: 0.000208178 | Time: 8770.1 ms [2020-11-10 19:26:19 main:574] : INFO : Epoch 149 | loss: 0.000202794 | val_loss: 0.00020295 | Time: 8771.62 ms [2020-11-10 19:26:29 main:574] : INFO : Epoch 150 | loss: 0.000196431 | val_loss: 0.0001961 | Time: 9862.37 ms [2020-11-10 19:26:39 main:574] : INFO : Epoch 151 | loss: 0.000190924 | val_loss: 0.000191574 | Time: 10578.1 ms [2020-11-10 19:26:50 main:574] : INFO : Epoch 152 | loss: 0.000185638 | val_loss: 0.000185084 | Time: 10192 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce RTX 2070 SUPER) [2020-11-10 19:27:02 main:435] : INFO : Set logging level to 1 [2020-11-10 19:27:02 main:441] : INFO : Running in BOINC Client mode [2020-11-10 19:27:02 main:444] : INFO : Resolving all filenames [2020-11-10 19:27:02 main:452] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 (exists = 1) [2020-11-10 19:27:02 main:452] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-1604982737-17426-0_3_r231725764_1 (exists = 1) [2020-11-10 19:27:02 main:452] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-1604982737-17426-0_3_r231725764_0 (exists = 0) [2020-11-10 19:27:02 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2020-11-10 19:27:02 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2020-11-10 19:27:02 main:472] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 [2020-11-10 19:27:02 main:474] : INFO : Configuration: [2020-11-10 19:27:02 main:475] : INFO : Model type: LSTM [2020-11-10 19:27:02 main:476] : INFO : Validation Loss Threshold: 0.0001 [2020-11-10 19:27:02 main:477] : INFO : Max Epochs: 192 [2020-11-10 19:27:02 main:478] : INFO : Batch Size: 32 [2020-11-10 19:27:02 main:479] : INFO : Learning Rate: 0.001 [2020-11-10 19:27:02 main:480] : INFO : Patience: 10 [2020-11-10 19:27:02 main:481] : INFO : Hidden Width: 64 [2020-11-10 19:27:02 main:482] : INFO : # Recurrent Layers: 4 [2020-11-10 19:27:02 main:483] : INFO : # Backend Layers: 2 [2020-11-10 19:27:02 main:484] : INFO : # Threads: 1 [2020-11-10 19:27:02 main:486] : INFO : Preparing Dataset [2020-11-10 19:27:02 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:27:02 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce RTX 2070 SUPER) [2020-11-10 19:40:51 main:435] : INFO : Set logging level to 1 [2020-11-10 19:40:51 main:441] : INFO : Running in BOINC Client mode [2020-11-10 19:40:51 main:444] : INFO : Resolving all filenames [2020-11-10 19:40:51 main:452] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 (exists = 1) [2020-11-10 19:40:51 main:452] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-1604982737-17426-0_3_r231725764_1 (exists = 1) [2020-11-10 19:40:51 main:452] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-1604982737-17426-0_3_r231725764_0 (exists = 0) [2020-11-10 19:40:51 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2020-11-10 19:40:51 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2020-11-10 19:40:51 main:472] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 [2020-11-10 19:40:51 main:474] : INFO : Configuration: [2020-11-10 19:40:51 main:475] : INFO : Model type: LSTM [2020-11-10 19:40:51 main:476] : INFO : Validation Loss Threshold: 0.0001 [2020-11-10 19:40:51 main:477] : INFO : Max Epochs: 192 [2020-11-10 19:40:51 main:478] : INFO : Batch Size: 32 [2020-11-10 19:40:51 main:479] : INFO : Learning Rate: 0.001 [2020-11-10 19:40:51 main:480] : INFO : Patience: 10 [2020-11-10 19:40:51 main:481] : INFO : Hidden Width: 64 [2020-11-10 19:40:51 main:482] : INFO : # Recurrent Layers: 4 [2020-11-10 19:40:51 main:483] : INFO : # Backend Layers: 2 [2020-11-10 19:40:51 main:484] : INFO : # Threads: 1 [2020-11-10 19:40:51 main:486] : INFO : Preparing Dataset [2020-11-10 19:40:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:40:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:40:52 load:106] : INFO : Successfully loaded dataset of 4096 examples into memory. [2020-11-10 19:40:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:40:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0006-train-val-dataset.hdf5 into memory [2020-11-10 19:40:52 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2020-11-10 19:40:52 main:494] : INFO : Creating Model [2020-11-10 19:40:52 main:507] : INFO : Preparing config file [2020-11-10 19:40:52 main:511] : INFO : Found checkpoint, attempting to load... [2020-11-10 19:40:52 main:512] : INFO : Loading config [2020-11-10 19:40:52 main:514] : INFO : Loading state [2020-11-10 19:40:53 main:559] : INFO : Loading DataLoader into Memory [2020-11-10 19:40:53 main:562] : INFO : Starting Training [2020-11-10 19:41:03 main:574] : INFO : Epoch 153 | loss: 0.000196787 | val_loss: 0.000178346 | Time: 9585.25 ms [2020-11-10 19:41:12 main:574] : INFO : Epoch 154 | loss: 0.000174638 | val_loss: 0.000176455 | Time: 9087.26 ms [2020-11-10 19:41:21 main:574] : INFO : Epoch 155 | loss: 0.000171892 | val_loss: 0.000171641 | Time: 9234.98 ms [2020-11-10 19:41:30 main:574] : INFO : Epoch 156 | loss: 0.000167625 | val_loss: 0.000168096 | Time: 8913.94 ms [2020-11-10 19:41:39 main:574] : INFO : Epoch 157 | loss: 0.000163503 | val_loss: 0.000163527 | Time: 9296.06 ms [2020-11-10 19:41:49 main:574] : INFO : Epoch 158 | loss: 0.000159013 | val_loss: 0.000159412 | Time: 9688.75 ms [2020-11-10 19:41:58 main:574] : INFO : Epoch 159 | loss: 0.00015483 | val_loss: 0.000152781 | Time: 9362.79 ms [2020-11-10 19:42:08 main:574] : INFO : Epoch 160 | loss: 0.000150721 | val_loss: 0.000151466 | Time: 9521.36 ms [2020-11-10 19:42:17 main:574] : INFO : Epoch 161 | loss: 0.00014711 | val_loss: 0.000147657 | Time: 9073.11 ms [2020-11-10 19:42:26 main:574] : INFO : Epoch 162 | loss: 0.00014414 | val_loss: 0.000143222 | Time: 9097.89 ms [2020-11-10 19:42:35 main:574] : INFO : Epoch 163 | loss: 0.000140161 | val_loss: 0.00013995 | Time: 9205.3 ms [2020-11-10 19:42:45 main:574] : INFO : Epoch 164 | loss: 0.000136952 | val_loss: 0.000136799 | Time: 9313.82 ms [2020-11-10 19:42:54 main:574] : INFO : Epoch 165 | loss: 0.000134027 | val_loss: 0.000135701 | Time: 9239.04 ms [2020-11-10 19:43:03 main:574] : INFO : Epoch 166 | loss: 0.000130907 | val_loss: 0.00013184 | Time: 9334.07 ms [2020-11-10 19:43:12 main:574] : INFO : Epoch 167 | loss: 0.000127749 | val_loss: 0.000127743 | Time: 9132.71 ms [2020-11-10 19:43:22 main:574] : INFO : Epoch 168 | loss: 0.000125533 | val_loss: 0.000125788 | Time: 9098.36 ms [2020-11-10 19:43:31 main:574] : INFO : Epoch 169 | loss: 0.00012257 | val_loss: 0.000123597 | Time: 9095.31 ms [2020-11-10 19:43:40 main:574] : INFO : Epoch 170 | loss: 0.000120142 | val_loss: 0.000121588 | Time: 8865.55 ms [2020-11-10 19:43:49 main:574] : INFO : Epoch 171 | loss: 0.000117469 | val_loss: 0.00011916 | Time: 9079.59 ms [2020-11-10 19:43:58 main:574] : INFO : Epoch 172 | loss: 0.00011485 | val_loss: 0.000117373 | Time: 9038.37 ms [2020-11-10 19:44:07 main:574] : INFO : Epoch 173 | loss: 0.000113054 | val_loss: 0.000115267 | Time: 9447.77 ms [2020-11-10 19:44:17 main:574] : INFO : Epoch 174 | loss: 0.000111183 | val_loss: 0.000111845 | Time: 9393.97 ms [2020-11-10 19:44:26 main:574] : INFO : Epoch 175 | loss: 0.000108659 | val_loss: 0.000111131 | Time: 9352.41 ms [2020-11-10 19:44:35 main:574] : INFO : Epoch 176 | loss: 0.000106565 | val_loss: 0.000106526 | Time: 9298.02 ms [2020-11-10 19:44:45 main:574] : INFO : Epoch 177 | loss: 0.000104681 | val_loss: 0.000106796 | Time: 9351.62 ms [2020-11-10 19:44:54 main:574] : INFO : Epoch 178 | loss: 0.000102637 | val_loss: 0.000104138 | Time: 9006.13 ms [2020-11-10 19:45:03 main:574] : INFO : Epoch 179 | loss: 0.000100884 | val_loss: 0.000101797 | Time: 9235.34 ms [2020-11-10 19:45:12 main:574] : INFO : Epoch 180 | loss: 9.93028e-05 | val_loss: 0.000101408 | Time: 9027.41 ms [2020-11-10 19:45:21 main:574] : INFO : Epoch 181 | loss: 9.77326e-05 | val_loss: 9.97309e-05 | Time: 9193.68 ms [2020-11-10 19:45:30 main:574] : INFO : Epoch 182 | loss: 9.57484e-05 | val_loss: 9.75034e-05 | Time: 9264.41 ms [2020-11-10 19:45:39 main:574] : INFO : Epoch 183 | loss: 9.43227e-05 | val_loss: 9.51011e-05 | Time: 8995.6 ms [2020-11-10 19:45:49 main:574] : INFO : Epoch 184 | loss: 9.28333e-05 | val_loss: 9.41703e-05 | Time: 9186.29 ms [2020-11-10 19:45:58 main:574] : INFO : Epoch 185 | loss: 9.15223e-05 | val_loss: 9.08035e-05 | Time: 8942.77 ms [2020-11-10 19:46:07 main:574] : INFO : Epoch 186 | loss: 8.95501e-05 | val_loss: 9.01768e-05 | Time: 9011.48 ms [2020-11-10 19:46:16 main:574] : INFO : Epoch 187 | loss: 8.84035e-05 | val_loss: 8.93172e-05 | Time: 9146.09 ms [2020-11-10 19:46:25 main:574] : INFO : Epoch 188 | loss: 8.69818e-05 | val_loss: 8.76591e-05 | Time: 8987.81 ms [2020-11-10 19:46:34 main:574] : INFO : Epoch 189 | loss: 8.5721e-05 | val_loss: 8.75242e-05 | Time: 9279.99 ms [2020-11-10 19:46:43 main:574] : INFO : Epoch 190 | loss: 8.44223e-05 | val_loss: 8.59925e-05 | Time: 9092.93 ms [2020-11-10 19:46:52 main:574] : INFO : Epoch 191 | loss: 8.30597e-05 | val_loss: 8.449e-05 | Time: 9097.22 ms [2020-11-10 19:47:01 main:574] : INFO : Epoch 192 | loss: 8.17069e-05 | val_loss: 8.41793e-05 | Time: 8868.07 ms [2020-11-10 19:47:01 main:597] : INFO : Saving trained model to model-final.pt, val_loss 8.41793e-05 [2020-11-10 19:47:01 main:603] : INFO : Saving end state to config to file [2020-11-10 19:47:01 main:608] : INFO : Success, exiting.. 19:47:01 (2812): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)