Task 2824976
| Name | rand_automata_0012-1604983197-28710-0_2 |
| Workunit | 1263698 |
| Created | 10 Nov 2020, 23:31:35 UTC |
| Sent | 10 Nov 2020, 23:41:13 UTC |
| Report deadline | 11 Nov 2020, 22:11:13 UTC |
| Received | 11 Nov 2020, 5:55:12 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 3241 |
| Run time | 30 min 16 sec |
| CPU time | 29 min 34 sec |
| Validate state | Valid |
| Credit | 959.40 |
| Device peak FLOPS | 9,068.06 GFLOPS |
| Application version | Machine Learning Dataset Generator (test) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.41 GB |
| Peak swap size | 3.01 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version> <![CDATA[ <stderr_txt> Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce RTX 2070 SUPER) [2020-11-10 21:21:58 main:435] : INFO : Set logging level to 1 [2020-11-10 21:21:58 main:441] : INFO : Running in BOINC Client mode [2020-11-10 21:21:58 main:444] : INFO : Resolving all filenames [2020-11-10 21:21:58 main:452] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 (exists = 1) [2020-11-10 21:21:58 main:452] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-1604983197-28710-0_2_r1507950206_1 (exists = 0) [2020-11-10 21:21:58 main:452] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-1604983197-28710-0_2_r1507950206_0 (exists = 0) [2020-11-10 21:21:58 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2020-11-10 21:21:58 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0) [2020-11-10 21:21:58 main:472] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 [2020-11-10 21:21:58 main:474] : INFO : Configuration: [2020-11-10 21:21:58 main:475] : INFO : Model type: LSTM [2020-11-10 21:21:58 main:476] : INFO : Validation Loss Threshold: 0.0001 [2020-11-10 21:21:58 main:477] : INFO : Max Epochs: 192 [2020-11-10 21:21:58 main:478] : INFO : Batch Size: 32 [2020-11-10 21:21:58 main:479] : INFO : Learning Rate: 0.001 [2020-11-10 21:21:58 main:480] : INFO : Patience: 10 [2020-11-10 21:21:58 main:481] : INFO : Hidden Width: 64 [2020-11-10 21:21:58 main:482] : INFO : # Recurrent Layers: 4 [2020-11-10 21:21:58 main:483] : INFO : # Backend Layers: 2 [2020-11-10 21:21:58 main:484] : INFO : # Threads: 1 [2020-11-10 21:21:58 main:486] : INFO : Preparing Dataset [2020-11-10 21:21:58 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 into memory [2020-11-10 21:21:58 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 into memory [2020-11-10 21:22:00 load:106] : INFO : Successfully loaded dataset of 4096 examples into memory. [2020-11-10 21:22:00 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 into memory [2020-11-10 21:22:00 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 into memory [2020-11-10 21:22:00 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2020-11-10 21:22:00 main:494] : INFO : Creating Model [2020-11-10 21:22:00 main:507] : INFO : Preparing config file [2020-11-10 21:22:00 main:519] : INFO : Creating new config file [2020-11-10 21:22:00 main:559] : INFO : Loading DataLoader into Memory [2020-11-10 21:22:00 main:562] : INFO : Starting Training [2020-11-10 21:22:10 main:574] : INFO : Epoch 1 | loss: 0.0662561 | val_loss: 0.0659852 | Time: 9840.95 ms [2020-11-10 21:22:20 main:574] : INFO : Epoch 2 | loss: 0.0654985 | val_loss: 0.0639824 | Time: 9505.49 ms [2020-11-10 21:22:29 main:574] : INFO : Epoch 3 | loss: 0.0622795 | val_loss: 0.0606376 | Time: 9476.64 ms [2020-11-10 21:22:39 main:574] : INFO : Epoch 4 | loss: 0.0595072 | val_loss: 0.0585025 | Time: 9534.81 ms [2020-11-10 21:22:48 main:574] : INFO : Epoch 5 | loss: 0.0577333 | val_loss: 0.0570975 | Time: 9475.41 ms [2020-11-10 21:22:58 main:574] : INFO : Epoch 6 | loss: 0.0563805 | val_loss: 0.0558324 | Time: 9480.71 ms [2020-11-10 21:23:07 main:574] : INFO : Epoch 7 | loss: 0.05526 | val_loss: 0.0547612 | Time: 9426.41 ms [2020-11-10 21:23:17 main:574] : INFO : Epoch 8 | loss: 0.0541734 | val_loss: 0.0537104 | Time: 9418.12 ms [2020-11-10 21:23:26 main:574] : INFO : Epoch 9 | loss: 0.0531175 | val_loss: 0.0528288 | Time: 9483.02 ms [2020-11-10 21:23:35 main:574] : INFO : Epoch 10 | loss: 0.0520071 | val_loss: 0.0517338 | Time: 9164.21 ms [2020-11-10 21:23:44 main:574] : INFO : Epoch 11 | loss: 0.0508477 | val_loss: 0.0504188 | Time: 8893.94 ms [2020-11-10 21:23:54 main:574] : INFO : Epoch 12 | loss: 0.049525 | val_loss: 0.0492684 | Time: 9513.22 ms [2020-11-10 21:24:03 main:574] : INFO : Epoch 13 | loss: 0.0481145 | val_loss: 0.047788 | Time: 9555.12 ms [2020-11-10 21:24:13 main:574] : INFO : Epoch 14 | loss: 0.046636 | val_loss: 0.0464847 | Time: 9546.62 ms [2020-11-10 21:24:22 main:574] : INFO : Epoch 15 | loss: 0.0448706 | val_loss: 0.0448683 | Time: 9577.18 ms [2020-11-10 21:24:32 main:574] : INFO : Epoch 16 | loss: 0.0431775 | val_loss: 0.0434998 | Time: 9560.4 ms [2020-11-10 21:24:42 main:574] : INFO : Epoch 17 | loss: 0.041319 | val_loss: 0.0414365 | Time: 9583.4 ms [2020-11-10 21:24:51 main:574] : INFO : Epoch 18 | loss: 0.0392898 | val_loss: 0.0399705 | Time: 9466.9 ms [2020-11-10 21:25:01 main:574] : INFO : Epoch 19 | loss: 0.0372864 | val_loss: 0.0381919 | Time: 9584.2 ms [2020-11-10 21:25:10 main:574] : INFO : Epoch 20 | loss: 0.0352082 | val_loss: 0.0356344 | Time: 9633.13 ms [2020-11-10 21:25:20 main:574] : INFO : Epoch 21 | loss: 0.033206 | val_loss: 0.0333378 | Time: 9803.4 ms [2020-11-10 21:25:30 main:574] : INFO : Epoch 22 | loss: 0.0306238 | val_loss: 0.031437 | Time: 9494.85 ms [2020-11-10 21:25:39 main:574] : INFO : Epoch 23 | loss: 0.0290188 | val_loss: 0.0303875 | Time: 9616.22 ms [2020-11-10 21:25:49 main:574] : INFO : Epoch 24 | loss: 0.0271763 | val_loss: 0.0358595 | Time: 9575.48 ms [2020-11-10 21:25:58 main:574] : INFO : Epoch 25 | loss: 0.0438219 | val_loss: 0.0356273 | Time: 9547.12 ms [2020-11-10 21:26:08 main:574] : INFO : Epoch 26 | loss: 0.0340184 | val_loss: 0.0650778 | Time: 9543.71 ms [2020-11-10 21:26:18 main:574] : INFO : Epoch 27 | loss: 0.0631141 | val_loss: 0.0625851 | Time: 9678.31 ms [2020-11-10 21:26:27 main:574] : INFO : Epoch 28 | loss: 0.0623384 | val_loss: 0.0622452 | Time: 9682.01 ms [2020-11-10 21:26:37 main:574] : INFO : Epoch 29 | loss: 0.062071 | val_loss: 0.0620236 | Time: 9762.8 ms [2020-11-10 21:26:47 main:574] : INFO : Epoch 30 | loss: 0.0618671 | val_loss: 0.0618486 | Time: 9591.43 ms [2020-11-10 21:26:56 main:574] : INFO : Epoch 31 | loss: 0.0616903 | val_loss: 0.0616697 | Time: 9429.45 ms [2020-11-10 21:27:06 main:574] : INFO : Epoch 32 | loss: 0.0615206 | val_loss: 0.0615111 | Time: 9632.27 ms [2020-11-10 21:27:15 main:574] : INFO : Epoch 33 | loss: 0.0613491 | val_loss: 0.0613174 | Time: 9558.73 ms [2020-11-10 21:27:25 main:574] : INFO : Epoch 34 | loss: 0.0611641 | val_loss: 0.061124 | Time: 9522.2 ms [2020-11-10 21:27:34 main:574] : INFO : Epoch 35 | loss: 0.0609665 | val_loss: 0.0609227 | Time: 9508.84 ms [2020-11-10 21:27:44 main:574] : INFO : Epoch 36 | loss: 0.0607743 | val_loss: 0.0607259 | Time: 10044.2 ms [2020-11-10 21:27:54 main:574] : INFO : Epoch 37 | loss: 0.0605693 | val_loss: 0.0604926 | Time: 9509.34 ms [2020-11-10 21:28:03 main:574] : INFO : Epoch 38 | loss: 0.0603335 | val_loss: 0.0602419 | Time: 9498.18 ms [2020-11-10 21:28:13 main:574] : INFO : Epoch 39 | loss: 0.060074 | val_loss: 0.0599665 | Time: 9507.21 ms [2020-11-10 21:28:23 main:574] : INFO : Epoch 40 | loss: 0.0598029 | val_loss: 0.0596785 | Time: 9551.7 ms [2020-11-10 21:28:32 main:574] : INFO : Epoch 41 | loss: 0.0595278 | val_loss: 0.0593915 | Time: 9704.14 ms [2020-11-10 21:28:42 main:574] : INFO : Epoch 42 | loss: 0.059233 | val_loss: 0.0590667 | Time: 9610.66 ms [2020-11-10 21:28:51 main:574] : INFO : Epoch 43 | loss: 0.0588961 | val_loss: 0.0586855 | Time: 9616.06 ms [2020-11-10 21:29:01 main:574] : INFO : Epoch 44 | loss: 0.058449 | val_loss: 0.058194 | Time: 9538.36 ms [2020-11-10 21:29:10 main:574] : INFO : Epoch 45 | loss: 0.0579138 | val_loss: 0.0576167 | Time: 9086.75 ms [2020-11-10 21:29:19 main:574] : INFO : Epoch 46 | loss: 0.0572764 | val_loss: 0.0569139 | Time: 8920.57 ms [2020-11-10 21:29:28 main:574] : INFO : Epoch 47 | loss: 0.0564262 | val_loss: 0.0559708 | Time: 9068.44 ms [2020-11-10 21:29:37 main:574] : INFO : Epoch 48 | loss: 0.0552851 | val_loss: 0.0546919 | Time: 8762.62 ms [2020-11-10 21:29:46 main:574] : INFO : Epoch 49 | loss: 0.0537094 | val_loss: 0.0529755 | Time: 8895.8 ms [2020-11-10 21:29:55 main:574] : INFO : Epoch 50 | loss: 0.0514848 | val_loss: 0.0505018 | Time: 8789.91 ms [2020-11-10 21:30:03 main:574] : INFO : Epoch 51 | loss: 0.0476834 | val_loss: 0.0453982 | Time: 8789.85 ms [2020-11-10 21:30:12 main:574] : INFO : Epoch 52 | loss: 0.0411856 | val_loss: 0.0379981 | Time: 8800.45 ms [2020-11-10 21:30:21 main:574] : INFO : Epoch 53 | loss: 0.0343776 | val_loss: 0.0386685 | Time: 8783.82 ms [2020-11-10 21:30:30 main:574] : INFO : Epoch 54 | loss: 0.0309553 | val_loss: 0.0296523 | Time: 8733.77 ms [2020-11-10 21:30:40 main:574] : INFO : Epoch 55 | loss: 0.0383685 | val_loss: 0.0556659 | Time: 10100.1 ms [2020-11-10 21:30:49 main:574] : INFO : Epoch 56 | loss: 0.0426284 | val_loss: 0.0292474 | Time: 9114.4 ms [2020-11-10 21:30:58 main:574] : INFO : Epoch 57 | loss: 0.0264759 | val_loss: 0.0304466 | Time: 8706.55 ms [2020-11-10 21:31:07 main:574] : INFO : Epoch 58 | loss: 0.0231582 | val_loss: 0.0237017 | Time: 8973.16 ms [2020-11-10 21:31:18 main:574] : INFO : Epoch 59 | loss: 0.024132 | val_loss: 0.0643277 | Time: 11730.9 ms [2020-11-10 21:31:30 main:574] : INFO : Epoch 60 | loss: 0.0610191 | val_loss: 0.0596486 | Time: 11230.6 ms [2020-11-10 21:31:41 main:574] : INFO : Epoch 61 | loss: 0.0584288 | val_loss: 0.0572801 | Time: 11164.7 ms [2020-11-10 21:31:52 main:574] : INFO : Epoch 62 | loss: 0.0552866 | val_loss: 0.0531385 | Time: 10703.8 ms [2020-11-10 21:32:02 main:574] : INFO : Epoch 63 | loss: 0.0488371 | val_loss: 0.043342 | Time: 10896.4 ms [2020-11-10 21:32:11 main:574] : INFO : Epoch 64 | loss: 0.0392045 | val_loss: 0.0458283 | Time: 8792.65 ms [2020-11-10 21:32:20 main:574] : INFO : Epoch 65 | loss: 0.0310015 | val_loss: 0.0249065 | Time: 9069.91 ms [2020-11-10 21:32:31 main:574] : INFO : Epoch 66 | loss: 0.0219287 | val_loss: 0.028072 | Time: 10419.5 ms [2020-11-10 21:32:40 main:574] : INFO : Epoch 67 | loss: 0.0215994 | val_loss: 0.0177278 | Time: 9340.25 ms [2020-11-10 21:32:49 main:574] : INFO : Epoch 68 | loss: 0.0229038 | val_loss: 0.018304 | Time: 8987.6 ms [2020-11-10 21:32:59 main:574] : INFO : Epoch 69 | loss: 0.0162218 | val_loss: 0.0160174 | Time: 10034.6 ms [2020-11-10 21:33:08 main:574] : INFO : Epoch 70 | loss: 0.0136303 | val_loss: 0.0131873 | Time: 8756.92 ms [2020-11-10 21:33:17 main:574] : INFO : Epoch 71 | loss: 0.0222742 | val_loss: 0.0144487 | Time: 8741.81 ms [2020-11-10 21:33:25 main:574] : INFO : Epoch 72 | loss: 0.0263614 | val_loss: 0.0194249 | Time: 8705.36 ms [2020-11-10 21:33:34 main:574] : INFO : Epoch 73 | loss: 0.0467966 | val_loss: 0.0353105 | Time: 8727.11 ms [2020-11-10 21:33:43 main:574] : INFO : Epoch 74 | loss: 0.0250254 | val_loss: 0.0175088 | Time: 8703.75 ms [2020-11-10 21:33:52 main:574] : INFO : Epoch 75 | loss: 0.0159901 | val_loss: 0.0138962 | Time: 8723.18 ms [2020-11-10 21:34:01 main:574] : INFO : Epoch 76 | loss: 0.0191283 | val_loss: 0.0515804 | Time: 8975.15 ms [2020-11-10 21:34:09 main:574] : INFO : Epoch 77 | loss: 0.0327217 | val_loss: 0.0426256 | Time: 8745.23 ms [2020-11-10 21:34:18 main:574] : INFO : Epoch 78 | loss: 0.0443144 | val_loss: 0.0469613 | Time: 8762.92 ms [2020-11-10 21:34:27 main:574] : INFO : Epoch 79 | loss: 0.0311978 | val_loss: 0.023206 | Time: 8783.64 ms [2020-11-10 21:34:36 main:574] : INFO : Epoch 80 | loss: 0.0257104 | val_loss: 0.0244495 | Time: 8799.78 ms [2020-11-10 21:34:45 main:574] : INFO : Epoch 81 | loss: 0.015383 | val_loss: 0.0127766 | Time: 9196.12 ms [2020-11-10 21:34:55 main:574] : INFO : Epoch 82 | loss: 0.0115357 | val_loss: 0.01164 | Time: 10095.4 ms [2020-11-10 21:35:04 main:574] : INFO : Epoch 83 | loss: 0.0105879 | val_loss: 0.010628 | Time: 9081.95 ms [2020-11-10 21:35:13 main:574] : INFO : Epoch 84 | loss: 0.00994433 | val_loss: 0.0101486 | Time: 8728.4 ms [2020-11-10 21:35:22 main:574] : INFO : Epoch 85 | loss: 0.00952637 | val_loss: 0.00978481 | Time: 8744.13 ms [2020-11-10 21:35:30 main:574] : INFO : Epoch 86 | loss: 0.00915289 | val_loss: 0.00942102 | Time: 8701.93 ms [2020-11-10 21:35:39 main:574] : INFO : Epoch 87 | loss: 0.0088488 | val_loss: 0.00908323 | Time: 8699.13 ms [2020-11-10 21:35:48 main:574] : INFO : Epoch 88 | loss: 0.00856579 | val_loss: 0.00891268 | Time: 8688.07 ms [2020-11-10 21:35:57 main:574] : INFO : Epoch 89 | loss: 0.00835754 | val_loss: 0.00852184 | Time: 8847.21 ms [2020-11-10 21:36:06 main:574] : INFO : Epoch 90 | loss: 0.00812203 | val_loss: 0.00829884 | Time: 8977.23 ms [2020-11-10 21:36:14 main:574] : INFO : Epoch 91 | loss: 0.00790927 | val_loss: 0.00809924 | Time: 8739.91 ms [2020-11-10 21:36:23 main:574] : INFO : Epoch 92 | loss: 0.0078571 | val_loss: 0.00856675 | Time: 8696.87 ms [2020-11-10 21:36:32 main:574] : INFO : Epoch 93 | loss: 0.00766805 | val_loss: 0.00811037 | Time: 8896.14 ms [2020-11-10 21:36:41 main:574] : INFO : Epoch 94 | loss: 0.00748647 | val_loss: 0.00749924 | Time: 8972.27 ms [2020-11-10 21:36:50 main:574] : INFO : Epoch 95 | loss: 0.00752604 | val_loss: 0.00733195 | Time: 8779.33 ms [2020-11-10 21:36:59 main:574] : INFO : Epoch 96 | loss: 0.0070378 | val_loss: 0.00708496 | Time: 8800.3 ms [2020-11-10 21:37:07 main:574] : INFO : Epoch 97 | loss: 0.00682931 | val_loss: 0.00693737 | Time: 8732.29 ms [2020-11-10 21:37:16 main:574] : INFO : Epoch 98 | loss: 0.00665954 | val_loss: 0.00703994 | Time: 8672.14 ms [2020-11-10 21:37:25 main:574] : INFO : Epoch 99 | loss: 0.00654366 | val_loss: 0.00650083 | Time: 8650.55 ms [2020-11-10 21:37:33 main:574] : INFO : Epoch 100 | loss: 0.00631624 | val_loss: 0.00631367 | Time: 8659.77 ms [2020-11-10 21:37:42 main:574] : INFO : Epoch 101 | loss: 0.00614828 | val_loss: 0.00614197 | Time: 8655.29 ms [2020-11-10 21:37:51 main:574] : INFO : Epoch 102 | loss: 0.00598957 | val_loss: 0.00621257 | Time: 9008.52 ms [2020-11-10 21:38:00 main:574] : INFO : Epoch 103 | loss: 0.00587266 | val_loss: 0.00581706 | Time: 8880.41 ms [2020-11-10 21:38:09 main:574] : INFO : Epoch 104 | loss: 0.00567208 | val_loss: 0.00565396 | Time: 8688.46 ms [2020-11-10 21:38:17 main:574] : INFO : Epoch 105 | loss: 0.0055099 | val_loss: 0.00549785 | Time: 8693.48 ms [2020-11-10 21:38:27 main:574] : INFO : Epoch 106 | loss: 0.00534939 | val_loss: 0.00533749 | Time: 9813.42 ms [2020-11-10 21:38:37 main:574] : INFO : Epoch 107 | loss: 0.00519317 | val_loss: 0.00517484 | Time: 10390 ms [2020-11-10 21:38:47 main:574] : INFO : Epoch 108 | loss: 0.00503845 | val_loss: 0.00501935 | Time: 9555.43 ms [2020-11-10 21:38:57 main:574] : INFO : Epoch 109 | loss: 0.00489759 | val_loss: 0.00489672 | Time: 9500.77 ms [2020-11-10 21:39:06 main:574] : INFO : Epoch 110 | loss: 0.00473854 | val_loss: 0.00471634 | Time: 9506.08 ms [2020-11-10 21:39:16 main:574] : INFO : Epoch 111 | loss: 0.00458516 | val_loss: 0.00457231 | Time: 9517.67 ms [2020-11-10 21:39:25 main:574] : INFO : Epoch 112 | loss: 0.00443492 | val_loss: 0.00441837 | Time: 9484.38 ms [2020-11-10 21:39:35 main:574] : INFO : Epoch 113 | loss: 0.00428889 | val_loss: 0.00427336 | Time: 10075.2 ms [2020-11-10 21:39:45 main:574] : INFO : Epoch 114 | loss: 0.00414561 | val_loss: 0.0041292 | Time: 9521.04 ms [2020-11-10 21:39:54 main:574] : INFO : Epoch 115 | loss: 0.00401216 | val_loss: 0.0042302 | Time: 9506.34 ms [2020-11-10 21:40:04 main:574] : INFO : Epoch 116 | loss: 0.0038948 | val_loss: 0.0038511 | Time: 9511.7 ms [2020-11-10 21:40:13 main:574] : INFO : Epoch 117 | loss: 0.00372483 | val_loss: 0.00371895 | Time: 9497.82 ms [2020-11-10 21:40:23 main:574] : INFO : Epoch 118 | loss: 0.0035923 | val_loss: 0.00358671 | Time: 9512.66 ms [2020-11-10 21:40:32 main:574] : INFO : Epoch 119 | loss: 0.0034623 | val_loss: 0.00345897 | Time: 9517.65 ms [2020-11-10 21:40:42 main:574] : INFO : Epoch 120 | loss: 0.0033355 | val_loss: 0.0033295 | Time: 9519.85 ms [2020-11-10 21:40:51 main:574] : INFO : Epoch 121 | loss: 0.00321225 | val_loss: 0.00320562 | Time: 9630.4 ms [2020-11-10 21:41:01 main:574] : INFO : Epoch 122 | loss: 0.0030922 | val_loss: 0.00308759 | Time: 9489.36 ms [2020-11-10 21:41:10 main:574] : INFO : Epoch 123 | loss: 0.00297503 | val_loss: 0.00297266 | Time: 9509.1 ms [2020-11-10 21:41:20 main:574] : INFO : Epoch 124 | loss: 0.00286078 | val_loss: 0.00285921 | Time: 9515.62 ms [2020-11-10 21:41:30 main:574] : INFO : Epoch 125 | loss: 0.00275028 | val_loss: 0.00274979 | Time: 9506.03 ms [2020-11-10 21:41:39 main:574] : INFO : Epoch 126 | loss: 0.00264212 | val_loss: 0.00264017 | Time: 9531.14 ms [2020-11-10 21:41:49 main:574] : INFO : Epoch 127 | loss: 0.00253745 | val_loss: 0.00253702 | Time: 9503.07 ms [2020-11-10 21:41:58 main:574] : INFO : Epoch 128 | loss: 0.00243563 | val_loss: 0.00243619 | Time: 9493.39 ms [2020-11-10 21:42:08 main:574] : INFO : Epoch 129 | loss: 0.00233723 | val_loss: 0.00233649 | Time: 9521.02 ms [2020-11-10 21:42:17 main:574] : INFO : Epoch 130 | loss: 0.00224117 | val_loss: 0.00224149 | Time: 9496.14 ms [2020-11-10 21:42:27 main:574] : INFO : Epoch 131 | loss: 0.00214813 | val_loss: 0.00214851 | Time: 10012.6 ms [2020-11-10 21:42:37 main:574] : INFO : Epoch 132 | loss: 0.00205737 | val_loss: 0.00206012 | Time: 10329.7 ms [2020-11-10 21:42:47 main:574] : INFO : Epoch 133 | loss: 0.00197064 | val_loss: 0.00197029 | Time: 9516.26 ms [2020-11-10 21:42:57 main:574] : INFO : Epoch 134 | loss: 0.00188514 | val_loss: 0.00188837 | Time: 9515.89 ms [2020-11-10 21:43:06 main:574] : INFO : Epoch 135 | loss: 0.00180386 | val_loss: 0.00180469 | Time: 9498.37 ms [2020-11-10 21:43:16 main:574] : INFO : Epoch 136 | loss: 0.00172369 | val_loss: 0.00172505 | Time: 9513.63 ms [2020-11-10 21:43:25 main:574] : INFO : Epoch 137 | loss: 0.00164621 | val_loss: 0.00164902 | Time: 9519.79 ms [2020-11-10 21:43:35 main:574] : INFO : Epoch 138 | loss: 0.0015723 | val_loss: 0.0015785 | Time: 9538.59 ms [2020-11-10 21:43:44 main:574] : INFO : Epoch 139 | loss: 0.00150082 | val_loss: 0.00150482 | Time: 9505.45 ms [2020-11-10 21:43:54 main:574] : INFO : Epoch 140 | loss: 0.00143109 | val_loss: 0.00143449 | Time: 9492.46 ms [2020-11-10 21:44:03 main:574] : INFO : Epoch 141 | loss: 0.00136566 | val_loss: 0.00136921 | Time: 9508.27 ms [2020-11-10 21:44:13 main:574] : INFO : Epoch 142 | loss: 0.00130139 | val_loss: 0.00130391 | Time: 9521.81 ms [2020-11-10 21:44:22 main:574] : INFO : Epoch 143 | loss: 0.00124068 | val_loss: 0.00124494 | Time: 9480.59 ms [2020-11-10 21:44:32 main:574] : INFO : Epoch 144 | loss: 0.00118197 | val_loss: 0.00118683 | Time: 9508.25 ms [2020-11-10 21:44:41 main:574] : INFO : Epoch 145 | loss: 0.00112563 | val_loss: 0.00113151 | Time: 9530.85 ms [2020-11-10 21:44:51 main:574] : INFO : Epoch 146 | loss: 0.00107315 | val_loss: 0.00107646 | Time: 9508.81 ms [2020-11-10 21:45:00 main:574] : INFO : Epoch 147 | loss: 0.00102257 | val_loss: 0.00102687 | Time: 9702.25 ms [2020-11-10 21:45:11 main:574] : INFO : Epoch 148 | loss: 0.000974257 | val_loss: 0.000978636 | Time: 10122.2 ms [2020-11-10 21:45:20 main:574] : INFO : Epoch 149 | loss: 0.000927736 | val_loss: 0.000929907 | Time: 9547.86 ms [2020-11-10 21:45:30 main:574] : INFO : Epoch 150 | loss: 0.000884325 | val_loss: 0.000886315 | Time: 9523.38 ms [2020-11-10 21:45:39 main:574] : INFO : Epoch 151 | loss: 0.000842159 | val_loss: 0.000847744 | Time: 9516.01 ms [2020-11-10 21:45:49 main:574] : INFO : Epoch 152 | loss: 0.000802937 | val_loss: 0.000804955 | Time: 9527.67 ms [2020-11-10 21:45:58 main:574] : INFO : Epoch 153 | loss: 0.000765673 | val_loss: 0.000768556 | Time: 9505.1 ms [2020-11-10 21:46:08 main:574] : INFO : Epoch 154 | loss: 0.000729109 | val_loss: 0.000734165 | Time: 9533.03 ms [2020-11-10 21:46:17 main:574] : INFO : Epoch 155 | loss: 0.000695673 | val_loss: 0.000698142 | Time: 9529.05 ms [2020-11-10 21:46:27 main:574] : INFO : Epoch 156 | loss: 0.000663715 | val_loss: 0.00066944 | Time: 9514.48 ms [2020-11-10 21:46:37 main:574] : INFO : Epoch 157 | loss: 0.000632819 | val_loss: 0.00063574 | Time: 9612.7 ms [2020-11-10 21:46:46 main:574] : INFO : Epoch 158 | loss: 0.000603369 | val_loss: 0.000607043 | Time: 9518.28 ms [2020-11-10 21:46:55 main:574] : INFO : Epoch 159 | loss: 0.000576133 | val_loss: 0.000582818 | Time: 8782.94 ms [2020-11-10 21:47:04 main:574] : INFO : Epoch 160 | loss: 0.000548871 | val_loss: 0.000554795 | Time: 8837.35 ms Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce RTX 2070 SUPER) [2020-11-10 21:48:58 main:435] : INFO : Set logging level to 1 [2020-11-10 21:48:58 main:441] : INFO : Running in BOINC Client mode [2020-11-10 21:48:58 main:444] : INFO : Resolving all filenames [2020-11-10 21:48:58 main:452] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 (exists = 1) [2020-11-10 21:48:58 main:452] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-1604983197-28710-0_2_r1507950206_1 (exists = 1) [2020-11-10 21:48:58 main:452] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-1604983197-28710-0_2_r1507950206_0 (exists = 0) [2020-11-10 21:48:58 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 0) [2020-11-10 21:48:58 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 1) [2020-11-10 21:48:58 main:472] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 [2020-11-10 21:48:58 main:474] : INFO : Configuration: [2020-11-10 21:48:58 main:475] : INFO : Model type: LSTM [2020-11-10 21:48:58 main:476] : INFO : Validation Loss Threshold: 0.0001 [2020-11-10 21:48:58 main:477] : INFO : Max Epochs: 192 [2020-11-10 21:48:58 main:478] : INFO : Batch Size: 32 [2020-11-10 21:48:58 main:479] : INFO : Learning Rate: 0.001 [2020-11-10 21:48:58 main:480] : INFO : Patience: 10 [2020-11-10 21:48:58 main:481] : INFO : Hidden Width: 64 [2020-11-10 21:48:58 main:482] : INFO : # Recurrent Layers: 4 [2020-11-10 21:48:58 main:483] : INFO : # Backend Layers: 2 [2020-11-10 21:48:58 main:484] : INFO : # Threads: 1 [2020-11-10 21:48:58 main:486] : INFO : Preparing Dataset [2020-11-10 21:48:58 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 into memory [2020-11-10 21:48:58 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 into memory [2020-11-10 21:49:00 load:106] : INFO : Successfully loaded dataset of 4096 examples into memory. [2020-11-10 21:49:00 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 into memory [2020-11-10 21:49:00 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/rand_automata_0012-train-val-dataset.hdf5 into memory [2020-11-10 21:49:00 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2020-11-10 21:49:00 main:494] : INFO : Creating Model [2020-11-10 21:49:00 main:507] : INFO : Preparing config file [2020-11-10 21:49:00 main:511] : INFO : Found checkpoint, attempting to load... [2020-11-10 21:49:00 main:512] : INFO : Loading config [2020-11-10 21:49:00 main:514] : INFO : Loading state [2020-11-10 21:49:01 main:559] : INFO : Loading DataLoader into Memory [2020-11-10 21:49:01 main:562] : INFO : Starting Training [2020-11-10 21:49:10 main:574] : INFO : Epoch 157 | loss: 0.000655214 | val_loss: 0.00062519 | Time: 9372.38 ms [2020-11-10 21:49:21 main:574] : INFO : Epoch 158 | loss: 0.000586686 | val_loss: 0.00057963 | Time: 10608.9 ms [2020-11-10 21:49:32 main:574] : INFO : Epoch 159 | loss: 0.000543419 | val_loss: 0.000539146 | Time: 10681.2 ms [2020-11-10 21:49:41 main:574] : INFO : Epoch 160 | loss: 0.000502888 | val_loss: 0.000498629 | Time: 9398.04 ms [2020-11-10 21:49:50 main:574] : INFO : Epoch 161 | loss: 0.000467618 | val_loss: 0.000466678 | Time: 9017.41 ms [2020-11-10 21:49:59 main:574] : INFO : Epoch 162 | loss: 0.000435128 | val_loss: 0.000433341 | Time: 8901.84 ms [2020-11-10 21:50:08 main:574] : INFO : Epoch 163 | loss: 0.000407013 | val_loss: 0.000406981 | Time: 8645.58 ms [2020-11-10 21:50:17 main:574] : INFO : Epoch 164 | loss: 0.000380866 | val_loss: 0.000380047 | Time: 9053.46 ms [2020-11-10 21:50:27 main:574] : INFO : Epoch 165 | loss: 0.000358562 | val_loss: 0.000358005 | Time: 9787.06 ms [2020-11-10 21:50:36 main:574] : INFO : Epoch 166 | loss: 0.000338443 | val_loss: 0.000340621 | Time: 9033.27 ms [2020-11-10 21:50:45 main:574] : INFO : Epoch 167 | loss: 0.000320805 | val_loss: 0.000320262 | Time: 9088.33 ms [2020-11-10 21:50:54 main:574] : INFO : Epoch 168 | loss: 0.000303974 | val_loss: 0.00030318 | Time: 9327.77 ms [2020-11-10 21:51:05 main:574] : INFO : Epoch 169 | loss: 0.000288501 | val_loss: 0.000289757 | Time: 10511.8 ms [2020-11-10 21:51:14 main:574] : INFO : Epoch 170 | loss: 0.000274728 | val_loss: 0.000277939 | Time: 9758.08 ms [2020-11-10 21:51:23 main:574] : INFO : Epoch 171 | loss: 0.000262636 | val_loss: 0.00026403 | Time: 8618.72 ms [2020-11-10 21:51:32 main:574] : INFO : Epoch 172 | loss: 0.000251051 | val_loss: 0.000250731 | Time: 8609.17 ms [2020-11-10 21:51:40 main:574] : INFO : Epoch 173 | loss: 0.000239606 | val_loss: 0.00024222 | Time: 8633.79 ms [2020-11-10 21:51:49 main:574] : INFO : Epoch 174 | loss: 0.000229751 | val_loss: 0.000230003 | Time: 8618.67 ms [2020-11-10 21:51:58 main:574] : INFO : Epoch 175 | loss: 0.000220715 | val_loss: 0.000221988 | Time: 8938.65 ms [2020-11-10 21:52:07 main:574] : INFO : Epoch 176 | loss: 0.000211675 | val_loss: 0.000213081 | Time: 9340.28 ms [2020-11-10 21:52:17 main:574] : INFO : Epoch 177 | loss: 0.000204415 | val_loss: 0.000208165 | Time: 9835.62 ms [2020-11-10 21:52:26 main:574] : INFO : Epoch 178 | loss: 0.000196694 | val_loss: 0.00019993 | Time: 9105.8 ms [2020-11-10 21:52:35 main:574] : INFO : Epoch 179 | loss: 0.000189386 | val_loss: 0.000190677 | Time: 8624.49 ms [2020-11-10 21:52:43 main:574] : INFO : Epoch 180 | loss: 0.000182687 | val_loss: 0.000184303 | Time: 8635.98 ms [2020-11-10 21:52:52 main:574] : INFO : Epoch 181 | loss: 0.000176311 | val_loss: 0.000179079 | Time: 8630.98 ms [2020-11-10 21:53:02 main:574] : INFO : Epoch 182 | loss: 0.000170291 | val_loss: 0.000171966 | Time: 9627.21 ms [2020-11-10 21:53:12 main:574] : INFO : Epoch 183 | loss: 0.000164877 | val_loss: 0.000165598 | Time: 10089.3 ms [2020-11-10 21:53:22 main:574] : INFO : Epoch 184 | loss: 0.000159167 | val_loss: 0.000161141 | Time: 9915.12 ms [2020-11-10 21:53:33 main:574] : INFO : Epoch 185 | loss: 0.000154287 | val_loss: 0.000157694 | Time: 10976.8 ms [2020-11-10 21:53:43 main:574] : INFO : Epoch 186 | loss: 0.000150144 | val_loss: 0.000150604 | Time: 9869.84 ms [2020-11-10 21:53:52 main:574] : INFO : Epoch 187 | loss: 0.000144553 | val_loss: 0.000146606 | Time: 9661.18 ms [2020-11-10 21:54:02 main:574] : INFO : Epoch 188 | loss: 0.000140311 | val_loss: 0.000141775 | Time: 9537.57 ms [2020-11-10 21:54:11 main:574] : INFO : Epoch 189 | loss: 0.000136705 | val_loss: 0.000137724 | Time: 9561.92 ms [2020-11-10 21:54:21 main:574] : INFO : Epoch 190 | loss: 0.000132471 | val_loss: 0.00013533 | Time: 9554.39 ms [2020-11-10 21:54:30 main:574] : INFO : Epoch 191 | loss: 0.000128735 | val_loss: 0.000131805 | Time: 9552.69 ms [2020-11-10 21:54:40 main:574] : INFO : Epoch 192 | loss: 0.000125187 | val_loss: 0.00012711 | Time: 9675.56 ms [2020-11-10 21:54:40 main:597] : INFO : Saving trained model to model-final.pt, val_loss 0.00012711 [2020-11-10 21:54:40 main:603] : INFO : Saving end state to config to file [2020-11-10 21:54:40 main:608] : INFO : Success, exiting.. 21:54:40 (15220): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)