Task 5115795
| Name | ParityModified-1607233796-4364-42_5 |
| Workunit | 3057036 |
| Created | 28 Apr 2021, 16:36:52 UTC |
| Sent | 1 May 2021, 21:17:12 UTC |
| Report deadline | 8 May 2021, 21:17:12 UTC |
| Received | 2 May 2021, 19:47:39 UTC |
| Server state | Over |
| Outcome | Validate error |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 11140 |
| Run time | 40 min 10 sec |
| CPU time | |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 13,358.80 GFLOPS |
| Application version | Machine Learning Dataset Generator (test) v9.80 (amdrocm) x86_64-pc-linux-gnu |
| Peak working set size | 1.43 GB |
| Peak swap size | 8.06 GB |
| Peak disk usage | 2.25 GB |
Stderr output
<core_client_version>7.16.16</core_client_version> <![CDATA[ <stderr_txt> DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__rocm -c --maxepoch 128 nthreads: 1 gpudev: 0 Re-exec()-ing to set environment correctly 14:20:04 (128095): start_timer_thread(): pthread_create(): 22Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: Vega 10 XL/XT [Radeon RX Vega 56/64]) [2021-05-02 14:20:04 main:442] : INFO : Set logging level to 1 [2021-05-02 14:20:04 main:448] : INFO : Running in BOINC Client mode [2021-05-02 14:20:04 main:451] : INFO : Resolving all filenames [2021-05-02 14:20:04 main:459] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 (exists = 1) [2021-05-02 14:20:04 main:459] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42_5_r1367993063_1 (exists = 0) [2021-05-02 14:20:04 main:459] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42_5_r1367993063_0 (exists = 0) [2021-05-02 14:20:04 main:459] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42 (exists = 1) [2021-05-02 14:20:04 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0) [2021-05-02 14:20:04 main:479] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 [2021-05-02 14:20:04 main:481] : INFO : Configuration: [2021-05-02 14:20:04 main:482] : INFO : Model type: GRU [2021-05-02 14:20:04 main:483] : INFO : Validation Loss Threshold: 0.0001 [2021-05-02 14:20:04 main:484] : INFO : Max Epochs: 128 [2021-05-02 14:20:04 main:485] : INFO : Batch Size: 128 [2021-05-02 14:20:04 main:486] : INFO : Learning Rate: 0.01 [2021-05-02 14:20:04 main:487] : INFO : Patience: 10 [2021-05-02 14:20:04 main:488] : INFO : Hidden Width: 12 [2021-05-02 14:20:04 main:489] : INFO : # Recurrent Layers: 4 [2021-05-02 14:20:04 main:490] : INFO : # Backend Layers: 4 [2021-05-02 14:20:04 main:491] : INFO : # Threads: 1 [2021-05-02 14:20:04 main:493] : INFO : Preparing Dataset [2021-05-02 14:20:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 14:20:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 14:20:05 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2021-05-02 14:20:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 14:20:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 14:20:05 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2021-05-02 14:20:05 main:501] : INFO : Creating Model [2021-05-02 14:20:05 main:514] : INFO : Preparing config file [2021-05-02 14:20:05 main:526] : INFO : Creating new config file [2021-05-02 14:20:05 main:545] : INFO : This is a continuation WU, loading previous network [2021-05-02 14:20:05 main:566] : INFO : Loading DataLoader into Memory [2021-05-02 14:20:05 main:569] : INFO : Starting Training [2021-05-02 14:20:30 main:581] : INFO : Epoch 1 | loss: 0.0311017 | val_loss: 0.0312467 | Time: 24145.4 ms [2021-05-02 14:20:49 main:581] : INFO : Epoch 2 | loss: 0.0310536 | val_loss: 0.0312488 | Time: 19018.8 ms [2021-05-02 14:21:08 main:581] : INFO : Epoch 3 | loss: 0.0310385 | val_loss: 0.0312629 | Time: 19075.7 ms [2021-05-02 14:21:27 main:581] : INFO : Epoch 4 | loss: 0.031026 | val_loss: 0.0312708 | Time: 19051.1 ms [2021-05-02 14:21:46 main:581] : INFO : Epoch 5 | loss: 0.0310299 | val_loss: 0.0312677 | Time: 18982.6 ms [2021-05-02 14:22:05 main:581] : INFO : Epoch 6 | loss: 0.0310297 | val_loss: 0.0312625 | Time: 19155.2 ms [2021-05-02 14:22:24 main:581] : INFO : Epoch 7 | loss: 0.0310278 | val_loss: 0.0312723 | Time: 18989.6 ms [2021-05-02 14:22:43 main:581] : INFO : Epoch 8 | loss: 0.031026 | val_loss: 0.0312727 | Time: 19194.6 ms [2021-05-02 14:23:03 main:581] : INFO : Epoch 9 | loss: 0.0310266 | val_loss: 0.0312743 | Time: 19501.5 ms [2021-05-02 14:23:21 main:581] : INFO : Epoch 10 | loss: 0.0310228 | val_loss: 0.0312596 | Time: 18932.4 ms [2021-05-02 14:23:40 main:581] : INFO : Epoch 11 | loss: 0.0310202 | val_loss: 0.0312746 | Time: 19064.3 ms [2021-05-02 14:24:00 main:581] : INFO : Epoch 12 | loss: 0.0310209 | val_loss: 0.031269 | Time: 19166 ms [2021-05-02 14:24:19 main:581] : INFO : Epoch 13 | loss: 0.031024 | val_loss: 0.031262 | Time: 18904 ms [2021-05-02 14:24:38 main:581] : INFO : Epoch 14 | loss: 0.0310332 | val_loss: 0.031259 | Time: 19054.7 ms [2021-05-02 14:24:57 main:581] : INFO : Epoch 15 | loss: 0.0310287 | val_loss: 0.0312828 | Time: 18898.2 ms [2021-05-02 14:25:15 main:581] : INFO : Epoch 16 | loss: 0.031025 | val_loss: 0.031253 | Time: 18941.1 ms [2021-05-02 14:25:34 main:581] : INFO : Epoch 17 | loss: 0.0310298 | val_loss: 0.0312843 | Time: 18968.4 ms [2021-05-02 14:25:54 main:581] : INFO : Epoch 18 | loss: 0.0310318 | val_loss: 0.0312535 | Time: 19071.5 ms [2021-05-02 14:26:12 main:581] : INFO : Epoch 19 | loss: 0.0310284 | val_loss: 0.031273 | Time: 18936.3 ms [2021-05-02 14:26:31 main:581] : INFO : Epoch 20 | loss: 0.0310193 | val_loss: 0.031272 | Time: 18845.4 ms [2021-05-02 14:26:50 main:581] : INFO : Epoch 21 | loss: 0.0310224 | val_loss: 0.031275 | Time: 18806 ms [2021-05-02 14:27:09 main:581] : INFO : Epoch 22 | loss: 0.0310241 | val_loss: 0.0312711 | Time: 18794 ms [2021-05-02 14:27:28 main:581] : INFO : Epoch 23 | loss: 0.03109 | val_loss: 0.0312241 | Time: 18960.1 ms [2021-05-02 14:27:47 main:581] : INFO : Epoch 24 | loss: 0.0310659 | val_loss: 0.0312482 | Time: 18945.1 ms [2021-05-02 14:28:06 main:581] : INFO : Epoch 25 | loss: 0.0310361 | val_loss: 0.0312513 | Time: 19049.2 ms [2021-05-02 14:28:25 main:581] : INFO : Epoch 26 | loss: 0.031029 | val_loss: 0.0312483 | Time: 19034.2 ms [2021-05-02 14:28:44 main:581] : INFO : Epoch 27 | loss: 0.0310244 | val_loss: 0.0312775 | Time: 18985.3 ms [2021-05-02 14:29:03 main:581] : INFO : Epoch 28 | loss: 0.0310209 | val_loss: 0.0312653 | Time: 18967.4 ms [2021-05-02 14:29:22 main:581] : INFO : Epoch 29 | loss: 0.031019 | val_loss: 0.0312714 | Time: 19019.5 ms [2021-05-02 14:29:41 main:581] : INFO : Epoch 30 | loss: 0.0310285 | val_loss: 0.0312528 | Time: 18944.4 ms [2021-05-02 14:30:00 main:581] : INFO : Epoch 31 | loss: 0.0310296 | val_loss: 0.0312589 | Time: 19070 ms [2021-05-02 14:30:19 main:581] : INFO : Epoch 32 | loss: 0.0310212 | val_loss: 0.0312779 | Time: 18980.3 ms [2021-05-02 14:30:38 main:581] : INFO : Epoch 33 | loss: 0.0310286 | val_loss: 0.0312546 | Time: 19009.4 ms [2021-05-02 14:30:57 main:581] : INFO : Epoch 34 | loss: 0.0310289 | val_loss: 0.0312794 | Time: 19226 ms [2021-05-02 14:31:16 main:581] : INFO : Epoch 35 | loss: 0.0310287 | val_loss: 0.0312544 | Time: 18936.7 ms [2021-05-02 14:31:35 main:581] : INFO : Epoch 36 | loss: 0.0310357 | val_loss: 0.0312676 | Time: 18960.4 ms [2021-05-02 14:31:54 main:581] : INFO : Epoch 37 | loss: 0.0310312 | val_loss: 0.0312585 | Time: 18995.7 ms [2021-05-02 14:32:13 main:581] : INFO : Epoch 38 | loss: 0.0310294 | val_loss: 0.0312663 | Time: 18975.6 ms [2021-05-02 14:32:32 main:581] : INFO : Epoch 39 | loss: 0.0310283 | val_loss: 0.0312656 | Time: 19006.8 ms [2021-05-02 14:32:51 main:581] : INFO : Epoch 40 | loss: 0.0310214 | val_loss: 0.0312691 | Time: 19093.9 ms [2021-05-02 14:33:10 main:581] : INFO : Epoch 41 | loss: 0.0310173 | val_loss: 0.0312788 | Time: 19072.4 ms [2021-05-02 14:33:29 main:581] : INFO : Epoch 42 | loss: 0.0310252 | val_loss: 0.0312694 | Time: 19033.8 ms [2021-05-02 14:33:48 main:581] : INFO : Epoch 43 | loss: 0.0310207 | val_loss: 0.0312721 | Time: 19052.2 ms [2021-05-02 14:34:07 main:581] : INFO : Epoch 44 | loss: 0.0310329 | val_loss: 0.0312473 | Time: 19219.1 ms [2021-05-02 14:34:27 main:581] : INFO : Epoch 45 | loss: 0.0310397 | val_loss: 0.0312827 | Time: 19076.4 ms [2021-05-02 14:34:45 main:581] : INFO : Epoch 46 | loss: 0.0310257 | val_loss: 0.0312813 | Time: 18927.3 ms [2021-05-02 14:35:04 main:581] : INFO : Epoch 47 | loss: 0.031021 | val_loss: 0.0312675 | Time: 18907 ms [2021-05-02 14:35:23 main:581] : INFO : Epoch 48 | loss: 0.0310206 | val_loss: 0.0312847 | Time: 18848.5 ms [2021-05-02 14:35:42 main:581] : INFO : Epoch 49 | loss: 0.0310195 | val_loss: 0.0312696 | Time: 18991.3 ms [2021-05-02 14:36:01 main:581] : INFO : Epoch 50 | loss: 0.0310236 | val_loss: 0.0312832 | Time: 18920.7 ms [2021-05-02 14:36:20 main:581] : INFO : Epoch 51 | loss: 0.0311232 | val_loss: 0.0312123 | Time: 18946.9 ms [2021-05-02 14:36:39 main:581] : INFO : Epoch 52 | loss: 0.0311395 | val_loss: 0.0312063 | Time: 18973.6 ms [2021-05-02 14:36:58 main:581] : INFO : Epoch 53 | loss: 0.0311174 | val_loss: 0.0312217 | Time: 18969.1 ms [2021-05-02 14:37:17 main:581] : INFO : Epoch 54 | loss: 0.0310925 | val_loss: 0.0312193 | Time: 19024.5 ms [2021-05-02 14:37:36 main:581] : INFO : Epoch 55 | loss: 0.0310673 | val_loss: 0.0312403 | Time: 18992.5 ms [2021-05-02 14:37:55 main:581] : INFO : Epoch 56 | loss: 0.0310486 | val_loss: 0.031252 | Time: 18970.2 ms [2021-05-02 14:38:14 main:581] : INFO : Epoch 57 | loss: 0.0310327 | val_loss: 0.0312538 | Time: 18900.7 ms [2021-05-02 14:38:33 main:581] : INFO : Epoch 58 | loss: 0.0310279 | val_loss: 0.0312601 | Time: 19014.5 ms [2021-05-02 14:38:52 main:581] : INFO : Epoch 59 | loss: 0.0310245 | val_loss: 0.0312799 | Time: 19037.7 ms [2021-05-02 14:39:11 main:581] : INFO : Epoch 60 | loss: 0.0310311 | val_loss: 0.0312726 | Time: 19006.7 ms [2021-05-02 14:39:30 main:581] : INFO : Epoch 61 | loss: 0.0310328 | val_loss: 0.0312588 | Time: 18835.7 ms [2021-05-02 14:39:49 main:581] : INFO : Epoch 62 | loss: 0.0310283 | val_loss: 0.0312699 | Time: 18986.3 ms [2021-05-02 14:40:08 main:581] : INFO : Epoch 63 | loss: 0.0310292 | val_loss: 0.0312581 | Time: 18977.2 ms [2021-05-02 14:40:27 main:581] : INFO : Epoch 64 | loss: 0.0310298 | val_loss: 0.0312739 | Time: 18950.8 ms [2021-05-02 14:40:46 main:581] : INFO : Epoch 65 | loss: 0.0310237 | val_loss: 0.0312696 | Time: 19084.9 ms [2021-05-02 14:41:05 main:581] : INFO : Epoch 66 | loss: 0.0310161 | val_loss: 0.0312703 | Time: 19000.7 ms [2021-05-02 14:41:24 main:581] : INFO : Epoch 67 | loss: 0.0310187 | val_loss: 0.0312869 | Time: 18789.8 ms [2021-05-02 14:41:42 main:581] : INFO : Epoch 68 | loss: 0.0310153 | val_loss: 0.0312637 | Time: 18898.5 ms [2021-05-02 14:42:01 main:581] : INFO : Epoch 69 | loss: 0.0310238 | val_loss: 0.0312719 | Time: 18801.6 ms [2021-05-02 14:42:20 main:581] : INFO : Epoch 70 | loss: 0.031029 | val_loss: 0.0312743 | Time: 18806.7 ms [2021-05-02 14:42:39 main:581] : INFO : Epoch 71 | loss: 0.0310269 | val_loss: 0.0312709 | Time: 18775.5 ms [2021-05-02 14:42:58 main:581] : INFO : Epoch 72 | loss: 0.0310295 | val_loss: 0.031269 | Time: 18820.9 ms [2021-05-02 14:43:18 main:581] : INFO : Epoch 73 | loss: 0.0310305 | val_loss: 0.0312693 | Time: 20038.8 ms [2021-05-02 14:43:41 main:581] : INFO : Epoch 74 | loss: 0.0310293 | val_loss: 0.0312754 | Time: 22862.7 ms [2021-05-02 14:44:01 main:581] : INFO : Epoch 75 | loss: 0.0310266 | val_loss: 0.03126 | Time: 20673.6 ms [2021-05-02 14:44:20 main:581] : INFO : Epoch 76 | loss: 0.0310207 | val_loss: 0.0312736 | Time: 18449.7 ms [2021-05-02 14:44:38 main:581] : INFO : Epoch 77 | loss: 0.0310227 | val_loss: 0.0312897 | Time: 18332.3 ms DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__rocm -c --maxepoch 128 nthreads: 1 gpudev: 0 Re-exec()-ing to set environment correctly 14:58:37 (3650): start_timer_thread(): pthread_create(): 22Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: Vega 10 XL/XT [Radeon RX Vega 56/64]) [2021-05-02 14:58:37 main:442] : INFO : Set logging level to 1 [2021-05-02 14:58:37 main:448] : INFO : Running in BOINC Client mode [2021-05-02 14:58:37 main:451] : INFO : Resolving all filenames [2021-05-02 14:58:37 main:459] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 (exists = 1) [2021-05-02 14:58:37 main:459] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42_5_r1367993063_1 (exists = 1) [2021-05-02 14:58:37 main:459] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42_5_r1367993063_0 (exists = 0) [2021-05-02 14:58:37 main:459] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42 (exists = 1) [2021-05-02 14:58:37 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0) [2021-05-02 14:58:37 main:479] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 [2021-05-02 14:58:37 main:481] : INFO : Configuration: [2021-05-02 14:58:37 main:482] : INFO : Model type: GRU [2021-05-02 14:58:37 main:483] : INFO : Validation Loss Threshold: 0.0001 [2021-05-02 14:58:37 main:484] : INFO : Max Epochs: 128 [2021-05-02 14:58:37 main:485] : INFO : Batch Size: 128 [2021-05-02 14:58:37 main:486] : INFO : Learning Rate: 0.01 [2021-05-02 14:58:37 main:487] : INFO : Patience: 10 [2021-05-02 14:58:37 main:488] : INFO : Hidden Width: 12 [2021-05-02 14:58:37 main:489] : INFO : # Recurrent Layers: 4 [2021-05-02 14:58:37 main:490] : INFO : # Backend Layers: 4 [2021-05-02 14:58:37 main:491] : INFO : # Threads: 1 [2021-05-02 14:58:37 main:493] : INFO : Preparing Dataset [2021-05-02 14:58:37 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 14:58:38 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 14:58:39 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2021-05-02 14:58:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 14:58:39 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 14:58:39 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2021-05-02 14:58:39 main:501] : INFO : Creating Model [2021-05-02 14:58:39 main:514] : INFO : Preparing config file [2021-05-02 14:58:39 main:526] : INFO : Creating new config file [2021-05-02 14:58:39 main:545] : INFO : This is a continuation WU, loading previous network [2021-05-02 14:58:39 main:566] : INFO : Loading DataLoader into Memory [2021-05-02 14:58:39 main:569] : INFO : Starting Training [2021-05-02 14:59:04 main:581] : INFO : Epoch 1 | loss: 0.0311165 | val_loss: 0.0312553 | Time: 25054 ms [2021-05-02 14:59:23 main:581] : INFO : Epoch 2 | loss: 0.031065 | val_loss: 0.0312541 | Time: 18727.4 ms [2021-05-02 14:59:41 main:581] : INFO : Epoch 3 | loss: 0.0310414 | val_loss: 0.0312541 | Time: 18716.3 ms [2021-05-02 15:00:00 main:581] : INFO : Epoch 4 | loss: 0.0310331 | val_loss: 0.0312551 | Time: 18828.2 ms [2021-05-02 15:00:19 main:581] : INFO : Epoch 5 | loss: 0.0310265 | val_loss: 0.0312697 | Time: 18964.3 ms [2021-05-02 15:00:38 main:581] : INFO : Epoch 6 | loss: 0.0310249 | val_loss: 0.0312544 | Time: 18907.4 ms [2021-05-02 15:00:57 main:581] : INFO : Epoch 7 | loss: 0.03103 | val_loss: 0.0312699 | Time: 18791.8 ms [2021-05-02 15:01:16 main:581] : INFO : Epoch 8 | loss: 0.0310238 | val_loss: 0.0312675 | Time: 18813.5 ms [2021-05-02 15:01:34 main:581] : INFO : Epoch 9 | loss: 0.0310222 | val_loss: 0.0312718 | Time: 18646.3 ms [2021-05-02 15:01:57 main:581] : INFO : Epoch 10 | loss: 0.0310203 | val_loss: 0.0312857 | Time: 22633.3 ms [2021-05-02 15:02:23 main:581] : INFO : Epoch 11 | loss: 0.0310239 | val_loss: 0.0312869 | Time: 25977.5 ms DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__rocm -c --maxepoch 128 nthreads: 1 gpudev: 0 Re-exec()-ing to set environment correctly 15:07:25 (44126): start_timer_thread(): pthread_create(): 22Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: Vega 10 XL/XT [Radeon RX Vega 56/64]) [2021-05-02 15:07:25 main:442] : INFO : Set logging level to 1 [2021-05-02 15:07:25 main:448] : INFO : Running in BOINC Client mode [2021-05-02 15:07:25 main:451] : INFO : Resolving all filenames [2021-05-02 15:07:25 main:459] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 (exists = 1) [2021-05-02 15:07:25 main:459] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42_5_r1367993063_1 (exists = 1) [2021-05-02 15:07:25 main:459] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42_5_r1367993063_0 (exists = 0) [2021-05-02 15:07:25 main:459] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1607233796-4364-42 (exists = 1) [2021-05-02 15:07:25 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0) [2021-05-02 15:07:25 main:479] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 [2021-05-02 15:07:25 main:481] : INFO : Configuration: [2021-05-02 15:07:25 main:482] : INFO : Model type: GRU [2021-05-02 15:07:25 main:483] : INFO : Validation Loss Threshold: 0.0001 [2021-05-02 15:07:25 main:484] : INFO : Max Epochs: 128 [2021-05-02 15:07:25 main:485] : INFO : Batch Size: 128 [2021-05-02 15:07:25 main:486] : INFO : Learning Rate: 0.01 [2021-05-02 15:07:25 main:487] : INFO : Patience: 10 [2021-05-02 15:07:25 main:488] : INFO : Hidden Width: 12 [2021-05-02 15:07:25 main:489] : INFO : # Recurrent Layers: 4 [2021-05-02 15:07:25 main:490] : INFO : # Backend Layers: 4 [2021-05-02 15:07:25 main:491] : INFO : # Threads: 1 [2021-05-02 15:07:25 main:493] : INFO : Preparing Dataset [2021-05-02 15:07:25 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 15:07:25 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 15:07:26 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2021-05-02 15:07:26 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 15:07:26 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2021-05-02 15:07:26 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2021-05-02 15:07:26 main:501] : INFO : Creating Model [2021-05-02 15:07:26 main:514] : INFO : Preparing config file [2021-05-02 15:07:26 main:526] : INFO : Creating new config file [2021-05-02 15:07:26 main:545] : INFO : This is a continuation WU, loading previous network [2021-05-02 15:07:26 main:566] : INFO : Loading DataLoader into Memory [2021-05-02 15:07:26 main:569] : INFO : Starting Training [2021-05-02 15:07:50 main:581] : INFO : Epoch 1 | loss: 0.0311071 | val_loss: 0.0312665 | Time: 23637.4 ms [2021-05-02 15:08:08 main:581] : INFO : Epoch 2 | loss: 0.0310559 | val_loss: 0.0312706 | Time: 18613.3 ms [2021-05-02 15:08:27 main:581] : INFO : Epoch 3 | loss: 0.0310346 | val_loss: 0.0312556 | Time: 18651.7 ms [2021-05-02 15:08:46 main:581] : INFO : Epoch 4 | loss: 0.0310321 | val_loss: 0.0312647 | Time: 18772 ms [2021-05-02 15:09:05 main:581] : INFO : Epoch 5 | loss: 0.0310281 | val_loss: 0.0312679 | Time: 19119.9 ms [2021-05-02 15:09:24 main:581] : INFO : Epoch 6 | loss: 0.0310232 | val_loss: 0.0312703 | Time: 18719.4 ms [2021-05-02 15:09:42 main:581] : INFO : Epoch 7 | loss: 0.0310264 | val_loss: 0.0312712 | Time: 18650.4 ms [2021-05-02 15:10:01 main:581] : INFO : Epoch 8 | loss: 0.03103 | val_loss: 0.0312586 | Time: 18714.6 ms [2021-05-02 15:10:20 main:581] : INFO : Epoch 9 | loss: 0.0310534 | val_loss: 0.0312551 | Time: 18711.1 ms [2021-05-02 15:10:38 main:581] : INFO : Epoch 10 | loss: 0.0310386 | val_loss: 0.0312554 | Time: 18702.6 ms [2021-05-02 15:10:58 main:581] : INFO : Epoch 11 | loss: 0.0310288 | val_loss: 0.0312792 | Time: 19572.5 ms [2021-05-02 15:11:17 main:581] : INFO : Epoch 12 | loss: 0.031023 | val_loss: 0.0312747 | Time: 18680.3 ms [2021-05-02 15:11:35 main:581] : INFO : Epoch 13 | loss: 0.0310192 | val_loss: 0.0312666 | Time: 18778.1 ms [2021-05-02 15:11:54 main:581] : INFO : Epoch 14 | loss: 0.0310265 | val_loss: 0.0312657 | Time: 18983.5 ms [2021-05-02 15:12:13 main:581] : INFO : Epoch 15 | loss: 0.0310221 | val_loss: 0.0312825 | Time: 18950.7 ms [2021-05-02 15:12:32 main:581] : INFO : Epoch 16 | loss: 0.0310269 | val_loss: 0.031254 | Time: 18893.4 ms [2021-05-02 15:12:51 main:581] : INFO : Epoch 17 | loss: 0.0310298 | val_loss: 0.0312715 | Time: 18748.1 ms [2021-05-02 15:13:10 main:581] : INFO : Epoch 18 | loss: 0.0310264 | val_loss: 0.0312716 | Time: 18780.7 ms [2021-05-02 15:13:29 main:581] : INFO : Epoch 19 | loss: 0.0310205 | val_loss: 0.031266 | Time: 18826.2 ms [2021-05-02 15:13:47 main:581] : INFO : Epoch 20 | loss: 0.0310204 | val_loss: 0.0312722 | Time: 18690.2 ms [2021-05-02 15:14:06 main:581] : INFO : Epoch 21 | loss: 0.0310194 | val_loss: 0.0312635 | Time: 18708.8 ms [2021-05-02 15:14:25 main:581] : INFO : Epoch 22 | loss: 0.0310209 | val_loss: 0.0312623 | Time: 18717.9 ms [2021-05-02 15:14:43 main:581] : INFO : Epoch 23 | loss: 0.0310251 | val_loss: 0.0312722 | Time: 18589.7 ms [2021-05-02 15:15:02 main:581] : INFO : Epoch 24 | loss: 0.0310288 | val_loss: 0.0312677 | Time: 18847.8 ms [2021-05-02 15:15:21 main:581] : INFO : Epoch 25 | loss: 0.0310241 | val_loss: 0.0312756 | Time: 18718.5 ms [2021-05-02 15:15:40 main:581] : INFO : Epoch 26 | loss: 0.0310196 | val_loss: 0.0312529 | Time: 18756.1 ms [2021-05-02 15:15:58 main:581] : INFO : Epoch 27 | loss: 0.0310219 | val_loss: 0.0312809 | Time: 18641.2 ms [2021-05-02 15:16:17 main:581] : INFO : Epoch 28 | loss: 0.0310244 | val_loss: 0.031278 | Time: 18745.5 ms [2021-05-02 15:16:36 main:581] : INFO : Epoch 29 | loss: 0.0310277 | val_loss: 0.0312664 | Time: 18646 ms [2021-05-02 15:16:54 main:581] : INFO : Epoch 30 | loss: 0.0310245 | val_loss: 0.0312551 | Time: 18681.8 ms [2021-05-02 15:17:13 main:581] : INFO : Epoch 31 | loss: 0.0310259 | val_loss: 0.031271 | Time: 18696.7 ms [2021-05-02 15:17:32 main:581] : INFO : Epoch 32 | loss: 0.0310249 | val_loss: 0.0312526 | Time: 18726.9 ms [2021-05-02 15:17:51 main:581] : INFO : Epoch 33 | loss: 0.0310289 | val_loss: 0.0312532 | Time: 18715.6 ms [2021-05-02 15:18:09 main:581] : INFO : Epoch 34 | loss: 0.0310286 | val_loss: 0.0312796 | Time: 18849 ms [2021-05-02 15:18:28 main:581] : INFO : Epoch 35 | loss: 0.0310285 | val_loss: 0.0312706 | Time: 18726.1 ms [2021-05-02 15:18:47 main:581] : INFO : Epoch 36 | loss: 0.031021 | val_loss: 0.0312647 | Time: 18755.2 ms [2021-05-02 15:19:06 main:581] : INFO : Epoch 37 | loss: 0.0310211 | val_loss: 0.0312767 | Time: 18744 ms [2021-05-02 15:19:24 main:581] : INFO : Epoch 38 | loss: 0.0310288 | val_loss: 0.0312619 | Time: 18839.8 ms [2021-05-02 15:19:43 main:581] : INFO : Epoch 39 | loss: 0.0310307 | val_loss: 0.0312506 | Time: 18811.8 ms [2021-05-02 15:20:02 main:581] : INFO : Epoch 40 | loss: 0.0310247 | val_loss: 0.031279 | Time: 18771.3 ms [2021-05-02 15:20:21 main:581] : INFO : Epoch 41 | loss: 0.0310241 | val_loss: 0.0312684 | Time: 18691.2 ms [2021-05-02 15:20:39 main:581] : INFO : Epoch 42 | loss: 0.0310169 | val_loss: 0.031283 | Time: 18658.2 ms [2021-05-02 15:20:58 main:581] : INFO : Epoch 43 | loss: 0.0310236 | val_loss: 0.0312676 | Time: 18730.3 ms [2021-05-02 15:21:17 main:581] : INFO : Epoch 44 | loss: 0.0310335 | val_loss: 0.0312684 | Time: 18797 ms [2021-05-02 15:21:36 main:581] : INFO : Epoch 45 | loss: 0.0310327 | val_loss: 0.0312527 | Time: 18680.8 ms [2021-05-02 15:21:54 main:581] : INFO : Epoch 46 | loss: 0.0310246 | val_loss: 0.0312649 | Time: 18666.3 ms [2021-05-02 15:22:13 main:581] : INFO : Epoch 47 | loss: 0.0310218 | val_loss: 0.0312666 | Time: 18718.8 ms [2021-05-02 15:22:32 main:581] : INFO : Epoch 48 | loss: 0.0310276 | val_loss: 0.0312769 | Time: 18700.1 ms [2021-05-02 15:22:51 main:581] : INFO : Epoch 49 | loss: 0.0310175 | val_loss: 0.0312789 | Time: 18893.3 ms [2021-05-02 15:23:09 main:581] : INFO : Epoch 50 | loss: 0.0310176 | val_loss: 0.0312898 | Time: 18731.1 ms [2021-05-02 15:23:28 main:581] : INFO : Epoch 51 | loss: 0.0310244 | val_loss: 0.0312638 | Time: 18700 ms [2021-05-02 15:23:47 main:581] : INFO : Epoch 52 | loss: 0.0310275 | val_loss: 0.0312541 | Time: 18702.6 ms [2021-05-02 15:24:05 main:581] : INFO : Epoch 53 | loss: 0.0310264 | val_loss: 0.0312694 | Time: 18669.3 ms [2021-05-02 15:24:24 main:581] : INFO : Epoch 54 | loss: 0.0310287 | val_loss: 0.0312772 | Time: 18652.7 ms [2021-05-02 15:24:43 main:581] : INFO : Epoch 55 | loss: 0.0310279 | val_loss: 0.0312969 | Time: 18917 ms [2021-05-02 15:25:02 main:581] : INFO : Epoch 56 | loss: 0.0310207 | val_loss: 0.0312677 | Time: 18672.9 ms [2021-05-02 15:25:20 main:581] : INFO : Epoch 57 | loss: 0.031023 | val_loss: 0.0312786 | Time: 18579.5 ms [2021-05-02 15:25:39 main:581] : INFO : Epoch 58 | loss: 0.0310202 | val_loss: 0.0312861 | Time: 18652 ms [2021-05-02 15:25:58 main:581] : INFO : Epoch 59 | loss: 0.0310248 | val_loss: 0.03125 | Time: 18843.8 ms [2021-05-02 15:26:17 main:581] : INFO : Epoch 60 | loss: 0.0310271 | val_loss: 0.0312609 | Time: 18848.1 ms [2021-05-02 15:26:35 main:581] : INFO : Epoch 61 | loss: 0.0310231 | val_loss: 0.031274 | Time: 18652.5 ms [2021-05-02 15:26:54 main:581] : INFO : Epoch 62 | loss: 0.0310299 | val_loss: 0.0312626 | Time: 18779.8 ms [2021-05-02 15:27:13 main:581] : INFO : Epoch 63 | loss: 0.0310289 | val_loss: 0.0312545 | Time: 18811.8 ms [2021-05-02 15:27:31 main:581] : INFO : Epoch 64 | loss: 0.0310258 | val_loss: 0.0312641 | Time: 18674.7 ms [2021-05-02 15:27:50 main:581] : INFO : Epoch 65 | loss: 0.0310196 | val_loss: 0.0312662 | Time: 18731.8 ms [2021-05-02 15:28:09 main:581] : INFO : Epoch 66 | loss: 0.0310372 | val_loss: 0.0312376 | Time: 18640.1 ms [2021-05-02 15:28:28 main:581] : INFO : Epoch 67 | loss: 0.0310477 | val_loss: 0.0312549 | Time: 18764.9 ms [2021-05-02 15:28:46 main:581] : INFO : Epoch 68 | loss: 0.0310304 | val_loss: 0.0312659 | Time: 18654 ms [2021-05-02 15:29:05 main:581] : INFO : Epoch 69 | loss: 0.0310263 | val_loss: 0.0312749 | Time: 18875.9 ms [2021-05-02 15:29:24 main:581] : INFO : Epoch 70 | loss: 0.0310201 | val_loss: 0.0312781 | Time: 18666.1 ms [2021-05-02 15:29:42 main:581] : INFO : Epoch 71 | loss: 0.0310239 | val_loss: 0.0312774 | Time: 18569.5 ms [2021-05-02 15:30:01 main:581] : INFO : Epoch 72 | loss: 0.0310309 | val_loss: 0.0312652 | Time: 18661.3 ms [2021-05-02 15:30:20 main:581] : INFO : Epoch 73 | loss: 0.0310357 | val_loss: 0.0312515 | Time: 18719.5 ms [2021-05-02 15:30:38 main:581] : INFO : Epoch 74 | loss: 0.0310339 | val_loss: 0.0312704 | Time: 18656.6 ms [2021-05-02 15:30:57 main:581] : INFO : Epoch 75 | loss: 0.0310338 | val_loss: 0.03125 | Time: 18683.7 ms [2021-05-02 15:31:16 main:581] : INFO : Epoch 76 | loss: 0.0310296 | val_loss: 0.0312888 | Time: 18627 ms [2021-05-02 15:31:34 main:581] : INFO : Epoch 77 | loss: 0.0310252 | val_loss: 0.0312494 | Time: 18720.6 ms [2021-05-02 15:31:53 main:581] : INFO : Epoch 78 | loss: 0.0310286 | val_loss: 0.0312836 | Time: 18833.4 ms [2021-05-02 15:32:12 main:581] : INFO : Epoch 79 | loss: 0.0310249 | val_loss: 0.0312865 | Time: 18635.6 ms [2021-05-02 15:32:30 main:581] : INFO : Epoch 80 | loss: 0.0310211 | val_loss: 0.0312713 | Time: 18619.8 ms [2021-05-02 15:32:49 main:581] : INFO : Epoch 81 | loss: 0.0310151 | val_loss: 0.0312881 | Time: 18650.4 ms [2021-05-02 15:33:08 main:581] : INFO : Epoch 82 | loss: 0.0310168 | val_loss: 0.0312902 | Time: 18677.5 ms [2021-05-02 15:33:26 main:581] : INFO : Epoch 83 | loss: 0.0310218 | val_loss: 0.0312651 | Time: 18665.4 ms [2021-05-02 15:33:45 main:581] : INFO : Epoch 84 | loss: 0.0310265 | val_loss: 0.0312761 | Time: 18652.3 ms [2021-05-02 15:34:04 main:581] : INFO : Epoch 85 | loss: 0.0310277 | val_loss: 0.0313001 | Time: 18709 ms [2021-05-02 15:34:23 main:581] : INFO : Epoch 86 | loss: 0.0310222 | val_loss: 0.0312675 | Time: 18859 ms [2021-05-02 15:34:41 main:581] : INFO : Epoch 87 | loss: 0.0310214 | val_loss: 0.0312659 | Time: 18774.7 ms [2021-05-02 15:35:00 main:581] : INFO : Epoch 88 | loss: 0.0310213 | val_loss: 0.0312821 | Time: 18782.5 ms [2021-05-02 15:35:19 main:581] : INFO : Epoch 89 | loss: 0.0310184 | val_loss: 0.0312754 | Time: 18759.1 ms [2021-05-02 15:35:38 main:581] : INFO : Epoch 90 | loss: 0.0310203 | val_loss: 0.0312719 | Time: 18680 ms [2021-05-02 15:35:57 main:581] : INFO : Epoch 91 | loss: 0.0310156 | val_loss: 0.0312804 | Time: 18825.6 ms [2021-05-02 15:36:15 main:581] : INFO : Epoch 92 | loss: 0.0310173 | val_loss: 0.031268 | Time: 18625 ms [2021-05-02 15:36:34 main:581] : INFO : Epoch 93 | loss: 0.0310224 | val_loss: 0.0312601 | Time: 18693.5 ms [2021-05-02 15:36:53 main:581] : INFO : Epoch 94 | loss: 0.0310225 | val_loss: 0.0312865 | Time: 18802.7 ms [2021-05-02 15:37:11 main:581] : INFO : Epoch 95 | loss: 0.0310193 | val_loss: 0.0312745 | Time: 18776.8 ms [2021-05-02 15:37:30 main:581] : INFO : Epoch 96 | loss: 0.0310199 | val_loss: 0.0312748 | Time: 18702.2 ms [2021-05-02 15:37:49 main:581] : INFO : Epoch 97 | loss: 0.0310213 | val_loss: 0.0312763 | Time: 18663.4 ms [2021-05-02 15:38:08 main:581] : INFO : Epoch 98 | loss: 0.0310193 | val_loss: 0.0312616 | Time: 18782.6 ms [2021-05-02 15:38:26 main:581] : INFO : Epoch 99 | loss: 0.0310322 | val_loss: 0.031269 | Time: 18663.5 ms [2021-05-02 15:38:45 main:581] : INFO : Epoch 100 | loss: 0.0310301 | val_loss: 0.031269 | Time: 18629.5 ms [2021-05-02 15:39:04 main:581] : INFO : Epoch 101 | loss: 0.031028 | val_loss: 0.0312794 | Time: 18627.9 ms [2021-05-02 15:39:22 main:581] : INFO : Epoch 102 | loss: 0.0310245 | val_loss: 0.031263 | Time: 18710.6 ms [2021-05-02 15:39:41 main:581] : INFO : Epoch 103 | loss: 0.0310256 | val_loss: 0.0312709 | Time: 18638.1 ms [2021-05-02 15:40:00 main:581] : INFO : Epoch 104 | loss: 0.031022 | val_loss: 0.0312627 | Time: 18728 ms [2021-05-02 15:40:18 main:581] : INFO : Epoch 105 | loss: 0.0310274 | val_loss: 0.03126 | Time: 18700 ms [2021-05-02 15:40:37 main:581] : INFO : Epoch 106 | loss: 0.0310218 | val_loss: 0.0312776 | Time: 18690.1 ms [2021-05-02 15:40:56 main:581] : INFO : Epoch 107 | loss: 0.0310184 | val_loss: 0.0312781 | Time: 18782.6 ms [2021-05-02 15:41:15 main:581] : INFO : Epoch 108 | loss: 0.0310207 | val_loss: 0.0312564 | Time: 18775.8 ms [2021-05-02 15:41:33 main:581] : INFO : Epoch 109 | loss: 0.0310169 | val_loss: 0.0312828 | Time: 18657.1 ms [2021-05-02 15:41:52 main:581] : INFO : Epoch 110 | loss: 0.0310239 | val_loss: 0.0312537 | Time: 18737.5 ms [2021-05-02 15:42:11 main:581] : INFO : Epoch 111 | loss: 0.0310177 | val_loss: 0.0312832 | Time: 18605.9 ms [2021-05-02 15:42:29 main:581] : INFO : Epoch 112 | loss: 0.0310145 | val_loss: 0.0312783 | Time: 18601 ms [2021-05-02 15:42:48 main:581] : INFO : Epoch 113 | loss: 0.0310139 | val_loss: 0.0312791 | Time: 18742.8 ms [2021-05-02 15:43:07 main:581] : INFO : Epoch 114 | loss: 0.0310233 | val_loss: 0.0312578 | Time: 18644.9 ms [2021-05-02 15:43:25 main:581] : INFO : Epoch 115 | loss: 0.031022 | val_loss: 0.0312762 | Time: 18601.2 ms [2021-05-02 15:43:44 main:581] : INFO : Epoch 116 | loss: 0.0310218 | val_loss: 0.0312648 | Time: 18767 ms [2021-05-02 15:44:03 main:581] : INFO : Epoch 117 | loss: 0.0310181 | val_loss: 0.0312723 | Time: 18899.9 ms [2021-05-02 15:44:22 main:581] : INFO : Epoch 118 | loss: 0.0310274 | val_loss: 0.0312808 | Time: 18753.3 ms [2021-05-02 15:44:40 main:581] : INFO : Epoch 119 | loss: 0.0310186 | val_loss: 0.0312691 | Time: 18745.8 ms [2021-05-02 15:44:59 main:581] : INFO : Epoch 120 | loss: 0.0310228 | val_loss: 0.0312624 | Time: 18803.5 ms [2021-05-02 15:45:18 main:581] : INFO : Epoch 121 | loss: 0.0310253 | val_loss: 0.0312672 | Time: 18743.1 ms [2021-05-02 15:45:37 main:581] : INFO : Epoch 122 | loss: 0.0310186 | val_loss: 0.031283 | Time: 18669.6 ms [2021-05-02 15:45:55 main:581] : INFO : Epoch 123 | loss: 0.031025 | val_loss: 0.0312764 | Time: 18936.2 ms [2021-05-02 15:46:14 main:581] : INFO : Epoch 124 | loss: 0.0310329 | val_loss: 0.0312557 | Time: 18688.1 ms [2021-05-02 15:46:33 main:581] : INFO : Epoch 125 | loss: 0.0310308 | val_loss: 0.0312765 | Time: 18787 ms [2021-05-02 15:46:52 main:581] : INFO : Epoch 126 | loss: 0.031024 | val_loss: 0.0312791 | Time: 18667.2 ms [2021-05-02 15:47:11 main:581] : INFO : Epoch 127 | loss: 0.0310224 | val_loss: 0.0312668 | Time: 18960.7 ms [2021-05-02 15:47:30 main:581] : INFO : Epoch 128 | loss: 0.0310252 | val_loss: 0.0312724 | Time: 19148.1 ms [2021-05-02 15:47:30 main:604] : INFO : Saving trained model to model-final.pt, val_loss 0.0312724 [2021-05-02 15:47:30 main:610] : INFO : Saving end state to config to file [2021-05-02 15:47:30 main:615] : INFO : Success, exiting.. 15:47:30 (44126): called boinc_finish(0) </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)