Task 3453877
| Name | ParityModified-1605453627-20422-6_7 |
| Workunit | 1630998 |
| Created | 28 Dec 2020, 16:49:47 UTC |
| Sent | 28 Dec 2020, 17:48:38 UTC |
| Report deadline | 4 Jan 2021, 17:48:38 UTC |
| Received | 28 Dec 2020, 18:01:37 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Aborted by user |
| Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
| Computer ID | 7050 |
| Run time | 6 min 21 sec |
| CPU time | 6 min 20 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 4,233.48 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.80 (cuda10200) x86_64-pc-linux-gnu |
| Peak working set size | 1.83 GB |
| Peak swap size | 13.39 GB |
| Peak disk usage | 2.98 GB |
Stderr output
<core_client_version>7.16.6</core_client_version> <![CDATA[ <message> aborted by user</message> <stderr_txt> DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 1024 nthreads: 1 gpudev: 0 Re-exec()-ing to set environment correctly Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: GeForce GTX 1060 3GB) [2020-12-28 18:48:44 main:442] : INFO : Set logging level to 1 [2020-12-28 18:48:44 main:448] : INFO : Running in BOINC Client mode [2020-12-28 18:48:44 main:451] : INFO : Resolving all filenames [2020-12-28 18:48:44 main:459] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 (exists = 1) [2020-12-28 18:48:44 main:459] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1605453627-20422-6_7_r164966966_1 (exists = 0) [2020-12-28 18:48:44 main:459] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1605453627-20422-6_7_r164966966_0 (exists = 0) [2020-12-28 18:48:44 main:459] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1605453627-20422-6 (exists = 1) [2020-12-28 18:48:44 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0) [2020-12-28 18:48:44 main:479] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 [2020-12-28 18:48:44 main:481] : INFO : Configuration: [2020-12-28 18:48:44 main:482] : INFO : Model type: GRU [2020-12-28 18:48:44 main:483] : INFO : Validation Loss Threshold: 0.0001 [2020-12-28 18:48:44 main:484] : INFO : Max Epochs: 1024 [2020-12-28 18:48:44 main:485] : INFO : Batch Size: 128 [2020-12-28 18:48:44 main:486] : INFO : Learning Rate: 0.01 [2020-12-28 18:48:44 main:487] : INFO : Patience: 10 [2020-12-28 18:48:44 main:488] : INFO : Hidden Width: 12 [2020-12-28 18:48:44 main:489] : INFO : # Recurrent Layers: 4 [2020-12-28 18:48:44 main:490] : INFO : # Backend Layers: 4 [2020-12-28 18:48:44 main:491] : INFO : # Threads: 1 [2020-12-28 18:48:44 main:493] : INFO : Preparing Dataset [2020-12-28 18:48:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2020-12-28 18:48:44 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2020-12-28 18:48:45 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2020-12-28 18:48:45 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2020-12-28 18:48:45 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2020-12-28 18:48:45 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2020-12-28 18:48:45 main:501] : INFO : Creating Model [2020-12-28 18:48:45 main:514] : INFO : Preparing config file [2020-12-28 18:48:45 main:526] : INFO : Creating new config file [2020-12-28 18:48:45 main:545] : INFO : This is a continuation WU, loading previous network [2020-12-28 18:48:45 main:566] : INFO : Loading DataLoader into Memory [2020-12-28 18:48:45 main:569] : INFO : Starting Training [2020-12-28 18:48:47 main:581] : INFO : Epoch 1 | loss: 0.0312393 | val_loss: 0.0312134 | Time: 1896.99 ms [2020-12-28 18:48:49 main:581] : INFO : Epoch 2 | loss: nan | val_loss: nan | Time: 1735.95 ms [2020-12-28 18:48:51 main:581] : INFO : Epoch 3 | loss: nan | val_loss: nan | Time: 1669.14 ms [2020-12-28 18:48:52 main:581] : INFO : Epoch 4 | loss: nan | val_loss: nan | Time: 1708.17 ms [2020-12-28 18:48:54 main:581] : INFO : Epoch 5 | loss: nan | val_loss: nan | Time: 1727.63 ms [2020-12-28 18:48:56 main:581] : INFO : Epoch 6 | loss: nan | val_loss: nan | Time: 1726.7 ms [2020-12-28 18:48:58 main:581] : INFO : Epoch 7 | loss: nan | val_loss: nan | Time: 1762.46 ms [2020-12-28 18:48:59 main:581] : INFO : Epoch 8 | loss: nan | val_loss: nan | Time: 1686.25 ms [2020-12-28 18:49:01 main:581] : INFO : Epoch 9 | loss: nan | val_loss: nan | Time: 1635.26 ms [2020-12-28 18:49:03 main:581] : INFO : Epoch 10 | loss: nan | val_loss: nan | Time: 1690.62 ms [2020-12-28 18:49:04 main:581] : INFO : Epoch 11 | loss: nan | val_loss: nan | Time: 1699.93 ms [2020-12-28 18:49:06 main:581] : INFO : Epoch 12 | loss: nan | val_loss: nan | Time: 1689.13 ms [2020-12-28 18:49:08 main:581] : INFO : Epoch 13 | loss: nan | val_loss: nan | Time: 1705.97 ms [2020-12-28 18:49:09 main:581] : INFO : Epoch 14 | loss: nan | val_loss: nan | Time: 1737.55 ms [2020-12-28 18:49:11 main:581] : INFO : Epoch 15 | loss: nan | val_loss: nan | Time: 1698.81 ms [2020-12-28 18:49:13 main:581] : INFO : Epoch 16 | loss: nan | val_loss: nan | Time: 1665.3 ms [2020-12-28 18:49:14 main:581] : INFO : Epoch 17 | loss: nan | val_loss: nan | Time: 1700.57 ms [2020-12-28 18:49:16 main:581] : INFO : Epoch 18 | loss: nan | val_loss: nan | Time: 1703.74 ms SIGSEGV: segmentation violation Stack trace (4 frames): ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x55c92bdf4f9c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f9e1d1123c0] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0x50)[0x7f9d9ea3c0e0] ./libtorch_cuda.so(+0x7d5e13)[0x7f9d9f7eae13] Exiting... DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 1024 nthreads: 1 gpudev: 0 Re-exec()-ing to set environment correctly Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: GeForce GTX 1060 3GB) [2020-12-28 18:53:51 main:442] : INFO : Set logging level to 1 [2020-12-28 18:53:51 main:448] : INFO : Running in BOINC Client mode [2020-12-28 18:53:51 main:451] : INFO : Resolving all filenames [2020-12-28 18:53:51 main:459] : INFO : Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 (exists = 1) [2020-12-28 18:53:51 main:459] : INFO : Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1605453627-20422-6_7_r164966966_1 (exists = 1) [2020-12-28 18:53:51 main:459] : INFO : Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1605453627-20422-6_7_r164966966_0 (exists = 0) [2020-12-28 18:53:51 main:459] : INFO : Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityModified-1605453627-20422-6 (exists = 1) [2020-12-28 18:53:51 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0) [2020-12-28 18:53:51 main:479] : INFO : Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 [2020-12-28 18:53:51 main:481] : INFO : Configuration: [2020-12-28 18:53:51 main:482] : INFO : Model type: GRU [2020-12-28 18:53:51 main:483] : INFO : Validation Loss Threshold: 0.0001 [2020-12-28 18:53:51 main:484] : INFO : Max Epochs: 1024 [2020-12-28 18:53:51 main:485] : INFO : Batch Size: 128 [2020-12-28 18:53:51 main:486] : INFO : Learning Rate: 0.01 [2020-12-28 18:53:51 main:487] : INFO : Patience: 10 [2020-12-28 18:53:51 main:488] : INFO : Hidden Width: 12 [2020-12-28 18:53:51 main:489] : INFO : # Recurrent Layers: 4 [2020-12-28 18:53:51 main:490] : INFO : # Backend Layers: 4 [2020-12-28 18:53:51 main:491] : INFO : # Threads: 1 [2020-12-28 18:53:51 main:493] : INFO : Preparing Dataset [2020-12-28 18:53:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2020-12-28 18:53:52 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2020-12-28 18:53:53 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory. [2020-12-28 18:53:53 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2020-12-28 18:53:53 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityModified-train-val-dataset.hdf5 into memory [2020-12-28 18:53:53 load:106] : INFO : Successfully loaded dataset of 512 examples into memory. [2020-12-28 18:53:53 main:501] : INFO : Creating Model [2020-12-28 18:53:53 main:514] : INFO : Preparing config file [2020-12-28 18:53:53 main:526] : INFO : Creating new config file [2020-12-28 18:53:53 main:545] : INFO : This is a continuation WU, loading previous network [2020-12-28 18:53:53 main:566] : INFO : Loading DataLoader into Memory [2020-12-28 18:53:53 main:569] : INFO : Starting Training [2020-12-28 18:53:55 main:581] : INFO : Epoch 1 | loss: nan | val_loss: nan | Time: 1950.42 ms [2020-12-28 18:53:57 main:581] : INFO : Epoch 2 | loss: nan | val_loss: nan | Time: 1764.63 ms [2020-12-28 18:53:59 main:581] : INFO : Epoch 3 | loss: nan | val_loss: nan | Time: 1748.81 ms [2020-12-28 18:54:00 main:581] : INFO : Epoch 4 | loss: nan | val_loss: nan | Time: 1667.49 ms [2020-12-28 18:54:02 main:581] : INFO : Epoch 5 | loss: nan | val_loss: nan | Time: 1696.09 ms [2020-12-28 18:54:04 main:581] : INFO : Epoch 6 | loss: nan | val_loss: nan | Time: 1684.33 ms [2020-12-28 18:54:05 main:581] : INFO : Epoch 7 | loss: nan | val_loss: nan | Time: 1709.56 ms [2020-12-28 18:54:07 main:581] : INFO : Epoch 8 | loss: nan | val_loss: nan | Time: 1758.42 ms [2020-12-28 18:54:09 main:581] : INFO : Epoch 9 | loss: nan | val_loss: nan | Time: 1726.65 ms [2020-12-28 18:54:11 main:581] : INFO : Epoch 10 | loss: nan | val_loss: nan | Time: 1726.97 ms [2020-12-28 18:54:12 main:581] : INFO : Epoch 11 | loss: nan | val_loss: nan | Time: 1724.74 ms [2020-12-28 18:54:14 main:581] : INFO : Epoch 12 | loss: nan | val_loss: nan | Time: 1724.31 ms [2020-12-28 18:54:16 main:581] : INFO : Epoch 13 | loss: nan | val_loss: nan | Time: 1684.84 ms [2020-12-28 18:54:17 main:581] : INFO : Epoch 14 | loss: nan | val_loss: nan | Time: 1664.06 ms [2020-12-28 18:54:19 main:581] : INFO : Epoch 15 | loss: nan | val_loss: nan | Time: 1699.28 ms [2020-12-28 18:54:21 main:581] : INFO : Epoch 16 | loss: nan | val_loss: nan | Time: 1660.24 ms [2020-12-28 18:54:22 main:581] : INFO : Epoch 17 | loss: nan | val_loss: nan | Time: 1669.19 ms [2020-12-28 18:54:24 main:581] : INFO : Epoch 18 | loss: nan | val_loss: nan | Time: 1696.42 ms [2020-12-28 18:54:26 main:581] : INFO : Epoch 19 | loss: nan | val_loss: nan | Time: 1674.02 ms [2020-12-28 18:54:27 main:581] : INFO : Epoch 20 | loss: nan | val_loss: nan | Time: 1709.69 ms [2020-12-28 18:54:29 main:581] : INFO : Epoch 21 | loss: nan | val_loss: nan | Time: 1710.43 ms [2020-12-28 18:54:31 main:581] : INFO : Epoch 22 | loss: nan | val_loss: nan | Time: 1665.55 ms [2020-12-28 18:54:33 main:581] : INFO : Epoch 23 | loss: nan | val_loss: nan | Time: 1734.37 ms [2020-12-28 18:54:34 main:581] : INFO : Epoch 24 | loss: nan | val_loss: nan | Time: 1758.14 ms [2020-12-28 18:54:36 main:581] : INFO : Epoch 25 | loss: nan | val_loss: nan | Time: 1738.31 ms [2020-12-28 18:54:38 main:581] : INFO : Epoch 26 | loss: nan | val_loss: nan | Time: 1747.1 ms [2020-12-28 18:54:40 main:581] : INFO : Epoch 27 | loss: nan | val_loss: nan | Time: 1754.98 ms [2020-12-28 18:54:41 main:581] : INFO : Epoch 28 | loss: nan | val_loss: nan | Time: 1668.09 ms [2020-12-28 18:54:43 main:581] : INFO : Epoch 29 | loss: nan | val_loss: nan | Time: 1675.58 ms [2020-12-28 18:54:45 main:581] : INFO : Epoch 30 | loss: nan | val_loss: nan | Time: 1669.79 ms [2020-12-28 18:54:46 main:581] : INFO : Epoch 31 | loss: nan | val_loss: nan | Time: 1665.48 ms [2020-12-28 18:54:48 main:581] : INFO : Epoch 32 | loss: nan | val_loss: nan | Time: 1681.24 ms [2020-12-28 18:54:50 main:581] : INFO : Epoch 33 | loss: nan | val_loss: nan | Time: 1689.91 ms [2020-12-28 18:54:51 main:581] : INFO : Epoch 34 | loss: nan | val_loss: nan | Time: 1693.41 ms [2020-12-28 18:54:53 main:581] : INFO : Epoch 35 | loss: nan | val_loss: nan | Time: 1674.56 ms [2020-12-28 18:54:55 main:581] : INFO : Epoch 36 | loss: nan | val_loss: nan | Time: 1672.45 ms [2020-12-28 18:54:56 main:581] : INFO : Epoch 37 | loss: nan | val_loss: nan | Time: 1700.22 ms [2020-12-28 18:54:58 main:581] : INFO : Epoch 38 | loss: nan | val_loss: nan | Time: 1718.81 ms [2020-12-28 18:55:00 main:581] : INFO : Epoch 39 | loss: nan | val_loss: nan | Time: 1730.81 ms [2020-12-28 18:55:02 main:581] : INFO : Epoch 40 | loss: nan | val_loss: nan | Time: 1774.6 ms [2020-12-28 18:55:03 main:581] : INFO : Epoch 41 | loss: nan | val_loss: nan | Time: 1780.83 ms [2020-12-28 18:55:05 main:581] : INFO : Epoch 42 | loss: nan | val_loss: nan | Time: 1742.77 ms [2020-12-28 18:55:07 main:581] : INFO : Epoch 43 | loss: nan | val_loss: nan | Time: 1668.15 ms [2020-12-28 18:55:08 main:581] : INFO : Epoch 44 | loss: nan | val_loss: nan | Time: 1673.51 ms [2020-12-28 18:55:10 main:581] : INFO : Epoch 45 | loss: nan | val_loss: nan | Time: 1659.46 ms [2020-12-28 18:55:12 main:581] : INFO : Epoch 46 | loss: nan | val_loss: nan | Time: 1662.82 ms [2020-12-28 18:55:13 main:581] : INFO : Epoch 47 | loss: nan | val_loss: nan | Time: 1703.82 ms [2020-12-28 18:55:15 main:581] : INFO : Epoch 48 | loss: nan | val_loss: nan | Time: 1752.09 ms [2020-12-28 18:55:17 main:581] : INFO : Epoch 49 | loss: nan | val_loss: nan | Time: 1711.95 ms [2020-12-28 18:55:19 main:581] : INFO : Epoch 50 | loss: nan | val_loss: nan | Time: 1790.77 ms [2020-12-28 18:55:20 main:581] : INFO : Epoch 51 | loss: nan | val_loss: nan | Time: 1668.68 ms [2020-12-28 18:55:22 main:581] : INFO : Epoch 52 | loss: nan | val_loss: nan | Time: 1670.06 ms [2020-12-28 18:55:24 main:581] : INFO : Epoch 53 | loss: nan | val_loss: nan | Time: 1664.97 ms [2020-12-28 18:55:25 main:581] : INFO : Epoch 54 | loss: nan | val_loss: nan | Time: 1663.48 ms [2020-12-28 18:55:27 main:581] : INFO : Epoch 55 | loss: nan | val_loss: nan | Time: 1667.57 ms [2020-12-28 18:55:29 main:581] : INFO : Epoch 56 | loss: nan | val_loss: nan | Time: 1750.93 ms [2020-12-28 18:55:31 main:581] : INFO : Epoch 57 | loss: nan | val_loss: nan | Time: 1703.17 ms [2020-12-28 18:55:32 main:581] : INFO : Epoch 58 | loss: nan | val_loss: nan | Time: 1664.52 ms [2020-12-28 18:55:34 main:581] : INFO : Epoch 59 | loss: nan | val_loss: nan | Time: 1724.48 ms [2020-12-28 18:55:36 main:581] : INFO : Epoch 60 | loss: nan | val_loss: nan | Time: 1730.74 ms [2020-12-28 18:55:37 main:581] : INFO : Epoch 61 | loss: nan | val_loss: nan | Time: 1705.74 ms [2020-12-28 18:55:39 main:581] : INFO : Epoch 62 | loss: nan | val_loss: nan | Time: 1667.76 ms [2020-12-28 18:55:41 main:581] : INFO : Epoch 63 | loss: nan | val_loss: nan | Time: 1705.41 ms [2020-12-28 18:55:42 main:581] : INFO : Epoch 64 | loss: nan | val_loss: nan | Time: 1724.77 ms [2020-12-28 18:55:44 main:581] : INFO : Epoch 65 | loss: nan | val_loss: nan | Time: 1684.92 ms [2020-12-28 18:55:46 main:581] : INFO : Epoch 66 | loss: nan | val_loss: nan | Time: 1727.82 ms [2020-12-28 18:55:48 main:581] : INFO : Epoch 67 | loss: nan | val_loss: nan | Time: 1741.57 ms [2020-12-28 18:55:49 main:581] : INFO : Epoch 68 | loss: nan | val_loss: nan | Time: 1786.82 ms [2020-12-28 18:55:51 main:581] : INFO : Epoch 69 | loss: nan | val_loss: nan | Time: 1778.12 ms [2020-12-28 18:55:53 main:581] : INFO : Epoch 70 | loss: nan | val_loss: nan | Time: 1715.22 ms [2020-12-28 18:55:55 main:581] : INFO : Epoch 71 | loss: nan | val_loss: nan | Time: 1671.78 ms [2020-12-28 18:55:56 main:581] : INFO : Epoch 72 | loss: nan | val_loss: nan | Time: 1747.56 ms [2020-12-28 18:55:58 main:581] : INFO : Epoch 73 | loss: nan | val_loss: nan | Time: 1784.02 ms [2020-12-28 18:56:00 main:581] : INFO : Epoch 74 | loss: nan | val_loss: nan | Time: 1732.25 ms [2020-12-28 18:56:02 main:581] : INFO : Epoch 75 | loss: nan | val_loss: nan | Time: 1663.83 ms [2020-12-28 18:56:03 main:581] : INFO : Epoch 76 | loss: nan | val_loss: nan | Time: 1672.86 ms [2020-12-28 18:56:05 main:581] : INFO : Epoch 77 | loss: nan | val_loss: nan | Time: 1666.78 ms [2020-12-28 18:56:07 main:581] : INFO : Epoch 78 | loss: nan | val_loss: nan | Time: 1674.82 ms [2020-12-28 18:56:08 main:581] : INFO : Epoch 79 | loss: nan | val_loss: nan | Time: 1665.44 ms [2020-12-28 18:56:10 main:581] : INFO : Epoch 80 | loss: nan | val_loss: nan | Time: 1681.5 ms [2020-12-28 18:56:12 main:581] : INFO : Epoch 81 | loss: nan | val_loss: nan | Time: 1680.16 ms [2020-12-28 18:56:13 main:581] : INFO : Epoch 82 | loss: nan | val_loss: nan | Time: 1670.17 ms [2020-12-28 18:56:15 main:581] : INFO : Epoch 83 | loss: nan | val_loss: nan | Time: 1679.63 ms [2020-12-28 18:56:17 main:581] : INFO : Epoch 84 | loss: nan | val_loss: nan | Time: 1736.8 ms [2020-12-28 18:56:18 main:581] : INFO : Epoch 85 | loss: nan | val_loss: nan | Time: 1697.24 ms [2020-12-28 18:56:20 main:581] : INFO : Epoch 86 | loss: nan | val_loss: nan | Time: 1667.98 ms [2020-12-28 18:56:22 main:581] : INFO : Epoch 87 | loss: nan | val_loss: nan | Time: 1665.77 ms [2020-12-28 18:56:23 main:581] : INFO : Epoch 88 | loss: nan | val_loss: nan | Time: 1674.94 ms [2020-12-28 18:56:25 main:581] : INFO : Epoch 89 | loss: nan | val_loss: nan | Time: 1669.88 ms [2020-12-28 18:56:27 main:581] : INFO : Epoch 90 | loss: nan | val_loss: nan | Time: 1683.97 ms [2020-12-28 18:56:28 main:581] : INFO : Epoch 91 | loss: nan | val_loss: nan | Time: 1675.36 ms [2020-12-28 18:56:30 main:581] : INFO : Epoch 92 | loss: nan | val_loss: nan | Time: 1679.82 ms [2020-12-28 18:56:32 main:581] : INFO : Epoch 93 | loss: nan | val_loss: nan | Time: 1668.19 ms [2020-12-28 18:56:33 main:581] : INFO : Epoch 94 | loss: nan | val_loss: nan | Time: 1689.71 ms [2020-12-28 18:56:35 main:581] : INFO : Epoch 95 | loss: nan | val_loss: nan | Time: 1688.45 ms [2020-12-28 18:56:37 main:581] : INFO : Epoch 96 | loss: nan | val_loss: nan | Time: 1670.15 ms [2020-12-28 18:56:38 main:581] : INFO : Epoch 97 | loss: nan | val_loss: nan | Time: 1670.17 ms [2020-12-28 18:56:40 main:581] : INFO : Epoch 98 | loss: nan | val_loss: nan | Time: 1690.83 ms [2020-12-28 18:56:42 main:581] : INFO : Epoch 99 | loss: nan | val_loss: nan | Time: 1668.71 ms [2020-12-28 18:56:43 main:581] : INFO : Epoch 100 | loss: nan | val_loss: nan | Time: 1669.59 ms [2020-12-28 18:56:45 main:581] : INFO : Epoch 101 | loss: nan | val_loss: nan | Time: 1664.31 ms [2020-12-28 18:56:47 main:581] : INFO : Epoch 102 | loss: nan | val_loss: nan | Time: 1679.78 ms [2020-12-28 18:56:48 main:581] : INFO : Epoch 103 | loss: nan | val_loss: nan | Time: 1667.78 ms [2020-12-28 18:56:50 main:581] : INFO : Epoch 104 | loss: nan | val_loss: nan | Time: 1667.84 ms [2020-12-28 18:56:52 main:581] : INFO : Epoch 105 | loss: nan | val_loss: nan | Time: 1667.55 ms [2020-12-28 18:56:53 main:581] : INFO : Epoch 106 | loss: nan | val_loss: nan | Time: 1676.5 ms [2020-12-28 18:56:55 main:581] : INFO : Epoch 107 | loss: nan | val_loss: nan | Time: 1681.31 ms [2020-12-28 18:56:57 main:581] : INFO : Epoch 108 | loss: nan | val_loss: nan | Time: 1700.49 ms [2020-12-28 18:56:59 main:581] : INFO : Epoch 109 | loss: nan | val_loss: nan | Time: 1700.1 ms [2020-12-28 18:57:00 main:581] : INFO : Epoch 110 | loss: nan | val_loss: nan | Time: 1681.37 ms [2020-12-28 18:57:02 main:581] : INFO : Epoch 111 | loss: nan | val_loss: nan | Time: 1697.83 ms [2020-12-28 18:57:04 main:581] : INFO : Epoch 112 | loss: nan | val_loss: nan | Time: 1709.34 ms [2020-12-28 18:57:05 main:581] : INFO : Epoch 113 | loss: nan | val_loss: nan | Time: 1753.09 ms [2020-12-28 18:57:07 main:581] : INFO : Epoch 114 | loss: nan | val_loss: nan | Time: 1665.01 ms [2020-12-28 18:57:09 main:581] : INFO : Epoch 115 | loss: nan | val_loss: nan | Time: 1662.64 ms [2020-12-28 18:57:10 main:581] : INFO : Epoch 116 | loss: nan | val_loss: nan | Time: 1674.03 ms [2020-12-28 18:57:12 main:581] : INFO : Epoch 117 | loss: nan | val_loss: nan | Time: 1717.29 ms [2020-12-28 18:57:14 main:581] : INFO : Epoch 118 | loss: nan | val_loss: nan | Time: 1700.14 ms [2020-12-28 18:57:16 main:581] : INFO : Epoch 119 | loss: nan | val_loss: nan | Time: 1680.16 ms [2020-12-28 18:57:17 main:581] : INFO : Epoch 120 | loss: nan | val_loss: nan | Time: 1667.77 ms [2020-12-28 18:57:19 main:581] : INFO : Epoch 121 | loss: nan | val_loss: nan | Time: 1677.48 ms [2020-12-28 18:57:21 main:581] : INFO : Epoch 122 | loss: nan | val_loss: nan | Time: 1676.11 ms [2020-12-28 18:57:22 main:581] : INFO : Epoch 123 | loss: nan | val_loss: nan | Time: 1661.76 ms [2020-12-28 18:57:24 main:581] : INFO : Epoch 124 | loss: nan | val_loss: nan | Time: 1665.26 ms [2020-12-28 18:57:26 main:581] : INFO : Epoch 125 | loss: nan | val_loss: nan | Time: 1686.81 ms [2020-12-28 18:57:27 main:581] : INFO : Epoch 126 | loss: nan | val_loss: nan | Time: 1684.93 ms [2020-12-28 18:57:29 main:581] : INFO : Epoch 127 | loss: nan | val_loss: nan | Time: 1668.5 ms [2020-12-28 18:57:31 main:581] : INFO : Epoch 128 | loss: nan | val_loss: nan | Time: 1666.9 ms [2020-12-28 18:57:32 main:581] : INFO : Epoch 129 | loss: nan | val_loss: nan | Time: 1661.88 ms [2020-12-28 18:57:34 main:581] : INFO : Epoch 130 | loss: nan | val_loss: nan | Time: 1666.35 ms [2020-12-28 18:57:36 main:581] : INFO : Epoch 131 | loss: nan | val_loss: nan | Time: 1669.45 ms [2020-12-28 18:57:37 main:581] : INFO : Epoch 132 | loss: nan | val_loss: nan | Time: 1676.58 ms [2020-12-28 18:57:39 main:581] : INFO : Epoch 133 | loss: nan | val_loss: nan | Time: 1669.68 ms [2020-12-28 18:57:41 main:581] : INFO : Epoch 134 | loss: nan | val_loss: nan | Time: 1670.2 ms [2020-12-28 18:57:42 main:581] : INFO : Epoch 135 | loss: nan | val_loss: nan | Time: 1678.54 ms [2020-12-28 18:57:44 main:581] : INFO : Epoch 136 | loss: nan | val_loss: nan | Time: 1667 ms [2020-12-28 18:57:46 main:581] : INFO : Epoch 137 | loss: nan | val_loss: nan | Time: 1669.38 ms [2020-12-28 18:57:47 main:581] : INFO : Epoch 138 | loss: nan | val_loss: nan | Time: 1664.35 ms [2020-12-28 18:57:49 main:581] : INFO : Epoch 139 | loss: nan | val_loss: nan | Time: 1666.2 ms [2020-12-28 18:57:51 main:581] : INFO : Epoch 140 | loss: nan | val_loss: nan | Time: 1665.61 ms [2020-12-28 18:57:52 main:581] : INFO : Epoch 141 | loss: nan | val_loss: nan | Time: 1662.71 ms [2020-12-28 18:57:54 main:581] : INFO : Epoch 142 | loss: nan | val_loss: nan | Time: 1670.54 ms [2020-12-28 18:57:56 main:581] : INFO : Epoch 143 | loss: nan | val_loss: nan | Time: 1669.66 ms [2020-12-28 18:57:57 main:581] : INFO : Epoch 144 | loss: nan | val_loss: nan | Time: 1662.46 ms [2020-12-28 18:57:59 main:581] : INFO : Epoch 145 | loss: nan | val_loss: nan | Time: 1667.37 ms [2020-12-28 18:58:01 main:581] : INFO : Epoch 146 | loss: nan | val_loss: nan | Time: 1668.9 ms [2020-12-28 18:58:02 main:581] : INFO : Epoch 147 | loss: nan | val_loss: nan | Time: 1663.43 ms [2020-12-28 18:58:04 main:581] : INFO : Epoch 148 | loss: nan | val_loss: nan | Time: 1664.36 ms [2020-12-28 18:58:06 main:581] : INFO : Epoch 149 | loss: nan | val_loss: nan | Time: 1666.07 ms [2020-12-28 18:58:07 main:581] : INFO : Epoch 150 | loss: nan | val_loss: nan | Time: 1660.59 ms [2020-12-28 18:58:09 main:581] : INFO : Epoch 151 | loss: nan | val_loss: nan | Time: 1668.92 ms [2020-12-28 18:58:11 main:581] : INFO : Epoch 152 | loss: nan | val_loss: nan | Time: 1692.23 ms [2020-12-28 18:58:12 main:581] : INFO : Epoch 153 | loss: nan | val_loss: nan | Time: 1664.87 ms [2020-12-28 18:58:14 main:581] : INFO : Epoch 154 | loss: nan | val_loss: nan | Time: 1667.86 ms [2020-12-28 18:58:16 main:581] : INFO : Epoch 155 | loss: nan | val_loss: nan | Time: 1679.04 ms [2020-12-28 18:58:17 main:581] : INFO : Epoch 156 | loss: nan | val_loss: nan | Time: 1673.77 ms [2020-12-28 18:58:19 main:581] : INFO : Epoch 157 | loss: nan | val_loss: nan | Time: 1672.3 ms [2020-12-28 18:58:21 main:581] : INFO : Epoch 158 | loss: nan | val_loss: nan | Time: 1701.11 ms [2020-12-28 18:58:22 main:581] : INFO : Epoch 159 | loss: nan | val_loss: nan | Time: 1663.84 ms [2020-12-28 18:58:24 main:581] : INFO : Epoch 160 | loss: nan | val_loss: nan | Time: 1691.23 ms [2020-12-28 18:58:26 main:581] : INFO : Epoch 161 | loss: nan | val_loss: nan | Time: 1673.97 ms [2020-12-28 18:58:27 main:581] : INFO : Epoch 162 | loss: nan | val_loss: nan | Time: 1682.87 ms [2020-12-28 18:58:29 main:581] : INFO : Epoch 163 | loss: nan | val_loss: nan | Time: 1705.62 ms [2020-12-28 18:58:31 main:581] : INFO : Epoch 164 | loss: nan | val_loss: nan | Time: 1696.57 ms [2020-12-28 18:58:32 main:581] : INFO : Epoch 165 | loss: nan | val_loss: nan | Time: 1676.3 ms [2020-12-28 18:58:34 main:581] : INFO : Epoch 166 | loss: nan | val_loss: nan | Time: 1664.97 ms [2020-12-28 18:58:36 main:581] : INFO : Epoch 167 | loss: nan | val_loss: nan | Time: 1669.53 ms [2020-12-28 18:58:38 main:581] : INFO : Epoch 168 | loss: nan | val_loss: nan | Time: 1683.69 ms [2020-12-28 18:58:39 main:581] : INFO : Epoch 169 | loss: nan | val_loss: nan | Time: 1674.07 ms [2020-12-28 18:58:41 main:581] : INFO : Epoch 170 | loss: nan | val_loss: nan | Time: 1694.93 ms [2020-12-28 18:58:43 main:581] : INFO : Epoch 171 | loss: nan | val_loss: nan | Time: 1677.08 ms [2020-12-28 18:58:44 main:581] : INFO : Epoch 172 | loss: nan | val_loss: nan | Time: 1663.26 ms [2020-12-28 18:58:46 main:581] : INFO : Epoch 173 | loss: nan | val_loss: nan | Time: 1692.25 ms [2020-12-28 18:58:48 main:581] : INFO : Epoch 174 | loss: nan | val_loss: nan | Time: 1697.22 ms [2020-12-28 18:58:49 main:581] : INFO : Epoch 175 | loss: nan | val_loss: nan | Time: 1675.37 ms [2020-12-28 18:58:51 main:581] : INFO : Epoch 176 | loss: nan | val_loss: nan | Time: 1670.14 ms [2020-12-28 18:58:53 main:581] : INFO : Epoch 177 | loss: nan | val_loss: nan | Time: 1662.84 ms [2020-12-28 18:58:54 main:581] : INFO : Epoch 178 | loss: nan | val_loss: nan | Time: 1696.18 ms [2020-12-28 18:58:56 main:581] : INFO : Epoch 179 | loss: nan | val_loss: nan | Time: 1675.9 ms [2020-12-28 18:58:58 main:581] : INFO : Epoch 180 | loss: nan | val_loss: nan | Time: 1726.42 ms [2020-12-28 18:58:59 main:581] : INFO : Epoch 181 | loss: nan | val_loss: nan | Time: 1741.82 ms [2020-12-28 18:59:01 main:581] : INFO : Epoch 182 | loss: nan | val_loss: nan | Time: 1773.08 ms [2020-12-28 18:59:03 main:581] : INFO : Epoch 183 | loss: nan | val_loss: nan | Time: 1716.12 ms [2020-12-28 18:59:05 main:581] : INFO : Epoch 184 | loss: nan | val_loss: nan | Time: 1775.82 ms [2020-12-28 18:59:06 main:581] : INFO : Epoch 185 | loss: nan | val_loss: nan | Time: 1729.76 ms [2020-12-28 18:59:08 main:581] : INFO : Epoch 186 | loss: nan | val_loss: nan | Time: 1724.82 ms [2020-12-28 18:59:10 main:581] : INFO : Epoch 187 | loss: nan | val_loss: nan | Time: 1737.49 ms [2020-12-28 18:59:12 main:581] : INFO : Epoch 188 | loss: nan | val_loss: nan | Time: 1764.03 ms [2020-12-28 18:59:13 main:581] : INFO : Epoch 189 | loss: nan | val_loss: nan | Time: 1781.38 ms [2020-12-28 18:59:15 main:581] : INFO : Epoch 190 | loss: nan | val_loss: nan | Time: 1777.83 ms [2020-12-28 18:59:17 main:581] : INFO : Epoch 191 | loss: nan | val_loss: nan | Time: 1692.88 ms [2020-12-28 18:59:19 main:581] : INFO : Epoch 192 | loss: nan | val_loss: nan | Time: 1737.42 ms [2020-12-28 18:59:20 main:581] : INFO : Epoch 193 | loss: nan | val_loss: nan | Time: 1710.94 ms [2020-12-28 18:59:22 main:581] : INFO : Epoch 194 | loss: nan | val_loss: nan | Time: 1671.91 ms [2020-12-28 18:59:24 main:581] : INFO : Epoch 195 | loss: nan | val_loss: nan | Time: 1691.17 ms [2020-12-28 18:59:25 main:581] : INFO : Epoch 196 | loss: nan | val_loss: nan | Time: 1663.72 ms [2020-12-28 18:59:27 main:581] : INFO : Epoch 197 | loss: nan | val_loss: nan | Time: 1693.37 ms [2020-12-28 18:59:29 main:581] : INFO : Epoch 198 | loss: nan | val_loss: nan | Time: 1671.55 ms [2020-12-28 18:59:30 main:581] : INFO : Epoch 199 | loss: nan | val_loss: nan | Time: 1672.77 ms [2020-12-28 18:59:32 main:581] : INFO : Epoch 200 | loss: nan | val_loss: nan | Time: 1681.44 ms [2020-12-28 18:59:34 main:581] : INFO : Epoch 201 | loss: nan | val_loss: nan | Time: 1663.34 ms [2020-12-28 18:59:35 main:581] : INFO : Epoch 202 | loss: nan | val_loss: nan | Time: 1700.95 ms [2020-12-28 18:59:37 main:581] : INFO : Epoch 203 | loss: nan | val_loss: nan | Time: 1723.59 ms [2020-12-28 18:59:39 main:581] : INFO : Epoch 204 | loss: nan | val_loss: nan | Time: 1661.6 ms [2020-12-28 18:59:41 main:581] : INFO : Epoch 205 | loss: nan | val_loss: nan | Time: 1723.09 ms [2020-12-28 18:59:42 main:581] : INFO : Epoch 206 | loss: nan | val_loss: nan | Time: 1730.56 ms [2020-12-28 18:59:44 main:581] : INFO : Epoch 207 | loss: nan | val_loss: nan | Time: 1746.88 ms [2020-12-28 18:59:46 main:581] : INFO : Epoch 208 | loss: nan | val_loss: nan | Time: 1674.91 ms [2020-12-28 18:59:47 main:581] : INFO : Epoch 209 | loss: nan | val_loss: nan | Time: 1675.94 ms [2020-12-28 18:59:49 main:581] : INFO : Epoch 210 | loss: nan | val_loss: nan | Time: 1680.21 ms [2020-12-28 18:59:51 main:581] : INFO : Epoch 211 | loss: nan | val_loss: nan | Time: 1673.32 ms [2020-12-28 18:59:52 main:581] : INFO : Epoch 212 | loss: nan | val_loss: nan | Time: 1677.92 ms [2020-12-28 18:59:54 main:581] : INFO : Epoch 213 | loss: nan | val_loss: nan | Time: 1659.53 ms [2020-12-28 18:59:56 main:581] : INFO : Epoch 214 | loss: nan | val_loss: nan | Time: 1662.57 ms [2020-12-28 18:59:57 main:581] : INFO : Epoch 215 | loss: nan | val_loss: nan | Time: 1708.77 ms [2020-12-28 18:59:59 main:581] : INFO : Epoch 216 | loss: nan | val_loss: nan | Time: 1739.29 ms [2020-12-28 19:00:01 main:581] : INFO : Epoch 217 | loss: nan | val_loss: nan | Time: 1758.25 ms [2020-12-28 19:00:03 main:581] : INFO : Epoch 218 | loss: nan | val_loss: nan | Time: 1717.93 ms [2020-12-28 19:00:04 main:581] : INFO : Epoch 219 | loss: nan | val_loss: nan | Time: 1715.17 ms [2020-12-28 19:00:06 main:581] : INFO : Epoch 220 | loss: nan | val_loss: nan | Time: 1703.17 ms [2020-12-28 19:00:08 main:581] : INFO : Epoch 221 | loss: nan | val_loss: nan | Time: 1746.94 ms [2020-12-28 19:00:10 main:581] : INFO : Epoch 222 | loss: nan | val_loss: nan | Time: 1775.87 ms [2020-12-28 19:00:11 main:581] : INFO : Epoch 223 | loss: nan | val_loss: nan | Time: 1744.51 ms SIGSEGV: segmentation violation Stack trace (39 frames): ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x55e78332ff9c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f277f9773c0] /lib/x86_64-linux-gnu/libc.so.6(+0x499f6)[0x7f27012a09f6] /lib/x86_64-linux-gnu/libc.so.6(on_exit+0x0)[0x7f27012a0be0] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379b25)[0x55e78332bb25] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379bee)[0x55e78332bbee] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f277f9773c0] /lib/x86_64-linux-gnu/libcuda.so.1(+0x1dcfd0)[0x7f26e3ad7fd0] /lib/x86_64-linux-gnu/libcuda.so.1(+0x1bab22)[0x7f26e3ab5b22] ./libcudnn_adv_infer.so.8(+0x3561d6)[0x7f2684fd91d6] ./libcudnn_adv_infer.so.8(+0x34818c)[0x7f2684fcb18c] ./libcudnn_adv_infer.so.8(+0x331c83)[0x7f2684fb4c83] ./libcudnn_adv_infer.so.8(+0x377a3b)[0x7f2684ffaa3b] ./libcudnn_adv_infer.so.8(+0x7794c)[0x7f2684cfa94c] ./libcudnn_adv_infer.so.8(+0xe89f)[0x7f2684c9189f] ./libcudnn_adv_infer.so.8(_Z18RNN_forwardGenericP12cudnnContextP14cudnnRNNStructiPKP17cudnnTensorStructPKvS4_S8_S4_S8_P17cudnnFilterStructS8_S6_PvS4_SB_S4_SB_SB_mSB_m18cudnnForwardMode_t+0x5d7)[0x7f2684cd58c7] ./libcudnn_adv_train.so.8(cudnnRNNForwardTraining+0x20e)[0x7f267261661e] ./libtorch_cuda.so(_ZN2at6native10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS1_EElS3_S3_S3_lllbdbbNS5_IlEES3_+0x2214)[0x7f270208fa64] ./libtorch_cuda.so(+0x85face)[0x7f27020d9ace] ./libtorch_cuda.so(+0x895988)[0x7f270210f988] ./libtorch_cpu.so(_ZN2at10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS0_EElS2_S2_S2_lllbdbbNS4_IlEES2_+0x2bc)[0x7f277a64851c] ./libtorch_cpu.so(+0x21cf2d4)[0x7f277ba122d4] ./libtorch_cpu.so(+0x21edb28)[0x7f277ba30b28] ./libtorch_cpu.so(_ZN2at10_cudnn_rnnERKNS_6TensorEN3c108ArrayRefIS0_EElS2_S2_S2_lllbdbbNS4_IlEES2_+0x2bc)[0x7f277a64851c] ./libtorch_cuda.so(+0x80f0fd)[0x7f27020890fd] ./libtorch_cuda.so(+0x80fb0a)[0x7f2702089b0a] ./libtorch_cpu.so(+0x97fe93)[0x7f277a1c2e93] ./libtorch_cpu.so(_ZN2at6native3gruERKNS_6TensorES3_N3c108ArrayRefIS1_EEbldbbb+0xea)[0x7f277a1a69ba] ./libtorch_cpu.so(+0xeb0a96)[0x7f277a6f3a96] ./libtorch_cpu.so(+0x22e3c92)[0x7f277bb26c92] ./libtorch_cpu.so(+0xedd579)[0x7f277a720579] ./libtorch_cpu.so(_ZN2at3gruERKNS_6TensorES2_N3c108ArrayRefIS0_EEbldbbb+0x1b1)[0x7f277a625ee1] ./libtorch_cpu.so(_ZN5torch2nn7GRUImpl14forward_helperERKN2at6TensorES5_S5_lS3_+0x43a)[0x7f277c58f4ca] ./libtorch_cpu.so(_ZN5torch2nn7GRUImpl7forwardERKN2at6TensorES3_+0xb0)[0x7f277c58f630] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x92fa8)[0x55e783044fa8] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x9a0e6)[0x55e78304c0e6] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8ad10)[0x55e78303cd10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f270127e0b3] ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x55e78303875a] Exiting... </stderr_txt> ]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)