Task 12599696
| Name | ParityModified-1645997577-11439-1-0_0 |
| Workunit | 9757169 |
| Created | 6 Mar 2022, 8:42:46 UTC |
| Sent | 10 Mar 2022, 2:16:26 UTC |
| Report deadline | 18 Mar 2022, 2:16:26 UTC |
| Received | 5 Apr 2022, 5:24:45 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 193 (0x000000C1) EXIT_SIGNAL |
| Computer ID | 20699 |
| Run time | |
| CPU time | |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 8,325.09 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.80 (cuda10200) x86_64-pc-linux-gnu |
| Peak working set size | 1.81 GB |
| Peak swap size | 8.92 GB |
| Peak disk usage | 2.99 GB |
Stderr output
<core_client_version>7.16.17</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)</message>
<stderr_txt>
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-09 18:24:40 main:442] : INFO : Set logging level to 1
[2022-03-09 18:24:40 main:448] : INFO : Running in BOINC Client mode
[2022-03-09 18:24:40 main:451] : INFO : Resolving all filenames
[2022-03-09 18:24:40 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-09 18:24:40 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-09 18:24:40 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-09 18:24:40 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-09 18:24:40 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-09 18:24:40 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-09 18:24:40 main:481] : INFO : Configuration:
[2022-03-09 18:24:40 main:482] : INFO : Model type: GRU
[2022-03-09 18:24:40 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-09 18:24:40 main:484] : INFO : Max Epochs: 2048
[2022-03-09 18:24:40 main:485] : INFO : Batch Size: 128
[2022-03-09 18:24:40 main:486] : INFO : Learning Rate: 0.01
[2022-03-09 18:24:40 main:487] : INFO : Patience: 10
[2022-03-09 18:24:40 main:488] : INFO : Hidden Width: 12
[2022-03-09 18:24:40 main:489] : INFO : # Recurrent Layers: 4
[2022-03-09 18:24:40 main:490] : INFO : # Backend Layers: 4
[2022-03-09 18:24:40 main:491] : INFO : # Threads: 1
[2022-03-09 18:24:40 main:493] : INFO : Preparing Dataset
[2022-03-09 18:24:40 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-09 18:24:41 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-10 22:10:04 main:442] : INFO : Set logging level to 1
[2022-03-10 22:10:04 main:448] : INFO : Running in BOINC Client mode
[2022-03-10 22:10:04 main:451] : INFO : Resolving all filenames
[2022-03-10 22:10:04 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-10 22:10:04 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-10 22:10:04 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-10 22:10:04 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-10 22:10:04 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-10 22:10:04 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-10 22:10:04 main:481] : INFO : Configuration:
[2022-03-10 22:10:04 main:482] : INFO : Model type: GRU
[2022-03-10 22:10:04 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-10 22:10:04 main:484] : INFO : Max Epochs: 2048
[2022-03-10 22:10:04 main:485] : INFO : Batch Size: 128
[2022-03-10 22:10:04 main:486] : INFO : Learning Rate: 0.01
[2022-03-10 22:10:04 main:487] : INFO : Patience: 10
[2022-03-10 22:10:04 main:488] : INFO : Hidden Width: 12
[2022-03-10 22:10:04 main:489] : INFO : # Recurrent Layers: 4
[2022-03-10 22:10:04 main:490] : INFO : # Backend Layers: 4
[2022-03-10 22:10:04 main:491] : INFO : # Threads: 1
[2022-03-10 22:10:04 main:493] : INFO : Preparing Dataset
[2022-03-10 22:10:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-10 22:10:04 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-18 20:12:06 main:442] : INFO : Set logging level to 1
[2022-03-18 20:12:06 main:448] : INFO : Running in BOINC Client mode
[2022-03-18 20:12:06 main:451] : INFO : Resolving all filenames
[2022-03-18 20:12:06 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-18 20:12:06 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-18 20:12:06 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-18 20:12:06 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-18 20:12:06 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-18 20:12:06 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-18 20:12:06 main:481] : INFO : Configuration:
[2022-03-18 20:12:06 main:482] : INFO : Model type: GRU
[2022-03-18 20:12:06 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-18 20:12:06 main:484] : INFO : Max Epochs: 2048
[2022-03-18 20:12:06 main:485] : INFO : Batch Size: 128
[2022-03-18 20:12:06 main:486] : INFO : Learning Rate: 0.01
[2022-03-18 20:12:06 main:487] : INFO : Patience: 10
[2022-03-18 20:12:06 main:488] : INFO : Hidden Width: 12
[2022-03-18 20:12:06 main:489] : INFO : # Recurrent Layers: 4
[2022-03-18 20:12:06 main:490] : INFO : # Backend Layers: 4
[2022-03-18 20:12:06 main:491] : INFO : # Threads: 1
[2022-03-18 20:12:06 main:493] : INFO : Preparing Dataset
[2022-03-18 20:12:06 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-18 20:12:06 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-19 11:44:19 main:442] : INFO : Set logging level to 1
[2022-03-19 11:44:19 main:448] : INFO : Running in BOINC Client mode
[2022-03-19 11:44:19 main:451] : INFO : Resolving all filenames
[2022-03-19 11:44:19 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-19 11:44:19 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-19 11:44:19 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-19 11:44:19 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-19 11:44:19 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-19 11:44:19 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-19 11:44:19 main:481] : INFO : Configuration:
[2022-03-19 11:44:19 main:482] : INFO : Model type: GRU
[2022-03-19 11:44:19 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-19 11:44:19 main:484] : INFO : Max Epochs: 2048
[2022-03-19 11:44:19 main:485] : INFO : Batch Size: 128
[2022-03-19 11:44:19 main:486] : INFO : Learning Rate: 0.01
[2022-03-19 11:44:19 main:487] : INFO : Patience: 10
[2022-03-19 11:44:19 main:488] : INFO : Hidden Width: 12
[2022-03-19 11:44:19 main:489] : INFO : # Recurrent Layers: 4
[2022-03-19 11:44:19 main:490] : INFO : # Backend Layers: 4
[2022-03-19 11:44:19 main:491] : INFO : # Threads: 1
[2022-03-19 11:44:19 main:493] : INFO : Preparing Dataset
[2022-03-19 11:44:19 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-19 11:44:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-19 11:56:59 main:442] : INFO : Set logging level to 1
[2022-03-19 11:56:59 main:448] : INFO : Running in BOINC Client mode
[2022-03-19 11:56:59 main:451] : INFO : Resolving all filenames
[2022-03-19 11:56:59 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-19 11:56:59 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-19 11:56:59 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-19 11:56:59 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-19 11:56:59 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-19 11:56:59 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-19 11:56:59 main:481] : INFO : Configuration:
[2022-03-19 11:56:59 main:482] : INFO : Model type: GRU
[2022-03-19 11:56:59 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-19 11:56:59 main:484] : INFO : Max Epochs: 2048
[2022-03-19 11:56:59 main:485] : INFO : Batch Size: 128
[2022-03-19 11:56:59 main:486] : INFO : Learning Rate: 0.01
[2022-03-19 11:56:59 main:487] : INFO : Patience: 10
[2022-03-19 11:56:59 main:488] : INFO : Hidden Width: 12
[2022-03-19 11:56:59 main:489] : INFO : # Recurrent Layers: 4
[2022-03-19 11:56:59 main:490] : INFO : # Backend Layers: 4
[2022-03-19 11:56:59 main:491] : INFO : # Threads: 1
[2022-03-19 11:56:59 main:493] : INFO : Preparing Dataset
[2022-03-19 11:56:59 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
HDF5: infinite loop closing library
L,D_top,S_top,T_top,F,P,P,Z,FD,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-19 12:00:30 main:442] : INFO : Set logging level to 1
[2022-03-19 12:00:30 main:448] : INFO : Running in BOINC Client mode
[2022-03-19 12:00:30 main:451] : INFO : Resolving all filenames
[2022-03-19 12:00:30 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-19 12:00:30 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-19 12:00:30 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-19 12:00:30 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-19 12:00:30 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-19 12:00:30 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-19 12:00:30 main:481] : INFO : Configuration:
[2022-03-19 12:00:30 main:482] : INFO : Model type: GRU
[2022-03-19 12:00:30 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-19 12:00:30 main:484] : INFO : Max Epochs: 2048
[2022-03-19 12:00:30 main:485] : INFO : Batch Size: 128
[2022-03-19 12:00:30 main:486] : INFO : Learning Rate: 0.01
[2022-03-19 12:00:30 main:487] : INFO : Patience: 10
[2022-03-19 12:00:30 main:488] : INFO : Hidden Width: 12
[2022-03-19 12:00:30 main:489] : INFO : # Recurrent Layers: 4
[2022-03-19 12:00:30 main:490] : INFO : # Backend Layers: 4
[2022-03-19 12:00:30 main:491] : INFO : # Threads: 1
[2022-03-19 12:00:30 main:493] : INFO : Preparing Dataset
[2022-03-19 12:00:30 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-19 12:00:30 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-19 12:06:57 main:442] : INFO : Set logging level to 1
[2022-03-19 12:06:57 main:448] : INFO : Running in BOINC Client mode
[2022-03-19 12:06:57 main:451] : INFO : Resolving all filenames
[2022-03-19 12:06:57 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-19 12:06:57 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-19 12:06:57 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-19 12:06:57 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-19 12:06:57 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-19 12:06:57 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-19 12:06:57 main:481] : INFO : Configuration:
[2022-03-19 12:06:57 main:482] : INFO : Model type: GRU
[2022-03-19 12:06:57 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-19 12:06:57 main:484] : INFO : Max Epochs: 2048
[2022-03-19 12:06:57 main:485] : INFO : Batch Size: 128
[2022-03-19 12:06:57 main:486] : INFO : Learning Rate: 0.01
[2022-03-19 12:06:57 main:487] : INFO : Patience: 10
[2022-03-19 12:06:57 main:488] : INFO : Hidden Width: 12
[2022-03-19 12:06:57 main:489] : INFO : # Recurrent Layers: 4
[2022-03-19 12:06:57 main:490] : INFO : # Backend Layers: 4
[2022-03-19 12:06:57 main:491] : INFO : # Threads: 1
[2022-03-19 12:06:57 main:493] : INFO : Preparing Dataset
[2022-03-19 12:06:57 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-19 12:06:58 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-19 15:03:43 main:442] : INFO : Set logging level to 1
[2022-03-19 15:03:43 main:448] : INFO : Running in BOINC Client mode
[2022-03-19 15:03:43 main:451] : INFO : Resolving all filenames
[2022-03-19 15:03:43 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-19 15:03:43 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-19 15:03:43 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-19 15:03:43 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-19 15:03:43 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-19 15:03:43 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-19 15:03:43 main:481] : INFO : Configuration:
[2022-03-19 15:03:43 main:482] : INFO : Model type: GRU
[2022-03-19 15:03:43 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-19 15:03:43 main:484] : INFO : Max Epochs: 2048
[2022-03-19 15:03:43 main:485] : INFO : Batch Size: 128
[2022-03-19 15:03:43 main:486] : INFO : Learning Rate: 0.01
[2022-03-19 15:03:43 main:487] : INFO : Patience: 10
[2022-03-19 15:03:43 main:488] : INFO : Hidden Width: 12
[2022-03-19 15:03:43 main:489] : INFO : # Recurrent Layers: 4
[2022-03-19 15:03:43 main:490] : INFO : # Backend Layers: 4
[2022-03-19 15:03:43 main:491] : INFO : # Threads: 1
[2022-03-19 15:03:43 main:493] : INFO : Preparing Dataset
[2022-03-19 15:03:43 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
HDF5: infinite loop closing library
L,D_top,S_top,T_top,F,P,P,Z,FD,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
double free or corruption (!prev)
SIGABRT: abort called
Stack trace (30 frames):
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x561ab117df9c]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f226798b520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0xf8)[0x7f22679df828]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f226798b476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd7)[0x7f22679717b7]
/lib/x86_64-linux-gnu/libc.so.6(+0x89606)[0x7f22679d2606]
/lib/x86_64-linux-gnu/libc.so.6(+0xa0afc)[0x7f22679e9afc]
/lib/x86_64-linux-gnu/libc.so.6(+0xa286c)[0x7f22679eb86c]
/lib/x86_64-linux-gnu/libc.so.6(free+0x55)[0x7f22679edd05]
/lib/x86_64-linux-gnu/libc.so.6(+0x4560f)[0x7f226798e60f]
/lib/x86_64-linux-gnu/libc.so.6(on_exit+0x0)[0x7f226798e660]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379b25)[0x561ab1179b25]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x379bee)[0x561ab1179bee]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f226798b520]
/lib/x86_64-linux-gnu/libc.so.6(read+0x4c)[0x7f2267a5c96c]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x1a7af4)[0x561ab0fa7af4]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x1a14ce)[0x561ab0fa14ce]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x18e541)[0x561ab0f8e541]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x190f67)[0x561ab0f90f67]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x341b58)[0x561ab1141b58]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x34398d)[0x561ab114398d]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x16986d)[0x561ab0f6986d]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x169bfd)[0x561ab0f69bfd]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x107700)[0x561ab0f07700]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xc0536)[0x561ab0ec0536]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0xc0c51)[0x561ab0ec0c51]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x89c77)[0x561ab0e89c77]
/lib/x86_64-linux-gnu/libc.so.6(+0x29fd0)[0x7f2267972fd0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x7d)[0x7f226797307d]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x561ab0e8675a]
Exiting...
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-19 15:06:42 main:442] : INFO : Set logging level to 1
[2022-03-19 15:06:42 main:448] : INFO : Running in BOINC Client mode
[2022-03-19 15:06:42 main:451] : INFO : Resolving all filenames
[2022-03-19 15:06:42 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-19 15:06:42 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-19 15:06:42 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-19 15:06:42 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-19 15:06:42 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-19 15:06:42 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-19 15:06:42 main:481] : INFO : Configuration:
[2022-03-19 15:06:42 main:482] : INFO : Model type: GRU
[2022-03-19 15:06:42 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-19 15:06:42 main:484] : INFO : Max Epochs: 2048
[2022-03-19 15:06:42 main:485] : INFO : Batch Size: 128
[2022-03-19 15:06:42 main:486] : INFO : Learning Rate: 0.01
[2022-03-19 15:06:42 main:487] : INFO : Patience: 10
[2022-03-19 15:06:42 main:488] : INFO : Hidden Width: 12
[2022-03-19 15:06:42 main:489] : INFO : # Recurrent Layers: 4
[2022-03-19 15:06:42 main:490] : INFO : # Backend Layers: 4
[2022-03-19 15:06:42 main:491] : INFO : # Threads: 1
[2022-03-19 15:06:42 main:493] : INFO : Preparing Dataset
[2022-03-19 15:06:42 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-19 15:06:43 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-19 15:25:48 main:442] : INFO : Set logging level to 1
[2022-03-19 15:25:48 main:448] : INFO : Running in BOINC Client mode
[2022-03-19 15:25:48 main:451] : INFO : Resolving all filenames
[2022-03-19 15:25:48 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-19 15:25:48 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-19 15:25:48 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-19 15:25:48 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-19 15:25:48 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-19 15:25:48 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-19 15:25:48 main:481] : INFO : Configuration:
[2022-03-19 15:25:48 main:482] : INFO : Model type: GRU
[2022-03-19 15:25:48 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-19 15:25:48 main:484] : INFO : Max Epochs: 2048
[2022-03-19 15:25:48 main:485] : INFO : Batch Size: 128
[2022-03-19 15:25:48 main:486] : INFO : Learning Rate: 0.01
[2022-03-19 15:25:48 main:487] : INFO : Patience: 10
[2022-03-19 15:25:48 main:488] : INFO : Hidden Width: 12
[2022-03-19 15:25:48 main:489] : INFO : # Recurrent Layers: 4
[2022-03-19 15:25:48 main:490] : INFO : # Backend Layers: 4
[2022-03-19 15:25:48 main:491] : INFO : # Threads: 1
[2022-03-19 15:25:48 main:493] : INFO : Preparing Dataset
[2022-03-19 15:25:48 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
HDF5: infinite loop closing library
L,D_top,S_top,T_top,F,P,P,Z,FD,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-19 15:28:47 main:442] : INFO : Set logging level to 1
[2022-03-19 15:28:47 main:448] : INFO : Running in BOINC Client mode
[2022-03-19 15:28:47 main:451] : INFO : Resolving all filenames
[2022-03-19 15:28:47 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-19 15:28:47 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-19 15:28:47 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-19 15:28:47 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-19 15:28:47 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-19 15:28:47 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-19 15:28:47 main:481] : INFO : Configuration:
[2022-03-19 15:28:47 main:482] : INFO : Model type: GRU
[2022-03-19 15:28:47 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-19 15:28:47 main:484] : INFO : Max Epochs: 2048
[2022-03-19 15:28:47 main:485] : INFO : Batch Size: 128
[2022-03-19 15:28:47 main:486] : INFO : Learning Rate: 0.01
[2022-03-19 15:28:47 main:487] : INFO : Patience: 10
[2022-03-19 15:28:47 main:488] : INFO : Hidden Width: 12
[2022-03-19 15:28:47 main:489] : INFO : # Recurrent Layers: 4
[2022-03-19 15:28:47 main:490] : INFO : # Backend Layers: 4
[2022-03-19 15:28:47 main:491] : INFO : # Threads: 1
[2022-03-19 15:28:47 main:493] : INFO : Preparing Dataset
[2022-03-19 15:28:47 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-19 15:28:48 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-29 21:38:20 main:442] : INFO : Set logging level to 1
[2022-03-29 21:38:20 main:448] : INFO : Running in BOINC Client mode
[2022-03-29 21:38:20 main:451] : INFO : Resolving all filenames
[2022-03-29 21:38:20 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-29 21:38:20 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-29 21:38:20 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-29 21:38:20 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-29 21:38:20 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-29 21:38:20 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-29 21:38:20 main:481] : INFO : Configuration:
[2022-03-29 21:38:20 main:482] : INFO : Model type: GRU
[2022-03-29 21:38:20 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-29 21:38:20 main:484] : INFO : Max Epochs: 2048
[2022-03-29 21:38:20 main:485] : INFO : Batch Size: 128
[2022-03-29 21:38:20 main:486] : INFO : Learning Rate: 0.01
[2022-03-29 21:38:20 main:487] : INFO : Patience: 10
[2022-03-29 21:38:20 main:488] : INFO : Hidden Width: 12
[2022-03-29 21:38:20 main:489] : INFO : # Recurrent Layers: 4
[2022-03-29 21:38:20 main:490] : INFO : # Backend Layers: 4
[2022-03-29 21:38:20 main:491] : INFO : # Threads: 1
[2022-03-29 21:38:20 main:493] : INFO : Preparing Dataset
[2022-03-29 21:38:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-29 21:38:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-29 21:43:55 main:442] : INFO : Set logging level to 1
[2022-03-29 21:43:55 main:448] : INFO : Running in BOINC Client mode
[2022-03-29 21:43:55 main:451] : INFO : Resolving all filenames
[2022-03-29 21:43:55 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-29 21:43:55 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-29 21:43:55 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-29 21:43:55 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-29 21:43:55 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-29 21:43:55 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-29 21:43:55 main:481] : INFO : Configuration:
[2022-03-29 21:43:55 main:482] : INFO : Model type: GRU
[2022-03-29 21:43:55 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-29 21:43:55 main:484] : INFO : Max Epochs: 2048
[2022-03-29 21:43:55 main:485] : INFO : Batch Size: 128
[2022-03-29 21:43:55 main:486] : INFO : Learning Rate: 0.01
[2022-03-29 21:43:55 main:487] : INFO : Patience: 10
[2022-03-29 21:43:55 main:488] : INFO : Hidden Width: 12
[2022-03-29 21:43:55 main:489] : INFO : # Recurrent Layers: 4
[2022-03-29 21:43:55 main:490] : INFO : # Backend Layers: 4
[2022-03-29 21:43:55 main:491] : INFO : # Threads: 1
[2022-03-29 21:43:55 main:493] : INFO : Preparing Dataset
[2022-03-29 21:43:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-29 21:43:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-30 20:46:05 main:442] : INFO : Set logging level to 1
[2022-03-30 20:46:05 main:448] : INFO : Running in BOINC Client mode
[2022-03-30 20:46:05 main:451] : INFO : Resolving all filenames
[2022-03-30 20:46:05 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-30 20:46:05 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-30 20:46:05 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-30 20:46:05 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-30 20:46:05 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-30 20:46:05 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-30 20:46:05 main:481] : INFO : Configuration:
[2022-03-30 20:46:05 main:482] : INFO : Model type: GRU
[2022-03-30 20:46:05 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-30 20:46:05 main:484] : INFO : Max Epochs: 2048
[2022-03-30 20:46:05 main:485] : INFO : Batch Size: 128
[2022-03-30 20:46:05 main:486] : INFO : Learning Rate: 0.01
[2022-03-30 20:46:05 main:487] : INFO : Patience: 10
[2022-03-30 20:46:05 main:488] : INFO : Hidden Width: 12
[2022-03-30 20:46:05 main:489] : INFO : # Recurrent Layers: 4
[2022-03-30 20:46:05 main:490] : INFO : # Backend Layers: 4
[2022-03-30 20:46:05 main:491] : INFO : # Threads: 1
[2022-03-30 20:46:05 main:493] : INFO : Preparing Dataset
[2022-03-30 20:46:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-30 20:46:05 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-03-31 18:33:20 main:442] : INFO : Set logging level to 1
[2022-03-31 18:33:20 main:448] : INFO : Running in BOINC Client mode
[2022-03-31 18:33:20 main:451] : INFO : Resolving all filenames
[2022-03-31 18:33:20 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-03-31 18:33:20 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-03-31 18:33:20 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-03-31 18:33:20 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-03-31 18:33:20 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-03-31 18:33:20 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-03-31 18:33:20 main:481] : INFO : Configuration:
[2022-03-31 18:33:20 main:482] : INFO : Model type: GRU
[2022-03-31 18:33:20 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-03-31 18:33:20 main:484] : INFO : Max Epochs: 2048
[2022-03-31 18:33:20 main:485] : INFO : Batch Size: 128
[2022-03-31 18:33:20 main:486] : INFO : Learning Rate: 0.01
[2022-03-31 18:33:20 main:487] : INFO : Patience: 10
[2022-03-31 18:33:20 main:488] : INFO : Hidden Width: 12
[2022-03-31 18:33:20 main:489] : INFO : # Recurrent Layers: 4
[2022-03-31 18:33:20 main:490] : INFO : # Backend Layers: 4
[2022-03-31 18:33:20 main:491] : INFO : # Threads: 1
[2022-03-31 18:33:20 main:493] : INFO : Preparing Dataset
[2022-03-31 18:33:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-03-31 18:33:20 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
Machine Learning Dataset Generator v9.80 (Linux/x86_64) (libTorch: release/1.7 GPU: NVIDIA GeForce RTX 3060 Ti)
[2022-04-04 16:26:28 main:442] : INFO : Set logging level to 1
[2022-04-04 16:26:28 main:448] : INFO : Running in BOINC Client mode
[2022-04-04 16:26:28 main:451] : INFO : Resolving all filenames
[2022-04-04 16:26:28 main:459] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-04 16:26:28 main:459] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-04-04 16:26:28 main:459] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-04 16:26:28 main:459] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-04 16:26:28 main:459] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-04-04 16:26:28 main:479] : INFO : Dataset filename: dataset.hdf5
[2022-04-04 16:26:28 main:481] : INFO : Configuration:
[2022-04-04 16:26:28 main:482] : INFO : Model type: GRU
[2022-04-04 16:26:28 main:483] : INFO : Validation Loss Threshold: 0.0001
[2022-04-04 16:26:28 main:484] : INFO : Max Epochs: 2048
[2022-04-04 16:26:28 main:485] : INFO : Batch Size: 128
[2022-04-04 16:26:28 main:486] : INFO : Learning Rate: 0.01
[2022-04-04 16:26:28 main:487] : INFO : Patience: 10
[2022-04-04 16:26:28 main:488] : INFO : Hidden Width: 12
[2022-04-04 16:26:28 main:489] : INFO : # Recurrent Layers: 4
[2022-04-04 16:26:28 main:490] : INFO : # Backend Layers: 4
[2022-04-04 16:26:28 main:491] : INFO : # Threads: 1
[2022-04-04 16:26:28 main:493] : INFO : Preparing Dataset
[2022-04-04 16:26:28 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-04 16:26:29 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: unknown error
Exception raised from current_device at /home/mlcbuild/git/pytorch-build/build-cuda/pytorch-prefix/src/pytorch/c10/cuda/CUDAFunctions.h:40 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7efcbaadd99b in ./libc10.so)
frame #1: at::cuda::getCurrentDeviceProperties() + 0x167 (0x7efc3d2333b7 in ./libtorch_cuda.so)
frame #2: <unknown function> + 0x88018 (0x561652e88018 in ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200)
frame #3: <unknown function> + 0x29fd0 (0x7efc3c3c0fd0 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: __libc_start_main + 0x7d (0x7efc3c3c107d in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: <unknown function> + 0x8675a (0x561652e8675a in ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200)
SIGABRT: abort called
Stack trace (14 frames):
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x56165317df9c]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7efc3c3d9520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0xf8)[0x7efc3c42d828]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7efc3c3d9476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd7)[0x7efc3c3bf7b7]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x135)[0x56165322f7f5]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398846)[0x561653198846]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398891)[0x561653198891]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x3968c4)[0x5616531968c4]
./libtorch_cuda.so(_ZN2at4cuda26getCurrentDevicePropertiesEv+0x1bd)[0x7efc3d23340d]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x88018)[0x561652e88018]
/lib/x86_64-linux-gnu/libc.so.6(+0x29fd0)[0x7efc3c3c0fd0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x7d)[0x7efc3c3c107d]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x561652e8675a]
Exiting...
DEBUG: Args: ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200 -c --maxepoch 2048
nthreads: 1 gpudev: 0
Re-exec()-ing to set environment correctly
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: unknown error
Exception raised from current_device at /home/mlcbuild/git/pytorch-build/build-cuda/pytorch-prefix/src/pytorch/c10/cuda/CUDAFunctions.h:40 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fe89fa7b99b in ./libc10.so)
frame #1: at::cuda::getCurrentDeviceProperties() + 0x167 (0x7fe8221d13b7 in ./libtorch_cuda.so)
frame #2: <unknown function> + 0x88018 (0x564876088018 in ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200)
frame #3: <unknown function> + 0x29fd0 (0x7fe82135efd0 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: __libc_start_main + 0x7d (0x7fe82135f07d in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: <unknown function> + 0x8675a (0x56487608675a in ../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200)
SIGABRT: abort called
Stack trace (14 frames):
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x37df9c)[0x56487637df9c]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fe821377520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0xf8)[0x7fe8213cb828]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fe821377476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd7)[0x7fe82135d7b7]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x135)[0x56487642f7f5]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398846)[0x564876398846]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x398891)[0x564876398891]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x3968c4)[0x5648763968c4]
./libtorch_cuda.so(_ZN2at4cuda26getCurrentDevicePropertiesEv+0x1bd)[0x7fe8221d140d]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x88018)[0x564876088018]
/lib/x86_64-linux-gnu/libc.so.6(+0x29fd0)[0x7fe82135efd0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x7d)[0x7fe82135f07d]
../../projects/www.mlcathome.org_mlcathome/mlds-gpu_9.80_x86_64-pc-linux-gnu__cuda10200(+0x8675a)[0x56487608675a]
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)