Task 14535667
| Name | ParityModified-1645997675-18601-3-0_1 |
| Workunit | 11189101 |
| Created | 7 Apr 2022, 5:14:13 UTC |
| Sent | 7 Apr 2022, 5:47:51 UTC |
| Report deadline | 15 Apr 2022, 5:47:51 UTC |
| Received | 7 Apr 2022, 7:10:56 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | -529697949 (0xE06D7363) Unknown error code |
| Computer ID | 4313 |
| Run time | 2 min 12 sec |
| CPU time | 1 min 25 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 9,464.73 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 1.76 GB |
| Peak swap size | 3.79 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 3765269347 (0xe06d7363)</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1080)
[2022-04-06 23:08:13 main:435] : INFO : Set logging level to 1
[2022-04-06 23:08:13 main:441] : INFO : Running in BOINC Client mode
[2022-04-06 23:08:13 main:444] : INFO : Resolving all filenames
[2022-04-06 23:08:13 main:452] : INFO : Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-06 23:08:13 main:452] : INFO : Resolved: model.cfg => model.cfg (exists = 0)
[2022-04-06 23:08:13 main:452] : INFO : Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-06 23:08:13 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-06 23:08:13 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-04-06 23:08:13 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-04-06 23:08:13 main:474] : INFO : Configuration:
[2022-04-06 23:08:13 main:475] : INFO : Model type: GRU
[2022-04-06 23:08:13 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-04-06 23:08:13 main:477] : INFO : Max Epochs: 2048
[2022-04-06 23:08:13 main:478] : INFO : Batch Size: 128
[2022-04-06 23:08:13 main:479] : INFO : Learning Rate: 0.01
[2022-04-06 23:08:13 main:480] : INFO : Patience: 10
[2022-04-06 23:08:13 main:481] : INFO : Hidden Width: 12
[2022-04-06 23:08:13 main:482] : INFO : # Recurrent Layers: 4
[2022-04-06 23:08:13 main:483] : INFO : # Backend Layers: 4
[2022-04-06 23:08:13 main:484] : INFO : # Threads: 1
[2022-04-06 23:08:13 main:486] : INFO : Preparing Dataset
[2022-04-06 23:08:13 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-06 23:08:14 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-06 23:08:17 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-04-06 23:08:17 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-06 23:08:17 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-06 23:08:17 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-04-06 23:08:17 main:494] : INFO : Creating Model
[2022-04-06 23:08:17 main:507] : INFO : Preparing config file
[2022-04-06 23:08:17 main:519] : INFO : Creating new config file
[2022-04-06 23:08:17 main:538] : INFO : This is a continuation WU, loading previous network
[2022-04-06 23:08:19 main:559] : INFO : Loading DataLoader into Memory
[2022-04-06 23:08:19 main:562] : INFO : Starting Training
[2022-04-06 23:08:26 main:574] : INFO : Epoch 1 | loss: 0.0311506 | val_loss: 0.0311785 | Time: 7518.98 ms
[2022-04-06 23:08:34 main:574] : INFO : Epoch 2 | loss: 0.0311223 | val_loss: 0.0311816 | Time: 7706.63 ms
[2022-04-06 23:08:41 main:574] : INFO : Epoch 3 | loss: 0.0311167 | val_loss: 0.0311859 | Time: 7163.71 ms
[2022-04-06 23:08:48 main:574] : INFO : Epoch 4 | loss: 0.0311147 | val_loss: 0.0311818 | Time: 6657.18 ms
[2022-04-06 23:08:55 main:574] : INFO : Epoch 5 | loss: 0.0311191 | val_loss: 0.0311823 | Time: 7274.69 ms
[2022-04-06 23:09:02 main:574] : INFO : Epoch 6 | loss: 0.0311173 | val_loss: 0.0311781 | Time: 6796 ms
[2022-04-06 23:09:09 main:574] : INFO : Epoch 7 | loss: 0.031117 | val_loss: 0.0311824 | Time: 7370.86 ms
[2022-04-06 23:09:17 main:574] : INFO : Epoch 8 | loss: 0.0311179 | val_loss: 0.0311862 | Time: 7261.05 ms
[2022-04-06 23:09:23 main:574] : INFO : Epoch 9 | loss: 0.0311172 | val_loss: 0.0311813 | Time: 6542.72 ms
[2022-04-06 23:09:32 main:574] : INFO : Epoch 10 | loss: 0.0311163 | val_loss: 0.0311842 | Time: 8226.12 ms
[2022-04-06 23:09:39 main:574] : INFO : Epoch 11 | loss: 0.031113 | val_loss: 0.0311834 | Time: 6633.68 ms
[2022-04-06 23:09:46 main:574] : INFO : Epoch 12 | loss: 0.0311191 | val_loss: 0.031182 | Time: 7935.05 ms
[2022-04-06 23:09:53 main:574] : INFO : Epoch 13 | loss: 0.0311166 | val_loss: 0.0311854 | Time: 6739.36 ms
[2022-04-06 23:10:01 main:574] : INFO : Epoch 14 | loss: 0.0311166 | val_loss: 0.0311821 | Time: 7350.43 ms
[2022-04-06 23:10:08 main:574] : INFO : Epoch 15 | loss: 0.0311176 | val_loss: 0.03118 | Time: 6986.53 ms
[2022-04-06 23:10:14 main:574] : INFO : Epoch 16 | loss: 0.0311172 | val_loss: 0.0311804 | Time: 6684.49 ms
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFA35C94F69
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.17.0
Dump Timestamp : 04/06/22 23:10:17
Install Directory : D:\Program Files\BOINC\
Data Directory : D:\ProgramData\BOINC
Project Symstore :
LoadLibraryA( D:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( D:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: D:\ProgramData\BOINC\slots\4;D:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome
ModLoad: 00000000fedf0000 00000000003df000 D:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000038490000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 00000000371b0000 00000000000be000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000035c60000 00000000002c8000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000033420000 0000000000090000 C:\WINDOWS\SYSTEM32\apphelp.dll (6.2.19041.1320) (-exported- Symbols Loaded)
Linked PDB Filename : apphelp.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000d3f30000 0000000000494000 C:\WINDOWS\SYSTEM32\AcLayers.DLL (6.2.19041.1266) (-exported- Symbols Loaded)
Linked PDB Filename : AcLayers.pdb
File Version : 10.0.19041.1266 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1266
ModLoad: 0000000037f70000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.19041.546
ModLoad: 0000000036840000 00000000001a0000 C:\WINDOWS\System32\USER32.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000035be0000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.1586) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.19041.1586 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1586
ModLoad: 0000000036f90000 000000000002b000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.19041.1202 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1202
ModLoad: 0000000035f30000 000000000010b000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 00000000363a0000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 00000000362a0000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 0000000036780000 0000000000055000 C:\WINDOWS\System32\SHLWAPI.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shlwapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000379d0000 00000000000ae000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000365b0000 000000000009c000 C:\WINDOWS\System32\sechost.dll (6.2.19041.1586) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000037080000 0000000000125000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000e3bb0000 0000000000003000 C:\WINDOWS\SYSTEM32\sfc.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : sfc.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000222b0000 0000000000094000 C:\WINDOWS\SYSTEM32\WINSPOOL.DRV (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : winspool.pdb
File Version : 10.0.19041.1023 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1023
ModLoad: 00000000142d0000 0000000000012000 C:\WINDOWS\SYSTEM32\sfc_os.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : sfc_os.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000d3f10000 0000000000011000 C:\WINDOWS\SYSTEM32\SortWindows61.dll (6.2.19041.1) (-exported- Symbols Loaded)
Linked PDB Filename : SortWindows61.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000036580000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000009b50000 0000000000058000 D:\ProgramData\BOINC\slots\4\c10.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000002e390000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.19041.867 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.867
ModLoad: 0000000028f60000 0000000024b49000 D:\ProgramData\BOINC\slots\4\torch_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000015fa0000 000000000001b000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.29.30135.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\a01\_work\4\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
File Version : 14.29.30135.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.29.30135.0
ModLoad: 0000000015fc0000 000000000008d000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.29.30135.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\a01\_work\4\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
File Version : 14.29.30135.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.29.30135.0
ModLoad: 00000000157f0000 000000000000c000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.29.30135.0) (-exported- Symbols Loaded)
Linked PDB Filename : d:\a01\_work\4\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
File Version : 14.29.30135.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.29.30135.0
ModLoad: 000000005afa0000 00000000030a5000 D:\ProgramData\BOINC\slots\4\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1031
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA SPARSE BLAS Library
Product Version : 6,14,11,1031
ModLoad: 0000000025aa0000 00000000034b2000 D:\ProgramData\BOINC\slots\4\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA RAND Library
Product Version : 6,14,11,1012
ModLoad: 000000001aeb0000 000000000abe7000 D:\ProgramData\BOINC\slots\4\torch_cpu.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000f6960000 000000001b3a4000 D:\ProgramData\BOINC\slots\4\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,10020
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.86 CUDNN Library
Product Version : 6,14,11,10020
ModLoad: 0000000011d10000 000000000919f000 D:\ProgramData\BOINC\slots\4\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA FFT Library
Product Version : 6,14,11,1012
ModLoad: 00000000f2c00000 0000000003d5b000 D:\ProgramData\BOINC\slots\4\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 00000000817d0000 000000000020d000 D:\ProgramData\BOINC\slots\4\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
Linked PDB Filename : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
File Version : 20200205
Company Name : Intel Corporation
Product Name : Intel(R) OpenMP* Runtime Library
Product Version : 5.0
ModLoad: 00000000f0d30000 0000000001eca000 D:\ProgramData\BOINC\slots\4\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 00000000e2360000 0000000000045000 D:\ProgramData\BOINC\slots\4\c10_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000028990000 0000000000010000 D:\ProgramData\BOINC\slots\4\nvToolsExt64_1.dll (-exported- Symbols Loaded)
Linked PDB Filename : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb
ModLoad: 0000000067fd0000 00000000002db000 D:\ProgramData\BOINC\slots\4\fbgemm.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000121e0000 0000000000040000 D:\ProgramData\BOINC\slots\4\asmjit.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000033ae0000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 000000005acd0000 00000000002cd000 C:\WINDOWS\SYSTEM32\nvcuda.dll (30.0.15.1179) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_75\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
File Version : 30.0.15.1179
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.6.110 driver
Product Version : 30.0.15.1179
ModLoad: 00000000ef690000 00000000016a0000 C:\WINDOWS\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_3b12ac0f95b18b9d\nvcuda64.dll (30.0.15.1179) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_75\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
File Version : 30.0.15.1179
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.6.110 driver
Product Version : 30.0.15.1179
ModLoad: 00000000300a0000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000035790000 0000000000012000 C:\WINDOWS\SYSTEM32\msasn1.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msasn1.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 000000002fe60000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : cryptnet.pdb
File Version : 10.0.19041.906 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.906
ModLoad: 0000000036140000 0000000000156000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.1320) (-exported- Symbols Loaded)
Linked PDB Filename : crypt32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 000000002adc0000 0000000000148000 C:\WINDOWS\SYSTEM32\drvstore.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : drvstore.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000035950000 0000000000033000 C:\WINDOWS\SYSTEM32\devobj.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : devobj.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000035c10000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : cfgmgr32.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 0000000035610000 000000000002c000 C:\WINDOWS\SYSTEM32\wldp.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : WLDP.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000035560000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cryptbase.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000da340000 0000000000768000 C:\WINDOWS\SYSTEM32\nvapi64.dll (30.0.15.1179) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_75\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
File Version : 30.0.15.1179
Company Name : NVIDIA Corporation
Product Name : NVIDIA Windows drivers
Product Version : 30.0.15.1179
ModLoad: 0000000037280000 0000000000744000 C:\WINDOWS\System32\SHELL32.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : shell32.pdb
File Version : 10.0.19041.964 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.964
ModLoad: 0000000036a40000 0000000000470000 C:\WINDOWS\System32\SETUPAPI.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : setupapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000035bb0000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000028980000 0000000000009000 D:\ProgramData\BOINC\slots\4\caffe2_nvrtc.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000ee6c0000 0000000000fca000 D:\ProgramData\BOINC\slots\4\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6.14.11.9000
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.89 NVRTC Library
Product Version : 6.14.11.9000
ModLoad: 00000000360b0000 0000000000082000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.1415) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.19041.1415 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1415
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 3652, Write: 381, Other 1750
- I/O Transfers Counters -
Read: 6778501, Write: 40613, Other 41046
- Paged Pool Usage -
QuotaPagedPoolUsage: 3492696, QuotaPeakPagedPoolUsage: 3493496
QuotaNonPagedPoolUsage: 54800, QuotaPeakNonPagedPoolUsage: 55136
- Virtual Memory Usage -
VirtualSize: -258293760, PeakVirtualSize: 352980992
- Pagefile Usage -
PagefileUsage: -258293760, PeakPagefileUsage: -258228224
- Working Set Size -
WorkingSetSize: 1857961984, PeakWorkingSetSize: 1857966080, PageFaultCount: 648679
*** Dump of thread ID 5152 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 29375000.000000, User Time: 490312512.000000, Wait Time: 13101561.000000
- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFA35C94F69
- Registers -
rax=0000000000000000 rbx=00000000253262f0 rcx=0000000000000000 rdx=0000000000000000 rsi=000000003cee3e80 rdi=0000000019930520
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000006000 r13=0000000000000000
r14=000000003cee4bc0 r15=000000003cee4b40 rip=0000000035c94f69 rsp=000000003cee3d20 rbp=000000003cee4728
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
- Callstack -
ChildEBP RetAddr Args to Child
3cee3df0 15fa6480 acf2a70f 00000000 3cee4bc0 2047a014 KERNELBASE!RaiseException+0x0
3cee3e50 20a75272 acf2a7af 00000000 000007bd 00000116 VCRUNTIME140!_CxxThrowException+0x0
3cee3ea0 20a756d4 00000000 e3df36c0 23a9d290 2c7c5f68 torch_cpu!at::Tensor::zero_+0x0
3cee4700 2bd89ebe 3cee4930 2c7c6850 2c7c6880 000002bc torch_cpu!_THError+0x0
3cee4740 2bd85412 3cee4930 3cee4890 e683f8f0 2cbfa720 torch_cuda!__THCudaCheck+0x0
3cee4780 207259ff 3cee4b00 3cee4930 3cee4e00 09b897b0 torch_cuda!at::native::set_storage_cuda_+0x0
3cee4b10 208c726f 2c3ff630 2047ea63 00000001 09b6ef8f torch_cpu!at::native::empty_cpu+0x0
3cee4b50 20880615 3cee0000 3cee4bc8 3cee4dc0 09b57ce6 torch_cpu!at::native::mkldnn_sigmoid_+0x0
3cee4b90 2088204a 3cee0000 3cee4c00 e6673800 2b0c6ae5 torch_cpu!at::native::mkldnn_sigmoid_+0x0
3cee4c20 20880615 00000000 2047ea63 e6673840 3cee4e00 torch_cpu!at::native::mkldnn_sigmoid_+0x0
3cee4c60 20951eb0 3cee0000 00000000 3cee4de8 15ff8573 torch_cpu!at::native::mkldnn_sigmoid_+0x0
3cee4d00 20962eec 4765d657 09b89c50 3cee4e60 09b53bd6 torch_cpu!at::bucketize_out+0x0
3cee4d50 2b48f160 3cee0000 00000000 3cee4e60 00000000 torch_cpu!at::empty+0x0
3cee50f0 2b492de8 09b89c50 20480528 28ea2d00 09b89c50 torch_cuda!at::native::cummin_helper_cuda+0x0
3cee52d0 2b494379 3cee53e0 bfd10670 3cee57e0 bfd10688 torch_cuda!at::native::cummin_helper_cuda+0x0
3cee5510 2b493d64 3cee0000 3cee57e0 3cee5620 3cee57e0 torch_cuda!at::native::cat_out_cuda+0x0
3cee5710 2bd31dd2 3cee5768 00000001 3cee5768 3cee57e0 torch_cuda!at::native::cat_cuda+0x0
3cee5790 2bd44179 00000001 3cee57d0 28ea1318 04000000 torch_cuda!at::native::set_storage_cuda_+0x0
3cee57d0 20951d7d 15891750 257d0640 00000080 3cee5b40 torch_cuda!at::native::set_storage_cuda_+0x0
3cee5850 20930d81 00000000 09b73994 3cee6020 c68917b0 torch_cpu!at::bucketize_out+0x0
3cee5890 2073b4a5 3cee58d0 28ea2900 3cee59a0 3cee5a10 torch_cpu!at::_cat+0x0
3cee5960 20a227bd 3cee5a30 00000101 3cee59b8 09ba1ea8 torch_cpu!at::native::cat+0x0
3cee59e0 204e29ae 385d6620 3ceff830 fffffb30 fffffb30 torch_cpu!at::zeros_out+0x0
3cee5a20 20951d7d 3cee5a60 3cee5ad0 3cf00000 3cece000 torch_cpu!torch::nn::functional::BatchNormFuncOptions::~BatchNormFuncOptions+0x0
3cee5aa0 209575d1 3cee6040 257d0640 3cee6020 c68917b0 torch_cpu!at::bucketize_out+0x0
3cee5ae0 20747d4c 3cee5d70 00000000 3cee5d70 00000000 torch_cpu!at::cat+0x0
3cee5ca0 20a3c38d 3cee5cf8 00000001 3cee5cf8 3cee5d70 torch_cpu!at::native::stack+0x0
3cee5d20 204e29ae 3cee5f10 3cee5f30 3cee6020 c68917b0 torch_cpu!at::zeros_out+0x0
3cee5d60 20951d7d 3cee5da0 3cee5e10 00000000 236f1c8f torch_cpu!torch::nn::functional::BatchNormFuncOptions::~BatchNormFuncOptions+0x0
3cee5de0 209a02e1 3cee5f10 09b73720 3cee6040 257d0640 torch_cpu!at::bucketize_out+0x0
3cee5e20 21d33839 3cee5ea0 28e9bce0 28e9bce0 3cee5f30 torch_cpu!at::stack+0x0
3cee5ff0 204e29ae 09ba1ea8 3cee0000 00000000 3cee60e0 torch_cpu!torch::autograd::GraphRoot::apply+0x0
3cee6030 20951d7d 3cee6070 3cee60e0 3cee6170 24244d20 torch_cpu!torch::nn::functional::BatchNormFuncOptions::~BatchNormFuncOptions+0x0
3cee60b0 209a02e1 3cee6220 20479cac 1042c490 c68917b0 torch_cpu!at::bucketize_out+0x0
3cee60f0 fedfff7e 3cee61b8 24244d20 24244d20 3cee6300 torch_cpu!at::stack+0x0
3cee6190 fee02ef8 fedffe30 3cee61d0 3cee61d0 3cee6220 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::addCallback+0x0
3cee61f0 fee03ea4 c2712be0 3ceff830 3cee6390 fffffb30 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Object::getSlot+0x0
3cee6330 fedfd3d2 d42ee750 384e164a 3cee6390 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::markCompleted+0x0
3cee6360 fee04541 3cee6370 384e15e2 00000000 df5b6b60 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Object::operator=+0x0
3cee6510 fee0743a 3cee65e0 00000000 3cee7020 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::markCompleted+0x0
3cee6a10 fee0b950 3cee6b10 3cee6b10 00000000 ca26a140 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0
3ceff7b0 ff04c6f8 e80b6550 00000000 00000000 e3cd3330 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
3ceff7f0 371c7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
3ceff820 384e2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
3ceff8a0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 26612 (state: Waiting): ***
- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 13101554.000000
- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=000000003d3ffcb0 rsi=0000000000000000 rdi=0000000000000064
r8=000000003d3fe36e r9=000000000000006e r10=0000000000000004 r11=000000003d3fe9e7 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=000000003852d3f4 rsp=000000003d3ffc88 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
3d3ffc80 35ca962e 3d3ffd48 00000000 00000000 00000000 ntdll!NtDelayExecution+0x0
3d3ffd20 fee2a07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0
3d3ffd50 371c7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
3d3ffd80 384e2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
3d3ffe00 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 8696 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 13101560.000000
- Registers -
rax=000000000000005b rbx=000000000000000d rcx=000000000000000d rdx=000000003d4ff460 rsi=0000000000000000 rdi=000000000000000d
r8=0000000000000001 r9=0000000000000000 r10=0000000000000000 r11=000000003d4ff3d0 r12=0000000000000064 r13=000000003d4ff460
r14=000000003d4ff160 r15=0000000000000000 rip=000000003852d8c4 rsp=000000003d4ff108 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
3d4ff100 35cacb20 00000000 000004d0 385fa4d0 384e1d9c ntdll!ZwWaitForMultipleObjects+0x0
3d4ff3f0 35caca1e 384e2651 00000000 0000000d 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0
3d4ff430 ef86d554 e8831990 384c01a7 35c653dc efbafbee KERNELBASE!WaitForMultipleObjects+0x0
3d4ff670 ef9158cd 00000000 e8839c30 00000000 00000000 nvcuda64!+0x0
3d4ff720 ef86cf23 a472b347 a1ba7b7d e88b43b0 00000000 nvcuda64!cuProfilerStop+0x0
3d4ff750 efba4594 e88b43b0 00000000 00000000 00000000 nvcuda64!+0x0
3d4ff780 371c7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
3d4ff7b0 384e2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
3d4ff830 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 14772 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 13101556.000000
- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=0000000000000238 rdx=0000000000000000 rsi=000000003d5ff938 rdi=0000000000000238
r8=000000003d5ff9a8 r9=0000000000000011 r10=0000000000000000 r11=0000000000000246 r12=00000000e883a8b0 r13=0000000000000000
r14=0000000000000238 r15=0000000000000001 rip=000000003852cdf4 rsp=000000003d5ff908 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
3d5ff900 35c81a5e 0000001a 3685d489 3d5ff9a8 00000000 ntdll!ZwWaitForSingleObject+0x0
3d5ff9a0 efa95d0e e7e04150 00000000 00000000 00000238 KERNELBASE!WaitForSingleObjectEx+0x0
3d5ffbe0 ef86cf23 00000000 e88b4f10 ef690000 e8e024e0 nvcuda64!cuProfilerStop+0x0
3d5ffc10 efba4594 e88b4f10 00000000 00000000 00000000 nvcuda64!+0x0
3d5ffc40 371c7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
3d5ffc70 384e2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
3d5ffcf0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 27108 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 6718750.000000, User Time: 323593760.000000, Wait Time: 13101540.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=000000000ec43138 rdx=0000000000000000 rsi=0000000000000000 rdi=000000000ec43138
r8=0000000000000001 r9=0000000000000000 r10=0000000000000038 r11=000000003d6ff190 r12=0000000000000000 r13=0000000000000000
r14=000000003d6ff740 r15=000000000ec430e8 rip=0000000038530764 rsp=000000003d6ff718 rbp=000000003d6ff770
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
3d6ff710 384f4021 3d6ffa10 00000000 3d6ff810 00000000 ntdll!NtWaitForAlertByThreadId+0x0
3d6ff790 35cccfa9 0ec43130 0ec43128 0ec430e0 220905b4 ntdll!RtlSleepConditionVariableSRW+0x0
3d6ff7d0 15fd2959 00000000 08591490 00000000 00000000 KERNELBASE!SleepConditionVariableSRW+0x0
3d6ff800 15fd2bea 61dfa3c5 00000000 3d6ffa10 0ec43128 MSVCP140!std::_Winerror_message+0x0
3d6ff830 2209064b 0ec430e0 3d6ff930 0ec430e0 08591470 MSVCP140!_Cnd_wait+0x0
3d6ff870 220925b1 07f7b230 ffffffff 08591470 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0
3d6ffb60 22092471 255b6d10 3d6ffba0 255b6d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0
3d6ffbb0 22089744 00080001 2858c4f0 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0
3d6ffbe0 362c1bb2 2858c4f0 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0
3d6ffc10 371c7034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0
3d6ffc40 384e2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
3d6ffcc0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)