Task 14624587

Name ParityModified-1647049068-15951-3-0_1
Workunit 11308656
Created 19 Apr 2022, 3:33:33 UTC
Sent 19 Apr 2022, 3:33:38 UTC
Report deadline 27 Apr 2022, 3:33:38 UTC
Received 19 Apr 2022, 4:41:38 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -529697949 (0xE06D7363) Unknown error code
Computer ID 4313
Run time 5 min 29 sec
CPU time 3 min 50 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 9,464.73 GFLOPS
Application version Machine Learning Dataset Generator (GPU) v9.75 (cuda10200)
windows_x86_64
Peak working set size 1.73 GB
Peak swap size 3.76 GB
Peak disk usage 1.54 GB

Stderr output

<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 3765269347 (0xe06d7363)</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: NVIDIA GeForce GTX 1080)
[2022-04-18 20:35:27	                main:435]	:	INFO	:	Set logging level to 1
[2022-04-18 20:35:27	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-04-18 20:35:27	                main:444]	:	INFO	:	Resolving all filenames
[2022-04-18 20:35:27	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-04-18 20:35:27	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 0)
[2022-04-18 20:35:27	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-04-18 20:35:27	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-18 20:35:27	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-04-18 20:35:27	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-04-18 20:35:27	                main:474]	:	INFO	:	Configuration: 
[2022-04-18 20:35:27	                main:475]	:	INFO	:	    Model type: GRU
[2022-04-18 20:35:27	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-04-18 20:35:27	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-04-18 20:35:27	                main:478]	:	INFO	:	    Batch Size: 128
[2022-04-18 20:35:27	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-04-18 20:35:27	                main:480]	:	INFO	:	    Patience: 10
[2022-04-18 20:35:27	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-04-18 20:35:27	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-04-18 20:35:27	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-04-18 20:35:27	                main:484]	:	INFO	:	    # Threads: 1
[2022-04-18 20:35:27	                main:486]	:	INFO	:	Preparing Dataset
[2022-04-18 20:35:27	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-18 20:35:28	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-18 20:35:34	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-04-18 20:35:34	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-18 20:35:35	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-18 20:35:35	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-04-18 20:35:35	                main:494]	:	INFO	:	Creating Model
[2022-04-18 20:35:35	                main:507]	:	INFO	:	Preparing config file
[2022-04-18 20:35:35	                main:519]	:	INFO	:	Creating new config file
[2022-04-18 20:35:35	                main:538]	:	INFO	:	This is a continuation WU, loading previous network
[2022-04-18 20:35:39	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-04-18 20:35:40	                main:562]	:	INFO	:	Starting Training
[2022-04-18 20:35:48	                main:574]	:	INFO	:	Epoch 1 | loss: 0.031363 | val_loss: 0.0311929 | Time: 8921.99 ms
[2022-04-18 20:35:55	                main:574]	:	INFO	:	Epoch 2 | loss: 0.03119 | val_loss: 0.0311693 | Time: 6838.34 ms
[2022-04-18 20:36:02	                main:574]	:	INFO	:	Epoch 3 | loss: 0.0311686 | val_loss: 0.0311738 | Time: 7120.22 ms
[2022-04-18 20:36:09	                main:574]	:	INFO	:	Epoch 4 | loss: 0.0311678 | val_loss: 0.0311736 | Time: 6480.94 ms
[2022-04-18 20:36:16	                main:574]	:	INFO	:	Epoch 5 | loss: 0.0311645 | val_loss: 0.0311735 | Time: 6858.39 ms
[2022-04-18 20:36:23	                main:574]	:	INFO	:	Epoch 6 | loss: 0.031161 | val_loss: 0.031169 | Time: 7578.46 ms
[2022-04-18 20:36:30	                main:574]	:	INFO	:	Epoch 7 | loss: 0.0311588 | val_loss: 0.0311677 | Time: 6963.46 ms
[2022-04-18 20:36:38	                main:574]	:	INFO	:	Epoch 8 | loss: 0.0311584 | val_loss: 0.0311648 | Time: 7082.4 ms
[2022-04-18 20:36:45	                main:574]	:	INFO	:	Epoch 9 | loss: 0.0311592 | val_loss: 0.0311669 | Time: 6810 ms
[2022-04-18 20:36:52	                main:574]	:	INFO	:	Epoch 10 | loss: 0.0311566 | val_loss: 0.0311665 | Time: 6948.03 ms
[2022-04-18 20:36:59	                main:574]	:	INFO	:	Epoch 11 | loss: 0.0311569 | val_loss: 0.0311643 | Time: 6952.88 ms
[2022-04-18 20:37:06	                main:574]	:	INFO	:	Epoch 12 | loss: 0.0311567 | val_loss: 0.0311709 | Time: 6954.27 ms
[2022-04-18 20:37:12	                main:574]	:	INFO	:	Epoch 13 | loss: 0.0311607 | val_loss: 0.0311692 | Time: 6668.73 ms
[2022-04-18 20:37:20	                main:574]	:	INFO	:	Epoch 14 | loss: 0.0311582 | val_loss: 0.0311704 | Time: 7216.87 ms
[2022-04-18 20:37:27	                main:574]	:	INFO	:	Epoch 15 | loss: 0.0311593 | val_loss: 0.0311681 | Time: 7788.49 ms
[2022-04-18 20:37:35	                main:574]	:	INFO	:	Epoch 16 | loss: 0.0311559 | val_loss: 0.0311627 | Time: 7211.52 ms
[2022-04-18 20:37:41	                main:574]	:	INFO	:	Epoch 17 | loss: 0.0311573 | val_loss: 0.031165 | Time: 6401.45 ms
[2022-04-18 20:37:48	                main:574]	:	INFO	:	Epoch 18 | loss: 0.031157 | val_loss: 0.0311645 | Time: 6622.2 ms
[2022-04-18 20:37:55	                main:574]	:	INFO	:	Epoch 19 | loss: 0.0311565 | val_loss: 0.0311623 | Time: 6922.99 ms
[2022-04-18 20:38:02	                main:574]	:	INFO	:	Epoch 20 | loss: 0.0311555 | val_loss: 0.031165 | Time: 6723.01 ms
[2022-04-18 20:38:09	                main:574]	:	INFO	:	Epoch 21 | loss: 0.0311542 | val_loss: 0.0311659 | Time: 7308.31 ms
[2022-04-18 20:38:16	                main:574]	:	INFO	:	Epoch 22 | loss: 0.0311549 | val_loss: 0.0311627 | Time: 6751.79 ms
[2022-04-18 20:38:23	                main:574]	:	INFO	:	Epoch 23 | loss: 0.0311547 | val_loss: 0.0311634 | Time: 7694.12 ms
[2022-04-18 20:38:31	                main:574]	:	INFO	:	Epoch 24 | loss: 0.0311532 | val_loss: 0.0311646 | Time: 6942.14 ms
[2022-04-18 20:38:37	                main:574]	:	INFO	:	Epoch 25 | loss: 0.0311517 | val_loss: 0.031165 | Time: 6380.59 ms
[2022-04-18 20:38:44	                main:574]	:	INFO	:	Epoch 26 | loss: 0.0311524 | val_loss: 0.0311636 | Time: 6647.9 ms
[2022-04-18 20:38:51	                main:574]	:	INFO	:	Epoch 27 | loss: 0.0311517 | val_loss: 0.031162 | Time: 6998.54 ms
[2022-04-18 20:38:57	                main:574]	:	INFO	:	Epoch 28 | loss: 0.0311519 | val_loss: 0.0311657 | Time: 6596.12 ms
[2022-04-18 20:39:04	                main:574]	:	INFO	:	Epoch 29 | loss: 0.0311523 | val_loss: 0.0311632 | Time: 6052.97 ms
[2022-04-18 20:39:10	                main:574]	:	INFO	:	Epoch 30 | loss: 0.0311521 | val_loss: 0.0311639 | Time: 6774.56 ms
[2022-04-18 20:39:17	                main:574]	:	INFO	:	Epoch 31 | loss: 0.0311505 | val_loss: 0.0311634 | Time: 6945.29 ms
[2022-04-18 20:39:25	                main:574]	:	INFO	:	Epoch 32 | loss: 0.0311553 | val_loss: 0.0311662 | Time: 7472.38 ms
[2022-04-18 20:39:31	                main:574]	:	INFO	:	Epoch 33 | loss: 0.0311591 | val_loss: 0.0311654 | Time: 6655.4 ms
[2022-04-18 20:39:38	                main:574]	:	INFO	:	Epoch 34 | loss: 0.0311556 | val_loss: 0.0311667 | Time: 6801.7 ms
[2022-04-18 20:39:45	                main:574]	:	INFO	:	Epoch 35 | loss: 0.0311528 | val_loss: 0.0311636 | Time: 6933.89 ms
[2022-04-18 20:39:52	                main:574]	:	INFO	:	Epoch 36 | loss: 0.0311542 | val_loss: 0.0311623 | Time: 7068.18 ms
[2022-04-18 20:39:59	                main:574]	:	INFO	:	Epoch 37 | loss: 0.0311524 | val_loss: 0.0311645 | Time: 6849.16 ms
[2022-04-18 20:40:05	                main:574]	:	INFO	:	Epoch 38 | loss: 0.0311528 | val_loss: 0.0311644 | Time: 5761.58 ms
[2022-04-18 20:40:12	                main:574]	:	INFO	:	Epoch 39 | loss: 0.0311531 | val_loss: 0.0311673 | Time: 6323.45 ms
[2022-04-18 20:40:18	                main:574]	:	INFO	:	Epoch 40 | loss: 0.031153 | val_loss: 0.031162 | Time: 6532.6 ms
[2022-04-18 20:40:24	                main:574]	:	INFO	:	Epoch 41 | loss: 0.0311502 | val_loss: 0.0311685 | Time: 5683.32 ms
[2022-04-18 20:40:28	                main:574]	:	INFO	:	Epoch 42 | loss: 0.0311491 | val_loss: 0.031164 | Time: 4340.18 ms
[2022-04-18 20:40:33	                main:574]	:	INFO	:	Epoch 43 | loss: 0.0311512 | val_loss: 0.0311642 | Time: 4343.41 ms
[2022-04-18 20:40:37	                main:574]	:	INFO	:	Epoch 44 | loss: 0.0311521 | val_loss: 0.0311658 | Time: 4315.29 ms
[2022-04-18 20:40:41	                main:574]	:	INFO	:	Epoch 45 | loss: 0.0311539 | val_loss: 0.0311725 | Time: 4405.94 ms
[2022-04-18 20:40:46	                main:574]	:	INFO	:	Epoch 46 | loss: 0.0311549 | val_loss: 0.031167 | Time: 4428.14 ms


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFB50604F69

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.17.0


Dump Timestamp    : 04/18/22 20:40:51
Install Directory : D:\Program Files\BOINC\
Data Directory    : D:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( D:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\ProgramData\BOINC\slots\0;D:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome


ModLoad: 000000008f500000 00000000003df000 D:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000528f0000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.1566) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 10.0.19041.1566 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1566

ModLoad: 00000000518d0000 00000000000be000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.1645) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 10.0.19041.1566 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1566

ModLoad: 00000000505d0000 00000000002c8000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.1645) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernelbase.pdb
    File Version          : 10.0.19041.1566 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1566

ModLoad: 000000004d890000 0000000000090000 C:\WINDOWS\SYSTEM32\apphelp.dll (6.2.19041.1320) (-exported- Symbols Loaded)
    Linked PDB Filename   : apphelp.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000f6200000 0000000000494000 C:\WINDOWS\SYSTEM32\AcLayers.DLL (6.2.19041.1266) (-exported- Symbols Loaded)
    Linked PDB Filename   : AcLayers.pdb
    File Version          : 10.0.19041.1266 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1266

ModLoad: 0000000051550000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.19041.546

ModLoad: 00000000509e0000 00000000001a0000 C:\WINDOWS\System32\USER32.dll (6.2.19041.1503) (-exported- Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000503d0000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.1645) (-exported- Symbols Loaded)
    Linked PDB Filename   : win32u.pdb
    File Version          : 10.0.19041.1645 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1645

ModLoad: 0000000051990000 000000000002a000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.1620) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 10.0.19041.1620 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1620

ModLoad: 00000000502c0000 000000000010b000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.1620) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32full.pdb
    File Version          : 10.0.19041.1620 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1620

ModLoad: 0000000050430000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.789

ModLoad: 0000000050010000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.789

ModLoad: 00000000511b0000 0000000000055000 C:\WINDOWS\System32\SHLWAPI.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 0000000052660000 00000000000ae000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.1466) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 0000000051440000 000000000009c000 C:\WINDOWS\System32\sechost.dll (6.2.19041.1586) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 0000000051080000 0000000000125000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.1645) (-exported- Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000557e0000 0000000000003000 C:\WINDOWS\SYSTEM32\sfc.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : sfc.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000321d0000 0000000000094000 C:\WINDOWS\SYSTEM32\WINSPOOL.DRV (6.2.19041.1566) (-exported- Symbols Loaded)
    Linked PDB Filename   : winspool.pdb
    File Version          : 10.0.19041.1023 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1023

ModLoad: 0000000032af0000 0000000000012000 C:\WINDOWS\SYSTEM32\sfc_os.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : sfc_os.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000002a1d0000 0000000000011000 C:\WINDOWS\SYSTEM32\SortWindows61.dll (6.2.19041.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : SortWindows61.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 0000000050ee0000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000001ee20000 0000000000058000 D:\ProgramData\BOINC\slots\0\c10.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000049ed0000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.19041.867 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.867

ModLoad: 0000000043c40000 0000000024b49000 D:\ProgramData\BOINC\slots\0\torch_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000035110000 000000000000c000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.29.30135.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\a01\_work\4\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
    File Version          : 14.29.30135.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.29.30135.0

ModLoad: 0000000034f90000 000000000001b000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.29.30135.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\a01\_work\4\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
    File Version          : 14.29.30135.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.29.30135.0

ModLoad: 00000000357c0000 000000000008d000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.29.30135.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\a01\_work\4\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
    File Version          : 14.29.30135.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.29.30135.0

ModLoad: 000000001edd0000 0000000000045000 D:\ProgramData\BOINC\slots\0\c10_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000040b90000 00000000030a5000 D:\ProgramData\BOINC\slots\0\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1031
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA SPARSE BLAS Library
    Product Version       : 6,14,11,1031

ModLoad: 00000000257e0000 000000001b3a4000 D:\ProgramData\BOINC\slots\0\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,10020
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.86 CUDNN Library
    Product Version       : 6,14,11,10020

ModLoad: 0000000045800000 0000000000010000 D:\ProgramData\BOINC\slots\0\nvToolsExt64_1.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb

ModLoad: 0000000022320000 00000000034b2000 D:\ProgramData\BOINC\slots\0\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA RAND Library
    Product Version       : 6,14,11,1012

ModLoad: 000000001e5c0000 0000000003d5b000 D:\ProgramData\BOINC\slots\0\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000139d0000 000000000abe7000 D:\ProgramData\BOINC\slots\0\torch_cpu.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000007af30000 0000000001eca000 D:\ProgramData\BOINC\slots\0\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 000000009ef60000 00000000002db000 D:\ProgramData\BOINC\slots\0\fbgemm.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000009f240000 000000000020d000 D:\ProgramData\BOINC\slots\0\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
    Linked PDB Filename   : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
    File Version          : 20200205
    Company Name          : Intel Corporation
    Product Name          : Intel(R) OpenMP* Runtime Library
    Product Version       : 5.0

ModLoad: 0000000028a00000 0000000000040000 D:\ProgramData\BOINC\slots\0\asmjit.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000000a830000 000000000919f000 D:\ProgramData\BOINC\slots\0\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA FFT Library
    Product Version       : 6,14,11,1012

ModLoad: 000000004df50000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000009b290000 00000000002cd000 C:\WINDOWS\SYSTEM32\nvcuda.dll (30.0.15.1179) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_75\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
    File Version          : 30.0.15.1179
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.6.110 driver
    Product Version       : 30.0.15.1179

ModLoad: 0000000009190000 00000000016a0000 C:\WINDOWS\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_3b12ac0f95b18b9d\nvcuda64.dll (30.0.15.1179) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_75\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
    File Version          : 30.0.15.1179
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.6.110 driver
    Product Version       : 30.0.15.1179

ModLoad: 000000004b910000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000004fc00000 0000000000012000 C:\WINDOWS\SYSTEM32\msasn1.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msasn1.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000004b790000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptnet.pdb
    File Version          : 10.0.19041.906 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.906

ModLoad: 0000000050160000 0000000000156000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.1320) (-exported- Symbols Loaded)
    Linked PDB Filename   : crypt32.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000467a0000 0000000000148000 C:\WINDOWS\SYSTEM32\drvstore.dll (6.2.19041.1566) (-exported- Symbols Loaded)
    Linked PDB Filename   : drvstore.pdb
    File Version          : 10.0.19041.1566 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1566

ModLoad: 000000004fdf0000 000000000002c000 C:\WINDOWS\SYSTEM32\devobj.dll (6.2.19041.1620) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 10.0.19041.1620 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1620

ModLoad: 0000000050110000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.1620) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 10.0.19041.1620 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1620

ModLoad: 000000004fa80000 000000000002c000 C:\WINDOWS\SYSTEM32\wldp.dll (6.2.19041.1566) (-exported- Symbols Loaded)
    Linked PDB Filename   : WLDP.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 000000004f9d0000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000043f60000 0000000000768000 C:\WINDOWS\SYSTEM32\nvapi64.dll (30.0.15.1179) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_75\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
    File Version          : 30.0.15.1179
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA Windows drivers
    Product Version       : 30.0.15.1179

ModLoad: 0000000051aa0000 0000000000744000 C:\WINDOWS\System32\SHELL32.dll (6.2.19041.1566) (-exported- Symbols Loaded)
    Linked PDB Filename   : shell32.pdb
    File Version          : 10.0.19041.964 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.964

ModLoad: 00000000521f0000 000000000046a000 C:\WINDOWS\System32\SETUPAPI.dll (6.2.19041.1620) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 0000000050400000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcrypt.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000457c0000 0000000000009000 D:\ProgramData\BOINC\slots\0\caffe2_nvrtc.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000081c0000 0000000000fca000 D:\ProgramData\BOINC\slots\0\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6.14.11.9000
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.89 NVRTC Library
    Product Version       : 6.14.11.9000

ModLoad: 0000000050540000 0000000000082000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.1415) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1415



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 3681, Write: 521, Other 2307

- I/O Transfers Counters -
Read: 6737584, Write: 140475, Other 43954

- Paged Pool Usage -
QuotaPagedPoolUsage: 3492856, QuotaPeakPagedPoolUsage: 3509240
QuotaNonPagedPoolUsage: 55208, QuotaPeakNonPagedPoolUsage: 55208

- Virtual Memory Usage -
VirtualSize: -241717248, PeakVirtualSize: 378736640

- Pagefile Usage -
PagefileUsage: -241717248, PeakPagefileUsage: -241688576

- Working Set Size -
WorkingSetSize: 1875615744, PeakWorkingSetSize: 1875619840, PageFaultCount: 653490

*** Dump of thread ID 4864 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 47968752.000000, User Time: 1325312512.000000, Wait Time: 32480632.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFB50604F69

- Registers -
rax=0000000000000000 rbx=00000000476bb320 rcx=0000000000000000 rdx=0000000000000000 rsi=00000000f22e45f0 rdi=0000000019930520
r8=0000000000001fa0 r9=00000000217406c0 r10=0000000000000023 r11=000000005587a18c r12=0000000005c1e200 r13=0000000015800000
r14=0000000005c14800 r15=0000000000633450 rip=0000000050604f69 rsp=00000000f22e4110 rbp=00000000f22e4380
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
f22e41e0 34f96480 47491fc8 474917f0 00000312 00000000 KERNELBASE!RaiseException+0x0 
f22e4240 469b9724 4748dbb8 14a00000 f22e43e8 14400000 VCRUNTIME140!_CxxThrowException+0x0 
f22e4720 46a125ea f22e47e8 00000001 f22e47e8 00000001 torch_cuda!at::native::_cudnn_rnn+0x0 
f22e4800 46a20cbf f22e4ef0 f22e66f8 f22e5840 5294164a torch_cuda!at::native::set_storage_cuda_+0x0 
f22e48e0 1941b9fc f22e4ef0 48e7c390 00000000 f62077bd torch_cuda!at::native::set_storage_cuda_+0x0 
f22e4a60 194531a2 52a60000 528f0000 001f5000 52a62a24 torch_cpu!at::native::mkldnn_sigmoid_+0x0 
f22e4d40 1a702613 00000000 00000000 00000003 85f17818 torch_cpu!at::_cudnn_rnn+0x0 
f22e5240 1a71080b 7fcb55a8 f22e5200 52a60000 00000004 torch_cpu!torch::autograd::NotImplemented::~NotImplemented+0x0 
f22e5320 1941b9fc f22e58a0 5294164a f22e5480 091b7a77 torch_cpu!torch::autograd::GraphRoot::apply+0x0 
f22e54a0 194531a2 582f6fe0 00000000 7fcb5658 478da3f0 torch_cpu!at::native::mkldnn_sigmoid_+0x0 
f22e5780 469b69a1 f22e0001 00000001 00000004 00000004 torch_cpu!at::_cudnn_rnn+0x0 
f22e5950 469c0117 00000000 f22e66f8 1edd2e0d f22e5a70 torch_cuda!at::native::_use_cudnn_ctc_loss+0x0 
f22e59e0 191a107b 1e2f1ac0 f22e5d00 be1d1dac 09387783 torch_cuda!at::native::_cudnn_rnn_flatten_weight+0x0 
f22e5c50 191d3b16 00000000 f22e66f8 00000000 f62077bd torch_cpu!at::native::fbgemm_pack_quantized_matrix+0x0 
f22e5de0 1954ae89 f22e5e58 00000001 f22e5e58 00000001 torch_cpu!at::native::gru+0x0 
f22e5e70 1a839fa3 f22e5f70 00000001 f22e60f0 f22e66f8 torch_cpu!at::zeros_out+0x0 
f22e5f00 1953dd35 1ee71ea8 55887190 00000000 1ee71ea8 torch_cpu!torch::autograd::GraphRoot::apply+0x0 
f22e5f80 194188ec 1e2f75c8 528f0000 1c8b4d98 06707380 torch_cpu!at::zeros_out+0x0 
f22e6080 1948c1bb 0000000c 00000080 00000004 a6147500 torch_cpu!at::native::mkldnn_sigmoid_+0x0 
f22e6100 1af77294 f22e6220 06707380 f22e6210 f22e63b8 torch_cpu!at::gru+0x0 
f22e6330 1af762ab f22e66d0 193a030e f22e66d0 f22e66f8 torch_cpu!torch::nn::GRUImpl::forward_helper+0x0 
f22e6420 8f512621 0000000b f22e7130 00000000 1ee71ea8 torch_cpu!torch::nn::GRUImpl::forward+0x0 
f22e6620 8f51705a 00000000 f22e0000 f22e0601 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::error+0x0 
f22e6b20 8f51b950 f22e6c20 f22e6c20 00000000 3ba07990 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0 
f22ff8c0 8f75c6f8 59d33da0 00000000 00000000 55922c40 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
f22ff900 518e7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
f22ff930 52942651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
f22ff9b0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 27640 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 32480630.000000

- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=00000000f27ff6d0 rsi=0000000000000000 rdi=0000000000000064
r8=00000000f27ff6a8 r9=0000000000000000 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=000000005298d3f4 rsp=00000000f27ff6a8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
f27ff6a0 5061962e f27ff768 00000000 00000000 00000000 ntdll!NtDelayExecution+0x0 
f27ff740 8f53a07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0 
f27ff770 518e7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
f27ff7a0 52942651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
f27ff820 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 14148 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 32480630.000000

- Registers -
rax=000000000000005b rbx=000000000000000d rcx=000000000000000d rdx=00000000f28ff440 rsi=0000000000000000 rdi=000000000000000d
r8=00000000596fdb30 r9=0000000000000000 r10=0000000000000000 r11=00000000f28ff3b0 r12=0000000000000064 r13=00000000f28ff440
r14=00000000f28ff140 r15=0000000000000000 rip=000000005298d8c4 rsp=00000000f28ff0e8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
f28ff0e0 5061cb10 00000000 000004d0 52a5a4d0 52941d9c ntdll!ZwWaitForMultipleObjects+0x0 
f28ff3d0 5061ca0e 52942651 00000000 0000000d 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0 
f28ff410 0936d554 5a43d750 529201a7 505d53dc 096afbee KERNELBASE!WaitForMultipleObjects+0x0 
f28ff650 094158cd 00000000 5a44d830 00000000 00000000 nvcuda64!+0x0 
f28ff700 0936cf23 a3ec3880 a25dc8d6 5a4d1850 00000000 nvcuda64!cuProfilerStop+0x0 
f28ff730 096a4594 5a4d1850 00000000 00000000 00000000 nvcuda64!+0x0 
f28ff760 518e7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
f28ff790 52942651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
f28ff810 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 4992 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 32460048.000000

- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=0000000000000254 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000254
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=000000005a44da10 r13=0000000000000000
r14=0000000000000254 r15=0000000000000000 rip=000000005298cdf4 rsp=00000000f29ffb08 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
f29ffb00 505f1a5e 00000000 00001f80 00000000 00000000 ntdll!ZwWaitForSingleObject+0x0 
f29ffba0 09595d0e 5a4d0f20 00000000 00000000 00000254 KERNELBASE!WaitForSingleObjectEx+0x0 
f29ffde0 0936cf23 00000000 5a4d0f20 09190000 5aa206d0 nvcuda64!cuProfilerStop+0x0 
f29ffe10 096a4594 5a4d0f20 00000000 00000000 00000000 nvcuda64!+0x0 
f29ffe40 518e7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
f29ffe70 52942651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
f29ffef0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 24936 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 22343750.000000, User Time: 909062528.000000, Wait Time: 32480502.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=0000000092422848 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000092422848
r8=0000000000000000 r9=0000000000000988 r10=0000000052a60000 r11=00000000000004c4 r12=0000000000000000 r13=0000000000000000
r14=00000000f2aff540 r15=00000000924227f8 rip=0000000052990764 rsp=00000000f2aff518 rbp=00000000f2aff570
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
f2aff510 52954021 f2aff810 00000000 f2aff610 00000000 ntdll!NtWaitForAlertByThreadId+0x0 
f2aff590 5063cf99 92422840 92422838 924227f0 1abb05b4 ntdll!RtlSleepConditionVariableSRW+0x0 
f2aff5d0 357d2959 00000000 a6ed0ff0 00000000 00000000 KERNELBASE!SleepConditionVariableSRW+0x0 
f2aff600 357d2bea 53de0388 00000000 f2aff810 92422838 MSVCP140!std::_Winerror_message+0x0 
f2aff630 1abb064b 924227f0 f2aff730 924227f0 a6ed0fd0 MSVCP140!_Cnd_wait+0x0 
f2aff670 1abb25b1 82dc2130 ffffffff a6ed0fd0 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0 
f2aff960 1abb2471 1e0d6d10 f2aff9a0 1e0d6d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0 
f2aff9b0 1aba9744 00080001 a6e32920 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0 
f2aff9e0 50031bb2 a6e32920 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0 
f2affa10 518e7034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0 
f2affa40 52942651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
f2affac0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 21876 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 32479096.000000

- Registers -
rax=00000000000001d3 rbx=0000000053e4c790 rcx=0000000000000014 rdx=0000000053e4c410 rsi=0000000055870b50 rdi=0000000053e4c790
r8=0000000000000a5d r9=0000000000000000 r10=000000005217d000 r11=0000000000000a5d r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000529907c4 rsp=00000000f23ff438 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
f23ff430 52942dc7 55870c68 55870c68 55870b50 55870b68 ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
f23ff730 518e7034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0 
f23ff760 52942651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
f23ff7e0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 440 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 32480504.000000

- Registers -
rax=00000000000001d3 rbx=0000000053e4dd50 rcx=0000000000000104 rdx=0000000053e4d9d0 rsi=00000000558a2490 rdi=0000000053e4dd50
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000529907c4 rsp=00000000f24ffaf8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
f24ffaf0 52942dc7 00000000 00000000 558a2490 00000000 ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
f24ffdf0 518e7034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0 
f24ffe20 52942651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
f24ffea0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>


©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)