Task 4315770

Name ParityMachine-1612383333-25226-4_1
Workunit 2250289
Created 5 Mar 2021, 1:40:16 UTC
Sent 5 Mar 2021, 2:29:53 UTC
Report deadline 12 Mar 2021, 2:29:53 UTC
Received 5 Mar 2021, 15:23:44 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status 203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID 9697
Run time 2 min 34 sec
CPU time 2 min 11 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 2,491.95 GFLOPS
Application version Machine Learning Dataset Generator (GPU) v9.75 (cuda10200)
windows_x86_64
Peak working set size 1.64 GB
Peak swap size 3.60 GB
Peak disk usage 1.53 GB

Stderr output

<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce GTX 1050 Ti)
[2021-03-05 23:21:14	                main:435]	:	INFO	:	Set logging level to 1
[2021-03-05 23:21:14	                main:441]	:	INFO	:	Running in BOINC Client mode
[2021-03-05 23:21:14	                main:444]	:	INFO	:	Resolving all filenames
[2021-03-05 23:21:14	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 (exists = 1)
[2021-03-05 23:21:14	                main:452]	:	INFO	:	Resolved: model.cfg => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1612383333-25226-4_1_r1355926207_1 (exists = 0)
[2021-03-05 23:21:14	                main:452]	:	INFO	:	Resolved: model-final.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1612383333-25226-4_1_r1355926207_0 (exists = 0)
[2021-03-05 23:21:14	                main:452]	:	INFO	:	Resolved: model-input.pt => ../../projects/www.mlcathome.org_mlcathome/ParityMachine-1612383333-25226-4 (exists = 1)
[2021-03-05 23:21:14	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2021-03-05 23:21:14	                main:472]	:	INFO	:	Dataset filename: ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5
[2021-03-05 23:21:14	                main:474]	:	INFO	:	Configuration: 
[2021-03-05 23:21:14	                main:475]	:	INFO	:	    Model type: GRU
[2021-03-05 23:21:15	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2021-03-05 23:21:15	                main:477]	:	INFO	:	    Max Epochs: 2048
[2021-03-05 23:21:15	                main:478]	:	INFO	:	    Batch Size: 128
[2021-03-05 23:21:15	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2021-03-05 23:21:15	                main:480]	:	INFO	:	    Patience: 10
[2021-03-05 23:21:15	                main:481]	:	INFO	:	    Hidden Width: 12
[2021-03-05 23:21:15	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2021-03-05 23:21:15	                main:483]	:	INFO	:	    # Backend Layers: 4
[2021-03-05 23:21:15	                main:484]	:	INFO	:	    # Threads: 1
[2021-03-05 23:21:15	                main:486]	:	INFO	:	Preparing Dataset
[2021-03-05 23:21:15	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-03-05 23:21:16	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-03-05 23:21:18	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2021-03-05 23:21:18	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-03-05 23:21:18	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from ../../projects/www.mlcathome.org_mlcathome/ParityMachine-train-val-dataset.hdf5 into memory
[2021-03-05 23:21:18	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2021-03-05 23:21:18	                main:494]	:	INFO	:	Creating Model
[2021-03-05 23:21:19	                main:507]	:	INFO	:	Preparing config file
[2021-03-05 23:21:19	                main:519]	:	INFO	:	Creating new config file
[2021-03-05 23:21:19	                main:538]	:	INFO	:	This is a continuation WU, loading previous network
[2021-03-05 23:21:20	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2021-03-05 23:21:20	                main:562]	:	INFO	:	Starting Training
[2021-03-05 23:21:23	                main:574]	:	INFO	:	Epoch 1 | loss: 0.0310775 | val_loss: 0.0311333 | Time: 2506.14 ms
[2021-03-05 23:21:25	                main:574]	:	INFO	:	Epoch 2 | loss: 0.0310497 | val_loss: 0.0311444 | Time: 2020.43 ms
[2021-03-05 23:21:27	                main:574]	:	INFO	:	Epoch 3 | loss: 0.0310434 | val_loss: 0.0311349 | Time: 2025.07 ms
[2021-03-05 23:21:29	                main:574]	:	INFO	:	Epoch 4 | loss: 0.0310423 | val_loss: 0.0311378 | Time: 2017.62 ms
[2021-03-05 23:21:31	                main:574]	:	INFO	:	Epoch 5 | loss: 0.0310412 | val_loss: 0.0311379 | Time: 2035.98 ms
[2021-03-05 23:21:33	                main:574]	:	INFO	:	Epoch 6 | loss: 0.0310432 | val_loss: 0.0311418 | Time: 2024.43 ms
[2021-03-05 23:21:36	                main:574]	:	INFO	:	Epoch 7 | loss: 0.0310438 | val_loss: 0.0311407 | Time: 2015.8 ms
[2021-03-05 23:21:38	                main:574]	:	INFO	:	Epoch 8 | loss: 0.0310418 | val_loss: 0.0311409 | Time: 2009.19 ms
[2021-03-05 23:21:40	                main:574]	:	INFO	:	Epoch 9 | loss: 0.0310404 | val_loss: 0.0311403 | Time: 2015.17 ms
[2021-03-05 23:21:42	                main:574]	:	INFO	:	Epoch 10 | loss: 0.031041 | val_loss: 0.0311441 | Time: 2042.08 ms
[2021-03-05 23:21:44	                main:574]	:	INFO	:	Epoch 11 | loss: 0.0310393 | val_loss: 0.0311486 | Time: 2019.74 ms
[2021-03-05 23:21:46	                main:574]	:	INFO	:	Epoch 12 | loss: 0.0310405 | val_loss: 0.0311423 | Time: 2014.1 ms
[2021-03-05 23:21:48	                main:574]	:	INFO	:	Epoch 13 | loss: 0.0310478 | val_loss: 0.0311403 | Time: 2068.84 ms
[2021-03-05 23:21:50	                main:574]	:	INFO	:	Epoch 14 | loss: 0.0310449 | val_loss: 0.0311386 | Time: 2001.8 ms
[2021-03-05 23:21:53	                main:574]	:	INFO	:	Epoch 15 | loss: 0.0310412 | val_loss: 0.0311392 | Time: 2033.29 ms
[2021-03-05 23:21:55	                main:574]	:	INFO	:	Epoch 16 | loss: 0.0310398 | val_loss: 0.0311382 | Time: 2043 ms
[2021-03-05 23:21:57	                main:574]	:	INFO	:	Epoch 17 | loss: 0.0310384 | val_loss: 0.0311443 | Time: 2057.95 ms
[2021-03-05 23:21:59	                main:574]	:	INFO	:	Epoch 18 | loss: 0.0310434 | val_loss: 0.0311353 | Time: 2012.81 ms
[2021-03-05 23:22:01	                main:574]	:	INFO	:	Epoch 19 | loss: 0.0310411 | val_loss: 0.0311409 | Time: 2042.71 ms
[2021-03-05 23:22:03	                main:574]	:	INFO	:	Epoch 20 | loss: 0.0310352 | val_loss: 0.0311447 | Time: 2073.96 ms
[2021-03-05 23:22:05	                main:574]	:	INFO	:	Epoch 21 | loss: 0.0310342 | val_loss: 0.03116 | Time: 2030.73 ms
[2021-03-05 23:22:08	                main:574]	:	INFO	:	Epoch 22 | loss: 0.0310427 | val_loss: 0.0311385 | Time: 2010.63 ms
[2021-03-05 23:22:10	                main:574]	:	INFO	:	Epoch 23 | loss: 0.0310415 | val_loss: 0.0311366 | Time: 2066.61 ms
[2021-03-05 23:22:12	                main:574]	:	INFO	:	Epoch 24 | loss: 0.0310475 | val_loss: 0.0311392 | Time: 2018.96 ms
[2021-03-05 23:22:14	                main:574]	:	INFO	:	Epoch 25 | loss: 0.0310495 | val_loss: 0.0311537 | Time: 2024.02 ms
[2021-03-05 23:22:16	                main:574]	:	INFO	:	Epoch 26 | loss: 0.0310493 | val_loss: 0.0311399 | Time: 2017.61 ms
[2021-03-05 23:22:18	                main:574]	:	INFO	:	Epoch 27 | loss: 0.0310488 | val_loss: 0.0311536 | Time: 2024.21 ms
[2021-03-05 23:22:20	                main:574]	:	INFO	:	Epoch 28 | loss: 0.031045 | val_loss: 0.0311424 | Time: 2007.34 ms
[2021-03-05 23:22:22	                main:574]	:	INFO	:	Epoch 29 | loss: 0.0310571 | val_loss: 0.0311332 | Time: 2020.26 ms
[2021-03-05 23:22:25	                main:574]	:	INFO	:	Epoch 30 | loss: 0.0310578 | val_loss: 0.0311395 | Time: 2059.7 ms
[2021-03-05 23:22:27	                main:574]	:	INFO	:	Epoch 31 | loss: 0.0310518 | val_loss: 0.0311416 | Time: 2011.5 ms
[2021-03-05 23:22:29	                main:574]	:	INFO	:	Epoch 32 | loss: 0.03105 | val_loss: 0.031135 | Time: 2032.19 ms
[2021-03-05 23:22:31	                main:574]	:	INFO	:	Epoch 33 | loss: 0.0310486 | val_loss: 0.0311435 | Time: 2111.01 ms
[2021-03-05 23:22:33	                main:574]	:	INFO	:	Epoch 34 | loss: 0.0310494 | val_loss: 0.0311414 | Time: 2034.86 ms
[2021-03-05 23:22:35	                main:574]	:	INFO	:	Epoch 35 | loss: 0.0310448 | val_loss: 0.0311312 | Time: 2026.59 ms
[2021-03-05 23:22:37	                main:574]	:	INFO	:	Epoch 36 | loss: 0.0310408 | val_loss: 0.031136 | Time: 2032.72 ms
[2021-03-05 23:22:40	                main:574]	:	INFO	:	Epoch 37 | loss: 0.0310408 | val_loss: 0.031143 | Time: 2341.97 ms
[2021-03-05 23:22:42	                main:574]	:	INFO	:	Epoch 38 | loss: 0.0310404 | val_loss: 0.0311413 | Time: 2021.86 ms
[2021-03-05 23:22:44	                main:574]	:	INFO	:	Epoch 39 | loss: 0.0310488 | val_loss: 0.0311357 | Time: 2020.13 ms
[2021-03-05 23:22:46	                main:574]	:	INFO	:	Epoch 40 | loss: 0.0310477 | val_loss: 0.0311517 | Time: 2082.31 ms
[2021-03-05 23:22:48	                main:574]	:	INFO	:	Epoch 41 | loss: 0.0310814 | val_loss: 0.0311356 | Time: 2049.51 ms
[2021-03-05 23:22:50	                main:574]	:	INFO	:	Epoch 42 | loss: 0.0311074 | val_loss: 0.0311298 | Time: 2012.54 ms
[2021-03-05 23:22:53	                main:574]	:	INFO	:	Epoch 43 | loss: 0.0310988 | val_loss: 0.031127 | Time: 2080.5 ms
[2021-03-05 23:22:55	                main:574]	:	INFO	:	Epoch 44 | loss: 0.0310964 | val_loss: 0.0311289 | Time: 2056.8 ms
[2021-03-05 23:22:57	                main:574]	:	INFO	:	Epoch 45 | loss: 0.0310895 | val_loss: 0.0311376 | Time: 2014.15 ms
[2021-03-05 23:22:59	                main:574]	:	INFO	:	Epoch 46 | loss: 0.0310859 | val_loss: 0.0311335 | Time: 2056.92 ms
[2021-03-05 23:23:01	                main:574]	:	INFO	:	Epoch 47 | loss: 0.031083 | val_loss: 0.031131 | Time: 2098.75 ms
[2021-03-05 23:23:03	                main:574]	:	INFO	:	Epoch 48 | loss: 0.0310828 | val_loss: 0.0311309 | Time: 2030.02 ms
[2021-03-05 23:23:05	                main:574]	:	INFO	:	Epoch 49 | loss: 0.0310809 | val_loss: 0.0311248 | Time: 2027.26 ms
[2021-03-05 23:23:08	                main:574]	:	INFO	:	Epoch 50 | loss: 0.0310798 | val_loss: 0.0311282 | Time: 2064.46 ms
[2021-03-05 23:23:10	                main:574]	:	INFO	:	Epoch 51 | loss: 0.0310756 | val_loss: 0.0311297 | Time: 2051.35 ms
[2021-03-05 23:23:12	                main:574]	:	INFO	:	Epoch 52 | loss: 0.0310705 | val_loss: 0.0311256 | Time: 2005.77 ms
[2021-03-05 23:23:14	                main:574]	:	INFO	:	Epoch 53 | loss: 0.0310692 | val_loss: 0.0311315 | Time: 2057.2 ms
[2021-03-05 23:23:16	                main:574]	:	INFO	:	Epoch 54 | loss: 0.0310678 | val_loss: 0.0311284 | Time: 2091.53 ms
[2021-03-05 23:23:18	                main:574]	:	INFO	:	Epoch 55 | loss: 0.031065 | val_loss: 0.0311319 | Time: 2041.55 ms
[2021-03-05 23:23:21	                main:574]	:	INFO	:	Epoch 56 | loss: 0.0310648 | val_loss: 0.0311344 | Time: 2046.3 ms
[2021-03-05 23:23:23	                main:574]	:	INFO	:	Epoch 57 | loss: 0.0310703 | val_loss: 0.0311331 | Time: 2089.16 ms
[2021-03-05 23:23:25	                main:574]	:	INFO	:	Epoch 58 | loss: 0.0310655 | val_loss: 0.0311347 | Time: 2014.34 ms
[2021-03-05 23:23:27	                main:574]	:	INFO	:	Epoch 59 | loss: 0.0310584 | val_loss: 0.0311315 | Time: 2013.69 ms
[2021-03-05 23:23:29	                main:574]	:	INFO	:	Epoch 60 | loss: 0.0310585 | val_loss: 0.0311294 | Time: 2128.7 ms
[2021-03-05 23:23:31	                main:574]	:	INFO	:	Epoch 61 | loss: 0.0310604 | val_loss: 0.0311336 | Time: 2107.01 ms
[2021-03-05 23:23:34	                main:574]	:	INFO	:	Epoch 62 | loss: 0.0310604 | val_loss: 0.0311295 | Time: 2021.7 ms


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFA6A1F9AD2

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.17.0


Dump Timestamp    : 03/05/21 23:23:36
Install Directory : D:\BOINC\program\
Data Directory    : D:\BOINC\data
Project Symstore  : 
LoadLibraryA( D:\BOINC\data\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\BOINC\data\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\BOINC\data\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\BOINC\data\version.dll ): GetLastError = 126
Loaded Library    : version.dll
[2021-03-05 23:23:36	                main:574]	:	INFO	:	Epoch 63 | loss: 0.0310613 | val_loss: 0.0311309 | Time: 2083.97 ms
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\BOINC\data\slots\2;D:\BOINC\data\projects\www.mlcathome.org_mlcathome


ModLoad: 000000001a130000 00000000003df000 D:\BOINC\data\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000006c630000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 000000006ae90000 00000000000bd000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.804) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 10.0.19041.804 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.804

ModLoad: 000000006a130000 00000000002c9000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.804) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernelbase.pdb
    File Version          : 10.0.19041.804 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.804

ModLoad: 000000006b730000 00000000001a0000 C:\WINDOWS\System32\USER32.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 000000006a500000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : win32u.pdb
    File Version          : 10.0.19041.844 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.844

ModLoad: 000000006ac40000 000000000002a000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 10.0.19041.746 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.746

ModLoad: 000000006a580000 000000000010b000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32full.pdb
    File Version          : 10.0.19041.746 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.746

ModLoad: 0000000069e50000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.789

ModLoad: 000000006a400000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.789

ModLoad: 000000006b4c0000 00000000000ac000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.610) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 000000006adf0000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 7.0.19041.546

ModLoad: 000000006b570000 000000000009c000 C:\WINDOWS\System32\sechost.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.1

ModLoad: 000000006b390000 000000000012b000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 0000000056680000 0000000000058000 D:\BOINC\data\slots\2\c10.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000053620000 0000000000099000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.22.27821.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\3\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
    File Version          : 14.22.27821.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Visual Studio? 2017
    Product Version       : 14.22.27821.0

ModLoad: 0000000068380000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.804) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.19041.804 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.804

ModLoad: 00000000536c0000 0000000000016000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.22.27821.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\3\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
    File Version          : 14.22.27821.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Visual Studio? 2017
    Product Version       : 14.22.27821.0

ModLoad: 0000000064bf0000 000000000000b000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.22.27821.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\agent\_work\3\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
    File Version          : 14.22.27821.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Visual Studio? 2017
    Product Version       : 14.22.27821.0

ModLoad: 00000000d6590000 000000000abe7000 D:\BOINC\data\slots\2\torch_cpu.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000027cd0000 00000000002db000 D:\BOINC\data\slots\2\fbgemm.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000134c0000 000000000020d000 D:\BOINC\data\slots\2\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
    Linked PDB Filename   : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
    File Version          : 20200205
    Company Name          : Intel Corporation
    Product Name          : Intel(R) OpenMP* Runtime Library
    Product Version       : 5.0

ModLoad: 0000000055d20000 0000000000040000 D:\BOINC\data\slots\2\asmjit.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000b1a40000 0000000024b49000 D:\BOINC\data\slots\2\torch_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000049ce0000 0000000000045000 D:\BOINC\data\slots\2\c10_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000ef3d0000 00000000034b2000 D:\BOINC\data\slots\2\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA RAND Library
    Product Version       : 6,14,11,1012

ModLoad: 0000000064be0000 0000000000010000 D:\BOINC\data\slots\2\nvToolsExt64_1.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb

ModLoad: 00000000ec320000 00000000030a5000 D:\BOINC\data\slots\2\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1031
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA SPARSE BLAS Library
    Product Version       : 6,14,11,1031

ModLoad: 00000000adce0000 0000000003d5b000 D:\BOINC\data\slots\2\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000abe10000 0000000001eca000 D:\BOINC\data\slots\2\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000a2c70000 000000000919f000 D:\BOINC\data\slots\2\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA FFT Library
    Product Version       : 6,14,11,1012

ModLoad: 0000000075b30000 000000001b3a4000 D:\BOINC\data\slots\2\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,10020
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.86 CUDNN Library
    Product Version       : 6,14,11,10020

ModLoad: 000000006c3a0000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000686a0000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000006a7d0000 0000000000469000 C:\WINDOWS\System32\Setupapi.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 000000006a530000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000069e20000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.662) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcrypt.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 0000000069b20000 000000000002c000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000006a0d0000 0000000000060000 C:\WINDOWS\System32\WINTRUST.dll (6.2.19041.804) (-exported- Symbols Loaded)
    Linked PDB Filename   : wintrust.pdb
    File Version          : 10.0.19041.804 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.804

ModLoad: 0000000069f70000 000000000015f000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : crypt32.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 0000000069960000 0000000000012000 C:\WINDOWS\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msasn1.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.546

ModLoad: 000000006b8d0000 0000000000742000 C:\WINDOWS\System32\Shell32.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : shell32.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 0000000067b20000 0000000000790000 C:\WINDOWS\SYSTEM32\windows.storage.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : Windows.Storage.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 000000006c040000 0000000000355000 C:\WINDOWS\System32\combase.dll (6.2.19041.844) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 00000000697e0000 000000000002c000 C:\WINDOWS\SYSTEM32\Wldp.dll (6.2.19041.662) (-exported- Symbols Loaded)
    Linked PDB Filename   : WLDP.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 000000006ad40000 00000000000ae000 C:\WINDOWS\System32\SHCORE.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : shcore.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 000000006acd0000 0000000000055000 C:\WINDOWS\System32\shlwapi.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 0000000055dc0000 000000000029d000 C:\WINDOWS\system32\nvcuda.dll (27.21.14.6140) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r460\r461_36\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
    File Version          : 27.21.14.6140
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.2.135 driver
    Product Version       : 27.21.14.6140

ModLoad: 0000000061c70000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000f3f90000 00000000015c2000 C:\WINDOWS\system32\DriverStore\FileRepository\nvdmi.inf_amd64_b706d4e32506a94f\nvcuda64.dll (27.21.14.6140) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r460\r461_36\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
    File Version          : 27.21.14.6140
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.2.135 driver
    Product Version       : 27.21.14.6140

ModLoad: 000000006b320000 000000000006b000 C:\WINDOWS\System32\WS2_32.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : ws2_32.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 0000000061c30000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptnet.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000069750000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000669d0000 000000000009e000 C:\WINDOWS\system32\uxtheme.dll (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : UxTheme.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 0000000054d70000 00000000006f1000 C:\WINDOWS\SYSTEM32\nvapi64.dll (27.21.14.6140) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r460\r461_36\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
    File Version          : 27.21.14.6140
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA Windows drivers
    Product Version       : 27.21.14.6140

ModLoad: 000000006af50000 000000000012a000 C:\WINDOWS\System32\OLE32.DLL (6.2.19041.746) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 10.0.19041.561 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.561

ModLoad: 0000000064320000 0000000000009000 D:\BOINC\data\slots\2\caffe2_nvrtc.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000f9930000 0000000000fca000 D:\BOINC\data\slots\2\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6.14.11.9000
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.89 NVRTC Library
    Product Version       : 6.14.11.9000

ModLoad: 0000000069ef0000 0000000000080000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.662) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.19041.662 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.19041.662



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 3689, Write: 536, Other 1258

- I/O Transfers Counters -
Read: 6545308, Write: 23635, Other 32636

- Paged Pool Usage -
QuotaPagedPoolUsage: 3504056, QuotaPeakPagedPoolUsage: 3504056
QuotaNonPagedPoolUsage: 52216, QuotaPeakNonPagedPoolUsage: 52624

- Virtual Memory Usage -
VirtualSize: -428244992, PeakVirtualSize: 190398464

- Pagefile Usage -
PagefileUsage: -428244992, PeakPagefileUsage: -427040768

- Working Set Size -
WorkingSetSize: 1766068224, PeakWorkingSetSize: 1766281216, PageFaultCount: 632707

*** Dump of thread ID 14500 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 7529330.000000

- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFA6A1F9AD2

- Registers -
rax=0000000000000000 rbx=0000000000000001 rcx=000000001a3e8c40 rdx=00000000af4fe4d0 rsi=0000000000000000 rdi=0000000000000000
r8=00000000af4fe4d0 r9=000000001a3e8c30 r10=0000000000000fff r11=0000000000000ff0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=000000006a1f9ad2 rsp=00000000af4fe4a8 rbp=0000000000000000
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
af4fe4a0 1a169100 00000001 af4fe4d0 af4fe4d0 1a3e8c30 KERNELBASE!DebugBreak+0x0 
af4fe8e0 1a169ce4 00000000 af4fec44 00000000 00000324 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
af4ff840 1a16a084 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
af4ff870 6aea7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
af4ff8a0 6c682651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
af4ff920 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 5516 (state: Running): ***

- Information -
Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 39218752.000000, User Time: 766093760.000000, Wait Time: 7529330.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=000000008bc93a88 rdx=0000000000000000 rsi=0000000000000000 rdi=000000008bc93a88
r8=0000000029031440 r9=0000000000000001 r10=0000000000008000 r11=00000000aedb5f40 r12=0000000000000000 r13=0000000000000000
r14=00000000aedb5770 r15=000000008bc93a38 rip=000000006c6d0604 rsp=00000000aedb5748 rbp=00000000aedb57a0
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
aedb5740 6c694021 2b53f3c0 56687ce6 aedb5970 f415f6a9 ntdll!ZwWaitForAlertByThreadId+0x0 
aedb57c0 6a19ba89 8bc93a30 8bc93a78 00000000 00008000 ntdll!RtlSleepConditionVariableSRW+0x0 
aedb5800 5365640d aedb5970 00000000 00008000 aedb5420 KERNELBASE!SleepConditionVariableSRW+0x0 
aedb5840 536324b8 00000000 aedb5420 49ce2e0d 566a2433 MSVCP140!__crtSetThreadpoolWait+0x0 
aedb5870 53632380 aedb5d20 53632b87 00000000 dbb60231 MSVCP140!std::_Winerror_message+0x0 
aedb58e0 dd77064b 9af20f00 000001c0 8bc93a30 8bc93a30 MSVCP140!std::_Winerror_message+0x0 
aedb5920 dd7725b1 290e0a00 9af20ef0 9af20ef0 00000000 torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0 
aedb5c10 dd76f15d aedb5ef0 aedb5ef0 aedb5ef0 290e0a00 torch_cpu!torch::autograd::Engine::thread_main+0x0 
aedb5e80 dd76eea0 a06cd000 e0c96d10 fffffffe 00000000 torch_cpu!torch::autograd::Engine::execute_with_graph_task+0x0 
aedb5fb0 dd7656f9 9c6cca20 9c6cca20 aedb5fa0 aedb60a8 torch_cpu!torch::autograd::Engine::execute+0x0 
aedb64d0 dd765086 00000000 aedb6938 9c6ccb00 aedb65a0 torch_cpu!torch::autograd::Engine::make_anomaly_metadata+0x0 
aedb6570 dda609be 8b714240 9c6cca28 00000000 aedb6510 torch_cpu!torch::autograd::backward+0x0 
aedb6650 dd491cd0 2b156870 00000000 00000000 00000000 torch_cpu!torch::autograd::VariableType::allCUDATypes+0x0 
aedb6680 dc130a06 00000000 00000000 00010001 00000001 torch_cpu!torch::autograd::GraphRoot::apply+0x0 
aedb68b0 1a147166 aedb6980 aedb69c0 aedb6980 aedb6f20 torch_cpu!at::Tensor::backward+0x0 
aedb6db0 1a14b950 aedb6eb0 aedb6eb0 00000000 74fbeda0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0 
aedcfb50 1a38c6f8 2b1f0600 00000000 00000000 290db970 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
aedcfb90 6aea7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
aedcfbc0 6c682651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
aedcfc40 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 13472 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 7529328.000000

- Registers -
rax=000000000000005b rbx=000000000000000d rcx=000000000000000d rdx=00000000af5ffbd0 rsi=0000000000000000 rdi=000000000000000d
r8=00000000af5ff878 r9=000000002b275970 r10=0000000000000000 r11=0000000000000246 r12=0000000000000064 r13=00000000af5ffbd0
r14=00000000af5ff8d0 r15=0000000000000000 rip=000000006c6cd764 rsp=00000000af5ff878 rbp=000000002b275970
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
af5ff870 6a180d40 00000000 aeefe000 6c79a4d0 aeefe000 ntdll!ZwWaitForMultipleObjects+0x0 
af5ffb60 6a180c3e 290e4030 004e0049 0000000d 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0 
af5ffba0 f4146714 2b27b730 00000000 00000000 00000000 KERNELBASE!WaitForMultipleObjects+0x0 
af5ffde0 f41de55d 00000000 2b275970 00000000 00000000 nvcuda64!+0x0 
af5ffe90 f41460e3 ead21c82 9738492f 2ba51b70 00000000 nvcuda64!cuProfilerStop+0x0 
af5ffec0 f445d9d4 2ba51b70 00000000 00000000 00000000 nvcuda64!+0x0 
af5ffef0 6aea7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
af5fff20 6c682651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
af5fffa0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 1348 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 7520388.000000

- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=00000000000002b8 rdx=0000000000000000 rsi=0000000000000000 rdi=00000000000002b8
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=00000000000002b8 r15=0000000000000000 rip=000000006c6ccc94 rsp=00000000af6ffaf8 rbp=000000002b2d55c0
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
af6ffaf0 6a14a34e 00000000 00000000 00000000 00000000 ntdll!NtWaitForSingleObject+0x0 
af6ffb90 f435cfae 2ba51c60 6a152cb5 00000000 000002b8 KERNELBASE!WaitForSingleObjectEx+0x0 
af6ffc00 f41460e3 2b2d55c0 2b61a6e0 2ba51c60 00000000 nvcuda64!cuProfilerStop+0x0 
af6ffc30 f445d9d4 2ba51c60 00000000 00000000 00000000 nvcuda64!+0x0 
af6ffc60 6aea7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
af6ffc90 6c682651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
af6ffd10 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 14304 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 7529314.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=000000002b390df8 rdx=00000000af7ffe78 rsi=00000000af7ffde0 rdi=0000000000000000
r8=0000000000000001 r9=00000000000037e0 r10=000000002b541be0 r11=00000000fbff4318 r12=00000000af7ffe78 r13=0000000000000000
r14=000000002b390df8 r15=000000002b541be0 rip=000000006c6d0604 rsp=00000000af7ffdb8 rbp=00000000af7ffe10
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
af7ffdb0 6c694295 00000102 00000102 00000000 af7ffde0 ntdll!ZwWaitForAlertByThreadId+0x0 
af7ffe20 6a19b9d9 2b53f3c0 00000000 00000000 00000000 ntdll!RtlSleepConditionVariableCS+0x0 
af7ffe50 f40c4a9c 2b53f3c0 6c6807b0 2b53f3c0 ffb3b4c0 KERNELBASE!SleepConditionVariableCS+0x0 
af7ffeb0 f41460e3 00000001 eae792e7 00000000 00000000 nvcuda64!+0x0 
af7ffee0 f445d9d4 2ba52230 00000000 00000000 00000000 nvcuda64!+0x0 
af7fff10 6aea7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
af7fff40 6c682651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
af7fffc0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 13692 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 33593752.000000, User Time: 476562496.000000, Wait Time: 7529317.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=000000008220ac08 rdx=0000000000000000 rsi=0000000000000000 rdi=000000008220ac08
r8=0000000000000000 r9=0000000000000000 r10=0000000000008000 r11=00000000af8fb210 r12=0000000000000000 r13=0000000000000000
r14=00000000af8ff520 r15=000000008220abb8 rip=000000006c6d0604 rsp=00000000af8ff4f8 rbp=00000000af8ff550
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
af8ff4f0 6c694021 8bc93a78 8bc93a78 af8ff630 8bc93a78 ntdll!ZwWaitForAlertByThreadId+0x0 
af8ff570 6a19ba89 8220abb0 8220abf8 00000000 00000000 ntdll!RtlSleepConditionVariableSRW+0x0 
af8ff5b0 5365640d 1bff5940 1bff5940 1bff5940 00000000 KERNELBASE!SleepConditionVariableSRW+0x0 
af8ff5f0 536324b8 9af20f20 00000000 00000000 dd7705b4 MSVCP140!__crtSetThreadpoolWait+0x0 
af8ff620 53632380 00000000 53632b87 00000000 9af20f20 MSVCP140!std::_Winerror_message+0x0 
af8ff690 dd77064b 9af20f00 000001c0 8220abb0 8220abb0 MSVCP140!std::_Winerror_message+0x0 
af8ff6d0 dd7725b1 a0b89990 ffffffff 9af20f00 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0 
af8ff9c0 dd772471 e0c96d10 af8ffa00 e0c96d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0 
af8ffa10 dd769744 00080001 9fa2e9d0 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0 
af8ffa40 6a421bb2 9fa2e9d0 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0 
af8ffa70 6aea7034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0 
af8ffaa0 6c682651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
af8ffb20 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>


©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)