Task 11532353

Name ParityModified-1639952775-13356-2-0_1
Workunit 8348095
Created 13 Feb 2022, 15:20:20 UTC
Sent 13 Feb 2022, 15:45:12 UTC
Report deadline 21 Feb 2022, 15:45:12 UTC
Received 13 Feb 2022, 16:54:01 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID 571
Run time 5 min 38 sec
CPU time 5 min 12 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 5,441.63 GFLOPS
Application version Machine Learning Dataset Generator (GPU) v9.75 (cuda10200)
windows_x86_64
Peak working set size 2.10 GB
Peak swap size 4.34 GB
Peak disk usage 1.54 GB

Stderr output

<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code -1073741819 (0xc0000005)</message>
<stderr_txt>
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce GTX 1660 Ti)
[2022-02-13 16:48:27	                main:435]	:	INFO	:	Set logging level to 1
[2022-02-13 16:48:27	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-02-13 16:48:27	                main:444]	:	INFO	:	Resolving all filenames
[2022-02-13 16:48:27	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-13 16:48:27	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 0)
[2022-02-13 16:48:27	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-13 16:48:28	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-02-13 16:48:28	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-02-13 16:48:28	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-02-13 16:48:28	                main:474]	:	INFO	:	Configuration: 
[2022-02-13 16:48:28	                main:475]	:	INFO	:	    Model type: GRU
[2022-02-13 16:48:28	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-02-13 16:48:28	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-02-13 16:48:28	                main:478]	:	INFO	:	    Batch Size: 128
[2022-02-13 16:48:28	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-02-13 16:48:28	                main:480]	:	INFO	:	    Patience: 10
[2022-02-13 16:48:28	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-02-13 16:48:28	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-02-13 16:48:28	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-02-13 16:48:28	                main:484]	:	INFO	:	    # Threads: 1
[2022-02-13 16:48:29	                main:486]	:	INFO	:	Preparing Dataset
[2022-02-13 16:48:29	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-13 16:48:29	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-13 16:48:34	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-02-13 16:48:34	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-13 16:48:34	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-13 16:48:34	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-02-13 16:48:34	                main:494]	:	INFO	:	Creating Model
[2022-02-13 16:48:35	                main:507]	:	INFO	:	Preparing config file
[2022-02-13 16:48:35	                main:519]	:	INFO	:	Creating new config file
[2022-02-13 16:48:35	                main:538]	:	INFO	:	This is a continuation WU, loading previous network
[2022-02-13 16:48:38	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-02-13 16:48:42	                main:562]	:	INFO	:	Starting Training
[2022-02-13 16:48:45	                main:574]	:	INFO	:	Epoch 1 | loss: 0.0309591 | val_loss: 0.0312652 | Time: 3145.08 ms
[2022-02-13 16:48:48	                main:574]	:	INFO	:	Epoch 2 | loss: 0.0309178 | val_loss: 0.0312558 | Time: 2591.66 ms
[2022-02-13 16:48:50	                main:574]	:	INFO	:	Epoch 3 | loss: 0.0308879 | val_loss: 0.0312588 | Time: 2675.84 ms
[2022-02-13 16:48:53	                main:574]	:	INFO	:	Epoch 4 | loss: 0.0308762 | val_loss: 0.0312546 | Time: 2715.58 ms
[2022-02-13 16:48:56	                main:574]	:	INFO	:	Epoch 5 | loss: 0.0308782 | val_loss: 0.0312783 | Time: 2537.17 ms
[2022-02-13 16:48:58	                main:574]	:	INFO	:	Epoch 6 | loss: 0.0308787 | val_loss: 0.0312784 | Time: 2622.61 ms
[2022-02-13 16:49:01	                main:574]	:	INFO	:	Epoch 7 | loss: 0.0308711 | val_loss: 0.0312545 | Time: 2584.9 ms
[2022-02-13 16:49:04	                main:574]	:	INFO	:	Epoch 8 | loss: 0.0308847 | val_loss: 0.0312573 | Time: 2443.53 ms
[2022-02-13 16:49:06	                main:574]	:	INFO	:	Epoch 9 | loss: 0.03087 | val_loss: 0.0312702 | Time: 2381.31 ms
[2022-02-13 16:49:09	                main:574]	:	INFO	:	Epoch 10 | loss: 0.0308683 | val_loss: 0.0312393 | Time: 2590.5 ms
[2022-02-13 16:49:12	                main:574]	:	INFO	:	Epoch 11 | loss: 0.0308711 | val_loss: 0.0312436 | Time: 3062.63 ms
[2022-02-13 16:49:14	                main:574]	:	INFO	:	Epoch 12 | loss: 0.0308636 | val_loss: 0.0312629 | Time: 2507.23 ms
[2022-02-13 16:49:17	                main:574]	:	INFO	:	Epoch 13 | loss: 0.0308785 | val_loss: 0.0312794 | Time: 2655.02 ms
[2022-02-13 16:49:20	                main:574]	:	INFO	:	Epoch 14 | loss: 0.0308739 | val_loss: 0.0312726 | Time: 2701.01 ms
[2022-02-13 16:49:22	                main:574]	:	INFO	:	Epoch 15 | loss: 0.0308641 | val_loss: 0.0312459 | Time: 2437.6 ms
[2022-02-13 16:49:25	                main:574]	:	INFO	:	Epoch 16 | loss: 0.0308587 | val_loss: 0.0312623 | Time: 2422.67 ms
[2022-02-13 16:49:27	                main:574]	:	INFO	:	Epoch 17 | loss: 0.0308869 | val_loss: 0.031259 | Time: 2580.62 ms
[2022-02-13 16:49:30	                main:574]	:	INFO	:	Epoch 18 | loss: 0.0308841 | val_loss: 0.0312597 | Time: 2612.16 ms
[2022-02-13 16:49:33	                main:574]	:	INFO	:	Epoch 19 | loss: 0.0308822 | val_loss: 0.0312662 | Time: 2663.15 ms
[2022-02-13 16:49:35	                main:574]	:	INFO	:	Epoch 20 | loss: 0.0308823 | val_loss: 0.0312654 | Time: 2605.18 ms
[2022-02-13 16:49:38	                main:574]	:	INFO	:	Epoch 21 | loss: 0.030876 | val_loss: 0.0312644 | Time: 2695.37 ms
[2022-02-13 16:49:41	                main:574]	:	INFO	:	Epoch 22 | loss: 0.0308874 | val_loss: 0.0312411 | Time: 2578.94 ms
[2022-02-13 16:49:43	                main:574]	:	INFO	:	Epoch 23 | loss: 0.0308678 | val_loss: 0.0312435 | Time: 2444.14 ms
[2022-02-13 16:49:46	                main:574]	:	INFO	:	Epoch 24 | loss: 0.0308595 | val_loss: 0.0312625 | Time: 2543.17 ms
[2022-02-13 16:49:48	                main:574]	:	INFO	:	Epoch 25 | loss: 0.0308712 | val_loss: 0.0312306 | Time: 2494.98 ms
[2022-02-13 16:49:51	                main:574]	:	INFO	:	Epoch 26 | loss: 0.0308656 | val_loss: 0.0312411 | Time: 2636.27 ms
[2022-02-13 16:49:54	                main:574]	:	INFO	:	Epoch 27 | loss: 0.0308807 | val_loss: 0.0312319 | Time: 2697.01 ms
[2022-02-13 16:49:57	                main:574]	:	INFO	:	Epoch 28 | loss: 0.0308748 | val_loss: 0.0312412 | Time: 2792.88 ms
[2022-02-13 16:49:59	                main:574]	:	INFO	:	Epoch 29 | loss: 0.0308944 | val_loss: 0.0312207 | Time: 2541.43 ms
[2022-02-13 16:50:02	                main:574]	:	INFO	:	Epoch 30 | loss: 0.0309024 | val_loss: 0.031255 | Time: 2427.67 ms
[2022-02-13 16:50:04	                main:574]	:	INFO	:	Epoch 31 | loss: 0.0308747 | val_loss: 0.0312726 | Time: 2530.49 ms
[2022-02-13 16:50:07	                main:574]	:	INFO	:	Epoch 32 | loss: 0.0308654 | val_loss: 0.0312609 | Time: 2614.13 ms
[2022-02-13 16:50:09	                main:574]	:	INFO	:	Epoch 33 | loss: 0.030864 | val_loss: 0.0312641 | Time: 2461 ms
[2022-02-13 16:50:12	                main:574]	:	INFO	:	Epoch 34 | loss: 0.0308669 | val_loss: 0.0312538 | Time: 2500.15 ms
[2022-02-13 16:50:15	                main:574]	:	INFO	:	Epoch 35 | loss: 0.0308688 | val_loss: 0.031273 | Time: 2532.12 ms
[2022-02-13 16:50:17	                main:574]	:	INFO	:	Epoch 36 | loss: 0.0308626 | val_loss: 0.0312626 | Time: 2618.69 ms
[2022-02-13 16:50:20	                main:574]	:	INFO	:	Epoch 37 | loss: 0.0308587 | val_loss: 0.0312587 | Time: 2653 ms
[2022-02-13 16:50:23	                main:574]	:	INFO	:	Epoch 38 | loss: 0.0308609 | val_loss: 0.0312581 | Time: 2639.16 ms
[2022-02-13 16:50:25	                main:574]	:	INFO	:	Epoch 39 | loss: 0.0308667 | val_loss: 0.0312652 | Time: 2442.44 ms
[2022-02-13 16:50:27	                main:574]	:	INFO	:	Epoch 40 | loss: 0.0308603 | val_loss: 0.0312649 | Time: 2451.68 ms
[2022-02-13 16:50:30	                main:574]	:	INFO	:	Epoch 41 | loss: 0.0308664 | val_loss: 0.0312813 | Time: 2662.01 ms
[2022-02-13 16:50:33	                main:574]	:	INFO	:	Epoch 42 | loss: 0.0308689 | val_loss: 0.0312605 | Time: 2542.89 ms
[2022-02-13 16:50:35	                main:574]	:	INFO	:	Epoch 43 | loss: 0.0308651 | val_loss: 0.0312844 | Time: 2514 ms
[2022-02-13 16:50:38	                main:574]	:	INFO	:	Epoch 44 | loss: 0.0308525 | val_loss: 0.0312958 | Time: 2562.23 ms
[2022-02-13 16:50:41	                main:574]	:	INFO	:	Epoch 45 | loss: 0.0308576 | val_loss: 0.0312764 | Time: 2691.57 ms
[2022-02-13 16:50:43	                main:574]	:	INFO	:	Epoch 46 | loss: 0.03086 | val_loss: 0.0312528 | Time: 2629.51 ms
[2022-02-13 16:50:46	                main:574]	:	INFO	:	Epoch 47 | loss: 0.030859 | val_loss: 0.0312555 | Time: 2516.82 ms
[2022-02-13 16:50:48	                main:574]	:	INFO	:	Epoch 48 | loss: 0.0308665 | val_loss: 0.0312895 | Time: 2601.35 ms
[2022-02-13 16:50:51	                main:574]	:	INFO	:	Epoch 49 | loss: 0.0308619 | val_loss: 0.0312533 | Time: 2587.44 ms
[2022-02-13 16:50:54	                main:574]	:	INFO	:	Epoch 50 | loss: 0.0308685 | val_loss: 0.0312815 | Time: 2778.92 ms
[2022-02-13 16:50:57	                main:574]	:	INFO	:	Epoch 51 | loss: 0.030856 | val_loss: 0.0312912 | Time: 2674.15 ms
[2022-02-13 16:50:59	                main:574]	:	INFO	:	Epoch 52 | loss: 0.0308692 | val_loss: 0.0312804 | Time: 2478.66 ms
[2022-02-13 16:51:02	                main:574]	:	INFO	:	Epoch 53 | loss: 0.0308607 | val_loss: 0.0312706 | Time: 2613.14 ms
[2022-02-13 16:51:05	                main:574]	:	INFO	:	Epoch 54 | loss: 0.0308555 | val_loss: 0.0312701 | Time: 2960.28 ms
[2022-02-13 16:51:07	                main:574]	:	INFO	:	Epoch 55 | loss: 0.030861 | val_loss: 0.0312543 | Time: 2409.32 ms
[2022-02-13 16:51:10	                main:574]	:	INFO	:	Epoch 56 | loss: 0.0308692 | val_loss: 0.0312678 | Time: 2647.71 ms
[2022-02-13 16:51:13	                main:574]	:	INFO	:	Epoch 57 | loss: 0.0308671 | val_loss: 0.0312623 | Time: 2691.25 ms
[2022-02-13 16:51:15	                main:574]	:	INFO	:	Epoch 58 | loss: 0.0308707 | val_loss: 0.0312363 | Time: 2565.49 ms
[2022-02-13 16:51:18	                main:574]	:	INFO	:	Epoch 59 | loss: 0.0308696 | val_loss: 0.0312542 | Time: 2635.57 ms
[2022-02-13 16:51:21	                main:574]	:	INFO	:	Epoch 60 | loss: 0.0308624 | val_loss: 0.0312526 | Time: 2693.02 ms
[2022-02-13 16:51:23	                main:574]	:	INFO	:	Epoch 61 | loss: 0.0308566 | val_loss: 0.0312875 | Time: 2427.69 ms
[2022-02-13 16:51:26	                main:574]	:	INFO	:	Epoch 62 | loss: 0.0308585 | val_loss: 0.0312633 | Time: 2428.59 ms
[2022-02-13 16:51:28	                main:574]	:	INFO	:	Epoch 63 | loss: 0.0308624 | val_loss: 0.0312571 | Time: 2468.73 ms
[2022-02-13 16:51:31	                main:574]	:	INFO	:	Epoch 64 | loss: 0.0308596 | val_loss: 0.0312774 | Time: 2679.67 ms
[2022-02-13 16:51:34	                main:574]	:	INFO	:	Epoch 65 | loss: 0.0308945 | val_loss: 0.0312201 | Time: 2718.87 ms
[2022-02-13 16:51:36	                main:574]	:	INFO	:	Epoch 66 | loss: 0.0309037 | val_loss: 0.0312615 | Time: 2562.72 ms
[2022-02-13 16:51:39	                main:574]	:	INFO	:	Epoch 67 | loss: 0.0308774 | val_loss: 0.0312707 | Time: 2625.64 ms
[2022-02-13 16:51:42	                main:574]	:	INFO	:	Epoch 68 | loss: 0.0308659 | val_loss: 0.0312536 | Time: 2781.75 ms
[2022-02-13 16:51:44	                main:574]	:	INFO	:	Epoch 69 | loss: 0.0308679 | val_loss: 0.0312525 | Time: 2405.75 ms
[2022-02-13 16:51:47	                main:574]	:	INFO	:	Epoch 70 | loss: 0.0308717 | val_loss: 0.0312595 | Time: 2454.82 ms
[2022-02-13 16:51:49	                main:574]	:	INFO	:	Epoch 71 | loss: 0.0308733 | val_loss: 0.0312599 | Time: 2625.35 ms
[2022-02-13 16:51:52	                main:574]	:	INFO	:	Epoch 72 | loss: 0.0308635 | val_loss: 0.0312612 | Time: 2531.39 ms
[2022-02-13 16:51:55	                main:574]	:	INFO	:	Epoch 73 | loss: 0.0308665 | val_loss: 0.0312569 | Time: 2665.23 ms
[2022-02-13 16:51:57	                main:574]	:	INFO	:	Epoch 74 | loss: 0.0308719 | val_loss: 0.0312427 | Time: 2626.99 ms
[2022-02-13 16:52:00	                main:574]	:	INFO	:	Epoch 75 | loss: 0.0308768 | val_loss: 0.0312781 | Time: 2627.56 ms
[2022-02-13 16:52:03	                main:574]	:	INFO	:	Epoch 76 | loss: 0.0308756 | val_loss: 0.0312462 | Time: 2601.85 ms
[2022-02-13 16:52:05	                main:574]	:	INFO	:	Epoch 77 | loss: 0.0308736 | val_loss: 0.0312665 | Time: 2471.57 ms
[2022-02-13 16:52:08	                main:574]	:	INFO	:	Epoch 78 | loss: 0.0308528 | val_loss: 0.0312878 | Time: 2385.66 ms
[2022-02-13 16:52:10	                main:574]	:	INFO	:	Epoch 79 | loss: 0.0308566 | val_loss: 0.031281 | Time: 2617.68 ms
[2022-02-13 16:52:13	                main:574]	:	INFO	:	Epoch 80 | loss: 0.0308771 | val_loss: 0.0312798 | Time: 2581.93 ms
[2022-02-13 16:52:15	                main:574]	:	INFO	:	Epoch 81 | loss: 0.0308765 | val_loss: 0.0312983 | Time: 2536.59 ms
[2022-02-13 16:52:18	                main:574]	:	INFO	:	Epoch 82 | loss: 0.0308739 | val_loss: 0.0312766 | Time: 2603.86 ms
[2022-02-13 16:52:21	                main:574]	:	INFO	:	Epoch 83 | loss: 0.0308784 | val_loss: 0.0312506 | Time: 2646.98 ms
[2022-02-13 16:52:24	                main:574]	:	INFO	:	Epoch 84 | loss: 0.0308686 | val_loss: 0.031271 | Time: 2928.59 ms
[2022-02-13 16:52:26	                main:574]	:	INFO	:	Epoch 85 | loss: 0.0308745 | val_loss: 0.0312818 | Time: 2443.57 ms
[2022-02-13 16:52:29	                main:574]	:	INFO	:	Epoch 86 | loss: 0.0308666 | val_loss: 0.031278 | Time: 2399.59 ms
[2022-02-13 16:52:31	                main:574]	:	INFO	:	Epoch 87 | loss: 0.0308543 | val_loss: 0.031301 | Time: 2654.36 ms
[2022-02-13 16:52:34	                main:574]	:	INFO	:	Epoch 88 | loss: 0.0308696 | val_loss: 0.0312815 | Time: 2639.11 ms
[2022-02-13 16:52:37	                main:574]	:	INFO	:	Epoch 89 | loss: 0.0308591 | val_loss: 0.03125 | Time: 2709.26 ms
[2022-02-13 16:52:39	                main:574]	:	INFO	:	Epoch 90 | loss: 0.0308652 | val_loss: 0.0312773 | Time: 2507.27 ms
[2022-02-13 16:52:42	                main:574]	:	INFO	:	Epoch 91 | loss: 0.0308587 | val_loss: 0.0312746 | Time: 2632.19 ms
[2022-02-13 16:52:44	                main:574]	:	INFO	:	Epoch 92 | loss: 0.0308741 | val_loss: 0.0312833 | Time: 2502.36 ms
[2022-02-13 16:52:47	                main:574]	:	INFO	:	Epoch 93 | loss: 0.030869 | val_loss: 0.0312615 | Time: 2459.97 ms
[2022-02-13 16:52:50	                main:574]	:	INFO	:	Epoch 94 | loss: 0.0308799 | val_loss: 0.0312364 | Time: 2631.72 ms
[2022-02-13 16:52:52	                main:574]	:	INFO	:	Epoch 95 | loss: 0.0308774 | val_loss: 0.0312572 | Time: 2622.83 ms
[2022-02-13 16:52:55	                main:574]	:	INFO	:	Epoch 96 | loss: 0.0308695 | val_loss: 0.0312214 | Time: 2550.41 ms
[2022-02-13 16:52:58	                main:574]	:	INFO	:	Epoch 97 | loss: 0.0308893 | val_loss: 0.0312124 | Time: 2682.92 ms
[2022-02-13 16:53:00	                main:574]	:	INFO	:	Epoch 98 | loss: 0.0308855 | val_loss: 0.0312883 | Time: 2482.5 ms
[2022-02-13 16:53:03	                main:574]	:	INFO	:	Epoch 99 | loss: 0.03088 | val_loss: 0.031288 | Time: 2546.17 ms
[2022-02-13 16:53:05	                main:574]	:	INFO	:	Epoch 100 | loss: 0.030877 | val_loss: 0.0312782 | Time: 2437.29 ms
[2022-02-13 16:53:08	                main:574]	:	INFO	:	Epoch 101 | loss: 0.0308606 | val_loss: 0.0312771 | Time: 2483.69 ms
[2022-02-13 16:53:10	                main:574]	:	INFO	:	Epoch 102 | loss: 0.0308761 | val_loss: 0.0312309 | Time: 2707.55 ms
[2022-02-13 16:53:13	                main:574]	:	INFO	:	Epoch 103 | loss: 0.0308839 | val_loss: 0.0312815 | Time: 2655.8 ms
[2022-02-13 16:53:16	                main:574]	:	INFO	:	Epoch 104 | loss: 0.0308619 | val_loss: 0.0312866 | Time: 2661.94 ms
[2022-02-13 16:53:19	                main:574]	:	INFO	:	Epoch 105 | loss: 0.0308629 | val_loss: 0.0312949 | Time: 2730.95 ms
[2022-02-13 16:53:21	                main:574]	:	INFO	:	Epoch 106 | loss: 0.0308603 | val_loss: 0.0312776 | Time: 2606.19 ms
[2022-02-13 16:53:24	                main:574]	:	INFO	:	Epoch 107 | loss: 0.0308527 | val_loss: 0.0312856 | Time: 2471.2 ms
[2022-02-13 16:53:26	                main:574]	:	INFO	:	Epoch 108 | loss: 0.0308519 | val_loss: 0.0312808 | Time: 2474.14 ms
[2022-02-13 16:53:29	                main:574]	:	INFO	:	Epoch 109 | loss: 0.0308648 | val_loss: 0.0312863 | Time: 2516.63 ms
[2022-02-13 16:53:32	                main:574]	:	INFO	:	Epoch 110 | loss: 0.030866 | val_loss: 0.0312487 | Time: 2672.12 ms
[2022-02-13 16:53:34	                main:574]	:	INFO	:	Epoch 111 | loss: 0.0308623 | val_loss: 0.031277 | Time: 2753.16 ms
[2022-02-13 16:53:37	                main:574]	:	INFO	:	Epoch 112 | loss: 0.0308569 | val_loss: 0.0312411 | Time: 2667.03 ms
[2022-02-13 16:53:40	                main:574]	:	INFO	:	Epoch 113 | loss: 0.0308613 | val_loss: 0.0312592 | Time: 2725.8 ms
[2022-02-13 16:53:43	                main:574]	:	INFO	:	Epoch 114 | loss: 0.0308512 | val_loss: 0.0312879 | Time: 2503.46 ms
[2022-02-13 16:53:45	                main:574]	:	INFO	:	Epoch 115 | loss: 0.0308518 | val_loss: 0.0312648 | Time: 2447.63 ms
[2022-02-13 16:53:48	                main:574]	:	INFO	:	Epoch 116 | loss: 0.0308454 | val_loss: 0.0313052 | Time: 2548.19 ms
[2022-02-13 16:53:50	                main:574]	:	INFO	:	Epoch 117 | loss: 0.0308767 | val_loss: 0.031273 | Time: 2541.57 ms


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00007FF9F2CEE7F5 read attempt to address 0xFFFFFFFF

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.17.0


Dump Timestamp    : 02/13/22 16:53:51
Install Directory : H:\Program Files\BOINC\
Data Directory    : H:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( H:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( H:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( H:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( H:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: H:\ProgramData\BOINC\slots\122;H:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome


ModLoad: 000000006fa30000 00000000003df000 H:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000342c0000 00000000001f0000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.18362.815) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000032670000 00000000000b2000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 10.0.18362.959 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.959

ModLoad: 0000000031d60000 00000000002a4000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernelbase.pdb
    File Version          : 10.0.18362.959 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.959

ModLoad: 00000000324b0000 0000000000194000 C:\WINDOWS\System32\USER32.dll (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000032070000 0000000000021000 C:\WINDOWS\System32\win32u.dll (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : win32u.pdb
    File Version          : 10.0.18362.959 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.959

ModLoad: 0000000032840000 0000000000026000 C:\WINDOWS\System32\GDI32.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 00000000320a0000 0000000000196000 C:\WINDOWS\System32\gdi32full.dll (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32full.pdb
    File Version          : 10.0.18362.959 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.959

ModLoad: 0000000031b70000 000000000009e000 C:\WINDOWS\System32\msvcp_win.dll (6.2.18362.815) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.18362.815 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.815

ModLoad: 00000000319f0000 00000000000fa000 C:\WINDOWS\System32\ucrtbase.dll (6.2.18362.815) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.18362.815 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.815

ModLoad: 0000000033b20000 00000000000a3000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.18362.752) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000032740000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.18362.1

ModLoad: 0000000033be0000 0000000000097000 C:\WINDOWS\System32\sechost.dll (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000032aa0000 0000000000120000 C:\WINDOWS\System32\RPCRT4.dll (6.2.18362.628) (-exported- Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 00000000f8c40000 0000000000058000 H:\ProgramData\BOINC\slots\122\c10.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000f0f90000 0000000000091000 H:\ProgramData\BOINC\slots\122\MSVCP140.dll (14.27.29112.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
    File Version          : 14.27.29112.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.27.29112.0

ModLoad: 000000001e700000 00000000001f4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000000c470000 0000000000019000 H:\ProgramData\BOINC\slots\122\VCRUNTIME140.dll (14.27.29112.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
    File Version          : 14.27.29112.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.27.29112.0

ModLoad: 000000000b660000 000000000000c000 H:\ProgramData\BOINC\slots\122\VCRUNTIME140_1.dll (14.27.29112.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
    File Version          : 14.27.29112.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.27.29112.0

ModLoad: 00000000d6a20000 000000000abe7000 H:\ProgramData\BOINC\slots\122\torch_cpu.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000e91c0000 00000000002db000 H:\ProgramData\BOINC\slots\122\fbgemm.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000f51b0000 0000000000040000 H:\ProgramData\BOINC\slots\122\asmjit.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000e8fb0000 000000000020d000 H:\ProgramData\BOINC\slots\122\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
    Linked PDB Filename   : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
    File Version          : 20200205
    Company Name          : Intel Corporation
    Product Name          : Intel(R) OpenMP* Runtime Library
    Product Version       : 5.0

ModLoad: 00000000a4bd0000 0000000024b49000 H:\ProgramData\BOINC\slots\122\torch_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000f0f40000 0000000000045000 H:\ProgramData\BOINC\slots\122\c10_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000e4bd0000 00000000034b2000 H:\ProgramData\BOINC\slots\122\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA RAND Library
    Product Version       : 6,14,11,1012

ModLoad: 00000000e1b20000 00000000030a5000 H:\ProgramData\BOINC\slots\122\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1031
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA SPARSE BLAS Library
    Product Version       : 6,14,11,1031

ModLoad: 000000000aed0000 0000000000010000 H:\ProgramData\BOINC\slots\122\nvToolsExt64_1.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb

ModLoad: 00000000a0e70000 0000000003d5b000 H:\ProgramData\BOINC\slots\122\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 000000009efa0000 0000000001eca000 H:\ProgramData\BOINC\slots\122\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 0000000083bf0000 000000001b3a4000 H:\ProgramData\BOINC\slots\122\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,10020
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.86 CUDNN Library
    Product Version       : 6,14,11,10020

ModLoad: 000000007aa50000 000000000919f000 H:\ProgramData\BOINC\slots\122\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA FFT Library
    Product Version       : 6,14,11,1012

ModLoad: 0000000032240000 0000000000080000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.18362.836) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.18362.836 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.836

ModLoad: 0000000033de0000 000000000002e000 C:\WINDOWS\System32\IMM32.DLL (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 0000000031180000 0000000000011000 C:\WINDOWS\System32\kernel.appcore.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000002fe10000 0000000000020000 C:\WINDOWS\SYSTEM32\dxcore.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : DXCore.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000031b20000 000000000004a000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 00000000334b0000 0000000000335000 C:\WINDOWS\System32\combase.dll (6.2.18362.900) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000033e10000 0000000000470000 C:\WINDOWS\System32\Setupapi.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000031af0000 0000000000026000 C:\WINDOWS\System32\bcrypt.dll (6.2.18362.267) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcrypt.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000030f10000 000000000002a000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 0000000032010000 000000000005c000 C:\WINDOWS\System32\WINTRUST.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : wintrust.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 00000000311a0000 0000000000012000 C:\WINDOWS\System32\MSASN1.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msasn1.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000031c10000 0000000000149000 C:\WINDOWS\System32\CRYPT32.dll (6.2.18362.592) (-exported- Symbols Loaded)
    Linked PDB Filename   : crypt32.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000032bc0000 00000000006e6000 C:\WINDOWS\System32\Shell32.dll (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : shell32.pdb
    File Version          : 10.0.18362.900 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.900

ModLoad: 00000000332b0000 00000000000a9000 C:\WINDOWS\System32\shcore.dll (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : shcore.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000031260000 0000000000782000 C:\WINDOWS\System32\windows.storage.dll (6.2.18362.959) (-exported- Symbols Loaded)
    Linked PDB Filename   : Windows.Storage.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 00000000311c0000 0000000000023000 C:\WINDOWS\System32\profapi.dll (6.2.18362.693) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : profapi.pdb
    File Version          : 10.0.18362.693 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.693

ModLoad: 00000000311f0000 000000000004a000 C:\WINDOWS\System32\powrprof.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : powrprof.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000031170000 0000000000010000 C:\WINDOWS\System32\UMPDC.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : UMPDC.pdb

ModLoad: 00000000327e0000 0000000000052000 C:\WINDOWS\System32\shlwapi.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000031240000 0000000000017000 C:\WINDOWS\System32\cryptsp.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptsp.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000017a30000 0000000000267000 C:\WINDOWS\system32\nvcuda.dll (27.21.14.5730) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r457_22\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
    File Version          : 27.21.14.5730
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.1.114 driver
    Product Version       : 27.21.14.5730

ModLoad: 000000002f150000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.DLL (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 00000000f2b80000 000000000152c000 C:\WINDOWS\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_5dcb5bbf5c3edcf2\nvcuda64.dll (27.21.14.5730) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r457_22\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
    File Version          : 27.21.14.5730
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.1.114 driver
    Product Version       : 27.21.14.5730

ModLoad: 00000000338d0000 000000000006f000 C:\WINDOWS\System32\WS2_32.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : ws2_32.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 000000002f120000 000000000002f000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptnet.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000030ab0000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 000000002f4a0000 0000000000099000 C:\WINDOWS\system32\uxtheme.dll (6.2.18362.449) (-exported- Symbols Loaded)
    Linked PDB Filename   : UxTheme.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 00000000295b0000 00000000006d6000 C:\WINDOWS\SYSTEM32\nvapi64.dll (27.21.14.5730) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r455\r457_22\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
    File Version          : 27.21.14.5730
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA Windows drivers
    Product Version       : 27.21.14.5730

ModLoad: 0000000033c80000 0000000000157000 C:\WINDOWS\System32\OLE32.DLL (6.2.18362.693) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 10.0.18362.530 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.530

ModLoad: 0000000006180000 0000000000009000 H:\ProgramData\BOINC\slots\122\caffe2_nvrtc.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000024840000 0000000000fca000 H:\ProgramData\BOINC\slots\122\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6.14.11.9000
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.89 NVRTC Library
    Product Version       : 6.14.11.9000



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 3740, Write: 770, Other 2363

- I/O Transfers Counters -
Read: 7024398, Write: 36109, Other 40910

- Paged Pool Usage -
QuotaPagedPoolUsage: 3535464, QuotaPeakPagedPoolUsage: 3535464
QuotaNonPagedPoolUsage: 59152, QuotaPeakNonPagedPoolUsage: 59624

- Virtual Memory Usage -
VirtualSize: 364781568, PeakVirtualSize: 984969216

- Pagefile Usage -
PagefileUsage: 364781568, PeakPagefileUsage: 364781568

- Working Set Size -
WorkingSetSize: -2043699200, PeakWorkingSetSize: -2041344000, PageFaultCount: 850791

*** Dump of thread ID 23576 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 68437504.000000, User Time: 1236875008.000000, Wait Time: 354734592.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00007FF9F2CEE7F5 read attempt to address 0xFFFFFFFF

- Registers -
rax=0000000000000000 rbx=00000000982a4110 rcx=00000000b53f95b0 rdx=00000000982a4110 rsi=000000006babc1c0 rdi=00000000b53f95b0
r8=0000000000000080 r9=0000000000000001 r10=0000000000000018 r11=0000000000000094 r12=0000000000000001 r13=0000000000000001
r14=00000000982a4110 r15=0000000000000003 rip=00000000f2cee7f5 rsp=00000000b53f9448 rbp=00000000b53f95a0
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246

- Callstack -
ChildEBP RetAddr  Args to Child
b53f9440 f2bb8363 982a4110 b53f95b0 b53f94a8 6babc1c0 nvcuda64!+0x0 
b53f9490 f2bb890c 6babc1d0 6babc1c0 982a4110 6babc1d0 nvcuda64!+0x0 
b53f9720 f2d81108 00008000 84b80730 00000000 84994de7 nvcuda64!+0x0 
b53f9890 849929b6 00000001 00000080 00000000 84b80730 nvcuda64!cuLaunchKernel+0x0 
b53f99a0 8498fdf4 84b80730 6a40ffd0 982a4110 b53f9bc0 cudnn64_7!cudnnSetDropoutDescriptor+0x0 
b53f9b20 84b73951 84b80730 00000000 84b80730 00000000 cudnn64_7!cudnnSetDropoutDescriptor+0x0 
b53f9c90 84afe604 0bc05310 3d148090 39a80000 000000c0 cudnn64_7!cudnnSetDropoutDescriptor+0x0 
b53f9ec0 84b0f99d 00000000 85e01500 00000000 84b80730 cudnn64_7!cudnnSetDropoutDescriptor+0x0 
b53f9f20 84ad703b 00000000 00000000 0bc05310 b53fa030 cudnn64_7!cudnnSetDropoutDescriptor+0x0 
b53fa3b0 849f43bc 00000000 b53fa49c 00000000 ce934020 cudnn64_7!cudnnSetDropoutDescriptor+0x0 
b53fa680 849f700e b53fa990 8499bc3c b53fa6d8 84994ce0 cudnn64_7!cudnnSetDropoutDescriptor+0x0 
b53fa770 848f82ca ce934020 00000000 00000000 0000000c cudnn64_7!cudnnSetDropoutDescriptor+0x0 
b53fa840 848f62b3 00000000 0000000c b53faa80 00000002 cudnn64_7!cudnnSetAttnDescriptor+0x0 
b53fa990 848f9536 00000024 886b36a0 c76d1930 00000000 cudnn64_7!cudnnSetAttnDescriptor+0x0 
b53faaa0 84925f37 39a80000 caeddeb0 00000000 00000000 cudnn64_7!cudnnSetAttnDescriptor+0x0 
b53fb0e0 8491eafa 00000004 caeddeb0 00000004 00000000 cudnn64_7!cudnnRNNForwardTrainingEx+0x0 
b53fb2a0 84939738 caeddeb0 00000400 cef9b800 886b36a0 cudnn64_7!cudnnRNNForwardTrainingEx+0x0 
b53fd4b0 a794af5a b53feaa0 39a80000 b53feaa0 886b36a0 cudnn64_7!cudnnRNNBackwardData+0x0 
b53fda40 a7949c90 00000010 00000000 b53fe3d0 b53fe3b8 torch_cuda!at::native::_cudnn_rnn_backward+0x0 
b53fdc30 a79a2808 b53fde20 b53feb08 b53fdd10 00000004 torch_cuda!at::native::_cudnn_rnn_backward+0x0 
b53fdd40 a79b103d b53fe3d0 b53fe3b0 b53feae8 00000000 torch_cuda!at::native::set_storage_cuda_+0x0 
b53fde50 dc46c201 b53fe3d0 dbfeea63 b9730fa0 b53fe518 torch_cuda!at::native::set_storage_cuda_+0x0 
b53fdfe0 dc4bbfea b53feaa0 00000000 00000007 df50f880 torch_cpu!at::native::mkldnn_sigmoid_+0x0 
b53fe110 dc4a34da e1344668 b53fe350 b53feb08 b53fe1f0 torch_cpu!at::bucketize_out+0x0 
b53fe200 dd79e0a9 b53fe350 b53feaa0 b53fe340 00000000 torch_cpu!at::_cudnn_rnn_backward+0x0 
b53fe4e0 dd7aa94d b97305a0 ddc1794a 00000020 00000004 torch_cpu!torch::autograd::GraphRoot::apply+0x0 
b53fe5f0 dc46c201 b53feaf8 b9731ca0 b53fe780 ddc13e3f torch_cpu!torch::autograd::GraphRoot::apply+0x0 
b53fe780 dc4bbfea b53feaa0 00000000 00000000 00000000 torch_cpu!at::native::mkldnn_sigmoid_+0x0 
b53fe8b0 dc4a34da e1344668 b53fec20 b53feb08 b53fe990 torch_cpu!at::bucketize_out+0x0 
b53fe9a0 dd6a994d b53fec20 9e49a2a0 b53feae0 cf4093e8 torch_cpu!at::_cudnn_rnn_backward+0x0 
b53fee40 dd6956b1 c76a4c80 b53ff148 00000000 f8c43bd6 torch_cpu!torch::autograd::generated::CudnnRnnBackward::apply+0x0 
b53ff050 ddbfd1ea b53ff160 c76a4c80 b53ff160 c76a4c80 torch_cpu!torch::autograd::Node::operator()+0x0 
b53ff230 ddbfdbdd 00000001 00000001 00000001 00000001 torch_cpu!torch::autograd::Engine::add_thread_pool_task+0x0 
b53ff710 ddc02812 00000000 00000000 c76a4c70 ffffffff torch_cpu!torch::autograd::Engine::evaluate_function+0x0 
b53ffa00 ddc02471 e1126d10 b53ffa40 e1126d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0 
b53ffa50 ddbf9744 00080001 ceed6820 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0 
b53ffa80 31a10e82 ceed6820 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0 
b53ffab0 32687bd4 00000000 00000000 00000000 00000000 ucrtbase!_beginthreadex+0x0 
b53ffae0 3432ce51 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
b53ffb60 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 27992 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 206250000.000000, User Time: 1603124992.000000, Wait Time: 354734592.000000

- Registers -
rax=00000000000001ca rbx=0000000000000000 rcx=00000000c70d1e18 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
r8=000000006a40ff40 r9=0000000000000001 r10=0000000000008000 r11=00000000b4ae1ea0 r12=0000000000000004 r13=00000000000001c0
r14=00000000c70d1dc8 r15=0000000000000000 rip=000000003435f9a4 rsp=00000000b4ae5908 rbp=00000000c70d1e18
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
b4ae5900 34320590 b4ae5b20 f2d44249 00000000 6b858fc0 ntdll!ZwWaitForAlertByThreadId+0x0 
b4ae5970 31d858fd c70d1dc0 00000000 00000000 00008000 ntdll!RtlSleepConditionVariableSRW+0x0 
b4ae59b0 f0fcba8d b4ae5b20 00000000 00008000 b4ae55d0 KERNELBASE!SleepConditionVariableSRW+0x0 
b4ae59f0 f0fa2848 00000000 b4ae55d0 f0f42e0d f8c62433 MSVCP140!__crtSetThreadpoolWait+0x0 
b4ae5a20 f0fa2710 b4ae5ed0 f0fa2f17 00000000 dbff0231 MSVCP140!std::_Winerror_message+0x0 
b4ae5a90 ddc0064b c76a4c80 000001c0 c70d1dc0 c70d1dc0 MSVCP140!std::_Winerror_message+0x0 
b4ae5ad0 ddc025b1 69c8f7f0 c76a4c70 c76a4c70 00000000 torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0 
b4ae5dc0 ddbff15d b4ae60a0 b4ae60a0 b4ae60a0 69c8f7f0 torch_cpu!torch::autograd::Engine::thread_main+0x0 
b4ae6030 ddbfeea0 a684a8c0 e1126d10 fffffffe 00000000 torch_cpu!torch::autograd::Engine::execute_with_graph_task+0x0 
b4ae6160 ddbf56f9 c76d16c0 c76d16c0 b4ae6150 b4ae6258 torch_cpu!torch::autograd::Engine::execute+0x0 
b4ae6680 ddbf5086 53af6868 dc5c5dc2 00000000 b4ae6750 torch_cpu!torch::autograd::Engine::make_anomaly_metadata+0x0 
b4ae6720 ddef09be b97329a0 c76d16c8 00000000 b4ae66c0 torch_cpu!torch::autograd::backward+0x0 
b4ae6800 dd921cd0 6b6c12a0 00000000 00000000 00000000 torch_cpu!torch::autograd::VariableType::allCUDATypes+0x0 
b4ae6830 dc5c0a06 00000000 00000000 00010001 00000001 torch_cpu!torch::autograd::GraphRoot::apply+0x0 
b4ae6a60 6fa47166 b4ae6b30 b4ae6b70 b4ae6b30 b4ae70d0 torch_cpu!at::Tensor::backward+0x0 
b4ae6f60 6fa4b950 b4ae7060 b4ae7060 00000000 352778e0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0 
b4affd00 6fc8c6f8 6b606520 00000000 00000000 6b66ce60 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
b4affd40 32687bd4 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
b4affd70 3432ce51 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
b4affdf0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 9204 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 354734592.000000

- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=00000000b4fff7f0 rsi=0000000000000000 rdi=0000000000000064
r8=00000000b4fff7c8 r9=0000000000000000 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=000000003435c6f4 rsp=00000000b4fff7c8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
b4fff7c0 31da6891 b4fff888 00000000 00000000 00000000 ntdll!ZwDelayExecution+0x0 
b4fff860 6fa6a07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0 
b4fff890 32687bd4 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
b4fff8c0 3432ce51 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
b4fff940 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 8972 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 354734592.000000

- Registers -
rax=000000000000005b rbx=000000000000000b rcx=000000000000000b rdx=00000000b50ffb60 rsi=0000000000000000 rdi=000000000000000b
r8=000000006bab7090 r9=000000006b83bf00 r10=0000000000000000 r11=00000000b50ffad0 r12=0000000000000064 r13=00000000b50ffb60
r14=00000000b50ff850 r15=0000000000000000 rip=000000003435cbc4 rsp=00000000b50ff7f8 rbp=000000006b83bf70
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
b50ff7f0 31db8037 33ca3880 343337d4 00000000 b48e7000 ntdll!NtWaitForMultipleObjects+0x0 
b50ffaf0 31db7f1e f3074b48 69c93ae0 0000000b 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0 
b50ffb30 f2d2bb54 6b840ef0 00000000 6b839750 00000000 KERNELBASE!WaitForMultipleObjects+0x0 
b50ffd70 f2dc09ed 00000000 00000004 00000000 00000000 nvcuda64!+0x0 
b50ffe20 f2d2b523 2c4b939a 6c89bb70 6bd15080 00000000 nvcuda64!cuProfilerStop+0x0 
b50ffe50 f303a750 6bd15080 00000000 00000000 00000000 nvcuda64!+0x0 
b50ffe80 32687bd4 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
b50ffeb0 3432ce51 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
b50fff30 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 6992 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 354713984.000000

- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=0000000000000260 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000260
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=000000006b820ad0 r13=0000000000000000
r14=0000000000000260 r15=0000000000000000 rip=000000003435c0f4 rsp=00000000b51ff998 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
b51ff990 31d98b03 00000000 00000000 00000000 00000000 ntdll!ZwWaitForSingleObject+0x0 
b51ffa30 f2f3bf48 00000000 31d84975 00000000 00000260 KERNELBASE!WaitForSingleObjectEx+0x0 
b51ffaa0 f2d2b523 00000001 6c2b29f0 6bd14ea0 00000000 nvcuda64!cuProfilerStop+0x0 
b51ffad0 f303a750 6bd14ea0 00000000 00000000 00000000 nvcuda64!+0x0 
b51ffb00 32687bd4 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
b51ffb30 3432ce51 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
b51ffbb0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 19336 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 354734592.000000

- Registers -
rax=00000000000001ca rbx=0000000000000000 rcx=000000006baba258 rdx=00000000b52ffda8 rsi=00000000b52ffd10 rdi=0000000000000000
r8=0000000000000001 r9=0000000000004b88 r10=000000006babe9d8 r11=000000006a00730f r12=00000000b52ffda8 r13=0000000000000000
r14=000000006baba258 r15=000000006babe9d8 rip=000000003435f9a4 rsp=00000000b52ffce8 rbp=00000000b52ffd40
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
b52ffce0 34320815 00000102 00000102 00000000 b52ffd10 ntdll!ZwWaitForAlertByThreadId+0x0 
b52ffd50 31ddc2b9 6babc1c0 00000000 00000000 00000000 ntdll!RtlSleepConditionVariableCS+0x0 
b52ffd80 f2cb223a 6babc1c0 6babc1c0 6babc1c0 ffb3b4c0 KERNELBASE!SleepConditionVariableCS+0x0 
b52ffde0 f2d2b523 00000004 2c492de4 2c492de4 00000000 nvcuda64!+0x0 
b52ffe10 f303a750 6bd14c00 00000000 00000000 00000000 nvcuda64!+0x0 
b52ffe40 32687bd4 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
b52ffe70 3432ce51 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
b52ffef0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>


©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)