Task 10032607

Name ParityModified-1639965493-6773-0_0
Workunit 7411192
Created 20 Dec 2021, 1:58:16 UTC
Sent 6 Jan 2022, 0:12:16 UTC
Report deadline 14 Jan 2022, 0:12:16 UTC
Received 3 Feb 2022, 21:31:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -529697949 (0xE06D7363) Unknown error code
Computer ID 16731
Run time 1 hours 3 min 51 sec
CPU time 1 hours 2 min 54 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 6,452.20 GFLOPS
Application version Machine Learning Dataset Generator (GPU) v9.75 (cuda10200)
windows_x86_64
Peak working set size 2.10 GB
Peak swap size 4.35 GB
Peak disk usage 1.54 GB

Stderr output

<core_client_version>7.16.20</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 3765269347 (0xe06d7363)</message>
<stderr_txt>
2492 | val_loss: 0.03125 | Time: 2050.69 ms
[2022-01-06 02:04:59	                main:574]	:	INFO	:	Epoch 1621 | loss: 0.0312501 | val_loss: 0.0312499 | Time: 2059.13 ms
[2022-01-06 02:05:02	                main:574]	:	INFO	:	Epoch 1622 | loss: 0.0312499 | val_loss: 0.0312496 | Time: 2069.97 ms
[2022-01-06 02:05:04	                main:574]	:	INFO	:	Epoch 1623 | loss: 0.0312491 | val_loss: 0.0312494 | Time: 2091.21 ms
[2022-01-06 02:05:06	                main:574]	:	INFO	:	Epoch 1624 | loss: 0.0312491 | val_loss: 0.0312489 | Time: 2096.84 ms
[2022-01-06 02:05:08	                main:574]	:	INFO	:	Epoch 1625 | loss: 0.031249 | val_loss: 0.0312488 | Time: 2076.51 ms
[2022-01-06 02:05:10	                main:574]	:	INFO	:	Epoch 1626 | loss: 0.0312491 | val_loss: 0.031249 | Time: 2041.49 ms
[2022-01-06 02:05:12	                main:574]	:	INFO	:	Epoch 1627 | loss: 0.0312491 | val_loss: 0.0312497 | Time: 2077.82 ms
[2022-01-06 02:05:14	                main:574]	:	INFO	:	Epoch 1628 | loss: 0.031249 | val_loss: 0.0312492 | Time: 2100.47 ms
[2022-01-06 02:05:16	                main:574]	:	INFO	:	Epoch 1629 | loss: 0.0312493 | val_loss: 0.0312505 | Time: 2149.06 ms
[2022-01-06 02:05:18	                main:574]	:	INFO	:	Epoch 1630 | loss: 0.0312491 | val_loss: 0.0312488 | Time: 2132.51 ms
[2022-01-06 02:05:21	                main:574]	:	INFO	:	Epoch 1631 | loss: 0.031249 | val_loss: 0.0312489 | Time: 2096.08 ms
[2022-01-06 02:05:23	                main:574]	:	INFO	:	Epoch 1632 | loss: 0.0312495 | val_loss: 0.0312494 | Time: 2067.08 ms
[2022-01-06 02:05:25	                main:574]	:	INFO	:	Epoch 1633 | loss: 0.0312489 | val_loss: 0.0312491 | Time: 2111 ms
[2022-01-06 02:05:27	                main:574]	:	INFO	:	Epoch 1634 | loss: 0.031249 | val_loss: 0.031249 | Time: 2087.69 ms
[2022-01-06 02:05:29	                main:574]	:	INFO	:	Epoch 1635 | loss: 0.0312496 | val_loss: 0.0312497 | Time: 2079.78 ms
[2022-01-06 02:05:31	                main:574]	:	INFO	:	Epoch 1636 | loss: 0.0312496 | val_loss: 0.0312489 | Time: 2116.18 ms
[2022-01-06 02:05:33	                main:574]	:	INFO	:	Epoch 1637 | loss: 0.031249 | val_loss: 0.0312491 | Time: 2077.86 ms
[2022-01-06 02:05:35	                main:574]	:	INFO	:	Epoch 1638 | loss: 0.0312492 | val_loss: 0.031249 | Time: 2111.46 ms
[2022-01-06 02:05:37	                main:574]	:	INFO	:	Epoch 1639 | loss: 0.031249 | val_loss: 0.0312492 | Time: 2080.63 ms
[2022-01-06 02:05:39	                main:574]	:	INFO	:	Epoch 1640 | loss: 0.03125 | val_loss: 0.0312493 | Time: 2089.44 ms
[2022-01-06 02:05:42	                main:574]	:	INFO	:	Epoch 1641 | loss: 0.0312492 | val_loss: 0.0312493 | Time: 2126.48 ms
[2022-01-06 02:05:44	                main:574]	:	INFO	:	Epoch 1642 | loss: 0.0312496 | val_loss: 0.0312489 | Time: 2093.42 ms
[2022-01-06 02:05:46	                main:574]	:	INFO	:	Epoch 1643 | loss: 0.0312494 | val_loss: 0.031249 | Time: 2115.91 ms
[2022-01-06 02:05:48	                main:574]	:	INFO	:	Epoch 1644 | loss: 0.0312491 | val_loss: 0.0312493 | Time: 2078.11 ms
[2022-01-06 02:05:50	                main:574]	:	INFO	:	Epoch 1645 | loss: 0.0312494 | val_loss: 0.031249 | Time: 2087.9 ms
[2022-01-06 02:05:52	                main:574]	:	INFO	:	Epoch 1646 | loss: 0.0312494 | val_loss: 0.0312489 | Time: 2075.72 ms
[2022-01-06 02:05:54	                main:574]	:	INFO	:	Epoch 1647 | loss: 0.0312501 | val_loss: 0.0312495 | Time: 2109.57 ms
[2022-01-06 02:05:56	                main:574]	:	INFO	:	Epoch 1648 | loss: 0.0312501 | val_loss: 0.0312504 | Time: 2093.49 ms
[2022-01-06 02:05:58	                main:574]	:	INFO	:	Epoch 1649 | loss: 0.0312497 | val_loss: 0.0312496 | Time: 2089.65 ms
[2022-01-06 02:06:01	                main:574]	:	INFO	:	Epoch 1650 | loss: 0.0312492 | val_loss: 0.0312493 | Time: 2093.8 ms
[2022-01-06 02:06:03	                main:574]	:	INFO	:	Epoch 1651 | loss: 0.0312498 | val_loss: 0.0312498 | Time: 2113.27 ms
[2022-01-06 02:06:05	                main:574]	:	INFO	:	Epoch 1652 | loss: 0.0312496 | val_loss: 0.031249 | Time: 2127.33 ms
[2022-01-06 02:06:07	                main:574]	:	INFO	:	Epoch 1653 | loss: 0.0312493 | val_loss: 0.0312491 | Time: 2072.78 ms
[2022-01-06 02:06:09	                main:574]	:	INFO	:	Epoch 1654 | loss: 0.0312493 | val_loss: 0.0312489 | Time: 2094.74 ms
[2022-01-06 02:06:11	                main:574]	:	INFO	:	Epoch 1655 | loss: 0.03125 | val_loss: 0.0312493 | Time: 2083.93 ms
[2022-01-06 02:06:13	                main:574]	:	INFO	:	Epoch 1656 | loss: 0.0312503 | val_loss: 0.0312492 | Time: 2103.04 ms
[2022-01-06 02:06:15	                main:574]	:	INFO	:	Epoch 1657 | loss: 0.0312499 | val_loss: 0.0312495 | Time: 2100.91 ms
[2022-01-06 02:06:18	                main:574]	:	INFO	:	Epoch 1658 | loss: 0.031251 | val_loss: 0.0312516 | Time: 2098.51 ms
[2022-01-06 02:06:20	                main:574]	:	INFO	:	Epoch 1659 | loss: 0.0312511 | val_loss: 0.031251 | Time: 2084.98 ms
[2022-01-06 02:06:22	                main:574]	:	INFO	:	Epoch 1660 | loss: 0.0312516 | val_loss: 0.0312534 | Time: 2085.31 ms
[2022-01-06 02:06:24	                main:574]	:	INFO	:	Epoch 1661 | loss: 0.0312511 | val_loss: 0.0312495 | Time: 2101.74 ms
[2022-01-06 02:06:26	                main:574]	:	INFO	:	Epoch 1662 | loss: 0.0312507 | val_loss: 0.0312529 | Time: 2113.93 ms
[2022-01-06 02:06:28	                main:574]	:	INFO	:	Epoch 1663 | loss: 0.031251 | val_loss: 0.0312499 | Time: 2142.75 ms
[2022-01-06 02:06:30	                main:574]	:	INFO	:	Epoch 1664 | loss: 0.0312495 | val_loss: 0.0312491 | Time: 2090.04 ms
[2022-01-06 02:06:32	                main:574]	:	INFO	:	Epoch 1665 | loss: 0.0312505 | val_loss: 0.0312501 | Time: 2078.25 ms
[2022-01-06 02:06:34	                main:574]	:	INFO	:	Epoch 1666 | loss: 0.0312503 | val_loss: 0.0312498 | Time: 2096.4 ms
[2022-01-06 02:06:37	                main:574]	:	INFO	:	Epoch 1667 | loss: 0.0312501 | val_loss: 0.0312493 | Time: 2115.86 ms
[2022-01-06 02:06:39	                main:574]	:	INFO	:	Epoch 1668 | loss: 0.0312492 | val_loss: 0.0312493 | Time: 2119.07 ms
[2022-01-06 02:06:41	                main:574]	:	INFO	:	Epoch 1669 | loss: 0.0312492 | val_loss: 0.0312489 | Time: 2108.95 ms
[2022-01-06 02:06:43	                main:574]	:	INFO	:	Epoch 1670 | loss: 0.0312493 | val_loss: 0.0312494 | Time: 2097.62 ms
[2022-01-06 02:06:45	                main:574]	:	INFO	:	Epoch 1671 | loss: 0.0312497 | val_loss: 0.0312492 | Time: 2126.17 ms
[2022-01-06 02:06:47	                main:574]	:	INFO	:	Epoch 1672 | loss: 0.0312491 | val_loss: 0.0312495 | Time: 2106.45 ms
[2022-01-06 02:06:49	                main:574]	:	INFO	:	Epoch 1673 | loss: 0.031249 | val_loss: 0.0312496 | Time: 2102.15 ms
[2022-01-06 02:06:51	                main:574]	:	INFO	:	Epoch 1674 | loss: 0.0312497 | val_loss: 0.0312505 | Time: 2094.27 ms
[2022-01-06 02:06:53	                main:574]	:	INFO	:	Epoch 1675 | loss: 0.0312507 | val_loss: 0.0312507 | Time: 2092.78 ms
[2022-01-06 02:06:56	                main:574]	:	INFO	:	Epoch 1676 | loss: 0.0312504 | val_loss: 0.0312505 | Time: 2083.23 ms
[2022-01-06 02:06:58	                main:574]	:	INFO	:	Epoch 1677 | loss: 0.0312492 | val_loss: 0.031249 | Time: 2081.88 ms
[2022-01-06 02:07:00	                main:574]	:	INFO	:	Epoch 1678 | loss: 0.0312504 | val_loss: 0.0312504 | Time: 2135.18 ms
[2022-01-06 02:07:02	                main:574]	:	INFO	:	Epoch 1679 | loss: 0.0312491 | val_loss: 0.0312491 | Time: 2119.01 ms
[2022-01-06 02:07:04	                main:574]	:	INFO	:	Epoch 1680 | loss: 0.0312492 | val_loss: 0.031249 | Time: 2143.29 ms
[2022-01-06 02:07:06	                main:574]	:	INFO	:	Epoch 1681 | loss: 0.0312491 | val_loss: 0.031249 | Time: 2107.07 ms
[2022-01-06 02:07:08	                main:574]	:	INFO	:	Epoch 1682 | loss: 0.0312492 | val_loss: 0.0312498 | Time: 2137.81 ms
[2022-01-06 02:07:11	                main:574]	:	INFO	:	Epoch 1683 | loss: 0.0312489 | val_loss: 0.031249 | Time: 2146.92 ms
[2022-01-06 02:07:13	                main:574]	:	INFO	:	Epoch 1684 | loss: 0.0312503 | val_loss: 0.0312491 | Time: 2123.96 ms
[2022-01-06 02:07:15	                main:574]	:	INFO	:	Epoch 1685 | loss: 0.0312499 | val_loss: 0.0312493 | Time: 2116.56 ms
[2022-01-06 02:07:17	                main:574]	:	INFO	:	Epoch 1686 | loss: 0.0312491 | val_loss: 0.031249 | Time: 2161.56 ms
[2022-01-06 02:07:19	                main:574]	:	INFO	:	Epoch 1687 | loss: 0.0312488 | val_loss: 0.0312514 | Time: 2170.09 ms
[2022-01-06 02:07:21	                main:574]	:	INFO	:	Epoch 1688 | loss: 0.03125 | val_loss: 0.0312491 | Time: 2131.65 ms
[2022-01-06 02:07:23	                main:574]	:	INFO	:	Epoch 1689 | loss: 0.0312493 | val_loss: 0.0312509 | Time: 2145.29 ms
[2022-01-06 02:07:26	                main:574]	:	INFO	:	Epoch 1690 | loss: 0.0312513 | val_loss: 0.0312525 | Time: 2113.74 ms
[2022-01-06 02:07:28	                main:574]	:	INFO	:	Epoch 1691 | loss: 0.0312502 | val_loss: 0.0312507 | Time: 2166.24 ms
[2022-01-06 02:07:30	                main:574]	:	INFO	:	Epoch 1692 | loss: 0.0312503 | val_loss: 0.0312527 | Time: 2163.77 ms
[2022-01-06 02:07:32	                main:574]	:	INFO	:	Epoch 1693 | loss: 0.0312497 | val_loss: 0.0312511 | Time: 2131.71 ms
[2022-01-06 02:07:34	                main:574]	:	INFO	:	Epoch 1694 | loss: 0.031251 | val_loss: 0.0312507 | Time: 2139.59 ms
[2022-01-06 02:07:36	                main:574]	:	INFO	:	Epoch 1695 | loss: 0.0312507 | val_loss: 0.0312503 | Time: 2157.68 ms
[2022-01-06 02:07:39	                main:574]	:	INFO	:	Epoch 1696 | loss: 0.0312501 | val_loss: 0.0312491 | Time: 2130.88 ms
[2022-01-06 02:07:41	                main:574]	:	INFO	:	Epoch 1697 | loss: 0.0312495 | val_loss: 0.0312509 | Time: 2112.39 ms
[2022-01-06 02:07:43	                main:574]	:	INFO	:	Epoch 1698 | loss: 0.0312497 | val_loss: 0.0312493 | Time: 2118.46 ms
[2022-01-06 02:07:45	                main:574]	:	INFO	:	Epoch 1699 | loss: 0.0312492 | val_loss: 0.0312506 | Time: 2094.1 ms
[2022-01-06 02:07:47	                main:574]	:	INFO	:	Epoch 1700 | loss: 0.0312493 | val_loss: 0.0312505 | Time: 2146.6 ms
[2022-01-06 02:07:49	                main:574]	:	INFO	:	Epoch 1701 | loss: 0.0312495 | val_loss: 0.0312506 | Time: 2106.82 ms
[2022-01-06 02:07:51	                main:574]	:	INFO	:	Epoch 1702 | loss: 0.0312493 | val_loss: 0.0312491 | Time: 2136.92 ms
[2022-01-06 02:07:54	                main:574]	:	INFO	:	Epoch 1703 | loss: 0.0312493 | val_loss: 0.0312491 | Time: 2123.9 ms
[2022-01-06 02:07:56	                main:574]	:	INFO	:	Epoch 1704 | loss: 0.0312495 | val_loss: 0.03125 | Time: 2098.73 ms
[2022-01-06 02:07:58	                main:574]	:	INFO	:	Epoch 1705 | loss: 0.0312493 | val_loss: 0.0312491 | Time: 2105.56 ms
[2022-01-06 02:08:00	                main:574]	:	INFO	:	Epoch 1706 | loss: 0.0312496 | val_loss: 0.0312492 | Time: 2114.02 ms
[2022-01-06 02:08:02	                main:574]	:	INFO	:	Epoch 1707 | loss: 0.0312494 | val_loss: 0.0312498 | Time: 2121.18 ms
[2022-01-06 02:08:04	                main:574]	:	INFO	:	Epoch 1708 | loss: 0.0312508 | val_loss: 0.0312505 | Time: 2085.28 ms
[2022-01-06 02:08:06	                main:574]	:	INFO	:	Epoch 1709 | loss: 0.03125 | val_loss: 0.0312506 | Time: 2116.29 ms
[2022-01-06 02:08:08	                main:574]	:	INFO	:	Epoch 1710 | loss: 0.0312498 | val_loss: 0.0312497 | Time: 2126.99 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce RTX 2060)
[2022-02-03 13:16:15	                main:435]	:	INFO	:	Set logging level to 1
[2022-02-03 13:16:15	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-02-03 13:16:15	                main:444]	:	INFO	:	Resolving all filenames
[2022-02-03 13:16:15	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-03 13:16:15	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-02-03 13:16:15	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-03 13:16:15	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 0)
[2022-02-03 13:16:15	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-02-03 13:16:15	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-02-03 13:16:15	                main:474]	:	INFO	:	Configuration: 
[2022-02-03 13:16:15	                main:475]	:	INFO	:	    Model type: GRU
[2022-02-03 13:16:15	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-02-03 13:16:15	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-02-03 13:16:15	                main:478]	:	INFO	:	    Batch Size: 128
[2022-02-03 13:16:15	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-02-03 13:16:15	                main:480]	:	INFO	:	    Patience: 10
[2022-02-03 13:16:15	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-02-03 13:16:15	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-02-03 13:16:15	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-02-03 13:16:15	                main:484]	:	INFO	:	    # Threads: 1
[2022-02-03 13:16:15	                main:486]	:	INFO	:	Preparing Dataset
[2022-02-03 13:16:15	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-03 13:16:16	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-03 13:16:19	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-02-03 13:16:19	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-03 13:16:20	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-03 13:16:20	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-02-03 13:16:20	                main:494]	:	INFO	:	Creating Model
[2022-02-03 13:16:20	                main:507]	:	INFO	:	Preparing config file
[2022-02-03 13:16:20	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-02-03 13:16:20	                main:512]	:	INFO	:	Loading config
[2022-02-03 13:16:20	                main:514]	:	INFO	:	Loading state
[2022-02-03 13:16:23	                main:559]	:	INFO	:	Loading DataLoader into Memory
[2022-02-03 13:16:23	                main:562]	:	INFO	:	Starting Training
[2022-02-03 13:16:26	                main:574]	:	INFO	:	Epoch 1711 | loss: 0.0312642 | val_loss: 0.031252 | Time: 3162.27 ms
[2022-02-03 13:16:28	                main:574]	:	INFO	:	Epoch 1712 | loss: 0.0312501 | val_loss: 0.0312495 | Time: 2181.91 ms
[2022-02-03 13:16:30	                main:574]	:	INFO	:	Epoch 1713 | loss: 0.0312504 | val_loss: 0.0312493 | Time: 2129.81 ms
[2022-02-03 13:16:33	                main:574]	:	INFO	:	Epoch 1714 | loss: nan | val_loss: nan | Time: 2158.89 ms
[2022-02-03 13:16:35	                main:574]	:	INFO	:	Epoch 1715 | loss: nan | val_loss: nan | Time: 2134.26 ms
[2022-02-03 13:16:37	                main:574]	:	INFO	:	Epoch 1716 | loss: nan | val_loss: nan | Time: 2152.23 ms
[2022-02-03 13:16:39	                main:574]	:	INFO	:	Epoch 1717 | loss: nan | val_loss: nan | Time: 2126.06 ms
[2022-02-03 13:16:41	                main:574]	:	INFO	:	Epoch 1718 | loss: nan | val_loss: nan | Time: 2205.92 ms
[2022-02-03 13:16:43	                main:574]	:	INFO	:	Epoch 1719 | loss: nan | val_loss: nan | Time: 2170.62 ms
[2022-02-03 13:16:46	                main:574]	:	INFO	:	Epoch 1720 | loss: nan | val_loss: nan | Time: 2144.92 ms
[2022-02-03 13:16:48	                main:574]	:	INFO	:	Epoch 1721 | loss: nan | val_loss: nan | Time: 2135.82 ms
[2022-02-03 13:16:50	                main:574]	:	INFO	:	Epoch 1722 | loss: nan | val_loss: nan | Time: 2156.2 ms
[2022-02-03 13:16:52	                main:574]	:	INFO	:	Epoch 1723 | loss: nan | val_loss: nan | Time: 2180.94 ms
[2022-02-03 13:16:54	                main:574]	:	INFO	:	Epoch 1724 | loss: nan | val_loss: nan | Time: 2142.07 ms
[2022-02-03 13:16:56	                main:574]	:	INFO	:	Epoch 1725 | loss: nan | val_loss: nan | Time: 2140.85 ms
[2022-02-03 13:16:59	                main:574]	:	INFO	:	Epoch 1726 | loss: nan | val_loss: nan | Time: 2181.08 ms
[2022-02-03 13:17:01	                main:574]	:	INFO	:	Epoch 1727 | loss: nan | val_loss: nan | Time: 2142.03 ms
[2022-02-03 13:17:03	                main:574]	:	INFO	:	Epoch 1728 | loss: nan | val_loss: nan | Time: 2152.67 ms
[2022-02-03 13:17:05	                main:574]	:	INFO	:	Epoch 1729 | loss: nan | val_loss: nan | Time: 2124.74 ms
[2022-02-03 13:17:07	                main:574]	:	INFO	:	Epoch 1730 | loss: nan | val_loss: nan | Time: 2240.68 ms
[2022-02-03 13:17:10	                main:574]	:	INFO	:	Epoch 1731 | loss: nan | val_loss: nan | Time: 2146.27 ms
[2022-02-03 13:17:12	                main:574]	:	INFO	:	Epoch 1732 | loss: nan | val_loss: nan | Time: 2189.56 ms
[2022-02-03 13:17:14	                main:574]	:	INFO	:	Epoch 1733 | loss: nan | val_loss: nan | Time: 2187.01 ms
[2022-02-03 13:17:16	                main:574]	:	INFO	:	Epoch 1734 | loss: nan | val_loss: nan | Time: 2160.18 ms
[2022-02-03 13:17:18	                main:574]	:	INFO	:	Epoch 1735 | loss: nan | val_loss: nan | Time: 2121.6 ms
[2022-02-03 13:17:20	                main:574]	:	INFO	:	Epoch 1736 | loss: nan | val_loss: nan | Time: 2150.65 ms
[2022-02-03 13:17:23	                main:574]	:	INFO	:	Epoch 1737 | loss: nan | val_loss: nan | Time: 2186.26 ms
[2022-02-03 13:17:25	                main:574]	:	INFO	:	Epoch 1738 | loss: nan | val_loss: nan | Time: 2222.42 ms
[2022-02-03 13:17:27	                main:574]	:	INFO	:	Epoch 1739 | loss: nan | val_loss: nan | Time: 2197.72 ms
[2022-02-03 13:17:29	                main:574]	:	INFO	:	Epoch 1740 | loss: nan | val_loss: nan | Time: 2153.17 ms
[2022-02-03 13:17:31	                main:574]	:	INFO	:	Epoch 1741 | loss: nan | val_loss: nan | Time: 2176.61 ms
[2022-02-03 13:17:34	                main:574]	:	INFO	:	Epoch 1742 | loss: nan | val_loss: nan | Time: 2138.98 ms
[2022-02-03 13:17:36	                main:574]	:	INFO	:	Epoch 1743 | loss: nan | val_loss: nan | Time: 2135.73 ms
[2022-02-03 13:17:38	                main:574]	:	INFO	:	Epoch 1744 | loss: nan | val_loss: nan | Time: 2139.83 ms
[2022-02-03 13:17:40	                main:574]	:	INFO	:	Epoch 1745 | loss: nan | val_loss: nan | Time: 2123.23 ms
[2022-02-03 13:17:42	                main:574]	:	INFO	:	Epoch 1746 | loss: nan | val_loss: nan | Time: 2203.84 ms
[2022-02-03 13:17:44	                main:574]	:	INFO	:	Epoch 1747 | loss: nan | val_loss: nan | Time: 2149.51 ms
[2022-02-03 13:17:47	                main:574]	:	INFO	:	Epoch 1748 | loss: nan | val_loss: nan | Time: 2176.16 ms
[2022-02-03 13:17:49	                main:574]	:	INFO	:	Epoch 1749 | loss: nan | val_loss: nan | Time: 2159.98 ms
[2022-02-03 13:17:51	                main:574]	:	INFO	:	Epoch 1750 | loss: nan | val_loss: nan | Time: 2129.3 ms
[2022-02-03 13:17:53	                main:574]	:	INFO	:	Epoch 1751 | loss: nan | val_loss: nan | Time: 2128.76 ms
[2022-02-03 13:17:55	                main:574]	:	INFO	:	Epoch 1752 | loss: nan | val_loss: nan | Time: 2123.04 ms
[2022-02-03 13:17:57	                main:574]	:	INFO	:	Epoch 1753 | loss: nan | val_loss: nan | Time: 2159.71 ms
[2022-02-03 13:18:00	                main:574]	:	INFO	:	Epoch 1754 | loss: nan | val_loss: nan | Time: 2155.53 ms
[2022-02-03 13:18:02	                main:574]	:	INFO	:	Epoch 1755 | loss: nan | val_loss: nan | Time: 2155.52 ms
[2022-02-03 13:18:04	                main:574]	:	INFO	:	Epoch 1756 | loss: nan | val_loss: nan | Time: 2208.75 ms
[2022-02-03 13:18:06	                main:574]	:	INFO	:	Epoch 1757 | loss: nan | val_loss: nan | Time: 2176.7 ms
[2022-02-03 13:18:08	                main:574]	:	INFO	:	Epoch 1758 | loss: nan | val_loss: nan | Time: 2152.73 ms
[2022-02-03 13:18:11	                main:574]	:	INFO	:	Epoch 1759 | loss: nan | val_loss: nan | Time: 2145.47 ms
[2022-02-03 13:18:13	                main:574]	:	INFO	:	Epoch 1760 | loss: nan | val_loss: nan | Time: 2150.17 ms
[2022-02-03 13:18:15	                main:574]	:	INFO	:	Epoch 1761 | loss: nan | val_loss: nan | Time: 2170.27 ms
[2022-02-03 13:18:17	                main:574]	:	INFO	:	Epoch 1762 | loss: nan | val_loss: nan | Time: 2151.36 ms
[2022-02-03 13:18:19	                main:574]	:	INFO	:	Epoch 1763 | loss: nan | val_loss: nan | Time: 2145.9 ms
[2022-02-03 13:18:21	                main:574]	:	INFO	:	Epoch 1764 | loss: nan | val_loss: nan | Time: 2146.92 ms
[2022-02-03 13:18:24	                main:574]	:	INFO	:	Epoch 1765 | loss: nan | val_loss: nan | Time: 2160.61 ms
[2022-02-03 13:18:26	                main:574]	:	INFO	:	Epoch 1766 | loss: nan | val_loss: nan | Time: 2158.96 ms
[2022-02-03 13:18:28	                main:574]	:	INFO	:	Epoch 1767 | loss: nan | val_loss: nan | Time: 2188.55 ms
[2022-02-03 13:18:30	                main:574]	:	INFO	:	Epoch 1768 | loss: nan | val_loss: nan | Time: 2137.35 ms
[2022-02-03 13:18:32	                main:574]	:	INFO	:	Epoch 1769 | loss: nan | val_loss: nan | Time: 2132.78 ms
[2022-02-03 13:18:35	                main:574]	:	INFO	:	Epoch 1770 | loss: nan | val_loss: nan | Time: 2181.3 ms
[2022-02-03 13:18:37	                main:574]	:	INFO	:	Epoch 1771 | loss: nan | val_loss: nan | Time: 2162.01 ms
[2022-02-03 13:18:39	                main:574]	:	INFO	:	Epoch 1772 | loss: nan | val_loss: nan | Time: 2166.37 ms
[2022-02-03 13:18:41	                main:574]	:	INFO	:	Epoch 1773 | loss: nan | val_loss: nan | Time: 2162.42 ms
[2022-02-03 13:18:43	                main:574]	:	INFO	:	Epoch 1774 | loss: nan | val_loss: nan | Time: 2145.34 ms
[2022-02-03 13:18:45	                main:574]	:	INFO	:	Epoch 1775 | loss: nan | val_loss: nan | Time: 2188.65 ms
[2022-02-03 13:18:48	                main:574]	:	INFO	:	Epoch 1776 | loss: nan | val_loss: nan | Time: 2155.45 ms
[2022-02-03 13:18:50	                main:574]	:	INFO	:	Epoch 1777 | loss: nan | val_loss: nan | Time: 2153.88 ms
[2022-02-03 13:18:52	                main:574]	:	INFO	:	Epoch 1778 | loss: nan | val_loss: nan | Time: 2211.39 ms
[2022-02-03 13:18:54	                main:574]	:	INFO	:	Epoch 1779 | loss: nan | val_loss: nan | Time: 2136.86 ms
[2022-02-03 13:18:56	                main:574]	:	INFO	:	Epoch 1780 | loss: nan | val_loss: nan | Time: 2174.8 ms
[2022-02-03 13:18:59	                main:574]	:	INFO	:	Epoch 1781 | loss: nan | val_loss: nan | Time: 2185.15 ms
[2022-02-03 13:19:01	                main:574]	:	INFO	:	Epoch 1782 | loss: nan | val_loss: nan | Time: 2174.19 ms
[2022-02-03 13:19:03	                main:574]	:	INFO	:	Epoch 1783 | loss: nan | val_loss: nan | Time: 2157.66 ms
[2022-02-03 13:19:05	                main:574]	:	INFO	:	Epoch 1784 | loss: nan | val_loss: nan | Time: 2113.22 ms
[2022-02-03 13:19:07	                main:574]	:	INFO	:	Epoch 1785 | loss: nan | val_loss: nan | Time: 2143.61 ms
[2022-02-03 13:19:09	                main:574]	:	INFO	:	Epoch 1786 | loss: nan | val_loss: nan | Time: 2184.97 ms
[2022-02-03 13:19:12	                main:574]	:	INFO	:	Epoch 1787 | loss: nan | val_loss: nan | Time: 2149.18 ms
[2022-02-03 13:19:14	                main:574]	:	INFO	:	Epoch 1788 | loss: nan | val_loss: nan | Time: 2153.72 ms
[2022-02-03 13:19:16	                main:574]	:	INFO	:	Epoch 1789 | loss: nan | val_loss: nan | Time: 2191.09 ms
[2022-02-03 13:19:18	                main:574]	:	INFO	:	Epoch 1790 | loss: nan | val_loss: nan | Time: 2132.93 ms
[2022-02-03 13:19:20	                main:574]	:	INFO	:	Epoch 1791 | loss: nan | val_loss: nan | Time: 2131.91 ms
[2022-02-03 13:19:22	                main:574]	:	INFO	:	Epoch 1792 | loss: nan | val_loss: nan | Time: 2137.33 ms
[2022-02-03 13:19:25	                main:574]	:	INFO	:	Epoch 1793 | loss: nan | val_loss: nan | Time: 2125.14 ms
[2022-02-03 13:19:27	                main:574]	:	INFO	:	Epoch 1794 | loss: nan | val_loss: nan | Time: 2199.21 ms
[2022-02-03 13:19:29	                main:574]	:	INFO	:	Epoch 1795 | loss: nan | val_loss: nan | Time: 2158.17 ms
[2022-02-03 13:19:31	                main:574]	:	INFO	:	Epoch 1796 | loss: nan | val_loss: nan | Time: 2148.85 ms
[2022-02-03 13:19:33	                main:574]	:	INFO	:	Epoch 1797 | loss: nan | val_loss: nan | Time: 2136.14 ms
[2022-02-03 13:19:35	                main:574]	:	INFO	:	Epoch 1798 | loss: nan | val_loss: nan | Time: 2171.84 ms
[2022-02-03 13:19:38	                main:574]	:	INFO	:	Epoch 1799 | loss: nan | val_loss: nan | Time: 2153.27 ms
[2022-02-03 13:19:40	                main:574]	:	INFO	:	Epoch 1800 | loss: nan | val_loss: nan | Time: 2130.01 ms
[2022-02-03 13:19:42	                main:574]	:	INFO	:	Epoch 1801 | loss: nan | val_loss: nan | Time: 2159.13 ms
[2022-02-03 13:19:44	                main:574]	:	INFO	:	Epoch 1802 | loss: nan | val_loss: nan | Time: 2172.68 ms
[2022-02-03 13:19:46	                main:574]	:	INFO	:	Epoch 1803 | loss: nan | val_loss: nan | Time: 2146.96 ms
[2022-02-03 13:19:48	                main:574]	:	INFO	:	Epoch 1804 | loss: nan | val_loss: nan | Time: 2223.6 ms
[2022-02-03 13:19:51	                main:574]	:	INFO	:	Epoch 1805 | loss: nan | val_loss: nan | Time: 2145.73 ms
[2022-02-03 13:19:53	                main:574]	:	INFO	:	Epoch 1806 | loss: nan | val_loss: nan | Time: 2146.53 ms
[2022-02-03 13:19:55	                main:574]	:	INFO	:	Epoch 1807 | loss: nan | val_loss: nan | Time: 2177.39 ms
[2022-02-03 13:19:57	                main:574]	:	INFO	:	Epoch 1808 | loss: nan | val_loss: nan | Time: 2133.88 ms
[2022-02-03 13:19:59	                main:574]	:	INFO	:	Epoch 1809 | loss: nan | val_loss: nan | Time: 2160.24 ms
[2022-02-03 13:20:01	                main:574]	:	INFO	:	Epoch 1810 | loss: nan | val_loss: nan | Time: 2175.22 ms
[2022-02-03 13:20:04	                main:574]	:	INFO	:	Epoch 1811 | loss: nan | val_loss: nan | Time: 2151.92 ms
[2022-02-03 13:20:06	                main:574]	:	INFO	:	Epoch 1812 | loss: nan | val_loss: nan | Time: 2164.61 ms
[2022-02-03 13:20:08	                main:574]	:	INFO	:	Epoch 1813 | loss: nan | val_loss: nan | Time: 2144.64 ms
[2022-02-03 13:20:10	                main:574]	:	INFO	:	Epoch 1814 | loss: nan | val_loss: nan | Time: 2231.6 ms
[2022-02-03 13:20:12	                main:574]	:	INFO	:	Epoch 1815 | loss: nan | val_loss: nan | Time: 2138.92 ms
[2022-02-03 13:20:14	                main:574]	:	INFO	:	Epoch 1816 | loss: nan | val_loss: nan | Time: 2123.98 ms
[2022-02-03 13:20:17	                main:574]	:	INFO	:	Epoch 1817 | loss: nan | val_loss: nan | Time: 2126.75 ms
[2022-02-03 13:20:19	                main:574]	:	INFO	:	Epoch 1818 | loss: nan | val_loss: nan | Time: 2140.49 ms
[2022-02-03 13:20:21	                main:574]	:	INFO	:	Epoch 1819 | loss: nan | val_loss: nan | Time: 2138.86 ms
[2022-02-03 13:20:23	                main:574]	:	INFO	:	Epoch 1820 | loss: nan | val_loss: nan | Time: 2147.1 ms
[2022-02-03 13:20:25	                main:574]	:	INFO	:	Epoch 1821 | loss: nan | val_loss: nan | Time: 2135.09 ms
[2022-02-03 13:20:27	                main:574]	:	INFO	:	Epoch 1822 | loss: nan | val_loss: nan | Time: 2132.57 ms
[2022-02-03 13:20:30	                main:574]	:	INFO	:	Epoch 1823 | loss: nan | val_loss: nan | Time: 2145.69 ms
[2022-02-03 13:20:32	                main:574]	:	INFO	:	Epoch 1824 | loss: nan | val_loss: nan | Time: 2137.76 ms
[2022-02-03 13:20:34	                main:574]	:	INFO	:	Epoch 1825 | loss: nan | val_loss: nan | Time: 2139.17 ms
[2022-02-03 13:20:36	                main:574]	:	INFO	:	Epoch 1826 | loss: nan | val_loss: nan | Time: 2137.16 ms
[2022-02-03 13:20:38	                main:574]	:	INFO	:	Epoch 1827 | loss: nan | val_loss: nan | Time: 2151.93 ms
[2022-02-03 13:20:40	                main:574]	:	INFO	:	Epoch 1828 | loss: nan | val_loss: nan | Time: 2166.84 ms
[2022-02-03 13:20:43	                main:574]	:	INFO	:	Epoch 1829 | loss: nan | val_loss: nan | Time: 2191.09 ms
[2022-02-03 13:20:45	                main:574]	:	INFO	:	Epoch 1830 | loss: nan | val_loss: nan | Time: 2148.22 ms
[2022-02-03 13:20:47	                main:574]	:	INFO	:	Epoch 1831 | loss: nan | val_loss: nan | Time: 2178.65 ms
[2022-02-03 13:20:49	                main:574]	:	INFO	:	Epoch 1832 | loss: nan | val_loss: nan | Time: 2190.15 ms
[2022-02-03 13:20:51	                main:574]	:	INFO	:	Epoch 1833 | loss: nan | val_loss: nan | Time: 2124.53 ms
Machine Learning Dataset Generator v9.75 (Windows/x64) (libTorch: release/1.6 GPU: GeForce RTX 2060)
[2022-02-03 13:29:03	                main:435]	:	INFO	:	Set logging level to 1
[2022-02-03 13:29:03	                main:441]	:	INFO	:	Running in BOINC Client mode
[2022-02-03 13:29:03	                main:444]	:	INFO	:	Resolving all filenames
[2022-02-03 13:29:03	                main:452]	:	INFO	:	Resolved: dataset.hdf5 => dataset.hdf5 (exists = 1)
[2022-02-03 13:29:03	                main:452]	:	INFO	:	Resolved: model.cfg => model.cfg (exists = 1)
[2022-02-03 13:29:03	                main:452]	:	INFO	:	Resolved: model-final.pt => model-final.pt (exists = 0)
[2022-02-03 13:29:03	                main:452]	:	INFO	:	Resolved: model-input.pt => model-input.pt (exists = 0)
[2022-02-03 13:29:03	                main:452]	:	INFO	:	Resolved: snapshot.pt => snapshot.pt (exists = 1)
[2022-02-03 13:29:03	                main:472]	:	INFO	:	Dataset filename: dataset.hdf5
[2022-02-03 13:29:03	                main:474]	:	INFO	:	Configuration: 
[2022-02-03 13:29:03	                main:475]	:	INFO	:	    Model type: GRU
[2022-02-03 13:29:03	                main:476]	:	INFO	:	    Validation Loss Threshold: 0.0001
[2022-02-03 13:29:03	                main:477]	:	INFO	:	    Max Epochs: 2048
[2022-02-03 13:29:03	                main:478]	:	INFO	:	    Batch Size: 128
[2022-02-03 13:29:03	                main:479]	:	INFO	:	    Learning Rate: 0.01
[2022-02-03 13:29:03	                main:480]	:	INFO	:	    Patience: 10
[2022-02-03 13:29:03	                main:481]	:	INFO	:	    Hidden Width: 12
[2022-02-03 13:29:03	                main:482]	:	INFO	:	    # Recurrent Layers: 4
[2022-02-03 13:29:03	                main:483]	:	INFO	:	    # Backend Layers: 4
[2022-02-03 13:29:03	                main:484]	:	INFO	:	    # Threads: 1
[2022-02-03 13:29:03	                main:486]	:	INFO	:	Preparing Dataset
[2022-02-03 13:29:03	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xt from dataset.hdf5 into memory
[2022-02-03 13:29:03	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yt from dataset.hdf5 into memory
[2022-02-03 13:29:06	                load:106]	:	INFO	:	Successfully loaded dataset of 2048 examples into memory.
[2022-02-03 13:29:06	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Xv from dataset.hdf5 into memory
[2022-02-03 13:29:06	load_hdf5_ds_into_tensor:28]	:	INFO	:	Loading Dataset /Yv from dataset.hdf5 into memory
[2022-02-03 13:29:06	                load:106]	:	INFO	:	Successfully loaded dataset of 512 examples into memory.
[2022-02-03 13:29:06	                main:494]	:	INFO	:	Creating Model
[2022-02-03 13:29:06	                main:507]	:	INFO	:	Preparing config file
[2022-02-03 13:29:06	                main:511]	:	INFO	:	Found checkpoint, attempting to load... 
[2022-02-03 13:29:06	                main:512]	:	INFO	:	Loading config


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFFB9674F69

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.17.0


Dump Timestamp    : 02/03/22 13:29:06
Install Directory : C:\Program Files\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\8;C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome


ModLoad: 00000000da080000 00000000003df000 C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000bbd50000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.1288) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 10.0.19041.1023 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1023

ModLoad: 00000000bacd0000 00000000000be000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.1348) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 10.0.19041.1348 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1348

ModLoad: 00000000b9640000 00000000002c8000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernelbase.pdb
    File Version          : 10.0.19041.1348 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1348

ModLoad: 00000000bada0000 00000000001a1000 C:\WINDOWS\System32\USER32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000b9d60000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : win32u.pdb
    File Version          : 10.0.19041.1387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1387

ModLoad: 00000000bb590000 000000000002b000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 10.0.19041.1202 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1202

ModLoad: 00000000b99a0000 000000000010d000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32full.pdb
    File Version          : 10.0.19041.1387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1387

ModLoad: 00000000b4bf0000 0000000000058000 C:\ProgramData\BOINC\slots\8\c10.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000b9ab0000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.789

ModLoad: 00000000b9c60000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.19041.789 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.789

ModLoad: 00000000bb0e0000 00000000000ac000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.1052) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000bb4f0000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.19041.546

ModLoad: 00000000ba540000 000000000009b000 C:\WINDOWS\System32\sechost.dll (6.2.19041.1415) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000ba410000 000000000012a000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.1288) (-exported- Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000a2300000 000000000008d000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.29.30133.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
    File Version          : 14.29.30133.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.29.30133.0

ModLoad: 00000000b7a80000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.19041.867 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.867

ModLoad: 0000000013050000 000000000abe7000 C:\ProgramData\BOINC\slots\8\torch_cpu.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000a25a0000 000000000001b000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.29.30133.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
    File Version          : 14.29.30133.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.29.30133.0

ModLoad: 00000000a1b90000 000000000000c000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.29.30133.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\a01\_work\2\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
    File Version          : 14.29.30133.0 built by: vcwrkspc
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Visual Studio&#174;
    Product Version       : 14.29.30133.0

ModLoad: 0000000063da0000 00000000002db000 C:\ProgramData\BOINC\slots\8\fbgemm.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000064080000 000000000020d000 C:\ProgramData\BOINC\slots\8\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
    Linked PDB Filename   : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
    File Version          : 20200205
    Company Name          : Intel Corporation
    Product Name          : Intel(R) OpenMP* Runtime Library
    Product Version       : 5.0

ModLoad: 00000000b4bb0000 0000000000040000 C:\ProgramData\BOINC\slots\8\asmjit.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000001dc40000 0000000024b49000 C:\ProgramData\BOINC\slots\8\torch_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000b3b00000 0000000000045000 C:\ProgramData\BOINC\slots\8\c10_cuda.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 000000000fb90000 00000000034b2000 C:\ProgramData\BOINC\slots\8\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA RAND Library
    Product Version       : 6,14,11,1012

ModLoad: 000000004f450000 00000000030a5000 C:\ProgramData\BOINC\slots\8\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1031
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA SPARSE BLAS Library
    Product Version       : 6,14,11,1031

ModLoad: 00000000b4cd0000 0000000000010000 C:\ProgramData\BOINC\slots\8\nvToolsExt64_1.dll (-exported- Symbols Loaded)
    Linked PDB Filename   : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb

ModLoad: 00000000eb640000 000000000919f000 C:\ProgramData\BOINC\slots\8\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1012
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA FFT Library
    Product Version       : 6,14,11,1012

ModLoad: 00000000e78e0000 0000000003d5b000 C:\ProgramData\BOINC\slots\8\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000dd0a0000 0000000001eca000 C:\ProgramData\BOINC\slots\8\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,1022
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA BLAS Library
    Product Version       : 6,14,11,1022

ModLoad: 00000000f47e0000 000000001b3a4000 C:\ProgramData\BOINC\slots\8\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
    Linked PDB Filename   : 
    File Version          : 6,14,11,10020
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 10.2.86 CUDNN Library
    Product Version       : 6,14,11,10020

ModLoad: 00000000ba7e0000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000b7ca0000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000b9f90000 0000000000472000 C:\WINDOWS\System32\Setupapi.dll (6.2.19041.1237) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000b9460000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 10.0.19041.1151 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1151

ModLoad: 00000000b94b0000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcrypt.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000b9200000 0000000000034000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.19041.1151) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 10.0.19041.1151 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1151

ModLoad: 00000000b9b50000 0000000000060000 C:\WINDOWS\System32\WINTRUST.dll (6.2.19041.1266) (-exported- Symbols Loaded)
    Linked PDB Filename   : wintrust.pdb
    File Version          : 10.0.19041.1266 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1266

ModLoad: 00000000b94e0000 0000000000156000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.1320) (-exported- Symbols Loaded)
    Linked PDB Filename   : crypt32.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000b9040000 0000000000012000 C:\WINDOWS\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : msasn1.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000bb5c0000 0000000000744000 C:\WINDOWS\System32\Shell32.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : shell32.pdb
    File Version          : 10.0.19041.964 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.964

ModLoad: 00000000b7220000 0000000000794000 C:\WINDOWS\SYSTEM32\windows.storage.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : Windows.Storage.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000bb190000 0000000000355000 C:\WINDOWS\System32\combase.dll (6.2.19041.1348) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.19041.1320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1320

ModLoad: 00000000b8dd0000 000000000002e000 C:\WINDOWS\SYSTEM32\Wldp.dll (6.2.19041.1320) (-exported- Symbols Loaded)
    Linked PDB Filename   : WLDP.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000b9ec0000 00000000000ad000 C:\WINDOWS\System32\SHCORE.dll (6.2.19041.1387) (-exported- Symbols Loaded)
    Linked PDB Filename   : shcore.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000b9e60000 0000000000055000 C:\WINDOWS\System32\shlwapi.dll (6.2.19041.1023) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000628e0000 000000000029d000 C:\WINDOWS\system32\nvcuda.dll (27.21.14.6089) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r460\r460_74\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
    File Version          : 27.21.14.6089
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.2.66 driver
    Product Version       : 27.21.14.6089

ModLoad: 00000000b1b90000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.DLL (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 0000000053c30000 00000000015c2000 C:\WINDOWS\system32\DriverStore\FileRepository\nv_dispui.inf_amd64_40cf23587feed9e7\nvcuda64.dll (27.21.14.6089) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r460\r460_74\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
    File Version          : 27.21.14.6089
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA CUDA 11.2.66 driver
    Product Version       : 27.21.14.6089

ModLoad: 00000000ba5e0000 000000000006b000 C:\WINDOWS\System32\WS2_32.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : ws2_32.pdb
    File Version          : 10.0.19041.1081 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1081

ModLoad: 00000000b1b50000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptnet.pdb
    File Version          : 10.0.19041.906 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.906

ModLoad: 00000000b8d40000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.19041.546 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.546

ModLoad: 00000000b6b60000 000000000009e000 C:\WINDOWS\system32\uxtheme.dll (6.2.19041.1266) (-exported- Symbols Loaded)
    Linked PDB Filename   : UxTheme.pdb
    File Version          : 10.0.19041.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1

ModLoad: 00000000ae230000 00000000006f0000 C:\WINDOWS\SYSTEM32\nvapi64.dll (27.21.14.6089) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\dvs\p4\build\sw\rel\gpu_drv\r460\r460_74\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
    File Version          : 27.21.14.6089
    Company Name          : NVIDIA Corporation
    Product Name          : NVIDIA Windows drivers
    Product Version       : 27.21.14.6089

ModLoad: 00000000ba650000 000000000012a000 C:\WINDOWS\System32\OLE32.DLL (6.2.19041.1202) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 10.0.19041.1320 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1320

ModLoad: 00000000b9910000 0000000000082000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.1415) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.19041.1415 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.19041.1415



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 3629, Write: 307, Other 954

- I/O Transfers Counters -
Read: 6695750, Write: 6037, Other 29994

- Paged Pool Usage -
QuotaPagedPoolUsage: 3454944, QuotaPeakPagedPoolUsage: 3454944
QuotaNonPagedPoolUsage: 38208, QuotaPeakNonPagedPoolUsage: 38272

- Virtual Memory Usage -
VirtualSize: -1255297024, PeakVirtualSize: -650063872

- Pagefile Usage -
PagefileUsage: -1255297024, PeakPagefileUsage: -1255297024

- Working Set Size -
WorkingSetSize: 1233842176, PeakWorkingSetSize: 1233846272, PageFaultCount: 429685

*** Dump of thread ID 5984 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 6093750.000000, User Time: 22187500.000000, Wait Time: 194650.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x00007FFFB9674F69

- Registers -
rax=0000000018be8b93 rbx=00000000da3e4910 rcx=00000000414e5d30 rdx=0000000000000001 rsi=00000000414e6670 rdi=0000000019930520
r8=00000000414e5dc8 r9=0000000000000000 r10=0000000000000000 r11=000000006a335200 r12=00000000071ecc30 r13=00000000414e6880
r14=000000000e615bb0 r15=0000000000000000 rip=00000000b9674f69 rsp=00000000414e6510 rbp=00000000414e6800
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
414e65e0 da2fa884 00000040 0000004f 071ed400 0000024c KERNELBASE!RaiseException+0x0 
414e6640 da0a1ca6 00000040 1d440000 06d7d5c8 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
414e66f0 da0a593b 00000000 00000030 00000020 00000008 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
414e6840 da0a1a78 00000000 00000000 414e6b70 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
414e6910 da0a1cfc 414e6b70 00000000 00000080 414e6e40 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
414e69d0 da0ac4dd 414e6e40 414e6e40 414e6e40 414e6e40 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
414e6cc0 da09b2fe 414e72d8 414e6e40 00000000 00000080 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
414ffa60 da2dc6f8 1f07a970 00000000 00000000 1f0208c0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
414ffaa0 bace7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
414ffad0 bbda2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
414ffb50 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 6052 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 194431.000000

- Registers -
rax=00000000000001d3 rbx=000000001d445bc0 rcx=0000000000000070 rdx=000000001d445840 rsi=000000001d4439c0 rdi=000000001d445bc0
r8=0000000000000008 r9=0000000000000040 r10=0000000000000580 r11=00000000fd3ffbbf r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000bbdf07c4 rsp=00000000415ffaa8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
415ffaa0 bbda2dc7 00000000 00000000 1d4439c0 1d445bc0 ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
415ffda0 bace7034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0 
415ffdd0 bbda2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
415ffe50 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 11040 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 194430.000000

- Registers -
rax=00000000000001d3 rbx=000000001d446040 rcx=0000000000000070 rdx=000000001d445cc0 rsi=000000001d4439c0 rdi=000000001d446040
r8=000000000000004c r9=000000001d443fc4 r10=000000000000006c r11=000000000000006c r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000bbdf07c4 rsp=00000000416ffac8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
416ffac0 bbda2dc7 00000000 00000000 1d4439c0 1d446040 ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
416ffdc0 bace7034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0 
416ffdf0 bbda2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
416ffe70 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 11168 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 194430.000000

- Registers -
rax=00000000000001d3 rbx=000000001d4463e0 rcx=0000000000000070 rdx=000000001d446060 rsi=000000001d4439c0 rdi=000000001d4463e0
r8=0000000000000359 r9=00000000bb1709c8 r10=000000001f0e7c10 r11=00000000bb177ea8 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000bbdf07c4 rsp=00000000417ff5b8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
417ff5b0 bbda2dc7 00000000 00000000 1d4439c0 1d4463e0 ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
417ff8b0 bace7034 00000000 00000000 00000000 00000000 ntdll!TpReleaseCleanupGroupMembers+0x0 
417ff8e0 bbda2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
417ff960 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 10844 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 194649.000000

- Registers -
rax=0000000000000034 rbx=0000000000000000 rcx=0000000000000000 rdx=00000000419ffa00 rsi=0000000000000000 rdi=0000000000000064
r8=00000000419ff9d8 r9=0000000000000000 r10=0000000000000000 r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000bbded3f4 rsp=00000000419ff9d8 rbp=0000000000000000
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
419ff9d0 b968962e 419ffa98 00000000 00000000 00000000 ntdll!NtDelayExecution+0x0 
419ffa70 da0ba07f 00000000 00000000 c00000bb 00000000 KERNELBASE!SleepEx+0x0 
419ffaa0 bace7034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0 
419ffad0 bbda2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
419ffb50 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 10060 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 194650.000000

- Registers -
rax=000000000000005b rbx=000000000000000e rcx=000000000000000e rdx=0000000041aff8c0 rsi=0000000000000000 rdi=000000000000000e
r8=0000000000000001 r9=000000001f1fbb00 r10=0000000000000000 r11=0000000041aff830 r12=0000000000000064 r13=0000000041aff8c0
r14=0000000041aff5c0 r15=0000000000000000 rip=00000000bbded8c4 rsp=0000000041aff568 rbp=000000001f1fbba0
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
41aff560 b968cb20 00000000 412b9000 bbeba4d0 412b9000 ntdll!ZwWaitForMultipleObjects+0x0 
41aff850 b968ca1e 1d454890 004e0049 0000000e 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0 
41aff890 53de66d4 1f1ff080 00000000 00000000 00000000 KERNELBASE!WaitForMultipleObjects+0x0 
41affad0 53e7e51d 00000000 1f1fbba0 00000000 00000000 nvcuda64!+0x0 
41affb80 53de60a3 1518d09d 137ba588 1f7dab50 00000000 nvcuda64!cuProfilerStop+0x0 
41affbb0 540fd794 1f7dab50 00000000 00000000 00000000 nvcuda64!+0x0 
41affbe0 bace7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
41affc10 bbda2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
41affc90 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 11060 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 194497.000000

- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=00000000000002a0 rdx=0000000000000000 rsi=0000000000000000 rdi=00000000000002a0
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=00000000000002a0 r15=0000000000000000 rip=00000000bbdecdf4 rsp=0000000041bff818 rbp=000000001f1dfef0
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
41bff810 b9661a5e 00000000 00000000 00000000 00000000 ntdll!ZwWaitForSingleObject+0x0 
41bff8b0 53ffcd6e 1f7da460 b966a395 00000000 000002a0 KERNELBASE!WaitForSingleObjectEx+0x0 
41bff920 53de60a3 1f1dfef0 1f6258d0 1f7da460 00000000 nvcuda64!cuProfilerStop+0x0 
41bff950 540fd794 1f7da460 00000000 00000000 00000000 nvcuda64!+0x0 
41bff980 bace7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
41bff9b0 bbda2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
41bffa30 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 

*** Dump of thread ID 11180 (state: Waiting): ***

- Information -
Status: Wait Reason: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 194631.000000

- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=000000001f47f298 rdx=0000000041cff938 rsi=0000000041cff8a0 rdi=0000000000000000
r8=0000000041cff918 r9=0000000000000000 r10=0000000000000000 r11=0000000000000246 r12=0000000041cff938 r13=0000000000000000
r14=000000001f47f298 r15=000000001f483e70 rip=00000000bbdf0764 rsp=0000000041cff878 rbp=0000000041cff8d0
cs=0033  ss=002b  ds=0000  es=0000  fs=0000  gs=0000             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
41cff870 bbdb4295 00000102 00000102 00000000 41cff8a0 ntdll!NtWaitForAlertByThreadId+0x0 
41cff8e0 b96acdd9 1f481650 00000000 00000000 00000000 ntdll!RtlSleepConditionVariableCS+0x0 
41cff910 53d64a5c 1f481650 bbda07b0 1f481650 ffb3b4c0 KERNELBASE!SleepConditionVariableCS+0x0 
41cff970 53de60a3 00000001 14f716bf 00000000 00000000 nvcuda64!+0x0 
41cff9a0 540fd794 1f7daaf0 00000000 00000000 00000000 nvcuda64!+0x0 
41cff9d0 bace7034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0 
41cffa00 bbda2651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
41cffa80 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>


©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)