Task 14178413
| Name | ParityModified-1647046470-1887-2-0_0 |
| Workunit | 11240532 |
| Created | 1 Apr 2022, 5:33:34 UTC |
| Sent | 5 Apr 2022, 18:57:25 UTC |
| Report deadline | 13 Apr 2022, 18:57:25 UTC |
| Received | 7 Apr 2022, 12:58:32 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED |
| Computer ID | 4503 |
| Run time | 8 hours 8 min 28 sec |
| CPU time | 13 min 46 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 10,073.24 GFLOPS |
| Application version | Machine Learning Dataset Generator (GPU) v9.75 (cuda10200) windows_x86_64 |
| Peak working set size | 2.08 GB |
| Peak swap size | 4.37 GB |
| Peak disk usage | 1.54 GB |
Stderr output
<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
exceeded elapsed time limit 29305.33 (4000000.00G/136.49G)</message>
<stderr_txt>
-06 22:28:51 main:452] : INFO : Resolved: model-input.pt => model-input.pt (exists = 1)
[2022-04-06 22:28:51 main:452] : INFO : Resolved: snapshot.pt => snapshot.pt (exists = 0)
[2022-04-06 22:28:51 main:472] : INFO : Dataset filename: dataset.hdf5
[2022-04-06 22:28:51 main:474] : INFO : Configuration:
[2022-04-06 22:28:51 main:475] : INFO : Model type: GRU
[2022-04-06 22:28:51 main:476] : INFO : Validation Loss Threshold: 0.0001
[2022-04-06 22:28:51 main:477] : INFO : Max Epochs: 2048
[2022-04-06 22:28:51 main:478] : INFO : Batch Size: 128
[2022-04-06 22:28:51 main:479] : INFO : Learning Rate: 0.01
[2022-04-06 22:28:51 main:480] : INFO : Patience: 10
[2022-04-06 22:28:51 main:481] : INFO : Hidden Width: 12
[2022-04-06 22:28:51 main:482] : INFO : # Recurrent Layers: 4
[2022-04-06 22:28:51 main:483] : INFO : # Backend Layers: 4
[2022-04-06 22:28:51 main:484] : INFO : # Threads: 1
[2022-04-06 22:28:51 main:486] : INFO : Preparing Dataset
[2022-04-06 22:28:51 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xt from dataset.hdf5 into memory
[2022-04-06 22:28:55 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yt from dataset.hdf5 into memory
[2022-04-06 22:29:07 load:106] : INFO : Successfully loaded dataset of 2048 examples into memory.
[2022-04-06 22:29:07 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Xv from dataset.hdf5 into memory
[2022-04-06 22:29:11 load_hdf5_ds_into_tensor:28] : INFO : Loading Dataset /Yv from dataset.hdf5 into memory
[2022-04-06 22:29:12 load:106] : INFO : Successfully loaded dataset of 512 examples into memory.
[2022-04-06 22:29:13 main:494] : INFO : Creating Model
[2022-04-06 22:29:13 main:507] : INFO : Preparing config file
[2022-04-06 22:29:14 main:519] : INFO : Creating new config file
[2022-04-06 22:29:14 main:538] : INFO : This is a continuation WU, loading previous network
[2022-04-06 22:29:21 main:559] : INFO : Loading DataLoader into Memory
[2022-04-06 22:29:35 main:562] : INFO : Starting Training
[2022-04-06 22:31:10 main:574] : INFO : Epoch 1 | loss: 0.0311412 | val_loss: 0.0312101 | Time: 94961.8 ms
[2022-04-06 22:32:26 main:574] : INFO : Epoch 2 | loss: 0.0311028 | val_loss: 0.0312236 | Time: 75931.4 ms
[2022-04-06 22:33:30 main:574] : INFO : Epoch 3 | loss: 0.0310976 | val_loss: 0.0312161 | Time: 64492.1 ms
[2022-04-06 22:36:08 main:574] : INFO : Epoch 4 | loss: 0.0310909 | val_loss: 0.0312074 | Time: 147929 ms
[2022-04-06 22:39:47 main:574] : INFO : Epoch 5 | loss: 0.0310925 | val_loss: 0.0312097 | Time: 212538 ms
[2022-04-06 22:41:28 main:574] : INFO : Epoch 6 | loss: 0.0310906 | val_loss: 0.0312327 | Time: 100133 ms
[2022-04-06 22:41:57 main:574] : INFO : Epoch 7 | loss: 0.0310886 | val_loss: 0.031208 | Time: 24591.5 ms
[2022-04-06 22:45:12 main:574] : INFO : Epoch 8 | loss: 0.0310849 | val_loss: 0.0312326 | Time: 194075 ms
[2022-04-06 22:47:37 main:574] : INFO : Epoch 9 | loss: 0.0310834 | val_loss: 0.031219 | Time: 145032 ms
[2022-04-06 22:50:55 main:574] : INFO : Epoch 10 | loss: 0.0310866 | val_loss: 0.0312334 | Time: 191895 ms
[2022-04-06 22:53:38 main:574] : INFO : Epoch 11 | loss: 0.0310836 | val_loss: 0.0312022 | Time: 162633 ms
[2022-04-06 22:55:52 main:574] : INFO : Epoch 12 | loss: 0.0310865 | val_loss: 0.0312259 | Time: 134040 ms
[2022-04-06 22:59:05 main:574] : INFO : Epoch 13 | loss: 0.0310875 | val_loss: 0.0312197 | Time: 181697 ms
[2022-04-06 22:59:37 main:574] : INFO : Epoch 14 | loss: 0.0310868 | val_loss: 0.0312307 | Time: 27180.3 ms
[2022-04-06 23:01:01 main:574] : INFO : Epoch 15 | loss: 0.0310835 | val_loss: 0.0312218 | Time: 83779.7 ms
[2022-04-06 23:03:37 main:574] : INFO : Epoch 16 | loss: 0.0310872 | val_loss: 0.0312295 | Time: 150653 ms
[2022-04-06 23:06:30 main:574] : INFO : Epoch 17 | loss: 0.0310839 | val_loss: 0.0312266 | Time: 172948 ms
[2022-04-06 23:10:00 main:574] : INFO : Epoch 18 | loss: 0.03109 | val_loss: 0.03121 | Time: 204239 ms
[2022-04-06 23:12:51 main:574] : INFO : Epoch 19 | loss: 0.0310883 | val_loss: 0.0312215 | Time: 170978 ms
[2022-04-06 23:16:50 main:574] : INFO : Epoch 20 | loss: 0.0310832 | val_loss: 0.0312351 | Time: 239825 ms
[2022-04-06 23:18:44 main:574] : INFO : Epoch 21 | loss: 0.0310829 | val_loss: 0.0312142 | Time: 113968 ms
[2022-04-06 23:21:43 main:574] : INFO : Epoch 22 | loss: 0.0310827 | val_loss: 0.0312104 | Time: 178594 ms
[2022-04-06 23:23:21 main:574] : INFO : Epoch 23 | loss: 0.0310818 | val_loss: 0.0312428 | Time: 97676.2 ms
[2022-04-06 23:23:28 main:574] : INFO : Epoch 24 | loss: 0.0310889 | val_loss: 0.031233 | Time: 7208.1 ms
[2022-04-06 23:25:47 main:574] : INFO : Epoch 25 | loss: 0.0310913 | val_loss: 0.0312088 | Time: 133027 ms
[2022-04-06 23:29:09 main:574] : INFO : Epoch 26 | loss: 0.0310911 | val_loss: 0.0312176 | Time: 192399 ms
[2022-04-06 23:31:36 main:574] : INFO : Epoch 27 | loss: 0.0310938 | val_loss: 0.0312077 | Time: 142135 ms
[2022-04-06 23:33:35 main:574] : INFO : Epoch 28 | loss: 0.0310983 | val_loss: 0.0312138 | Time: 114350 ms
[2022-04-06 23:35:57 main:574] : INFO : Epoch 29 | loss: 0.0310988 | val_loss: 0.0312153 | Time: 141933 ms
[2022-04-06 23:38:06 main:574] : INFO : Epoch 30 | loss: 0.0310896 | val_loss: 0.0312123 | Time: 128441 ms
[2022-04-06 23:40:42 main:574] : INFO : Epoch 31 | loss: 0.0310883 | val_loss: 0.0312241 | Time: 155705 ms
[2022-04-06 23:43:01 main:574] : INFO : Epoch 32 | loss: 0.031084 | val_loss: 0.0312355 | Time: 139118 ms
[2022-04-06 23:44:08 main:574] : INFO : Epoch 33 | loss: 0.0310845 | val_loss: 0.0312077 | Time: 61311.7 ms
[2022-04-06 23:44:54 main:574] : INFO : Epoch 34 | loss: 0.0310869 | val_loss: 0.0312192 | Time: 46173.2 ms
[2022-04-06 23:45:01 main:574] : INFO : Epoch 35 | loss: 0.0310844 | val_loss: 0.0312411 | Time: 6497.61 ms
[2022-04-06 23:48:14 main:574] : INFO : Epoch 36 | loss: 0.0310834 | val_loss: 0.0312219 | Time: 193770 ms
[2022-04-06 23:48:21 main:574] : INFO : Epoch 37 | loss: 0.0310853 | val_loss: 0.0312264 | Time: 6010.6 ms
[2022-04-06 23:49:18 main:574] : INFO : Epoch 38 | loss: 0.0310831 | val_loss: 0.0312234 | Time: 57077.4 ms
[2022-04-06 23:51:08 main:574] : INFO : Epoch 39 | loss: 0.031084 | val_loss: 0.031224 | Time: 105372 ms
[2022-04-06 23:53:17 main:574] : INFO : Epoch 40 | loss: 0.0310966 | val_loss: 0.0312182 | Time: 122901 ms
[2022-04-06 23:56:12 main:574] : INFO : Epoch 41 | loss: 0.0310923 | val_loss: 0.0312173 | Time: 175400 ms
[2022-04-07 00:00:17 main:574] : INFO : Epoch 42 | loss: 0.0310864 | val_loss: 0.0312232 | Time: 239890 ms
[2022-04-07 00:03:33 main:574] : INFO : Epoch 43 | loss: 0.0310831 | val_loss: 0.0312242 | Time: 191011 ms
[2022-04-07 00:05:55 main:574] : INFO : Epoch 44 | loss: 0.0310833 | val_loss: 0.0312126 | Time: 142297 ms
[2022-04-07 00:07:39 main:574] : INFO : Epoch 45 | loss: 0.0310822 | val_loss: 0.0312224 | Time: 103788 ms
[2022-04-07 00:09:48 main:574] : INFO : Epoch 46 | loss: 0.031081 | val_loss: 0.0312304 | Time: 123763 ms
[2022-04-07 00:11:20 main:574] : INFO : Epoch 47 | loss: 0.0310752 | val_loss: 0.031224 | Time: 86144.5 ms
[2022-04-07 00:15:18 main:574] : INFO : Epoch 48 | loss: 0.0310776 | val_loss: 0.0312128 | Time: 231566 ms
[2022-04-07 00:16:21 main:574] : INFO : Epoch 49 | loss: 0.0310822 | val_loss: 0.0312288 | Time: 58017.7 ms
[2022-04-07 00:18:12 main:574] : INFO : Epoch 50 | loss: 0.031077 | val_loss: 0.0312344 | Time: 110532 ms
[2022-04-07 00:18:37 main:574] : INFO : Epoch 51 | loss: 0.0310735 | val_loss: 0.0312124 | Time: 24290.3 ms
[2022-04-07 00:19:56 main:574] : INFO : Epoch 52 | loss: 0.0310798 | val_loss: 0.0312187 | Time: 78780.1 ms
[2022-04-07 00:21:01 main:574] : INFO : Epoch 53 | loss: 0.0310756 | val_loss: 0.0312289 | Time: 59284.1 ms
[2022-04-07 00:22:10 main:574] : INFO : Epoch 54 | loss: 0.0310763 | val_loss: 0.031214 | Time: 64141.9 ms
[2022-04-07 00:23:42 main:574] : INFO : Epoch 55 | loss: 0.0310766 | val_loss: 0.031222 | Time: 91763.1 ms
[2022-04-07 00:27:39 main:574] : INFO : Epoch 56 | loss: 0.0310748 | val_loss: 0.0312434 | Time: 231597 ms
[2022-04-07 00:31:12 main:574] : INFO : Epoch 57 | loss: 0.0310712 | val_loss: 0.0312329 | Time: 207805 ms
[2022-04-07 00:33:41 main:574] : INFO : Epoch 58 | loss: 0.0310693 | val_loss: 0.0312299 | Time: 148247 ms
[2022-04-07 00:35:52 main:574] : INFO : Epoch 59 | loss: 0.031069 | val_loss: 0.031227 | Time: 131513 ms
[2022-04-07 00:39:12 main:574] : INFO : Epoch 60 | loss: 0.0310737 | val_loss: 0.0312284 | Time: 199354 ms
[2022-04-07 00:40:48 main:574] : INFO : Epoch 61 | loss: 0.0310774 | val_loss: 0.0312203 | Time: 91156.1 ms
[2022-04-07 00:42:21 main:574] : INFO : Epoch 62 | loss: 0.031079 | val_loss: 0.0312389 | Time: 87595.1 ms
[2022-04-07 00:46:03 main:574] : INFO : Epoch 63 | loss: 0.0310816 | val_loss: 0.0312246 | Time: 219988 ms
[2022-04-07 00:49:35 main:574] : INFO : Epoch 64 | loss: 0.0310845 | val_loss: 0.031216 | Time: 207931 ms
[2022-04-07 00:50:06 main:574] : INFO : Epoch 65 | loss: 0.0310809 | val_loss: 0.0312282 | Time: 30681.4 ms
[2022-04-07 00:53:00 main:574] : INFO : Epoch 66 | loss: 0.031074 | val_loss: 0.0312159 | Time: 173755 ms
[2022-04-07 00:55:16 main:574] : INFO : Epoch 67 | loss: 0.0310739 | val_loss: 0.0312201 | Time: 131212 ms
[2022-04-07 00:55:54 main:574] : INFO : Epoch 68 | loss: 0.0310742 | val_loss: 0.0312274 | Time: 38534 ms
[2022-04-07 00:57:55 main:574] : INFO : Epoch 69 | loss: 0.031086 | val_loss: 0.0312065 | Time: 119696 ms
[2022-04-07 01:01:04 main:574] : INFO : Epoch 70 | loss: 0.0310945 | val_loss: 0.0312005 | Time: 184078 ms
[2022-04-07 01:03:38 main:574] : INFO : Epoch 71 | loss: 0.0310878 | val_loss: 0.0312176 | Time: 149160 ms
[2022-04-07 01:04:20 main:574] : INFO : Epoch 72 | loss: 0.0310813 | val_loss: 0.0312045 | Time: 41819.2 ms
[2022-04-07 01:05:09 main:574] : INFO : Epoch 73 | loss: 0.0310794 | val_loss: 0.031224 | Time: 49309.7 ms
[2022-04-07 01:06:47 main:574] : INFO : Epoch 74 | loss: 0.0310837 | val_loss: 0.031214 | Time: 93477 ms
[2022-04-07 01:10:44 main:574] : INFO : Epoch 75 | loss: 0.031089 | val_loss: 0.0312211 | Time: 231162 ms
[2022-04-07 01:12:15 main:574] : INFO : Epoch 76 | loss: 0.0310866 | val_loss: 0.0312092 | Time: 91300.1 ms
[2022-04-07 01:12:22 main:574] : INFO : Epoch 77 | loss: 0.0310812 | val_loss: 0.031224 | Time: 5956.57 ms
[2022-04-07 01:15:40 main:574] : INFO : Epoch 78 | loss: 0.0310832 | val_loss: 0.031217 | Time: 197938 ms
[2022-04-07 01:19:08 main:574] : INFO : Epoch 79 | loss: 0.0310784 | val_loss: 0.0312107 | Time: 199990 ms
[2022-04-07 01:21:55 main:574] : INFO : Epoch 80 | loss: 0.031077 | val_loss: 0.0312229 | Time: 161389 ms
[2022-04-07 01:22:26 main:574] : INFO : Epoch 81 | loss: 0.0310863 | val_loss: 0.0312208 | Time: 31402.4 ms
[2022-04-07 01:22:35 main:574] : INFO : Epoch 82 | loss: 0.0310845 | val_loss: 0.0312074 | Time: 8285.63 ms
[2022-04-07 01:26:14 main:574] : INFO : Epoch 83 | loss: 0.0310816 | val_loss: 0.0312335 | Time: 214329 ms
[2022-04-07 01:30:09 main:574] : INFO : Epoch 84 | loss: 0.0310763 | val_loss: 0.031206 | Time: 235105 ms
[2022-04-07 01:33:35 main:574] : INFO : Epoch 85 | loss: 0.0310751 | val_loss: 0.0312322 | Time: 206573 ms
[2022-04-07 01:35:58 main:574] : INFO : Epoch 86 | loss: 0.0310744 | val_loss: 0.031224 | Time: 142243 ms
[2022-04-07 01:39:26 main:574] : INFO : Epoch 87 | loss: 0.0310801 | val_loss: 0.0312135 | Time: 208113 ms
[2022-04-07 01:39:53 main:574] : INFO : Epoch 88 | loss: 0.0310781 | val_loss: 0.0312068 | Time: 27210 ms
[2022-04-07 01:41:00 main:574] : INFO : Epoch 89 | loss: 0.0310851 | val_loss: 0.0312099 | Time: 66729.6 ms
[2022-04-07 01:44:54 main:574] : INFO : Epoch 90 | loss: 0.0310962 | val_loss: 0.0312196 | Time: 229177 ms
[2022-04-07 01:47:45 main:574] : INFO : Epoch 91 | loss: 0.0310856 | val_loss: 0.031225 | Time: 171135 ms
[2022-04-07 01:51:08 main:574] : INFO : Epoch 92 | loss: 0.0310824 | val_loss: 0.0312343 | Time: 196525 ms
[2022-04-07 01:53:44 main:574] : INFO : Epoch 93 | loss: 0.0310859 | val_loss: 0.0312128 | Time: 156741 ms
[2022-04-07 01:56:21 main:574] : INFO : Epoch 94 | loss: 0.031084 | val_loss: 0.0312092 | Time: 156105 ms
[2022-04-07 02:00:06 main:574] : INFO : Epoch 95 | loss: 0.0310775 | val_loss: 0.0312332 | Time: 220501 ms
[2022-04-07 02:00:14 main:574] : INFO : Epoch 96 | loss: 0.0310735 | val_loss: 0.0312358 | Time: 8311.11 ms
[2022-04-07 02:03:45 main:574] : INFO : Epoch 97 | loss: 0.0310735 | val_loss: 0.031221 | Time: 201928 ms
[2022-04-07 02:07:03 main:574] : INFO : Epoch 98 | loss: 0.0310814 | val_loss: 0.0312236 | Time: 198253 ms
[2022-04-07 02:10:12 main:574] : INFO : Epoch 99 | loss: 0.0310846 | val_loss: 0.0312272 | Time: 181704 ms
[2022-04-07 02:12:59 main:574] : INFO : Epoch 100 | loss: 0.0310841 | val_loss: 0.0312199 | Time: 157318 ms
[2022-04-07 02:16:55 main:574] : INFO : Epoch 101 | loss: 0.0310948 | val_loss: 0.0312087 | Time: 231576 ms
[2022-04-07 02:19:28 main:574] : INFO : Epoch 102 | loss: 0.0310898 | val_loss: 0.0312177 | Time: 148256 ms
[2022-04-07 02:20:51 main:574] : INFO : Epoch 103 | loss: 0.0310864 | val_loss: 0.0312386 | Time: 77953.3 ms
[2022-04-07 02:23:04 main:574] : INFO : Epoch 104 | loss: 0.0310802 | val_loss: 0.0312209 | Time: 128229 ms
[2022-04-07 02:26:22 main:574] : INFO : Epoch 105 | loss: 0.0310846 | val_loss: 0.0312259 | Time: 197709 ms
[2022-04-07 02:28:27 main:574] : INFO : Epoch 106 | loss: 0.0310931 | val_loss: 0.031206 | Time: 125025 ms
[2022-04-07 02:29:41 main:574] : INFO : Epoch 107 | loss: 0.0310883 | val_loss: 0.0312305 | Time: 74346 ms
[2022-04-07 02:32:08 main:574] : INFO : Epoch 108 | loss: 0.0310825 | val_loss: 0.0312123 | Time: 146606 ms
[2022-04-07 02:33:20 main:574] : INFO : Epoch 109 | loss: 0.0310877 | val_loss: 0.0312102 | Time: 71809.6 ms
[2022-04-07 02:35:54 main:574] : INFO : Epoch 110 | loss: 0.0310911 | val_loss: 0.0312104 | Time: 149368 ms
[2022-04-07 02:37:45 main:574] : INFO : Epoch 111 | loss: 0.0310865 | val_loss: 0.0312068 | Time: 110666 ms
[2022-04-07 02:40:49 main:574] : INFO : Epoch 112 | loss: 0.0310821 | val_loss: 0.0312217 | Time: 184100 ms
[2022-04-07 02:40:56 main:574] : INFO : Epoch 113 | loss: 0.0310782 | val_loss: 0.0312272 | Time: 6871.53 ms
[2022-04-07 02:43:06 main:574] : INFO : Epoch 114 | loss: 0.031083 | val_loss: 0.0312097 | Time: 128499 ms
[2022-04-07 02:46:02 main:574] : INFO : Epoch 115 | loss: 0.0310865 | val_loss: 0.031228 | Time: 168653 ms
[2022-04-07 02:48:21 main:574] : INFO : Epoch 116 | loss: 0.031081 | val_loss: 0.0312193 | Time: 131345 ms
[2022-04-07 02:51:27 main:574] : INFO : Epoch 117 | loss: 0.031079 | val_loss: 0.0312405 | Time: 180484 ms
[2022-04-07 02:53:51 main:574] : INFO : Epoch 118 | loss: 0.0310773 | val_loss: 0.0312229 | Time: 144502 ms
[2022-04-07 02:55:36 main:574] : INFO : Epoch 119 | loss: 0.0310746 | val_loss: 0.0312233 | Time: 104785 ms
[2022-04-07 02:55:41 main:574] : INFO : Epoch 120 | loss: 0.031073 | val_loss: 0.0312387 | Time: 5439.1 ms
[2022-04-07 02:58:25 main:574] : INFO : Epoch 121 | loss: 0.0310743 | val_loss: 0.0312422 | Time: 161921 ms
[2022-04-07 02:59:09 main:574] : INFO : Epoch 122 | loss: 0.0310702 | val_loss: 0.0312402 | Time: 36895.9 ms
[2022-04-07 03:01:20 main:574] : INFO : Epoch 123 | loss: 0.0310705 | val_loss: 0.0312377 | Time: 131518 ms
[2022-04-07 03:03:41 main:574] : INFO : Epoch 124 | loss: 0.031079 | val_loss: 0.0312375 | Time: 135937 ms
[2022-04-07 03:04:44 main:574] : INFO : Epoch 125 | loss: 0.0310938 | val_loss: 0.031214 | Time: 62758.5 ms
[2022-04-07 03:07:09 main:574] : INFO : Epoch 126 | loss: 0.0310901 | val_loss: 0.031226 | Time: 140158 ms
[2022-04-07 03:10:20 main:574] : INFO : Epoch 127 | loss: 0.0310883 | val_loss: 0.0312291 | Time: 184936 ms
[2022-04-07 03:12:39 main:574] : INFO : Epoch 128 | loss: 0.0310891 | val_loss: 0.03123 | Time: 134680 ms
[2022-04-07 03:16:44 main:574] : INFO : Epoch 129 | loss: 0.0310888 | val_loss: 0.0312171 | Time: 238682 ms
[2022-04-07 03:18:56 main:574] : INFO : Epoch 130 | loss: 0.0310805 | val_loss: 0.031227 | Time: 132844 ms
[2022-04-07 03:19:55 main:574] : INFO : Epoch 131 | loss: 0.0310747 | val_loss: 0.031219 | Time: 58168.2 ms
[2022-04-07 03:21:57 main:574] : INFO : Epoch 132 | loss: 0.0310752 | val_loss: 0.0312185 | Time: 122545 ms
[2022-04-07 03:23:49 main:574] : INFO : Epoch 133 | loss: 0.0310921 | val_loss: 0.0312125 | Time: 111685 ms
[2022-04-07 03:27:55 main:574] : INFO : Epoch 134 | loss: 0.0312186 | val_loss: 0.0312381 | Time: 246082 ms
[2022-04-07 03:31:19 main:574] : INFO : Epoch 135 | loss: 0.0312339 | val_loss: 0.0312292 | Time: 204229 ms
[2022-04-07 03:33:49 main:574] : INFO : Epoch 136 | loss: 0.0312271 | val_loss: 0.0312231 | Time: 149266 ms
[2022-04-07 03:36:34 main:574] : INFO : Epoch 137 | loss: 0.0312189 | val_loss: 0.031218 | Time: 164999 ms
[2022-04-07 03:40:44 main:574] : INFO : Epoch 138 | loss: 0.0312155 | val_loss: 0.0312151 | Time: 245203 ms
[2022-04-07 03:43:18 main:574] : INFO : Epoch 139 | loss: 0.0312114 | val_loss: 0.0312117 | Time: 153635 ms
[2022-04-07 03:46:43 main:574] : INFO : Epoch 140 | loss: 0.0312089 | val_loss: 0.0312095 | Time: 200233 ms
[2022-04-07 03:46:49 main:574] : INFO : Epoch 141 | loss: 0.0312056 | val_loss: 0.0312075 | Time: 6627.67 ms
[2022-04-07 03:47:32 main:574] : INFO : Epoch 142 | loss: 0.0312041 | val_loss: 0.0312064 | Time: 40731.9 ms
[2022-04-07 03:48:54 main:574] : INFO : Epoch 143 | loss: 0.0312021 | val_loss: 0.0312056 | Time: 81401.2 ms
[2022-04-07 03:50:55 main:574] : INFO : Epoch 144 | loss: 0.0312003 | val_loss: 0.0312053 | Time: 116339 ms
[2022-04-07 03:51:05 main:574] : INFO : Epoch 145 | loss: 0.0311994 | val_loss: 0.0312031 | Time: 9716.11 ms
[2022-04-07 03:52:18 main:574] : INFO : Epoch 146 | loss: 0.0311975 | val_loss: 0.0312057 | Time: 73356.1 ms
[2022-04-07 03:53:48 main:574] : INFO : Epoch 147 | loss: 0.0311964 | val_loss: 0.0312022 | Time: 90023.7 ms
[2022-04-07 03:57:37 main:574] : INFO : Epoch 148 | loss: 0.0311955 | val_loss: 0.0312015 | Time: 223232 ms
[2022-04-07 03:59:19 main:574] : INFO : Epoch 149 | loss: 0.031195 | val_loss: 0.0312015 | Time: 102305 ms
[2022-04-07 03:59:28 main:574] : INFO : Epoch 150 | loss: 0.0311929 | val_loss: 0.0311944 | Time: 8630.6 ms
[2022-04-07 04:01:55 main:574] : INFO : Epoch 151 | loss: 0.0311898 | val_loss: 0.031194 | Time: 141312 ms
[2022-04-07 04:03:47 main:574] : INFO : Epoch 152 | loss: 0.0311893 | val_loss: 0.0311929 | Time: 107754 ms
[2022-04-07 04:05:59 main:574] : INFO : Epoch 153 | loss: 0.0311878 | val_loss: 0.0311943 | Time: 131801 ms
[2022-04-07 04:10:34 main:574] : INFO : Epoch 154 | loss: 0.0311857 | val_loss: 0.0311924 | Time: 274601 ms
[2022-04-07 04:10:54 main:574] : INFO : Epoch 155 | loss: 0.0311854 | val_loss: 0.0311921 | Time: 20232.3 ms
[2022-04-07 04:14:45 main:574] : INFO : Epoch 156 | loss: 0.0311845 | val_loss: 0.0311898 | Time: 220885 ms
[2022-04-07 04:16:12 main:574] : INFO : Epoch 157 | loss: 0.0311834 | val_loss: 0.0311871 | Time: 87021.2 ms
[2022-04-07 04:16:23 main:574] : INFO : Epoch 158 | loss: 0.0311815 | val_loss: 0.0311874 | Time: 10431.3 ms
[2022-04-07 04:20:04 main:574] : INFO : Epoch 159 | loss: 0.0311804 | val_loss: 0.0311867 | Time: 214964 ms
[2022-04-07 04:20:54 main:574] : INFO : Epoch 160 | loss: 0.0311802 | val_loss: 0.0311947 | Time: 45281.5 ms
[2022-04-07 04:23:56 main:574] : INFO : Epoch 161 | loss: 0.0311842 | val_loss: 0.0311869 | Time: 182052 ms
[2022-04-07 04:26:28 main:574] : INFO : Epoch 162 | loss: 0.031182 | val_loss: 0.0311915 | Time: 152077 ms
[2022-04-07 04:29:29 main:574] : INFO : Epoch 163 | loss: 0.0311831 | val_loss: 0.031191 | Time: 176106 ms
[2022-04-07 04:33:29 main:574] : INFO : Epoch 164 | loss: 0.0311792 | val_loss: 0.0311844 | Time: 228721 ms
[2022-04-07 04:35:57 main:574] : INFO : Epoch 165 | loss: 0.0311789 | val_loss: 0.0311837 | Time: 143207 ms
[2022-04-07 04:40:32 main:574] : INFO : Epoch 166 | loss: 0.0311774 | val_loss: 0.0311841 | Time: 274946 ms
[2022-04-07 04:42:33 main:574] : INFO : Epoch 167 | loss: 0.031176 | val_loss: 0.0311844 | Time: 114664 ms
[2022-04-07 04:45:14 main:574] : INFO : Epoch 168 | loss: 0.0311753 | val_loss: 0.0311867 | Time: 161436 ms
[2022-04-07 04:48:51 main:574] : INFO : Epoch 169 | loss: 0.0311761 | val_loss: 0.0311842 | Time: 212578 ms
[2022-04-07 04:50:23 main:574] : INFO : Epoch 170 | loss: 0.0311758 | val_loss: 0.0311851 | Time: 91481.5 ms
[2022-04-07 04:52:01 main:574] : INFO : Epoch 171 | loss: 0.0311745 | val_loss: 0.0311846 | Time: 92341.1 ms
[2022-04-07 04:53:53 main:574] : INFO : Epoch 172 | loss: 0.0311768 | val_loss: 0.0311873 | Time: 111534 ms
[2022-04-07 04:56:34 main:574] : INFO : Epoch 173 | loss: 0.03118 | val_loss: 0.0311867 | Time: 161369 ms
[2022-04-07 05:00:15 main:574] : INFO : Epoch 174 | loss: 0.0311796 | val_loss: 0.031189 | Time: 216137 ms
[2022-04-07 05:03:51 main:574] : INFO : Epoch 175 | loss: 0.0311781 | val_loss: 0.0311856 | Time: 215526 ms
[2022-04-07 05:05:53 main:574] : INFO : Epoch 176 | loss: 0.0311741 | val_loss: 0.031184 | Time: 122324 ms
[2022-04-07 05:08:50 main:574] : INFO : Epoch 177 | loss: 0.0311736 | val_loss: 0.0311843 | Time: 176892 ms
[2022-04-07 05:10:48 main:574] : INFO : Epoch 178 | loss: 0.0311727 | val_loss: 0.0311862 | Time: 117546 ms
[2022-04-07 05:13:59 main:574] : INFO : Epoch 179 | loss: 0.0311731 | val_loss: 0.0311861 | Time: 191245 ms
[2022-04-07 05:14:18 main:574] : INFO : Epoch 180 | loss: 0.0311718 | val_loss: 0.0311856 | Time: 13293.1 ms
[2022-04-07 05:17:05 main:574] : INFO : Epoch 181 | loss: 0.0311726 | val_loss: 0.0311874 | Time: 166827 ms
[2022-04-07 05:20:57 main:574] : INFO : Epoch 182 | loss: 0.0311713 | val_loss: 0.0311867 | Time: 225881 ms
[2022-04-07 05:21:58 main:574] : INFO : Epoch 183 | loss: 0.0311702 | val_loss: 0.0311841 | Time: 61136.6 ms
[2022-04-07 05:23:28 main:574] : INFO : Epoch 184 | loss: 0.0311719 | val_loss: 0.0311851 | Time: 90602.3 ms
[2022-04-07 05:25:25 main:574] : INFO : Epoch 185 | loss: 0.0311723 | val_loss: 0.0311855 | Time: 116941 ms
[2022-04-07 05:25:38 main:574] : INFO : Epoch 186 | loss: 0.0311713 | val_loss: 0.0311872 | Time: 13005.6 ms
[2022-04-07 05:28:15 main:574] : INFO : Epoch 187 | loss: 0.0311713 | val_loss: 0.0311857 | Time: 156733 ms
[2022-04-07 05:29:04 main:574] : INFO : Epoch 188 | loss: 0.0311705 | val_loss: 0.0311839 | Time: 48546.4 ms
[2022-04-07 05:32:53 main:574] : INFO : Epoch 189 | loss: 0.0311697 | val_loss: 0.0311843 | Time: 229187 ms
[2022-04-07 05:33:50 main:574] : INFO : Epoch 190 | loss: 0.0311692 | val_loss: 0.0311862 | Time: 56666.2 ms
[2022-04-07 05:34:00 main:574] : INFO : Epoch 191 | loss: 0.0311692 | val_loss: 0.0311867 | Time: 4596.53 ms
[2022-04-07 05:35:51 main:574] : INFO : Epoch 192 | loss: 0.0311678 | val_loss: 0.0311855 | Time: 109570 ms
[2022-04-07 05:37:30 main:574] : INFO : Epoch 193 | loss: 0.0311713 | val_loss: 0.0311828 | Time: 98978.8 ms
[2022-04-07 05:39:26 main:574] : INFO : Epoch 194 | loss: 0.0311711 | val_loss: 0.0311809 | Time: 111352 ms
[2022-04-07 05:42:06 main:574] : INFO : Epoch 195 | loss: 0.0311688 | val_loss: 0.0311813 | Time: 154828 ms
[2022-04-07 05:44:35 main:574] : INFO : Epoch 196 | loss: 0.0311695 | val_loss: 0.0311811 | Time: 143398 ms
[2022-04-07 05:44:41 main:574] : INFO : Epoch 197 | loss: 0.0311687 | val_loss: 0.0311818 | Time: 5769.13 ms
[2022-04-07 05:46:48 main:574] : INFO : Epoch 198 | loss: 0.0311679 | val_loss: 0.0311819 | Time: 127493 ms
[2022-04-07 05:50:56 main:574] : INFO : Epoch 199 | loss: 0.0311665 | val_loss: 0.0311836 | Time: 236845 ms
[2022-04-07 05:53:06 main:574] : INFO : Epoch 200 | loss: 0.03117 | val_loss: 0.0311815 | Time: 130065 ms
[2022-04-07 05:54:00 main:574] : INFO : Epoch 201 | loss: 0.031172 | val_loss: 0.0311822 | Time: 53596.8 ms
[2022-04-07 05:56:55 main:574] : INFO : Epoch 202 | loss: 0.0311691 | val_loss: 0.0311827 | Time: 174732 ms
[2022-04-07 05:57:45 main:574] : INFO : Epoch 203 | loss: 0.0311672 | val_loss: 0.0311815 | Time: 50692.9 ms
[2022-04-07 05:58:24 main:574] : INFO : Epoch 204 | loss: 0.0311664 | val_loss: 0.0311813 | Time: 38799.3 ms
[2022-04-07 06:02:07 main:574] : INFO : Epoch 205 | loss: 0.0311661 | val_loss: 0.0311825 | Time: 217298 ms
[2022-04-07 06:03:49 main:574] : INFO : Epoch 206 | loss: 0.0311669 | val_loss: 0.0311821 | Time: 102102 ms
[2022-04-07 06:05:51 main:574] : INFO : Epoch 207 | loss: 0.0311672 | val_loss: 0.0311824 | Time: 122270 ms
[2022-04-07 06:08:45 main:574] : INFO : Epoch 208 | loss: 0.0311659 | val_loss: 0.031181 | Time: 168763 ms
[2022-04-07 06:12:17 main:574] : INFO : Epoch 209 | loss: 0.0311652 | val_loss: 0.0311809 | Time: 205428 ms
[2022-04-07 06:15:06 main:574] : INFO : Epoch 210 | loss: 0.0311651 | val_loss: 0.031181 | Time: 169366 ms
[2022-04-07 06:19:05 main:574] : INFO : Epoch 211 | loss: 0.0311645 | val_loss: 0.0311805 | Time: 233941 ms
[2022-04-07 06:20:55 main:574] : INFO : Epoch 212 | loss: 0.0311653 | val_loss: 0.0311823 | Time: 109396 ms
[2022-04-07 06:21:07 main:574] : INFO : Epoch 213 | loss: 0.0311643 | val_loss: 0.0311806 | Time: 12850.7 ms
[2022-04-07 06:22:07 main:574] : INFO : Epoch 214 | loss: 0.0311644 | val_loss: 0.0311812 | Time: 54986.7 ms
[2022-04-07 06:23:57 main:574] : INFO : Epoch 215 | loss: 0.0311662 | val_loss: 0.0311848 | Time: 104437 ms
[2022-04-07 06:27:04 main:574] : INFO : Epoch 216 | loss: 0.0311709 | val_loss: 0.0311852 | Time: 187076 ms
[2022-04-07 06:31:00 main:574] : INFO : Epoch 217 | loss: 0.0311727 | val_loss: 0.031187 | Time: 236221 ms
[2022-04-07 06:32:37 main:574] : INFO : Epoch 218 | loss: 0.0311709 | val_loss: 0.031179 | Time: 92637.4 ms
[2022-04-07 06:32:46 main:574] : INFO : Epoch 219 | loss: 0.0311692 | val_loss: 0.0311788 | Time: 8273.61 ms
[2022-04-07 06:37:02 main:574] : INFO : Epoch 220 | loss: 0.0311674 | val_loss: 0.0311779 | Time: 255156 ms
[2022-04-07 06:38:58 main:574] : INFO : Epoch 221 | loss: 0.0311669 | val_loss: 0.031179 | Time: 116334 ms
[2022-04-07 06:40:27 main:574] : INFO : Epoch 222 | loss: 0.031167 | val_loss: 0.0311801 | Time: 89181.4 ms
[2022-04-07 06:40:35 main:574] : INFO : Epoch 223 | loss: 0.031168 | val_loss: 0.0311804 | Time: 7632.7 ms
[2022-04-07 06:42:45 main:574] : INFO : Epoch 224 | loss: 0.0311693 | val_loss: 0.0311813 | Time: 111713 ms
[2022-04-07 06:44:05 main:574] : INFO : Epoch 225 | loss: 0.0311685 | val_loss: 0.0311793 | Time: 79995.6 ms
[2022-04-07 06:47:36 main:574] : INFO : Epoch 226 | loss: 0.0311665 | val_loss: 0.0311777 | Time: 211397 ms
[2022-04-07 06:50:53 main:574] : INFO : Epoch 227 | loss: 0.0311652 | val_loss: 0.0311771 | Time: 191748 ms
[2022-04-07 06:53:30 main:574] : INFO : Epoch 228 | loss: 0.0311658 | val_loss: 0.0311782 | Time: 151992 ms
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFF273290D2
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.17.0
Dump Timestamp : 04/07/22 06:54:59
Install Directory : C:\Program Files\BOINC\
Data Directory : C:\ProgramData\BOINC
Project Symstore :
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\16;C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome
ModLoad: 000000002c280000 00000000003df000 C:\ProgramData\BOINC\projects\www.mlcathome.org_mlcathome\mlds-gpu_9.75_windows-x86_64__cuda10200.exe (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000294d0000 00000000001f5000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000028900000 00000000000be000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000027260000 00000000002c8000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 00000000283d0000 00000000001a0000 C:\WINDOWS\System32\USER32.dll (6.2.19041.1503) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000026c90000 0000000000022000 C:\WINDOWS\System32\win32u.dll (6.2.19041.1586) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.19041.1586 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1586
ModLoad: 0000000027740000 000000000002b000 C:\WINDOWS\System32\GDI32.dll (6.2.19041.1202) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.19041.1202 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1202
ModLoad: 0000000026f90000 000000000010b000 C:\WINDOWS\System32\gdi32full.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000026bf0000 000000000009d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 00000000270a0000 0000000000100000 C:\WINDOWS\System32\ucrtbase.dll (6.2.19041.789) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.19041.789 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.789
ModLoad: 0000000028850000 00000000000ae000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000293e0000 000000000009e000 C:\WINDOWS\System32\msvcrt.dll (7.0.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.19041.546
ModLoad: 0000000027770000 000000000009c000 C:\WINDOWS\System32\sechost.dll (6.2.19041.1586) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000027870000 0000000000125000 C:\WINDOWS\System32\RPCRT4.dll (6.2.19041.1466) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000aad10000 000000000abe7000 C:\ProgramData\BOINC\slots\16\torch_cpu.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000019820000 0000000000058000 C:\ProgramData\BOINC\slots\16\c10.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000007af60000 0000000024b49000 C:\ProgramData\BOINC\slots\16\torch_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000007c40000 0000000000092000 C:\WINDOWS\SYSTEM32\MSVCP140.dll (14.28.29334.0) (-exported- Symbols Loaded)
Linked PDB Filename : D:\a01\_work\6\s\\binaries\amd64ret\bin\amd64\\msvcp140.amd64.pdb
File Version : 14.28.29334.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29334.0
ModLoad: 000000000cd10000 0000000000019000 C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (14.28.29334.0) (-exported- Symbols Loaded)
Linked PDB Filename : D:\a01\_work\6\s\\binaries\amd64ret\bin\amd64\\vcruntime140.amd64.pdb
File Version : 14.28.29334.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29334.0
ModLoad: 000000000d570000 000000000000c000 C:\WINDOWS\SYSTEM32\VCRUNTIME140_1.dll (14.28.29334.0) (-exported- Symbols Loaded)
Linked PDB Filename : D:\a01\_work\6\s\\binaries\amd64ret\bin\amd64\\vcruntime140_1.amd64.pdb
File Version : 14.28.29334.0 built by: vcwrkspc
Company Name : Microsoft Corporation
Product Name : Microsoft® Visual Studio®
Product Version : 14.28.29334.0
ModLoad: 000000001a230000 00000000001e4000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.19041.867) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.19041.867 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.867
ModLoad: 00000000ca300000 00000000002db000 C:\ProgramData\BOINC\slots\16\fbgemm.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 0000000004700000 0000000000045000 C:\ProgramData\BOINC\slots\16\c10_cuda.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000e4610000 000000000020d000 C:\ProgramData\BOINC\slots\16\libiomp5md.dll (5.0.2020.205) (-exported- Symbols Loaded)
Linked PDB Filename : O:\promo\20200205\tmp\win_32e-rtl_int_5_nor_dyn.rel.c0.s0.t1..h1.w1-FXILAB103\libiomp5md.pdb
File Version : 20200205
Company Name : Intel Corporation
Product Name : Intel(R) OpenMP* Runtime Library
Product Version : 5.0
ModLoad: 00000000bd1e0000 00000000034b2000 C:\ProgramData\BOINC\slots\16\curand64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA RAND Library
Product Version : 6,14,11,1012
ModLoad: 00000000219e0000 0000000000010000 C:\ProgramData\BOINC\slots\16\nvToolsExt64_1.dll (-exported- Symbols Loaded)
Linked PDB Filename : D:\bld\lib\nvtx\v1\_bin\win32_x64_release\nvToolsExt64_1.pdb
ModLoad: 00000000ba130000 00000000030a5000 C:\ProgramData\BOINC\slots\16\cusparse64_10.dll (6.14.11.1031) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1031
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA SPARSE BLAS Library
Product Version : 6,14,11,1031
ModLoad: 000000005fbb0000 000000001b3a4000 C:\ProgramData\BOINC\slots\16\cudnn64_7.dll (6.14.11.10020) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,10020
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.86 CUDNN Library
Product Version : 6,14,11,10020
ModLoad: 0000000006160000 0000000000040000 C:\ProgramData\BOINC\slots\16\asmjit.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 000000005be50000 0000000003d5b000 C:\ProgramData\BOINC\slots\16\cublas64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 00000000b6e20000 0000000001eca000 C:\ProgramData\BOINC\slots\16\cublasLt64_10.dll (6.14.11.1022) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1022
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA BLAS Library
Product Version : 6,14,11,1022
ModLoad: 0000000052cb0000 000000000919f000 C:\ProgramData\BOINC\slots\16\cufft64_10.dll (6.14.11.1012) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6,14,11,1012
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA FFT Library
Product Version : 6,14,11,1012
ModLoad: 0000000028710000 0000000000030000 C:\WINDOWS\System32\IMM32.DLL (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000024a40000 0000000000012000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 00000000289c0000 0000000000470000 C:\WINDOWS\System32\Setupapi.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : setupapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000271a0000 000000000004e000 C:\WINDOWS\System32\cfgmgr32.dll (6.2.19041.1151) (-exported- Symbols Loaded)
Linked PDB Filename : cfgmgr32.pdb
File Version : 10.0.19041.1151 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1151
ModLoad: 0000000026cc0000 0000000000027000 C:\WINDOWS\System32\bcrypt.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000269b0000 0000000000033000 C:\WINDOWS\SYSTEM32\DEVOBJ.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : devobj.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 00000000271f0000 0000000000068000 C:\WINDOWS\System32\WINTRUST.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : wintrust.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 0000000026d80000 0000000000156000 C:\WINDOWS\System32\CRYPT32.dll (6.2.19041.1320) (-exported- Symbols Loaded)
Linked PDB Filename : crypt32.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000267d0000 0000000000012000 C:\WINDOWS\SYSTEM32\MSASN1.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : msasn1.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000027a50000 0000000000744000 C:\WINDOWS\System32\Shell32.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : shell32.pdb
File Version : 10.0.19041.964 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.964
ModLoad: 0000000024d20000 0000000000794000 C:\WINDOWS\SYSTEM32\windows.storage.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : Windows.Storage.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000028e30000 0000000000354000 C:\WINDOWS\System32\combase.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : combase.pdb
File Version : 10.0.19041.1320 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1320
ModLoad: 0000000026650000 000000000002c000 C:\WINDOWS\SYSTEM32\Wldp.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : WLDP.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 0000000027690000 00000000000ad000 C:\WINDOWS\System32\SHCORE.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : shcore.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000275a0000 0000000000055000 C:\WINDOWS\System32\shlwapi.dll (6.2.19041.1023) (-exported- Symbols Loaded)
Linked PDB Filename : shlwapi.pdb
File Version : 10.0.19041.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1
ModLoad: 00000000c9340000 00000000002cd000 C:\WINDOWS\system32\nvcuda.dll (30.0.15.1165) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_37\drivers\gpgpu\cuda\loader\_out\wddm2_amd64_release\nvcuda_loader.pdb
File Version : 30.0.15.1165
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.6.99 driver
Product Version : 30.0.15.1165
ModLoad: 00000000a7620000 00000000016a0000 C:\WINDOWS\system32\DriverStore\FileRepository\nvhdc.inf_amd64_d11413f6fd8fe259\nvcuda64.dll (30.0.15.1165) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_37\drivers\gpgpu\_out\wddm2_amd64_release\nvcuda.pdb
File Version : 30.0.15.1165
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 11.6.99 driver
Product Version : 30.0.15.1165
ModLoad: 00000000216e0000 000000000000a000 C:\WINDOWS\SYSTEM32\VERSION.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 000000001dc70000 0000000000031000 C:\WINDOWS\SYSTEM32\cryptnet.dll (6.2.19041.906) (-exported- Symbols Loaded)
Linked PDB Filename : cryptnet.pdb
File Version : 10.0.19041.906 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.906
ModLoad: 0000000018c60000 0000000000148000 C:\WINDOWS\SYSTEM32\drvstore.dll (6.2.19041.1566) (-exported- Symbols Loaded)
Linked PDB Filename : drvstore.pdb
File Version : 10.0.19041.1566 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1566
ModLoad: 00000000265c0000 000000000000c000 C:\WINDOWS\SYSTEM32\cryptbase.dll (6.2.19041.546) (-exported- Symbols Loaded)
Linked PDB Filename : cryptbase.pdb
File Version : 10.0.19041.546 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.546
ModLoad: 0000000052540000 0000000000768000 C:\WINDOWS\SYSTEM32\nvapi64.dll (30.0.15.1165) (-exported- Symbols Loaded)
Linked PDB Filename : C:\dvs\p4\build\sw\rel\gpu_drv\r510\r511_37\drivers\nvapi\gpu\_out\wddm2_amd64_release\nvapi64.pdb
File Version : 30.0.15.1165
Company Name : NVIDIA Corporation
Product Name : NVIDIA Windows drivers
Product Version : 30.0.15.1165
ModLoad: 0000000019dc0000 0000000000009000 C:\ProgramData\BOINC\slots\16\caffe2_nvrtc.dll (-exported- Symbols Loaded)
Linked PDB Filename :
ModLoad: 00000000a9d40000 0000000000fca000 C:\ProgramData\BOINC\slots\16\nvrtc64_102_0.dll (6.14.11.9000) (-exported- Symbols Loaded)
Linked PDB Filename :
File Version : 6.14.11.9000
Company Name : NVIDIA Corporation
Product Name : NVIDIA CUDA 10.2.89 NVRTC Library
Product Version : 6.14.11.9000
ModLoad: 0000000026cf0000 0000000000082000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.19041.1415) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.19041.1415 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.19041.1415
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 3848, Write: 3051, Other 8270
- I/O Transfers Counters -
Read: 7543824, Write: 6891468, Other 57336
- Paged Pool Usage -
QuotaPagedPoolUsage: 3533544, QuotaPeakPagedPoolUsage: 3533816
QuotaNonPagedPoolUsage: 58336, QuotaPeakNonPagedPoolUsage: 58744
- Virtual Memory Usage -
VirtualSize: 399704064, PeakVirtualSize: 1016348672
- Pagefile Usage -
PagefileUsage: 399704064, PeakPagefileUsage: 399704064
- Working Set Size -
WorkingSetSize: 1967140864, PeakWorkingSetSize: -2055716864, PageFaultCount: 841116
*** Dump of thread ID 19956 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 160409632.000000
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFF273290D2
- Registers -
rax=0000000000000000 rbx=0000000000000001 rcx=000000002c538c40 rdx=00000000da1fe870 rsi=0000000000000000 rdi=0000000000000000
r8=00000000da1fe870 r9=000000002c538c30 r10=0000000000000fff r11=0000000000000ff0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000273290d2 rsp=00000000da1fe848 rbp=0000000000000000
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
da1fe840 2c2b9100 00000001 da1fe870 da1fe870 2c538c30 KERNELBASE!DebugBreak+0x0
da1fec80 2c2b9ce4 cd800000 da1fefe4 00000000 00000324 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
da1ffbe0 2c2ba084 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
da1ffc10 28917034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
da1ffc40 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
da1ffcc0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 8004 (state: Ready): ***
- Information -
Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 221562496.000000, User Time: 4798593536.000000, Wait Time: 160409344.000000
- Registers -
rax=0000000000000382 rbx=00000000bc2e93b0 rcx=000000003cc41b80 rdx=000000005155b5a0 rsi=00000000d9cdfa80 rdi=000000003cc41b80
r8=00000000d9cdfa80 r9=000000003c1e68a0 r10=0000000000000000 r11=000000005155b58c r12=0000000000000001 r13=0000000000000000
r14=0000000000000088 r15=000000003cc0fc40 rip=00000000a7879c0d rsp=00000000d9cdf9c0 rbp=000000003c4c6620
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000206
- Callstack -
ChildEBP RetAddr Args to Child
d9cdfa10 a76947ec bc2e93b0 3cc41b80 3cc41b80 3b8c4520 nvcuda64!cuProfilerStop+0x0
d9cdfa70 a798c6cb ad647700 5155b5a0 bc2e93b0 3c783930 nvcuda64!+0x0
d9cdfab0 a77b1815 3c4c6620 5155b58c d9cdfbc0 3c4c6620 nvcuda64!cuProfilerStop+0x0
d9ce00f0 a7962e95 d9ce0170 d9ce0170 d9ce0170 d9ce0170 nvcuda64!+0x0
d9ce0230 a765c464 8cd08d20 d9ce04a0 8cd08d20 d9ce06c0 nvcuda64!cuProfilerStop+0x0
d9ce0300 a765ca57 d9ce0410 3c4c6620 d9ce0410 8cd08d20 nvcuda64!+0x0
d9ce0670 a785c8bc 00000001 00004000 60bd0b00 60954de7 nvcuda64!+0x0
d9ce07e0 609529b6 00000000 00000001 00000100 60bd0b00 nvcuda64!cuEventRecordWithFlags+0x0
d9ce08f0 6094fdf4 60bd0b00 3ac7ffd0 8cd08d20 d9ce0b10 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d9ce0a70 60bbbb21 60bd0b00 00000000 60bd0b00 00000000 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d9ce0be0 60abde74 3c290510 3d75d800 3c200000 00000090 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d9ce0e10 60acf99d 00000000 61dc1400 00000000 60bd0b00 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d9ce0e70 60a9703b 00000000 00000000 3c290510 d9ce0f80 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d9ce1300 609b43bc 00000000 d9ce13ec 00000000 bc2d1af0 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d9ce15d0 609b700e 00000000 ce4dce20 00000006 00000001 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d9ce16c0 608b82ca bc2d1af0 00000000 00000000 00000024 cudnn64_7!cudnnSetDropoutDescriptor+0x0
d9ce1790 608b62b3 3c212000 00000024 d9ce19d0 000000eb cudnn64_7!cudnnSetAttnDescriptor+0x0
d9ce18e0 608b9536 3c212000 220aadc0 00000000 00000000 cudnn64_7!cudnnSetAttnDescriptor+0x0
d9ce19f0 608cb06f 000000ec 3c275000 00000000 00000000 cudnn64_7!cudnnSetAttnDescriptor+0x0
d9ce1ea0 608beff1 d9ce4188 220aadc0 00000005 cd879370 cudnn64_7!cudnnSetRNNProjectionLayers+0x0
d9ce2040 608c7d8e cf120200 00000000 cd879370 d9ce2308 cudnn64_7!cudnnSetRNNProjectionLayers+0x0
d9ce21e0 608ddcd3 00000000 00000000 cee12d20 d9ce2300 cudnn64_7!cudnnSetRNNProjectionLayers+0x0
d9ce4370 7dcd89fd cf12ace0 3c200000 cf12ace0 220aadc0 cudnn64_7!cudnnRNNForwardTraining+0x0
d9ce4850 7dd325ea d9ce4918 00000001 d9ce4918 00000001 torch_cuda!at::native::_cudnn_rnn+0x0
d9ce4930 7dd40cbf d9ce5020 d9ce6828 d9ce5970 294fb86b torch_cuda!at::native::set_storage_cuda_+0x0
d9ce4a10 b075b9fc d9ce5020 00000000 bc33e4d0 bc33e4a0 torch_cuda!at::native::set_storage_cuda_+0x0
d9ce4b90 b07931a2 00000001 b1f067c2 d9ce4bc4 294fb86b torch_cpu!at::native::mkldnn_sigmoid_+0x0
d9ce4e70 b1a42613 00000000 00000000 00000003 b0d9aca8 torch_cpu!at::_cudnn_rnn+0x0
d9ce5370 b1a5080b 1c9adc41 cda00d90 d9ce54a0 00000004 torch_cpu!torch::autograd::NotImplemented::~NotImplemented+0x0
d9ce5450 b075b9fc d9ce59d0 b1b75878 d9ce55b0 a7647a77 torch_cpu!torch::autograd::GraphRoot::apply+0x0
d9ce55d0 b07931a2 cd879110 00000000 1c9ad9d1 7ebfa3f0 torch_cpu!at::native::mkldnn_sigmoid_+0x0
d9ce58b0 7dcd69a1 d9ce0001 00000001 00000004 00000004 torch_cpu!at::_cudnn_rnn+0x0
d9ce5a80 7dce0117 00000000 d9ce6828 04702e0d d9ce5ba0 torch_cuda!at::native::_use_cudnn_ctc_loss+0x0
d9ce5b10 b04e107b b5631ac0 d9ce5e30 94bb2437 a7817783 torch_cuda!at::native::_cudnn_rnn_flatten_weight+0x0
d9ce5d80 b0513b16 00000001 d9ce6828 00000080 cf397328 torch_cpu!at::native::fbgemm_pack_quantized_matrix+0x0
d9ce5f10 b088ae89 d9ce5f88 00000001 d9ce5f88 00000001 torch_cpu!at::native::gru+0x0
d9ce5fa0 b1b79fa3 d9ce60a0 00000001 d9ce6220 d9ce6828 torch_cpu!at::zeros_out+0x0
d9ce6030 b087dd35 19871ea8 3bb6f170 00000000 19871ea8 torch_cpu!torch::autograd::GraphRoot::apply+0x0
d9ce60b0 b07588ec b56375c8 21ecb120 ce4dcf50 d9ce62d0 torch_cpu!at::zeros_out+0x0
d9ce61b0 b07cc1bb 0000000c 00000080 00000004 cda00200 torch_cpu!at::native::mkldnn_sigmoid_+0x0
d9ce6230 b22b7294 d9ce6350 21ecb120 d9ce6340 d9ce64e8 torch_cpu!at::gru+0x0
d9ce6460 b22b62ab d9ce6800 b06e030e d9ce6800 d9ce6828 torch_cpu!torch::nn::GRUImpl::forward_helper+0x0
d9ce6550 2c292621 0000000e d9ce7260 00000000 19871ea8 torch_cpu!torch::nn::GRUImpl::forward+0x0
d9ce6750 2c29705a 00000000 d9ce0000 d9ce0601 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::error+0x0
d9ce6c50 2c29b950 2c2cd95f 2c2cd95f 00000000 065e8de0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::then+0x0
d9cff9f0 2c4dc6f8 3c0104c0 00000000 00000000 3bfe54f0 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
d9cffa30 28917034 00000000 00000000 00000000 00000000 mlds-gpu_9.75_windows-x86_64__c!c10::ivalue::Future::wait+0x0
d9cffa60 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
d9cffae0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 6272 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 156250.000000, Wait Time: 160409616.000000
- Registers -
rax=000000000000005b rbx=000000000000000e rcx=000000000000000e rdx=00000000da2ff4e0 rsi=0000000000000000 rdi=000000000000000e
r8=000000003c4c4c70 r9=000000003c22f500 r10=0000000000000000 r11=00000000da2ff450 r12=0000000000000064 r13=00000000da2ff4e0
r14=00000000da2ff1e0 r15=0000000000000000 rip=000000002956d8c4 rsp=00000000da2ff188 rbp=000000003c22f530
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
da2ff180 272acb20 00000000 d9a7d000 2963a4d0 294fb86b ntdll!ZwWaitForMultipleObjects+0x0
da2ff470 272aca1e 3a570820 004e0049 0000000e 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x0
da2ff4b0 a77fd554 da2ff568 3c4c8cd8 3c4c8cd8 00001880 KERNELBASE!WaitForMultipleObjects+0x0
da2ff6f0 a78a58cd 00000000 3c22f530 00000000 00000000 nvcuda64!+0x0
da2ff7a0 a77fcf23 abe89c99 fb42f7ce 3c8159f0 00000000 nvcuda64!cuProfilerStop+0x0
da2ff7d0 a7b344c4 3c8159f0 00000000 00000000 00000000 nvcuda64!+0x0
da2ff800 28917034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
da2ff830 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
da2ff8b0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 2164 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 158466576.000000
- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=0000000000000298 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000298
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=0000000000000000 r12=000000003c1e4f40 r13=0000000000000000
r14=0000000000000298 r15=0000000000000000 rip=000000002956cdf4 rsp=00000000da3ff818 rbp=0000000000000000
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
da3ff810 27281a5e 00000000 00001f80 00000000 00000000 ntdll!ZwWaitForSingleObject+0x0
da3ff8b0 a7a25d0e 3c815a50 00000000 00000000 00000298 KERNELBASE!WaitForSingleObjectEx+0x0
da3ffaf0 a77fcf23 00000000 3c815a50 a7620000 3c24c6e0 nvcuda64!cuProfilerStop+0x0
da3ffb20 a7b344c4 3c815a50 00000000 00000000 00000000 nvcuda64!+0x0
da3ffb50 28917034 00000000 00000000 00000000 00000000 nvcuda64!cuProfilerStop+0x0
da3ffb80 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
da3ffc00 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 6060 (state: Waiting): ***
- Information -
Status: Wait Reason: Unknown, , Kernel Time: 132812496.000000, User Time: 3108124928.000000, Wait Time: 160409184.000000
- Registers -
rax=00000000000001d0 rbx=0000000000000000 rcx=00000000b0adad18 rdx=0000000000000000 rsi=0000000000000000 rdi=00000000b0adad18
r8=0000000049b4e2e8 r9=000000000010097c r10=0000000000000008 r11=00000000da4fbe50 r12=0000000000000000 r13=0000000000000000
r14=00000000da4ff330 r15=00000000b0adacc8 rip=0000000029570764 rsp=00000000da4ff308 rbp=00000000da4ff360
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
da4ff300 29534021 bc9c83e8 07c52f97 bc682c20 bc682c10 ntdll!NtWaitForAlertByThreadId+0x0
da4ff380 272ccfa9 b0adad10 b0adad08 b0adacc0 00000000 ntdll!RtlSleepConditionVariableSRW+0x0
da4ff3c0 07c7bf3d bc682c40 00000000 00000000 b1ef05b4 KERNELBASE!SleepConditionVariableSRW+0x0
da4ff400 07c52798 00000000 bc682c40 00000000 00000000 MSVCP140!__crtSetThreadpoolWait+0x0
da4ff430 07c52a4a 8336d910 00000000 da4ff640 b0adad08 MSVCP140!std::_Winerror_message+0x0
da4ff460 b1ef064b b0adacc0 da4ff560 b0adacc0 bc682c20 MSVCP140!_Cnd_wait+0x0
da4ff4a0 b1ef25b1 cf406400 ffffffff bc682c20 ffffffff torch_cpu!torch::utils::Future<std::vector<at::Tensor,std::allocator<at::Tensor> > >::markCompletedInternal+0x0
da4ff790 b1ef2471 b5416d10 da4ff7d0 b5416d10 00000001 torch_cpu!torch::autograd::Engine::thread_main+0x0
da4ff7e0 b1ee9744 00080001 ceeb91b0 00000000 00000000 torch_cpu!torch::autograd::Engine::thread_init+0x0
da4ff810 270c1bb2 ceeb91b0 00000000 00000000 00000000 torch_cpu!torch::autograd::Engine::get_base_engine+0x0
da4ff840 28917034 00000000 00000000 00000000 00000000 ucrtbase!_configthreadlocale+0x0
da4ff870 29522651 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
da4ff8f0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
</stderr_txt>
]]>
©2022 MLC@Home Team
A project of the Cognition, Robotics, and Learning (CORAL) Lab at the University of Maryland, Baltimore County (UMBC)