Jul 03 19:36:27 fedora kernel: NVRM: _kgspLogXid119: ********************************* GSP Timeout **********************************
Jul 03 19:36:27 fedora kernel: NVRM: _kgspLogXid119: Note: Please also check logs above.
Jul 03 19:36:27 fedora kernel: NVRM: GPU at PCI:0000:03:00: GPU-be9b4836-5a63-a9ef-5b03-721c54957016
Jul 03 19:36:27 fedora kernel: NVRM: GPU Board Serial Number: 0
Jul 03 19:36:27 fedora kernel: NVRM: Xid (PCI:0000:03:00): 119, pid=3182, name=memtest_vulkan, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) (0x2080012b 0x230).
Jul 03 19:36:27 fedora kernel: NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) and data 0x000000002080012b 0x0000000000000230.
Jul 03 19:36:27 fedora kernel: NVRM: GPU0 RPC history (CPU -> GSP):
Jul 03 19:36:27 fedora kernel: NVRM: entry function data0 data1 ts_start ts_end duration actively_polling
Jul 03 19:36:27 fedora kernel: NVRM: 0 76 GSP_RM_CONTROL 0x000000002080012b 0x0000000000000230 0x000639059ff19625 0x0000000000000000 y
Jul 03 19:36:27 fedora kernel: NVRM: -1 103 GSP_RM_ALLOC 0x0000000000009072 0x000000000000000c 0x000639059ff193a8 0x000639059ff19516 366us
Jul 03 19:36:27 fedora kernel: NVRM: -2 76 GSP_RM_CONTROL 0x0000000020800a5d 0x0000000000000008 0x000639059ff192d6 0x000639059ff193a2 204us
Jul 03 19:36:27 fedora kernel: NVRM: -3 103 GSP_RM_ALLOC 0x0000000000009072 0x000000000000000c 0x000639059ff19176 0x000639059ff192c8 338us
Jul 03 19:36:27 fedora kernel: NVRM: -4 76 GSP_RM_CONTROL 0x0000000020800a5d 0x0000000000000008 0x000639059ff19090 0x000639059ff19170 224us
Jul 03 19:36:27 fedora kernel: NVRM: -5 103 GSP_RM_ALLOC 0x0000000000009072 0x000000000000000c 0x000639059ff18f28 0x000639059ff19081 345us
Jul 03 19:36:27 fedora kernel: NVRM: -6 76 GSP_RM_CONTROL 0x0000000020800a5d 0x0000000000000008 0x000639059ff18e42 0x000639059ff18f24 226us
Jul 03 19:36:27 fedora kernel: NVRM: -7 103 GSP_RM_ALLOC 0x0000000000009072 0x000000000000000c 0x000639059ff18c5f 0x000639059ff18e34 469us
Jul 03 19:36:27 fedora kernel: NVRM: GPU0 RPC event history (CPU <- GSP):
Jul 03 19:36:27 fedora kernel: NVRM: entry function data0 data1 ts_start ts_end duration during_incomplete_rpc
Jul 03 19:36:27 fedora kernel: NVRM: 0 4099 POST_EVENT 0x00000000000000a2 0x0000000000000000 0x000639055f5df98a 0x000639055f5df9a8 30us
Jul 03 19:36:27 fedora kernel: NVRM: -1 4099 POST_EVENT 0x00000000000000a2 0x0000000000000000 0x000639055f4eb18c 0x000639055f4eb199 13us
Jul 03 19:36:27 fedora kernel: NVRM: -2 4099 POST_EVENT 0x00000000000000a2 0x0000000000000000 0x000639055f3f699e 0x000639055f3f69a9 11us
Jul 03 19:36:27 fedora kernel: NVRM: -3 4099 POST_EVENT 0x00000000000000a2 0x0000000000000000 0x000639055f3021ab 0x000639055f3021be 19us
Jul 03 19:36:27 fedora kernel: NVRM: -4 4099 POST_EVENT 0x00000000000000a2 0x0000000000000000 0x000639055f20d9cb 0x000639055f20d9d8 13us
Jul 03 19:36:27 fedora kernel: NVRM: -5 4099 POST_EVENT 0x00000000000000a2 0x0000000000000000 0x000639055f1191e4 0x000639055f1191f1 13us
Jul 03 19:36:27 fedora kernel: NVRM: -6 4099 POST_EVENT 0x00000000000000a2 0x0000000000000000 0x000639055f0249e6 0x000639055f0249f3 13us
Jul 03 19:36:27 fedora kernel: NVRM: -7 4099 POST_EVENT 0x00000000000000a2 0x0000000000000000 0x000639055ef301e9 0x000639055ef301f7 14us
Jul 03 19:36:27 fedora kernel: CPU: 31 UID: 1000 PID: 3182 Comm: memtest_vulkan Tainted: G S OE 6.15.3-200.fc42.x86_64 #1 PREEMPT(lazy)
Jul 03 19:36:27 fedora kernel: Tainted: [S]=CPU_OUT_OF_SPEC, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Jul 03 19:36:27 fedora kernel: Hardware name: Xioaxi X99-Special /X99 Taichi, BIOS P1.80 04/06/2018
Jul 03 19:36:27 fedora kernel: Call Trace:
Jul 03 19:36:27 fedora kernel: <TASK>
Jul 03 19:36:27 fedora kernel: dump_stack_lvl+0x5d/0x80
Jul 03 19:36:27 fedora kernel: _kgspRpcRecvPoll+0x593/0x760 [nvidia]
Jul 03 19:36:27 fedora kernel: _issueRpcAndWait+0xd2/0x900 [nvidia]
Jul 03 19:36:27 fedora kernel: ? osGetCurrentThread+0x26/0x60 [nvidia]
Jul 03 19:36:27 fedora kernel: rpcRmApiControl_GSP+0x76f/0x940 [nvidia]
Jul 03 19:36:27 fedora kernel: ? _tlsThreadEntryGet+0x82/0x90 [nvidia]
Jul 03 19:36:27 fedora kernel: ? osGetCurrentThread+0x26/0x60 [nvidia]
Jul 03 19:36:27 fedora kernel: rmresControl_Prologue_IMPL+0xd4/0x1e0 [nvidia]
Jul 03 19:36:27 fedora kernel: resControl_IMPL+0xd6/0x1b0 [nvidia]
Jul 03 19:36:27 fedora kernel: ? _tlsEntryAcquire+0x29/0xd0 [nvidia]
Jul 03 19:36:27 fedora kernel: serverControl+0x47e/0x590 [nvidia]
Jul 03 19:36:27 fedora kernel: _rmapiRmControl+0x544/0x820 [nvidia]
Jul 03 19:36:27 fedora kernel: rmapiControlWithSecInfo+0x79/0x140 [nvidia]
Jul 03 19:36:27 fedora kernel: rmapiControl+0x24/0x40 [nvidia]
Jul 03 19:36:27 fedora kernel: kgrobjPromoteContext_IMPL+0x2e8/0x350 [nvidia]
Jul 03 19:36:27 fedora kernel: kgrobjConstruct_IMPL+0x27a/0x480 [nvidia]
Jul 03 19:36:27 fedora kernel: __nvoc_objCreate_KernelGraphicsObject+0x132/0x240 [nvidia]
Jul 03 19:36:27 fedora kernel: __nvoc_objCreateDynamic+0x4a/0x70 [nvidia]
Jul 03 19:36:27 fedora kernel: ? _portMemAllocNonPagedUntracked+0x2c/0x40 [nvidia]
Jul 03 19:36:27 fedora kernel: ? os_alloc_mem+0x104/0x120 [nvidia]
Jul 03 19:36:27 fedora kernel: resservResourceFactory+0xc5/0x240 [nvidia]
Jul 03 19:36:27 fedora kernel: ? _tlsEntryAcquire+0x93/0xd0 [nvidia]
Jul 03 19:36:27 fedora kernel: _clientAllocResourceHelper+0x2aa/0x660 [nvidia]
Jul 03 19:36:27 fedora kernel: ? _tlsThreadEntryGet+0x82/0x90 [nvidia]
Jul 03 19:36:27 fedora kernel: ? tlsEntryGet+0x31/0x70 [nvidia]
Jul 03 19:36:27 fedora kernel: serverAllocResourceUnderLock+0x33b/0xa10 [nvidia]
Jul 03 19:36:27 fedora kernel: ? portSyncSpinlockAcquire+0x18/0x30 [nvidia]
Jul 03 19:36:27 fedora kernel: ? portThreadGetCurrentThreadId+0x1d/0x30 [nvidia]
Jul 03 19:36:27 fedora kernel: ? os_acquire_rwlock_write+0x2b/0x40 [nvidia]
Jul 03 19:36:27 fedora kernel: ? portThreadGetCurrentThreadId+0x1d/0x30 [nvidia]
Jul 03 19:36:27 fedora kernel: ? rmclientValidateLocks_IMPL+0x21/0x90 [nvidia]
Jul 03 19:36:27 fedora kernel: ? _serverLockClientWithLockInfo.constprop.0+0x106/0x260 [nvidia]
Jul 03 19:36:27 fedora kernel: serverAllocResource+0x2b4/0x5c0 [nvidia]
Jul 03 19:36:27 fedora kernel: rmapiAllocWithSecInfo+0x1f0/0x420 [nvidia]
Jul 03 19:36:27 fedora kernel: rmapiAllocWithSecInfoTls+0x65/0x90 [nvidia]
Jul 03 19:36:27 fedora kernel: Nv04AllocWithAccessSecInfo+0x6f/0x80 [nvidia]
Jul 03 19:36:27 fedora kernel: ? security_capable+0x50/0x150
Jul 03 19:36:27 fedora kernel: RmIoctl+0xac3/0xda0 [nvidia]
Jul 03 19:36:27 fedora kernel: ? os_acquire_spinlock+0x12/0x30 [nvidia]
Jul 03 19:36:27 fedora kernel: ? portSyncSpinlockAcquire+0x18/0x30 [nvidia]
Jul 03 19:36:27 fedora kernel: rm_ioctl+0x66/0x4f0 [nvidia]
Jul 03 19:36:27 fedora kernel: nvidia_ioctl.isra.0+0x450/0x810 [nvidia]
Jul 03 19:36:27 fedora kernel: nvidia_unlocked_ioctl+0x1d/0x30 [nvidia]
Jul 03 19:36:27 fedora kernel: __x64_sys_ioctl+0x97/0xc0
Jul 03 19:36:27 fedora kernel: do_syscall_64+0x7b/0x160
Jul 03 19:36:27 fedora kernel: ? do_syscall_64+0x87/0x160
Jul 03 19:36:27 fedora kernel: ? nvidia_unlocked_ioctl+0x1d/0x30 [nvidia]
Jul 03 19:36:27 fedora kernel: ? __x64_sys_ioctl+0x97/0xc0
Jul 03 19:36:27 fedora kernel: ? syscall_exit_to_user_mode+0x10/0x210
Jul 03 19:36:27 fedora kernel: ? do_syscall_64+0x87/0x160
Jul 03 19:36:27 fedora kernel: ? nvidia_unlocked_ioctl+0x1d/0x30 [nvidia]
Jul 03 19:36:27 fedora kernel: ? __x64_sys_ioctl+0x97/0xc0
Jul 03 19:36:27 fedora kernel: ? syscall_exit_to_user_mode+0x10/0x210
Jul 03 19:36:27 fedora kernel: ? do_syscall_64+0x87/0x160
Jul 03 19:36:27 fedora kernel: ? filp_flush+0x5b/0x80
Jul 03 19:36:27 fedora kernel: ? syscall_exit_to_user_mode+0x10/0x210
Jul 03 19:36:27 fedora kernel: ? do_syscall_64+0x87/0x160
Jul 03 19:36:27 fedora kernel: ? syscall_exit_to_user_mode+0x10/0x210
Jul 03 19:36:27 fedora kernel: ? do_syscall_64+0x87/0x160
Jul 03 19:36:27 fedora kernel: ? exc_page_fault+0x7e/0x1a0
Jul 03 19:36:27 fedora kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 03 19:36:27 fedora kernel: RIP: 0033:0x7fc2ec30eaad
Jul 03 19:36:27 fedora kernel: Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
Jul 03 19:36:27 fedora kernel: RSP: 002b:00007ffed0b757f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Jul 03 19:36:27 fedora kernel: RAX: ffffffffffffffda RBX: 0000000000000030 RCX: 00007fc2ec30eaad
Jul 03 19:36:27 fedora kernel: RDX: 00007ffed0b75950 RSI: 00000000c030462b RDI: 000000000000000b
Jul 03 19:36:27 fedora kernel: RBP: 00007ffed0b75840 R08: 00007ffed0b75950 R09: 00007ffed0b75978
Jul 03 19:36:27 fedora kernel: R10: 00007fc2d5b17b54 R11: 0000000000000246 R12: 000000000000000b
Jul 03 19:36:27 fedora kernel: R13: 00000000c030462b R14: 000000000000002b R15: 00007ffed0b75850
Jul 03 19:36:27 fedora kernel: </TASK>
Jul 03 19:36:27 fedora kernel: NVRM: _kgspLogXid119: ********************************************************************************
Jul 03 19:36:27 fedora kernel: NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 76!
Jul 03 19:36:27 fedora kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kgrobjPromoteContext(pGpu, pKernelGraphicsObject, pKernelGraphics) @ kernel_graphics_object.c:223
Jul 03 19:36:33 fedora kernel: NVRM: Xid (PCI:0000:03:00): 119, pid=3182, name=memtest_vulkan, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 103 (GSP_RM_ALLOC) (0xcab5 0x8).
Jul 03 19:36:33 fedora kernel: NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 103!
Jul 03 19:36:33 fedora kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d0005e; hParent=0xbeef0100; hObject=0xbeefa0b5; hClass=0x0000cab5; paramsSize=0x00000008; paramsStatus=0x00000000; status=0x00000065
Jul 03 19:36:39 fedora kernel: NVRM: Xid (PCI:0000:03:00): 119, pid=3182, name=memtest_vulkan, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 103 (GSP_RM_ALLOC) (0xcab5 0x8).
Jul 03 19:36:39 fedora kernel: NVRM: nvAssertFailedNoLog: Assertion failed: Back to back GSP RPC timeout detected! GPU marked for reset @ kernel_gsp.c:2314
Jul 03 19:36:39 fedora kernel: NVRM: _issueRpcAndWait: rpcRecvPoll timedout for fn 103!
Jul 03 19:36:39 fedora kernel: NVRM: rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d0005e; hParent=0xbeef0101; hObject=0xbeef8500; hClass=0x0000cab5; paramsSize=0x00000008; paramsStatus=0x00000000; status=0x00000065
Jul 03 19:36:45 fedora kernel: NVRM: Rate limiting GSP RPC error prints for GPU at PCI:0000:03:00 (printing 1 of every 30). The GPU likely needs to be reset.
Jul 03 19:36:51 fedora kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_INTERNAL_LOG_OOB_XID, ¶ms, sizeof(params)) @ gpu.c:6468
Jul 03 19:36:51 fedora kernel: NVRM: Xid (PCI:0000:03:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)
Jul 03 19:36:57 fedora kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kgrobjPromoteContext(pGpu, pKernelGraphicsObject, pKernelGraphics) @ kernel_graphics_object.c:223
Jul 03 19:37:09 fedora kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 03 19:37:09 fedora kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 03 19:37:09 fedora kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 03 19:37:21 fedora kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:179
Jul 03 19:37:27 fedora kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:538
Jul 03 19:37:45 fedora kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:179
Jul 03 19:37:51 fedora kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:538