b***@gmail.com
2019-03-26 11:39:57 UTC
There is a kernel level driver installed on a terminal server.It works fine for certain period of time on that terminal sever. later on
that terminal server itself getting into freezed state where noboday can RDP & web console to connect with server. In my case,
CPU is always hitting to 100% in freezed state and i had to hard reboot only by using VM option "power off". After unstalling that driver the terminal server works fine or even responds properly always.Even if it is 100% CPU usage and gets slow but still reponds to the RDP & web console.
That scenario is kind of hard to reproduce it. but still i got successful to fetch complete memory dump out of that machine in that scenario then i analyzed full memory dump using microsoft WinDbg tool. WinDbg tool displayed faulty driver module name and call stack as below
Module Name: MMTEProxy (Installed Driver)
0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
NMI_HARDWARE_FAILURE (80)
This is typically due to a hardware malfunction. The hardware supplier should
be called.
Arguments:
Arg1: 00000000004f4454
Arg2: 0000000000000000
Arg3: 0000000000000000
Arg4: 0000000000000000
Debugging Details:
------------------
KEY_VALUES_STRING: 1
PROCESSES_ANALYSIS: 1
SERVICE_ANALYSIS: 1
STACKHASH_ANALYSIS: 1
TIMELINE_ANALYSIS: 1
DUMP_CLASS: 1
DUMP_QUALIFIER: 402
BUILD_VERSION_STRING: 9600.17415.amd64fre.winblue_r4.141028-1500
SYSTEM_MANUFACTURER: VMware, Inc.
VIRTUAL_MACHINE: VMware
SYSTEM_PRODUCT_NAME: VMware Virtual Platform
SYSTEM_VERSION: None
BIOS_VENDOR: Phoenix Technologies LTD
BIOS_VERSION: 6.00
BIOS_DATE: 04/05/2016
BASEBOARD_MANUFACTURER: Intel Corporation
BASEBOARD_PRODUCT: 440BX Desktop Reference Platform
BASEBOARD_VERSION: None
DUMP_TYPE: 0
BUGCHECK_P1: 4f4454
BUGCHECK_P2: 0
BUGCHECK_P3: 0
BUGCHECK_P4: 0
CPU_COUNT: 2
CPU_MHZ: bb8
CPU_VENDOR: GenuineIntel
CPU_FAMILY: 6
CPU_MODEL: 3e
CPU_STEPPING: 4
CPU_MICROCODE: 6,3e,4,0 (F,M,S,R) SIG: 42C'00000000 (cache) 42C'00000000 (init)
DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT
BUGCHECK_STR: 0x80
PROCESS_NAME: svchost.exe
CURRENT_IRQL: 0
ANALYSIS_SESSION_HOST: INPN01LAP107
ANALYSIS_SESSION_TIME: 03-26-2019 16:30:13.0120
ANALYSIS_VERSION: 10.0.18317.1001 amd64fre
LAST_CONTROL_TRANSFER: from fffff8005ae205b2 to fffff8009a6601a7
STACK_TEXT:
nt!KxWaitForLockOwnerShip+0x27
MMTEProxy!SVSessionLutTranslatePort+0x2c2 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c @ 873]
MMTEProxy!PerformProxySocketRedirection+0xba7 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\filteralebindredirect.c @ 247]
MMTEProxy!TriggerProxyByALERedirectInline+0x244 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\filteralebindredirect.c @ 690]
MMTEProxy!DDProxyBindRedirectClassify+0x537 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\filteralebindredirect.c @ 881]
THREAD_SHA1_HASH_MOD_FUNC: 03f7fb5fd041c46c9b4dff8f1685ccff753d3642
THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 7f4a5e830d38804e610244f134268d53640c97a0
THREAD_SHA1_HASH_MOD: 2a8f232a3e3c38ad2a6b44b0d2253b97c2ac4b2a
FOLLOWUP_IP:
MMTEProxy!SVSessionLutTranslatePort+2c2 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c @ 873]
fffff800`5ae205b2 c644244000 mov byte ptr [rsp+40h],0
FAULT_INSTR_CODE: 402444c6
FAULTING_SOURCE_LINE: c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c
FAULTING_SOURCE_FILE: c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c
FAULTING_SOURCE_LINE_NUMBER: 873
FAULTING_SOURCE_CODE:
No source found for 'c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c'
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: MMTEProxy!SVSessionLutTranslatePort+2c2
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: MMTEProxy
IMAGE_NAME: MMTEProxy.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 5a60d5f0
STACK_COMMAND: .thread ; .cxr ; kb
BUCKET_ID_FUNC_OFFSET: 2c2
FAILURE_BUCKET_ID: 0x80_MMTEProxy!SVSessionLutTranslatePort
BUCKET_ID: 0x80_MMTEProxy!SVSessionLutTranslatePort
PRIMARY_PROBLEM_CLASS: 0x80_MMTEProxy!SVSessionLutTranslatePort
TARGET_TIME: 2019-02-26T11:15:36.000Z
OSBUILD: 9600
OSSERVICEPACK: 0
SERVICEPACK_NUMBER: 0
OS_REVISION: 0
SUITE_MASK: 16
PRODUCT_TYPE: 3
OSPLATFORM_TYPE: x64
OSNAME: Windows 8.1
OSEDITION: Windows 8.1 Server TerminalServer
OS_LOCALE:
USER_LCID: 0
OSBUILD_TIMESTAMP: 2014-10-29 06:08:48
BUILDDATESTAMP_STR: 141028-1500
BUILDLAB_STR: winblue_r4
BUILDOSVER_STR: 6.3.9600.17415.amd64fre.winblue_r4.141028-1500
ANALYSIS_SESSION_ELAPSED_TIME: 685
ANALYSIS_SOURCE: KM
FAILURE_ID_HASH_STRING: km:0x80_MMTEProxy!svsessionluttranslateport
FAILURE_ID_HASH: {c64b7e97-0bf3-daf1-ad95-9f39cbf37a9a}
Followup: MachineOwner
---------
Since i am not expert in kernel level driver development,But i tried to google about driver. Internally it uses the following lock to perform any operation at process table or session table
#Code snippet
PLIST_ENTRY processTableListHead = NULL;
{
....
KLOCK_QUEUE_HANDLE processTableLockHandle;
KLOCK_QUEUE_HANDLE sessionTableLockHandle;
PLIST_ENTRY tempNode = 0;
....
...
KeAcquireInStackQueuedSpinLock(&gProcessTableLock,&processTableLockHandle);
tempNode = processTableListHead;
...
...
..
//Releases lock
KeReleaseInStackQueuedSpinLock(&sessionTableLockHandle);
KeReleaseInStackQueuedSpinLock(&processTableLockHandle);
}
With help of WinDbg tool, What i observed here, Mostly it is failling at source line no where assinging the value to a variables and that variables defined before accuiring the lock. You can see it in above driver code snippet. my driver is a WFP ALE filtered driver. it inspects traffic it works in a multhreaded environment and my driver allocates/freed memory in non-paged pool
And I also checked that there is no any deadlock condition or curretnly held lock by any thread. still i am not getting what causing this issue. whether its lock is not handled properly at code level or some particular situation.
Can you please help me with pointer or direction?
that terminal server itself getting into freezed state where noboday can RDP & web console to connect with server. In my case,
CPU is always hitting to 100% in freezed state and i had to hard reboot only by using VM option "power off". After unstalling that driver the terminal server works fine or even responds properly always.Even if it is 100% CPU usage and gets slow but still reponds to the RDP & web console.
That scenario is kind of hard to reproduce it. but still i got successful to fetch complete memory dump out of that machine in that scenario then i analyzed full memory dump using microsoft WinDbg tool. WinDbg tool displayed faulty driver module name and call stack as below
Module Name: MMTEProxy (Installed Driver)
0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
NMI_HARDWARE_FAILURE (80)
This is typically due to a hardware malfunction. The hardware supplier should
be called.
Arguments:
Arg1: 00000000004f4454
Arg2: 0000000000000000
Arg3: 0000000000000000
Arg4: 0000000000000000
Debugging Details:
------------------
KEY_VALUES_STRING: 1
PROCESSES_ANALYSIS: 1
SERVICE_ANALYSIS: 1
STACKHASH_ANALYSIS: 1
TIMELINE_ANALYSIS: 1
DUMP_CLASS: 1
DUMP_QUALIFIER: 402
BUILD_VERSION_STRING: 9600.17415.amd64fre.winblue_r4.141028-1500
SYSTEM_MANUFACTURER: VMware, Inc.
VIRTUAL_MACHINE: VMware
SYSTEM_PRODUCT_NAME: VMware Virtual Platform
SYSTEM_VERSION: None
BIOS_VENDOR: Phoenix Technologies LTD
BIOS_VERSION: 6.00
BIOS_DATE: 04/05/2016
BASEBOARD_MANUFACTURER: Intel Corporation
BASEBOARD_PRODUCT: 440BX Desktop Reference Platform
BASEBOARD_VERSION: None
DUMP_TYPE: 0
BUGCHECK_P1: 4f4454
BUGCHECK_P2: 0
BUGCHECK_P3: 0
BUGCHECK_P4: 0
CPU_COUNT: 2
CPU_MHZ: bb8
CPU_VENDOR: GenuineIntel
CPU_FAMILY: 6
CPU_MODEL: 3e
CPU_STEPPING: 4
CPU_MICROCODE: 6,3e,4,0 (F,M,S,R) SIG: 42C'00000000 (cache) 42C'00000000 (init)
DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT
BUGCHECK_STR: 0x80
PROCESS_NAME: svchost.exe
CURRENT_IRQL: 0
ANALYSIS_SESSION_HOST: INPN01LAP107
ANALYSIS_SESSION_TIME: 03-26-2019 16:30:13.0120
ANALYSIS_VERSION: 10.0.18317.1001 amd64fre
LAST_CONTROL_TRANSFER: from fffff8005ae205b2 to fffff8009a6601a7
STACK_TEXT:
nt!KxWaitForLockOwnerShip+0x27
MMTEProxy!SVSessionLutTranslatePort+0x2c2 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c @ 873]
MMTEProxy!PerformProxySocketRedirection+0xba7 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\filteralebindredirect.c @ 247]
MMTEProxy!TriggerProxyByALERedirectInline+0x244 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\filteralebindredirect.c @ 690]
MMTEProxy!DDProxyBindRedirectClassify+0x537 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\filteralebindredirect.c @ 881]
THREAD_SHA1_HASH_MOD_FUNC: 03f7fb5fd041c46c9b4dff8f1685ccff753d3642
THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 7f4a5e830d38804e610244f134268d53640c97a0
THREAD_SHA1_HASH_MOD: 2a8f232a3e3c38ad2a6b44b0d2253b97c2ac4b2a
FOLLOWUP_IP:
MMTEProxy!SVSessionLutTranslatePort+2c2 [c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c @ 873]
fffff800`5ae205b2 c644244000 mov byte ptr [rsp+40h],0
FAULT_INSTR_CODE: 402444c6
FAULTING_SOURCE_LINE: c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c
FAULTING_SOURCE_FILE: c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c
FAULTING_SOURCE_LINE_NUMBER: 873
FAULTING_SOURCE_CODE:
No source found for 'c:\users\dkelone\git\MMTE\MMTE\MMTEdriver\sessionlut.c'
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: MMTEProxy!SVSessionLutTranslatePort+2c2
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: MMTEProxy
IMAGE_NAME: MMTEProxy.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 5a60d5f0
STACK_COMMAND: .thread ; .cxr ; kb
BUCKET_ID_FUNC_OFFSET: 2c2
FAILURE_BUCKET_ID: 0x80_MMTEProxy!SVSessionLutTranslatePort
BUCKET_ID: 0x80_MMTEProxy!SVSessionLutTranslatePort
PRIMARY_PROBLEM_CLASS: 0x80_MMTEProxy!SVSessionLutTranslatePort
TARGET_TIME: 2019-02-26T11:15:36.000Z
OSBUILD: 9600
OSSERVICEPACK: 0
SERVICEPACK_NUMBER: 0
OS_REVISION: 0
SUITE_MASK: 16
PRODUCT_TYPE: 3
OSPLATFORM_TYPE: x64
OSNAME: Windows 8.1
OSEDITION: Windows 8.1 Server TerminalServer
OS_LOCALE:
USER_LCID: 0
OSBUILD_TIMESTAMP: 2014-10-29 06:08:48
BUILDDATESTAMP_STR: 141028-1500
BUILDLAB_STR: winblue_r4
BUILDOSVER_STR: 6.3.9600.17415.amd64fre.winblue_r4.141028-1500
ANALYSIS_SESSION_ELAPSED_TIME: 685
ANALYSIS_SOURCE: KM
FAILURE_ID_HASH_STRING: km:0x80_MMTEProxy!svsessionluttranslateport
FAILURE_ID_HASH: {c64b7e97-0bf3-daf1-ad95-9f39cbf37a9a}
Followup: MachineOwner
---------
Since i am not expert in kernel level driver development,But i tried to google about driver. Internally it uses the following lock to perform any operation at process table or session table
#Code snippet
PLIST_ENTRY processTableListHead = NULL;
{
....
KLOCK_QUEUE_HANDLE processTableLockHandle;
KLOCK_QUEUE_HANDLE sessionTableLockHandle;
PLIST_ENTRY tempNode = 0;
....
...
KeAcquireInStackQueuedSpinLock(&gProcessTableLock,&processTableLockHandle);
tempNode = processTableListHead;
...
...
..
//Releases lock
KeReleaseInStackQueuedSpinLock(&sessionTableLockHandle);
KeReleaseInStackQueuedSpinLock(&processTableLockHandle);
}
With help of WinDbg tool, What i observed here, Mostly it is failling at source line no where assinging the value to a variables and that variables defined before accuiring the lock. You can see it in above driver code snippet. my driver is a WFP ALE filtered driver. it inspects traffic it works in a multhreaded environment and my driver allocates/freed memory in non-paged pool
And I also checked that there is no any deadlock condition or curretnly held lock by any thread. still i am not getting what causing this issue. whether its lock is not handled properly at code level or some particular situation.
Can you please help me with pointer or direction?