Help needed for 3 different causes for BSOD

[j.a. harriman]

I haven't done a lot with kernel dumps, but here's what I have:

This server normally sits idle as it's a failover server should the main
server go down. It's only rebooted when patches MS (and others) are applied
(on average 1/ month).

Any help with tracking this down further would be appreciated!

These are the results from the bugcheck analysis by date (using the MINI
dumps):

On 09/17:
0: kd> !analyze -
*******************************************************************************
*
*
* Bugcheck Analysis
*
*

*******************************************************************************

DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is
caused by drivers that have corrupted the system pool. Run the driver
verifier against any new (or suspect) drivers, and if that doesn't turn up
the culprit, then use gflags to enable special pool.
Arguments:
Arg1: 00000000, memory referenced
Arg2: d0000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 8089c4bb, address which referenced memory

Debugging Details:
------------------


BUGCHECK_STR: 0xC5_D0000002

CURRENT_IRQL: 2

FAULTING_IP:
nt!ExDeferredFreePool+1d7
8089c4bb 8937 mov dword ptr [edi],esi

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

PROCESS_NAME: smlogsvc.exe

IRP_ADDRESS: f7f43800

TRAP_FRAME: b8feb918 -- (.trap 0xffffffffb8feb918)
ErrCode = 00000002
eax=f7c03270 ebx=00000000 ecx=000001ff edx=f7c03000 esi=00000000 edi=00000000
eip=8089c4bb esp=b8feb98c ebp=b8feb9c4 iopl=0 nv up ei ng nz ac pe cy
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010297
nt!ExDeferredFreePool+0x1d7:
8089c4bb 8937 mov dword ptr [edi],esi
ds:0023:00000000=????????
Resetting default scope

LAST_CONTROL_TRANSFER: from 8089c4bb to 80836df5

STACK_TEXT:
b8feb918 8089c4bb badb0d00 f7c03000 8083d60c nt!KiTrap0E+0x2a7
b8feb9c4 8089c5a0 808b7600 00000000 f7f43800 nt!ExDeferredFreePool+0x1d7
b8feba1c 808495b0 f7f43800 00000000 82ee04c0 nt!ExFreePoolWithTag+0x57f
b8feba38 80840492 f7f43800 f7f43840 f7dd8950 nt!IopFreeIrp+0xe9
b8feba88 80840ac9 f7f43840 b8febad4 b8febac8 nt!IopCompleteRequest+0x3db
b8febad8 80a84199 00000000 00000000 00000000 nt!KiDeliverApc+0xbb
b8febaf8 80a843d9 f7dd8901 00000000 00000000
hal!HalpDispatchSoftwareInterrupt+0x49
b8febb14 80a84456 00000001 f7dd8900 b8febb40
hal!HalpCheckForSoftwareInterrupt+0x81
b8febb24 8083d60c f7dd8950 f7f43840 808b9900 hal!KfLowerIrql+0x62
b8febb40 80840d7f f7f43840 f7f43800 00000000 nt!KiExitDispatcher+0x130
b8febb60 80840055 f7f43840 82ee04c0 00000000 nt!KeInsertQueueApc+0x57
b8febb94 8090eccd f7f43800 8a7ec030 f7f43800 nt!IopfCompleteRequest+0x200
b8febc3c 80840153 8a7ecd48 f7f43800 f7dd8b58 nt!WmipIoControl+0x71a
b8febc50 8092b50f f7f439b4 82ee04c0 f7f43800 nt!IofCallDriver+0x45
b8febc64 8092b444 8a7ecd48 f7f43800 82ee04c0
nt!IopSynchronousServiceTail+0x10b
b8febd00 8092b564 00000240 00000244 00000000 nt!IopXxxControlFile+0x60f
b8febd34 80833bef 00000240 00000244 00000000 nt!NtDeviceIoControlFile+0x2a
b8febd34 7c82860c 00000240 00000244 00000000 nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
0091f314 00000000 00000000 00000000 00000000 0x7c82860c


STACK_COMMAND: kb

FOLLOWUP_IP:
nt!ExDeferredFreePool+1d7
8089c4bb 8937 mov dword ptr [edi],esi

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nt!ExDeferredFreePool+1d7

FOLLOWUP_NAME: Pool_corruption

IMAGE_NAME: Pool_Corruption

DEBUG_FLR_IMAGE_TIMESTAMP: 0

MODULE_NAME: Pool_Corruption

FAILURE_BUCKET_ID: 0xC5_D0000002_nt!ExDeferredFreePool+1d7

BUCKET_ID: 0xC5_D0000002_nt!ExDeferredFreePool+1d7

Followup: Pool_corruption
---------

On 10/14:

3: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************

KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003. This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG. This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG. This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: f7022b53, The address that the exception occurred at
Arg3: b7b16b5c, Trap Frame
Arg4: 00000000

Debugging Details:
------------------


EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx"
referenced memory at "0x%08lx". The memory could not be "%s".

FAULTING_IP:
mfehidk+13b53
f7022b53 ff500c call dword ptr [eax+0Ch]

TRAP_FRAME: b7b16b5c -- (.trap 0xffffffffb7b16b5c)
ErrCode = 00000000
eax=00000000 ebx=ff98a008 ecx=ff98a01c edx=00000000 esi=ff98a01c edi=00000000
eip=f7022b53 esp=b7b16bd0 ebp=b7b16bf4 iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
mfehidk+0x13b53:
f7022b53 ff500c call dword ptr [eax+0Ch]
ds:0023:0000000c=00000000
Resetting default scope

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR: 0x8E

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from f70237f4 to f7022b53

STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be
wrong.
b7b16bf4 f70237f4 00014be0 b7b16c34 e162bcc8 mfehidk+0x13b53
b7b16c0c f7010ba8 00014be0 82f403a8 8922a538 mfehidk+0x147f4
b7b16c20 f70111af b7b16c34 0000000c 808b5d80 mfehidk+0x1ba8
b7b16c40 809676e8 0000095c 00014be0 00000000 mfehidk+0x21af
b7b16c64 8090e4d5 00000001 00000008 82d45a20 nt!PspExitProcess+0x5e
b7b16cf0 809206e4 00000000 00000000 82f403a8 nt!PspExitThread+0x528
b7b16d08 8090e466 82d45a20 00000000 00000001
nt!PspTerminateThreadByPointer+0x4b
b7b16d38 f702bebf 00000000 00000000 b7b16d64 nt!NtTerminateProcess+0x138
b7b16d54 80833bef ffffffff 00000000 17b2fbfc mfehidk+0x1cebf
b7b16d54 7c82860c ffffffff 00000000 17b2fbfc nt!KiFastCallEntry+0xfc
17b2fbfc 00000000 00000000 00000000 00000000 0x7c82860c


STACK_COMMAND: kb

FOLLOWUP_IP:
mfehidk+13b53
f7022b53 ff500c call dword ptr [eax+0Ch]

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: mfehidk+13b53

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: mfehidk

IMAGE_NAME: mfehidk.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 49dbf0c6

FAILURE_BUCKET_ID: 0x8E_mfehidk+13b53

BUCKET_ID: 0x8E_mfehidk+13b53

Followup: MachineOwner
---------


On 12/18:

2: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000000, memory referenced
Arg2: d000001b, IRQL
Arg3: 00000001, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only on
chips which support this level of status)
Arg4: 8083d658, address which referenced memory

Debugging Details:
------------------


WRITE_ADDRESS: 00000000

CURRENT_IRQL: 1b

FAULTING_IP:
nt!KiUnwaitThread+13
8083d658 890a mov dword ptr [edx],ecx

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR: 0xA

PROCESS_NAME: cqmgserv.exe

TRAP_FRAME: b8af5af0 -- (.trap 0xffffffffb8af5af0)
ErrCode = 00000002
eax=fb217008 ebx=80a84000 ecx=00000000 edx=00000000 esi=8718adb0 edi=ff67cc78
eip=8083d658 esp=b8af5b64 ebp=b8af5b68 iopl=0 nv up ei pl nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010206
nt!KiUnwaitThread+0x13:
8083d658 890a mov dword ptr [edx],ecx
ds:0023:00000000=????????
Resetting default scope

LAST_CONTROL_TRANSFER: from 8083d658 to 80836dfd

STACK_TEXT:
b8af5af0 8083d658 badb0d00 00000000 00000000 nt!KiTrap0E+0x2a7
b8af5b68 808406fd 00000000 8718adb0 ff67cc78 nt!KiUnwaitThread+0x13
b8af5b8c 80840d6f e104c9e8 e52c4338 80a840b4 nt!KiInsertQueueApc+0x21f
b8af5bac 80925593 ff67cc78 0000010c e52c4338 nt!KeInsertQueueApc+0x47
b8af5bdc 809254ed e104c9e8 00000000 00000004 nt!CmpPostNotify+0x1a0
b8af5c14 80921134 e123d2e0 00009484 e1009198 nt!CmpReportNotifyHelper+0xcf
b8af5c3c 8092cb7d e123d2e0 e1009198 8002d230 nt!CmpReportNotify+0x72
b8af5cb4 8092f551 e123d2e0 b8af5d0c 00000003 nt!CmSetValueKey+0x4d6
b8af5d44 80833bef 00000390 00d9fe7c 00000000 nt!NtSetValueKey+0x241
b8af5d44 7c82860c 00000390 00d9fe7c 00000000 nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
00d9fe4c 00000000 00000000 00000000 00000000 0x7c82860c


STACK_COMMAND: kb

FOLLOWUP_IP:
nt!KiUnwaitThread+13
8083d658 890a mov dword ptr [edx],ecx

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nt!KiUnwaitThread+13

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nt

IMAGE_NAME: ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 4a79a70a

FAILURE_BUCKET_ID: 0xA_nt!KiUnwaitThread+13

BUCKET_ID: 0xA_nt!KiUnwaitThread+13

Followup: MachineOwner
---------

[Ivan Brugiolo [MSFT]]

The first BSOD, if it had been a full dump, would have been interesting,
because you could have seen the full device stack inolved, for example,
by taking the DEVICE_OBJECT passed to IofCallDriver.
From the device stack, finding a list of culprits would have been easy.

The second BSOD seems to indicate that the Deamon Tools drivers are
involved.
Now, what is the reason to have any driver that is not needed to
run your hardware in a production server is quite unexplicable to me.
I would start from removing anything that is not an in-box driver, or
any driver the is not needed to run your network card and/or SCSI/HBA
adapter.
--
--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by j.a. harriman
This server normally sits idle as it's a failover server should the main
server go down. It's only rebooted when patches MS (and others) are applied
(on average 1/ month).
Any help with tracking this down further would be appreciated!
These are the results from the bugcheck analysis by date (using the MINI
0: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is
caused by drivers that have corrupted the system pool. Run the driver
verifier against any new (or suspect) drivers, and if that doesn't turn up
the culprit, then use gflags to enable special pool.
Arg1: 00000000, memory referenced
Arg2: d0000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 8089c4bb, address which referenced memory
------------------
BUGCHECK_STR: 0xC5_D0000002
CURRENT_IRQL: 2
nt!ExDeferredFreePool+1d7
8089c4bb 8937 mov dword ptr [edi],esi
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME: smlogsvc.exe
IRP_ADDRESS: f7f43800
TRAP_FRAME: b8feb918 -- (.trap 0xffffffffb8feb918)
ErrCode = 00000002
eax=f7c03270 ebx=00000000 ecx=000001ff edx=f7c03000 esi=00000000 edi=00000000
eip=8089c4bb esp=b8feb98c ebp=b8feb9c4 iopl=0 nv up ei ng nz ac pe cy
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
efl=00010297
8089c4bb 8937 mov dword ptr [edi],esi
ds:0023:00000000=????????
Resetting default scope
LAST_CONTROL_TRANSFER: from 8089c4bb to 80836df5
b8feb918 8089c4bb badb0d00 f7c03000 8083d60c nt!KiTrap0E+0x2a7
b8feb9c4 8089c5a0 808b7600 00000000 f7f43800 nt!ExDeferredFreePool+0x1d7
b8feba1c 808495b0 f7f43800 00000000 82ee04c0 nt!ExFreePoolWithTag+0x57f
b8feba38 80840492 f7f43800 f7f43840 f7dd8950 nt!IopFreeIrp+0xe9
b8feba88 80840ac9 f7f43840 b8febad4 b8febac8 nt!IopCompleteRequest+0x3db
b8febad8 80a84199 00000000 00000000 00000000 nt!KiDeliverApc+0xbb
b8febaf8 80a843d9 f7dd8901 00000000 00000000
hal!HalpDispatchSoftwareInterrupt+0x49
b8febb14 80a84456 00000001 f7dd8900 b8febb40
hal!HalpCheckForSoftwareInterrupt+0x81
b8febb24 8083d60c f7dd8950 f7f43840 808b9900 hal!KfLowerIrql+0x62
b8febb40 80840d7f f7f43840 f7f43800 00000000 nt!KiExitDispatcher+0x130
b8febb60 80840055 f7f43840 82ee04c0 00000000 nt!KeInsertQueueApc+0x57
b8febb94 8090eccd f7f43800 8a7ec030 f7f43800 nt!IopfCompleteRequest+0x200
b8febc3c 80840153 8a7ecd48 f7f43800 f7dd8b58 nt!WmipIoControl+0x71a
b8febc50 8092b50f f7f439b4 82ee04c0 f7f43800 nt!IofCallDriver+0x45
b8febc64 8092b444 8a7ecd48 f7f43800 82ee04c0
nt!IopSynchronousServiceTail+0x10b
b8febd00 8092b564 00000240 00000244 00000000 nt!IopXxxControlFile+0x60f
b8febd34 80833bef 00000240 00000244 00000000 nt!NtDeviceIoControlFile+0x2a
b8febd34 7c82860c 00000240 00000244 00000000 nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
0091f314 00000000 00000000 00000000 00000000 0x7c82860c
STACK_COMMAND: kb
nt!ExDeferredFreePool+1d7
8089c4bb 8937 mov dword ptr [edi],esi
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: nt!ExDeferredFreePool+1d7
FOLLOWUP_NAME: Pool_corruption
IMAGE_NAME: Pool_Corruption
DEBUG_FLR_IMAGE_TIMESTAMP: 0
MODULE_NAME: Pool_Corruption
FAILURE_BUCKET_ID: 0xC5_D0000002_nt!ExDeferredFreePool+1d7
BUCKET_ID: 0xC5_D0000002_nt!ExDeferredFreePool+1d7
Followup: Pool_corruption
---------
3: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003. This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG. This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG. This will let us see why this breakpoint is
happening.
Arg1: c0000005, The exception code that was not handled
Arg2: f7022b53, The address that the exception occurred at
Arg3: b7b16b5c, Trap Frame
Arg4: 00000000
------------------
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx"
referenced memory at "0x%08lx". The memory could not be "%s".
mfehidk+13b53
f7022b53 ff500c call dword ptr [eax+0Ch]
TRAP_FRAME: b7b16b5c -- (.trap 0xffffffffb7b16b5c)
ErrCode = 00000000
eax=00000000 ebx=ff98a008 ecx=ff98a01c edx=00000000 esi=ff98a01c edi=00000000
eip=f7022b53 esp=b7b16bd0 ebp=b7b16bf4 iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
efl=00010246
f7022b53 ff500c call dword ptr [eax+0Ch]
ds:0023:0000000c=00000000
Resetting default scope
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR: 0x8E
CURRENT_IRQL: 0
LAST_CONTROL_TRANSFER: from f70237f4 to f7022b53
WARNING: Stack unwind information not available. Following frames may be
wrong.
b7b16bf4 f70237f4 00014be0 b7b16c34 e162bcc8 mfehidk+0x13b53
b7b16c0c f7010ba8 00014be0 82f403a8 8922a538 mfehidk+0x147f4
b7b16c20 f70111af b7b16c34 0000000c 808b5d80 mfehidk+0x1ba8
b7b16c40 809676e8 0000095c 00014be0 00000000 mfehidk+0x21af
b7b16c64 8090e4d5 00000001 00000008 82d45a20 nt!PspExitProcess+0x5e
b7b16cf0 809206e4 00000000 00000000 82f403a8 nt!PspExitThread+0x528
b7b16d08 8090e466 82d45a20 00000000 00000001
nt!PspTerminateThreadByPointer+0x4b
b7b16d38 f702bebf 00000000 00000000 b7b16d64 nt!NtTerminateProcess+0x138
b7b16d54 80833bef ffffffff 00000000 17b2fbfc mfehidk+0x1cebf
b7b16d54 7c82860c ffffffff 00000000 17b2fbfc nt!KiFastCallEntry+0xfc
17b2fbfc 00000000 00000000 00000000 00000000 0x7c82860c
STACK_COMMAND: kb
mfehidk+13b53
f7022b53 ff500c call dword ptr [eax+0Ch]
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: mfehidk+13b53
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: mfehidk
IMAGE_NAME: mfehidk.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 49dbf0c6
FAILURE_BUCKET_ID: 0x8E_mfehidk+13b53
BUCKET_ID: 0x8E_mfehidk+13b53
Followup: MachineOwner
---------
2: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arg1: 00000000, memory referenced
Arg2: d000001b, IRQL
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only on
chips which support this level of status)
Arg4: 8083d658, address which referenced memory
------------------
WRITE_ADDRESS: 00000000
CURRENT_IRQL: 1b
nt!KiUnwaitThread+13
8083d658 890a mov dword ptr [edx],ecx
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR: 0xA
PROCESS_NAME: cqmgserv.exe
TRAP_FRAME: b8af5af0 -- (.trap 0xffffffffb8af5af0)
ErrCode = 00000002
eax=fb217008 ebx=80a84000 ecx=00000000 edx=00000000 esi=8718adb0 edi=ff67cc78
eip=8083d658 esp=b8af5b64 ebp=b8af5b68 iopl=0 nv up ei pl nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
efl=00010206
8083d658 890a mov dword ptr [edx],ecx
ds:0023:00000000=????????
Resetting default scope
LAST_CONTROL_TRANSFER: from 8083d658 to 80836dfd
b8af5af0 8083d658 badb0d00 00000000 00000000 nt!KiTrap0E+0x2a7
b8af5b68 808406fd 00000000 8718adb0 ff67cc78 nt!KiUnwaitThread+0x13
b8af5b8c 80840d6f e104c9e8 e52c4338 80a840b4 nt!KiInsertQueueApc+0x21f
b8af5bac 80925593 ff67cc78 0000010c e52c4338 nt!KeInsertQueueApc+0x47
b8af5bdc 809254ed e104c9e8 00000000 00000004 nt!CmpPostNotify+0x1a0
b8af5c14 80921134 e123d2e0 00009484 e1009198 nt!CmpReportNotifyHelper+0xcf
b8af5c3c 8092cb7d e123d2e0 e1009198 8002d230 nt!CmpReportNotify+0x72
b8af5cb4 8092f551 e123d2e0 b8af5d0c 00000003 nt!CmSetValueKey+0x4d6
b8af5d44 80833bef 00000390 00d9fe7c 00000000 nt!NtSetValueKey+0x241
b8af5d44 7c82860c 00000390 00d9fe7c 00000000 nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
00d9fe4c 00000000 00000000 00000000 00000000 0x7c82860c
STACK_COMMAND: kb
nt!KiUnwaitThread+13
8083d658 890a mov dword ptr [edx],ecx
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: nt!KiUnwaitThread+13
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: nt
IMAGE_NAME: ntkrnlmp.exe
DEBUG_FLR_IMAGE_TIMESTAMP: 4a79a70a
FAILURE_BUCKET_ID: 0xA_nt!KiUnwaitThread+13
BUCKET_ID: 0xA_nt!KiUnwaitThread+13
Followup: MachineOwner
---------

...

[j.a. harriman]

Sorry Ivan, I should have stated that I do have the memory.dmp files for all
these. I was making the assumption that the mini-dumps are supposed to
contain more information.

I also wanted to add that the OS is: Windows Server 2003 Kernel Version 3790
(Service Pack 2) MP (4 procs) Free x86 compatible.

Based on your experience, are these related to the same issue or are they
likely multiple issues?

Can you offer suggestions as to what info I can extract to track this down?
Thanks. Jeff

Here is some additional info from what I could gather from the 'help' files
on DEVICE_OBJECT .

0: kd> !devnode
Dumping IopRootDeviceNode (= 0x8a790008)
DevNode 0x8a790008 for PDO 0x8a791888
Parent 0000000000 Sibling 0000000000 Child 0x8a790c88
InstancePath is "HTREE\ROOT\0"
State = DeviceNodeStarted (0x308)
Previous State = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[06] = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[05] = DeviceNodeStarted (0x308)
StateHistory[04] = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[03] = DeviceNodeStarted (0x308)
StateHistory[02] = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[01] = DeviceNodeStarted (0x308)
StateHistory[00] = DeviceNodeUninitialized (0x301)
StateHistory[19] = Unknown State (0x0)
StateHistory[18] = Unknown State (0x0)
StateHistory[17] = Unknown State (0x0)
StateHistory[16] = Unknown State (0x0)
StateHistory[15] = Unknown State (0x0)
StateHistory[14] = Unknown State (0x0)
StateHistory[13] = Unknown State (0x0)
StateHistory[12] = Unknown State (0x0)
StateHistory[11] = Unknown State (0x0)
StateHistory[10] = Unknown State (0x0)
StateHistory[09] = Unknown State (0x0)
StateHistory[08] = Unknown State (0x0)
StateHistory[07] = Unknown State (0x0)
Flags (0x00000131) DNF_MADEUP, DNF_ENUMERATED,
DNF_IDS_QUERIED, DNF_NO_RESOURCE_REQUIRED
DisableableDepends = 9 (from children)

0: kd> !drvobj
Driver object (e1008030) is for:
e1008030: is not a driver object

How do I determine what parameter (address) to pass to !drvobj?

Same for !devstack

Post by Ivan Brugiolo [MSFT]
The first BSOD, if it had been a full dump, would have been interesting,
because you could have seen the full device stack inolved, for example,
by taking the DEVICE_OBJECT passed to IofCallDriver.
From the device stack, finding a list of culprits would have been easy.
The second BSOD seems to indicate that the Deamon Tools drivers are
involved.
Now, what is the reason to have any driver that is not needed to
run your hardware in a production server is quite unexplicable to me.
I would start from removing anything that is not an in-box driver, or
any driver the is not needed to run your network card and/or SCSI/HBA
adapter.
--
--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by j.a. harriman
This server normally sits idle as it's a failover server should the main
server go down. It's only rebooted when patches MS (and others) are applied
(on average 1/ month).
Any help with tracking this down further would be appreciated!
These are the results from the bugcheck analysis by date (using the MINI
0: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
DRIVER_CORRUPTED_EXPOOL (c5)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is
caused by drivers that have corrupted the system pool. Run the driver
verifier against any new (or suspect) drivers, and if that doesn't turn up
the culprit, then use gflags to enable special pool.
Arg1: 00000000, memory referenced
Arg2: d0000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 8089c4bb, address which referenced memory
------------------
BUGCHECK_STR: 0xC5_D0000002
CURRENT_IRQL: 2
nt!ExDeferredFreePool+1d7
8089c4bb 8937 mov dword ptr [edi],esi
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME: smlogsvc.exe
IRP_ADDRESS: f7f43800
TRAP_FRAME: b8feb918 -- (.trap 0xffffffffb8feb918)
ErrCode = 00000002
eax=f7c03270 ebx=00000000 ecx=000001ff edx=f7c03000 esi=00000000 edi=00000000
eip=8089c4bb esp=b8feb98c ebp=b8feb9c4 iopl=0 nv up ei ng nz ac pe cy
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
efl=00010297
8089c4bb 8937 mov dword ptr [edi],esi
ds:0023:00000000=????????
Resetting default scope
LAST_CONTROL_TRANSFER: from 8089c4bb to 80836df5
b8feb918 8089c4bb badb0d00 f7c03000 8083d60c nt!KiTrap0E+0x2a7
b8feb9c4 8089c5a0 808b7600 00000000 f7f43800 nt!ExDeferredFreePool+0x1d7
b8feba1c 808495b0 f7f43800 00000000 82ee04c0 nt!ExFreePoolWithTag+0x57f
b8feba38 80840492 f7f43800 f7f43840 f7dd8950 nt!IopFreeIrp+0xe9
b8feba88 80840ac9 f7f43840 b8febad4 b8febac8 nt!IopCompleteRequest+0x3db
b8febad8 80a84199 00000000 00000000 00000000 nt!KiDeliverApc+0xbb
b8febaf8 80a843d9 f7dd8901 00000000 00000000
hal!HalpDispatchSoftwareInterrupt+0x49
b8febb14 80a84456 00000001 f7dd8900 b8febb40
hal!HalpCheckForSoftwareInterrupt+0x81
b8febb24 8083d60c f7dd8950 f7f43840 808b9900 hal!KfLowerIrql+0x62
b8febb40 80840d7f f7f43840 f7f43800 00000000 nt!KiExitDispatcher+0x130
b8febb60 80840055 f7f43840 82ee04c0 00000000 nt!KeInsertQueueApc+0x57
b8febb94 8090eccd f7f43800 8a7ec030 f7f43800 nt!IopfCompleteRequest+0x200
b8febc3c 80840153 8a7ecd48 f7f43800 f7dd8b58 nt!WmipIoControl+0x71a
b8febc50 8092b50f f7f439b4 82ee04c0 f7f43800 nt!IofCallDriver+0x45
b8febc64 8092b444 8a7ecd48 f7f43800 82ee04c0
nt!IopSynchronousServiceTail+0x10b
b8febd00 8092b564 00000240 00000244 00000000 nt!IopXxxControlFile+0x60f
b8febd34 80833bef 00000240 00000244 00000000 nt!NtDeviceIoControlFile+0x2a
b8febd34 7c82860c 00000240 00000244 00000000 nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
0091f314 00000000 00000000 00000000 00000000 0x7c82860c
STACK_COMMAND: kb
nt!ExDeferredFreePool+1d7
8089c4bb 8937 mov dword ptr [edi],esi
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: nt!ExDeferredFreePool+1d7
FOLLOWUP_NAME: Pool_corruption
IMAGE_NAME: Pool_Corruption
DEBUG_FLR_IMAGE_TIMESTAMP: 0
MODULE_NAME: Pool_Corruption
FAILURE_BUCKET_ID: 0xC5_D0000002_nt!ExDeferredFreePool+1d7
BUCKET_ID: 0xC5_D0000002_nt!ExDeferredFreePool+1d7
Followup: Pool_corruption
---------
3: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003. This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG. This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG. This will let us see why this breakpoint is
happening.
Arg1: c0000005, The exception code that was not handled
Arg2: f7022b53, The address that the exception occurred at
Arg3: b7b16b5c, Trap Frame
Arg4: 00000000
------------------
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx"
referenced memory at "0x%08lx". The memory could not be "%s".
mfehidk+13b53
f7022b53 ff500c call dword ptr [eax+0Ch]
TRAP_FRAME: b7b16b5c -- (.trap 0xffffffffb7b16b5c)
ErrCode = 00000000
eax=00000000 ebx=ff98a008 ecx=ff98a01c edx=00000000 esi=ff98a01c edi=00000000
eip=f7022b53 esp=b7b16bd0 ebp=b7b16bf4 iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
efl=00010246
f7022b53 ff500c call dword ptr [eax+0Ch]
ds:0023:0000000c=00000000
Resetting default scope
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR: 0x8E
CURRENT_IRQL: 0
LAST_CONTROL_TRANSFER: from f70237f4 to f7022b53
WARNING: Stack unwind information not available. Following frames may be
wrong.
b7b16bf4 f70237f4 00014be0 b7b16c34 e162bcc8 mfehidk+0x13b53
b7b16c0c f7010ba8 00014be0 82f403a8 8922a538 mfehidk+0x147f4
b7b16c20 f70111af b7b16c34 0000000c 808b5d80 mfehidk+0x1ba8
b7b16c40 809676e8 0000095c 00014be0 00000000 mfehidk+0x21af
b7b16c64 8090e4d5 00000001 00000008 82d45a20 nt!PspExitProcess+0x5e
b7b16cf0 809206e4 00000000 00000000 82f403a8 nt!PspExitThread+0x528
b7b16d08 8090e466 82d45a20 00000000 00000001
nt!PspTerminateThreadByPointer+0x4b
b7b16d38 f702bebf 00000000 00000000 b7b16d64 nt!NtTerminateProcess+0x138
b7b16d54 80833bef ffffffff 00000000 17b2fbfc mfehidk+0x1cebf
b7b16d54 7c82860c ffffffff 00000000 17b2fbfc nt!KiFastCallEntry+0xfc
17b2fbfc 00000000 00000000 00000000 00000000 0x7c82860c
STACK_COMMAND: kb
mfehidk+13b53
f7022b53 ff500c call dword ptr [eax+0Ch]
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: mfehidk+13b53
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: mfehidk
IMAGE_NAME: mfehidk.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 49dbf0c6
FAILURE_BUCKET_ID: 0x8E_mfehidk+13b53
BUCKET_ID: 0x8E_mfehidk+13b53
Followup: MachineOwner
---------
2: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************
IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arg1: 00000000, memory referenced
Arg2: d000001b, IRQL
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only on
chips which support this level of status)
Arg4: 8083d658, address which referenced memory
------------------
WRITE_ADDRESS: 00000000
CURRENT_IRQL: 1b
nt!KiUnwaitThread+13
8083d658 890a mov dword ptr [edx],ecx
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR: 0xA
PROCESS_NAME: cqmgserv.exe
TRAP_FRAME: b8af5af0 -- (.trap 0xffffffffb8af5af0)
ErrCode = 00000002

...