Please Help Me Debug Recurring Bluescreen

[transitionality]

My OS is Windows XP Pro SP3. I've recently been getting seemingly random bluescreens that I can't associate with any particular application or device. The error message is always the same.

IRQL_NOT_LESS_OR_EQUAL

0x0000000A (0x0000001C, 0x00000002, 0x00000001, 0x806E6A16)

I'd be very appreciative if somebody could take a look at a few recent minidumps and tell me what to suspect.

Thanks.

minidumps.zip

[cluberti]

It looks like the Outpost firewall driver is crashing your system:

0: kd> k
ChildEBP RetAddr  
b8877bb4 806644a2 hal!KeAcquireInStackQueuedSpinLock+0x26
b8877bd4 f5374131 nt!VfIrpDatabaseEntryReleaseLock+0xae
WARNING: Stack unwind information not available. Following frames may be wrong.
b8877be4 f53636c0 afwcore+0x12131
b8877c04 f5363cd6 afwcore+0x16c0
b8877c1c 804ef19f afwcore+0x1cd6
b8877c2c 80658128 nt!IopfCallDriver+0x31
b8877c50 8057f982 nt!IovCallDriver+0xa0
b8877c64 805807f7 nt!IopSynchronousServiceTail+0x70
b8877d00 80579274 nt!IopXxxControlFile+0x5c5
b8877d34 8054162c nt!NtDeviceIoControlFile+0x2a
b8877d34 7c8fe4f4 nt!KiFastCallEntry+0xfc
042bf914 00000000 0x7c8fe4f4

0: kd> .bugcheck
Bugcheck code 1000000A
Arguments 0000001c 00000002 00000001 806e6a16

In a stop 0x1A or 0xA, the first parameter is the memory address referenced - in this case, it's 0x0000001c. Since the first 64K of VA for any process is marked as PAGE_NO_ACCESS, and 0x0000001c would fall into said 64K, this access is going to fail. Worse, because the second parameter is the IRQ Level at the time of the dump, and the machine is at IRQL2 (dispatch), any page fault or memory access error at this IRQL or higher is actually *supposed* to cause a bugcheck, because otherwise the system could conceivably get stuck in a dispatch loop and do a hard hang at this point.

Also, note that the EIP register is invalid here as well, and EIP is the CPU register that tells the CPU what instruction to execute next. Since this is invalid, bad things are going to happen - ultimately, this is what started the whole mess:

0: kd> kv
ChildEBP RetAddr  Args to Child			  
b8877bb4 806644a2 83235f98 f7aaa870 f7aaa890 hal!KeAcquireInStackQueuedSpinLock+0x26 (FPO: [0,1,0])
b8877bd4 f5374131 8394a6c0 f53740da b8877c04 nt!VfIrpDatabaseEntryReleaseLock+0xae (FPO: [Non-Fpo]) (CONV: fastcall)
WARNING: Stack unwind information not available. Following frames may be wrong.
b8877be4 f53636c0 8598bbe8 8394a6c0 8598bbe8 afwcore+0x12131
b8877c04 f5363cd6 8598bbe8 8394a6c0 8598bbe8 afwcore+0x16c0
b8877c1c 804ef19f 8598bbe8 8394a6c0 806e6428 afwcore+0x1cd6
b8877c2c 80658128 854b23a0 806e6410 8394a6c0 nt!IopfCallDriver+0x31 (FPO: [0,0,0]) (CONV: fastcall)
b8877c50 8057f982 8394a730 854b23a0 8394a6c0 nt!IovCallDriver+0xa0 (FPO: [Non-Fpo]) (CONV: fastcall)
b8877c64 805807f7 8598bbe8 8394a6c0 854b23a0 nt!IopSynchronousServiceTail+0x70 (FPO: [Non-Fpo]) (CONV: stdcall)
b8877d00 80579274 000001d0 000005a8 00000000 nt!IopXxxControlFile+0x5c5 (FPO: [Non-Fpo]) (CONV: stdcall)
b8877d34 8054162c 000001d0 000005a8 00000000 nt!NtDeviceIoControlFile+0x2a (FPO: [Non-Fpo]) (CONV: stdcall)
b8877d34 7c8fe4f4 000001d0 000005a8 00000000 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ b8877d64)
042bf914 00000000 00000000 00000000 00000000 0x7c8fe4f4

0: kd> .trap b8877d64
ErrCode = 00000000
eax=00185d98 ebx=00000000 ecx=7c82ff82 edx=7ffab000 esi=042bf964 edi=00000000
eip=7c8fe4f4 esp=042bf8b4 ebp=042bf914 iopl=0		 nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000			 efl=00000202
001b:7c8fe4f4 ??			  ???

0: kd> dc eip L1
7c8fe4f4  ????????							 ????

With all that said, the machine crashed because the kernel-mode driver afwcore, running inside process acs.exe, caused a memory fault at DPC dispatch level, thus the machine bugchecks. All 3 dumps are the same, meaning that the crashes appear random, but are not. I believe acs.exe and afwcore.sys are a part of the Outpost Firewall, so upgrading that may help - here is the current version:

0: kd> lmvm afwcore
start	end		module name
f5362000 f5399a80   afwcore  T (no symbols)		   
	Loaded symbol image file: afwcore.sys
	Image path: afwcore.sys
	Image name: afwcore.sys
	Timestamp:		Mon Jun 30 09:16:08 2008 (4868DC98)
	CheckSum:		 0003CC9D
	ImageSize:		00037A80
	Translations:	 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Note that if the problem exists in any newer version, upgrading obviously isn't going to resolve the problem, and your only real recourse is to uninstall the Outpost firewall and file a bug with the vendor to have this fixed. Good luck.

[transitionality]

That's an amazing depth of analysis in a very short time. Thank you very much.

I'd been suspecting overheating and bad ram, both of which are tricky things to deal with in a laptop, but Outpost is no big deal. I'll update, and if that doesn't solve my problem, I'll switch vendors.

Many thanks.

[cluberti]

No worries. Glad to have you onboard.