I would like to display a message box if a page error exception (0Eh) occurs in a NTVDM process in winxp.
Normally - and unlike GPFs - in case of page errors the NTVDM process terminates silently. So I wrote a simple VDD (which is just a normal PE dll running in user mode in the NTVDM process), which sets a standard exception registration record at FS:[0]. But this doesnt work, the handler routine is never executed, the process still terminates silently.

So far I have verified that the dll is loaded and runs in the same thread as the code causing the page error.

Has anyone some experience in this topic?

Posted on 2004-01-11 17:58:13 by japheth
Perhaps the vectored exception handling stuff in XP can be used? Just a (bad) guess though, if SEH doesn't catch it VEH probably won't, either.
Posted on 2004-01-19 06:55:57 by f0dder
Hi f0dder,

havent heard about this vector stuff.
But I surely would prefer a solution which works for 2k and NT as well, because there the behaviour is identical.

Currently my guess is that NTVDM.EXE is able to change the standard exception handling. Thats done by calling ntdll function NtVdmControl, which of course is totally undocumented and I havent found anything about it in the net as of yet. And this function is too complicated for RE, at least at the current "priority".

Posted on 2004-01-19 08:02:13 by japheth
VEH is some new stuff included in XP, and I know next to nothing about it - apart from the name. Should be fairly well documented in PlatformSDK though?

NtVdmControl - heh... probably a lot of weird stuff happening there. Have you tried asking at tsehp's board? (I guess the name reference is okay since there's no link, and I doubt you're up to malicious plans with NTVDM).

Btw, what kind of pagefault is it you want to trap? A process running in NTVDM, or the NTVDM process itself?
Posted on 2004-01-19 08:59:41 by f0dder
Here is a msdn article on VEH, in case anybody is interested:

Posted on 2004-01-19 09:46:57 by Jibz
Thanks for the links/names.

Its the NTVDM process itself, no WOW, just a DPMI app running inside.

With a debugger (WinDbg, for example) I have no problem trapping the exception, and all registers are displayed correctly.

Posted on 2004-01-20 05:57:32 by japheth