EDIT: This part is solved. Read: http://www.asmcommunity.net/board/index.php?topic=29234.msg206503#msg206503

I can't figure out how to overcome stack overflow errors with SEH.
I know that the Exception Handler can't run without stack space, but there seems to be a way.

I tried writing a try/catch block in C++ and it seems to work for them, but the code is a little tricky so I can't really figure what's different.

A stack overflow such as this:

while (1)
    push eax
endw

Will be caught in a Cpp try block, but not in my asm program (to be more specific, the program will just exit without notice). Any ideas?

Example asm code that exits the program:

	push offset TimerExceptionHandler1
push fs:[0]
mov fs:[0], esp

.while 1
push eax
.endw

TimerExceptionHandler1:

invoke MessageBox, NULL, addr ClassName, NULL, MB_OK

ret


However, if I would use for instance - mov ecx, 0 - mov dword ptr , 0 - it would MessageBox.
Posted on 2008-12-13 09:34:54 by ekx
Hm, you could perhaps switch to a safe stack in the exception handler?
Posted on 2008-12-13 10:44:49 by f0dder
Yes f0dder, I managed to get this far now. That was indeed part of the problem.

My problem is now that this works for me if it's done in the main thread of the program. If the same thing is done in a thread (even one that's freshly created), the program just closes, which is why I was having so much trouble with this since I was testing across those two mediums. I'll try to gather some more information...
Posted on 2008-12-13 10:52:15 by ekx
Figured it out.

invoke CreateThread, NULL, 1024*1024, offset TTimer, NULL, NULL, offset TimerThreadId

Will crash in event of a stack overflow, but

invoke CreateThread, NULL, 1024*1024 + 24, offset TTimer, NULL, NULL, offset TimerThreadId

Will recover successfully. It seems that pushing on the extra 24 bytes raises an exception, but the exception procedure can use those 24 bytes for it's arguments??

invoke CreateThread, NULL, NULL, offset TTimer, NULL, NULL, offset TimerThreadId

of course also works, because it's the default setting.
Posted on 2008-12-13 11:15:13 by ekx
I assume you *are* setting the SEH per-thread? (Not calling you stupid or anything, just to rule out all possibilities :)).

How are you switching to safe stack? If you're allocating it on the heap or through static buffers, remember that something] defines the "safe" range ESP can lie within, and that your thread will be terminated if ESP goes outside this range.
Posted on 2008-12-13 11:15:17 by f0dder
Yes I set it per thread. I actually wasn't using any safe stack when testing this.

I just had a

Exception:
while 1
endw

loop running in the exception procedure, which is the step before even messing with the stack.

If the program closed, then it would mean the exception didn't go trough. If it looped endlessly, the thread froze, but didn't force a program shutdown.

The program closes on the thread with a stack of 1024*1024 when processing the exception (so Exception: never gets called), but works fine with 1024*1024+24 (the thread will freeze).

I'm not sure if I'm missing something, I don't know that much about FS and the rest of the stuff you mentioned. I think I can figure it out from this point on though. Thanks a lot.

I guess I'll post some code if you ever want to know what I'm doing:



TTimer Proc

push offset TimerExceptionHandler
push fs:[0]
mov fs:[0], esp

.while 1
push eax
.endw

TimerExceptionHandler:
.while 1

.endw

TTimer endp

1. invoke CreateThread, NULL, 1024*1024, offset TTimer, NULL, NULL, offset TimerThreadId ; program will crash
2. invoke CreateThread, NULL, 1024*1024+24, offset TTimer, NULL, NULL, offset TimerThreadId ; program will not crash, thread will freeze

Posted on 2008-12-13 11:25:35 by ekx
My main problem right now is that a stack overflow gets only successfully raised once. The second time that it overflows, an exception doesn't get raised, and the stack frame used for the exception handler which follows gets corrupted making the program crash. I have tried using VirtualProtect , VirtualFree, etc, to somehow re-enable guarding of the last stack page, but couldn't get it to work, or to raise exceptions on guarded/no-access pages.

The following code shows that the program crashes only after a second stack overflow:


push offset TimerExceptionHandler1
push fs:[0]
mov fs:[0], esp
   
mov TimerExceptionSafe, offset Restart
mov TimerExceptionStack, esp
mov TimerExceptionEbp, ebp
   
Restart:
invoke MessageBoxA, 0, offset szStackOverflow, 0, MB_OK

.while 1
push eax
.endw

ret

TimerExceptionHandler1 PROC C pExcept:DWORD, pFrame:DWORD, pContext:DWORD, pDispatch:DWORD

mov eax, pContext
mov ecx, TimerExceptionSafe
mov .CONTEXT.regEip, ecx
mov ecx, TimerExceptionStack
mov .CONTEXT.regEsp, ecx
mov ecx, TimerExceptionEbp
mov .CONTEXT.regEbp, ecx

mov eax, ExceptionContinueExecution

ret

TimerExceptionHandler1 endp


This is some of the things I have tried to restore the stack overflow protection:



.data?
TimerMBI MEMORY_BASIC_INFORMATION {}
OldProt dd 2 dup (?)

.code

TimerExceptionHandler1 PROC C pExcept:DWORD, pFrame:DWORD, pContext:DWORD, pDispatch:DWORD

mov eax, pExcept
mov ecx, .EXCEPTION_RECORD.ExceptionInformation ; address of accessed memory

invoke VirtualQuery, ecx, offset TimerMBI, sizeof TimerMBI

mov eax, TimerMBI.AllocationProtect
or eax, PAGE_GUARD

;invoke VirtualFree, TimerMBI.BaseAddress, 4096, MEM_DECOMMIT
invoke VirtualProtect, TimerMBI.BaseAddress, eax, 4096, offset OldProt

mov eax, pContext
mov ecx, TimerExceptionSafe
mov .CONTEXT.regEip, ecx
mov ecx, TimerExceptionStack
mov .CONTEXT.regEsp, ecx
mov ecx, TimerExceptionEbp
mov .CONTEXT.regEbp, ecx

mov eax, ExceptionContinueExecution

TimerExceptionHandler1 endp


Any ideas are welcome.
Posted on 2008-12-15 15:00:08 by ekx