Hi. I just wrote a nifty little asm function that calculates square root with a root table. I wanted to test my library in Visual C++ and found that my asm function somehow mangles the registers or something and causes cin to read memory from outer space!

I'm using the invoke directive with stdcall and I thought that the only registers I had to worry about preserving were esp and ebp (or something like that) My function only uses the general purpose registers and the floating point unit so what gives? (and also makes calls to GetProcessHeap and HeapAlloc)

The asm code for the offending function looks like this:

InitRootTable proc Precision:DWORD
    LOCAL ProcHeap:HANDLE
    LOCAL TableSize:DWORD ; size of RootTable in bytes
    LOCAL Root:DWORD

    .IF Precision>23
        mov Precision,23
    .ENDIF
   
    ; calculate TableSize
    mov ecx,Precision
    add ecx,2
    mov eax,1
    shl eax,cl
    mov TableSize,eax
   
    ; calculate MantissaMask (This is "MantissaMask DWORD ?" in my .DATA? section)
    mov eax,0ffffffffh
    mov ecx,32
    sub ecx,Precision
    shl eax,cl
    shr eax,9
    mov MantissaMask,eax
   
    ; allocate TableSize bytes and make RootTable point to it
    invoke GetProcessHeap
    mov ProcHeap,eax
    invoke HeapAlloc,ProcHeap,0,TableSize
    mov RootTable,eax

    ; fill RootTable
    mov ebx, TableSize
    shr ebx,2
    .WHILE ebx != 0
        dec ebx
        mov eax,ebx
        mov edx,TableSize
        shr edx,2
        add eax,edx
        mov Root,eax
        ;fild Root
        ;fsqrt
        ;fstp Root
        mov eax,Root
        and eax,007fffffh
        mov ,eax
    .ENDW
    ret
InitRootTable endp

Ok, that's a little long but I assure you all I use is eax-edx, and make a call to GetProcessHeap and HeapAlloc, and use fld, fild, and fsqrt.

The C++ program looks like this:

extern "C" _stdcall InitRootTable(unsigned int);

int main(int argc, char* argv[])
{
int x;
InitRootTable(8);
while (cin >> x) {
}
return 0;
}

At this point I'm guessing that the calls to GetProcessHeap and HeapAlloc are somehow mangling some of the segment registers and this is what is messing up cin... (actually the access violation occurs in istream.cpp)

Any help would be greatly appreciated!
Posted on 2005-05-01 06:42:10 by wildgnu
I think the problem lies in that you did not preserve the value in ebx.
Posted on 2005-05-01 06:47:54 by roticv
So which registers can you leave mangled and which need to be preserved? What's so special about ebx?
Posted on 2005-05-01 07:15:40 by wildgnu
I tried saving eax-edx, ebp and esp and it didn't change anything. When debugging the program with Visual C++ it shows that the only registers that change after calling InitRootTable are eip and efl. (and MM0 whatever that is) I removed the calls to GetProcessHeap and HeapAlloc and it worked fine. I tried replacing them with calls to malloc and this didn't change anything. (it still failed)
Posted on 2005-05-01 07:29:14 by wildgnu
That's weird. MM0 is a MMX register. Usually ebx, edi, esi, ebp (though the assembler normal preserve for you) needs to be preserved. Do you have any exe to post? Maybe I can debug it.
Posted on 2005-05-01 07:50:45 by roticv
Here's the exe with debugging info if you want to give it a shot. I had to change the extension to .txt since it won't let me upload exe's. I changed the model to flat,c instead of flat, stdcall and that seemed to help a little bit, now it just jumps into outer space instead of trying to read from outer space... Not much of an improvement though! :-)
Attachments:
Posted on 2005-05-01 08:32:56 by wildgnu
Um, you can try and debug that if you're bored but I think I figured out the problem.

BTW, don't you hate it when things mysteriously start working the way you'd expect them to?
Posted on 2005-05-01 08:42:52 by wildgnu
Debugging rule #9 : "If you didn't fix it, it ain't fixed"  :D

www.debuggingrules.com
Posted on 2005-05-01 13:47:32 by Dr. Manhattan
wildgnu,

Just try what Victor suggested and preserve EBX. What I would be tempted to do is preserve ALL of the  registers except EAX and restore them at the end of the algo to see if you can get it working then remove them one at a time to see which one causes the problem. ESP & EBP are already handled by the PROC layout of the masm procedure so preserve,


push ebx
push ecx
push edx
push esi
push edi

; run your code

pop edi
pop esi
pop edx
pop ecx
pop ebx


Also make sure that you prototype the masm proc as EXTERN C according to the compiler syntax you are using and get the calling convention correct.
Posted on 2005-05-01 18:53:59 by hutch--
The general rule of thumb is that you should not touch segment registers under windows

Anyway all I did was to change your


...
push cs
push ds
...
pop ds
pop cs
...

to

...
push ebx
nop ;this is used because i was editing the exe in ollydbg
...
nop ;same as above comments
pop ebx

And it worked fine for me. I guess you did not read my comments. ebp is automatically preserved by masm (you can verify by looking in a debugger), so there is no need to preserve it.
Posted on 2005-05-01 22:33:38 by roticv