So, i have the following code assembled with NASM 64-bit mode:

extern GetCurrentProcess
extern SetPriorityClass
extern ExitProcess

section .data
hProcess dq 0
NORMAL_PRIORITY_CLASS equ 00000020h
REALTIME_PRIORITY_CLASS equ 00000100h

section .text
global _main

_main:
call GetCurrentProcess
mov , rax

;call _dummy

mov rcx, rax
mov rdx, REALTIME_PRIORITY_CLASS
call SetPriorityClass

mov rcx, 0
call ExitProcess

_dummy:
push rsp
and spl, 0F0h

mov rcx,
mov rdx, REALTIME_PRIORITY_CLASS
call SetPriorityClass

pop rsp
ret


As you can see, i have the line that says {call    _dummy} commented out.
The reason is because when i call SetPriorityClass() from an outside procedure it crashes.
I don't get the reason why it is crashing when i call it from a procedure outside _main.
Any idea what is going wrong? I even tried aligning the stack but no luck.
Posted on 2011-02-05 01:56:47 by banzemanga

So, i have the following code assembled with NASM 64-bit mode:

extern GetCurrentProcess
extern SetPriorityClass
extern ExitProcess

section .data
hProcess dq 0
NORMAL_PRIORITY_CLASS equ 00000020h
REALTIME_PRIORITY_CLASS equ 00000100h

section .text
global _main

_main:
call GetCurrentProcess
mov , rax

;call _dummy

mov rcx, rax
mov rdx, REALTIME_PRIORITY_CLASS
call SetPriorityClass

mov rcx, 0
call ExitProcess

_dummy:
push rsp
and spl, 0F0h

mov rcx,
mov rdx, REALTIME_PRIORITY_CLASS
call SetPriorityClass

pop rsp
ret


As you can see, i have the line that says {call    _dummy} commented out.
The reason is because when i call SetPriorityClass() from an outside procedure it crashes.
I don't get the reason why it is crashing when i call it from a procedure outside _main.
Any idea what is going wrong? I even tried aligning the stack but no luck.


You must always allocate register shadow space on the stack when calling Win64 functions even if they don't appear to be used.  Windows may internally be accessing those areas.  Thus in your case, RSP may be overwritten causing the fault.


    sub    rsp, 32
    mov  rcx,
    mov  rdx, REALTIME_PRIORITY_CLASS
    call    SetPriorityClass
    add    rsp, 32


There are optimizations that can be made to this scheme but this should get you started.
Posted on 2011-02-05 10:16:56 by p1ranha


    sub    rsp, 32
    mov   rcx,
    mov   rdx, REALTIME_PRIORITY_CLASS
    call    SetPriorityClass
    add    rsp, 32


There are optimizations that can be made to this scheme but this should get you started.



Thanks it works. I remember reading about fastcall and moving the adjust pointer as a callee but never had a problem with it until now.

Can you do explain me why 32? I can see it is 3*8 meaning 3 8-byte variables on the stack. But SetPriorityClass() only has 8*2 arguments.
Posted on 2011-02-05 14:18:26 by banzemanga

Actually it's 4*8 = 32 as Win64 uses a max of 4 64-bit registers, thus I always subtract a minimum of 32 bytes from RSP for all functions with 1 to 4 parameters.
Also very important to remember for fastcall is that the stack must always be properly aligned.  So even if you only have 3 parameters you MUST evenly align the stack by multiples of 16 prior to making the call.  The macros in the NASMX package goes through great pains to ensure proper stack alignment when defining procedures and invoking function calls.

Posted on 2011-02-06 11:40:32 by p1ranha