So I'm supposed to do this homework assignment where I take a 4 digit number from the user and determine whether or not it is prime.
Secondly, I need to find the prime numbers on either side of the prime number.

I've written this code and since I'm not really sure how to use WinDbg to figure out what line is causing the trouble, I figured I'd ask here.
Apparently I've written an "Integer Overflow Error" somewhere in here.
Anyway I'm aware this is basically the equivalent of assembly garbage code, but I'm just trying to get through this assignment.

Thanks for your help, as it's due Tuesday afternoon (short time schedule I know but I've been beating my head in on this one).

Oh one more thing...I think one of the div instructions could be the cause, but I'm not sure how.

.386
.MODEL FLAT

ExitProcess PROTO NEAR32 stdcall, dwExitCode:DWORD
INCLUDE io.h              ;header file for input/output

cr    EQU  0dh                  ;carriage return character
Lf  EQU  0ah                  ;line feed

.STACK 4096                  ;reserve 4096-byte stack

.DATA                    ;reserve storage for data

prompt1  BYTE  "Please enter a four digit integer:  ", 0
prompt2  BYTE  cr, Lf, " is not a prime number.", 0
prompt3 BYTE  cr, Lf, " is a prime number.", 0
prompt4 BYTE  cr, Lf, "Largest prime number smaller than ", 0
prompt5 BYTE  cr, Lf, "Smallest prime number larger than ", 0
string  BYTE  11 DUP (?)
orig  WORD  ?
lower  WORD  ?
higher  WORD  ?

.CODE
_start:

mov bx, 2
output prompt1
input string, 6
atoi string      ; put input in ax
mov orig, ax  ; copy ax to orig

origtest:

cmp bx, ax      ; subtract input from bx
je origprime  ; if input and bx are = then jump
cwd            ; convert to word
div bx        ; divide ax by bx
cmp dx, 0      ; sub ax minus 0
je orignotprime  ; if 0 and dx are = then jump
inc bx        ; increment bx register
jmp origtest  ; restart loop

lowerloop:

sub ax, 1
jmp lowerprimetest

lowerprimetest:

cmp bx, ax
je lowerprime
cwd
div bx
cmp dx, 0
je lowerloop
inc bx
jmp lowerprimetest

lowerprime:

mov lower, ax
jmp highestloop

highestloop:

jmp higherprimetest

higherprimetest:

cmp bx, ax
je higherprime
cwd
div bx
cmp dx, 0
je highestloop
inc bx
jmp higherprimetest

higherprime:

mov higher, ax
jmp finalout

origprime:

mov bx, 2
mov ax, orig
itoa string, ax ; Convert ax to Ascii
output string  ; Output: ax
output prompt2  ; is a prime number
jmp lowerloop

orignotprime:

mov bx, 2
mov ax, orig
itoa string, ax
output string
output prompt3  ; is not a prime number
jmp lowerloop

finalout:

output prompt4  ; Largest number smaller than
mov ax, lower
itoa string, ax
output string
output prompt5  ; Smallest number greather than
mov ax, higher
itoa string, ax
output string

INVOKE ExitProcess, 0

PUBLIC _start

END
Posted on 2007-10-22 21:14:57 by Kuroi Kaze
You must set edx to zero before you perform a divide.

The div opcode works like this:
ax = (dx : ax) / (reg/mem)  <-- div
dx = (dx : ax) % (reg/mem)  <-- mod !!!

That means even when working with 16 bit registers, the numerator is 32 bits, dx:ax !!!
Just perform a xor dx,dx or equivalent to prevent getting weird div results.
Posted on 2007-10-23 00:16:22 by Homer

http://www.ollydbg.de/

(Or, push F9 and it'll stop at the problem.)

*Basically, DIV opperate on DX:AX, but you ignore DX prior to DIV. You should set it to zero.
Posted on 2007-10-23 00:17:00 by bitRAKE
Yeah I grabbed OllyDbg...it's a bit complicated for a beginner like me, but I generally see that div operations are causing the exceptions.

Also my logic is flawed in places where I'm relying on EAX because div changes the values there and it's no longer original input.  I need to use ECX in some places to fix that.

I'll be working hard on this all day and if I have more questions I'll be back.
Posted on 2007-10-23 09:28:45 by Kuroi Kaze
Okay here's the re-written overflow-free error code.

Unfortunately it doesn't work right.

1059 is my "control" test and the answers that should come out are not what come out of my program.

When I put in 1059 I should get 1051 and 1061 are the two primes.
Instead I get 1053 and 1060?!?

.386
.MODEL FLAT

ExitProcess PROTO NEAR32 stdcall, dwExitCode:DWORD
INCLUDE io.h ;header file for input/output

cr EQU 0dh ;carriage return character
Lf EQU 0ah ;line feed

.STACK 4096 ;reserve 4096-byte stack

.DATA ;reserve storage for data

prompt1 BYTE "Please enter a four digit integer:  ", 0
prompt2 BYTE cr, Lf, " is not a prime number.", 0
prompt3 BYTE cr, Lf, " is a prime number.", 0
prompt4 BYTE cr, Lf, "Largest prime number smaller than ", 0
prompt5 BYTE cr, Lf, "Smallest prime number larger than ", 0
string BYTE 11 DUP (?)
orig DWORD ?
lower  DWORD ?
higher  DWORD ?

.CODE
_start:

mov ebx, 2
output prompt1
input string, 6
atod string ; put input in eax
mov orig, eax ; copy eax to orig
mov ecx, eax

origtest:

cmp ebx, eax ; subtract input from ebx
je origprime ; if input and ebx are = then jump
XOR edx,edx ; zero out edx
div ebx ; divide eax by ebx
cmp edx, 0 ; sub eax minus 0
je orignotprime ; if 0 and edx are = then jump
inc ebx ; increment ebx register
mov eax, ecx ; put the original test back in eax
jmp origtest ; restart loop

lowerloop:

sub ecx, 1
jmp lowerprimetest

lowerprimetest:

mov eax, ecx
cmp ebx, eax
je lowerprime
XOR edx,edx
div ebx
cmp edx, 0
je lowerloop
inc ebx
jmp lowerprimetest

lowerprime:

mov lower, ecx
mov ecx, orig
jmp highestloop

highestloop:

jmp higherprimetest

higherprimetest:

mov eax, ecx
cmp ebx, eax
je higherprime
XOR edx,edx
div ebx
cmp edx, 0
je highestloop
inc ebx
jmp higherprimetest

higherprime:

mov higher, ecx
jmp finalout

origprime:

mov ebx, 2
dtoa string, ecx ; Convert ecx to Ascii
output string ; Output: ecx
output prompt2 ; is a prime number
jmp lowerloop

orignotprime:

mov ebx, 2
dtoa string, ecx
output string
output prompt3 ; is not a prime number
jmp lowerloop

finalout:

output prompt4 ; Largest number smaller than
mov eax, lower
dtoa string, eax
output string
output prompt5 ; Smallest number greather than
mov eax, higher
dtoa string, eax
output string

INVOKE ExitProcess, 0

PUBLIC _start

END

Posted on 2007-10-23 12:25:47 by Kuroi Kaze