hi all,
i'm getting a little trouble on getting messages trough POP3 protocol:

Actually the way i'm retrieving messages is:

2) STAT -> i get total size to do GLobalAlloc (total size)
3) RETR 1

After RETR command, i perform a cycle to receive the entire message, block by block. I check on ev ery block the last 5 char, if they correspond to the termination octet CRLF.CRLF
But the problem is that i receive messages that are always greater then the value i get after RETR:

RETR 1 +OK 2357 octets 2421
RETR 2 +OK 10992 octets 11298
RETR 3 +OK 699 octets 710

so i always get a value greater then the one i've estimated. In this way i go over the limit of memory i allocate getting exception in writing memory.

I really appreciate any suggestion about the correct way to proceed on downloading.
Posted on 2003-10-29 01:50:01 by Bit7
Show us your source, it's probably a silly assumption in your program.
Posted on 2003-10-29 03:27:55 by Homer
thx evil,

follow the routine that download messages

; ?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?
proc ReceiveMessages uses ebx esi edi, hDlg:dword


; (00) preparo message counter

xor ebx, ebx
mov edi, [m_msgStartOffset]

@@nextMsgDwn: inc ebx

; (01) update message number, preparo el "RETR" command (retrieve)

call WriteBinToEdit, [hDlg], IDC_LOG_MD, ebx

call AngeloZeroMem, offset m_pop3_RETR + 5, 12
call _wsprintfA, offset m_pop3_RETR + 5, offset g_szFormat2, ebx
add esp, 12
call AppendCRLF, offset m_pop3_RETR

call AngeloFindByte, offset m_pop3_RETR, 0
call send, [g_skt], offset m_pop3_RETR, ecx, NULL

call AngeloZeroMem, offset g_szTemporary, 32
call recv, [g_skt], offset g_szTemporary, XD_NET_SIZETOREAD, NULL

; (02) ricevo e estrapolo el "message size"

mov esi, offset g_szTemporary
add esi, 4
call AngeloFindByte, esi, SD_VAL_SPACE

call AngeloStrToNum, esi, ecx
mov [BYTESTOREAD], eax

shl eax, 16
call SendDlgItemMessageA, [hDlg], IDC_LOG_PROGRESS1, PBM_SETRANGE, 0, eax

call SendDlgItemMessageA, [hDlg], IDC_LOG_PROGRESS1, PBM_SETPOS, 0, 0

; (03) preparo e salvo el message header

mov eax, [BYTESTOREAD]
mov [msgHdr.msgsize], eax
mov [msgHdr.state], 'U'
call AngeloCopyMem0, offset g_szInBox, offset msgHdr.folder

call AngeloCopyMem, offset msgHdr, edi, size MSGHEADER
add edi, size MSGHEADER

; (04) scarigo el messaggio e lo metto in memoria

call WriteBinToEdit, [hDlg], IDC_LOG_MBT, [msgHdr.msgsize]

xor esi, esi
@@nxtPart: call recv, [g_skt], edi, [msgHdr.msgsize], NULL
add edi, eax
add esi, eax

; .. updating message progress

call WriteBinToEdit, [hDlg], IDC_LOG_MBD, esi
call SendDlgItemMessageA, [hDlg], IDC_LOG_PROGRESS1, PBM_SETPOS, esi, 0

; looking for "CRLF.CRLF"

mov eax, edi
sub eax, 5
call AngeloCmpString, eax, offset m_szEOM, 5
cmp ecx, 0
jne @@nxtPart

@@msgRcvd: ; (05) cancello o no ?

cmp [g_delMsgFromServ], 1
jne @@goOnNext
call AngeloZeroMem, offset m_pop3_DELE + 5, 12
call _wsprintfA, offset m_pop3_DELE + 5, offset g_szFormat2, ebx
add esp, 12
call AppendCRLF, offset m_pop3_DELE

call AngeloFindByte, offset m_pop3_DELE, 0
call send, [g_skt], offset m_pop3_DELE, ecx, NULL

call recv, [g_skt], offset g_szTemporary, XD_NET_SIZETOREAD, NULL

call SendDlgItemMessageA, [hDlg], IDC_LOG_PROGRESS2, PBM_SETPOS, ebx, 0

; (06) check for end of messages, and return

@@goOnNext: cmp ebx, [dInfo.totMessages]
jl @@nextMsgDwn


endp ReceiveMessages

In these routine i receive the messages,
by before colling this routin i allocate memory after a STAT command, once for all the messages

; (2) STAT --- how many message to receive and total size...

call SendCommandAndLog, [hRichEditLog], offset m_pop3_STAT, offset m_szSendingStat
call ReceiveAndLog, [hDlg], [hRichEditLog], [recvBuff]

mov esi, offset dInfo
call GetDownloadInfo, esi, [recvBuff]
cmp [dInfo.totMessages], 0
je @@quit

; ?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?
proc PrepareMemoryForMessages

; preparo lo spazio che me servi: tot bytes to write + message headers

mov ecx, [dInfo.totBytes]
mov eax, size msgHdr
add eax, 35
mov ebx, [dInfo.totMessages]
mul ebx
add ecx, eax
mov [m_newMessagesSize], ecx

call GlobalAlloc, GMEM_FIXED, [m_newMessagesSize]

mov [m_msgStartOffset], eax


endp PrepareMemoryForMessages

; ?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?=?
proc GetDownloadInfo uses esi, downloadInfo:dword, statBuff:dword

mov ebx, [downloadInfo]

call AngeloFindByte, [statBuff], 32
inc ecx
add [statBuff], ecx

call StatFindAndConvert, [statBuff], SD_VAL_SPACE

mov [(DWNINFO ebx).totMessages], eax

inc esi
add [statBuff], esi
call StatFindAndConvert, [statBuff], 0Dh

mov [(DWNINFO ebx).totBytes], eax


endp GetDownloadInfo

Hope you'll understand my method.
Thanks B7
Posted on 2003-10-29 12:45:22 by Bit7
anyway, i've read some posts on various forums:
Th issue seems to be that the size of every message i receive from the pop3 server after a RETR command is always wrong:

RETR 1 +OK 12090 octets i get 12424
RETR 2 + OK 6679 octrts i get 6857
RETR 3 + OK 24335 octets i get 25043

An article i've found says, to be sure to have enough space, to allocate always 1500bytes more than the size initially received.
So, for this reason propably is better i allocate memory before every message, then receiving, saving on file and free memory (now i was allocating all the messages total space).
Posted on 2003-10-30 01:21:02 by Bit7
Hello, you really should not make assumptions - always check return values from api functions !!

For example, here's what micro$oft have to say about calling RECV...

Return Values
If no error occurs, recv returns the number of bytes received. If the connection has been gracefully closed, the return value is zero. Otherwise, a value of SOCKET_ERROR is returned, and a specific error code can be retrieved by calling WSAGetLastError.

Even though you told it to receive N bytes, it could return a different amount, and it is the actual amount you are interested in, not the perceived amount...
A suggestion here is to always call ioctlsocket,hSock,FIONREAD,addr cbRead
before you call RECV - this will return the actual #bytes that are waiting in the network stack to be read - then hand the value in cbRead now to RECV - much safer !!

In conclusion, I suggest that you should ALWAYS check return values of api functions and never blindly assume anything...

Have a nice day :)
Posted on 2003-10-31 07:52:26 by Homer
thanks again evil,

the ioctlsocket function should be very useful, i could allocate space for every block i receive and save it directly to file. It's just an idea. Anyway i really would like the relation between octest declared from POP3 and the greater size that i receive... i've read RFC, it says that on multiline response it append a CRLF at the end of every line. Could it be the CRLF's that increment the size ?
Posted on 2003-11-01 03:45:48 by Bit7
I can't give you a definitive answer for why the packet sizes are larger than the purported payload, but that sounds very likely to me.
Yes, I use ioctlsocket in the way you describe - I allocate the block after calling ioctlsocket, read into the allocated block, and then generally I keep a list of pointers to allocated blocks, and create another thread whose ONLY job is to process and then release the blocks in the same order they are encountered.
This is called a "forked receiver", but it's just one possible way to do it.
Have fun, play around with a few different ideas, see what works best for you in your situation, experiment, and if it's not fun, don't do it !! :)
Posted on 2003-11-02 17:52:00 by Homer
thanks again,

yes the dynamic allocation method seems the most correct... for now i leave the 1500bytes more allocation for every message, seems wark fine now, just to test the rest of the program.

Thx B7
Posted on 2003-11-04 02:22:26 by Bit7
actually i've improoved the easiest and hope safest solution.. also if not the best one:

I allocate for every message the POP3 declared size * 2

These becouse the server seems don't count CRLF as 2 chars, but as one, so if i have a message completely just full of CRLF i should receive it correctly... i will try :)
Posted on 2003-11-07 16:16:50 by Bit7