Im writing this code that enumerates all subkeys of a specific registry keys and is supposed to send it to the client side. i am using Winsock 1.1 and the sockets are both in the non-blocking mode. what is the best way of transferring the enumerated keys over to the client side ?

Btw the getsockopt() with SO_SNDBUF returns the value of 8192 so i guess i can't try to transfer the whole string in one shot can i ?
Posted on 2006-04-27 22:55:11 by XCHG
Yeah - send as much data in a single chunk as you like.. the network subsystem will package your data into 1500-ish byte packets, which under tcp, are guaranteed to arrive in the correct order... make the client responsible for reconstructing the data - and here's a hint.. add several bytes to the "front" of the data which indicate the expected payload size, so the client can figure out which is the last "partial payload packet".. make sense?

Note : what you just did is implement your own transport protocol on top of tcp - your 'custom protocol packetheader' describes the total number of bytes in a given 'send job'..
Posted on 2006-04-28 00:40:15 by Homer
Be sure to check return-values from Send(), though - you *could* risk not being able to send all data in one go. I prefer sending "chunked" anyway, to not have to lock too much kernel mode memory for the send buffer.
Posted on 2006-04-28 04:43:52 by f0dder
So basically i shouldnt care how big the data that im trying to send through the socket is huh ? would it be okay if i send each of the enumerated keys one by one while using the registry enumeration APIs ?

btw, thanks for the replies guys
Posted on 2006-05-01 20:56:50 by XCHG
No - that will result in lots of small packets being sent, with 54 bytes of tcp header for each packet, I know I'm pedantic, but that's a waste of bandwidth.
The network subsystem will cut up oversized packets for you, but it won't cache undersized packets for you.. you should do this yourself, here's how:

I recommend collecting up the results in a small buffer and sending data whenever the buffer contains more than 1.5kb... and then finally sending whatever's left in the buffer when enumeration is completed.
This will guarantee that the fewest possible packets are sent, meaning the smallest amount of 'tcp packet header overhead' being sent.
It will mean the packets you are sending are "full".

Of course, you could just collect up ALL the data into a LARGE buffer, and Send it all in one hit, leaving the network subsystem to chop up the data into max. sized packets, but that depends on you knowing how large a buffer is required, which you may not be able to easily determine (as in your case)... and there's still no guarantee that the network subsystem will be able to "swallow your load" - for example, if we try to send 118kb of data in a single send, it might report that 12kb was sent (depends on what else your tcp/ip stack is doing at the time). Always check how much actually got sent !!

Posted on 2006-05-02 01:15:07 by Homer

No - that will result in lots of small packets being sent, with 54 bytes of tcp header for each packet, I know I'm pedantic, but that's a waste of bandwidth.
The network subsystem will cut up oversized packets for you, but it won't cache undersized packets for you.. you should do this yourself, here's how:

Nagle's Algorithm?


Of course, you could just collect up ALL the data into a LARGE buffer, and Send it all in one hit, leaving the network subsystem to chop up the data into max. sized packets,

Pretty bad to do if you have a lot of data, because the buffer will have to be physmem locked until the send is done.
Posted on 2006-05-02 02:29:51 by f0dder

Yeah, I was only really stating the obvious, that cutting corners is a bad idea, that the 'easy way' to do something is usually the 'worst way' imaginable, and that managing your own buffers is generally a good idea.

I'd forgotten about Nagle's algorithm, is that enabled by default though?
Posted on 2006-05-02 04:47:44 by Homer
Yep, Nagle is on by default - you have to turn it off if you don't want it (things like SSH/telnet connections might not want it).

But I agree, you should "do the right thing" and not depend on Nagle. Better performance, less kernel/usermode transitions.
Posted on 2006-05-02 05:09:19 by f0dder
Or, if you could use Winsock 2, then you might find the TransmitFile and TransmitPacket functions make things easier.
Posted on 2006-05-02 09:07:40 by stormix
Hmm interesting, TransmitFile exists even on win9x and NT3.51? :O

A thing to keep in mind: iirc TransmitFile() is limited to two concurrent transmits on NT workstation, but can do any amount on server versions of windows.
Posted on 2006-05-02 09:16:20 by f0dder
Cool thang, thanks for all the help
Posted on 2006-05-04 22:41:38 by XCHG
Uh, i have this protocol which consumes the first four bytes of a packet which tells either the client or the server what the content of the package is, i guess i can not send the whole packet in one shot because the four byte signature would only be transferred the first time right ?
Posted on 2006-05-07 00:53:12 by XCHG
Sure you can.
This protocol you are working on has a special purpose for the first four bytes, but that doesn't change anything, other than adding four bytes to each packet you send.
When this sent packet is received, the receiver must check the first four bytes, and then strip it from the front of the received data..

Posted on 2006-05-07 03:04:31 by Homer
XCHG, if you're using TCP, realize that you are ***NOT*** sending "packets", you're sending a stream of data. So if you do, say, 5 sends of 100 bytes, it could show up as one read() of 5000 bytes, or sortof any combination of differently sized buffers.
Posted on 2006-05-07 04:32:50 by f0dder
My experience has shown that this is ALWAYS true, not just of tcp, but of udp, and of raw ethernet. You should ALWAYS "expect the unexpected" (after all, you're a programmer!) - in your current example, this 4 bytes thing is being used to separate the actual DATA PAYLOADS being sent in the 'tcp stream' - use them to determine whetehr you just received (for example) 2 and a HALF packets, instead of the amount you maybe expected.
ITS COMMON TO RECEIVE INCOMPLETE DATA and unless you know how many bytes are EXPECTED, you have no way to verify this !! THAT is the purpose of your 4 bytes in your given example, regardless of whether it contains just a protocol identifier that identifies the start of each packet, or whether it contains the #bytes in the packet, it serves the same purpose. Typically, such headers are larger than 4 bytes, but theres NO standard here, you are "inventing your own protocol that sits on top of tcp" and as such you are making up the rules to suit yourself - whatever works, eh? :)

PS : Don't use "network peek" methods to check how much data is available to receive, because for various reasons they are unreliable and inefficient.. just receive ALL incoming data into a "First In, First Out) (FIFO) buffer and then process any COMPLETE packets as you see fit.

PPS : I can qualify that last statement apon request.
Posted on 2006-05-07 07:06:30 by Homer
Network peek == bad, indeed.

If your "4 bytes" is just some ID, then that won't be sufficient, if your data stream is binary - you'll need to either prefix a length as well (whether it be dword or qword :p), unless you can 100% guarantee your ID won't show up in the raw binary data.

Or well, if you send fixed-size structures you might not need to send the length, but just keep in mind you can get incomplete as well as too large results with read().
Posted on 2006-05-07 08:01:09 by f0dder
Well the data is not having a fixed length. but the thing is that all the receiving actions take place in one single thread in the client. Thus i just made it peek at the first four bytes of each incoming stream to realize what that stream of data is made for.
But sometimes the stream i am sending grows as big as 500 KB thus makes all the packets but the first one to lose their four bytes signature which could only be attached to the beginning of the stream. and i dont think it is a good idea to attach the four bytes to each of the streams.

Right now i have done a little bit of a trick. i have attached another four bytes after the four-byte signature at the beginning of the stream, which indicates the size of the data that is to follow, and inside the client thread i start another thread for downloading the data and suspend the main thread until the data is completely read.
Posted on 2006-05-15 23:05:49 by XCHG
XCHG, once again: TCP is not packet-based, it's stream-based. You *need* to prefix a length (your "hack") if you're not dealing with fixed-size items. Using two threads for this sounds like bad design...
Posted on 2006-05-16 04:30:54 by f0dder
I just fixed that one, 4 bytes for the signature, 4 bytes the length of the *stream* and the rest is the data

stream stream stream, i've gotta keep saying this  :)
Posted on 2006-05-20 22:50:50 by XCHG
How the client should handle the data received by a TransmitFile call from the server?
Can anyone post an example of usage of this API?

Thanks
Posted on 2007-01-23 16:37:05 by lectus