Hi everybody =)

I would like to write a filetransfer application, nothing special yet, just an exercise for me. But I would like to add a feature: a compression, which I use in "real time". This means I read a file, compress the data, and send it. I want to gain extra speed in transferring files without manually zip them :)

So I ask you for some advice :)

I heard that the ethernet packet limit is 1500 bytes. This means, I should approx. take 200bytes away to get a "optimal" size for my data for each single packet. right?

And has anyone of you a tutorial on file operations in win32?

Thank you very much!
Posted on 2001-12-12 12:53:13 by nyook

Hi everybody =)

I would like to write a filetransfer application, nothing special yet, just an exercise for me. But I would like to add a feature: a compression, which I use in "real time". This means I read a file, compress the data, and send it. I want to gain extra speed in transferring files without manually zip them :)

So I ask you for some advice :)

I heard that the ethernet packet limit is 1500 bytes. This means, I should approx. take 200bytes away to get a "optimal" size for my data for each single packet. right?


I don't think you should think in packets & packet sizes, at least not IP packets.. Of course all data is cut into packets but this is done by winsock so just send the data in any (reasonable) size..

However I think you should compress the data in blocks of say 16kb, but this depends on the algorithm used. If you would compress each say 400 bytes one by one, the overhead of a compression algo could make it very slow, so it's better to compress chunks of data, then add some header stuff about the packed & unpacked size, which file it comes from etc, and send it to the client program. The client should read your header, decompress it and store it at the right place in some file.


And has anyone of you a tutorial on file operations in win32?


I can't remember if Iczelion had some tutorials about files, but at least there's one on memory mapping & file I/O..

Thomas
Posted on 2001-12-12 13:34:37 by Thomas
For filexfer, you should use TCP and not UDP. TCP is stream-based,
so don't think about packetsize. Also, remember what Thomas
says about choosing a resonable compression buffer size.
Posted on 2001-12-12 13:53:44 by f0dder
Thanks,

of course I won't compress each packet one by one.
But if I sent.. for example 2000bytes, then winsock would send one "full" packet and a half.

Another question:

I read in ICZ's tutorials that after a send command you get the number of bytes actually sent. But in non-blocking sockets, you theoretically should get 0 bytes, because the transfer is still in progress. How can I get the number of bytes sent then?

I hope that I've read the tut well, and this isnt a stupid question :)
Posted on 2001-12-13 01:22:28 by nyook

of course I won't compress each packet one by one.
But if I sent.. for example 2000bytes, then winsock would send one "full" packet and a half.


Winsock has internal buffers and tries to send optimal packets, so don't bother about that, when you just keep sending data to winsock, winsock will just buffer the data so it can send a full package..


I read in ICZ's tutorials that after a send command you get the number of bytes actually sent. But in non-blocking sockets, you theoretically should get 0 bytes, because the transfer is still in progress. How can I get the number of bytes sent then?


Although unlikely, 0 is a valid return value for send. But I think it only occurs when you also specify 0 as length parameter for send. So it doesn't have to mean the transfer is still in progress.
The only 'invalid' return value is SOCKET_ERROR. And this is important when using non-blocking socket: If you get this error, call WSAGetLastError. If the error is WSAEWOULDBLOCK, then the transfer is still in progress and you'll have to try again later (when you received a notification that the other side is ready for writing again). So this error is normal and can occur very often.
The number of bytes sent is simply the return value of send (except for SOCKET_ERROR).

Thomas
Posted on 2001-12-13 05:19:45 by Thomas

when you just keep sending data to winsock, winsock will just buffer the data so it can send a full package..


Really? Cool... :)


Hm... I'm not sure if I understood you correctly: When I get a WSAEWOULDBLOCK error, this means I have to re-send my data?
I thought this just means that the data isnt immediately send, I just have to wait a second or so...
Posted on 2001-12-14 03:59:11 by nyook
Thank you very much, I'm goint to start to program my FileTransfer progg, using Iczelions wonderful tutorials =)

Hmm.. I remember you posted some benchmark-like results in using different sorts of Winsock (blocking, non-blocking, threads).

I would like to use the fastest method, but how can I now what is generally the fastest? maybe I can't, but can you give me any tips? :)

Thanks again.
Posted on 2001-12-14 04:03:15 by nyook
Hm... I'm not sure if I understood you correctly: When I get a WSAEWOULDBLOCK error, this means I have to re-send my data?


Yes.

I thought this just means that the data isnt immediately send, I just have to wait a second or so...


No that's wrong, the return value of send indicates how many bytes are really sent, that's the only thing you can and should rely on. A WSAEWOULDBLOCK actually means 'I would have to block (i.e. wait until I can send data) to execute this function but as you set the socket to non-blocking I'm not allowed to do this so try again later' :).

Hmm.. I remember you posted some benchmark-like results in using different sorts of Winsock (blocking, non-blocking, threads).


It's the 'asynchronous recv/send' (iirc) thread, the examples should still be there..

I would like to use the fastest method, but how can I now what is generally the fastest? maybe I can't, but can you give me any tips?


The benchmarks are in that thread to.. I think the fastest one is using WSAEventSelect (I'm using that one in my webserver), but this requires a little bit more programming and design than the other ones. WSAAsyncSelect is the easiest to use (apart from blocking functions), but is slow because it uses windows messages, and.. well windows messages are slow :). When using it for internet applications that use some bandwith like 1024kbps, this is not really a problem but for local networks with speeds up to 100Mbit, it really is a bottleneck.
You could also try blocking sockets inside a thread, but I never use blocking sockets because I want full control and I don't have that when some function blocks (read: hangs :) ).

If you don't require speeds of multiple megabytes per second, WSAAsyncSelect will do, and it's easier to program with.
If you need higher speeds, WSAEventSelect seems to be doing well, but it just requires a more difficult design and some more programming.
Take a look at the examples in the other thread, they all seem very easy but remember that the only thing they have to do is keep sending data..

If you're interested, I can mail you the source of my old httpft (file transfer using the http protocol) program.. The executable can be downloaded from my site. It's one of my first programs so the design is not optimal and some winsock things are not implemented as they should be (causing 100% processor use while transfering .. :rolleyes: ) but maybe it's useful to you.

Thomas
Posted on 2001-12-14 13:48:35 by Thomas
i see. but then I have to do a loop until my send-call doesnt give back a Wsaewouldblock, so does that mean it's nearly the same as blocking sockets, because my loop would block my program :D

Hm.. EventSelect loocks interesting, they arent faster than blocking sockets? good :)

Are there any tutorials on EventSelect aside your Example Code (for which I'm going to seek for right after)?

Thanks again!
Posted on 2001-12-14 14:25:34 by nyook

i see. but then I have to do a loop until my send-call doesnt give back a Wsaewouldblock, so does that mean it's nearly the same as blocking sockets, because my loop would block my program :D


If you'd repeat the send call this would happen but you shouldn't do that.
This is exactly why WSAAsyncSelect (and others) are created. These functions give you notifcations of network events like 'ready for writing again', or 'data is waiting for you to be received'.
When you look at the documentation of WSAAsyncSelect, you will see you can set which events you want to receive (FD_READ, FD_WRITE and so on). These events happen in two cases:


    [*]Right after you called WSAAsyncSelect, all events that are true at that moment ('data available', 'ready for sending') are sent..This normally happens one time (or multiple times if you call WSAAsyncSelect more than once)
    [*]And of course when one of the events happens


    When you receive a notification of some kind, say FD_READ, it will not be send again until you re-enable it. A notification can be re-enabled in various ways (all described in the WSAAsyncSelect docs), for example when you call 'recv', the FD_READ notification is re-enabled.
    Same for the FD_WRITE notification, it is not re-enabled until you call 'send'.
    (edit: wrote 'FD_SEND', should be 'FD_WRITE', corrected now) :rolleyes:
    An important thing to remember is that the notifcation is re-enabled, regardless of the success of the re-enabling function: even if send fails, the notification is re-enabled.

    In your case, this is what you should do:
    (assumed you use WSAAsyncSelect)
    - call WSAASyncSelect to be notified of the FD_WRITE message
    - The FD_WRITE notification is sent right after your call when the other side is ready for writing
    - call send to send some data, the FD_WRITE msg is re-enabled
    - wait for the next FD_WRITE msg
    - call send again...etc

    "Wait for the next FD_WRITE msg" is actually just quiting the window procedure, you will receive your custom defined winsock message as soon as you can send data again.

    Even if you'r not going to use WSAAsyncSelect, read the documentation about it (you can find sock2.hlp on my site) carefully, it explains a lot of these things...


    Hm.. EventSelect loocks interesting, they arent faster than blocking sockets? good


    In my benchmark it was faster than blocking sockets. Also I think you should never use blocking sockets, unless you're writing a console app like telnet or something. 'blocking' and 'multithreaded windows application' just don't go together well :) ...

    Are there any tutorials on EventSelect aside your Example Code (for which I'm going to seek for right after)?


    I don't know any tutorials about it but I didn't look for it either. Download the winsock 2 reference from my site, that's a good start.
    The only example code I have is the example in the other thread and of course my new webserver but I'm not going to release that yet and it's far too complex for an example anyway (it's actually a combination of windows messages (very few though), event objects and my own thread message queue).

    However it isn't that hard to use, it's best to use it in a seperate thread, as you can use waiting functions like WaitForSingleObject to wait for an event, without blocking the main program.
    A big difference with WSAAsyncSelect is that WSAAsyncSelect gives you the event that happens in the lower (or higher, can't remember right now) word of lParam when you receive the custom message. WSAEventSelect just activates an event object when *one or more of the events* happened, you'll have to figure out *which* yourself. You can do this with WSAEnumNetworkEvents.

    Thomas
Posted on 2001-12-14 14:59:52 by Thomas
One sidenote: EventSelect is a 'new' function that is available only in Winsock version 2 and higher.

However you can safely use that version (I never used anything else) as it's already supported in win98 (and win95 with the appropriate patch), and NT 3.5 iirc.

Just remember to use ws2_32.dll/lib/inc for version 2 instead of wsock32 (used in Iczelion's tutors)
Posted on 2001-12-14 15:11:07 by Thomas
I seee =) you're really a Code Warrior :)



WSAEventSelect just activates an event object when *one or more of the events* happened, you'll have to figure out *which* yourself. You can do this with WSAEnumNetworkEvents.


But if you have to do this manually, doesn't it consume as much time as windows messages either way?
And why are Windowsmessages so slow, are they really?

Thanks^2
Posted on 2001-12-14 15:37:46 by nyook

But if you have to do this manually, doesn't it consume as much time as windows messages either way?


No, with window messages, each event causes a custom message to be sent to your window (very slow).
With EventSelect, an object's state is changed to signaled (very fast), your thread can be in an almost 0% cpu time consuming state while waiting for that signal. The only thing you then have to do is to check which events happened, and internally this is just stored somewhere in winsock's memory..
With asyncSelect, the bottleneck is *using windows messages*, nothing else.


And why are Windowsmessages so slow, are they really?


Messages are continiously sent for almost every possible event that happens, for example a mousemove, a repaint message, etc. So network events aren't the only messages that are sent to your window.
But it's not that bad when you receive an event now and then. However when using a local network, say you send 1KB per send call. The local network reaches about 5 MB/s. After each send call, another notification message (=windows message) is sent.
This makes 5 MB / 1KB = 5120 messages per second. You can imagine that's quite a lot to process in the message queue and another problem is that your window may be too busy with the winsock events causing low response to user actions or even an hanging app.


Thomas
Posted on 2001-12-14 17:55:53 by Thomas
you're my god =)
Posted on 2001-12-15 14:31:01 by nyook