Hey guys

I am in need of writing a server capable of handling many client. By many I say it should be able to handle atleast 2000 or even more.
Because of needs to support more OS's then Windows (especially linux in future) I'd like to use blocking sockets to do this.
I think it could be inefficient to start a new thread for every client so I wanted to do something else.
Maybe one of you know some reading resources on such?

I saw a server recently in a book which used blocking sockets and it used select() to poll client sockets if they were in ready state but so far I could see select got a limitation of 64 sockets at a time. This could ofcourse be changed to poll single sockets and then copy the ready handles to another thread which would do the processing but maybe there exists some better way?

Atleast this is what I came up with for now:

Thread 1 polls for new clients using accept. When a new client is connected it's socket handle is put into a linked list.

Thread 2 now scans all the nodes of the linked list doing a select() on each of them. If a client is ready for reading or writing then the node would be copied to some other list which would specify that the client needs processing.

Thread 3 takes care of all the ready clients and processes their request by either reading or writing to the socket.

I just wonder if it won't be too slow with select(). Since new clients are appended to the end it could take some time before the scanner thread would come to it once there are many clients connected.
Also that the worker thread which needs to process each client could have many requests so the last client added could have to wait some time before it gets it's turn.

Another thing is if there is a timeout on tcp sockets. I plan to use tcp only and what if a client doesn't send anything for do any requests for like 30 min. Would the socket handle still be valid?

I'm really open for better suggestions.

// CyberHeg
Posted on 2003-01-14 01:56:47 by CyberHeg
I am in need of writing a server capable of handling many client. By many I say it should be able to handle atleast 2000 or even more.
Because of needs to support more OS's then Windows (especially linux in future) I'd like to use blocking sockets to do this.
I think it could be inefficient to start a new thread for every client so I wanted to do something else.
Maybe one of you know some reading resources on such?

Blocking mode isn't the most efficient way of handling requests in windows, but if you need easy portability, it's probably the way to go.

I saw a server recently in a book which used blocking sockets and it used select() to poll client sockets if they were in ready state but so far I could see select got a limitation of 64 sockets at a time. This could ofcourse be changed to poll single sockets and then copy the ready handles to another thread which would do the processing but maybe there exists some better way?

Atleast this is what I came up with for now:

Thread 1 polls for new clients using accept. When a new client is connected it's socket handle is put into a linked list.

Thread 2 now scans all the nodes of the linked list doing a select() on each of them. If a client is ready for reading or writing then the node would be copied to some other list which would specify that the client needs processing.

Thread 3 takes care of all the ready clients and processes their request by either reading or writing to the socket.

I just wonder if it won't be too slow with select(). Since new clients are appended to the end it could take some time before the scanner thread would come to it once there are many clients connected.
Also that the worker thread which needs to process each client could have many requests so the last client added could have to wait some time before it gets it's turn.

Whatever model you choose, you can't handle all the requests at the same time. So there will always be some time before you reach the latest request.

The model you came up with seems okay, although I don't think you need thread 2. Thread 3 (could be multiple worker threads actually to serve multiple request at the same time) can get (/check/wait for) the next request in the linked list when it got nothing to do. However I wouldn't process a *full* request at once. That way if you have two worker threads and two people download a big file (in the case of a HTTP server, for example), all the other requests would have to wait for those two people (which may download for an hour)! So at least send/recv a piece of data and then handle the next bit of the next request (simulating the processing of all requests at the same time, just like multitasking). The problem with blocking sockets is that they block :) So if you call recv it might wait for a minute (although unlikely), blocking the processing of other requests. If you would use non-blocking sockets it would be a lot easier. You could just say: 'send this data to that socket and tell me when you're ready while I'm doing more important things'. That's a lot more efficient since, winsock will just queue your data until the other side is ready (which is an easy operation that requires nearly no cpu time), while you have to dedicate a full thread for it. The thread doesn't use much cpu either, but you could have used the thread for something more active (like handling other requests). Just opening 1000 threads is very inefficient.

Btw be sure to do some synchronization on the linked list so it doesn't get corrupted when multiple threads use it at the same time (same goes for all other shared data).

Another thing is if there is a timeout on tcp sockets. I plan to use tcp only and what if a client doesn't send anything for do any requests for like 30 min. Would the socket handle still be valid?


The socket *handle* will stay valid until you close it with closesocket. However the socket might have been *disconnected* earlier. The handle will still be valid but send and recv will fail because the socket isn't connected anymore.

Thomas
Posted on 2003-01-14 13:12:05 by Thomas


Whatever model you choose, you can't handle all the requests at the same time. So there will always be some time before you reach the latest request.

The model you came up with seems okay, although I don't think you need thread 2. Thread 3 (could be multiple worker threads actually to serve multiple request at the same time) can get (/check/wait for) the next request in the linked list when it got nothing to do.



Thanks for the feedback. I can see now it's not needed with thread 2.


However I wouldn't process a *full* request at once. That way if you have two worker threads and two people download a big file (in the case of a HTTP server, for example), all the other requests would have to wait for those two people (which may download for an hour)! So at least send/recv a piece of data and then handle the next bit of the next request (simulating the processing of all requests at the same time, just like multitasking).



I assume you mean sending/recieving small chunks like 10 bytes at a time and then have a header in each packet which would say how much data there really is. So if there is 88 bytes to send from client to server it would be sent in 1 chunk to the server and the server would recieve it in 9 requests. Similar a 55 bytes packet from the server to client would be sent to the client in 6 requests where the client would collect the data it recieves making it back to 1 real packet.


The problem with blocking sockets is that they block :) So if you call recv it might wait for a minute (although unlikely), blocking the processing of other requests. If you would use non-blocking sockets it would be a lot easier. You could just say: 'send this data to that socket and tell me when you're ready while I'm doing more important things'. That's a lot more efficient since, winsock will just queue your data until the other side is ready (which is an easy operation that requires nearly no cpu time), while you have to dedicate a full thread for it. The thread doesn't use much cpu either, but you could have used the thread for something more active (like handling other requests).



I could see if the client side is ready to recieve next block with select() but it's a bit time consuming.
Fortunatly I don't think I will need to do more transfering then a few hundred bytes between the clients and no big file downloading :)


Btw be sure to do some synchronization on the linked list so it doesn't get corrupted when multiple threads use it at the same time (same goes for all other shared data).


Yes I am aware of that. If it's not done you run into serious problems in no time.


The socket *handle* will stay valid until you close it with closesocket. However the socket might have been *disconnected* earlier. The handle will still be valid but send and recv will fail because the socket isn't connected anymore.


Is there a way to see what this timeout is? Like what if there is no data transfered in like 5 min?
The server I want to create the clients allocate "shared" resources on the server so as soon as the client is disconnected expectedly or unexpectedly the resources which was allocated for that specific client should be freed immediately. I thought the easiest way is to have a socket open all the time. If the socket would become invalid on the server for the client the client would be disconnected and resources freed.

What about checkumming of packets?
I've seen protocols in past doing chekcumming of data but I guess since tcp is safe communication there is no way on earth the data could be wrong unless a hacker would modify the content intentionally.

Thanks in advance.

// CyberHeg
Posted on 2003-01-23 09:44:01 by CyberHeg
I assume you mean sending/recieving small chunks like 10 bytes at a time and then have a header in each packet which would say how much data there really is. So if there is 88 bytes to send from client to server it would be sent in 1 chunk to the server and the server would recieve it in 9 requests. Similar a 55 bytes packet from the server to client would be sent to the client in 6 requests where the client would collect the data it recieves making it back to 1 real packet.

Don't think in packets when working with winsock. Think of it as a two way continuous stream, but one which may not be ready to read/write data from/to at any time. The TCP/IP packets are managed by winsock, it may pack multiple sends in one packet, or distribute one packet over multiple recvs. However it's not of your concern. If you use your own protocol you probably need some 'packet header' like you said (I prefer to call these protocol 'packets' messages, and their headers message headers, to avoid confusion) with the size of the message to read from the 'continuous stream'.

Is there a way to see what this timeout is? Like what if there is no data transfered in like 5 min?

Sorry I don't know... But to prevent timeouts you could send short 'keep alive' messages every minute or so, to keep the connection at least a bit active so it won't time out.

What about checkumming of packets?
I've seen protocols in past doing chekcumming of data but I guess since tcp is safe communication there is no way on earth the data could be wrong unless a hacker would modify the content intentionally.

TCP/IP is a reliable protocol, the data you receive should be exactly the same as the data sent. So luckily this is not of your concern either :). Even if someone would modify the packets without correcting the checksum the packet would probably be dropped by the first router or gateway it reaches... You can't avoid intentional data manipulation (although it's not easy to do that), but data corruption is quite easy to detect, and winsock will do this for you.

Thomas
Posted on 2003-01-25 06:23:31 by Thomas
i just want to add that ive made a multithreaded chat server, each sock has its own thread that recv data and will spawn a new thread to process that data, and depeneding on the command, say a send chat command will spawn a thread for every conneced sock to send the proper response too, and all in blocking mode.
its made in c++ though. althogh it will support a limited number of threads, i made a app do 4k threads in 1 xp app, so i would say 400 users, altough you could run 2 apps and have one central server that will distrobute load over each app... and take a look at this thread for syncronized access to shared data. http://www.asmcommunity.net/board/index.php?topic=10072 i have to use those apis my server with 100 user load hasnt crashed yet cuz of 2 threads accessing the same memory address.
Posted on 2003-01-25 17:57:55 by Qages

i just want to add that ive made a multithreaded chat server, each sock has its own thread that recv data and will spawn a new thread to process that data, and depeneding on the command, say a send chat command will spawn a thread for every conneced sock to send the proper response too, and all in blocking mode.
its made in c++ though. althogh it will support a limited number of threads, i made a app do 4k threads in 1 xp app, so i would say 400 users, altough you could run 2 apps and have one central server that will distrobute load over each app... and take a look at this thread for syncronized access to shared data. http://www.asmcommunity.net/board/index.php?topic=10072 i have to use those apis my server with 100 user load hasnt crashed yet cuz of 2 threads accessing the same memory address.


So if I understood you right you can have a maximum of 3 threads per user? One recv, process and one send thread.
Are you making the send threads as the data is about to be sent? Doesn't that make big delays? I remember in my experiments long ago I saw that a time to start a new thread was anything between a few milliseconds and a few seconds.
I plan to use C++ too since I see many advantages by doing so.

What bugs me most is that with blocking sockets one client could block anothers requests for a period of time hence not using processing time most optimally. I don't think makeing a thread per client would be ok though since it could be even more resource taking.

// CyberHeg
Posted on 2003-01-26 02:14:28 by CyberHeg
no, there is 1 thread always open, when it recvs data it will spawn a temp thread to spawn say 50 threads(for 50 users) each sending to a diffrent client, so there is no delay. and took about 1 second to spawn 4000 threads. and i have fast computer, and my server takes no more then 3mb of memory with 50 users connected.
Posted on 2003-01-26 15:46:51 by Qages
Thanks for the tip.

It's indeed an interesting idea. I can see this could come handy for solving all problems with "blockin each other" issues like Thomas pointed out.

One thing I have been looking for for some time is actually proper reading material. I'm supprised every socket book tells how to open a socket and write a single threaded server but nothing more then that.

Thanks again to both of you.

// CyberHeg
Posted on 2003-01-27 13:47:18 by CyberHeg