Hi all ... if any of this is too confusing ... just ignore me ... I've confused myself :)

I'm busy writing the communication component for a massive server. The server will control access to and from data between clients and a SQL 2000 database and must be capable of inter socket communication. eg. 1 User sends a message to another user. They are not really separate entities like in an FTP server. The server will need to be able to take WinSock to the max with between 20000 and 25000 simultaneous connections ... so I need a robust, fast and stable Socket handling component ... I've been working with WinSock for a long time and know all the API's etc. I'm also quite capable of creating dynamic arrays with the lowest level API calls (VirtualAlloc, VirtualFree) ... I think more of you guys should use these API's for allocating dynamic memory ... all other API calls eventually call these (check the attached image file from MSDN) ... including HeapAlloc, GlobalAlloc, LocalAlloc, malloc etc. NB: Don't be fooled by the name, VirtualAlloc doesn't mean your memory will go directly to the paging file! I will need the dynamic arrays for many reasons including send and receive buffers because of the size of data I will often be sending ... I'm also very familiar with the multithreading API's and syncronization methods ... Events, Mutexes and Semaphores etc.

My problems relate to putting all I know into a proper working model first time ... and not having nightmares down the line ...
All these issues are what people must face when designing a big server so I don't feel so bad asking you guys !
And hopefully this could help some other people down the line too ...

Problem: Desiding whether to use a thread per connection or not !
--- Assumes I use WSAEventSelect to retrieve network events (Has anyone run a thread per socket with WSAAsyncSelect ?)
Against: The issue of inter-thread communication will be a headache ... if not impossible to solve !!! (as far as good performance is concerned)
--- How do I notify one thread "efficiently" that it must now send data while it is still monitoring for received data or a close event ???
Against: The overhead might be a little too much at 25000 threads in my app ? What do you say ? Anyone ever had so many Threads ?
Against: Syncronization will be a headache ! But I'm capable of solving the syncronization issues ... it'll just take extra time and overhead ...
Against: This is how I would have liked to run my server ... 1 thread per socket ... except for one big issue ...
--- Using WSAEventSelect for FD_READ, FD_WRITE and FD_CLOSE ... these are the primary events ...
--- the problem is FD_WRITE event will cause this Thread & API call to continue immediately for about 20000 connections because they are all ready to send data !
For: "Ease of Use" if I could only solve the problem of FD_WRITE

Problem: I could run a thread per connection ... and only use WSAEventSelect to check FD_READ and FD_CLOSE
--- Because remember one socket/thread needs to communicate with another ...
--- I could use a governor thread ... or a thread that just sends data to any connection ???
Against: Sending data to the socket ... how do I handle this in a thread per socket environment if I'm not monitoring FD_WRITE ?

Problem: I could use a thread for every 64 connections and use the WSAEventSelect and WSAWaitForMultipleEvents API's.
Against: I will need to create additional dynamic arrays and know what Threads are handling what sockets.
Against: How do I tell a thread to start monitoring for a new connection while it is waiting for network events for it's other connections ?
Against: The issue of sending data on the thread is still an issue ... how do I let a thread know that there is data for it to send for one of it's sockets ?
--- I can solve this problem by allowing the threads to routinely check if there is data for them to send ... but remember ...
--- 25000 connections / 64 sockets per thread = 390 threads checking if there is data for one of their sockets to send !

Problem: You cannot use both WSAEventSelect AND WSAAsyncSelect simultaneously to notify you of Network events !
Why: Because the one cancels the requests of the other ...
--- I could possibly alternate ... use WSAEventSelect for FD_READ & FD_CLOSE and use WSAAsynSelect for FD_WRITE ...
--- But how do I control this ??? the WSAEventSelect will run in a thread ... the WSAAsynSelect will probably not be running in a thread !

I'm sorry if all this confuses people !!! If it does just ignore me !!!
If anyone has designed a server with all these aspects in mind ... please let me know what you did so I may consider it !
Posted on 2002-11-20 08:30:26 by SubEvil
The amount of connections you mention is quite high, this page from the winsock FAQ has some details about the system limitions on the number of sockets. According to that article, even the system will have a hard time handling that many connections... So I think it will be quite hard to write a program for it as well..

I don't have much experience with server applications with high loads but I'll try to give some comments anyway:
Problem: Desiding whether to use a thread per connection or not !
--- Assumes I use WSAEventSelect to retrieve network events (Has anyone run a thread per socket with WSAAsyncSelect ?)
Against: The issue of inter-thread communication will be a headache ... if not impossible to solve !!! (as far as good performance is concerned)

What kind of communcation do you need between the threads? You could probably just use some additional event objects for that..

How do I notify one thread "efficiently" that it must now send data while it is still monitoring for received data or a close event ???

Again, not just wait for the winsock event but also for your own event object so you can tell the thread to do something.

Against: The overhead might be a little too much at 25000 threads in my app ? What do you say ? Anyone ever had so many Threads ?

Although I never really tried I think windows will have some trouble with 25000 threads..

the problem is FD_WRITE event will cause this Thread & API call to continue immediately for about 20000 connections because they are all ready to send data !

I can't follow you here, each write event is specific to the socket it is associated with, why would they all continue at the same time then?


Problem: I could use a thread for every 64 connections and use the WSAEventSelect and WSAWaitForMultipleEvents API's.
Against: I will need to create additional dynamic arrays and know what Threads are handling what sockets.

That shouldn't be a problem..

Against: How do I tell a thread to start monitoring for a new connection while it is waiting for network events for it's other connections ?

Use another custom event to tell the thread to wait for the new connection.

Problem: You cannot use both WSAEventSelect AND WSAAsyncSelect simultaneously to notify you of Network events !
Why: Because the one cancels the requests of the other ...
--- I could possibly alternate ... use WSAEventSelect for FD_READ & FD_CLOSE and use WSAAsynSelect for FD_WRITE ...
--- But how do I control this ??? the WSAEventSelect will run in a thread ... the WSAAsynSelect will probably not be running in a thread !

You shouldn't use WSAAsyncSelect, WSAEventSelect is much faster... asyncselect would be very bad for the performance..

Thomas
Posted on 2002-11-23 14:39:57 by Thomas
Thomas has made alot of good comments already. But I just wanted
to input one or two things.

Think of a web-server and how that handles the connections it gets.
Alot of webservers can have tons of connections at the same time.
And I don't think that a web-server would generate a single thread for
every connection that is requested. That would just put the machine
into a performance hell.

Creating some sortof queue system would be better. Like for instance:
You have 300 open sockets(should work on an nt-system), and when
the requests reaches 300 the rest goes into the queue system.
Awaiting a socket being freed-up.

Furthermore, I really dont believe that you will ever have 20-25.000
simultaneous connections. That just doesnt sound reasonable.
Whats the probability for 20-25.000 people simultaneously deciding
to connect to that server?

If 20.000 users were to connect to the server at the same time
it would have caused a problem with any server. There will always
be a couple of seconds delay caused by their location,isp, etc.
And those small delays would be enough for the server to finish
processing the last request leaving a free socket.

This is what i believe, feel free to correct me if I am wrong. I have
never made the server type your talking about.
Posted on 2002-11-24 05:55:32 by natas
WSAWaitForMultipleEvents :)
Similar to a FD_SET (used to monitor a socket-array)
this one monitors a set of event objects which are network events triggered by winsock2 for you. This lets you monitor all the common FD socket messages for all sockets at the same time. You will be handed the socket handle who triggered the network event.
I found this one handy when coding a multisocketed client app (a download manager of sorts) where I wanted to resist the urge to thread per socket or generate a bunch of wm's.
Posted on 2002-11-24 06:55:12 by Homer
To Thomas

What evidence do you have that "WSAEventSelect is much faster" than WSAAsyncSelect ?
Besides message overhead I suppose ! I suspected it may be a little faster ... but if you could give me a little technical detail it would be great !
The problem with using WSAEventSelect is it can only listen to a max of 64 sockets ...
So I have to create a new thread for every 64 connections ...
Do you recommend I use an WSAEventSelect per socket ?
I could be wrong but that would probably mean I have to create a thread per socket anyway !

To Natas

They won't be connectING at the same time ... but there will eventually be between 10000 and 20000 people connectED ...
Bessides, windows can only support 5 people connecting at precisely the same time ... 5 pending connections ...
That's the limit of a listening socket!

To EvilHomer2k

I have a question on the use of WSAWaitForMultipleEvents ... and WSAEventSelect
I'm fully aware of it's use ... but never really tried it ... I've always used WSAAsyncSelect ...
How did you control the FD_WRITE event ... did you ask windows to check FD_READ, FD_WRITE and FD_CLOSE at the same time ?
Because if you ask windows to notify you of an FD_WRITE event, it will return immediately if the socket is free to send info ...
Or did you only ask to monitor this event when you wanted to send data to see if the socket was ready to send ...
Sorry if this is confusing ... I just want to know a bit more of the design principle you used ... if you could explain it to me I'd be greatful !
Posted on 2002-11-25 23:57:37 by SubEvil
Mate, for a trully scalable server you have to go the IO completion ports way.
I didn't read all your posts but it doesn't seem like you have mentioned them.

25000 connections are not a big problem, but 25000 threads?? are you joking right ?? ;)

There's plenty on the net that covers winsock's completion ports. Not too difficult, although learning curve may be a little steep.

sorry I don't have time to help you out more with details, but at least I hope this hint will help you and show you the light!! ;)

Random
Posted on 2002-11-27 06:14:07 by random
How did you control the FD_WRITE event ... did you ask windows to check FD_READ, FD_WRITE and FD_CLOSE at the same time ?

Well, that's a good question.
Once yoiu have established which socket triggered the event,
use WSAEnumNetworkEvents to establish which network event(s) occurred.

invoke WSAEnumNetworkEvents,hSock,hEvent,addr wse
.if eax!=SOCKET_ERROR
mov eax,wse.lNetworkEvents

now eax contains the FD message code(s)...
you just handle it as per usual, but expect multiple FD messages OR'd together !!
For example, you will have to handle FD_READ separately to FD_READ+FD_CLOSE, since the latter indicates the last event on a closing socket.
Your handling will be determined by your requirements.

I found that eax can be ZERO meaning NO network events occurred, even though one triggered the thread... something to watch out for :)
Posted on 2002-11-30 07:05:51 by Homer
How many threads would be possible from a practical point of view then?

I'm thinking of creating a similar server but here I "only" need a maximum of about 2000 clients so would 2000 threads be ok for a normal NT system or is it too much?
MSDN says NT can handle about 2028 threads with the thread stack of 1 mb but you can freely reduce this since which would allow you to create more threads.
Further more I'd like my server to work on linux too so the more code reuse I can do the better.

Unlike the discribed system here I won't need any inter socket communication.

Thanks in advance.

// CyberHeg
Posted on 2002-12-01 03:51:09 by CyberHeg
In my opinion a well written server shouldn't use more than 1 to 3 dozen worker threads. Of course it depends a bit. But consider 2 points:

- creating a thread is a somewhat time consuming task for OS
- each thread needs its own stack, which possibly can be less than 1MB but if you define 256 kB stack size this would make for 2000 threads 2000*256 = 512 MB. And thats just the stack, not to mention other memory requests (malloc() ...)
Posted on 2002-12-01 14:02:54 by japheth
<off topic> woohoooo!!!! another South African asm programmer!!!!!

have you considered using a cluster of machines to service the connections?

20000 simul for 1 machine is quite a bit even if you use a queing system.

maybe have 10 machines with 2000 connections and 1 dedicated one between them

maye setup a 'master' machine to quickly redirect the requests to the slaves.
Posted on 2002-12-02 01:48:59 by Terab


They won't be connectING at the same time ... but there will eventually be between 10000 and 20000 people connectED ...
Bessides, windows can only support 5 people connecting at precisely the same time ... 5 pending connections ...
That's the limit of a listening socket!




just a little note, that is only true for win9x systems, win2k server supposedly can handle 200 pending connections although I have never tested this. my servers never need to handle more an about 60 at a time.
Posted on 2002-12-02 01:52:30 by Terab
This might be offtopic with the previous discussion, then again maybe not.

I'm trying to find a working model for creating a server which can handle alot clients using blocking sockets.

With existing servers I've written which didn't need too many clients I've written something like this:

Thread 1:

LOOP
client = accept(...)
Spawn new thread with client socket
END LOOP

Thread 2:

LOOP
select(...) on client socket

if (except) -> kill connection and exit thread
if (readability) -> read incomming packet and make a response or see if client exited gracefully
if (writeability) -> if the client is ready to accept data (it's assumed that a packet came in and got interpretted before this happens)
END LOOP

I normally use blocking sockets since I think about future compatibility with other OS's like linux.
The problem here is that a new thread is spawned for every client but they all end up in a simple select() loop.

I thought about another design which is a bit more complex (and untested for me atleast).

It's assumed we have a linked list where each node holds a structure and one of the structure members is a socket.

In thread 1:

Client gets accepted and a new node is appended to the client list where the socket gets set of that client as the node data (among other data).

Then in thread 2 there is a loop which runs through the client list and does select on each socket to see if there are things about to happen from the clients like a incomming packet.

If there is, either the client request could be handled by the same thread or the request could be put into a queue and be handled by a 3rd thread.

The only real problem about this model (besides that it needs more code) is that it could take some time from the time the poll on client 1 happens till poll on client 5000 happens.

Maybe someone have a good document with suggestions on how to handle clients? Most Tcp/Ip books I've seen just explain how to make a simple server and how to use the socket api's.

Thanks in advance.

// CyberHeg
Posted on 2002-12-02 07:04:11 by CyberHeg
To Terab:

I think inter socket communication would be very difficult in the Multi-Server environment, the Master server for example would need to assertain whether a particular user is logged in or not ... sending a message to all the clients ... It'll just be a pain to sort out over a single server solution, bessides, I would rather buy one big multi-CPU server than 2 or 3 or more ... but thanx for the suggestion, maybe it'll suite someone else ...

To random:

I'm going the IO completion ports way ... you said the learning curve was steep ... it's like K2 man ...
Almost everything I know of WinSock falls away ... but IOCP can't be used on 9x machines ...
So I have to develop Client and Server strategies seperately ...
But I've been doing a lot of reading on the subject and it's the recommended route for servers of this scale!

If anyone has or knows where to get Asm source for an IOCP Server, I'd love to get it as I can only get C/C++ source etc.

Hopefully I'll have an Asm WinSock server using IOCP's myself in the not so distant future :)
Posted on 2002-12-03 01:06:11 by SubEvil