In order to perform asynchronous networking operations with an IOCP, we need to use 'overlapped io'.
And in order to use overlapped socket io, we need to use a structure called WSAOVERLAPPED.
This structure is a wrapper for another structure called WSABUF, which is a container for a relatively small IO buffer used to send and receive data.
Most of the asynchronous socket api require that we supply these 'overlapped' structs rather than simply handing a pointer to a buffer.
It is standard practice to extend the WSAOVERLAPPED structure in order to keep more information about the contents of the buffer... I called mine XOVL - 'extended overlapped'.
But I didn't want to be working with mere structures... I wanted to make the XOVL into an OA32 class object, so that I could associate code methods with this data structure. Thus IOJob was born.

IOJob is a wrapper for XOVL which is a wrapper for WSAOVERLAPPED which is a wrapper for WSABUF which is a wrapper for our data buffer... phew!

Each IOJob contains a pointer to the Client which issued it, so when an io operation completes, NetEngine can marshal the completion notification to the very Client who 'owns' it. This is quite logical to me, and means that a Client can have any number of concurrent jobs pending completion.
Furthermore, IOJob is tagged with an 'operation code' which tells us whether the IOJob represents a recv, send, accept or some other kind of operation.

There's not a whole lot to see in this class as its mainly a container for data - next post will show the Client object, which is vastly more interesting.
Posted on 2008-11-07 20:08:31 by Homer
The real action happens in the Client class.

Here we can find methods for performing network operations ('queueing iojobs'), which we can think of as 'event sourcing'.
There are also methods for responding to 'completion notifications', which we can think of as 'event sinking'.

If we call a Client method such as 'QueueRead' or 'QueueWrite', an IOJob is allocated and an asynchronous socket api is called.
NetEngine has one 'Worker Thread' per logical cpu, these threads are essentially sleeping, waiting for an IOJob to complete.
When the IOJob completes, one of these 'Worker Threads' will wake up, and it will determine which Client issued that job, and notify that Client via one of the 'OnXXXCompleted' methods, passing the Client a pointer to the completed IOJob.
The Client will then pass that notification on to its Protocol Handler, which has a similar set of 'event sink' (OnXXXCompleted) methods.
What we do in response to this notification depends a lot apon the protocol we are implementing.
For example, let's say we received some data (a Read job completed).
The Client who issued that Read job will notify the Protocol handler, who can then examine the data - we might want to send some data back in response by issueing a Write job. Protocol handler might also communicate with the Application, so that the user can see something on the application's window.

So now we have a fairly complete picture of what is happening in regards to NetEngine's notification marshaling scheme, and we can see that the Protocol handler class is where the actual logic is implemented.

I've provided an abstract class called NetworkProtocol which allows NetEngine to communicate with classes that you, the programmer, derived from NetworkProtocol... ie, its a Base class,h your own protocol handling class or classes should inherit from NetworkProtocol, overriding its methods as you see fit. Of course you can add extra methods in your derived protocol handlers, that's completely up to you.

Attached are the classes called Client and NetworkProtocol.

Just because I am a nice guy, I have attached two more classes called EchoClientProtocol and EchoServerProtocol.
These example classes can be used to implement example applications which implement a simple echo client and echo server.
If the client sends some data to the server, the server sends it back to that client.
The client protocol handler just communicates with an application in order to fetch data (perhaps from an EDIT control) to send, and to display data that has been echoed by the server.
This is not really a network protocol - just the most simple example of what a derived protocol handler class might look like.

With these files you should be able to put together a working application.
If you'd like to see an example application project that uses these files, I can provide that too, the echo client and server projects will certainly be bundled with NetEngine in future releases of ObjAsm32 :)

Posted on 2008-11-08 01:34:08 by Homer
Several small issues in NetEngine have been addressed over the past couple of days, including some 'potential' memory leaks, and some protection against untimely thread switching screwing up critical operations (the thread scheduler can be a real pain, but we don't want completely synchronous behaviour or we may as well not be using IOCP at all).

Some code in the Client class has been moved into a new method called IOJob.Cut, this method can trim a given number of bytes from the start of that job's buffer, or if we try to cut all the buffered data, or more than that, then IOJob.Reset will be called, which quickly resets the IOJob to default state.

More interestingly, I've made some serious progress with the upnp nat traversal protocol, and embedded it directly into NetEngine.
Now when we initialize NetEngine, a udp broadcast is sent (discovery of upnp device controller), which will then either trigger a tcp query of the upnp device controller for the external ip address (needed when we want to create port mappings aka port-forwarding) and NetEngine.Init will return pThis=success, or the udp broadcast will time-out, and NetEngine.Init will return zero=failed.

I'll probably make this a little more friendly, perhaps inform the user about the failure through a messagebox, including possible reasons why upnp is not working and how to fix it - because really this only affects applications which want to accept incoming connections (servers and server-clients, not simple outbound client apps).

Now I just need to implement code to automatically create port-mappings (port forwarding) whenever the NetEngine.Listen method is called (and if I'm nice, to remove them when the listening port is closed).
This is what I really wanted!

Better still, I've made major improvements to the way that NATUPNPProtocol parses http packets.
And this forms a good foundation for a robust general-purpose HTTPProtocol class, which is something I'd like to do in a future project.

I will post an update of all files as soon as I've added the code I mentioned, and tested it thoroughly.

Posted on 2008-11-10 05:44:43 by Homer
I've finished implementing automatic forwarding of Listening ports via upnp.

It works great, however I did a little more work than was necessary.
You don't actually need to discover the external IP address !!!
You only need to discover the upnp controller's ip and port via the udp broadcast packet, and then you're ready to create port mappings.

Furthermore, there's no harm in 'recreating' (or editing) an existing port mapping - the response is identical to that returned when creating a 'new' one.

Now I just have to get in touch with my beta testers, ensure it all works for them too, and then I can post this stuff.

Posted on 2008-11-13 18:01:09 by Homer
An important addition was made to NetEngine today.
Remember me saying that IOCP can be used for more than just socket io?
It is now possible for a user to define their own io job types.

NetEngine's defined job types are now:


Any positive value outside of this range is available for any purpose you see fit.
New methods Client.OnUserIOCompleted and NetworkProtocol.OnUserIOCompleted have been added.
Therefore your user-defined job types will be passed to your protocol handler apon completion.

In order to trigger such a completion event, do the following:
-Call Client.AllocateJob to obtain an IOJob
-Set the IOJob.xovl.operation field to your user-defined operation value
-call PostQueuedCompletionStatus api function.

Your completion event will be serviced asynchronously as soon as possible, generally by another thread.
Example uses for user-defined completion notifications are playing sound effects, and reacting to collisions.
This allows you to write eventing methods with names like MyGameProtocol.OnPlayerDied which won't block the thread that issued the event!

Posted on 2008-11-13 23:23:23 by Homer
The new OnUserIOCompleted method has allowed me to better control the sequencing of operations under a network protocol - in fact, it will become a critical tool for the user to develop their own network protocol handlers to be immune to the bad effects of the thread scheduler.

Furthermore, I've added a final fallback test to NetEngine.Init method - if the user requests 'automatic port forwarding', and upnp discovery fails, NetEngine will check if the local machine's ip address falls within one of the three IANA-assigned private ip blocks.
If it does, NetEngine will alert the user via a YES/NO messagebox, telling them that their server can only work on their LAN, and cannot accept clients from the internet.

Everything is falling into place :)

Now I await feedback from my betatesters, then I'll post updates of all the files.

Biterider has been putting together his own works based strongly on mine, thats fine as long as this beast is available to OA32 programmers in some form, heh.

Honestly I can't wait until this project is complete.. With the new 'user-defined operations' methods in place, it is looking like the heart and soul of a game engine, which is my goal !!

I think OA32 is very suited to game development, and that's what I wanna do next.

Posted on 2008-11-15 03:03:55 by Homer
So far I have reports back from half of my beta testers, and so far no problems!
I've just added some basic protection against 'zombie denial of service' attacks which likely I will make a bit smarter in the near future... currently the tolerance to pain is hardcoded, it would be nice if the server side of the engine was able to detect these attacks, and scale down its tolerance to pain accordingly, for as long as the attack persists... then start to relax when the attack ceases.

Posted on 2008-11-18 09:10:02 by Homer
Here is a current update of all the NetEngine components, with the exception of the EchoServer and EchoClient demos and related protocol plugins.
For those of you who are truly interested, this will present hardly any problem.
The rest of you will have to wait for the official release in the next ObjAsm32 update.

This project is still in a state of growth, so I will not yet say this is the final state of the code, but it is certainly the current state, and I can't give more than I am giving now. This is two years of my spare time, and counting.

Have fun!

Posted on 2008-11-19 01:44:53 by Homer
I noticed that I missed IOJObPool.
Posted on 2008-11-19 23:49:43 by Homer
My next task will be to write LobbyServer and LobbyClient demos.
A LobbyServer is a server whose purpose is to track GameServers and statistics of players, and perform Authentication (aka Logins) - it is the root server of a multihomed gaming network.
People who wish to run a GameServer will connect to the LobbyServer to advertise their presence.
GameClients will connect to the LobbyServer in order to receive a list of available GameServers.
This means that individual GameServers are not the ultimate authority, and if we detect that people are running 'hacked servers', they can be pruned from the network efficiently.

The GameClient would contain both LobbyServer and LobbyClient components, allowing players to Host a game, and simultaneously allow them to Join a game, on their own GameServer, or on another GameServer.
Given appropriate bandwidth, it would be ok to run multiple GameServers, and multiple GameClients from a single ip.

GameServers interact via the LobbyServer, which makes it possible to create a huge virtual world, where players are passed from one GameServer to another when they pass a border of the game environment (Massively Multiplayer Online Games).

The abbreviated terms MMOG and MMORPG are heavily abused by marketing monkeys, and it's easy to forget what they mean... take WolfTeam for example, a dozen or so players per game is NOT massively multiplayer, no matter what they say - it's not relevant how many individual games are running, it's about how many players can fit into the same virtual space - and multihomed networks are the key to implementing truly massive worlds which are inhabited by massive numbers of players.

Posted on 2008-11-21 03:26:27 by Homer
A small change made to the Client cleanup code.. the Done and Goodbye methods are slightly altered to prevent the (Client)Pool from attempting to release object resources more than once when the application is closed.

Also, I've added some comments regarding the use of SO_LINGER with respect to nonblocking sockets and a nonzero linger time (see Client.Goodbye)... this seems to be a very much misunderstood aspect of socket programming (msdn recommends not doing it, but doesn't do a good job of explaining the implications).

Please note that Client.Goodbye should not be called directly by the user - instead, you should be calling NetEngine.GoodbyeClient, which will ensure that the object is recycled correctly.
Posted on 2008-11-21 23:25:45 by Homer
I've made a few more small bugfixes and improvements.

While playing around with client http sessions using the KEEP-ALIVE semantic, I found that webservers will sometimes terminate your session, and completely ignore the fact that we requested the session to remain open.
This can be a problem if we were not expecting it / were not finished talking to that server.

There's a new method in the Client and NetworkProtocol classes, its name is OnDisconnected.
If we detect that a connection has been lost, this method sinks that event, which gives us a way to recover from unexpected session termination.
For example, if we determine that the session terminated prematurely, we might want to try to reconnect to the server in order to finish what we were doing.

I won't post it just yet because I want to test the recovery mechanism a little more.

Posted on 2008-11-26 17:36:14 by Homer
The server side can terminate the connection at any time. Some do it because of security reasons, some do it because they're bugged, some do it because of network problems. Please take that into consideration if you want to "shutdown gracefully" in response to abnormal situations.
Posted on 2008-11-26 21:21:35 by ti_mo_n
Yep, I'm quite aware of that (now)... although strictly speaking what I was describing is a violation of the http 1.1 protocol, the fact remains that it CAN happen, and the Client needs to be stateful enough to at least attempt to resume a session that was terminated unexpectedly.
In fact, one of the new features of NetEngine is automatic pruning of Zombie Clients to avoid DoS attacks... a Client must send one complete and valid 'protocol packet' to the Server within 5 seconds of connecting, or it gets booted, regardless of the Protocol being used... a 'supervisor' thread enforces this asynchronously. This means that an attacker would need to be aware of the details of the mechanism in order to perform a successful DoS attack, and if I see that ever happen I will immediately add a second layer of protection and make an update available.

Today I moved a number of the object definitions from their individual class files into, which allows all these classes to act as 'friends', to know each other more intimately. The reason was to allow Client.Done to check whether NetEngine is closing down before attempting to release client-specific resources, and avoid a potential GPF there... on the upside, it is now possible for all the affected classes to call methods and to read and write the class variables in all the 'friend' classes.... I may even extend this to make ALL the classes in this project friendly to each other.

I know that Biterider has been using such a scheme in his version of this project for some time now, for the very same reason... but I had avoided it because I don't like seeing all the object definitions in one file, and code separate from them in other files - I've requested that he modify the MakeObjects macro to check for ".h" files and defer the inclusion of .inc files until ALL the headers have been collected, but I also know that he has not got a lot of free time, and I know that it would probably require a new macro to 'finalize' the build process.

Perhaps MakeHeaders and MakeObjects would be suitable to avoid this situation?

Anyway, since all the objects in NetEngine are exclusive to this project, I am willing to pull those headers out into as I have described... but it's not a solution for more generic objects.
Posted on 2008-11-29 23:21:26 by Homer
Last night I made an important and interesting change to the NetworkProtocol class.
I've moved all the 'event sinking' methods into a new ancestor class:

;NetEventSink decribes the 'event sinking' methods of NetworkProtocol,
;it's been defined separately in case you wish to implement
;an eventsink in the APPLICATION, that your derived protocol handler
;can forward event notifications to.. which can be a handy thing.
NAES equ 346766
Object NetEventSink,NAES,Primer
  DynamicAbstract OnReadCompleted, Pointer,Pointer,dword,dword ;pClient, pXOVL, dwBytes, dErrorCode
  DynamicAbstract OnWriteCompleted,          Pointer,Pointer,dword        ;pClient, pXOVL, dwBytes
  DynamicAbstract OnConnectCompleted, Pointer,dword             ;-> Client, dErrorCode
  DynamicAbstract OnDisconnected,            Pointer,dword                ;-> Client, dErrorCode
  DynamicAbstract OnAcceptCompleted,          Pointer,dword             ;-> Client, dErrorCode
  DynamicAbstract OnUserIOCompleted,          Pointer,Pointer,dword        ;pClient, pJob, dError
  DynamicAbstract OnClientQuit, Pointer             ;-> Client

NetworkProtocolID equ 23543
Object NetworkProtocol,NetworkProtocolID,NetEventSink
  RedefineMethod    Init,                      Pointer, Pointer    ;pOwner, lpAppEventSink

  DynamicMethod   FindDelimiter,             Pointer,Pointer ;pClient,pxovl
  DynamicMethod   ProcessReceivedData, Pointer    ;pClient

  RedefineMethod    OnReadCompleted, Pointer,Pointer,dword,dword ;pClient, pXOVL, dwBytes, dErrorCode
  RedefineMethod    OnWriteCompleted,          Pointer,Pointer,dword        ;pClient, pXOVL, dwBytes
  RedefineMethod OnConnectCompleted, Pointer,dword     ;-> Client, dErrorCode
  RedefineMethod    OnDisconnected,            Pointer,dword        ;-> Client, dErrorCode
  RedefineMethod    OnAcceptCompleted,          Pointer,dword     ;-> Client, dErrorCode
  RedefineMethod    OnUserIOCompleted,          Pointer,Pointer,dword  ;pClient, pJob, dError
  RedefineMethod OnClientQuit, Pointer     ;-> Client

  DefineVariable    lpAppEventSink,            Pointer, NULL

The idea is that you can add a derived NetEventSink in your application, and inform NetworkProtocol via the Init method... If you do this,  NetworkProtocol will notify the Application's event sink after the Protocol methods have executed and before NetEngine sees the result.
This arrangement has two benefits.
1: We have a general-purpose mechanism for passing notifications to the Application.
2: We have a means to override some of the default behaviour of the protocol handler, even if the protocol handler is enshrined in the form of a pluggable object such as a DLL, without screwing around with the code in there.

Posted on 2008-11-30 17:58:27 by Homer
As mentioned, I've been working on a demo 'Lobby Server' and client.
I'm interested in developing online games, but this could be used for a multihomed chat system such as Yahoo or IRC, it could be used for a centralized filesharing network, or dozens of other purposes.

A database of user accounts has been implemented, and Authentication of account via user/pass is also implemented.

User Account records contain username, password, and optionally, email address.
If you attempt to authenticate a username which does not exist, the account will be created automatically for you, and your client will be notified of this.

You don't have to supply an email address, but if you don't (or its not actually valid), and you forget your password, there will be no way for you to reclaim your account !!!

I haven't implemented password recovery yet, because I don't have any email protocol handler written at this stage.

If a client fails authentication (due to wrong password, the only possible reason), the session is terminated.
But if auth succeeds, the session remains open so that the client can make further requests, such as obtaining a list of potential hosts, advertising as a host, or enquiring as to the current location of a specific user.

Again, nothing beyond account authentication / creation has been implemented yet, however it won't take long to add the code to handle all these tasks.

Also, I am thinking about replacing the EMAIL record with an 'answer to secret question' as some other existing servers already do.
This would allow the user to recover (or at least reset) their password *without* requiring an email address, or me having to write a protocol handler for email.
Finally, I am thinking about replacing the plaintext password with a hash - you wouldn't be able to recover your password, because the server would not keep the password, only the hash... it also means that bad guys can't capture your password with a packet sniffer - they could still perform 'replay attacks', I guess no system is perfect but sending plaintext is a bad idea - maybe encrypting the entire session with a random session key would be the way to fly, that would certainly prevent replay attacks, and make it difficult to reverse engineer the protocol.

Posted on 2008-11-30 22:15:34 by Homer
Having moved past the authentication stuff (I'm sure I'll revisit that), the next step was to implement the code for clients to advertise as a host server, for the lobbyserver to add host servers, for the clients to fetch a list of host servers, and for the lobbyserver to build and send that list.
All of this has been done, however there's some simple design flaws that need to be taken care of.
I've noticed that the Server is not setting the socket address of the remote clients it accepts.... their ip address appears to be, which is obviously wrong.
That's cool, I'll call getpeeraddress after accept completes (in NetEngine) and that will take care of it.
It's good to find this out, it's obviously slipped under the radar.
By tomorrow, I'll basically have the LobbyServer and LobbyClient stuff completed, except for the security aspects, which I have already mentioned.

That will mean I can temporarily shut the book on the Lobby stuff, and move on to the more interesting (for me) GameServer/GameClient, which is my goal !!!
Multihomed massively multiplayer online gaming, here I come !!!
Posted on 2008-12-01 23:26:15 by Homer

I do not like the concept of the server storing the passwords. I think it is much better for the server to store the hash of the password. As to resetting password, why not regenerate a random password and send to the email? Allow the user to change only after he logs in with the new generated password.
Posted on 2008-12-01 23:53:43 by roticv
These are all ideas I have already mentioned, and I completely agree.
I also mentioned replay attacks, where a hash is not good enough by itself - there needs to be a session-key which encrypts the data, then it doesn't really matter if its a hash or a password, except in terms of the size of the data.

Since the session key (which forms the basis of the encryption key) is only valid within the context of a single session, a replay attack simply won't work - the Server will attempt to use a new session key to decrypt the client's data (which has been encrypted with an OLD session key) - resulting in garbage, which will be rejected by the protocol handler and result in the client being disconnected.

Posted on 2008-12-02 00:00:17 by Homer
An important change was made to the Client.OnAcceptCompleted method - I'm now using the GetAcceptExSockaddrs api to grab both the local and remote socket address structures and copying them into the newly-accepted Client. This is very important since we can't know what the remote ip address of a client is until accept has completed, and we cannot use the getpeername api on unbound sockets.
Now that I can obtain the ip of the remote client, I can do things like comparing it against a black/white list if I decide I want to ban people for some reason.

Posted on 2008-12-02 08:01:34 by Homer