Calling all those with knowledge of IO Completion ports and it's use with WinSock!

I have downloaded and studied about 10 examples of this type of server implemented in various ways in C/C++ code ... Basically, my server is almost complete, except for one small problemo :(

I can successfully create the IO Completion port using CreateIoCompletionPort() (still waiting patiently for Hutch to add this API to the Kernel32.Lib and Kernel32.Inc files so I don't have to add it at runtime :( ), I can successfully add listening or connected sockets to the IO Completion Port I created as well! But the problem basically is that WinSock doesn't seem to signal the Port when network events such as FD_ACCEPT, FD_WRITE, FD_READ and FD_CLOSE occur!

So my calls in the working threads to the GetQueuedCompletionStatus() API never release! If I use PostQueuedCompletionStatus() API to send the Completion Port a message however, it releases and returns with my message! So everything is as it should be ... I think :confused:

My calls to CreateIoCompletionPort(), GetQueuedCompletionStatus() and PostQueuedCompletionStatus() don't return any errors ... everything seems normal and as it should be ... just that WinSock isn't sending network event messages to the port ... can someone maybe explain to me how WinSock is supposed to know that it must send messages there (To the Completion Port) ? It just doesn't seem to know that the sockets are using a Completion Port for communication !

I can fix up my code leaving only the necessary stuff and post it to you for analysis if it helps?

Thanx!

PS: I posted it here for 2 reasons :) The mods will probably move it to the NetWorking section so I get both sections and Completion Ports are used by other areas such as ReadFileEx and WriteFileEx API's .... not just WinSock :)
Posted on 2002-12-11 00:46:59 by SubEvil
You should probably familiarize yourself intimately with the other socket IO models before you begin tackling IOCPs.

Anyways, the reason you aren't getting FD_xxxxx notifications is because that's not how it's done with the IOCP model. You have to set your own event aliases and pass them as an appendage to the lpOverlapped structure when you queue an event on the port.
Posted on 2002-12-11 05:35:52 by iblis
Hi Iblis,

I'm intimately familiar with the other methods ... well the WSAAsyncSelect/WSAEventSelect methods ... I can do them in my sleep! I just used the FD_xxx thing to try and relate where my problems lay! I have made many successful multi-threading servers with the other two methods, I even use my other servers to try and connect to my new IOCP server ... to no avail :( I even made a sorta WSAEventSelect (For the Listening Socket) IOCP Hybrid (For the Connected Sockets) ...

Anyway ... so back to this setting an event alias ... I have seen something similar being done in some examples ... but didn't quite get it !
How exactly do you tell WinSock to use these events? I mean ... how does WinSock know to post a message in the IOCP queue? You said "when you queue an event on the port"??? That's scarry ... I though WinSock would be queing events ... not me!

Question: What if you only want "FD_READ" events posted on a particular socket? In WSAAsyncSelect you "or" the FD_xxx values together ... so how do you tell the IOCP just to notify you when data has arrived on a socket ?
Question: Does WinSock change the OVERLAPPED structure?
Question: What is the use of the OVERLAPPED structure ? You don't actually need it do you ? I will know which OVERLAPPED structure is used by which socket from the Completion Key generally! But is there anything in the OVERLAPPED structure of use ?
Question: How does WinSock know what messages to post to the IOCP for a particular socket? Just by adding the socket to the IOCP ??? Is that enough? Because I successfully do that but WinSock isn't sending any messages to my IOCP!

You see the issue is I can make an IOCP, I can connect Sockets to it ... but WinSock doesn't seem to Post any messages to it! So when I call GetQueuedCompletionStatus() to wait for messages, it never releases when a client sends me data! It just sits there waiting ... but I can post it manual messages and it works fine :(

Perhaps you could briefly explain how the messaging system works in the IOCP model in comparison to the others?

I would really appreciate your help! There doesn't seem to be many here that know about IOCP's. If you have some ASM source code I could look at I'd love that ... otherwise just a little help understanding!
Posted on 2002-12-11 07:50:47 by SubEvil
SubEvil,
The third parameter of CreateIoCompletionPort allows a completion key to be specified along with the
socket handle to be associated. U can use this to pass context information that is associated with the
socket. It is a pointer to your own EXTENDED OVERLAPPED structure.
GetQueuedCompletionStatus will return the same pointer when the operation completes, i.e.
GetQueuedCompletionStatus will return the same pointer to your own EXTENDED OVERLAPPED
structure when the operation completes. That's all.
No FD_ACCEPT, FD_WRITE, FD_READ, FD_CLOSE and other bla, bla, blab..

What about accept API u use?
If you post a code I can help more

Regards,
Lingo
Posted on 2002-12-12 16:14:00 by lingo12
Question: Can a listening socket be added to an IOCP? Or only connected "normal" client sockets?

Question: Is the overlapped/extened overlapped structure absolutely necessary in IOCP? And why?

Question: How does WINSOCK know the socket is communicating through IOCP ???
eg. We added the socket to the IOCP via CreateIoCompletionPort() API ... so does WinSock now automatically send network status messages to the IOCP just by using this API? WinSock and this API don't seem to be related!


Anyway,
I've included a sample of my Server, took me long to simplify this code but I'm desperate to understand IOCP now!
You will see, the IOCP is created, no problems, the worker thread is created, no problem, WinSock is initialized, no problem, a listening socket is created and added to the IOCP, no problem!

BUT

When a client connects, WinSock doesn't notify the IOCP of the new client! So before I can issue an AcceptEx/Accept call on the new socket, the message is lost!

I added 3 dummy message posts to the IOCP to see if it's OK (you will see 3 messageboxes pop up when it starts, they are from the Worker Thread) ... no problems ... I can send custom messages to the IOCP but WinSock never does, it doesn't let the listening socket on the IOCP know we have a new connection!

You will notice the Worker Thread posts a simple message whenever it get's something, but WinSock has NEVER sent my Worker Thread ANY messages! Not on connect, not on data send or when data is received on a socket NEVER :( That's my only problem! WinSock doesn't seem to communicate through the IOCP! Even if I successfully add the socket to the IOCP :(

GetQueuedCompletionStatus will return the same pointer to your own EXTENDED OVERLAPPED
structure when the operation completes.

GetQueuedCompletionStatus has NEVER returned when a WinSock operation completes! = My problem
Posted on 2002-12-13 12:47:48 by SubEvil
I just briefly glanced over your code and noticed a few things right off the bat. Among them is that you apparently have no idea what you're doing. You should probably go get a book on Winsock 2 or something and save yourself the frustration.

Anyways, here are the biggies:


1) You create the IOCP, but never add any handles to it. If you want it to process socket events then you'll have to do that. An example of what you should do:

[size=12];First create the port.

invoke CreateIoCompletionPort, INVALID_HANDLE_VALUE, NULL, 0, 0
or eax, eax
jz error
mov hIOCP, eax

...

;Add socket handle to the port.
;(later on, once you have a socket)
invoke CreateIoCompletionPort, hsocket, hIOCP, addr SocketContext, 0
or eax, eax
jz error[/size]



Secondly, you never receive or send anything on those sockets. I.E.:

[size=12]    invoke WSASend, hsocket, addr wsabuf, 1, addr nBytes, 0, addr extOverlapped, NULL

cmp eax, SOCKET_ERROR
jne noerror
invoke WSAGetLastError
cmp eax, WSA_IO_PENDING
jne error
noerror:
...[/size]



There are of course more issues you missed, but you should probably grasp these concepts (and the extended overlapped structure Lingo and I were talking about) before continuing.
Posted on 2002-12-13 13:35:51 by iblis
SubEvil,
I saw your code and unfortunately iblis is right
I didn't see how you manage the clients,
accept the connections, send and receive data
ASYNCHRONOUSLY, close the connections, etc.
You need to learn the differences between listen
and accepted sockets too..
I suppose you can read C because your 286 assembly
is similar and here is an example about IOCP.
If your next code has all the attributes from it
I will continue to help you, else I'm not sure


#include "stdafx.h"
#define WIN32_LEAN_AND_MEAN
#include <winsock2.h>

#include "iocpserver.h"

unsigned short g_Port = 5001;
BOOL g_bEndServer = FALSE; // set to TRUE on CTRL-C
DWORD g_dwThreadCount = 0; //worker thread count
HANDLE g_hIOCP = NULL;
SOCKET g_sdListen = INVALID_SOCKET;
HANDLE g_hThreads[MAX_WORKER_THREAD];
CPtrList g_CtxtList;
CRITICAL_SECTION g_CriticalSection; // guard access to the global context list

#ifdef _DEBUG
#define new DEBUG_NEW
#undef THIS_FILE
static char THIS_FILE[] = __FILE__;
#endif

// The one and only application object
CWinApp theApp;
using namespace std;

int _tmain(int argc, TCHAR* argv[], TCHAR* envp[])
{
int nRetCode = 0;

// initialize MFC and print and error on failure
if (!AfxWinInit(::GetModuleHandle(NULL), NULL, ::GetCommandLine(), 0))
{
cerr << _T("Fatal Error: MFC initialization failed") << endl;
return (nRetCode = 1);
}
SYSTEM_INFO systemInfo;
WSADATA wsaData;
SOCKET sdAccept = INVALID_SOCKET;
PPER_SOCKET_CONTEXT lpPerSocketContext = NULL;
DWORD dwRecvNumBytes = 0;
DWORD dwFlags = 0;
DWORD numClients = 0;
int nRet;

g_bEndServer = FALSE;
GetSystemInfo(&systemInfo);
g_dwThreadCount = systemInfo.dwNumberOfProcessors * 2;
if ((nRet = WSAStartup(MAKEWORD(2,2), &wsaData)) != 0)
{
printf("WSAStartup failed: %d\n",nRet);
return 1;
}
InitializeCriticalSection(&g_CriticalSection);
g_hIOCP = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);
if (NULL == g_hIOCP)
{
printf("CreateIoCompletionPort failed: %d\n", GetLastError());
CleanUp();
return 1;
}
for (DWORD dwCPU=0; dwCPU < g_dwThreadCount; dwCPU++)
{
HANDLE hThread;
DWORD dwThreadId;

hThread = CreateThread(NULL, 0, EchoThread, g_hIOCP, 0, &dwThreadId);
if (hThread == NULL)
{
printf("CreateThread failed: %d\n", GetLastError());
CleanUp();
return 1;
}
g_hThreads[dwCPU] = hThread;
}
if (!CreateListenSocket())
{
CleanUp();
return 1;
}


//[B]; WHERE IS NEXT PART IN YOUR ASM CODE[/B]


while (g_bEndServer == FALSE)
{
sdAccept = WSAAccept(g_sdListen, NULL, NULL, NULL, 0);
if (SOCKET_ERROR == sdAccept)
{
printf("WSAAccept: %d\n", WSAGetLastError());
CleanUp();
return 1;
}
printf("Accept Listen socket: %d\n", numClients++ );
lpPerSocketContext = UpdateCompletionPort(sdAccept, ClientIoRead);
if (NULL == lpPerSocketContext)
{
CleanUp();
return 1;
}
// post initial receive on this socket
nRet = WSARecv(sdAccept, &(lpPerSocketContext->pIOContext->wsabuf), 1,
&dwRecvNumBytes, &dwFlags,
&(lpPerSocketContext->pIOContext->Overlapped), NULL);
if (nRet == SOCKET_ERROR && (ERROR_IO_PENDING != WSAGetLastError()))
{
printf("WSARecv Failed: %d\n", WSAGetLastError());
CloseClient(lpPerSocketContext);
}
} //while
CleanUp();
return 1;
} //main
// Create a listening socket.
BOOL CreateListenSocket(void)
{
SOCKADDR_IN si_addrlocal;
int nRet;
int nZero = 0;
g_sdListen = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);
if (INVALID_SOCKET == g_sdListen)
{
printf("WSASocket(g_sdListen): %d\n", WSAGetLastError());
return(FALSE);
}
si_addrlocal.sin_family = AF_INET;
si_addrlocal.sin_port = htons(g_Port);
si_addrlocal.sin_addr.s_addr = htonl(INADDR_ANY);
nRet = bind(g_sdListen, (struct sockaddr *)&si_addrlocal, sizeof(si_addrlocal));
if (SOCKET_ERROR == nRet)
{
printf("bind: %d\n", WSAGetLastError());
return(FALSE);
}
nRet = listen(g_sdListen, 5);
if (SOCKET_ERROR == nRet)
{
printf("listen: %d\n", WSAGetLastError());
return(FALSE);
}
return(TRUE);
}
// Worker thread that handles all I/O requests on any socket handle added to the IOCP.
//
DWORD WINAPI EchoThread (LPVOID WorkThreadContext)
{
HANDLE hIOCP = (HANDLE)WorkThreadContext;
BOOL bSuccess = FALSE;
int nRet;
LPOVERLAPPED lpOverlapped = NULL;
PPER_SOCKET_CONTEXT lpPerSocketContext = NULL;
PPER_IO_CONTEXT lpIOContext = NULL;
WSABUF buffRecv;
WSABUF buffSend;
DWORD dwRecvNumBytes = 0;
DWORD dwSendNumBytes = 0;
DWORD dwFlags = 0;
DWORD dwIoSize;
while (TRUE)
{
// continually loop to service io completion packets
bSuccess = GetQueuedCompletionStatus(hIOCP,
&dwIoSize, (LPDWORD)&lpPerSocketContext,
&lpOverlapped,INFINITE);
if (!bSuccess)
printf("GetQueuedCompletionStatus: %d\n", GetLastError());

if (lpPerSocketContext == NULL)
{
return(0);
}

if (g_bEndServer)
{
return 0;
}
if (!bSuccess || (bSuccess && (0 == dwIoSize)))
{
// client connection dropped, continue to service remaining (and possibly
// new) client connections
CloseClient(lpPerSocketContext);
continue;
}
// determine what type of IO packet has completed by checking the PER_IO_CONTEXT
// associated with this socket. This will determine what action to take.
lpIOContext = (PPER_IO_CONTEXT)lpOverlapped;
switch (lpIOContext->IOOperation)
{
case ClientIoRead:
// a read operation has completed, post a write operation to echo the
// data back to the client using the same data buffer.
lpIOContext->IOOperation = ClientIoWrite;
lpIOContext->nTotalBytes = dwIoSize;
lpIOContext->nSentBytes = 0;
lpIOContext->wsabuf.len = dwIoSize;
dwFlags = 0;
nRet = WSASend(lpPerSocketContext->Socket,
&lpIOContext->wsabuf, 1, &dwSendNumBytes,
dwFlags, &(lpIOContext->Overlapped), NULL);
if (SOCKET_ERROR == nRet && (ERROR_IO_PENDING != WSAGetLastError()))
{
printf("WSASend: %d\n", WSAGetLastError());
CloseClient(lpPerSocketContext);
}
break;
case ClientIoWrite:
// a write operation has completed, determine if all the data intended to be
// sent actually was sent.
lpIOContext->IOOperation = ClientIoWrite;
lpIOContext->nSentBytes += dwIoSize;
dwFlags = 0;
if (lpIOContext->nSentBytes < lpIOContext->nTotalBytes)
{
// the previous write operation didn't send all the data,
// post another send to complete the operation
buffSend.buf = lpIOContext->Buffer + lpIOContext->nSentBytes;
buffSend.len = lpIOContext->nTotalBytes - lpIOContext->nSentBytes;
nRet = WSASend (
lpPerSocketContext->Socket,
&buffSend, 1, &dwSendNumBytes,
dwFlags,
&(lpIOContext->Overlapped), NULL);
if (SOCKET_ERROR == nRet && (ERROR_IO_PENDING != WSAGetLastError()))
{
printf ("WSASend: %d\n", WSAGetLastError());
CloseClient(lpPerSocketContext);
}
}
else
{
// previous write operation completed for this socket, post another recv
lpIOContext->IOOperation = ClientIoRead;
dwRecvNumBytes = 0;
dwFlags = 0;
buffRecv.buf = lpIOContext->Buffer,
buffRecv.len = MAX_BUFF_SIZE;
nRet = WSARecv(
lpPerSocketContext->Socket,
&buffRecv, 1, &dwRecvNumBytes,
&dwFlags,
&lpIOContext->Overlapped, NULL);
if (SOCKET_ERROR == nRet && (ERROR_IO_PENDING != WSAGetLastError()))
{
printf ("WSARecv: %d\n", WSAGetLastError());
CloseClient(lpPerSocketContext);
}
}
break;
} //switch
} //while
return(0);
}
PPER_SOCKET_CONTEXT UpdateCompletionPort(SOCKET sd, IO_OPERATION ClientIo)
{
PPER_SOCKET_CONTEXT lpPerSocketContext;

// PER_SOCKET_CONTEXT
lpPerSocketContext = CtxtAllocate(sd, ClientIo);
if (lpPerSocketContext == NULL)
return NULL;

g_hIOCP = CreateIoCompletionPort((HANDLE)sd, g_hIOCP,
(DWORD)lpPerSocketContext, 0);
if (NULL == g_hIOCP)
{
printf("CreateIoCompletionPort: %d\n", GetLastError());
if (lpPerSocketContext->pIOContext)
free(lpPerSocketContext->pIOContext);
free(lpPerSocketContext);
return(NULL);
}
CtxtListAddTo(lpPerSocketContext);

return(lpPerSocketContext);
}
VOID CloseClient (PPER_SOCKET_CONTEXT lpPerSocketContext)
{
EnterCriticalSection(&g_CriticalSection);
if (lpPerSocketContext)
{
closesocket(lpPerSocketContext->Socket);
CtxtListDeleteFrom(lpPerSocketContext);
lpPerSocketContext = NULL;
}
else
{
printf("CloseClient: lpPerSocketContext is NULL\n");
}
LeaveCriticalSection(&g_CriticalSection);
return;
}
PPER_SOCKET_CONTEXT CtxtAllocate(SOCKET sd, IO_OPERATION ClientIO)
{
PPER_SOCKET_CONTEXT lpPerSocCon;

lpPerSocCon = (PPER_SOCKET_CONTEXT)malloc(sizeof(PER_SOCKET_CONTEXT));
if (lpPerSocCon)
{
lpPerSocCon->pIOContext = (PPER_IO_CONTEXT)malloc(sizeof(PER_IO_CONTEXT));
if (lpPerSocCon->pIOContext)
{
lpPerSocCon->Socket = sd;
memset(&lpPerSocCon->pIOContext->Overlapped, 0, sizeof(OVERLAPPED));
lpPerSocCon->pIOContext->IOOperation = ClientIO;
lpPerSocCon->pIOContext->nTotalBytes = 0;
lpPerSocCon->pIOContext->nSentBytes = 0;
lpPerSocCon->pIOContext->wsabuf.buf = lpPerSocCon->pIOContext->Buffer;
lpPerSocCon->pIOContext->wsabuf.len = MAX_BUFF_SIZE;
}
else
{
free(lpPerSocCon);
lpPerSocCon = NULL;
}
}
return(lpPerSocCon);
}
VOID CtxtListAddTo (PPER_SOCKET_CONTEXT lpPerSocketContext)
{
EnterCriticalSection(&g_CriticalSection);
g_CtxtList.AddTail(lpPerSocketContext);
LeaveCriticalSection(&g_CriticalSection);
return;
}
VOID CtxtListDeleteFrom(PPER_SOCKET_CONTEXT lpPerSocketContext)
{
EnterCriticalSection(&g_CriticalSection);
if (lpPerSocketContext)
{
POSITION pos = g_CtxtList.Find(lpPerSocketContext);
if (pos)
{
g_CtxtList.RemoveAt(pos);
if (lpPerSocketContext->pIOContext)
free(lpPerSocketContext->pIOContext);
free(lpPerSocketContext);
}
}
LeaveCriticalSection(&g_CriticalSection);
return;
}
VOID CtxtListFree()
{
EnterCriticalSection(&g_CriticalSection);
while (!g_CtxtList.IsEmpty())
{
PPER_SOCKET_CONTEXT lpPerSocketContext;

lpPerSocketContext = (PPER_SOCKET_CONTEXT)g_CtxtList.RemoveHead();
if (lpPerSocketContext->pIOContext)
free(lpPerSocketContext->pIOContext);
free(lpPerSocketContext);
}
LeaveCriticalSection(&g_CriticalSection);
return;
}
void CleanUp()
{
// Cause worker threads to exit
if (g_hIOCP)
{
for (DWORD i = 0; i < g_dwThreadCount; i++)
PostQueuedCompletionStatus(g_hIOCP, 0, 0, NULL);
}
//Make sure worker threads exits.
if (WAIT_OBJECT_0 != WaitForMultipleObjects( g_dwThreadCount, g_hThreads, TRUE, 1000))
printf("WaitForMultipleObjects failed: %d\n", GetLastError());
else
for (DWORD i = 0; i < g_dwThreadCount; i++)
{
if (g_hThreads[i] != INVALID_HANDLE_VALUE) CloseHandle(g_hThreads[i]);
g_hThreads[i] = INVALID_HANDLE_VALUE;
}
CtxtListFree();
if (g_hIOCP)
{
CloseHandle(g_hIOCP);
g_hIOCP = NULL;
}
if (g_sdListen != INVALID_SOCKET)
{
closesocket(g_sdListen);
g_sdListen = INVALID_SOCKET;
}
DeleteCriticalSection(&g_CriticalSection);
WSACleanup();
}


Regards,
Lingo
Posted on 2002-12-13 15:29:54 by lingo12
Quite frankly, I don't think I need help from the two of you because I believe I already know more about IOCP and how it works than both of you put together! I asked very simple and clear questions and neither of you answered them! As for my 286 assembly, nither of you understand that the code I posted was a simplified version of a massive server I'm converting from WSAAsyncSelect & WSAEventSelect and took me a few hours to put together for you because someone said they could help if I posted code here! My current server is too big and you would be lost if I posted it here! You don't understand that I just demonstrator one thing in this code, it just listens for incomming connections, DOESN'T accept them, DOESN'T send or receive either, just sets a listening socket, adds it to the IOCP and waits, when a connection comes, I'm supposed to get a MessageBox popup where I will add code to accept the connection etc. It is a simple server and if you guys knew anything about IOCP you would know what was happening, I made it clear in code and added titles to each section to make it more clear!

Iblis

If you actually read the code you would find I have your "create the port" code almost word for word in my tiny project! You would also see I add the listening socket to the IOCP with VERY similar code to yours, obviously just changing variable name etc. And you obviously don't undertand what my code is supposed to achieve (only listens), to demonstrate my problem, but you obviously have no clue what that problem is!

lingo12

I know the code you posted backwards, you have just made minor modifications to it but I have studied the origional complete echo server code in depth! If you think "Iblis is right", then you are just as clueless as he is about my problems! My code doesn't need "all the attributes from it" because I have a very specific problem which is demonstrated in my "286 assembly code"! I don't need to "manage my clients" to demonstrate my problem! If you think I have to write a whole complete IOCP server app to demonstrate my problem then you've missed it already!

For crying out load guys, I have a specific problem, I am 100000% knowledgable on WinSock and how it works, I don't have a WinSock specific problem, I have a WinSock interfacing with IOCP problem, I don't need to manage the clients, send and receive etc. to demonstrate my problem! Just assemble the code, it sits and waits for a connection, then use another WinSock app and connect, the client connects to it, you will see, but in my server's IOCP worker thread, I'm supposed to get a messagebox saying something like "IOCP have received a message from outerspace!", that just symbolises that IOCP has received a message, any message, it doesn't matter, but WinSock never sends a message to IOCP with the PostQueuedCompletionStatus() API. I send 3 test messages to the IOCP in the beginning of the code to test that the IOCP is setup ...
Posted on 2002-12-17 00:12:20 by SubEvil
SubEvil,

I didn't want to offend you, and your emotions are surprise for me
I just expected translation of working example from C to ASM

"I have studied the origional complete echo server code in depth! "

Would you be so kind to explain how your problem is solved
by the example?
Which sockets are associated with an IOCP, listen or accepted?
What about notification you looking for if you didn't initialize any events?
What means "overlapped I/O completion"?
What about the contents of your overlapped structures of notification packets for every socket?
What about the data in the linked list?

I can continue..

When an overlapped call is made, a pointer to an overlapped structure is passed as a parameter. GetQueuedCompletionStatus will return the same pointer when the operation completes succesfuly.
Where is your overlapped call?

Regards,
Lingo
Posted on 2002-12-17 02:36:00 by lingo12
My appologies Lingo12 ... I know you're trying to help and thanx! I must appologize for my emotions too, I'm not very good at taking criticism ... sorry ...

From the questions you are asking me ... I can tell I've still got some things to understand about IOCP and perhaps I'm asking the wrong questions myself! I will try to answer your questions as best I can ... then you can comment on my answers and explain things to me! You see I've only read example code ... I haven't found any good tutorials on WinSock IOCP ... not even from MSDN ... mostly they are just general IOCP tutorials!

Would you be so kind to explain how your problem is solved by the example?

The example doesn't add the Listening socket to the IOCP. This is a question I have already asked you ... Does the function of a listening socket get performed by the IOCP?

Which sockets are associated with an IOCP, listen or accepted?

Again, something I have asked you guys to answer, my answer would be ... accepted sockets get added by your example (Not mine, is that wrong?) I have tried this method already without success!

What about notification you looking for if you didn't initialize any events?

What exactly do you mean by this? This could be what is missing in my understanding of IOCP! Why do I need to initialize events? Do you mean with the CreateEvent() calls? What type of events are you talking about?

What means "overlapped I/O completion"?

Basically, multiple IO calls get processed together and the app is notified when they have completed!

What about the contents of your overlapped structures of notification packets for every socket?

I will solve this by having a dynamic array of connected sockets, I will use the same methods I've used before with my other server types.

What about the data in the linked list?

What data? The socket specific data will be stored in my dynamic arrays, eg. Mutexes, Sockets States, client specific data etc. Don't worry about this, this is the least of my problems!

Where is your overlapped call?

Now this could be the key to my IOCP understanding!
Let me ask YOU what calls perform overlapped operations? Only WSASend() & WSARecv()? What about AcceptEx()? Or just Accept()?
Maybe I have IOCP calls in my code! That would mean I never get any IOCP messages! Perhaps my understanding of what IOCP is used for with WinSock is flawed?

Do you have a good WinSock IOCP TUTORIAL ... not C example code for me ??? I have plenty of example code!

Thanx
Posted on 2002-12-17 04:33:54 by SubEvil
SubEvil, if I was not trying to help you then I would not have even bothered posting here. Teaching you the basics of completion ports is not my job nor do I owe you anything, so if you want to get angry at those trying to help you then I'll be happy to stop.


Anyways, regarding the listening socket, You *can* have it send connects down the port but you don't have to, and since you are new to this, to keep things simple you should just use listen() and accept() and then update the IOCP with the connected socket. If performance is what you're after, you won't lose any by doing this. Just do all your listening in another thread.

The extended OVERLAPPED structure contains "per socket data" meaning that anything you wish to associate with your socket, i.e. buffers, data size, etc. you need to 'extend' the overlapped structure by adding this data to the end of it. This includes IO operation type.

I have downloaded and studied about 10 examples of this type of server implemented in various ways in C/C++ code


I don't think you studied them very well. You should go back and look at them. Everything we are describing is most probably in there. Take your time and read them carefully. Look how the examples make use of the overlapped structure. I think things will seem clearer to you once you do this.


As a side note, I don't know what sort of miraculous server you hope to have with this. Just because the IOCP model claims the capability of handling many connections doesn't mean it necessarily will, and I think that if you ever finish this program you will be disappointed to find out that it doesn't do what you expect. There are a lot more issues to address than just what socket IO model to use. You really need to buy a book on this. It will help you immensely.
Posted on 2002-12-17 05:27:22 by iblis
SubEvil,

Here is the main thread and the working thread usage

Role of the main thread (human resources):

1. Create empty IOCP(the system) and working thread(supervisor)
Why?
Because if we have workers, someone must supervise them (see later)

2. Create new workers (accepted sockets) with WSAAccept() or AcceptEx()
with qualifications inherited from their father(listen socket)

Why workers?
Because they do the job(receive and send data from/to clients)
How?
Asynchronously->hence, we must use overlapped structure because of that

Where to keep overlapped structure and other info about every worker?
In his personal info file (his personal extended overlapped structure).

Why extended overlapped structure?
Because it contents a lot of bureaucracy:
- his overlapped structure
- his name (accepted socket handle)
- his current job (status flag); for instance:
1 - means I'm reading data now, 2 - means I'm writing now, etc.
- how many bytes must be read/write
- how many bytes are read
- etc.
You (the programmer) can extend the bureaucracy (extended overlapped structure)
with additional records if you or the supervisor (working thread) will need it later!

Who changes the status flag? For the first time after creation,
the main thread (human resources), next,
after every particularly finished job the supervisor(working thread)

Who needs a bureaucracy? Mainly the supervisor(working thread), see later

3. Associate them (new workers = accepted sockets) with IOCP (the system)

How?
- Create and fill new personal info file about new worker (his personal
extended overlapped structure).
- update his current job (status flag) to 1 (for instance: 1 - means I'm reading data now)
and post initial asynchronous receive on this worker (accepted socket) with WSARecv()
- Save the pointer of his file anywhere in memory for easy retrieving later (linked list),
and send this pointer to the IOCP (the system) with his name (accepted socket handle) as
parameters, via CreateIoCompletionPort() API

4. Jmp again to 2


Role of working thread (supervisor):

Worker thread (supervisor) handles all I/O requests on any socket handle added to the IOCP
by main thread, i.e. here we have one supervisor(working thread) and one or more
workers (accepted sockets)

A. Ask the system (IOCP) is there any particularly finished job via
GetQueuedCompletionStatus() API
If there is no completion packet (particularly finished job info) queued, the function waits
for a pending I/O operation associated with the completion port to complete.

B. The system (IOCP) returns the pointer of any worker's file
(i.e. pointer to his personal extended overlapped structure) with particularly finished job,
and number of bytes transferred during an I/O operation that has completed.

C. Determines what next action to take (WSARecv,WSASend or/and other server routine),
delegates it to this worker (accepted socket) and updates his personal info file
(i.e. his extended overlapped structure) accordingly.

D. Jmp again to A


Regards,
Lingo
Posted on 2002-12-17 22:50:59 by lingo12
lingo12, I have just tried to implement your code...
I get 10045 on WSARecv - odd?
Posted on 2004-07-21 06:51:58 by Homer