dunno exactly what advantages you could get with GDI hooking (except for perhaps immediate updates on LAN connections) - but if one thinks it through, it might be possible to come up with some clever tricks.

WM_PAIN hooking should be possible globally with a simple windows hook. SetWindowsHookEx() ...
Posted on 2003-04-08 15:02:29 by f0dder
another question:

should the server send the data as soon as new data is available (ie a new window opens).
or should the client always make a request (this would make sense on slow connections. not sure about lans: you don't wan't to pollute the intranet with stupid requests each 10ms (assumed that we want instant view of the actions taking place on the remote screen))


hmm conclusion:

if new data is available, the server sends it. after receiving the client must send an 'ack'-command. the server is only permitted to send new data if the ack of its last data is recevied. until then, the server holds the screen updates in a buffer.


edit: as far as i remember, the VNC implementation said that the client does have to make requests. IMHO this isn't the most suitable solution.
Posted on 2003-04-08 16:51:59 by nyook
well, catching WM_Paint doesn't fit all needs. This way we can only be told updates which windows did order. But (I think it has been already mentioned above) windows which paint some special things (like winamp does) aren't considered in this approach.

I think a hook to ReleaseDC makes sense, because I think applications use CreateDC, then do some painting stuff and immediately call ReleaseDC (Well, not 100% sure about this).
With this hook we could get an accurate hit on screen changes. (One must be aware that some parts of the screen change too often and therefore care must be taken not to waste bandwith with this minor information).
Posted on 2003-04-09 08:13:04 by nyook
things that are BitBlt'ed outside wm_paint will probably most of the time be likely candidates for "updated too fast" - so a WM_PAINT hook is probably still the best generic idea? Furthermore it should be relatively simple to do, so I guess it's at least worth a try doing an initial implementation like this.
Posted on 2003-04-09 08:33:41 by f0dder

things that are BitBlt'ed outside wm_paint will probably most of the time be likely candidates for "updated too fast" - so a WM_PAINT hook is probably still the best generic idea? Furthermore it should be relatively simple to do, so I guess it's at least worth a try doing an initial implementation like this.



You're right.

EliCZ' ApiHooks would be a nice tool for hooking the api (at a later step), although I dislike using something which I don't have the source of, because I'd like to know _how_ to do it. But I think that this would require a lot more learning.
Posted on 2003-04-09 08:56:16 by nyook
MSDN says global hooking is costly. Does anyone know if it is so much that the user can feel it? *implementing a wm_paint hook right now*
Posted on 2003-04-09 10:31:08 by nyook
Yes it can be very costly indeed, depends how well you will implement it, but from what i worked on such matters it will be quite noticeable on the server or hooked machine. The whole thingy about hooking is to -- presumed faster -- get the rectangles of WM_PAINT ...hmmm and get into hell for that :P

IMHO "the naive" approach (BTW used by VNC, PC AnyWhere,etc) is the best solution because doing a GetDC,0 and scanning the full screen bitmap for differences can not be THAT costly on today's fast CPU's. My guess is that one can identify that "rectangles" much faster in the accumulated global variable that is the screen...

PS
----
THis way you will catch all "other" BitBlted applications like WinAmp but not only... not caching them on remote screen will really make your application look "lame" IMHO, likewise running on WinNT/2K only :)
Posted on 2003-04-09 10:51:23 by BogdanOntanu
the more I think of it.... wm_paint isn't anywhere if you scroll down a document, right? hell, the naive implementation really makes sense. But a pixelwise check for changes wouldn't make sense. please post ideas for good algos.

"PC-Duo Remote Control is GDI Intercept based and uses a Mirror Driver on W2K and above."

I don't like that in these cases I always want to achieve something using the best possible way :D
WTH is a mirror driver?
Posted on 2003-04-09 12:08:29 by nyook
I don't think (yes, think - I haven't timed it) a global hook will be that bad. You check to see if the message is WM_PAINT, if not you call the "next hook in chain" as early as possible. Not much extra work going on before a message is dispatched as per usual. It's not like windows messaging is (or has to be) high-speed anyway.

Constantly scanning the screen for changes, on the other hand... either you do it a _lot_, which most likely _will_ eat a lot of CPU time - or you do it less times, which ends up in sluggish feel on the client side.

GDI mirror driver... I can only guess about this, but I would presume it's a form of GDI miniport driver (sorry I'm probably using wrong terms, it's been over a year since I digged around in NTDDK for this stuff) that intercepts GDI calls (at "native" level), does handling of the necessary functions, and then passes on to the "original" GDI(/miniport) driver. I think this might be the best-performing way of handling stuff, but also more complicated.

Bogdan, WM_PAINT hook ought to catch "BitBlt'ed" stuff as well, as long as it's well-behaved applications that redraw their UI on WM_PAINT. So, you should get the main winamp interface, though the oscilloscope might not be updated realtime.
Posted on 2003-04-10 02:03:16 by f0dder
Ok, here's an idea for a possible implementation.

Use a WH_CALLWNDPROCRET hook. This will give us all window messages, but not until after they're done (eg, we're certain that on a WM_PAINT the application will have redrawn when our hook is called - unless of course the app does something weird like creating a new thread for the drawing).

First: determine if it's a WM_PAINT (or other message we might be interested in - might very likely be a few others that are interesting, though kbd+mouse might require other hooks).

If it's a WM_PAINT, take note of the hwnd, and notify the "server process". (Perhaps it should also GetClientRect and send this along too - see notes below).

No, we shouldn't do processing inside the hook - after all, the hook is called for every windows message for every thread, so it's limited how much we can get away with doing here. Furthermore, a couple hundred threads processing and sending packets at the same time? I think not.

Since the hook DLL is injected into every process (and our server application, that installs the hook, is not), we cannot rely on a callback to notify the server process - we must rely on IPC. Memory Mapped Files are out of the question (unless you devise a scheme for mutexing the access to it), so pipes or window messages are probably the best solutions. I think a pipe would be best - assuming that data isn't "mixed" (ie, that a WriteFile to a pipe will not be interrupted in the middle by another WriteFile). Using a pipe or window messaging also acts as a serializer - we will only handle one message at a time.

This way, the hook doesn't do a terrible lot of processing, and shouldn't really slow down the system that much.

The server process reads a stream of HWNDs from the pipe. There's a whole lot of ways to go about stuff now, but the basic idea is to get a HWND, GetClientRect on it, grab bitmap data, and send to the client. We might want to send the ClientRect from the hook instead, to handle (imho pretty unlikely) situations where a window is moved _right_ after it has been redrawn, before the server grabs the bitmap data.

Sending out raw pixels is a bad idea - they should be compressed. Also, sending out the entire rect from the window is propbably superfluous - often only parts of a window will have been redrawn. For this purpose, we should maintain an internal memory bitmap of the entire screen. Instead of constantly re-grabbing it (bogdans idea), we will update our memory image with the bitmaps we grab in response to the WM_PAINT messages from the hook. Still following me? :). By doing it this way, we can (on WM_PAINT messages from the hook) compare the newly-grabbed bitmap data with our cache, and only send the delta across the network (and of course update our cache image afterwards).

also, instead of sending data at once, it might be smart to accumulate a bit, and only send X times per second. This, and whether to use UDP or TCP, are parameters that should be thought about, tested, and tuned.
Posted on 2003-04-10 13:06:16 by f0dder
Here's a little hook test. It sets a WH_CALLWNDPROC hook, and in the hook it paints the window rectangle purple. There's two DLLs included, rename one of them to hookdll.dll. The "100" version does a Sleep(100) so you can easily see the repaint procedure.

The main app will pop up a messagebox - when you hit OK the hook will unload.

Note that the app knows nothing about update regions, it fills the entire window rectangle - this means there will be a lot of puple on screen, as a lot of applications only update the parts that have changed. This shouldn't matter much in the context of remote desktop though, as there should be image-delta code too.

However, and this is alarming, it would seem that some things, like character input in an editbox, does not flag a WM_PAINT message. Hopefully other messages are sent that we can trap; otherwise WM_PAINT hooking could be combined with "poll foregrund window" or similar.

C++ binaries attached.
Posted on 2003-04-10 14:34:16 by f0dder
I programmed a hook also but mine isn't global. Maybe because I used WH_GETMESSAGE, but MSDN said this could also be made global.

You're prog is nice. A little problem is that if you drag a window outside the screen and back, not everything is overwritten purple (at least my explorer window, not sure about other apps). This may mess up the delta-calculation, because we may get a wrong bitmap with the old rect?

My hook implementation doesn't work correctly yet :( damn thing.

Delta calculation:

We could slice the screen in 80x80pixel tiles. then compare each tile to the old tile in the buffer. if a difference is found, mark the actual tile to be sent. continue with the other tiles. Another improvement would be to only scan every 2nd pixel.
Posted on 2003-04-10 15:13:15 by nyook
drag outside screen and back is handled by windows with bitblt, it caches client area to avoid having to issue a wm_paint (I think - at least it feels like that's what's happening).
Oh, I also don't trap non-client paint.

Obviously refinement will have to be done. But it should be possible. I can't stand the idea of continuously comparing the entire screen to a previously grabbed copy to find deltas, some form of hooking has to be faster.

C++ code attached.
Posted on 2003-04-10 15:20:41 by f0dder
Thanks for the code. But my own implementation still doesn't seem to hook globally :(
I will look at it again tomorrow.

AFAIK we don't catch the bitmap of the window's titlebar. just the client draw area. I don't know if this is a big problem to overcome (I'm not that much a windows guru ;))



I can't stand the idea of continuously comparing the entire screen to a previously grabbed copy to find deltas, some form of hooking has to be faster.


sure. if you get rects which are absolute, you better catch resize, move etc messages. then using stored rects, you can find the deltas.
Posted on 2003-04-10 15:56:19 by nyook

Oh, I also don't trap non-client paint.

...and it would probably be handled by additionally trapping WM_NCPAINT.

As for GDI mirror drivers, supposedly used for one of those projects:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/graphics/hh/graphics/dpyddi_33fr.asp
It's been a while (> 1 year, I think) since I had a look at all that stuff, but iirc, a GDI mirror driver miniport might be an interesting project to undertake.
Posted on 2003-04-10 15:59:37 by f0dder


...and it would probably be handled by additionally trapping WM_NCPAINT.


i see. so the question to my answer is obvious, thx :)


edit: your link is very interesting. it is worth a deeper look.
Posted on 2003-04-10 16:02:03 by nyook
I downloaded the DDK now. I'm interested in knowing if moving the cursor also generates writing output for the mirror driver or not (I hope it doesn't).
What I dislike is the installation of the driver, service etc.
So I maybe go better with some Message hooks. They don't catch all drawing operations, but the most important of it. Additionally, some GDI hooking would serve too.
Posted on 2003-04-12 02:44:49 by nyook
A mirror driver would be for a more "professional" product, I guess. I don't think having to install a driver is all that bad, if the product works well. It would feel cleaner than lots of hooks and stuff :)


I'm interested in knowing if moving the cursor also generates writing output for the mirror driver or not (I hope it doesn't).

I don't know, but the info ought to be somewhere in the NTDDK. Also, it wouldn't necessarily be a bad thing, depending of course on the type of write output it generates.
Posted on 2003-04-14 02:08:30 by f0dder
hm I didn't get it to work yet. It still captures only the messages dedicated to my own window. I tried then to put the Clientrect area into the clipboard, but only works 50%. Sad to say, I don't have more time to spend on this matter now.

<- beginner
Posted on 2003-06-21 14:36:32 by nyook