Hi guys, I need to ask yer advice :) if ye don?t mind.

I need to get something working, and as I?ve little experience I thought I?d discuss it first with people know this stuff much better than me.

Basically I need to get a load of computers on a LAN working on some bulk processing, unfortunately this isn?t a true parallel type of job so there?ll be a lot of communication between the Pcs.

In brief each Pc will do some processing and produce an output < 100kb. All these output need to be combined (summed basically) bit of additional processing to changes them and then have the new output (same size as original) sent back to each Pc

The LAN I?ll be working on only has the one server so that kinda simplifies things. Off hand I can think of two possible ways of implementing this. Call the server s, and lets say 6 Pcs a, b, c, d, e & f. The first and most obvious would be that all 6 sent their output straight to s who does some work and sends the new output back.

However a second idea occurred to me. If a sends output to b, c to d and e to f. Then b send combine output to d, f sends combined output to s. Finally d sends combine output to s. The new output could be sent back along to same route.

This idea is probably silly, I imagine it would probably work better over lets say the net, but on a single server LAN is the first simple solution the better.

Your thoughts?
Posted on 2002-12-12 18:32:57 by Eóin
Sounds a lot like a "Render farm"...
The preferable flowpath is determined by the network topography and the timing of the operation. I suggest mapping the estimated time requirements of each portion of the operation in order to better plan the operation timing. Think about this as if you were managing a large job and heading a team of workers... as an efficient manager, you would flowchart the job at various stages of completion, and you would allocate labor to each stage.
In a "Render Farm", one machine has the job of managing the outsourcing of workloads to participating LAN machines, and tracking the stored resources across those machines, since the data generated can be large, it can be left on the machines it is generated on, and recalled later. A good RenderFarm system can detect a filled hard drive at a Slave, and still be able to use the cpu resources of that machine, passing the data directly to another slave for storage.
The RenderFarm system consists of simply server and client applications, I recommend staying with the central server and not going with peer2peer networking for the application.. it gets messy and confusing !!

To implement parallelish behaviour without going peer2peerish, why not implement a hierarchy of servers instead of a server and clients?
All participating machines would be servers, but only one would be Root.
The hierarchy would resemble a tree, with N branches per node.

To implement "true" parallel behaviour without going peer2peer, why not implement a neural network of servers?
All participating machines would be servers and would communitcate with all the others, and would be equal except for one authoritative machine, which is the machine which starts the "session" (first machine speaking our protocol - all further machines will be "joining" the "session"...)
The earliest ethernet chat apps worked this way, they simply broadcast packets on the shared bnc cable, which participating machines would listen to.
Obviously this won't work if your router filters local broadcasts, but there's plenty of ways to skin a cat...

In conclusion, what you wish to implement is a more specialised form of render farm, more specialised because you wish to place certain constraints on the labor allocation, and also the final form and location of the data, and because the data processing is not necessarily anything to do with rendered images.

Sounds like fun :alright:
Posted on 2002-12-12 22:53:30 by Homer
This tutorial contains some Info about programming for clusters...

http://tldp.org/HOWTO/Beowulf-HOWTO.html

You could easily split the work over multiple computers under Linux with OpenMosix.

http://openmosix.sourceforge.net/

I am pretty sure that your programm is a console application, so you can easily port it to Linux (if your wrote it in C or assembly). And something well tested like OpenMosix is for sure better than something self-coded :grin:
Posted on 2002-12-13 08:13:58 by bazik
EvilHomer2k, your insight is much appreciated. I think I?ll go for the central server, everything else acting as nodes. At least that?ll be the simplest to try and if it works, then leave well enough alone.

Bazik, you?re an absolute wealth of information. Unfortunately Linux won?t be an option, Windows is the OS of the PCs on the network, including the Server (specifically I think that?s running Win2k server edition). That openMosix is really, really cool though.

BTW, I?ve spent pretty much the last two weeks trying to setup a Linux From Scratch on my Pc. It took two attempts, and four reinstalls of the host system, but its finally up and running. Now need to follow Beyond Linux from Scratch to get FluxBox working.
Posted on 2002-12-13 12:06:51 by Eóin

Bazik, you?re an absolute wealth of information. Unfortunately Linux won?t be an option, Windows is the OS of the PCs on the network, including the Server (specifically I think that?s running Win2k server edition). That openMosix is really, really cool though.


Ah, I see :)

Here is some information for clustering under Windows (2000):

http://research.microsoft.com/Scaleable/default.htm
http://www.microsoft.com/windows2000/hpc/
http://www.windowsclusters.org/
Posted on 2002-12-13 12:13:40 by bazik

BTW, I?ve spent pretty much the last two weeks trying to setup a Linux From Scratch on my Pc. It took two attempts, and four reinstalls of the host system, but its finally up and running. Now need to follow Beyond Linux from Scratch to get FluxBox working.


LFS is a pain in the ass, although you learn alot from the internal structure of Linux (in fact how the OS/Kernel works at all).
But for work-use I would suggest things like rocklinux or gentoo (easier to upgrade) :)
Posted on 2002-12-13 12:15:03 by bazik
Thanks for those links Bazik, I'll have to wait till later though to look into them, about to go to work now :( .

I'd never actually heard of rocklinux before, seems very like gentoo. The problem was that I wanted to install on a Pc with a slow internet connection. Gentoo wasn't an option there, maybe Rock would be, I'll look into it.

Linux from Scratch, yeah sure it was a pain in the ass first time. But I was (still am) a Linux newbie and I learned alot. Things went belly up true, but I'm sure it was something I messed up.

Second time around I worked my way through LFS, plus manage to get Xfree86 and truetype working in about four hours, which wasn't too bad. Couldn't get FluxBox to compile though, but I'm sure I'll manage. Think I'll need to setup some of the symlinks that Beyond Linux From Scratch suggests.
Posted on 2002-12-13 12:50:34 by Eóin