Do you know any good way to solve the latency problem when making an internet game? I'm trying to syncronize the movement of characters on all computers connected to a server but there must be some future prediction to make it correct. Is that the correct way to deal with it or is there any other solution?
Posted on 2001-08-23 02:04:43 by gliptic
Well of course it is ..


But it depends on game type...

General rules
========================
1. Send as little data as possible.
2. Use UDP(inet) or IPX (local). Never use TCP/IP!
(What I mean is use nonguaranteed protocols)
3.Make your own SYNC/ACK protocol on top of
the one used (*see rule 2)
4.Allow full loose of packets and still recover...



Additional Rules for RTS Games like we do in Hostile Encounter
========================================
1.Keep world on each computer and send only changes
2.Forget Client/Server use Peer to Peer ONLY
3.Never predict anything

Allways remember that curent inet latency is between 100ms up to 10seconds! or infinity
Posted on 2001-08-23 18:33:15 by BogdanOntanu
I'm doing a 2D action game which needs fast response. I can solve all problems by making the server process all keypresses and send the position, etc. of the characters to the clients, but that means that the characters doesn't respond until the server has recieved the keypresses and sent the actions back to the client. The game works even if some packets are lost so I could use UDP.

I have always wondered how games like Quake does it. The character can move without delays and everything is just perfect. Anyone who knows how inet engines for games like Quake works?
Posted on 2001-08-24 01:38:28 by gliptic
gliptic, client-side prediction. Which means that if you're lagged,
everything seems to be running fine, and suddenly you just die :).

peer-to-peer can be good, but as far as I can see, it gets really
messy when you have a lot of connections, since every computer
has to notify every other computer what it's doing. If you know of
an alternative way to handle this, let me know.

If you're implementing a fullblown syn/ack and error-correcting protocol
with UDP... then don't :). You're just reimplementing TCP, and might
as well use that (but perhaps disable the various buffering algorithms
so you get direct throughput).
Posted on 2001-08-24 08:57:20 by f0dder
Bogdan, why not use TCP/IP? I was going to use UDP in my game, I was just wondering why not? You obviously know something I don't :)
Posted on 2001-08-24 14:16:04 by Kenny
well, the TCP protocol keeps on sending data until the data gets
through. There are buffering schemes so if you send 10x1byte, it
will buffer until it can send a large chunk of data, wihch can obviously
be annoying in realtime situations (the various buffering methods
can be turned off, though).

Problem with UDP is... it lacks the features TCP has :). So if you want
semi-reliable UDP, you can easily end up rewriting TCP.
Posted on 2001-08-24 15:13:21 by f0dder
Oh I get it now... I knew UDP was a subset of TCP, I didn't think of it backwards :) I also never read up on the documentation :P
Posted on 2001-08-24 15:47:37 by Kenny
I guess you can view UDP as a subset of TCP. But I don't think TCP
implementations use UDP for their work. I could of course be wrong :)
Posted on 2001-08-24 17:44:04 by f0dder
Well

1. Why not use TCP/IP
=====================

TCP/IP (like C) is a prehistoric protocol that was designed to be reliable and offer almost zero control and be extra slow.

Games need speed and control.

TCP can wait even 60 second until it will give up (if it will ever) sending some data... and yes it will make decisions for you and not even let you know about it...this can be fatal in games

You MUST use UDP or IPX or raw sockets or ANY other NONGuaranteed protocol and makea a bare bone ACK syncronization on top of it...if you think a little you will see you have to do that esp in order to defeat LAG and dropouts and to gain most of the speed one needs

Remember that you can not advance to the next frame until you have not received SYNC (at least) from ALL other players.

This is unacceptable because lag is at least 100ms and with 10 players you will get 1FPS at maximum

The ONLY solution is to make SYNC every Nth frame and let each PC do its internal world actualizations until the next SYNC brings in some more data....or just a SYNC

Things are way more complicated, what is talk here is only and Overview,....

Most /ALL Real Time Games do so....but dont talk about it

2.NO Client/Server also
======================
Think a little can you afford a server? how many connections at full speed can a server serve until you need a second one and a third one and a forth one etc eh?

Remember Blizzard did not aford it!
Westwood did not also....
their servers are for CHAT only, a little trick to foul user, right after the game has started the server is OUT and each computer talks to each other via UDP !

They ALL use YOUR PC when you play on INTERNET and you dont even know it ;) then they come and talk about Client/Server Arhitecture...LOL

Central Servers are for MMO and IRC ONLY and for ppl that have 100.000 USD to spare...

Make use of the computers at your user's desk and his internet connection.... be wise dont make PC a TERMINAL ONLY..... PCs have a processor also, RAM, HDD etc

Why put a server calculate WORLD for 50-1000 Players? no single server can do that and send results back in realtime

Instead calculate wolds on each PC and send ONLY......you think what ;)

Yes IF your game will have more then 25 players in a single session you can have some problems...you will need OPTIMIZATIONS, but then again if you send only 10bytes of data ;) what can go wrong or slow?

In a quake like game things are easy because you can make "predictions" like this:

if a packet has not reached you for a long period of time, keep sending your packets (from that yet unreceived packed until present time) and keep your character go in the direction he was going when the last SYNC was received, also keep all other remote characters do the same,

Hopefully you will receive the lost packets (if your designed your protocol good) and will be able to adjust (via sudden jump or nice LIE/ interploated cruve) your opponents pozitions. You do NOT have to adjust your position...

This Quake style LIE WILL NOT WORK on a RTS, RTS have DRACONIC Multiplayer NEEDS. You will have to freeze movement on screen if you have not received SYNC from all other player s too much time ...but you can breathe a little and let each PC calculate movements and AI in BETWWEN 2 SYNC PULSES

We have done Network Multiplayer in HE...so i know what i am talking about, also there are some Gamasutra articles telling the same widely unknown story
Posted on 2001-08-24 23:18:11 by BogdanOntanu
Bogdan, from your talk I get the impression that you think a client/server
model requires the "server" to be one big machine hosting all games...
I dunno if I'm misinterpreting you, but this is not true. Client/server
simply means that one computer is appointed the "server" of a game,
and the rest of the computers a "clients" (the "server" will usually
also run a "client" itself). This makes stuff easier and takes less network
bandwith - a client only has to send info on what it wants to do to
*one* computer - the server. Unlike a peer-to-peer, where you will
have to send info on what you're doing to ALL the other computers...
which gets out of hand very very quickly.

Client/server also makes it harder to cheat, as you have a server
validating every more, and keeping things in sync.

You are right about TCP though, it wasn't meant for any form of
real-time action. It's wrong to call it (and C) prehistoric and slow, though.
Both serve their purposes *very* well.
Posted on 2001-08-24 23:35:48 by f0dder
Yes, i have something against C an TCP/IP but this is better to be discussed in the crusades section.

I try to keep an Open Mind and belive me if suddenly they will all be in favour of peer to peer and Pascal ;) i will start to preach Client/Server and ASM...ooops C..

But you are right they both serve their purpose well ...

About The Latency:

Problems with Client/Server:
========================

1. we have DOUBLE latency
---------------------------------------
because each client has to send a message to the server and ONLY then the server can send it it back to the other clients....this makes 2xLAG and this was all about (latency) in this post....

2. we have low safety
------------------------------
IF the server looses connection (and this will sure happen or has a congestion) ALL game is LOST, clients can not continue unless they all have a copy of the world and are able to calculate it and also can choose another server in no time = this makes it allmost perr to peer ;)

3. Server has to send BIG Data
----------------------------------------------
Usually the server updates the world and then sends the x,y coordinates of objects and other variables back to clients...but as number of objects in game increases more and more data has to be sent
Only one server to send and receive so much data from multiple clients becomes unacceptable in medium complex games.

Think to a RTS that has 200 units per player x 8 players ->
server has to send x,y coordinates for 1600 units

4.Bad usage of Internet and web topo
-------------------------------------------------
As Interned was designed for each computer to talk to each other on diffrent routes this client /server makes use of a CENTRAL (mainframe style) computer,

hack this single computer and you kill or modify the game.

Even if Client 1 has a better connection with Client 2 (so it happens if they are both in Australia) the can NOT use this they must send data to a server in USA and wait it to return ;)


Advantages of Client/Client as we do it in HE
----------------------------------------------------------

1.We send allmost zero data ;)
----------------------------------------
Please guess how ...,
and if everything goes OK we can have 1xLAG, because all clients have the same data and status variables they can send out very little data, this makes resending in case of a drop a piece of cake

2.Safety
----------------------
IF ANY Computer gets out .... the game can continue until there are at least 2 clients in the game, well even after...but then it will be a single player game

3.Hacking is hard up to impossible
----------------------------------------------
because each computer has its OWN WORLD and make a CRC32 (well or custom algorithm) of it every frame and then compares it with the CRC of ALL other Clients...
the Hacker gets INSTANT disconection ...no questions asked

even if one wants to hack it is MUCH harder to do it the SAME WAY in the SAME TIME on ALL Clients as one has no access to all client computers

There is a problem only with seeing the map (because exists on all clients) but client/server is hopeless here also

4. We dont pay for a server
------------------------------------
We dont need a server....

Of course one computer acts like a server in starting the game but after that there is no server anymore

I belive Client / Server is more like who gets to calculate the world? only one PC or all PCs? and also the path the data walks

However doing peer to peer as we do in HE is much HARDER then a simple Client/Server solution and this can also be a problem ;)

Conclusion
===================
Because of "double lag" and "loose of whole game" if server looses connection ....

one has to go peer to peer in games

until players in game gets greater then 20 or somthing....maybe even more

Client/Server has other usages....

Also no need to reimplement TCP/IP just a little custom protocol (think well) on top of UDP or IPX or whatever...
Posted on 2001-08-27 13:01:24 by BogdanOntanu
I know a little about Half-Lifes netcode (Counter-Strike) and here are some things to note. In the HL console you can type lots of commands that affects the netcode and some of them are:

cl_lw - turns on/off client side weapon prediction
cl_lc - turns on/off client side player prediction
cl_cmdbackup - sets the cmdbackup buffer size
cl_cmdrate - times/sec the client should update the server
cl_updaterate - times/sec the server should update your client
cl_rate - i belive this is max bandwidth allowed
ex_interp - player interpolation rate

Here is how it all works:

A player fires his weapon with weapon prediction on. This will make it look like the weapon fires the very same instant he press the trigger, the client will also draw blood on the wall behind the enemy if the client thinks you have scored a hit and will draw bullet holes if it thinks you missed him. Now the fire commands have also been sent to the server and have been affected by some lag, let's say 80ms. The server now performs its magic. It takes all players and calculates based on their speed and movement vectors where they were located 80ms ago then it performs the bullet hit checks. If you scored a kill the server will then relays it to the client.

All this means that in one game you can have 5 players with a ping of 1000ms and it won't affect your ping of 40ms or your fps or anything else. It also means that what you see on your screen is not the real thing. on your screen you might have scored a hit and blood was shown but the server might say you missed, so what you get is blood when you missed a shot. Ofcourse this just happens in extreme cases where there are lots of pingspikes and chokes. Also your bulletholes that the client shows you might not be the actual location where the bullet struck, it depends on your ping; if your ping is 40ms the server will calculate where you were positioned 40ms ago before it traces the bullet. And even if you stand still there is a small deviation where your holes and the holes the other players see are.

Did anyone understand any of this? ;)
I could explain more but now i have some stuff to attend to...
Posted on 2001-08-30 06:09:23 by Zynaps

Yes, i have something against C an TCP/IP but this is better to be discussed in the crusades section.

I try to keep an Open Mind and belive me if suddenly they will all be in favour of peer to peer and Pascal ;) i will start to preach Client/Server and ASM...ooops C..

But you are right they both serve their purpose well ...

About The Latency:

Problems with Client/Server:
========================

1. we have DOUBLE latency
---------------------------------------
because each client has to send a message to the server and ONLY then the server can send it it back to the other clients....this makes 2xLAG and this was all about (latency) in this post....


This depends on the type of game, peer 2 peer networks would be totally useless and unimplementable for a first-person shooter or any other fast paced game (RTS games are not fast paced, stressful maybe but not fast paced) while it would work fine with a RTS game. And the server does not have to wait for all clients to update the server before it can update all clients. This may be true to some extent for RTS games but for FPS/Racing/Simulator it would never work. Instead each client tells the server how often it should send updates, regardless of the status of the other clients. If a client then experiences some net congestion/choking the player prediction will take care of it.


2. we have low safety
------------------------------
IF the server looses connection (and this will sure happen or has a congestion) ALL game is LOST, clients can not continue unless they all have a copy of the world and are able to calculate it and also can choose another server in no time = this makes it allmost perr to peer ;)


True, but it does not happen often because no-one runs a multiplayer (more than 3) server on a 56k modem ;)
But transfering the whole world from server to client if the client has been disconnected would not take very long either. 4kb/s for 10 seconds = 40kb of data, surley every unit, crater, exploded building/veichle could be transfered in this time 10-30 seconds?


3. Server has to send BIG Data
----------------------------------------------
Usually the server updates the world and then sends the x,y coordinates of objects and other variables back to clients...but as number of objects in game increases more and more data has to be sent
Only one server to send and receive so much data from multiple clients becomes unacceptable in medium complex games.

Think to a RTS that has 200 units per player x 8 players ->
server has to send x,y coordinates for 1600 units


Again this applies to an RTS games only :)
And it would only be true if all players moved their 200 units at the exact same time... I mean no-one would program a server which sends info about units that are sitting idle?


4.Bad usage of Internet and web topo
-------------------------------------------------
As Interned was designed for each computer to talk to each other on diffrent routes this client /server makes use of a CENTRAL (mainframe style) computer,

hack this single computer and you kill or modify the game.


I've never ever heard about someone hacking a server while a game has been in progress ;) The advantage of server/client type of games are that each client does not know the IP adresses of the other clients. In an 8 player peer to peer network each client either knows the IP of all other clients or atleast 2 of the other clients. This makes it very easy to target individual players for a DoS attack. Sure you could DoS the server in a Server/Client game too but trust me, this is rare happenings :)
Posted on 2001-08-30 08:59:25 by Zynaps
Lets see


True, but it does not happen often because no-one runs a multiplayer (more than 3) server on a 56k modem


well here we talk about the bEST way to avoid latency,

IF i CAN do a 8 players RTS (200 units per player) on a 56K Modem then it just means my method is better and IF the lag is smaller and conections are better (like T1) the game will run even faster and smoother...any game any time is better to be able to do it rather them to require trick to make the user "belive" you do it ;)

And yeah Starcraft can DO it (8players via modem peer to peer)
and HE will be able to do it also maybe even 12 players because the NET is faster today ;)
Posted on 2001-08-30 17:59:30 by BogdanOntanu
bogdan, just how do you do the peer-to-peer? Does every machine
connect to every other machine to tell them what it does? Or is this
sort of "distributed", so that each machine only tells one or two other,
which then tell the rest?
Posted on 2001-08-30 18:24:15 by f0dder

This depends on the type of game, peer 2 peer networks would be totally useless and unimplementable for a first-person shooter or any other fast paced game (RTS games are not fast paced, stressful maybe but not fast paced) while it would work fine with a RTS game.


RTS multiplayer games are the harder to do games ever.

Get Starcraft v108, also get a saved replay, then make a multiplayer game with other 7 friends, start the replay and get the speed at maximum...i belive there are around 200fps there ;)
either single player either multiplayer...i have never seen a 8 multiplayer FPS game run at 200+fps,

i will tell you a trick :
a 3 hours replay has allmost the same size as a 2 minutes one (approx 120k map included)....guess what the game is actually replayed not a movie play ;)

There are many many multiplayer Doom/Quake/HalfLife clones out there (i mean the FPS/3rdPS engine) and they are all good...hence very easy to do today.

There are only a few real good multiplayer RTS out there....

RTS require VERY FAST network algorithms because one can not predict / lie about nothing. If methods from RTS will be aplied to FPS then a new era of multiplayer FPS will arrive...well i think some (Tribes) did it ... if not i will do it after HE.

And the server does not have to wait for all clients to update the server before it can update all clients. This may be true to some extent for RTS games but for FPS/Racing/Simulator it would never work. Instead each client tells the server how often it should send updates, regardless of the status of the other clients. If a client then experiences some net congestion/choking the player prediction will take care of it


Oh but its true for all games

IF you want to play the SAME GAME, otherwise we will play difrent games to some extent .
FPS get by with it because they are played in internet Cafes with very good connections or local LAN...but its time they grow up and stop this fakeing because they loose a lot of speed or gameplay doing so.

Prediction/fakeing has to be used because this Client Server architecture is easy to implement in programming, and its also very very slow for a game... path for a network message is 2 times longer average....this make the game 2 times slower average ... dont you like to double your multiplayer fps speed ?

RTS simply can not fake so they had to find a faster method to send syncro and data over the network...and they did!

I've never ever heard about someone hacking a server while a game has been in progress The advantage of server/client type of games are that each client does not know the IP adresses of the other clients. In an 8 player peer to peer network each client either knows the IP of all other clients or atleast 2 of the other clients. This makes it very easy to target individual players for a DoS attack. Sure you could DoS the server in a Server/Client game too but trust me, this is rare happenings


Well you have bever head of a SERVER beeing Hacked? .... oops
And what will it be much easy to DoS a single machine or 8 separate machines? besides if you kill the server you kill all other 8 machines . One hit = 8 kills nice record.

I have seen it done so many times, and seen FPS games hacked so many times exactly because of prediction...like: you can never kill me because i am in another place then where you fire.... but i allways shoot exactly between your eyes ;) esp if you have a faster connection and i fake a slow one ;)



But transfering the whole world from server to client if the client has been disconnected would not take very long either. 4kb/s for 10 seconds = 40kb of data, surley every unit, crater, exploded building/veichle could be transfered in this time 10-30 seconds?


Wrong again, What world to transfer is the server is down? besides worlds go more and more complex every day ...

But you are right at some point here (this is the only advantage) the client can reconect if the server is still up .... useless in RTS as his base will be in flames by the time he reconects ;)

This gives me an idea for HE ... maybe i can protect its base from destruction until he reconects...hmmm

Thx man
Posted on 2001-08-30 18:27:55 by BogdanOntanu
I would guess it is client/server that is dynamically decided. I don't know though.

P.S. this is completely off topic, but why is the HE PE so large?
Posted on 2001-08-30 18:28:52 by Kenny
Kenny

you have a way to get me unprotected ;)

Yeah HE PE is so large because we left a LOTs of ZEROs inside, in the preinitialized .data section...while we do testin we feel much safer like that

We had some problems with .data? section so we decided to let it like that until we finish testing

However you can compress HE PE to about 150K either with ZIP or with any PE compressor available ;)
Posted on 2001-08-30 18:48:23 by BogdanOntanu
Un protected huh :)

lol It just so happens I am a RTS buff, and I also have done tons of research programming to learn how to program them (I always like to learn before getting into the code and getting frustrated).

I've also been following your project for quite a while now. I remember the release you did even before the demo version. It was like a green thing with some bitmaps on it that moved, and that was it :)
Posted on 2001-08-30 18:55:00 by Kenny
Fodder

HE networking is Peer to Peer

Each machine sends its data to all other machines in a game, we can send very little data (but not optimized this yet) even if many units are in game, actually its independent of the units number in game...

its a special technology... after optimizations (no compression involved) we will be able to send about 32 bytes or less every frame ;) even if we have 200x12 units in game


i belive .... but i am not sure that Starcraft does the same ;)

However i am sure that Starcraft is also peer to peer...

Even if this HE demo uses only 2 players (no intro screen to choose more is available) internally HE allways thinks there are 8 players in a game (yes even network game) just that some of them do not do nothing for now (but they are calculated)
Posted on 2001-08-30 18:58:59 by BogdanOntanu