The whole point of this project, as I may have mentioned before, is to develop the servers, the backend, for the roleplaying framework, not the client. Of course, developing one or more clients is to be expected as I work on the project, because much like you'd expect from a blind painter, it helps to have a visual verification that the result is at all accurate; I could work on the server-side code all I want and HOPE it works, but I'd really want to SEE that it works eventually.
Knowing that, I knew that some day I'd be working on a client -- textbased, 2d, 3d, whatever -- but I never thought it'd be so soon. In fact, I'm actually writing a text-based console client before I get any of the "useful" part of MMORF written: even the server-side code that I'm writing to test the client is just login server stuff right now. Where's the good stuff??
Of course, writing the client means that I've had to address the issue of the protocol between the client and the server, which I've briefly discussed before, but have spent a bit of time thinking about already. Two things are important in a protocol, when all things are said and done: it's efficient, and it's debuggable. The second part can always be done regardless of how the protocol is implemented, but its readability corresponds to how easily it is debugged. Sure, you can write loggers and decoders to translate some binary stream to meaningful words for the developer, but when first developing the system, you're better off making it human-readable to start, and worry about the efficiency later, provided you coded the networking subsystems to handle a complete change to the protocol. This means that all communication is handled through functions that generate the protocol packets via a drop-in system, one which can be replaced without anyone being the wiser. In fact, having the client being able to support every drop-in system you've devised based on its ability to detect what's coming down the pipe means no rewriting later in the development cycle.
When I talk about a human-readable protocol, I typically mean an interchange format such as XML, JSON, YAML or even Metaplace's MetaMarkup. It's something that balances readability by humans and parseability by computers. Myself, I tend to lean towards XML, because it handles hierarchical data very well (both structurally and visually), it copes with escaping characters used by the protocol WITHIN the protocol (something which was a recurring theme with MetaMarkup), and because I've had some experience with it as a network protocol.
The only unfun part of developing a protocol is ... developing a protocol. When you get to the binary level and are trying to squeeze as much data through as possible, THEN it might actually be fun, but at the XML level, it can be tedious coming up with the schema for all of the different packets that are going to fly by.
That's why I was quite pleased to find out that the serialization of C# objects results in: XML! Instead of having to come up with an XML protocol schema, all I have to do is come up with an object schema based on all of the communication needed between client-and-server and server-and-server, then pass these objects through the network and let them pop out as objects that need to be handled. Even better, handlers for these objects can actually be part of the object itself.
It's a pretty slick system. It took a bit of designing to come up with a general server framework, allowing the client to handle various login servers (local (object-based) and remote) and client servers (local and remote), but this is done and mostly implemented. I now have a console client that launches its own local login and client servers (internal objects, not even using a loopback network connection) and can send login packets (or rather, serialized login objects). I had to design none of the protocol, instead letting the .NET Serialization system do it for me, and I had to neither write out nor read in the stream of data, deconstituting or reconstituting the meaning. Instead, I create a Login object, call my LoginServerConnection's Send() function with it, and on the other side the LoginServer sees a Login object pop out, which it may deal with as it pleases.
Just think: I'm almost at the point where I can actually design some of the game server and see it working. Amazing.
Showing posts with label protocols. Show all posts
Showing posts with label protocols. Show all posts
Thursday, July 01, 2010
Friday, April 08, 2005
Never trust the client
A few days ago, in the #ddo IRC channel on SorceryNet, the topic of packet sniffers came up, with respect to the upcoming Dungeons & Dragons Online game.
The issue goes back to the more general rule, "never trust the client", or "The client is in the hands of the enemy". The fact is that no matter how much you try to hide or obfuscate the operation of your client, there are people out there that will figure it out, and they will write a cheat program for your users to use.
The solution, then, is "never trust the client" - that is, anything your client says to do, you verify as "correct" or possible. Instead of taking orders from the client, you take suggestions. And client trust doesn't even have to be in communication, where you are listening to what the client is saying. You can't even trust the client to perform in a commanded way, such as display something it should or hide something it shouldn't. The earliest case I had first heard of (as a player in the community) was in Ultima Online, where the game servers would tell the clients to darken the screen, because it was nighttime or because the player was in a dark dungeon. It was meant to provide atmosphere, but also, I presume, to add more challenge to adventuring in the darkness. I don't know how long it took, but individuals had figured out the packets being passed back and forth between the server and client, had isolated the one that said "make the screen dark", and stripped it out (or modified it to say "make the screen bright").
In the case of the conversation in #ddo, there were the issues of being able to see traps that you hadn't detected yet, or knowing of the presence of monsters that were hiding in the shadows. I, of course, stated flat out that this kind of information shouldn't be in the hands of the client until the player is supposed to know about these. This started a flurry of responses, from "yeah, but every other MMO has had the problem" to "it would be too laggy if you didn't send this stuff beforehand".
Now, I'm one of the first to grit my teeth when I'm reading gaming forums and see people say "they should add feature X. It's really easy, they just need to ..." Invariably, these people are not programmers, and if they are, they're not good ones, and if they are good, then they still don't have the knowledge to say how easy it is to add a feature to someone else's codebase. Almost as bad are people who seem to know what *can't* be done. This is, however, what the discussion became, me included.
One party insisted that game developers are going to keep doing it, even though (I assured them that) experienced developers know about these previous mistakes in this industry, and would know better. Another party insisted that they couldn't get away from sending early information to the client because of lag problems (which is a little better argument than "they'll do it because all games have it").
I disagree with them both. One, at least, is a programmer. And while I find many of the vocal "easy" people on forums try claiming programming skill as well, I usually call bollocks on their abilities, and chalk them up as know-it-alls that know nothing. This chatroom member, though, I'm willing to give credit as a capable programmer, and thus I just disagree with him.
First of all, the "there's always something that a packet sniffer can find" argument is just crap. Yes, it might be that every game so far has had some client vulnerability, but to argue that every game in the future must therefore have the same is ludicrous. The client is just an interface to the information send by and to the server. It should never have extra information that isn't displayed. Enough of that argument.
But the point about lag is valid. The example went something like "if a dozen goblins were sneaking up on you, and suddenly stepped out of the shadows, the sudden surge of data from the server to the client would cause the client to lag and the player to die (or be at a disadvantage)". Fair enough, that might happen. But, this brings up a few points:
Please, prove that one chatroomer wrong.
The issue goes back to the more general rule, "never trust the client", or "The client is in the hands of the enemy". The fact is that no matter how much you try to hide or obfuscate the operation of your client, there are people out there that will figure it out, and they will write a cheat program for your users to use.
The solution, then, is "never trust the client" - that is, anything your client says to do, you verify as "correct" or possible. Instead of taking orders from the client, you take suggestions. And client trust doesn't even have to be in communication, where you are listening to what the client is saying. You can't even trust the client to perform in a commanded way, such as display something it should or hide something it shouldn't. The earliest case I had first heard of (as a player in the community) was in Ultima Online, where the game servers would tell the clients to darken the screen, because it was nighttime or because the player was in a dark dungeon. It was meant to provide atmosphere, but also, I presume, to add more challenge to adventuring in the darkness. I don't know how long it took, but individuals had figured out the packets being passed back and forth between the server and client, had isolated the one that said "make the screen dark", and stripped it out (or modified it to say "make the screen bright").
In the case of the conversation in #ddo, there were the issues of being able to see traps that you hadn't detected yet, or knowing of the presence of monsters that were hiding in the shadows. I, of course, stated flat out that this kind of information shouldn't be in the hands of the client until the player is supposed to know about these. This started a flurry of responses, from "yeah, but every other MMO has had the problem" to "it would be too laggy if you didn't send this stuff beforehand".
Now, I'm one of the first to grit my teeth when I'm reading gaming forums and see people say "they should add feature X. It's really easy, they just need to ..." Invariably, these people are not programmers, and if they are, they're not good ones, and if they are good, then they still don't have the knowledge to say how easy it is to add a feature to someone else's codebase. Almost as bad are people who seem to know what *can't* be done. This is, however, what the discussion became, me included.
One party insisted that game developers are going to keep doing it, even though (I assured them that) experienced developers know about these previous mistakes in this industry, and would know better. Another party insisted that they couldn't get away from sending early information to the client because of lag problems (which is a little better argument than "they'll do it because all games have it").
I disagree with them both. One, at least, is a programmer. And while I find many of the vocal "easy" people on forums try claiming programming skill as well, I usually call bollocks on their abilities, and chalk them up as know-it-alls that know nothing. This chatroom member, though, I'm willing to give credit as a capable programmer, and thus I just disagree with him.
First of all, the "there's always something that a packet sniffer can find" argument is just crap. Yes, it might be that every game so far has had some client vulnerability, but to argue that every game in the future must therefore have the same is ludicrous. The client is just an interface to the information send by and to the server. It should never have extra information that isn't displayed. Enough of that argument.
But the point about lag is valid. The example went something like "if a dozen goblins were sneaking up on you, and suddenly stepped out of the shadows, the sudden surge of data from the server to the client would cause the client to lag and the player to die (or be at a disadvantage)". Fair enough, that might happen. But, this brings up a few points:
- Is the protocol so "bulky" that the information about these dozen goblins (or whatever information suddenly because available) will cause such a discernable lag? If so, can the protocol be optimized? There's a reason why MMOs aren't "twitch" games -- because of the latency of the Internet (discussed previously) and disparate speeds of players' computers - is the game too twitchy if this sudden information is a problem?
- Does all the information have to be sent immediately? Can the server not say "draw some shadowy figures - I'll let you know what they are in a sec"? Are full texture descriptions being sent up, as one of my debaters suggested, instead of them being pre-existing on the client, or send a little later?
- Can the game be designed so that any information sent early (and thus hacked) is of insignificant value? In the case of the darkness if UO, I think it might have been changed so "dark" wasn't really so bad, and it was moved from a game-affecting feature (and thus an advantage to the cheaters) to solely a mood-lighting, visual effect.
Please, prove that one chatroomer wrong.
Friday, February 25, 2005
Lag
I've been giving a lot of thought to the problem of lag in MMOs lately. Part of the reason, of course, is that I still play one, and thus experience it, but also because it seems to be quite a large problem that plagues developers. I haven't read anything about the issue, so I'm probably rehashing the old and restating the obvious...
There are, as I see it, a few different issues that all get lumped under the same title of "lag". "Lag", by definition, is to "fail to keep pace". Lag is "experienced" when the action fails to keep pace with a player's commands; when a player fails to keep pace with another player; and when game time fails to keep pace with realtime.
The server
Lag on the server is an example of game time failing to keep up with real time. Players get used to being able to move this distance in this amount of real time, and because the server is so "lagged", they aren't. This usually happens when there is so much action happening on a specific server (for worlds that run multiple servers per world). In-game events, such as a city invasion by hundreds of MOBs, leads to a lot of AI churning away at the server, and an unusually high concentration of players in one spot, fending them off. Also, unsanctioned events can lead to more of a load on a server than developers might expect. The fault: aging hardware, unrealistic computing requirements, underestimation of popularity... mostly things in the developer's realm.
The client
As MMOs get more and more complex, and more and more 3D, the video card is getting taxed to the limit. While this problem will never rear its head in a text-based MUD, graphics are becoming more and more involved, and the creation of them is being done on the client more and more. From environmental textures and effects, to world objects that rotate, flicker and swing, to the numerous MOBs and players that run by with their armor glinting in ten sources of light, the days of static graphics are going. Well, maybe not going, but are falling by the wayside. Here the fault can be prescribed 50/50: users might need to keep up with their state-of-the-art hardware, or developers might be expecting too much of their userbase.
And inbetween...
The biggest culprit of them all -- the Internet. How many miles of cable and fibre do the bits of information between client and server go through? How many pieces of hardware (routers) have to shunt it this way and that? What quality is the medium, and how quick the messenger? This is the biggest bugbear, because it's out of the hands of everyone involved. Neither players nor developers can choose which path their packets should take. Sure, the player can move to a better ISP, or the company can sit on a thicker backbone, but no backbone will be perfect for everyone, and some places still have more hops than desired to get from A to B.
Until the internet is completely pervasive across the globe, and routing technology because near-instantaneous and universally-installed thusly, this will be a problem for the developers to worry about. You need to
By "out-of-sync", I mean our second definition of lag above, where one layer fails to keep pace with another (and in this case, a player could include a MOB).
Make it impossible for two people to become out-of-sync
Force instructions from the player and events from the server to happen based on a "tick" from some clock which keeps track of time. The length of a tick would be the maximum latency of all the players connected to the game. Everyone is forced to play at the speed of the slowest player, in what basically becomes a turn-based game with a timer per turn. Everyone has X seconds to figure out what they're doing this "turn", with the slowest player having the least amount of time to act (because he or she spent most of their turn sending their last action and waiting for results). Doesn't make for much of a game...
Make it so players don't notice if they become out-of-sync
Support on-going commands ("walk north") that will take place on the server, even if the player isn't able to continually send the command. If the server receives a "stop" command a little too late (because it walked them a bit further), then "snap" them back the where they think they should be. This can lead to other players seeing the lagging player hop back a few steps or continue going in the same direction too much (which seems to be what you see in World of Warcraft); or, the player's client, which has been smoothly drawing the action that it's hoping and assuming is happening will suddenly "rubberband" the player back to reality - that is, the spot that he or she is really standing in (now that the client has gotten new information). This is what you see if Ultima Online.
Make it so it doesn't matter if they become out-of-sync
Design the game so it's less of a "twitch" game, where fast reflexes are required to play. Ultima Online PvP used to be all about who had the fastest connection (once the combatants all knew the "best" methods). With one expansion, it changed so that your items were more important than your strategy, thus allowing the server to decide the winner, even if one of the participants couldn't keep up as well as the other. It didn't remove the need for skill in combat, but it helped reduce the effects of differing connectivity between players.
PvM is a bit different. MOBs are designed with their AI as-is, so they don't have different tactics. To make monster X challenging to a 13 level wizard, it has these skills, these stats, these items. But challenging to what level of lag? You want the guy playing down the street with the 5ms latency to have to back off once or twice to heal himself, but does that make it too hard for the woman playing across the continent with the local ISP with bad copper lines underground? Do you make that same monster easier, so the lagged player can still back off in time, thus making it very easy for the low-lag player -- too easy, for the reward?
Or can combat be made even less "twitch" dependant, so a lagged player can decide early on that they're outmatched and thus escape, while there's no benefit to a character of the same strength with a good connection? Do we boil it down to
You have chosen to attack monster X. Do you wish to proceed?
> YES
You have died! / You have vanquished your foe!
where the same result is going to happen to any player, whether their internet connection tells them immediately or half a second later?
Combat, whether PvP, PvM or PvE (versus the environment, such as jumping onto the swaying bridge), is where most of the problems occur. Role-playing is still effective even if your dialog is half a second behind the others -- not everyone's a professional typist! Shopping in-game, wandering the lands, and interacting with NPCs are all things that can happen with leisure. Combat is where it's going to be most noticeable, because that's where the consequence of lag is going to be the most dire: character death, usually.
So the fault for this kind of lag? No fault. But, the developers are the ones that might be able to do something about it.
But that's only half of the problem with the lag between client and server. In addition to the delay between client and server for moving the information, there's also the volume of information.
In Ultima Online, player houses can have items locked down and viewable by passers-by. This means that as a player walks through an area, information about item type and location from many objects are being sent to that player's client. The more items on-screen, the more data being sent. The faster they're moving (or, in this case, trying to move), the more items are coming on screen, and thus even more data. Some houses would be so "decorated" that a noticeable delay would occur as information about hundreds of items was suddenly shoved towards the client. There was even a case years ago where you could make a "black hole" in the game, where if you got too close to a stack of items (a very, very numerous stack), your client would crash. Whether that was from an inability to cope with the number of items, or just too much network data, I don't know. Probably the former, but can you imagine the latter?
In an old text MUD, InFiNiTy CoMpLeX, you could, with the right weapon, take out 15-16 enemies in one shot (it was a rocket launcher). This, I suspect, caused nearly all the different kinds of lag: the server had to handle the firing of the weapon, the damage dealt to numerous MOBs, and their deaths (which included giving experience and broadcasting the event to everyone in the complex) - this was all on a single machine that was running the rest of the BBS on which the game was hosted; the amount of text that had to be send to every client added up to hundreds of bytes, which was significant at 2400 baud; and the sheer number of characters every player received upon this event took some time to display on their slow, 8086 DOS machine (if they were lucky).
Fault: Developers, I suppose: it's up to them to cut down on the size and number of packets being sent back and forth.
Of course, I supply no hard solutions to these problems yet -- I'm still learning. But it's a start, at least, that I know these problems are out there, and that I'm going to have to work around them, if not try to quash them completely.
There are, as I see it, a few different issues that all get lumped under the same title of "lag". "Lag", by definition, is to "fail to keep pace". Lag is "experienced" when the action fails to keep pace with a player's commands; when a player fails to keep pace with another player; and when game time fails to keep pace with realtime.
The server
Lag on the server is an example of game time failing to keep up with real time. Players get used to being able to move this distance in this amount of real time, and because the server is so "lagged", they aren't. This usually happens when there is so much action happening on a specific server (for worlds that run multiple servers per world). In-game events, such as a city invasion by hundreds of MOBs, leads to a lot of AI churning away at the server, and an unusually high concentration of players in one spot, fending them off. Also, unsanctioned events can lead to more of a load on a server than developers might expect. The fault: aging hardware, unrealistic computing requirements, underestimation of popularity... mostly things in the developer's realm.
The client
As MMOs get more and more complex, and more and more 3D, the video card is getting taxed to the limit. While this problem will never rear its head in a text-based MUD, graphics are becoming more and more involved, and the creation of them is being done on the client more and more. From environmental textures and effects, to world objects that rotate, flicker and swing, to the numerous MOBs and players that run by with their armor glinting in ten sources of light, the days of static graphics are going. Well, maybe not going, but are falling by the wayside. Here the fault can be prescribed 50/50: users might need to keep up with their state-of-the-art hardware, or developers might be expecting too much of their userbase.
And inbetween...
The biggest culprit of them all -- the Internet. How many miles of cable and fibre do the bits of information between client and server go through? How many pieces of hardware (routers) have to shunt it this way and that? What quality is the medium, and how quick the messenger? This is the biggest bugbear, because it's out of the hands of everyone involved. Neither players nor developers can choose which path their packets should take. Sure, the player can move to a better ISP, or the company can sit on a thicker backbone, but no backbone will be perfect for everyone, and some places still have more hops than desired to get from A to B.
Until the internet is completely pervasive across the globe, and routing technology because near-instantaneous and universally-installed thusly, this will be a problem for the developers to worry about. You need to
- Make it impossible for two people to become out-of-sync
- Make it so players don't notice if they become out-of-sync
- Make it so it doesn't matter if they become out-of-sync
By "out-of-sync", I mean our second definition of lag above, where one layer fails to keep pace with another (and in this case, a player could include a MOB).
Make it impossible for two people to become out-of-sync
Force instructions from the player and events from the server to happen based on a "tick" from some clock which keeps track of time. The length of a tick would be the maximum latency of all the players connected to the game. Everyone is forced to play at the speed of the slowest player, in what basically becomes a turn-based game with a timer per turn. Everyone has X seconds to figure out what they're doing this "turn", with the slowest player having the least amount of time to act (because he or she spent most of their turn sending their last action and waiting for results). Doesn't make for much of a game...
Make it so players don't notice if they become out-of-sync
Support on-going commands ("walk north") that will take place on the server, even if the player isn't able to continually send the command. If the server receives a "stop" command a little too late (because it walked them a bit further), then "snap" them back the where they think they should be. This can lead to other players seeing the lagging player hop back a few steps or continue going in the same direction too much (which seems to be what you see in World of Warcraft); or, the player's client, which has been smoothly drawing the action that it's hoping and assuming is happening will suddenly "rubberband" the player back to reality - that is, the spot that he or she is really standing in (now that the client has gotten new information). This is what you see if Ultima Online.
Make it so it doesn't matter if they become out-of-sync
Design the game so it's less of a "twitch" game, where fast reflexes are required to play. Ultima Online PvP used to be all about who had the fastest connection (once the combatants all knew the "best" methods). With one expansion, it changed so that your items were more important than your strategy, thus allowing the server to decide the winner, even if one of the participants couldn't keep up as well as the other. It didn't remove the need for skill in combat, but it helped reduce the effects of differing connectivity between players.
PvM is a bit different. MOBs are designed with their AI as-is, so they don't have different tactics. To make monster X challenging to a 13 level wizard, it has these skills, these stats, these items. But challenging to what level of lag? You want the guy playing down the street with the 5ms latency to have to back off once or twice to heal himself, but does that make it too hard for the woman playing across the continent with the local ISP with bad copper lines underground? Do you make that same monster easier, so the lagged player can still back off in time, thus making it very easy for the low-lag player -- too easy, for the reward?
Or can combat be made even less "twitch" dependant, so a lagged player can decide early on that they're outmatched and thus escape, while there's no benefit to a character of the same strength with a good connection? Do we boil it down to
You have chosen to attack monster X. Do you wish to proceed?
> YES
You have died! / You have vanquished your foe!
where the same result is going to happen to any player, whether their internet connection tells them immediately or half a second later?
Combat, whether PvP, PvM or PvE (versus the environment, such as jumping onto the swaying bridge), is where most of the problems occur. Role-playing is still effective even if your dialog is half a second behind the others -- not everyone's a professional typist! Shopping in-game, wandering the lands, and interacting with NPCs are all things that can happen with leisure. Combat is where it's going to be most noticeable, because that's where the consequence of lag is going to be the most dire: character death, usually.
So the fault for this kind of lag? No fault. But, the developers are the ones that might be able to do something about it.
But that's only half of the problem with the lag between client and server. In addition to the delay between client and server for moving the information, there's also the volume of information.
In Ultima Online, player houses can have items locked down and viewable by passers-by. This means that as a player walks through an area, information about item type and location from many objects are being sent to that player's client. The more items on-screen, the more data being sent. The faster they're moving (or, in this case, trying to move), the more items are coming on screen, and thus even more data. Some houses would be so "decorated" that a noticeable delay would occur as information about hundreds of items was suddenly shoved towards the client. There was even a case years ago where you could make a "black hole" in the game, where if you got too close to a stack of items (a very, very numerous stack), your client would crash. Whether that was from an inability to cope with the number of items, or just too much network data, I don't know. Probably the former, but can you imagine the latter?
In an old text MUD, InFiNiTy CoMpLeX, you could, with the right weapon, take out 15-16 enemies in one shot (it was a rocket launcher). This, I suspect, caused nearly all the different kinds of lag: the server had to handle the firing of the weapon, the damage dealt to numerous MOBs, and their deaths (which included giving experience and broadcasting the event to everyone in the complex) - this was all on a single machine that was running the rest of the BBS on which the game was hosted; the amount of text that had to be send to every client added up to hundreds of bytes, which was significant at 2400 baud; and the sheer number of characters every player received upon this event took some time to display on their slow, 8086 DOS machine (if they were lucky).
Fault: Developers, I suppose: it's up to them to cut down on the size and number of packets being sent back and forth.
Of course, I supply no hard solutions to these problems yet -- I'm still learning. But it's a start, at least, that I know these problems are out there, and that I'm going to have to work around them, if not try to quash them completely.
Subscribe to:
Posts (Atom)
Posts
