Difference between revisions of "Category:AI War 2: All About Multiplayer"

Initial Caveats And Related Resources

• Please note that multiplayer is currently in alpha status. If you have not read about that, or the current status of where it is is, please see this page.

• If you are testing during alpha, we have a variety of questions for you that we would definitely appreciate you taking a look at.

• If you want to know what the known outstanding issues are, a pretty good summary of that is here. In some cases, particularly for features we might add, that can be rather vague, so more data is on our our mantis bugtracker.

Reporting Bugs Or Suggestions

• Any bugs or requests should go to our mantis bugtracker
  • If you need to submit log files, those can generally be found under your PlayerData folder in the folder your game is installed in. The most relevant one is called ArcenDebugLog.txt. You can send us the whole thing, or just strip out relevant parts.
  • In rare cases, mainly if your entire game crashes (that almost never happens), we will need your unity player log. That gets overwritten the next time you run the game after a crash, unlike the other log. These can be found here:
    • Windows: C:\Users\username\AppData\LocalLow\Arcen Games, LLC\AIWar2\Player.log
    • macOS: ~/Library/Logs/Arcen Games, LLC/AIWar2/Player.log
    • Linux: ~/.config/unity3d/Arcen Games, LLC/AIWar2/Player.log

• If you have been generating logging information that is networking-specific, then that will also likely be in the PlayerData folder, in some cases in subfolders.
  • Generally with those sorts of logs, we need the equivalent log from the host and at least one client (whichever client had the problem).
  • But, also, we really don't expect you to be generating these sorts of logs for us typically, so these really are going above and beyond if you are sending those in.

• In other cases, if you suspect something is wrong, then opening the escape menu during the game and taking a screenshot of the stats there to send to us is welcome.

Reporting Lag Or Connection Problems, Specifically

• Relating to multiplayer performance problem reports:
  • Knowing how long (in actual realtime, not gametime) you were playing in a given session to go along with the amount of data transmitted and performance you were seeing is useful.
  • Knowing rough distances between the players (intercontinental, same country, same metro area, or same building is also useful if you're having performance-related problems.
  • Similarly, knowing the general strength of the internet connections in question is useful if it's a performance problem, as well as which networking framework (Steam, GOG, or LiteNetLib) was used is also helpful. If one or more of you is running this over a VPN, that's also important for us to know.

Multiplayer, At A Glance

This game is a client-server model, with one host, and any number of clients who are connected to that host. There are several networking frameworks available to use, but the host and clients must all be using the same one.

About The Host

• If the host severs the connection, then the game is halted for everyone else. A new host is not automatically chosen.
• Only the host can save the game, at present. Even with our improvements to multiplayer sync, clients are too likely to have slightly-garbled versions of the game.
• The host must start up the game first, either in the lobby, or by loading an existing save or quick start. The clients are then able to connect, based on whatever the rules are of the networking framework everyone is using.
• The host should have the most powerful CPU, as it runs a lot of background AI threads that are too time-consuming and long-running to run in the main simulation, and which do not run on clients.
  • This is similar to AI War Classic, but many threads instead of just one.
• The host should also have the most robust network, as they have to send approximately 2n the data that a client does, where n is the number of clients.
  • So in a two player game, the host is probably sending and receiving roughly 4 times as much data as the sole client.
  • In a three player game, the host is sending and receiving roughly 6 times as much data as either of the two clients.
  • A spectator client adds roughly the same amount of bandwidth requirements as a regular player.

About Clients

• Each client connects to a host, and passes the name of their current Player Profile (as configured in AI War 2 itself).
• Clients (and for that matter the host) are linked back to their prior faction based on the player profile's name, so changing that midway through a campaign with friends is not a great idea.
• Clients are in general in a state of slightly-wrong-sync with the host, though the game does its best to keep everything consistent.
  • There's a rolling sync process that sweeps through and corrects inconsistencies within 3-4 seconds on each client, and so in cases of high desync clients may see things jump to correct themselves to match the host.
  • This is not something the host will see, as the host is treated as the "central correct record" for everything.
• It's worth noting that if a client is lagging behind, it will slow the host and other clients to wait for them. BUT if a client is disconnected entirely, then the game will continue without them (though any other player can pause it; they can then save, stop, or continue).
  • It's also worth noting that the host and the other clients do not have to disconnect or restart anything in order for the missing client to connect back. When the missing player returns, there is usually just a 1-5 second pause (depending on the size of the savegame) while the client gets caught back up to speed. But you don't even have to explicitly pause the game.
  • If sync is ever atrociously off on one client, that client can always use this to their advantage to disconnect and reconnect and have perfect sync again (although sync being that bad off is not supposed to ever happen).

Bandwidth/CPU Requirements And Player Counts

• There is a built-in limit of 254 clients connecting to a single host, but for practical purposes there's really not going to be remotely that many players supported.
• How many clients are actually supported is a matter of bandwidth between them and the host, as well as CPU processing power for the resulting game.
  • If there are 40 clients all sharing one faction, then this is far less of a load than 40 clients all with their own starting planet and unique faction. It's several orders of magnitude less CPU processing, to the point where the former might be barely possible on a nice LAN, but the latter is not possible with current hardware.

• As the years pass and multi-core processors become stronger and also add more cores, AI War 2 will naturally spread out to use those capabilities with no code changes required. It would probably take 16 or so cores before you stop seeing any tangible benefit.

• At the moment, with one client and one host in a fairly average starting galaxy with one AI and two player factions, the bandwidth requirements are less than 100KB per second on the host (in many cases closer to 10KB per second). The client ranges from half to a quarter of that amount, depending on what stage of the game we're talking about.
  • With just two players, far more important than the speed of the connection is the reliability of it. Lots of dropped packets and resends will cause delays even if the actual connection is plenty robust.
  • With two clients and one host (aka 3 players), the client requirements stay the same, but the host bandwidth requirements go up to something in the 200KB range.
  • It's actually expected that, for most practical cases with more than 10 total players, the CPU load of preparing all the extra network messages will probably cause more slowdown than the actual sending of the messages. This is something that we have worked around as much as we can, but also not something that more CPU cores will help given the linear nature of the data we are sending.

• The bottom line is that, in most situations, you should have a fine time as long as you have a 10 or fewer players, but we'd be interested in hearing what sort of experiences you have.
  • For practical purposes, the game itself will probably bog down with more than 4-6 human factions on current hardware.

Which Networking Framework To Use?

The game supports three networking frameworks at present, although more could be added in by ourselves or modders.

What Is A Network Framework?

A networking framework is responsible for transmitting data between players, and for getting players connected in the first place, but it is completely unaware of what data is being transmitted.

Conversely, the game itself is responsible for preparing and handing off all the actual messages it wants to orchestrate between players, and it has no idea where those messages go (at a low technical level) or how they get from the client to different hosts.

Some networking frameworks are more efficient than others in terms of how they use CPU and RAM on the client and host, and also in terms of how they use your network card and available bandwidth. They also can differ in what sort of networks they use, and so again can have a big impact from that area alone.

Steam

In this networking framework, you don't have to worry about IP addresses at all. As long as all of the clients are Steam friends with the host, you're good to go. Firewalls and network configurations don't matter much.

The host starts a game while logged into Steam, but they do not have to be in Online status on the Steam Friends section. Even someone who appears Offline in Steam Friends can be a host or client.

The clients will see a list of their Steam friends, sorted first by those who are currently online and playing AI War 2, and then by those who are online, and then those who (at least seem to be) offline. You can click connect next to any of the friends and try to connect to their game, even if they appear to be offline. It won't hurt their machine or yours, and if they have a game open even while Steam Friends is set to offline, you can connect to them. Please do only connect when invited, though!

It's worth noting that all of the network traffic is routed through Valves relay servers at all times. This will typically give you better performance than the general Internet backbone, according to Valve. You CAN also use this while on the same LAN as other players, but your performance and ping will be as if you were on different networks in the same city. Aka, far worse than if you were on the LAN together, but still plenty great for most purposes.

GOG Galaxy

In this networking framework, you also don't have to worry about IP addresses. As long as all of the clients are GOG friends with the host, you're good to go. Here again, firewalls and network configurations don't matter much.

The clients will see a list of their GOG Galaxy friends, sorted first by those who are currently online and have an active AI War 2 multiplayer session open. You will then see the others who are offline, or in game, etc. The various statuses don't really matter except for the ones that show up as purple, who have a game actively open at the moment. You won't see a connect button except for players in that status, so be sure to refresh the list of friends frequently if one of them just opened a game and you want to connect.

We're not sure exactly how the network traffic is routed. There are relay servers involved if a direct connection can't be made, but we're not sure if a direct connection is used in cases where it can be. Performance seems great in general, but definitely is relay-based if you are on the same LAN.

LiteNetLib

If you're on the same LAN as the other players, then this is a no-brainer choice. If you're on different networks connected to the Internet, then it's your choice; to use this in that situation would probably require port forwarding on your routers, and that's something you can easily look up for multiplayer games in general if you want to go that route.

The game's network tab of the settings menu shows you your public internet IP address, as well as your local IPv4 and IPV6 addresses. You can also configure what port the game runs on (it is using UDP). Only the host would need to set up port forwarding for that port if you are playing across the Internet.

The host would need to give their public IP address (for internet play) or their local IP address (for LAN play) to all of the intended clients, and they then type that into the connection textbox and connect to the host after the host has started or loaded a savegame/quickstart or the lobby.

If you are on a VPN network together, then it works much like a LAN does, but whether or not your IP address for the VPN's virtual network adapter will show up in the game interface is dependent on the VPN service itself. If not, then most VPN services prominently show both your virtual IP address as well as the virtual IP addresses of any other folks in the same session with you.

VPN services usually try to forge a direct connection if they can (which is almost as fast as the port forwarding route, but not quite because of encryption), but fall back on a relay server network of their own if that is not possible. The speed of these relay networks varies hugely depending on the VPN service. Some of them are extremely fast and suitable for gaming, others are direly bad.

So What Should I Use?

• Are you playing the Steam version across the Internet? Use the Steam network framework. It's simple and really fast.
• Are you playing the GOG Galaxy version across the Internet? Use the GOG Galaxy network framework. It's also simple and quite fast.
• Are you playing on a LAN? Consider using LiteNetLib, as it's the most direct and will be the fastest by far. If you have trouble, you can fall back to Steam or GOG.
• Do you feel like avoiding Steam or GOG Galaxy, and you're an expert with your router? Use LiteNetLib with port forwarding for maximum speed.
• Do you feel like avoiding Steam or GOG Galaxy, and you're not a networking wizard? Use LiteNetLib with a VPN service like Hamachi or similar. There are many available.

How Do Expansions (DLC) and Mods Affect Multiplayer?

In general, players must have the same expansions and mods turned on when they are playing multiplayer together. If a client tries to connect to a host with not-enough or too-many things installed, it will inform them of the differences and what changes either they or the host should make to bring them in line.

It is our goal that people never be forced to buy DLC/Expansion content in order to be able to play with friends who have that content. However, while you're playing with those friends, they will have to disable the DLC/Expansion content.

You can disabled expansions/DLC or mods, as needed on the client or the host under the Game tab of the settings menu. That then does require an application restart, as a lot of the data that is included in those packages is very tightly integrated into the main game and can alter things in unexpected ways without the restart.

Compatibility?

All official content from Arcen, ranging from the base game to the Expansions/DLC is compatible with multiplayer. If there are any problems with any of it, that's a bug that we will look into and get fixed for you.

Mods, on the other hand, are not something that we can control in this way, and are something of a wild card. This caveat also includes mods that we distribute with the game, as we are making them more accessible to you, but not taking responsibility for them being bug-free.

That said, mods that are just xml or just visual components are pretty much guaranteed to be compatible in multiplayer, as they are using core game engine features that we designed. So, for instance, there's no risk of the Spire Railgun Shop messing up something in multiplayer.

Mods that include code of any sort tend to be the ones at risk of lack of compatibility, and the more complicated the code is, the more likely problems are. Civilian Industries, which we also include with the game, is a great example of this. We do our best to give tools to the mod authors so that they can make sure that their mods work well in multiplayer, but ultimately that still requires testing on their part, or feedback from people playing their mods in multiplayer.

The sync-correction process that is a core part of the game in general will keep things from a mod working smoothly, or at least not diverging too heavily. But it's also possible for a mod to introduce a memory leak (client side only, or else it's a problem with the mod not limited to just multiplayer) by not handling the sync code properly, or to introduce exceptions during sync itself, etc. It's also possible for a mod to fall prey to the same sort of "random threading exceptions" that other parts of the game have to guard against, through no fault of the mod author.

The most likely thing to happen in a mod with some incompatibilities is that certain data in tooltips will be inconsistent or missing on multiplayer clients. For instance, using Civilian Industries as an example again, it may be that the resources collected by various civilian ships are not reflected in the tooltips on the client the way that they are on the host. This is something that we will work with mod authors on to ensure is as correct as possible wherever they are willing and such an issue is noticed.

How Can Players Participate In A Multiplayer Session?

There are several ways that players can join in. In order to discuss this, we need to make three pieces of terminology clear:

• A player is a specific person, at their computer, who is going to be either the host or one of the clients.
• A player faction is kind of like an empire. In single player games, there is always one empire for the one player. But that's not always the case in multiplayer.
• A player profile is the name and preferred-color you give yourself on your machine. You can have multiple of these, with different names and colors, but only one can be active at a given time.

Example time!

If we have BobX as a player profile on the host machine, and he starts a new custom game, he will immediately be put in charge of player faction 1, which will also be called BobX unless he changes it (via the empire name field on the player faction). The faction color will be set to match his preferred player profile color, again unless he changes that on the settings for the player faction he's now in charge of. So far this is just like a single-player game.

Now if we have SallyB as a player profile on a client machine, and she connects into BobX, then a second player faction will be generated, this one for her. It will get the name SallyB, and take on her preferred color, but it does not have to stay that way. And that's where we start getting into a slight bit of complexity.

Everyone With Their Own Faction

This is the simplest and most obvious way to play, and what happens by default. If you have three clients join the host, then you wind up with four total player factions. Each one has a name that matches faction to player, and that's that.

Although, still, there are some questions worth addressing:

• What if several of the players' preferred colors are too similar or the same?
  • This won't affect anything, gameplay-wise. The color is just a color, and won't merge factions or anything like that.
  • From a practical standpoint, it will be confusing, however. So changing the colors up a bit via either the factions tab in the lobby, or the factions screen from the escape menu during gameplay, might be a good idea. You can always change the colors of any of the factions at any time, so it's never to late to shift things up if you're finding it hard to tell someone apart.

• What if players choose the same name?
  • This is not allowed. Out in the world, globally, people can reuse names as much as they want. But in a specific multiplayer campaign, there can only be one "BobX" and one "SallyB"
  • Fun fact, if BobX and SallyB save and quit the game, then rename their player profiles (or switch player to alternate profiles on their machine that have those other names), then they can come back into the game as each other's factions.
    • Put another way: If BobX is now SallyB, and opens up the savegame to host on his computer again, he will now be in control of SallyB's faction. If she tries to reconnect as SallyB, she'll be told that someone else is already playing with that name. If she tries to connect as BobX to that save she and he were previously playing, then now she has his faction, but is still the client.
    • Why you would want to do this is not entirely clear.
  • More realistically, if there is a second BobX trying to connect to a game with BobX and SallyB, then that second Bob will have to come up with a new name. They can either rename their existing player profile, or just create a new one and use that when they are playing with this specific group. Maybe this new person continues to go by the profile name BobX most of the time, but in this campaign they switch over to a profile name of BobZ so that they can be uniquely identified.

• Why not use something like people's IP addresses or Steam or GOG name or something else like that to solve the above conflict?
  • First of all, because the above problem is really kind of an edge case. Most people choose fairly unique names, especially among groups of friends.
  • But secondly, we assume that people may have their IP addresses change, or may switch around what network framework they use, or any number of other factors.
  • And finally, we actually view it as a really nice feature that anybody can impersonate someone else in a savegame, to see what things look like from their point of view.
    • This is useful for us as developers, but it's also neat for anyone who wants to share savegames on the Internet for others to play.
    • If there's a really challenging savegame with three players where people post it going "can anyone beat this??", then they can give the names that people connect as (Be BobX if you want the big empire up nort, but that one is getting hammered by waves. Be SallyB if you want all those upgraded golems in the east. Be BobZ if you want that almost-dead last planet in the center. Can you win where we lost?) That sort of thing is neat.

At Least One Spectator

A spectator is a player who controls no factions, and thus can do nothing except watch what everyone else sees.

Fun fact, it's possible to play this game in multiplayer, but still have it be single-player in all respects that matter. Have the host be an actual player, and the sole client be a spectator, and now you have a second person who is just watching you play but can't do anything. A twitch stream might be a little more straightforward, but a spectator can look at different things from what you are seeing.

The one thing that spectators can do is chat. They can click around to all the planets that any other human players can view, but they don't see anything extra that the others would not. So you can't use a spectator to cheat, but unlike a twitch stream a spectator can look in places you are not seeing and can alert you to threats. They can thumb through intel, journal entries, watch border planets you don't have time to glance at, and so on.

To become a spectator, or make someone else in your lobby become a spectator, simply remove the faction that is tied to that person. It won't assign them to another faction unless one is added back manually for them.

Alpha Note: we will be expanding the interface a bit to make it easier to set yourself up as a spectator, later during the alpha.

Multiple Players On One Faction

Alpha Note: the interface for setting this up is not yet ready.

It's possible to have multiple players share control of a single faction. There's no explicit limit on how many players can be sharing that control, and not all of the players have to be present for gameplay to proceed.

By default if BobX and SallyB share control of a faction, it will name it BobX and SallyB. With three players, it becomes BobX, SallyB, and BobZ. Beyond that, it adds "and others." But really, if you have more than one player controlling a faction, it might be nice to set the empire name to be something thematic like "Empire of the Rising Bobs."

In chat, it will identify players by their profile name, but the color of their faction. That way you can tell if it's BobX or SallyB talking, despite the fact that they are sharing a faction at the moment.

Many, Many Uses For This!

There are NO RESTRICTIONS on what two players can do if they share a faction together. Essentially they both have equal control of the resources and units, and their own independent copy of the interface. If a friend who is new to the game wants to learn, this is a great way to do it. You are essentially playing a "single player game" in terms of balance, but with two brains and four hands and four eyes controlling the shared empire. If you want to set up an arrangement where certain fleets are used by certain players, that needs to just be an agreement between you.

Various folks have asked us about things like "Champions" or "Helpers" from the first AI War, and this ability to share factions really handles the bulk of the use cases of those old roles, but with more flexibility. Essentially if you want to have one friend share in your empire but just drive a Botnet Golem around causing chaos, then that can be their sole responsibility.

Any Combination Is Fine!

You can mix and match this however you like. Two players can share one faction while a third has their own faction. Yet another player can be a spectator. It doesn't matter which one of them is the host (the host can actually remove themselves from the original faction once a second player is in).

Empty (From The Start) Player Factions

It's possible to add an extra player faction that... isn't controlled by anyone right now. It will just kind of hang out on its planet and gather resources, and not have any intelligence behind it. Its ships will fire on things that come into range, but they won't be in pursuit mode by default.

The only real reason to do this seems to be if you have a two-person group today, but next week it will be three people and you don't mind defending that third planet until then.

Alpha Note: it's possible that late-assigning a player to this won't work properly yet, in the alpha.

Abandoned (Forever Or For A While) Player Factions

Last week, perhaps you had three players who wanted to play, each with their own factions. This week, you want to continue that campaign, but you only have two of the players present. You can load up that savegame and continue with the third faction just sitting there doing whatever its last orders were to do, and that won't be a problem.

If next week the third player is back, you can either go back to where you were when they last played, or you can jump them into the future where they are probably behind and disorganized, but hopefully still living.

What Multiplayer-Specific Features Exist?

Generally speaking, if you can do something in single-player, then you can do the same thing in multiplayer. Overall our goal is not a radically different experience when playing multiplayer, but rather "it's AI War 2, but with more people, and fun." That said, to hit the "and fun" bit, certain things had to be taken into account specific for multiplayer.

Alpha Note: We're going to need new things in here based on how answers to our various 'questions to MP alpha testers' come out.

Copies Of Science And Hacking Points

In the original AI War, every player individually had to gather their own knowledge and hacking points. That was actually really annoying and tedious.

In AI War 2, if any player has a unit gathering science or hacking points, then all players get a copy of those points. So there's never a case of us having to coordinate over gathering science from a distant temporary planet we are holding. If you gather it, I get a copy, and if I go there later, there's nothing left for me to gather since I already have it.

Shared Scouting And Intel

Our scouting levels, and our intel sidebar, are all shared. This is the same as the original AI War.

One of the nice things about the intel sidebar in particular, however, is that any player can right-click the items to change their priority from normal to high or low, and then all players see that. So if there are things that everyone is wanting to coordinate, people can adjust those priorities together until they agree upon them. Or one or more players can be designated to evaluate intel and recommend what to pursue and what to ignore for everyone.

See Where Other Players Are On The Galaxy Map

It's really handy to be able to say "see this planet up here where I am?" This is something you can do by simply clicking the planet, and other players will then see it as highlighted. If they mouseover the planet, they can see your player name (since with many players, this can be confusing as to who is where). This is meant as a visual aid to allow players to see where each other are and thus coordinate a bit better.

What Do I Need To Know About Balance In Multiplayer?

Alpha note: this is actively under discussion, and one reason for doing the alpha.

Overall it's worth noting that if you add in multiple players to a single player faction, that's the same as having one player faction in single-player, and it always will be. The game is definitely not meant to react to having more players, ever, but only to having more player factions.

When it comes to having more player factions, right now the big change in balance compared to single player is that in order to support those factions, more planets will need to be taken, and more fleets captured. This is going to raise the AI Progress (AIP) substantially more than in single-player. But since the AI is not inherently scaling up in any other way in response to multiple player factions, this might actually work out just great. At the moment we don't have enough data to know either way.

In the questions to alpha testers section, we have posed some questions about what people would like to see regarding hacks in particular. But bear in mind that in a two-player-faction game, you get double science and double hacking points for free, plus double starting fleets and starting planets and so on.

The original AI War was really rigid in its number of factions (always exactly two AIs), but the sequel is not. To some extent it may be best if we don't do too much scaling to multiple player factions, and instead let them tune to taste. But if we did that, then essentially every quick start would be extremely easy for four players compared to one, as just one example problem. This again is one of those things that we want more data on from people actually playing: how it feels to them, versus what we can guess at on paper.

Technical Explanation Of The Multiplayer Model

We've written about this extensively over time, and done a few videos, but it's nice to collect this into one place in a brief format.

AI War Classic: The Lockstep Model

The original AI War used a lockstep design along the lines of what you can see in that link. That is indeed what multiplayer RTS games have been doing essentially from the dawn of them existing. We used the same networking model in Tidalis and in Skyward Collapse.

We WERE planning on using the same model in AI War 2, but ultimately we found ourselves getting caught out by the extremely multi-threaded nature of this game. We just can't guarantee absolute determinism between clients and the host, or clients and each other, and so in 2018 we decided to stop trying for absolute rigor in that regard.

We do still mostly use the lockstep approach, and we do our best to keep things deterministic as much as possible, but divergences are expected -- unlike with the original AI War. When a desync was detected in the first AI War, it would stop the game, ask you to save and reconnect, and suggest you give a report to the developers. We would then spend sometimes hours or days chasing down the problem, until it could be fixed and run well again.

(It's worth noting that that level of rigor was also something we realized we could never enforce in mods for AI War 2, and that we'd never be able to tell if a desync was a mod or the core game, so we were really going to be messed up if a desync was a crisis like it was in the first game. There were just a whole host of reasons why a pure lockstep model could no longer work for us.)

(As one other aside, we also really needed to use some floating point math for accuracy in certain places, as our fixed-integer math code was showing its age. We needed this for speed and accuracy with certain kinds of calculations in particular. Floating point numbers are a definite source of desync, which is why AI War Classic doesn't use any of them. Here's a fun rabbit hole, if you want to know more.)

Entity Prediction And Extrapolation?

In our titles A Valley Without Wind 1 and 2, we took our existing lockstep code and changed it into something more appropriate for an action game. We added in our own form of entity prediction (on the client and server side), and error correction interpolation. Since it was a co-op game (not competitive), and cheating wasn't a concern, there was a lot that we were able to do to make that go more smoothly. However, this sort of approach really only works when there are small numbers of entities, a few dozen rather than tens of thousands, and only so many states they can have on them (a single ship in AI War 2 can have several dozen states in its data in hundreds of thousands of possible combinations).

So for the most part, the sort of entity interpolation and input prediction that most action games use (that link is an excellent article on how Valve's Source Engine does it) is a complete no-go for us here.

As one aside, however, back when we were first designing AI War 2 at all, we did use concepts from this in order to handle the entire "visual layer" of AI War 2, and this has had many benefits for us in both single-player and multiplayer.

Essentially, we're running our extremely demanding game simulation at a mere 10fps, no matter what, which is very similar to how things are done in most physics simulations in most major games (AAA and indie) regardless of if they are singleplayer of multi-player. Those other games wind up interpolating between physics steps at a higher framerate, and we do the same thing with our visual display and camera and player input reacting at a much higher framerate than our simulation itself.

Essentially we use that same sort of time-slicing that both Valve and Unity discuss (and PhysX does the same thing), but we do it in a slightly novel way to handle how things are drawn. There are probably other RTs games that do the same thing, but mainly it's something that an MMO would do, since a MMO is another rare case of a game showing an individual player just a small part of a larger simulated world.

But we digress. At any rate, this sort of off-main-thread simulation is both exceedingly common but also unusual in how we implemented it, and it buys us a lot of processing power and ability to multithread that lets us also make small corrections "just happen" when there are sync errors. So that's what we'll get into next.

Sync Correction Code

In AI War 2, we're simulating tens or hundreds of thousands of agents, and a lot of other data at a faction or world or arbitrary-modder-data level. So in a time-sliced fashion, we have the host simply go over a lot of the data and forcibly sync that to clients. In other cases, the host sends a few bits of data to clients for them to compare to their own entities and then make requests for full entity resyncs based on any divergences.

This makes it so that our simulation can react to the simulation being out of sync and self-heal, and in cases of fairly minor discrepancies (the sort you might see from a floating point error) it will actually do so in such a way that our existing "visual later interpolation code" will automatically and smoothly handle the interpolation so that you don't even notice it. In other cases, the sync is more "catastophic" in nature, where ship number 345466 is a nanocaust ship on plant 5 on the host, but a BobX player ship on planet 17 on the client. In those cases, it winds up finding that out within a few seconds (on average within 4), and deletes both wrong ships on the client and recreates them in the image that the host has.

If this was a competitive game, or a game where we were worried about malicious hosts cheating, then this would be big trouble. But as this is a co-op game typically played with friends, we're able to take the host in good faith and deal with sync data in a way that assumes lockstep works most of the time (and it does), but non-destructively fixes divergences when those arise (and they do).

In essence, we're combining several different networking models for maximum performance in as large a simulation as we can possible muster.

Alpha Notes: During alpha, the sync code may be triggered too frequently because of things that could better be in lockstep, and we can improve performance in various areas by improving the determinism of the underlying code, or by providing tools for specific factions or mods to note that they need extra sync at times. At any rate, this is one of those things that we can only really address as we see what sort of data we get, so if there's a particular savegame that is throwing off tons of divergences to an unreasonable degree, that's a great one to throw on the bugtracker for us to look at.

Various Logging Tools For Multiplayer

There are several different methods of enabling several different kinds of logging. All of them are aimed at different purposes, and most of them slow the execution of the game down. So you'll want to enable with care. Let's talk about each of them: