Difference between revisions of "Category:AI War 2: All About Multiplayer"

Discord Channel For Multiplayer

• A great place to meet folks for play or to discuss ideas is our discord channel for AI War 2 multiplayer!

Reporting Bugs Or Suggestions

• Any bugs or requests should go to our mantis bugtracker
  • If you need to submit log files, those can generally be found under your PlayerData folder in the folder your game is installed in. The most relevant one is called ArcenDebugLog.txt. You can send us the whole thing, or just strip out relevant parts.
  • In rare cases, mainly if your entire game crashes (that almost never happens), we will need your unity player log. That gets overwritten the next time you run the game after a crash, unlike the other log. These can be found here:
    • Windows: C:\Users\username\AppData\LocalLow\Arcen Games, LLC\AIWar2\Player.log
    • macOS: ~/Library/Logs/Arcen Games, LLC/AIWar2/Player.log
    • Linux: ~/.config/unity3d/Arcen Games, LLC/AIWar2/Player.log

• If you have been generating logging information that is networking-specific, then that will also likely be in the PlayerData folder, in some cases in subfolders.
  • Generally with those sorts of logs, we need the equivalent log from the host and at least one client (whichever client had the problem).
  • But, also, we really don't expect you to be generating these sorts of logs for us typically, so these really are going above and beyond if you are sending those in.

• In other cases, if you suspect something is wrong, then opening the escape menu during the game and taking a screenshot of the stats there to send to us is welcome.

Reporting Lag Or Connection Problems, Specifically

• For this in particular, there are detailed logs on the client and host machine, in the ArcenDebugLog.txt files. It states how long you were playing for, how much of each type of data was transmitted, what the ping was near the end, and some other general statistics.
  • At the moment, we're no longer seeing any issues that seem to be purely lag-related except in extreme circumstances where that would be warranted (people way too far apart with giant games, where it's just a matter of the speed of light and the many routers between them that make the problem).

Multiplayer, At A Glance

This game is a client-server model, with one host, and any number of clients who are connected to that host. There are several networking frameworks available to use, but the host and clients must all be using the same one.

About The Host

• If the host severs the connection, then the game is halted for everyone else. A new host is not automatically chosen.
• Only the host can save the game, at present. Even with our improvements to multiplayer sync, clients are too likely to have slightly-garbled versions of the game.
• The host must start up the game first, either in the lobby, or by loading an existing save or quick start. The clients are then able to connect, based on whatever the rules are of the networking framework everyone is using.
• The host should have the most powerful CPU, as it runs a lot of background AI threads that are too time-consuming and long-running to run in the main simulation, and which do not run on clients.
  • This is similar to AI War Classic, but many threads instead of just one.
• The host should also have the most robust network, as they have to send approximately 2n the data that a client does, where n is the number of clients.
  • So in a two player game, the host is probably sending and receiving roughly 4 times as much data as the sole client.
  • In a three player game, the host is sending and receiving roughly 6 times as much data as either of the two clients.
  • A spectator client adds roughly the same amount of bandwidth requirements as a regular player.

About Clients

• Each client connects to a host, and passes the name of their current Player Profile (as configured in AI War 2 itself).
• Clients (and for that matter the host) are linked back to their prior faction based on the player profile's name, so changing that midway through a campaign with friends is not a great idea.
• Clients are in general in a state of slightly-wrong-sync with the host, though the game does its best to keep everything consistent.
  • There's a rolling sync process that sweeps through and corrects inconsistencies within 3-4 seconds on each client, and so in cases of high desync clients may see things jump to correct themselves to match the host.
  • This is not something the host will see, as the host is treated as the "central correct record" for everything.
• It's worth noting that if a client is lagging behind, it will slow the host and other clients to wait for them. BUT if a client is disconnected entirely, then the game will continue without them (though any other player can pause it; they can then save, stop, or continue).
  • It's also worth noting that the host and the other clients do not have to disconnect or restart anything in order for the missing client to connect back. When the missing player returns, there is usually just a 1-5 second pause (depending on the size of the savegame) while the client gets caught back up to speed. But you don't even have to explicitly pause the game.
  • If sync is ever atrociously off on one client, that client can always use this to their advantage to disconnect and reconnect and have perfect sync again (although sync being that bad off is not supposed to ever happen).

Bandwidth/CPU Requirements And Player Counts

• There is a built-in limit of 254 clients connecting to a single host, but for practical purposes there's really not going to be remotely that many players supported.
• How many clients are actually supported is a matter of bandwidth between them and the host, as well as CPU processing power for the resulting game.
  • If there are 40 clients all sharing one faction, then this is far less of a load than 40 clients all with their own starting planet and unique faction. It's several orders of magnitude less CPU processing, to the point where the former might be barely possible on a nice LAN, but the latter is not possible with current hardware.

• As the years pass and multi-core processors become stronger and also add more cores, AI War 2 will naturally spread out to use those capabilities with no code changes required. It would probably take 16 or so cores before you stop seeing any tangible benefit.

• At the moment, with one client and one host in a fairly average starting galaxy with one AI and two player factions, the bandwidth requirements are less than 100KB per second on the host (in many cases closer to 10KB per second). The client ranges from half to a quarter of that amount, depending on what stage of the game we're talking about.
  • With just two players, far more important than the speed of the connection is the reliability of it. Lots of dropped packets and resends will cause delays even if the actual connection is plenty robust.
  • With two clients and one host (aka 3 players), the client requirements stay the same, but the host bandwidth requirements go up to something in the 200KB range.
  • It's actually expected that, for most practical cases with more than 10 total players, the CPU load of preparing all the extra network messages will probably cause more slowdown than the actual sending of the messages. This is something that we have worked around as much as we can, but also not something that more CPU cores will help given the linear nature of the data we are sending.

• The bottom line is that, in most situations, you should have a fine time as long as you have a 10 or fewer players, but we'd be interested in hearing what sort of experiences you have.
  • For practical purposes, the game itself will probably bog down with more than 4-6 human factions on current hardware.

Which Networking Framework To Use?

The game supports three networking frameworks at present, although more could be added in by ourselves or modders.

What Is A Network Framework?

A networking framework is responsible for transmitting data between players, and for getting players connected in the first place, but it is completely unaware of what data is being transmitted.

Conversely, the game itself is responsible for preparing and handing off all the actual messages it wants to orchestrate between players, and it has no idea where those messages go (at a low technical level) or how they get from the client to different hosts.

Some networking frameworks are more efficient than others in terms of how they use CPU and RAM on the client and host, and also in terms of how they use your network card and available bandwidth. They also can differ in what sort of networks they use, and so again can have a big impact from that area alone.

Steam

In this networking framework, you don't have to worry about IP addresses at all. As long as all of the clients are Steam friends with the host, you're good to go. Firewalls and network configurations don't matter much.

The host starts a game while logged into Steam, but they do not have to be in Online status on the Steam Friends section. Even someone who appears Offline in Steam Friends can be a host or client.

The clients will see a list of their Steam friends, sorted first by those who are currently online and playing AI War 2, and then by those who are online, and then those who (at least seem to be) offline. You can click connect next to any of the friends and try to connect to their game, even if they appear to be offline. It won't hurt their machine or yours, and if they have a game open even while Steam Friends is set to offline, you can connect to them. Please do only connect when invited, though!

It's worth noting that all of the network traffic is routed through Valves relay servers at all times. This will typically give you better performance than the general Internet backbone, according to Valve. You CAN also use this while on the same LAN as other players, but your performance and ping will be as if you were on different networks in the same city. Aka, far worse than if you were on the LAN together, but still plenty great for most purposes.

GOG Galaxy

In this networking framework, you also don't have to worry about IP addresses. As long as all of the clients are GOG friends with the host, you're good to go. Here again, firewalls and network configurations don't matter much.

The clients will see a list of their GOG Galaxy friends, sorted first by those who are currently online and have an active AI War 2 multiplayer session open. You will then see the others who are offline, or in game, etc. The various statuses don't really matter except for the ones that show up as purple, who have a game actively open at the moment. You won't see a connect button except for players in that status, so be sure to refresh the list of friends frequently if one of them just opened a game and you want to connect.

We're not sure exactly how the network traffic is routed. There are relay servers involved if a direct connection can't be made, but we're not sure if a direct connection is used in cases where it can be. Performance seems great in general, but definitely is relay-based if you are on the same LAN.

Presently there is no way to directly make a connection between a GOG Galaxy version of the game and a Steam version of the game using the GOG Galaxy framework. That's a case where you'd need to use LiteNetLib (see below).

LiteNetLib

If you're on the same LAN as the other players, then this is a no-brainer choice. If you're on different networks connected to the Internet, then it's your choice; to use this in that situation would probably require port forwarding on your routers, and that's something you can easily look up for multiplayer games in general if you want to go that route.

The game's network tab of the settings menu shows you your public internet IP address, as well as your local IPv4 and IPV6 addresses. You can also configure what port the game runs on (it is using UDP). Only the host would need to set up port forwarding for that port if you are playing across the Internet.

The host would need to give their public IP address (for internet play) or their local IP address (for LAN play) to all of the intended clients, and they then type that into the connection textbox and connect to the host after the host has started or loaded a savegame/quickstart or the lobby.

If you are on a VPN network together, then it works much like a LAN does, but whether or not your IP address for the VPN's virtual network adapter will show up in the game interface is dependent on the VPN service itself. If not, then most VPN services prominently show both your virtual IP address as well as the virtual IP addresses of any other folks in the same session with you.

VPN services usually try to forge a direct connection if they can (which is almost as fast as the port forwarding route, but not quite because of encryption), but fall back on a relay server network of their own if that is not possible. The speed of these relay networks varies hugely depending on the VPN service. Some of them are extremely fast and suitable for gaming, others are direly bad.

So What Should I Use?

• Are you playing the Steam version across the Internet? Use the Steam network framework. It's simple and really fast.
• Are you playing the GOG Galaxy version across the Internet? Use the GOG Galaxy network framework. It's also simple and quite fast.
• Are you playing on a LAN? Consider using LiteNetLib, as it's the most direct and will be the fastest by far. If you have trouble, you can fall back to Steam or GOG.
• Do you feel like avoiding Steam or GOG Galaxy, and you're an expert with your router? Use LiteNetLib with port forwarding for maximum speed.
• Do you feel like avoiding Steam or GOG Galaxy, and you're not a networking wizard? Use LiteNetLib with a VPN service like Hamachi or similar. There are many available.

What If I Only Have One Steam Or GOG Galaxy Account?

If you are planning to use Steam networking, or GOG Galaxy networking, you need a unique account on the respective service for each machine that is going to use it. They each need to own the base game, although the DLC can easily be pirated by just copying it from one machine to the other. (Please don't, but literally piracy is as easy as copying thing, since we don't believe in DRM).

The only reason that you have to have unique accounts is because the connections in these cases are account-based. And the reason that you have to own those games is that those services check for that when you use them -- the services are paying for the bandwidth usage of games that are run on them, and they'd rather not have games not owned on their service run multiplayer through their relays, which seems reasonable.

Anyhow, so for testing purposes Arcen has a few Steam and GOG accounts to run between multiple computers, as one example. But if we were to play with family members, they would just each need a Steam key or GOG key for their own accounts.

The LiteNetLib networking framework has absolutely zero restrictions, so you can put the game on a thumb drive and copy it to 4000 people and they can all play each other in multiplayer without any sort of checks or restrictions. Again, please don't pirate our stuff, but it's stupidly easy to because we're not trying to stop you.

Questions About Specific Errors

Steamworks: ProblemDetectedLocally

If you run into this error during play, this is the steam client passing back to our code (through the Facepunch Steamworks wrapper) a "we had some sort of problem, no idea what" error message. There's not anything we can do to debug this, as it's either a bug in the steam client itself, some sort of networking issue that the Steam client was facing, or otherwise. The best suggestion is to use GOG to connect, or some other form of networking framework (LiteNetLib, for instance).

Steamworks can be a really convenient method for networking, but it also has a lot of infrastructure underneath it, and it doesn't always work properly. LiteNetLib, by contrast, is... well, very light. It has its own code, and then just direct sockets connections from one machine to the other. There are many fewer things that can go wrong, and you aren't dependent on the state of the nearest Valve datacenter to you.

Problems Connecting Or Right After The Lobby

The very first thing to do in all cases is to verify your local steam file cache (right click the game, hit properties, click to local files, and then hit the button that says "Verify integrity of game files...") on the client and the host.

Sometimes Steam in particular doesn't download all files quite correctly, which can cause issues in single player or multiplayer, but multiplayer is even more sensitive to it. If the client and the host have slightly different files for some reason, then you'll get kind of nonsensical errors when the world state is transmitted from the host to the client.

A Giant Rash Of Inexplicable Problems Happened (Error Cascade)

What Happens

You're playing, and then you get one error popup. You hit ignore, but then after that a bunch more errors suddenly appear, endlessly, and/or parts of the interface stop working. Cats and dogs start living together, units disappear, the AI starts speaking in tongues, etc.

What That Means

The initial error is the problem. Please send us a report with it, and we'll take care of it. Everything that happens after that is... maybe a problem? But usually not. Please do send the entire log so that we can make a determination either way, but often there's not much to do with the later errors. If you let your log accumulate errors for too long (it loops after 4 MB or so to keep from flooding your disk), then we won't be able to see the initial error.

At any rate, once you have an initial error like this, the game is going to be horribly unstable until you disconnect the client and reconnect, anyway.

Can I Have An Analogy?

Right, so multiplayer and the threading model are like a room packed full of very complex machines, all spinning away at once. Sometimes, one of the machines will throw an error, and it's not a big deal. A little piece falls off this one machine, and so that machine stops doing whatever it was supposed to, but beyond that the rest of the game -- the other machines -- all work just fine.

What happens in THIS case is very different, though: whatever machine failed, it flung some gears or sprockets into other machines that were functioning fine up until they had hot shards of metal flung into them. Now they are breaking and also probably throwing out slag of their own. Pretty soon the entire room is a shambles and on fire.

When possible, we put up protections to protect one machine from another, but we're generally faced with the problem of trying to get as many machines as possible into a small space, and have them run as fast and hot as possible, so there are limits to the ability to do that. This does mean that one rare bug that a player hits can make the entire game look incredibly unstable, because it's flinging debris and setting the room on fire.

"Rare Bug?" This Happens Every Time I Play!

A "rare" bug just means that it hasn't been encountered before and reported to us, not that it's an inconsistent bug. You may have a savegame where, every time you restart it, the room catches on fire within moments. Everyone else seems to be having a fine time but you for some reason. In this situation, please just do send us the savegame as well as the bug logs from the client in particular (host is rarely the issue, but if the host has an error that is also useful). If there are steps required to recreate the bug, please let us know what to do. With the save in hand, or sometimes just the log, we're able to fix the initial issue with usually only a line or two of code, as well as put up some extra shielding.

Without having the logs in hand, or the save, or both, there's really nothing we can do, though. We've run lots of multiplayer sessions for lengthy amounts of time with zero issues, and so it's always a matter of some specific confluence of events that combine in just the wrong way. When that leads to the entire room catching on fire, it can make it seem like multiplayer itself isn't stable at all, but that's not quite how I would characterize it. We try to handle this sort of thing quickly, but the game is huge and the number of multiplayer enthusiasts is comparably low relative to single player, so you may be the unlucky "first" to find some sprocket-flinging locale. Thank you for your help!

How Do Expansions (DLC) and Mods Affect Multiplayer?

In general, players must have the same expansions and mods turned on when they are playing multiplayer together. If a client tries to connect to a host with not-enough or too-many things installed, it will inform them of the differences and what changes either they or the host should make to bring them in line.

It is our goal that people never be forced to buy DLC/Expansion content in order to be able to play with friends who have that content. However, while you're playing with those friends, they will have to disable the DLC/Expansion content.

You can disabled expansions/DLC or mods, as needed on the client or the host under the Game tab of the settings menu. That then does require an application restart, as a lot of the data that is included in those packages is very tightly integrated into the main game and can alter things in unexpected ways without the restart.

Compatibility?

All official content from Arcen, ranging from the base game to the Expansions/DLC is compatible with multiplayer. If there are any problems with any of it, that's a bug that we will look into and get fixed for you.

Mods, on the other hand, are not something that we can control in this way, and are something of a wild card. This caveat also includes mods that we distribute with the game, as we are making them more accessible to you, but not taking responsibility for them being bug-free.

That said, mods that are just xml or just visual components are pretty much guaranteed to be compatible in multiplayer, as they are using core game engine features that we designed. So, for instance, there's no risk of the Spire Railgun Shop messing up something in multiplayer.

Mods that include code of any sort tend to be the ones at risk of lack of compatibility, and the more complicated the code is, the more likely problems are. Civilian Industries, which we also include with the game, is a great example of this.

During the course of the alpha of multiplayer, we made a number of improvements that make most sync considerations for mods kind of irrelevant:

• If a mod does not properly coordinate actions between the client and hosts, the game will notice that within a few seconds and fix it.
• If there is data from the mod that is out of sync with the clients, but it doesn't affect actions of ships or units, then there's no problem in the first place.
• If there is data from the mod that would affect tooltips on the clients, that's no longer a worry because as soon as clients hover over a unit to get a tooltip it now gets an updated sync copy from the server. This eliminates almost all potential problems, and uses very little data.

So, the sync-correction process will keep almost everything with a mod working just fine. But it's also possible for a mod to introduce a memory leak (client side only, or else it's a problem with the mod not limited to just multiplayer) by not handling the sync code properly, or to introduce exceptions during sync itself, etc. It's also possible for a mod to fall prey to the same sort of "random threading exceptions" that other parts of the game have to guard against, through no fault of the mod author. These things are usually pretty quick fixes once they are identified by players.

Distribution Over The Network In The Future?

We don't have any plans to make it so that mods or DLC can be downloaded from the host to clients, or vice-versa, in the future.

When it comes to DLC, there are a couple of things that stick out there. For one, if the visuals were to be downloaded, this usually means sending something like 1GB of data from the host to the clients (or vice-versa) to make this work. The format of the visuals and so on is platform-specific (OSX, linux, or windows), so if you are playing between different OSes, the host or client might not even have compatible files for one another to send.

Secondly, aside from the technical hurdles on DLC, this gets generous to the point of shooting ourselves in the foot when it comes to DLC. It's extremely easy to pirate the DLC from one client to another (just copy the files) if you really want to do that, but if you're a group and you'd like to play with the DLC, we hope you'll consider actually supporting us instead. If you're playing with folks newer to the game, and you understandably don't want to have them go through extra investment into DLC before you're sure you enjoy playing it together, we've made it really easy for you to turn off your DLC while you are playing with friends so that you can do that without anyone having to spend any money.

Lastly, and this applies to both DLC and mods, but you'll notice that enabling or disabling any of them requires a restart of the game, anyway. This is because these mods and DLC can run extremely deep when they are making changes to the game, including changing or adding content to things that were in the base game. There's not a way for us to be sure that you are back on "baseline" without that restart and a fresh l oad of the data files. So even if we were to set up the ability to send a mod over the wire from the host to the client, or vice-versa, applying the content from that mod would not be a simple matter without rebooting and breaking the connection anyhow.

How Can Players Participate In A Multiplayer Session?

There are several ways that players can join in. In order to discuss this, we need to make three pieces of terminology clear:

• A player is a specific person, at their computer, who is going to be either the host or one of the clients.
• A player faction is kind of like an empire. In single player games, there is always one empire for the one player. But that's not always the case in multiplayer.
• A player profile is the name and preferred-color you give yourself on your machine. You can have multiple of these, with different names and colors, but only one can be active at a given time.

Example time!

If we have BobX as a player profile on the host machine, and he starts a new custom game, he will immediately be put in charge of player faction 1, which will also be called BobX unless he changes it (via the empire name field on the player faction). The faction color will be set to match his preferred player profile color, again unless he changes that on the settings for the player faction he's now in charge of. So far this is just like a single-player game.

Now if we have SallyB as a player profile on a client machine, and she connects into BobX, then a second player faction will be generated, this one for her. It will get the name SallyB, and take on her preferred color, but it does not have to stay that way. And that's where we start getting into a slight bit of complexity.

Everyone With Their Own Faction

This is the simplest and most obvious way to play, and what happens by default. If you have three clients join the host, then you wind up with four total player factions. Each one has a name that matches faction to player, and that's that.

Although, still, there are some questions worth addressing:

• What if several of the players' preferred colors are too similar or the same?
  • This won't affect anything, gameplay-wise. The color is just a color, and won't merge factions or anything like that.
  • From a practical standpoint, it will be confusing, however. So changing the colors up a bit via either the factions tab in the lobby, or the factions screen from the escape menu during gameplay, might be a good idea. You can always change the colors of any of the factions at any time, so it's never to late to shift things up if you're finding it hard to tell someone apart.

• What if players choose the same name?
  • This is not allowed. Out in the world, globally, people can reuse names as much as they want. But in a specific multiplayer campaign, there can only be one "BobX" and one "SallyB"
  • Fun fact, if BobX and SallyB save and quit the game, then rename their player profiles (or switch player to alternate profiles on their machine that have those other names), then they can come back into the game as each other's factions.
    • Put another way: If BobX is now SallyB, and opens up the savegame to host on his computer again, he will now be in control of SallyB's faction. If she tries to reconnect as SallyB, she'll be told that someone else is already playing with that name. If she tries to connect as BobX to that save she and he were previously playing, then now she has his faction, but is still the client.
    • Why you would want to do this is not entirely clear.
  • More realistically, if there is a second BobX trying to connect to a game with BobX and SallyB, then that second Bob will have to come up with a new name. They can either rename their existing player profile, or just create a new one and use that when they are playing with this specific group. Maybe this new person continues to go by the profile name BobX most of the time, but in this campaign they switch over to a profile name of BobZ so that they can be uniquely identified.

• Why not use something like people's IP addresses or Steam or GOG name or something else like that to solve the above conflict?
  • First of all, because the above problem is really kind of an edge case. Most people choose fairly unique names, especially among groups of friends.
  • But secondly, we assume that people may have their IP addresses change, or may switch around what network framework they use, or any number of other factors.
  • And finally, we actually view it as a really nice feature that anybody can impersonate someone else in a savegame, to see what things look like from their point of view.
    • This is useful for us as developers, but it's also neat for anyone who wants to share savegames on the Internet for others to play.
    • If there's a really challenging savegame with three players where people post it going "can anyone beat this??", then they can give the names that people connect as (Be BobX if you want the big empire up nort, but that one is getting hammered by waves. Be SallyB if you want all those upgraded golems in the east. Be BobZ if you want that almost-dead last planet in the center. Can you win where we lost?) That sort of thing is neat.

At Least One Spectator

A spectator is a player who controls no factions, and thus can do nothing except watch what everyone else sees.

Fun fact, it's possible to play this game in multiplayer, but still have it be single-player in all respects that matter. Have the host be an actual player, and the sole client be a spectator, and now you have a second person who is just watching you play but can't do anything. A twitch stream might be a little more straightforward, but a spectator can look at different things from what you are seeing.

The one thing that spectators can do is chat. They can click around to all the planets that any other human players can view, but they don't see anything extra that the others would not. So you can't use a spectator to cheat, but unlike a twitch stream a spectator can look in places you are not seeing and can alert you to threats. They can thumb through intel, journal entries, watch border planets you don't have time to glance at, and so on.

To become a spectator, or make someone else in your lobby become a spectator, simply remove the faction that is tied to that person. It won't assign them to another faction unless one is added back manually for them.

Multiple Players On One Faction

It's possible to have multiple players share control of a single faction. There's no explicit limit on how many players can be sharing that control, and not all of the players have to be present for gameplay to proceed.

By default if BobX and SallyB share control of a faction, it will name it BobX and SallyB. With three players, it becomes BobX, SallyB, and BobZ. Beyond that, it adds "and others." But really, if you have more than one player controlling a faction, it might be nice to set the empire name to be something thematic like "Empire of the Rising Bobs."

In chat, it will identify players by their profile name, but the color of their faction. That way you can tell if it's BobX or SallyB talking, despite the fact that they are sharing a faction at the moment.

Many, Many Uses For This!

There are NO RESTRICTIONS on what two players can do if they share a faction together. Essentially they both have equal control of the resources and units, and their own independent copy of the interface. If a friend who is new to the game wants to learn, this is a great way to do it. You are essentially playing a "single player game" in terms of balance, but with two brains and four hands and four eyes controlling the shared empire. If you want to set up an arrangement where certain fleets are used by certain players, that needs to just be an agreement between you.

Various folks have asked us about things like "Champions" or "Helpers" from the first AI War, and this ability to share factions really handles the bulk of the use cases of those old roles, but with more flexibility. Essentially if you want to have one friend share in your empire but just drive a Botnet Golem around causing chaos, then that can be their sole responsibility.

Any Combination Is Fine!

You can mix and match this however you like. Two players can share one faction while a third has their own faction. Yet another player can be a spectator. It doesn't matter which one of them is the host (the host can actually remove themselves from the original faction once a second player is in).

Empty (From The Start) Player Factions

It's possible to add an extra player faction that... isn't controlled by anyone right now. It will just kind of hang out on its planet and gather resources, and not have any intelligence behind it. Its ships will fire on things that come into range, but they won't be in pursuit mode by default.

The only real reason to do this seems to be if you have a two-person group today, but next week it will be three people and you don't mind defending that third planet until then.

Abandoned (Forever Or For A While) Player Factions

Last week, perhaps you had three players who wanted to play, each with their own factions. This week, you want to continue that campaign, but you only have two of the players present. You can load up that savegame and continue with the third faction just sitting there doing whatever its last orders were to do, and that won't be a problem.

If next week the third player is back, you can either go back to where you were when they last played, or you can jump them into the future where they are probably behind and disorganized, but hopefully still living.

What Multiplayer-Specific Features Exist?

Generally speaking, if you can do something in single-player, then you can do the same thing in multiplayer. Overall our goal is not a radically different experience when playing multiplayer, but rather "it's AI War 2, but with more people, and fun." That said, to hit the "and fun" bit, certain things had to be taken into account specific for multiplayer.

Copies Of Science And Hacking Points

In the original AI War, every player individually had to gather their own knowledge and hacking points. That was actually really annoying and tedious.

In AI War 2, if any player has a unit gathering science or hacking points, then all players get a copy of those points. So there's never a case of us having to coordinate over gathering science from a distant temporary planet we are holding. If you gather it, I get a copy, and if I go there later, there's nothing left for me to gather since I already have it.

It is worth noting that when you have more player empires, everyone does get FEWER hacking points per planet than they otherwise would.

Shared Scouting And Intel

Our scouting levels, and our intel sidebar, are all shared. This is the same as the original AI War.

One of the nice things about the intel sidebar in particular, however, is that any player can right-click the items to change their priority from normal to high or low, and then all players see that. So if there are things that everyone is wanting to coordinate, people can adjust those priorities together until they agree upon them. Or one or more players can be designated to evaluate intel and recommend what to pursue and what to ignore for everyone.

See Where Other Players Are On The Galaxy Map

It's really handy to be able to say "see this planet up here where I am?" This is something you can do by simply clicking the planet, and other players will then see it as highlighted. If they mouseover the planet, they can see your player name (since with many players, this can be confusing as to who is where). This is meant as a visual aid to allow players to see where each other are and thus coordinate a bit better.

Map Pings

By hitting F1, you can go into "ping mode," or there's a button on the galaxy map at the bottom of the screen. You can put as many pings as you want, and they last for six seconds. A little notice in the upper right corner while in ping mode notes some modifier keys that allow you to change the color of pings if you are trying to differentiate a few things.

In general, pings are for letting you "point at" things on the galaxy map or a planet view. In the planet view, pings get smaller over time, so you can generally indicate direction with them.

Gifting Fleet Lines Between Players

This is so easy that you might miss it! Simply go into the normal interface for shifting ship lines around between fleets, and you will find that you can also shift them to/from other player empires. The mark levels of the ships will reset to be whatever is appropriate for the target player's unlocked techs (higher, lower, or the same).

Gifting Fleets/Planets Between Players

If you go into the fleet interface, there's a button there that will allow you to gift any fleet, other than your home command station fleet itself, to an ally. If there were any science upgrades made to a fleet, those are refunded to you as the fleet changes ownership. Sharing science between players is not permitted under any circumstances, because everyone already has a copy of the science for their own purposes.

One-Time Gifts Between Players

You can gift bulk amounts of metal or hacking points between players at any time. Hit F2 to bring up the factions interface to your empire's page without going through the escape menu. On this screen, you can make gifts to help out a struggling ally, or you can even go in and make gifts on their behalf to you or to someone else. It is really much more polite to ask permission first, though!

Ongoing Gifts Between Players

You can gift ongoing per-second amounts of metal, or ongoing energy volumes, between players. Hit F2 to bring up the factions interface to your empire's page without going through the escape menu. On this screen, you can make gifts to let an economic powerhouse fund another empire that is churning through battles, or whatever else makes sense.

What Do I Need To Know About Balance In Multiplayer?

Overall it's worth noting that if you add in multiple players to a single player faction, that's the same as having one player faction in single-player, and it always will be. The game is definitely not meant to react to having more players, but only to having more player factions.

When it comes to having more player factions, right now the big change in balance compared to single player is that in order to support those factions, more planets will need to be taken, and more fleets captured. This is going to raise the AI Progress (AIP) substantially more than in single-player. Overall things will probably be a bit easier than if you were playing solo against a given AI, but it's definitely not as hard as adding multiple AIs, or cranking up the AI difficulty.

When you have multiple human empires, extra capturables are seeded around the galaxy map so that everyone has enough. It's probably not a great idea to go over about 4 human empires, though, or balance is likely to start to really break down and there may not be enough room for everyone on large maps. You can definitely do more than four human empires, but you might have a better experience if at least some of the extra players share empires, instead.

Technical Explanation Of The Multiplayer Model

We've written about this extensively over time, and done a few videos, but it's nice to collect this into one place in a brief format.

AI War Classic: The Lockstep Model

The original AI War used a lockstep design along the lines of what you can see in that link. That is indeed what multiplayer RTS games have been doing essentially from the dawn of them existing. We used the same networking model in Tidalis and in Skyward Collapse.

We WERE planning on using the same model in AI War 2, but ultimately we found ourselves getting caught out by the extremely multi-threaded nature of this game. We just can't guarantee absolute determinism between clients and the host, or clients and each other, and so in 2018 we decided to stop trying for absolute rigor in that regard.

We do still mostly use the lockstep approach, and we do our best to keep things deterministic as much as possible, but divergences are expected -- unlike with the original AI War. When a desync was detected in the first AI War, it would stop the game, ask you to save and reconnect, and suggest you give a report to the developers. We would then spend sometimes hours or days chasing down the problem, until it could be fixed and run well again.

(It's worth noting that that level of rigor was also something we realized we could never enforce in mods for AI War 2, and that we'd never be able to tell if a desync was a mod or the core game, so we were really going to be messed up if a desync was a crisis like it was in the first game. There were just a whole host of reasons why a pure lockstep model could no longer work for us.)

(As one other aside, we also really needed to use some floating point math for accuracy in certain places, as our fixed-integer math code was showing its age. We needed this for speed and accuracy with certain kinds of calculations in particular. Floating point numbers are a definite source of desync, which is why AI War Classic doesn't use any of them. Here's a fun rabbit hole, if you want to know more.)

Entity Prediction And Extrapolation?

In our titles A Valley Without Wind 1 and 2, we took our existing lockstep code and changed it into something more appropriate for an action game. We added in our own form of entity prediction (on the client and server side), and error correction interpolation. Since it was a co-op game (not competitive), and cheating wasn't a concern, there was a lot that we were able to do to make that go more smoothly. However, this sort of approach really only works when there are small numbers of entities, a few dozen rather than tens of thousands, and only so many states they can have on them (a single ship in AI War 2 can have several dozen states in its data in hundreds of thousands of possible combinations).

So for the most part, the sort of entity interpolation and input prediction that most action games use (that link is an excellent article on how Valve's Source Engine does it) is a complete no-go for us here.

As one aside, however, back when we were first designing AI War 2 at all, we did use concepts from this in order to handle the entire "visual layer" of AI War 2, and this has had many benefits for us in both single-player and multiplayer.

Essentially, we're running our extremely demanding game simulation at a mere 10fps, no matter what, which is very similar to how things are done in most physics simulations in most major games (AAA and indie) regardless of if they are singleplayer of multi-player. Those other games wind up interpolating between physics steps at a higher framerate, and we do the same thing with our visual display and camera and player input reacting at a much higher framerate than our simulation itself.

Essentially we use that same sort of time-slicing that both Valve and Unity discuss (and PhysX does the same thing), but we do it in a slightly novel way to handle how things are drawn. There are probably other RTs games that do the same thing, but mainly it's something that an MMO would do, since a MMO is another rare case of a game showing an individual player just a small part of a larger simulated world.

But we digress. At any rate, this sort of off-main-thread simulation is both exceedingly common but also unusual in how we implemented it, and it buys us a lot of processing power and ability to multithread that lets us also make small corrections "just happen" when there are sync errors. So that's what we'll get into next.

Sync Correction Code

In AI War 2, we're simulating tens or hundreds of thousands of agents, and a lot of other data at a faction or world or arbitrary-modder-data level. So in a time-sliced fashion, we have the host simply go over a lot of the data and forcibly sync that to clients. In other cases, the host sends a few bits of data to clients for them to compare to their own entities and then make requests for full entity resyncs based on any divergences.

This makes it so that our simulation can react to the simulation being out of sync and self-heal, and in cases of fairly minor discrepancies (the sort you might see from a floating point error) it will actually do so in such a way that our existing "visual later interpolation code" will automatically and smoothly handle the interpolation so that you don't even notice it. In other cases, the sync is more "catastophic" in nature, where ship number 345466 is a nanocaust ship on plant 5 on the host, but a BobX player ship on planet 17 on the client. In those cases, it winds up finding that out within a few seconds (on average within 4), and deletes both wrong ships on the client and recreates them in the image that the host has.

If this was a competitive game, or a game where we were worried about malicious hosts cheating, then this would be big trouble. But as this is a co-op game typically played with friends, we're able to take the host in good faith and deal with sync data in a way that assumes lockstep works most of the time (and it does), but non-destructively fixes divergences when those arise (and they do).

In essence, we're combining several different networking models for maximum performance in as large a simulation as we can possible muster.

We also have made it so that clients can request updated information for certain units, and they now do that specifically for anything you're getting a tooltip about. This makes it so that any ancillary data about a unit is automatically synced when you go to look at its details.

Various Logging Tools For Multiplayer

There are several different methods of enabling several different kinds of logging. All of them are aimed at different purposes, and most of them slow the execution of the game down. So you'll want to enable with care. Let's talk about each of them, and when you would use them. You're going to get basically no information of use from most of them, so it's a matter of when you might use them in order to send logs to us.

Log All Decoded Network Sync Data To Disk

This is really only useful if a client and the host both have this on, ideally from the start of the client connecting to the host. If you are seeing repeated errors on the client or the host that pop up and say something about "serialization" or "sync" or something to that effect, then there may be a problem with the sync data serializing properly, and this would be a way to document what is actually happening between computers.

1. First of all, you need to have a savegame that your host is able to load, and that once the client connects to it, it reliably causes the problem within 30 seconds of realtime at an absolute minimum. If it requires unpausing and clicking around in some specific fashion, that's fine, but make sure you document what those steps are and if the client or host has to do it.

2. Now that you have this save in hand, look in your PlayerData folder and see if there is a NetworkSyncMessages subfolder. On both the client and the host, if there is anything in that folder, delete it.

3. Now make sure that this "Log All Decoded Network Sync Data To Disk" setting is on on the host AND the client. It's under the Network tab of your personal settings window.

4. Have the host load the savegame that demonstrates your problem. You will note that nothing gets logged to disk on the host at this point, because there is no client to share sync data with.

5. Have the client connect to the host, and as soon as that connection is complete it will start logging a tremendous amount of data to your disk. It's something along the lines of 200kb every second, which is one reason you really can't let this run for too long or you'll have hundreds of MB of data.

6. Quickly unpause and do whatever it is that causes the problem. Once the problem has happened, have the client and the host both kill the game. If you're running in windowed mode, then hitting the X in the top corner of the window (or red dot on OSX) will do it. On windows, you can use Alt+F4, or on OSX you can use Command+Q.

7. On each of the two machines, zip up the entire contents of the NetworkSyncMessages subfolder, and be sure to label which one was from the host machine and which one is from the client machine. We will later unzip both into separate folders, and compare them with a tool like WinMerge, which will show us all of the differences in what the host thinks it sent and what the client thinks it received (and vice-versa in some cases). Somewhere there is some form of typo or inconsistency, and this will let us find and fix that.

In order for us to verify the fix, we will also need your savegame from the host that you used to reproduce this, and it would probably be helpful to have the ArcenDebugLog.txt file from both the client and the host. But it's really not required.

As far as logging goes, this is pretty intensive, so this is not something we expect most people to do most of the time.

Also! At this point things seem pretty reliable as far as the consistency between the client and the host in terms of this sort of messaging. So if the client is actively wrong, aka has to many units or fleets or something compared to the host, then probably that is not a bug that this would capture (since this is just about what the host tells the client), but rather a bug in the client not properly cleaning up after itself, or something of that nature.

Log Decoded Network Ship Sync Data To Disk

This works exactly the same as the "Log All Decoded Network Sync Data To Disk" above, but this just focuses in on ship syncs themselves. If you see for certain that the bugs are somewhere in the GameEntitySquad serialization, then you could use this one, as it definitely logs less data. But frankly it's going to be hard for you to know when it's only limited to this area, so if you're going to all the trouble of logging, you should probably go ahead and log the whole thing.

Network Debugging Mode

This logs a bunch of stuff to your local ArcenDebugLog.txt file, but outside of testing a new networking framework there's not much purpose for this. This is for testing and demonstrating the reliability of messages, not their contents, and that is critical when implementing a new networking framework (which, as noted, is something a modder or external developer could do if they want support for more than Steam, GOG, and LiteNetLib), but that's when you'd be using this.

Log Network Traffic To Disk

This is a more extreme version of the above, which logs in even more detail. So very useful for implementing new networking frameworks, and for when we were first developing the protocols that AI War 2 uses to talk back and forth in general one layer above that, but it's no longer useful for most situations for folks.

Network Logging Includes Frame Auths

Normally the above logging excludes frame auths, which happen 10x a second, but this turns them back on if they were needed for some reason.

Network Logging Includes Command Batches

These also are so frequent and reliable that they are too much of a clutter, but then aside from sync data this is basically all of the game data that is being passed back and forth. With this off, very little is logged beyond the actual initial connection logic (and now the sync data also).

Network Logging Includes Sync Checks

Same notes as above, essentially. This is voluminous and frequent data, but then again it's most of the data that makes the game actually work.

Log Human-Readable Ship Network Syncs To Disk

Only relevant on multiplayer clients, not the host. Will majorly slow the game down, but dumps a huge amount of networking info to the disk in the NetworkHumanReadablyShipSyncLog.txt file in the PlayerData folder. This gives a message for each ship that was changed or deleted and why that was done, as well as what planet it was on and who owned it. When syncs are doing funky things, this is a way to manually review it and find out why.

Long-term, once the sync code has been in use for a long time, it's unlikely that very actionable data will be found from this. Any discrepancies would again be code errors directly on the client, not in how the data is coming to the client. But for now, this gives us a robust and helpful view of what the client thinks it is supposed to be doing.

Write Network World Serialization Logs

This is something you would use if the client has a corrupt world immediately upon connection to the host. To be useful, both the host and client would need this setting on, and then we'd need the NetworkWorldSerialization.txt from the host, and the corresponding NetworkWorldDeserialization.txt from the client. We would then look at the differences to see what is wrong with that initial world sync.

Show Network Sync Details In Escape Menu

Ah, now this is something useful! It has a very different output on the host and on each client. This data is being collected no matter what (in memory, not on disk), so having it show causes no extra CPU load or slowness. The data is all just aggregated integer counts, so it's very inexpensive for us to collect in the first place. The main reason this field exists at all is to reduce the clutter on the interface.

There are times where we might want to see screenshots of this data from the escape menu on the host and clients, so that we can have a general sense of potentially why performance is bad in some savegame. Basically this is kind of broadly diagnostic, telling us if there are too many syncs happening, but not why. Looking at this data yourself, if you see the PKID or ship sync numbers skyrocketing suddenly, then that might be a good time to make a savegame and send it to us, because something that is breaking lockstep is happening.

You only see these numbers go up AFTER lockstep is broken, however, so it can be a bit too late by the time you think to save (unless it's an ongoing process that is breaking lockstep). But if it's something like "at every wave there's a major amount of desyncs that it has to fix," then you can make that observation yourself and then give us a save from right before a wave unleashes, or something like that.

If you are suspicious that network traffic spikes at certain times, this will also give you hard data to back up your suspicions. You can watch the network traffic totals, and if those spike after certain events, or when you give certain orders or do certain types of battles or something, then that can give you confirmation that your hunch is right, and you can send a savegame to us for analysis.

cmd:dump data tables

Type cmd:dump data tables into the text chat on any client, and it will then freeze the game for 20-30 seconds (please warn other players before doing this) and fill up your respective PlayerData/DataTableExports folders.

You can then copy the folders from different machines all to be on one machine, and use a tool like WinMerge to do a folder-level comparison of the results from each machine. If there are divergences, then this is going to be causing a lot of desyncs, probably.

Most of the time there should not be any differences here, so this is probably not worth doing very often at all, because these files are really huge and also unlikely to be wrong. But if you are playing with a large and complicated mod you are suspicious of... it could be worth checking.

cmd:dump external data

Type cmd:dump external data into the text chat on any client, and it will then freeze the game for probably only 1-2 seconds (please still warn other players before doing this) and fill up your respective PlayerData/ExternalDataExports folders.

Comparing Machine Sync

You can then copy the folders from different machines all to be on one machine, and use a tool like WinMerge to do a folder-level comparison of the results from each machine. There will always be divergences between the host and the clients, and if the clients joined at different times then also between clients.

Divergences are probably not a crisis! However, if things are VERY divergent, or the things that are divergent on the clients are very stale (aka have not been updated in a long time), then this can give you really useful info.

Looking For Client Memory Leaks

If a client has been seeing performance that gets progressively worse, or memory is notably going up in usage, then it's possible that either the main devs or a modder made a mistake in how a collection of data is being synced to clients from the host. The host is not going to see this particular problem.

After you have run the export, which may take substantially longer than usual if a client actually is having trouble, then you can open up these logs in something like Notepad++ and do a search for the text "BIGLIST". There are some cases where you will see perfectly-valid versions of these, particularly BIGLIST-A and BIGLIST-B. However, if you're suspicious of these, you can always show your logs to the developers or to the mod author for review.

What is rarely correct is BIGLIST-C, and BIGLIST-D is almost NEVER correct. BIGLIST-E is not something we can think of any valid reason to have happen. Please report it to the appropriate party (developers or mod author).