Difference between revisions of "Category:AI War 2: All About Multiplayer"

Discord Channel For Multiplayer

• A great place to meet folks for play or to discuss ideas is our discord channel for AI War 2 multiplayer!

Initial Caveats And Related Resources

• Please note that multiplayer is in two different statuses:
  • If you are playing with multiple player factions, this is currently in alpha status. Please see below for details. The bottom line is that there are known issues and missing features.
  • If you are playing with a single shared player faction and any number of players, this is now in beta status. That means there will still be bugs and polish needed, but it is feature-complete.

• If you are testing during alpha or beta, we have a variety of questions for you that we would definitely appreciate you taking a look at.

• If you want to know what the known outstanding issues are, a pretty good summary of that is here. In some cases, particularly for features we might add, that can be rather vague, so more data is on our our mantis bugtracker.

What Does The Alpha Vs Beta Status Mean?

If you are playing with however many players, and you are all sharing a single human player faction, then this is in beta status. That means that that mode of the game is feature-complete, and should be fully functional. It definitely does not mean it is bug-free, but it does mean that we're pretty much just in the mode of fixing bugs for this mode, and adding any other bits of polish that come up. This is a very exciting milestone for us!

On the other hand, if you are playing with more than one human player faction, then that form of multiplayer is still in alpha status. This means that we do want your feedback on it, and you can play it, but there are a number of things with it that we know are actively broken or incomplete.

Known Issues

• Multi-faction multiplayer known issues:
  • It is not yet possible to gift fleets between factions (but you can gift fleet lines).
  • It is not yet possible to gift metal between factions, or set up an ongoing metal flow from one faction to another.
  • In multi-faction games, we have not yet solved the balance issue of players having too many hacking points when you add all those up.
  • Everything else, to our knowledge, is working great from a design standpoint, so these are the issues keeping mult-faction MP from being in beta mode.

We have a number of people who play both single-faction and multi-faction multiplayer regularly, and who are having a great time despite whatever hiccups in the latter in particular. We would definitely love it if you join them! Please report any issues or problems you encounter on our mantis bugtracker, ideally with a savegame and/or reproduction steps, as well as any relevant error logs.

It's really critical to remember that there are a ton of edge cases where bugs can pop up, so even if there are dozens of people playing different multiplayer games during the alpha, you and your group may be the only ones to hit a specific issue. Please actually make reports to us, rather than assuming it will get fixed because someone else reported it. If none of the groups happen to report it, despite knowing about it, then we'll eventually proceed to beta with those bugs still in place because we'll have no idea they exist. The game is just too large, and there are too many things that are situationally-dependent, for us to find everything on our own without user reports.

Also:

• Below you will see a variety of questions aimed at testers.
  • Please see if there are any that you have feelings on, and feel free to offer suggestions ideally on mantis bugtracker, either in new topics or on existing topics that are open that someone else raised.
• Further below, you will see a number of "todo before beta" issues.
  • If an issue that you are running into is mentioned there, then it might not warrant a report, but we'd rather have extra reports than not enough; use your own judgement, and we certainly won't fuss at you for giving us extra data.

Questions For Multiplayer Alpha/Beta Testers

• For every possible hack that you can do in the game and its various expansions, does it feel fair what results from the hack?
  • If not, what doesn't feel fair about it and how might we address it? For example, in many cases, just the hacking player expended hacking points (HaP), so just them getting a benefit sort of makes sense. Given that, one potential would be to make those structures able to be hacked once per player, and they each get the benefit if they choose to spend HaP there. That might be the way to retain the most freedom for players, especially since they all get their own "copy" of the HaP (so in a two player game, your team has 2x the total HaP you would have in a solo game).
  • There are, of course, other consequences to various hacks, ranging from exos to other angry AI or faction responses, to even AIP going up. Is there anything we're missing with specific ones that seem off to you?

• For every capturable, such as new fleets/golems/arks/battlestations/citadels/etc in particular, is that feeling like it is fairly distributed and not resulting in too much AIP?
  • The AIP of a multiplayer game should probably be expected to run a bit higher, but you have double or more in terms of metal and energy income, as well as double science and hacking points, as well as a lot of extra firepower in terms of starting fleets, so to some extent that's actually a good thing.
  • But there's a difference from being balanced and feeling balanced. If it doesn't feel right, even if it is right, then we still need to at least look at it.

• Are you getting any "random" errors popping up, particularly on clients?
  • These usually will be nullref exceptions or out of index exceptions in what may seem like completely random parts of the game code.
  • There is a whole new set of cross-threading conditions that exist only on the clients in a multiplayer game that can cause these, and they are extremely dependent on millisecond-level timings (in some cases, literally nanosecond-level timings). So to say that not everyone will hit all of these is an understatement. If you run into any of these issues, even as a one-off thing that doesn't occur again even for you, please do let us know with a log of the error.

• What features are missing for you sharing things between player factions?
  • You want to give X to your ally, or have them give you X, and you can't do it -- what is X, and is there anything else we know about that?

• Particular areas of multiplayer that need testing:
  • Every faction needs to be played for a while, to see if they give any new exceptions in version 2.642 (November 30th) or later.
  • Nomad planets, making sure the linkages work the same on both sides.

Reporting Bugs Or Suggestions

• Any bugs or requests should go to our mantis bugtracker
  • If you need to submit log files, those can generally be found under your PlayerData folder in the folder your game is installed in. The most relevant one is called ArcenDebugLog.txt. You can send us the whole thing, or just strip out relevant parts.
  • In rare cases, mainly if your entire game crashes (that almost never happens), we will need your unity player log. That gets overwritten the next time you run the game after a crash, unlike the other log. These can be found here:
    • Windows: C:\Users\username\AppData\LocalLow\Arcen Games, LLC\AIWar2\Player.log
    • macOS: ~/Library/Logs/Arcen Games, LLC/AIWar2/Player.log
    • Linux: ~/.config/unity3d/Arcen Games, LLC/AIWar2/Player.log

• If you have been generating logging information that is networking-specific, then that will also likely be in the PlayerData folder, in some cases in subfolders.
  • Generally with those sorts of logs, we need the equivalent log from the host and at least one client (whichever client had the problem).
  • But, also, we really don't expect you to be generating these sorts of logs for us typically, so these really are going above and beyond if you are sending those in.

• In other cases, if you suspect something is wrong, then opening the escape menu during the game and taking a screenshot of the stats there to send to us is welcome.

Reporting Lag Or Connection Problems, Specifically

• For this in particular, opening the escape menu during the game and taking a screenshot of the stats there to send to us is welcome.
  • This is a case where turning on the extra info in the escape menu, Show Network Sync Details In Escape Menu (via your personal settings, networking section), is extra helpful.

• Relating to multiplayer performance problem reports:
  • Knowing how long (in actual realtime, not gametime) you were playing in a given session to go along with the amount of data transmitted and performance you were seeing is useful.
  • Knowing rough distances between the players (intercontinental, same country, same metro area, or same building is also useful if you're having performance-related problems.
  • Similarly, knowing the general strength of the internet connections in question is useful if it's a performance problem, as well as which networking framework (Steam, GOG, or LiteNetLib) was used is also helpful. If one or more of you is running this over a VPN, that's also important for us to know.

Multiplayer Remaining Todo List

If something disappeared from here, you might want to check if it has been recently fixed.

Before Beta

• Multi-faction multiplayer known issues:
  • It is not yet possible to gift metal between factions, or set up an ongoing metal flow from one faction to another.
  • In multi-faction games, we have not yet solved the balance issue of players having too many hacking points when you add all those up.
  • Everything else, to our knowledge, is working great from a design standpoint, so these are the issues keeping mult-faction MP from being in beta mode.

• "Reset to defaults" on the host causes an error on the client side, and player account destruction.

• Look at all the hacks to account for multiple player factions.

Before Full Launch

• Allow clients to save the game by having a gamecommand created (as always), but having the host then create the save and send it as a package deal to the client, who saves it.

• Get it so that clients are kept up to date on the status of other clients, so that they can make a call similar to how OnServer_GetIsConnected() works on the host.
  • This is only relevant for games with 3+ players, but is a nice convenience. If the host is waiting excessively long a specific client, then it might be nice to send that information, but might be hard to get there in time for most situations.

Lower Priority

• Sync improvements:
  • Do we need to sync journal entries and world history? If so, given them their own time slice, but it seems unlikely.

• Spurious "Fixed attempt to read more faction data than we had factions on the client." on disconnect.

• client side disconnection (need help replicating)

• add a debug menu button to force an immediate full sync.

What Has Changed Lately?

Please see the release notes for details!

Multiplayer, At A Glance

This game is a client-server model, with one host, and any number of clients who are connected to that host. There are several networking frameworks available to use, but the host and clients must all be using the same one.

About The Host

• If the host severs the connection, then the game is halted for everyone else. A new host is not automatically chosen.
• Only the host can save the game, at present. Even with our improvements to multiplayer sync, clients are too likely to have slightly-garbled versions of the game.
• The host must start up the game first, either in the lobby, or by loading an existing save or quick start. The clients are then able to connect, based on whatever the rules are of the networking framework everyone is using.
• The host should have the most powerful CPU, as it runs a lot of background AI threads that are too time-consuming and long-running to run in the main simulation, and which do not run on clients.
  • This is similar to AI War Classic, but many threads instead of just one.
• The host should also have the most robust network, as they have to send approximately 2n the data that a client does, where n is the number of clients.
  • So in a two player game, the host is probably sending and receiving roughly 4 times as much data as the sole client.
  • In a three player game, the host is sending and receiving roughly 6 times as much data as either of the two clients.
  • A spectator client adds roughly the same amount of bandwidth requirements as a regular player.

About Clients

• Each client connects to a host, and passes the name of their current Player Profile (as configured in AI War 2 itself).
• Clients (and for that matter the host) are linked back to their prior faction based on the player profile's name, so changing that midway through a campaign with friends is not a great idea.
• Clients are in general in a state of slightly-wrong-sync with the host, though the game does its best to keep everything consistent.
  • There's a rolling sync process that sweeps through and corrects inconsistencies within 3-4 seconds on each client, and so in cases of high desync clients may see things jump to correct themselves to match the host.
  • This is not something the host will see, as the host is treated as the "central correct record" for everything.
• It's worth noting that if a client is lagging behind, it will slow the host and other clients to wait for them. BUT if a client is disconnected entirely, then the game will continue without them (though any other player can pause it; they can then save, stop, or continue).
  • It's also worth noting that the host and the other clients do not have to disconnect or restart anything in order for the missing client to connect back. When the missing player returns, there is usually just a 1-5 second pause (depending on the size of the savegame) while the client gets caught back up to speed. But you don't even have to explicitly pause the game.
  • If sync is ever atrociously off on one client, that client can always use this to their advantage to disconnect and reconnect and have perfect sync again (although sync being that bad off is not supposed to ever happen).

Bandwidth/CPU Requirements And Player Counts

• There is a built-in limit of 254 clients connecting to a single host, but for practical purposes there's really not going to be remotely that many players supported.
• How many clients are actually supported is a matter of bandwidth between them and the host, as well as CPU processing power for the resulting game.
  • If there are 40 clients all sharing one faction, then this is far less of a load than 40 clients all with their own starting planet and unique faction. It's several orders of magnitude less CPU processing, to the point where the former might be barely possible on a nice LAN, but the latter is not possible with current hardware.

• As the years pass and multi-core processors become stronger and also add more cores, AI War 2 will naturally spread out to use those capabilities with no code changes required. It would probably take 16 or so cores before you stop seeing any tangible benefit.

• At the moment, with one client and one host in a fairly average starting galaxy with one AI and two player factions, the bandwidth requirements are less than 100KB per second on the host (in many cases closer to 10KB per second). The client ranges from half to a quarter of that amount, depending on what stage of the game we're talking about.
  • With just two players, far more important than the speed of the connection is the reliability of it. Lots of dropped packets and resends will cause delays even if the actual connection is plenty robust.
  • With two clients and one host (aka 3 players), the client requirements stay the same, but the host bandwidth requirements go up to something in the 200KB range.
  • It's actually expected that, for most practical cases with more than 10 total players, the CPU load of preparing all the extra network messages will probably cause more slowdown than the actual sending of the messages. This is something that we have worked around as much as we can, but also not something that more CPU cores will help given the linear nature of the data we are sending.

• The bottom line is that, in most situations, you should have a fine time as long as you have a 10 or fewer players, but we'd be interested in hearing what sort of experiences you have.
  • For practical purposes, the game itself will probably bog down with more than 4-6 human factions on current hardware.

Which Networking Framework To Use?

The game supports three networking frameworks at present, although more could be added in by ourselves or modders.

What Is A Network Framework?

A networking framework is responsible for transmitting data between players, and for getting players connected in the first place, but it is completely unaware of what data is being transmitted.

Conversely, the game itself is responsible for preparing and handing off all the actual messages it wants to orchestrate between players, and it has no idea where those messages go (at a low technical level) or how they get from the client to different hosts.

Some networking frameworks are more efficient than others in terms of how they use CPU and RAM on the client and host, and also in terms of how they use your network card and available bandwidth. They also can differ in what sort of networks they use, and so again can have a big impact from that area alone.

Steam

In this networking framework, you don't have to worry about IP addresses at all. As long as all of the clients are Steam friends with the host, you're good to go. Firewalls and network configurations don't matter much.

The host starts a game while logged into Steam, but they do not have to be in Online status on the Steam Friends section. Even someone who appears Offline in Steam Friends can be a host or client.

The clients will see a list of their Steam friends, sorted first by those who are currently online and playing AI War 2, and then by those who are online, and then those who (at least seem to be) offline. You can click connect next to any of the friends and try to connect to their game, even if they appear to be offline. It won't hurt their machine or yours, and if they have a game open even while Steam Friends is set to offline, you can connect to them. Please do only connect when invited, though!

It's worth noting that all of the network traffic is routed through Valves relay servers at all times. This will typically give you better performance than the general Internet backbone, according to Valve. You CAN also use this while on the same LAN as other players, but your performance and ping will be as if you were on different networks in the same city. Aka, far worse than if you were on the LAN together, but still plenty great for most purposes.

GOG Galaxy

In this networking framework, you also don't have to worry about IP addresses. As long as all of the clients are GOG friends with the host, you're good to go. Here again, firewalls and network configurations don't matter much.

The clients will see a list of their GOG Galaxy friends, sorted first by those who are currently online and have an active AI War 2 multiplayer session open. You will then see the others who are offline, or in game, etc. The various statuses don't really matter except for the ones that show up as purple, who have a game actively open at the moment. You won't see a connect button except for players in that status, so be sure to refresh the list of friends frequently if one of them just opened a game and you want to connect.

We're not sure exactly how the network traffic is routed. There are relay servers involved if a direct connection can't be made, but we're not sure if a direct connection is used in cases where it can be. Performance seems great in general, but definitely is relay-based if you are on the same LAN.

Presently there is no way to directly make a connection between a GOG Galaxy version of the game and a Steam version of the game using the GOG Galaxy framework. That's a case where you'd need to use LiteNetLib (see below).

LiteNetLib

If you're on the same LAN as the other players, then this is a no-brainer choice. If you're on different networks connected to the Internet, then it's your choice; to use this in that situation would probably require port forwarding on your routers, and that's something you can easily look up for multiplayer games in general if you want to go that route.

The game's network tab of the settings menu shows you your public internet IP address, as well as your local IPv4 and IPV6 addresses. You can also configure what port the game runs on (it is using UDP). Only the host would need to set up port forwarding for that port if you are playing across the Internet.

The host would need to give their public IP address (for internet play) or their local IP address (for LAN play) to all of the intended clients, and they then type that into the connection textbox and connect to the host after the host has started or loaded a savegame/quickstart or the lobby.

If you are on a VPN network together, then it works much like a LAN does, but whether or not your IP address for the VPN's virtual network adapter will show up in the game interface is dependent on the VPN service itself. If not, then most VPN services prominently show both your virtual IP address as well as the virtual IP addresses of any other folks in the same session with you.

VPN services usually try to forge a direct connection if they can (which is almost as fast as the port forwarding route, but not quite because of encryption), but fall back on a relay server network of their own if that is not possible. The speed of these relay networks varies hugely depending on the VPN service. Some of them are extremely fast and suitable for gaming, others are direly bad.

So What Should I Use?

• Are you playing the Steam version across the Internet? Use the Steam network framework. It's simple and really fast.
• Are you playing the GOG Galaxy version across the Internet? Use the GOG Galaxy network framework. It's also simple and quite fast.
• Are you playing on a LAN? Consider using LiteNetLib, as it's the most direct and will be the fastest by far. If you have trouble, you can fall back to Steam or GOG.
• Do you feel like avoiding Steam or GOG Galaxy, and you're an expert with your router? Use LiteNetLib with port forwarding for maximum speed.
• Do you feel like avoiding Steam or GOG Galaxy, and you're not a networking wizard? Use LiteNetLib with a VPN service like Hamachi or similar. There are many available.

What If I Only Have One Steam Or GOG Galaxy Account?

If you are planning to use Steam networking, or GOG Galaxy networking, you need a unique account on the respective service for each machine that is going to use it. They each need to own the base game, although the DLC can easily be pirated by just copying it from one machine to the other. (Please don't, but literally piracy is as easy as copying thing, since we don't believe in DRM).

The only reason that you have to have unique accounts is because the connections in these cases are account-based. And the reason that you have to own those games is that those services check for that when you use them -- the services are paying for the bandwidth usage of games that are run on them, and they'd rather not have games not owned on their service run multiplayer through their relays, which seems reasonable.

Anyhow, so for testing purposes Arcen has a few Steam and GOG accounts to run between multiple computers, as one example. But if we were to play with family members, they would just each need a Steam key or GOG key for their own accounts.

The LiteNetLib networking framework has absolutely zero restrictions, so you can put the game on a thumb drive and copy it to 4000 people and they can all play each other in multiplayer without any sort of checks or restrictions. Again, please don't pirate our stuff, but it's stupidly easy to because we're not trying to stop you.

How Do Expansions (DLC) and Mods Affect Multiplayer?

In general, players must have the same expansions and mods turned on when they are playing multiplayer together. If a client tries to connect to a host with not-enough or too-many things installed, it will inform them of the differences and what changes either they or the host should make to bring them in line.

It is our goal that people never be forced to buy DLC/Expansion content in order to be able to play with friends who have that content. However, while you're playing with those friends, they will have to disable the DLC/Expansion content.

You can disabled expansions/DLC or mods, as needed on the client or the host under the Game tab of the settings menu. That then does require an application restart, as a lot of the data that is included in those packages is very tightly integrated into the main game and can alter things in unexpected ways without the restart.

Compatibility?

All official content from Arcen, ranging from the base game to the Expansions/DLC is compatible with multiplayer. If there are any problems with any of it, that's a bug that we will look into and get fixed for you.

Mods, on the other hand, are not something that we can control in this way, and are something of a wild card. This caveat also includes mods that we distribute with the game, as we are making them more accessible to you, but not taking responsibility for them being bug-free.

That said, mods that are just xml or just visual components are pretty much guaranteed to be compatible in multiplayer, as they are using core game engine features that we designed. So, for instance, there's no risk of the Spire Railgun Shop messing up something in multiplayer.

Mods that include code of any sort tend to be the ones at risk of lack of compatibility, and the more complicated the code is, the more likely problems are. Civilian Industries, which we also include with the game, is a great example of this.

During the course of the alpha of multiplayer, we made a number of improvements that make most sync considerations for mods kind of irrelevant:

• If a mod does not properly coordinate actions between the client and hosts, the game will notice that within a few seconds and fix it.
• If there is data from the mod that is out of sync with the clients, but it doesn't affect actions of ships or units, then there's no problem in the first place.
• If there is data from the mod that would affect tooltips on the clients, that's no longer a worry because as soon as clients hover over a unit to get a tooltip it now gets an updated sync copy from the server. This eliminates almost all potential problems, and uses very little data.

So, the sync-correction process will keep almost everything with a mod working just fine. But it's also possible for a mod to introduce a memory leak (client side only, or else it's a problem with the mod not limited to just multiplayer) by not handling the sync code properly, or to introduce exceptions during sync itself, etc. It's also possible for a mod to fall prey to the same sort of "random threading exceptions" that other parts of the game have to guard against, through no fault of the mod author. These things are usually pretty quick fixes once they are identified by players.

Distribution Over The Network In The Future?

We don't have any plans to make it so that mods or DLC can be downloaded from the host to clients, or vice-versa, in the future.

When it comes to DLC, there are a couple of things that stick out there. For one, if the visuals were to be downloaded, this usually means sending something like 1GB of data from the host to the clients (or vice-versa) to make this work. The format of the visuals and so on is platform-specific (OSX, linux, or windows), so if you are playing between different OSes, the host or client might not even have compatible files for one another to send.

Secondly, aside from the technical hurdles on DLC, this gets generous to the point of shooting ourselves in the foot when it comes to DLC. It's extremely easy to pirate the DLC from one client to another (just copy the files) if you really want to do that, but if you're a group and you'd like to play with the DLC, we hope you'll consider actually supporting us instead. If you're playing with folks newer to the game, and you understandably don't want to have them go through extra investment into DLC before you're sure you enjoy playing it together, we've made it really easy for you to turn off your DLC while you are playing with friends so that you can do that without anyone having to spend any money.

Lastly, and this applies to both DLC and mods, but you'll notice that enabling or disabling any of them requires a restart of the game, anyway. This is because these mods and DLC can run extremely deep when they are making changes to the game, including changing or adding content to things that were in the base game. There's not a way for us to be sure that you are back on "baseline" without that restart and a fresh l oad of the data files. So even if we were to set up the ability to send a mod over the wire from the host to the client, or vice-versa, applying the content from that mod would not be a simple matter without rebooting and breaking the connection anyhow.

How Can Players Participate In A Multiplayer Session?

There are several ways that players can join in. In order to discuss this, we need to make three pieces of terminology clear:

• A player is a specific person, at their computer, who is going to be either the host or one of the clients.
• A player faction is kind of like an empire. In single player games, there is always one empire for the one player. But that's not always the case in multiplayer.
• A player profile is the name and preferred-color you give yourself on your machine. You can have multiple of these, with different names and colors, but only one can be active at a given time.

Example time!

If we have BobX as a player profile on the host machine, and he starts a new custom game, he will immediately be put in charge of player faction 1, which will also be called BobX unless he changes it (via the empire name field on the player faction). The faction color will be set to match his preferred player profile color, again unless he changes that on the settings for the player faction he's now in charge of. So far this is just like a single-player game.

Now if we have SallyB as a player profile on a client machine, and she connects into BobX, then a second player faction will be generated, this one for her. It will get the name SallyB, and take on her preferred color, but it does not have to stay that way. And that's where we start getting into a slight bit of complexity.

Everyone With Their Own Faction

This is the simplest and most obvious way to play, and what happens by default. If you have three clients join the host, then you wind up with four total player factions. Each one has a name that matches faction to player, and that's that.

Although, still, there are some questions worth addressing:

• What if several of the players' preferred colors are too similar or the same?
  • This won't affect anything, gameplay-wise. The color is just a color, and won't merge factions or anything like that.
  • From a practical standpoint, it will be confusing, however. So changing the colors up a bit via either the factions tab in the lobby, or the factions screen from the escape menu during gameplay, might be a good idea. You can always change the colors of any of the factions at any time, so it's never to late to shift things up if you're finding it hard to tell someone apart.

• What if players choose the same name?
  • This is not allowed. Out in the world, globally, people can reuse names as much as they want. But in a specific multiplayer campaign, there can only be one "BobX" and one "SallyB"
  • Fun fact, if BobX and SallyB save and quit the game, then rename their player profiles (or switch player to alternate profiles on their machine that have those other names), then they can come back into the game as each other's factions.
    • Put another way: If BobX is now SallyB, and opens up the savegame to host on his computer again, he will now be in control of SallyB's faction. If she tries to reconnect as SallyB, she'll be told that someone else is already playing with that name. If she tries to connect as BobX to that save she and he were previously playing, then now she has his faction, but is still the client.
    • Why you would want to do this is not entirely clear.
  • More realistically, if there is a second BobX trying to connect to a game with BobX and SallyB, then that second Bob will have to come up with a new name. They can either rename their existing player profile, or just create a new one and use that when they are playing with this specific group. Maybe this new person continues to go by the profile name BobX most of the time, but in this campaign they switch over to a profile name of BobZ so that they can be uniquely identified.

• Why not use something like people's IP addresses or Steam or GOG name or something else like that to solve the above conflict?
  • First of all, because the above problem is really kind of an edge case. Most people choose fairly unique names, especially among groups of friends.
  • But secondly, we assume that people may have their IP addresses change, or may switch around what network framework they use, or any number of other factors.
  • And finally, we actually view it as a really nice feature that anybody can impersonate someone else in a savegame, to see what things look like from their point of view.
    • This is useful for us as developers, but it's also neat for anyone who wants to share savegames on the Internet for others to play.
    • If there's a really challenging savegame with three players where people post it going "can anyone beat this??", then they can give the names that people connect as (Be BobX if you want the big empire up nort, but that one is getting hammered by waves. Be SallyB if you want all those upgraded golems in the east. Be BobZ if you want that almost-dead last planet in the center. Can you win where we lost?) That sort of thing is neat.

At Least One Spectator

A spectator is a player who controls no factions, and thus can do nothing except watch what everyone else sees.

Fun fact, it's possible to play this game in multiplayer, but still have it be single-player in all respects that matter. Have the host be an actual player, and the sole client be a spectator, and now you have a second person who is just watching you play but can't do anything. A twitch stream might be a little more straightforward, but a spectator can look at different things from what you are seeing.

The one thing that spectators can do is chat. They can click around to all the planets that any other human players can view, but they don't see anything extra that the others would not. So you can't use a spectator to cheat, but unlike a twitch stream a spectator can look in places you are not seeing and can alert you to threats. They can thumb through intel, journal entries, watch border planets you don't have time to glance at, and so on.

To become a spectator, or make someone else in your lobby become a spectator, simply remove the faction that is tied to that person. It won't assign them to another faction unless one is added back manually for them.

Multiple Players On One Faction

It's possible to have multiple players share control of a single faction. There's no explicit limit on how many players can be sharing that control, and not all of the players have to be present for gameplay to proceed.

By default if BobX and SallyB share control of a faction, it will name it BobX and SallyB. With three players, it becomes BobX, SallyB, and BobZ. Beyond that, it adds "and others." But really, if you have more than one player controlling a faction, it might be nice to set the empire name to be something thematic like "Empire of the Rising Bobs."

In chat, it will identify players by their profile name, but the color of their faction. That way you can tell if it's BobX or SallyB talking, despite the fact that they are sharing a faction at the moment.

Many, Many Uses For This!

There are NO RESTRICTIONS on what two players can do if they share a faction together. Essentially they both have equal control of the resources and units, and their own independent copy of the interface. If a friend who is new to the game wants to learn, this is a great way to do it. You are essentially playing a "single player game" in terms of balance, but with two brains and four hands and four eyes controlling the shared empire. If you want to set up an arrangement where certain fleets are used by certain players, that needs to just be an agreement between you.

Various folks have asked us about things like "Champions" or "Helpers" from the first AI War, and this ability to share factions really handles the bulk of the use cases of those old roles, but with more flexibility. Essentially if you want to have one friend share in your empire but just drive a Botnet Golem around causing chaos, then that can be their sole responsibility.

Any Combination Is Fine!

You can mix and match this however you like. Two players can share one faction while a third has their own faction. Yet another player can be a spectator. It doesn't matter which one of them is the host (the host can actually remove themselves from the original faction once a second player is in).

Empty (From The Start) Player Factions

It's possible to add an extra player faction that... isn't controlled by anyone right now. It will just kind of hang out on its planet and gather resources, and not have any intelligence behind it. Its ships will fire on things that come into range, but they won't be in pursuit mode by default.

The only real reason to do this seems to be if you have a two-person group today, but next week it will be three people and you don't mind defending that third planet until then.

Alpha Note: it's possible that late-assigning a player to this won't work properly yet, in the alpha.

Abandoned (Forever Or For A While) Player Factions

Last week, perhaps you had three players who wanted to play, each with their own factions. This week, you want to continue that campaign, but you only have two of the players present. You can load up that savegame and continue with the third faction just sitting there doing whatever its last orders were to do, and that won't be a problem.

If next week the third player is back, you can either go back to where you were when they last played, or you can jump them into the future where they are probably behind and disorganized, but hopefully still living.

What Multiplayer-Specific Features Exist?

Generally speaking, if you can do something in single-player, then you can do the same thing in multiplayer. Overall our goal is not a radically different experience when playing multiplayer, but rather "it's AI War 2, but with more people, and fun." That said, to hit the "and fun" bit, certain things had to be taken into account specific for multiplayer.

Alpha Note: We're going to need new things in here based on how answers to our various 'questions to MP alpha testers' come out.

Copies Of Science And Hacking Points

In the original AI War, every player individually had to gather their own knowledge and hacking points. That was actually really annoying and tedious.

In AI War 2, if any player has a unit gathering science or hacking points, then all players get a copy of those points. So there's never a case of us having to coordinate over gathering science from a distant temporary planet we are holding. If you gather it, I get a copy, and if I go there later, there's nothing left for me to gather since I already have it.

Shared Scouting And Intel

Our scouting levels, and our intel sidebar, are all shared. This is the same as the original AI War.

One of the nice things about the intel sidebar in particular, however, is that any player can right-click the items to change their priority from normal to high or low, and then all players see that. So if there are things that everyone is wanting to coordinate, people can adjust those priorities together until they agree upon them. Or one or more players can be designated to evaluate intel and recommend what to pursue and what to ignore for everyone.

See Where Other Players Are On The Galaxy Map

It's really handy to be able to say "see this planet up here where I am?" This is something you can do by simply clicking the planet, and other players will then see it as highlighted. If they mouseover the planet, they can see your player name (since with many players, this can be confusing as to who is where). This is meant as a visual aid to allow players to see where each other are and thus coordinate a bit better.

What Do I Need To Know About Balance In Multiplayer?

Alpha note: this is actively under discussion, and one reason for doing the alpha.

Overall it's worth noting that if you add in multiple players to a single player faction, that's the same as having one player faction in single-player, and it always will be. The game is definitely not meant to react to having more players, ever, but only to having more player factions.

When it comes to having more player factions, right now the big change in balance compared to single player is that in order to support those factions, more planets will need to be taken, and more fleets captured. This is going to raise the AI Progress (AIP) substantially more than in single-player. But since the AI is not inherently scaling up in any other way in response to multiple player factions, this might actually work out just great. At the moment we don't have enough data to know either way.

In the questions to alpha testers section, we have posed some questions about what people would like to see regarding hacks in particular. But bear in mind that in a two-player-faction game, you get double science and double hacking points for free, plus double starting fleets and starting planets and so on.

The original AI War was really rigid in its number of factions (always exactly two AIs), but the sequel is not. To some extent it may be best if we don't do too much scaling to multiple player factions, and instead let them tune to taste. But if we did that, then essentially every quick start would be extremely easy for four players compared to one, as just one example problem. This again is one of those things that we want more data on from people actually playing: how it feels to them, versus what we can guess at on paper.

Technical Explanation Of The Multiplayer Model

We've written about this extensively over time, and done a few videos, but it's nice to collect this into one place in a brief format.

AI War Classic: The Lockstep Model

The original AI War used a lockstep design along the lines of what you can see in that link. That is indeed what multiplayer RTS games have been doing essentially from the dawn of them existing. We used the same networking model in Tidalis and in Skyward Collapse.

We WERE planning on using the same model in AI War 2, but ultimately we found ourselves getting caught out by the extremely multi-threaded nature of this game. We just can't guarantee absolute determinism between clients and the host, or clients and each other, and so in 2018 we decided to stop trying for absolute rigor in that regard.

We do still mostly use the lockstep approach, and we do our best to keep things deterministic as much as possible, but divergences are expected -- unlike with the original AI War. When a desync was detected in the first AI War, it would stop the game, ask you to save and reconnect, and suggest you give a report to the developers. We would then spend sometimes hours or days chasing down the problem, until it could be fixed and run well again.

(It's worth noting that that level of rigor was also something we realized we could never enforce in mods for AI War 2, and that we'd never be able to tell if a desync was a mod or the core game, so we were really going to be messed up if a desync was a crisis like it was in the first game. There were just a whole host of reasons why a pure lockstep model could no longer work for us.)

(As one other aside, we also really needed to use some floating point math for accuracy in certain places, as our fixed-integer math code was showing its age. We needed this for speed and accuracy with certain kinds of calculations in particular. Floating point numbers are a definite source of desync, which is why AI War Classic doesn't use any of them. Here's a fun rabbit hole, if you want to know more.)

Entity Prediction And Extrapolation?

In our titles A Valley Without Wind 1 and 2, we took our existing lockstep code and changed it into something more appropriate for an action game. We added in our own form of entity prediction (on the client and server side), and error correction interpolation. Since it was a co-op game (not competitive), and cheating wasn't a concern, there was a lot that we were able to do to make that go more smoothly. However, this sort of approach really only works when there are small numbers of entities, a few dozen rather than tens of thousands, and only so many states they can have on them (a single ship in AI War 2 can have several dozen states in its data in hundreds of thousands of possible combinations).

So for the most part, the sort of entity interpolation and input prediction that most action games use (that link is an excellent article on how Valve's Source Engine does it) is a complete no-go for us here.

As one aside, however, back when we were first designing AI War 2 at all, we did use concepts from this in order to handle the entire "visual layer" of AI War 2, and this has had many benefits for us in both single-player and multiplayer.

Essentially, we're running our extremely demanding game simulation at a mere 10fps, no matter what, which is very similar to how things are done in most physics simulations in most major games (AAA and indie) regardless of if they are singleplayer of multi-player. Those other games wind up interpolating between physics steps at a higher framerate, and we do the same thing with our visual display and camera and player input reacting at a much higher framerate than our simulation itself.

Essentially we use that same sort of time-slicing that both Valve and Unity discuss (and PhysX does the same thing), but we do it in a slightly novel way to handle how things are drawn. There are probably other RTs games that do the same thing, but mainly it's something that an MMO would do, since a MMO is another rare case of a game showing an individual player just a small part of a larger simulated world.

But we digress. At any rate, this sort of off-main-thread simulation is both exceedingly common but also unusual in how we implemented it, and it buys us a lot of processing power and ability to multithread that lets us also make small corrections "just happen" when there are sync errors. So that's what we'll get into next.

Sync Correction Code

In AI War 2, we're simulating tens or hundreds of thousands of agents, and a lot of other data at a faction or world or arbitrary-modder-data level. So in a time-sliced fashion, we have the host simply go over a lot of the data and forcibly sync that to clients. In other cases, the host sends a few bits of data to clients for them to compare to their own entities and then make requests for full entity resyncs based on any divergences.

This makes it so that our simulation can react to the simulation being out of sync and self-heal, and in cases of fairly minor discrepancies (the sort you might see from a floating point error) it will actually do so in such a way that our existing "visual later interpolation code" will automatically and smoothly handle the interpolation so that you don't even notice it. In other cases, the sync is more "catastophic" in nature, where ship number 345466 is a nanocaust ship on plant 5 on the host, but a BobX player ship on planet 17 on the client. In those cases, it winds up finding that out within a few seconds (on average within 4), and deletes both wrong ships on the client and recreates them in the image that the host has.

If this was a competitive game, or a game where we were worried about malicious hosts cheating, then this would be big trouble. But as this is a co-op game typically played with friends, we're able to take the host in good faith and deal with sync data in a way that assumes lockstep works most of the time (and it does), but non-destructively fixes divergences when those arise (and they do).

In essence, we're combining several different networking models for maximum performance in as large a simulation as we can possible muster.

We also have made it so that clients can request updated information for certain units, and they now do that specifically for anything you're getting a tooltip about. This makes it so that any ancillary data about a unit is automatically synced when you go to look at its details.

Various Logging Tools For Multiplayer

There are several different methods of enabling several different kinds of logging. All of them are aimed at different purposes, and most of them slow the execution of the game down. So you'll want to enable with care. Let's talk about each of them, and when you would use them. You're going to get basically no information of use from most of them, so it's a matter of when you might use them in order to send logs to us.

Log All Decoded Network Sync Data To Disk

This is really only useful if a client and the host both have this on, ideally from the start of the client connecting to the host. If you are seeing repeated errors on the client or the host that pop up and say something about "serialization" or "sync" or something to that effect, then there may be a problem with the sync data serializing properly, and this would be a way to document what is actually happening between computers.

1. First of all, you need to have a savegame that your host is able to load, and that once the client connects to it, it reliably causes the problem within 30 seconds of realtime at an absolute minimum. If it requires unpausing and clicking around in some specific fashion, that's fine, but make sure you document what those steps are and if the client or host has to do it.

2. Now that you have this save in hand, look in your PlayerData folder and see if there is a NetworkSyncMessages subfolder. On both the client and the host, if there is anything in that folder, delete it.

3. Now make sure that this "Log All Decoded Network Sync Data To Disk" setting is on on the host AND the client. It's under the Network tab of your personal settings window.

4. Have the host load the savegame that demonstrates your problem. You will note that nothing gets logged to disk on the host at this point, because there is no client to share sync data with.

5. Have the client connect to the host, and as soon as that connection is complete it will start logging a tremendous amount of data to your disk. It's something along the lines of 200kb every second, which is one reason you really can't let this run for too long or you'll have hundreds of MB of data.

6. Quickly unpause and do whatever it is that causes the problem. Once the problem has happened, have the client and the host both kill the game. If you're running in windowed mode, then hitting the X in the top corner of the window (or red dot on OSX) will do it. On windows, you can use Alt+F4, or on OSX you can use Command+Q.

7. On each of the two machines, zip up the entire contents of the NetworkSyncMessages subfolder, and be sure to label which one was from the host machine and which one is from the client machine. We will later unzip both into separate folders, and compare them with a tool like WinMerge, which will show us all of the differences in what the host thinks it sent and what the client thinks it received (and vice-versa in some cases). Somewhere there is some form of typo or inconsistency, and this will let us find and fix that.

In order for us to verify the fix, we will also need your savegame from the host that you used to reproduce this, and it would probably be helpful to have the ArcenDebugLog.txt file from both the client and the host. But it's really not required.

As far as logging goes, this is pretty intensive, so this is not something we expect most people to do most of the time.

Also! At this point things seem pretty reliable as far as the consistency between the client and the host in terms of this sort of messaging. So if the client is actively wrong, aka has to many units or fleets or something compared to the host, then probably that is not a bug that this would capture (since this is just about what the host tells the client), but rather a bug in the client not properly cleaning up after itself, or something of that nature.

Log Decoded Network Ship Sync Data To Disk

This works exactly the same as the "Log All Decoded Network Sync Data To Disk" above, but this just focuses in on ship syncs themselves. If you see for certain that the bugs are somewhere in the GameEntitySquad serialization, then you could use this one, as it definitely logs less data. But frankly it's going to be hard for you to know when it's only limited to this area, so if you're going to all the trouble of logging, you should probably go ahead and log the whole thing.

Network Debugging Mode

This logs a bunch of stuff to your local ArcenDebugLog.txt file, but outside of testing a new networking framework there's not much purpose for this. This is for testing and demonstrating the reliability of messages, not their contents, and that is critical when implementing a new networking framework (which, as noted, is something a modder or external developer could do if they want support for more than Steam, GOG, and LiteNetLib), but that's when you'd be using this.

Log Network Traffic To Disk

This is a more extreme version of the above, which logs in even more detail. So very useful for implementing new networking frameworks, and for when we were first developing the protocols that AI War 2 uses to talk back and forth in general one layer above that, but it's no longer useful for most situations for folks.

Network Logging Includes Frame Auths

Normally the above logging excludes frame auths, which happen 10x a second, but this turns them back on if they were needed for some reason.

Network Logging Includes Command Batches

These also are so frequent and reliable that they are too much of a clutter, but then aside from sync data this is basically all of the game data that is being passed back and forth. With this off, very little is logged beyond the actual initial connection logic (and now the sync data also).

Network Logging Includes Sync Checks

Same notes as above, essentially. This is voluminous and frequent data, but then again it's most of the data that makes the game actually work.

Log Human-Readable Ship Network Syncs To Disk

Only relevant on multiplayer clients, not the host. Will majorly slow the game down, but dumps a huge amount of networking info to the disk in the NetworkHumanReadablyShipSyncLog.txt file in the PlayerData folder. This gives a message for each ship that was changed or deleted and why that was done, as well as what planet it was on and who owned it. When syncs are doing funky things, this is a way to manually review it and find out why.

Long-term, once the sync code has been in use for a long time, it's unlikely that very actionable data will be found from this. Any discrepancies would again be code errors directly on the client, not in how the data is coming to the client. But for now, this gives us a robust and helpful view of what the client thinks it is supposed to be doing.

Write Network World Serialization Logs

This is something you would use if the client has a corrupt world immediately upon connection to the host. To be useful, both the host and client would need this setting on, and then we'd need the NetworkWorldSerialization.txt from the host, and the corresponding NetworkWorldDeserialization.txt from the client. We would then look at the differences to see what is wrong with that initial world sync.

Show Network Sync Details In Escape Menu

Ah, now this is something useful! It has a very different output on the host and on each client. This data is being collected no matter what (in memory, not on disk), so having it show causes no extra CPU load or slowness. The data is all just aggregated integer counts, so it's very inexpensive for us to collect in the first place. The main reason this field exists at all is to reduce the clutter on the interface.

There are times where we might want to see screenshots of this data from the escape menu on the host and clients, so that we can have a general sense of potentially why performance is bad in some savegame. Basically this is kind of broadly diagnostic, telling us if there are too many syncs happening, but not why. Looking at this data yourself, if you see the PKID or ship sync numbers skyrocketing suddenly, then that might be a good time to make a savegame and send it to us, because something that is breaking lockstep is happening.

You only see these numbers go up AFTER lockstep is broken, however, so it can be a bit too late by the time you think to save (unless it's an ongoing process that is breaking lockstep). But if it's something like "at every wave there's a major amount of desyncs that it has to fix," then you can make that observation yourself and then give us a save from right before a wave unleashes, or something like that.

If you are suspicious that network traffic spikes at certain times, this will also give you hard data to back up your suspicions. You can watch the network traffic totals, and if those spike after certain events, or when you give certain orders or do certain types of battles or something, then that can give you confirmation that your hunch is right, and you can send a savegame to us for analysis.

cmd:dump data tables

Type cmd:dump data tables into the text chat on any client, and it will then freeze the game for 20-30 seconds (please warn other players before doing this) and fill up your respective PlayerData/DataTableExports folders.

You can then copy the folders from different machines all to be on one machine, and use a tool like WinMerge to do a folder-level comparison of the results from each machine. If there are divergences, then this is going to be causing a lot of desyncs, probably.

Most of the time there should not be any differences here, so this is probably not worth doing very often at all, because these files are really huge and also unlikely to be wrong. But if you are playing with a large and complicated mod you are suspicious of... it could be worth checking.

cmd:dump external data

Type cmd:dump external data into the text chat on any client, and it will then freeze the game for probably only 1-2 seconds (please still warn other players before doing this) and fill up your respective PlayerData/ExternalDataExports folders.

Comparing Machine Sync

You can then copy the folders from different machines all to be on one machine, and use a tool like WinMerge to do a folder-level comparison of the results from each machine. There will always be divergences between the host and the clients, and if the clients joined at different times then also between clients.

Divergences are probably not a crisis! However, if things are VERY divergent, or the things that are divergent on the clients are very stale (aka have not been updated in a long time), then this can give you really useful info.

Looking For Client Memory Leaks

If a client has been seeing performance that gets progressively worse, or memory is notably going up in usage, then it's possible that either the main devs or a modder made a mistake in how a collection of data is being synced to clients from the host. The host is not going to see this particular problem.

After you have run the export, which may take substantially longer than usual if a client actually is having trouble, then you can open up these logs in something like Notepad++ and do a search for the text "BIGLIST". There are some cases where you will see perfectly-valid versions of these, particularly BIGLIST-A and BIGLIST-B. However, if you're suspicious of these, you can always show your logs to the developers or to the mod author for review.

What is rarely correct is BIGLIST-C, and BIGLIST-D is almost NEVER correct. BIGLIST-E is not something we can think of any valid reason to have happen. Please report it to the appropriate party (developers or mod author).