AI War 2:Serialization Gates

This whole page is aimed almost more at developers than modders, but it's important to know for both to a certain extent.

All of the data in the savegame files, profiles, input files, and settings files are stored as a continuous stream of ints, floats, fints (fixed-ints), bools, and strings. It's imperative that the data be read out exactly as it was written in, or else the whole thing falls apart. We don't have tags in there saying what field is what, because the amount of data is too massive for that. So if you switch reading int 1 and int 2, there's no protection against that.

So! Thankfully, we've made it really easy to simply mimic reads and writes, in well-ordered serialization and deserialization methods. So long as those look okay at the start and a test save and load functions, all is well. (Unless data is really nested, and not present, as the one exception.)

The big question, of course, is what if you need to add a field or remove a field? Wouldn't that break all savegames? That's where serialization gates come in. But first, some slight background:

GameVersion structure

There is a GameVersion xml file, which has entries like this:

<game_version name="Unity2018_2" major_version="0" minor_version="749" release_date_text="July 9th, 2018" />

<game_version name="MoreFactionDataForTechsEtc" major_version="0" minor_version="756" release_date_text="July 25th, 2018" />

The name is pointless, but must be unique. It's just so that we can remember what it is supposed to be. You can't change it once it has been set and things have been saved with it, though.

The real important part is in the major version and minor version, both of which are ints. It's assumed that the minor version won't be outside 0-999. The major version can be 0-n.

Making a SerializationGate from a GameVersion

Only developers can do this, at the moment. There is an enum called SerializationGate defined in the EnumsGame file. It looks along these lines:

   public enum SerializationGate
   {
       None,
       _0_756_MoreFactionDataForTechsEtc,
       Length
   }

The underscores in there are really important. It must start with an underscore (because enum names cannot start with a number), and then it must have the major version, another underscore, the minor version, another underscore, and then some sort of text. The text can be anything you want, it doesn't matter. But again it helps for readability if it's something recognizable.

The code does a split('_') on the underscore there and winds up with an array that contains the major and minor versions at expected places. So it's able to match to the proper game version.

Game versions themselves have a meythod for comparing themselves to gates:

MyGameVersion.GetMeets( SerializationGate._0_756_MoreFactionDataForTechsEtc )

Easy peasy. But how do we actually use this in some practical scenarios?

Practical Examples of Serialization Gate Checking

This is equally for modders and developers, I suppose.

Adding A Field

Let's suppose you want to serialize something new. You would set up the serialization just like usual... but if you do the deserialization like normal, then you've just broken all old savegames. So instead we use a serialization gate in order to only deserialize if the savegame is new enough. Like this:

if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_103_SetupOnly_StartingPlanetIndex ) ) this.SetupOnly_StartingPlanetIndex = Buffer.ReadInt32();

And you're not limited to just single fields. You can do sub-objects, lists, whatever. If you think something isn't possible, you aren't thinking creatively enough. ;) We've been using this for half a decade now, so it's definitely possible.

Here's how lists work:

           if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_607_RefuseToWait ) )
           {
               int countToExpect = Buffer.ReadInt32();
               for ( int i = 0; i < countToExpect; i++ )
               {
                   FInt strength = Buffer.ReadFInt();
                   if ( i < result.UnengagedMobileStrengthByHopCount.Length )
                       result.UnengagedMobileStrengthByHopCount[i] = strength;
               }
           }

Overall that's not really different from the first example.

It's worth also noting that sometimes not only will you want to deserialize something like a sub-object, but if the game version is too old and doesn't have that data, you want to fill it in with something you auto-generate. You can probably guess the code, but here's an example:

           if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_105_ControlledByPlayerAccounts ) )
               this.GalaxySpaceboxDefinition = SpaceboxDefinitionTable.Instance.DeserializeFrom( Buffer );
           else
               this.GalaxySpaceboxDefinition = SpaceboxDefinitionTable.Instance.GrabBagGalaxyMap.PickRandomItemAndReplace( Engine_Universal.PermanentQualityRandom );