AI War 2:Serialization Gates

This whole page is aimed almost more at developers than modders, but it's important to know for both to a certain extent.

All of the data in the savegame files, profiles, input files, and settings files are stored as a continuous stream of ints, floats, fints (fixed-ints), bools, and strings. It's imperative that the data be read out exactly as it was written in, or else the whole thing falls apart. We don't have tags in there saying what field is what, because the amount of data is too massive for that. So if you switch reading int 1 and int 2, there's no protection against that.

So! Thankfully, we've made it really easy to simply mimic reads and writes, in well-ordered serialization and deserialization methods. So long as those look okay at the start and a test save and load functions, all is well. (Unless data is really nested, and not present, as the one exception.)

The big question, of course, is what if you need to add a field or remove a field? Wouldn't that break all savegames? That's where serialization gates come in. But first, some slight background:

GameVersion structure

There is a GameVersion xml file, which has entries like this:

<game_version name="Unity2018_2" major_version="0" minor_version="749" release_date_text="July 9th, 2018" />

<game_version name="MoreFactionDataForTechsEtc" major_version="0" minor_version="756" release_date_text="July 25th, 2018" />

The name is pointless, but must be unique. It's just so that we can remember what it is supposed to be. You can't change it once it has been set and things have been saved with it, though.

The real important part is in the major version and minor version, both of which are ints. It's assumed that the minor version won't be outside 0-999. The major version can be 0-n.

Making a SerializationGate from a GameVersion

We got rid of the concept of serialization gates, and instead always use this sort of code:

if ( Buffer.FromGameVersion.GetGreaterThanOrEqualTo( 0, 105 ) ) //ControlledByPlayerAccounts

Chris likes to leave a little handy comment (optional) that gives the context for what the name of the version was, but that's optional.

You can also do this:

if ( Buffer.FromGameVersion.GetLessThan( 0, 105 ) ) //ControlledByPlayerAccounts

This is identical to just using a ! to invert the first example, but this is a lot clearer to read in code because those ! signs are easy to miss in such a long statement.

As a convention to ease readability, we use one method or the other rather than inverting them.

Practical Examples of Serialization Gate Checking

This is equally for modders and developers, I suppose.

Adding A Field

Let's suppose you want to serialize something new. You would set up the serialization just like usual... but if you do the deserialization like normal, then you've just broken all old savegames. So instead we use a serialization gate in order to only deserialize if the savegame is new enough. Like this:

       if ( Buffer.FromGameVersion.GetGreaterThanOrEqualTo( 0, 103 ) ) //StartingPlanetIndex
              this.SetupOnly_StartingPlanetIndex = Buffer.ReadInt32();

And you're not limited to just single fields. You can do sub-objects, lists, whatever. If you think something isn't possible, you aren't thinking creatively enough. ;) We've been using this for half a decade now, so it's definitely possible.

Here's how lists work:

           if ( Buffer.FromGameVersion.GetGreaterThanOrEqualTo( 0, 607 ) ) //RefuseToWait
           {
               int countToExpect = Buffer.ReadInt32();
               for ( int i = 0; i < countToExpect; i++ )
               {
                   FInt strength = Buffer.ReadFInt();
                   if ( i < result.UnengagedMobileStrengthByHopCount.Length )
                       result.UnengagedMobileStrengthByHopCount[i] = strength;
               }
           }

Overall that's not really different from the first example.

It's worth also noting that sometimes not only will you want to deserialize something like a sub-object, but if the game version is too old and doesn't have that data, you want to fill it in with something you auto-generate. You can probably guess the code, but here's an example:

           if ( Buffer.FromGameVersion.GetGreaterThanOrEqualTo( 0, 105 ) ) ) //ControlledByPlayerAccounts
               this.GalaxySpaceboxDefinition = SpaceboxDefinitionTable.Instance.DeserializeFrom( Buffer );
           else
               this.GalaxySpaceboxDefinition = SpaceboxDefinitionTable.Instance.GrabBagGalaxyMap.PickRandomItemAndReplace( Engine_Universal.PermanentQualityRandom );

Removing A Field

Old data? Changed formats to something new? No problem. We still use a serialization gate, but this time we check if it is NOT met. In that case, we read in all the old data like we used to, but we just toss it away. Data structures are mildly more tricky in that you have to retain a shell of those objects if the nesting there went too deep, but when in doubt you can just delete everything on the old objects except the deserialization method, and then have them not assign data to anything.

A lot of times we make it a convention to put a comment next to each piece of data we're reading saying what it used to be, to make debugging easier if there's ever a problem (there shouldn't be, but better safe than sorry). Here's a simple example with a list of what used to be a very simple sub-object, where we just inlined the sub-object's fields:

           if ( Buffer.FromGameVersion.GetLessThan( 0, 615 ) ) //DifficultyToExternal
           {
               countToExpect = Buffer.ReadInt32();
               for ( int i = 0; i < countToExpect; i++ )
               {
                   //this.AIPChangeHistory.Add( AIPChange.DeserializeFrom( Buffer ) );
                   Buffer.ReadInt32(); //result.GameSecond = 
                   Buffer.ReadFInt(); //result.Change = 
                   Buffer.ReadInt32(); //result.Reason = (AIPChangeReason)
                   Buffer.ReadString(); // result.RelatedEntityTypeData = GameEntityTypeDataTable.Instance.DeserializeFrom( Buffer );
               }
           }

Other times, you'll find that you add something in one version, gating it positively for a while, and then later you remove it and wind up gating it negatively. That works like any typical boolean logic:

        if ( Buffer.FromGameVersion.GetGreaterThanOrEqualTo( 0, 123 ) //UnlockedAIDesigns
            && Buffer.FromGameVersion.GetLessThan( 0,601 ) ) //PerAIUnlocks
            Buffer.ReadInt32(); //this.SpentAIUnlockPoints =

And it's of course possible to get more complicated with that concept, where part of something was added, and then a larger whole including that thing is removed:

           if ( Buffer.FromGameVersion.GetLessThan( 0, 601 ) ) //PerAIUnlocks 
           {
               countToExpect = Buffer.ReadInt32();
               for ( int i = 0; i < countToExpect; i++ )
                   GameEntityTypeDataTable.Instance.DeserializeFrom( Buffer ); //this.CorruptedAIDesigns.Add();
               if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_123_UnlockedAIDesigns ) )
               {
                   countToExpect = Buffer.ReadInt32();
                   for ( int i = 0; i < countToExpect; i++ )
                       GameEntityTypeDataTable.Instance.DeserializeFrom( Buffer ); //this.UnlockedAIDesigns.Add();
               }
           }