AI War 2:Serialization Gates
This whole page is aimed almost more at developers than modders, but it's important to know for both to a certain extent.
All of the data in the savegame files, profiles, input files, and settings files are stored as a continuous stream of ints, floats, fints (fixed-ints), bools, and strings. It's imperative that the data be read out exactly as it was written in, or else the whole thing falls apart. We don't have tags in there saying what field is what, because the amount of data is too massive for that. So if you switch reading int 1 and int 2, there's no protection against that.
So! Thankfully, we've made it really easy to simply mimic reads and writes, in well-ordered serialization and deserialization methods. So long as those look okay at the start and a test save and load functions, all is well. (Unless data is really nested, and not present, as the one exception.)
The big question, of course, is what if you need to add a field or remove a field? Wouldn't that break all savegames? That's where serialization gates come in. But first, some slight background:
Contents
GameVersion structure
There is a GameVersion xml file, which has entries like this:
<game_version name="Unity2018_2" major_version="0" minor_version="749" release_date_text="July 9th, 2018" />
<game_version name="MoreFactionDataForTechsEtc" major_version="0" minor_version="756" release_date_text="July 25th, 2018" />
The name is pointless, but must be unique. It's just so that we can remember what it is supposed to be. You can't change it once it has been set and things have been saved with it, though.
The real important part is in the major version and minor version, both of which are ints. It's assumed that the minor version won't be outside 0-999. The major version can be 0-n.
Making a SerializationGate from a GameVersion
Only developers can do this, at the moment. There is an enum called SerializationGate defined in the EnumsGame file. It looks along these lines:
public enum SerializationGate
{
None,
_0_756_MoreFactionDataForTechsEtc,
Length
}
The underscores in there are really important. It must start with an underscore (because enum names cannot start with a number), and then it must have the major version, another underscore, the minor version, another underscore, and then some sort of text. The text can be anything you want, it doesn't matter. But again it helps for readability if it's something recognizable.
The code does a split('_') on the underscore there and winds up with an array that contains the major and minor versions at expected places. So it's able to match to the proper game version.
Game versions themselves have a meythod for comparing themselves to gates:
MyGameVersion.GetMeets( SerializationGate._0_756_MoreFactionDataForTechsEtc )
Easy peasy. But how do we actually use this in some practical scenarios?
What If I Need A Gate But Am Not A Developer?
Right, that enum is in the internal non-moddable code. Enums aren't really something that can be modded, which is why we moved to xml tables most places.
Fortunately, the actual serialization gate is just a convenience, to be honest. You can easily use other methods on the gameversion to do the same thing. For instance, these are functionally identical:
if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_105_ControlledByPlayerAccounts ) )
and
if ( Buffer.FromGameVersion.GetGreaterThanOrEqualTo( 0, 105 ) ) //ControlledByPlayerAccounts
The second case even has a little handy comment (optional) that gives the same context that the first one has.
Practical Examples of Serialization Gate Checking
This is equally for modders and developers, I suppose.
Adding A Field
Let's suppose you want to serialize something new. You would set up the serialization just like usual... but if you do the deserialization like normal, then you've just broken all old savegames. So instead we use a serialization gate in order to only deserialize if the savegame is new enough. Like this:
if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_103_SetupOnly_StartingPlanetIndex ) ) this.SetupOnly_StartingPlanetIndex = Buffer.ReadInt32();
And you're not limited to just single fields. You can do sub-objects, lists, whatever. If you think something isn't possible, you aren't thinking creatively enough. ;) We've been using this for half a decade now, so it's definitely possible.
Here's how lists work:
if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_607_RefuseToWait ) )
{
int countToExpect = Buffer.ReadInt32();
for ( int i = 0; i < countToExpect; i++ )
{
FInt strength = Buffer.ReadFInt();
if ( i < result.UnengagedMobileStrengthByHopCount.Length )
result.UnengagedMobileStrengthByHopCount[i] = strength;
}
}
Overall that's not really different from the first example.
It's worth also noting that sometimes not only will you want to deserialize something like a sub-object, but if the game version is too old and doesn't have that data, you want to fill it in with something you auto-generate. You can probably guess the code, but here's an example:
if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_105_ControlledByPlayerAccounts ) )
this.GalaxySpaceboxDefinition = SpaceboxDefinitionTable.Instance.DeserializeFrom( Buffer );
else
this.GalaxySpaceboxDefinition = SpaceboxDefinitionTable.Instance.GrabBagGalaxyMap.PickRandomItemAndReplace( Engine_Universal.PermanentQualityRandom );
Removing A Field
Old data? Changed formats to something new? No problem. We still use a serialization gate, but this time we check if it is NOT met. In that case, we read in all the old data like we used to, but we just toss it away. Data structures are mildly more tricky in that you have to retain a shell of those objects if the nesting there went too deep, but when in doubt you can just delete everything on the old objects except the deserialization method, and then have them not assign data to anything.
A lot of times we make it a convention to put a comment next to each piece of data we're reading saying what it used to be, to make debugging easier if there's ever a problem (there shouldn't be, but better safe than sorry). Here's a simple example with a list of what used to be a very simple sub-object, where we just inlined the sub-object's fields:
if ( !Buffer.FromGameVersion.GetMeets( SerializationGate._0_615_DifficultyToExternal ) )
{
countToExpect = Buffer.ReadInt32();
for ( int i = 0; i < countToExpect; i++ )
{
//this.AIPChangeHistory.Add( AIPChange.DeserializeFrom( Buffer ) );
Buffer.ReadInt32(); //result.GameSecond =
Buffer.ReadFInt(); //result.Change =
Buffer.ReadInt32(); //result.Reason = (AIPChangeReason)
Buffer.ReadString(); // result.RelatedEntityTypeData = GameEntityTypeDataTable.Instance.DeserializeFrom( Buffer );
}
}
Other times, you'll find that you add something in one version, gating it positively for a while, and then later you remove it and wind up gating it negatively. That works like any typical boolean logic:
if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_123_UnlockedAIDesigns ) && !Buffer.FromGameVersion.GetMeets( SerializationGate._0_601_PerAIUnlocks ) ) Buffer.ReadInt32(); //this.SpentAIUnlockPoints =
And it's of course possible to get more complicated with that concept, where part of something was added, and then a larger whole including that thing is removed:
if ( !Buffer.FromGameVersion.GetMeets( SerializationGate._0_601_PerAIUnlocks ) )
{
countToExpect = Buffer.ReadInt32();
for ( int i = 0; i < countToExpect; i++ )
GameEntityTypeDataTable.Instance.DeserializeFrom( Buffer ); //this.CorruptedAIDesigns.Add();
if ( Buffer.FromGameVersion.GetMeets( SerializationGate._0_123_UnlockedAIDesigns ) )
{
countToExpect = Buffer.ReadInt32();
for ( int i = 0; i < countToExpect; i++ )
GameEntityTypeDataTable.Instance.DeserializeFrom( Buffer ); //this.UnlockedAIDesigns.Add();
}
}
