AI War 2:The Great Refactor

Known Issues

• Any bugs or requests should go to our mantis bugtracker
  • If you need to submit log files, those can generally be found under your PlayerData folder in the folder your game is installed in. The most relevant one is called ArcenDebugLog.txt. You can send us the whole thing, or just strip out relevant parts.
  • In rare cases, mainly if your entire game crashes (that almost never happens), we will need your unity player log. That gets overwritten the next time you run the game after a crash, unlike the other log. These can be found here:
    • Windows: C:\Users\username\AppData\LocalLow\Arcen Games, LLC\AIWar2\Player.log
    • macOS: ~/Library/Logs/Arcen Games, LLC/AIWar2/Player.log
    • Linux: ~/.config/unity3d/Arcen Games, LLC/AIWar2/Player.log

• Multiplayer is in public beta, as noted below. There is a detailed multiplayer guide that we are working on building up.

• Feel free to join discussions on discord!

What Does Multiplayer Beta Mean?

Please see this link for details on multiplayer. This wound up taking up too much space in this document, so all of the multiplayer-relevant bits have been moved to the other page.

What's this phase all about?

We hit a point, while working on the beta branch, where we realized we just needed to really rework a lot of the underlying code to make multiplayer and modding and so on be as bug-free and compatible as we wanted for the future. So this was supposed to be a digression that would take a week or two at most, but wound up taking closer to four. And there are then after-effects for a while, on top of that. But the good news is that it dramatically improves... performance, multiplayer correctness, how easy or hard the game is to code to, etc.

The goal of this phase is to finish up reworking all of the things that we need to, for maximum long-term enjoyment of the game. This also means getting multiplayer absolutely completely finished and working. After that, we'll be shifting back into work on DLC3 (The Neinzul Abyss), which will be the final DLC for the game. The various remaining kickstarter items will also be taken care of, plus some other bits that we just always wanted to add for the game, and after that we'll be moving on to other projects aside from bugfixes and balance tweaks and modder-support throughout 2022.

The target release for DLC3 is in January of 2022, and that is when we'll also have the AI War 2 Complete Edition. After that, it will just be fixing material defects, and supporting modders. We expect it to have a long life if AI War Classic is anything to go by. The game is in a really good state and only getting better, and doing a refactor of this scale near the end of the project development cycle is a sign of our commitment to that longevity. Even if you aren't giving us more money in the future for this title, we want you to be able to enjoy it for years and years.

3.744

(Not yet released -- we're still working on it!)

• Added a new FleetSimLogic, which moves some of our fleet logic out of the closed-source Core dll into the open-source BaseInfo External dll.
  • No logic was changed at this time (by design), it was just moved.
  • There is also one for the fleet memberships, but for now that is empty. The one method that would have gone in there so far is more efficient when on the FleetSimLogic class.
  • The purpose of these changes is primarily so that we don't ever have to have anything pathfinder-related in Core.

• Orbital AI -- new feature!
  • No more stationary turrets, except for the wormhole sentinels. Instead, the Orbital Guardposts now spawn turrets that orbit the guardpost. These can be completely random or from a specific set to complement a given guardpost.

Pathfinding Performance Improvements, And ThreadStatic RAM Improvements

• Added a new ThreadStaticWrapperedPooledItem, which is a way for us to keep some sort of complex object -- often a collection, but more things than that are planned -- in a threadstatic, safe way.
  • ...What?
    • Okay, so ThreadStatic. Basically, any time a new thread is using that variable, it can initialize its own copy. This is hyper useful.
    • However, with random background worker threads, this does mean that we wind up with some things initialized on threads that may never be used the same way again.
      • I'm referring here to things like Task<>.Run() or things that happen inside Parallel.For() or similar. Or older-school ThreadPool calls, whatever.
    • Additionally, we pool and reuse our own threads between loads of the game (meaning out to the main menu and back), for things like factions.
      • So if thread 554 was Dark Spire in one game, and you go to the main menu and come back into the same game, it might wind up being the AI Hunter thread instead.
    • Essentially this works vaguely like a memory leak, even though it is in no way a memory leak, because we wind up with stuff that has been allocated but will never be used again.
      • During our "before mapgen or savegame load clear," it makes more sense to recall all the threadstatic stuff, and pull those into central pools, and then start over fresh using them again.
  • How we do exactly that is... with this new class, ThreadStaticWrapperedPooledItem!
    • This wrapper item is instead now the thing that is threadstatic, and it communicates under the hood with a pool.
    • We can put anything in here. Collections, pathfinders, and so on.
      • A side benefit of this is that for things where we previous had a scope that was a "context" (that can be VERY confusing), the scope can instead be the thread.
      • Recently, we've been having some issues with pathfinders being called from more than one contextual location at a time, which might have been a false alarm or might have been a very dangerous bug. Either way, this will fix that.
        • But even more to the point, we might have been having some pathfinder bugs if people ever decided to use one from a given context in a Task or Parallel call. I don't think anyone was doing that yet, but if they did it would have given scrambled results.
    • A side effect of these changes is that we can allocate fewer lists and pathfinders in general, because not every faction needs them in the first place. And what we do allocate can get much more reuse.
  • If we felt like we needed to, we could extend this concept further and push kind of a library "check in, check out" sort of data structure that keeps things for less long than between map-generations.
    • That said, I think that there is potentially performance loss from that, and I don't expect the RAM usage of my current approach to be enough of a problem to warrant that extra CPU usage.
    • If I get proved wrong later, then I can make a second version of this general concept that prioritizes RAM savings instead of CPU savings, and it will be easy to swap out the ones where it's a problem.

• The types of end pathfinders have been condensed down to three: PlanetPathfinderBasic, PlanetPathfinderConservative, and PlanetPathfinderWarden.
  • There is nothing in there about any sort of long-term or short-term differences now.
  • Additionally, when you find a path, it now requires you to pass in the faction and any extra debug info when you do so.

• Added a new DictionaryOfProtectedValDictionaries, which is presently unused. Thought I would need it on CacheForPathingMode, but those are being pooled by map-reload, so no.
  • In general, a lot of the pathing code is now pooled way better between loads of the game. Previously... possibly it was lost and leaked? Maybe it was not leaking, but just kind of allocated in a useless place? Either way, fixed.

• Man, there's a whole bunch of other changes where things with pathfinding were really confusing to me. I didn't have time to document them all.
  • However, the new approach is really clear and has comments or naming that explains what the rules are, and why. Also the code reads linearly from top to bottom, rather than jumping all over the place. So that also helps.
  • For an end-faction-developer, very little is different. It does ask you for a "debug addendum" string, but that's just so that error messages will be more informatively precise if you should run into any.

• The sum total of these changes saves a LOT of memory, as well as making things load faster, as well as also eliminating the pathfinders-from-multiple-contexts bug. It also leads to some CPU speed improvements in several spots.
  • It also solves what was definitively still a memory leak in terms of pathfinders.
    • After 10 loads of the game prior to these changes, there were 169,706 PathBetweenPlanetsForFaction-PathToReadOnly collections, with 25,661,700 capacity and 115,592 stored items. This is absolutely freaking huge!
    • Now, after 8 loads of the game, there are 79 collections, 420 total capacity, and 19 stored items. This is... uh... quite an improvement.

• Thanks to tom.prince and Badger for reporting the pathfinder errors that led to all these improvements. This was a tough nut to crack.

Beta 3.743 Return Of The Imperial Spire

(Released November 2nd, 2021)

Improvements

• The FitListToSize_OrCreateWithDefault_OrThrowErrorIfEmpty() and MakeListAtLeastThisLongWithDefaults() support methods have been moved to ArcenDynamicTableRow since they can potentially do a lot of good where all XML-based objects can use them.
• Additions to the EntityTypeDrawingBag (ETDB):
  • It is now possible to define mixed bags of parse types - as in spawning a certain entity by its name and then random entities based on tags.
  • The same way differing count types (raw count, strenth-based, metal/energy/AI cost) can now be mixed.
  • Additionally there is 2 new lists: duplicate_pick_handling_list (GenerallyAllowed, AllowedForDifferentTiers, GenerallyUnallowed) allows to set whether or not duplicates may be picked, while multi_pick_list defines how many separate, if need be unique entity types of a "tier" are picked (say that there's a tag with 3 units, and 2 should be chosen - this is where the "2" would go to make that happen). The maximum that can be picked is the maximum possible individual entries per "tier".
• Outguard now instead of the primary/secondary/tertiary XML fields uses the ETDB, with some new ones:
  • min_spawn_count_increase_per_aip_interval is a list that defines how much is added to the min/max values of the spawn counts, while the intervals themselves are defined by the spawn_count_increase_aip_threshold list.
  • All XML was adjusted to compensate.
  • Thanks to NR SirLimbo for adding!

• Rename the Random Faction display name so as to get it toward the top of the Game Lobby 'Add Faction' menu. Otherwise it's a hassle when adding a couple random factions
  • Thanks to Badger for updating!

• Refactor the code for generating spire city and fleet names.
  • Thanks to tom.prince for updating.

• Brutal Lair: slow down the rate of Brutal Guardian spawning
  • Thanks to Badger for updating!

• Add a couple new Elderlings for diversity
  • Thanks to Badger for updating!

• Elderlings now grant essence when killed
  • Thanks to Badger for updating!

• Add a new Feeble Elderling that's seeded near the player in necromancer mode
  • Thanks to Badger for updating!

• Minor buff to sappear beachhead constructors
  • Thanks to Badger for updating!

• Slothful and Wrathful elderlings can now be hacked for necromancer blueprints to allow new flagship transformations.
  • There's an explanation in the XML for how one would set up other Elderlings to be hacked for blueprints
  • Thanks to Badger for updating!

• When the Necromancer is added to the game, it now also adds a copy of the Elderlings (AI-allied) and the Templar (AI-allied). These are visible in the game lobby so players can adjust the difficulty.
  • Thanks to Badger for updating!

• Gave the Templar and Elderlings some new icons
  • Thanks to Badger for updating!

• More Outguard work:
  • The spawn_delay XML field now actually works. Groups called in may take a delay in time until they arrive. Sidebar, tooltips and XML have been adjusted.
  • With the exception of the DLC 2 Outguard "Hyperion" all are still the same as before. Hyperion, as per original concept, now takes 30 seconds to warp in.
  • Thanks to NR SirLimbo for adding!

• Tweak the descriptions for macrophages and elderligns a bit
  • Thanks to Badger for updating!

• Add some new options for the Necromancer to make the game easier.
  • You can A. start with bonus resources
  • B. Start with higher available skeletons
  • C. Start with higher available wights
  • D. Start with a bonus ship line of wights
  • The goal is to make sure that even weaker players can enjoy playing the necromancer.
  • Thanks to Badger for adding!

• Add a couple of extension methods to ArcenCharacterBuffer for displaying human readable times.
  • AddHoursAndMinutes will display times like 5h 13m 20s, for use in description and tooltip text.
  • AddSecondsRemaining will display times like 13:20, for use in notification panels. By default, this will show times <20s in red and <60s in yellow.
  • I also went and changed a bunch of places to use the new functions. Most of the changes haven't been tested.
  • Thanks to tom.prince for updating!

• Convert a few more notification panels to use AddSecondsRemaining.
  • Also add an option to AddSecondsRemaining to use the 10 minute intensity variation from nomad planets.
  • Thanks to tom.prince for updating!

• There is a new GetShouldBeHidden() on the notification code that lets programmers specify that something should be hidden based on whatever they want.
  • Essentially, if a timer was counting down and would be negative, but we find that the notification is still in the list for a little bit, then instead this can just say "yeah, hide me," and it will disappear from the list at a proper time. This is particularly important for multiplayer clients, which might get data slightly late.
  • Overall nobody needs to go crazy with this function, but it exists as a new safety valve to help keep things from showing stale data. We already have this same sort of thing already on most of our UI element, so extending it here is an easy leap and quite consistent with everything else. It's only needed when it's needed, but it's nice to have it for those cases.
  • Thanks to tom.prince for the discussion that inspired this addition.

• List<> and ProtectedList<> now both have an IsHardened bool property on them.
  • If a list is hardened, then index out of range exceptions are considered fine. Basically those are threading issues that should just return default(T), which is generally going to be null.
  • This does not help with inserts, adds, writes, or clears. It applies to for, foreach, contains, find, findall, findlast, get-index, etc.
  • If you have a list that is mostly used one place, and is only written to on a single given thread (main sim probably), then you can just flag it as IsHardened = true and read from it from other threads. If one of those threads happens to go what would be out of bounds for the list (generally because something was removed from the list on its main thread while the other thread was reading from it), that's no problem. You'll just get a null back, which you can check for.
  • In general when reading from lists, read into a variable with one indexing call and then use that value after you've verified it's not null. This is better practice for performance anyhow. It then keeps your surface area much smaller for the list. No need for try/catch around the index into the list now.
  • Thanks to NR SirLimbo for inspiring this change.

• Since OutguardSpawnRequest is going to be seen on the UI for longer than it might exist in the actual working pool, it makes sense to put in the time-based pool like we do for units, shots, etc. This has been converted over, and it thus then calls its Clear() object when it goes out of quarantine and back into the main pool, which is 2-3 seconds after going into the pool (the UI list refresh will happen every 1 second, so that's plenty of time).
  • Please note that this does not have any effect on the discussion of the way that the lists are filled, or how the lists are accessed -- the things that NR SirLimbo and I were discussing. Pooling strategy is one thing, and cross-thread read/write strategy is another, and the two can be considered independently.
  • Thanks to tom.prince for suggesting.

• More under-the-hood Outguard work:
  • Put in the code for the new hardened protected list as primary list for the Outguard Spawn Requests. This should prevent any potential cross-threading exceptions when it's being read on an UI thread.
  • Clients are being kept (hopefully) perfectly up-to-date with Outguard spawn requests, so their sidebars also refresh as timely as possible.
  • Additionally it should no longer be possible to spawn the same group twice on hosts and clients by clicking basically at the same time, where the UI doesn't yet block the client from sending the game command to spawn the group.
  • Thanks to NR SirLimbo for implementing!

Bugfixes

• Mark items as coming out of quarantine when going into the main pool. (TimeBasedPool)
  • Thanks to tom.prince for fixing!

• Make spire transciever work again, and add a global cap to them.
  • Thanks to tom.prince for fixing!

• Make imperial spire spawn from transceiver as well.
  • This does make the imperial spire twice as powerful, in a solo game, but that is really incidental, since they already have overwhelming force. This is mostly for flavor reasons, I was surprised when testing out the transceiver that they were at least initially arriving from the transceiver that called them.
  • Thanks to tom.prince for fixing!

• Elderlings now properly join their allies in minor faction teams
  • Elderlings now mark up properly. They initially start with a 3 planet territory, then add one new planet when the Elderling marks up
  • Elderlings now try to max out at 3 Elderlings per Territory planet. They are now more organized in how to lay eggs, and to make sure that elderlings don't get too cluttered.
  • Thanks to Badger for fixing!

• Fireteams: fix a bug presumably from the memory-leak changes where we had two almost identically named copies of a list, and would get confused as to which we were using. There is now only one list
  • Thanks to Badger for fixing!

• Fix a bug where too many elderlings were winding up in the same space
  • Thanks to Badger for fixing!

• Templar Encampments spawn a bit less frequently, but should be a bit stronger
  • Thanks to Badger for fixing!

• AI-allied high-intensity elderlings will be more concentrated. Hopefully fix a null reference exception in the wave code
  • Thanks to Badger for fixing!

• Fixed an exception that could happen due to a race condition in HandleLODsAndShipPartAnimations/
  • Thanks to Eluthena for reporting.

• Fixed an issue where OutguardSpawnRequest was never being returned to the pool. It now centrally uses a ProtectedList rather than a List, which means it should automatically head back to the pool when we're done with it. (This was a memory leak.)
  • Thanks to tom.prince for reporting.

• Fixed an error in my build scripts for GOG Galaxy that was leading to the updates on the beta branch being stuck on version 3.313. I had thought that things were working fine since it was still having something to download every time a new version was uploaded, but that was not the case.
  • What happened was, I upgraded to a new computer a few months ago, and when I did the paths changed for where my build scripts pointed to. All my steam scripts use relative path names, but the GOG ones are apparently hardcoded and were looking to a wrong drive. I would have expected some sort of error, but there was not one.
  • I've now corrected the path names to look to the correct drive, and folks on GOG can now get the latest betas properly for testing again.
  • Thanks to slake-moth for reporting.

• In Disband_JustTheDeepInfoPortion, the "GetExternalBaseInfoAs<ScourgePerUnitBaseInfo>()" (which throws an error if it's the wrong type) now is just "TryGetExternalBaseInfoAs<ScourgePerUnitBaseInfo>()" (which is able to silently return null when it's not a match).
  • This is basically old code, as Badger notes, but the way that I had translated it into the new framework was suboptimal.
  • Thanks to Badger for reporting.

• Fixed several groups of errors that could happen when deleting campaigns, relating to exceptions and having the wrong campaign selected, etc. Some of this was indeed because we were using an asynchronous callback when -- after a delete in particular -- we needed to use a synchronous one for the sake of accuracy.
  • Thanks to Crabby for the report, and tom.prince for the diagnosis and suggested solution.

• Fixed a bug where if the campaign name was null, and you were NOT in the tutorial, it would start saving as "Tutorial".
  • That's been fixed so that it will now save as General, and if you're in a tutorial it will save as Tutorial.
  • Thanks to Daniexpert, Eluthena, and tom.prince for reporting.

• Fixed a bug in GameCommand_DeleteAllSaveGames where it was using the current campaign name instead of the one that was passed in.

• Put in error tracking in a number of places so that it will fail to save (and tell you about that) if the campaign name is blank.
  • Fixed a variety of places where the savegame or campaign name could be null and just cause problems or save into an odd space without warning. Now those places error, instead, so that we'll know when they happen.
  • When pooling the World object, we no longer ever reset the CampaignName. The first thing we do when using it out of the pool is set a new one, anyhow.
    • This should stop a variety of bugs with saving doing funky things since the great refactor.
  • Thanks to Daniexpert and tom.prince for reporting.

• Fixed a deserialization bug in wormhole invasions.
  • Thanks to Daniexpert for the report with the matching serialization logs that made this super easy to see in action!

• Corrected Nucleophilic turret count on Webbed battlestation template 5 -> 25
  • Thanks to CRCGamer for fixing!

• ArcenUI_AxisAlignment is now a struct rather than a class. It doesn't actually solve a leak or anything.

Beta 3.742 "Classic Map" For DLC1

(Released October 30th, 2021)

• Add a minimal UI for changing which fleet is being bolstered.
  • Thanks to Tom for adding!

• DLC3: Add a new Elderlings fashion.
  • This faction starts with an Egg. After some time the egg hatches into an Elderling.
  • Elderlings patrol the area around their initial egg. Every so often an Elderling can lay another egg.
  • Able to be allied to player/ai/minor faction.
  • Thanks to Badger for adding!

New Map Type For DLC1!

• DLC1 has just not been popular enough with folks, so we're deciding to sweeten the deal and add some cool new things to it. You'll see some minor new factions, some notable cool new features, new units, and a new map type, among other things.
  • We're going to be renaming that DLC in order to make it more clear that this is a majorly expanded DLC compared to what it originally was, and when DLC3 comes out, we'll also be raising the price of DLC1 to match that of DLCs 2 and 3. Essentially, they're all going to be suitably giant to match DLC2. So if you want to get it for a couple of bucks cheaper, now is a good time actually!
  • In the meantime, the new map type has been added to DLC1, courtesy of Badger: Classic.
    • Practically, it's a mix of bunch of clusters and lonely planets with lots of connections; it has a bit of flavor of the maps from AI War Classic.

Bugfixes And General Improvements

• Use AddedMarkLevelsForFleet_FromScience rather than CurrentMarkLevel to determine mark of Spire ships.
  • Thanks to Tom for updating!

• Create second spire fleet at first Mark 3 city.
  • Previously, it would always spawn at the Capitol.
  • Thanks to Tom for updating!

• Include planet name in Spire City upgrade text.
  • Thanks to Tom for updating!

• Fix incorrect fleet name showing up in several tooltips.
  • In a number of places, it would always show the command station fleet of the current planet.
  • Thanks to Tom for fixing!

• Display the city each Spire Fleet ship line is bolstered from in the tooptip.
  • Thanks to Tom for adding!

• Fixed a place where I missed a fourth codepath for the "SaveGameData Create" log, which was what led to the log spam. I didn't actually forget to compile anything, looks like.
  • Thanks to Daniexpert for reporting.

• Fixed a bug in the dark zenith deserialization that was preventing savegames with them from being loaded.
  • Thanks to Daniexpert for reporting.

• Fixed a harmless but annoying typo in the logging for Zenith Architrave data.
  • Thanks to Daniexpert for reporting.

• Fixed another annoyance with log spam, this time with nomads spamming the log.
  • Thanks to Daniexpert for reporting.

• Territory dispute factions no longer clear their fireteams before calling FireteamBaseUtility.DeserializeFireteamsAndDiscardAnyExtraLeftovers, since that would lead to spotty data in MP.

• Fixed an exception (a boolean inversion) that was preventing any fireteams from being deserialized.
  • Thanks to Daniexpert for reporting.

• Put in more instrumentation into ChooseWaveTarget, so that if it has an exception it will now be more clear where that was, and also still potentially return something marginally useful.
  • Also then fixed an exception that could happen when validly reporting "no target from X location." The code was looking for an old-style answer that no longer is possible since these became pooled, but has been corrected to look for "valid no-data" answers in the new way.

• Fixed an exception that could happen in ShipVisualizer.RenderTarget.

• Necromancer bolstering seems to work. Big thanks to Tom for his fallen spire work that I copied.
  • Thanks to Badger for adding.

• Necromancer flagships are now better at seeing when they have hacks.

• Fix a bug where we were breaking out of the wrong loop in the hacking display code.
  • Thanks to Badger for fixing.

• Mummies now load into flagships.

• Fix a bug where each necropolis bolstering a fleet was clobbering the 'how many ships are granted' data.
  • Added a mention of this to the Fall Spire code in case someone else sees the same problem; I'm not investigating further for the fallen spire though

• Clear tooltips before clearing pools, when returning to the main menu.
  • Thanks to Tom for fixing.

• Make various functions in Window_InGameHoverEntityInfo take GameEntity_Squad instead of GameEntity_Base.
  • As it is, that is all they handle anyway, so make that clear from the signature.
  • Thanks to Tom for updating!

• Simplify some code in GalaxyViewSelector.cs around type conversions.
  • Thanks to Tom for updating!

Major Time-Based Pool Improvements

• Previously, the game used ITimeBasedPoolable and TimeBasedPoolable for any sort of pooled objects, even if they were going in the ConcurrentPool.
  • This was a bit confusing (potentially, anyway), and also limiting (definitely), since the two types of pools now have different needs.
  • So we now have IConcurrentPoolable and ConcurrentPoolable, which are used in the vast majority of cases now, since mostly we use ConcurrentPools.

• Time-based pooling has been majorly expanded to now have a three-stage process.
  • There's a quarantine phase that is far more clear now, and at the end of that there's now an extra clear step rather than there only being "enter pool" and "exit pool" clears.
    • This lets us hang on to certain references that would otherwise cause errors at various parts of the application, while still easing the objects out of the simulation in other ways.
  • Additionally, the "advancement through buckets" for time-based pools now happens on a background worker thread instead of the main or sim threads, which is WAY more efficient. This keeps the framerate and sim speeds up.
  • Also? It turns out that, all this time, time-based pooling has been running too fast, largely because I was trying to avoid excessive calls to take and put from the quarantine batches. This led to all sorts of anomalies.
    • Now that I'm running the time pool logic on a new random background worker thread, we can afford the extra time to make sure it's done right.
    • In general, this particular thing also is not quite such a large load as I'd once feared it might be (then again, I have the luxury of saying that when it's not blocking the sim, right?)

• The squad pool was set to be stupidly long (30 increments of 12 seconds each, wow).
  • It is now set to be 3 increments of 20 seconds, which is a lot more efficient.

• Shots, transported-ships, fleet-memberships, and external and base infos have all had basic translations over to the new time-based pooling for maximum efficiency and minimum chance of errors.

• Squads/ships have had the largest transition, and a FAR larger amount of their stuff now happens on the new Mid phase.
  • This both speeds up the deaths and reuse of squads (major boon on many threads), but also helps avoid lots of random errors that could otherwise happen in LRP threads or in the UI, particularly notifications.
  • While in this part of the code, I also discovered that there was a leak with the external baseinfo/deepinfo for squads, and I fixed that. Whoops.

• Fireteam history is no longer in a time-based pool. Because it's a sub-object of fireteams, it makes more sense to just be a concurrentpool, which is how we handle systems on squads.

• Fireteams themselves were updated like shots, fleets, etc, were.

• Squads now have a PermaDebugNonSimID on them, which never changes and which has no sim-meaning. It's different between the same units on different computers, and between different runs of the same game on one computer, rather like a hashcode for many objects in C#.
  • We can use this to trace the life of a literal squad object as it changes over time, if we need to, even as its identity is rewritten around itself.
  • This has been used to verify that the timings on the time-based pooling actually finally work properly for apparently the first time ever.

• Thanks to Tom for the discussion that led to this change in how we do things with time-based pools, as this will make a LOT of code way less prone to random exceptions.
  • As he points out, this is a little bit similar to RCU, which is a neat coincidence but a great way to think of it it: https://www.kernel.org/doc/html/latest/RCU/whatisRCU.html

Memory Leak And Performance Work (Ongoing Part 2)

• Added the ability for our dictionaries to expand in a non-doubling fashion, same as we have for our lists. It's not used very often, but in certain cases it can make for more efficient memory usage.

• My CrossThreadSwizzleLists for ship-to-ship beams, etc, now use a lot less memory and will also consequently no longer even try to draw more than 9000 beams of a single type in one frame.
  • If more than that are requested, it just ignores the requests beyond that number. We've never had anywhere close to that number requested at once, and it would tank pretty much any GPU, so this makes sense. The limit before was 90,000, and there was no safeguard if it was exceeded, because it could not feasibly ever be reached.

• I'm making many various other little updates to collection capacities and capacity growth curves as I go through Daniexpert's core dump. It's not worth noting them all, but it's just little tunings here and there that make things work better.

• I decided to take a break and do a manual review of all of the classes in ArcenUniversal, of which there are apparently 464 at the moment.
  • I reviewed them all manually to see if they were suspicious or what, particularly paying attention to things in the realm of UI. For those that were suspicious, I've added some instrumentation so that we'll see in the future if they leak.
  • For four others, I find them worthy of further investigation, and I've just noted them down for now.

• Removed the PlanetFactions list on the Faction object, as it turns out it was never filled and also never used (thankfully). The PlanetFactionEntityLists list is what is actually used, and that's been left alone.

• Made it so that parallel processing of planets is a lot more efficient in terms of both CPU usage and RAM usage.
  • I had been making a working copy to iterate over, and there was a much better way to handle that, sigh.

• The Queue class also now supports non-doubling increases in capacity, like our List class.

• Lots more various improvements in how various collections allocate their capacity.

• TemporaryBuffer has been removed from ArcenCharacterBuffer, because it was not threadsafe. Also it was not even used in this game!

• ArcenCharacterBuffer can no longer be directly created, but instead is borrowed from a pool and then expected to be put back.
  • This is one of the largest memory leaks that we definitively still had, and it is now fixed.
  • Any time we were writing a key combo to the screen it was a major leak of these, turns out.

• ArcenXMLSerializationBuffer now is created, used for a while, and then disposed, versus being kept for the long-term.
  • We're no longer doing reference tracking on these, since they don't have any data unique to themselves. Everything is part of the ArcenCharacterBuffer except a thin wrapper here around it.
  • In fact, these are now structs, since they contain so little data.

• A whole bunch of debug and tracing output code has been updated to use the updated ArcenCharacterBuffers that are pooled.
  • The method now is more efficient, and also more robust against small typos.
  • For example, it turns out that there was an infinite-appending memory leak in code that was meant to be just for tracing in CheckForWarFooting for Zenith Architrave, but instead it was leaking infinite strings over time even when tracing was off.
    • There were some other examples like this, but this was the most notable one where it looks like it would cause gradual memory degradation over time until the game is restarted (or until actual tracing was turned on for the ZA).
    • In the event of something along these lines in the future, we'll see one of two things: 1) an exception (yay!), or 2) a rising number of ArcenCharacterBuffers being allocated and never put back. Prior to today, it was just invisible ram usage since we're not keeping track of "excessively long strings" or something of that sort.
      • That said, in the normal case of operations (aka not tracing something), it should just show up as an exception, which is the best result. Leaking character buffers would likely only happen during tracing, if it were going to from tracing activities.
  • It's vaguely possible that there are some leaks of character buffers when doing tracing. Given that this is only something developers or modders are likely to do, and only for brief times, this does not seem to be a big deal. However, there also should be a minimal number of these (it was a lot of work).
    • Actually, despite all my work, it's pretty likely there are some. At least they are minor.
  • It's also likely that there are now some random nullref exceptions scattered in the code that were previously infinitely-growing strings that memory leaked. That's unfortunate, but better than the alternative. I examined all of them by hand, but there were 1158 of them, so I had to skim kind of quickly.
    • Found some of those in planning wave (very rare), ZA (super hyper very very frequent), templars (minor), dark spire (moderate), scourge (very frequent), dark zenith (very frequent), base-oriented warden (minor),

• Fixed several tooltips in the resource bar and the bottom left menu that would leak memory (and ArcenCharacterBuffers) like crazy when you hovered over them.

• If a blank or null string is passed to SingleLineQuickDebug or ArcenDebugLogSingleLine, these are now ignored.
  • This is useful to have more brief code with certain tracing code. Sometimes it would accidentally be sending something blank, and that's fine and should just be easily ignored.

Beta 3.741 Clogging Holes

(Released October 28th, 2021)

• Fixed two typos related to Necromancer

Memory Leak And Performance Work (Ongoing)

• Added a new ProtectedArray collection, which simply lets us have automation with things like items going back into the pool, etc.

• The MarkLevelStats on GameEntityTypeData is now pooled, and uses the ProtectedArray, so that when the game reloads all its xml it should now also properly flush this out.
  • Same deal with EntitySystemTypeData itself, and MarkLevelStats on EntitySystemTypeData.
  • And also HackData, on GameEntityTypeData.

• Added a new ProtectedValue "collection" (of exactly one item), which again gives us that automated protection.
  • This is now used on the EntityTypeDrawingBag references on GameEntityTypeData so that they properly get pooled when there's a reload.
  • Please note! We're using this on things like PeriodicSpawn_EntityTypeDrawingBag because this IS the data, and so we want to make sure and clear it as well as the reference to it. So we DO use ProtectedValue.
  • Meanwhile, for things like UnitToMakeWithBuildPointsFromDamageTaken, that's just a reference to data that exists independently elsewhere. We'd want to clear the reference to it, sure, but actually cleaning up the distant data is not our call on GameEntityTypeData; it's not in charge of that data. So it would NOT use ProtectedValue.
  • More often than not, the proper decision is to NOT use ProtectedValue.

• Added better error handling for "DoForAllElementsIncludingParentsAndPartialsInForwardOrderFromOldest"
  • If it has an exception, it now tells you what the sub-node was that it was looking at, and also doesn't stop processing of the rest of its row (thus preventing an error cascade that gives misleading errors that would go away if you fix the actual core error).

• Working on memory leak hunting:
  • ArcenDynamicTableIndexSerializationInfo allocation capacities are now more tame. Also, these were leaking, it turns out, so they are now pooled and reused instead.
    • Same with ArcenNonTableIndexSerializationInfo on all fronts.
  • Thanks to Daniexpert for the core dump that led to these.

• Yesterday I split ArbitaryArcenOption so that it was an abstract base class and then also had an ArbitaryArcenOptionStandalone sub-class for being used.
  • It turns out that this was a terrible idea, and on top of that I had some other classes that were inheriting from ArbitaryArcenOption in a way that was really not helpful.
  • So, I'm combining ArbitaryArcenOptionStandalone back into ArbitaryArcenOption, and then making that class sealed (so nothing can inherit from it, because that's "crossing the streams" and gives very bad behavior).
  • Faction configurations and galaxy options continue to now use ArbitaryArcenOption directly, and it works as expected.
  • CampaignOrQuickstartFolder, CampaignOrQuickstart_SubFolder, and SaveGameData all now directly inherit from ArcenOwnedRowBase, which is the root behind both ArbitaryArcenOption and ArcenDynamicTableRow.
    • They also then implement IOption, that excellent idea that Tom suggested yesterday, and pool themselves without any worry about strange inheritance patterns.
    • I've also made these sealed, so nobody is ever tempted to inherit from them and cause the same problem again, just at a new inheritance depth.
  • These are verified to no longer be leaking, but these are definitely not the largest part of the leak that is in the core dump from Daniexpert. I haven't gotten to that yet, but I'm going through methodically. I've also fixed up a number of capacities to be more sensible based on seeing real-world core-dump data, and the savegames now load faster than ever, especially on second load and beyond.
  • Thanks to Daniexpert for the core dumps for this and other related items, and to Tom for the IOption idea.

Beta 3.740 Code Panopticon

(Released October 27th, 2021)

• Lost Spire Frigates now have "Roam if instructed" set by default
  • Thanks to Wuffell_1, TheIronBird and all the others for suggesting, and to Daniexpert for fixing!

• Add some new integers, StoredFactionResource[One|Two|Three] to the faction object. This is for future modding support as well as the necromancer.
  • The Necromancer now treats FactionResourceOne as "Essence", which is obtained by hacking Rifts.
  • Essence is used to do all necropolis and flagship operations, which are spent through the Hacking sidebar.
  • Thanks to Badger for adding.

• DZ Svikari: Support AI allied variant. Currently only in DLC3 for testing
  • Thanks to Badger for adding.

• Allow stationary flagships to travel to specific planet locations with modifier.
  • Thanks to Tom for updating!

• Add support for autobuilding forcefields.
  • While we are here, refactor the autobuild code somewhat, so we only loop over the command stations once, rather once per type of automation.
  • This could probably do with being somewhat more data driven, though.
  • Thanks to Tom for adding!

UI Updates

• The planet sidebar menu now shows the actual hull points of crippled watched/local flagships, so you can easily monitor their repairs. It still includes the text 'Crippled' though, so players should not get confused.
  • Thanks to Badger for updating.

NA Balance changes

• Magnet ward starts at mark 1 (instead of mark 5).
• Sabot ward now fires in a burst of 3
• Angelic ward strength multiplier reduced from 5 to 3.
  • Thanks to Democracy! for pointing out this issue!

ZO Balance Changes

• Cruisers stat adjustments. Minimium armor thickness from 50 mm - 175mm. This fixes several cruisers, most notably the starfish, which was vulnerable to all sources of weapon jamming.
  • Thanks to Democracy! for pointing out this issue!

Bugfixes

• Fixed a missing XML field in the Brutal Guardian Lair, causing it to complain about not finding any Brutal Guardians.
  • Thanks to Badger for reporting.

• Fix loading NecromancerUpgrade table when DLC3 isn't available.
  • Thanks to Tom for fixing!

• Mark orbital guard posts ship groups with `can_be_empty_or_filled_and_no_complaints`.
  • This allows them to be loaded without error without DLC3.
  • Thanks to Tom for fixing!

• Fix some map description typos for Swirl and Bubbles.
  • Thanks to Tom for fixing!

• Make sure that deepstrike eligibility is updated even when there are no deep strikes.
  • This was causing deepstrikes to never happen, as no planets would be declared eligible.
  • Thanks to Tom for fixing!

• Save the name of PlayerCustomCity fleets. This avoids Spire City Fleets from having generic names after loading a save game. This breaks the savegame format for games with Spire Cities in them.
  • Thanks to Tom for fixing!

• Clamp maximum distance to point in BadgerUtilityMethods::GetExtremelySafePointNearPoint to 5 times the desired distance.
  • If the distance the point can be placed at is not limited, it is possible to end up with maps where one or several satellite planets (in the X or Linked Rings map types), are an order of magnitude or more farther away then the rest of the planets in the galaxy.
  • Thanks to Tom for fixing, and cml, Wuffell_1, and Ecthelon for reporting.

• Clear the savegame selection when deleting a save in the load menu.
  • This prevents errors when clickling the `Start` or `Lobby` buttons afterwards.
  • Thanks to Tom for fixing!

• Don't randomize the number of spokes in a perfectly symmetric X map.
  • Thanks to Tom for fixing!

• Achievements for the DZ now properly occur for the DZ instead of the ZA
  • Thanks to Tom for pointing the ZA were stealing the DZ's achievements, and zeus for fixing!

• Count the number of planets in X map subtrees correctly.
  • Always divide extra planets among all children.
  • Ensure than main branches of symmetrical X map have the same number of planets.
  • Thanks to Tom for fixing!

• Don't error when trying to get `NumberToSeed` faction configuration when chekcing achievements.
  • In particular, the Hunter Fleet doesn't have it, so checking for completion of "Kites Get Struck By Lightning, You Know" was causing an exception after you won the game.
  • Thanks to Tom for fixing!

• Convert more places that use UniqueTypeDataDifferentiatorForDuplicates to use int.
  • The field was converted to an int in r14429, but there were a bunch more places that used or accpeted this value that needed to be changed. I think I've got all the remaining ones.
  • Thanks to Tom for fixing!

• Add FleetCategory.GetCanHaveDuplicateLines to track whether fleets can have duplicate lines.
  • This is a followup to r14429, which changed how Spire fleets work. They use `FleetCategory.PlayerCustomCityFedMobile` fleets, and each city provides ships with a `UniqueTypeDataDifferentiatorForDuplicates` corresponding to the city's `FleetID`. Thus, that category of fleet needs to support the `*_WithUniqueIDForDuplicates` method variants. The new method is used to control this.
  • Thanks to Tom for adding.

• Move the logic for per-city frigates to be attached to the hub.
  • I added a new tag `SpireFeedsFleet` to all Spire building that provide ship capacity, and changed the Spire Hub and Capitol to provide the per-city frigate.
  • The old logic was incorrect, since the per-city frigate wasn't additive withfrigates from other structures.
  • Thanks to Tom for fixing.

• Calculate the stats of PlayerCustomCity fleets as player fleets.
  • In particular, this calculates MaxMarkLevelOfAnyInFleet, which is used to determine Spire Fleet strength.
  • Thanks to Tom for fixing.

• Remove duplicate centerpiece upgrading for Spire Fleets.
  • There were two copies of this code, one from before splitting fleets, which recorded a journal entry, and one after, which didn't. I've removed the former, along with the journal entries, as they were now incorrect, and somewhat redundant with the city mark level journal entries which are granted at the same time.
  • Thanks to Tom for fixing.

• Mark Spire Hubs as granting science upgrades to the fleet.
  • Though they can't be upgraded with science by the player, the city upgrade uses science to indicate the mark level of the fleet.
  • Thanks to Tom for fixing.

• For PlayerCustomCityFedMobile fleets, use the mark from the bolstering city.
  • Thanks to Tom for fixing.

• Don't grant science to non-centerpice ships in Spire Fleets.
  • Now that ships in spire fleets get mark from the cities they are fed from, we should not grant them from the fleet mark as well.
  • Thanks to Tom for fixing.

• Use "science" to upgrade Spire Fleet Flagships.
  • The code previously called SetCurrentMarkLevel, but that only changes a transient value that gets overwritten on the next update.
  • Thanks to Tom for fixing.

• Renable building Spire City structures.
  • Thanks to Tom for fixing!

• Make spire cities automaticly bolster nearest fleet fleet.
  • This is at least partly a stop-gap until it is possible to change which fleet is being bolstered by a city.
  • Thanks to Tom for fixing!

• Upgraded the engine to Unity 2020.3.21f1, from the prior version 2020.3.20f1 we had been using.
  • This is a pretty fast turnaround on engine version upgrades for us, but they fixed a memory leak that affected OSX players.
  • It seems likely that we've fixed our own memory leaks that affected all platforms, but at any rate let's not have that confused with an engine-level one.

• Fix isGalaxyFullyConnected to not sometimes report a false negative.
  • The previous version tried to save allocations by having the caller pass in a reference to a list of planets to use for tracking a connected component. However, since it also need to pass the list to a delgate, it needed to create a local copy of the parameter. It looks like there was some confusion between the two variables that lead to the wrong list being used.
  • I changed it to take the list of connected planets as a regular parameter, rather than a `ref` parameter[1]. If the caller (i.e. `makeSureGalaxyIsFullyConnected`) want access to a connected component, or to reuse the list, it can allocate and pass in a list. However, if the caller (i.e. `removeSomeLinksBetweenPlanets`) doesn't care about either of those things, it can just pass in `null`.
  • I also added some documentation for the method, which explains the usage of the `connectedPlanets`, along with caveat about the behavior if it isn't already connected.
  • Thanks to Tom for fixing!

• Fixed several cases where we could get spam of "Look rotation viewing vector is zero" invisibly in the player log, which led to memory rising as well as choppiness.

• Fix a couple of errors in ArcenUniversal.List where passed in Lists were not initialized.
  • Thanks to Tom for fixing!

• Fix a divide by zero in the Anti-AI Zombie patrol code.
  • Thanks to Tom for fixing!

The Code Panopticon

• Added a new form of instrumentation where we can track all collections and see if things are leaking via successive dumps, which includes finding out WHERE something is dumping from.
  • This does require folks to specify what the use of their stuff is, and it also makes it so that (for the time beinng, when the logging is on), no collections are ever garbage-collected.
  • Honestly, I don't trust the garbage collector to do a good job in the first place. "Temp collections" should be using ThreadStatic, not just created and then dropped.
  • This does require code changes in a variety of places, including in code mods, but this is kind of the ultimate in memory leak detection and repair. It also forces programmers to think a bit about what sort of data they're just tossing away. At best, it's something that adds slight hiccups from the garbage collector, and at worst it's a memory leak.
    • Please bear in mind that I'm concerned not just with the memory leak we have right now (we had one in July, which was different and I fixed it, for example), but any future ones, including client-only ones in MP, and mod-specific ones. Having a true solution that lets us detect them -- in fact, lets a modder detect them -- is something that's important to longevity.

• We had several methods for profiling the performance of the game, added over the years because the unity profiler is not great for multithreaded games (it is getting better, but still kinda stinks).
  • Some of these were potentially a minor performance drain, and we don't even use them anymore, so they've been removed. The StopwatchSegment stuff was the oldest one, and is now gone.

• The collections in general, and some other classes that depend on them, now require you to pass in a string LocationNameForTracing,
  • One of MANY places where this affects things is also the ArcenDoubleCharacterBuffer and similar, which I've known for a while we have been churning out like crazy.
  • These are being abused all over the place, sometimes just through casual usage ("this is infrequent and won't matter much"), and other times by accident (a sub-object that gets lost when an outer collections of collections is cleared).
    • In that second case, overall my goal is to have data structures for those types of collections of collections so that the internal collections get pooled within the outer one, thus keeping code simple.
  • The downside for programmers is that they need to give a name to a lot of things that they did not previously have to. The upside is that this helps us realize "hey, if I'm naming this, I'd probably better not throw it away, because that might be a memory leak." And the super duper upside is that when we do wind up with a memory leak, we can see it, by name, and then just go directly where it is and fix it.
    • I originally tried to use stack traces to accomplish this same thing, but that make loading the game take 90 seconds instead of 8 seconds on my machine (ow), and loading a savegame increased from 2 seconds to 260 seconds. So that obviously wasn't a workable approach.

• A new DictionaryOfDictionaries has been added, which works a lot like the DictionaryOfLists in that it has internal pooling to prevent leaks that otherwise were rampant when programmers used this sort of structural design.
  • The toFromPlanetCache now uses this, which gets rid of one notable potential memory leak.

• Added a new ProtectedListArray, which is less critical but does prevent some accidents with a ProtectedList[]. It also makes that outer code simpler, like these other compound collections, which is good.
  • Same deal with a new ListArray.

• For the sake of brevity and ease of use, ArcenRandomDrawBag is now just DrawBag.

• Added a new DictionaryOfDrawBags, again for the same sort of "don't leak a collection" purposes.

• Added a new DoFor() method on EnumIndexedArray.

• Added a new EnumIndexedArrayOfLists, again for anti-leak purposes.

• Added a new EnumIndexedArrayOfArcenOverLinkedLists, again anti-leak compound data structure.

• Now we also have EnumIndexedArrayOfProtectedKeyDictionaries.

• Interesting, a new ConcurrentDictionaryOfLists. Had not expected to need that, but it's useful.

• Also added a new DictionaryOfDoubleBufferedLists, which is mostly used for some logic on the macrophage, but it's vastly more flexible and effient than what I had set up previously to run them.

• SortedDictionaryOfSortedDictionaries has been added, and this may be replacing one of the areas where a larger leak was possible.
  • Interestingly, in adding this and then making that in use throughout ShipListerUtils, I found something that looked like it would lead to incorrect counts of ships in some circumstances. We've had sporadic reports of that, so this may be solved now.

• Added DictionaryOfArrays, which I don't actually need right now, but I figured why not. Because...
  • Added SortedDictionaryOfArrays, which is slightly more complicated but based on the above one, and this one I actually do need.

• Fixed various random issues that were discovered because of the above changes requiring lots of code review:
  • There was a code path that cause ArcenRandomDrawBag to not actually initialize its tracker properly.
  • Fixed up the ArcenLessLinkedList and ArcenOverLinkedList to use structs for their enumerators, like the internal official classes do. Way more efficient.
  • GetRange() and FindAll() on the standard List class now work in a way that does not generate any garbage, and is faster to run in general. This requires you to pass in a list when using them.
  • Internally improved the performance of the ConcurrentBag that Microsoft provided, making it so that it doesn't need to generate lists and garbage when it is stealing items from one thread's pool to another.
    • Also improved its performance in the same way for when it needs to do an enumeration over a list that happens to be empty right now.
  • Improved the performance of my own logging from background threads to the main thread. It was generating some needless garbage and cost.
  • Huh, interesting, it was possible to really improve the ConcurrentBag and ConcurrentQueue enumerations and CopyTo and so on even more.
  • Made the xml import a LOT more efficient because of how we handle creation of working lists and sub-lists and things. This one is definitely on me in terms of why it was not as efficient as ideal.
    • A new FillList method that is very efficient is now on ConcurrentDrawBag and saves a lot of locks and list allocations and so on that we had to do without it.
  • Now using DictionaryOfLists in xml import. And for input actions by category. And music tracks. And RowsByTag and RowsByPlanetSeedingType.
  • Now initializing a far smaller list of sub-collections as we import various xml elements and attributes. Yay efficiency!
  • When dropdowns change what they are showing, there is more efficiency there, too.
  • A few places where we use RefPair in Core have been changed over to KeyValuePair, which is more efficient. We had very few instances of using RefPair in general, outside of the Draw Bags where it is actually needed in the form that it is.
  • Various more cases where we now use the compound classes.
  • Fixed up the templars so that they properly clear their data now and don't leak any data. I had marked that with a data_todo but hadn't gone back to them since then.
  • Fixed a possible memory (substantial) leak in AOE attacks, although it's not for sure that was happening. Might be a false alarm, but either way it is good now.
  • Noted that ExoOptions, which were set up as a struct, had reference types (lists) as part of them, which kind of defeated the point of these being structs. It does cut down on the amount of gc being hit, but still has a notable hit. These are now pooled and tracked; these always get sent to SendExoGalacticAttack, so that properly returns them to the pool when it is done with them (or earlier, when it early-outs on an MP-client).
  • Certain methods, like GetPlanetsForExtragalacticBudget(), no longer return a list of things. Instead, they take in a parameter that is a list-to-fill, and they do that. This pattern is required in places like this (it's always a judgement call) so that multiple callers can work with the same method and not have scrambled results. ThreadStatic is another way to go, but then you run into problems with serial calls to the method on the same thread. So since this a return list that we don't know how long will be kept or used, it's better to let the list be something that we don't originate in a method like that.
  • There was some funky logic that was hard to follow with a few collections being passed around from waves to notifiers and being used for potentially too long. It probably was not a problem, but now it definitely is not.
  • Got the sapper unit data so that it properly cleans itself up. Here again I had left a note to myself, but not gone back to it yet.
  • A bunch of common game commands, like wormhole movement and regular ship movement, are now at worst more efficient, and at best also no longer leak memory.
  • GetOptions() no longer returns a list of options

• Most references to Linq have been removed from ArcenUniversal.
  • Linq is inherently messy and create a lot of garbage and is slow and is not something to be used very frequently. I discovered this back in 2008, with the original AI War, and have maintained that position ever since.
  • That said, for our ObjectDumper, DiffLib, and a few other key things that are infrequently-used, we can get away with some limited use of it.

• Added a new BetweenXmlReloadPool, which provides a new way for us to more efficiently hang on to certain data between reloads of the xml (reloads of the xml only happen when you change around your enabled expansions or mods, but it's still good to do).
  • Previously I was relying on certain types of data being garbage collected, but now I'm not going to trust that.
  • For now this is just used on MapSettingOption and MapSettingOptionChoice, but later it can be extended to table rows and similar.
  • There is a new enum that is used for the ConcurrentPool and the TimeBasedPool which specifies that things either will reload when the game is cleared (like going to the main menu, mapgen, etc), or when xml is reloaded, or not at all.

• ArbitaryArcenOption is now pooled between xml loads. OptionWrapper also. Okay... the big one is ArcenDynamicTableRow, and it also now complies with this. This is going to be a pain in my rear, but there's just no other way to avoid memory leaks on xml reloads. How severe those would be is hard to be certain, but it's not great, so here we are.

• CreateNewForPool() has been removed from the various pools. They now instead take a GetNewEntryFromAny<TValue> delegate, which is vastly more flexible and makes certain kinds of inheritance possible for the first time. This was the big blocker behind dynamic table rows.
  • Okay... to handle dynamic table rows, we really need to have something more automated, even it's more slow. This is just during xml reload, not during gameplay, so even if it's 10x slower it's still going to be plenty fast. During gameplay we need all the speed we can get, so we don't automate things.
    • This has led to a reflection-based approach that is able to copy all of the values of an uninitialized dynamic table row of a certain type (like GameEntityTypeData) and set them to their defaults. The idea is to kind of brute force this.
  • There's a new ArcenTypeAnalyzer class that we can now use for looking at a type and making a blank version and then applying the defaults from it to other copies of it.

• EntityMetalFlowEntry has been converted from a class to a struct, using the better patterns that we have established for this sort of thing.
  • There was some confusion on my part in discord a few weeks ago, and I had to really re-learn how to properly use structs in certain classes of cases, but actually the microsoft collection enumeraters were a big help in showing big practices (and their KeyValuePair). This now follows those best practices.
  • IncreasingPrep has also gotten the same treatment.

• Everything that inherits from ArcenDynamicTableRow is now forced to use a certain flow for creation that uses a private constructor and either uses a pool (normal case) or explicitly and clearly opts not to use a pool.
  • Previously it was really easy to accidentally wind up with multiple ways to create from these, some pool-based and others not.

• A couple of memory leaks that were in various dropdowns, most notably in the Galaxy Options and Personal Settings windows, are now fixed.
  • It's unclear if these were true memory leaks in the sense that memory would never be given back, or if it was just a performance drain and hit to the GC. Either way it's fixed, and this is not likely to be the smoking gun of the main leak.
  • Actually, these particular windows were really just full to the brim of stuff like that, but at this point they have been restructured so that they now give compiler errors if someone tries to do the sort of things that we were previoulsy having problems with. A lot of this was I think my code, but from a long time ago.

• PlanetWaveComposition has been converted from a struct to a class, and is now pooled. This is another one where it was leaking pretty fiercely, but exactly how much of a problem it used to be is hard to tell.
  • If I had to guess as to a main source of a leak from what I've seen so far, this is a high candidate, at least. Between this and some of the stuff in the beam weapon attacks, those seem like the potential worst offenders I've seen thus far.
  • Added a new ProtectedKeyList to help with some of the collections affected here.

• Added a new DictionaryOfSortedDictionaries, which is used mainly for distant ships to be brought to the local planet.

• All of the savegame and formatting code has been split out into their own files. The savegame stuff was not being pooled previously, but now is. Note that I'm referring here to representations of savegames on disk, not the actual contents of them.

• In general, the number of ArcenDoubleCharacterBuffers has been dramatically reduced by making a lot of them static rather than instance variables. Other similar changes have also been made that help keep RAM usage lower and CPU usage more efficient.

• Fixed a couple of places in the squad visualizer that were initializing absolutely insanely huge arrays. This was not a "memory leak" per se, but it was allocating two orders of magnitude more RAM than it should have been, which would wind up looking like a memory leak.
  • If this is what was causing the memory leak, that will be quite interesting. This is something that I accidentally introduced in the last few weeks, either way. I'll need to improve my tooling in general, because this particular thing would not have been detected by my memory leak detection code.

• Added a new GetTotalCurrentCapacityForItemsForTracing() on TraceableCollectionBase, because the more I thought about this since last night, it's entirely possible that this entire "memory leak" (this time) was merely a matter of over-allocating to a frankly insane degree on each ship in a squad.
  • This only applies to ships that are being visualized, in the example yesterday. But let's look at the numbers. The ShipsLiving on each individual squad object visible in the game was allocating a list of 250,000 in size for ShipsLiving, and 10,000 in size for ShipsDying. These are good numbers for TOTAL ship counts, not per-squad.
    • I should mention that we still treat ships as being in squads, but just a squad of one, at the visual layer. This is minorly helpful, but mostly a remnant of past designs.
    • So, we have only as many squads as we need to draw on screen at one time, when it comes to these particular objects. This is just a visualization layer thing, not part of the simulation (where the core squad objects exist).
    • If we have a battle where there are 8,000 ships on screen at once, then that would mean 8,000 of these objects. Normally not a problem at all. Let's do the math here.
    • We'll simplify a bit, but internal to a list is an array. We're on 64bit systems, so each reference is 8 bytes. Even the unfilled references use 8 bytes. An actual object would be way more than that, but that's fine. There's a few other heads and things, but those always exist, so let's ignore those.
      • That gives us (8000 * 8 * 250000) + (8000 * 8 * 10000). In other words, 16,000,000,000 + 640,000,000, or 16,640,000,000 bytes. So that's 16,250,000 kilobytes, which means 15,869 megabytes. That's... 15.49 gigabytes. Yeah, that's the memory leak all right. Not a leak at all, just an insane over-allocation.
    • With that in mind, I'm going to be monitoring the over-allocations as well as for actual leaks.

• Some of the calculations for fireteams and budgets now do their work a lot more efficiently in terms of reusing existing collections instead of creating a bunch of new ones each cycle and then having to garbage collect them.

• Added a new ThrowawayListCanMemLeak class, which is like List<> but not tracked in the same way (it's still tracked numerically, but not by reference), and able to be garbage collected. It includes the following comment in its header:
  • READ THIS BEFORE USE! This class is very very close to being like the general List<> class from Microsoft. We have our own implementation of List<>, though, and that is both more efficient and also NEVER GARBAGE COLLECTED. That's appropriate for almost all purposes we could have, because we really want to resuse collections.
  • Creating collections just to toss them away is a really bad idea... 99% of the time, anyway. The other 1% is why this ThrowawayListCanMemLeak<> exists. This is exactly the same List<> class, minus the tracking and a few other minor features, and plus the ability to be garbage collected.
  • Please use this sparingly, and probably mostly in mapgen. There's certain kinds of recursive code that you just can't do without local variables of some sort, and this allows for that sort of thing. If you're using it in MapGen, great. If you're using it ANYWHERE else, I hope you have thought things through and have a good reason why. If so, great!
  • If you're using it frequently during normal runtime operations, on the other hand, then you're probably not doing performance any favors. Consider a refactor in that case.

• The wasteful version of FindAll has been added to ThrowawayListCanMemLeak, since a bit of map code wanted that. Why not.

• ThrowawayListCanMemLeak is now used in most of the mapgen code.
  • There's a few bits that still use List, and there are yet other parts that now use IList -- not Microsoft's version, but an extended version of my own that allows for more functionality to be shared between the two types of lists.

• Put in a possible fix for isGalaxyFullyConnected not working the first time it was called, but have not had a chance to test it yet.
  • Thanks to Tom for alerting us to the issue.

• When exos are spawned against you during the fallen spire campaign, it now actually does the splitting of that into two exos, one on the relic, and the other on something else of value to you.
  • Previously it looked like that logic was partially there, but never actually fully completed.

• Improved the mapgen logic in a few ways where you don't have to pass in a list that won't be used, but instead can pass in null and it will still work fine.
  • Similar changes were also made to wave spawning logic. It gives more flexibility on getting things back, without having to declare unused lists.

• Improved some various faction utility methods so that they internally use threadstatic lists rather than requiring you to send in a managed list that will never be used for anything else.

• PlannedWaveOptions is now a pooled class, rather than being a struct. This is required because it has a dictionary on it.

• FireteamRegiment is also now pooled, and should always be stored in a ProtectedValDictionary so that it gets automatically put back into the pool when the dictionary is cleared.

• Added a new DictionaryOfDictionaryOfLists, which is our most-compound data structure yet in terms of the general ones. This one is mainly used by the outguard.
  • The compound data structures really are non-optional, if you're a modder and you're wondering. You're going to leak memory if you don't use them. If you need one that's not provided, then I can add it.
  • In general, though, compound data structures are sometimes not really as needed as they might seem. This is a good case of one where, yes, it really is actually needed.
  • That said, using a dictionary that has a dictionary of lists, in raw form, is going to leak both dictionaries and lists. So the compound structure that pools them is really non-optional as the way to handle it.
    • In the past, using Microsoft's version of these data structures, they would get picked up by the garbage collector (so we're talking a performance hit, not a memory leak). But my version of them are tracked permanently, so they will literally be a leak. That's part of the panopticon nature of all this. The code really needs to be able to observe itself in order for us to know when we're leaking or over-allocating. And we should be hitting the GC as little as possible (this is true for absolutely any game), anyway, for the smoothest end-user experience.

• Added a new AddRange_IgnoreNulls on List, so that we can pass in a collection/list that is potentially null and just have it ignore the null if it is. This simplifies some other code.

• The ArcenXmlImporter is now pooled. I had originally tried using ThreadStatic and static to handle this, but that turned out to be a disaster.

• Added a new DictionaryOfArcenOverLinkedLists, as this is really required for any sort of good performance on EntityCollection's EntitiesByType_OrNull.
  • The old way of doing this was likely a pretty solid performance drain, yikes. I'm still not in love with the way this is done as a general concept, but... it's fine enough for now, I suppose.

• We now use a LOT more of the [ThreadStatic] attribute throughout our codebase. But the downside with this is that it can be applied to non-static variables, in which case it... does nothing, and the variable works entirely incorrectly. I found a few cases of this (maybe three?) over the last week of reworking the game's code, and so had resolved to do a full review of all the code and see what I could find now, after the revamp.
  • It turns out that we have exactly 254 uses of [ThreadStatic] between our five main dlls, which is a curiously-significant number (being almost a power two, must have mystical power, right?). I had to review those all by hand, because the attribute goes on the line above the actual text, so I can't see if the line below it is declared properly directly in my find results screen.
  • Errors found out of those 254? ...zero. Go team, I guess! Good to have checked it, even if it was all fine. I did tweak the formatting on a few of them so that they are slightly more consistently placed in code.

• ListWithLogging has been removed, as it was simply not something that was very efficient, and I can now get a similar effect in the future with the main list if I truly ever need to.

• Fixed up a number of collections that inherited from CountedPoolBase to instead inherit from TraceableCollectionBase, since that's technically more accurate.

• Fixed dictionaries throwing exceptions when empty and I tried to dump them.

• For clarity, renamed ProtectedKeyList to ProtectedKeyValuePairList.

• The internals of most of our collections have been shifted around a bit to allow for them to report on what they have in them even better.
  • Additionally, when it comes to the compound collections, they now just report once at an outer level versus repeatedly over-reporting each sub-collection as well.
  • We're able to get a better picture of the information, with less duplicative information.

• Using the core data dumps, optimized the following collection capacities which were a bit too high:
  • ShipTypesThatThisBenefits on techs.
    • Also TagsList and TagsDict. These numbers seemed reasonable, but in practice did not turn out to be so.
  • CyclicalArrayPool turned out to be using as much as 380MB of space even early in the game, which was a huge waste. This is for drawing ships and such. And this would only get worse with time.
    • This is not a new problem, from the look of things, and I'm not sure how it was not more of a problem in the past, to be honest. At any rate, I've fixed this up so that now it uses VERY little RAM, mostly allocating in a lazy fashion as it goes, rather than preallocating way too much.
  • The dynamic table row counts also were using substantial size.
  • Map setting options and choices were also silly large.
  • Bundle cached objects, also.
  • Xml parsing working lists, too.
  • Some AIShipGroup collections.
  • The central pool list collections.
  • Between all of these various improvements, the game load time is back down into the 7 second timeframe for me (down from 10ish), despite the extra instrumentation.
    • Additionally, my starting RAM usage dropped from about 3GB to about 1.5GB.

• Improved the capacity reporting of several collections (for the core dump file) to do a better job of explaining what capacity they have actually have allocated in memory, versus what they would if they get called on later.

• Made the core dump report more brief in a few ways, more focused on the data I actually find useful (as it turns out).

• In the event of some issues with the cyclical array pools, the game now gives a warning rather than freaking out and having endless display issues. These should never happen in the first place, but due to some internal changes and testing, it started happening.

• Fixed a memory leak in our core dump code, irony of ironies.

• Fixed some arcen double character buffer gc churn from the ui, which in this new build had turned into a full-blown memory leak because of the structural changes around it.
  • This does solve a leak that was persistent and annoying in the past, and should lead to better performance than before. And in the new version, also it then doesn't memory leak on us.

• OptionWrapper, which was a pooled class, has been removed, and converted into IOption, which is way simpler.
  • One of the coolest things is that this uses explicit interface definitions (https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/interfaces#explicit-interface-member-implementations) to avoid cluttering up the classes that it is implemented on with seemingly-duplicative methods.
  • Thanks to Tom for figuring out how much better this approach would be, and suggesting it.

• ArcenSerializationLogger is now pooled, and thus in general is a lot more efficient. Before it was "just" hitting the GC, but there are several areas of code that should be picked up by the GC, but which from memory usage in general I can tell are not being picked up. Pooling it is for these ones.

• BufferTracker is now pooled and tracked. This was something else that was leaking on deserialization.

• FragmentedNetworkMessage is now pooled and tracked. This is good for large messages in multiplayer.

• ArcenDeserializationBufferModern is now pooled and tracked. This was a notable memory leak whenever a new savegame was loaded.
  • I still need to handle the serialization buffer as well as the arcen character buffer, since those seem to be leaking. And there's other stuff I need to instrument. But this is enough to get it closer to playable without too many leaks.
  • The Steam networking transport layer is now particularly more efficient because of this, too, as it turns out. GOG was already quite efficient in this area. This was because of my own methods of implementing them, not because of the inherent nature of either platform.

• More work will be needed to make this work fully correctly again, but various improvements have been made to the new "rows are pooled" logic when you make mod/dlc changes in the settings menu.
  • At the moment, you can use those settings, but you may have to restart the game after doing so. It seems like it will often let you do a reload a single time, but not twice. Or at least it's based around certain things. Disabling DLC3 works, but enabling it does not. Disabling DLC2 doesn't work, etc. I just have more work to do with them in general.

Press F7 To Core Dump

• The old "Dump External Hierarchy Data" hotkey has been removed, as it is being replaced with a newer and better thing that includes its info plus more.

• There's a new hotkey called "Do Core Dump", bound to F7 by default.
  • If you suspect you're having a memory leak, then use this and send the result to us. When pressed, it will output a bunch of data about the current state of the game into [InstallLocation]/GameData/CoreDumps/[Date][Time][Status].txt.
  • A large bulk of the data is what we were exporting before with the thing we just replaced, but over time we'll add to it. And in the meantime, this data has been made smaller as well as more readable.

Beta 3.712 Loading Hotfix

(Released October 15th, 2021)

• Add a new option for DLC3 for Enhanced Overlords. This is off by default for HA but enabled for Expert/Deathwish
  • With this enabled, as soon as you unleash the phase 2 overlord the AI gets a CPA against you.
  • Also, as long as the overlord phase 2 lives, the AI will launch exos and wormhole invasions against you.
  • Thanks to Badger for adding.

• Created the new EntityTypeDrawingBag (GEDB) and its logic:
  • This drawing bag combines drawing from:
    • Direct internal names of GameEntityTypData
    • Tags used by GameEntityTypeData
    • XML-defined AI ship groups
    • Categories of AI ship groups
    • The budget types the AI uses for its reinforcements and various sub-factions
    • XML-designed fleets
    • SpecialEntityTypes
  • They are defined as sub-nodes in XML in 3 categories:
    • entity_type_drawing_bag_used_on_death -> Spawns something when the entity this is on dies
    • entity_type_drawing_bag_used_on_periodic_spawning -> Spawns something as part of periodic spawning (i.e. Brutal Guardian Lairs)
    • entity_type_drawing_bag_used_on_spawn -> Spawns containing things similar to how guard posts work. This mechanic was created in addition to all the entity type drawing bag logic.
  • The following categories can be defined:
    • display_name -> This is used by tooltips and notifications as replacement for the built-in write-to-buffer method of the GEDB.
    • data_list -> This can be entity names, tags, the names of ship groups, fleet design templates, etc.
    • parsed_data_type -> This defines as what the data_list is parsed. Possible are:
      • Disabled (this is used to stop spawning something for entities copying or partial recording something)
      • EntityNames
      • Tags
      • AIShipGroups
      • AIShipGroupCategory
      • AIBudgetCategory
      • FleetDesignTemplates
      • SpecialEntityTypes
    • When using AIBudgetCategory above, ai_budget_subcategories may be defined. Possible are:
      • NormalAIShipGroup
      • GuardPostAIShipGroup
      • DireGuardPostAIShipGroup
      • UnarmedGuardPostAIShipGroup
      • TurretAIShipGroup
      • NonTurretDefenseAIShipGroup
      • GuardianAIShipGroup
      • WormholeSentinelAIShipGroup
      • ForcefieldGuardianAIShipGroup
      • DecloakerAIShipGroup
      • DireGuardianAIShipGroup
      • SingularFreakySurprisesAIShipGroup
      • ExoLeaderAIShipGroup
      • All
    • When using FleetDesignTemplates above, fleet_design_template_subcategories may be defined. Possible are:
      • Strike
      • Frigate
      • Turret
      • OtherDefense
      • Centerpiece
      • Civilian
      • All
    • spawn_count_type refers to how the min and max values for spawning are iterpreted. Note that for anything except for raw counts the results are rounded down. Possible are:
      • RawCount
      • AIBudget
      • Strength_BaseMark
      • Strength_CurrentMark
      • MetalCost
      • EnergyCost
    • Finally min_spawn_value_list and max_spawn_value_list define the actual values worked with.
  • By far not all values need to be defined, as they default, lists auto-adjust to inputs and things should mostly fall into place.
  • Vanilla, Expansion and Mod files have been updated to include these changes. As far as tested everything should work.
  • Thanks to NR SirLimbo for adding!

• Added the Brutal Carrier Guardian - This one carries now weapon but spawns with between 22.5 and 37.5 strength worth of Mk7 ships inside. The type of ship depends on what type of AI is spawning the carrier, but they use the same ship group category that guard posts are reinforced with.
  • Thanks to NR SirLimbo for adding!

• Add 'Adaptive Music' to the game. This is an Audio setting, defaulting to "Off"
  • When you are in a battle of reasonable size then the game will prefer to play Exciting music when it chooses the next track to play. This currently means 5 + AIP/20 strength on both sides. So at 100 AIP a battle with 10 strength on both sides will trigger combat. This is set in AudioAndSimilarHandler.
  • The goal is that you don't get really chill tracks playing during exciting battles, or up-tempo tracks during a refleet.
  • To set a track to "Exciting" add this field to the Music Definitions xml:

tags="Exciting"

• • Thanks to Badger for adding this feature, and to Puffin for identifying all the exciting tracks.

• When in an energy brownout, your factories will work at half-speed. Untested.
  • Thanks to Badger for adding!

Bugfixes

• Fix a bug with "LoadAllFoundXmlFilesIntoMemory debugStage 6000: System.Exception: Passed in Dictionary<> capacity less than or equal to zero. in the log" and similar preventing the game from loading.
  • Thanks to Tom for reporting.

• Templar defensive code.

Beta 3.711 More Fixes

(Released October 15th, 2021)

• Fixed some further bugs where fireteams could still get "preferred speeds" less than zero.
  • Thanks to BoBy for reporting.

• Could not duplicate the "GetStringValueForCustomFieldOrDefaultValue: The faction AI has 11 custom fields on it, but none of them were named 'StartingAILayout'" error, but have at least silenced it. In places where that error would have happened, it just treats it as "small clusters".
  • Thanks to BoBy for reporting.

• CreateExternalBaseInfo<T>() on Squads is now more efficient, on par with GetExternalBaseInfoAs<T>().
  • Same for the DeepInfo equivalents.
  • We do so many sanity-check versions of Create rather than Get that they really should be equal in performance. Basically, they now are, and they are also identical in performance save for one fact: if the data is missing when you call Create, it will add it. If it's missing when you call Get, it will error.
  • Thanks to Badger for inspiring this change.

• Updated the scourge to be less likely to throw errors, and to have better error-handling in SendUnitToUpgradeStructureIfPossible.
  • Thanks to BoBy for reporting.

• Updated our custom implementation of Dictionary to work a bit better, so that now when doing a foreach over them, it will have a compile-time error on invalid casts, rather than a runtime one. This is how it has always been before, but it was a bit trickier than it sounds.
  • It turns out we were only down to one last one of these anyhow, in the fleet management window. At any rate, that one is now fixed!
  • Thanks to Daniexpert for reporting.

• Improved the DictionaryOfLists substantially, so that it has more functions and also has an enumerator (so you can run foreach against it).
  • This is something that keeps the internal lists from having to be discarded by pooling them.

• Civilian Industries has been recompiled to probably work again with the new collections. It uses the DictionaryOfLists, making it the first part of the game to do that.
  • In general, a bunch of things that were previously creating collections and throwing them away very inefficiently now use threadstatic collections instead.

Beta 3.710 Hotfixes Serialization and Performance

(Released October 15th, 2021)

• Fixed a number of invalid cast exceptions and similar relating to the changes to SpeedGroup in the prior game. Essentially this made it so that you could not save, or really play all that long. Facepalm.
  • Thanks to Daniexpert for reporting.

• Added a new and more efficient method for TryRemove on ConcurrentDictionary that allows for multiple tries at once, and doesn't force you to take an out value.
  • If you only want it to try once, just pass in any number less than 2.

• The death registry now supports resurrections, heh. Units that are put into a transport are considered "dead," but when they are brought back out they are now considered brought back to life as their same self.
  • This fixes an error in the death registry code that would happen around transports, which I had anticipated yesterday and then got sidetracked into all that drone business and didn't actually test it.
  • Thanks to Badger and Daniexpert for reporting.

• In preparation for getting MP working again on the beta, the game now fast-blasts death registry data to the clients.
  • This includes removals and additions to the death registry, and slots them into the fast-blast order in the appropriate spot where we won't get death registry warnings.
  • For units being killed, this also proactively kills them on the client in the same way they died on the host, which is quite handy.

• The sidebars, and many other parts of the game that involve adding an unknown number of items, headers, and sub-items, all now populate themselves in a time-sliced manner. This means that when you first open the build tab, for instance, you may see a slight flicker as it fills out all its entries.
  • Previously, I used a bit of an opposite approach, and it was not a good idea. I was trying to pre-create a lot of extra items to prevent lag spikes, but this meant that there was one big lag spike (often lasting a few seconds) when you opened up those sidebars for the first time.
  • This makes the interface more responsive in the early game, and reduces some bits of CPU and RAM usage (both in relatively minor ways) without leading to later lag spikes.
  • Thanks to StarKelp for reminding us of how annoyingly slow these were to open.

• Fixed an issue with some of the new custom flagship types that made them unable to hack. This had broken the necromancer faction's ability to have their flagships hack, in particular.
  • Thanks to Daniexpert for reporting.

• Removed the constructor on my custom version of List<> which took an IEnumerable collection. This was inefficient to begin with in Microsoft's version, and generally used to work around things like the limitations of "can't be iterating over a dictionary and add or remove to/from it" (we now can do that with my version of dictionary).
  • In general, I am already planning on cracking down a lot on how collections are initialized and disposed for memory leak purposes in the very near future, so this is in line with my plans there, anyway.
  • In the short term, more to the point, there was a bug in my version of the list that took a collection as an initializer, and this was likely causing some issues in xml parsing being fully applied (not confirmed, but it seems likely), and also definitely was causing the necromancer hacks to come up blank, and may have also messed with some of the scourge logic and two map types.
  • Thanks to Daniexpert for reporting.

• Fixed some exceptions that could happen in DoOnAnyDeathInCombatLogic_AfterFullDeathOrPartOfStackDeath if the fleet had already been nulled out.
  • Thanks to Daniexpert for reporting.

Beta 3.709 Upgrades And Serialization Fix

(Released October 14th, 2021)

• For performance and accuracy reasons, the SquadWrapper has been retired, and everything that was using it is now using the LazyLoadSquadWrapper.
  • The lazy-load version does not require a secondary fill pass on deserialization. We've been having guards breaking ranks and becoming threat after savegame load recently, and this was most likely to do with them being released because of the post-process of the squad wrapper not being called properly. These now use lazy-load squad wrappers, which should protect against that.
  • To make really sure that I understand what is happening with our data flows with this, there's now a SquadWrapperStats class that we can set to be populated, and which then gets shown if you right-click the clock and look at the debug stats there.
  • There's still more I need to update with this, but this is a start.

• Adjusted the wormhole invasion LRP thread to run every 15 seconds instead of every 30.
  • Anything set over about 25 seconds will wind up showing inactive thread warnings to players on busy games. Anything 30 and up will show inactive warnings on EVERY game, periodically.
  • The inactive warning starts showing after 30 seconds of not running, and LRP threads may naturally start up to 4-5 seconds after their "don't run for this long" interval is over, just depending on how busy everything else is.
  • Most factions have an interval of 1-5 seconds, but for those that really don't need to happen frequently, 15 seconds is fine. If something doesn't need to happen frequently AND is super cheap to run, you could set it as high as 20, but there's not a lot of benefit to doing that since by definition it is quick to run.

• Speed groups now also use lazy load squad wrappers, and thus no longer require a post-processing pass after deserialization.

• Fixed a performance issue in gameplay for squad wrappers where things that were set to ID 0 (meaning empty) were able to try repeatedly to look in the central dictionary, uselessly.

• There is now a central "death registry" for squads that have died.
  • This was something I had been wanting to do for a while for multiplayer, to solve the ghosts issue once and for all in a different way, but it also has benefits in code protection on loading savegames in single-player or on the host.
  • Normally if a lazy load squad wrapper tries to look up an entity by ID in the central registry and cannot find it, then it marks itself as null and stops trying. This is... problematic in edge cases, and may contribute to strange behaviors like the golem guards turning aggressive.
    • Now if we come back with null from the central registry, we check the death registry. If the unit is properly marked as dead, then it's time to just do the above logic (yay!). However, if the unit is not in the death registry, we're going to wait 10 realtime seconds and keep checking the central registry as-needed. We may be waiting for data from the host, or we may be single-player and waiting for data to unpack fully from our savegame. Either way, if the data is simply incoming, we will wind up getting it. If it goes the full 10 seconds with no resolution, we now throw an error, because something is clearly amiss and needs to be looked into. This error won't mess up your save, but it will help us find out what is going on.

• Fixed a serialization bug from the last few versions that was related to some new death-effect data.
  • Additionally, while I was at it, made those serialize by index rather than name, since that will be more efficient given the nature of their data.
  • Note that for hack types we're still serializing by name, since that's more efficient there -- there are many hacks, and few used at once, which is not a good candidate for by-index. But death effects are few in type but used many at a time, which is a great candidate.
  • Thanks to Eluthena, HarryT, Badger, and Daniexpert for reporting.

• Both health_change_per_damage_dealt and health_change_by_max_health_divided_by_this_per_attack are now no longer on game entities but on individual weapon systems, which means it is now possible to design ships that have specific weapons that heal or damage them, instead of having to balance all weapons around that mechanic.
  • Updated code and tooltips to reflect this change. The health change based on max health so far didn't even appear in tooltips, it is now present named "Self-Assembly" (for healing) and "Disassembly" (self-damaging). If it's damaging itself so much that it'll insta-kill itself after just one attack it's named "Self-Destruct" instead.
  • Updated all existing normal and mod content to reflect these changes.
    • Tethuida Drones had a -100 health per damage dealt, which was most likely to mimic a self-destruct. But given something with high enough damage reduction they could actually fail, so they've been given health_change_by_max_health_divided_by_this_per_attack="-1" to make certain they always self-destruct on attacks.
    • Starbursts have an invisible melee weapon that causes them to move in close - however this also would take their health if it got to actually attack. Now it will no longer potentially do that.
    • Dark Splinters from the ESV mod no longer have the ability to self-destruct on the first revenge shot they fire. Only them crashing into enemies and doing damage does so.
    • There is a chance that I missed something due to copy and partial record changes. I was trying to be as thourough as possible - but chances are that I missed something. Please notify me if that happened.
      • At least all DLCs load without issue.
• More Brutal Guardian Work:
  • Reduced the size of the Brutal Amalgam Alpha's head and tail a bit, so they aren't as large as the actual thing. Also fixed the Tail's description.
  • Added the Brutal Amalgam Beta Guardian. A combination of Dire Vampire, Forcefield, Tesla and Gravity guardians. Even while cloaked it protects allies and disrupts enemy formations with its forcefield, siphoning health with each vampiric attack while also slowing enemy engines and weapons in range. On death it splits into a Vampire-Forcefield head and Tesla-Gravity tail. When those die they will turn into their respective Dire Guardians.
  • Thanks to SirLimbo for adding!

• Updated the tooltip for self-heal and self-damage to be a bit cleaner.
  • Thanks to CRCGamer for writing.

• Experimented with moving around Drones and transported ships, but that did NOT work out. The performance hit is pretty massive.
  • Also experimented around with only as-need initializing a bunch of collections on squad, but that adds a ton of confusing code, and I already have put in lazy-initialization INSIDE collections, so that was utterly pointless.
  • Overall, wasted several hours on this, and reverted all of it. Except the two bits below.

• Added a new DictionaryOfLists, which is a handy data structure we may or may not use.

• Added a few new List methods, like RemoveAndReturn and RemoveAtAndReturnLast, which make common operations a bit more efficient and also less code.

Upgrade To Unity 2020.3.20f1

• The game engine has been upgraded from Unity 2020.3.15f1 to Unity 2020.3.20f1. Improvements of note from Unity:
  • Graphics: Improvements to scene culling CPU performance - Shadow casters are rejected earlier if their shadow casting is disabled which should improve performance in scenes with scenes with lots of renderable objects that are set to not cast shadows. (This applies to AI War 2)
  • Graphics: D3D12 player will not crash/freeze when switching between full screen modes
  • UI: Backout a optimization of caching the change count where the layout is changed during a layout call. (Apparently this could cause UI scales to not be quite correct)
  • Graphics: Improve Line/TrailRenderer quality when points are close together.
  • Graphics: Reduced the Input delay in Vulkan standalone when VSync is enabled.
  • Graphics: Fixed a crash that was caused by a memory error when using Vulkan.
  • Windows: Fixed an issue for mouse deltas always being 0 when running Windows in a VM.
  • Graphics: Fixed FrameTimingManager produced incorrect frame time on Vulkan and DirectX12
  • Graphics: Reduced main thread hitching caused by Shader loading
  • Asset Bundles: Fixed Asset Bundle determinism (this makes it so that when we make updates, they won't require such large downloads on GOG and Steam)
  • Graphics: Fixed performance spikes in OSX while using an external monitor. Metal editor on OSX now uses CVDisplayLink to time frame presents.
  • Scripting: Fixed a performance regression when activating/deactivating panels in a canvas. (Yikes, probably)
  • Graphics: Fixed a crash when uploading large number of texture assets in DX12.
  • Graphics: Fixed a race condition deadlock when loading textures synchronously. (This was likely hitting us on game start; this was finally fixed on October 8th)

Beta 3.708 Bugfixes And Death Spawn

(Released October 13th, 2021)

• Update the some Tips about the wormhole invasion to explain the new mechanics.
  • Thanks to Badger for adding.

• Added the ability to spawn multiple entities on death.
  • The XML fields "spawns_on_death" and "count_to_spawn_on_death" are now both lists. They must be of equal size, spawns_on_death must contain the name of entities and count_to_spawn_on_death the amount of entities spawned for each type, which mus be 1 or greater.
  • Clearing these out will no longer work with the "None" keyword - instead simply empty out the lists (= spawns_on_death="" count_to_spawn_on_death="") if you want to prevent a copied entity type from spawning more stuff on death.
  • Thanks to NR SirLimbo for adding.

• The XML field spawns_on_death it seems can't be cleared with just nothing between the "". Instead, it can now be cleared by using "None".
  • This now fixes an ininitely splitting Hydral Hobbyist outguard. It already has enough heads.
• More work on Brutal Guardian Lairs:
  • Fixed Brutal Guardian Lairs spawning against minor factions - I simply forgot to disable that in XML.
  • The Brutal Darkness Guardian now has 0.7 instead of just 0.6 albedo (it's cloaked), and has individual metal and energy costs. Before all of this was just copied off the Brutal Adaptive Guardian.
  • Added the Brutal EMP Guardian. When encountering the first enemy planet with a structure of 4 tx or more it self-destructs, leaving behind a depleted version of itself that attacks with a stun blast, and an EMP bomb that will explode after 5 seconds, massively stunning next to everything on the planet.
  • Added the Brutal Amalgam Alpha Guardian, a combination of the Dire Pike, Nucleophilic, Concussion and Tritium Sniper Guardians. When destroyed splits into a head that carries the Pike and Nucleophilic weaponry, and a tail that carries the Concussion and Tritium Sniper weaponry. When those are destroyed they split into the relevant Dire Guardians.
  • Thanks to NR SirLimbo for adding.

Bugfixes

• Fixed a missing empty space after the cloaking-loss-per-shot tooltip section.
  • Thanks to NR SirLimbo for fixing.

• Fixed an issue where string-based dropdowns in the in-game galaxy options screen would not properly save their value. It worked fine in the lobby, but not in-game.
  • The new wormhole invasion settings are the first settings to ever use this feature, so this is our initial discovery of a bug dating back to at least June, if not prior.
  • Thanks to Badger for reporting.

• Fixed an issue with my list collections from two builds ago where FindAll and GetRange could throw an exception if they were to return a zero-length list.
  • This was causing an exception in the Swirl map type.

• From June 18th of this year, I made it so that map options were a lot more robust, but at the same time introduce a possibility where maps would not have an underlying related_int set properly.
  • Because of how this was done, this made it really easy for modders to shoot themselves in the foot, and I also shot myself in the foot with a ton of the existing maps as I tried to convert the data over. I essentially broke the sub-options on a lot of map types at that point.
  • To make sure that these don't just slide by unnoticed in the future, if the map generation algorithm tries to read an uninitialized integer or fixed-integer, it will now throw an exception.
  • Finishing the fix in the short term, then went through and fixed up the xml for tons of map types from the base game and dlcs, which have been quietly not having their sub-options working for all this time, it turns out.
  • Thanks to Eluthena and tom.prince for reporting, and tom.prince for noting that the xml was not set up correctly.

• Broke savegame serialization again, but made future serialization more likely to be correct -- or at least error at the correct spot.
  • Also cleaned up some serialization logging flags that were slightly inconsistent and which thus would mask errors.
  • Thanks to Eluthena, HarryT, Badger, Daniexpert, SirLimbo, and BoBy for reporting.

• FireteamRequiredTargets are now properly pooled; previously they were small objects that just leaked.

• WormholeInvasionData was previously treated as a struct rather than a class, but after a lot of discussion it looks simpler to treat things like this as poolable structs.
  • These also really need to use DeserializeUncertainNumberOfEntriesIntoExistingList in order to avoid a memory leak in multiplayer on clients, and now do.

• ProtectedDictionary has been split into ProtectedKeyDictionary and ProtectedValDictionary, since we might one to protect one or the other.
  • Also, these just now use IProtectedListable, rather than IProtectedDictionaryable.

• WormholeWaveData has also had a similar update.

• Found a general serialization bug in StoredAIPurchaseCostByBudgetForSpecificUnits, which was blanking out all of the budget types on save to disk or network.
  • The way this is stored and processed is now more efficient in MP and SP, but especially on MP clients.

• OutguardSpawnRequest is also now pooled, and also no longer leaks on MP clients thanks to DeserializeUncertainNumberOfEntriesIntoExistingList.

• I think that there are still other serialization errors for me to deal with, unfortunately. But things are better now than they were.

Beta 3.706 Expanded Wormhole Invasions And Raid Engines

(Released October 12th, 2021)

• The game is now forced to load the game entity data in a single thread, since it was consistently during this step that people were running into deadlocks on game load. I am a bit surprised I did not think to do this before, but I had shifted the actual loading of graphics to be direct rather than asynchronous. It may be possible that this lockup is unrelated to unity jobs after all, and perhaps there's just something that was prone to (relatively, in the grand scheme of things) rare deadlocks. I've seen it myself, but only one in every 50 or so times I start the game, if that. (I start the game an average of 30+ times a day, depending on what sort of coding I'm doing).
  • Anyhow, we'll see if this helps, fingers crossed.
  • Thanks to Mac, Badger, and NR SirLimbo for the most recent reports.

Badger's Wormhole Invasion Redo

• The player can now choose via a Galaxy Setting what mode of Wormhole Invasions they want. There are 3 options, the first two are simple:
  • Immediate Invasion: The current behaviour
  • Waves: The AI will spend its wormhole invasion budget on normal waves
  • Then there's the Planet Linking mode.
    • When in planet linking mode, the AI will spawn a Wormhole Projector on one of their planets. In a few seconds the projector will create a wormhole from its current planet to a planet that is (hopefully) behind your defenses. Then several Waves will be spawned.
    • Destroying the Projector will cause it to destabilize, and removes the wormhole in 60 seconds.

• Add a new cheat, cmd: instainvasion, to trigger a planet-linking wormhole invasion. It takes an optional additional integer for the strength of the invasion. So cmd: instainvasion, 60 would trigger a wormhole invasion of strength 60. Without the number it will send an invasion of strength 20

SirLimbo's Raid Engine Mechanic Extensions

• Gave the Raid Engine mechanic a few more options to work with, namely a few more XML fields:
  • "alert_wave_range_to_trigger" (default is integer 0, which is only the current planet). If set to above 0 then waves will be launched as soon as enemies are only X hops away. With this, Raid Engines could have a range.
  • "alert_wave_min_strength_to_trigger" (default is integer 0). If the base strength points (1000 bsp = 1 ingame strength) are below this, the Raid Engine won't fire.
  • "alert_wave_triggers_only_on_occupation" (default is bool false). This changes the behavior from triggering on enemy presence to only trigger on enemy occupation. If combined with alert_wave_min_strength_to_trigger then the occupied planet must have at least X amount of strength.
  • "alert_wave_spawns_unit_with_tag_instead_of_wave" (default is empty string). If set to something else than null or empty instead of launching a wave it'll spawn a random unit with said tag. This is how Raid Engines can quickly be turned into Dire Guardian Lairs.
  • Ditto form "alert_wave_spawns_unit_of_type_instead_of_wave", only with a special type (such as "AIDireGuardian"). This also supercedes the above!
  • "alert_wave_unit_name_override_for_display" overrides the name of the tag or entity type displayed (if it's set), so this is more readable for players.
  • "alert_wave_faction_for_unit_for" (accepts the following strings: "Sentinels", "Hunter", "Warden", "Praetorian", but could easily be extended if needed). If the "Raid Engine" spawns units based on tags instead of waves, the newly created units will be spawned for that faction.
    • Tooltips and notifications have been updated to work with these changes, such as being able to rigger even when the planet is not yet revealed (raid engines or Dire Guardian Lairs spawning from outside of visible range is a legitimate reason to be scared).
    • Raid engines (or any other entity with such a mechanic) no longer need their planet on alert level 4 if they trigger at a range. If they only trigger for the planet they are on nothing changes.
  • For a few examples: With this new logic you can:
    • Make a fully functional Dire Guardian Lair.
    • Give the AI Overlord Stage (stage 1 and otherwise) the ability to spawn exowar units for its Praetorian faction when a player settles within X amount of hops.
    • Make the AI spawn endless amounts of frigates so long as there is but one enemy faction with more than 0 strength in the entire galaxy.

• Even more "Raid Engine" mechanic updates NOTE: This brakes savegames - but we're on beta anyway:
  • Code-internally it has been renamed to "PeriodicSpawn".
  • All but the mechanics of the Vanilla Raid Engine are now gated behind DLC 3. New stuff is highlighted with _NEW_.
  • The maximum timer periodic spawns can work with is now no longer 128 seconds but 32767 (some ~9.1 hrs, this should be enough for basically anything).
  • Since it was a bit confusing and XML fields have been renamed, this is the end result:
    • "periodic_spawn_initial_delay" (Vanilla) = the time before the spawner becomes active for the first time.
    • "periodic_spawn_delay_between_spawns" (Vanilla) = the time between each new spawning event.
    • "periodic_spawn_never_stop_once_triggered" (DLC 3) = once the first even has fired it will never stop. Basically, the max range is set to the highest possible value, the minimum strength is set to 0. The player-only and occupation-only factors, if set, still stand though. _NEW_.
    • "periodic_spawn_creates_wave" (Vanilla) = creates a wave when triggering. Compatible with exo spawning, incompatible with entity spawning.
    • "periodic_spawn_creates_exo_strike" (DLC 3) = creates an exo strike when triggering. Compatible with wave spawning, incompatible with entity spawning. _NEW_.
    • "periodic_spawn_wave_or_exo_size_multiplier" (Vanilla) = when spawning waves or exo strikes they are this much more powerful than base waves or exo strikes.
    • "periodic_spawn_max_hops_to_trigger" (DLC 3) = if set to more than 0 also scans nearby planets within the hop limit for valid targets to trigger against.
    • "periodic_spawn_min_hostile_strength_to_trigger" (DLC 3) = when scanning for planets to trigger against at least this much hostile strength must be present. This is in base points of strength (1000 BSP = 1 ingame Strength).
    • "periodic_spawn_only_trigger_on_occupation" (DLC 3) = when scanning for planets to trigger against, they must not only have enemy strength on them but also be occupied by an enemy faction.
    • "periodic_spawn_only_trigger_against_player" (DLC 3) = when scaning for planets to trigger against only player factions are regarded. _NEW_.
    • "periodic_spawn_create_unit_of_type" (DLC 3) = creates units of a certain SpecialEntityType when spawning. Takes priority over "periodic_spawn_create_unit_with_tag", incompatible with wave and exo spawning.
    • "periodic_spawn_create_unit_with_tag" (DLC 3) = creates units with a certain tag when spawning. Overwritten by "periodic_spawn_create_unit_of_type", incompatible with wave and exo spawning.
    • "periodic_spawn_min_units_to_spawn" (DLC 3) = when creating units, spawns at least this many. _NEW_.
    • "periodic_spawn_max_units_to_spawn" (DLC 3) = when creating units, spawns at most this many. If set to less than the minimum unit count it's set to the minimum unit count. If both min and max are the same, exactly this many units will be spawned per triggering. _NEW_.
    • "periodic_spawn_unit_name_override_for_display" (DLC 3) = overwrites the SpecialEntityType and tag units are spawned by with this in tooltips and notifications. Only exists for clarity.
    • "Periodic_spawn_faction_for_unit" (DLC 3) = can spawn the unit for a specific AI sub-faction. Allowed are "Sentinels", "Hunter", "Warden", "Praetorian", "BorderAggression", "CPA" and "Wave". The latter 3 are _NEW_.
  • Internal code, including tooltips and notifications were updated to reflect these changes.
  • There is new fields on GameEntityTypeData: HasPeriodicSpawn, PeriodicallySpawnsUnit, PeriodicallySpawnsEvent. These can be used to quickly identify what the entity does.

• Added a new XML field "keep_damage_and_debuffs_on_transformation" which does just that: All damage to hull and shield as well as engine/weapon slow, paralysis and damage amplification will carry over. It defaults to false, which makes things work as they are now. When transforming the old entity type's XML flag being set to true or false counts, the not the entity type.
  • Affected transformations are: The mechanic AI Eyes or Gridlock/Deadlock use, time-based transformations such as warping-in structures, centerpieces tranformed via hacking points, necromancer transformations, Spire Telium transformations, Spire City flagship transformations, Neinzul Migrant wormhole transformations, Templar castle upgrades. Again: None of these are forced to have damage and debuffs carry over now, but the possibility to enable it via XML alone exists now.
  • Things that could be considered debuffs but aren't kept are time spent in a specific state-of-matter and death effects.
• Fixed a bit bad formating with entity tooltips relating to the description of multi-hitting beams and shots.
• Fixes and improvements to the periodic spawn mechanic:
  • Fixed the periodic_spawn_initial_delay and periodic_spawn_delay_between_spawns XML fields still being only bytes instead of shorts, limiting them to a maximum number of 128.
  • Fixed a bug where if the minimum strength to trigger on was set to 0 it'd always trigger - mostly against Outguard, unless more allied minor factions exist. In addition, when factions have the same amount of strength the raid engine or any other structure will spawn against the first faction with that amount of strength, which usually will be a player.
  • Fixed a bit of text overflow for Raid Engine notifications.
  • The mechanic will now work for all AI sub-factions as well, not just AI sentinels.
• When DLC 3 is installed the AI will now spawn 3 Brutal Guardian Lairs somewhere in the galaxy, at least 5 hops from the player homeworld(s). Currently they may not even spawn on maps such as X if the player is in the center.
  • These react to any player presence, ignoring any other faction type. When a player unit comes within 1 hop range a countdown of 5 minutes will begin. If the player retreats it will stop, and if need be continue again.
  • Once the 5-minute time has run out a Brutal Guardian will spawn for the Hunter Fleet of the AI owning the Lair. Currently 5 types exist:
    • Brutal Entrenching Guardian: If outnumbered on a planet changes to being an immobile, stronger variant with more firepower and shields. When transforming does not restore health and shields, and all debuffs will stay as well.
    • Brutal Warp Gate Guardian: Acts as a warp gate and Raid Engine in one. Only triggers against occupied planets, not simply all enemy presence. Especially dangerous if lurking as part of a Hunter fleetball right next door to a player.
    • Brutal Attrition Guardian: High attrition damage and shield-depleting ability combined into one terrifying ship.
    • Brutal Adaptive Guardian: Has adaptive multishot and high generalist DPS, which means it can equally strike against swarmers and big targets of all types alike.
    • Brutal Darkness Guardian: Heavily cloaked, very fast and able to attack while cloaked for almost 40 seconds with all guns active. Its light and heavy sabot weaponry strikes well against most things as big or bigger than turrets.
  • Once the spawning has started it will not stop until the Lair is destroyed. Killing it will inflict 15 AIP, but also grant 500 science and 15 hacking points. The Lair will also self-destruct if the AI command station of its planet dies.
• Outguard can no longer have the AI send waves against them. It makes no sense to do so anyway.

Beta 3.705 Blazing Collections And The New Spire

(Released October 6th, 2021)

Bugfixes

• Corrected some base game starting battlestation templates having incorrectly divided by five twice over turret counts. Small turret counts returned to their correct 25 count values.
  • Thanks to Mac for reporting.

• Fixed yet another exception that could happen in the faction windows when swapping between factions. It was kind of a timing/race condition thing with the UI refreshing.
  • Thanks to BoBy for reporting.

• Fixed another exception in long-range-planning for the scourge.
  • Thanks to BoBy for reporting.

• Since long before the great refactor, we've had occasional issues with fireteams getting strange preferred speeds set on them. I suspect it's some sort of integer overflow, or some other related issue. At any rate, there are now protections in place to keep from generating an absurdly high speed (anything higher than the highest speed available (unless everything is slower than 550, in which case it still accelerates to that), and also avoiding anything that is negative or zero. This should solve some strange errors that would occasionally come out of fireteams.
  • Thanks to BoBy for the most recent report.

• The serialization of all prior savegames has now been broken again, because I made a major structural change to how we save entities into the list. (Remember, we are still in that breaking-savegames timeframe, but that won't last much longer).
  • Essentially, the old approach was inefficient on the host in multiplayer, and for the player in general when saving to disk, and it occasionally was prone to cross-threading issues when it comes to units dying on a background thread while trying to be sent over the network or to disk on the foreground thread.
  • The new approach uses one extra bit per entity being serialized (though it saves 32 bits before serializing all of them), so it uses very slightly more space and network bandwidth, but it requires only one loop over the entity lists rather than two (this is a really good improvement), as well as being a lot more robust against those cross-threading issues. The old approach was kind of asking for trouble, even though it was still rare to actually HAVE trouble.
  • This is in relation to the "faction A on planet B had X entities to serialize but only Y were actually serialized" errors that you could get, among other ails.
  • Thanks to BoBy for reporting.

• Hardened EntityOrder.CalculateStrengthCountingData against cross-threading issues that could happen at stage 5000. Those should not be possible anymore.
  • Thanks to BoBy for reporting.

• Fixed a visual issue where a wrong markup tag was causing the entire planet tooltip text to be bold.

• Random Factions, a super cool feature, work once again! These have been broken since the great refactor, but it was not nearly as big a break as previously thought!

• Fixed a bug introduced in yesterday's build where if you loaded up savegames or started new games after playing for a while in a different one, you'd have perma-vision on random planets. At least it was a two-line fix, but this was an error on my part as I went through the performance improvement work.
  • Thanks to Badger for discovering.

• can_be_selected_by_any_player_even_if_not_owner=true" has been removed, as that is now universally true no matter what, anyway.
  • can_be_given_orders_by_any_player_even_if_not_owner="true" has replaced that. This is used mainly for the spire relic right now.
  • With some tweaks in this area, the "cannot give orders to spire relic" bug seems to be fixed. This was not something that was super reliable to reproduce, though, so it may still turn up again.
  • Thanks to Vinco for reporting.

Improved Handling Of Textboxes

• The way that we were listening for TextMeshPro actions when a textbox is highlighted has been stripped out. It was not working reliably.
  • Essentially, we were listening for the escape key or the enter key to either clear or submit things, and this was too flaky to keep using it. It made sending chats recently (which also means entering cheat codes) a real exercise in frustration.

• There is now a gets_sent_to_focused_textboxes="true" that can be added to any custom input (like open menu, open chat, whatever).
  • Things that have this one it will still not do their normal execution, but instead will be sent to the textbox that is currently open. The textbox gets to choose what to do with it, and then can also say "execute as normal" or "ignore it."
  • This gives us the flexibility to override things, and not have unexpected things happen, but at the same time to actually make these respond reliably.
  • This is now applied to OpenSystemMenu (Escape key), and Return (the Enter key). If we (or a modder) ever needs more than this to punch through, that's fine. I can't see what the use would be right now, though, as people are generally typing.8

• In general when we were trying to unfocus text fields, it was not always working fully. This is apparently a known unity bug with unity textmeshpro.
  • I implemented the hack workaround from here, and that fixed it: https://answers.unity.com/questions/1638173/removing-focus-from-a-tmp-inputfield.html

• Based on the above changes, plus adjustments all over the place:
  • Hitting enter in the textbox on the save menu after you've typed your text now saves the game right away, versus you having to click the button.
  • Hitting enter in the chat textbox now works reliably (omg this was rage-inducing for me before).
  • Hitting escape in the chat textbox clears whatever text is there as well as unfocusing. Hitting escape a second time closes the window.
  • In the galaxy map filter box, hitting enter leaves the text you have typed, but removes focus from the field. So you can then pan around or what have you. That keeping focus until you clicked was also hyper frustrating.
  • In the galaxy map filter box, hitting escape clears the text you have typed, and also removes focus from the field. This way you don't have to hit the little X button in there. Thanks to Badger for suggesting this long ago.
  • In the lobby, when you hit enter in the campaign name textbox, it now starts the game. Escape just clears focus.
  • In the quick start menu, when you hit enter in the campaign name textbox, it now starts the game. Escape just clears focus.
  • In the MP lobby, hitting enter in the chat box now works reliably, and hitting escape now clears you text.
  • Everything else seems to pretty much behave as you would expect.

• In the multiplayer lobby, if you try to send a chat message and nobody is there, it no longer gives you an error message about failing to send to anyone.
  • This is a longstanding gripe of mine, finally fixed. It was an overcautious bit of error-handling on my part.

Improving The Fallen Spire and the Necromancer

• Added two new special_entity_type options: MobileCustomFleetFlagship and CityCenter.
  • Removed becomes_centerpiece_of_custom_fleet_if_exists="true" and is_quasi_centerpice_for_stationary_structures_in_fleets="true"
  • Both the fallen spire and necromancer have been converted to use this.
  • A couple of spire-specific and necromancer-specific rollups have been removed as redundant, as well.

• Added a new PlayerCustomCity FleetCategory.

• A new CityBolstersFleetID has been added on fleets, only for PlayerCustomCity.

• The compound "city and mobile fleet" concept from the fallen spire has been removed from the game. That was such a mess, and made many things like "destroy distant fleet members" not work correctly.
  • The entire premise was just very not-helpful. There will instead now be separate fleets for the city and for the mobile forces.

• Added Pluralize.NET to our source code (it is MIT license), and added a new ArcenStrings.Pluralize() method, which efficiently wrappers it.
  • If you have a word that is singular that you want to make plural, this is how to do it -- don't just add "s" one the end, or what have you.
  • This aids in our dynamic text generation.

• We now have name_for_city_sockets="string" and name_for_city_center="string" for cities and city sockets, which are required for any unit that uses those features.
  • This allows us to properly talk about spire city sockets, and necropolis bone holes (wow that sounded wrong) or whatever we want to.

• "City Sockets" in Necropolises are now called "Necropolis Hexes," at least for now.
  • In spire cities, these are also now called "Spire Casings."
  • I specifically chose "city sockets" in the past because it was generic, but now we can be specific.

• At this point, things compile and the spire cities have been separated from their fleet portions, but not combined back in. This means fallen spire is effectively broken, but that won't last long.
  • Similarly, the necromancers spawn their base necropolis just fine, but their core fleet stuff has also not been split out properly yet. This breaks them, too.
  • I made enough changes that I was nervous about continuing to make more without first checking these in, though.

• Spire cities (and necropolises) can now be renamed if you wish. I'm not sure if that was possible before, but I think it was not.

• When you create a spire city, you now only get a city itself, not a half-city-half-fleet.
  • You will get up to three fleets (shared between all human players as they wish) depending on how many cities you have.
  • The first spire city gives you a first fleet. The first time you get any spire city to mark 3, you get another spire fleet. The first time you get any spire city to mark 5, you get a third spire fleet.
  • This starts you off with a more manageable and limited number of spire fleets, and keeps them separate from the cities (which work the same as ever), while letting you grow your fleets over time so that you can eventually cover many areas of the map.
  • Overall, thanks to this separation, the fleets for the cities and the actual fleets themselves work... way more like you'd expect. It's so much easier to deal with, now.
  • More changes are needed to adapt this whole thing into this new idiom, so stay tuned, but this is a good start.

• The city interface now has the start of where you can control which mobile spire fleet (or necromancer fleet) is being "bolstered" by the current spire city / necropolis. The idea is that each city-type-thing gives its units to one fleet or another, and that can be changed as you get more fleets.
  • The full functionality of being able to actually change that is not in yet, but the basic structure is, including auto-bolstering any new spire fleet from any cities that are not bolstering anything yet.
  • There's still more to do there, and the logic is per-faction, so I'm going to be having some hooks in there so that the necromancer can do whatever it needs to do, which is likely to be slightly different from the fallen spire. This is again something that is very mod-friendly now, so similar sorts of things can be set up in mods in the future. (So many possibilities, SirLimbo...!)

• The FleetCategory PlayerCustomMobile has been split into two parts: PlayerCustomCityFedMobile and PlayerCustomUnattachedMobile.

• For the city-fed fleets, there is now a FedHereFromCityFleetID on each fleet.

• The Necromancer is back to being more functional, with a Necrofleet being spawned with every Necropolis.
  • The term Necrofleet is obviously subject to change, and right now I'm not sure that all of the ships that should be spawning in the fleet are doing so -- they may still be spawning in the Necropolis itself.
  • But at any rate, it's progress.

• Removed the DoForFleetsParallel methods, because those are basically never going to pan out as a positive performance boost.

• Added a new DoForCityCenterFleets() and a new DoForCityFedMobileFleets(), and the underlying data structures needed to support making those run quickly.
  • Also made it vastly faster to look up fleets by ID, partially because in some areas we will be doing that a bit more (but honestly I should have done this a while ago, now that I look at it).
  • Also made it faster to loop over just player fleets.

• Necromaner ship attritioning and updating has been moved from DeepInfo to BaseInfo. This will be much better for multiplayer.

• More necromancer logic has been moved into BaseInfo. This is stuff that needed to be present for both the client and the host to work properly in MP.

• The necropolis mobile bits have been split out more, and the city should be feeding them. There are still some glitchy bits with it, but I think it's because I'm getting some extra fleets right now.

• More necromancer updates to try to get it to populate the mobile fleets from the city fleets. No dice quite yet.

• Success! The necromancer no longer creates double-fleets (meaning quadruple total) when it creates the major necropolises and the attached necrofleets.
  • Additionally, the skeleton and wight caps and so on happen on the necrofleet, fed by the necropolis bolstering them, rather than being created directly.
  • As far as getting back to prior functionality baseline of necromancer, this seems like we are pretty much there. More things will need to be adjusted in the coming weeks, but this is a great start.

• UniqueTypeDataDifferentiatorForDuplicates is now an int, rather than a byte. We're using this to delineate which ship lines are bolstered from other fleets, now, to make things easier.

• Added a new FleetQualifier string on fleets, which lets us identify certain fleets even if their fleet leader has died.

• In the fallen spire campaign, it now upgrades all of your flagships to battleships after you have your first mark 3 city. Once you have your first mark 6 city, they all become dreadnoughts.

• In the fallen spire campaign, all flagships of the spire are simply the mark level of the highest-mark city you have. This is simpler than the other system we had discussed on discord, and this faction is complex enough to play already.
  • This way you don't have to worry about spire flagship booster buildings, that's not a thing now.

• In the past, there used to be one fleet led by a spire flagship for each city you had. This is no longer the case.
  • You now start out with only one fleet. Once you have your first mark 3 city, then you get a second fleet. Once you get your first mark 4 city, you get a third fleet.

• In the fallen spire campaign, your cities properly bolster the fleets they are assigned to, and give more units in them at the mark level of the city.
  • There's still some functionality to add here, but the generalities are in place now.

• From the very start, each spire city now gives one spire frigate to the fleet it is bolstering.
  • This does make them a bit more powerful, but you are getting fewer flagships than before, and it's nice to have a bit of new ships from the start with every city.

Code Refactoring For Clarity

• Fleet.Membership has been changed into FleetMembership.
  • This is mostly for ease of file organization, but also ease of typing it.
  • In your mods, just do a simple search and replace that is case-sensitive and whole-word for "Fleet.Membership" to "FleetMembership"

• Civilian Industries (the mod) has been updated to function with the above change.

• The "PlayerPlanetaryDefense" FleetCategory has been renamed to "PlayerBattlestation"
  • This is way easier to understand in the code and know what you are looking at versus having to guess.

• General cleanup. Removed some deprecated code and data.

• SpecialEntityType: MobileCustomFleetFlagship has been changed to MobileCustomCityFedFleetFlagship.
  • MobileCustomStrikeCombatFleetFlagship has been changed to MobileCustomUnattachedFleetFlagship
  • These were set to be confusing as heck. The latter is not actually used right now (but is useful for mods), and the former is used for spire and necromancer stuff.
  • This separation also allows for me to start putting some specific sub-data on MobileCustomCityFedFleetFlagship fleets.

Collection Performance Improvements

• I have reimplemented a custom version of the List<> class from Microsoft's generics, and made various improvements to make it faster.
  • Overall it ranges from 3% to 10% faster on average, but it also no longer takes a poor-performing parameterless constructor. That was about 300% slower than a proper use case.
  • Sorting is also about 10% faster than before.

• A new test framework for testing how well various data structures work IN UNITY has been added.
  • Thanks to SirLimbo for originating the base versions of it.

• ArcenListPool has been removed, as we can do the same thing much better with ThreadStatic.
  • This was used only in the layouter rebuilder for the UI, and was a source of some slowness.
  • Got rid of TMP_ArcenListPool for the same reason.

• Removed Debugging_ArcenReadonlyList, because we simply did not even use it and it's not useful.
  • Oh, also Debugging_ArcenReadonlyArray.
  • Also RefPairCollection.

• The Queue<> class has been moved into ArcenUniversal. For the moment there were not really any performance improvements to make to this.
  • However, we may make some in the future, and because of the fact that we moved List<> into the codebase and we want to replace that, we need to also handle the other Generics collections.
  • So, same deal with Stack<>.
  • Oh, I did make the Clear() method a bit more efficient on both of them.

• Dictionary<> has also been moved into our codebase and made slightly more efficient.
  • Also one plus is that if you call in to get a key that doesn't exist, it gives the default value for that type, rather than throwing an exception. This is the same as our ArcenSparseLookup.

• ConcurrentDictionary has been pulled into our codebase, along with the other collections, so that it can use our own versions of the enumerators consistently, and so that if we need to make performance changes to it later we can.
  • At the moment I'm not super aware of any performance issues with these, and they're used only in certain circumstances anyhow. But being able to take control of our most central collections and correct errors that we do find is useful.
  • This of course means that ConcurrentQueue and ConcurrentBag also are making the jump in.
  • After looking at the code for ConcurrentQueue vs ConcurrentBag, I can see why there were such differences in performance that some programmers on the internet were reporting in the past.
    • These have completely different philosophies on how to build handle themselves and their data, and the fact that ConcurrentQueue was faster than ConcurrentBag in practical uses is probably not just a "whoops there was a small error there" sort of issue. Supposedly it was fixed so that Bag is now as fast or faster than Queue in the concurrent classes, but I've never seen any benchmarks, and I have not yet run any of my own.
    • From just looking at the code, however, I am now deeply suspicious of ConcurrentBag. It really throws off a lot of extra objects at the very least. It is notably simpler code (sort of) compared to Queue, but the thing about this simplicity is that it is missing some of the support data structures on Queue that just _happen_ to seem to make that likely to be faster.
    • In other words, I'm not convinced that Bag can ever be faster than Queue without a complete rewrite, or if it is then it will be highly context-specific. You would think that the fact that the Queue is ordered would make it perform better than the unordered Bag, but to me with a bit of a look through things, it looks like a lot more intensive thought went into Queue than Bag. Which is a shame.
    • We currently use some of both, but we are really going to need to test out performance in a real-world scenario and see if we can actually get an answer on which seems to be faster for those purposes, minus the noise of other things that fluctuate. I'm not convinced that any meaningful test-bench type tests would give us realistic expectations or results when it comes to the concurrent collections. It would be really easy to accidentally design tests that favor one or the other artificially.

• On ConcurrentDictionary, I have specifically avoided implementation of certain methods and properties, such as Values and Keys (the properties that give you IEnumerators or similar).
  • Those particular properties are handy in the abstract, but really are just kind of a convenience to make outer code the TINIEST bit more brief. But what happens is, under the hood, rather than doing a wonderful iteration over the internal buckets with a yield statement -- like the normal ConcurrentDictionary enumerator is able to do -- it has to pull things out into a separate list, and then enumerate that. Translation: lots of extra garbage being generated, and some performance loss. So, yeah, we can do without that feature.
  • In general with all of these collections, if there was something that is likely to shoot your in the foot with performance without you knowing it, then that's missing. If you're trying to do something and find you can't now... well, there's an easy mostly-equivalent still there, and it's going to perform much better than whatever it was you were trying to do.
  • Another notable example on all of these is CopyToArray() or similar. That's something that is rarely really useful in a runtime sense. It can be a handy shortcut when you're doing a debug dump or something... but just loop over things and use the extra performance during said debug dump (it will be slower than CopyTo or CopyFrom would have been, but that's okay because this is a debug dump). The thing I want to avoid is someone trying to do a CopyTo, thinking that's okay to do during runtime, because it's really not. There is a method to my madness! Often programmers make choices without understanding the hidden costs in the classes they are calling, and my goal is generally to help avoid that. Microsoft's goal is maximum flexibility, by contrast, and so you can shoot yourself in the foot all day as far as they are concerned. We have different goals, so provide different things.

• One side effect of a lot of these changes is that you can't declare "var" as the type of the thing you are doing in most foreach calls. The reason why is a bit complicated, but since this leads to more-readable code (that also performs faster in visual studio when you're editing, as a side bonus), I'm not going to complain or worry about it.

• Another side effect is that I can (and have) now put in base classes that let me control more things in the future, and which make my custom version of ObjectDumper now give a vastly better snapshot of game RAM when I do a dump of that sort.

• I swear, I have no idea where certain code was coming from, like TryDequeue on the regular Queue. That is in the Microsoft specifications, but not in their published source code. It must be added by some sort of other dll.
  • There have been some odd cases in the past for me where I had a Clear() method only some of the time for certain collections (this is weeks ago), probably based on which using directives I had imported. That was annoying as heck. All of the stuff is now directly implemented. If there wasn't an implementation, I made one.

• Side effect of some of these changes: the game itself may load more quickly now (getting to the main menu), and map generation may be a tad faster (probably not much), and savegame loading might be faster (closer to the speed of a second load rather than a first-load).
  • Depending on the specifics of your machine, it may make very little difference or it might be really noticeable.
  • Also during actual runtime, it should run more smoothly for basically everyone, but here again this is... cleaning up a lot of tiny paper cuts all over the body, essentially. It's not some giant wound or something. So seeing exactly how that has an effect in aggregate is always interesting.
  • On the MP client in particular (still not functional yet in the latest beta, but coming soon) there's some speedups with where I had been using the .Keys sub-listing of ConcurrentDictionary, having no idea that that performs terribly.

• I did not port over Hashset, so I have implemented RemoveWhere() onto dictionary, since that's something that StarKelp seems to find handy. (And yeah, that is handy.)

• If you have code of your own (for instance a mod), you'll want to remove the following references. You can do a global search and replace to make them blank:
  • using System.Collections.Generic;
  • using System.Collections.Concurrent;
  • using System.Collections; (this last one is optional -- you may need a couple of things from this, like IEnumerator, so you can leave it if you want).
  • Assuming that you use a using reference to Arcen.Universal, that's all you need now.
  • You can then expect to see some errors of the following kinds:
    • Collections that you didn't specify a starting capacity for asking you for one. Go ahead and set those up to be a little bit larger than you need, because RAM is cheaper than CPU on the scale we're talking about (small ram, big cpu to expand).
    • Any place you used var in a foreach will need to use the actual type name.
    • Any attempts to loop over the .Values list of a dictionary will also be busted and you should just loop over the dictionary itself.
    • Any places where you are using "IEnumerable<" should become "System.Collections.Generic.IEnumerable<" Easy global search and replace.

• Our regular dictionary now lazy-loads its internals, so that we can specify a desired size without actually having to do any work if it's not going to be used for a while. But at the same time, it doesn't initialize to a too-small value that then has performance issues during gameplay. The loading time after getting all this running was truly abysmal, double what it used to be with these changes, and RAM usage was also too high.
  • Most of that had to do with initializing a lot of lists and dictionaries to sizes that are sufficiently large for use, but which are very often only contextually-used, if used at all.
  • There are actually a ton of lists and dictionaries in our codebase that get initialized "just in case" of future need, and by lazy-loading those instead of aggressively front-loading them, we can save ram and performance and the end code never knows the difference.
  • This is the sort of cool thing that we can do now that we've taken control of our own collections, versus letting them be just part of the core .net framework.

• The KeyCollection and ValueCollection have been removed from the Dictionary class, which means that .Keys and .Values are also gone.
  • We did not use these super frequently, but they were a definite performance drain both when initializing a dictionary and also when making these calls. These essentially made the entire game a bit slower, and so now we just loop over the core, same as with the concurrent dictionary.
  • Also, this prevents programmers from making some... uh... questionable array accesses inside loops for things like serialization. Again a speed boost.
    • Actually this was happening in more than just serialization. So, yay, speed boosts in a variety of places, then.

• The Stack class now lazy-loads, giving it the same performance boost as dictionary now has.
  • Queue now, also.
  • And then the big one: list. Performance is back to what I would hope for all this, and is now finally better than the old standard.

Sparse Lookup Replacement

• Dictionaries, lists, and so forth all now support doing a foreach on them even when changes happen.
  • This means that you can remove from a dictionary you are foreaching over and not get an error. Similarly, if another thread touches the data, it no longer will error a foreach.

• Added a new ProtectedDictionary, which does much the same sort of thing that ProtectedArcenSparseLookup does, but for Dictionary.
  • Most things that used to use the ArcenSparseLookup are being converted to Dictionary, so this also needs to exist now.

• Ported ClearAndCopyFrom() from ArcenSparseLookup to Dictionary.

• ArcenSparseLookup has internal improvements from both our new List and Dictionary classes, but it's still stupidly slow because it does both things at once.
  • Essentially, the dictionary class now does everything that this one did, except faster, with one exception: sorting. So in the event you need that, you need this. Otherwise Dictionary is the way to go now.
  • With that in mind, this has been renamed from ArcenSparseLookup to InefficientSortableDictionary.
  • Unless you need sorting capabilities, please be sure to use Dictionary from now on!
  • If you are converting things over, you can do a global search for "ArcenSparseLookupPair" with "KeyValuePair".
    • Also replace "ArcenSparseLookupPair" with "KeyValuePair"
    • And replace "ArcenSparseLookup" with "Dictionary"
  • You will find some cases where previously we took advantage of the fact that ArcenSparseLookupPair was a reference type. The old code from ChooseWaveTarget() is a good example of this.
    • That code will no longer work, because as a value type, trying to assign into the KeyValuePair is readonly on purpose (it prevents programmer error). You would now instead need to assign to that in a constructor, or assign directly into the dictionary itself, depending on what you are doing. This is a very good thing, it's way more performant and also error-proof.
      • See the above method (check diffs if you want) as an example of how to deal with it. The code is just as legible now, if not more so, and it also doesn't throw any garbage onto the heap.

• Added a new extremely-efficient CopyToList() function on Dictionary, which works via a special hook I've set up with List<> to make that work really really well.
  • This is an early example of the sort of power we've now got to (either shoot ourselves in the foot if we make an error or) do really cool things that allow for better functionality and also improve performance while we're at it.

• Added a new GenericDictionaryBase, which defines some abstract methods that are normally implemented by dictionary.
  • This allows me to make ProtectedDictionary inherit from Dictionary, rather than wrapper it. This is notably more efficient, and also more functional.
  • This will be a nice little performance boost in general, and since I'm using abstract methods, it's just as fast. Note that I could have gotten the same effect by marking dictionary methods as virtual, but if you look at the language specs, that is slower.

• Okay, so a minor case study. One of the things that we needed to sort previously was EnergyConsumedPerFleet_ForUIOnly. This is on factions, and was an ArcenSparseLookup.
  • This is now just a dictionary. Previously, it was aggregating the data during sim AND sorting it (which it even noted was redundant, sigh). It no longer sorts this data until you actually view these details, in the UI subwindow where it was already re-sorting before.
  • But wait! We can't sort a dictionary... what's happening, then? Well... this is where we get into the magic and vastly better performance (and code that is also simple.
  • Before, the code looked like this: localFaction.EnergyConsumedPerFleet_ForUIOnly.Sort( ArcenSparseLookupPair <Fleet, EnergyUsageDataHolder> ). And then we'd iterate over the internal list (hyper inefficient, sigh) that the SparseLookup had.
  • Now we do the following in the UI code when we want to draw this in a sorted way:
    • Declare a List<KeyValuePair<Fleet, EnergyUsageDataHolder>> energyConsumedPerFleetForSorting, which is one line of code. This is very efficient.
    • Call this: localFaction.EnergyConsumedPerFleet_ForUIOnly.CopyToList( energyConsumedPerFleetForSorting, 0 );
    • At this point we have a list copy of all our dictionary data, and it was done using Array.Copy internally, and allocated nothing new. It's super duper efficient, and only runs when we actually need it.
    • Of course, now that we have a list, we can just sort it. And we can do this when we need to, but skip any other times.

• Okay, let's follow up the above case study. That is indeed not TOO bad, and you can do that still, but it's more code than I want to deal with. Generally there's a 1:1 relationship between the thing you want to sort and the dictionary, right? So let's build that into dictionary instead, a bit more directly but with that same degree of high efficiency.
  • The TLDR is that I don't feel like typing that much code each time I want to do a sort, and since I control the dictionary class now, I don't have to. Let's see how much shorter we can make the code while not incurring even a tiny bit more performance:
  • Define a holder working variable in the local method like this: List<KeyValuePair<Fleet, EnergyUsageDataHolder>> energyConsumedPerFleetSortedList = null;
  • Then assign into it in one call like this: energyConsumedPerFleetSortedList = localFaction.EnergyConsumedPerFleet_ForUIOnly.SortIntoList( yourSortHere )
  • Then just loop over your energyConsumedPerFleetSortedList, and ignore it other than that. Much more quick to type! Same efficiency as before.

• You know what this means, I think: goodbye InefficientSortableDictionary! There is absolutely no reason to have it or its variants now. The above case study for sorting is far more efficient.

• While I have been going through the dictionaries to fix these up (well, the ones that were arcen sparse lookups in a prior life), I also have fixed a number of places where we were doing extra allocations and hitting the GC. Basically just stuff from before we knew about ThreadStatic, or where somebody didn't bother to use it for whatever reason.

• Added a new SortedDictionary class, which mostly just uses dictionary EXCEPT that it is expects to be sorted and will enumerate over the internals of the sorted list rather than the dictionary itself.
  • This is used in very few places in our code, but this is the only way I could really think of to make sure that code remained readable and also high-performance and without being prone to bugs. My first couple of ideas on this were terrible, so I dropped back to this, which works very well.
  • This is mainly for the strong-against and weak-against code, and some other ship lister results, and I haven't tested it yet. If there's a bug, that's where it will be. It's also used for the "list of ships that will benefit from hacking a tech vault." Also the list of planets that have threat on them when you mouse over the threat tooltip. Oh, also the list of ships that will benefit from upgrading a fleet by science directly. Aaaand it's also used for the ships that benefit from regular techs, too. If those tooltips are broken or not sorted nicely, you'll know it's this.
  • Okay, have tested these out now, and they do work (I had a few fixes to do before they did).

• For the sake of efficiency and code brevity, I've added a SortIntoListWithoutSort() and GetRandomKeyValuePair( RandomGenerator Generator ) onto the dictionary classes.
  • This makes it easier to do some things that we used to only be able to do with the sparse lookup. It's less efficient than not doing these things on a dictionary, but it's more efficient than sparse lookup used to be.

Beta 3.704 Unexpected Turbo Sim Speed

(Released October 1st, 2021)

• In the lobby, the AIs now have a nicer display on the left sidebar of their difficulty and their type.

• There was some OLD logic in our "read strings from files in a folder" logic that said to look in an "Override" folder inside that folder if you wanted to, well, override it. That's been stripped out.
  • This logic instead now checks all the mod folders for the folder name in question, and if it exists then it pulls from all of those mods (if there are more than one with the same folder name) and ignores the main game.
  • Translation: this now allows for mods to be created that have fleet names or planet names. These can be additional sets, or they can be replacement sets for our existing "Standard" option or what have you.
  • Thanks to RocketAssistedPuffin and NR SirLimbo for reminding me that this was one hole in our modding toolkit.

• Added a new "League+" name set for planet names, created by Angel of death.

• The fleet management window has been updated slightly, to make room for modular unit controls.

• The start of modular ships -- aka, ship loadouts -- is in place now.
  • There is an is_modular="true" field, which can be assigned to any unit from DLC1 or DLC3, or any mod unit if you have DLC1 installed (loadouts/modularity are going to be considered a DLC1 feature -- a few other DLC1 features are coming, along with a later price increase on DLC1, to make all three DLCs equal in scope and value).
  • Right now the modular flag doesn't do much, but it's the gateway to the next things I'll be adding on those units.
  • Of note, I wound up looking at how I was thinking about changing things like forcefields to work via systems, and... definitely decided against that. I'm going to wind up with a split system for handling modules, one for weapons (easy), and one for "other random stuff" like extra health, or bubble forcefields, etc. I have a good design in mind, and it will be higher-performance and not require breaking existing saves in order to do it. All in all that should make it faster to implement, too.

• The fallen spire ships have all been split up so that they now have a base root unit for each class which is what gives the general stats and such to the imperial spire versions and regular and flagship versions, but no weapons.
  • Right now, the regular, flagship, and imperial spire versions all have the same weapons, but they have some other different stats. The imperial versions are used by NPCs and are marked as not-modular.
  • The regular and flagship versions are going to become modular soon, and will have potentially different modules than one another, so hence having a divergence from a new common base.

Bugfixes

• Fixed exceptions that could happen with CustomData_StartingAILayout, CustomData_StartingAIPlanetRatio, and CustomData_WormholeBorerIncomeModifier if the data for them was blank. Now it just uses the default value in those case. Not entirely sure how the data can be blank, but in some fashion or other it was just data that was initialized without the player actually making a choice about it.
  • Thanks to BoBy for reporting.

• Fixed an exception that could happen in Marauders stage2 with lazy-initialization of their external data.
  • Thanks to BoBy for reporting.

• Fixed an issue where the lobby and in-game factions screens were showing the same information in the tabs on the left for every faction beyond the first for any given type of faction. So for instance, each AI showed the info from the first, etc.
  • This actually finally explains how the attachedfaction used to be null in that area, which is nice to know I wasn't crazy. I couldn't see how it was null, but it was using something outdated that I forgot existed. Now it uses the expected central items.

• When I set up the AI data and subdata in as part of the great refactor, I accidentally wound up having two copies of the central AI type and difficulty. One on the main object (not needed), and one on the sentinels sub-object. Most things were reading from the latter, but the latter was also always set to the defaults rather than the values you chose.
  • Now there is only one set, and everything reads from that, and this works great.
  • Thanks to Arctic for reporting.

• At least since the great refactor, but maybe longer, the escape menu where it shows the list of factions has been a bit of a mess, with extra intensities showing and so forth. Fixed up those bits, and then also just made it look a bit nicer than it did in the past for the AI types in particular.
  • Thanks to Arctic for reporting.

• Fixed a nullref exception that could happen in SpecialFactionPlanning.GetNeedsToRun().
  • Thanks to BoBy for reporting.

• Fixed a cluster of issues that could result in the AI sub-types or sub-difficulties not being set properly when loading savegames in some cases.
  • This then fixes all sorts of other random issues that would show up in other spots if this happened. The central logic for these is also now set to error if they still remain null, so that we know what the true problem is rather than worrying with random other issues.

• I can't (yet) duplicate the error, but there was a pathfinder being used from two contexts incorrectly, and I've now put in a lot more code around making that not happen. It appears that this was caused during an exogalactic attack, but that was honestly the part that looked correct from the log -- there was some prior call from the AI sentinels to the wrong pathfinder, and that set things up for the later error. I now have it so that it will detect the FIRST wrong call, and not just when there are calls from two contexts. I've done a manual code review of all the affected areas, and can't see how it would have a problem, and I've done some testing for a while as well, so I guess we'll just have to wait for it to trip again. Next time will be more informative, or should be.
  • Thanks to whimsee and Vinco for reporting.

• HandleExogalacticAttacks has been updated so that it is now using threadstatic working collections so that it is fully threadsafe. An error in a log may have been due to that method being called by two factions at once, or it might have been related to the pathfinding issues. Either way, I made the method threadsafe, made it slightly more efficient, and then also put in extra logging spots so that we will know which it is if it ever has an issue again.
  • Thanks to Vinco for reporting.

• Fixed an exception in the scourge that could happen during long-range-planning if their warriors were not properly assigned their BaseInfo data yet. Honestly this seems risky from a threading point of view to have it handled this way, but apparently it worked prior to now. At any rate, now it should still work.

• Further fixes to the scourge. Their data is now initialized in Stage2 if it is missing, which should avoid any LRP issues or other cross-threading issues, and some followup exceptions that I saw.

• Improved the error detection when deserializing faction data.

• Previously, if you were trying to hack something but got an error about having too many hackers to choose from, there were the following problems, all now fixed:
  • It would choose one and hack anyway, which was potentially going to be quite a problem.
  • It would only show you this warning AFTER you clicked "yes I'm sure," rather than before, which is just plain frustrating.
  • Thanks to Vinco for reporting.

• Fixed a variety of errors that could happen with astro trains based on them not having their external data properly initialized. At least one or two of these were translation errors on my part during the great refactor.
  • Thanks to whimsee for reporting.

• The internal nomenclature for getting network-attached settings is now a lot more clear. It says NetworkAttached, specifically.
  • In order to trigger journal entries properly in multiplayer, the network-attached settings idiom is still really useful for our automation settings. At first it seemed superfluous, but it's really not.
  • Found and fixed a problem (maybe from the great refactor, not sure), where network-attached settings were not properly being sent. This affects singleplayer as well as multiplayer.
  • With that, watchmen frigates now auto-build again, and any other non-default automation settings also work properly.
  • Thanks to whimsee and Vinco for reporting.

• The timings window now shows us how many player accounts are in a game (way down below the threading info), and which one is the local account, if any.

• Found and fixed that pathfinder issue! It turns out it would only happen after you had saved and loaded a few times, or better yet had generated a couple of different maps. It was a pooling issue.
  • Thanks to whimsee and Vinco for reporting.

• Fixed a rather subtle bug that had to do with singleton cleanup on external faction data in general. Essentially, it was not being cleaned up when going back into a pool, but only on coming back out.
  • The notes in the code make it more clear why I now handle factions and squads in an opposite manner from one another, but at any rate factions have been adjusted so that there are no longer lingering effects of absent factions after you remove a faction.
  • The most visible of these was nomad planets, where if you added that faction and then removed it, you would see nomad planets in all your maps until you restarted the entire game program. The nomad faction was indeed gone, but its singleton was still there and so it was kind of half-alive.
    • There would have bene other factions (fallen spire, etc) that also had this sort of characteristic, but they no longer do.
  • Thanks to whimsee for reporting.

• Reworked the lobby faction tab so that it no longer has any chance of doing funky stuff when factions are removed or the faction you are viewing is changed. Previously it could throw exceptions in some very rare and unlucky cases. I don't think the factions window in-game (after starting a campaign) can do this, but if we see it then I'll make similar changes there.
  • Thanks to whimsee for reporting.

• Fixed an exception that could happen in nanocaust logic because I had switched one of their data structures from an ArcenSparseLookup to a Dictionary. Rather than trying to recode all that logic (bad idea), I just returned it to being an ArcenSparseLookup. This should solve the issue.
  • Thanks to nas1m for reporting.

• Fixed a post-refactor exception that could happen in QueueChatMessageOrCommand based on null being passed in for SFXItemInternalName instead of empty string.
  • The reason this was only happening post-refactor is that there used to be different message signatures for passing in a Context object, and the method call in the DZ thought it was passing null context, not null SFXItemInternalName. But since the change to no longer pass context, it wound up effectively passing null SFXItemInternalName.
  • I have just made it properly handle null SFXItemInternalName, so any other cases of that are now fine.
  • Thanks to Badger for reporting.

• Fix a DZ bug where the PerUnit data was not clearing all of its fields.
  • Thanks to Badger for fixing, and HarryT for reporting.

• Hardened all of the methods in AllegianceHelper against errors during game quit. There were a number of places where they were vulnerable to cross-threading exceptions during game teardown.
  • Thanks to Badger for reporting.

• The game is now a lot better about killing threads when it wants to go back to the main menu, or exit the game in general. Before it was possible that they could linger for a moment and throw errors.
  • This is one of those "how did that ever work without errors all the time on exit?" sort of situations. Nevertheless, it was still rare even now.
  • Thanks to Badger for reporting.

• Okay, took a much more aggressive approach with on-quit errors. The fact is, we simply have too many threads with too much error-handling to guarantee that none will error a we pull down the walls around them. And there's no guarantee that a mod will properly handle ThreadAbortExceptions, let alone anything else more specific. So let's handle this centrally!
  • When you quit the game or quit to the main menu, after it has done anything that we might care about (like saving an ironman save or something like that), it now gives a 3-second moratorium where it ignores any and all exceptions that are thrown. Those are all assumed to be spurious (because they are), and that means we don't have to worry about the fact that we can't halt a Parallel thread easily (that was one of many problems), or that certain code might not be set up properly for the abort codes.
  • This has no net negative effect on the game, because essentially this is just stuff that was running that did not need to be, and it's not going to hurt any data.
  • Thanks to Badger for reporting.

• The default planet naming style is now set to Standard.

• When you hit Reset To Defaults, previously it was not setting the planet naming style to the default, but now it does. Same with the planet count needing to be set to 80, and also with the map type needing to be set to the default (which is Realistic). Previously it was just randomizing the map type.
  • Thanks to Zweihand for reporting.

Simulation Performance Improvements

• Improved the efficiency of the strengthcounting background thread when it comes to looking at incoming waves and adding their strength to your totals.

• In general, improved the efficiency of strengthcounting by making it use a ton more background threads. This is one of those areas of the game that can be a real sim-speed limiter, it turns out.
  • In particular, with higher numbers of planets, this can peg even really new, high-speed high-core CPUs. I have it spreading out amongst CPU cores better now with more threads, but I still see it using too much, so I am going to make some further improvements.
  • This and metal flows planning are two areas where I should be able to make the simulation run a lot better for everyone, particularly in giant games, now that I can see that they are spiking up as problem points pretty frequently even on my Ryzen 7 4800H.

• MAJOR refactor of the strength counting logic to time-slice its work over the span of 10 frames (1 second).
  • As part of this, also discovered that too much Parallel looping over planets was a major performance problem inside this thread.
  • Even on an 8-core machine, this was spending more work getting those threads up and down than it was getting benefit out of the threads themselves.
  • Also, as part of this, I've discovered that the strength counting itself is one of those things that has a MAJOR impact on the game the more planets there are. I had wondered where all our performance was going with high numbers of planets in the past, and... well, here's the answer. This is one of the most longstanding mysteries of the game, solved.
    • Even now, with the awesome new time-slicing, the difference between me running this on 80 planets and 110 planets is "always buttery smooth" and "kind of choppy at times, but not so bad as before." This was with two AI factions and the fallen spire.
    • When I stick with 80 planets but add 10 AI factions and one each of most every other faction available in all 3 DLCs, plus two of a few of those factions, then the performance of this -- from second one onwards -- is abysmally low. I go from having something that takes about 26ms on average (that's lower than 100ms by a lot, so that's good), to something that takes 300ms on average (that makes for instant incredibly high sim lag) with spikes to 600ms. This is AFTER all my existing optimizations.
  • I think that what I'm going to do is actually spread this out even further, to time-slice this over 3 seconds rather than 1 second. My scenario with such an insane count of factions is not a reasonable one to play, but I have a very high-end CPU, and I would like for the base case to have a lot of extra wiggle room where it's not even coming close to performing poorly.

• More strength counting performance improvements:
  • The final step of copying the strength aggregation data from the working location to the final data spots that are used has been moved to the same background thread as the rest of the strength counting.
  • In my prior version, I had the planet strength counting split over two time slices. That is now split over 5.
  • In my prior version, I had the aggregation strength counting split over four time slices. That is now split over 20.
  • With this setup, I now get vastly better performance in my unreasonably huge test bed, and notably better performance in my regular use cases. However, there are some major downsides, so I'm going to redo this yet again from another angle.
    • Firstly, in the huge testbed case there are still some outlier spikes, which get worse as time goes on. So an initial sim speed that was better deteriorates over half a minute or so. I'm not entirely sure why that is, but I have some guesses.
    • Secondly, the human-facing data is only being updated every 3 seconds now, which really does not feel right. When I explore the map via a cheat, it taking 3 realtime seconds to appear feels very off-putting.
  • Don't worry, I have a plan. :)

• Okay! A whole new approach has been implemented for the strength counting thread. The short answer is that performance is through the roof.
  • Before, I was trying to time-slice at manual spots that I was personally selecting. That just wasn't working out very well, for a variety of reasons. We would have some slices that ran well over 100ms, and thus slowed down the sim, and others that were still a lot higher than I wanted, while others were almost no time at all. But... even worse, which parts are the most expensive is something that depends on the specific savegame (how many units, how full specific planets are, and many other things). So there's literally no way to make a generalized solution to this in a hand-designed fashion. It will always fail somewhere, making it completely impossible to solve that way. So...
  • The new solution that I implemented is ADAPTIVE time-slicing. Here's how it works: essentially, at certain juncture points it decides if it has been running too long, and if so then it terminates. Wherever it is, it doesn't matter. The next time it is run, it resumes right where it left off. To a non-programmer this might sound easy and obvious, but when you're talking about a linear function it is really not a common pattern to do something like this.
  • The metric that I wound up using was 40ms. If the current iteration is taking more than 40ms, then it stops for now. I want the remainder of that 100ms to be used for the rest of the sim. And honestly, since I can only check but so frequently how long it has been it will sometimes hit 60-70ms per slice like this, anyway. In a suitably rough location in a suitably huge save, it can get a bit over 100ms, but rarely. I've iteratively optimized that down so it doesn't happen much.
  • Overall this now means that the strength-counting data is slightly out of sync on clients and hosts, and it means that it is not up-to-the-frame accurate anymore (well, not always). In small savegames, if it can run all in one frame, it still will, of course. But for really huge savegames, it will take a bit longer.
  • I had a couple of test cases, and both of them improved dramatically.
    • One was a basic game with two AIs and the fallen spire. Under the hood, this meant 23 factions. I was testing on 110 planets, which earlier today was feeling a bit choppy even though it was close to being perfect sim speed. Now it's running without the chop, at full sim speed. It takes about 300ms (76ms of calculation-time, which is 40% of realtime) for the data to be updated now, versus forcing it to be updated within the initial 100ms span. Previously, this had meant that there was only 24ms of time to do ALL the rest of each sim step, but now there's lots more, so it runs well.
    • The second test case was a truly insane one, with 80 planets and 10 AIs and one of every faction along with two of a couple. Under the hood, this makes for 111 factions. Prior to this specific update, I was getting about 40% sim speed on my very nice CPU, and it was so choppy it was almost like a slideshow. Now I get well over 60fps, and almost perfect sim speed (95% or more), and it takes about 1.5 seconds for the strength counting data to be updated (670ms of calculation time, which is 40% of realtime).
  • This is... an incredibly dramatic improvement. A ton of game scenarios that were never possible before are now potentially playable, based on this early data. And since the code is adaptive, it should handle many cases such as super high unit counts or super high planet counts or whatever else. I'm going to test the latter of those next, just to see what it's like.

Options For Larger Maps

• The planet count max for all map types has been increased to 300 from a prior range of 120-160.
  • With 300 planets, just starting the game used to give very poor sim speed, but now that problem is solved. Framerate still takes a big hit even when you aren't looking at anything in particular, but it is north of 40fps for me on my dev rig, so it seems like an option worth having (with warnings in the lobby of likely performance issues if you don't have a very very nice machine).
    • I am reminded of the Supreme Commander maps that were at the largest possible scale, where they ran poorly on basically any machine that existed when the game released, but now they run fine. This is kind of like that, although it is playable on current machines if you don't have too many factions.
  • I tried with up to 500 planets, and it took 15 seconds to generate the map and then I was getting literally 20fps even though the sim speed was about 80%. That is still a titanic improvement from the past, where it would have been sub-10%. Nevertheless, 500 planets just feels way too laggy and likely always will given how the game is structured. I'm not quite sure where the new bottleneck is in the code, but the game just wasn't designed for that sort of planet count, anyway.
  • Planet counts higher than 170 and 220 now give you different warnings about possible performance issues if your machine is not very good. So you have the option for these mega maps, but hopefully also know what you're getting into.

Beta 3.703 Smooth Sim

(Released September 29th, 2021)

Improved Simulation Handling And Speed

• Making some changes to the threading model for AI War 2 to make things a bit more clear:
  • ProcessSimStep is now called ProcessArbitraryFrameOnMainThread, because that is FAR more clear and accurate. Even I was confused by the old name.
  • DoActualSimStep is, similarly, now called ProcessCoreLogicForArbitraryFrameOnMainThread.
  • Fixed a bit of logic that was running twice on the host and in single-player, eating up some extra processing time. Not much, but nice to clear that out (this happened as part of The Great Refactor).
    • Actually, this might have caused flickery lines, not sure.
  • OnClient_SecondsStoredUpForNextFrame is now OnClientOrHost_TimeStartedLastFrame.
  • All of the short-term sim context work is now handled on the main sim thread, rather than on a secondary worker thread managed by that.
  • These short-term threads are also now executed on the main sim thread via a Parallel call, which makes it so that they are able to happen much more efficiently in terms of how they get started and finish.
    • This is a MAJOR adjustment to the game, and likely to improve performance quite a lot, potentially on older machines the most (but really for anyone).
  • A whole bunch of other logic that I've hated for a long time relating to how the simulation kicks off threads and checks for gamecommands and so forth is a lot simpler now.
    • The idea is that we want to make sure there is minimal input lag in multiplayer, and that we're seeing as close to zero lag between "finished one thing" and "started another" as possible.
    • Previously, if you had a choppy main thread framerate for whatever reason (low powered graphics card and giant battle?), it could make for your sim speed being affected since that was being time-sliced over frames. That sort of time-slicing is no longer done, which leads to more stable simulation kickoffs.
  • This whole thing actually simplifies several areas that were really complicated before, and which were impossible to profile well, because they had noise from cross-frame data in them that made it impossible to know what was just general frame time, and what was actual processing time on these themselves.
    • This simplification should lead to some performance benefits in intense savegames in general, and in multiplayer basically all the time. Not that multiplayer is ready to go again yet since The Great Refactor, but this will be a big benefit for it once it is back up.

• In the thread timings details window (right-click the timer in the bottom left window), you can now see the last sim time in milliseconds, and the average sim times of the last 20 sim frames (2 seconds' worth).
  • If this is above 100ms, then you will have a slow sim because of the sim itself. If it's below 100ms, then if your sim is seeming slow, it's because of factors outside of the sim itself. That could be frame timings or other factors, but this is giving you the details on JUST the sim itself, which is useful for isolating what the problem is. In multiplayer, "sim speed" can be low because of network issues, for example, and this lets you see that.
  • Interestingly, I was seeing some performance ratios hitting down into the 80% range when looking at the traditional view of things, but I can at the same time see that the slowest actual sim frame time was 36ms. That is faster is actually needed in order to have a perfect 100% sim speed. So either the calculation is off, or something else is causing sim speed to stutter.
  • Until today's refactor of the central sim timings, I would have had no way to tell which. Now I can actually investigate it.

• Fixed an issue (that has been incredibly longstanding -- years and years) where after the lobby, your game would be acting like the sim steps were extra slow.
  • Also fixed a much more recent issue where after the lobby it would say there were a bunch of faction threads that had been inactive for a long while.
  • This sort of thing is trivial, but at the same time could be alarming to new players and is not the sort of thing we want people seeing right as they start a new game.

• When mousing over the clock in the bottom left and seeing the simulation speed tooltip, it could have a pretty wicked flicker. Fixed.
  • Additionally, when mousing over this or clicking into it, it now also shows you the framerate that we have calculated you are getting. Previously we only showed that on the main menu for whatever reason.

• The way that I have been calculating the simulation speed (for visual display) over the last years has been... cough... truly asinine.
  • It was some strange logic where it was kind of half calculating itself based on the time that it was called in the UI when the frame number had changed, rather than just doing a proper calculation from "frame number counts up to next time frame number counts up."
  • This has been corrected, and the timings are now vastly more accurate.

• Because of the combination of changes in this release, I now can see that I am often tripping the sim to start slightly slower than I should (about 95% instead of 100%), even while running at 120fps and with a sim load that is 30% of what is available, which is leading to some slight choppiness as well, probably because of how I am handling this between frame-timings.
  • Ironically, unity themselves had something along these lines themselves recently: https://blog.unity.com/technology/fixing-time-deltatime-in-unity-2020-2-for-smoother-gameplay-what-did-it-take
  • I can't blame my own frame timings being off on them, but the mistake that I made is very similar to theirs. I'll have to get that straightened out now, too.

• Well now! Here's another REALLY nice improvement, which will have a positive impact on multiplayer and single-player.
  • Essentially, I smoothed out the sim-steps by sampling what recent frame-times were (the most recent 30 only, no matter the framerate), and then taking the average of them. If the next sim step is not yet ready to start based on our target of "every 100ms," but 55% of our average recent frame time WOULD push us past the sim step start time... then I run the frame on this sim step anyhow.
  • This essentially works out so that we get between 99-102% sim speed on average, with a slight bias towards being a tiny bit fast (that's why 55% instead of 50%), whereas before it was averaging only 95% pretty much all the time.
    • Why the bias towards being fast? Well, ship movement gets slightly choppy when the sim is running slow. The UI layer is expecting things to be running at exactly 10fps on the sim, and is basing its linear interpolations on that being correct. Being slightly behind on a lerp will still look smooth, but being even a bit slow (well, more than 1% slow anyhow) introduces micro-stutters.
  • It has been quite noticeable when you watch ships move around, how they have micro-stutters in past versions, and it has always bugged me. That micro-stutter is now gone.
  • This also has positive effects for multiplayer, and really single-player also, in making the clock move at a consistent speed rather than slightly too slowly, or at slightly different rates on computers with different framerates. I believe that in the past the drop in sim speed, and the micro stutters, were worse on lower framerates. When the frame times are larger (more coarse), it's harder to get exactly to the 100ms interval I'm seeking.
    • At 30fps, every frame is 33.3ms long, for instance, so previously the sim steps would be off by a range of 1-33ms. In the new system, by splitting the difference the way I am, the divergence can only ever be 1-16ms at 30fps. This is still large, but it's now 16% of the total runtime instead of the (huge) 30% of the runtime of a sim step.
    • At 100fps, we get 10ms per frame, and before it ranged from 1-10ms off. Now it just ranges from 1-5ms off, which is much better.
  • TLDR: gameplay in general is now smoother visually, on all hardware and all framerates, and multiplayer will diverge less between machines of different power levels.

Bugfixes

• Quieted the error "Entity in CenterPlanetViewOnEntity null!", which could happen in some circumstances when clicking through wormholes. It's not an error that is actually worth talking about; we should just ignore it.

• Fixed an exception that would happen whenever you moused over wave notifications. This was another case of my efficiency-improvement code hitting a legacy case which errors on purpose to let us know. It was supposed to give us compiler-level errors, but unfortunately that didn't work as hoped, so now we have to find them in the runtime.
  • Thanks to CRCGamer for reporting.

• Fixed an oversight in the great refactor, where I forgot to actually clear the AIP info between loads of the game! So that meant that starting a new game after having been in a game with AIP would leave you with extra AIP and a strange AIP history. Loading saves was fine.
  • Thanks to Arctic for reporting.

• Fixed the bug (starting from just before the great refactor) where every time you load a savegame it would give you another 10 AIP. Yikes!
  • Thanks to Lord Of Nothing, Badger, Strategic Sage, Arctic, and HTL9000 for reporting.

• Fixed a pre-refactor bug that was introduced when I redid the notifications system. The spire debris notification always showed zero, and would trigger errors when clicked.
  • Thanks to Vinco for reporting.

Youtube Threading & Multiplayer Modding/Programming Guide

Coding With Chris: AI War 2's Threading & Multiplayer Part 1: Challenges

Coding With Chris: AI War 2's Threading & Multiplayer Part 2: Data Flow For Unusual Solutions

Modders: Do you need to watch all of the above?

1. Not one bit if you're not coding anything. If you're making xml mods, you can completely ignore the above.

2. If you already think you understand how RTS networking works, then just read the below images and then skip to about the 30-minute mark in the second video.

New Diagram For Modders/Programmers: AI War 2 Threading Model

New Diagram For Modders/Programmers: Multiplayer Data Transmission Model

Beta 3.702 Bugfixes And Mod Conversion

(Released September 28th, 2021)

Bugfixes

• Fixed a bug with all hacks where if the Fallen Spire was not enabled or installed, you'd get endless errors instead of the hacking screen.
  • Thanks to Arctic and UncleYar for reporting.

• The Zenith Architrave now does a lot more verbose logging of its externaldata in case anything is wrong with it.
  • From analyzing a specific savegame, it looked like there was something wrong with it, but now that does not seem to be the case.

• Fixed a minor nanocaust deserialization issue that would cause multiplayer clients to drop their constants and reload them every second or so. This was found during code review while looking for something else. Not a biggie, but could have been annoying.

• Fixed two deserialization errors that meant that any savegames with nanocaust in them could not be loaded.
  • Thanks to Magiik for reporting.

• Fixed a serialization issue in the dyson sphere faction that would throw warning errors, but not actually cause any problems.
  • Fixed a similar one in Nanocaust.
  • Thanks to Magiik for reporting.

• The scourge as a faction were throwing errors and shutting down in Stage2 if they were included in your save in certain circumstances.
  • The scourge have had their Stage2 instrumented more to give us more informative feedback when that happens.
  • It appears that some Scourge Spawners could sometimes be missing their externaldata. This is now fixed.
  • Thanks to Magiik for reporting.

• Okay, when I reorganized the entire savegame file, one of the things that I changed was how galaxy info was deserialized relative to the entities in it. Normally entities are units -- ships, structures, etc. But sometimes they are wormholes.
  • Long story short, I had an oversight where I was calculating which wormholes are linked to which other planets BEFORE we had loaded any wormholes out of the savegame, meaning that each planet thought it had no links to any others. That meant that your movement commands between planets would fail after you loaded a save, but not after you started a fresh game.
  • Fixed, and this also fixes existing saves if that matters.
  • Thanks to Daniexpert and Andriy Mykhaylyk for reporting.

Mod Updates

• Civilian Industries has been converted over to work in the new framework, although it has not yet been tested. There is a four and a half hour youtube video showing the entire process, with commentary and advice on a variety of coding things.
  • https://youtu.be/3Aw34XjO9nk

• The following mods have been hidden for now, since they are nonfunctional for the moment.
  • AMU (SirLimbo is actively working on this. You can follow him on the modding discord channel. This may be quite a while coming.
  • Devourer Chrysalis (Again, NR SirLimbo is working on this. It's more than 100 hours of work, so it may take a month or two).
  • Kaizer's Marauders (Same notes as the above two, SirLimbo is still working on it).
  • Macrophage Histiocytes (and spire integration) (These are by StarKelp, and I may have time to fix them someday, or he might get around to it, it's hard to be sure yet).
  • The Reprocessors (this is a CRZgatecrusher mod that needs updating, and I haven't heard from him yet).
  • AI Shield Generators (this is a cml mod, and this one is going to be retired, with the author's permission, as its functions are going to be built into the base game with consultation with the mod author).
  • This should solve any confusion about which mods do and do not work. All of the other mods that you see are working so far as we know, although of course with Civilian Industries that has not been tested at all yet, but I have done the conversion as part of this build.

Beta 3.701 Natural Object Order

(Released September 23rd, 2021)

• Made it so that saves from 3.700 are no longer valid to load, because they are SO broken there is nothing new we can learn from them now.

Bugfixes

• Apparently we had never really used bool toggles on factions prior to now, because there were some errors lingering in there from way on back. During TGR, I converted the "debug mode" options for various factions to all use those, and that then led to errors in the faction screens for them. Fallen Spire "Super Easy Mode" is one example of those.
  • This all works now!
  • Thanks to Vinco and Magiik for reporting.

• Adjusted things so that the "Skipped an extra mapgen call that was substantially the same as one that was run X seconds ago" won't be hit if there are changes in the number of factions compared to now versus then. Just in case. I am not positive this was a problem before, but it might have been, and now it can't be.

• The warning for things like "Roaming Enclave Migrant is 23 characters long, and needs a display_name_for_sidebar variant that is 18 characters are less." is now an outright error, rather than a quiet thing in the log. Since we are making sure that mods are updated and such anyhow, this is a good time to make sure that we don't have things like this sneaking through.
  • Fixed the two instances of this sort of thing that were previously in the codebase in DLC3.

• Fixed an issue where NaturalObject (no owner) faction was not always set to faction index 0 anymore. This was something I was already aware of, but wasn't sure if it would be a problem.
  • It turns out that this was a major problem, and would cause various ailments such as units not reacting to aggro and releasing units, to all sorts of other potentially unknown stuff. This will not fix existing 3.700 saves that had this problem, but any new saves will not have the problem.
  • Additionally, this was causing units to not respond to move commands properly between planets. This again isn't fixed in existing 3.700 saves, but any new saves no longer exhibit the problem.
  • Thanks to Vinco and Daniexpert for reporting.

• Fixed some general issues with GetControllingPlanetFaction() and similar methods where they could now return null at some points, rather than returning naturalobject when nobody owns a world.
  • Thanks to Daniexpert for reporting.

• Fixed an exception where if you tried to change the main AI color in the lobby, it would error.
  • Thanks to Magiik for reporting.

• Fixed a rather egregious and VERY longstanding bug where if an unclaimed unit (a capturable unit) was placed into a LazyLoadSquadWrapper, it would return null from that squad wrapper.
  • This was potentially never a real problem before, but we now use lazy load squad wrappers on fleet centerpieces, which meant that this bug now showed itself by deleting all capturable golems, arks, transports, and so on.
  • This bug is at least 3 years old, and who knows what it was affecting (maybe nothing, honestly), but it just had now found new expression. Glad to have that fixed.
  • Thanks to Vinco for reporting.

• Fixed exceptions that could happen in a number of places if you tried to get the wave info, because it was using the old ToString() method rather than the new efficient method that writes to a buffer without causing a lot of garbage collection.

• Fixed an exception that would happen in the AIP history window, because it was using the old ToString() method rather than the new efficient method that writes to a buffer without causing a lot of garbage collection.
  • Thanks to Daniexpert for reporting.

• Fixed a typo in the MarauderOutpostRaiderPerUnitBaseInfo that had it linked to the outguard class instead, which then led to errors. This will still be broken in old savegames, but is fixed in any new ones.
  • Thanks to StarKelp for reporting.

Youtube Deep Modding/Programming Guide

Okay, here's the modding guide for AI War 2 now: https://www.youtube.com/playlist?list=PLc2dK5ROW0WCjIHUBRJPp5oiJkTIMZupL

Modders: Do you need to watch all of the above? No, absolutely not. The timestamps do point out some interesting things that you may want to listen to, though, if a topic strikes your fancy. Even the regular code developers, like Badger, may find some of those sub topics rather edifying. You can click the timestamps in the video description, and it takes you right to where I am examining and discussing some particular piece of code. This jumps all over the place for different examples as appropriate, and does some work on AMU but not remotely the whole thing. It also shows some simple and complex faction data from the base game, and discusses both the theory behind these designs, as well as the practical uses of it. I feel like a professor trying to impart an entire textbook in a 3 hour lecture, but such is life. I will be available for questions and consultations. I want people to have a good experience with their transition to the better format, and also understand the sheer power and performance gained by doing the transition.

Beta 3.700 Bulk Of The Great Refactor

(Released September 22nd, 2021)

Known Issues

• This is a brand new version of the game in a whole lot of senses, so it is expect there are going to be a bunch of random errors that are not known. Please don't stress about it. If you just report them, most should be very quick to fix. With that said, here's the list of things we do know are not working at the moment:

• Tutorials are temporarily broken.

• Beacons for factions are temporarily removed. They will be re-implemented in a more efficient manner soon.

• Most mods are probably broken completely or partially.

• The kills and deaths data that you could get is no longer present. It was limited and stupid and boring, and I'm going to give you cool data, with graphs, instead.

• Random Factions are broken.

• All savegames prior to this version will no longer load in this new version, which is by design: we have completely reworked the savegame format to be more efficient.
  • I should warn you that I am going to break even NEW saves several more times prior to 4.0. This will be in pursuit of making the Fallen Spire and Necromancer factions feel better to play with, and to add ship loadouts (the AI War 2 version of modules) with minimal CPU overhead during gameplay, and a few other reasons like that. I will let you know when I'm done breaking saves, but until that point please consider any save as potentially invalidated by the next version coming out.

Random Features And Fixes

• Added a new is_immune_to_reconquest_waves="true", that can be used to make it so that a given faction won't have reconquest waves launched against them (other waves are fine).
  • This is also newly applied to both flavors of dyson sphere, which is new. This should let them get a lot less hassled.
  • Thanks to easytarget for inspiring this addition, and Badger for the notes on how to quickly add it.

• The "A Tale Of Two Stars" quickstart has been removed, because with the Dyson Sphere changes coming in this build, that scenario will no longer work. Likely there will be even more changes to Dyson Spheres over time, so it would probably just get a bit more busted with time.
  • Thanks to Puffin for reporting.

• For purposes of damages/penalties to damage output of weapons when under a friendly bubble forcefield, weapons now also consider themselves to be under a bubble forcefield if their parent unit is the one emitting said bubble forcefield.
  • For example, a Riot Cruiser that has a bubble forcefield and 10 guns on itself, if its bubble forcefields are up at all, its guns will work better unless it has the usual things that let it not cause that. Previously, it would only get the "under bubble forcefield" status on its weapons if some OTHER unit was providing said bubble shield.
  • Thanks to CRCGamer for reporting.

• Moved the mark level scaling data for Fleet Officers from being ZO only to also being applied to the base game to simplify balancing.
  • Additional revised offensive scaling of Fleet Officers to cap at 6x base damage instead of x7. This actually does cut a few of the more outrageously upgraded options by 50-70K DPS.

• Adjusted Combat Factory internal drone metal rate up from fixed 1000 to 1500 plus 250 per mark level.
  • Combat Factories have been pretty slow at replacing their drones for a while and unlike most of the support factories they have a much larger number they maintain.
• Adjusted Scrapper Factory internal drone metal rate to gain 250 per mark level.
  • Since this factory type essentially keeps a stock of knock-off autobombs as drones it just made sense to let it make replacements faster with higher mark levels.

• Adjusted Shieldwall Battlestation to be a much more compelling option overall.
  • It no longer gives units under its bubble shield a damage penalty.
  • Said shield also no longer shrinks when taking damage.
  • As it uses classic scaling and thus defenses only improve by up to 4x base value by Mark 7 it has been given an additional 25% base shield value.
  • A new mechanic where as long as its shield is up it takes reduced damage to said shield has been applied. Base 10% damage reduction with an additional 4% per mark level. This effectively means that at Mark 7 it has 34% damage reduction to shields.

• Adjusted Grenade Corvette strength bonus modifier slightly upwards to account for higher damage reliability with shot compression.

• Bumped up Harasser variant of MLRS Corvette base damage from 18 to 20. (Note that these ships can fire 8 shots of that damage per salvo)

• Previous adjustment downwards on target cap for Electric Bomber wasn't quite sufficient alone after shot compression made reliability higher. Further adjustment of base damage down from 900 to 750.

Quick Starts Data Overhaul

• Added a new ultra-micro format for "last lobby settings" and for quickstarts.
  • This isn't really so much for performance purposes as it is for forward-compatibility purposes.
  • I'm about to absolutely revamp the entire savegame format and how a lot of data is stored for basically every faction that exists, and I want quick starts to persist past something even that drastic.
  • Why the huge revamp of data? Primarily multiplayer, but honestly this is something that will likely yield performance benefits and added correctness in single-player, too.
  • We basically never do such huge revamps of data, because we absolutely loathe breaking savegame formats, but at the moment the time for half-measures is past. So we'll do it once, and make it really count.

• Starting to recreate all of the quickstarts. The first one recreated is now Helping Hands, which is identical in form and function to what it used to be, but now part of the micro-save format so that it will survive the upcoming purge night.

• All of the quickstarts from the base game have been updated to the new micro-save format that will be the only thing future-compatible after this build.

• Fixed a bug with Helping Hands 2 from The Spire Rises, where it was not working if you didn't also have DLC2 and DLC3 installed.
  • It is expected that there were other similar errors in some of the other quickstarts in DLC, but I am just fixing them all as I move them to the micro-save format, so I can't say for sure.

• All of the The Spire Rises quickstarts are now safely in the new micro-save format.

• All of the DLC2 quick starts are now fully recreated in the micro-save format.

Make Your Own Quickstarts!

• makequickstart, [folder name], [quickstart name]
  • Causes a new quickstart to be saved to the quickstarts folder in the new ultra-brief quickstarts format.
  • This quickstart format is required if you want to have future-proof data that does not break with future versions of the game.
  • Example usage:
    • cmd:makequickstart,1-Basic,Helping Hands
      • This would make a replacement for the basic helping hands quickstart.
  • Please note that you need to still add a .tooltip file next to any quickstart. You'll have to do that in your filesystem on your own, but it's just an ASCII text file with narrative information inside it about what the quickstart is about. Otherwise that tooltip will come up blank.
  • When you do this process, it also gives you a note telling you which dlc and mods are going to be required for someone to play that quick start. We had lots of cases in the past of accidentally requiring more than we meant to.

Bugfixes

• Improved the handling of DoForPlanetsParallel() so that it can't spam errors when you're exiting the game, or exiting to the main menu.

• Put in some extra debugging in DefinePlanetFactionDefenseTypesIfNeeded(), as it could throw exceptions in some cases.

• Fixed things up better so that any DLC can use features from any other DLC without incident, despite mods being required to have the parent DLC installed.
  • We originally had some cool chain lightning units in DLC3, which got removed after we introduced the restrictions, but we'd like for you to have those back!
  • There's also a detailed writeup on all this here: https://wiki.arcengames.com/index.php?title=AI_War_2:Features_From_DLCs

• Fixed what looked to be a bug in recent versions with outguard not actually spawning. This was based on code analysis, not testing, so let us know if something is off.

• Fixed a number of potential cross-threading issues with nomad planets using the main thread context on a background thread.
  • And some with wormhole borers, though fewer of those.

• Discovered and marked a bug where MP clients cannot cancel hacks in progress, but have not fixed it yet.

• Been meaning to do this for a while: multiplier_to_all_fleet_caps has been removed from the game.
  • It was previously on all turrets, set to 0.2. It was only there for historical savegame purposes, and messed with the unit cap math badly.
  • There are other modifiers in place about weapons strength or hull or shield health, and those are used in a variety of cases and we're leaving those intact and in place.
  • The ship cap one being removed means that if you have a mod that is seeding turrets, you'll want to remove this multiplier and then probably divide your number of seeded turrets by 5.
  • This should solve a number of longstanding but rare bugs that we keep seeing with the TSS and other turret-granters, but we'll see. It certainly won't hurt.

• Fixed typo in "ADVICE: Next Steps".
  • Thanks to Ben Parrish for reporting.

• Fixed a few grammatical errors in EncirclingSpider and IrefulMLRSCorvette descriptions.
  • Thanks to Fenrakk101 for reporting.

Rip And Tear For ExternalData

• All savegames and quickstarts older than 3.700 are now broken in the latest beta branch, and there is now a popup message explaining that any older saves you have should be finished on the older versions.
  • This is something that was not done lightly, and we're going to keep breaking beta-branch saves for at least the rest of this week as we refactor things to run in a high-performance, MP-friendly, modder-friendly, non-error-prone way. No more half measures.

• The StatisticsSet class, which was mostly unused except for some small bits of data that were stored in a way that was hard to get at, has been removed.
  • The actual data about offline achievements has not been in here for years, for example. As has the info about achievements won during a given specific save.

• Our ArcenExternalDataLookup class, ArcenExternalData class, ArcenExternalDataPatternTable class, ArcenExternalSubManagedData interface, and ArcenExternalSubSubUnmanagedData interface have all been removed. ArcenExternalDataPatternImplementationBase is also gone, and all the related stuff, really.
  • The generic "let's talk about external data attached anonymously to an object" concept that has plagued this game since our earliest versions is now going away.
  • For those who have coded factions or mods, please don't stress -- the vast bulk of your content will still work once I'm done rebuilding this in a more productive and direct way.
  • The problems with the old approach (of connecting the Core and External data parts together) were many, and reflect that it was originally kind of a prototype concept that never got refactored significantly enough because we didn't want to break savegames. I've spent the last year and a half trying half-measures to work with this stuff, but now it's time to go all the way.
    • Among problems with the old method: memory leaks, bad performance, inefficient data storage and transmission, no deliniation of data that we really consider central versus data that is just needed for host-only execution, and more.
    • There were also shifting and evolving standards in use over the years, leading to better and better code over the years, but still pulling along old bad code. Everything is getting a facelift out of this.
  • Last year and earlier this year you can see a number of cases of me trying to bring this to heel in a better fashion, but it's just crossing a line now where it's way too wild.
  • My one rule is generally "don't break legacy savegames," so you know this is serious business at this point. Time to get a lot done all at once to make this worth it.

• Old concept being removed: you can have "externaldata" which has any amount of sub-data on it, attached to various objects.
  • The precise nature of how that connection is forged is actually something that exactly zero people fully understood, and at any rate it was a very ineffective method for gathering this data. There were three levels of pooled wrappers in there, it was nasty.
  • The types of objects that could have externaldata were World, Planet, PlayerAccount (never needed that, though), Unit (this was used all the time, but then much less over time), and Faction (this is the really meaty one that tends to matter, but even this is problematic in the extreme for older factions in particular).
  • Please note that PlayerAccount is permanently losing an kind of external-data analogue, since that was never actually needed. The new network-enabled personal settings actually are much closer to what we had once thought we might do with these, and those are highly organized data that works really well.

• A LOT of code that was dedicated to maintaining old savegame compatibility has been removed.
  • This was thousands of lines of code that made the codebase that much harder to work with.
  • This alone is going to help performance for network transfers and savegame loads, given just how many versions there have been.

• Stripped out some of the excess of paranoid canaries that I had placed in game units a number of versions back. There are still some in there, but it's more efficient having fewer, and the reduced count will be sufficient for our purposes at this point.

• Previously, game units could only have a maximum of 15 systems, per their savegame format. This was very very sensible in a time before loadouts, but now this is being expanded to 63 instead.
  • This allows for us to have a great number of optional systems on a unit for loadouts. We still think 63 is an absurd number, but that's the sort of buffer zone we want to be in, really.

• Removed DoomData, which hasn't been used ever but was kept as an example bit of code for a long time.

• In our external code, right now we have some temporary classes as bases for things that will later be split off into other files and have their data style changed around.
  • Right now the goal is just getting things compiling, but having the external factions be braindead. The places where many things will change from the root up are marked with EXTERNALDATA_TODO for now.
  • It's going to be several days at minimum before we have a truly functional build again.
  • It's worth noting that a lot of giant gaps are there right now, like faction allegiances even being off once you load. This is because the config data doesn't have a pathway to the external data at the moment.
  • At the moment, however, you CAN load up the lobby and everything there is 100% functional. It doesn't actually kick off factions after the lobby is left, but that's not the point -- the point is nicely-siloed data that we understand the boundaries of.
  • At this point, after you load a quick start or click start from the lobby, it runs part of mapgen, but most of it is disabled. You then instantly win the game, and errors spam the world.
    • Here again, this is kind of the point... it's impossible for us to reliably have known prior to now what a client would or would not have on it in MP, and everything was very messy. These errors likely were the same ones that were going to show up on a client after getting a map from the host in the recent betas, but we would not have known until testing.
    • My work tomorrow is going to be centered around building in stable and clear data bridges between Core and External: this will have to be well-organized, easy to understand at glance, not prone to memory leakage, and equally as powerful as the tools that they are replacing.
    • The idea from that point on is to start getting the most-core data linked up, that stuff which is used by every faction, or by the human or AI factions in particular. And the idea will be to separate out that which is present for everyone, and that which only the host has, and that which the host automatically shares with the client as it is dirtied.
      • After that is done, and we have the basic factions running again, then it's a matter of going faction-by-faction and converting them to this new format. How much work that will be remains to be seen, but I anticipate being done with most of that this week.
      • To be clear, this should then have covered essentially all of the major things that multiplayer is currently lacking in data transmissions, and also have improved the performance of getting to said data in a single-player context as well.

• The old ShipStats information had data on how many ships of each sort you had killed and lost, by mark level. This was actually stored on every faction.
  • This is not very interesting data, and is very very hard to show. We're going to track and show much more interesting data, in a completely different way. So this is being removed.
  • This was the data that was "experimental" when you right-clicked the metal view. For the moment that's blank, but soon I'm going to have actual graphs.

• Adjusted all of the ExternalData serialization to no longer have the old data for deserialization logic. This was less to remove in the main game, but still several hundred lines of code.
  • The idea is making this cleaner and easier to read since we're in the process of breaking saves anyway, and it also makes it easier for me to move forward with splitting and cutting up the data.

• Removed some ultra old rally point code that hasn't been used since fleets were added.

• (Almost) all of our older code and folders and such which used to say "Mercenary" or "Merc" or similar have been updated to say Outguard instead.
  • This simply makes things more clear, particularly since we've been calling them both things in code for a few years now.
  • There are still a few references to mercenaries in a few files and filenames, but it's WAY lesser now, which is nice for avoiding confusion in the future in particular.

• Removed old features that had been retained only for really really old savegames:
  • ship_cap_multiplier_strikecraft, ship_cap_multiplier_turret, ship_cap_multiplier_frigate. These were all tied to the IGC in older versions.
    • There was a LOT of related code for this, which is all removed now, hooray.
  • The deprecated custom-flagships xml.
  • FCE xml and a bit of related code.
  • supercharges_strikecraft_ship_cap_of_rest_of_player_fleet and related code.

• Old logic for removing AI frigates has been removed. We no longer need to deal with those savegames.

• Removed the HostOnlyAggregatePlanning background thread, which I added a week or so ago but never wound up needing.

• Removed some FleetFormation logic and xml that had never been used and never been tested. It was an old prototype just clogging things up minorly.

Reconstructing ExternalData Links

• FactionCenterColor and FactionTrimColor have been removed as direct variables from the Faction class. So has FactionType Type and and SpecialFactionData SpecialFactionData.
  • In place of these, ConfigurationForFaction Config is now always permanently on the faction, and it has a direct link to this data.
  • Previously it was frustratingly common to have two copies of various pieces of configuration data, or even three, and we want to avoid that. Trying to keep that in sync serves no good purpose, versus having one place where each piece of data can be queried is very useful.
  • However, it's nice to have nice wrappers around all of these so that the same code will work as always have, so this is what we've done.
  • This may also fix a bug we've had for a while with the "subsequent faction colors not being set on factions of the same type" for some subfactions, but if not it at least makes that easier to fix in the future.

• The "MissingFaction" is set as the default entry for the factions table so that it will automatically be chosen in cases where a faction is, well, missing. Removes the need for some code.

• Added in some more wrapper methods for looking up custom field data for factions in a direct sense, rather than potentially getting stale data from one of the secondary places it is stored. We never want it to be the case that the UI is lying to you while having a secret separate value behind it, so this is part of that. We had accidental cases of that above-mentioned problem before.

• ArcenSimContextBase now has an abstract AmIAHostOnlySubContext on itself, and a ShouldThisMethodBeSkippedBecauseThisMachineIsClient() direct method.
  • This lays the groundwork for us removing some of the RNG divergences that were happening in multiplayer prior to now. This also serves as a guide for how we're going to be splitting our External dlls. Honestly it's making me review all the methods and make it really clear what can do what, which is pretty darn useful to have to do right in advance of that other surgery.
  • Added a new ArcenClientOrHostSimContextCore, and ArcenSimContext and ArcenSimContextBase are getting a TON of additions and changes. Way too many to note the details of in this format.
  • Added a new ArcenHostOnlySimContext that is for the host-only sections of the game.
  • QueueLogJournalEntryToSidebar and QueueChatMessageOrCommand no longer require contexts to be passed to them (they were not using it anyhow).
  • Lots of methods now either require a ArcenHostOnlySimContext, or a ArcenClientOrHostSimContextCore.
    • It is very rare to now require a root ArcenSimContext on a primary logic method, and this is part of how we're now controlling what methods run on the client or host (which we were already doing, but more haphazardly), AND controlling which random generator is used (this is the bit that is new, which will help stop that "units go over here and then zip back over on the MP client" bug).
    • That said, ArcenSimContext is absolutely still used on all the various worker methods that are called from a variety of contexts. Aka some code that might be used in any and all shared circumstances.
    • To make things even more clear, and prevent un-examined code from upgrading into this without examination, ArcenSimContext is now called ArcenSimContextAnyStatus to denote we really mean that.
  • ILongRangePlanningContext is now ILongRangePlanningHostContext.
  • ArcenSimPlanningContext is now ArcenSimPlanningCommonCoreContext_DoNotReferenceDirectly. It should never be used directly anymore, as it's ambiguous as to whether it's on the client or the host.
    • There are now intermediate ArcenHostOnlySimPlanningContext and ArcenClientOrHostSimPlanningContext classes that inherit from it. Mostly this is a formality, but it does add clarity and keeps consistency with the rest of the code.
  • A metric ton of code has been adjusted for all of this, in some cases removing things that were passing around some form of Context that had no business doing that. These are meant to stay in some reasonable sync, and sometimes even some UI choices could cause problems.
  • Because of the way inheritance works from classes in C# (single inheritance only, interfaces aside), there are a few methods that are duplicated across a couple of central classes.
  • THAT said, this leads to less duplicated code elsewhere in end-implementation classes, AND the inheritance tree of the "context" types makes it so that people literally can't call methods from improper spots.
    • If you're a modder and you find yourself trying to work around a method call that doesn't have a compatible context type, then either you just need to call yourCurrentContext.GetHostOnlyContext() and try that, or else give up and do something else because this would be a MP sync issue or other data invalidation issues would be caused. Or let me know, and I can always look into making something more permissive. But honestly other than people forgetting they need to call .GetHostOnlyContext(), I don't expect any troubles.
  • So what's up with calling .GetHostOnlyContext()? Isn't that kind of a waste of end-coder time? Actually... no! It's super important. The host-only context is used for parts of the code that only run on the host, and it has its own separate RNG that the shared clientAndHost context does not have. This means that all of what runs on the client and host has the same number of RNG calls, yielding a deterministic result. But peppered in there are some host-only calls, which previously were breaking sync but now just require a call to .GetHostOnlyContext() and sync is preserved while still keeping the same code structure you're used to.
    • This is round one of several rounds of fixes that could also be described as idiot-proofing (where all of us are equally idiots, and not meaning that in a pejorative way). The code is now demonstrably clear about when it cares what sort of caller is around, which is a big step forward for the next round of this, which will involve splitting things between dlls in order to separate out a new client-accessible layer of data from lots of host-only code that runs against it.
      • Put another way, this is getting around the problem of having to replace tooltip appenders with calls to the host, or doing all of the notification calculation on the host and then transmitting lots of complicated data. We're instead handling this in a much more tri-layered fashion, based in general on the MCV (Model-View-Controller) design pattern.

• DoOnSelfBuildingCompleteLogic has been moved from the scenario to the faction class, so that it's able to properly do its job. Before, this would have been a mighty big annoyance for any modders who made a new faction.
  • Same deal with DoOnInternalConstructionCompleteLogic, although nothing actually uses this one yet.

• The "ExternalCode" project, which is the main open-source AI meat of AI War 2, has been split into two parts.
  • One part remains in the existing dll, which is the most "permissive" part.
    • Anything in this dll is stuff that we're intending the client in multiplayer to have access to. (That is a statement of design intent, but I have a lot of work to do before that part is actually fulfilled.
    • Also, anything in this part can be freely examined by the UI, on the host or the client. I mean, thread safety aside, the UI is allowed to know about these things, and these things exist on every computer in multiplayer (again, talking about design intent, not current status).
  • The new part is called "AIWarExternalDeepProcessingCode," and is meant to be only things that the host has access to. Also, the UI cannot directly query these things.
    • Right now we have moved the faction processing logic, "on death effect" code, the main mapgen code, the "hacking is happening" code, and the "long term planning" code in general.
    • Please do bear in mind! Right now there's data mixed up in many of the files that were moved, along with the logic that was moved. Some of this data is meant to be host-only, but a lot of it is not. The next job will be splitting up all of these files and restructuring how the data is found, and that goes hand in hand with also re-teaching the Core code from AI War 2 how to reach these new split-up forms of data and code.
  • Please note that all mods will be required to follow this same double-dll format, which we will be enforcing by a quick scan of your mod code as it is first loaded. More on this will be available when this conversion process is complete.
    • The goal with this structure is to make things multiplayer-safe, and separate the data from the code a bit better, and separate the host-only code a bit better from "everybody runs it" code.

• Updated our build scripts so that when we build an external project from within visual studio, it will now properly set them into ReliableDLLStorage. Prior to this point, any compile-breaking changes needed to first be copied there by running the entire command-line build script.

• My external code loading framework is being put to new use in terms of helping ensure data-correctness.
  • For classes that will go into deep processing or the client-safe data processing, they will all inherit from either IExternalDataAndSimCode or IExternalDeepProccessingCode.
  • The game now checks to make sure two of these same things are never in the same dll, and additionally, that no dlls that have deep processing code are referenced by any classes with DataAndSimCode in them.
  • This is really important for preventing errors that would affect multiplayer clients, among a few other maluses, and so by doing this very quick check on startup we can make sure that all of our own code, as well as any mods, all conform to the standard that prevents the most common MP-only issues from happening.
  • Put another way: if you've just coded a mod, and it works for single player, you want to be pretty darn sure that it "just works" in multiplayer without having to go test it there. It's the same with any changes we make to the core game, or in DLC. All of this code restructuring, along with these checks, are how I ensure that is the case at a very low level. If MP has a problem that is unique to it, I want that to be something that's my central problem to solve, not something that faction designers or modders are chasing on their own stuff.

Restructuring The Upwards Link From Core To External, Part 1

• Previously, we had a thing called "ExternalDataPattern," and some of the DLC content had to be included in the base game or else beacon factions would mess up the ability to load quickstarts. And last-settings would get messed up if you disabled a DLC and went back into the lobby, too.
  • Thankfully, now we don't have this problem at all because of the new "micro save" format that is now used for those two purposes as of this build. Therefore, the new equivalents of this old sort of data are now able to be stored in each DLC properly (this has the same implications for mods, so this is a good thing for mods being usable for sure).

• Added the following two sets of data/code pairs:
  • ExternalDataAndSimCode, which is for clients to know about and the ui to be able to see and use.
  • ExternalDeepProccessingCode, which is only for hosts and single-player, and contains all of the deeper logic and working variables and whatever else. We don't want this to accidentally get sent to a client at any time.
  • It is worth noting that the above things can have 1:1 pairs (faction data in the sim code that the ui can also see, and then deep processing for said faction), but in many other cases we don't need a faction to be involved at all, or have several sub-parts to a faction (unit data, etc), or whatever other designs we want.
  • As part of the two above sets of data/code pairs, added the following three categories:
    • World. This is something attached to the world, and it can run whatever logic you want to, kind of like a faction would have a chance to but without the faction.
      • Internally, some of the logic that we have very confusingly on "Human" right now -- there can be many human factions, and we generally want to run this stuff once for all of them -- will move here. This is also for the things that are human-faction-type agnostic.
      • Externally in mods, we expect to see the FakeExecutorFaction from AMU move into this. We should never need to have a fake faction again, as there is now a clean and fully supported method for doing that same sort of thing here.
    • Faction. This is where the meat of the game logic lives for anything that isn't just the autonomic style of unit AI.
      • Internally and externally there will be some swapping back and forth of what is loaded where, and the SpecialFactionLogic class in general will be subsumed into this.
    • Squad. This is data that goes on an actual individual ship/structure for whatever reason. Some factions are designed where this is needed, others are not.
      • Having deep processing available for individual ships is now possible, although that would be a little bit on the strange side. We have a way of handling it with efficiency, but it's an oddity for sure if you structure it that way.
  • ExternalDataPattern used to be loaded very early on in the load process, and these new ones are loaded only slightly later. Other xml will be referring to these.
  • So far, each of these is just blank slates, but they'll all have unique methods on them, and be the vessel through which most communication happens between Core and External and ExternalDeepProcessing.
    • It is worth noting that these all implement time-based pools that are incredibly direct and also centrally handled, so we can avoid the memory weak worry that was everpresent with MP clients in the past.
  • Oh, another thing to note: all of these are individual object data holders that can also contain code. None of them are singletons.
    • So for instance, if a faction defines a FactionDataA class that goes with itself, then if there are 3 instances of that faction TYPE, there will also be three instances of this faction data. This is very useful, because you can therefore store faction-specific working data in the deep processing in particular, or use static variables if you want to share across factions. This is very similar to the ethos behind ISpecialFactionImplementation, which will be leaving us before too long.

• Experimented with a new method of serializing data for objects. The data is really really sparse, though, because of "all the things that can be on a ship," for example, usually only one or two actually are. It's much the same for faction and even world.
  • I spent a goodly while coming up with wrapper classes to manage arrays based on the length of how many external linkages there are of any given type (which is based on how many DLC and/or mods you have installed), and was going to have some sort of very high limit (so that if you run with 100 mods on, someday when there actually are 100 mods), you don't have a problem.
    • However, that has many many drawbacks, because we wind up iterating a ton of arrays that are almost universally empty, or have a single item or two in there. And there is a finite limit on the number of mods you could have, even if it is very high.
    • The other big drawback is just how long it will take to iterate all of that, and for squads we have issues of how timely we can get those data updates from the host to a client after there's some change. Also, we have some order of operations questions with things like fast blast data potentially arriving before the external data, or vice-versa. Either way, it's not a good model.
  • The solution will to take this "vertical" approach and make it horizontal, and instead serialize on-the-fly in a more centralized fashion. This keeps our amount of iteration very very low. The downside is that I then have to make it very fast to link back and forth between the data and the thing it is attached to, but that's not the worst problem in the world. I was already potentially going to have to find a row index and then index into an array and then cast to the proper type. Instead we're likely looking at a concurrent dictionary lookup, which is at least a single operation rather than multiple (admittedly cheaper) ones.
  • Well. At any rate, my code for the old vertical method is not complete and does not compile (I was in the middle of it when I had this realization). I'm checking it in in case it is ever needed in the future, though. Next step will be to convert it to the horizontal format.

• Some rather mind-bending code for horizontally serializing and deserializing all of the external data for any world data, faction data, and squad data, all at the central faction level, has now been put in place.
  • This differs from a lot of our serialization code in that it's very... reactive... to what it finds. Essentially what happens is it iterates over every possible piece of data that can be sent (hey, macrophage ships can sometimes have data attached -- are there any in this game? If so, send those that are there). It's clearing out stale data as it goes, and as-before it gives the details of the serialization itself to the actual External code classes.
  • One of the chief differences this time around is that we don't have to serialize any table or row names -- we are using central indices that are synced from the client to the host -- and we're also using smaller ranges of integers for the PKIDs for factions and squads based on the differential between the largest and smallest ID in the particular list at any given time.
  • This will require testing once we can run the game again, to make sure that I didn't make any typos, but it uses a lot of generics and type inheritance in order to minimize code, so whatever problems are there should be of the "fix once, don't think about it again" variety.
  • At the moment, what is in the codebase is kind of impressive, but also kind of baffling. Here we have six identical tables and classes, all serialized using some very complicated code that is very centralized... but there's nothing different about any of these classes or tables. They're all super identical.
  • Where the magic really is going to come from is the fact that we can have all manner of different methods on each of those distinct six classes, and so we can set, retrieve, and manipulate data without having to do any type casting at all.
  • This a much more elegant solution than the old solution we had, but it strays slightly close to "clever code" (the sort of thing that is showing off for no reason, and thus is super hard to maintain and nobody else can read it).
    • That said, the reasons for this solutions design are pretty clear: keeping the guard rails on, not mis-identifying data (often happened before during load issues), minimal steps to get from Core to External, the ability to have lots and lots of very direct methods that we can can directly call, and pooling that is NOT in the hands of the External programmers, but instead is controlled by Core. There's also a goal for having substantially less code, and substantially clearer code, on the External side. That will start becoming evident tomorrow as I start wiring this up to them. On the Core side it takes some extra complexity in order to create more simplicity on the External side.
  • That's the idea, anyhow. I have long considered myself a code architect in terms of programming sub-specialties, and I've done a few dozen refactors of major programs, ranging between languages and business software and games. This one has so far been, to my surprise, the most complicated one I've had to do. There's a lot of pleasure in that, but at the same time: wow complex.
  • The main good news from today is that there is really a small amount of code (all things considered) where I could have made mistakes. So once I start this up, after connecting to the External code where I CAN start this up, any errors should make themselves apparent immediately and then be gone forever.
    • The old way of doing things was something I was never able to wrap my brain completely around because it had three layers of indirection that also required bidirectional communication and reading of the code. In other words, to understand that code, you had to read the entire path of it from Universal, through Core, all the way to External, and then follow a further chain back down the other direction. That was... code that worked, but was very very hard to be sure was not secretly broken. It had some arrays of objects in the middle, hitting the GC in a way I didn't like, and it was particularly hard to tell what it was doing o =n MP clients.
    • But the new code? It flows one direction, you can follow it clearly, there's no GC hits in the middle, and despite the code being a little bit odd syntactically (generics really can look odd), it's fairly straightforward to follow.
  • Tomorrow will be the real test, where I start wiring up the various pieces of External code to these new pieces, get them compiling again, and then eventually transform it all into the simplest possible version of itself that will be as error-free as possible in MP and SP as we want. It won't all happen in one go, likely, and I expect several more major refactoring passes next week to clear up various perennial bugbears like linked factions, the strange combined fleets/cities of the fallen spire, the odd (and not architecturally wise) way we have "faction common data" and similar structural oddities that make some of the externaldata have a larger learning curve than we'd prefer. Hopefully we have a working SP game this weekend, and hopefully we have a refactor-complete game by next weekend. We'll see how quickly things move tomorrow.

Restructuring The Upwards Link From Core To External, Part 2

• Okay, all that "clever" code yesterday? I said in my release notes that it was pushing the edge of being too clever for its own good (when programmers of a certain experience start talking about how "clever" code is, it's not a compliment -- if a professor tells you your code is clever, that's a compliment, but if a colleague does, they are worried about ever having to maintain it. Smart and cool and elegant are always compliments.).
  • The code was indeed the simplest possible implementation with the least amount of code for doing the sort of many-to-many relationships that I was assuming some of our data might have. But thankfully, no actual data is structured that way.
  • The old code that we used to have was allowing for many-to-many relationships between many kinds of objects and data, but thankfully everyone was sensible in both internal and mod code, and instead what we wound up with was a lot of one-to-one code. THIS ship has THAT data and no other, etc.
  • So! After talking to the folks who have been working on the bigger mods and the external factions and verifying that my sudden suspicion was correct, that means it was time to strip out pretty much most of the work I did yesterday (sigh, but it was a good intellectual exercise I suppose) and implement something new.
  • This new approach treats all of the World data as being singletons (aka only one copy of that external object, ever, period), and it even queries them "hey, you want in on this?" to which they can contextually say yes or no.
    • This is super duper useful, because you could actually have some sort of helper code turn itself on for a while to manage some process in the game, and then relieve itself of duty and stop using any bandwidth or CPU cycles. Or it could start out as off, but turn itself on once some situation happens, etc. Handy!
    • In general, one of the larger problems with external data assignments like this is "when do they get attached, and how do we know to do it?" In the past, a lot of that was "when ships are created in externaldata, we'll assign it contextually then, and hope we miss no cases. If found we did, then backfill." We may keep to that, but will also try to formalize it a bit. But for the world, simply asking the objects "you want in, or you good sitting out for the moment?" is feasible as well as most performant in this instance.
  • Factions and squads are still pooled, and so a lot of what was done yesterday (the easy parts) still apply for these.

• Map generation code is now flagged as deep processing, so it will be disallowed from being in any dll with the base data and sim code.

• ISpecialFactionImplementation, which has for so many years been the absolute centerpiece of faction and AI code, has been removed from the game.
  • It's functions are now being split between base-data-and-sim faction code, and deep-processing faction code objects, which has been one of the chief aims from the start of this rework.
  • There is no longer an "Implementation" object on the faction at all. Instead, there is now a "BaseInfo" and a "DeepInfo."
    • These objects serialize data to disk (like externaldata used to), as well as contain whatever logic we want them to. That's not quite an MVC philosophy, but it's following a similar and useful OOP design.
      • It's worth noting that the deep info ONLY ever serializes to disk, and never across the network to MP clients. It's also worth noting that the UI should not be trying to get any data out of the deep data unless it's purely for literal debugging purposes.

• For the first time, squads themselves now also have a BaseInfo and a DeepInfo, which mirrors what you're used to with the factions.
  • This doesn't replace anything on the "big logic processor side," but will probably fold in things like "description appenders" later. However, this replaces the notion of ExternalData that used to exist.
  • It's worth noting that this external data is no longer a collection of things, unlike before. There's only a single item attached to each squad, or zero items attached to it (if it's not some special faction other than humans or the AI, it's probably zero items).
  • Before we had a many-to-many relationship, but that was hyper inefficient and something you are better off handling with data at the faction or even world. There are no known cases of any internal or mod code actually needing that, though.

• Added ParallelDoForAllActiveEntries() onto TimeBasedPool, which gives us a comparably-quick way of doing some action VERY completely against all live objects in that pool.
  • Quick is a relative term in this one, but it is multithreaded, at least. This shouldn't be used in most cases for during-game things, but rather is for times like "when we reload all the xml" or things like that.
  • We also added a ParallelDoForAllEntriesEverEvenInactive, since that actually seemed kind of warranted for the xml case.

• GameProgramIsAboutToReloadExternalXmlAndSimilar() is now a thing that exists on all the external objects, rather than being limited to the faction data.
  • This is mainly as a courtesy to mods that may have different structures from our code. We don't actually use it, but it's useful as a thing that can be called for those that do.

• ExternalDataExports is now functional again, and lets us dump out all of the new-style external data.

• Everything that was called "ExternalXDataAndSimCode" is now called "ExternalXBaseInfo"
  • This is a lot easier to type, and is way easier to read at a glance.

• Everything that was called "ExternalXDeepProccessingCode" is now called "ExternalXDeepInfo"
  • Again easier to type and read.

• There was also kind of an awkward convention between the tables and rows describing the external data, and the external data itself.
  • That has now been clarified into the tables and rows being called "InfoSources", while the actual things themselves are "Info." Logical and straightforward at a glance.

• Faction xml (aka from the SpecialFaction folder) no longer wants or accepts a dll_name and type_name.
  • Instead, it now requires a base_info_source (referring to an entry from ExternalFactionBaseInfoSourceTable).
  • And then also a deep_info_source (that refers to ExternalFactionDeepInfoSourceTable).

• Squad xml now has the OPTION of specifying base_info_source or deep_info_source as a way of automatically attaching these types of external data to all entities of the type in question.
  • This is actually a really clean way to handle things, and then the data can be populated later by the external code as desired. Nobody HAS to use this, but it can be a nice thing.
  • Do note that this IS inherited via copy_from, so if you are copying a ship that happens to have base_info_source set, then on any that are being copied for use without that, you need to set base_info_source="None" on any direct children to blank that back out.

• Actually, it looks like the DescriptionAppenders and the AlternativeMoveOrderHandlers, and the EntitySpecialPerSecondLogic all can just stay as they are.
  • They are being flagged as IExternalBaseInfo so that they have to go into that dll rather than the deep dll (otherwise these would all break on MP clients), but that's about it.
  • These are a great example of something that can just stay flexible (no need for a rigid structure here), but now have a proper place to live that also guarantees what data access they have access to. Previously, these things would maybe work or maybe not work on MP clients, depending on the status of things. But generally trending towards not working. Now they can be verified to function just by testing them in single player, since they're in the dll where clients also get access to all the info.

• At this point, the Core dll has had all of the changes made to it that are required in order to handle the upwards-facing communications that the Core dll has with External code.
  • This means that the work of actually translating over the External dlls can now begin, and some things can be redone in passes, but not all things have to be.

Restructuring External Base Info, Part 1

• Started the process of transitioning some data from the Deep dll into the Base dll, with the idea being to separate out the running of things from the actual viewing of data related to those things.
  • Also, of course, wanting to isolate the sort of background and working data off in Deep away from the UI and any temptation to try to show that data, which would be explosive on the MP client and just a bad idea on single player or the host.
  • Moving over HunterFleetBudgetType and its related stuff is straightforward enough, and matches with the sort of thing I have in mind. That's just a copy-paste job, and hey I'll make new files for these for easier location of them in the future.
  • Then I turned my attention to HunterFleetType, and immediately hit a hard stop: it has something on it called IHunterFleetTypeImplementation Implementation, which defines both data (Base type) and logic (Deep type).
    • This is a bit of a conundrum, but not a crisis. Ultimately this mirrors the same sort of setup that I have going on in Core, and I'll need to split these things out in some fashion.
    • Right now my brain is feeling quite tired, so maybe I'll wait a bit on that problem, but essentially this is where we get to the "dream within a dream" level of inception, and this is actually more important than it might look on the surface.
      • First of all, modders and general faction designers need to be able to create new data tables in a subordinate fashion like this, and there needs to be a really central way of handling it. I can't idiot-proof it like I can with things coming out of Core, specifically because as the "dreamer" of the first dream, the modder/faction-coder has full control over how they structure their inner dream within a dream. That's both neccessary, and something I don't want to take away.
      • This actually does get at the generalized need to formalize that as much as I can, however. I can't make it idiot-proof, but I can give the proper tools to standardize dreams within dreams (going with that metaphor), AND also to make sure that they serialize in a way that does not cause a memory leak on MP clients. It's this last part that is the most challenging, and if I could make that one part idiot-proof, that would make me very happy. That's the one bit of power (accidental memory leaks) I'd like to take away from the dreamers in the top level dream.
      • Further compounding this is that I might actually need to support more depth, aka dreams within dreams within dreams -- three levels, or even four, or some arbitrary recursive number. I sure hope no-one has structured their data like that, but I guess we'll be finding out. First, though, I'm going to take a break and focus on some other compiler issues that need to get sorted out anyway.
  • Overall when it comes to looking at this, I have factions and squads which presently have a very similar -- but slightly different way of serializing. The faction MUST always have both Base and Deep info, and it serializes with that in mind. The squad may or may not have Base or Deep in any combination or none at all, and it serializes with THAT in mind.
    • The world, meanwhile, can have a great many base and deep infos attached to itself, but they are all singletons and are also all loosely, voluntarily attached.
    • The question will mainly be if I can formalize these relationships in a generic and encapsulated format that can be attached to arbitrary objects. The huntertype seems to be a singleton processor-type class, and the other classes that I've run into are similar.

• Added a new ArcenLongTermIntermittentPlanningContextBase in the BaseInfo dll.
  • This is going to be used in some of our "dream within a dream" data management cases.

• Fireteams have been partially moved into the BaseInfo dll, but parts of their code is clearly DeepInfo code.
  • That has been commented-out for now, and will be grafted back on in the DeepInfo dll once we can compile BaseInfo again.
  • Additionally, FireteamUtility has been split into one part that is FireteamBaseUtility that can run anywhere, and then the other part that stays behind in DeepInfo.

• In Universal, split up the interfaces that contribute to the external BaseInfo and DeepInfo so that we can have ones that are singletons and ones that are multiples properly denoted.

• All thirty-seven "Decription appenders" have been moved from their files in the DeepInfo dll into their own individual files inside the BaseInfo dll.
  • There were two old description appenders (hacker and science) that were not used and thus have been removed.

• All of the IAlternativeMoveOrderHandlers have been moved from their files in the DeepInfo dll into their own individual files inside the BaseInfo dll. There was only one.
  • In recent versions of multiplayer, clients were not going to be able to order spire relics around. Nice that we're doing this overhaul, as that was not on my radar and would have been a huge pain...

• FrenzyFleets from the nanocaust have been removed, as fireteams have far far replaced those.

• FactionUtilityMethods has been partially moved into the BaseInfo dll, and it is now a singleton rather than a static class.
  • Thanks to it being a singleton, this means that we can also add extension methods in the DeepInfo dll and not have to worry about having two classes with similar names that we wonder which item is on which.

• The "SpecialForces" faction has been fully renamed to Warden in the code. It was old legacy naming from 2018 or prior, and getting rid of that makes things a lot more readable.

• EntitySimLogic has been split into EntitySimLogicBaseInfo and EntitySimLogicDeepInfo.

• Moved over the bulk of the data that should be in BaseInfo rather than in DeepInfo.
  • There's still more that needs to come over, but it's part of larger classes that will need to be split.
  • At the moment, I've still got a lot of things commented out or using fake connections to mimic the old style. It's not functional, but it's hitting the point where I can compile again and thus move towards getting it working.

• A variety of new methods have been added to FactionUtilityMethods to make checking for faction types easier and more reliable.

• Current status: lots of things are moved around to be in a better spot, but a lot more work needs to be done.
  • There's still a lot of temporary code linking to the old style of doing things so that we can compile the main ExternalData dll (the base info one). That actually does compile now! But it's not linked up yet, and won't function yet.
  • The new deep info dll doesn't compile yet, and will require a lot more splitting of data into multiple classes before it does.
  • There are still some things that need to move one direction or the other, and then other things that need to be moved around or restructured so that data is grouped properly and can be accessed properly.
  • This is... quite a bit more tangled-up than I had expected, to be honest. But this is a really super good thing to be doing, and if this past week is any guide, it should be less than a week of work left to get this functional again.

Reconnecting External BaseInfo to Core, Part 1

• There was a "does nothing, but helps things compile" set of code that we had in the ExternalData (BaseInfo) dll since making the intial rip and tear changes to ExternalData as a whole.
  • This code has now been removed, which creates hundreds of errors but allows us to start replacing those hooks with new ones. Doing a massive code shift like this is an iterative process, and sometimes you just need "somewhere to plug things in for a bit so you can work on another part," if you think of it like working on a physical piece of machinery.

• The base and deep infos on factions are automatically created just like those ISpecialFactionImplemenations used to be, so that's not something that you ever have to worry about creating as an end programmer/modder in this game.
  • Similarly, the ones for the world are created as singletons and are never really attached to the world. The world just asks your code "hey, you can see what's up, do you want to run?" And if you say yes, then the world will run your code and serialize it, etc.
  • On the other hand, squads are a bit different. Historically we have added things to them as we went, as we wanted to, but that doesn't mean we can't centralize some of it. There are two modes of operation for the squad, therefore:
    • If you specify the xml for it, then it will automatically add them when the squad is created. This is not a bad idea! But this does assume that every unit of that type EVER will have this external data, and there are potentially cases where that is not true.
    • For those sorts of conditional cases, you can instead manually call CreateExternalBaseInfo() or CreateExternalDeepInfo() on a squad, and pass in the type of data you want it to initialize for you.
      • If this is already initialized to something else, then it will throw an error, since that is in fact an error and you need to take a look at what's going on.
      • If, however, this is already initialized to the same type you just asked for, then it just returns the existing item it has. Basically if there are cases where we might accidentally call the same method here more than once with the same code on it, then this doesn't hurt anything with the extra calls.

• Okay, here's a description of the pattern for changing things over from the old format to the new. This is mainly for modders to know, but also anyone working on the codebase it's useful to see.
  • We previously stored Wormhole Invasion Data (which goes on a squad) like this:
    • WormholeInvasionDataExternal was the outer wrapper that inherited from ArcenExternalDataPatternImplementationBase_Squad.
      • Structurally, this was clunky code, and very rarely had actual data of its own in there (in terms of general ExternalData for whatever function). There's no real analogue for this anymore in the new system.
      • IF you were doing the extremely old-school thing (for our codebase) of storing multiple sub-objects in here, you will have to choose one to become your new primary BaseInfo, and all the others will be sub-objects of that. Easy-peasy.
    • ExtensionMethodsFor_WormholeInvasionDataExternal was an optional (but always-used) convention that made it easier to get the wormhole invasion data sub-ojects from externaldata.
      • Structurally this was also a bit of a pain, even though it wasn't much code. A small typo, and dealing with the annoying patternindex (which no longer exists) could really foul things up. We can safely remove this, also.
      • If you really want some extension methods for THAT easy of access to BaseInfo or DeepInfo in the new system... then honestly instead maybe think about just how frequently you're calling such methods, and instead call it fewer times and pass the result around. Player CPUs will thank you.
    • Finally we come to the actual meat, which in this case was WormholeInvasionData, which inherited from ArcenExternalSubManagedData (which no longer exists). This is what will become BaseInfo.
      • The reason that this is BaseInfo instead of DeepInfo is that this is information that both the client and the host might like to see, in terms of it affecting the UI for both of them. In the event that there are some parts that are truly host-only, and never get shown on the ui except for maybe debugging purposes, then consider moving it to DeepInfo in a separate class IF it's substantial enough data to actually care about (why tell the client about things it never cares about?).
      • This class is now going to inherit from ExternalSquadBaseInfo, since this is what it is (base info, squad-attached). Let's talk about what THAT means separately, in a moment.
  • In terms of translating things over, for this type of data this is all you need to know from a bird's view.

• So now let's talk about details of what it means to handle this ExternalSquadBaseInfo, starting with WormholeInvasionData -- which let's rename to WormholeInvasionBaseInfo.
  • Trust me, it's worth it to rename this just to save yourself confusion later. Remember to do a global search and replace from old name to new name in both the BaseInfo and DeepInfo dlls (the DeepInfo almost always acts on BaseInfo).
  • Now that we have this class inheriting from ExternalSquadBaseInfo instead of ArcenExternalSubManagedData, there are some immediate things to change. (This will be identical for DeepInfo, and for faction and world data.)
    • DeserializeIntoSelf and SerializeTo both no longer are passed a SerializationCommandType SerializationType, so you can just remove those and those methods will then be happy. You can probably just run a global replace in your BaseInfo dll for ", SerializationCommandType SerializationType" and replace it with nothing and it will save you time (we had 226 instances that were replaced in our own innternal BaseInfo code).
    • We need to implement the base method Cleanup, which gets called whenever an object is being pulled back out of the pool.
      • This may or may not be important to actually fill, depending on your class. If you were previously setting some values in the constructor, and your code always expects that to be the starting state, then you should probably move that initialization code to Cleanup and have your constructor also call Cleanup(). You'll note that this is what we've done with WormholeInvasionBaseInfo.
  • At this point we have translated things over such that they CAN connect up, but there isn't an actual linkage being created yet. It's time to address that.

• Since we did a global search and replace in both projects, we can search for WormholeInvasionBaseInfo in both the BaseInfo and DeepInfo dlls.
  • In our case, there are no other references to this in BaseInfo -- that's just where the data is stored. (This is temporary and wrong, actually! We're going to move the notifiers over to BaseInfo to save processing and data, so later that will actually be there).
  • At any rate, there are two ways we might want to use this data now. We might want to get a reference to it, to change it or just write about it.
    • Previously that would have been done with the GetWormholeInvasionBaseInfo() extension method, or something similar.
    • Now this is done by calling GetExternalBaseInfoAs<WormholeInvasionBaseInfo>() on the squad, which will give you back a WormholeInvasionBaseInfo that already exists or give you an error. For DeepInfo, you can call GetExternalDeepInfoAs<YourTypeHere>(), and it works the same.
    • It is worth noting that if the base or deep info here is null, then these will just return null, so your end code should be prepared to handle that if you think that might be a possibility. In the past, most of the time our code just returned a new instance of the data type, which... well, that could have hid a LOT of code errors. We have no way of knowing. If someone calls these things on the wrong kinds of unit/squad/structure, we actually do want this to error, because that is almost certainly a logic error that then would lead to strange behavior. It's possible we will find some "new" bugs that are actually existing ones that had been hiding, in the coming weeks. This in turn would likely fix a lot of other rare bugs that were downstream from this problem, if that's actually something that happens.
  • In the old code, after changing some data, you'd MAYBE have to call SetWormholeInvasionBaseInfo(). There is no longer any such analogue.
    • Technically in the old code, this was not required for this particular class. But if you were using certain kinds of data, in certain situations, it was VERY much required. So end-coders wound up just calling it uselessly after making any changes in general, and thankfully now that's not a thing anymore.
  • Lastly (though technically firstly), there's the creation of this data. This would, presumably, happen when the squad itself is actually created, and in practice that is what the code does.
    • This can be handled two ways.
      • Option 1: You can set it up so that all of the units with the "ExogalacticWormhole" tag have this data linked via the new xml tabs, since that's apparently the criteria we're using.
        • Pros: the xml will be nice and clear about what is linked, and there's no code path where you might accidentally forget to add it.
        • Cons: properly transferring the existing code to hit all the proper entities is slightly tedious, and also introduces the possiblity, of errors.
      • Option 2: You can take what was a call to GetWormholeInvasionBaseInfo( ExternalDataRetrieval.CreateIfNotFound ) and turn that into a call to CreateExternalBaseInfo<WormholeInvasionBaseInfo>( "WormholeInvasionBaseInfo" ).
        • Pros: this keeps the same structure we had before, so there's fewer errors of short term errors in conversion. Requires no xml work. Can be used to add BaseInfo data onto units in a conditional manner -- for instance, usually just when they belong to a specific faction.
        • Cons: in general, if you forget to do this on all possible code paths that are expected to lead to the result you want, then the game is going to have new errors popping up where it used to not have that. In the vast majority of internal cases, there's only one central method that creates these things in the first place, so that's usually less of a risk, but sometimes a consideration.
    • Before anyone asks: yes, you CAN handle it "both ways." It won't complain if it already has the type you're asking to create. It will just return the type that is already there.
    • Also worth a thought: there isn't any sort of initialization callback on these right now, so if there's linkages you need to make on setup, you're going to need to at the very least call the GetExternalBaseInfoAs<WormholeInvasionBaseInfo>() method and then assign some stuff. So this again creates as situation where you must be sure to always catch every possible code path that could generate this sort of entity, though we may work on solving that in a couple of ways in the very near future. Probably before this build is out.
  • Okay, now that the process is outlined, it's time for me to repeat this another few hundred times on the main game code.

• Previously, there was some information stored on every faction. To get at this data, you would have to call GetCommonExternal() on the faction object.
  • This data has moved onto the BaseInfo object for factions in general, in Core.
  • There is a new IPlanetPathfinder in Core, which lets us handle a lot of this centrally now.

• Made some updates to methods in FactionUtilityMethods to use a more efficient way of finding the highest AI difficulty in various situations.
  • Also got rid of some duplicative methods of this from MapGen_Base.

• SpecialFaction_Human had a lot of random stuff on it that is moving elsewhere:
  • The audio cue suppression stuff, which was complex but is only really used for the king unit being damaged, has been moved to nonsim in the base game.
    • This is local data for a client or a host, and we don't use this in such a way that it warrants nearly the intensity it is generating.
    • Similarly, the data from HostOnlyAudioTracking has moved onto the planet itself and is no longer host-only (that was an error on my part a week or two ago).
    • The function is identical to before, but it's now far less code.
  • GetMaxPossibleStrengthOfAllHumanFlagships() has moved to a new StrengthHelper class in Core.
  • GetPathingModeForLocalPlayer() has moved to a new PathingHelper in External (BaseInfo).
    • Similarly, a bunch of stuff like InnerFindPath_Raw(), InnerFindPath_RawSinglePathfinder(), FindPathFreshOrFromCache(), and FindPathFreshOrFromCache_ShortTermPlanningOnly() moved to that PathingHelper out of BaseFaction.

• Also previously, there used to be a ExternalData_MinorFactionCommon.Primitives, which you could get from most factions via calling Ex_MinorFactionCommon_GetPrimitives().
  • This is now gone as well, with SOME of its data wrapped into the BaseInfo root just like what we did with the CommonExternal stuff.
  • The rest of this data is currently sitting (as of this writing) in a Temp.Historical class, and will have to be properly distributed to various classes that actually need this information. Right now there's a lot of junk in there that is not used at all, and a bunch of stuff that is faction-specific in a way that doesn't need to be central (the HRF intensity is not needed on every faction in the game, etc).
  • One thing that is moving is the Intensity being directly on any faction. This should instead be per-faction where needed (AIs don't use this, they use AIDifficulty, same with Wardens and Hunter and PG, for instance, and many others use nothing at all).
    • Instead, when trying to quickly get the intensity for a faction in general, which before was Ex_MinorFactionCommon_GetPrimitives( ExternalDataRetrieval.CreateIfNotFound ).Intensity, the code that should be called instead is BaseInfo.GetDifficultyOrdinal_OrNegativeOneIfNotRelevant(). This was by far the most common usage of primitives.
  • Actually, for the moment nothing has moved data-wise from here! The Allegiance field would be the one to move, but for now I've implemented a string get-only property called Allegiance on BaseInfo, which is just a wrapper around calling directly into the config file on the attached faction. This is getting back to (from last week) only having one copy of certain data.
  • HasPlayerGainedIntel has been removed, as there is a HasBeenSeenByPlayer on the core faction object now. This was only used by the devourer golem for one notification, anyhow. That now uses DoOnFirstSightingOfFactionByPlayer(), which was added long after the devourer golem faction was coded.
  • TotalMetalMetabolized has also been moved to the roof faction object in Core.

• Got rid of an old GetFactionIntensity() method from Scenario_BaseScenario. The proper code is now BaseInfo.GetDifficultyOrdinal_OrNegativeOneIfNotRelevant().
  • There was a pretty large bug in there that returned -1 for the intensity of any faction that actually used the central intensity settings (aka not AI or its subfactions).
  • This was causing achievements to not fire for beating other factions in general, or at least it should have been doing that.

• The "SpawnAnimationData" and "DespawnAnimationData" data classes have been entirely removed.
  • This was vaguely like the despawn info, which the game DOES use, but it was not actually ever data that was used!
  • It's vaguely possible that a few parts of this were actually functional, but it's honestly very hard to tell. If that turns out to be the case, then we'll have to implement that as more centralized data. The way it was handled before was super confusing.

• The "InstigatorData" faction info is our first example of a faction that needs to be converted to have BaseInfo and DeepInfo.
  • The SpecialFaction_Instigators class would be the basis of the DeepInfo, while the InstigatorData that inherits from ArcenExternalSubManagedData is the basis of our BaseInfo, but really we're going to be pulling both code and data from both of these for our new setup.
  • First off, let's rename "InstigatorData" to instead be InstigatorFactionBaseInfo.
  • Next we need to implement the Cleanup() method in the exact same fashion that we had for the squad baseinfo.
  • Also, because there is some wrapper data also being saved, you must rename DeserializeIntoSelf to DeserializeFactionIntoSelf, and SerializeTo into SerializeFactionTo.
  • IF you have any usage of ModdableGameCommandExecution() on your old SpecialFaction class, then you should port that over.
  • Again, IF you have any usage of DoOnSpecialEvent_OnMainThread_ClientOrHost() on your old SpecialFaction class, then port that over here.
  • You must implement GetDifficultyOrdinal_OrNegativeOneIfNotRelevant() now, even if that's just to return a -1. Ideally you would return a number between 1 and 10, unless your faction has no concept of a difficulty level (or intensity) associated with itself.
    • This method no longer has a faction object passed in. Use the AttachedFaction property to know what faction you're talking about.
    • This method was widely incorrect in the prior version of the game, so this should fix bugs like the "OnLoad: AI faction: X had a difficulty of -1, which is not helpful for setting overlord difficulty level."
    • Instigators are a great example of one with nothing like that.
  • IF you need a nonstandard implementation of FindPathFreshOrFromCache(), then be sure to migrate that over from your old SpecialFaction class. We use this on the hunter, warden, and similar, but not much else.
  • IF you used DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost in your SpecialFaction class, then move that over. Otherwise ignore.
    • Same with DoOnPerSecondWhilePausedNonSimUpdates_OnMainThread if you use it.
    • Also DoOnLocalStartNonSimUpdates_OnMainThread if you use it.
    • And WriteTextToSecondLineOfLeftSidebarInLobby if you need something beyond the defaults.
    • And GetShouldAttackNormallyExcludedTarget() if you use it.
    • And finally, SetStartingFactionRelationships() if you deviate from the defaults, which is the most likely out of this list.
  • Whew, that's everything. A lot of stuff had to be adjusted in order to get this first one working, but now future ones are pretty straightforward.

• Removed GroupTargetSorting (it's some random ExternalData), since that is not in use. It was some very legacy vestigial code from pre-pivot.

• Status time: at this point "all" that's left for the BaseInfo dll is to translate the remaining faction and squad data over. If there's any world data we run into during that process, that will also need to be ported over.
  • After that, there's going to be a needed conversion for all of the SpecialFaction_Whatever data from DeepInfo, but that is going to be a way more direct item to tackle.
  • Also todo is to move notifications back into BaseInfo where the client can generate them all, and turn back on a lot of things that were disabled for being-able-to-compile purposes when the data was not all in the correct place.
    • Some of that will then also probably involve moving some data from DeepInfo classes to their BaseInfo counterparts.
  • This is probably another 2-3 days of work. After that I can run this for the first time and see how badly things are broken. Hopefully that is less than one day's further work of fixing, and then we're back to actually having beta releases. That would put us at Friday, if the general estimate here holds.
  • I did previously have a worry about data-within-data, but most of that is (knock on wood) not an issue. There are still some interfaces in a few spots where a Base class wants to call a Deep class, thanks to SurrogateTables (this is things like the Warden Difficulty, etc). I don't have a firm solution for that yet, but at worst I can use the surrogate table itself to manage that. It's also possible that this form of nesting might not be problematic in the first place.

Reconnecting External BaseInfo to Core, Part 2

• There are now GetExternalBaseInfoAs() and GetExternalDeepInfoAs methods on the faction, to mirror those that are on the squad.
  • This helps us have a more orderly error if something goes wrong, and it also allows for the code to look that much more consistent, too.
  • It's fine to talk about the BaseInfo or DeepInfo directly on a faction, and if you want to then cast that to a specific implementation type that can also be done. But if you need to move from the base implementation to the specific implementation, the most robust way that errors would be reported is by calling these methods.

• LazyLoadSquadWrapper and SquadWrapper have both been updated to hide more of their internals, and thus have fewer bits of confusion.
  • Because these are structs, there's definite risk of confusion on the part of some end programmers, but now there is not. There is also now a Clear() method on both.
  • Related, also removed the SquadPrimaryKeyID_Neg1ToPos serialization variants. People should instead use SquadPrimaryKeyID_PosDef0.
    • In all cases a squad ID of 0 means null, and anything less than 0 will be saved as 0.
  • There are REALLY some notable cases in the external data that need to be using these two structs, specifically because without that the usage of ship pooling might cause some very strange artifacts.

• Fixed up a whole lot of references in the UI to be properly matched to the new format.

• Faction BaseInfo now has a virtual AppendStateForDebugDisplay() method, which does a much more efficient version of what GetAstroTrainsStateForDisplay() and similar used to do.

• All of the external base and deep infos now have a AppendStateForInterfaceDisplay() virtual method on them, and they both seal the ToString() method, fill it with an error, and mark it as obsolete.
  • The idea is that we should never be calling ToString(), as that's confusing and wasteful on the GC. Instead, we pass in a buffer in an ordered fashion.
  • The fact that it is now sealed and obsolete will in both cases help us automatically convert older code to the newer format, since rather than it being invisible (that was one problem before) it is now a compiler error.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • AIReservesPerUnitData is now AIReservesPerUnitBaseInfo
  • NebulaGuardiansPerUnitData is now NebulaGuardiansPerUnitBaseInfo
  • AstroTrainsPerDepotData is now AstroTrainsPerDepotBaseInfo
    • AstroTrainData has been renamed to AstroTrainBehaviorType. If you didn't already know what this was (I did not), it was incomprehensible in the code until you looked in the xml.
    • Fixed a minor issue with TrainsSpawned being serialized twice.
  • AstroTrainsGuardData is now AstroTrainsPerTrainGuardUnitBaseInfo
    • This now uses a LazyLoadSquadWrapper and also uses better serialization. This should solve several rare issues, and is also slightly more efficient during gameplay.

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • AIReservesData is now AIReservesFactionBaseInfo
  • NebulaGuardiansGlobalData is now NebulaGuardiansFactionBaseInfo
    • Fixed some pooling issues.
  • AstroTrainsGlobalData is now AstroTrainsFactionBaseInfo
    • A variety of data has been ported over from the SpecialFaction, and part of that also involved decoupling some of the settings data from the loading of xml. It was messy and prone to potential errors the other way.

• ExoData is... an interesting challenge. This is technically first-level faction subdata, but just something that can be run from a variety of different facions.
  • I don't think that pooling these would give us any notable benefit, so instead these should really just become... a secondary data structure that the game doesn't manage at all, and that is a readonly variable on the faction classes in question. That keeps them implicitly pooled with those factions (there are X ExoData classes if there are X faction classes in pools that have ExoData on them as a readonly property (readonly means the root object can't be reassigned; the data inside it can be changed all day long).
  • In other words, the best way to think about this as if it were just directly data on the faction objects. But of course, we don't want to LITERALLY do that, because then we're repeating code, so instead we have a readonly object that has the fields insde it. Handy!
  • The actual TLDR of this is that it's a challenging concept for a moment, but then has a deceptively simple solution. Hooray.

• Added a new DoGeneralAggregationsPausedOrUnpaused() virtual method on faction BaseInfo.
  • The idea is that there's various lookup information we might need to use from the front-end and the DeepInfo, and this is data that we can simply aggregate from other existing data.
  • This should be done asap on load to make sure the UI is correct from the start. Therefore, we call it right after we call DoOnLocalStartNonSimUpdates_OnMainThread, which is another vritual method you may or may not have a use for.
  • This also needs to stay up to the second accurate even when the game is paued, so it gets called right BEFORE we call DoOnPerSecondWhilePausedNonSimUpdates_OnMainThread() (so that method can use its results if need be).
  • Finally, this also needs to be up-to-the-sim-step accurate when the game is running, so it gets called right BEFORE we call DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost() (so that method can use its results if need be).
  • Reworked AstroTrainsFactionBaseInfo to use this, and it should be a lot more efficient as well as providing a better pattern for other factions in the future. The code also is more brief.

• Added a new DoubleBufferedList<T> to ArcenUniversal, which is a lot like a SwizzleList<T>, except easier to use and based around List<> rather than an array of T[].

• Added GetFactionBaseInfoOrNullAs_Safe<T> and GetFactionDeepInfoOrNullAs_Safe<T> onto squads, for quick acquisition of the faction external info as its native type.
  • Fixed an issue with Dark Spire Vengeance Generator description appenders, where in multi-DS games they would likely all show the information for the first DS, or something else equally wrong. Now they properly show the info for the DS faction of the actual VG you hovered over.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • FallenSpirePerUnitData is now FallenSpirePerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • FallenSpireGlobalData is now FallenSpireFactionBaseInfo
    • Updated the way that FallenSpireDifficultyTable stores its rows by intensity so that we can look them up more efficiently.
    • All of the various lists of cities, debris, and so forth have been updated to use the new DoubleBufferedList<T>.
    • Additionally, rather than having a simple DarkSpireEnabled boolean, this faction now keeps a double-buffered list of all the dark spire factions.
    • And finally for now, SpireDebris_ForUI and SpireDebris_ForUI have both been removed, as the new double-buffered lists that controls the central SpireDebris and SpireCities is now sufficient for our purposes here.
  • DarkSpireData is now DarkSpireFactionBaseInfo
    • DarkSpirePerPlanet is the first example of external data within external data. This really should be pooled, but we can't do that centrally.
      • The methodology for this is going to be up to individual modders and faction designers. I don't have a central solution for this, but I can provide code to replicate. On the worst case, if you do nothing but Clear() the list that is incoming, you could just throw the data away and it might be a memory leak, or the GC might take care of it. Deviating from the approach outline below by too much can introduce an MP-client-only leak, so please just follow the approach below. ;)
      • The better case is to add a new pool that is just for your object type, and then use a ProtectedList or ProtectedArcenSparseLookup where you need to centrally store these. That way when you clear, it goes right back in the pool instantly.
        • To copy this code, simply copy the Pooling section from DarkSpirePerPlanet and, replace "DarkSpirePerPlanet" with the name of your class. Additionally, make sure it inherits from the same interfaces as DarkSpireData does.
        • Next, PerPlanet on DarkSpireFactionBaseInfo was previously a ArcenSparseLookup<Int16, DarkSpirePerPlanet>. It is now a ProtectedArcenSparseLookup<Int16, DarkSpirePerPlanet>. That means that whenever it has entries removed or cleared, it will pop them back in the pool.
        • Finally, Clear() calls need to be Clear( true ) for that list, and anything that would call new DarkSpirePerPlanet() should instead call DarkSpirePerPlanet.GetFromPoolOrCreate(). Congratulations, you now have a leak-proof, MP-friendly, pooled, high-performance piece of sub-data.
    • By the by -- there used to be a method called World.Instance.GetDarkSpireFactionBaseInfoExt_AndCacheAfter(). There's no such thing anymore. That was designed as if there could only ever be one dark spire faction, and would cause problems if there were more.
    • Similarly, some faction helper methods like GetDarkSpireFaction() are now renamed to GetFirstDarkSpireFaction(), and honestly potentially they should not be used at all.

• Vast amounts of the fireteam data has been brought back to life and converted over.
  • It has also been split into three files, for now. One which has text export bits, one which has DeepInfo bits that currently live in BaseInfo (but might move), and then the core stuff.
    • Actually, for the parts we could go ahead and move into Deep, those are now in a FireteamExtensionMethods class in the Deep dll. All but about 80 lines of the deep-type code were able to be moved there. I'll circle back around later. Please note that from the perspective of calling code in the deep dll, there aren't any real differences here (other than the things to do with strings and the one method split below).
  • Additionally, the UpdateNonSerializedFields() method has been split into two parts: one that is run on clients, and one that is not.
  • A bajillion string concatenations have been turned into double character buffers for the sake of efficiency. So has the ShipsByPlanet_ForUI field.

• Added TryGetExternalBaseInfoAs<T> and TryGetExternalDeepInfoAs<T> on squads, along with TryGetFactionBaseInfoOrNullAs_Safe<T> and TryGetFactionDeepInfoOrNullAs_Safe<T>.
  • The idea here is that in a few pieces of code, mostly some of the central Fireteam handling code, it says something like "hey if you've got some scourge data on you, then also do this." But in the event there is no scourge data on the target, or the target's data is some other type, we don't want to complain or have errors!

• On the World BaseInfo, added in DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost as an abstract method, and Safe_DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost as a wrapper method to call it.
  • The various things alongside this which let us see any error data, and clear any error data.
  • This will allow us to handle things like notifications to being a client-and-host activity.
  • While I was added it, went ahead and added the host-only equivalent per-sim-step items on the world DeepInfo. Namely, DoPerSimStepLogic_OnMainThread_NonSim__HostOnly and Safe_DoPerSimStepLogic_OnMainThread_NonSim__HostOnly.
    • This will allow for things like autosave handling, which is host-only.

• Added a new LocalPlayerWorldBaseInfo, which is our very first ExternalWorldBaseInfo.
  • The logic from SpecialFaction_Human that has to do with hovering over planets and finding paths between them has been moved into this.

• Split things up so that we can once again call the ArcenLongTermContinuousPlanningContext.RunAllContexts_ForUnpausedOnly() from the simulation, even though that's a host-only thing and thus moved.
  • Further added a new ArcenLongTermContinuousPlanningClientOrHostContext, and moved PerSecondNonSimPlanning back into BaseInfo and have it now descending from that.

• DoPerSecondNonSimUpdate_OnBackgroundNonSimThread_NonBlocking_HostOnly has moved from faction DeepInfo and is now DoPerSecondNonSimUpdate_OnBackgroundNonSimThread_NonBlocking_ClientOrHost on BaseInfo for factions.
  • Same thing with DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_HostOnly, and it becoming DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_ClientOrHost.

• The seven notifiers that were still left in the old SpecialFaction_Human class have now been moved into their own files in BaseInfo, and are filled from the LocalPlayerWorldBaseInfo world BaseInfo class.

• All of the code for generating intel tab entries (internally called objectives) has been moved back to being run on the client and the host.
  • Sigh. This was a lot of work to make host-only, and the host was going to have to periodically share that data with clients, but at least that no longer has to happen.
  • This is now non-serialized data, as it always had been in the past, and doesn't go in savegames or across the wire to clients.

• Added a new DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_ClientOrHost onto ExternalWorldBaseInfo, which we can now use to generate notifications that are not faction-specific.
  • This is really important for purposes of things like players who are not human empires (solo ark, necromancer, etc). Some notifications would be shared across human player types, others would not, and now we have the choice to actually set that up.
  • Also? If you're a modder and you want to add a notification, and it's not connected to a faction of your own, then this now gives you a way to do it. Previously there was not a way other than adding a fake invisible faction or something else wasteful (but required) like that.

• Added DoPerSecondLogic_Stage1Clearing_OnMainThreadAndPartOfSim_ClientAndHost and DoPerSecondLogic_Stage2Aggregating_OnMainThreadAndPartOfSim_ClientAndHost on the BaseInfo for factions.
  • We still have the host-only versions on the DeepInfo, and that won't change. The host-only version also has Stage0, Stage3, and Stage4. The client section only ever gets its version of Stage1 and Stage2.
  • A lot of factions do aggregations that need to happen in Stage2 on the client and the host, so simply moving those over to BaseInfo while still having the option of having things split is a good idea.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • ScourgePerUnitData is now ScourgePerUnitBaseInfo
  • AstroTrainsPerTrainData is now AstroTrainsPerTrainBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • ScourgeGlobalData is now ScourgeFactionBaseInfo

• A bunch more of the Astro Train faction data has been split over to BaseInfo, partially to go ahead and get some of the notifications relating to them able to function. This really makes things more legible and organized, I have to say. It's still a mess because it's not done, but there's no more wondering which variables are available when, which is really nice.

• Status update! Another day done, and... well, a lot of framework and structure has been implemented today, and SOME data has been transitioned over.
  • In a lot of respects, I was underestimating just how much additional framework work (additions to BaseInfo to make things convenient and quick, and make the cross-talk work) there was when I gave my three-day estimate yesterday.
  • At this point I don't really have a good sense as to how much more framework refactoring there is going to be, but I have a couple of the simpler factions to the point where their BaseInfo is done, and the squad-level data is easy. And the world-level data is also off to the races now, so that's also good.
  • The main problem is that I still have at least a full day of more work on BaseInfo before I can fully start on DeepInfo, although I am making many DeepInfo changes as I go to lessen the impact of that later. I can't imagine that DeepInfo will be less than a day, but it could be two. BaseInfo could be 1-3 more days on its own, which really really stinks.
  • On the other hand, the code is looking GOOD. I can actually understand what's going on, and aside from the truly most complicated factions, I have all of the general relationships figured out. The AI itself is one of the nastiest things remaining, but I plan to try and be as literal of a straight-port as I can, get that working, and then later potentially improve its structure further.
  • Even so, I haven't been able to run the game for something like two weeks now, so we'll see how many runtime errors have piled up once I even get everything to compile, which is somewhere between 2-5 days of work left before I hit THAT point. I had been allocating a day-ish to get it functional enough to share with players again, but I wonder if that's optimistic thinking. We shall see.
  • Another bit of good news is that many MP issues that I was going to have to tackle, some of whcih were multi-day problems that were going to be super error-prone, are just... no longer a worry. And as I've been doing this, I've also been doing a systematic code review and have fixed a number of bugs that were affecting SP at least some of the time. So this is paying dividends all over the place, it's just... really long.

Reconnecting External BaseInfo to Core, Part 3

• Added a new GetDoesThisImplementInterface() on ExternalData_AbsoluteBase, which means that all of the new BaseInfo and DeepInfo classes are able to use this.
  • This is something that allows for a new form of generalized expansion of code capabilities from just the external dlls, without having to update Core.
  • This is useful for purposes of mods in particular, letting them add discoverable methods without having to have access to Core (or without having to alter it if the modder does have access to it).
  • Internally, this method checks to see if the end type that it is (for example, some Faction's BaseInfo type) ALSO happens to inherit from some interface of your choosing. It caches the results, so it's very efficient.
    • Please note, simply calling the normal "BaseInfoClass is YourInterface" to see if that is true will not always return true for interfaces, particularly if they are not directly assigned on the most recent class, or if you are checking for an interface that is a parent or grandparent of the interface which is directly assigned to your class.
      • Confused by that? Doesn't really matter -- just use this method to avoid any funny business with "is" or "as" keywords. Then when you get a true back from "SomeExternalData.GetDoesThisImplementInterface( typeof(YourInterface) )", make sure and do "(YourInterface)SomeExternalData" to get a YourInterface, NOT "SomeExternalData as YourInterface" in order to properly get the interface.
      • If you're curious, then calling "(YourInterface)SomeExternalData" when "SomeExternalData.GetDoesThisImplementInterface( typeof(YourInterface) )" returns false will give you an InvalidCastException. Technically you could wrapper a try/catch around that and use that to direct your code, but this is rightly considered very bad code and also is comparably very very slow.
  • So what can we DO with this?
    • Well, let's take ExoData as an example. As mentioned in yesterday's notes, sometimes there is an ExoData that is now directly assigned to a class (like FallenSpireFactionBaseInfo). Let's break down how this works.
      • If you know that FallenSpireFactionBaseInfo has ExoData and you want to look at it, you can just call faction.GetExternalBaseInfoAs<FallenSpireFactionBaseInfo>() and then immediately access the ExoData. Well that's simple!
      • But what if you DON'T know what type the BaseInfo is, for instance if you're iterating over a list of BaseInfos for various other factions for whatever reason. Maybe they are from other mods, even. Who knows?
      • Well, in that case we need a generalized way to ask ANY BaseInfo "Do you have an ExoData?" In the past, we could just call faction.GetExoDataExt( ExternalDataRetrieval.ReturnNullIfNotFound ), and what we want is an equivalent to that. So that's what we're creating here.
    • First of all, we need a new interface. In the file where the ExoData class exists, I have defined a new interface called IExoDataHolder, and gave it just a single method, called GetExoData(). See code for details, but it's super duper simple.
    • Secondly, any class that has ExoData on it that I want to expose in this generic fashion should now inherit from IExoDataHolder. For the moment, that's just FallenSpireFactionBaseInfo, but later it will be two others once I port them in.
      • Now on thos classes, I have to implement the very very simple GetExoData() method that I designed in IExoDataHolder. This is one line of internal code, just returning this.exoData.
    • Now, back in some other code where I would have previously called faction.GetExoDataExt( ExternalDataRetrieval.ReturnNullIfNotFound ), I can call the following:
      • "if ( otherFaction.BaseInfo.GetDoesThisImplementInterface( typeof(IExoDataHolder) ) )"
      • then: "exodata = ((IExoDataHolder)otherFaction.BaseInfo).GetExoData();"
  • And that's it! It would have been possible for me to make this code slightly less verbose, BUT that would have been at the expense of flexibility for faction designers and mod designers.
    • With the current approach, you can implement ANY arbitrary data-holders on any subset of classes you choose, and Core never needs to be altered in order to do so. The power of that is really useful.
    • Please bear in mind that for most sub-data, we don't remotely need to make use of this. We don't always CARE what factions use what feature. But for cases where we want to loop over all the factions and ask "which of you can give me this kind of subdata?", this new pattern lets you do so.

• NanocaustPerUnitData has been retired from the game, since it was noted as deprecated in 2019.
  • ConstructorData for the nanocaust was not marked as deprecated, but it sure was unused, so it's also gone.

• Added a new AllegianceHelper static class, which has things like enemyThisFactionToAll and such moved into itself.
  • Fixed a couple of small bugs in here that could lead to strange allegiances. One of which was related to multiplayer allegiances with players beyond the first.
  • Beyond that, there was a lot of special-case type code that was looking for specific factions and making choices about that. Those instead need to be triggered by xml data or this is not future-proof and mod-friendly.
    • Added allegiance_should_always_be_neutral_to_all="true" as a new option for factions, and this is now on the nebula guardians and territory dispute factions.
    • Added allegiance_should_always_be_friendly_to_all="true" as a new option for factions, and this is now on the Zenith Trader.
      • Fixed several ways in which factions could become hostile to the Trader in particular.
    • Also added allegiance_factions_i_always_hate="Faction1,Faction2" and allegiance_factions_i_always_ally="Faction1,Faction2".
      • This lets us centrally define some relationships that should always be friendly or enemy, using the name (InternalName) field from the factions in question.
      • Key note: these faction names are not checked, so they can be from mods or dlc that are not installed without problem. But if you have a typo, it may be hard to notice for that same reason.
      • This is not only more flexible (for mods and otherwise), but it also prevents the AllegianceHelper from accidentally overwriting entries if things like MakeEveryoneAnEnemy or whatever are called.
      • At present this is used to make Nebula Guardians and Champions hate one another, and Necromancers and Templars to always hate one another as well.
  • Also moved SetDefaultStartingFactionRelationships() into there, so that the defaults can be seen in External code (it had recently moved to Core) as well as changed as-needed.
    • Bear in mind you can override the defaults at a faction level via a mod, but you can't do that over what is in this helper class for ALL factions. But you can certainly rewrite faction alliances however you want via a mod.
  • As part of this, made it so that we never can override (via larger faction alliance methods) team affiliations (for the minor faction teams of three colors) or always-hate or always-ally settings. There were definitely some cases where someone could break off previously.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • MacrophagePerSporeData is now MacrophagePerSporeBaseInfo
  • MacrophagePerHarvesterData is now MacrophagePerHarvesterBaseInfo
  • MacrophagePerTeliumData is now MacrophagePerTeliumBaseInfo
  • OutguardUnitData is now OutguardPerUnitBaseInfo
    • Made OutguardGroupDataTable serialize by index, and thus improved the data transmission of this.
  • MarauderOutpostRaiderSpawnData is now MarauderOutpostRaiderPerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • MacrophageGlobalData is now MacrophageFactionBaseInfo
  • NanocaustMgr is now NanocaustFactionBaseInfo
  • OutguardGlobalData is now OutguardFactionBaseInfo
  • DysonData is now DysonSphereFactionBaseInfo
    • DysonAntagonizerData has been removed, and merged back into DysonSphereFactionBaseInfo. There was always a 1:1 ratio of these existing, and there is no benefit to having them separated at this time, while there are several downsides.
  • HRFData is now HumanResistanceFighterFactionBaseInfo
  • MarauderData is now MarauderFactionBaseInfo
    • The marauder UpdateAllegiance has been updated to be more robust and use the AllegianceHelper.
    • Eight constants from the marauders xml were messily being saved into the savegame, which was mildly wasteful and moderately confusing. This is all cleaned up now.
  • RiskAnalyzerData is now RiskAnalyzerFactionBaseInfo, and is on the risk analyzer faction rather than on a human faction (that was super confusing and likely led to errors).
    • It is now our second example of an IExoDataHolder.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • DarkZenithPerUnitData is now DarkZenithPerUnitBaseInfo
  • ZenithMinersPerUnitData is now ZenithMinersPerUnitBaseInfo
  • ZenithArchitravePerUnitData is now ZenithArchitravePerUnitBaseInfo
  • TemplarPerUnitData is now TemplarPerUnitBaseInfo
  • TerritoryDisputePerUnitBuildData is now TerritoryDisputePerBuilderBaseInfo
  • SappersPerUnitData is now SappersPerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • GlobalCPALogic is now AICrossPlanetAttackerBaseInfo
  • ScourgeFactionBaseInfo is actually fully set up now, apparently I got distracted in the middle of that one.
  • DarkZenithGlobalData is now DarkZenithFactionBaseInfo
  • ZenithArchitraveGlobalData is now ZenithArchitraveFactionBaseInfo
  • ZenithMinersGlobalData is now ZenithMinersFactionBaseInfo
  • TemplarGlobalData is now TemplarFactionBaseInfo
  • NecromancerGlobalData is now NecromancerFactionBaseInfo
  • TerritoryDisputeTeamData is now TerritoryDisputeTeamFactionBaseInfo
  • NomadPlanetsGlobalData is now NomadPlanetsFactionBaseInfo
  • SappersGlobalData is now SappersFactionBaseInfo
  • MigrantFleetsData is now MigrantFleetsFactionBaseInfo

• Went ahead and removed the champions and nebula guardians factions and their related data.
  • The history of them is available in svn, but as has been discussed elsewhere, necromancers and templars replaced these.
  • At the moment, the time cost of translating the rest of these over was going to be a waste of several hours.
  • EnableChampionResponse is now EnableNecromancerResponse, although that setting is not yet hooked up to do anything, so it's commented-out for now.
  • A lot of the central code that was related to champions has been renamed to SoloArk, and converted slightly in preparation for that kind of faction later on.

• We're now to the point where the only faction info remaining in BaseInfo that has not been converted is stuff relating directly to the AIs themselves!
  • There will still be a flood of other things needing to be adjusted after that, but that's a milestone that is approaching (and a milestone that has been hit, as well).

• The concept of ProcessFactionConfigCustomFields has been removed from the game.
  • This was something that was previously running right after you left the lobby and translating the settings from your "soft settings" into permanent saved data on the actual factions.
  • As of a few days ago, I made it so that we're now pulling much more directly from the "soft settings" in the first place, which is useful for making sure that the game always has your latest settings even after making changes in the post-lobby settings areas.
  • It also lets us avoid having to save the data a second time for the factions, so that the "soft settings" location is instead the only place the data is saved -- which is good, because when you save the same thing two places you are at BEST wasting some bandwidth/disk space (less than a kilobyte, but still), and at worst you secretly have an accidental disconnect from what front-end shows and what the back-end is working from.
  • In place of this, there's a new convention -- not enforced -- of DoRefreshFromSettings() on factions, where we can get the settings on a small random interval, called from the chain of DoGeneralAggregationsPausedOrUnpaused(). This keeps the CPU usage down, things nice and responsive, and the settings super up to date, all at once. It's also easier code to read.

• We now generate a completely random FactionRandomSeed on each faction when we're starting the game. This allows us to do things like random AI types that are unpredictable, but still deterministic per-campaign.
  • Actually we wound up not using this right now, but it might be useful in the future, and it's very little data, so what the heck; let's keep it.

• Serializing a table entry no longer confusingly takes two string variables, one optional and one not, when deserializing while taking only one for serialization. It now just requires one for both, which is the same one.
  • Additionally, that FieldNameForErrors is no longer optional for these.

• Work is ongoing with getting the AI and its sub-factions set up. A lot of this is coming together quite well.
  • There were a few surprises, like the way that I need to cache the AI Type after randomly choosing it, but that's not something you can change once the game starts, anyway (or if it's adaptive, it changes itself in the one spot here). Differentiating this from the other settings was not hard. Now it's just a matter of doing the rest, carefully.
  • There's a solid chance that this will implicitly fix some bugs that were recently reported about things like "individually chosen sub-difficulties not working properly when universal difficulty is off." That was likely an artifact of the order of operations craziness that was part of the older way of using ProcessFactionConfigCustomFields. Once it's ported over, it should be order-of-operation-independent.

For Modders: How To Handle Lists Of Sub-Data In Multiplayer-Friendly Way

• Added a new extension method to List<> and ProtectedList<> that makes deserialization into an existing uneven list waaaay simpler.
  • This is what needs to happen, over and over again, on clients when they are getting certain kinds of external data.
  • PlannedWaves are one of the many that needs this sort of logic, so formalizing it so that it isn't prone to errors (and is easier to read and understand as end-coder) is a win.
  • Basically, you use this when:
    • You have a list of stuff that is variable length over time, and the client is going to be getting a deserialized copy of it pretty frequently... AND
    • You don't want to just clear that list (because what a waste on the one hand, or other code might be having references to objects in it)... AND
    • You do want to refill the list with the most up to date data in host-order, and then discard any leftover husks that were on the client.
  • Example 1:
    • Client has a list of 10 incoming waves. Host sends an update that includes 7 waves for whatever reason.
      • Result from this process: Client deserializes those 7 into its first 7 slots (overwriting whatever is there), and then discards the last 3. If it's a ProtectedList<>, then those 3 are headed back to the pool automatically.
  • Example 2:
    • Client has a list of 1 incoming wave. Host sends and update that includes 5.
    • Result: Client deserializes the first 1 into the existing one it has, then adds 4 more.
  • Example 3 is simply when the number of waves match, whether or not the data in them does, and it just overwrite-assigns each one as it goes.
  • The idea is to handle all of that logic without having to write all of that code every time (and to avoid the typos and confusion that are inhernt in copy-pasting such code all over the place).

• Okay! Time to handle the first of the sub-objects that are pooled properly. Let's look at PlannedWave as an example.
  • We're going to want to have it inherit from the following: PlannedWave : TimeBasedPoolable<PlannedWave>, IProtectedListable<PlannedWave>
    • This sets us up to let us have it go in a variety of kinds of pools, as well as also function inside "protected lists" (where when you clear the list, it goes back in the pool for you automatically).
  • There is then a "#region Pooling" section, and you basically should copy that whole bit over and change and references to PlannedWave to be whatever your class is (same as with the inheritance bit above, of course).
    • In this section there is a DoAnyBelatedCleanupWhenComingOutOfPool(), which you can usually leave blank, but essentially it's helpful if there are some object references that might briefly be in use still when this gets put back into the pool, that you don't want to cause nullref exceptions from.
    • In this section there is also a DoEarlyCleanupWhenGoingBackIntoPool(), which tends to happen on background threads instead of one that is more busy, so anything you need to blank out that you can, should probably go here. You want to make sure and reset EVERY variable and list and so on that is a sub part of this object in this.
      • In this example case, to keep things simple and also consistent, I've got a SetDefaults() method that is being set in the constructor as well as on the DoEarlyCleanupWhenGoingBackIntoPool(). This is a good pattern to keep.
        • It is SUPER important that absolutely every property or variable on your class (PlannedWave in this case) is reset to its default value in SetDefaults(). Otherwise the next wave will have some stale data and act strangely when this gets pulled back out of the pool.
    • Please also note that there's a private constructor in here, and then no other constructors anywhere else in the class! We don't want any way of creating new objects of this sort other than calling the static method GetFromPoolOrCreate(). Anything else we call would be bypassing the pool and causing at least a mild memory leak.
  • At this point we have a poolable object that can only be retrieved from a pool, but we're not doing any retrieval or ever putting them back. So now let's do that.
    • On AISentinelsFactionBaseInfo, I would normally have a List<PlannedWave> WaveList. Instead we're going to use a ProtectedList<PlannedWave> WaveList, and it pretty much works identically.
    • This is a case of this list being the truly master list. If it's in this list, it's meant to exist. If we take it out, it's meant to be gone back to the pool. So, based on that, we don't ever have to think about calling ReturnToPool() manually on the PlannedWaves. That's handled for us. Sweet!
      • On Clear and Remove from this list, just be sure to pass in true as the required parameter, and it will do that handling for you. In the event that you wanted to use a protected list without it automatically sending stuff to the pool on removal, you could pass false... but that is not advised.
    • In that case, that means that the only thing we need to change beyond this is wherever we were previously calling "new PlannedWave()". That simply changes to "PlannedWave.GetFromPoolOrCreate()", and boom we're pooling. And we're done.
  • In the case of PlannedWave, I am using a ConcurrentPool, by the way. This is something that can be written to from any thread, and it has some nice advantages. But there's not "time out" cooldown before waves are reused from the pool. Because of the nature of how waves are created, this is not a problem.
    • In the case of a piece of data where you might have lingering references for a while, you might be better served using a TimeBasedPool instead. That works exactly like ConcurrentPool except that you can specify a cooldown time that the objects must stay in the pool before they can come back out. 2 intervals of 30 seconds each are enough for all long range planning to complete for every faction in a super pessimistic case, if that's what you want to base it on. Otherwise just give it a couple of seconds of delay. At any rate, PlannedWave doesn't need that.

Reworking Subfaction Relationships And The AI and Beacons, Part 1

• Added a new AISentinelsFactionBaseInfoExtensions class, with GetAISentinelsBaseInfo() on it as an extension of the faction class.
  • The reason I'm doing this for the sentinels and not any other factions is because of all its many subfactions, and their constant need to get the info of their parent.
  • I'm also planning on storing all of the meaningful subfaction data of any note on the parent itself, to further tighten these things together.

• Removed GetPrecedingAIFaction() and GetNextFactionOfSpecialFactionData() from the Faction class in Core.
  • Now that I'm working on the AI and its subfactions, the current situation with how they are linked is just really untenable. It was unreliable before, and would not always properly link the factions together exactly as expected. Time to change that.

• Just like we have FactionIndicesIAmHostileTo and FactionIndicesIAmAlliedWith, factions now also have FactionIndicesOfMyChildren and a new FactionIndexOfMyParentIfIHaveOne.
  • These let us keep track, explicitly, of which subfactions are linked to what parent faction, in a bidirectional sense.
  • There is a new MakeChildOfOtherFaction( OtherFaction ) method on the faction class that makes this a one-line call to initialize.
  • You can then just call the new GetParentFactionOrNull() on factions to get a reference to the parent faction if there is one.
  • Part of the idea here is to make it much simpler for our AI factions and subfactions, sure, but also the other part of the idea is to make it so that mods can now set up compound-factions like this if they need to. Or us with other related factions if need be.

• General improvements to some of the serialization code have been made.
  • Among these, planets and factions no longer have a serialization gate calculated for them. They are simply now capped at 511 for planets, and 255 for factions. These limits were already well enforced elsewhere.
  • Fixed a minor new SpecialFactionData serialization issue from it getting mangled in all the ripping and tearing. It would have been fast to find later, but this saves time.

• Getting rid of some other methods that we no longer use and so just clutter things up:
  • On faction: GetDoesThisFactionHaveBaseInfoType(), GetDoesThisFactionHaveDeepInfoType(). That new FactionRandomSeed.
  • On squad: GetFactionBaseInfoType_Safe(), GetFactionDeepInfoType_Safe().

• Split the BaseInfo and DeepInfo serialization out of the main faction methods for serialization into new SerializeExternalFactionData() and DeserializeExternalFactionData().
  • This is actually a pretty huge deal, because now we'll send the externaldata at the end of any transmission, and at the end of any savegame.
  • This means that factions, squads, planets, etc, all already exist when externaldata starts getting loaded. Previously a lot of that was not loaded yet, so externaldata had to use indices during deserialization rather than looking at real objects. This will simplify code, and speed up some bits of code, and reduce coding errors.
  • Same for BaseInfo and DeepInfo on squads, which is now based from SerializeExternalSquadData() and DeserializeExternalSquadData().

• The multiplayer fast-blast sync code has now been updated to handle the BaseInfo from host squads needing to be synced over.
  • The faction BaseInfo sync in multiplayer via PeriodicFactionExternalSyncDataThatJustOverrides has been updated to the new format, which was very simple.

• Split SerializePlanetFactionEntities and DeserializePlanetFactionEntities out from the core PlanetFaction serialization, since that was rather confusing code the way it was before.
  • Also, CurrentAttackMultiplier and CurrentSpeedMultiplier are no longer serialized at all (it's recalculated every frame).
  • And the shipgroups are no longer serialized, even in super-brief form, except for AI factions (they're the only ones who use it).
  • Uh, also a lot of other stuff moved around. It's more efficiently organized, less likely to cause errors, and easier to extend.
    • The really big one is that this completes the move of all the externaldata to the very end of a network transmission or a savegame save. For factions, squads, and the world. That way the external data can always freely talk about the Core data without worrying if it's been loaded or not yet.

• For the sake of clarity, the factions that are auto-added-always-there-but-only-one have been moved into one file, and the "factions that are auto-added-per-AI" have been moved into another.

• On factions, auto_add_one_of_faction_if_missing_on_save_load="true" has been changed to should_always_have_exactly_one_of_these_in_every_game="true"
  • This has been removed from several sub-factions of the AI, and instead is just on the things like Outguard, AI Reserves, Instigators, and the zombie factions.

• There was then an actually_seeds_multiple_so_do_not_complain="true" that was an answer to some of the factions that had must_be_at_most_one="true" on them but were not supposed to complain.
  • Those factions were erroneously marked that way in the first place (because of a long history of evolution of the game, it's complicated), so that actually_seeds_multiple_so_do_not_complain="true" has been removed.

• auto_add_one_of_faction_per_ai_if_missing_on_save_load="true" and remove_if_more_than_one_per_ai_prior_to_this="true" have both been removed, as they were kind of funky.
  • Instead, added a new is_a_subfaction_of="OtherFactionName", which lets us define the relationships directly, and which causes them to get created one-per-such faction, call MakeChildOfOtherFaction() on said factions, and disappear if that faction is not present. All speaking here about on the lobby load, or savegame load.
  • The warden, hunter, PG, CPA, relentlesss wave, and border aggression are now marked as subfactions of the AI.
  • The territory dispute teams are now marked as subfactions of the parent territory dispute faction.

• takes_on_color_of_the_first_ai_faction="true" is being left alone -- that's useful for instigators and so on -- but takes_on_center_color_of_the_preceding_ai_faction="true" is being removed.
  • We now instead have a takes_on_center_color_of_the_parent_faction="true" as an option which can work for any faction that is a child of another. For now it's just used with the AI subfactions: warden, hunter, PG, CPA, relentlesss wave, and border aggression.

• is_considered_visible_part_of_the_ai_faction_for_what_can_own_purposes="true" is removed, as it was not actually used at this point.

• takes_on_current_general_mark_level_of_the_ai_factions="true" has been renamed to takes_on_highest_current_general_mark_level_of_any_ai_faction="true", because that's what it does.
  • This is now used by the instigators, AI reserves, astro trains, and risk analyzers, but is no longe used by the AI subfactions.

• Added a new takes_on_current_general_mark_level_of_parent_faction="true" that is now used by the AI subfactions.
  • Previously if you had many AIs of different strengths, their subfactions would all be the mark level of the strongest AI, which was wrong.

• The way that "beacon factions" work is being completely reworked under the hood.
  • The user experience was never great (people didn't realize they could configure said factions prior to hacking the beacons but that it got more limited after that).
  • And there was a nontrivial performance drain, and some bugs, from having lots of these. Macrophage picking up spire shards when not in the game except in beacon form, etc.
  • In the next few weeks I will re-code the hacks for the beacons so that you can configure all the aspects of the beacon faction as you are summoning them. The beacons will from now on belong to the NaturalObjects faction, however.
  • In the meantime, all of the stuff related to a faction being present but "needing to be awakened" is being removed.
    • MustBeAwakenedByPlayer, MustBeAwakenedAndIsAllied, ShouldBeRetainedIntoLobbyLoadEvenIfMustBeAwakened, and HasBeenAwakenedByPlayer.

• auto_remove_faction_on_lobby_load_and_mapgen_when_other_faction_present="FactionName" is also being removed, as we don't need it anymore.

• auto_add_one_of_faction_when_other_faction_present has been converted into auto_add_singleton_of_faction_when_any_count_of_other_faction_present, to be more clear.
  • This is still used on tamed and enraged macrophage, but is also now used by the antagonized dyson sphere. This is also still used by the templars in response to a necromancer.

• When you load up a savegame into the lobby, or old lobby settings, or a quickstart into the lobby or for generation, it now strips out all the following:
  • Any subfactions of other factions. Any that should always have exactly one per game. Any that are automatically added when some other faction is present.
  • This is mostly equivalent to what we did in the past, but the general idea is we never want to see things like a dormant tamed macrophage with no main macrophage present, or extra hunters due to having fewer AIs present now, etc.
  • The only time these things are added BACK is when a full generation of the map happens (aka moving from the lobby into the game, or from the quickstart into creation of the game proper). Then all the magic fills out the relationships and should have exactly the faction list as desired.
  • We had half a dozen idle factions clogging up every savegame in the past, and sometimes more, which was potentially a waste of bandwidth and performance. It was also a barrier to adding beacons for more factions, since there was such a performance hit with each one.
  • All of the various code paths that do these sorts of things now seem to function, which is awesome.

• Removed GetDefaultFactionConfigurations() from the Scenario definition, since now most of our code is much more xml driven.

• One thing that isn't fully xml-driven, but nonetheless is now part of the Expert campaign type, is that you must have a minimum of two AIs to face.
  • A lot of the complexity of the game is lost when you only have one AI, because you don't get that ramp-up in challenge after defeating the first one, and have to figure out how to still deal with the second one.

• The seeding options for a couple of DLC2 factions that were "Only From Beacon" have been removed.
  • These will get beacons in the near future, along with some other factions that never had beacons, but they will work the new way.
  • Worth noting that this didn't play well with the Enable Beacons setting, and so could get kind of funky in Expert mode because of that.

• Also stripped out more code that was based around the concept of a faction being present as a beacon but not enabled.
  • Later, the logic will be that the beacon is present as a NaturalObject, and the only time the faction will arrive is after a hack adds them (letting you clearly choose settings for it before you complete the hack, too).

• Added in a new HasDoneInvasionStyleAction to factions, which is false by default.
  • Previously, we had a HasBeenAwakenedByPlayer which was used one way sometimes (for beacons) and another way other times (for delayed invasions). That had led to various bugs in the lobby.
  • Now we just have HasDoneInvasionStyleAction, which is used for invasions or not at all. Most factions ignore this one bool and just go about life.
    • This is primarily used by the nanocaust, but in theory other factions could use it later.

• Simplified some of the PlanetFaction data. FInts are the largest data format that we have for casual use, and each PlanetFaction object had three of them (AIPLeftFromControlling, AIPLeftFromCommandStation, and AIPLeftFromWarpGate).
  • In an 80 planet game with 21 factions under the hood (remember, one AI is a compound faction that is 6 under the hood alone), that would have been 5,040 FInts. Those each require somewhere in the ballpark of 64 bits to store, which would be a total of 40 kilobytes in a savegame that is maybe 200-600kb in total in the early stretch (so, that's as high as 20% of our entire data storage in the early game, for context. Granted, things get adjusted around a lot to avoid this, but I still wanted to minimize what is on PlanetFactions, and the above is a good enough example to show why.
  • We no longer have AIPLeftFromControlling at all, since that could be inferred from the other two. The other two are now stored as integer values, and specifically AIPLeftFromCommandStation is serialized as a 17-bit format, and AIPLeftFromWarpGate as a 9-bit format.
    • Assuming no other changes (ha -- there already were improvements elsewhere, but let's ignore those for this example), this would be 5.74 kilobytes of data now, instead of 40.
    • This matters much less in savegames than it does in multiplayer, where we are frequently sending things like the planetfactions from the host to each client just to make sure that things are synced up.
  • The bandwidth use by the game prior to this whole rewrite of the game's structure was pretty darn alarming (though still lower than a poor-quality youtube video, for instance). But that bandwidth usage made higher player counts harder for people to do, and put an extra strain on people who had a lot of packet loss or were physically very far apart.
  • This is just one of those "low hanging fruit" ways to reduce transmission amounts, but the whole rewrite of all this is aimed at least partially in that direction (other than bandwidth reduction, the rewrite is aimed at code correctness, client functionality parity, and ease of coding for modders with the game having to send less self-correction data.

• Added TryGetExternalBaseInfoAs() and TryGetExternalDeepInfoAs() on faction, mirroring the versions that are on squad.
  • Added a new TryGetAISentinelsBaseInfo() in AISentinelsFactionBaseInfoExtensions as an extension on the faction class. This more or less mirrors the "get or return null" style.
  • Essentially, in your mods you can do a global search and replace for "GetSentinelsExternal( ExternalDataRetrieval.ReturnNullIfNotFound )" to "TryGetAISentinelsBaseInfo()"

• All of the AI subfactions are now going to have sub-objects that are shared on the central AI's object. This greatly simplifies how they can access each other's information, but lets us still keep things ordered. As with ExoData, these will be readonly subobjects that we can then edit at will without ever replacing them, and so they don't need to be pooled, etc.
  • AISentinelsExternalData is now AISentinelsBaseInfo. This is now a readonly sub-object of AISentinelsFactionBaseInfo, which makes it really convenient for any sentinels faction or subfaction to get at the data. I'll be following the same pattern with the other data of this sort.

• TimeBasedPoolable now does the thing where it seals and obsoletes ToString(), and has a virtual method called AppendStateForInterfaceDisplay().
  • We really want to avoid having classes that are calling ToString(), because that's pretty much always GC churn and a waste.
  • EntityOrder now uses AppendStateForInterfaceDisplay instead of ToString(). It's a case of not even knowing it was a problem until it was made a compiler error.
  • PlannedWave also has been updated in this way, as has EntitySystem.

• ArcenCharacterBuffers of all sorts now support passing in an ArcenPoint and having that serialize nicely.

• ExtragalacticBudget has been updated to be poolable in the same pattern that was implemented for PlannedWave.
  • Fixed an issue that would cause deserialization errors in ExtragalacticBudget on MP clients. This was a longstanding issue, looks like, and very rare and confusing if it happened.
    • The fix for this was switching to the new DeserializeUncertainNumberOfEntriesIntoExistingList(), which is a great example of just why this new thing is so darn useful.
  • ExtragalacticBudget is an interesting case, because it was very complex with multiple constructors. But those simply are turned into static Create() methods, and then suddenly the complexity is gone and the program flow is the same. There are many similar examples in Core, but this is the first one in external code.

Reworking The AI And The Last Of BaseInfo, Part 1

• AISentinelsBaseInfo is now AISentinelsCoreData, because that naming was HYPER confusing. Naming anything BaseInfo or DeepInfo when it's not actually that is a really bad idea, turns out.
  • The other sub-classes of AISentinelsFactionBaseInfo are now being set up as well. Bear in mind that these are part of the sentinels faction data, not part of the sub-data of the sub-factions. They do have their own sub-objects, though, to keep things organized but accessible.
    • AIWardenExternalData is now AIWardenCoreData
    • HunterFleetExternalData is now AIHunterCoreData
    • HunterFleetExternalData is now AIHunterCoreData
    • PraetorianGuardExternalData is now AIPraetorianGuardCoreData

• AICommonExternalData is now GlobalAIWorldBaseInfo, which is something that I am attaching to the world object, because it's thing that is truly global between all of the AI factions that might exist.
  • This is slightly more efficient from a data standpoint, and is simpler to use. Any faction can do this sort of thing, particularly if there are going to many of that faction.
  • This also declares a singleton, so it's really easy for any code anywhere to now just go GlobalAIWorldBaseInfo.Instance.AIProgress_Effective or whatever.

• AIPChange is now being treated like PlannedWave, with the new TimeBasedPoolable and IProtectedListable inheritance.
  • This is probably now the simplest example of this pattern in use, because it's a fairly low-data-and-methods class to begin with.
  • There was some logic in GetAIP that previously was setting the AIP to the highest version based off of any AIP. The coder had been note the code as being a bit hacky. I've move that over into GlobalAIWorldBaseInfo, and it calculates those things now on that method.

• Added a new EnumIndexedArray in Universal, so that we can get rid of all those class-specific versions and add more functionality to them.
  • In particular there is a new ClearTo method. But there's also a reference tracker, so that we'll know how many there are in the game if we do a data dump. And there's just no longer a bunch of duplicate code all over the place so much.
  • Then removed the following classes, converting them to instead use this new EnumIndexedArray:
    • Universal: ArcenEnumIndexedArray_PerformanceSegment, ArcenEnumIndexedArray_FourDirection
    • Core: ArcenEnumIndexedArray_CoreFunction, ArcenEnumIndexedArray_EntityLineHardcodedType, ArcenEnumIndexedArray_MetalFlowPurpose, ArcenEnumIndexedArray_SpecialEntityType, ArcenEnumIndexedArray_OtherSpecialEntityType, ArcenEnumIndexedArray_EntityRollupType, ArcenEnumIndexedArray_FactionType, ArcenEnumIndexedArray_PlanetFactionBooleanFlag, ArcenEnumIndexedArray_EntityOrderType, ArcenEnumIndexedArray_FactionStance, ArcenEnumIndexedArray_ResourceType, ArcenEnumIndexedArray_EntityBehaviorType
    • External: ArcenEnumIndexedArray_AIBudgetType, ArcenEnumIndexedArray_HunterFleetBudgetType, ArcenEnumIndexedArray_PraetorianGuardBudgetType, ArcenEnumIndexedArray_AIWardenBudgetType, ArcenEnumIndexedArray_Code, ArcenEnumIndexedArray_ClientConnectionStage
  • Tip for modders: you can use a simple global search and replace for each type you remove, in this fasion: "ArcenEnumIndexedArray_AIBudgetType<" with "EnumIndexedArray<AIBudgetType,"

• The external data for factions and squads now has a virtual method that you can override called DoAnyInitializationImmediatelyAfterSquadAssigned() and DoAnyInitializationImmediatelyAfterFactionAssigned(), respectively.
  • On AISentinelsFactionBaseInfo, I am now using this to call "this.WardenInfo.InitializePathfinders( this.AttachedFaction );" and the same for the other subfactions.

• It turns out that ExoData is the perfect model for how to handle the AntiMinorFactionWaveData, so that's the pattern I'm going with, including having a IAntiMinorFactionWaveDataHolder.
  • MarauderFactionBaseInfo and NanocaustFactionBaseInfo have been updated to hold an AntiMinorFactionWaveData WaveData object.

• And that's the last of converting old style ExternalData over to BaseInfo!
  • At the time of this writing, there are 195 errors still remaining (newly revealed now that all the types are converted over) in the BaseInfo dll that need to be resolved before I can start truly working on DeepInfo.
  • There's also 85 "//EXTERNALDATA_TODO" tags where I commented out some code in BaseInfo to let it compile at one point last week. I'll have to uncommented those and get that all compiling, and then BaseInfo will be done-enough-until-DeepInfo-forces-more-changes, but at least I'll be able to compile it.

• Uncommented out all 85 of the //EXTERNALDATA tag sections, and now I'm up to 385 reported errors in the BaseInfo dll. Time to get cracking on those!

• Removed the PreferredAIFactionIndex from faction external base data.
  • This has been made obsolete by FactionIndexOfMyParentIfIHaveOne on the Faction object itself.

• BaseAIFaction.ChangeAIP() should now be GlobalAIWorldBaseInfo.Instance.ChangeAIP(), which has been converted to the new format.

• SpecialFaction_CPALogic.cpaDelayTime is now AICrossPlanetAttackerBaseInfo.CPA_DELAY_TIME.

• Rather than ever using factionExternal.AIProgress_Effective, it should now always be GlobalAIWorldBaseInfo.Instance.AIProgress_Effective.
  • Same with AIPChangeHistory and things like that.

• GetPrecedingAIFaction() should now be GetParentFactionOrNull().

• I've improved the way that we're able to call in and get faction configuration data. You can have it error when you ask for data from a field that does not exist on that faction, or just return the default value.
  • Most of the time you'll want it to error, because that's probably a glitch in your code that will cause unexpected behavior. It's the sort of thing where a small typo can just wreck your whole day, but now it's actually going to tell you about it and therefore it's super quick to fix.
  • Then again, there are other times where we're scanning past lots of factions, and saying something like "hey, do you have an intensity? I'll show it if so. Got an allegiance field? I'll show that, too, if you got one." And if the answer is "no I don't have one," that's not an error at all. Those are the cases where you'd pass in false to having it error on missing data.

• Added some extension methods where you can call faction.CustomData_GetSpawningBooleanOne(), and a variety of others that start with CustomData_[Whatever]
  • Many references to Ex_MinorFactionCommon_GetPrimitives have been expunged, with most being converted using the above, but others just directly referencing some variable in a new place. Still more to go.
  • This also pulled out a ton of code from ProcessFactionConfigCustomFields() that was very specific to just that one method, and now that's more formalized and something that can be called from more than just that one place.
  • The complete difficulty settings for the AI and all of its subfactions now get pulled out properly, and should be far more accurate (there were some reports before of univesal difficulty not being handled properly when disabled)
  • Also as part of this, many of the generic names for things (like "SpawningBooleanOne") have been renamed to actual sensible things (like "EnableFimbulwinter").
    • Side note: it was a huge pain in the rear to even try to add new proper field names before, so this is not a criticism of faction coders in the past. It was an archtectural challenge that no longer exists.

• Added a new DevourerFactionBaseInfo, which tracks the TimeLastExisted for that faction, as well as some other info that was in the DeepInfo dll but needed to come out.
  • Also added a new LoneWandererFactionBaseInfo, which will act as the underlying basis for handling both the Devourer and the Zenith Trader.
  • Please note, this is actually a great class to look at if you're thinking of doing something similar. It has a TON of extra comments in there, mostly about how to properly use DoubleBufferedList<>.
  • I am going to start having a habit of putting "TEACHING_MOMENT:" in the code anytime there's an example class with a lot of things that are explained. That way coders can search for "TEACHING_MOMENT" and find anything that I (or someone else) has put in there.
    • There are two teaching moments in this particular file. One for double buffered lists, and the other for Sealed methods and abstract submethods.

• Updated my DoubleBufferList so that the swap and clear actions are two separate things.
  • The reason for this is that at swap-time, it's very likely that another thread has a hold of the expiring display list. So clearing that would cause an exception.
  • By the time we loop back around and are ready to construct a new list, any other threads are fairly certain to all be using the new display list, so clearing that other list won't bother them. There's a much larger gap between list constructions than there is any display-list-accesses, by design. This is a common feeder pattern.
  • It's something I had realized the other day, and had just put on my long list of to-dos, but at this point I need more of these to use for things like keeping track of the Devourers running around, etc.

Reworking The AI And The Last Of BaseInfo, Part 2

• A variety of code translations to the new format that are not worth detailing, and finished the work of getting rid of Ex_MinorFactionCommon_GetPrimitives.

• Fixed an issue where we had apparently wished for the Macrophage to have the same "Seed On Nomad If Possible" option that the Dyson Sphere had, but the setting was partially missing.

• The central AI BaseInfo now has direct references to all of its subfactions in the form of handy names like SubFac_CPA and SubFac_Warden and so on.
  • This doubles as a self-checker to make sure that an AI faction is in proper health with all the expected subfactions and no extra subfactions hanging around.
  • This then lets us get rid of a lot of things like GetCPALogicForThisAI(), happily.

• Added a new AISubFactionBaseInfo, which acts vaguely like the LoneWandererFactionBaseInfo class.
  • This is now the base class of all of the AI subfactions, and it makes the act of finding the parent faction and its data super trivial, and also ensures proper linkages or it has a centralized freakout about that.

• All of the various factions in the game and DLCs now have proper BaseInfo classes, and the xml is all linked up for BaseInfo and DeepInfo.
  • As part of this, the three types of zombie factions now share a common base class, like how the lone wanderers and the AI subfactions do.
  • Also, the Svikari have become a subclass of DarkZenith rather than using DarkZenith quite so directly.

• All of the ISpecialFaction classes from DeepInfo have now been renamed to their new proper names, and inherit from ExternalFactionDeepInfo.
  • This leaves a lot of other fixing to do, but it gets things in proper place at least. This is things like SpecialFaction_Human becoming HumanEmpireFactionDeepInfo.

• The above two items also involved splitting a number of files, and setting up some template empty files for various factions, etc.
  • This gets all of the major objects into place, but just not yet with the correct formatting and linkages and so on.
  • It was about an hour and a half just to get things organized like this, and would have taken a lot longer if I did it in small stages. There are still probably tens of hours left on this conversion, but having things working by next weekend is looking increasingly feasible.

• At this point, there were still 374 lines in BaseInfo and DeepInfo that referred to ExternalDataRetrieval, which no longer exists.
  • A targeted set of 64 global search and replace commands got all of those fixed up. (Yeah, it was odd to me that it was exactly 64, too.)
  • Total errors reported by the compiler in BaseInfo are now down to 224, but this was fixing up a lot of others that had not been reported in that number yet, or which are in DeepInfo. The IDE tells me there are 3269 errors in all at the moment, but that's both overcounting and undercounting at the same time. It's progress, anyhow!

Reworking The AI And The Last Of BaseInfo, Part 3

• All of the general cases where I could update "Implementation is" or "Implementation as" in a global search and replace way are now fixed (I didn't count how many it took, but a couple dozen operations to get about 140 items corrected). I'll get the rest as I go through files in more detail.
  • So this leaves me with 220 errors reported by the compiler for BaseInfo. I spent a lot of the morning doing payroll, so let's see if I can knock out the rest of BaseInfo by the end of today. That really would be nice.

• All of the BaseInfo objects for factions that are only allowed to have a single version now have a static Instance on them for quick access.
  • Worth noting: if those factions are not active in the current game, those instance variables will be null! This is by design.
  • These classes also now all inherit from IExternalBaseInfo_Singleton, just to make it a bit more clear again what the intent is.
  • This applies to: Dark Spire, Fallen Spire, Astro Trains, antagonized dyson sphere, HRF, Risk Analyzers, tamed and enraged macrophage, naturalobject, ai reserves, instigators, all three zombie factions, outguard, zenith miners, dark zenith (which has had its class split from its core, for later benefit because of svikari), nomad planets, territory dispute parent, necromancer, and templars.
  • Might as well have a consistent pattern, and this makes converting a lot of other code easier.

• All of the "WAVES_TODO" markers are happily able to be removed, as they will now work properly on clients without any further effort thanks to this general restructuring of data.

• Spire relics can now be moved around properly by multiplayer clients. That was something I hadn't even realized was broken by my changes in July, but it was -- a great example of why this much more intentional split, as time-consuming as it was, is so much better in the long term. I can clearly see in the code itself when something is wrong, without having to wait for testing an exception to pop up.

• The ClientSafeExternal namespace has been removed, and its contents returned to External, because the separation is now in between dlls rather than just namespaces.
  • This is something that is no longer needed at all, and was a half-measure where now a full measure is being taken.

• Marauder outpost descriptions will now have proper info after you load a save and before you unpause the game.
  • Right now I'm making dozens of code changes that are not really worth detailing since they are just tidying things from one format to the other, so instead I'm mentioning things that happen to get fixed along the way.

• Added a new DoubleBufferedDictionary, which works just like the DoubleBufferedList does. Turns out this is needed for various factions, marauders among them.

• Fixed up some code from UpdatePowerLevel() in a ton of factions to keep them from having values that flickered for a few microseconds at a time. It probably wasn't a problem, but why tempt fate.

• The scourge now use vastly less GC every frame in general, and also less when showing their details for the UI.
  • Several minor formatting issues were fixed with scourge being displayed on the UI, and additionally it now shows the list of spawners like it does for armories. Before it meant to, from the look of it, but it only actually showed the count.

• Added a new DoubleBufferedArray, which is now used by various factions, including the scourge.

• Ported over a huge amount of the scourge data and the logic for its data aggregation.

• Added a new DoubleBufferedValue, which is basically like the other double-buffered things but just for a single integer or similar. This is also now used on the scourge data.
  • There are many other ways to handle this same problem, the simplest of which is a working variable. But in looking at the scourge class, where there was a mix of local variables and naked assignments, it's easy to see that that's hard to maintain properly.

• At this point, down to 191 compiler errors reported in BaseInfo.
  • It's worth noting that the bug count in DeepInfo falls dramatically with each of these checkins, too, but those are not numbers I'll ever actually see as high as they are, since I can't attempt a true compile of DeepInfo until BaseInfo works.

• Macrophage turns out to be a really complicated class in terms of how the base, tamed, and enraged variants relate to one another.
  • There were a number of parsing things that I don't think were quite right, and probably led to things like the enraged version always having an effective intensity of 0.
    • I've adjusted things so that tamed still has the intensity of 10 that someone put in there (wow, nice), and the enraged version has the intensity to match the highest regular macrophage now.

• Also to support the macrophage, because they do this sort of cross-faction aggregation to fill what used to be called SporesPerPlanet, but which I'm now calling CrossFaction_SporesPerPlanet, I've had to make a number of new little things.
  • For one thing, the way that this data structure gets filled across faction lines is now vastly more efficient (blind Add calls and one-way caching, versus expensive Contains lookups).
  • This then required me to added a new Stage2APostAllFactionAggregating method, though, that lets me run the final SwitchConstructionToDisplay() on the DoubleBufferList array that is CrossFaction_SporesPerPlanet. That needs to happen once per cycle, after all of the other factions have done their stuff.

• While I was at it, I noticed that the central logic had not yet been set up to actually call the BaseInfo Stage1 and Stage2, they were just calling DeepInfo Stage1 and Stage2.
  • Now it calls both, although I'm unsure if I'll keep Stage1 and Stage2 on DeepInfo. I see pros and cons to that, but it will likely just remain, but be empty for all of our built-in factions. I could see some modders needing the feature.

• Added a DoubleBufferedArcenSparseLookup, because I know how fond some programmers are of the ArcenSparseLookup's extended capabilities.
  • I'd prefer not leaning on this too terribly much, because it comes at a bit of an extra cost compared to a regular Dictionary, but such is life. Sometimes we need those capabilities.
  • As an aside, I built some of the "freedom to call into keys that don't exist without error and get a default value back" capabilities into all of my DoubleBuffered structures. It's one of the favorite advantages of ArcenSparseLookup over Dictionary, and it's something you can use with DoubleBufferedDictionary without having to go all the way to DoubleBufferedArcenSparseLookup.
  • In my opinion, the only real good reason to go to DoubleBufferedArcenSparseLookup is if you need it to be a dictionary but also have the ability to sort, like ArcenSparseLookup has.
    • I almost switched the list of Telia to being a DoubleBufferedArcenSparseLookup because of how it is used in SporeDescriptionAppender, but then realized that I could accomplish what I wanted just by using a foreach rather than a for statement if I kept it DoubleBufferedDictionary. This will be a tad more performant, so that's what I went with.

• Also, I'm no longer keeping any concept of "Wilds" for macrophage. That seemed to be a mix of enraged and normal macrophage for some reason.
  • I now have a singleton for both tamed and enraged, and then a full list of the regular macrophage. They can be queried separately or in series. Sticking the enraged in with the regulars didn't sit right with me, it felt like a problem waiting to happen.
  • Unlike pretty much every other faction I've seen so far, Macrophage is having to have a bit more surgery to parts of its actual logic, partly because it's just that old, and partly because it's been maintained by a revolving list of programmers over the years. Please let me know what issues you find once we're back to being able to play on the new builds again.

• And with that I'm down to 161 stated compiler errors in BaseInfo. But once again, the day has had lots of progress on DeepInfo, so it's not going quite as slowly as it seems to by that error count number alone. My hope, knock on wood, is that DeepInfo will just fly by fast after BaseInfo is all finished up.

• A bunch of fallen spire stuff got moved over into BaseInfo. That was the easiest yet because it's more or less my code, most of it.

• Same with the Zenith Architrave, mainly because they're just a really modern faction in terms of their code.

• Also the same with the Migrant Fleets faction, which is very well done from what I see so far.
  • One thing I fixed was converting cachedMovementTargets from an ArcenSparseLookup into a ConcurrentDictionary, because both the UI and the long-range-planning threads use this, so a cross-threading conflict was otherwise just a matter of time. Now the existing logic will work fine.

• Down to 147 reported compiler errors in BaseInfo.

• Some more general fixes and translations, getting the error count lower.

• Fixed a variety of xml filenames that said BaseInfo instead of DeepInfo. It didn't hurt anything, but was confusing.

• Trying to get the host to trigger SharkB (which must happen in DeepInfo) in response to something that happens in a gamecommand that the client also needs to execute (in BaseInfo) is an early and interesting test of a pattern that is not very common thus far.
  • Overall most of the control code is in DeepInfo, and so we don't need to "call up" like this. Instead, deep just calls down and can see everything, which is the idea. But these exceptions do exist, and so having a formalized way of handling them has always been part of the plan.
  • In this particular case, we're getting our first DeepInfo world-attached object, which I'm calling GlobalGeneralDeepInfoCommandHandler. It's just for a variety of general things, over time.
  • Additionally, the ExternalData_World base object now has its first method of interest, which is called DoAfterEntityDeath(), and lets you pass in a string ActionCode to specify one or more types of things.
    • The flexibility of this is that we can have more than one thing handled in our central library, but also modders can set up their own objects of this sort and also call them. We don't need one object per type of on-death call, since that would be excessive.
      • It's important to know that these are called manually, from the bottom-up, when the sim code detects something it wants to react to. This is the same pattern SharkB was already using.
  • There's another "TEACHING_MOMENT" piece of code in there (I've added several of those today, as well as flagging some comments of Badger's that were teaching moments as being such so people can search for them).

• Imported a bunch of Dark Zenith logic (the stage 2 aggregation). This was more complex because it was having to do way more work since it did not have double-buffered collections in the past.
  • Some things also were able to be simplified because of the new architecture where you can actually get planets and squads and such immediately on deserialization, whereas before this version you would not. So for instance, serializedFimbulwinterEligibleTime is able to be ditched, and we just write straight to FimbulwinterEligibleTime, saving ourselves some hassle.
  • Same with OriginalPlanetIdxs going away, and just having OriginalPlanets, instead.

• Okay, that's it for today. Down to 134 errors in BaseInfo. It took a while to convert over the Dark Zenith, mainly because it's such a big complicated class. But thankfully, the newer logic (double buffers, mainly, but also the deserialization order) really make the code a lot more legible and cut down on extra work.
  • I'm sure it's going to be mildly frustrating to Badger that he didn't have those tools when he originally coded this faction, but I wouldn't have thought of adding them without seeing the same patterns over and over again and realizing I could simplify it based on that.

Reworking The AI And The Last Of BaseInfo, Part 4

• GameCommands used to be all in one giant monolithic file. They have now been split out into a variety of files, by loose purpose or association. This makes it a lot easier to find a specific GameCommand, or to look at several GameCommands that are related to one another.
  • As part of this, and because I needed to, two of the GameCommands (both dealing with changing the galaxy map, as part of ZO) have been moved to DeepInfo, where they will only be run by the host, and where they can access data and methods that only the host would have.

• The dyson antagonizer faction logic has been ported properly between base and deep info now.
  • Same with the the main dyson sphere faction, including wiht the ability to handle multiple spheres around the galaxy, now using the same pattern for that which macrophage are using.

• All the cheat codes work again (yeah that was a thing).

• All of the description appenders work again.

• The central faction processing code is now split out fully into DeepInfo, which is excellent. It's still kicked off from logic in BaseInfo, but the actual execution (only on the host) is in DeepInfo now.

• A number of faction methods that used to be FOR UI ONLY and very problematic if used otherwise are now using ThreadStatic internal markers on their lists to make themselves threadsafe. These are now labeled as being threadsafe, and can be used by the UI, main thread, LRP, whatever -- all at the same time, even!
  • The various notifications that used these ui-only versions of the methods have been updated to point to the new ones, and a TEACHING_MOMENT flag has been put in next to a comment about ThreadStatic at one point.

• The HostOnlyJournalsAndAutosaveAndSimilarHandler has been moved into DeepInfo.

• AIUtilityMethods has been moved from DeepInfo into BaseInfo.

• Fixed a likely bug where nanocaust intel entries have not been shown ever since frenzy fleets were removed (back around the time of fireteams).

• Imported some various things from deepinfo and sappers and such into BaseInfo that were going to move here soon enough anyhow.

• Okay! Well, BaseInfo now compiles, which means that I'm able to get going on DeepInfo properly.
  • There's a lot to translate over there, but I've been steadily doing that as I went, too, so we'll see how long this really takes.
    • There's a fair number of things which are kind of mindless search-and-replace things, like a lot of the ExternalData faction methods no longer have the faction being passed in. That's now handled as AttachedFaction as an embedded field in the class itself.
    • To trick the IDE into fixing over 800+ references that would otherwise have to be done by hand, I simply did a search and replace for the methods in question, which then made them have references to (typically) "Faction faction" that no longer exists. So I made it exist, in the base class. Then used the IDE's rename function (F2) to rename that to AttachedFaction, and presto. Then just delete that temporary variable you made. You can do the same in your mods, if you wish.
    • The reason for gettting rid of Faction as a property on this methods is that it creates ambiguity and the potential for errors. It's just cleaner not to have it.
  • At the moment, I have about 200 compiler errors in DeepInfo, and 2934 errors in visual studio, which is probably vaguely accurate. It's several thousand lines that need correcting or adjusting in some fashion. There's still over 240 references to SpecialFaction_[Whatever], and those will take a mixture of search and replace (when it's checking is or as again), and manual adjustments as I link up the base and deep infos of each faction in particular.
  • Based on the speed of things lately, and the somewhat unknown volume of code that has to be corrected, I think this will easily take me another 2-3 days. I worked through almost all of last weekend, and I hope not to do that again. The picture will become more clear tomorrow and the day after as I get further along.

• Okay, I couldn't quite leave it for the night. I got autosaves working there's a new AutosaveHandlerDeepInfo class in DeepInfo which replaces what used to be there.
  • Really, all the code is the same, but it just doesn't have any chance of going to a client, and isn't attached to any random factions. The translation of this was literally under five minutes, nice.

• Got a little over halfway through setting up quick-links between the DeepInfo objects and their BaseInfo counterparts. This includes setting up the DeepInfo objects as singletons where need be, for handy access.
  • As I was doing this, I also made various other corrections and shifts to code as I saw it, and in all the number of errors in DeepInfo are plummeting. There's still plenty there -- visual studio says 2400, which seems about correct at this point -- but the code is thinning out thanks to being better organized (data and data access in one place, logic in another place, and in consistent order and consistently organized in both).
  • I don't want to get ahead of myself, but after a few weeks of being overly optimistic on schedules (or is that just a terminal case for me, one could wonder), it actually looks like I was -- potentially, and majorly knock on wood -- overly pessimistic, this time.
  • THAT said, assuming I do come in ahead of what I was projecting, that's still just going to be leaving more time for me testing for runtime errors and fixing those. It's not like as soon as this compiles it's good to go. I've been coding for almost three weeks now on very long days and over weekends without testing any of it -- because I couldn't -- and if you're a programmer, you know how horrifying that is. All things considered I think I've been very clean with my work in this time, but I expect to find some major gaffes.
  • There's a vague, outside possibility I could have this compiling tomorrow. At the current pace I have observed, it's not outside of the realm of possibility. It's not hugely likely, either, but it's what I'm going to shoot for. I want those two days prior to the weekend for testing and fixing the runtime.

• Removed some probably-unused-and-unworking code for ships warping in and out over distances. As Badger notes, we have way cooler things to look at these days, and we haven't noticed this actually functioning for a few years now. It was taking up data space, though, and eating some CPU processing, so out it goes.

Moving Into DeepInfo, Part 1

• ModdableGameCommandExecution has been moved out of BaseInfo and into DeepInfo, after seeing a bit more how it is used and thinking about it.
  • Some nice comments have also been put in there, with two TEACHING_MOMENT entries: one about what the moddable game command things do, and another about how to get data from the host to clients in general.

• Thinking of trying to minimize multiplayer bandwidth, I decided to make a second Stage3, which does happen in BaseInfo and thus on clients.
  • We want to be relatively careful what logic we put in there, to not make MORE desyncs. But for simple logic in particular -- like zombies attritioning -- it's going to be way more of a desync if this DOESN'T happen on the client.
  • There's a TEACHING_MOMENT note in there for one of the zombie factions.

• I also changed my mind and am removing DoPerSecondLogic_Stage1Clearing_OnMainThreadAndPartOfSim_HostOnly and DoPerSecondLogic_Stage2Aggregating_OnMainThreadAndPartOfSim_HostOnly from the DeepInfo.
  • We just don't need it, and I don't think any mod that would need that is structured properly. These methods are specifically design around working on the data that is held in BaseInfo, so we don't need a DeepInfo version.

• Somewhat related, Stage0 -- aka DoPerSecondLogic_Stage0Clearing_OnMainThreadAndPartOfSim_OncePerFactionTypeEvenForFactionsNotInGame_HostOnly -- has been removed from the game.
  • That was simply not something that was ever needed at this point. We have better ways of handling similar things, and the new Stage2A is part of that.
  • Specifically, also a lot of the double-buffered lists and similar make this completely unneeded.

• The remaining 16 factions (of of 37) have now been fully linked up from DeepInfo to BaseInfo. I was feeling really happy and motivated last night and got the first 19 out of the joy of it.

• Down to 141 direct compiler errors at this point, and 1782 visual studio errors (which is likely to be pretty close to accurate). There's still a chance I can knock the rest of this out today, but if not then tomorrow is pretty much a certainty.

• Added a new ExternalFactionBaseInfoRoot, which now sits under all of the BaseInfos for factions.
  • It was becoming really clear that there were some commonalities that needed to become shared, plus a few things that needed to be able to happen on a schedule that is centrally controlled (to avoid some mapgen errors I suddenly realized were possible thanks to a comment by StarKelp in one of his faction files).
  • First of all, we now are ensuring that DoRefreshFromSettings() and DoGeneralAggregationsPausedOrUnpaused() get called from mapgen.
  • Secondly, something that was just an optional convention that I had been using (that being DoRefreshFromSettings() itself) is now a required part of faction base info. It allows for great flexibility without overtaxing the processor.
  • Next, the base implementations for GetRaidDesirability() and GetRaidTraversalDifficulty() now have somewhere appropriate to live.
  • Lastly (for now), this allows us to implement a proper GetFireteamById() rather than just GetFireteamBaseById(), mirroring how faction code used to work.

• The call to link the faction externaldata into the factions, and also to call DoAnyInitializationImmediatelyAfterFactionAssigned(), has been moved around substantially.
  • The idea is that we want more of the data available as we're deserializing savegames, but when it comes to mapgen we also need to be after-all-factions-but-before-actually-doing-any-mapgen.
  • This should save quite a few headaches for me from after I get this compiling again (I would have had to make these order of operation changes in general).

• GetFireteamById() has been moved over into BaseInfo where it belongs.

• All of the pathfinding logic has been adjusted to be where it should or call the new locations of relevance.

• Added a new AIRelentlessAndBorderAggressionFactionBaseInfoRoot, which is a common ancestor now to AIRelentlessWaveFactionBaseInfo and AIBorderAggressionFactionBaseInfo, mirroring the setup we have in DeepInfo.

• Stage2 logic was still hanging around in DeepInfo for the following factions and has been migrated over (along with a bunch of other data):
  • Cross Planet Attacker AI subfaction.
  • AI Relentless And Border Aggression root.
  • AI reserves
  • six more to go, at the moment, but AI reserves took... a lot.

• The AI Reserves faction no longer has dual lists for "sim" and "lrp." It just calculates their data with double-buffered lists, instead.
  • This also includes a new teaching note, and some code that has been adjusted to exist in DeepInfo only, along with some guidelines on why exactly it works this way versus some other way.
  • In other words: how do you know which place a variable or collection should go? This lays it out, with the example of DeepstrikeEligiblePlanets as a guide.

• Added a new SortThreadsafeCopyOfDisplayList() to double buffered lists, along with an optional TryClearThreadsafeSortedCopyOfDisplayList().
  • See, the problem with the double buffered lists is that normally you sort them ONCE right before you finish constructing them and flipping them active.
  • However, after they are the display list, any number of threads might be reading from them, so to sort then would not be threadsafe in the slightest.
  • To work around that, this method allows any thread to sort a copy of the display list, in a perfectly threadsafe way.
  • This will create some extra data in RAM, very small amounts, which you can clear with TryClearThreadsafeSortedCopyOfDisplayList() if you feel like it. Or it will just be cleared the next time you call SortThreadsafeCopyOfDisplayList() on the same list from the same thread. Either way.
  • This is another addition that has been made thanks to AI Reserves and some of the complicated things they want to do. Their list of eligible planets gets sorted over and over based on distance from other planets they are thinking about, but this is all on a super low priority background LRP thread, so it's fine... except for the need for thread safety.
  • Yep there's another TEACHING_MOMENT in the code about this.

• Split out some of the waves logic into a new WavesHelper static class that variable factions can use. AI Reserves needed it.

• "Set the preferred AI faction" logic has been removed from the remaining locations where it existed, since that is now obsolete and handled as parent/child factions. There were over a dozen copies of the same lengthy code all over the place.
  • That then left me with 21 references to PreferredAIFactionIndex, which have also now been converted over.

• Four instances of DoOnFirstSightingOfFactionByPlayer needed adjusting, and have gotten it.

• Rather than being a static class, WavesHelper is now a singleton.
  • This allows us to extend that singleton in DeepInfo, and add DeployComposition into there where only DeepInfo can reach.
  • We use the same pattern for FactionUtilityMethods, if you're curious.

• Necromancer stage 2 and various data has been ported to BaseInfo properly.
  • Good grief this was complicated. It provides a couple of TEACHING_MOMENT items, too.
  • This also shows one of the examples of how we no longer need to precalculate a string (and hit the heap when we do), by instead changing a ToString() method to one that takes in an ArcenDoubleCharacterBuffer and writes to that on-demand.
    • A lot of other classes also do this exact thing now, but SkeletonVariantPercentagesByFleet_ToString() and similar provide a great example of it in action.

• The necromancer faction is mostly working again in general, but it has uncovered a bit of a problem in my logic with hacking in particular.
  • Right now, hacks are only scheduled to run on the host, but there's a lot of info about "can we start a hack" and such that the client needs to be able to see. I'll have to deal with that.
  • The necromaner sacrifice unit hack in particular is also problematic, because it executes on all clients and the host (well, or just on the host), but is using local-to-one-machine data (selection info). That's going to require some refactoring on my part to figure out how to handle cases like this.
  • I have a similar problem coming up with the sub-types for the sentinels, warden, hunter, and PG. They essentially need some of their stuff in BaseInfo, and some other bits in DeepInfo. I mentioned this last week actually as a concern point that I didn't know how to solve at the time. At the moment, I have a fairly decent idea of a way to handle it, but I need to examine the code a bit more first.
  • I'm down to 2009 errors at this point. Not sure if I'll finish tomorrow or not, at this rate. Which really bites, but it's the nature of the beast, I guess. At the moment I'm fixing some things that were always going to work incorrectly in multiplayer anyhow, so it's not just architecture for the sake of nothing.

• Moving hacking back into BaseInfo. I'll build in hooks into DeepInfo as needed, but the vast majority of this code really needs to be run as part of the sim.
  • One thing that is interesting about this whole revamp process is that the way that data is available to the UI or where it needs to be is really becoming increasingly clearer, even to me. I knew in pretty great detail what I wanted to do, but there have still been many cases where my intuition was wrong. This is one of them.

• The necromancer "sacrifice" code and hacks has all been removed. That was going to be complicated, but Badger noted that was deprecated in favor of the Rift hacks. This saves a lot of time! But still helped to identify a problem with how I was handling hacking in general in DeepInfo, so was useful.

• Now that hacking has been moved back to BaseInfo, there were several hundred errors that needed to be fixed (they were going to need to be dealt with anyway, but now they prevent BaseInfo from compiling again until I fix them).
  • As part of this, moved around a lot of things that are not really worth detailing, and fixed a number of references that were still old-style. It's hundreds of lines of code, but there's nothing really surprising in this batch worth reporting in detail.

• TerritoryDisputeTeamFactionDeepInfo has had its Stage2 merged into BaseInfo properly. This was needed for some of the hacking stuff, but also just required in general.

• Added ConstructionContainsKey() to DoubleBufferedDictionary, and ConstructionHasKey() to DoubleBufferedArcenSparseLookup.
  • This is needed to support some of the territory dispute logic.
  • Ah, and it turns out we also need versions for the display dictionary, so those were also added.
  • And it turns out that we also need DisplayContains() and ConstructionContains() on the DoubleBufferedList, so that's also in now!

• Added a new table called ExternalDeepLink, which lets us define singletons in BaseInfo, which can then be initialized via xml, and then assign their reference back to BaseInfo from DeepInfo.
  • Essentially, this answers the question "how can a modder or faction designer add arbitrary code calls from BaseInfo to DeepInfo without modifying Core code?
  • I actually already use this sort of pattern all over the codebase, and have since 2018. Lots of external classes register themselves with more information than the raw table that loads them has access to.
    • This is kind of complicated to explain in a written note like this, so it will be much better to show examples. I'll have some teaching moments in there, too.
  • The first and simplest example is ExoGalacticDeepLinkRoot / ExoGalacticDeepLink.
    • ExoGalacticDeepLinkRoot is an abstract class defined in BaseInfo. It defines what methods it has, and has a static Instance.
    • Xml states to load ExoGalacticDeepLink, which is in DeepInfo. In its constructor it assigns itself to the Instance variable on ExoGalacticDeepLinkRoot.
    • After this, any BaseInfo code can just call ExoGalacticDeepLinkRoot.Instance.YourMethodHere(), and that will be directed straight to DeepInfo and do whatever it is you want it to.
  • Ultimately it's a really simple pattern, but it's just hard to explain in text like this.

• We now also have a DysonUtilityMethodsDeepLink, for supporting a number of hacks.

• Fixed an old and undiscovered pooling issue with ZenithArchitravePerUnitBaseInfo, where it would get into a funky state after being hacked.

• There's also now a NecromancerDeepLink. And a MacrophageDeepLink.
  • These are stupidly simple to add (just a couple of lines of code), and allow for full control of how you can talk from BaseInfo to DeepInfo. Really pleased with how this turned out.

• Stopping for the night. At this point, all of the hacking stuff should work now, and is back in BaseInfo but using DeepLink calls where needed to call into DeepInfo-only areas.
  • There are presently 92 compiler errors reported for DeepInfo, which is getting pleasantly low, and last I saw the visual studio error count it was just under 2000 now.
  • This is definitely a tense schedule, but there's a substantial chance I could get this done tomorrow -- to the point where I can start running it and finding all the runtime errors on Friday, anyhow.
  • We'll see how it goes. Honestly there is deceptively high numbers of things that need attention or conversion, but the new DeepLink tool solves a HUGE number of the problems that I was otherwise having trouble figuring out a graceful solution to.

Moving Into DeepInfo, Part 2

• BuildTerritoryCollection() was called a variety of times in the territory dispute faction classes.
  • This basically just made a lookup, by faction, of what territory each faction controlled. This sort of thing used to be required, but now we can access the factions so easily and they all have double-buffered territory lists already. So we can, happily, skip this and use data we already have.
  • Getting rid of this sort of thing (which previously existed because of either threading concerns or because of difficulty finding the correct other classes' data) is a major win of the new framework. Saves a lot of CPU, and in many cases also some minor RAM GC churn.
  • Note that I'm not particularly calling out the territory dispute faction anymore than I've meant to call out any other faction. They're all coded very well in the limitations of the old architecture. But at various points, various factions illustrate different benefits of the new architecture.
  • Added a new DoForAllTeams() method to TerritoryDisputeParentFactionBaseInfo that also helps offset the need for this method, and methods like BuildTeamCollection().
  • GetColonyShipTargets() and similar are also now a great example of how to reduce CPU usage and GC churn by using threadstatic lists. These are thus also threadsafe methods. Please see the teaching_moment related to threadstatic if you want more info.

• The "Stage4 Advanced Allegiance Code" that I added a while back to support the territory dispute faction has now been moved into BaseInfo.

• Added an extension method for DoFor() to Dictionary, giving it that handy same method that ArcenSparseLookup contains.

• The territory dispute factions (both the parent and the subfaction) are now fully ported over, including its notifications.
  • This was a very interesting use case, because it's very modern and written very well, but also has some unusual stuff like the sub-teams. It's probably pretty obvious, but the architecture has been continually expanding as I see what each faction author wrote in detail and how I need to improve my central tools for them to do their same work in less code.
  • There is a "StarKelp_Question: hey, I was unclear on this logic." note in there at one point, which StarKelp you should feel free to either correct or confirm whenever you like.

• Down to 63 compiler errors in DeepInfo, and 1818 errors reported errors in visual studio. I'm determined to get through this today so that it all at least compiles!

• Stage2 logic has now been moved over for the last four factions that needed that pulled out of DeepInfo and into BaseInfo:
  • Dark Spire (also got rid of some ForUI variants of lists, and some list variants "for all factions" when there can only be one dark spire).
    • This also has a great example of some of the old-style codebase stuff, where it had to have a count of spire cities for itself. That's been removed, and it just asks the spire, which is able to say that really clearly now.
    • Also removed the pointless FactionIndex, which was just referring to itself it turned out; that was actually hard to be sure of for a bit. This faction is _really_ old. At one point we probably legitimately needed that to know how the external data connected in to its faction.
  • Templars (also got rid of some ForUI variants of lists).
    • A teaching_moment has been added in there, along with a bit of logic shifting on how rifts are spawned, essentially working around a cross-threading error that would happen but rarely, and it explains how to notice it. The double-buffered lists make it a lot more obvious, since you should not be calling add to a construction list outside of the construction process, and you should NEVER be adding to a display list.
  • Zenith Architrave (removed many ForUI variants, also made serialization of territory and planets ever taken direct now rather than first sent to IDx).
    • Also got it so that their BaseInfo now properly cleans itself up before pooling. (When it was originally coded, it wasn't meant to be pooled.)
    • The way that territory is calculated, and a number of other things beyond that, have been simplified a lot, and hopefully not mangled in the process. Before there were some _severe_ order of operations constraints across multiple passes that these revisions aim to make just happen in order, 1, 2, 3, much simpler.
    • Gah, this one was really a cluster. This is new top prize for hardest faction to transition over. I don't think I messed anything up, but Badger will have to review it, sigh. The logic is probably easier to follow now, though, on the bright side.
  • Nanocaust (okay, this one was shockingly easy).
    • Got rid of yet a bit more really super old deprecated code, and repointed a few variables and it is what it is.

• Moved over more notififications logic from DeepInfo to BaseInfo, which is one of those really lovely things that lets us have functionality that works properly in multiplayer:
  • 2 from Dark Spire.
  • 2 from ZA.
  • There are still 10 more I need to grab.

• Many various other redirections from old code spots to new, nothing terribly exciting with most of that.

• Dark Spire faction is now fully compiler-error-free.
  • Same with nanocaust and Zenith Architrave, and a bunch of other factions.
  • We're down to 44 compilter errors now, and 1594 in visual studio.
  • The Zenith Architrave and the Territory Dispute factions were both way more challenging than I had expected, and the largest faction of them all, Dark Zenith, still needs a lot of translaton work, as does the AI. We'll see what I can do...

• AI Reserves notifier moved over from Deep to Base.
  • The last bits of the AI Reserves DeepInfo errors are also resolved.

• Many various other random small improvements all over the place. This is going to last well into tomorrow, sigh.

Moving Into DeepInfo, Part 3

• Many various cleanups and fixes to the new format.

• Added a new DoFor on GamEntity_Squad that works on a List<>, like the one that works on ArcenLessLinkedList<>.

• Added a new HasSimStepRunSinceGameLoad on BaseInfo for each faction, which is set after Stage4 has run on each of them.
  • This is really useful, because Nanocaust had some stuff that it really didn't want to do on long-range-planning threads (it would make wrong choices) until at least one sim frame had elapsed.
  • While looking at that, it became really obvious that this is a problem that other factions would now share, if they didn't already (most already probably did).
  • So now the long range planning threads are blocked from running at all until the first sim-step pass has happened since the game was last loaded (savegame load, quickstart, game start, whatever).
  • To be clear, this is a delay of about 100ms, so it's not really noticeable. But it will make it so that the initial run of long-range-planning is never acting strange because of a lack of data.

• Nanocaust data is now properly cleaned up when it goes back into the pool.
  • Also removed yet more old frenzy variables that are not used since the advent of fireteams.

• The nanocaust now uses double buffered lists in place of the linked lists or other formats.
  • Also, the nanocaust has some cross-faction lists, along the lines of what the dyson spheres do.

• The way that the nanocaust aggregates its data has been converted into the modern format. Well, the NEW modern format, but also the modern-since-DLC1 format, where Stage2 is used for aggregations.
  • Whoof, some of its logic in general, such as for attrition or for constructor upgrades, has been moved to baseinfo and converted a bit.
    • I had to go back and check the older versions of the code on some of this, because the constructor upgrade logic applies to non-constructors and that is how it was coded. I'm leaving it because that seems to be what has been the historical function even if that's not what the code names would seem to indicate was intended.
  • The nanocaust faction is now fully up and ready to run (or at least compile). Ultimately the surgery was not too bad, and things should be more clear than they were before when looking at that code.

• Moved the remaining notifications over from DeepInfo to BaseInfo, along with the code for generating them:
  • 5 from Fallen Spire.
  • 2 from Dark Zenith.
  • 1 from Nomad Planets (that's the last of any compile errors with nomad planets gone!)
  • 1 from Zenith Miners (and with about six other corrections, all compile errors with the ZM are gone also!)
  • None from sappers, but some outer scaffolding that was prepping for them has been moved.
  • At this point, notifications should again work properly on multiplayer clients, which was one of the sticking points that led to this whole overhaul in the first place.

• HasSimStepRunSinceGameLoad has been renamed to HasPerSecondSimRunSinceGameLoad.
  • This is already what it did, but it's now correctly named. It's important to note that the long range planning threads won't start for an entire second, not just 100ms. This is of no gameplay significance from a player point of view, but it keeps the data and the factions happen.
  • Both marauders and HRF had nanocaust-style "have I loaded yet?" checks to enforce this sort of thing in an ad-hoc manner. As suspected, this affected a lot more than just the nanocaust.

• FireteamRegiments have been moved out of the BaseInfo and back into DeepInfo.
  • After consultation with Badger, it became clear that this is purely host-only information, and not something that would need to be seen on the ui in a non-debugging fashion. Therefore, into DeepInfo it goes, to avoid confusion.

• A number of things on the Marauders faction that had been keeping an index of planets now keep the actual planets since that is now possible. Makes for some added simplicity.

• After quite a number of changes, marauders now fully compile.

• The HRF now also compile, with the changes to make them use planets directly rather than planet indices.

• IPraetorianGuardTypeImplementation, IHunterFleetTypeImplementation, and IAIWardenTypeImplementation have all been removed.
  • This was a really funky way of setting things up in general, and Puffin had already made a todo in the code stating that we should move this into xml long ago. As he noted, it took him forever to find these balance levers.
  • In the more recent short term, this whole design just really wasn't fitting with the new setup anyhow, so it was doubly time to adjust this stuff.
  • AISubFactionDeepInfo now also has been dismantled, since this was just a vehicle for the three above.
  • Xml for all of the above are now present, and you can change a lot of things very easily now that were previously very hard to do! As a modder, this can majorly let you change how the hunter, warden or praetorians act.
  • A lot of the code that was tied to this whole set of structures needs to be reconnected, but I have to get some other things in place first.

• Lots of other random minor fixes and rearrangements. That got rid of the last of the low-number compiler-blocking errors... revealing a further 1446 errors. That's about what visual studio had been predicting. Woo...

• Got rid of the last 5 references to "is SpecialFaction_", down from hundreds initially.
  • That left 8 more references to "as SpecialFaction_", also down from hundreds initially. Those are also gone now.

• The AI type implementations have been moved back into BaseInfo rather than DeepInfo (sigh), and split out into multiple files so that they are easier to keep track of.
  • Technically this stuff is only used from DeepInfo, but it's a real pain to try to sequester it there, and there's nothing about the data of this that would break any of the client/host rules, so... fine. BaseInfo it is.
  • The ability to add per-sim-step or per-second logic on AI types has been removed, incidentally. That was never used.

• A bunch of the AI Sentinels DeepInfo methods were static for some reason, potentially because we thought we might want to cross-call from other locations or something.
  • Thankfully we were not actually doing that (that would have been a nightmare), so these have been made regular methods and are already private, so they have quicker access to the data of the AI they are dealing with.

• The way that AI spending ratios are assigned to budgets no longer generates a bunch of garbage for the GC to take care of.

• DoOnAnyDeathLogic_HostOnly on factions has been renamed to DoOnAnyDeathLogic_MyFactionUnitsOnly_HostOnly, because honestly it was not clear if that was how it works or not.
  • There's another method called DoOnAnyDeathLogic_FromCentralLoop_NotJustMyOwnShips_HostOnly which runs on any unit, so it's vaguely implied that the above one is only for its own ships if you see that, but it's not clear enough for my taste.
  • Because of that, there's a number of cases of indirection (that existed prior to this rework) that I'm able to simplify away. Just working more directly with object references makes things faster, and makes the code more readable.

• Fixed what seems to have been a bit of a glitch with macrophage not getting any metal income from them killing enemies in some circumstances -- maybe if they did not have a spire telium?

• Zenith Trader now fully compiles.

• The way that the macrophage keeps track of its entities has been updated to a more modern interpretation.

• Added a new UniqueNameForFactionToAvoidThreadConflicts, which defines intentional collisions for "don't run these faction's LRP threads at the same time."
  • This allows us to ensure thread safety even amongst factions with inheritance chains, like the macrophage, lone wanderers, dark zenith, and so on.
  • We had something kinda like this in the past, but it wasn't as good and didn't cross types like this. It had made it so that rare LRP cross-threading issues were possible, that now are not.

• The macrophage code now compiles, and is a real showcase of how a complicated (but understandable) faction should work in the new code framework.
  • There are a number of teaching moments, and a number of threading things that get explored here.
  • There are also two "BADER_TODO / STARKELP_TODO" entries with questions where I was curious about the logic that was pre-existing, and woudl like more clarity myself.
  • This one is a great one to look at to understand the new format better, albeit with something that is on the more middle-complex side

• At this point, I'm down to 1124 compiler errors. This is taking MUCH longer than expected, as everything seems to have with this process, but it's worth it.
  • There are some things related to hunter, warden, and PG that are just commented out right now, which I'll be re-enabling later as I get their patterns fixed up to hook them back in.

Moving Into DeepInfo, Part 4

• Got rid of the visual studio solutions that had the base game and dlc xml in themselves separately. Instead, combined both into a single solution that has the separate projects from each of them in it. This makes searching and replacing a lot easier, and I don't know why I never did this before.

• The death effects have been fixed up in DeepInfo:
  • Zombificiation of all sorts.
  • Metabolization and Necromancy were already fine.
  • Nanocaustation.

• The AI Sentinels DeepInfo class is now down to 5 errors (from hundreds), and uses its new structure to the fullest. It should be much easier to read, and it doesn't have to do subfaction lookups and such nearly so often.
  • The remaining 5 errors are related to parts of subfactions that I have temporarily disabled, so I need to get those subfactions all fixed up and then those 5 will go away.
  • Total error count is now down to 1090.

• Set up a new AISubFactionCoreDataRoot to live in BaseInfo under the sub-data for Warden, Hunter, and PG, since those all have a certain type of data involved.
  • This makes it so that they can now reach back out and talk about their parent faction with ease, rather than having to have a faction passed in and then fumble around to find it on the faction like was the case in the old architecture.
  • This actually pulls in pretty much all of the code that used to live in IndependentFleet, and translates it into the new format. It is vastly more brief code now, and I put like five different TEACHING_MOMENTS in there.
    • In particular there's some great teaching moments about the threading model of the entire game, and the System.Threading.Interchanged class and how to use that, and cross-thread-safety concerns in general.
  • This section of code (and the three subfactions that ride upon it) was the one I was most worried about coming into the weekend, so I'm glad to see it looking so well.

• Warden logic that was previously on IAIWardenTypeImplementation has now been moved to AIWardenCoreData.
  • ReceiveDonation is a really straightforward copy-paste from where it used to live. Now it's in BaseInfo.
  • The rest of those other things now have migrated over to live on top of the framework that AISubFactionCoreDataRoot has built. It's wonderfully simple by comparison to what it used to be, and the code is also much closer together and doesn't have an interface in between the definition and the implementation.

• Added a new MapgenDeepLink, which lets me handle things like seeding the base-oriented warden secret ninja bases. See how handy this pattern is? It literally can be used for anything.

• The "HunterFleetHelper", which I had created only for a little time, has gone away. This is what was on IHunterFleetTypeImplementation at one point.
  • The translation across for this was a ten minute job, after spending hours getting the general framework up to make this and warden and such actually work. Nice.

• The Praetorian data went even faster than that, excellent.

• The SetSpecialTargetPlanet command has been removed from the game, because it existed just to send data to the client which was basically host-only (and which is set in a different threadsafe manner now from the host).
  • There's several teaching moments in the code related to this, but essentially it was a pointless game command that just increased complexity a bit without actually solving any issue in the way multiplayer and the faction logic actually turned out.
  • Speaking of, in this check-in the CurrentTargetPlanetIndex moved from being a core thing on every external faction's BaseInfo, to only being a thing that the hunter, warden, and Praetorian share. Nothing else uses it.

• The UpdateFireteams() method on factions has been removed, because it was possible for someone to unknowingly just directly set NumFireteams.
  • Numfireteams is now a property whose setter does the logic that UpdateFireteams() used to do.

• The UpdateFireteamSizes game command has also been removed.
  • On the surface, this might seem a bit like bad form for multiplayer. We are changing something sim-affecting, after all... kinda.
  • But the fireteam logic all pretty much just runs on the host, and -- again -- the faction data is transmitted very frequently to the client, anyway.
  • It's more economical in all senses to just directly set the NumFireteams on the faction directly from the external code that used to call the gamecommand.

• Those last errors in the AI sentinels faction code that I mentioned earlier today are now gone, and the hunter faction code is now also error-free.
  • Praetorian faction code is now zero errors, and so is the warden.

• Also, while I was at it, in the xml for the warden I took their default multipliers for how they gauge enemy strength and their own strength and moved those from being 1.5x enemy, 1.0 self, to instead being 1.0 enemy, 1.1 self.
  • This will make them considerably more willing to help out, even on planets where they may be outmatched. We've had off and on comments for a long time about the Warden being a bit too passive, and this will change that. If it's too much, we can tweak the xml, or make variants, or whatever else.
  • Since this is controlled by xml now, end users can experiment with it with just a text editor, or modders can make their own versions of the warden who act differently, etc. No code required.

• Stopping for the evening and my total remaining compiler error count is now 935. That should be pretty accurate, and is the first time we've been sub-1000 errors since starting this whole process a long while back. At the height it was over 7000.
  • I still have some pretty heft factions to deal with (Fallen Spire, Dark Zenith, Scourge, and Sappers are the big ones, with Dyson Sphere also clocking in a lot of errors right now as well). A few other factions have a handful of errors (Migrant Fleets, Astro Trains, etc), and some mapgen code also needs a few fixes. But it's definitely feeling like the downhill slope now.
  • The AI cluster of factions was the thing I was most worried about working on, and I commented about my worry about the Subfaction interfaces a couple of weeks ago when I ran into those and remembered them (they were my doing, I believe, but I had forgotten about them). As of today, that part is all done, which is a giant weight off!

• Fixed up how the AI Start Sizes for AI empires work in mapgen now that the code for getting their settings has been adjusted.

• The other handful of compiler errors in mapgen are also now fixed.

• A handful of errors in various factions that were otherwise okay are now fixed, leaving mainly the big factions to deal with.
  • Also two small errors in TargetListPlanning.
  • And nine in EntitySimLogicImplementation_DeepInfo.
  • Six from FireteamUtility.
  • Four from FireteamExtensionMethods.
  • Two in SpecialFactionPlanning.
  • One in ExternalFactionDeepInfoRoot.

• Okay, now actually it's time to deal with some Fireteam change that are required, because they give funky results on the clients.
  • TargetPlanetIdx is gone, just leaving the actual TargetPlanet. The only reason for TargetPlanetIdx was because of the order of serialization in the past.
  • Same deal with TargetID, LurkPlanetIdx,
  • Also, DoEarlyCleanupWhenGoingBackIntoPool() and InitializeToDefaults() have had their code merged to make them not duplicate one another.
  • Made a few of the serialized variables that are stored here more efficient, too, since there are a lot of fireteams being transmitted around.

• FireteamRequiredTarget has been made more efficient to serialize, and on the CPU. It requires fewer lookups during runtime, and has the data it needs on demand.
  • In order to fully make this work for the AgainstSquad, that now uses a LazyLoadSquadWrapper, but for the factions and such that's already handled.
  • This in turn prevents us from having to cache some information on the Fireteam itself that was just kind of copies of this.
    • Those fields on Fireteam are now instead properties that wrapper the FireteamRequiredTarget. This will actually work properly on MP clients without having to have them run UpdateNonSerializedFields(), which was a problem up until now.

• Whew, okay, fireteams are really problematic. They have a lot of host-only stuff that gets fed into the UI, and it's really hard to tell the parts apart.
  • I've improved things quite a bit so far, but there's nothing there that will keep the problem from recurring at the moment (it's a case of easy to shoot yourself in the foot when adding functionality to fireteams, right now).
  • I'm currently checking in what I have on this front so far (with 904 errors left, incidentally), and then seeing what I can do to actually split this out so that not all of the Fireteams functionality exists in BaseInfo. There are parts that frankly just need to exist in DeepInfo so that we don't do this foot-shooting thing.
  • Right now it's mainly a matter of the MP clients are going to be incredibly spotty in what they show properly in their tooltips about fireteams; the actual gameplay of them acting properly is fine for MP and SP.

• The hunter and warden types now properly load again from the settings in the lobby. This was just one of those things I had not ported over.
  • Vassal status now also loads properly, and centrally for all factions in an optional way, which is nice.

• The starting fleets and battlestations and such for human empires now load properly again. This again was a matter of porting over the existing stuff.

• And finally, the last of the lobby fields, the trim colors for AI subfactions, are also now ported over.
  • There's a slight chance they may be a bit janky either in the lobby or after, but I think it will work this way, and it's more efficient. I have a couple of lines commented out that will fix it (but do more CPU work) if it's actually a problem.

• Fixed a few more things here and there, Devourer and some exogalactic bits.

• The Scourge faction has been mostly updated, except for the parts where the LRP thread does some things that would violate the normal double-buffering pattern. I'll deal with those when I am feeling more fresh.
  • For now, got rid of several hundred errors in the Scourge for compilation, and now there are 4 left. One of those is related to some fireteams changes I still need to make (I have a great idea on how to do it now, but I'd like everything else cleaned up first so I can see my work more easily).
  • Overall error count now down to 613. That's really it for me tonight, now.

• The Sappers collection on the Sappers faction is now DoubleBufferedConcurrentList. All the others were fine without the concurrency.
  • Oops, no, actually FloweredCrystals also needed that.
  • Okay, the sappers are fully compiling and ready to go. That's us down to 73 compiler errors.

• Hjarnum on Dark Zenith is now DoubleBufferedConcurrentList. It seems to be the only one there that needs that pattern!
  • Sigh. That said, it needs to be sorted, which is incompatible. So we also now have a SortedHjarnum that we build at the same time and sort. This is apparently only used in the UI, which is handy at least.
  • Almost all the errors are out of the Dark Zenith faction now, but there are a few things that I still need to look at in the morning when I am more fresh. I was working until 1:30am last night, and then back at it at about 8:30am this morning, so I'm not the most fresh.
  • That said, the total compiler error count is now... 6.
  • Tomorrow morning I'll need to sort out these last issues with the DZ, and then get the fireteams split I have planned done (it shouldn't be too bad, knock on wood), and then correct a couple of vis-dll errors that should take under 30 minutes, and then I'm on to actually... runtime testing!
  • This whole process has taken enormously longer than I ever expected, but the end is finally in sight. If I don't have at least a compiling version tomorrow, then something is very very wrong. Honestly if there's not one by lunch I'll be shocked.

Moving Into DeepInfo, Part 5

• Removed the ConcurrentDictionaryExtensions class, and merged that into the general DictionaryExtensions class.
  • Added a new DoFor() extension to ConcurrentDictionary via extension, the same as we had previously added that to Dictionary<>.

• Astro Trains had a few enough number of errors in its file -- 63 -- to actually have them shown properly in visual studio.
  • Oof, but its BaseInfo was not using double buffered lists, and the way that it was aggregating train guards was... fraught. So let's fix all that up, and also serialize less while we're in the process.
    • AstroTrainsPerTrainBaseInfo now has a pair of double buffered collections, which is the first time we've used that on per-unit data, but it's actually a great way to store things in the new framework. I haven't needed this before now, but it's a very useful form to use.
    • We also no longer have DeployedGuardsPerTrainSim on the BaseInfo. That's now being handled on the actual trains, rather than via an indirect lookup. This solves a lot of code complexity and chances for orphaned data. It also uses less CPU to calculate in the new format, though in the old format of the game it would have been debatable.
    • This actually has turned into a rather fascinating study of nested data. It's quick and easy to understand and a pattern to be emulated. This is actually kind of a great showpiece for the power of the new system.
    • This turned out to be SO useful that I've put a teaching moment in there.
    • Improved the description appenders for trains to be a bit more informative now that they are able to be.
  • Okay, back to the astro trains deepinfo file, how many errors do we have now? 71, okay.
    • Well, we can safely get rid of DeployedGuardsPerTrainLRP, StationsLRP, and TrainDepotsLRP.
    • TimeLastDepotSpawned was not consistently called, so it has been moved into SpawnNewTrainDepot. Same with AstroTrainsPerDepotBaseInfo being created and attached to it.
    • initializeDepotData has been removed and made part of the depot creation process.
  • The guards code was really confusing because things were not named very clearly. The guards on the train are now called StoredGuardsInsideThisTrain and DeployedGuardsOfThisTrain, so that it's clear that's how this works.
  • The way that guards are stored for finding lost guards, and the way that they attrition, has been moved into BaseInfo (and thus runs on the client as well as the host, which is good), and is also a lot simpler to follow now. Same general logic, but it won't miss anything by accident.
  • StoredGuardsInsideThisTrain was an ArcenSparseLookup, which was a cross-threading problem waiting to happen, but it can't use the double-buffered style collections because that's not what it is (it's actual stored data). So this is now a ConcurrentDictionary, and there's a teaching moment in there explaining why.

• Silly me, I thought that Astro Trains were going to be a very easy faction. But it turns out there was quite a lot going on there that was really really out of date for the codebase, so... here we are.
  • Down to 551 compiler errors left.

• Okay, let's see what's going on in the Fallen Spire deep infor class. I see 85 errors at the moment, mainly so low because I've been in here before quite a bit, but not with my full attention.
  • Most of that melted like butter in under five minutes. There was a funky thing with the journal logging that wasn't very clear, but that was something that the svn history let me clear up and then document pretty easily.
  • Transceiver needed to be converted into a DoubleBufferedValue. From the look of this class, this was before I added that specific item (in the distant memory of last week).
  • Oh... wow, what an error on my part. Coding while tired, I guess. This was also one of the earlier factions I started the conversion on, and then I had set it aside. All the BaseInfo Stage2 stuff was in DoFactionGeneralAggregationsPausedOrUnpaused instead, which runs WAY too frequently. Talk about a performance drain. Glad that code never released.
    • I also untangled the double-buffered clears and switches from the logic that fills them, which is what all the other classes I've done more recently were like. It's amazing how that takes this from unreadable to really nicely legible.
  • Okay, even so with all the oddness in BaseInfo, this gets me down to two errors remaining in the DeepInfo class... and they are the same sort of nastiness that I've seen in several classes when working with my nice new double-buffered lists. They just... aren't sufficient for a lot of the code that exists in actual DeepInfo classes. I'm particularly unhappy with the butchering that I did to Zenith Architrave; I think I may revert back out that entire class to before my last pass on it, and redo it more nicely. But before that, I need a double-buffered collection that supports concurrency; one that is used for spire cities and debris (thus solving my last two errors here), and which then is also used on Zenith Architrave (thus returning its logic to what it used to be). What I had expected to be an edge case actually turned out to be a really common case, so new data collection it is.
  • I'm down to 468 errors, but I'm going to pause on forward-progress for a bit, add my new data structure, and then undo all of the likely damage I did to Zenith Architrave.

• Added a new Engine_Universal.DEFAULT_CONCURRENCY_LEVEL, which is set to 4. This should be passed into every ConcurrentDictionary in the program as the default concurrency level (and then choose yourself what the starting capacity for that dictionary should be).
  • Why does this matter? https://blog.getpaint.net/2017/06/30/concurrentdictionary-allocates-a-lot/
  • Engine_Universal.DEFAULT_CONCURRENCY_LEVEL is the default level of concurrency that we are likely to need in AI War 2, based on my designs for threading. If more threads than this are accessing a ConcurrentDictionary at once, then performance will suffer slightly, but not much.
  • The default concurrency level is actually 4x the logical processor count, which is... insanely too much, and wastes RAM and performance when it comes to high-CPU computers. I only have 8 phyical cores with hyperthreading for 16 logical processors, but that means that it is creating (for me) literally 16x more RAM usage and locks than are needed. In the future, as computers advance, the design of this program is such that it will still use plenty of processors, but not on an individual dictionary basis.
  • So we'd start seeing MAJOR performance degredation and memory spikes if 128 logical processors were ever a commonplace thing, etc. Now we won't. Yay futureproofing. And honestly, it helps for performance NOW, too.

• Added a new DoubleBufferedConcurrentList. This is a lot like DoubleBufferedList, but less efficient in some ways, and more powerful.
  • This one also needs to support the display list taking additions and working as a working list for background threads. This tends to be a case where something says "oh hey, let me look at what is there, and also let me add this one," and then the rest of that background thread needs to have access to thing that was added RIGHT NOW.
    • For example, this sort of thing could happen on Stage3 or in LRP or both, who knows or cares. Either of those is still happening while the UI is using the same DisplayList as the others.
  • There's no way to handle that without using a concurrent list, and you might think that would be enough to avoid flicker in the UI, but it's not. The flicker would come when the construction list is torn down and then rebuilt, since that is often on a (different) background thread.
    • So we need to always have persistent lists, and support ad-hoc additions to the display one as we go, while also supporting the common cycle of construct-clear, construct, flip-to-display.
  • Sidebar: Internally, we are using a ConcurrentDictionary to power this. What... the heck?
    • Well, there is no "list" class in the .NET Concurrent collections, because the idea of an ordered list is antithetical to cross-thread ad-hoc addition and removal. They have ConcurrentBag and ConcurrentQueue and a few other things. ConcurrentQueue is vaguely ordered, but the main point is that it supports the foreach operator, and also supports direct removal and additions. Throughout the codebase, I have preferred ConcurrentQueue to ConcurrentBag because of profiling work that various other programmers have done, demonstrating that Microsoft's implementation of ConcurrentQueue is slightly more performant. That said, neither of those classes -- nor ConcurrentStack -- support the Contains operation. There isn't any equivalent of a ConcurrentHashSet, either. The only collection that really supports all of what we want to do is ConcurrentDictionary, which has some performance drawbacks, but not too severe to be worth it.
    • If you CAN use DoubleBufferedList<>, then of course use that instead. That thing flies. Start with DoubleBufferedList<>, and if you find you need to graduate to DoubleBufferedConcurrentList<>, then do so.
  • Implementation notes:
    • You will notice that there is no GetDisplayList() like you are used to seeing all over the place with DoubleBufferedList<>. Instead, you must use DorFor on it to do the equivalent of a foreach loop (which you can break out of as-needed).
    • There IS a Contains method, but there is a (very very) low chance that it will return false when the answer is really true.
      • Because of that, there is an int TimesToRecheckIfSeemsLikeNotThere = 0 parameter to this method. If you REALLY need an accurate Contains, you can set that to something larger than 0, but be aware that has a peformance impact that is equal to calling the Contains methods however many extra times you say.
    • Since this is a dictionary under the hood, you can't put the same item in the list more than once. Or rather, if you do, they will just collapse down to the unique set.
      • This is actually useful, since typically adding something repeatedly is an error anyhow. We specifically have some things like AddPlanetIfNotAlreadyInList(), but you do NOT have to do that. That's implicit in this structure. For goodness say, definitely do NOT call "if not contains X, then add X," because you just wasted some CPU cycles for nothing in that case. Just add it.
    • Since we are specifically wanting to be able to modify the display list on the fly (that's why this variant of DoubleBufferedList<> exists at all!), there is of course also a new AddToDisplayList() method.
    • As a consequence of this being a concurrent collection, you can forget about any sorting methods. The normal DoubleBufferedList<> supports sorting, but this version does not. You'll have to feed the results of a DoFor operation into some local working list and then sort that if you really must combine sorting with this sort of list. But in general, I haven't seen any cases where that would be relevant.
    • You may have noticed that none of the DoubleBuffered collections suport IEnumerable (so you can't do foreach on them). This class is no different. The reason is because that would be a highly ambiguous call. Someone might do a foreach while thinking they are doing list construction, or they might do it while thinking they are looking at the display results. Lack of clarity like this is very bad, because a second programmer following behind you can't even tell with ease if you made a mistake or are doing what you mean to. So instead we have the DoFor variants.

• Went ahead and added a Display_DoFor onto DoubleBufferedList<>, so that it has the same style of functionality as the concurrent one from that angle.

• Added a new Display_GetRandomItem() on both DoubleBufferedList<> (just to save code, and it's very efficient), and to DoubleBufferedConcurrentList<> (this is not very efficient at all, but it works; and in most cases of relevance in our codebase, these lists are so small that it's plenty efficient ENOUGH).
  • Normally you can't index into a dictionary in the way we would want to in order to get a random return value, but our code would often need to do this on the LRP in particular with DoubleBufferedConcurrentLists, so now it's possible.

• For Astro Trains, they are now the pilot for DoubleBufferedConcurrentList, with TrainDepots and TrainsEnRoute now using them.
  • This was definitely a major diversion off to the side, but this keeps the code for the end product of the game as close as possible to what it was in the past, and thus lowers our risk of new logic errors or just incomprehensible code. It's time for me to completely redo the travesty of what I visited upon the Zenith Architrave a few days ago, now that I have this tool.

• Reverted the Zenith Architrave to svn revision 14224. Ugh I hate undoing work, but that was really a hash. This brings me back up from 468 errors to now 539 errors.
  • My goal this time is to use the double buffered concurrent lists to accomplish keeping the logic of the ZA intact, rather than trying to rewrite it and hash it up like I did on Thursday night. I had noted even in my checkin notes on Thursday that it was a hash, which is never what you want to see in your checkin notes.

• Fixed all of the non-controversial errors with the Zenith Architrave, which took only 20 minutes by my reckoning (I'm getting much faster!). I had temporary working variables to quiet certain errors related to the central lookups that are now going to need to change.
  • I am first checking in this version that has all those changes in place before I start changing the bits that need the new collection types. With the temporary variables back out of there, I'm back up to 75 errors from this file, so that's quite a lot related to them.
  • Also worth noting is that I'm not messing with the LRP collections at all. This class is complicated enough that they may as well just keep doing what they do. I don't really want Badger to have to come in here and review this code like I had asked him to before, and at this point I don't see a reason why he should have to.
  • 543 errors in total for now.

• Okay, got the next batch of ZA fixes in place. All of them except the WarpingInSpawners and the Castra are able to use the normal DoubleBufferedList. That's easy enough.
  • If I were to extend this to the LRP variables, which I think I did before, then that would cause some need to use the concurrent version more. But that would be a net negative in terms of performance, so I'll just leave LRP doing its own thing, giving more room for the sim and UI to share work/benefits with one another.
  • At this point there are 17 remaining ZA errors, and those will be solved by me converting the Castra and WarpingInSpawners to the concurrent double buffered list.
  • This time around, the flow of logic is preserved exactly as Badger wrote it, and there's no longer a need for a code review of this faction.

• WarpingInSpawners and Castra are both now DoubleBufferedConcurrentLists.
  • WarpingInSpawners previously was sorting itself after the initial creation of its list, but that's no longer a possibility with the new collection format.
    • The logic that was relying on this sorted list was kind of complicated, so it was easiest just to implement a secondary SortedWarpingInSpawners that we feed from the concurent list and then sort like we did before.
  • That's a wrap on ZA. They are now error-free, and no longer in need of a code review any more than any other part of the codebase now is.
  • Back down to 468 errors.

• Clearing "two" errors on the fallen spire took 35 minutes and generated tons of other things that had to be converted. But nevertheless, the fallen spire is now error-free:
  • Spire Cities and Spire Debris are now using a DoubleBufferedConcurrentList.
    • However, because spire cities in particular ALSO need to be sorted half the time (sigh), there is now a SortedSpireCities that is built for mostly UI purposes.

• GetScourgeStateForDisplay now includes information on fortresses. Previously it just had armories and spawners.

• The three kinds of "warping in" collections on scourge now all use DoubleBufferedConcurrentList. This was about a 10 minute job for "three errors."
  • Other than one fireteam-related error which I am intentionally leaving to the last (it's a structural thing that I need to be able to see in isolation to fix), Scourge are now error-free.
  • Total error count now 463.

• Fixed right about 60 errors that were remaining in the migrant fleets faction code (22 were obscured from the compiler until I fixed the ForEach they were in.
  • This was jolly easy and pleasant, nice. 425 errors remaining.

• Fixed what was reportedly originally 38 antagonized dyson sphere errors, but wow that was kind of a mess in general. They're all fixed up now, and probably for the first time ever we're not mixing up data of the antagonizer and the dyson spheres themselves, which makes the whole thing a lot easier to follow. Previously they had kinda-sorta shared an external data object, but not fully, and that made it a bit confusing to untangle.
  • Anyhow, I happened to also notice that AntagonizedDysonDescriptionAppender is not actually used on dyson spheres at any point from what I can tell. Maybe this was deprecated on purpose, but it would be functional if someone wants to attach that to a dyson sphere that's being antagonized.
  • Error total is now 394. Other than the fireteam stuff that will be at the very last (and some vis-layer stuff that is incredibly minor), this is down to mostly the Dark Zenith and the Sappers. Oh, and apparently the Dyson Sphere proper has 63 errors, too, I now see.

• Well, that was again pleasantly straightforward: dyson sphere faction is now error free. New compiler error count is 332. Time for the last two big guys.

• Decided to make the safe changes first, and hit low-hanging fruit, in case I make another mistake again like I did with the Zenith Architrave.
  • Sappers are down from 119 errors to 45, and what remains to be done with the rest of them is a lot more clear.
  • Dark Zenith are down from 211 errors to 71. It looks like Sappers and Hjarnum will need to be concurrent, but none of the other lists, which is excellent. Additionally, a couple of methods need some adjustment based on working lists, but that's not too bad. All told, this faction was less intimidating than it had seemed, thanks to judicious search and replace. Zenith Architrave was the more challenging one structurally, even though Dark Zenith is more complicated in general.
  • Total compiler errors are now down to 117, which is... insane! I was happy just to get to three digits today, after being in the high four digits for weeks. Now we're almost down to two digits!

Hunting Runtime Errors, Part 1

• The last of the non-fireteam compiler errors are now fixed. Time to get the last fireteams bits fixed up and that will be that!

• The text-export portions of fireteams are being merged back into the main file of Fireteams.
  • A new FireteamHonoraryDeepInfo object has been created, which exists as an immutable sub-part of every fireteam. On MP clients, this sub-part is ignored and unfilled.
  • There are detailed teaching moments that explain the thought process behind this, and how to mimic it for other data if you ever need to (as a main developer or a modder).
  • PreferredSpeed has been moved to be a serialized field, so that it can be shown on the UI on clients and hosts.
  • Removed some string construction that was happening overly-frequently on fireteams even on the host/singleplayer cases. It now only happens if you're actually looking at the debug menu in question.
  • For the first time since it has existed, our DeepInfo dll (AIWarExternalDeepProcessingCode) actually compiles!

• The 23 errors from the external vis dll are also all now fixed.
  • Running our complete build scripts to verify that everything compiles... yep! We are officially on to runtime errors now. Time to start up the game for the first time in three weeks (time for that to be possible for the first time in three weeks, internally).

• Fixed a bug in GetIsDescendentOfInterface() that was causing false positives.

• Corrected GameEntityTypeData to not require BaseInfo or DeepInfo for units. Most of them will have neither, and so far no units have needed DeepInfo.

• Fixed a couple of xml typos that were preventing proper parsing.

• For the sake of clarity, IArcenExternalClassBase has been renamed to IArcenExternalClassRoot.
  • We don't want this to get confused with BaseInfo, because that's not what it is (but it was sort of what it looked like).
  • ArcenCachedExternalTypeBase is now ArcenCachedExternalTypeRoot for the same reason.'

• Verified that my logic is working for checking if BaseInfo and DeepInfo (true DeepInfo, not honorary) are living together in the same dll.
  • As I had kind of expected, it's tripping on my own code at the moment, which mostly means I have some things miscategorized.
  • This exception checking is here to prevent accidental mixing of this data that is supposed to be separate, including dlls referencing dlls that they should not have access to.
    • Essentially this sets up a boundary where it tells programmers (modders, later, but for the moment myself) "hey, you're either arranging this wrong or you've tagged things wrong, but either way fix it because bad things would happen in multiplayer if you do not."

• Now that I have verified that the above works, a whole bunch of debugging output that was related to that has been turned back off.

• Fixed all the map definitions still pointing to the wrong dll from their xml.
  • Ditto for the TargetEvaluators.
  • Ditto the AIDefensePlacers.
  • And DeathEffects.
  • And GuardpostAndCommandPlacers.

• Fixed an issue where the xml tag multiplier_to_all_fleet_caps="0.2" was still being used on a variety of outguard units, which I don't think had any relevance for that at all.
  • Also some sappers turrets, and a ZA turret, etc.

• Then there was an unexpected multiplier_to_all_fleet_caps="1.34" on:
  • Golden Wasp Drones, which made their cap 536 instead of 400. Let's make that 540 just to look tidier.
  • Queen Bee Drone, normal and Dark Zenith versions, ditto.

• Milestone: I can get to the main menu without any errors at all! It took just over 1 hour from the point of being able to compile at all to get to that point. Now I'm going to break for lunch, but it will be interesting to see how long it takes to get the rest working. It's nice to see the main menu again!

• Fixed an issue with the Source not being properly linked to each of the four PrototypeObjectForGlobalLobbyClearingOnly objects for the new externaldata, which led to an immediate exception when trying to start a game.

• Fixed an one-line code error that broke the new sub-faction linkages for the AI. They needed to be after a loop, not inside it, and it was one curly brace too early.

• Fixed another one-line error that mixed up relentless wave linkages from DeepInfo to BaseInfo.

• Fixed an exception where Outguard mistakenly thought they were supposed to have an Intensity (they are not).

• Fixed a minor issue with order of operations with the lobby partial-map-generations not linking up parentages properly.
  • This was technically working by design, but this was from a few weeks ago before I knew the shape of things to come. It needed to be changed a bit in light of the final design.

• Fixed another issue with Allegieances, where every faction was expected to have one (but many invisible ones don't, in particular).

• Milestone: I can now open the lobby without errors. I took a long lunch and ran an errand, so this was about 1 hour of work since I got things working to the main menu.

• Fixed up how the faction external data was linked to its "attached object" (faction or squad).
  • Previously it did it way too late, and therefore some errors were resulting.

• Started tracking the way that faction external data is linked to its attached faction, because I am getting some strange nullref exceptions.
  • By assigning cross-thread-safe unique IDs, I can see that the version of the faction data that we have in the lobby for use is very different from the one that previously had itself assigned. Curious!
  • Stuff like this is central and breaks every faction, and is just part of the framework itself being tested out for the first time. This is the sort of thing I was hoping to see, and not a thousand little per-faction errors. I may yet see a tide of those later, but at any rate this current part is just a normal part of unit testing after coding for three weeks without being able to test any of it.
  • At this point, I can see that we have a minor memory leak of factions again, without having to even look the memory dumps and diff them (which I will do, later, when all of this is working better). Nice to find this out sooner than later.

• BaseInfo and DeepInfo on factions and squads are now carefully-managed and sent back to their pools more aggressively.
  • ...That did nothing, unfortunately, to help my actual problem. It does make the code easier to read, and it's hard to say beyond that.

• Fixed things up so that BaseInfo and DeepInfo actually get put back into their pools when they are supposed to.
  • This helps a little bit, but there's still a central leak with my pooling framework, which is a surprising thing to learn at this point. I thought I had established that as solid a while back.
  • Then again, back when I was establishing it as solid, I had noticed that it would increase to about twice what I expected, and then stop. So it was using extra memory, but not infinite endless memory. This may be related to what I'm seeing now, which I am going to fix because it causes problems for me on several levels.

• Added a new GetItemsWaitingInPoolRightNow() and GetItemsInTheForeverList() onto TimeBasedList, to help me see what's going on as the factions are pulling external data out of their pools (right now they create too many objects, and I want to know why; after that I'll still have to deal with the problem of improper item teardown, but one thing at at a time).

• Discovered that a variety of pools were not using their original entity (the "creator entity") for actual data storage. For the larger objects, where there are just one or two per type, this is very wasteful. Read: factions.
  • In the past, I've always just really cared about things like entity orders, or squads, or whatever else -- and the difference of literally one object is super stupidly negligible. For a faction, it's an extra 100% usage of RAM, by comparison.
  • These pool types now include the creator entity as a normal object to use, just to remove this confusion and slight waste (it can still do its job as an object factory no matter what else is happening): TimeBasedPool<>, BetweenMapGenPool<>, and ConcurrentPool<>. I think that's all the major ones of relevance.

• Immediately after that, started having some exceptions in Recalculate_PreSystemRecalculate() on initial game load (before getting to the main menu), which is curious but not completely out of character with some OLD issues I've seen off and on sine 2018. Let's get that fixed.
  • For now, Recalculate_PreSystemRecalculate is being instrumented.
  • Based on this, fixed an issue with AIWar2GalaxySettingTable.GetFauxFIntValueFromSettingByName_DefaultOne_DuringGame() and all similar methods where it would give an error if those were called prior to the game being fully initialized.
  • Based on this, also fixed an issue with GetResourceProductionBeforeAnyBonuses() where it would also error if called prior to the world being initialized.
  • Same with GetFrameSizeMultiplierAsDoubleNonSim().
  • Did a further bunch of hardening of a bunch of other places, several dozen; mostly windows.
  • Worth noting that none of this code was changed during this refactor. This is an old issue that is only triggered by certain rare circumstances that -- for whatever reason -- become common during this pooling shift.
    • On the whole, this is not the sort of issue I was expecting to chase today at all (something unrelated to the work of the last three weeks), but there's always the chance that this has been secretly breaking for a rare subset of players, so let's make sure it works.

• I think that the world is actually being set to null for some reason, possibly because of the world itself being pooled in a funky way...
  • So let's fix THAT: The three pool types noted above all now have a CreatorEntryIsAddedToThePool parameter. Now it's up to the calling code to decide how to handle it.
  • True on everything except: PlayerAccount, World (the first entry in that is a "dummy world!"), ConfigurationForFaction, EntitySystem, SpeedGroup, PlayerAccount_AIW2, and World_AIW2.
  • The game now loads just fine to the main menu again.
  • Also, the lobby faction tab no longer spams errors about a missing faction link! So I DID solve the problem I was originally chasing, but just in a really roundabout way.
    • This made me suspect that entering the lobby multiple times in a row, or otherwise serially regenerating map types would be a problem. But nope, those work great also.
    • Adding multiple AI factions works great, and adding extra human factions also works.
    • However, after adding two extra AI factions and then another human faction, I did get an error where the AI subfactions were not linked to their parent AI faction in the case of all three AIs. So I'll have to look into that, after I can duplicate it more.
    • Also, reading last-lobby-settings seems to be broken at the moment, but that's fine. Just adding that to my list.
    • Starting a new game from the lobby is also broken, but this is probably unrelated to anything I've done thus far; this is the first time I've tried clicking that button today, and the place it errors looks like a sensible place for a one-line typo.
  • So, I'm going to call this current line of investigation complete, and move on to the new list of errors that I have accrued to look into. (As kind of an aside, I was fully expecting to have serialization be broken in the new version, as that's really hard to do completely correctly en-masse. I built in tools a couple of years ago that makes it trivial to find those errors, generally, but it's not my top priority right this second).

• Milestone: I can click around the lobby, add factions and such, with very few errors (though not zero yet).

• The game now does a handy dump of all the faction basics when it runs into parentage issues.
  • See, back when I was originally designing the parentage stuff a few weeks ago, I had not wanted the subfactions clogging up the lobby for very good reason -- it's the reason I'm having some exceptions now, in fact; old stale sub-factions get left around. Having the subfactions earlier than later is actually better overall, I just need to have them clean themselves up between mapgens now, basically.

• Using the above, the game now properly removes and adds all the various factions and subfactions. No more errors on that front, or mis-parenting of factions, or duplicate factions, using any routes that I can tell.
  • I do see there is still an issue with null attachedfactions sometimes on factions added via the lobby, but hitting the map regeneration button fixes it. Still needs to be fixed to not happen at all.

• Made mapgen a bit quieter in the log when it's not having problems.

• Fixed an exception that would happen when it tried to add nomad planets if you did not have the nomad faction enabled.

• Fixed a minor issue where the game had a default of "Normal" for the AI Layout field, when that was not an option. The default is now Small Clusters, as intended, and "Normal" is also taken as meaning "Small Clusters" just to be on the safe side.

• Fixed another exception that would happen in the escape menu when trying to describe factions. It was being too strict and expecting all factions to have a NumberToSeed, which of course they do not.

• Updated the version number to 3.700, so that the game can actually TRY to load saves it makes (it won't try for anything lower than 3.7).

• Milestone: Game is nominally playable. Currently in single-player, you are not properly assigned to the first human faction, but you can start the game anyway. Once you are in, all the menus seem to work and you can assign yourself to that faction. Ships obey orders, fighting works, factions seem to be doing what they are supposed to, and there are no errors so far. Bear in mind this is just with the human and the AI and the auto-added factions, not with anything from DLCs yet, but that's still pretty awesome.

• Fixed an issue with "Super Easy Mode" on the Fallen Spire (debug mode) not being a proper bool. I had converted all the others, but this one was erroring because I missed that part of the xml.

• Fixed a couple of serialization issues with savegames, but I have a few more to fix still. Doesn't look like anything super major, knock on wood.
  • There is less data being sent now, partially because I am reading things from the Config for factions directly rather than sending/storing that multiple times, and that now fully works.
  • From the look of things, the "last stored settings" for the lobby also work again. Have not tested quickstarts yet.

• One of the benefits of breaking all the savegames is that I can change the order of things around to be more efficient/clear with how we load things. Previously I used that in order to put all of the ExternalData at the very end of the savegame, which makes deserialization a lot easier there.
  • Now I am switching fleets and factions, so that factions are deserialized first. In order to do some more-efficient conditional deserialization of fleets (sending less data based on the type of faction in question), we need to have the factions loaded first. Now we do!
  • As part of this, we no longer have to have a loadingFactionIndex on fleets. We can just directly load the faction.

• Similarly, I have now split it so that the planets and the galaxy are now loaded before fleets. However, the PlanetFactions (and all the units inside them) are not loaded until AFTER fleets.
  • This split makes even more things easier to load, which is really nice. When I originally designed the order of these things back in 2018 or so, the full scope of what their needs would be was not remotely clear.
  • We now no longer have to do the loadingPlanetIndex on fleets, and we can also set the PlanetFaction immediately, too.
  • PlanetFaction PFaction on fleets has been removed, as it was not actually ever directly needed and it was slow to calculate.

• On fleets, we no longer have a Centerpiece_DeserID. Also, Centerpiece is no longer a GameEntity_Squad -- it's now a LazyLoadSquadWrapper.

• Thanks to the changes above, we no longer need FinishPostFactionAndPlanetDeserializationLinkages() to run after fleets are deserialized. This is good for speed in general of loading savegames, but also for multiplayer CPU processing on the clients as they get fleet data.

• Because of the order of operations change (factions loading before fleets), I can no longer serialize the LooseFleet ID for a faction directly on the faction itself -- the fleets don't exist yet.
  • I _could_ go in and make some indiretion (store the fleetID, then later link it), but there is a much better solution.
  • We now just store a serialized IsLooseFleetForFaction bool on the fleet itself, and since it loads after the faction, it can just set itself as the LooseFleet with ease.
    • Fun fact: this takes one bit of data, rather than 32 bits. Also it's faster to do, because we don't need to do a lookup at any point; we already have a reference to the faction object.

• Necromancer data on the fleets and factions now only serializes for the necromancer faction.
  • Similarly, a bunch of player-specific faction data only serializes for players in general.

• PlanetFactionBooleanFlags are also now only sent for player planetfactions.

• In general cleaned up a few fields in serialization code, and put in a few more markers for telling where we are in the logs. These don't affect non-logged transmissions and saves.

• Fixed up some more code in the Templars, just correcting a few things that I mised and thus were not quite up to the latest standard.
  • There's still an exception in MarkUpCastles, but I'll deal with that tomorrow.

• There's also still at least a few issues with deserialization of savegames and network data, but it's getting closer to being done, and is more efficient, and is easier code to read. So that's progress!

Hunting Runtime Errors, Part 2

• Fixed up most of the core problems with savegame/network serialization that were just a consequence of me changing the up the format so much. I also... changed up the format even more, a bit. Just adding bits of extra efficiency on things that are sent more frequently. As one random example, for instance, we now use 38 bits to send the offset-coordinates of each step in a unit's guard-path rather than using 64 bits to do the same.
  • There's still something up with the serialization, but now it's to the point where it is at the external info and it's hitting an infinite loop on load. That shouldn't take too long to track down. From the serialization logs, it looks like most of the data that I would expect to be in there is actually being saved, which is nice. Right now the deserilization log itself cuts off in the middle of writing, however, which I can tell means there's a buffer that hasn't fully been flushed by the time it hits the place where the infinite loop is. So I'm going to work on making that buffer flush faster, which will take me right to the problem code.
  • At the moment, these are still the sort of structural problems that you fix-once-and-then-not-again, so I'm happy about that so far.

• The serialization logger now flushes its buffers to disk much more rapidly, so that if there is a problem that interrupts the program or throws it into a loop, the output isn't behind.

• Fixed two silly lines of code that were accidentally infinite-recursive, and caused deserialization of faction external data to not work. It didn't actually take that long to find, but there have been a lot of other secondary distractions and things today.

• Fixed the last of the deserialization errors that were new structural problems introduced in The Great Refactor.

• Milestone: The game can now save savegames, and then load them.
  • Reminder 1: older savegames from prior to The Great Refactor will not work in the new format. Trust me, it is absolutely worth it for all the improvements throwing out that compatibility has lent us. Much as I hate losing access to all those older saves.
  • Reminder 2: until we hit version 4.0, hopefully within a few weeks, I still reserve the right to continually break NEW savegames as well. There are some structural things with the Fallen Spire in particular that I need to redo, and that was too big of a thing to tack on DURING The Great Refactor. So those will be coming right on the tail of it.

• The debug setting "Write Deserialization Data Sizes" is no longer considered an advanced field.
  • When this option is on (it was added a little after DLC1 came out), it gives a detailed breakdown in WorldDeserializationDataSizes.txt of how much room each part of a savegame takes up, and what kind of data it is.
  • This has been adjusted today a fair bit to make sure it really captures things properly. Looking at the data, it's looking good! I see a few places where I can wring out a bit more efficiency, but honestly that was something that already happened a lot during The Great Refactor, and it shows.

• Discovered that I had put in the sort of funky working-list stuff in Templars for Rifts, not just in Zenith Architrave for their stuff.
  • Fixed up Templars to instead use the much-nicer and also much-less-likely-to-error DoubleBufferedConcurrentList.

• Fixed an exception in MarkUpCastles for Templars, where essentially the data for the castles was not attached onto them at the time of creation, but instead was randomly attached to them during this method.
  • There will likely be other places like this in the code, from all of us different programmers, and you can handle it in one of two different ways in any given case:
    • Option A: When you first create some entity (squad) that will later need some BaseInfo or DeepInfo of a certain point... just populate it then!
      • That's what the game mostly assumes you will do, but you also don't have to.
      • It's possible you might have some units that only need extra data SOME of the time, and in that case, that's a great reason to wait and not add it now, for instance.
    • Option B: When you need to do some sort of operation on an entity (squad), then just call CreateExternalBaseInfo<YourType>( "YourType" ), rather than calling GetExternalBaseInfoAs<YourType>() or TryGetExternalBaseInfoAs<YourType>().
      • This is also an acceptable choice, and it has the benefit of working really well with old savegames (which we have zero of at the moment, but in the future if you're trying to deal with code additions to things that now need data that didn't have it before, then this is the way to do it).
      • This approach does have the downside, though, that the Create() method has a bit more overhead than the Get() or TryGet() methods (Get and TryGet are equal in overhead, but the former errors when data is missing and the latter does not). The extra overhead of the Create() call is really just a matter of a table lookup, which means a dictionary call, and then one additional string comparison check. So it's not exactly massively slow or something.

• Fixed a longstanding minor bug in how battles were calculated for the galaxy map, with them overestimating the scale of your allies. This is for the little ripple thingies on planets.
  • This was found while fixing a really pointless warning message that could be sent to the log now when loading savegames post-Refactor.

• Now that my pooling works really well and extends fully into the externaldata realm, there are inevitably going to be some things that require some adjustment just because of having been missed and not previously being expected to pool.
  • One of those was pathfinders, which would throw exceptions after the second time you loaded a world. That's now fixed, and at the moment I'm not seeing anything else that has this sort of issue (Pathfinders were always kind of an odd case, like EntitySystems are an odd case, because those two categories of data retain attached to their parent object but still pool).

• When you are clicking the randomize button on the map screen, on solo play it now has a delay of only 0.2 seconds rather than 1 second. At the host in MP, only 1 second rather than 3. As a client, only 1.5 seconds instead of 5.
  • The idea is to prevent a bunch of regeneration requests from queuing up, which used to be a problem, but this all runs faster now and it was just annoyingly slow to let you click more, before.

• Fixed an issue (not sure how long this has been a thing) where when you hit reset to defaults in the lobby, it was not actually resetting the map tab items.
  • Specifically, it would reset the map type, but not any sub-options for the map type, and not the number of planets. This probably is new as of the current beta series, but not The Great Refactor, if I had to guess.

• Removed ControlledByPlayerAccounts_DuringGame from on the Faction class.
  • This was duplicative of something in the ConfigurationForFaction class, so we could already get that information a better way.

• Also removed/hid ControlledByPlayerAccounts_OnlyUseInLobby on the ConfigurationForFaction class, instead making this a private field that is only something that can be touched via properties and methods on its class:
  • bool IsFactionControlledByAnyPlayer tells you that.
  • bool IsFactionControlledByLocalPlayer tells you THAT.
  • int CountOfPlayersFactionControllingFaction gives you that answer.
  • byte GetPKIDOfControllingPlayerAtIndex( int Index ) has the expected answers.
  • PlayerAccount GetAccountOfControllingPlayerAtIndex( int Index ) again tells you the expected bit.
  • int GetPKIDOfFirstControllingPlayer() is predictable also.
  • PlayerAccount GetAccountOfFirstControllingPlayerOrNull(), ditto.
  • bool GetDoesPlayerControlThisFaction( byte PlayerPrimaryKeyID ) answers as you'd figure.
  • bool GetDoesPlayerControlThisFaction( PlayerAccount Player ) does too.
  • void RemovePlayerAccountFromControllingThisFaction( byte PlayerPrimaryKeyID ) is clear.
  • void ClearAllPlayerAccountsFromControllingThisFaction() is also.
  • void SetPlayerAccountInControlOfThisFaction( byte PlayerPrimaryKeyID ) is the last one, which does as you would expect.
    • Crucially, when things don't work as we expect, we can now insturment any and all of these and figure out WHY.
  • string GetNameOfPlayerOwnersOrEmptyString() is the actual last one, since the Faction class can't really do this anymore with much ease.
  • void SetPlayerAccountsToViewStartingPlanetAfterMapgen() is another "actual last" one, this time pulling in from mapgen.

• Okay... I just cannot get the faction ownership thing to work fully correctly. We have a lobby version and a long-term version, and the long-term version is right but the lobby version is wrong even after I set it to be right. I... my brain hurts. It's been hours staring at this.
  • I think that a further simplification to the lobby config settings is incoming. I will have to delay and do a release tomorrow. In the meantime I have further broken tutorials, but those will need some slight structural adjustments in general.
  • The only other really big bug that concerns me before a release is the factions external data not always having attached factions properly, but I think that can also be solved by simplifying the central configuration. This will need to be tomorrow, as I am likely to make matters worse rather than better if I try to fix it tonight.

Hunting Runtime Errors, Part 3

• Okay! After mulling this a lot last night, I have figured out that I hate the current design for our lobby configuration. It's my design, so I can say that. ;)
  • This is a design from 2018 or something along those lines, and it was in response to severe problems that I had when working with the lobby in AI War Classic in 2008-2009. It DOES solve the problem, but it created new ones. It's not the simplest-possible solution.
  • So here's the problem, and let's see if you think of the new solution before I tell you what it is. When framed like this, it's deceptively simple, but remember I'm framing it for you in a way that points toward the answer. Don't think too poorly of my past self, heh.
    • 1. Map generation takes time. How much time varies by the map and the computer, but it's a lot longer than a frame. It's something like 200ms at best, and 1.5 seconds on the super horrible end of things.
    • 2. During map generation, we need to have a copy of all the settings you chose from the lobby prior to that generation call, and those settings need to not change. So for instance, how many planets you asked for, any other map-related bits, anything do with factions that affects the map, etc.
    • 3. We obviously want to run map generation on a background worker thread, because we're not going have the game seem to freeze for 200ms - 1.5s while you wait for mapgen. For one that's just really annoying and bad form, but it also would halt all network communications during that time.
    • 4. Given that, players will then be trying to mess with the lobby settings, potentially, while the map is generating. 200ms is not enough time to do much, but 1.5s is.
      • One solution to this is to temporarily make all of the lobby controls read-only, which was my solution in AI War Classic, but that's really messy and requires a lot of extra code (a bit of code for every stinking field). It's very annoying.
        • Another limitation to THAT solution is that the client doesn't know that the host is in the process of map generation when using that solution, and so we can't lock all their controls. In AI War Classic, this is why most of the controls for map generation could not be edited on clients.
      • The second solution to this is to just have two copies of the data. There's one copy for display, which mapgen doesn't use. It's what players see, and they can edit it from anywhere at any time. When it's time to submit it to mapgen, it makes a copy of itself and we're off to the races.
        • This is the solution that AI War 2 has been using since about 2018, but the problem is -- well, now we've got two copies of the data. It's possible for them to get out of sync, which is what has been frustrating me for several hours yesterday.
        • The good news is that once you solve that problem once, it's solved until the next time you make a structural change to the entire program, but it's complicated enough that it will deter you from making structural changes to the program because of what a huge pain this is.
  • All right, so what are some potential solutions that are better than the above? This is what I've been thinking about since last night.
    • Firstly, rather than having a lobby-copy and a permanent-copy (which is really what I've been doing all this time in AI War 2), maybe the sole copy of the config just makes a copy of itself that gets passed around in mapgen.
      • I don't like this, though, for a couple of reasons. One, it's entirely possible for a mapgen programmer to still go reach the central data by accident, and thus have a threading error or worse. Two, it requires adjusting how all of the mapgen code passes data around. And three, it is a minor pooling nightmare, since we would want to make sure that all of this gets pooled properly, even when mapgen fails or exits early, etc. There are "valid" mapgen failure cases, such as "there's not enough room on this size of map for all the factions you chose," for instance.
    • Okay, when that's out... hey, you know what? Our mapgen code really is more like 200ms to 400ms at the moment. How much clicking CAN a player do during that time? One or two at most, generally.
      • This is the solution I'm going with: the UI is still responsive and listens to players during mapgen, but no GameCommands get processed during mapgen. They all get processed right after, instead. Problem solved, and a single copy of configuration now.
      • Background: when players click around and make changes in the lobby, they don't directly change data. Instead, all of these things are generating GameCommands, which are what then actually change the data. There's normally a 100ms lag on this, because networking framework and sim cycle.
      • The leap from a 100ms lag to 200-1500ms of lag in the event that a new map is actively generating is really not far at all. And realistically it's typically more like 200-400ms, as noted. This lets players continue to have a responsive UI, and I expect in almost all cases they won't even notice the difference at all. It keeps threading issues away from mapgen, lets mapgen continue to work as before, and removes the complexity of having two copies of data.

• Milestone able to properly start a game, and reset a game, with all of the proper settings and things working as expected. Editing faction and galaxy option data in the lobby and in-game both work as expected.
  • New problems: not put in charge of any faction after loading savegame; savegames from yesterday broken; quickstarts broken.
  • Reminder to self: tutorials still broken, check to see if there are still issues with random faction external datas in lobby not being properly attached to a faction (this is unknown, it might have been fixed by the work from this morning).
  • Observation: the lobby feels more responsive to me, but it's hard to be sure if that's just me imagining it.

• Fixed a single line in the deserialization code that was causing quickstarts to not load properly.
  • They still don't load fully properly after they initialize, but the data I need gets off disk now, and so the only thing left with them is making sure their faction boot up properly. The fact that the data on disk is good for actually getting that started now is the important thing.

• Milestone: Quickstarts once again fully work, including loading them into the lobby or starting them directly. They properly strip out and recreate the bits that need that sort of thing, and all seems to be well from what I can tell.

• Fixed a bug where savegames did not remember who was in control of each faction on load.

• Fixed a load exception for the astro trains faction, and another for tamed macrophage. Just one-liner translation type issues.
  • Got another one for the dark spire fixed.

• Fixed an issue that made it so that when you added a new faction via the lobby, it did not actually fully generate things such that they had their needed attached info.

• Fixed another one-liner that was the vassal flag for units being read wrong.

• Fixed another one-liner that made it so that the second time astro trains were refreshed on a game load, they'd freak out.

• WorkThreadOnly_StartNewGame and WorkThreadOnly_GenerateNewGame and DoFinalBitsAfterMapGenerationOrFixingSavegame are all now instrumented to report errors better.

• Discovered that Random Factions are broken at the moment, but that's okay; I will get that fixed after this build.

• Fixed an error in Dark Spire where HandleTerminiiAndEpistyles assumed that DarkZenithPerUnitBaseInfo would be initialized, but it was not always (this sort of thing was me being hopeful, but my fallback was explained yesterday).

• Fixed a pair of errors in the enraged and tamed macrophage trying to update their allegiances like the regular macrophage.

• Fixed a one-line bug (this was not new, I just happened to find it) where Neinzul Sappers would not properly get their intensity if you put them on a minor faction team.

• It looks like random factions actually DO work, after a fashion, but are invisible in the lobby for some reason. So more will need to go into looking at that.

• Added the following new option to the debug settings in personal settings, in Debug, Detailed Logs: Log Fleet Setups For Custom ARS Types
  • When this is on, during startup it will log to the debug log any information about AddedToFleet_MinorFactionStrikecraft and AddedToFleet_MinorFactionFrigate. If you are getting an error with some custom faction structure that lets you hack it like an ARS, this will let you see what is wrong with your xml.
  • As you may be able to guess, there's a data problem here at the moment. It looks related to the dark spire, and I have no idea if this is new, or if this is an old problem that actually just gets reported now. But it sure spams the game once it starts, at the moment.

• Added a new IsSvikari onto the dark zenith core BaseInfo. This is used to let the Svikari not try to find the SpawningOptions field.

• I had corrected the internal name of the Dark Spire to be DarkSpire instead of "Dark Spire", but I had not fully corrected that throughout the rest of the xml. That was leading to the flood of exceptions, which I could now easily find thanks to the new debug setting that I put in. Those are all now fixed.
  • Additionally, I'm not sure if I made the same fix to Fallen Spire or not, but either way the random factions had a wrong value for them, and I corrected that also now, too.

• Discovered that the Zenith Architrave also do not always proactively initialize their units' external data. Fixed so that it works around that.
  • Same for macrophage teliums. Same fix applied. I suspect there will be a variety of this sort of thing showing up for factions this week.
  • And some more for the Dark Zenith.
  • The old structure for the game kind of encouraged lazy-loading of externaldata, so... here we are. We can either fix them to proactively attach said data (that would be nice, but has not too much point), or we can set it up to lazy-load as-needed here, too (which is what I'm doing now).
    • Despite the pejorative-sounding name, there are sometimes benefits to lazy loading (sometimes EXTREME benefits), so this isn't really a criticism of anyone. It's just an observational about differences in architectural assumptions.

• Fixed a minor linkage issue between the territory dispute subfactions and their parent. One-line of code type thing (most of these are one line of code fixes).

• Fixed a variety of conversions in the Dark Zenith resources converter table that were using the dark-zenith-specific variant rather than the DarkZenithFactionBaseInfoRoot that lives underneath both the dark zenith and the Svikari. This led to exceptions on the Svikari. Easily fixed with some search and replacing.

• Fixed a really silly issue with purging dead fireteam units, where I meant to check IF they were scourge, not assume they HAD to be scourge.

• Milestone: with over 20 factions added, I can now run the game for over twenty minutes without any errors popping up. They are fighting, and doing what they do, and etc. All of the errors thus far have been of the sort of one-line translation things that are fixed in about 10 seconds. Writing the change log note takes as long as finding the error in the code and fixing it combined.
  • Factions tested so far: AI and all of the auto-added factions, human empire, AI risk analyzers, astro trains, dark spire, dark zenith, dark zenith svikari (multiple at once), devourer golem (multiple at once), macrophage (multiple at once), marauders, nanocaust, nomad planets, territory dispute, zenith architrave (multiple), zenith miners, and zenith trader.
  • Not yet tested in that way: necromancer, fallen spire, HRF, migrant fleets, sappers, scourge, zenith dyson sphere.

• SphereNoLongerAntagonized() has been instrumented for better error reporting.
  • Fixed what seems to have been two translation errors on my part (the worst ones I've seen yet, but they were still quick to fix, knock on wood) where all dyson spheres were being treated as if they were antagonized, and constantly trying to revert to normal.

• Milestone: I have now run with all the other factions that I mentioned before, PLUS the necromancer, the fallen spire, the HRF, migrant fleets, sappers, scourge, and zenith dyson sphere. None of the others had any errors. That is every faction in the base game and all of the DLCs, even the unreleased one.
  • Please do note I didn't actually start the spire quests or anything like that, so there may be errors once we get into that more. It's hard to do a quick test of fallen spire. But at any rate, the factions in general there all lied for 15 minutes without any errors now.

• The faction utility methods to find either a human king or AI king are a lot more explicit now, and will only throw errors if you say they should. Additionally, if you ask them to throw an error, that error will be very visible. This eliminates any chance of invisible log spam that slows down your game, while keeping the flexibility to error if we really need to (we pretty much never need to in the game itself, but some mod might).

Mod Updates

• In general, expect all code mods to be broken for a bit. All xml mods should mostly be fine, but it depends on the mod.

• AMU:
  • More functions for the ManagedFleetHandler:
    • It now has a list of tasks to do, with their own on-failure and on-flagship-death finisher actions, and settings such as whether flagships are carriers, unload ships to attack enemies, etc, etc...
    • The OnSuccess/Failure/FlagshipDeath extra logic virtual functions have been renamed to OnEvery[something], and fire on every [something].
    • Additionally there now is a function for every entity removal that is triggered by the ManagedFleetHandler.
    • Tasks will now be archived and can be looped for perpetually repeating orders with the matching OnSuccess or OnSuccess_Fallback action.
    • The ManagedFleetHandler now has default settings for all tasks, so to reduce the hassle of setting up new tasks. This comes with a bunch of variants on the constructor to set up tasks easier.
  • Fixed one more reference to The Neinzul Abyss being DLC 4 instead of 3 in AMU.
  • The AMU_BattleshipBase entity type and all entities that inherit from it now use large health bars (visual change only).
• Devourer Chrysalis:
  • Updated to work with the above changes.
  • This hopefully fixes all remaining cases where the Devourer Hunters warp out but don't refund their strength as budget.
• NEW MOD! More Frigates:
  • Adds several starships from AI War Classic to 2 in frigate form.
  • Adds two new starting fleets composed solely of these frigates.
• NEW MOD! Super Saiyan Overlord:
  • This mod replaces the default overlord with a new one that is equipped with new weapons and receives a massive stat buff.
  • The Overlord Phase 2 also now sends raiding fleets against the player as long as it is engaged in combat with the player's units.
  • This mod is meant to be the ultimate test of your skills.
• More Starting Fleets:
  • Moved several fleets into their respective DLCs in order to allow players to use the mod even without said DLC (they will be missing the fleets that uses ships from the DLCs).
  • Reworked/Buffed/Nerfed several fleets and removed some that were too similar to other fleets.
• More Starting Fleets Plus:
  • Due to lack of demand and technical support, said mod will be disabled and sunsetted.
  • May return at a later date but currently is too similar to More Starting Fleets without bringing anything unique to the table.

Prior Release Notes

AI War 2: Finalizing Multiplayer