AI War 2:The Great Refactor

Known Issues

• Any bugs or requests should go to our mantis bugtracker
  • If you need to submit log files, those can generally be found under your PlayerData folder in the folder your game is installed in. The most relevant one is called ArcenDebugLog.txt. You can send us the whole thing, or just strip out relevant parts.
  • In rare cases, mainly if your entire game crashes (that almost never happens), we will need your unity player log. That gets overwritten the next time you run the game after a crash, unlike the other log. These can be found here:
    • Windows: C:\Users\username\AppData\LocalLow\Arcen Games, LLC\AIWar2\Player.log
    • macOS: ~/Library/Logs/Arcen Games, LLC/AIWar2/Player.log
    • Linux: ~/.config/unity3d/Arcen Games, LLC/AIWar2/Player.log

• Multiplayer is in public beta, as noted below. There is a detailed multiplayer guide that we are working on building up.

• Feel free to join discussions on discord!

What Does Multiplayer Beta Mean?

Please see this link for details on multiplayer. This wound up taking up too much space in this document, so all of the multiplayer-relevant bits have been moved to the other page.

What's this phase all about?

We hit a point, while working on the beta branch, where we realized we just needed to really rework a lot of the underlying code to make multiplayer and modding and so on be as bug-free and compatible as we wanted for the future. So this was supposed to be a digression that would take a week or two at most, but wound up taking closer to four. And there are then after-effects for a while, on top of that. But the good news is that it dramatically improves... performance, multiplayer correctness, how easy or hard the game is to code to, etc.

The goal of this phase is to finish up reworking all of the things that we need to, for maximum long-term enjoyment of the game. This also means getting multiplayer absolutely completely finished and working. After that, we'll be shifting back into work on DLC3 (The Neinzul Abyss), which will be the final DLC for the game. The various remaining kickstarter items will also be taken care of, plus some other bits that we just always wanted to add for the game, and after that we'll be moving on to other projects aside from bugfixes and balance tweaks and modder-support throughout 2022.

The target release for DLC3 is in January of 2022, and that is when we'll also have the AI War 2 Complete Edition. After that, it will just be fixing material defects, and supporting modders. We expect it to have a long life if AI War Classic is anything to go by. The game is in a really good state and only getting better, and doing a refactor of this scale near the end of the project development cycle is a sign of our commitment to that longevity. Even if you aren't giving us more money in the future for this title, we want you to be able to enjoy it for years and years.

3.771

(Not yet released -- we're still working on it!)

Expert Mode Updates

• Add some more logic for seeding Fuel Stations:
  • Make sure random fuel stations are at least 3 hops out from a human homeworld. (Balanced stations are at least 5 hops out)
  • Ensure that fuel stations seeded near homeworlds are not on the same planets as those seeded randomly.
  • Thanks to tom.prince for updating!

• Make sure Fuel settings use new defaults, rather than the initial values.
  • Thanks to tom.prince for updating!

• Add an option to make beachheading more expensive.
  • When enabled, turrets will be more expensive on hostile planets.
  • 10% more expensive per mark-level of the planet
  • 25% more expensive on planets adjacent to a human planet
  • 50% more epxensive on planets not adjacent to a human planet
  • Thanks to tom.prince for updating!

Mod Review

• Mod updates (how that we're exiting beta soon, we can't have a bunch of broken mods showing as available):

• Fixed up the civilian industries mod so that it can load to the main menu and even start a game, but at the moment it still has exceptions when the game is actually run.

• The Civilian Industries mod is fully back up and going! Big thanks to StarKelp for getting that fully finished adapting into the new framework.

• Mod statuses:
  • AMU has been down for a while, but a lot of the secondary functions of it have been added to be the main game by -NR-SirLimbo.
  • Kaizer's Marauders, and the Devourer Chrysalis, are super complicated and rely on AMU as well, so those have been down for a while.
  • The AI Core Shield Generators mod has been intentionally taken down after talking to the mod author a while back, as (with permission) that will soon be folded into the main game as a direct option).
  • The Reprocessors mod by CRZgatecrusher has been disabled for a while, since it will require many updates before it can work in the new framework.
  • The Macrophage Histiocytes mod by StarKelp has been disabled for a while, since it has a lot of updates required, and he was thinking of just building that into the main game after a discussion we've had.
  • Capturable Dreadnoughts by Zweihand is an xml-only mod and works just fine in the new version.
  • Exotic Ships by -NR-SirLimbo needs xml updates, and also relies on AMU, which is out of commission for now. So it has been disabled.
  • Frigate Focus by CRCGamer seems to work great, and is xml-only, so it's fine to use!
  • Micro Mod Collection by -NR-SirLimbo needed only small xml updates, so it has been updated and now seems to work great!
  • More Frigates by Zweihand needed only small xml updates, so it has been updated and now seems to work great!
  • More Starting Fleets by Zweihand is an xml-only mod and works just fine in the new version.
  • More System Defenders by CRCGamer needs more extensive xml updates than I can do right now, so it's been disabled until someone wants to update those.
  • Powerful Command Stations by ArnaudB needed only small xml updates, so it has been updated and now seems to work great!
  • Spire Railgun Shop by Lord of Nothing need too many xml updates, and is going to be wrapped into the main game (with permission) as modular spirecraft are introduced in the next few months. So it's disabled.
  • The three Strategic Rebalance mods by Zweihand all seem to work great!
  • Super Saiyan Overlord by Zweihand needs a lot more xml updates, and so for now has been disabled.
  • Tame Dark Spire by ArnaudB already seemed to work great, as it was just xml changes, so that's excellent!
  • Raising The Floor Multi AI Adjustment by ArnaudB already seemed to work great, as it was just xml changes, so that's excellent!
  • Supercharge Raiders by ArnaudB already seeemed to work great, as it was just xml changes, so that's excellent!
  • Out of the mods distributed with the game, I think that's a full accounting at the moment.

Beta 3.770 Bug Roasting

(Released December 20th, 2021)

• Add an option to the Fallen Spire that returns player-controlled Spire ships to their slower speed.
  • Thanks to tom.prince for adding.

• Update starting fuel values per Strategic Sage.
  • Thanks to tom.prince for updating.

Bugfixes

• Fixed several errors that could happen during wave generation, which were also causing waves to either not be launched or to be launched with zero strength.
  • Thanks to tom.prince for identifying the problem code, and to tom.prince, motai, UFO, dEAdOnE77, Puffin, and WanoU for reporting.

• Fixed another error that could happen if you were hovering of the wave tooltip right as the wave hit.
  • Thanks to dEAdOnE77 for reporting.

• Fixed spurious "The PrimaryKeyID [X] could not be found for the last 10 seconds, but it is not in the death registry. Where did it go!?" messages that could pop up when you had loaded a savegame and the unit that a ship had been targeting during the last save was already dead and thus not loaded into the new save. We only care about cases where the ship was literally alive and then disappeared.
  • Thanks to Puffin for the report.

• Put in some extra protections in DoPerSecondLogic_Stage2Aggregating_OnMainThreadAndPartOfSim_ClientAndHost against an exception that could happen after extended play with the macrophage. If anyone is still seeing such an error, then we'd need a save to reproduce it.
  • Thanks to dEAdOnE77 for the report.

• Finally fixed the tech upgrade preview window going nuts with counts scrolling up and down over time, or just counting infinitely up. This was a tricky bug that has been in the beta for a few months, but it's been increasingly hard to trigger.
  • Thanks to motai for the save that finally let me resolve it, and also to RedPine for reporting.

• Fixed Engine_Universal.PermanentQualityRandom to be multithreading-friendly. Very oddly, this has never come up before now, but it could lead to an exception in settings-checks for any faction.
  • Thanks to dEAdOnE77 for reporting.

• Fixed an exception that would happen when trying to view the tech history because of a data conversion that had been partially completed.
  • Thanks to Linux-nörtti for reporting.

• Fixed an exception on all AI progress reduction intel entries on the intel tab.

• Fully fixed the "PlannedWave.AppendStateForInterfaceDisplay exception hit. debugCode 500" error.
  • Thanks to Puffin for the report.

• Reworked quite a few things to get the intel tab priorities working again, and also make it so that for items that are invalid for having their priorities changed, it now says that instead of saying you can change it when you can't.
  • Thanks to Linux-nörtti for reporting.

• The entire death registry and general ship registry have both been re-coded more or less from scratch, and combined.
  • There were simply too many little odd cases of ships going missing, even during gameplay with no save loads, so the premise has been changed. Rather than ever getting rid of a reference to a dead ship (until exiting the game and reloading it), it just specifies what happened to it (why it is dead). This then helps with self-repair and so forth, which will hopefully work more reliably this time.
  • This had been an intermittent problem in single player, but a very constant one on multiplayer clients, so hopefully this stamps it out on both.
  • Thanks to Puffin for the latest reports.

• Made some general ram allocation improvements based on a general core dump from StarKelp. Thanks for that!

• Wormhole names will no longer wrap in the planet view. That's been a thing for a little while, mostly accidentally, and it was pretty annoying.
  • Thanks to Viss Valdyr and Mac for letting us know.

• Three of the laser beams have gotten a bit less bright, and two of them have shifted from being a kind of fake-looking lance shape to just being the more traditional laser that is actually realistic. That looked neat when it was not scaled up, but did not work at scale.
  • Thanks to zeus for mentioning the brightness issue.

• Added some new tools the memory profiling section of the escape menu. It now shows more about the pathfinder counts and related things (those are leaking at the moment), and it also shows the total amount of RAM controlled by the GC.
  • Essentially, it's the garbage collector's estimation of how large the "heap" is in managed memory inside the game. The game might be using 4.5 GB of RAM total, but only 0.5GB to 1GB is generally needed for "managed data" that run our simulations and so forth. The rest is all texture data, audio data, graphics buffers on the CPU side, and things of that nature.
  • This is a useful thing in general, but in the short term it confirms what we were already suspecting about pathfinders.
  • Thanks to Puffin and StarKelp for reporting and helping to narrow down the problem.

• Fixed a spam of mostly-invisible unity errors into enginedebugging that was happening since last version because of a call I actually put in the wrong method. It wasted a bit of RAM and CPU, but not much.
  • Thanks to Democracy for reporting.

• Fixed a ridiculous memory leak that I introduced on 12/17 when I converted pathfinders to be pooled instead of cached. I didn't quite complete the job correctly, and so they were leaking infinitely and quickly. Figure several GB per hour. Sigh. Fixed now!
  • Thanks again to Puffin and StarKelp for noticing it and bringing it up. All else looks to be well, memory-wise.

• Fixed a number of issues where engineers would not properly repair things that they were in range of, or would not be assertive enough to go to an assistance target out of range.
  • Thanks to Viss Valdyr for reporting.

Multiplayer Fixes

• The way that you can cancel hacks by clicking the hack in the notifications toolbar should now work properly on multiplayer clients for the first time ever.

DLC3

Neinzul

• Roaming Enclave had their shield removed and added to their hull.
• Roaming Enclave had their ranged damage resistance removed.
• Clanlings now fire their Younglings in bursts
  • This should assist with them getting stuck in endless Youngling dueling loops.

Neinzul Custodians

• Fixed incorrect math in custom lobby settings.
• Make it so Enclaves will properly retreat when hurt, unless they're on the planet with their paired hive, in which case they will fight to the death.
• Second draft at Difficulty settings, generally lowering the Clanling count and increasing spawn timers.

Beta 3.769 Neinzul Custodians And Total Conversions

(Released December 19th, 2021)

• On the main menu and on the loading screen, if you have a total conversion mod installed and active, it will now show the name of that mod (it gets mighty confusing otherwise, even if you intentionally loaded into that total conversion).

Expert Mode

• Add `fuel_use` and `fuel_use_type` to type data, to allow overriding fuel use.
  • Also have units that can't go through wormholes treated use Radon by default.
• Use the new options to adjust some of the basic player non-turret defense to have the appropriate fuel usage.
  • Sentry Frigates, Tachon Arrays, Tractor Arrays, and Focused Gravity Generators all use Radon.
  • Combat Sentry Frigates use Argon.
  • Thanks to tom.prince for adding!

• Make the Fuel info in the resource bar slightly more readable.
  • Thanks to tom.prince for this improvement!

Bugfixes

• Hardened some of the wave launch code against a couple of types of exceptions that were possible.
  • Thanks to UFO for reporting.

• Change `IOption.GetDescription()` to `IOption.AddDescription(ArcenCharacterBufferBase)`.
  • This avoid building strings in a number of places.
  • Thanks to tom.prince for this improvement!

• Remove long-unused hacking and metal costs of ouguard groups.
  • Thanks to tom.prince for this improvement!

• Replace NecromancerUpgrade.ToString with NecromancerUprgade.ToDebugString.
  • Thanks to tom.prince for this improvement!

• Allow bolstering info to be set from any DLC.
  • Thanks to tom.prince for the fix!

• Improved a lot of the code around how galaxy map planets are hidden when things like hidden galaxy are used.
  • In the galaxy, it now shows the planet you are starting on and its adjacent planets, but also there are just some bugfixes in here.
  • Thanks to Sounds for the request.

• Fixed an infinite loop that could happen in makeSureGalaxyIsFullyConnected when the map topology was invalid.
  • Discovered this as a bug where if you were on the galaxy map when you exited one game, then map generation would hang the next time you entered the lobby.

• Improve the mapgen logging to be more informative, and also be written as it goes, so that if it has exceptions or other problems, it shows you where it stopped.
  • Also made it so that it logs them all into a MapGenLog folder, rather than just one file. Sometimes several mapgens happen rapidly back to back, and this is the only way to see them all.

• Exiting the game and going back into the lobby now keeps your prior map seed.

• Fixed another kind of crazy bug in general that was causing some of the planets to be thought of as destroyed when you exited the game from the galaxy map and went straight back to the lobby, which then led to various non-deterministic map results.

• Updated the guard post and command station placer to error out if it can't find the proper entries it needs in xml, and state what those are, rather than taking forever to generate a suboptimal map.
  • Thanks to Puffin for reporting.

• Fixed a regression from earlier this week where the cursor on the planet plane of play was really jerky, making it look like a low framerate. It was particularly noticeable during build mode.

• Fleet leaders that are crippled can now properly self build.
  • Thanks to Puffin for reporting that this did not work. It's not done in the main game.

• Fixed an issue in ArcenCharacterBuffer with floating point number formats that was causing all of the positional text to be shoved way over to the left in the C-click details of units that can be upgraded by techs, etc.
  • Thanks to UFO for reporting.

• Refactored the ShipVisualizer to no longer use the lazy load wrapper, and to keep drawing even if the related in-game squad is dead. Between these two sets of changes, explosions of ships had stopped happening. They now work again, and should do so without risk of cross-threading exceptions.

• Total conversion mods no longer load the Expansion or GameVersion folders. Those are exclusively loaded from the main game itself.
  • In particular, the GameVersion folder is important so that the game can deserialize properly without the mod having to be updated every time there is a new push.
  • Thanks to Puffin for alerting us to this issue!

DLC3

• Add a description to the necromancer upgrade options available in the lobby.
  • Thanks to tom.prince for adding!

• Neinzul Custodians are being added to replace the Territory Dispute faction, which was not working as well as we had liked. This new faction is much more interesting in practice.
  • More info will be coming about them later, but essentially they function like an allied Warden for you, with offensive capabilities as well, when you use them as allies. The Neinzul Sappers that Badger added are more focused on economy and defense, but the Custodians are a lot more outward-focusing. The pair of them can make for some really compelling vassals.
  • Thanks to StarKelp for implementing!

• Neinzul
  • Rebalanced Clanlings slightly
    • Mammoth damage increased, bubble shield removed
    • Bear damage decreased
  • Thanks to StarKelp for updating!

• Neinzul Custodians
  • Fixed a bug where Enclave deaths would not always destroy their attached Hive
  • Fixed a bug where 2 Enclaves from every attack could get stuck on their target planet
  • Stat Rebalances
    • Custodians were currently instant winning pretty much every engagement due to vast overtuned stats, making for subpar testing
    • Roaming Enclave Health/Shields 750k -> 500k
    • Roaming Enclave Shots per Salvo 50 -> 10
    • Justice/Bravery Damage Per Shot 1000 -> 100
    • Bravery damage mult per 10% hull lost 1 -> 2
    • Justice shots per salvo 50 -> 10
    • Clanling Mammoth Health/Shields 100k -> 50k
    • Clanling Mammoth Beam Damage 5000 -> 1500
    • Clanling Mammoth Beam Reload 1 -> 5
    • Clanling Mammoth max beam targets 30 -> 10
    • Clanling Bear Health/Shields 100k -> 75k
    • Clanling Bear Claw Damage 2000 -> 1000
    • Clanling Bear Roar Damage 2000 -> 500
    • Clanling Bear Roar max targets 50 -> 10
  • Thanks to StarKelp for updating!

• Neinzul Custodians: Approaching ideal behavior for player aligned --npc aligned still a bit weird but also much better.
  • Added a new Overkill Ratio setting, which determines how much of an advantage they desire on planets before they consider it a victory
  • Added the ability for them to flush units from Reinforcement points and gave them anti-cloaking burst
  • Updated their Catchup logic to be far more responsive, was previously reliant on Fireteams updating their per planet ship lists
  • Removed the unused 'alliedDefense' list from their targeting code
  • Increased the speed of Roaming Enclaves to match that of Clanlings
  • Thanks to StarKelp for updating!

• Fixed a bug where Migrants would try to warp right to the destination planet, and thus not move at all.
  • Thanks to StarKelp for updating!

• Disabled Fireteam's built in anti-deathballing logic for Neinzul Custodians.
  • Thanks to StarKelp for updating!

• Disabled Custodian's forceful retargeting due to their inability to ever gather quick enough before they reset.
  • Thanks to StarKelp for updating!

• Made Custodian Tachyon Blast silent.
  • Thanks to StarKelp for updating!

• Fixed a bug where the AI would try, and fail, to send waves against Custodians.
  • Thanks to StarKelp for updating!

• Implemented hostile versions of planetary attack/speed boosters
  • hostile_speed_multiplier, hostile_speed_multiplier_added_per_mark
  • hostile_attack_multiplier, hostile_attack_multiplier_added_per_mark
  • Thanks to StarKelp for adding!

• Neinzul Hives
  • Removed Gnats and replaced them with the new planetary attack/speed debuffs
  • Make it so hives makes their planet angry, even if you one-shot them
  • Thanks to StarKelp for updating!

Beta 3.768 Ironman And Doomsday

(Released December 18th, 2021)

• The AI Reserves will not try to defend mark 1 planets. This should remove deepstrike danger for hostile-planet starts like we plan to have for Ark Empires, although the latter part is not tested yet.
  • Thanks to Badger for adding.

• Added a new dire_singular_freaky_ship_group_cat in addition to the usual singular_freaky_ship_group_cat.
  • This is completely optional, and if left blank it will just pull from the regular category.
  • In general, the game now will seed items from the dire group on AI home and bastion worlds, if a collection for that is present.
  • Thanks to tadrinth for requesting.

Base Game - Ironman Mode

• The base game now has Ironman Mode

DLC1 - Doomsday Mode

• Doomsday mode is enabled by setting an int value. Approximately every X minutes a planet in the galaxy will be destroyed. It will not be a planet with a player flagship or a planet owned by a player. Also not AI Homeworlds, Bastion Worlds, Dark Zenith homeworlds.
• You will get no warnings, so you can't try to game the system by protecting planets with flagships.
• Planets destroyed in this way don't generate AIP from anything; command stations, black hole generators, etc
  • Thanks to Puffin for the idea and Badger for the implementation

DLC3

• The Necromancer can now upgrade its buildable defensive structures with Science

• Templar Rifts are no longer killable
  • Thanks to zeus for reporting

• When a necromancer flagship is crippled, the player loses 5 Hacking points
  • This behaviour can be disabled with an in-game setting

• When a necromancer flagship is crippled, it can no longer raise ships with necromancy
  • This behaviour can be disabled with an in-game setting

• Templars no longer send assault waves every 4 minutes, but on more sensible intervals

• Player-allied elderligns no longer spawn Feeble Elderlings when the player is a necromancer

• Tracking an Elderling with a hack now also shows when it will mark up and next lay eggs

• Additional adjustments to the skull pile. Now actually does have a unique skeleton per mark. Costs 2 hexes from 1.

• Stat adjustment to the Catapult, now that this actually gains marks

• Reworked the Hedgehog into the Lightning Tower. Feels much more thematic. Now has a chain lightning weapon. Short range compensated by arcing weapon that can greatly extend range

• Basic Necomancer flagships stat adjustments:
  • hull/shields doubled to make it less prone to being crippled.
  • Range increased but ROF reduced

• Adjusted necromancer upgrades. Generally increases the capsize increase from rift hack for skeleton units (more for the weaker types) and wights.

• Special Necromancer Flagships have had hull/shields adjusted across the board. Buffs all around, more so for the elderling types

• Adjusted science costs of defensive structures to be cheap for early investment but more expensive at end. Generally, defensive structures gain additional benefits at higher marks, making it a good idea to invest at least a few marks
• Mummy science costs dramatically increased to be more in line with the others. They were super cheap, making them a braindead choice to mark up all the way

• Reduced the science costs of Igor at the higher marks, since this is a utility unit. Shouldn't brank the bank to invest this all the way

• Do a bit of buffing of the Elderlings a bit less made of tissue paper. The units still need a proper design though.

• Standardized the hull/shields of the necromancer units buildings.

• The Skeleton/Wight Launcher (created by hacking a Templar defense structure) are reworked into Forts. They were far too fragile/weak to be worthwhile
  • Skeleton Launcher is now the Desecrated Keep. Has a powerful MRLS like weapon
  • The Wight Launcher is now the Haunted Keep. Has a modified Drainer citadel weapon. (Fixed it to use wight drones instead of skeleton drones)

New "Speedster" AI Type

• Add Speedster AI type - uses fast ships in waves, has speed magnifiers on all planets, and has special speed booster guardians in the warden fleet.
  • Thanks to tadrinth for adding!

Expert Mode

• Change some details of Fuels per Strategic Sage:
  • Remove minor fuel stations and bump generation of major stations to 600k each.
  • Remove "Major " from the names of fuel stations, since there are not currently multiple types.
  • Display fuel stations in the galaxy map.
  • Reduce the number of fuel stations seeded near each AI homeworld to 1.
  • Remove the fuel stations seeded near player homeworlds.
  • Remove the fuel provided by the ZPG and Lost Energy Reactor (which were added for early testing before fuel stations were added).
  • Add a number of options to enable tweaking the seeding of fuel stations.
  • Thanks to tom.prince for updating!

• Add some more fuel options.
  • Add a Balanced Fuel Station (with one seeded per galaxy).
  • Strategic Sage isn't sure if the is necessary or desirable. I've added the required C# code to seed them, so they are easy to test.
  • Add an setting to control how often multiple stations will be seed on a planet (this only applies to the randomly seeded stations, and not the ones near AI Homeworlds).
  • Thanks to tom.prince for adding!

• Split the Balanced Fuel Station into two variants.
  • One has slightly more Argon and one has slightly more Xenon.
  • They have the same tag used for seeding, so which one you get is random.
  • Also, remove the description text for Fuel Stations about them being harder to defend, since there are no Minor Fuel Stations for that to be in contrast with.
  • Thanks to tom.prince for adding!

Bugfixes

• Fixed an exception in the tooltip for the "locate enemy overlords" tooltip.
  • Thanks to marty651 for reporting.

• The annoying "Error! Duplicate planet fleet" that has been around intermittently for the last month or so is now gone.
  • Instead, there are a few other errors that now can be thrown if invalid data is passed in to fleet creation, which would be more informative but also prevent some of the real errors if those happen.
  • And additionally, there is now a "Warning Only: Skipped creating duplicate fleet" that is preventative rather than actually letting the problem happen and then complaining.

• Added a bit of protective code that does less work on partial map generation and should also avoid even getting the warnings about avoiding duplicate fleets.

• Fixed an intermittent exception that could happen in GetMetalCost() on FleetMembership as of yesterday's build.
  • Thanks to Badger for reporting.

• Fixed some text overlap in the top of the build menu if necropolis or spire city names were long.
  • Thanks to Badger for reporting.

• The way that "is this threat" is calculated is now a lot more robust and easier to extend in general if we need to.
  • Building upon that, now things that are orbital are now calculated properly as being threat or not threat.
  • This solves the issue of there seemingly being a ton of threat early in games with an orbital AI type, or in general having too much threat when there are a few orbitals hanging around no matter what.
  • Thanks to Zeus and Badger for reporting.

• Removed StringBuilderCache, as that was less efficient and almost never used, and our own ArcenCharacterBuffer is a lot more efficient and does the same thing.

• All of the uses of [ThreadStatic] in Arcen.Universal have been reviewed, and have either been replaced or marked as "ok because of their infrequency of use."

• A variety of uses of ThreadStaticWrapperedPooledItem have been replaced with implementations that require you to get a temporary list from a central pool instead, and then return it.
  • Because of the sheer volume of background threads, the ThreadStaticWrapperedPooledItem is not the best idea because it can rack up a huge amount of ram usage during a long game even though it's not technically a memory leak.

• Pathfinders no longer use ThreadStatic or ThreadStaticWrapperedPooledItem. These were huge amounts of RAM that could be orphaned on worker threads for long periods, but now it's all centrally pooled.

• The Core dll has moved from having 38 remaining ThreadStatic objects, of which most were used super frequently, to only having 8 remaining.
  • 6 of those remaining 8 would be ideal to refactor, but are not all that important compared to what was already refactored.

• There are now only 4 remaining threadstatic objects in the Core dlls. These ones would be destructive to try to remove or replace.

• There were 187 threadstatic objects in the various External dlls.
  • 63 of these were lists of planets, although actually there were some others that were just static (and potentially some subtle bugs with multiple factions of the same type, therefore) rather than even being threadstatic.
  • All in all, after this update this gets me down to 121 threadstatic items in External. That math does not check out, but I actually fixed a number of just-plain-static planet lists, and some other collections that were not planet lists.
    • I believe it's quite possible that I fixed a few race conditions and bugs that were lurking to come out in just random periods, that are now fixed.

• Added a new RapidAntiLeakPool, which is a lot like ConcurrentPool but with added anti-memory-leak bits added in.
  • Specifically, ConcurrentPool is meant for long term or arbitrary-length times of usage of items. But unlike that, RapidAntiLeakPool is purely for temp objects/collections/etc that will be checked out and given back rapidly. This makes memory leaks very simple to detect with them.
  • This is a very handy safety net, and even lets us know when things like tracing logs are leaking if tracing is enabled.

• At this point, there were still 121 ThreadStatic items in External code. Time to keep chopping onto those.
  • After several hours, it's down to 73. Oooof.
  • On the plus side, I keep seeing little errors that these are fixing, although the errors were probably rarely going to pop up.

• Okay, so starting the morning with 73 external mentions of threadstatic.
  • First starting focusing on some of the dictionaries, some of which are static but not threadstatic.
  • Some of the mapgen logic has been updated so that it is more robust and can properly handle capturable seed caps done in multiple passes, rather than expecting them to all be in one pass.
  • Down to 67.

• All of the static dictionaries in External are now converted over if they need to be, whether they were threadstatic or not. 53 actual threadstatic objects remain.

• The various tooltip appenders and pieces all now support either regular character buffers or double character buffers.
  • Lots of conversion of threadstatic character buffers of various sorts. Down to 29 threadstatic objects in External, one of which is now marked as okay. So 28 to still investigate.

• Fixed a code-todo item that might have been behind ships not always being willing to move from one planet to another when you clicked a planet on the galaxy map.
  • This was done while fixing up some of the various threadstatic items, so a nice bonus.

• The External code is now down to (giant sigh) 3 remaining threadstatic members. Two are okay and marked as such, while the other one is already a different form of memory leak from the look of things and needs a larger refactor.

• Added a new ListOfLists, which is now used for writing the details of outguard when they are in tooltips.
  • This solves a very minor memory leak that was present there previously, as well as removing the last of the threadstatic items that needed to go. At last!

• Fixed a minor memory leak with VassalMissions in DLC3.

• Added a new AddIfNotAlreadyIn to List, which does the equivalent of "if not contains, then add" in one quick call.

• The way that faction influence on planets is set has been updated so that it now does it all in one batch and thus does not introduce a flicker.
  • Previously, it was not flickering on the UI or on the main simulation because it was setting this as part of the main thread, but LRP threads and other worker threads could have experienced flicker that would cause them to behave inappropriately.

• Fixed a bug where if you did not have DLC2 installed since the great refactor, when you started a game you'd get an error about StartingAIPlanetRatio missing.
  • Thanks to tadrinth for reporting.

• Updated the civilian industries mod to the most recent code changes from this build, since it's fresh in my mind.
  • Overall there were 16 threadstatic collections that I had to translate over (I had been the one that originally added those anyhow).

• I still have yet to be able to duplicate this myself, but several people have reported it, so if folks can verify this is fixed that would be appreciated. There was a bug where various ship counts you could capture, or that you have, or that units were strong or weak against could count up rapidly. This should be fixed now, but it was intermittent before and I never could trigger it.
  • Thanks to UFO for the most recent report.

• Fixed an exception in GetHostileBlackHoleGenerator().
  • Thanks to motai for reporting.

• Put in a bit of defensive code against a rare serialization error that could happen in GetUltraEfficientStyleData for some reason.
  • Thanks to UFO for reporting.

Beta 3.767 Elderling Sanity

(Released December 16th, 2021)

• Hitting the return/enter key now closes OK popups, and also confirms yes/no popups. Hitting escape continues to mean No on yes/no popups, and also still closes Ok popups. This was actually super annoying not to have in place, not sure why it got left like this so long.
  • Thanks to tadrinth for suggesting.

• Added a new metal_cost_multiplier_list on MarkLevelScaleStyle.
  • This allows for ships to have different metal costs at different mark levels. We're not planning on using this in the base game, but it's useful for certain mods.
  • Thanks to Puffin for requesting.

DLC3

• The necromancer Phylactery now starts with 6 hexes. This lets them build all the structures they'd get from taking the first rift and also building their lobby utility choice. It felt kinda bad when you couldn't build all your initial stuff

• Sapper crystals will flower more quickly when there's only one Habitat; this should let them get started a bit faster. Otherwise they can take a while to get started. This slow start is especially hard for necromancer-allied sappers
  • Thanks to Tom for the balance feedback

• Changed the MLRS frigate back into the Chain Lightning frigate. Medium Splash frigate that does powerful single target damage, and minor secondary damage, more so to units with energy >=7K
  • This was always my intention but it seemed that the CL Mechanic was gated behind DLC2. This is no longer so

• Skull Pile changes:
  • Skull Piles are now defensive structures. They shoot Weak Skeletons as weapons
  • Skull Pile now shoots another type of skeleton with each mark, to make this an interesting defensive tech choice.
  • Added "Caged" versions of skeletons. These are simply variants of the other skeletons that exist to be used by the Skull Pile
  • Skull Piles can only be built at a Defensive Necropolis. Having an additional defensive tool should make Defensive Necropolis more enticing, I hope
    • Thanks to discussion with Tom, Zeus and Badger. Thanks to Zeus for the unit XML

New "Panopticon" AI Type

• Add Panopticon AI type - has AI Eyes on most planets, using the singular_freaky_ship_group_cat Reinforcement budget.
  • Add Tachyon, Gravitic, Sabot, and Hive AI Eyes - DLC3 but no DLC3 mechanics.
  • Add Aggressive, Hunter, Warden, and Dire Guardian AI Eyes - Spawn ships when alerted rather than directly attacking, using DLC3 raid engine mechanics
  • Add Dagger, Eyebot, and Praetorian Extragalactic War Eyes - Used only by the Panopticon AI Type.
  • Thanks to tadrinth for adding!

Elderling Rebalancing

The goal of this is to make the Elderling lifecycle a bit more interesting. I want it to feel like they have more of a real life cycle, and also to make it easier for players to get more Essence. There should be lots of weak Elderlings to hunt, but still to keep the "OMG that's terrifying" feel of the High-Tier Elderlings.

• Elderlings will now generally start as weak elderlings and get stronger over time (first marking up to 7, then evolving into more powerful forms). Elderlings gain experience over time generally, and bonus experience if there are enemies in their Territory.

• If there are too many elderlings on a planet, the weaker ones will lose Sanity. When an Elderling runs out of Sanity it will go mad
  • They will either fly away to attack an enemy faction's king, or becoming Maddened.
  • Maddened elderlings are hostile to literally everything. When a Maddened elderling is slain, other Elderlings on the same planet involve get bonus experience.
  • The goal of this mechanic is to make sure that the game spawns lots of Elderlings, but they don't get too thick on the ground. It's important for the Necromancer to have lots of Elderlings to hunt for Essence
  • Player-allied elderlings will never become Maddened

• Elderlings now mark up based on Experience rather than time.
  • Right now Elderlings get experience over time or when enemies are in their Territory.
  • Other experience generators may be granted as well.

• When a mark 7 Elderling would mark up, it instead becomes a more powerful Elderling
  • Low tier Elderlings become mid tier Elderlings, and mid tier Elderlings become high tier Elderlings

• Also add the ability to Track elderlings; a player can hack an Elderling to have the Elderling display its Territory in the Elderling's hovertext
  • Gives the player another way to interact with Elderlings

Neinzul Wild Hives Updates

Neinzul Wild Hives have received a number of updates shoring up their core gameplay loop to feel far tighter, and make them feel much more like a fast acting, and reacting, infection. These changes help alleviate some issues related to them leaving half of the galaxy as a graveyard before you even arrive.

• Saves Broken

• Younglings now decay far, far quicker.
• Wild Hives now consider a planet of theirs under attack for a full minute, instead of immediately thinking its safe.
• Wild Hives and Enclaves are now targetable by other factions when they spawn soldiers, and remain targetable until they consider a planet safe.
• Wild Hives and Workers now rapidly regenerate on planets that they consider safe.
• Soldiers now decay much more rapidly on safe planets.

• At Higher Intensities, Wild Hives are now able to consume other neutral entities.
  • At a high enough intensity, they are even capable of taking over entities like Major Data Centers and Lost Spire Frigates, requiring you to enrage their hive to gain access to them.

• Removed their existing Mark Up Logic
  • Instead, all seeded Neinzul Hives spawn in at Mark 1.
  • When a Neinzul Hive has enough points to spawn a Worker, they spawn it at their current mark level, and than the Hive marks up.
  • Soldiers are still spawned in as their parent entity's mark level.

• Various bugs fixed that were found by DEMOCRACY? DEMOCRACY! and RocketAssistedPuffin during testing.

Bugfixes

• Fixed the raycasts against unit icons not working properly in the planet view since The Great Refactor.

• Added in a new PerSecondSimRepair thread that does checking for seeing if there are squads, shots, or wormholes missing in the main dictionary lookup that are in the entity collections of planet factions, which would be the normal genesis of the "ghosts" problem in multiplayer. As well as the "such and such missing from death registry" bug, too.
  • So far nothing has happened, but it should self-correct any cases like that, if they still happen.

• NewTimingsBeingBuilt and similar have been removed from the codebase, as they have not been used in a really long time (a couple of years) and we have better timings-checks in place now. This will probably improve performance to a mild degree.

• Fixed a couple of bugs with the intel sidebar. One where things were flipping around in an unhelpful way since the great refactor, and another where some were blinking in and out of view over top of others since that same time.
  • This was all just a case of an older style of performance-enhancing caching clashing with the new kind, and has been resolved.
  • Thanks to Sounds and dEAdOnE77 for reporting.

• Fixed a bug where range rings from build mode would continue drawing into the galaxy map if you tabbed over to there while in build mode.

• Added hardening to prevent errors in GetStringValueForCustomFieldOrDefaultValue even if there's something like a misconfigured faction field that has a subtable that is empty or whatever.
  • Thanks to Puffin for reporting.

• UtilityMethods in DeepInfo is now a public class, rather than an internal one, so that modders can use it and inherit from it and all that sort of thing.
  • Thanks to Puffin for reporting.

• DoOnAnyDeathLogic_HostOnly_AfterFullDeathOrPartOfStackDeath on scenario is now a virtual method, which lets sub-classes inherit from it.
  • Thanks to Puffin for requesting.

• Tutorials have been busted since the great refactor started, and they are now functional again. There may be some errors that pop up after playing them for sufficiently long, or in certain hover circumstances, so I'm still looking into those bits. But by and large, they seem to work now.

• Fixed an issue on the galaxy map where if you clicked certain parts of the ui, your clicks would go through to the space underneath. This was problematic for a number of reasons, but most notably it actually made step 5 of the second tutorial impossible to finish.

• The text of step 5 of tutorial 4 has been made more brief so that it fully fix on the screen.
  • A few other minor tutorial text tweaks, as well.

• Tutorials now seem fully functional!

• Fixed an issue where log messages were not being written properly when you loaded a game too quickly after having just quit another one.

• Fixed a minor tutorial bit of invisible log clutter where it was complaining about certain planets being assigned to nobody (by design, it was a false warning).

• Fixed an issue where tutorial starting planets sometimes had a blank background instead of a proper starfield.

Beta 3.766 The Red Harvest

(Released December 15th, 2021)

Balance

• Raid frigates and variants engine power increased to 20 to resist Blackhole Effect.
  • Thanks to zeus for updating!

• Raider strike craft and variants engine power increased to 20 to resist Blackhole Effect
  • Thanks to zeus for updating!

• When the player gets 4 spire cities, the AI will spawn a Nadir Dire Venator; this is a fast, tanky melee brawler designed to be particularly good against Spire Ships
  • This only takes effect if there's an AI at difficulty 6 or higher in the game
  • Thanks to Badger for adding! Unit name and design suggestion from zeus.

Necromancers

• The necromancer now gets Weak Skeletons or Weak Wights from drones. Also the Heroic AI Type no longer gets regular Harbingers (and heroic harbingers are weaker)
  • Thanks to Badger for updating.

• Mummy homes now cost 4 hexes; Tom reported that mummy bodyguards were too easy to get previously
  • Thanks to Badger for updating.

• Slightly tweak the essence cost of various Necromancer direct upgrades.
  • Before the change to scaling essence cost, the total cost for mark 7 was 90, afterwards it was 160.
  • For the phylactery and major necropolis, I decreased the cost of the initial level slightly, but scaled it a bit more quickly, so the cost for the final upgrade ended up the same. The new total cost to upgrade to mark 7 in 153.
  • For the minor necropolis, I decreased the initial cost and the rate of scaling. Now the total cost to upgrade to mark 7 is 109. A Mark 7 Minor Necropolis has 8 hexes, compared to 15 for a Mark 6 Major necropolis (which costs 108 essence).
  • For the necromancer flagships, I decreased the initial cost even more, and the rate of scaling by the same amount. The new total cost to upgrade to mark 7 is 99. In my experience, flagship upgrades are significantly less impactful than necropolis upgrades.
  • I've not touched defensive necropolis upgrades, since I don't have as much experience with them, and none with upgrading them. However, my initial thought is that both the upgrade cost and the number of hexes per mark should be decreased, inline with minor necropolises. Given that they can only build weapons and shipyards, so I suspect most hexes will be for weapons. Thus, each additional hex represents significantly more defensive power than for the other forms, but also significantly less offensive power. By decreasing the additional hexes per mark, that makes it harder for them to become impenetrable planets, and decrease the cost to compensate for their limited utility beyond that.
  • Thanks to tom.prince for updating!

• Templar balance adjustments.
  • The templar now have a lot more things scale effectively with their faction Difficulty.
  • This includes number/mark level of wave leaders, wave rate, and range of defensive structures.
  • The Templar's 3 defensive structures now help planets at 1, 1 and 2 hops away on lower difficulties (suggested by Tom), and at more hops at higher difficulties
  • Thanks to Badger for updating!

• Spawn a few more feeble elderlings, hopefully to counter reports of insufficient essence for the necromancer
  • Thanks to Badger for updating!

• Add necromancer harvest count; the necromancer now keeps track of how many units and of what types it has raised during a battle. This information is shown in the combat Notification hovertext. The goal is to make the player feel powerful when they see how much many ships they've taken.
  • Thanks to Badger for updating!

Sappers

New turrets for the Sappers

• Jasper: Wormhole only Turret. Hybrid of Ambush and Deathgrip. Cloaking is to keep it alive as it does its initial burst of damage (as such Turrets are sitting in the alpha strike area) to the things held in tractors from the Array further below.
• Garnet: Solely around Important Structures. Since these structures may well have Forcefields, and some may have penalties to units within, this one has the Bastille bonus to ensure it isn't a concern for players, particularly as you can't exactly stop the Sappers doing this.
• Hematite: combination of anti Albedo and anti-Mass to deal with things immune to tractors and cloaked units. Can focus fire its weapon on singular targets
• Lazurite: Intended as the beachheader. Using the Classic Fusion Ruffian Turret as a base. Radar Dampening (vanilla Ruffian has this), long range (same), but then bonuses against structures. As a curious experiment, giving it a minor ambush bonus as well, since enemies coming out of Guard Posts, and incoming Warden are viable victims.

• Thank you to Puffin for replaciing the placeholders with some shiny new turrets!

Bugfixes

• Clear planet lists before collecting them when determing planets to reinforce.
  • This may address the issue of some planets getting reinfored way to often, for one of the new NA AI types.
  • Thanks to tom.prince for updating!

• Fixed an issue with my ArcenTypeAnalyzer where it was incorrectly resetting arrays during xml reloads.
  • There's now a subset of the ApplyDefaults code that explicitly handles arrays.

• Fixed another issue with ArcenTypeAnalyzer where if something was a list but initially null, it was not being handled quite right, either.

• Fixed an issue with how surrogate tables (external tables) were reloaded in xml since the great refactor. This had been making all of them come through blank, but now they're fine.

• FleetItemCategory now implements IClearableDuringXmlReload, which means that the entire xml is now able to reload properly again!

• There are now new debug settings that allow for doing the table exports that I used to find and fix all these issues.

• The R-view now has more spacing so that the icon does not overlap any text.
  • Thanks to Daniexpert for reporting.

Beta 3.765 Easier Upgrades And Necromancer Surge

(Released December 13th, 2021)

• Fleets that can be upgraded by science now show up in your regular tech menu (as well as still in the fleet window).
  • This is a super useful learning tool for new players in general, who otherwise might not know they can upgrade fleets. And it's a time-saver for everyone, as well as making it more clear what all your options are on one screen.
  • On the tech window, you can see all the usual details you do about tech upgrades, but this time for fleets (how many lines affected, how much strength gain in mobile and defensive forces, etc). This is a lot of new information, which before was only available via C-clicking and not in aggregate; it's actually possible to see just how powerful these upgrades are now.
  • On the fleet popout, it now includes more of the details there in that window as well, just to make things even all around.

• Add an integer for Salvage to the Planet object; done to make Puffin's like easier
  • Thanks to Badger for adding.

• The game now tracks whether orders are from HumanPlayer, FRD or Other. The Wave and Templar factions now allow for the LRP code to override FRD orders to do 'more exciting' behaviour, like attacking a command station or bypassing your defenses to hit another planet.
  • Thanks to Tom for pointing out that there was a race condition between targets being assigned by the FRD code and the Wave LRP code
  • Thanks to Badger for adding.

Necromancer Improvements

• Guard posts that are not marked as combatants will no longer be reanimated by players.
  • Thanks to zeus and Mac for reporting how useless (and expensive!) they are.

• Don't endlessly clone player-owned guards posts when they die to remains.
  • Thanks to tom.prince for the fix!

• Copy flagship orders to units summoned by necromancy.
  • Thanks to tom.prince for the improvement!

• Added a new mapgen_should_all_guard_posts_be_armed tag for player types.
  • This is now on for the necromancer empire and sidekick. It makes all guard posts armed, so that there's something interesting to capture at all times for these necromancers.

• Necromancer factions now start with two effigies already built, and one starting shipyard, so that they have more firepower than just their starting necromancer flagship when they are getting started.
  • Chris was finding it kind of impenetrable at the start, because until some buildings were constructed, there was not a lot to do and everything was very slow and frustrating. So starting the player off similar to how human empires have buildings in place to build things instantly makes the necromancer empire feel more familiar.

• Necromancers now actually start with some skeletons and wights in their initial necrofleet, so that the initial stages of the game doesn't play out as if you were in molasses.
  • Necromancers are essentially a faction of inertia -- once you have stuff you can convert enemies with, you can convert more, and hooray. But if you have very little to help with the converting and destruction, then it's very very painful.
  • Most notably, now that all guard posts of the enemy are armed when necromancers are around, it pretty much kills all the Ikiryo and the flagship just trying to take out one. You can go hunting for skeletons for a while, but this is not exactly a stimulating start. So let's basically start the necromancer in motion, and get to the good part faster.
  • We thus give them 30 base skeletons, 10 skeleton archers, and 20 skeleton warriors, along with 5 base wights, 2 ravenous wights, and 3 infectious wights.
    • The starting necrofleet starts out in transport mode so that these don't all attrition away from the start, and so that the player can quickly just fly over to an enemy planet and start the first beatdown.
  • I don't really have a sense if this makes the larger necromancer game easier or not, but it seems like this sort of starting momentum is just going to be something that speeds up the early game without making the endgame any easier.

• Necromancer games now seed some capturables.
  • There are now Skeleton/Wight/Mummy amplifiers; having one of those on a planet with a major/minor necropolis now grants the bolstered fleet a chance of getting bonus skeletons/wights/mummies.
  • The goal of this is to give the necromancer some actual objectives and things they might want to hold.
  • Thanks to Badger for adding.

• More templar units are in Attacker_Full when they are created
  • Thanks to Badger for updating.

• Add short names for city sockets, for use in build sidebar.
  • Add display of hexes (used and available) in direct build sidebar.
  • Thanks to tom.prince for adding!

• Seed starting skeleton/wight home, instead of support unit tomb.
  • Thanks to tom.prince for updating!

• Make it so that Necromancer Amplifiers can be rebuilt.
  • Also, make them immune to player damage.
  • Thanks to tom.prince for updating!

• Make guard posts converted by Necromancers have the same mark as originally, rather than all being Mark 1.
  • Thanks to tom.prince for fixing!

• Reduce the metal cost of Necromancer Amplifiers.
  • Given that they currently don't stop functioning when they have been turned to remains, having them take a long time to rebuild isn't particularly impactful, and just requires micro to avoid them being prioritized for repair.
  • Thanks to tom.prince for updating!

• The hacks for upgrading necromancer flagships and city-centers have been deprecated, as we'll get the exact same result a nicer way.

• Player types now allow for extra info to be written at the top of the hacking and tech windows, and the necromancers now use that to write their essence.

• Essence is now a reddish color instead of bluish. Green was definitely not the correct color, but blue also seemed too much like science. Red like blood is good.

• Where possible, "resource one" is now named and defined as part of the PlayerType. Same with "resource two."
  • In other cases, players beyond the actual affected faction (like regular human friends of the necromancer) need to see a tooltip about something that grants essence, and in those cases we also now have a global working variable on World_AIW2 that names, colors, and so forth what Resource1 and 2 are.

• Necromancer fleets of all sorts are now directly upgraded by techs, like most other fleets of that sort.
  • This lets us set really direct cost upgrade curves, as well as use a whole bunch of other code that is really intensive and that we wouldn't want to try to duplicate (how many ship lines are drawn, etc).

• Techs now have an upgrade_resource option on them, which can be blank or Science (the default), or set to Resource1 or Resource2.
  • This allows for certain techs, like the necromancer upgrades, to cost something custom (like Resource1 / Essence) instead of Science.
  • This touches a lot of UIs, but in the grand scheme it's more flexible and less code than trying to add a third branch that brings in hacks instead of tech upgrades.

• If two different factions are set up as player types which use different names for resource 1 or 2, then the game won't allow the player to be started with them together in the same galaxy.
  • This mainly applies to mods in the future.

• We already had a galaxy_wide_cap_for_players_constructing, now renamed to base_galaxy_wide_cap_for_players_constructing, but now we have the following:
  • added_galaxy_wide_cap_for_players_constructing_per_x_planets defines how much is added to the cap every X planets that you control (can be zero)
  • galaxy_wide_cap_for_players_constructing_x_planet_variable defines what X is in the above formula.
  • So, you could say there's a base galaxy cap of 5, and that you then get 5 more cap for every 4 planets you control. It will round down. Or you can do something way more or less granular, which is nice.
  • No galaxy cap will be in effect if you don't set the base galaxy cap to at least 1. If that's set to 0 or below, then the whole rest of things is ignored.
  • Also please note, all planets controlled by any player will be counted for this.
  • You can also now see the galaxy cap inside the tooltip for units with one again, plus for the first time (since it's now a thing) you can see how the cap will increase.
  • There is also now a galaxy_wide_cap_match_string that lets us have a global cap be match between several units that are variants but essentially the same.

• The necromancer major necropolis can now only have one at the start, plus an additional one per 4 planets you control.
  • The idea is to make players have to take more territory for the major necropolises, although this may be too stingy. Feedback welcome.

• On the player type data, there is a new uses_energy_and_fuel and uses_metal.
  • When this is set to false, then energy is always assumed to be fully available, and will not be shown. For metal, same deal, but essentially the metal amounts on things affect how fast they build.
  • This is something that we want to be able to turn off mainly for the necromancer factions, which just don't have any practical purpose for these bits.

• No necromancer ships have metabolization anymore (they don't need metal). Instead, they have a minor necromancy effect, whereas the larger necromancer units maintain their major necromancy effect.

• A whole bunch of unit tooltip info now turns off when you're the necromancer, relating to metal and fuel and energy and such. So much more readable!

• Fuel is no longer shown in the top bar even when fuel is enabled if you're playing a necromancer faction. More changes to come soon, but this is a quick fix to overwhelmingness.

• A bunch of different galaxy map filters relating to fuel and metal and energy and so forth are also no longer shown in necromancer mode.

• A bunch of metal and energy and fuel things in the planet tooltips also no longer show for necromancers.
  • The big thing will be the reorganization of the top resource bar, but that's not in this build.

Bugfixes

• Don't give orders to Assisters (like Engineers or Necromancer shipyards) if their target is in assist range.
  • For Necromancer Shipyards in particular, this would often (always?) result in orders well outside of combat space.
  • Thanks to tom.prince for fixing!

• Don't allow orbital guardposts to build turrets while they are remains.
  • Thanks to tom.prince for the fix!

• Fix display of rate of building of orbital descendants.
  • Thanks to tom.prince for the fix!

• Guard post seeding has been updated substantially, and there should no longer be oddities on larger gravity well sizes.
  • Thanks to Puffin for reporting, and tom.prince for narrowing down the problem and suggesting a refactor.

• Adaptive music no longer triggers outside of normal gameplay. The Esc menu now gives more information about game state with adaptive music enabled
  • Thanks to Badger for fixing.

• When you don't have enough city sockets to build something, then it no longer shows a message saying city sockets. It now says necromancer hexes or whatever.
  • It also has the general suggestion "Typically upgrade your city to get more" so that people will know that they can expand, they just need to do something else first.

ZO Balance changes

• Added a prefix to the different Zenith Miners to make it clear these get stronger over time
  • Thanks to zeus for updating!

• Buffs to the later tiers of the Zenith miners weapons
  • Thanks to zeus for updating!

NA Improvements

• Orbital Turrets no longer die when guardpost dies
  • Thanks to zeus for updating!

• Orbital Guardposts are less likely to spawn with turrets.
  • Thanks to zeus for updating!

• Exotic Destroyer's ward now only spawns the first copy, so it doesn't lose out their harmonic trait
  • Thanks to zeus for updating!

• Redo the vassal mission code to provide for more functionality, and to serve as a reference for Chris for the things I plan to support
  • Thanks to Badger for adding.

Beta 3.763 Necromancer Reanimation And Refinement

(Released December 9th, 2021)

• Add hooks for custom build sidebar category logic.
  • This is used to replace the existing logic for where command stations and necropolises can be built.
    • Command stations can be built on any planet without a command station. (Previously, this was on any *unowned* planet without a command station, but Necromancer's can now own planets.
    • Necropolises can be built on any planet without and Necropolis, and not controlled by a hostile faction.
  • This is used to control what type of DirectBuildable is created, for entities not associated with a specific FleetMembership, removing the special case code for Necropolises.
  • This also removes the `only_shows_when_planet_not_owned_by_anyone` and `only_shows_when_planet_not_influenced_by_local_player` settings on build sidebar categories, which were used to implement the old logic.
  • I've also changed the code so that entries added by sidebar categories with granted_by_any_mobile_fleet are only added to that sidebar category, and not every category that the type belongs to.
  • Thanks to tom.prince for updating!

• Add ArcenCharacterBuffer.ToStringAndReturnToPool and use it in a number of places.
  • Thanks to tom.prince for adding!

Necromancer Improvements

• Encampments, Fastnesses and Castles are now higher targets for FRD ships
  • Thanks to Tom for the feedback

• Modify templar imcome; move some fields to the XML and remove others. This is a net nerf to the Templar
  • Thanks to Tom for the feedback

• Minor buffs to some necromancer units; the early game seems to weak

• Necromancers are now able to reanimate guard posts of the AI that they kill, and these guard posts then serve as the main defenses on a lot of necromancer planets.
  • Details!
    • At the moment, each one costs 10k metal per mark level of the guard post (so, 30k for a mark 3 post) to build, and they come in at 1 health and 0 shields. For dires, it is 40k metal per mark, instead. This may need to be shifted to be different on different difficulty levels, or in some other fashion.
      • These are able to self-build immediately, so they can join the fight and make for an automatic beachhead when you are attacking weak planets in particular. This is very interesting, but if it proves too powerful, we can easily make these be remains instead, which would not build unless you take the planet.
    • If you scrap a guard post, or if one is destroyed on a planet when no necromancer flagship or phylactory or necropolis is present, then it is gone forever. Them's the breaks. Necromancer is fast and brutal.
    • If a guard post that you control dies, it dies to remains, even if your units are not present. In order to repair it from remains, you'll have to actually take the planet, I believe, so this means that you can't infinitely aggro planets you don't own.
    • Note that the necromancer units themselves don't have to kill the guard post. It doesn't matter who does. A necromancer simply has to be on the planet.
    • In the event that you have no necromancer guard-post-converter unit on a planet, but you DO have necromancer units in general, then a guard post will be converted to remains when you kill it.
      • It's easy for your flagship to die and be bailed out of the planet and for you to then just casually destroy guard posts that are gone forever, otherwise.
    • If multiple necromancer factions exist, then the one who owns the planet gets the guard post. If none own the planet, then the one with the most guard-post-capturers will get it. If there are two factions with equal numbers, then the one that has the lower faction index (added first in the lobby) will get it.
  • This whole thing is going to need tweaking and tuning, but it's off to quite an interesting start. This is the last major function of the necromancer, so far as we know pending further testing.
  • Thanks to Puffin for suggesting this guard post capture mechanic!

• The game now supports setting custom metal costs for units via code. This is useful for purposes of when we capture a guard post via a necromancer and want to have it cost something custom. The interface is now updated to show all of this sort of thing, and it makes the whole experience more clear.

• Use LazyLoadSquadWrapper for tracking Templar home castles.
  • This should make the logic of castle defenders despawning when the castle is destroyed work.
  • Thanks to tom.prince for the fix!

• Fix checking for if a new Necropolis should be able to bolster Necromancer Fleets.
  • When switching building Necropolises to use DirectBuildable, I missed that the check for Minor Necropolises was against a tag which didn't match the type name. Update the check to use a tag explicitly for bolstering.
  • Thanks to tom.prince for the fix!

• Update the checks for available city sockets to properly handle FleetMemberships with no ships in them.
  • Because some of the Necropolises were listed as being on the Necropolis build menu, they were added as empty fleet memberships, which caused the count of available sockets to be inflated (and inflated per mark level of the city even). In addition to fixing the logic for calculating sockets, I removed the remaining `NecromanceBuildMenu` tag from necropolises.
  • Thanks to tom.prince for the fix!

• Disable seeding of some more things that Necromancer Empires can't use.
  • Thanks to tom.prince for the fix!

• Remove Build Necropolis hacks, now that building them is done through the build sidebar.
  • Thanks to tom.prince for the fix!

• Make the various self-hacks that Necromancers can do as only working on their own units.
  • Thanks to tom.prince for the fix!

• Update the message for officer self-hacks to reflect that they can be done on Necromancer owned planets.
  • Thanks to tom.prince for the update!

• Add an option to control whether city centers grant tech levels to the members of city-fed fleets.
  • Currently this is turned on for all city centers, but allows experiementing with different choices. In particular, I am wondering if this is too strong for Necrofleets.
  • Thanks to tom.prince for the update!

• Show Templar Rifts on Capturables galaxy display mode.
  • Thanks to tom.prince for the update!

Player Type Improvements And Additions

• Add tags to player type data.
  • This will be used to control accesss to things like build sidebar categories and tech upgrades.
  • Thanks to tom.prince for adding!

• Add `for_player_type_tags` to build sidebar categories.
  • This is used to replace the necromancer specific attributes used for the same purpose.
  • Thanks to tom.prince for adding!

• Replace 'for_player_types' on tech upgrades with 'for_player_type_tags'.
  • Also remove 'uses_tech_upgrades_from_player_types' which was used for the same purpose.
  • Thanks to tom.prince for adding!

• Replace 'for_player_types_or_blank_if_for_all_empire_style_types' on tech upgrades with 'for_player_type_tags_or_blank_if_for_all_empire_style_types'.
  • Also remove 'uses_hacking_types_from_player_types' which was used for the same purpose.
  • Thanks to tom.prince for adding!

Mapgen

• Added mapgen_should_fuel_seed_for_me_if_fuel_is_enabled and mapgen_should_arses_seed_for_me to the PlayerTypes.
  • Fuel now only seeds at all if there's at least one player faction that uses fuel. So, not spectators or suzerains, for example.
    • Additionally, fuel seeds near any player faction that uses fuel, not just empire-style factions. So this means there's extra fuel near a necro sidekick now, for example.
  • For ARSes, those now only seed at all if there is at least one faction that wants them. And then beyond that, it seeds them based on the number of players that want them, rather than the number of empires.
    • Necromancer empires don't use ARSes, for instance, but in theory someday there could be a suzerain-style faction that does get them. Right now suzerains do NOT get them.

• Fuel depots have been fixed to always seed as un-owned. This way zombies and minor factions won't attack them when they are under the control of the AI.
  • We want all these sorts of things to be able to attack it once you have control of it, so this was just an initial-seeding issue rather than a targeting issue. Thematically the AI should not have owned these, anyway.
  • Thanks to RedPine for reporting.

• Added a new SeedSpecialEntities_LateAfterAllFactionSeeding_CustomForPlayerType on IPlayerTypeSpecificCodeDeepInfo, which allows for completely custom late-seeding of "stuff in the galaxy" from whatever player type.
  • If there are any truly custom things that are capturables for a faction, then this is probably where to put them.

• The EarlyOfficers tag has been removed, and there is now AnyEarlyOfficer and NonArkOfficer.

• There is now a LoneSpire tag, and one of the three lone spire frigates now seeds based on that again.

• Previously, other officer fleets (golems and arks) were seeding by the special entity type MobileOfficerCombatFleetFlagship.
  • They now instead seed based on the tag AnyOfficer, which is far more flexible.

• Previously, regular fleets were seeding by the special entity type MobileStrikeCombatFleetFlagship.
  • They now instead seed based on the tag RegularStrike, which is far more flexible.

• Previously, support fleets were seeding by the special entity type MobileSupportFleetFlagship.
  • They now instead seed based on the tag SupportFleet, which is far more flexible.

• Previously, citadels were seeding by the special entity type BattlestationCitadel.
  • They now instead seed based on the tag SeededCitadel, which is far more flexible.

• There is a new mapgen_should_coprocessors_seed_for_me tag on player types.
  • Coprocessors now only seed if at least one of the players in the game has a player type where this is true (all other prior seeding restrictions still apply, such as there must be at least 2 AIs).

• There are new mapgen_should_zenithpowergenerators_seed_for_me and mapgen_should_zenithmatterconverters_seed_for_me tags on player types.
  • These only seed if at least one of the players in the game has a player type where this is true.
  • Works the same for mapgen_should_distributionnodes_seed_for_me, mapgen_should_minorcapturables_seed_for_me.
  • Also mapgen_should_datacenters_seed_for_me, mapgen_should_majordatacenters_seed_for_me, mapgen_should_superterminals_seed_for_me.
  • Also mapgen_should_tsses_seed_for_me, mapgen_should_odsses_seed_for_me.
  • Also mapgen_should_techvaults_seed_for_me, mapgen_should_frses_seed_for_me, mapgen_should_cruisers_seed_for_me, mapgen_should_destroyers_seed_for_me, and mapgen_should_lonespire_seed_for_me.

• There is a new tag called mapgen_require_minorcapturable_tag.
  • If that is set to anything other than blank for any player type in a game, then it will switch to using that player type's tag for seeding rather than the usual tag of "MinorCapturable"
  • If there are multiple factions with different required tags set for their player type, then the one that is added last will prevail.
  • The purpose of this is to allow for various factions to override the normal seeding, but it's not expected to be done all that much.

• Same as the items above, there are pairs for mapgen_should_officerfleets_seed_for_me and mapgen_require_officerfleet_tag.
  • When there is an Ark Empire in the game, the game will no longer seed any Arks as officers, only golems, which is quite interesting.

• Then the same sort of thing is for mapgen_should_citadels_seed_for_me and mapgen_require_citadel_tag, but this only is for potential modding or future expansion in terms of the tag options.
  • Same for mapgen_should_supportfleets_seed_for_me and mapgen_require_supportfleet_tag.
  • And also mapgen_should_earlyofficers_seed_for_me and mapgen_require_earlyofficer_tag. For the Ark Empire, again the Arks are filtered out on this one.
  • And lastly, also mapgen_should_mobilestrikefleets_seed_for_me and mapgen_require_mobilestrikefleet_tag.

• The distance from the player homeworld that "early officers" seed on difficulties 7+ is now 3-8 rather than being 6-16 hops out. For 6 and down it remains 1-4 hops out.

• Added mapgen_should_spirearchive_seed_for_me and mapgen_should_outguard_seed_for_me, so that these can be turned off for the factions that can't use them.
  • Thanks to tom.prince for suggesting.

Ship Encyclopedia

• Added new Thematic Groups and Thematic Group Types, which I had been planning for a long time for purposes of the unit encylopedia (which is coming soon), but which in the short term is also really useful for searching for units by name on the galaxy map.
  • Essentially, when units are added to thematic groups, those groups also provide text matches (for example: "Arks", or "Neinzul"). The group types are simply for later use in the encylopedia.
  • For now, the three group types are Race, Coalition, and Unit Class.

• The Arks and Golems from the base game, DLC1 and 2, and the outguard from DLC2, have all now had their thematic groups assigned.
  • There's tons more units that need thematic groups assigned for the ship encyclopedia, but hopefully xml-oriented folks can help out with that, as that's a big job.
  • For now, one of the most useful things is that you can search for "ark" in the "Find Units" part of the galaxy map filters and actually get all the arks. Previously, ones like Nodorian Tortoid or Grand Salvage would not show up because Ark is not in their name.

• Any units that are not part of any thematic groups now go into the Unsorted thematic group.
  • This is a signal that someone ought to sort these, so it creates a nice bit of a todo list for us or for mod authors.

• Drones that are not marked as the thematic group Drone now get marked as that thematic group automatically. That's not enough to keep them from still being in the Unsorted group on top of that, though.

Bugfixes

• Fixed a bug where difficulty 3 for AIs had an 8x multiplier to border aggression (that was going to be apocalyptically difficult) rather than 0.8x.
  • Thanks to Puffin for discovering.

• Fixed an issue where total conversion mods were unable to find icon xml or code dlls in the original GameData folder from the game. They should now do that if they don't find it first in the total conversion folder.
  • Thanks to Puffin for reporting.

• Make fuel stations actually capturable. (untested)
  • Thanks to tom.prince for the fix!

• Remove some duplicate checks for finding hacker types.
  • Thanks to tom.prince for the fix!

• Allow human home arks to hack.
  • Thanks to Evil Bistro for reporting an issue related to this, and tom.prince for fixing!

• Hide some really common errors that were introduced in r14956+r14957.
  • They are still accessible behind a tracing flag, so they can be debugged, but this should stop the duplicate reports we are receiving because of them.
  • For future debuggers: I suspect part of what is happening is that we are trying to return things to the pool twice, partially since we are stopping showing ships one frame early.
  • Thanks to tom.prince for the fix!

Beta 3.762 Expert Necromancers

(Released December 6th, 2021)

• The wiki now has a page for AI Types for AIW2, to go with the page for AIWC. It's not fully filled out yet though.
  • Thanks to Badger for setting it up and Lord of Nothing for filling out most of the fields we have so far

Expert Mode

• All of the home arks, and the home command station, now produce 500k argon and radon rather than 100k.
  • Thanks to Strategic Sage for suggesting.

• Fuel depots -- the base seeding of them -- is now more sparse than before.
  • However, there are now some extra fuel depots seeded relatively near to every human empire, and every AI homeworld.
    • These tend to make for some highly-valuable "megaclusters" that are very enticing. Thanks to Puffin for suggesting the megacluster part.

• Overall, the major fuel depots now give more, and the minor ones now give less. In general making for more variation as you look around the galaxy map.
  • The idea is to make it more interesting as to which planets to take for fuel purposes.

• Fixed a typo where the major fuel depots were labeled as minor. Confusing!

• Major and minor fuel depots previously had an AIP cost on death of 10. That was a copy-paste error and not desired.
  • Minor now have an on death AIP of 1, and major have a cost of 3. You're going to be hurting enough from losing these as it is, and probably have to take more planets.

• The icon offsets for the fuel and necromancer resources are now properly offset to fit inline with the rest of the text they are with.
  • This is just a handy little xml piece of work.

• Zenith power generators now generate a lot more fuel, making them more exciting in expert mode.

• Added a new "Fuel Availability" galaxy map filter that only appears in games with fuel enabled.
  • Shows how much fuel is available at each planet, for each type of fuel. Unit icons shown under the planet are one per fuel generator.

DLC 3

Necromancer saves are broken in this patch (all other saves are fine! Necromancers are an experimental thing at times still.)

• Fix a bug with the first templar encampment's markups

• The game tries to make sure a Rift doesn't have too many options that a necromancer doesn't have high enough mark levels for; it's frustrating if all the upgrades in a Rift are things you can't get

• The necromancer now builds a necropolis via the build menu.

• Add a new utility ship for the necromancer, the Ikiryo. This ship line specializes in preventing enemies from fleeing

• The necromancer gets a bonus 'utility' ship at the beginning of the game. They can choose in the game lobby which ship type they would like
  • They can currently choose between an anti-cloaking ship line (the Banshee), an Engineer/Support shipline (the Igor), and the Ikiryo, a ship that focuses on preventing enemies from fleeing

• Add a lore journal for the necromancer and templar

• Add a bunch more necropolis names (there's now about 60)

• Add a target evaluator to help ships with necromancy be more effective. Some balance tweaks to the Necromancer Tank Flagship
  • Thanks to Tom for this code

• Elderling Eggs are no longer auto-targeted by ships; this way players can choose whether to let an Elderling egg hatch or not.
  • Thanks to Tom for suggesting

• If the necromancer has built enough structures to have a > 100% chance of getting various sorts of skeletons/wights/etc, the odds are now scaled down to a 1 in 100 chance

• Some tweaks to the text for the way DLC3-specific resources are shown in tooltips
• Some tweaks to the text for bodyguard mark level increase upgrades

• Give proper support for Mummies (gotten from killing Large/Scary ships). The goal of this is that Mummies should work very similarly to Skeletons and Wights.
  • There are now 3 variants; Royal, Decaying and Withered.
  • The player unlocks them through Rift Hacks.
    • The can build Mummy Variant Homes to give them the ability to get those ships; they also give a single bodyguard each.

• The Templar Ships given to the Heroic and Jabberwock AI Types now have their own XML so they can be balanced separately

• Higher Tier Elderlings will appear earlier in the game
  • Thanks to Tom for the feedback

• Make the Necromancer Tank flagship less overpowered.
  • This removes the starburst weapon from it, and makes it available earlier.
  • Thanks to tom.prince for updating.

Bugfixes And Improvements

• Factor out the logic of what cateogry of ship to summon from Necromancy.
  • I'd like to re-use the logic to prioritize targeting, so I've separated out the logic that picks between skeleton, wight and mummy, from the logic that picks a specific type of ship from that category to spawn.
  • Thanks to tom.prince for updating!

• Add a target evaluator that priortizes targets that can be converted via Necromancy.
  • I've also applied it to most of the mobile ships with Necromancy. There are a few that also have other effects that impact targeting that I've not modified.
  • Thanks to tom.prince for updating!

• The IExtraDamageModifierImplementation now also gets the system that is trying to calculate the damage passed as a parameter.
  • Thanks to NR SirLimbo for adding! From a discussion with RocketAssistedPuffin.

• Remove a level of indirection in GameCommand_PlaceSelfBuildingUnit.
  • Thanks to tom.prince for updating!

• Use DoubleBufferedConcurrentList for Necropoleis.
  • This allows adding newly built Necropolises to it, to prevent being able to build multiple necropolises on a planet all at once.
  • Thanks to tom.prince for updating!

• Make direct-building self-building units slightly more moddable.
  • This replaces `Engine_AIW2.Instance.PlacingEntityType` (which was really a `FleetMembership`) with a new `DirectBuildable` type. The behavior can be controlled by providing a new implementation of `IDirectBuildableImplementation`.
  • Most of the code in PlanetViewSelector that knew details about what to build was moved to `IDirectBuildableImplementation`. In particular, details about finding a builder for Command Stations, and constructing a game command to do the build were moved.
  • A new game command to build a Necropolis was added.
  • Minor regressions:
    • After placing a command station, you are no longer taken out of build mode. I've left a comment in PlanetViewSelector about how to fix this.
    • Placing a Necropolis now doesn't require the Necropolis to self-build again. I've left a note in the new gamecommand suggesting a fix for this.
  • Thanks to tom.prince for updating!

• Remove the now unused NecromancerPerUnitBaseInfo.
  • This was superseded by the DirectBuldable code for customizing how Necropolises are built.
  • Thanks to tom.prince for updating!

• Add some nerfs to the Transcendental Elderling.
  • Reduce the Essence gained to 50 (from 100).
  • Spawn the elderling on a random planet, instead of the current one.
  • Thanks to tom.prince for updating!

Beta 3.761 Savegame Safety

(Released December 3rd, 2021)

"Total Conversion" Mod Support

• The game now supports "total conversion" mods.
  • A total conversion mod is a singular thing that ignores absolutely all of our xml, and has the option of loading a whole bunch of other code. This is not a mod that plays well with other mods. You can only have one total conversion mod on at a time.
  • Total conversion mods must currently be enabled through the filesystem, using the TotalConversionModOption.txt file in the PlayerData folder. If we actually start having some total conversion mods (Puffin has been working on one for a long time), then I'll make an interface for enabling these in-game.
    • Realistically, any total conversion mod is going to be going through a lot of growing pains until we're closer to being done with the Complete Edition of the game, so mostly this is just setup so that folks can do more with it in 2022, although it is functional now.
  • A total conversion mod loads none of the normal mods, however it does have support for having sub-mods that are created like anyone else.
  • Your settings and player data is completely separate between the normal game and any total conversions you use, so you don't have to worry about incompatibilities or one thing corrupting the other.
  • Total conversions don't "see" DLCs that you have installed in the traditional way, but they can pull art and music and such out of them just like normal, making things very convenient.
  • Total conversions don't "see" any code from normal game mods, nor do they "see" any xml outside of their own total conversion folder. It is totally possible to make a version of your mod for the main game and for a popular total conversion, but for many good reasons you'd likely have to maintain those separately.
  • Thanks to RocketAssistedPuffin for requesting.

Other Modding Extensions

• There is a new ExtraDamageModifierTypeData, which allows mods to interject extra logic in the "how much damage does a weapon do against a target" code.
  • This is interjected in such a way that targeting logic will pick it up, meaning that if you give bonuses or penalties to ships, autotargeting will "just work" and know what it's doing.
  • This requires you to compile your own external C# dll to apply whatever logic you have in mind, but you declare basically a single class that inherits from IExtraDamageModifierImplementation and then gets to evaluate any time any unit does any damage to another unit.
    • If you're not intending to modify the output of all units, then just return the same value that is passed in to you on any given cases that are not relevant to you. The game has no idea what is relevant to you, so it just runs everything by you to see what you want to do.
    • The method signature is FInt ApplyAnyDamageMultipliersAgainst( FInt NearFinalResult, GameEntity_Squad Target, ArcenCharacterBuffer tracingBuffer, bool IncludeOutsideFactors, int OverridingDamage, int forCompressedShots )
  • The folder for the xml for this is ExtraDamageModifiers. This can be defined in a mod or in the base game or DLC or whatever. Right now we don't use it.
    • For example xml, see the TargetEvaluator folder, because we're not actually using this. This is just for mods.
      • For example code that implements ITargetEvaluatorImplementation and its CalculateNormalizedAdditionToCoreImportance in this same sort of singleton fashion, see the open source parts of the game that that xml references.
  • This whole thing is untested, but is very simple code so there's no reason it should not work.
    • This wasn't the highest-priority thing for me to work on today, but Puffin and I were talking about it, and at this point it took 10 minutes to implement and it lets me get this out of my brain as one less thing to have to remember, so here it is.
  • Thanks to Puffin for requesting.

Ark Empire

• The various Arks for the Ark Empire now produce 2200 (+400 per mark) metal rather than 1500 (+300), and 350k (+25k) energy rather than 50k (+10k).
  • This makes them all the equivalent of the home command station when it is not in distributed economy mode. For the other command stations and such, distributed economy on or off will still work for Ark Empires, but I don't feel like the starting Ark getting changes is really a required thing.

• The metal storage for the empire arks is now equal to that of the home command station, making them much more viable to play.
  • Thanks to Puffin for reporting how much of a problem the lack of that was.

Expert Mode

• Added a new "Command Station Direct Upgrade Costs More" option in the Human Balance section of galaxy options which is normally optional, but becomes mandatory in expert mode.
  • Normally, command station upgrades cost 500, 1000, 1500, 2500, 2500, and then 3000. So the total cost of upgrading to mark 7 is a whopping 10,100, but the early marks are extraordinarily cheap.
  • With this option on, the first two direct upgrades to command stations cost 1500, and then the rest cost 2500. So to get to mark 7 costs more -- 13,000 science -- but the first couple of marks is not quite as severe a jump as the rest.
  • This setting defaults to off in the simpler campaign types, but then becomes mandatory.
  • Thanks to Strategic Sage for suggesting (although I did adjust the first two marks down from his suggestion of 2500 to instead be 1500, so people might still do it some. We'll see what testing brings).

Fuel!

• Three kinds of fuel are now defined for later use in expert mode (sooner than later): Argon, Radon, and Xenon.
  • Tom had already started on the fuel implementation, which is SUPER helpful, but now this is split out by type.
  • Immobile combatants all now use Radon in the amount of what energy requirements would normally be. We can later override this as desired.
  • Officer ships (arks, golems, spire lone wolves) and elites (destroyers and cruisers) do the same thing, but with Xenon.
  • All other mobile ships (so strikecraft, frigates, etc) also do the same thing, but with Argon.
  • The various maluses that Tom had put in (not being able to build more of a thing, having the thing become weaker in attack and defense, etc) all still apply when you are in negative fuel of whichever sort.

• The icon for necromancer essence has been integrated into the game, along with a color for it. It's still in the ugly temp spot on the resource bar, but that's okay.
  • Same deal for the three fuel icons. Those are still also in the not-awesome spot way over to the side, but they use their icons and colors now at least, and also are split out into three properly.
  • Later these things will have fully proper displays, their own tooltips, etc.

• Home command stations and home arks now produce 100k each of argon and radon, but no xenon.
  • Some of the capturables out in the galaxy now produce a substantial amount of xenon, but none produce radon.
  • The main sources of further resources are meant to be new structures that are captured, but these things get you off to a start.
    • It is intentional that additional fuel (of any sort) is NOT produced by upgrading your command stations.

• New option in the galaxy options tab, under Balance, Human Balance: Human Ships Use Fuel
  • This has been moved from Tom's Experimental Fuel mod into the main game now, and is forced on for Expert mode.
  • Description:
    • This is a feature with HUGE impact on the overall flow of the game. It's required for Expert mode and up, but you can use it at any level.
    • This is probably the most-core feature of Expert mode, and it essentially gives you a variety of permament-consequence weakspots that the AI can attack, as well as a bit more to think about when deciding how to expand your empire.
    • With fuel enabled, you not only have to think about Metal and Energy for your ships, but you also have to think about fuel.
    • Who uses fuel? Most small and midsize military craft consume Argon, which is relatively common. Officers (Arks and Golems) and Elites (destroyers and cruisers) use Xenon, which is harder to come by. Turrets and forcefields and simiar use Radon, which is about as common as Argon.
    • Where do you get fuel? You start with a baseline of Argon and Radon. You'll need to capture more by taking planets and holding these structures from them. These structures are also where Xenon comes from.
    • The balance on this is INCREDIBLY preliminary right now -- it's basically a proof of concept. The UI is also not complete at all. But it's a start, for testing purposes.
  • Thanks to tom.prince for the initial implementation of this, and to Strategic Sage for designing this with me all the way back in May or so.

• New option in the galaxy options tab, under Balance, Human Balance: Can use more than available fuel
  • This has been moved from Tom's Experimental Fuel mod into the main game now, and is forced off for Expert mode.
  • Description:
    • Defaults to off, and on Expert and above it is forced to be off.
    • Normally when you are out of fuel of a certain sort, you cannot build any more ships that rely on that fuel. With this setting on, you can over-use fuel.
    • Whether this option is on or off, you may find yourself short on fuel (usually because of loss of a fuel structure). In those situations, the ships using that fuel get weaker hulls and shields, and weaker attack power, which gets worse depending on how large your fuel crisis is.
    • A fuel crisis is a problem for you no matter what, but this option lets you intentionally put yourself into a fuel crisis, or still build ships of the affected sort while in a fuel crisis.
  • Thanks to tom.prince for adding this!

• Minor and Major fuel stations are now seeded semi-randomly throughout the galaxy for all three of the fuel types. There's typically at most one per planet, but a 10% chance of seeing two per planet.
  • The minor fuel stations have the benefit of being mobile (but very slow), so you can move them wherever you need to.
  • The major fuel station provide way more resources, but are completely immobile.
  • In the case of both kinds, if they are destroyed, they are gone forever and you need to capture a new one.
  • This hasn't been tested yet, and is basically hot off the presses. The balance is probably incredibly off, but it's a starting point. There may be other things mechanically wrong, and the interface is definitely not ideal yet.
    • With that said, we look forward to the earliest feedback on this, and hope that this first version provides an interesting first look at a way that this can evolve quickly over the next few weeks.
  • Big thanks to Strategic Sage for helping design this, almost half a year ago now.

DLC 3

• Necromancer Upgrades can now be gated on the following:
  • Flagship level: No flagship below X can claim this upgrade
  • Min Necropolis Level: A player must have a necropolis at this level or above to claim this upgrade; the flagship doesn't matter
  • This is necessary because flagships and necropoleis can be at different mark levels
    • Thanks to Tom for pointing this out to Badger, and Badger for updating.

• The player must capture and then hack showdown devices to trigger the showdown with the AI.
  • Thanks to Badger for updating!

• Mark Necromancer Phylactery as factory for Necromancer ships.
  • Thanks to tom.prince for updating!

• Always show Necromancer flagships, even on planets without intel.
  • Without this, crippled flagships could disappear when crippled.
  • Thanks to tom.prince for updating!

• Include a long description for the phylactery.
  • Thanks to tom.prince for updating!

• Make the essence option from the Templar Rifts always the first option.
  • Thanks to tom.prince for updating!

• Don't offer Flagship blueprints that the Necromancer has already received in Rifts.
  • Thanks to tom.prince for updating!

• Add a NecromanceFeedsFleet tag to all buildings that provide ships to a bolstered fleet.
  • Thanks to tom.prince for updating!

• Reduce the range that Templar castles will defend.
  • If there are a string of systems, all with castles, it can be overwhelming to try and make your way through those systems, if all castles within range 5 will defend each planet.
  • This redueces the distance they will defend at max tier to be 3, as a first step. I'm not sure what the right value is here (I had tried 1/1/2, which seemed difficult but not completely overwhelming).
  • Thanks to tom.prince for updating!

• The range of templar structures is now in the TemplarDifficulty.
  • Thanks to Badger for updating!

• Necromancer Lobby improvements
  • The Necromancer now can choose its starting skeleton or wight in the game lobby
  • The Necromancer now has an option to start with every ship line. Intended to testing or sandbox play.
    • Added by Badger

Icon Work!

• The Zenith Architrave Portal and Warp Point now have revised icons in DLC2, because the main icon that the Portal was based on work so much better for the necromancer in DLC3. Now they all look quite cool.
  • The lesser icon overlay for these is now also used for some of the necromancer skeletons in DLC3.

• 3 new icons for the new fuels (Argon, Radon, and Xenon) for expert mode are ready to go, but not used in the game itself yet.
  • These are both in the format of production facility ship icons, and for ui use.

• The necromancer Essence resource also is now in the game in both formats mentioned for the fuels, but also not integrated yet.
  • I have some work to do with the top bar (the resource bar) to make it dynamic for both of these purposes. I was going to do that today, but decided to do more icons instead while I was in that mode.

• 18 new icons for the templars are now in place.

• The overlays for the lesser and greater spire fortress and cradle are now much more pleasant.

• 7 new unused ship icons are now in place (may be used later for DLC3 or DLC1 additions, or else just remain available for modders, depending).

• 11 new overlays are now in place for the various kinds of skeleton the neceromancer has, plus 2 icons for skeleton-related things.

• 11 new overlays are now in place for the various kinds of wight the neceromancer has, plus 2 icons for wight-related things.

• 21 other icons are now in place for the various other units of the necromancer, and 13 icon overlays as well.

• All of the Necromancer and Templar icons are now complete and integrated, although I might swap out and/or improve some of them with time and testing.

• Showdown device progress by Badger.

• The Necromancer can now choose their starting Skeleton or Wight in the game lobby.
  • Add a "Please give me all upgrades at the start" option for the necromancer. This is intended for testing or sandbox play
  • Thanks to Badger for adding!

Bugfixes / Tweaks

• Adjusted some serialization code to be slightly more optimal. It breaks saves, but this should be the last day of that.

• Some old code relating to alternate player data folders (this was for purposes of multiple copies of the game running on one computer to play multiplayer with themselves in prior games to AI War 2) have been removed.

• Fixed 2 bugs with the EntityTypeDrawingBag:
  • It was impossible to create ETDBs with subcategories (both for AI Budgets and Fleet Design Templates), due to a bug where the subcategory lists were never created and actually null.
  • The XML field "periodic_spawn_faction_for_unit" was accidentally capitalized, which wasn't as much a bug as it was simply very confusing.
    • Thanks to tadrinth for finding, and N RSirLimbo for fixing!

• Treat everything that has factory metal flows as a factory.
  • In particular, this makes the Necromancer Phylactery show factory status in its hover text.
  • Thanks to tom.prince for updating!

• Don't allow things that have died to remains to build drones.
  • This is relevant for the buildins the Necromancer hacks from the Templar.
  • Thanks to tom.prince for updating!

• data_extension on GameEntityTypeData is now able to be added properly to shots and wormholes, not just ships/squads/structures.
  • Thanks to Puffin for requesting.

Beta 3.760 Chaotic Maps And Ark Empires

(Released November 30th, 2021)

New DLC1 Map Type: Chaotic

• Add Chatoic map type.
  • Planets are chaoticly arranged, with not planar links between them.
  • Note: Due to the highly interconnected nature of this map, it can be hard to judge proximity.
  • Thanks to tom.prince for adding this new feature!!

QoL

• You can now toggle quickstarts as Cleared, so you can keep track of which ones you've done!
  • Requested by many people, thanks to Badger for the code.

• Display necromance essence on the resource bar.
  • This is ugly, as it displays it in the empty space on the right of the bar, but provides an at-a-glance view.
  • Thanks to tom.prince for adding!

• Show more detail in skeleton/wight home structures.
  • Thanks to tom.prince for adding.

• Periodic spawning (Raid Engines, Brutal Guardian Lairs, etc) notifications now have the following priority level:
  • Starts out at 0 (informational).
  • If it's the local player, increase by 2. Otherwise, if the target spawned against a faction allied to the local player (including other players) increase by 1.
  • If the spawn is within the next 10 seconds or 20% of the timer, whichever is less, incrase by 2. Otherwise, if the spawn is within the next 60 seconds or 5% of the timer, whichever is less, increase by 1.
  • This way periodic spawning notifications dynamically adjust their priority based on the situation.
    • Thanks to Badger for suggesting, and NR SirLimbo for adding!

• Include strength of stacks in planet sidebar strength calculations.
  • Thanks to tom.prince for adding!

Balance

• If you had DLC2 installed, then previously the golems and Arks from the base game and DLC1 had different stats.
  • This was something we had tried out while we were rebalancing DLC2, but never meant to leave it exclusive to that. The following changes have now been made to the core content:
    • Base Game:
      • Artillitery Golem can be hacked for increased weapon power OR increased durability (I believe in DLC2 it was just weapon power, and in base game just durability)
      • Regenerator Golem deals health change per Vampiric Doubleshot from 1 to 0.5.
      • Cursed Golem shots per salvo at mark 1 down from 60 to 40, but gains 20 shots per salvo per mark. Paralysis effect twice as effective. Claim cost and metal cost from 2m to 5m. Energy usage from 200k to 50k. Secondary weapon gains +1 shot per salvo per mark, and does twice as much paralysis damage.
      • Botnet Golem stats were unchanged and so are unchanged.
      • Black Widow Golem can be hacked for increased weapon power OR increased durability (I believe in DLC2 it was just weapon power, and in base game just durability). Shots per salvo at mark 1 down from 80 to 40, but gains 20 per mark now. Also it now jams weapons rather than disrupts engines, and its tractor beam can grab up to albedo 0.5 instead of 0.4.
      • Hive Golem from 1.4m to 1.8m health, shields 0.7m to 0.9m, drone build speed from 7 to 4 at mark 1, but +2 per mark. Also 1 added shot per salvo as mark level increases.
      • AI variants of all golems get 1.33x to all costs, 1.5x to all healths, and 1.5x to all weapons.
      • Rescue Spore Drone moved from DLC2 to base game.
      • Orchid Ark gains a rescue spore drone launcher, and repair speeds 5x higher.
      • Ark One health from 0.65m to 0.5m, and shields from 0.35m to 0.5m. Shield stays larger, unlike what DLC2 had before. Range on the spider cannon increased.
      • Gyrn, the Voidhome energy production from 150k to 500k, and energy consumption from 100k to 0. Damage lowered from 16k to 10k on main weapon, now has a 2x bonus against ships of at least mass 5 tX. Factory construction speed slightly lower at mark 1, but increases faster as marks go up.
      • Thanatos Ark main weapon damage lowered from 12k to 8k, now has a 2x bonus against ships of at least mass 5 tX.
    • DLC1:
      • Belle Prime Ark strength multiplier is now 5%, meaning it does not look so overpowered. This is just for display and AI purposes, not actual damage output. Main weapon damage from 400 to 300, but the multipliers for damage based on time at a planet greatly increased and expanded. It's ability to ambush or deal with ambushers is higher than ever.
      • Great A'Thomek Ark now has the Distracting Metallics system, which pulls in shots to itself that were headed toward other friendlies.
      • Nodorian Tortoid Ark strength multiplier is now 200%, meaning it does not look so underpowered. This is just for display and AI purposes, not actual damage output.
      • Sol Ater Ark gains a cloaking device. Its main weapon loses a lot of power, but gains a 4x bonus if it has been on the current planet for at least 90 seconds. It also gets a further multiplier against enemies that are new here. Also it is an even better hacker now, with the AI response dropping from 85% to 70%.
      • Grand Salvage Ark health and shields up about 20%, and rather than its primary weapon doing AOE knockback damage to up to 50 ships in an area, it now fires salvos of 50 shots that knockback; this is notably more effective. It's secondary weapon also now has some paralysis effects.
  • Thanks to ArnaudB for adding all of these changes, and to Puffin for noticing that these were accidentally gated behind DLC2.

Improvements

• In the tooltips for the player type, it now shows the details for each player type.

• If there are problems that would prevent you from starting the game, it now says so in the lobby in the upper right hand corner, and also says so if you click Start Game.

• Several pieces of code that were all lumped together in one file in mapgen have been split out into multiple files for easier management.

• There was some very complicated and gross code related to the way that I added the randomization option for the starting player fleets and such. I've stripped out all of that, and made it so that Core Table Subsets and other similar things can now specify "I have a random option and this is its text" without having to create junk data.
  • Additionally, there was some severe redundancy in fields with the starting fleets and such, and that has all been cleaned up. This is all mod-friendly style stuff now, and also makes it much easier for me to implement what I want tomorrow for the Ark Empire, etc.

• The support fleets now have better names that are actually more brief and thematic, rather than long and descriptive.

• All of the starting fleets that you wind up with (battlestations, starting fleet, and support fleet) now use the name of their actual fleet template when you start the game. Previously, it was just using randomly assigned names based on your starting planet, which was not helpful at all.
  • See, we frequently choose a random battlestation in particular, and if you did that, then there was no way to ever know which one you had chosen and if you really liked it or what. Now you can actually tell all these fleets apart when you use them, which feels really good.

• The game now loads every GameEntityTypeData list of units with a tag into a "core table subset" with a name of "ETag_[TagName]"
  • What this allows for is dropdowns in the settings menu or faction controls that contain specific lists of units.
  • If you wish to have a random option, you can use the new random name, displayname, and description tags on the field itself.
  • This is a pretty major expansion of flexibility that can be used for many things in mods and in the main game itself!

• There is a new galaxy option in Advanced Dangers under AI Behavior called "Brutal Guardian Lairs" if you have DLC3 installed.
  • This is OFF by default, as we never want to make the game just blanket harder because you install some DLC. However, it is currently forced on for Expert mode and above. We're open to feedback on that decision, but it seems interesting.
  • Description (may need some work):
    • Enabling this makes it so that the AI will spawn 3 Brutal Guardian Lairs somewhere in the galaxy, at least 5 hops from the player homeworld(s). Currently they may not even spawn on maps such as X if the player is in the center.
    • Once the 5-minute time has run out a Brutal Guardian will spawn for the Hunter Fleet of the AI owning the Lair.
    • Once the spawning has started, it will not stop until the Lair is destroyed. Killing it will inflict 15 AIP, but also grant 500 science and 15 hacking points. The Lair will also self-destruct if the AI command station of its planet dies.
  • Thanks to NR SirLimbo for adding this cool feature!
• More Brutal Guardian work:
  • The Amalgam Gamma variant now has its own icon overlay, distinct from the Beta variant.
  • Added the Amalgam Delta variant: A fusion of Dire Stealth and Plasma, and Heavy Beam and Shredder guardians. Do not let it near your command stations and anything else that relies on shields and/or forcefields to protect itself.
  • Thanks to NR SirLimbo for updating!

• Removed the AgileCursedGolem, which was unused.

• The start of renaming a lot of the systems of ships to be shorter names that don't mean anything, other than just what weapon they are (W1, W2, etc).
  • Essentially, this is the internal name that is never seen. The visible name is frequently very different from the underlying name, which is just used for matching to and upgrading on. It makes savegames smaller to have these be just symbolic, and it also makes the xml easier to read.

• The rest of the systems in the game (all 1400 or so of them) have all been given one or two letter designations.
  • The game will now throw errors on load if a mod or dlc or whatever tries to use a system with a longer name (again, this is the internal designation, not the visible display name).
  • To reiterate, this is about performance and clarity.
    • Clarity: One thing that can never change about a system is its name. So if we start out with a system named FusionBlaster as its internal name, it must stay that forever, even if we rename the actual system to be "Fusion Cutter" or "Thor's Giant Smiting Hand" or whatever. It's actually remarkable how confusing it can be to see something that seems meaningful (FusionBlaster) in the xml for the internal name, and then to see the real name (Arcane Saw or whatever) is something completely different. It's jarring and makes it hard to read. Seeing things like W1 and W2 instead of that is a bit easier, because that's clearly just a designation -- "weapon 1, weapon 2, okay, what is weapon 1 called?" At no point do I get confused if the name is "W1" or "icy black hand of death."
      • I would have preferred to use a very consistent naming scheme, but unfortunately that would have led to more errors, particularly in places where there are a ton of copy-from descendants who then modify what their parent had. So there are a number of designations like B1 through D9, and same with V and Q and R, and those have no special meaning. I also used W and then every letter of the alphabet following it.
  • Performance:
    • We have to store these names somewhere in order to make sure we map to them correctly (the internal designations, not the longer players-see-it version of names). That list of names is then in every savegame, and transmitted over the network every time. That list is the full list of possible system names from any unit in the game, cobbled down. If there are 600ish unique names, averaging about 8 characters each, and each character takes us about 1 byte to represent in our format, then that's 4800 bytes, or 4.6kb. That's not super duper sizeable. But that was because we had a lot of reuse. If modders take to adding lots of custom names, and don't realize that the internal designations can be as simple as W1, W2, W3 for all the cool things they come up with, then it's really possible to see this sort of data just skyrocket in size. (I mean, what would we ever do if the game had a one-time transmission and storage of... 40 kb! Yes, I am joking. This doesn't save remotely as much data as I'd like, even projected forward.)
  • So I guess this was just about clarity with all the frequent renames we do. It's enormously common for us to create a new ship with some generic copies of some weapon, and then later give it unique names and art. Having old and mismatched internal names was not a welcome part of that process, and it was everpresent and confusing. Since changing those things breaks saves, this was my last chance to rectify it.

• Hacks and techs are now serialized by index rather than name. They just barely cross the line to where they are more efficient to serialize that way. All the other things that are serialized by name are so rarely done that to store the lookup would be larger than the data of just sending the single name used.

• ReactToHumansGettingFirstIntelOfPlanet is now actually called properly on the DeepInfo of factions when appropriate. And older logic that lets the dyson sphere and nanocaust react to this, or play an audio notification, are now triggered in the new format by this.

Player-Type Related

• A new IPlayerTypeSpecificCode provides a linkage (rather like description appenders or similar) where some worker class can provide playertype-specific code.
  • This is separate from the actual BaseInfo/DeepInfo that player types use because we may need to call this sort of code from all sorts of places, and there's also no good reason that those factions can't otherwise just share a BaseInfo or DeepInfo in some cases. In other words, this leads to simpler code and more readable code.
  • That said, in the end we wound up needing to make a BaseInfo and DeepInfo version of these code executor interfaces.

• There was still a bunch of remnant code for a "solo ark" faction that the game was going to have. That has all been removed. There will be a solo ark player type coming up soon, but it won't require any of the mechanics that were in our other faction code for it, and it will instead require some other new bits.

• Some of the game logic for the earliest parts of the necromancer faction has been split out into the new IPlayerTypeSpecificCode code.
  • This is cleaner as a place for the logic in general, but it also is going to differ between lone necromancer and necromancer empire, so it's good to have separated anyhow.

• Lone Necromancers are no longer auto-seeded on a random human empire starting planet. After all, the lone necromancer can exist now in games without any human empire present at all.
  • They seed on whatever planet you've selected for them in the lobby, and further details will be coming for them in the near future.

• Overall the various player types are closer now to seeding the correct things on campaign start now, but it's still not quite right. Should be tomorrow for that!
  • At the moment, from what I can tell, the lobby behavior is correct for the various player types, except for the bit about not yet having all of the new lobby options I want to add for some of the new types.

• Adjust the logic for notifications of homeworld attack to handle Home Arks.
  • This adjust both the notification, and the resource bar icon, to notify if any planet with a unit that causes a loss is under attack. This means:
    • If your ark is on a hostile planet, you will get a notification
    • If your starting planet is under attack, but doesn't have your ark, you won't get a notification.
  • Thanks to tom.prince for updating!

• Add `uses_tech_upgrades_from_player_types` to player types.
  • This allows related player types to share tech upgrades, without needing to modify the base records of the tech upgrade.
  • In particular, this allows player types added in expansions and mods to use tech upgrades from base game player types, without needing to modify/enumerate them.
  • Thanks to tom.prince for adding!

• Allow necropolises to control planets.
  • This add a city_centers_can_control setting to player types, that allows city centers of those players to control planets.
  • Thanks to tom.prince for adding!

• Allow GameEnitity types to explictly mark themselves as king units.
  • In particular, this makes the Home Necropolis a king unit (without changing it from a city center), which allows Neinzul Sappers to spawn on the homeworld of a Solo necromancer.
  • Thanks to tom.prince for adding!

• Split out a few DLC3 faction xml files for easier editing. Same with some of the other partial-record additions for DLC2.

• requires_player_type on custom fields is now requires_player_types, a list of multiple player types. This allows for some fields to be shared between player types.
  • This is most useful for some of the player types that will just be slight variations on one another. Specifically the Necromancer Sidekick, which is easier for players to find and understand when it's a distinct type rather than just having it as an option under the lone necromancer itself.

• Added can_lose_game on player types, which is normally true. But for helper-only or spectator factions, this is set to false because essentially they cannot lose at all.

• WriteTextToSecondLineOfLeftSidebarInLobby is now implemented on the IPlayerTypeSpecificCodeBaseInfo, so that each player type uses that instead of the equivalent function from ExternalFactionBaseInfo.
  • Also added WriteFactionTooltipForSidebarInLobby, which handles mouseover of the faction tabs. Additionally it now just writes the player type name and description, and if there are any obvious problems with the current setup that are identifiable at the playertype-level (without custom code), then it goes ahead and writes all of that in the tooltip now, too.

• A new GetDisplayNameWithoutPlayerNames() has been added on faction and faction config, and most places in the code now use that instead of the older SpecialFactionData.DisplayName.
  • Essentially, this gets either the faction name or the player type name if it's a human faction.

• The game no longer requires at least one human player homeworld in order for the game to proceed.

• The game really does need at least one human faction if you want to be a spectator, but it now just adds that in belatedly if there are no human factions listed.
  • This allows for the same user experience regarding spectators in multiplayer and single-player, while keeping things as-required on the behind the scenes side.

• Added prevents_game_win_or_loss, reveals_entire_map_at_start, and must_be_sole_player_faction, all of which combine to complete the spectator player type's functionality in the base game.

• Player factions are always now removable. Previously, the first player faction could never be removed. It is now possible to start a game with no player factions at all.

• The Spectator player type is now fully complete, making it the second player type to reach that status (behind human empire).

• Added a new hide_all_faction_fields_other_than_player_type field for player types, which is only useful for the spectator. This basically just keeps the spectator interface clean, since there's nothing relevant beyond "I am a spectator"
  • This fully completes the spectator player type, so that it's as simple and clean as I can make it.

• For factions that cannot lose (right now only the necro sidekick), mapgen now will bar them from starting the game if they aren't partnered with another player who can actually lose. Otherwise the game itself can't be lost, which is obviously nonideal.

• For factions that require vassals in order to play, the game now prevents starting when there are not enough vassals set up. It also gives a list of available vassals so that the player doesn't have to wonder which ones are available for that.

• In the faction setup, when you have at least one faction selected that requires vassals (aka Suzerain is the only one for now), then the Vassal option no longer is hidden under advanced options.
  • Additionally, in the list of potential factions under Add Faction in this same circumstance, it shows the text "Can Be Vassal" next to each valid option.

• When factions are selected to be a vassal, they are now forced to an allegiance of Friendly To Players, and they also have their Invasion Time forced to Immediate, if relevant.
  • Additionally, these fields just no longer appear on the interface for factions once you have made them a vassal, as they are not relevant.

• Vassals now automatically share their vision with human players, regardless of other settings.

• On the sidebar in the lobby, vassals now show up with their allegiance as vassal, rather than Friendly To Players.

• The Dark Zenith "Spawning Options" field has been renamed to properly be "Invasion Time"
  • This allows for it to be properly overridden and hidden, etc.

• Allow player types to specify other player types they use hacks from.
  • This allows Expansion and Mod player types to use hacks from the base game, without needing to list them in the base game hacking types.
  • Use this to remove access to a bunch of hacks from the Necromancer.
    • All the hacks that grant ship lines or tech, since they are not relevant to the necromancer.
    • Spire Archive and hacks to sabatoge AI structures, for balance reasons, since the Necromancer currently has effectively unlimited hacking points.
    • I had originally considered removing a bunch of minor faction hacks for similiar balance reasons, but decided to leave them in for now. (Note that these are only available to the Necromance Empire already)
  • Thanks to tom.prince for adding!

• Make all the upgrade/build Necromancer hacks complete instantly.
  • Thanks to tom.prince for updating!

• Get rid of the special case for building things in cities.
  • This removes the granted_by_any_custom_fleets attribute of build sidebar categories. It seems that this exist (at least after the city refactor) that the only reason this was necessary is because spire hubs were not considered `BuildMenuPopulator`s. However, that turns out to be because they did not grant any fleet design templates or logic; but there is no reason to have that limition.
  • Thus, by removing the check for fleet design templates/logic, we can remove the special case code.
  • It also turns out that the city logic had slightly different logic (it only showed things with a non-zero *explicit* cap, as opposed to a *effective* cap. I discovered this when I removed the unused fleet design for necropolises, and noticed some of the buildings could no longer be built, despite the fleet design being empty.
  • Thanks to tom.prince for updating!

• Rework how bolstering is handled, to use external fleet base info.
  • When I originally added support for changing which fleet is bolstered, I used base-info on the centerpiece to stand-in for fleet base-info. Now that we can put base-info on fleets, it make more sense to use that directly.
  • While we are touching this code, make it so that only minor necropolises can change which fleet they are bolstering. Major necropolises always bolster the fleet the spawn, and defensive necropolis don't bolster anything.
  • Thanks to tom.prince for updating!

• Only show bolstering UI if there is a bolstered fleet, or the city can change which fleet is bolstered.
  • Thanks to tom.prince for updating!

• Refactor calculation of bolstered fleets contents, to share code between Spire and Necromancer.
  • This also adjust some code in both of those factions so that they don't get as confused if there is a necromancer in a spire campaign. I don't know if I addressed all the issues, but I fixed the ones I ran into.
  • Thanks to tom.prince for updating!

• Scourge: add an option to smooth upgrading out. Intended for puffin
  • Thanks to Badger for adding!

Ark Empire

• Ark Empires now spawn a new king-style Ark instead of a home command station and all the things that go with it.

• The Ark Empire now allows for you to select from a list of all the Arks available for that mode, or to have it assign you one at random (this is using the new ETag_ functionality).
  • Over time, I could use some help from volunteers/modders/any-players who might have feelings about adjustments to the tooltip descriptions for these versions of the Arks, plus health or weaponry boosts for them.
    • Right now not all Arks are equal, and they probably never will be, but giving players more information (and/or somewhat closing the gap in stats) while at the same time not giving them the full insanely huge tooltip would be the goal. Things like "power level 1/5, 3/5, etc" might be one way to do it if people are thinking about it. But also closing the stats gap some, just for the Ark-Empire versions (not the main capturable versions).

• Add some energy and metal production to Home Arks (for the Ark Empire).
  • In particular, without energy, you will immediately enter brownout at the beginning of the game.
  • Also, mark Home Arks as causing a loss, if they die.
  • Thanks to tom.prince for adding!

• Ark Empires are not ready yet, but they are getting closer! You could play with them and potentially win, although it's not balanced or fully set up yet.
  • Your initial fleet for your Ark now comes complete with five lines of ships, which are individually something you can choose in the lobby.
    • This is a build-your-own-fleet feature, which is pretty darn exciting, I have to say. I'd love more input from folks on what they'd like to see as options for each of the five categories, but I started out with a pretty good starting selection just pulling from the base game.
      • I've been really tempted to add in a frigate line, and maybe even... a cruiser line also if you have DLC2. The problem with the cruiser line is that it would be conditional on DLC2, and thus would make balance shift SHARPLY if it was not present. So maybe just frigates instead, or there's a 6th line which either has stupidly-high caps of frigates and relatively-low caps of cruisers to choose from?
    • Each of the five categories has specific ship lines that are set up via tags in xml (aka, they are easy to mod, easy to add to without code), and this actually demonstrates using custom xml embedded on units (which could be handled via a partial record if we wanted to, but is not for this case). This shows several really powerful things that would be useful for various mods in the future, in short.
  • In addition to the above, your starting ark now acts as a factory with double the power of a normal single factory, allowing for your ark to build for you no matter where you are.
  • Major items still to do for the Ark Empire:
    • More ship choices for the five lines, possibly including that frigate/cruiser mix as a sixth line. Help would be appreciated.
    • I need to get it showing something different for the player other than the custom fleet in the left tab of the factions screen, since that is not even relevant for the Ark Empire.
    • You should have to fight to take your starting planet, versus it being cleared out.
    • The game should not consider you deep-striking in the early game of this player type.
    • The amount of metal and energy produced by the empire versions of these arks needs a look. Help is also welcome on that.
    • Ark Empires need to have some pop-out mobile home-style defenses that they can use as-desired, like forcefields. I need to add a few features for that.
    • After this short list of items, I believe the Ark Empire is fully ready to go!

Necromancer Changes

• The "Home Necropolis" for necromancers is now called a Phylactory. The actual logic for how these work in terms of hacking and such is not in place yet, but the description updates are in, along with the journal entry by Badger, and the name change.
  • Thanks to Zeus for coming up with the name, and Badger for suggesting the functionality.

• All of the necromancer journal entries are pretty much now for the tips tab, since they are more about advice than about lore dumps.

• The necromancer can now use a hack to move its Phylactery (ie its home necropolis). To move the player must pay Essence and have a target necropoolis that is not crippled. When it moves, the old necropolis moves swaps places with the phylactery. When this location change happens, all stationary structures for both are destroyed
  • Thanks to Badger for adding!

• More health to the necromancer flagships
  • Thanks to Badger for adding!

• Modify necromancer economy to give more income but also higher prices. The intent is to not change the current behaviour too much, but having larger numbers is more fun for players.
  • The Templar can now use 'Elite' units that don't grant resources to the necromancer once the necromancer gets strong enough. This is intended to reduce the extreme amounts of science the necromancer can get in the mid-late game
  • Thanks to Badger for adding!

• Give an Elderling a very expensive hack to summon a Transcendant Elderling; an extra scary elderling that can be hunted for a ton of Essence.
  • The goal is to give the Necromancer some fun things to spend their excess Hacking points on.
  • Thanks to Badger for adding!

• Some templar income nerfs. Phylactory movement should now happen automatically when it would die
  • Thanks to Badger for updating.

• The templar can't rebuild a defensive structure on a planet until 20 minutes have passed since the last time they owned it. This rule does not apply to structures built to defend a Rift
  • Continued work on Phylactery movement
  • The necromancer empire now correctly dies when its phylactery is crippled
  • Thanks to Badger for adding!

• The phylactery for necromancer empire should be shown on the galaxy map, since it can move between planets
  • Thanks to Badger for updating.

• Make necromancer upgrades a bit more expensive for the higher mark levels
  • Thanks to Badger for updating.

• Necromancer building improvements.
  • First, the Necromancer can now build Skeleton and Wight Drone producers. Hopefully this will go a long way toward giving the player defensive resources. The drones scale in power with your existing science options (So science invested into skeleton warriors will improve skeleton warrior drones).
  • Secondly, A Defensive necropolis now can only build defensive or utility structures. Now they actually have defensive options!
  • Thanks to Badger for adding!

• The necromancer drone producers are now only obtained by hacking Templar defenses. Chris has some plans for giving the necromancer other main necropolis defenses. This is now intended as a mechanism for letting players spend their extra hacking points
  • Thanks to Badger for adding!

New Specific Player Types

• A new Necromancer Sidekick player type (for DLC3) has been added, with the goal of this being a multiplayer-only thing for when a less-experienced player wants to be part of the game but not risk actually losing the game for the group.
  • Description:
    • The Necromancer Sidekick is a powerful, player contollable faction -- that cannot lose. It can only be used in tandem when allied with another player faction, typically in multiplayer.
    • The Necromancer has come to this region of the galaxy to harvest life force from Rifts, powerful energy sources, in order to make itself more powerful.
    • Playing as the Necromancer Sidekick will cause an additional copy of AI-Allied Elderlings to be spawn, as well as a unique faction called the Templar.
    • Using the Necromancer Sidekick is compatible with any playstyle, assuming that the sidekick isn't doing much. But if the sidekick is playing fully, then generally the AIP is going to be driven up a fair bit.
    • Please see the in-game Journals for how the necromancer faction works.

• The start of the Necromancer Empire player type (for DLC3) has been added, although bear in mind that it's not remotely ready just yet.
  • Description:
    • The Necromancer Empire is a powerful, non-traditional, empire-based player faction. It is not yet ready for testing -- not everything described in this tooltip yet exists!
    • The Necromancer has come to this region of the galaxy to harvest life force from Rifts, powerful energy sources, in order to make itself more powerful. Generally, a Necromancer must defeat their most powerful foes in order to become more powerful. Using the Necromancer Empire is not conducive to overly low-AIP playstyles.
    • Playing as the Necromancer Empire will cause an additional copy of AI-Allied Elderlings to be spawned, as well as a unique faction called the Templar.
    • The Necromancer Empire has both offensive and defensive components, but most of its defenses come from capturing AI guard posts and using them against their former masters. A Necromancer Empire is quite capable of standing on its own toe-to-toe with any enemies that a normal Human Empire could face, but is more complex to master.
    • Please see the in-game Journals for how the necromancer faction works.

• The necromancer commodore has been removed as a separate faction, because after discussion on discord we realized that basically it would just be a subset of the necromancer empire at this point.
  • Having the necromancer sidekick separate is still required because of its unique role in multiplayer, but other than that the way to play necromancer is just with the full necromancer empire, plus any options that may be desired to make things harder or easier. We don't need a separate player type for any of that stuff.

• Added the start of the last new DLC1 player type: Spire-Infused Empire
  • Description:
    • NOT YET FUNCTIONAL, JUST A STUB! Functions very much like a normal human empire, but also comes with spire cities and artifacts right from the start.
    • If you are familiar with the Fallen Spire faction, normally those act as a separate faction, and you can only grow them by gathering Spire Relics in various ways.
    • This faction skips the Spire Relic process, but keeps the rest of what makes Fallen Spire awesome:
    • If you like the quest-style attributes of Fallen Spire, then you should just use a regular human empire (or Ark Empire) and enable the Fallen Spire campaign.
    • If you just want Spire ships and a direct frontal assault type of game, then this is the faction for you.

• Added a new base game player type: Spectator
  • Description:
    • If you want to just see other factions duke it out, then this allows you to do so.
    • Please note: if you don't actually set up any enemies amongst other factions, you're going to have a boring time!
    • This player type is mostly useful for single-player. If you want to have a spectator in multiplayer, then just unassign that player's account from any player faction and they will be a spectator who can just view what the other player factions see.
    • By using this player faction, the game will be unable to be won or lost, and the entire game map will be visible from the start.

• Added in the last planned player type, the Suzerain (for DLC3), which I had forgotten to add the other day.
  • Description:
    • Humans exist only in a single Ark that cannot claim worlds or build other structures or ships. It's just you and your Ark for your faction.
    • For this reason, the Suzerain requires at least two Vassals to be chosen as subjects for it to order around, but you have the option of choosing more than two. Your victory hinges on your ability to effectively order around these NPCs.

Map Generation / Lobby Responsiveness

• No actual algorithmic changes yet, but all of the map configuration data -- what is needed to generate the structure of a map, but not its contents -- has been moved onto MapConfiguration. This is going to allow me to decouple the structural generation from its population in a way that will be great for lobby responsiveness, but step one is getting all the data into the proper encapsulated locations.
  • That in and of itself led to a bunch of changes in 26 files, and I didn't want to have that mixed up with actual algorithm changes in svn.

• There is a new thing called Map Populator in the galaxy options under MapGen:
  • Once the structure of a map is generated, it then needs to actually be filled with something. Normally there is a single central algorithm that handles this. However, it is possible for alternative algorithms to be added by mods, and you can then select those here.

• Related to that, there's a new MapPopulatorType xml folder, and there's a single populator in there called Standard.
  • The main reason for this is to make it so that mods can put in their own replacements if they want to do that.

• At this point, not much of the actual logic has been changed for map generation, but the population and the structural bits have been separated.
  • The goals of this are twofold.
    • Firstly, to create this separation so that soon we can run the structural parts only when those are changed (thus making the lobby more responsive).
    • Secondly, to make it so that our map population algorithm can be replaced by any modders who want to copy the existing one (it's open source) and make their own variant. Thanks to Puffin for requesting this part!

• MapPopulators have been added for the test chamber and tutorials, getting the test chamber back into functional format. Tutorials are still out of commission at the moment on the beta branch, but not because of this. Those are on my list for soon.

• PsuedoPlanetData is a data structure that we no longer needed, and it has been removed.

• There is now a CachedPlanetPoint and CachedPlanetConnection, which allow us to "regenerate" a galaxy for which the map structure has not changed in a very short amount of time by skipping the structure generation if we know it's just going to be a copy of the one we last cached this run.
  • Often this can cut generation time down from several hundred ms to just two dozen ms. Or in the case of excessively large galaxies with many factions, it can cut it down from multiple seconds to just a few.
  • Overall this creates a vast improvement in the responsiveness of adding factions and such in the lobby, but changing the map type or style or seed still has the same sort of wait as it did before.
  • For a complex map with 300 planets, on my machine this takes 9.5 seconds to generate the structure, but after that is done, it only takes 280ms per faction added or removed. On an 80 planet map, it's 400ms and 20ms, respectively. Prior to now it would take the larger number every time we did anything like adding or removing factions, which was very confusing and annoying.
  • Thanks to Eluthena for reminding us of how annoying it was to have these long waits on things like changing faction colors, etc.

• "Skipped an extra mapgen call that was substantially the same as one that was run X seconds ago" is no longer a feature (it's not needed).

• Map generation now actually happens on a background thread, which keeps the interface moving while it's doing what it does. This is very handy for the giant slow maps that sometimes people like to start.
  • This was long-since meant to be happening that way, but for some combination of reasons had changed so that we were not actually doing it that way anymore.

• "SerializeByIndex called on table with null IndexSerializationInfo" now makes an effort to fix itself before it happens.

• Removed some old "work thread" stuff that was still in place from Stars Beyond Reach, which we had intended to adapt into AI War 2, but wound up being set aside because we found even better ways to handle things. The upgrade to the newer version of .NET in 2017 was part of that.

• Engine_Universal.OnUpdateFromMainThread now gives more detailed error handling.
  • Same with UpdateGameController.
  • This was able to have some cross-threading errors now that I made these changes with mapgen. It only is a new risk during mapgen in the lobby itself, not during gameplay.

• The game now ignores some "UnityEngine.EventSystems.ExecuteEvents.GetEventList" exceptions that can happen during repeated mapgen clicks.
  • Also one with "UnityEngine.UI.Graphic.Raycast"

• The AIWarExternalCode dll (BaseInfo) now has a dependency on the ArcenAIW2Visualization dll.
  • There is no longer any base classes for the GalaxyMapPlanet and GalaxyMapPlanetLink.

• All of the code for setting up and updating GalaxyMapPlanets has been completely reorganized and all resides in a few simple to understand methods rather than being scattered around.
  • In general, previously the planets were responsible for updating the galaxy map view. Now the galaxy map view takes care of itself by examining the planets. This has many advantages.
    • One of the advantages is that the Galaxy object no longer needs to store NewPlanetsExploredIfNecessary, NomadHasMoved, NewPlanetsSpawned, or NomadHasMoved. Instead, the code is able to just notice these things directly.
  • Other bits of code are now just more readable in general, since they now happen with less indirection, meaning you can just read linearly what they do.
  • Another benefit of this is that the galaxy map no longer blinks out of existence and then back in wehn you're changing parameters on a map in the lobby. This makes it far easier to see what is happening with a map.

• Updated the lobby so that it now shows you (or the host in multiplayer, but not the clients) the progress of long mapgens as it gets the planets created and all the links populated.
  • This is great for long mapgens, such as the ones that take 15 seconds for 300 planets, although for the ones that take more like a third of a second it just feels flickery, so there's going to need to be a middle-ground here.

• The visuals of the lobby map generation now adaptively change their behavior based on how long it takes your specific computer to generate maps of a given type. This gives a certain amount of future-proofing so that the behavior will always make sense, and it also makes it so that it behaves properly even on very differently-powered CPUs..
  • Essentially, the game keeps track of the longest it took -- since starting the program today -- it has taken to generate the given map type you've chosen with roughly the same planet count.
    • If it doesn't know yet, or it's over 1 second, then it lets you watch the planets be assembled sooner than later, which gives you a view of what is happening as mapgen happens, versus the game just seeming to be frozen.
    • If it's doing a very quick regeneration of the existing structure, or past mapgens took less than 1 second, then it just freezes the view of the planets until they are finished generating. Usually this is something like 300ms, but sometimes even less. The goal here is to make it so that you don't see planets or links between them flickering around, and in particular so that as you change map options on relatively small maps, you can see the links move easily because you're not seeing the intermediate movements that otherwise would be visible.
  • So far as I can tell, so far this gives the optimal experience in all of the various settings combos I've been able to come up with.

• Adjusted the weights of the older galaxy map backgrounds so that they have more change to show up (or less in the case of one). All of Puffin's lobby BGs have a weight of 50k. Previously, all of my older ones had a weight of just 100. Now Black5 is 50k, SwirlyBlack is 10k, BlackHole is down to 10, and BrightBlack is 1k.
  • Thanks to Puffin for the help with this.

Bugfixes

• Fixed several bugs with dropdowns in general not showing tooltips properly when you hovered over them if they were closed. This was super annoying, and I seem to recall some folks complaining about it months ago, now.

• Updated ArcenExternalDllTypeInfoAndCache to be IClearableDuringXmlReload and thus properly reset itself during xml reload. Also solved a tiny memory leak relating to these on xml reloads.

• Fix the use of incorrect tracing buffer causing tracing to fail.
  • Thanks to tom.prince for the fix!

• Don't try to fire shots, if the number of shots has been reduced to 0.
  • This can happen if the target already has enough incoming shots to kill it.
  • Thanks to tom.prince for the fix!

• Put in a potential fix for Shot Compression killing all shots at a target due to the target being protected by a forcefield, but enough shots to kill the target if it weren't shielded are already on their way.
  • Thanks to tom.prince for finding and NR SirLimbo for fixing. Hopefully this fixes the bug, but only further testing will tell.

• Prevented Shot Compression from underestimating the damage of each shot due to rounding. This is only ever a single point of damage, but more precision is always good.
  • Thanks to tom.princefor figuring out the code could still do this and NR SirLimbo for fixing.

• Fix error when switch maptypes in lobby.
  • Thanks to tom.prince for fixing!

• Don't reset planet count when changing map type.
  • Thanks to tom.prince for fixing!

• If we are removing a SquadVisualizer because the squad is going back to the pool, clear the related entity.
  • I think this probably causes a bunch of errors; but, along with the next patch, seems to stop the visual clutter of stray ship circles, and dead entities that can be hovered over.
• As far as I can tell, the errors this produces are harmless to gameplay.
  • Thanks to tom.prince for fixing!

• Use LazyLoadSquadWrapper for references from the visualizer to squads.
  • This is the other part of the patch dealing with dead squads still being hoverable, and having ship circles.
  • I'm not sure if this is correct, but it seems like at least a reasonable stop-gap. In particular, when reviewing, I was unsure if the visual layer wants access to the squad, even when it is to be removed at end of frame[1].
  • Even though these patches causes a bunch of errors, I suspect it is worthwhile, as it makes those error easier to diagnose, since the accesses they represent would have caused issues anyway.
  • [1] Though I guess if it does want that, it would be possible to add an accessor to LazyLoadSquadWrapper that allowed access til at least end-of-frame.
    • Or to add a variant of LazyLoadSquadWrapper that had the behavior; and perhaps also doesn't fall back to looking up by primary key.
  • Thanks to tom.prince for fixing!

• Don't display 1000 as 1,0000.
  • Thanks to tom.prince for fixing and Ranakastrasz for reporting!

• Unity UI has been removed as a package from our code (it has not been updated since 2019), and has instead been directly integrated. This way we can make some adjustments to the source code in order to fix bugs as needed, since we're seeing a couple on the current internal build.

• Suppressed the "Internal error. Trying to destroy object that is already released to pool." error, and made it so that it now doesn't allow further execution of that action that would be erroneous. This is an error inside unity UI that is being triggered by something changing during a raycast, near as I can tell. Regardless, when it detects the error state, it's easy to just not follow through on actually doing something wrong, and not bother telling anyone about it since it becomes a non-issue.
  • Those using svn will need a fresh build of the game for their OS (specifically the UnityEngine.UI.dll in the Managed folder) before they'll stop seeing the error.

• Fixed some ErrorsReportedByEngine that were pointless and harmless.

• Fixed a couple of errors in ProtectedList that could happen when you tried to iterate beyond its proper range, among other things.

• Fixed a one-line typo in ToThrowawayListCanMemLeak_HugeWasteAndExpensive_AvoidIfAtAllPossible() where it was creating a throwaway list from a regular managed list, but then filling the contents of the managed list with the contents of the managed list rather than filling the throwaway list with the contents of the managed list.
  • This led to a pretty incredible infinite memory leak and crash to desktop in the rare cases where this is actually used, most notably in some of the map types. Swirl and linked rings were the main ones.
  • Thanks to Major Dullard, UFO, and Badger for reporting.

• ExternalizedPool has been removed, as thankfully nothing uses it anymore. It was used previously for some mapgen stuff, but I don't think it was working entirely correctly anyhow.

• The Dark Zenith per-unit base info is now properly being cleared between uses. It was not being properly cleaned up since the code rework until now.

• The Zenith Architrave now attritions or absorbs ships in BaseInfo rather than DeepInfo, to avoid conflicts in multiplayer.
  • The AI Reserves logic is more complicated, so it still damages ships in DeepInfo, but it flags them as needing sync to the client so that the client will find out immediately.
  • I evaluated the death effects because of their use of TakeDamageDirectly in DeepInfo, but they're already okay because they're in the process of being caught in fast-blast initial sends.
  • Zenith Miner transformations on the host are done in DeepInfo, but will be translated to clients faster. Another case where we have TakeDamageDirectly being called.
  • Neinzul Sappers now attrition their turrets on the client and the host in BaseInfo, to keep from having extra transmissions be required from them.
    • There was another case where the weakest sapper needs to take damage to build something, and this does need to stay in DeepInfo, but it's now flagged for immediate send of that data to the client so it will stay in sync.
  • I had flagged GetMaxStrengthForCastle as being a potential for moving to BaseInfo from DeepInfo for the templars, but in reality there's nothing there that is cause for that.
  • This concludes a full review of things that affect ship health in DeepInfo and might still need to move to BaseInfo. It's one of those things I had marked a few months ago to come back to, and now I have done so.

• Fix overlap in tooltip text for Spire(?) and Necromance structures grating ships.
  • Thanks to tom.prince for fixing, and Daniexpert for reporting.

• Add some missing tags for Necromancer flagships.
  • Thanks to tom.prince for fixing.

• Fix the type name for wight variants to transform into.
  • Thanks to tom.prince for fixing.

• Fix some indentation that was off by one space.
  • Thanks to tom.prince for fixing.

• Adjust the logic for allowing Necropolises to control planets, so allow for loss of control.
  • Previously it wouldn't detect if the city center was crippled.
  • Also remove the code for Necromancer to mark planets with influence. This is a left-over from when the faction was a minor faction. This prevented seeing the change in control in the UI.
  • That requires changing how Templars find a Necromancer to target. (This part is untested)
  • Thanks to tom.prince for fixing.

• Allow Templars to actually attack in the new setup.
  • Thanks to tom.prince for fixing.

• Remove Necromancer flagships from the Necropolis fleet membership, as it is meaningless.
  • This was a remenant of before cities were seperated from mobile fleets.
  • Thanks to tom.prince for fixing.

• Remove unused fleet design template for necropolises.
  • Thanks to tom.prince for removing.

• Fix spawning of initial necormancer fleet.
  • The code for spawning a necropolis was looking for specific tags to be passed to it, to detect whether it should spawn a fleet. However, the tags passed have changed a couple of times, breaking this. Instead, look for a specific (SpawnsNecromancerFleet) on the necropolis, rather than just the tag paased to pick it.
  • Thanks to tom.prince for fixing in the new setup.

• Fix a bug where the Praetorian Guard would try to kill the templar
  • Thanks to Badger for fixing.

• Properly track necromancer flagship upgrades, and respect ship cap increases for them.
  • Of note is that currently, ship cap increases of bodyguards impact lines from any bolstering necropolis, but only if that necropolis is providing *any* of that ship type already.
  • Thanks to tom.prince for fixing.

• Handle per-fleet upgrades of skeleton/wight soft caps.
  • Thanks to tom.prince for fixing.

• fix warp gate guardian timers
  • Thanks to tom.prince for fixing.

• Fix an annoying null reference exception in UpdateEntitiesShownAtPlanetDirectly
  • Thanks to Badger for fixing.

• Some minor tweaks to templar balance
  • Thanks to Badger for updating.

• Use GetStrengthOfSelfAndContents where we are currently open coding that calculation.
  • Thanks to tom.prince for fixing.

• Add a GetStrengthOfStack method to squads, and replace the places where we open-coded that.
  • Thanks to tom.prince for fixing.

• Include strength of stack in Human Resitance Fighters' budget refund when warping out.
  • Thanks to tom.prince for fixing!

• Make sure that in a couple of places, the templar spawn units in-bounds, that weren't before.
  • Thanks to tom.prince for fixing!

• Store Templar Constructor Location as ArcenPointFromCombatSpace.
  • Thanks to tom.prince for fixing!

• Don't include ship cap changes for mark level of ships in Necromancer fleets, when calculating strength changes.
  • Since Necromancer fleets don't have ship cap changes based on mark, make sure we don't confuse the player when calculating the effect of tech increases.
  • Thanks to tom.prince for fixing!

Beta 3.754 Player Types

(Released November 18th, 2021)

Improvements

• Templar Rifts will no longer spawn on planets with a lot of hostile strength
  • Thanks to Badger for adding.

• Don't allow cross-planet waves to be interrupted on their way to their target.
  • Thanks to tom.prince for improving, and Daniexpert for reporting!

• Cheat improvement: Trigger wormhole invasion instantly, on cheat request.
  • Also provide feedback that command was triggered.
  • Thanks to tom.prince for improving.

• Some nerfs to elderlings.
  • Thanks to Badger for updating.

Bugfixes

• Fix an exception with wave tracing.
  • Thanks to tom.prince for fixing!

• Fix serialization/deserialization of wormhole invasions.
  • Thanks to tom.prince for fixing!

• In general, CleanupForReuseAsNewObject is now WipeForReuseAsNewObject, and ClearDuringXmlReload is now WipeDuringXmlReload.
  • When I was typing Clear, intellisense was way to fast to give me the less-frequently-used other options, to huge annoyance.

• Added a new Faction.PoolFor_FactionList, which is a better model than the one I'm using for planets and a few other things. I will convert more things to this style in the near future, as we can save some RAM and some processing power by moving away from threadstatic with these sorts of things.

• Fixed a typo that was causing allegiance_factions_i_always_enemy to never be properly read in.

New DLC1 Feature: GameEntityTypeData Extensions

• Added the core of a new ExternalGameEntityTypeDataExtension, which is not yet actually in use for parsing anything, but which we can use to add custom things like what necromancer has for its external logic.
  • It is this sort of thing which allows for certain faction behaviors to be custom with all sorts of ships without having to mod Core, but this won't allow for things like new ship weapon behaviors, since that's more tightly integrated into Core.
  • A lot of how the necromancer works is really interesting and unique, and being able to have that as a mod-style extension (from a code sense) is useful for future expansion when it comes to actual mods. That said, a lot of core behaviors relating more to weapons and such will always need to be in Core.
  • This new feature in general is a part of DLC1. It's available to all three DLCs to use, but any mods that use this will error if DLC1 is not installed, so they should be set to require DLC1.

• All of the various necromancer features which were previously on GameEntityTypeData are now on an extension that is DLC3-wide.
  • The code and xml has really clear examples of how you can add these things (see data_extension name="DLC3" all over the place), and how they can even be made to conditionally-turn-on only in the presence of a mod or dlc (see data_extension name="DLC3" required_expansion_list="3_The_Neinzul_Abyss" on various AI Structures in the base game).
  • Essentially, this is a very powerful tool for modders, and also makes certain aspects of faction design both more compartmentalized and arguably more efficient.
  • An important design choice, which is noted by the code in a couple of places, is how many of these to have. My opinion is that the most efficient style is not to make this per-faction or whatever, but instead to make it global per-DLC or per-mod.
    • So if mod X has 14 factions in there, it would just have ModXGameEntityTypeDataExtender with a name of ModX, and thus would be very simple to get anything and everything of relevance from that mod's scope all at once.
  • It is also possible for code in general to look for the presence of one of these data extenders, and do something if it is found, and otherwise ignore it. This is really useful when you want a couple of mods to work together, but one mod defines some extra functionality that the second one uses if it's there, but ignores if it's not. That sort of thing.

New Feature: Player Types

• There is a new PlayerTypeData table, which in the base game (for now) just has "Human Empire" as the sole type.
  • In DLC3, "Lone Necromancer" is a new type that we have so far, but we'll also have some more types in DLC3, possibly one in DLC1, and also a spectator mode in the base game.
  • With DLC3 installed, mods can also now add custom player types!

• The game now only allows a single player faction type to be defined in the game, and it must be named Human. It throws an error if it finds multiple, instructing modders that they should implement their alternate player type as an actual PlayerType.
  • If people wind up trying to implement things in two different ways, we're going to get into some pretty nasty code situations in the future, plus it would just be darn confusing for players who want to look at all their options on how to play in one sensible list.

• On player factions, the base_info_source and deep_info_source now comes from the actual playertype, rather than coming from the faction definition.
  • This allows for all the faction code to be as heavily-divergent as desired in terms of execution, while still maintaining central organization.

• is_alternate_player_faction and alternate_player_type have been removed from the faction definition.
  • is_human_empire_faction, is_necromancer_faction, and is_solo_ark_faction have all been removed, too.
  • does_not_collect_science_or_hacking_with_other_players has moved permanently to the PlayerTypeData.

• Reactive Ship Group Triggers now have an option for trigger_on_player_type, which lets them trigger based on specific player types.

• On tech upgrades, for_alternate_player_faction and the related ForHumanEmpire filed are both removed, and have been replaced by the for_player_types list field.
  • On each player type, there is now a TechUpgradesForThisPlayerType, which is filled from the tech upgrade table in a helpfully-automated way.

• Rather than having manual lists on the World object of things like "HumanEmpireFactions," each PlayerType now has a list of CurrentFactionsInThisGame on it.
  • This allows for a similar level of immediacy for iterating over factions of a type, while still keeping them data-driven.

• Previously, human empires auto-gave metal from themselves to other human empires if they would be gathering an excess of metal when they were at cap.
  • Now, each player type defines a excess_metal_auto_gifts_to_player_types, which allows us to define which player types have that sort of relationship.
    • This is relevant partially because we'll have things like the Ark Empire which should totally share with normal empires, for example.
  • For now, if there are two lone necromancers, they can have metal flow between one another like that (because, hey, their balance is equivalent), but the empire-like player types can't donate back and forth to the solo-type faction (because their economies are way too different).

• is_subject_to_energy_brownouts is now specifically set on the player type, so that things like necromancer don't get hit with those (they don't use energy), while various other player types of course still have them.

• There is a new is_considered_a_full_empire on player types, which causes them to be added into a general EmpireStylePlayerFactions list.
  • This list is not for any specific type of player per se, but basically all of the big empire-like ones versus the solo-ones or helper-ones.
  • We'll definetely be having Ark Empire in DLC1, and Necromancer Empire in DLC3, and potentially a Spire-Infused Empire or something of that sort in DLC1, to go with the general Human Empire from the base game, and these are all functionally identical in SOME ways, but not all.
  • As a category, this turns out to matter for a few things like the way that science and hacking scaling happens in multiplayer, for instance. Also that these start with a homeworld of their own. And they affect how many nasty things seed out in the galaxy from various AI types and other factions. But I don't want to assume metal-gifting or certain other characteristics across all these empire types.

• Added a new hacks_fail_when_hacking_points_exhausted, because for some reason the necromancers were set to not do this. I'm not sure if this is truly the desired behavior or not, but at least it is controllable by xml now and not hidden in the code.

• There were a few very funky pieces of logic with the mark level setting and ship-cap-setting of some alternate-faction fleet members that I believe were not really used at all and have been safely stripped out, but if they were intended then I probably just slightly messed up how that works for the necromancer.
  • If that's the case, the logic was in the main Fleet class and is easy to put back with a conditional based on the PlayerTypeData, but it really seems like it was not actually used...

• Added ship_counts_go_up_from_upgrades, which is more descriptive than functional. This is set to false for necromaners, since their techs work so differently, and this keeps the ship lister utils reporting correct numbers without having to hardcode anything about that faction.

• Put a variety of helper methods in NecromancerFactionBaseInfo that allow us to do things for all necromancer factions without having anything hardcoded outside of the necromancer files themselves.
  • This consolidates logic and will make it easier for us to add the necromancer empire, etc.

• Added uses_any_random_hacker_for_hacks, which is true for necromancers for a variety of their hacks.

• Added reject_tech_upgrades_that_benefit_nothing, which necromancers set to false but the regular empires set to true. Data-driven rather than hardcoded.

• Added fleets_only_show_mobile_strength_in_sidebar, which necromancers set to true but the regular empires do not. Again making it data-driven.

• can_upgrade_city_fed_fleets_directly is another necromancer thing that is now data-driven.

• auto_add_singleton_of_faction_when_any_count_of_other_faction_present is now auto_add_singleton_of_faction_when_any_count_of_other_factions_present, and allows for a list of factions.
  • There is also a new auto_add_singleton_of_faction_when_any_count_of_other_player_types_present, which lets us do the same thing for player types.

• Okay! Lone Necromancers have been converted into being a player type, and in general things seem to work, with some oddities. Known issues at the moment:
  • You can't start a game with just a lone necromancer faction yet. I need to put in new logic for that.
  • When you start a game with a human empire and a lone necromancer, right now it's seeding the lone necromancer on its own planet, incorrectly, rather than putting them with the other player.
  • When you start a game with a lone necromancer, it's giving you a human home command station and cryogenic pods and such.
  • The descriptions for the player types are in place in the code, but do not visibly show yet.
  • It's possible that there are other issues that I'm not yet aware of, but those are all the ones I've seen thus far.

Beta 3.753 Necromancer Externalizing

(Released November 17th, 2021)

Improvements

• Added a new ExternalFleetBaseInfo, which basically works like the squad BaseInfo.
  • These are attached to fleets, rather than squads, however, and they don't have a DeepInfo counterpart as that does not seem needed.
  • Right now there's one set up that is empty for the Necromancer, but so far this is just structural work.
  • The idea behind this is that sometimes a specific faction or other feature will require fleet-specific custom data, and this provides a mechanism for doing that. In the vanilla game and DLCs, probably the only need we'll have for this is with the necromancer, but this should be quite useful for mods that people dream up in the future, too.

• Faction BaseInfo has a new DoWhenThisFactionIsToBeRewardedForKillingEntity, which basically allows us to collect a lot of the logic from DoOnAnyDeathLogic_HostOnly_AfterFullDeathOrPartOfStackDeath in the scenario and hand that off to specific factions.
  • This is called only when the faction itself is the killer and also determined to be the one "deserving a reward." At the moment it is unused, but it sure could be useful in the future.

• Faction BaseInfo also has a new CheckIfPlayerFactionShouldGetRewardBasedOnAUnitDying.
  • This is called on every player faction when any unit is killed, period. It identifies who the killing faction is, and allows for custom logic to be run.
  • Many times, this will be utterly unrelated to anything the player faction needs to do. But if the player faction is "the strongest faction of type X on that planet where the thing died," for instance, then this is a method where that sort of thing can be calculated and then some reward can be granted.
  • I have converted over some of the necromancer rewards to this, and this actually has a lot more flexibility on the reward structure than before. I left a "BADER_TODO GetShouldThisNecromancerFactionGetARewardBasedOnThisKill" in place of an older TODO, because this is supposed to benefit multiple factions in a few ways when Elderlings are killed. This is now possible without getting into list structures or non-faction-specific code, which is nice.

• Faction BaseInfo now has a new GetCustomHackingHistory().
  • If we want a given player faction to have a custom hacking history in place of the regular one, then we can write this here and return true.
  • If we want it to have ADDED hacking history in addition to the regular one, we can write it here and then return false.

• The various elements from the necromancer faction that were embedded in Fleet and Faction have been moved into external data BaseInfo, instead.
  • Right now things have just been linearly converted across into the new format that is modder-friendly (where anyone can create a faction like this, in other words), but my next step will be converting it from this into a player-type instead.

• Use a single single attribute to track the status of time-base pool items.
  • Thanks to tom.prince for updating!

• Add separate tracing for Warden sub-faction.
  • Thanks to tom.prince for adding!

Bugfixes

• Only generate periodic AIP increase if non-zero.
  • Thanks to tom.prince for fixing!

• Fix a divide-by-zero in the periodic AIP code
  • Thanks to Badger for fixing!

• Don't count energy usage twice when checking if self-building units can complete.
  • The energy of self-building units is already counted in total energy usage, so we don't need to check if there is enough remaining energy for the unit in addition.
  • This meant that you couldn't finish building a self-building unit you had enough energy to build, if you didn't have enough energy to power one additional one.
  • Thanks to tom.prince for fixing!

• Re-add per-second logic to Warden, Hunter and Praetorian Guard that was inadvertantly removed in r14244 (this was during the Great Refactor, on September 18th).
  • Thanks to tom.prince for fixing!

• Fix errors in fireteam tracing.
  • I'm not sure how *useful* these changes are, but it doesn't throw exceptions now.
  • Thanks to tom.prince for fixing!

• Get rid of duplicate list of fireteams on Hunter and Warden.
  • At least for the Hunter, this prevented the hunter from doing anything. It may have also affected the strength counting, if the Warden depended on that info.
  • Thanks to tom.prince for fixing!

• Move handling of IArcenUI_Element_Controller.HandleMouseover to OnMainThreadUpdate.
  • It was originally called in OnUpdateFromMainThread, but that meant that code that looked at values set by that were called in the next frame, rather than the same frame.
  • Thanks to tom.prince for fixing!

Balance

• Skeleton detonators are no longer options for game start (they cost too much metal). Update the necromancer faction description to suggest an allied faction. Give the necromancer a bit more essence
  • Thanks to Badger for updating!

• Add a new ship line to the Necromancer, the Banshee; this is a cloaked ship with a decloaker attached
  • Thanks to Badger for adding, and Dragoris for pointing out we had no decloakers in the Necromancer faction!

The Start Of Fuel For Expert Mode

• Add some *very* rough code implementing fuel.
  • This is intended to add more strategic depth to expert mode.
  • The setting to enable it currently lives in a mod to reflect the rough nature.
  • Thanks to tom.prince for adding this!

• Allow overusing fuel, but cause apply penalties.
  • Your turrets are less effective by a percentage of twice the percentage of overuse (capped at 99%).
  • Thanks to tom.prince for adding this!

Beta 3.752 Pathfinding Efficiency

(Released November 16th, 2021)

Improvements

• More work implementing AMU settings into Vanilla:
  • Each Human Empire may now set a starting amount of Metal, Science and Hacking. Like Cryopods and Settlements at campaign modes higher than Humanity Ascendant these lock into the current default values.
  • All values can go down to as little as 0 - meaning it's possible to start with literally nothing.
  • Science and Hacking can increase by 1/3rd over the current default values on HA, and vastly higher in Sandbox mode. This should still be relatively balanced, but certainly is a change potentially in favor of the player.
  • Since metal is already at the maximum capacity of the Home Command station there's no need to have any higher setting here.

• Split the Necromancer Flagships, Skeleton ships/structure and Wight ships/structure XML into their own files. A single file was getting too cumbersome
  • No functional change

• Reorganize the necromancer journals a bit
  • Thanks to Dragoris for the feedback

• Show essence in hacking sidebar for necromancer.
  • Thanks to tom.prince for adding!

Bugfixes

• Removed the unused AIP floor modifier external constant from the game, as it was never used.
• Continuing work on porting over AMU's extended galaxy settings into Vanilla:
  • Starting AIP is now a setting instead of an external constant. The default was and is 10.
  • It is now possible to set an AIP value per interval and the interval time in minutes. By default no periodic AIP is generated.
  • Fixed some bad and/or wrong descriptions in the categories for the Extended galaxy setting subcategories.
  • Thanks to NR SirLimbo for all of this!

• PathCache has been converted to be concurrent-poolable, rather than between-mapgen pooled.
  • The idea is that various threads only need these for a while and can give them back.
  • Since CacheForPathingMode is a subcomponent of this, it also now has the same behavior as well.
  • ...and PathBetweenPlanetsForFaction also gets the same treatment, goodness.
  • GetOrAddPathCacheForFactionInThisThread in PathingHelper has been removed, and a new PerFactionPathCache concurrent-poolable object has taken its place.
  • This whole rework basically keeps all the logic the same, but makes the "pathfinder didn't finish before it was run again" bug impossible to hit now. Apparently it became more rare in the prior build, but was still possible. Exactly why is hard for me to tell, but it's hard to be sure why.
  • The secondary effect of this is that there had been a pretty massive amount of RAM being used for pathfinding cache data, 300MB or so after maybe 20 minutes of play from what I can tell, and it might actually have qualified as a memory leak, temporarily. The number of tracked objects was into the hundreds and climbing. In the same sort of scenario, the number of tracked objects is now... 3. And RAM stays stable.
  • Thanks to ColdPrototype for reporting.

• Fixed five different pools that were only doing a pull-in of their items on xml reload, not on game reload.

• The Civilian Industries mod has been updated to compile again, and no longer leak certain kinds of memory. Whether it actually fully runs or not is anyone's guess thus far, though.

Beta 3.751 Faction Processing Groups

(Released November 15th, 2021)

Improvements

• When hovering ships you can take in an ARS, the game now tells you how many other ship lines you have with the same techs.
  • Thanks to Badger for adding!

• Necromancers and Human Empire can no longer assist with eachother's construction. They are balanced around different amounts of metal availability.
  • Thanks to Badger for adding.

• Begun to implement some AMU extended galaxy settings into Vanilla:
  • Science and Hacking points in the galaxy can now be edited, with the following settings for each in campaign types lower than . All of them are saved on a per-galaxy basis, with no changes in XML required, and by default no changes from Vanilla unless specifically set by the player:
    • The amount per planet can be set individually.
    • A variance to that amount can be applied and can range from 0 to 100%. At say 30 HaP per planet, and a variance of 50% the planets randomly have between 15 and 45 HaP.
    • The adjacency bonus for HaP can now be edited on a per-galaxy basis, and an adjacency bonus for SP is now possible at all.
    • There are now 4 options for the scaling of HaP and SP in Multiplayer: "Soft Decline" (as HaP do now), "Linear Decline" (received amount = 1 / player count), "No Decline" (as SP do now, each player gets 100% from all any-player owned planets), "Individual Harvest" (each player gets 100%, but only from their individually owned planets).
    • Per-AIP generation: (Fractions of) SP and HaP can now be granted whenever AIP is generated (except for the starting amount of AIP), similar to how HaP worked in AIWC.
  • The under-the-hood logic:
    • Created the AIWar2GalaxySettingQuickAccess class, which holds a few settings that are used very frequently used and directly stores their values so they need not be searched for whenever used.
    • This corresponds to the XML field has_quick_access_logic. Note that this does nothing alone, to work all quick access settings it need to be set up in C# code as well.
    • Planets now store a potential overriding HaP amount. This breaks savegames.
  • Thanks to NR SirLimbo for adding!

• Holding ctl when hovering ARS hackable ship lines now shows more detail about synergies
  • Thanks to Badger for adding!

Bugfixes

• Game Entities now track both their CalculatedSpeed and their CalculatedSpeedWithoutSpeedGroups. Fireteams use the second number when figuring out how fast they should go.
  • This prevents a positive feedback loop where a fireteam would think it needed to go faster and faster. Hopefully no more super-fast fireteams anymore! Thanks to Badger for the analysis and fix, and lots of people for reporting the bug.
• Thanks to Badger for finding and fixing!

• Fix a bug where Svikari description appenders weren't working
  • Thanks to Magiik for reporting and Badger for the fix.

• Fixed possible exceptions due to disabled EntityTypeDrawingBag entries on tooltips, for both periodic spawning and on-death spawns.
  • Thanks to Puffin for reporting and NR SirLimbo for fixing!

• Removed the unused pooling behavior of ArcenCachedExternalType.
  • Everything is now pooled elsewhere, and had no interesting behavior related to this pooling. The only significant behavior is that ExternalData_Base (and descendants) are no longer returned to multiple pools. This was, in particular, causing the weird behavior of multiple AI factions in the lobby.
  • Thanks to tom.prince for finding and fixing!

• Remove unusued `PrototypeObject`s from Base/DeepInfo, and a bit of other dead code.
  • Thanks to tom.prince for cleaning up.

• Fixed a bug with the energy display's non-C-Click part where it wouldn't show fleets consuming energy.
  • Thanks to Eluthen for reporting.
• Fixed a bug with the energy display in general where it didn't reset the energy produced per planet, thus counting up indefinitely.
  • Thanks to NR SirLimbo for the fix!

• Don't claim objects while the game is paused.
  • Thanks to tom.prince for the fix!

• Improved the pathfinding error handling debug info.

• Added a new processing_group="Name" on faction definitions that is required.
  • This allows us to aggregate "hey, am I allowed to run at the same time?" in the way we really need to. We had something like this before, but not enough.
  • This in turn requires new xml in a new folder called SpecialFactionProcessingGroup, and that fills a SpecialFactionProcessingGroup object in the code, which is mostly used for LRP threading locks.

• UniqueNameForFactionToAvoidThreadConflicts has been removed, replaced with a reference to the faction's SpecialFactionProcessingGroup.

• Okay! This has been a problem for longer than I realized, as it turns out; it just has been more obvious in recent months.
  • Essentially, the way we were handling factions that used shared resources for their "LRP processing" previously only attempted to not let them START in the same lrp sim cycle, but it was not preventing them from starting in multiple cycles and thus overlapping.
  • Good grief, this was more untangling than expected. This SHOULD prove to be the end of any sort of "pathfinder already running" errors, but we'll have to wait longer and see if folks still run into it. I really don't think they will, knock on wood.
    • This should also solve any MersenneTwister.GenerateUInt issues, which are also basically an indication of a single randomizer being called by two threads at once.
  • Thanks to Daniexpert, Puffin, and ColdPrototype for reporting.

• Fixed several faction shortnames being off.

DLC3

• Skull piles now appear in the right portion of the build menu
  • Thanks to Daniexpert for the report and Badger for the fix

• Fix a bug with Necropolis Bolstering, where swapping the bolstered necropolis wasn't happening correctly
  • Thanks to Puffin for the bug report and Badger for the fix

• Add strength indicator when hacking for necromancer ship lines
  • Thanks to Daniexpert for requesting and Badger for the code

• Show more detail in the ARS hacking tooltip to show how many other ship lines you have with tech synergies
  • Thanks to Asteroid for the bug report and Badger for the code

• Fix a bug where necromancer tech upgrades were incorrectly implying you'd get bonus ships from them
  • Thanks to Daniexpert for the report and Badger for the fix

• Remove "strength" text of Necromancer faction in lobby
  • Thanks to Daniexpert for the report and Badger for the fix

• Fix some fallout from my bolstering changes
  • Thanks to Daniexpert for the report and Badger for the fix

• Fix a Null Reference in the skeleton percentage code
  • Thanks to Daniexpert for the report and Badger for the fix

• Fix a bug where Wights weren't decaying properly
  • Thanks to Daniexpert for the report and Badger for the fix

• Add a necromancer upgrade history

• Fix a bug with the Squares map type not connecting properly
  • Thanks to Daniexpert for the report and Badger for the fix

Beta 3.750 Smooth Simulator

(Released November 11th, 2021)

Improvements

• Display more time delays with minutes and seconds.
  • Thanks to tom.prince for the improvement!

• Adjusted some ship group settings after complaints from Puffin and confusion from a user on Steam.
  • To be more exact reassigned AI ship groups for both Electric Bombers and Dark Mirrors. Removed Electric Bombers from one pre-made fleet group.
  • Thanks to CRCGamer for the improvement!

• Counter-sniper turrets are no longer in the simple turrets ship group, and attritioner strikecraft are no longer in the simple AI ship groups with even a 1/100 ticket count.
  • Thanks to Puffin for suggesting.

• The game now requires the following tags for each AI type, which are used to choose how to seed the four categories of "nasty pick":
  • big_gun_nasty_pick_tag, eye_nasty_pick_tag, support_structure_nasty_pick_tag, and wildcard_nasty_pick_tag.
  • Right now, these four all just point to the same tags that were globally used before. But the xml designers among us can now start splitting things out into tiers, or making specific versions for specific AIs, as desired.
    • The ratios of those four kinds of things are set based on the AI difficulty independent of the AI type, but you could always design an AI that used Eyes for all four categories instead of just the Eye category, for instance.
  • This is also a great way to allow for some very difficult nasty pick options, but only have those show up on AIs that are more challenging.
  • At the moment, there is no net effect on the game, but it gives xml designers and modders more power.
  • Thanks to Puffin and Zeus for suggesting.

• Right-clicking the clock now has an "Other Silent Error Data" section where we can see things that had some sort of silent error that the game just worked past.
  • Most of these are going to be cross-threading issues of no real consequence, but it's nice to be able to dig in and see them if we suspect an issue.

• In general, our "delayed logging" that happens from background threads now also logs the thread name and ID to aid in debugging of multithreading problems.
  • Thanks to Tom for the great suggestion, this makes a lot of things much easier.

• World-attached BaseInfo and DeepInfo both now support per-second sim-thread actions, in addition to the existing per-sim-step logic that they already had available to them.
  • Thanks to Badger for requesting.

• Other AI Types now get access to unique warden ships.
  • Thanks to Badger for adjusting!

• Allow a wormhole invasion to be invoked with either 'don't launch and invasion without a good target' (this is typically used for the periodic AI Budget-based wormhole invasions), and 'always launch this invasion, using a bad target if necessary' (this is typically used for event-related wormhole invasion responses, like during Enhanced Overlord)
  • Thanks to Badger for adding!

• StrengthCounting is no longer a part of the short-term planning (aka part of the sim directly). Instead it is now calculated as a long-term-continuous thread, and thus it no longer blocks the sim at all.
  • This was the largest source of sim slowdowns, and on slower computers and larger maps it would get progressively worse. I implemented time-slicing a while ago, but that only worked but so well. We still had occasional spikes.
  • The reason that this can be moved out of the sim is that all of the AI logic that uses this sort of data is run only on the host anyway, frankly. There are not per-ship decisions being made off this stuff. Other than that, this is all just for display purposes on the client and host (not entirely true, but close enough).
  • At any rate, the slight differences in results in MP, which are likely to be negligible if anything are not enough to warrant such a large load as part of the main sim. It's far easier for the host and client to handle it with error correction data when they need to.
  • For me, this takes my average frame time from 30-40ms per frame to 10ms per frame. Given we have a 100ms window per frame, this is truly excellent. Of course, this is combined with our other frame-timing improvements recently. This should have an outsized effect on older machines, which is great.

DLC3 Additions

• Vanguard and Heroic AI Types added to DLC3
  • Thanks to Badger for adding!

• Add Showdown Devices.
  • This is a Galaxy Setting under AI Behaviour.
  • When enabled, it seeds 4 Shutdown Devices in the galaxy.
  • If the player destroys all 4 of those then it triggers a countdown (currently 20 minutes)
  • During that interval, the AI will throw lots of Exos and Wormhole Invasions at you.
  • --
  • If you reach the end of the timer, several things happen.
  • First, all the warp gates and dire guard posts in the galaxy explode
  • Second all Guard posts will deploy all their units
  • Third, all AI ships in the galaxy will join the relentless wave faction.
  • Fourth, every Overlord Mark 1 will transform into a Galactic Command Ship
    • The command ship is going to be a reskinned Overlord Mark 2 (it uses the same balance mechanics)
  • Thanks to Badger for adding!

Bugfixes

• Removed the RelatedString from the OutguardSpawnRequest - it always ends up being the Outguard Group's internal name, and since the above already stores its group, no need to keep that field around.
  • This was likely only because it was needed in the GameCommand_QueueOutguard - this part has been adjusted too.
  • Thanks to NR SirLimbo for the simplification.

• Add a missing space to deepstrike description.
  • Thanks to tom.prince for fixing.

• Fix the ability to control which fleet is bolstered by a city.
  • When I first implemented the ability to control bolstering, I mistakenly attached `BolsteringFleetPerUnitBaseInfo` to the bolstered fleet rather than the bolstering fleet. I didn't catch this in testing, because I had some code to upgrade saves and apparently didn't test on a fresh game.
  • Thanks to tom.prince for fixing!

• Fix sometimes not being able to cancel a hack by not using a global to track the hacker.
  • Thanks to tom.prince for fixing!

• The warning message "Warning: skipped mod "_Vanilla" in XMLLoadingOrder.txt because we couldn't find any mod with that name" will no longer appear. That's just an example mod. Actual mods that are wrongly named will still show this error if this file is used (generally, this file is not needed).

• Further silenced a harmless error in TextMeshPro that was sometimes spamming the log, particularly on game exit but not always limited to that. ("UnityEngine.Material.GetFloat").
  • Last build I kept it out of the main log, but this time it was in the ErrorsReportedByEngine.txt file. Now this file ignores all of the same harmless errors that the main debug log does.
  • Thanks to Puffin for reporting.

• Fixed an exception in GetShouldAttackNormallyExcludedTarget() for the nanocaust that I unintentionally introduced on October 31st when trying to prevent memory leaks and reworking the character buffer usage. In this particular case, the old code was not a memory leak, but my error in handling this was a source of infinite exceptions if you had nanocaust active.
  • Thanks to Major Dullard for reporting.

• Fixed a random exception in CalculateSizeOfTextMeshProUGUIText. This was some funky code that was also a bit of a performance drain when tooltips were involved in general.

• Fixed things up so that when a save fails to load when you're loading it into the lobby, it doesn't lead to an error spiral.

• Fixed an issue where loading old savegame versions as the last-lobby-settings that are invalid into the lobby would pop up a message.

• Fixed yet another issue where quickstarts and last-lobby-settings could get invalidated in the micro-format even though their data was fine. It was just not allowing them to load.

• Fixed an issue with loading quickstarts into the lobby in general; it would error and not let you.
  • Also, related only somewhat, fixed it so that if you started typing a campaign name and then chose to edit a quickstart in the lobby, the name you typed will get copied over.

• If the game asks for an outguard state for a null outguard group, it now gives an appropriate error message about that. This applies to the Territory Dispute faction right now.
  • Thanks to tom.prince for reporting.

• Fixed some rather random-seeming exceptions that were suddenly showing up in SquadStackPlanning. It's possible that these were not new, but that they were somehow being masked previously. At any rate, it should be more robust against these now.

• Fixed a couple of nullrefs that could happen in FactoryConstructionForPlayerMobileFleets.

• Fixed an issue with TargetListPlanning where if it had exceptions, it would sometimes do funky things and send malformed gamecommands as a result. Now if it has an exception, it just stops trying to think about that round of targeting.

First Batch Of Threading-Related Fixes

• Do less pause-specific work in SimPlannerImplementation.
  • It turns out that some of the logic that was being run while paused was *also* being run in the background sim planning[1] threads. This could lead to race conditions, in particular, the disappearing turrets.
  • In particular:
    • `DoFleetPerStepCalculationsOnly` is no longer called (and the parameter it took for dealing with being called from other contexts was removed).
    • The logic for calling `DoGeneralAggregationsPausedOrUnpaused` from faction BaseInfo was removed, since this is done in the main "sim" logic even when paused.
    • The logic for calling `DoOnPerSecondWhilePausedNonSimUpdates_OnMainThread` from faction BaseInfo was removed; nothing implemented this anyway.
    • The logic for calling `Planet.DoPerSecondWhenPaused` was removed, since this is done in the main "sim" logic even when paused.
  • Note that per r14349, "sim planning" is a misnomer, since it handles lots of things that run even while paused.
  • Thanks to tom.prince for figuring all of this out and fixing it!

• Fleet and TypeData on FleetMembership have now gotten the same sort of treatment that I put in for TypeData and Planet and such on squads.
  • The idea here is that when things temporarily go null, we have some major issues that come out in random ways.
  • Same treatment for Planet and Faction on Fleet itself, too.
  • Thanks to UFO for a report of there being a FleetMembership error with TypeData after a number of map generations in mapgen.

• Fixed a persistent recent bug, which may have been a memory leak and also caused crashes, where essentially some debugging processes for "sub thread information" in strength counting were being run from multiple threads at once (bad), and also were running even when we were not showing the monitoring window (mostly pointless, but hey let's save a bit of CPU).
  • Thanks to UFO and others for the crash reports that included these.

• There is some sort of exception that keeps being possible in my list class when AddRange is called, which calls InsertRange, which does List.CopyTo. I think this is related to lazy-initialization, but I really can't tell what the heck is going on exactly.
  • What I've done now is made it so that it gives me way more detailed information on those operations if they fail, without too much in the way of cost, since these are not really used all that frequently to where they could be THAT large of CPU drains.
  • The end result of this will lead to incorrect data (aka the data won't be copied), but not an actual halt to program execution like it did previously.
  • Thanks to Major Dullard and others for the various reports that keep hitting this.

• The exception "CalculateRequestedFlows exception hit. debugCode 390" has been instrumented further to give us much more detailed data about what is wrong.
  • To be honest, given the state of the log in which this was found, I think that the cross-threading issue that tom.prince found and fixed in this new version is probably going to resolve this. But just in case.
  • Thanks to Major Dullard for reporting.

• Same deal with "CalculateRequestedFlows exception hit. debugCode 420." It's now a lot more detailed, but probably already resolved by tom.prince's fixes.
  • Thanks to Major Dullard for reporting.

• Updated PlannedFlowsByPlanet_Sorted_ForUI to be a bit more efficient, and a bit more error-proof during its sorts. This is a rare case where I'm actually leaning more on the GC (with a RefThreeTuple instead of a Pair) in exchange for the ability to precalculate some data that was otherwise calculated repeatedly in a loop within a sort, which is definitely a slowing sort of thing to do.
  • Thanks to Major Dullard for reporting an exception that was happening in this code.

• Altered our List.RemoveAt() code so that it now diverges from the Microsoft standard a bit more. It now takes an optional second parameter, ThrowErrors.
  • If that is true, then it works like it used to -- if the code tries to remove something out of range, it will freak out and die.
  • If that is false, which is the default, then the code will just silently return false. Previously it did not return anything.
  • There are times where we have one thread updating a targeting list and another thread reading from it, among many other circumstances, and it was possible to get errors in these cases. Now it will just gracefully keep going, instead.
  • Thanks to Major Dullard for the log with the report in it.

• Ugh! There's an error "Error occurred in c-o-h sim planning context", but I can't fix it just yet because I forgot to have it actually include any details about the error it experienced. Sigh. So the new version reports those details. I should then be able to fix it next build, if it's not accidentally fixed by the other fixes in this new build.
  • Thanks to cml for reporting.

• Silenced an error that could happen in trying to sort objectives in "ObjectiveCategory.SwapObjectivesFromBuildingToDisplay."
  • Almost certainly this was happening because of the bug that tom.prince fixed with this code being kicked off twice instead of once when the game was running twice, but it's good to make sure it's not interrupting game flow either way.
  • Thanks to cml for the report.

• A bit better error logging inside ArcenThread.Start, although I think a lot of the errors happening in there were secondary to other things that people were running into.
  • Thanks to Puffin for reporting.

• Fixed an invalid error that would happen when you were trying to load savegames, where it said "could load" because of a missing campaign name.
  • For one, fixed the campaign name to be read properly (it was right there), and for two fixed it to say "could not load" if it ever has that problem again.

• The objective sorting errors got worse, not better, so I've put in even more debug code with those to see what is happening.

• I was also able to get a bunch of the "c-o-h sim planning" errors, which at first seemed to be an error in BasicPool.Clear().
  • However, it looks like it's actually an error inside of some random BasicPoolable.Clear for a specific class. That is now made apparent in the error logs.

• Okay, both of the two errors above seem to have been flavors of the same general problem: the way that I'm returning data from the squad and such still sometimes being null, or changing unexpectedly. Essentially I could set the TypeData member variable to be a value after discovering it was null, but then still end up returning null out of it.
  • This probably has to do with the equivalent of why one would normally use the volatile keyword, but I don't wish to do that in these cases. Instead what I do is the usual hardening-strategy, which uses a local variable and returns THAT. It also tries to assign the member variable, but if that fails for some reason it's not the end of the world and won't lead to an exception.
  • We're talking differences of literally microseconds here in data, and it does seem to happen way more on AMD than Intel, but that's why I'm running an AMD rig as my main dev machine since this summer. Anyhow, shifting the pattern of these methods seemed to fix a wide class of bugs -- properly, finally. The prior public build claimed to do so, but I don't think it worked for AMD, only for Intel (and maybe not even that; it's really hard to be sure).

• BasicPoolable has been renamed to IBasicPoolable, because that's the proper naming convention. I'm not sure why I missed doing that initially.

• Fixed an exception that could happen at "DoFactionStepLogic_Stage1 error at debugStage 10600" with some cross-threading situations.

• Added robustness to ConcurrentDictionaryOfLists in order to prevent "Hit exception in BuildShipsLookup debugCode 400" errors.
  • The ConcurrentDictionary class really is only partly reliable, I have to say, based on observations so far. This explains a lot about the ghost ships and missing ships in general in that central dictionary.

• Unfortunately, there's just something that is changing the objective information during a sort operation, on a different thread, when it comes to the objective counting. I have done enough profiling to be pretty sure that this is not multiple threads running at once. One other possibility is that there are two threads sharing the same objective information instead of it properly being in the pool, but that would assumedly create more of a visual mess.
  • For now, I have it just counting how many errors of these types it has, and allowing the sort to fail. This is on a random background worker thread, so it shouldn't be interrupting game flow or causing anything overly strange.

• To prevent any sort of UI strangeness, if either of the sorts for objectives fail (this is on a random BG thread), it will try again up to 5 times. This should lead to perfect consistency, knock on wood.

• ActualObjective is being converted from IBasicPoolable to TimeBasedPoolable. There were some errors that are mathematically impossible to have occur, and that's a good sign that we have cross-threading shenanigans going on.

Rework Of Threading, V1, Lots Of Lack Of Success

• Okay, lots more debugging after taking a break overnight. There are multiple copies of some core threads kickinng off, which I am now able to detect and prevent, and warn about.
  • One core problem with the current setup (which is old and goes back to 2017) is that it's not very flexible when it comes to the separation of "context," thread, and code execution. With the ability to reload all the xml, we really do need that.
  • I'm not precisely sure how long this has been an issue, but in some form at least for a few weeks. I'm going to work on a midsize refactor of context-to-thread-to-code relationships, which won't affect end-programmer code, but will be more robust for the sort of reloading and pooling we do.

• Our debug logging is now a little bit clearer and more brief when it comes to adding in thread information.
  • Thanks to tom.prince and Badger for suggesting.

• IThreadStatusChecker was a supremely confusing name, and has simply been renamed to IArcenThread.

• GetThreadByUniqueName_CreateIfNotPresent now locks the list of threads to prevent any duplicates from ever being created.

• ArcenClientOrHostSimPlanningContext no longer has a readonly thread assigned to itself on initialization.
  • Instead, it now has a IArcenThread AssignedThread that I will initialize when I darn well feel like it.

• The sim contexts are now separated out from their threads more, as well as the code that runs on them.
  • They now terminate at a poolable object, not a code-executing object. These are then pooled by type, rather than having yet another sub-type inheriting from them.
  • It's a much more consistent pattern with the rest of the codebase, and makes things way easier to keep track of. In a lot of ways, this is more MVC-like, though it's not exactly that design pattern.

• MainThreadArcenClientOrHostSimContext is actually not pooled, because there should only ever be a single one of these, and it's for the existing main thread of the game.

• NonThreadBasedArcenClientOrHostSimContext no longer gets assigned to static variables, as we pull these out and use them for a little while before putting them back.

• ArcenShortTermPlanningContext is now a sealed class, rather than abstract.
  • A new ArcenShortTermPlanningCodeExecution has been split off of it that handles the code portions that are the various child classes, like StrengthCounting, etc.
    • None of these classes rely on static variables at all anymore, and additionally none of them are singletons now.

• ArcenSimPlanningFrameProfilingData and all its related code has been removed. It has not been used in quite a while, and it's not efficient at all. We have better ways of getting the same info now.

• The OuterLT_Intermittent thread is being removed, as it serves basically no purpose. I can get the same effect with less fuss by using Task.Run(), so there's no need to have this here.

• ArcenExecutionContext has been converted away from being part of the inheritance chain leading to SimExecution. That was needlessly confusing.
  • This is now a static class called ArcenExecutionManager and it does exactly that.

• A new ArcenLongTermContinuousPlanningClientOrHostManager has been split out of the core of ArcenLongTermContinuousPlanningClientOrHostContext, making the latter very very simple now.

• The display when you right-click the clock now gives some more info on some of the intermediate threads and their run statuses.
  • Additionally, those threads are now pooled and executed a bit more clearly than before, though I may still make more improvements later.

Rework Of Threading, V2, Much More Success

• ArcenThreadManagerBase has been removed, and a new ArcenThreading class has been introduced instead.
  • Overall, this new class provides ways to run anything we want on background threads, while at the same time allowing us to _block_ further runs of any background threads, and adjustments to the pools, etc.
  • Rather than ever calling Task.Run() -- a few exceptions in Universal aside, which are for my own reasons -- end-programmers should now call ArcenThreading.RunTaskOnBackgroundThread().
    • The functionality is identical, except for the fact that it will discard your request to run a task if the game is in the process of quitting to the main menu or exiting to the desktop.
    • If you use regular Task.Run(), the main risk is that you'll get errors in your code during those periods. It would be really nice not to have those, for your sake and mine, so hence it being very desirable to use this central thing.
  • Same exact deal with Parallel.For now being ArcenThreading.ParallelFor, and also Parallel.ForEach now being ArcenThreading.ParallelForEach.

• When the game is in its shutdown phase, moving from the request for threads to stop doing their work into whatever comes next, the following happens:
  • There's a 200ms period where it first just stops refusing to start new threads, and also stops givinng out anything from pools.
    • In places where either is requested on a non-main-thread, it throws a new ArcenPleaseStopThisThreadException which can be caught like older ThreadAbortExceptions and allows for code not not bother trying to log the exception.
    • Also during this period -- it is assumed that all saving has already happened, so we don't care about anything because we're headed to the main menu or the OS -- it stops logging any errors or other messages to the debug log from any non-main threads. There's nothing to say during this part.
  • After this initial period of 200ms, it actually does the "quit to main menu" logic, OR it does the exit to OS logic.
    • If it's headed to the OS, then it ceases any more logging from background threads, period, and just lets the game be reclaimed by the OS. Everything of importance on background threads is done prior to now, so no harm.
    • If it's headed to the main menu, then it starts clearing the world, but puts a 2 second moratorium extension on things coming out of pools or threads kicking off.
      • It's possible that there may be some edge cases in the UI where someone can click THAT fast back into the lobby or something, and I may need to handle that in some way. Or shorten this time period. Since I'm not actively murdering threads anymore, I have to give them a bit of time to error on their own as they see the world disappear around them, and in general that's something that we've been making threads hardened AGAINST, so it's a slight catch-22.
      • Anyhow, after this period it then resumes normal operations and we're good to go. If there's something I need to adjust during this period, it's only involving a few buttons on the main menu (since that's where you are now), so I can handle that with a fairly minimal surface area.
  • Thanks to tom.prince for lots of great ideas on how this shutdown phase should work in order to avoid timing-related pooling problems.

• All the various places that were handling ThreadAbortException now handle ArcenPleaseStopThisThreadException.

• The threading model in general has been completely ripped out, for those threads that still used it.
  • IArcenThread, ArcenThread, and ArcenThreadManager are all gone.
  • These were only used for the main sim thread, and the various long-term background threads in External code. The short-term planning code and various other things were already using Task or Parallel.
  • It was also used for the status checker for connecting to a host by IP address using LiteNetLib or similar, but that has also now been converted over.

• All of the places that had MyWorkIsDone now have IsCurrentlyWaitingOnThread. This is a lot more useful, and is consistent with the pattern I'm using elsewhere.

• ThreadCompletionType has been removed, as that was a needless complication in the new design of things.

• The interface that you can see when you right-click the clock no longer shows informations on threads, since those are no longer centrally tracked or handled by us directly (instead they are part of the C# TaskFactory, managed by mono).
  • You can still see the timings for everything, however, based on the contexts that we keep track of. I added some more pieces to that last night already, so it's at least as informative as it used to be.

• Added a new IContextForMonitoring interface, which I'm using to hook in to give us the proper warnings for slow or halted contexts -- it's the equivalent of monitoring slow and stalled threads, but not actually threads now (though each context does correspond to some work being done on a background thread, that thread changes pretty rapidly and is just some random threadpool worker).
  • This actually gives us some insight into a few areas that we could not see previously.

• Whatever DetailedProfilingEnabled and DetailedProfilingPassesBetweenDumpsToDisk used to be, they are now removed.
  • This sent up some alarm bells for me, because these were enabled for a couple of long-range threads. This might have been a source of something that appeared to be a memory leak or a performance churn, depending on what the loggin was like, and why it existed.
  • I'm pretty sure that this was something I set up at some point, but the actual parts that did whatever the logging were had already been stripped out (again, by me) at some point in the past. How far back, I'm not sure.
  • This was probably bad news at some point, these getting left on, and I just happened to find the remains of that at the moment without really being able to see how bad of news that was, whenever it was an issue.
  • It would certainly be darkly "hilarious" if this or something like it was the cause of some of the original memory leak reports sometime in the last few months. Regardless, I'm not going to waste time looking into it, because the codebase is so thoroughly evolved past that now, it was worth it either way. And I'd rather not resume grinding my teeht in my sleep.

• Fixed a completely benign spam of "Error! Duplicate planet fleet ID" messages that were simply new fleets finding themselves in the list. Why this is different now, compared to any time in the past, I have no idea. At any rate, when it properly checks to make sure it's not looking at itself, all is well on that front.

• Pools all now define a PoolBehaviorDuringShutdown, which has three possible values: AllowAllThreads (rarely used, but is needed sometimes), BlockAllThreads (needed more of the time), BlockBackgroundThreadsOnly (in theory could be useful).
  • The second one is what was described before, but it turns out that there are some pools that still need to act like normal during the shutdown period (things for temporary working lists, etc).

• Added a new ArcenThreadsAreShutDownRightNowSoYouCannotHaveThatException that is thrown instead of returning null from pools on the main thread when they're not allowed to get something.
  • This turns something from "some random nullref exception" to a very clear "oh, something funky is happening right there" issue.
  • This actually now happens when you try to run a thread from the main thread, versus it just silently returning, too.

• Improved code in a variety of places, including a couple on the main thread, to no longer use DoForPlanetsParallel. Instead, these places now use DoForPlanetsSingleThread.
  • Essentially, there's a certain spinup cost for each thread for the parallel version, and so if you're not doing enough work inside the delegate here, it's much much better to handle this all on one thread than it is to fire up many threads and then wait for them all to return.
  • Originally, I did this in order to keep from having to catch exceptions when the main thread was trying to run things in parallel while also shutting down, but it turns out this is a nontrivial performance boost also.

• In that same spirit, renamed DoForEntitiesParallel() to be DoForEntitiesParallel_OnlyUseForSuperHeavyThings(). And the same for the factions and planets ones.

• Overall the game is now much better at not doing as much work while it's trying to shut down to the main menu or exit the game.

• Improved the performance of the ship disabled checks and current order checks by splitting those out and having them happen in parallel, with the disabled checks being on a worker thread while the order checks happen on the main sim thread.
  • Please note that I don't actually wait for the ship disabled checks to finish before I move on to the other parallel parts of the short term sim context. It doesn't seem warranted, since it would only have a very occasional sync issue on multiplayer, and that will self-heal cheaper than the CPU cost of waiting here.

• The game has been upgraded to properly pool the host-side long-term-continuous threads, and to also make their execution visible to the UIs that show threads.

• A bit more information about the last time threads were run is also now present in the clock-right-clicked screen.

• Improved the cleanup of threads between one run of the game, going to the main menu, and running another.
  • At this time I'm still able to get stuck simulation threads, but it seems to be the entire simulation now, not some random smaller part of it. In some ways that is a big improvement, because it narrows down the surface area considerably.

• The right-click clock window now has more information about why it did not run the last sim step, if and when that happens. Sometimes it's a valid reason -- like being paused -- and other times it's something that got stuck. It's right up near the top, in fairly small print.

• The game now keeps track of the tasks that get spawned via ArcenThreading, and as those are completed it stops watching them.
  • Because of this, the logic for "when the game can again start spawning threads and getting things out of queues" has changed from "after two seconds of waiting, just to be sure" to "either two seconds of waiting, or when all the tasks are completed."
  • This seems to resolve itself in a matter of milliseconds, rather than having the annoying wait for two seconds.
  • It's fairly safe to say, this is the fastest and most responsive the game has ever been. It's _lightning_ fast at responding to things. Mapgen is still blocking the UI in an annoying way, but that's a job for another day. Also a job for another day is the fact that the main sim execution can still get stuck, although I have trouble duplicating that right now. The surface area for hunting for that is lower than it was before, at least.

• In the right-click-the-clock screen, the "Details" row now shows a whole bunch of extra information relating to reasons why the sim might be blocked if it is.
  • Additionally, there is a "Sim Spots" line under it now, which has cryptic acronyms that count up every time they are hit. If the sim stops, some of those numbers should keep going up, while the others stop, and that will have narrowed down the area in which we have to search considerably.
  • If you run into a stuck sim thread, and can see anything interesting on the Details line, and/or identify where the numbers stop counting on the Sim Spots line, then I can probably figure out what the heck is going on and fix it.
  • Current theory: it's related to some of the super-ancient way that mapgen does its work, which is also actually one of the reasons that the lobby is less responsive than I'd like. This was already on my list to rework so that the lobby is nice and responsive even with giant maps, but I am betting that this also has to do with why the sim gets stuck sometimes.
  • Alternative theory: it has something to do with error handling. When factions have errors, evidently maybe it's more likely to appear? I was seeing it a lot more before I fixed all the exceptions I was running into, for instance.

• Yet more logic has been put in place to figure out why the sim logic gets stuck. It turns out it was after all of my other instrumentation, which is a big surprise, because it means that none of my guesses as to what might have been wrong were remotely correct. Now I've instrumented things in the other direction, and it should give us a picture of where the problem is. This is lightweight enough code that I'll probably leave it in for good.
  • Thanks to tom.prince for the report and duplication.

• Corrected a couple of places where I was re-throwing errors in a way that really did not need to happen. It's possible that these were the things that were causing exceptions to stop the sim from running, but it's hard to be certain. In theory, this sort of issue should have had no effect, because the finally block should have handled it. But I'm going to not trust that, since things are being a bit funky. Similarly, in our catch blocks on these relevant areas, I put in calls to our thread teardown method just to make sure that things get handled okay. We'll see if this makes a difference or not.

• Since all of the threads are now thread pool worker by name, the name is no longer included in the delayed writes. Just the thread ID.

• Completely reworked how I handle the "thread is finished or not finished" logic to be based around the actual task at hand and looking at it to see if it is complete or not.
  • Previously, I was trying to rely on try/finally statements and Interlocked.Exchange, and that was just not reliably setting values back to what they should have been. It really seems like something is amiss with that, to be honest, but that's not my problem for now.
  • The new code does seem to work with no more thread stalls, but I am now finding some new errors that are popping up. Where were these a bit ago? Were they hidden for some reason because of the nature of how the threads died? It's quite mysterious. Either way, so long as it works now, things are pretty straightforward in terms of how they are set up, so that's good. This approach is simpler and better, but it's really not clear why the other approach did not work.

Rework Of Threading, V3, Holy Cow That's Complicated

• Okay... updated lots of the codebase to no longer use try/finally, but to only ever use try/catch/finally. This was behaving in unexpected ways in some places where it was happening.

• Put in some extra error handling in a few places, and removed a few finally blocks where those were simply not needed.

• Updated the way that I'm kicking off the short-term sim thread secondary threads to be Task.Run instead of Parallel.ForEach, and then am using Task.WaitAll on the calling background thread.
  • This seems to be... a lot more efficient!? I'm not sure I wholly believe the stats just yet, but it is getting the work done so far as I can tell. I'm going to rework the stats-checking on this, but it seems to be doing the same work in 1ish millisecond or less on my machine, rather than routinely taking 30+ milliseconds for just PART of the work. That... really smells suspicious.

• I found that even with using the Tasks that were returned from running, I just can't ensure that threads are not running over top of one another. So I put back in the Interlocked.Exchange method of tracking when a thread is busy in a code area -- but only in three central classes, not in as many places as it was before.
  • So far, with the combo of these two things, I now seem to always have threads running properly, and never doubling up. I don't see any threads stalling, either, but then again I was less prone to seeing those happen recently.

• I still have some investigation to do here, and I'm deeply suspicious of the timings that are being reported to me (they are way too optimistic), but I can move ships between planets and engage in battles and I see everything happen properly, so that's a big step forward.

• A variety of more debug logging has been added to the Sim Spots line in order to chase down the sim lockup that Daniexpert is still seeing happen.

• Previously, for two of our main thread tasks types, I was tracking the start of a thread from an outer thread context that spawned it, and then decrementing that when that thread ended. It seems highly likely that some of these threads for some reason were not actually spawning, and this means that of course they can't finish.
  • I've adjust it so that it no longer does the second stop-check or the initial increment until it's on a background thread. This should lead to it being impossible for it to get stuck in the way it was before (assuming that's what the problem actually was).

• Another two attempts to make sure that they always know when they are done.
  • Firstly, the teardown method's FIRST thing is to say it's done, rather than that being a secondary thing.
  • Secondly, it no longer keeps track of the Tasks that it is given back at all, just using the Exchange method only. The idea here is that a task might not be properly cleared, and might be running for unrelated reasons, or who knows.

• For the main simulation thread, it now treats itself as aborted after half a second of being stuck. There is now a tracker that will show up on its line item with the number of aborts that have happened, if any have happened.
  • We're not literally aborting any threads, but we are assuming that they died on their own in the woods and that we need to keep the game running. Should be interesting to see what this does.

• The game no longer has a collection of execution contexts where there is only one for the main sim. It just has a single object, and looks at it directly.
  • This slightly changes the error messages and so on, more than anything else. It's unlikely to fix anything, but it's a way to see that things are different, and it is probably very slightly more efficient, either way.

• Adjusted the setting of the abort timing to be set only on the inner thread.

• Also put in a new "Time Styles" line at the top of our right-click-clock menu.

• Lastly, put in a "would abort at" time for the main sim thread. Whatever that number is, it should be aborting the main thread if it's less than the NSGT timer.

• Split our timekeeper NonSetupGameTimeSinceLastLoadOrStartF into two parts.
  • NonSetupGeneralGameTimeSinceLastLoadOrStartF and NonSetupNetworkReadyGameTimeSinceLastLoadOrStartF.
  • The latter is blocked by things like the sim not running because of network lag, so that we don't get things like a bunch of threading errors because the simulation is waiting on other players in MP.
    • However, it was also being used to gate some things for the simulation itself, so at the very least it was preventing the threads from aborting properly. Was it also causing other issues? That is... really hard to know.
  • The former is new, and is now used mostly just for the aborts. If an abort happens with the sim frame, it should wind up progressively ahead of the other, I think. At any rate, the soft aborts should now work properly, regardless.
  • Thanks to Daniexpert for helping discover this error, which is probably somehow or other related to our locked threads from the start.

• The game no longer says "would abort at" next to the sim line. It only shows information for actual aborts, if any happen.

• It turns out that my split of things into ArcenShortTermPlanningCodeExecution was not the best. I've merged that back into ArcenShortTermPlanningContext, which is once again abstract instead of sealed.

• Fixed a new bug from today where the times for the short-term planning contexts were not being tracked properly.
  • This had also introduced some lag since strength counting was thus also no longer doing any time-slicing.

• In the thread tracker window, you can also now see how long it has been since the short-term planning contexts were run.

Threading Works, But Chasing A Deadlock

• The game now has a system of "accumulators" that run all the time, at very low CPU cost, which give information about parts of the simulation and similar that are running.
  • In the event that there is an abort of the sim, in the new style that we have, this is something that means we either have a deadlock or we have an infinite loop or who knows what. But it needs to be solved... where is it, though??
  • These accumulators act like lightbulbs for showing us at least generally where the problems are, and they get more specific as we need them to. Any that show up with an orange or red number are indicating a spot where the code was not reached because something immediately prior didn't exit properly.
    • If you've had any exceptions trigger, then this will lead to some red or orange numbers, and those are to be ignored. But if you've had aborts but no exceptions, then this is a way for detecting infinite loops.
  • To see these, you can just right-click the clock. If you have information on there at the top, then just do a screenshot and upload it for us. Or tell us which abbreviation(s) had the number, either way.
  • In the future, this can also be used by modders who accidentally shoot themselves in the foot with some sort of infinite loop. It happens to the best of us, and the normal ways of going about finding this sort of thing are really insufficient in unity, so here we are.

• The various short-term threads all now show how many times they have started and finished. If one deadlocks or gets into an infinite loop, we'll now see that in the right-click-clock menu.

• All of the code accumulator timings are set to only show the most recent 10,000ish out of their accumulator, but that's just a visual trick. Under the hood, it keeps the whole numbers, but visually it's easier to parse if these numbers can't get infinitely large.

• Combat is an area that is deadlocking, so now we have more information in accumulators for that, in their own section.
  • Additionally, when paused we no longer run some of the combat logic, which keeps things lower-load when paused. A lot of the sim runs even when paused (by design), but just doesn't move ships, etc. These are some calculations that really are not needed when paused, though.

• Added two sets of custom accumulators that can be used wherever someone wants to. They're on Engine_Universal, so they can be used in any of the dlls with ease.
  • These are meant to be set up by a programmer on a temporary basis, not used permanently. This gives modders the flexibility to instrument some code and see where the problem might be.
  • In my case, it also lets me quickly reuse some easy variables for testing purposes to narrow something down, without having to completely extend the framework in ways that might not be good for performance long-term. Win for everybody.

• Narrowed down one of the infinite loops (or deadlocks, whatever it is) to DoEntityStepLogic_Ship.
  • From there, narrowed it down to DoSystemStep.
  • And from there, to ActuallyFireSalvoAtTargetPriorityList.
  • And there to InternalCreateActualShotForSalvo. Good grief, what a rabbit hole.
  • ...and then into SpawnShot_ReturnNullIfMPClient.
  • ...and then the real kicker is that this eventually led me to find a deadlock in TimeBasedPool, specifically when it was asked to give a shot back. Sometimes it would just hang forever.
    • This is one of those super strange cases, because this seems to work fine on the first run of the game, but then gets progressively worse. Why would that be? It is not remotely clear.
    • That said, ever since I laid eyes on the ConcurrentBag's internal code, it has made me nervous. And the performance profile has not been any better than ConcurrentQueue. Internally to TimeBasedPool, I used... ConcurrentBag.
    • I've now changed over to using ConcurrentQueue internally to TimeBasedPool, and it now works perfectly. I was able to reliably get a deadlock in a specific place, and there was a secondary deadlock also frequently coming up that I had not even chased yet. I can't replicate either, after doing five times the unit testing I would normall do to replicate it just once. At this point I can't get the game to do a sim thread abort at all (this is a good thing), whereas I could replicate it in under 30 seconds, before. Okay, so I tested well more than 5x normal.

• We had 41 other places where we were using ConcurrentBag, mostly internally to pools, or as part of xml parsing, or for network messages.
  • Those have all been converted over to ConcurrentQueue.
  • Additionally, ConcurrentBag has been removed from our codebase.
    • It's possible that the bugs in this were something I introduced, rather than being something specific to the Microsoft version that I pulled from their open-source repository.
    • That said, the performance profile and functionality of it is extremely redundant with ConcurrentQueue, and you can just tell how well ConcurrentQueue is made compared to ConcurrentBag, which is way less impressive.
    • So, regardless of if this is my error or not, it doesn't really make sense to keep around both of these data structures. Nominally, ConcurrentBag exists to be better-performing than ConcurrentQueue, since it is unordered. However, in practice that just doesn't happen.
      • I have a strong suspicion that this was not a bug I introduced, but instead is actually a problem with ConcurrentBag in its original form when it's being called from too many threads. You can see in the code how ConcurrentQueue goes out of its way to handle some of that sort of thing, but Bag does none of that. Seems fishy. Either way, I don't need it.

Beta 3.746 Phantom Ship Syndrome

(Released November 5th, 2021)

• Previously, it was possible for you to have "full metal" of zero, because you could get a metal storage amount of 0. This made a game basically completely unable to be played further after you have lost all your command stations.
  • Players are instead now given 10,000 metal storage for free, which means that you can survive and limp along if you want to play post-loss. It also means that you'll never get a misleading "full metal" when that metal is zero.

• Move several DLC3 galaxy settings into base game.
  • This includes some additional roguelike options like Hidden Galaxy, and also Border Aggression is now on by default in base game.
  • Thanks to Badger for updating.

Bugfixes

• Updated the last seven compound collections to do the lazy-initialization of their innards.

• Fixed a logic error in CyclicalArrayPool that was based on my changes to the lazy-initialization yesterday.
  • This is an unusually complicated class, and I made a slightly wrong choice in how I implemented the new methods, and it caused exceptions and many visual artifacts.
  • Thanks to Badger for reporting.

• Put in some more defensive code in SquadVisualizer just in case units are in a wrong state when they get there.
  • Thanks to Daniexpert for reporting.

• Fixed an issue with the science sidebar where FactionDoesNotHaveEnoughScience was being considered a critical bug rather than a normal part of operations.
  • This was a consequence of centralizing some of the science checks, which was a good move to do, but wound up with this side effect.
  • Thanks to Badger and The Beast of Bognor for reporting.

• Fixed an exception in SortedDictionaryOfSortedDictionaries where it was not properly getting a placeholder empty list when it had nothing to report, and was instead erroring.
  • Thanks to Badger for reporting.

• Fixed the "Exception in entity tooltip text generation at stage 5" error that has been annoying folks really lately.
  • This is related to some of the quarantine code, which I still need to directly fix, but having this gone is good either way.
  • Thanks to Badger, Daniexpert, and UFO for reporting.

• Adjusted things, a bit experimentally, so that the DataForMark, TypeData, Planet, and PlanetFaction can never be set to null on squads, shots, or wormholes.
  • If the game attempts to set one to null, then it just ignores that. If they are already null (never properly initialized), then it tries to find a sort of random non-null value to give back.
  • This is pretty similar to what we're doing with other pooled data, where we don't want certain things to become stale. The quarantine period should have been enough to bypass this, but frankly sometimes something was getting over-aggressive and having the PlanetFaction in particular as null well before it ever reached the point of exiting quarantine.
  • Since everything is pooled, having stale data references in here should have no effect, because we're always updating these things to the proper values when we pull them out of the pool to use them, anyway. I've tested this over several savegames and fresh maps, and things seem to look sane, so we'll see. I'm not expecting a bunch of issues out of this, but you never know.
  • Thanks to tom.prince for helping to narrow down what the root issue was, although the actual root cause of it in the code is still a mystery. Ultimately, it doesn't matter -- having any section of the code able to torpedo the rest by setting a null like this is probably not wise. So now it should be protected against all of that.

• Fixed another error in CyclicalArrayPool, this one in Reset. It was causing exceptions during battles.

• Fix a bug with necromancer tech levels going to 8 by mistake
  • Thanks to Daniexpert for the bug report

• Tweak the necromancer's early game to make it a bit smoother

• Improved the display of debug "construction can't build" slightly.

• When you load a savegame that has already been won or lost, it no longer plays the win or loss animation when you first start it up. This made testing certain saves, or continuing to play past death, pretty annoying.

• Fixed some exceptions that could happen if you closed down the game during pathfinding and then reloaded a new one. The exceptions would happen in the new game, when it thought that a pathfinder was still running -- when in fact it had been validly shut down by a ThreadAbortException, probably.

• Fixed a bug that may have been new in this build, it's hard to be sure. At any rate, sometimes the DataForMark was for the wrong TypeData, which led to scaling issues on ships, visually, amongst other gameplay problems. This may have been a consequence of my earlier fix to things to deal with quarantining, but either way it's good to have this resolved.

• LoosePool has been removed from the game, as it was very clearly not needed compared to ConcurrentPool at this point.
  • I made this, and some other related changes to the names of some of the visualization components (for the sake of clarity in this case) while hunting the "ships still show after death" bug.
  • I really can't duplicate that, but I CAN duplicate the bug with forcefields sticking around after death, and even being visible on the galaxy map.
  • The good news is, after this changeover, now the forcefields are properly cleaned up after reloading into even the same savegame, let alone another one. So it's a less-severe leak, despite still being a leak.
  • Thanks to Eluthena for reporting.

• The "bug with multiple pathfinders" happening at once was happening super frequently with the Elderlings for some reason. This has been adjusted so that it now uses interlocked to determine if it's still running, and also sets timing to let us know how long ago it was started, too.
  • Since this is something that runs synchronously, and which is run on the LRP thread, which in this case is SUPPOSED to only run once per 10 seconds, this is all very strange. I was seeing the error roughly every 70 seconds in a test session; not sure what's up.

• Fixed a bug where various squads (sometimes with selection rings, other times with forcefields) were not properly cleaning themselves up and dying.
  • I really don't know why this changed, but this logic has always been squirrely. Right now, the gimbals (icons) are properly exploding, but the ships themselves just disappear, which looks bad. But it's much better than the other bug we had for a while. I'll deal with that one on Monday.
  • It's highly likely that this was also a performance drain, and potentially a memory leak, prior to now. Go figure.
  • Thanks to Badger, Eluthena, and in particular Daniexpert for a handy reproduction case.

• Put in extra error handling in GetAIToPurchaseCostPresentForBudgetType. Not sure if the actual root bug is fixed by other code adjustments or not, but if it ever has another problem, then we'll know exactly where said problem is.
  • Thanks to Daniexpert for reporting.

• Put in protection against a really rare and strange bug inside MersenneTwister (our random number generator algorithm, by Takuji Nishimura, ported to C# by Akihilo Kramot). This thing has been a gold standard in prng since 1997, but apparently I've found some multithreading edge case that can cause issues with it. That's fixed.
  • Thanks to Daniexpert for the very unexpected report.

• Fixed a fairly funky bug in Sappers UpdateConstructors, which actually was in BuildStructureIfPossible, but because of its placement must have been related to the tracing buffers or something. Or more code has changed since the original report than I realized.
  • At any rate, there's now a lot more instrumentation, so if the bug recurs, then we'll have a much more detailed idea of where it is. This is a whole class of bug that should largely be fixed by this release now, though.
  • Thanks to Daniexpert for reporting.

• Fixed a rare exception that could happen in GetStringValueForCustomFieldOrDefaultValue.
  • Thanks to Daniexpert for reporting.

• Fixed a nullref exception that could happen in ProtectedList.Clear.
  • Thanks to Daniexpert for reporting.

• Silenced a harmless error in TextMeshPro that was sometimes spamming the log, particularly on game exit but not always limited to that. ("UnityEngine.Material.GetFloat")
  • Thanks to Daniexpert for reporting.

• After attempting a fairly disastrous tasks-based replacement for our thread-based approach to the main background threads, I've instead started looking into improving the thread-based items.
  • First issue: if there's a missing faction or deepinfo as it tries to run LRP, it will have an error (if it gets to that part of the code) and will also show the thread name appropriately. Thanks to tom.prince for the info on this avenue. I don't know if this is solved or not, it's hard to tell just yet.
  • Second issue: It is possible for all of the faction threads to be unassigned to a faction. I don't yet know what causes this, but it seems to be repeat loads of various games. Daniexpert first reported it today, and I've seen it once so far. I will have to chase this further on Monday.
    • Reproduction steps: load one savegame twice. Start a new game. Quit to main menu. Load savegame again. Boom.
  • Third issue (solved): I never have been able to figure out why the threads seem to kick off so much more frequently than it seems like they should. That in and of itself seems to be a problem. It turns out that I was having these run at sim-speed, and I often play at 5x, so it was doing waaay too much on the background threads. Now it actually uses real time. This may also help with some theoretical issues with gametime-based "last kicked off" logic.
  • Fourth issue: for some unknown reason, various non-faction threads sometimes get stuck. Executioncontext is the main one, but there are others. In these cases, Daniexpert was able to log that we get to the "THREAD Kicked next cycle" code.
    • This... is significant, but I'm not quite sure of the nuances why. Essentially, the thread thinks it is done, but has not died or cleaned up. What I now have it doing is killing the thread if the thread has been in this state for at least 4 realtime seconds. I may tighten that down to a substantially smaller margin if it turns out to be messing with the simulation, but I'm not sure on that.
    • Related to this, it looks like the Kill method had a bit of an oversight. If the thread was terminal, then it was not nulling it out and just let things go. Now it nulls it out, which makes the logic above do the proper thing. Or should.
    • Speaking of, if we try to run a thread and it thinks it is complete but still exists, we now check if it's terminal and then null it out if it is. I suspect that when we kill threads by going to the main menu and back into the game, sometimes it was having issues related to being in the terminal state but never nulled. Why this never mattered until recently is a mystery, as this code has not changed much until now.
  • Overall I expect things to be improved, and I'd be very interested in which cases still exist in the new code.
  • Thanks to Daniexpert, tom.prince, and others for related reports to this.

• Fixed a random threading issue that could happen in RemoveFactionInfluencer.

Beta 3.745 Deadlock Removal

(Released November 4th, 2021)

• New cheat code: grantalltech
  • All techs are immediately upgraded to their highest levels.
  • Thanks to tom.prince for adding!

DLC 3

• Added Orbital versions of all existing nasty picks besides the eyes, including DLC2
  • Thanks to Zeus for adding!

• Added a New Nasty pick -- the Orbital Inertia Missile Battery!
  • Orbiting defensive structure that exploits long distances to deal massive damage to enemies. Repulses smaller ships that get too close with inertia missiles, knocking ships back and inflicting engine stun. Inertia missiles do more damage to ships that are immune to the knockback effect.
  • Thanks to Zeus for adding!

• Replace the Templar Castle temporary weapon with a new temporary weapon for improved balance testing
  • Thanks to Badger for the code, and Daniexpert for the report

• Added the Orbital Polarizer Guardpost, adapted from DLC2.
  • Thanks to Zeus for adding.

Improvements

• Display galaxy wide cap in build sidebar now.
  • This will show a galaxy wide cap if there is one. If the galaxy wide cap is less than or equal to the current fleets cap, then *only* the galaxy wide cap is show (while still indicating it is a galaxy wide cap), otherwise, both are shown.
  • Thanks to tom.prince for the addition.

• Adjust the color of the galaxy wide cap, when that cap has been reached.
  • Thanks to tom.prince for the addition.

• Move checking of faction type for tech upgrades to Faction.CanUnlockTech, rather than the UI.
  • Thanks to tom.prince for the refactor.

• Minor buff to Super fortress. (Zeus) Saw that its stats didn't seem so super

Bugfixes

• fixed an issue with the orbital guardposts using tags. These don't support lists
  • Thanks to Badger for pointing this out and Zeus for the fix!

• Fixed the orbital AI files not being properly included in the last build for DLC3.
  • Thanks to Zeus for the fix.

• Fix a bug where Major Necropolises weren't correctly updating their buildable ships when structures are built. Also tweak the base skeleton balance
  • Thanks to Badger for the fix, and Puffin for the report

• Fix several bugs with the orbital AI type structures
  • Thanks to Zeus for the fix, and Badger for the report

• Fix `mypotluck` cheat command, which was broken during the refactor.
  • Thanks to tom.prince for the fix.

• Fix display of which settings are changed from default in setup lobby options tab.
  • Thanks to tom.prince for the fix.

• Changed how the tracking of lists and references is handled so that they are now spun off on their own background task threads.
  • These were leading to some deadlock situations in certain cases, some of which were rare, but not rare enough for my taste.
  • This does slow down the initial application startup time from about 8 seconds to about 10 seconds on my machine -- so an extra 2 seconds of load time.
  • However, I no longer can get the deadlock to recur in any of the cases where it previously existed, so that's excellent.

• I also added in some tools for tracking thread status in detail. It shows up in the setting menu without having to have advanced options turned on now.
  • The next time someone runs into stuck threads, if they don't mind turning this option on for a bit, that would be appreciated. This will rapidly fill your log, so it's not something worth leaving on for a longer period.

• Adjusted all of the various collection types to now lazy-load.
  • Well, actually there are still 7 that I didn't have time to do, but I got the other 30-something of them.
  • The effect of this is to make for less RAM usage, and relatively faster startup (and/or other intializations) where possible.
  • At the moment, just loading into the main menu with all three DLCs installed but no mods, this drops our total items in collections from 8.4m to 0.74m, and total capacity from 6m to 5.5m, and total collections from 250k to 76k, and references tracked from 392k to 181k.
  • Overall this is one of those things that will benefit the game more and more as there are more mods installed, keeping it so that things only actually spin up data stores as they are needed, rather than piling on a ton right at the start.
  • This also gets my own loading times back down to around 8ish seconds, sometimes 9, rather than being 10ish seconds; so it's helping to compensate for the extra startup time that I introduced with today's earlier work.

Beta 3.744 Pathfinding

(Released November 3rd, 2021)

• Added a new FleetSimLogic, which moves some of our fleet logic out of the closed-source Core dll into the open-source BaseInfo External dll.
  • No logic was changed at this time (by design), it was just moved.
  • There is also one for the fleet memberships, but for now that is empty. The one method that would have gone in there so far is more efficient when on the FleetSimLogic class.
  • The purpose of these changes is primarily so that we don't ever have to have anything pathfinder-related in Core.

• Orbital AI -- new feature!
  • No more stationary turrets, except for the wormhole sentinels. Instead, the Orbital Guardposts now spawn turrets that orbit the guardpost. These can be completely random or from a specific set to complement a given guardpost.
  • Thanks to Zeus for adding!

• Show unused AIP reduction, and remaining AIP until next mark levels.
  • Thanks to tom.prince for adding!

• Necromancy is now more easily practiced; previously it was hard for a non-flagship to deal enough necromantic damage to a fleetship without the fleetship already dying.
  • The base flagship no longer has a beam weapon; the beam weapon is on the blaster flagship. Instead the base necromancer flagship has multi-shot, to make it easier to necromance more ships early.
  • Thanks to Badger for updating, and Puffin for the feedback

Bugfixes

• Fix notifications not displaying in the prior build.
  • Thanks to tom.prince for fixing!

• Elderlings: fix some additional problems with planet allocation
  • Thanks to Badger for updating.

• Fix collapsing sidepanel groups layout not working quite right.
  • Thanks to tom.prince for fixing!

• Enable AI types that are marked as kiting to actually kite.
  • Thanks to tom.prince for fixing!

• Fix dispaly of AI type difficulty in game setup.
  • Thanks to tom.prince for fixing!

• Change a number of APIs dealing with planet indexes to use Int16 rather than int.
  • Thanks to tom.prince for fixing!

• Improve handling of attack orders given on galaxy targetting ships/structures.
  • The code previously assumed that the entity being attacked was on the currently selected planet, which is no longer true, now that you can attack things from the galaxy map.
  • Thanks to tom.prince for fixing!

Pathfinding Performance Improvements, And ThreadStatic RAM Improvements

• Added a new ThreadStaticWrapperedPooledItem, which is a way for us to keep some sort of complex object -- often a collection, but more things than that are planned -- in a threadstatic, safe way.
  • ...What?
    • Okay, so ThreadStatic. Basically, any time a new thread is using that variable, it can initialize its own copy. This is hyper useful.
    • However, with random background worker threads, this does mean that we wind up with some things initialized on threads that may never be used the same way again.
      • I'm referring here to things like Task<>.Run() or things that happen inside Parallel.For() or similar. Or older-school ThreadPool calls, whatever.
    • Additionally, we pool and reuse our own threads between loads of the game (meaning out to the main menu and back), for things like factions.
      • So if thread 554 was Dark Spire in one game, and you go to the main menu and come back into the same game, it might wind up being the AI Hunter thread instead.
    • Essentially this works vaguely like a memory leak, even though it is in no way a memory leak, because we wind up with stuff that has been allocated but will never be used again.
      • During our "before mapgen or savegame load clear," it makes more sense to recall all the threadstatic stuff, and pull those into central pools, and then start over fresh using them again.
  • How we do exactly that is... with this new class, ThreadStaticWrapperedPooledItem!
    • This wrapper item is instead now the thing that is threadstatic, and it communicates under the hood with a pool.
    • We can put anything in here. Collections, pathfinders, and so on.
      • A side benefit of this is that for things where we previous had a scope that was a "context" (that can be VERY confusing), the scope can instead be the thread.
      • Recently, we've been having some issues with pathfinders being called from more than one contextual location at a time, which might have been a false alarm or might have been a very dangerous bug. Either way, this will fix that.
        • But even more to the point, we might have been having some pathfinder bugs if people ever decided to use one from a given context in a Task or Parallel call. I don't think anyone was doing that yet, but if they did it would have given scrambled results.
    • A side effect of these changes is that we can allocate fewer lists and pathfinders in general, because not every faction needs them in the first place. And what we do allocate can get much more reuse.
  • If we felt like we needed to, we could extend this concept further and push kind of a library "check in, check out" sort of data structure that keeps things for less long than between map-generations.
    • That said, I think that there is potentially performance loss from that, and I don't expect the RAM usage of my current approach to be enough of a problem to warrant that extra CPU usage.
    • If I get proved wrong later, then I can make a second version of this general concept that prioritizes RAM savings instead of CPU savings, and it will be easy to swap out the ones where it's a problem.

• The types of end pathfinders have been condensed down to three: PlanetPathfinderBasic, PlanetPathfinderConservative, and PlanetPathfinderWarden.
  • There is nothing in there about any sort of long-term or short-term differences now.
  • Additionally, when you find a path, it now requires you to pass in the faction and any extra debug info when you do so.

• Added a new DictionaryOfProtectedValDictionaries, which is presently unused. Thought I would need it on CacheForPathingMode, but those are being pooled by map-reload, so no.
  • In general, a lot of the pathing code is now pooled way better between loads of the game. Previously... possibly it was lost and leaked? Maybe it was not leaking, but just kind of allocated in a useless place? Either way, fixed.

• Man, there's a whole bunch of other changes where things with pathfinding were really confusing to me. I didn't have time to document them all.
  • However, the new approach is really clear and has comments or naming that explains what the rules are, and why. Also the code reads linearly from top to bottom, rather than jumping all over the place. So that also helps.
  • For an end-faction-developer, very little is different. It does ask you for a "debug addendum" string, but that's just so that error messages will be more informatively precise if you should run into any.

• The sum total of these changes saves a LOT of memory, as well as making things load faster, as well as also eliminating the pathfinders-from-multiple-contexts bug. It also leads to some CPU speed improvements in several spots.
  • It also solves what was definitively still a memory leak in terms of pathfinders.
    • After 10 loads of the game prior to these changes, there were 169,706 PathBetweenPlanetsForFaction-PathToReadOnly collections, with 25,661,700 capacity and 115,592 stored items. This is absolutely freaking huge!
    • Now, after 8 loads of the game, there are 79 collections, 420 total capacity, and 19 stored items. This is... uh... quite an improvement.

• Thanks to tom.prince and Badger for reporting the pathfinder errors that led to all these improvements. This was a tough nut to crack.

Beta 3.743 Return Of The Imperial Spire

(Released November 2nd, 2021)

Improvements

• The FitListToSize_OrCreateWithDefault_OrThrowErrorIfEmpty() and MakeListAtLeastThisLongWithDefaults() support methods have been moved to ArcenDynamicTableRow since they can potentially do a lot of good where all XML-based objects can use them.
• Additions to the EntityTypeDrawingBag (ETDB):
  • It is now possible to define mixed bags of parse types - as in spawning a certain entity by its name and then random entities based on tags.
  • The same way differing count types (raw count, strenth-based, metal/energy/AI cost) can now be mixed.
  • Additionally there is 2 new lists: duplicate_pick_handling_list (GenerallyAllowed, AllowedForDifferentTiers, GenerallyUnallowed) allows to set whether or not duplicates may be picked, while multi_pick_list defines how many separate, if need be unique entity types of a "tier" are picked (say that there's a tag with 3 units, and 2 should be chosen - this is where the "2" would go to make that happen). The maximum that can be picked is the maximum possible individual entries per "tier".
• Outguard now instead of the primary/secondary/tertiary XML fields uses the ETDB, with some new ones:
  • min_spawn_count_increase_per_aip_interval is a list that defines how much is added to the min/max values of the spawn counts, while the intervals themselves are defined by the spawn_count_increase_aip_threshold list.
  • All XML was adjusted to compensate.
  • Thanks to NR SirLimbo for adding!

• Rename the Random Faction display name so as to get it toward the top of the Game Lobby 'Add Faction' menu. Otherwise it's a hassle when adding a couple random factions
  • Thanks to Badger for updating!

• Refactor the code for generating spire city and fleet names.
  • Thanks to tom.prince for updating.

• Brutal Lair: slow down the rate of Brutal Guardian spawning
  • Thanks to Badger for updating!

• Add a couple new Elderlings for diversity
  • Thanks to Badger for updating!

• Elderlings now grant essence when killed
  • Thanks to Badger for updating!

• Add a new Feeble Elderling that's seeded near the player in necromancer mode
  • Thanks to Badger for updating!

• Minor buff to sappear beachhead constructors
  • Thanks to Badger for updating!

• Slothful and Wrathful elderlings can now be hacked for necromancer blueprints to allow new flagship transformations.
  • There's an explanation in the XML for how one would set up other Elderlings to be hacked for blueprints
  • Thanks to Badger for updating!

• When the Necromancer is added to the game, it now also adds a copy of the Elderlings (AI-allied) and the Templar (AI-allied). These are visible in the game lobby so players can adjust the difficulty.
  • Thanks to Badger for updating!

• Gave the Templar and Elderlings some new icons
  • Thanks to Badger for updating!

• More Outguard work:
  • The spawn_delay XML field now actually works. Groups called in may take a delay in time until they arrive. Sidebar, tooltips and XML have been adjusted.
  • With the exception of the DLC 2 Outguard "Hyperion" all are still the same as before. Hyperion, as per original concept, now takes 30 seconds to warp in.
  • Thanks to NR SirLimbo for adding!

• Tweak the descriptions for macrophages and elderligns a bit
  • Thanks to Badger for updating!

• Add some new options for the Necromancer to make the game easier.
  • You can A. start with bonus resources
  • B. Start with higher available skeletons
  • C. Start with higher available wights
  • D. Start with a bonus ship line of wights
  • The goal is to make sure that even weaker players can enjoy playing the necromancer.
  • Thanks to Badger for adding!

• Add a couple of extension methods to ArcenCharacterBuffer for displaying human readable times.
  • AddHoursAndMinutes will display times like 5h 13m 20s, for use in description and tooltip text.
  • AddSecondsRemaining will display times like 13:20, for use in notification panels. By default, this will show times <20s in red and <60s in yellow.
  • I also went and changed a bunch of places to use the new functions. Most of the changes haven't been tested.
  • Thanks to tom.prince for updating!

• Convert a few more notification panels to use AddSecondsRemaining.
  • Also add an option to AddSecondsRemaining to use the 10 minute intensity variation from nomad planets.
  • Thanks to tom.prince for updating!

• There is a new GetShouldBeHidden() on the notification code that lets programmers specify that something should be hidden based on whatever they want.
  • Essentially, if a timer was counting down and would be negative, but we find that the notification is still in the list for a little bit, then instead this can just say "yeah, hide me," and it will disappear from the list at a proper time. This is particularly important for multiplayer clients, which might get data slightly late.
  • Overall nobody needs to go crazy with this function, but it exists as a new safety valve to help keep things from showing stale data. We already have this same sort of thing already on most of our UI element, so extending it here is an easy leap and quite consistent with everything else. It's only needed when it's needed, but it's nice to have it for those cases.
  • Thanks to tom.prince for the discussion that inspired this addition.

• List<> and ProtectedList<> now both have an IsHardened bool property on them.
  • If a list is hardened, then index out of range exceptions are considered fine. Basically those are threading issues that should just return default(T), which is generally going to be null.
  • This does not help with inserts, adds, writes, or clears. It applies to for, foreach, contains, find, findall, findlast, get-index, etc.
  • If you have a list that is mostly used one place, and is only written to on a single given thread (main sim probably), then you can just flag it as IsHardened = true and read from it from other threads. If one of those threads happens to go what would be out of bounds for the list (generally because something was removed from the list on its main thread while the other thread was reading from it), that's no problem. You'll just get a null back, which you can check for.
  • In general when reading from lists, read into a variable with one indexing call and then use that value after you've verified it's not null. This is better practice for performance anyhow. It then keeps your surface area much smaller for the list. No need for try/catch around the index into the list now.
  • Thanks to NR SirLimbo for inspiring this change.

• Since OutguardSpawnRequest is going to be seen on the UI for longer than it might exist in the actual working pool, it makes sense to put in the time-based pool like we do for units, shots, etc. This has been converted over, and it thus then calls its Clear() object when it goes out of quarantine and back into the main pool, which is 2-3 seconds after going into the pool (the UI list refresh will happen every 1 second, so that's plenty of time).
  • Please note that this does not have any effect on the discussion of the way that the lists are filled, or how the lists are accessed -- the things that NR SirLimbo and I were discussing. Pooling strategy is one thing, and cross-thread read/write strategy is another, and the two can be considered independently.
  • Thanks to tom.prince for suggesting.

• More under-the-hood Outguard work:
  • Put in the code for the new hardened protected list as primary list for the Outguard Spawn Requests. This should prevent any potential cross-threading exceptions when it's being read on an UI thread.
  • Clients are being kept (hopefully) perfectly up-to-date with Outguard spawn requests, so their sidebars also refresh as timely as possible.
  • Additionally it should no longer be possible to spawn the same group twice on hosts and clients by clicking basically at the same time, where the UI doesn't yet block the client from sending the game command to spawn the group.
  • Thanks to NR SirLimbo for implementing!

Bugfixes

• Mark items as coming out of quarantine when going into the main pool. (TimeBasedPool)
  • Thanks to tom.prince for fixing!

• Make spire transciever work again, and add a global cap to them.
  • Thanks to tom.prince for fixing!

• Make imperial spire spawn from transceiver as well.
  • This does make the imperial spire twice as powerful, in a solo game, but that is really incidental, since they already have overwhelming force. This is mostly for flavor reasons, I was surprised when testing out the transceiver that they were at least initially arriving from the transceiver that called them.
  • Thanks to tom.prince for fixing!

• Elderlings now properly join their allies in minor faction teams
  • Elderlings now mark up properly. They initially start with a 3 planet territory, then add one new planet when the Elderling marks up
  • Elderlings now try to max out at 3 Elderlings per Territory planet. They are now more organized in how to lay eggs, and to make sure that elderlings don't get too cluttered.
  • Thanks to Badger for fixing!

• Fireteams: fix a bug presumably from the memory-leak changes where we had two almost identically named copies of a list, and would get confused as to which we were using. There is now only one list
  • Thanks to Badger for fixing!

• Fix a bug where too many elderlings were winding up in the same space
  • Thanks to Badger for fixing!

• Templar Encampments spawn a bit less frequently, but should be a bit stronger
  • Thanks to Badger for fixing!

• AI-allied high-intensity elderlings will be more concentrated. Hopefully fix a null reference exception in the wave code
  • Thanks to Badger for fixing!

• Fixed an exception that could happen due to a race condition in HandleLODsAndShipPartAnimations/
  • Thanks to Eluthena for reporting.

• Fixed an issue where OutguardSpawnRequest was never being returned to the pool. It now centrally uses a ProtectedList rather than a List, which means it should automatically head back to the pool when we're done with it. (This was a memory leak.)
  • Thanks to tom.prince for reporting.

• Fixed an error in my build scripts for GOG Galaxy that was leading to the updates on the beta branch being stuck on version 3.313. I had thought that things were working fine since it was still having something to download every time a new version was uploaded, but that was not the case.
  • What happened was, I upgraded to a new computer a few months ago, and when I did the paths changed for where my build scripts pointed to. All my steam scripts use relative path names, but the GOG ones are apparently hardcoded and were looking to a wrong drive. I would have expected some sort of error, but there was not one.
  • I've now corrected the path names to look to the correct drive, and folks on GOG can now get the latest betas properly for testing again.
  • Thanks to slake-moth for reporting.

• In Disband_JustTheDeepInfoPortion, the "GetExternalBaseInfoAs<ScourgePerUnitBaseInfo>()" (which throws an error if it's the wrong type) now is just "TryGetExternalBaseInfoAs<ScourgePerUnitBaseInfo>()" (which is able to silently return null when it's not a match).
  • This is basically old code, as Badger notes, but the way that I had translated it into the new framework was suboptimal.
  • Thanks to Badger for reporting.

• Fixed several groups of errors that could happen when deleting campaigns, relating to exceptions and having the wrong campaign selected, etc. Some of this was indeed because we were using an asynchronous callback when -- after a delete in particular -- we needed to use a synchronous one for the sake of accuracy.
  • Thanks to Crabby for the report, and tom.prince for the diagnosis and suggested solution.

• Fixed a bug where if the campaign name was null, and you were NOT in the tutorial, it would start saving as "Tutorial".
  • That's been fixed so that it will now save as General, and if you're in a tutorial it will save as Tutorial.
  • Thanks to Daniexpert, Eluthena, and tom.prince for reporting.

• Fixed a bug in GameCommand_DeleteAllSaveGames where it was using the current campaign name instead of the one that was passed in.

• Put in error tracking in a number of places so that it will fail to save (and tell you about that) if the campaign name is blank.
  • Fixed a variety of places where the savegame or campaign name could be null and just cause problems or save into an odd space without warning. Now those places error, instead, so that we'll know when they happen.
  • When pooling the World object, we no longer ever reset the CampaignName. The first thing we do when using it out of the pool is set a new one, anyhow.
    • This should stop a variety of bugs with saving doing funky things since the great refactor.
  • Thanks to Daniexpert and tom.prince for reporting.

• Fixed a deserialization bug in wormhole invasions.
  • Thanks to Daniexpert for the report with the matching serialization logs that made this super easy to see in action!

• Corrected Nucleophilic turret count on Webbed battlestation template 5 -> 25
  • Thanks to CRCGamer for fixing!

• ArcenUI_AxisAlignment is now a struct rather than a class. It doesn't actually solve a leak or anything.

Beta 3.742 "Classic Map" For DLC1

(Released October 30th, 2021)

• Add a minimal UI for changing which fleet is being bolstered.
  • Thanks to Tom for adding!

• DLC3: Add a new Elderlings fashion.
  • This faction starts with an Egg. After some time the egg hatches into an Elderling.
  • Elderlings patrol the area around their initial egg. Every so often an Elderling can lay another egg.
  • Able to be allied to player/ai/minor faction.
  • Thanks to Badger for adding!

New Map Type For DLC1!

• DLC1 has just not been popular enough with folks, so we're deciding to sweeten the deal and add some cool new things to it. You'll see some minor new factions, some notable cool new features, new units, and a new map type, among other things.
  • We're going to be renaming that DLC in order to make it more clear that this is a majorly expanded DLC compared to what it originally was, and when DLC3 comes out, we'll also be raising the price of DLC1 to match that of DLCs 2 and 3. Essentially, they're all going to be suitably giant to match DLC2. So if you want to get it for a couple of bucks cheaper, now is a good time actually!
  • In the meantime, the new map type has been added to DLC1, courtesy of Badger: Classic.
    • Practically, it's a mix of bunch of clusters and lonely planets with lots of connections; it has a bit of flavor of the maps from AI War Classic.

Bugfixes And General Improvements

• Use AddedMarkLevelsForFleet_FromScience rather than CurrentMarkLevel to determine mark of Spire ships.
  • Thanks to Tom for updating!

• Create second spire fleet at first Mark 3 city.
  • Previously, it would always spawn at the Capitol.
  • Thanks to Tom for updating!

• Include planet name in Spire City upgrade text.
  • Thanks to Tom for updating!

• Fix incorrect fleet name showing up in several tooltips.
  • In a number of places, it would always show the command station fleet of the current planet.
  • Thanks to Tom for fixing!

• Display the city each Spire Fleet ship line is bolstered from in the tooptip.
  • Thanks to Tom for adding!

• Fixed a place where I missed a fourth codepath for the "SaveGameData Create" log, which was what led to the log spam. I didn't actually forget to compile anything, looks like.
  • Thanks to Daniexpert for reporting.

• Fixed a bug in the dark zenith deserialization that was preventing savegames with them from being loaded.
  • Thanks to Daniexpert for reporting.

• Fixed a harmless but annoying typo in the logging for Zenith Architrave data.
  • Thanks to Daniexpert for reporting.

• Fixed another annoyance with log spam, this time with nomads spamming the log.
  • Thanks to Daniexpert for reporting.

• Territory dispute factions no longer clear their fireteams before calling FireteamBaseUtility.DeserializeFireteamsAndDiscardAnyExtraLeftovers, since that would lead to spotty data in MP.

• Fixed an exception (a boolean inversion) that was preventing any fireteams from being deserialized.
  • Thanks to Daniexpert for reporting.

• Put in more instrumentation into ChooseWaveTarget, so that if it has an exception it will now be more clear where that was, and also still potentially return something marginally useful.
  • Also then fixed an exception that could happen when validly reporting "no target from X location." The code was looking for an old-style answer that no longer is possible since these became pooled, but has been corrected to look for "valid no-data" answers in the new way.

• Fixed an exception that could happen in ShipVisualizer.RenderTarget.

• Necromancer bolstering seems to work. Big thanks to Tom for his fallen spire work that I copied.
  • Thanks to Badger for adding.

• Necromancer flagships are now better at seeing when they have hacks.

• Fix a bug where we were breaking out of the wrong loop in the hacking display code.
  • Thanks to Badger for fixing.

• Mummies now load into flagships.

• Fix a bug where each necropolis bolstering a fleet was clobbering the 'how many ships are granted' data.
  • Added a mention of this to the Fall Spire code in case someone else sees the same problem; I'm not investigating further for the fallen spire though

• Clear tooltips before clearing pools, when returning to the main menu.
  • Thanks to Tom for fixing.

• Make various functions in Window_InGameHoverEntityInfo take GameEntity_Squad instead of GameEntity_Base.
  • As it is, that is all they handle anyway, so make that clear from the signature.
  • Thanks to Tom for updating!

• Simplify some code in GalaxyViewSelector.cs around type conversions.
  • Thanks to Tom for updating!

Major Time-Based Pool Improvements

• Previously, the game used ITimeBasedPoolable and TimeBasedPoolable for any sort of pooled objects, even if they were going in the ConcurrentPool.
  • This was a bit confusing (potentially, anyway), and also limiting (definitely), since the two types of pools now have different needs.
  • So we now have IConcurrentPoolable and ConcurrentPoolable, which are used in the vast majority of cases now, since mostly we use ConcurrentPools.

• Time-based pooling has been majorly expanded to now have a three-stage process.
  • There's a quarantine phase that is far more clear now, and at the end of that there's now an extra clear step rather than there only being "enter pool" and "exit pool" clears.
    • This lets us hang on to certain references that would otherwise cause errors at various parts of the application, while still easing the objects out of the simulation in other ways.
  • Additionally, the "advancement through buckets" for time-based pools now happens on a background worker thread instead of the main or sim threads, which is WAY more efficient. This keeps the framerate and sim speeds up.
  • Also? It turns out that, all this time, time-based pooling has been running too fast, largely because I was trying to avoid excessive calls to take and put from the quarantine batches. This led to all sorts of anomalies.
    • Now that I'm running the time pool logic on a new random background worker thread, we can afford the extra time to make sure it's done right.
    • In general, this particular thing also is not quite such a large load as I'd once feared it might be (then again, I have the luxury of saying that when it's not blocking the sim, right?)

• The squad pool was set to be stupidly long (30 increments of 12 seconds each, wow).
  • It is now set to be 3 increments of 20 seconds, which is a lot more efficient.

• Shots, transported-ships, fleet-memberships, and external and base infos have all had basic translations over to the new time-based pooling for maximum efficiency and minimum chance of errors.

• Squads/ships have had the largest transition, and a FAR larger amount of their stuff now happens on the new Mid phase.
  • This both speeds up the deaths and reuse of squads (major boon on many threads), but also helps avoid lots of random errors that could otherwise happen in LRP threads or in the UI, particularly notifications.
  • While in this part of the code, I also discovered that there was a leak with the external baseinfo/deepinfo for squads, and I fixed that. Whoops.

• Fireteam history is no longer in a time-based pool. Because it's a sub-object of fireteams, it makes more sense to just be a concurrentpool, which is how we handle systems on squads.

• Fireteams themselves were updated like shots, fleets, etc, were.

• Squads now have a PermaDebugNonSimID on them, which never changes and which has no sim-meaning. It's different between the same units on different computers, and between different runs of the same game on one computer, rather like a hashcode for many objects in C#.
  • We can use this to trace the life of a literal squad object as it changes over time, if we need to, even as its identity is rewritten around itself.
  • This has been used to verify that the timings on the time-based pooling actually finally work properly for apparently the first time ever.

• Thanks to Tom for the discussion that led to this change in how we do things with time-based pools, as this will make a LOT of code way less prone to random exceptions.
  • As he points out, this is a little bit similar to RCU, which is a neat coincidence but a great way to think of it it: https://www.kernel.org/doc/html/latest/RCU/whatisRCU.html

Memory Leak And Performance Work (Ongoing Part 2)

• Added the ability for our dictionaries to expand in a non-doubling fashion, same as we have for our lists. It's not used very often, but in certain cases it can make for more efficient memory usage.

• My CrossThreadSwizzleLists for ship-to-ship beams, etc, now use a lot less memory and will also consequently no longer even try to draw more than 9000 beams of a single type in one frame.
  • If more than that are requested, it just ignores the requests beyond that number. We've never had anywhere close to that number requested at once, and it would tank pretty much any GPU, so this makes sense. The limit before was 90,000, and there was no safeguard if it was exceeded, because it could not feasibly ever be reached.

• I'm making many various other little updates to collection capacities and capacity growth curves as I go through Daniexpert's core dump. It's not worth noting them all, but it's just little tunings here and there that make things work better.

• I decided to take a break and do a manual review of all of the classes in ArcenUniversal, of which there are apparently 464 at the moment.
  • I reviewed them all manually to see if they were suspicious or what, particularly paying attention to things in the realm of UI. For those that were suspicious, I've added some instrumentation so that we'll see in the future if they leak.
  • For four others, I find them worthy of further investigation, and I've just noted them down for now.

• Removed the PlanetFactions list on the Faction object, as it turns out it was never filled and also never used (thankfully). The PlanetFactionEntityLists list is what is actually used, and that's been left alone.

• Made it so that parallel processing of planets is a lot more efficient in terms of both CPU usage and RAM usage.
  • I had been making a working copy to iterate over, and there was a much better way to handle that, sigh.

• The Queue class also now supports non-doubling increases in capacity, like our List class.

• Lots more various improvements in how various collections allocate their capacity.

• TemporaryBuffer has been removed from ArcenCharacterBuffer, because it was not threadsafe. Also it was not even used in this game!

• ArcenCharacterBuffer can no longer be directly created, but instead is borrowed from a pool and then expected to be put back.
  • This is one of the largest memory leaks that we definitively still had, and it is now fixed.
  • Any time we were writing a key combo to the screen it was a major leak of these, turns out.

• ArcenXMLSerializationBuffer now is created, used for a while, and then disposed, versus being kept for the long-term.
  • We're no longer doing reference tracking on these, since they don't have any data unique to themselves. Everything is part of the ArcenCharacterBuffer except a thin wrapper here around it.
  • In fact, these are now structs, since they contain so little data.

• A whole bunch of debug and tracing output code has been updated to use the updated ArcenCharacterBuffers that are pooled.
  • The method now is more efficient, and also more robust against small typos.
  • For example, it turns out that there was an infinite-appending memory leak in code that was meant to be just for tracing in CheckForWarFooting for Zenith Architrave, but instead it was leaking infinite strings over time even when tracing was off.
    • There were some other examples like this, but this was the most notable one where it looks like it would cause gradual memory degradation over time until the game is restarted (or until actual tracing was turned on for the ZA).
    • In the event of something along these lines in the future, we'll see one of two things: 1) an exception (yay!), or 2) a rising number of ArcenCharacterBuffers being allocated and never put back. Prior to today, it was just invisible ram usage since we're not keeping track of "excessively long strings" or something of that sort.
      • That said, in the normal case of operations (aka not tracing something), it should just show up as an exception, which is the best result. Leaking character buffers would likely only happen during tracing, if it were going to from tracing activities.
  • It's vaguely possible that there are some leaks of character buffers when doing tracing. Given that this is only something developers or modders are likely to do, and only for brief times, this does not seem to be a big deal. However, there also should be a minimal number of these (it was a lot of work).
    • Actually, despite all my work, it's pretty likely there are some. At least they are minor.
  • It's also likely that there are now some random nullref exceptions scattered in the code that were previously infinitely-growing strings that memory leaked. That's unfortunate, but better than the alternative. I examined all of them by hand, but there were 1158 of them, so I had to skim kind of quickly.
    • Found some of those in planning wave (very rare), ZA (super hyper very very frequent), templars (minor), dark spire (moderate), scourge (very frequent), dark zenith (very frequent), base-oriented warden (minor),

• Fixed several tooltips in the resource bar and the bottom left menu that would leak memory (and ArcenCharacterBuffers) like crazy when you hovered over them.

• If a blank or null string is passed to SingleLineQuickDebug or ArcenDebugLogSingleLine, these are now ignored.
  • This is useful to have more brief code with certain tracing code. Sometimes it would accidentally be sending something blank, and that's fine and should just be easily ignored.

Beta 3.741 Clogging Holes

(Released October 28th, 2021)

• Fixed two typos related to Necromancer

Memory Leak And Performance Work (Ongoing)

• Added a new ProtectedArray collection, which simply lets us have automation with things like items going back into the pool, etc.

• The MarkLevelStats on GameEntityTypeData is now pooled, and uses the ProtectedArray, so that when the game reloads all its xml it should now also properly flush this out.
  • Same deal with EntitySystemTypeData itself, and MarkLevelStats on EntitySystemTypeData.
  • And also HackData, on GameEntityTypeData.

• Added a new ProtectedValue "collection" (of exactly one item), which again gives us that automated protection.
  • This is now used on the EntityTypeDrawingBag references on GameEntityTypeData so that they properly get pooled when there's a reload.
  • Please note! We're using this on things like PeriodicSpawn_EntityTypeDrawingBag because this IS the data, and so we want to make sure and clear it as well as the reference to it. So we DO use ProtectedValue.
  • Meanwhile, for things like UnitToMakeWithBuildPointsFromDamageTaken, that's just a reference to data that exists independently elsewhere. We'd want to clear the reference to it, sure, but actually cleaning up the distant data is not our call on GameEntityTypeData; it's not in charge of that data. So it would NOT use ProtectedValue.
  • More often than not, the proper decision is to NOT use ProtectedValue.

• Added better error handling for "DoForAllElementsIncludingParentsAndPartialsInForwardOrderFromOldest"
  • If it has an exception, it now tells you what the sub-node was that it was looking at, and also doesn't stop processing of the rest of its row (thus preventing an error cascade that gives misleading errors that would go away if you fix the actual core error).

• Working on memory leak hunting:
  • ArcenDynamicTableIndexSerializationInfo allocation capacities are now more tame. Also, these were leaking, it turns out, so they are now pooled and reused instead.
    • Same with ArcenNonTableIndexSerializationInfo on all fronts.
  • Thanks to Daniexpert for the core dump that led to these.

• Yesterday I split ArbitaryArcenOption so that it was an abstract base class and then also had an ArbitaryArcenOptionStandalone sub-class for being used.
  • It turns out that this was a terrible idea, and on top of that I had some other classes that were inheriting from ArbitaryArcenOption in a way that was really not helpful.
  • So, I'm combining ArbitaryArcenOptionStandalone back into ArbitaryArcenOption, and then making that class sealed (so nothing can inherit from it, because that's "crossing the streams" and gives very bad behavior).
  • Faction configurations and galaxy options continue to now use ArbitaryArcenOption directly, and it works as expected.
  • CampaignOrQuickstartFolder, CampaignOrQuickstart_SubFolder, and SaveGameData all now directly inherit from ArcenOwnedRowBase, which is the root behind both ArbitaryArcenOption and ArcenDynamicTableRow.
    • They also then implement IOption, that excellent idea that Tom suggested yesterday, and pool themselves without any worry about strange inheritance patterns.
    • I've also made these sealed, so nobody is ever tempted to inherit from them and cause the same problem again, just at a new inheritance depth.
  • These are verified to no longer be leaking, but these are definitely not the largest part of the leak that is in the core dump from Daniexpert. I haven't gotten to that yet, but I'm going through methodically. I've also fixed up a number of capacities to be more sensible based on seeing real-world core-dump data, and the savegames now load faster than ever, especially on second load and beyond.
  • Thanks to Daniexpert for the core dumps for this and other related items, and to Tom for the IOption idea.

Beta 3.740 Code Panopticon

(Released October 27th, 2021)

• Lost Spire Frigates now have "Roam if instructed" set by default
  • Thanks to Wuffell_1, TheIronBird and all the others for suggesting, and to Daniexpert for fixing!

• Add some new integers, StoredFactionResource[One|Two|Three] to the faction object. This is for future modding support as well as the necromancer.
  • The Necromancer now treats FactionResourceOne as "Essence", which is obtained by hacking Rifts.
  • Essence is used to do all necropolis and flagship operations, which are spent through the Hacking sidebar.
  • Thanks to Badger for adding.

• DZ Svikari: Support AI allied variant. Currently only in DLC3 for testing
  • Thanks to Badger for adding.

• Allow stationary flagships to travel to specific planet locations with modifier.
  • Thanks to Tom for updating!

• Add support for autobuilding forcefields.
  • While we are here, refactor the autobuild code somewhat, so we only loop over the command stations once, rather once per type of automation.
  • This could probably do with being somewhat more data driven, though.
  • Thanks to Tom for adding!

UI Updates

• The planet sidebar menu now shows the actual hull points of crippled watched/local flagships, so you can easily monitor their repairs. It still includes the text 'Crippled' though, so players should not get confused.
  • Thanks to Badger for updating.

NA Balance changes

• Magnet ward starts at mark 1 (instead of mark 5).
• Sabot ward now fires in a burst of 3
• Angelic ward strength multiplier reduced from 5 to 3.
  • Thanks to Democracy! for pointing out this issue!

ZO Balance Changes

• Cruisers stat adjustments. Minimium armor thickness from 50 mm - 175mm. This fixes several cruisers, most notably the starfish, which was vulnerable to all sources of weapon jamming.
  • Thanks to Democracy! for pointing out this issue!

Bugfixes

• Fixed a missing XML field in the Brutal Guardian Lair, causing it to complain about not finding any Brutal Guardians.
  • Thanks to Badger for reporting.

• Fix loading NecromancerUpgrade table when DLC3 isn't available.
  • Thanks to Tom for fixing!

• Mark orbital guard posts ship groups with `can_be_empty_or_filled_and_no_complaints`.
  • This allows them to be loaded without error without DLC3.
  • Thanks to Tom for fixing!

• Fix some map description typos for Swirl and Bubbles.
  • Thanks to Tom for fixing!

• Make sure that deepstrike eligibility is updated even when there are no deep strikes.
  • This was causing deepstrikes to never happen, as no planets would be declared eligible.
  • Thanks to Tom for fixing!

• Save the name of PlayerCustomCity fleets. This avoids Spire City Fleets from having generic names after loading a save game. This breaks the savegame format for games with Spire Cities in them.
  • Thanks to Tom for fixing!

• Clamp maximum distance to point in BadgerUtilityMethods::GetExtremelySafePointNearPoint to 5 times the desired distance.
  • If the distance the point can be placed at is not limited, it is possible to end up with maps where one or several satellite planets (in the X or Linked Rings map types), are an order of magnitude or more farther away then the rest of the planets in the galaxy.
  • Thanks to Tom for fixing, and cml, Wuffell_1, and Ecthelon for reporting.

• Clear the savegame selection when deleting a save in the load menu.
  • This prevents errors when clickling the `Start` or `Lobby` buttons afterwards.
  • Thanks to Tom for fixing!

• Don't randomize the number of spokes in a perfectly symmetric X map.
  • Thanks to Tom for fixing!

• Achievements for the DZ now properly occur for the DZ instead of the ZA
  • Thanks to Tom for pointing the ZA were stealing the DZ's achievements, and zeus for fixing!

• Count the number of planets in X map subtrees correctly.
  • Always divide extra planets among all children.
  • Ensure than main branches of symmetrical X map have the same number of planets.
  • Thanks to Tom for fixing!

• Don't error when trying to get `NumberToSeed` faction configuration when chekcing achievements.
  • In particular, the Hunter Fleet doesn't have it, so checking for completion of "Kites Get Struck By Lightning, You Know" was causing an exception after you won the game.
  • Thanks to Tom for fixing!

• Convert more places that use UniqueTypeDataDifferentiatorForDuplicates to use int.
  • The field was converted to an int in r14429, but there were a bunch more places that used or accpeted this value that needed to be changed. I think I've got all the remaining ones.
  • Thanks to Tom for fixing!

• Add FleetCategory.GetCanHaveDuplicateLines to track whether fleets can have duplicate lines.
  • This is a followup to r14429, which changed how Spire fleets work. They use `FleetCategory.PlayerCustomCityFedMobile` fleets, and each city provides ships with a `UniqueTypeDataDifferentiatorForDuplicates` corresponding to the city's `FleetID`. Thus, that category of fleet needs to support the `*_WithUniqueIDForDuplicates` method variants. The new method is used to control this.
  • Thanks to Tom for adding.

• Move the logic for per-city frigates to be attached to the hub.
  • I added a new tag `SpireFeedsFleet` to all Spire building that provide ship capacity, and changed the Spire Hub and Capitol to provide the per-city frigate.
  • The old logic was incorrect, since the per-city frigate wasn't additive withfrigates from other structures.
  • Thanks to Tom for fixing.

• Calculate the stats of PlayerCustomCity fleets as player fleets.
  • In particular, this calculates MaxMarkLevelOfAnyInFleet, which is used to determine Spire Fleet strength.
  • Thanks to Tom for fixing.

• Remove duplicate centerpiece upgrading for Spire Fleets.
  • There were two copies of this code, one from before splitting fleets, which recorded a journal entry, and one after, which didn't. I've removed the former, along with the journal entries, as they were now incorrect, and somewhat redundant with the city mark level journal entries which are granted at the same time.
  • Thanks to Tom for fixing.

• Mark Spire Hubs as granting science upgrades to the fleet.
  • Though they can't be upgraded with science by the player, the city upgrade uses science to indicate the mark level of the fleet.
  • Thanks to Tom for fixing.

• For PlayerCustomCityFedMobile fleets, use the mark from the bolstering city.
  • Thanks to Tom for fixing.

• Don't grant science to non-centerpice ships in Spire Fleets.
  • Now that ships in spire fleets get mark from the cities they are fed from, we should not grant them from the fleet mark as well.
  • Thanks to Tom for fixing.

• Use "science" to upgrade Spire Fleet Flagships.
  • The code previously called SetCurrentMarkLevel, but that only changes a transient value that gets overwritten on the next update.
  • Thanks to Tom for fixing.

• Renable building Spire City structures.
  • Thanks to Tom for fixing!

• Make spire cities automaticly bolster nearest fleet fleet.
  • This is at least partly a stop-gap until it is possible to change which fleet is being bolstered by a city.
  • Thanks to Tom for fixing!

• Upgraded the engine to Unity 2020.3.21f1, from the prior version 2020.3.20f1 we had been using.
  • This is a pretty fast turnaround on engine version upgrades for us, but they fixed a memory leak that affected OSX players.
  • It seems likely that we've fixed our own memory leaks that affected all platforms, but at any rate let's not have that confused with an engine-level one.

• Fix isGalaxyFullyConnected to not sometimes report a false negative.
  • The previous version tried to save allocations by having the caller pass in a reference to a list of planets to use for tracking a connected component. However, since it also need to pass the list to a delgate, it needed to create a local copy of the parameter. It looks like there was some confusion between the two variables that lead to the wrong list being used.
  • I changed it to take the list of connected planets as a regular parameter, rather than a `ref` parameter[1]. If the caller (i.e. `makeSureGalaxyIsFullyConnected`) want access to a connected component, or to reuse the list, it can allocate and pass in a list. However, if the caller (i.e. `removeSomeLinksBetweenPlanets`) doesn't care about either of those things, it can just pass in `null`.
  • I also added some documentation for the method, which explains the usage of the `connectedPlanets`, along with caveat about the behavior if it isn't already connected.
  • Thanks to Tom for fixing!

• Fixed several cases where we could get spam of "Look rotation viewing vector is zero" invisibly in the player log, which led to memory rising as well as choppiness.

• Fix a couple of errors in ArcenUniversal.List where passed in Lists were not initialized.
  • Thanks to Tom for fixing!

• Fix a divide by zero in the Anti-AI Zombie patrol code.
  • Thanks to Tom for fixing!

The Code Panopticon

• Added a new form of instrumentation where we can track all collections and see if things are leaking via successive dumps, which includes finding out WHERE something is dumping from.
  • This does require folks to specify what the use of their stuff is, and it also makes it so that (for the time beinng, when the logging is on), no collections are ever garbage-collected.
  • Honestly, I don't trust the garbage collector to do a good job in the first place. "Temp collections" should be using ThreadStatic, not just created and then dropped.
  • This does require code changes in a variety of places, including in code mods, but this is kind of the ultimate in memory leak detection and repair. It also forces programmers to think a bit about what sort of data they're just tossing away. At best, it's something that adds slight hiccups from the garbage collector, and at worst it's a memory leak.
    • Please bear in mind that I'm concerned not just with the memory leak we have right now (we had one in July, which was different and I fixed it, for example), but any future ones, including client-only ones in MP, and mod-specific ones. Having a true solution that lets us detect them -- in fact, lets a modder detect them -- is something that's important to longevity.

• We had several methods for profiling the performance of the game, added over the years because the unity profiler is not great for multithreaded games (it is getting better, but still kinda stinks).
  • Some of these were potentially a minor performance drain, and we don't even use them anymore, so they've been removed. The StopwatchSegment stuff was the oldest one, and is now gone.

• The collections in general, and some other classes that depend on them, now require you to pass in a string LocationNameForTracing,
  • One of MANY places where this affects things is also the ArcenDoubleCharacterBuffer and similar, which I've known for a while we have been churning out like crazy.
  • These are being abused all over the place, sometimes just through casual usage ("this is infrequent and won't matter much"), and other times by accident (a sub-object that gets lost when an outer collections of collections is cleared).
    • In that second case, overall my goal is to have data structures for those types of collections of collections so that the internal collections get pooled within the outer one, thus keeping code simple.
  • The downside for programmers is that they need to give a name to a lot of things that they did not previously have to. The upside is that this helps us realize "hey, if I'm naming this, I'd probably better not throw it away, because that might be a memory leak." And the super duper upside is that when we do wind up with a memory leak, we can see it, by name, and then just go directly where it is and fix it.
    • I originally tried to use stack traces to accomplish this same thing, but that make loading the game take 90 seconds instead of 8 seconds on my machine (ow), and loading a savegame increased from 2 seconds to 260 seconds. So that obviously wasn't a workable approach.

• A new DictionaryOfDictionaries has been added, which works a lot like the DictionaryOfLists in that it has internal pooling to prevent leaks that otherwise were rampant when programmers used this sort of structural design.
  • The toFromPlanetCache now uses this, which gets rid of one notable potential memory leak.

• Added a new ProtectedListArray, which is less critical but does prevent some accidents with a ProtectedList[]. It also makes that outer code simpler, like these other compound collections, which is good.
  • Same deal with a new ListArray.

• For the sake of brevity and ease of use, ArcenRandomDrawBag is now just DrawBag.

• Added a new DictionaryOfDrawBags, again for the same sort of "don't leak a collection" purposes.

• Added a new DoFor() method on EnumIndexedArray.

• Added a new EnumIndexedArrayOfLists, again for anti-leak purposes.

• Added a new EnumIndexedArrayOfArcenOverLinkedLists, again anti-leak compound data structure.

• Now we also have EnumIndexedArrayOfProtectedKeyDictionaries.

• Interesting, a new ConcurrentDictionaryOfLists. Had not expected to need that, but it's useful.

• Also added a new DictionaryOfDoubleBufferedLists, which is mostly used for some logic on the macrophage, but it's vastly more flexible and effient than what I had set up previously to run them.

• SortedDictionaryOfSortedDictionaries has been added, and this may be replacing one of the areas where a larger leak was possible.
  • Interestingly, in adding this and then making that in use throughout ShipListerUtils, I found something that looked like it would lead to incorrect counts of ships in some circumstances. We've had sporadic reports of that, so this may be solved now.

• Added DictionaryOfArrays, which I don't actually need right now, but I figured why not. Because...
  • Added SortedDictionaryOfArrays, which is slightly more complicated but based on the above one, and this one I actually do need.

• Fixed various random issues that were discovered because of the above changes requiring lots of code review:
  • There was a code path that cause ArcenRandomDrawBag to not actually initialize its tracker properly.
  • Fixed up the ArcenLessLinkedList and ArcenOverLinkedList to use structs for their enumerators, like the internal official classes do. Way more efficient.
  • GetRange() and FindAll() on the standard List class now work in a way that does not generate any garbage, and is faster to run in general. This requires you to pass in a list when using them.
  • Internally improved the performance of the ConcurrentBag that Microsoft provided, making it so that it doesn't need to generate lists and garbage when it is stealing items from one thread's pool to another.
    • Also improved its performance in the same way for when it needs to do an enumeration over a list that happens to be empty right now.
  • Improved the performance of my own logging from background threads to the main thread. It was generating some needless garbage and cost.
  • Huh, interesting, it was possible to really improve the ConcurrentBag and ConcurrentQueue enumerations and CopyTo and so on even more.
  • Made the xml import a LOT more efficient because of how we handle creation of working lists and sub-lists and things. This one is definitely on me in terms of why it was not as efficient as ideal.
    • A new FillList method that is very efficient is now on ConcurrentDrawBag and saves a lot of locks and list allocations and so on that we had to do without it.
  • Now using DictionaryOfLists in xml import. And for input actions by category. And music tracks. And RowsByTag and RowsByPlanetSeedingType.
  • Now initializing a far smaller list of sub-collections as we import various xml elements and attributes. Yay efficiency!
  • When dropdowns change what they are showing, there is more efficiency there, too.
  • A few places where we use RefPair in Core have been changed over to KeyValuePair, which is more efficient. We had very few instances of using RefPair in general, outside of the Draw Bags where it is actually needed in the form that it is.
  • Various more cases where we now use the compound classes.
  • Fixed up the templars so that they properly clear their data now and don't leak any data. I had marked that with a data_todo but hadn't gone back to them since then.
  • Fixed a possible memory (substantial) leak in AOE attacks, although it's not for sure that was happening. Might be a false alarm, but either way it is good now.
  • Noted that ExoOptions, which were set up as a struct, had reference types (lists) as part of them, which kind of defeated the point of these being structs. It does cut down on the amount of gc being hit, but still has a notable hit. These are now pooled and tracked; these always get sent to SendExoGalacticAttack, so that properly returns them to the pool when it is done with them (or earlier, when it early-outs on an MP-client).
  • Certain methods, like GetPlanetsForExtragalacticBudget(), no longer return a list of things. Instead, they take in a parameter that is a list-to-fill, and they do that. This pattern is required in places like this (it's always a judgement call) so that multiple callers can work with the same method and not have scrambled results. ThreadStatic is another way to go, but then you run into problems with serial calls to the method on the same thread. So since this a return list that we don't know how long will be kept or used, it's better to let the list be something that we don't originate in a method like that.
  • There was some funky logic that was hard to follow with a few collections being passed around from waves to notifiers and being used for potentially too long. It probably was not a problem, but now it definitely is not.
  • Got the sapper unit data so that it properly cleans itself up. Here again I had left a note to myself, but not gone back to it yet.
  • A bunch of common game commands, like wormhole movement and regular ship movement, are now at worst more efficient, and at best also no longer leak memory.
  • GetOptions() no longer returns a list of options

• Most references to Linq have been removed from ArcenUniversal.
  • Linq is inherently messy and create a lot of garbage and is slow and is not something to be used very frequently. I discovered this back in 2008, with the original AI War, and have maintained that position ever since.
  • That said, for our ObjectDumper, DiffLib, and a few other key things that are infrequently-used, we can get away with some limited use of it.

• Added a new BetweenXmlReloadPool, which provides a new way for us to more efficiently hang on to certain data between reloads of the xml (reloads of the xml only happen when you change around your enabled expansions or mods, but it's still good to do).
  • Previously I was relying on certain types of data being garbage collected, but now I'm not going to trust that.
  • For now this is just used on MapSettingOption and MapSettingOptionChoice, but later it can be extended to table rows and similar.
  • There is a new enum that is used for the ConcurrentPool and the TimeBasedPool which specifies that things either will reload when the game is cleared (like going to the main menu, mapgen, etc), or when xml is reloaded, or not at all.

• ArbitaryArcenOption is now pooled between xml loads. OptionWrapper also. Okay... the big one is ArcenDynamicTableRow, and it also now complies with this. This is going to be a pain in my rear, but there's just no other way to avoid memory leaks on xml reloads. How severe those would be is hard to be certain, but it's not great, so here we are.

• CreateNewForPool() has been removed from the various pools. They now instead take a GetNewEntryFromAny<TValue> delegate, which is vastly more flexible and makes certain kinds of inheritance possible for the first time. This was the big blocker behind dynamic table rows.
  • Okay... to handle dynamic table rows, we really need to have something more automated, even it's more slow. This is just during xml reload, not during gameplay, so even if it's 10x slower it's still going to be plenty fast. During gameplay we need all the speed we can get, so we don't automate things.
    • This has led to a reflection-based approach that is able to copy all of the values of an uninitialized dynamic table row of a certain type (like GameEntityTypeData) and set them to their defaults. The idea is to kind of brute force this.
  • There's a new ArcenTypeAnalyzer class that we can now use for looking at a type and making a blank version and then applying the defaults from it to other copies of it.

• EntityMetalFlowEntry has been converted from a class to a struct, using the better patterns that we have established for this sort of thing.
  • There was some confusion on my part in discord a few weeks ago, and I had to really re-learn how to properly use structs in certain classes of cases, but actually the microsoft collection enumeraters were a big help in showing big practices (and their KeyValuePair). This now follows those best practices.
  • IncreasingPrep has also gotten the same treatment.

• Everything that inherits from ArcenDynamicTableRow is now forced to use a certain flow for creation that uses a private constructor and either uses a pool (normal case) or explicitly and clearly opts not to use a pool.
  • Previously it was really easy to accidentally wind up with multiple ways to create from these, some pool-based and others not.

• A couple of memory leaks that were in various dropdowns, most notably in the Galaxy Options and Personal Settings windows, are now fixed.
  • It's unclear if these were true memory leaks in the sense that memory would never be given back, or if it was just a performance drain and hit to the GC. Either way it's fixed, and this is not likely to be the smoking gun of the main leak.
  • Actually, these particular windows were really just full to the brim of stuff like that, but at this point they have been restructured so that they now give compiler errors if someone tries to do the sort of things that we were previoulsy having problems with. A lot of this was I think my code, but from a long time ago.

• PlanetWaveComposition has been converted from a struct to a class, and is now pooled. This is another one where it was leaking pretty fiercely, but exactly how much of a problem it used to be is hard to tell.
  • If I had to guess as to a main source of a leak from what I've seen so far, this is a high candidate, at least. Between this and some of the stuff in the beam weapon attacks, those seem like the potential worst offenders I've seen thus far.
  • Added a new ProtectedKeyList to help with some of the collections affected here.

• Added a new DictionaryOfSortedDictionaries, which is used mainly for distant ships to be brought to the local planet.

• All of the savegame and formatting code has been split out into their own files. The savegame stuff was not being pooled previously, but now is. Note that I'm referring here to representations of savegames on disk, not the actual contents of them.

• In general, the number of ArcenDoubleCharacterBuffers has been dramatically reduced by making a lot of them static rather than instance variables. Other similar changes have also been made that help keep RAM usage lower and CPU usage more efficient.

• Fixed a couple of places in the squad visualizer that were initializing absolutely insanely huge arrays. This was not a "memory leak" per se, but it was allocating two orders of magnitude more RAM than it should have been, which would wind up looking like a memory leak.
  • If this is what was causing the memory leak, that will be quite interesting. This is something that I accidentally introduced in the last few weeks, either way. I'll need to improve my tooling in general, because this particular thing would not have been detected by my memory leak detection code.

• Added a new GetTotalCurrentCapacityForItemsForTracing() on TraceableCollectionBase, because the more I thought about this since last night, it's entirely possible that this entire "memory leak" (this time) was merely a matter of over-allocating to a frankly insane degree on each ship in a squad.
  • This only applies to ships that are being visualized, in the example yesterday. But let's look at the numbers. The ShipsLiving on each individual squad object visible in the game was allocating a list of 250,000 in size for ShipsLiving, and 10,000 in size for ShipsDying. These are good numbers for TOTAL ship counts, not per-squad.
    • I should mention that we still treat ships as being in squads, but just a squad of one, at the visual layer. This is minorly helpful, but mostly a remnant of past designs.
    • So, we have only as many squads as we need to draw on screen at one time, when it comes to these particular objects. This is just a visualization layer thing, not part of the simulation (where the core squad objects exist).
    • If we have a battle where there are 8,000 ships on screen at once, then that would mean 8,000 of these objects. Normally not a problem at all. Let's do the math here.
    • We'll simplify a bit, but internal to a list is an array. We're on 64bit systems, so each reference is 8 bytes. Even the unfilled references use 8 bytes. An actual object would be way more than that, but that's fine. There's a few other heads and things, but those always exist, so let's ignore those.
      • That gives us (8000 * 8 * 250000) + (8000 * 8 * 10000). In other words, 16,000,000,000 + 640,000,000, or 16,640,000,000 bytes. So that's 16,250,000 kilobytes, which means 15,869 megabytes. That's... 15.49 gigabytes. Yeah, that's the memory leak all right. Not a leak at all, just an insane over-allocation.
    • With that in mind, I'm going to be monitoring the over-allocations as well as for actual leaks.

• Some of the calculations for fireteams and budgets now do their work a lot more efficiently in terms of reusing existing collections instead of creating a bunch of new ones each cycle and then having to garbage collect them.

• Added a new ThrowawayListCanMemLeak class, which is like List<> but not tracked in the same way (it's still tracked numerically, but not by reference), and able to be garbage collected. It includes the following comment in its header:
  • READ THIS BEFORE USE! This class is very very close to being like the general List<> class from Microsoft. We have our own implementation of List<>, though, and that is both more efficient and also NEVER GARBAGE COLLECTED. That's appropriate for almost all purposes we could have, because we really want to resuse collections.
  • Creating collections just to toss them away is a really bad idea... 99% of the time, anyway. The other 1% is why this ThrowawayListCanMemLeak<> exists. This is exactly the same List<> class, minus the tracking and a few other minor features, and plus the ability to be garbage collected.
  • Please use this sparingly, and probably mostly in mapgen. There's certain kinds of recursive code that you just can't do without local variables of some sort, and this allows for that sort of thing. If you're using it in MapGen, great. If you're using it ANYWHERE else, I hope you have thought things through and have a good reason why. If so, great!
  • If you're using it frequently during normal runtime operations, on the other hand, then you're probably not doing performance any favors. Consider a refactor in that case.

• The wasteful version of FindAll has been added to ThrowawayListCanMemLeak, since a bit of map code wanted that. Why not.

• ThrowawayListCanMemLeak is now used in most of the mapgen code.
  • There's a few bits that still use List, and there are yet other parts that now use IList -- not Microsoft's version, but an extended version of my own that allows for more functionality to be shared between the two types of lists.

• Put in a possible fix for isGalaxyFullyConnected not working the first time it was called, but have not had a chance to test it yet.
  • Thanks to Tom for alerting us to the issue.

• When exos are spawned against you during the fallen spire campaign, it now actually does the splitting of that into two exos, one on the relic, and the other on something else of value to you.
  • Previously it looked like that logic was partially there, but never actually fully completed.

• Improved the mapgen logic in a few ways where you don't have to pass in a list that won't be used, but instead can pass in null and it will still work fine.
  • Similar changes were also made to wave spawning logic. It gives more flexibility on getting things back, without having to declare unused lists.

• Improved some various faction utility methods so that they internally use threadstatic lists rather than requiring you to send in a managed list that will never be used for anything else.

• PlannedWaveOptions is now a pooled class, rather than being a struct. This is required because it has a dictionary on it.

• FireteamRegiment is also now pooled, and should always be stored in a ProtectedValDictionary so that it gets automatically put back into the pool when the dictionary is cleared.

• Added a new DictionaryOfDictionaryOfLists, which is our most-compound data structure yet in terms of the general ones. This one is mainly used by the outguard.
  • The compound data structures really are non-optional, if you're a modder and you're wondering. You're going to leak memory if you don't use them. If you need one that's not provided, then I can add it.
  • In general, though, compound data structures are sometimes not really as needed as they might seem. This is a good case of one where, yes, it really is actually needed.
  • That said, using a dictionary that has a dictionary of lists, in raw form, is going to leak both dictionaries and lists. So the compound structure that pools them is really non-optional as the way to handle it.
    • In the past, using Microsoft's version of these data structures, they would get picked up by the garbage collector (so we're talking a performance hit, not a memory leak). But my version of them are tracked permanently, so they will literally be a leak. That's part of the panopticon nature of all this. The code really needs to be able to observe itself in order for us to know when we're leaking or over-allocating. And we should be hitting the GC as little as possible (this is true for absolutely any game), anyway, for the smoothest end-user experience.

• Added a new AddRange_IgnoreNulls on List, so that we can pass in a collection/list that is potentially null and just have it ignore the null if it is. This simplifies some other code.

• The ArcenXmlImporter is now pooled. I had originally tried using ThreadStatic and static to handle this, but that turned out to be a disaster.

• Added a new DictionaryOfArcenOverLinkedLists, as this is really required for any sort of good performance on EntityCollection's EntitiesByType_OrNull.
  • The old way of doing this was likely a pretty solid performance drain, yikes. I'm still not in love with the way this is done as a general concept, but... it's fine enough for now, I suppose.

• We now use a LOT more of the [ThreadStatic] attribute throughout our codebase. But the downside with this is that it can be applied to non-static variables, in which case it... does nothing, and the variable works entirely incorrectly. I found a few cases of this (maybe three?) over the last week of reworking the game's code, and so had resolved to do a full review of all the code and see what I could find now, after the revamp.
  • It turns out that we have exactly 254 uses of [ThreadStatic] between our five main dlls, which is a curiously-significant number (being almost a power two, must have mystical power, right?). I had to review those all by hand, because the attribute goes on the line above the actual text, so I can't see if the line below it is declared properly directly in my find results screen.
  • Errors found out of those 254? ...zero. Go team, I guess! Good to have checked it, even if it was all fine. I did tweak the formatting on a few of them so that they are slightly more consistently placed in code.

• ListWithLogging has been removed, as it was simply not something that was very efficient, and I can now get a similar effect in the future with the main list if I truly ever need to.

• Fixed up a number of collections that inherited from CountedPoolBase to instead inherit from TraceableCollectionBase, since that's technically more accurate.

• Fixed dictionaries throwing exceptions when empty and I tried to dump them.

• For clarity, renamed ProtectedKeyList to ProtectedKeyValuePairList.

• The internals of most of our collections have been shifted around a bit to allow for them to report on what they have in them even better.
  • Additionally, when it comes to the compound collections, they now just report once at an outer level versus repeatedly over-reporting each sub-collection as well.
  • We're able to get a better picture of the information, with less duplicative information.

• Using the core data dumps, optimized the following collection capacities which were a bit too high:
  • ShipTypesThatThisBenefits on techs.
    • Also TagsList and TagsDict. These numbers seemed reasonable, but in practice did not turn out to be so.
  • CyclicalArrayPool turned out to be using as much as 380MB of space even early in the game, which was a huge waste. This is for drawing ships and such. And this would only get worse with time.
    • This is not a new problem, from the look of things, and I'm not sure how it was not more of a problem in the past, to be honest. At any rate, I've fixed this up so that now it uses VERY little RAM, mostly allocating in a lazy fashion as it goes, rather than preallocating way too much.
  • The dynamic table row counts also were using substantial size.
  • Map setting options and choices were also silly large.
  • Bundle cached objects, also.
  • Xml parsing working lists, too.
  • Some AIShipGroup collections.
  • The central pool list collections.
  • Between all of these various improvements, the game load time is back down into the 7 second timeframe for me (down from 10ish), despite the extra instrumentation.
    • Additionally, my starting RAM usage dropped from about 3GB to about 1.5GB.

• Improved the capacity reporting of several collections (for the core dump file) to do a better job of explaining what capacity they have actually have allocated in memory, versus what they would if they get called on later.

• Made the core dump report more brief in a few ways, more focused on the data I actually find useful (as it turns out).

• In the event of some issues with the cyclical array pools, the game now gives a warning rather than freaking out and having endless display issues. These should never happen in the first place, but due to some internal changes and testing, it started happening.

• Fixed a memory leak in our core dump code, irony of ironies.

• Fixed some arcen double character buffer gc churn from the ui, which in this new build had turned into a full-blown memory leak because of the structural changes around it.
  • This does solve a leak that was persistent and annoying in the past, and should lead to better performance than before. And in the new version, also it then doesn't memory leak on us.

• OptionWrapper, which was a pooled class, has been removed, and converted into IOption, which is way simpler.
  • One of the coolest things is that this uses explicit interface definitions (https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/interfaces#explicit-interface-member-implementations) to avoid cluttering up the classes that it is implemented on with seemingly-duplicative methods.
  • Thanks to Tom for figuring out how much better this approach would be, and suggesting it.

• ArcenSerializationLogger is now pooled, and thus in general is a lot more efficient. Before it was "just" hitting the GC, but there are several areas of code that should be picked up by the GC, but which from memory usage in general I can tell are not being picked up. Pooling it is for these ones.

• BufferTracker is now pooled and tracked. This was something else that was leaking on deserialization.

• FragmentedNetworkMessage is now pooled and tracked. This is good for large messages in multiplayer.

• ArcenDeserializationBufferModern is now pooled and tracked. This was a notable memory leak whenever a new savegame was loaded.
  • I still need to handle the serialization buffer as well as the arcen character buffer, since those seem to be leaking. And there's other stuff I need to instrument. But this is enough to get it closer to playable without too many leaks.
  • The Steam networking transport layer is now particularly more efficient because of this, too, as it turns out. GOG was already quite efficient in this area. This was because of my own methods of implementing them, not because of the inherent nature of either platform.

• More work will be needed to make this work fully correctly again, but various improvements have been made to the new "rows are pooled" logic when you make mod/dlc changes in the settings menu.
  • At the moment, you can use those settings, but you may have to restart the game after doing so. It seems like it will often let you do a reload a single time, but not twice. Or at least it's based around certain things. Disabling DLC3 works, but enabling it does not. Disabling DLC2 doesn't work, etc. I just have more work to do with them in general.

Press F7 To Core Dump

• The old "Dump External Hierarchy Data" hotkey has been removed, as it is being replaced with a newer and better thing that includes its info plus more.

• There's a new hotkey called "Do Core Dump", bound to F7 by default.
  • If you suspect you're having a memory leak, then use this and send the result to us. When pressed, it will output a bunch of data about the current state of the game into [InstallLocation]/GameData/CoreDumps/[Date][Time][Status].txt.
  • A large bulk of the data is what we were exporting before with the thing we just replaced, but over time we'll add to it. And in the meantime, this data has been made smaller as well as more readable.

Beta 3.712 Loading Hotfix

(Released October 15th, 2021)

• Add a new option for DLC3 for Enhanced Overlords. This is off by default for HA but enabled for Expert/Deathwish
  • With this enabled, as soon as you unleash the phase 2 overlord the AI gets a CPA against you.
  • Also, as long as the overlord phase 2 lives, the AI will launch exos and wormhole invasions against you.
  • Thanks to Badger for adding.

• Created the new EntityTypeDrawingBag (GEDB) and its logic:
  • This drawing bag combines drawing from:
    • Direct internal names of GameEntityTypData
    • Tags used by GameEntityTypeData
    • XML-defined AI ship groups
    • Categories of AI ship groups
    • The budget types the AI uses for its reinforcements and various sub-factions
    • XML-designed fleets
    • SpecialEntityTypes
  • They are defined as sub-nodes in XML in 3 categories:
    • entity_type_drawing_bag_used_on_death -> Spawns something when the entity this is on dies
    • entity_type_drawing_bag_used_on_periodic_spawning -> Spawns something as part of periodic spawning (i.e. Brutal Guardian Lairs)
    • entity_type_drawing_bag_used_on_spawn -> Spawns containing things similar to how guard posts work. This mechanic was created in addition to all the entity type drawing bag logic.
  • The following categories can be defined:
    • display_name -> This is used by tooltips and notifications as replacement for the built-in write-to-buffer method of the GEDB.
    • data_list -> This can be entity names, tags, the names of ship groups, fleet design templates, etc.
    • parsed_data_type -> This defines as what the data_list is parsed. Possible are:
      • Disabled (this is used to stop spawning something for entities copying or partial recording something)
      • EntityNames
      • Tags
      • AIShipGroups
      • AIShipGroupCategory
      • AIBudgetCategory
      • FleetDesignTemplates
      • SpecialEntityTypes
    • When using AIBudgetCategory above, ai_budget_subcategories may be defined. Possible are:
      • NormalAIShipGroup
      • GuardPostAIShipGroup
      • DireGuardPostAIShipGroup
      • UnarmedGuardPostAIShipGroup
      • TurretAIShipGroup
      • NonTurretDefenseAIShipGroup
      • GuardianAIShipGroup
      • WormholeSentinelAIShipGroup
      • ForcefieldGuardianAIShipGroup
      • DecloakerAIShipGroup
      • DireGuardianAIShipGroup
      • SingularFreakySurprisesAIShipGroup
      • ExoLeaderAIShipGroup
      • All
    • When using FleetDesignTemplates above, fleet_design_template_subcategories may be defined. Possible are:
      • Strike
      • Frigate
      • Turret
      • OtherDefense
      • Centerpiece
      • Civilian
      • All
    • spawn_count_type refers to how the min and max values for spawning are iterpreted. Note that for anything except for raw counts the results are rounded down. Possible are:
      • RawCount
      • AIBudget
      • Strength_BaseMark
      • Strength_CurrentMark
      • MetalCost
      • EnergyCost
    • Finally min_spawn_value_list and max_spawn_value_list define the actual values worked with.
  • By far not all values need to be defined, as they default, lists auto-adjust to inputs and things should mostly fall into place.
  • Vanilla, Expansion and Mod files have been updated to include these changes. As far as tested everything should work.
  • Thanks to NR SirLimbo for adding!

• Added the Brutal Carrier Guardian - This one carries now weapon but spawns with between 22.5 and 37.5 strength worth of Mk7 ships inside. The type of ship depends on what type of AI is spawning the carrier, but they use the same ship group category that guard posts are reinforced with.
  • Thanks to NR SirLimbo for adding!

• Add 'Adaptive Music' to the game. This is an Audio setting, defaulting to "Off"
  • When you are in a battle of reasonable size then the game will prefer to play Exciting music when it chooses the next track to play. This currently means 5 + AIP/20 strength on both sides. So at 100 AIP a battle with 10 strength on both sides will trigger combat. This is set in AudioAndSimilarHandler.
  • The goal is that you don't get really chill tracks playing during exciting battles, or up-tempo tracks during a refleet.
  • To set a track to "Exciting" add this field to the Music Definitions xml:

tags="Exciting"

• • Thanks to Badger for adding this feature, and to Puffin for identifying all the exciting tracks.

• When in an energy brownout, your factories will work at half-speed. Untested.
  • Thanks to Badger for adding!

Bugfixes

• Fix a bug with "LoadAllFoundXmlFilesIntoMemory debugStage 6000: System.Exception: Passed in Dictionary<> capacity less than or equal to zero. in the log" and similar preventing the game from loading.
  • Thanks to Tom for reporting.

• Templar defensive code.

Beta 3.711 More Fixes

(Released October 15th, 2021)

• Fixed some further bugs where fireteams could still get "preferred speeds" less than zero.
  • Thanks to BoBy for reporting.

• Could not duplicate the "GetStringValueForCustomFieldOrDefaultValue: The faction AI has 11 custom fields on it, but none of them were named 'StartingAILayout'" error, but have at least silenced it. In places where that error would have happened, it just treats it as "small clusters".
  • Thanks to BoBy for reporting.

• CreateExternalBaseInfo<T>() on Squads is now more efficient, on par with GetExternalBaseInfoAs<T>().
  • Same for the DeepInfo equivalents.
  • We do so many sanity-check versions of Create rather than Get that they really should be equal in performance. Basically, they now are, and they are also identical in performance save for one fact: if the data is missing when you call Create, it will add it. If it's missing when you call Get, it will error.
  • Thanks to Badger for inspiring this change.

• Updated the scourge to be less likely to throw errors, and to have better error-handling in SendUnitToUpgradeStructureIfPossible.
  • Thanks to BoBy for reporting.

• Updated our custom implementation of Dictionary to work a bit better, so that now when doing a foreach over them, it will have a compile-time error on invalid casts, rather than a runtime one. This is how it has always been before, but it was a bit trickier than it sounds.
  • It turns out we were only down to one last one of these anyhow, in the fleet management window. At any rate, that one is now fixed!
  • Thanks to Daniexpert for reporting.

• Improved the DictionaryOfLists substantially, so that it has more functions and also has an enumerator (so you can run foreach against it).
  • This is something that keeps the internal lists from having to be discarded by pooling them.

• Civilian Industries has been recompiled to probably work again with the new collections. It uses the DictionaryOfLists, making it the first part of the game to do that.
  • In general, a bunch of things that were previously creating collections and throwing them away very inefficiently now use threadstatic collections instead.

Beta 3.710 Hotfixes Serialization and Performance

(Released October 15th, 2021)

• Fixed a number of invalid cast exceptions and similar relating to the changes to SpeedGroup in the prior game. Essentially this made it so that you could not save, or really play all that long. Facepalm.
  • Thanks to Daniexpert for reporting.

• Added a new and more efficient method for TryRemove on ConcurrentDictionary that allows for multiple tries at once, and doesn't force you to take an out value.
  • If you only want it to try once, just pass in any number less than 2.

• The death registry now supports resurrections, heh. Units that are put into a transport are considered "dead," but when they are brought back out they are now considered brought back to life as their same self.
  • This fixes an error in the death registry code that would happen around transports, which I had anticipated yesterday and then got sidetracked into all that drone business and didn't actually test it.
  • Thanks to Badger and Daniexpert for reporting.

• In preparation for getting MP working again on the beta, the game now fast-blasts death registry data to the clients.
  • This includes removals and additions to the death registry, and slots them into the fast-blast order in the appropriate spot where we won't get death registry warnings.
  • For units being killed, this also proactively kills them on the client in the same way they died on the host, which is quite handy.

• The sidebars, and many other parts of the game that involve adding an unknown number of items, headers, and sub-items, all now populate themselves in a time-sliced manner. This means that when you first open the build tab, for instance, you may see a slight flicker as it fills out all its entries.
  • Previously, I used a bit of an opposite approach, and it was not a good idea. I was trying to pre-create a lot of extra items to prevent lag spikes, but this meant that there was one big lag spike (often lasting a few seconds) when you opened up those sidebars for the first time.
  • This makes the interface more responsive in the early game, and reduces some bits of CPU and RAM usage (both in relatively minor ways) without leading to later lag spikes.
  • Thanks to StarKelp for reminding us of how annoyingly slow these were to open.

• Fixed an issue with some of the new custom flagship types that made them unable to hack. This had broken the necromancer faction's ability to have their flagships hack, in particular.
  • Thanks to Daniexpert for reporting.

• Removed the constructor on my custom version of List<> which took an IEnumerable collection. This was inefficient to begin with in Microsoft's version, and generally used to work around things like the limitations of "can't be iterating over a dictionary and add or remove to/from it" (we now can do that with my version of dictionary).
  • In general, I am already planning on cracking down a lot on how collections are initialized and disposed for memory leak purposes in the very near future, so this is in line with my plans there, anyway.
  • In the short term, more to the point, there was a bug in my version of the list that took a collection as an initializer, and this was likely causing some issues in xml parsing being fully applied (not confirmed, but it seems likely), and also definitely was causing the necromancer hacks to come up blank, and may have also messed with some of the scourge logic and two map types.
  • Thanks to Daniexpert for reporting.

• Fixed some exceptions that could happen in DoOnAnyDeathInCombatLogic_AfterFullDeathOrPartOfStackDeath if the fleet had already been nulled out.
  • Thanks to Daniexpert for reporting.

Beta 3.709 Upgrades And Serialization Fix

(Released October 14th, 2021)

• For performance and accuracy reasons, the SquadWrapper has been retired, and everything that was using it is now using the LazyLoadSquadWrapper.
  • The lazy-load version does not require a secondary fill pass on deserialization. We've been having guards breaking ranks and becoming threat after savegame load recently, and this was most likely to do with them being released because of the post-process of the squad wrapper not being called properly. These now use lazy-load squad wrappers, which should protect against that.
  • To make really sure that I understand what is happening with our data flows with this, there's now a SquadWrapperStats class that we can set to be populated, and which then gets shown if you right-click the clock and look at the debug stats there.
  • There's still more I need to update with this, but this is a start.

• Adjusted the wormhole invasion LRP thread to run every 15 seconds instead of every 30.
  • Anything set over about 25 seconds will wind up showing inactive thread warnings to players on busy games. Anything 30 and up will show inactive warnings on EVERY game, periodically.
  • The inactive warning starts showing after 30 seconds of not running, and LRP threads may naturally start up to 4-5 seconds after their "don't run for this long" interval is over, just depending on how busy everything else is.
  • Most factions have an interval of 1-5 seconds, but for those that really don't need to happen frequently, 15 seconds is fine. If something doesn't need to happen frequently AND is super cheap to run, you could set it as high as 20, but there's not a lot of benefit to doing that since by definition it is quick to run.

• Speed groups now also use lazy load squad wrappers, and thus no longer require a post-processing pass after deserialization.

• Fixed a performance issue in gameplay for squad wrappers where things that were set to ID 0 (meaning empty) were able to try repeatedly to look in the central dictionary, uselessly.

• There is now a central "death registry" for squads that have died.
  • This was something I had been wanting to do for a while for multiplayer, to solve the ghosts issue once and for all in a different way, but it also has benefits in code protection on loading savegames in single-player or on the host.
  • Normally if a lazy load squad wrapper tries to look up an entity by ID in the central registry and cannot find it, then it marks itself as null and stops trying. This is... problematic in edge cases, and may contribute to strange behaviors like the golem guards turning aggressive.
    • Now if we come back with null from the central registry, we check the death registry. If the unit is properly marked as dead, then it's time to just do the above logic (yay!). However, if the unit is not in the death registry, we're going to wait 10 realtime seconds and keep checking the central registry as-needed. We may be waiting for data from the host, or we may be single-player and waiting for data to unpack fully from our savegame. Either way, if the data is simply incoming, we will wind up getting it. If it goes the full 10 seconds with no resolution, we now throw an error, because something is clearly amiss and needs to be looked into. This error won't mess up your save, but it will help us find out what is going on.

• Fixed a serialization bug from the last few versions that was related to some new death-effect data.
  • Additionally, while I was at it, made those serialize by index rather than name, since that will be more efficient given the nature of their data.
  • Note that for hack types we're still serializing by name, since that's more efficient there -- there are many hacks, and few used at once, which is not a good candidate for by-index. But death effects are few in type but used many at a time, which is a great candidate.
  • Thanks to Eluthena, HarryT, Badger, and Daniexpert for reporting.

• Both health_change_per_damage_dealt and health_change_by_max_health_divided_by_this_per_attack are now no longer on game entities but on individual weapon systems, which means it is now possible to design ships that have specific weapons that heal or damage them, instead of having to balance all weapons around that mechanic.
  • Updated code and tooltips to reflect this change. The health change based on max health so far didn't even appear in tooltips, it is now present named "Self-Assembly" (for healing) and "Disassembly" (self-damaging). If it's damaging itself so much that it'll insta-kill itself after just one attack it's named "Self-Destruct" instead.
  • Updated all existing normal and mod content to reflect these changes.
    • Tethuida Drones had a -100 health per damage dealt, which was most likely to mimic a self-destruct. But given something with high enough damage reduction they could actually fail, so they've been given health_change_by_max_health_divided_by_this_per_attack="-1" to make certain they always self-destruct on attacks.
    • Starbursts have an invisible melee weapon that causes them to move in close - however this also would take their health if it got to actually attack. Now it will no longer potentially do that.
    • Dark Splinters from the ESV mod no longer have the ability to self-destruct on the first revenge shot they fire. Only them crashing into enemies and doing damage does so.
    • There is a chance that I missed something due to copy and partial record changes. I was trying to be as thourough as possible - but chances are that I missed something. Please notify me if that happened.
      • At least all DLCs load without issue.
• More Brutal Guardian Work:
  • Reduced the size of the Brutal Amalgam Alpha's head and tail a bit, so they aren't as large as the actual thing. Also fixed the Tail's description.
  • Added the Brutal Amalgam Beta Guardian. A combination of Dire Vampire, Forcefield, Tesla and Gravity guardians. Even while cloaked it protects allies and disrupts enemy formations with its forcefield, siphoning health with each vampiric attack while also slowing enemy engines and weapons in range. On death it splits into a Vampire-Forcefield head and Tesla-Gravity tail. When those die they will turn into their respective Dire Guardians.
  • Thanks to SirLimbo for adding!

• Updated the tooltip for self-heal and self-damage to be a bit cleaner.
  • Thanks to CRCGamer for writing.

• Experimented with moving around Drones and transported ships, but that did NOT work out. The performance hit is pretty massive.
  • Also experimented around with only as-need initializing a bunch of collections on squad, but that adds a ton of confusing code, and I already have put in lazy-initialization INSIDE collections, so that was utterly pointless.
  • Overall, wasted several hours on this, and reverted all of it. Except the two bits below.

• Added a new DictionaryOfLists, which is a handy data structure we may or may not use.

• Added a few new List methods, like RemoveAndReturn and RemoveAtAndReturnLast, which make common operations a bit more efficient and also less code.

Upgrade To Unity 2020.3.20f1

• The game engine has been upgraded from Unity 2020.3.15f1 to Unity 2020.3.20f1. Improvements of note from Unity:
  • Graphics: Improvements to scene culling CPU performance - Shadow casters are rejected earlier if their shadow casting is disabled which should improve performance in scenes with scenes with lots of renderable objects that are set to not cast shadows. (This applies to AI War 2)
  • Graphics: D3D12 player will not crash/freeze when switching between full screen modes
  • UI: Backout a optimization of caching the change count where the layout is changed during a layout call. (Apparently this could cause UI scales to not be quite correct)
  • Graphics: Improve Line/TrailRenderer quality when points are close together.
  • Graphics: Reduced the Input delay in Vulkan standalone when VSync is enabled.
  • Graphics: Fixed a crash that was caused by a memory error when using Vulkan.
  • Windows: Fixed an issue for mouse deltas always being 0 when running Windows in a VM.
  • Graphics: Fixed FrameTimingManager produced incorrect frame time on Vulkan and DirectX12
  • Graphics: Reduced main thread hitching caused by Shader loading
  • Asset Bundles: Fixed Asset Bundle determinism (this makes it so that when we make updates, they won't require such large downloads on GOG and Steam)
  • Graphics: Fixed performance spikes in OSX while using an external monitor. Metal editor on OSX now uses CVDisplayLink to time frame presents.
  • Scripting: Fixed a performance regression when activating/deactivating panels in a canvas. (Yikes, probably)
  • Graphics: Fixed a crash when uploading large number of texture assets in DX12.
  • Graphics: Fixed a race condition deadlock when loading textures synchronously. (This was likely hitting us on game start; this was finally fixed on October 8th)

Beta 3.708 Bugfixes And Death Spawn

(Released October 13th, 2021)

• Update the some Tips about the wormhole invasion to explain the new mechanics.
  • Thanks to Badger for adding.

• Added the ability to spawn multiple entities on death.
  • The XML fields "spawns_on_death" and "count_to_spawn_on_death" are now both lists. They must be of equal size, spawns_on_death must contain the name of entities and count_to_spawn_on_death the amount of entities spawned for each type, which mus be 1 or greater.
  • Clearing these out will no longer work with the "None" keyword - instead simply empty out the lists (= spawns_on_death="" count_to_spawn_on_death="") if you want to prevent a copied entity type from spawning more stuff on death.
  • Thanks to NR SirLimbo for adding.

• The XML field spawns_on_death it seems can't be cleared with just nothing between the "". Instead, it can now be cleared by using "None".
  • This now fixes an ininitely splitting Hydral Hobbyist outguard. It already has enough heads.
• More work on Brutal Guardian Lairs:
  • Fixed Brutal Guardian Lairs spawning against minor factions - I simply forgot to disable that in XML.
  • The Brutal Darkness Guardian now has 0.7 instead of just 0.6 albedo (it's cloaked), and has individual metal and energy costs. Before all of this was just copied off the Brutal Adaptive Guardian.
  • Added the Brutal EMP Guardian. When encountering the first enemy planet with a structure of 4 tx or more it self-destructs, leaving behind a depleted version of itself that attacks with a stun blast, and an EMP bomb that will explode after 5 seconds, massively stunning next to everything on the planet.
  • Added the Brutal Amalgam Alpha Guardian, a combination of the Dire Pike, Nucleophilic, Concussion and Tritium Sniper Guardians. When destroyed splits into a head that carries the Pike and Nucleophilic weaponry, and a tail that carries the Concussion and Tritium Sniper weaponry. When those are destroyed they split into the relevant Dire Guardians.
  • Thanks to NR SirLimbo for adding.

Bugfixes

• Fixed a missing empty space after the cloaking-loss-per-shot tooltip section.
  • Thanks to NR SirLimbo for fixing.

• Fixed an issue where string-based dropdowns in the in-game galaxy options screen would not properly save their value. It worked fine in the lobby, but not in-game.
  • The new wormhole invasion settings are the first settings to ever use this feature, so this is our initial discovery of a bug dating back to at least June, if not prior.
  • Thanks to Badger for reporting.

• Fixed an issue with my list collections from two builds ago where FindAll and GetRange could throw an exception if they were to return a zero-length list.
  • This was causing an exception in the Swirl map type.

• From June 18th of this year, I made it so that map options were a lot more robust, but at the same time introduce a possibility where maps would not have an underlying related_int set properly.
  • Because of how this was done, this made it really easy for modders to shoot themselves in the foot, and I also shot myself in the foot with a ton of the existing maps as I tried to convert the data over. I essentially broke the sub-options on a lot of map types at that point.
  • To make sure that these don't just slide by unnoticed in the future, if the map generation algorithm tries to read an uninitialized integer or fixed-integer, it will now throw an exception.
  • Finishing the fix in the short term, then went through and fixed up the xml for tons of map types from the base game and dlcs, which have been quietly not having their sub-options working for all this time, it turns out.
  • Thanks to Eluthena and tom.prince for reporting, and tom.prince for noting that the xml was not set up correctly.

• Broke savegame serialization again, but made future serialization more likely to be correct -- or at least error at the correct spot.
  • Also cleaned up some serialization logging flags that were slightly inconsistent and which thus would mask errors.
  • Thanks to Eluthena, HarryT, Badger, Daniexpert, SirLimbo, and BoBy for reporting.

• FireteamRequiredTargets are now properly pooled; previously they were small objects that just leaked.

• WormholeInvasionData was previously treated as a struct rather than a class, but after a lot of discussion it looks simpler to treat things like this as poolable structs.
  • These also really need to use DeserializeUncertainNumberOfEntriesIntoExistingList in order to avoid a memory leak in multiplayer on clients, and now do.

• ProtectedDictionary has been split into ProtectedKeyDictionary and ProtectedValDictionary, since we might one to protect one or the other.
  • Also, these just now use IProtectedListable, rather than IProtectedDictionaryable.

• WormholeWaveData has also had a similar update.

• Found a general serialization bug in StoredAIPurchaseCostByBudgetForSpecificUnits, which was blanking out all of the budget types on save to disk or network.
  • The way this is stored and processed is now more efficient in MP and SP, but especially on MP clients.

• OutguardSpawnRequest is also now pooled, and also no longer leaks on MP clients thanks to DeserializeUncertainNumberOfEntriesIntoExistingList.

• I think that there are still other serialization errors for me to deal with, unfortunately. But things are better now than they were.

Beta 3.706 Expanded Wormhole Invasions And Raid Engines

(Released October 12th, 2021)

• The game is now forced to load the game entity data in a single thread, since it was consistently during this step that people were running into deadlocks on game load. I am a bit surprised I did not think to do this before, but I had shifted the actual loading of graphics to be direct rather than asynchronous. It may be possible that this lockup is unrelated to unity jobs after all, and perhaps there's just something that was prone to (relatively, in the grand scheme of things) rare deadlocks. I've seen it myself, but only one in every 50 or so times I start the game, if that. (I start the game an average of 30+ times a day, depending on what sort of coding I'm doing).
  • Anyhow, we'll see if this helps, fingers crossed.
  • Thanks to Mac, Badger, and NR SirLimbo for the most recent reports.

Badger's Wormhole Invasion Redo

• The player can now choose via a Galaxy Setting what mode of Wormhole Invasions they want. There are 3 options, the first two are simple:
  • Immediate Invasion: The current behaviour
  • Waves: The AI will spend its wormhole invasion budget on normal waves
  • Then there's the Planet Linking mode.
    • When in planet linking mode, the AI will spawn a Wormhole Projector on one of their planets. In a few seconds the projector will create a wormhole from its current planet to a planet that is (hopefully) behind your defenses. Then several Waves will be spawned.
    • Destroying the Projector will cause it to destabilize, and removes the wormhole in 60 seconds.

• Add a new cheat, cmd: instainvasion, to trigger a planet-linking wormhole invasion. It takes an optional additional integer for the strength of the invasion. So cmd: instainvasion, 60 would trigger a wormhole invasion of strength 60. Without the number it will send an invasion of strength 20

SirLimbo's Raid Engine Mechanic Extensions

• Gave the Raid Engine mechanic a few more options to work with, namely a few more XML fields:
  • "alert_wave_range_to_trigger" (default is integer 0, which is only the current planet). If set to above 0 then waves will be launched as soon as enemies are only X hops away. With this, Raid Engines could have a range.
  • "alert_wave_min_strength_to_trigger" (default is integer 0). If the base strength points (1000 bsp = 1 ingame strength) are below this, the Raid Engine won't fire.
  • "alert_wave_triggers_only_on_occupation" (default is bool false). This changes the behavior from triggering on enemy presence to only trigger on enemy occupation. If combined with alert_wave_min_strength_to_trigger then the occupied planet must have at least X amount of strength.
  • "alert_wave_spawns_unit_with_tag_instead_of_wave" (default is empty string). If set to something else than null or empty instead of launching a wave it'll spawn a random unit with said tag. This is how Raid Engines can quickly be turned into Dire Guardian Lairs.
  • Ditto form "alert_wave_spawns_unit_of_type_instead_of_wave", only with a special type (such as "AIDireGuardian"). This also supercedes the above!
  • "alert_wave_unit_name_override_for_display" overrides the name of the tag or entity type displayed (if it's set), so this is more readable for players.
  • "alert_wave_faction_for_unit_for" (accepts the following strings: "Sentinels", "Hunter", "Warden", "Praetorian", but could easily be extended if needed). If the "Raid Engine" spawns units based on tags instead of waves, the newly created units will be spawned for that faction.
    • Tooltips and notifications have been updated to work with these changes, such as being able to rigger even when the planet is not yet revealed (raid engines or Dire Guardian Lairs spawning from outside of visible range is a legitimate reason to be scared).
    • Raid engines (or any other entity with such a mechanic) no longer need their planet on alert level 4 if they trigger at a range. If they only trigger for the planet they are on nothing changes.
  • For a few examples: With this new logic you can:
    • Make a fully functional Dire Guardian Lair.
    • Give the AI Overlord Stage (stage 1 and otherwise) the ability to spawn exowar units for its Praetorian faction when a player settles within X amount of hops.
    • Make the AI spawn endless amounts of frigates so long as there is but one enemy faction with more than 0 strength in the entire galaxy.

• Even more "Raid Engine" mechanic updates NOTE: This brakes savegames - but we're on beta anyway:
  • Code-internally it has been renamed to "PeriodicSpawn".
  • All but the mechanics of the Vanilla Raid Engine are now gated behind DLC 3. New stuff is highlighted with _NEW_.
  • The maximum timer periodic spawns can work with is now no longer 128 seconds but 32767 (some ~9.1 hrs, this should be enough for basically anything).
  • Since it was a bit confusing and XML fields have been renamed, this is the end result:
    • "periodic_spawn_initial_delay" (Vanilla) = the time before the spawner becomes active for the first time.
    • "periodic_spawn_delay_between_spawns" (Vanilla) = the time between each new spawning event.
    • "periodic_spawn_never_stop_once_triggered" (DLC 3) = once the first even has fired it will never stop. Basically, the max range is set to the highest possible value, the minimum strength is set to 0. The player-only and occupation-only factors, if set, still stand though. _NEW_.
    • "periodic_spawn_creates_wave" (Vanilla) = creates a wave when triggering. Compatible with exo spawning, incompatible with entity spawning.
    • "periodic_spawn_creates_exo_strike" (DLC 3) = creates an exo strike when triggering. Compatible with wave spawning, incompatible with entity spawning. _NEW_.
    • "periodic_spawn_wave_or_exo_size_multiplier" (Vanilla) = when spawning waves or exo strikes they are this much more powerful than base waves or exo strikes.
    • "periodic_spawn_max_hops_to_trigger" (DLC 3) = if set to more than 0 also scans nearby planets within the hop limit for valid targets to trigger against.
    • "periodic_spawn_min_hostile_strength_to_trigger" (DLC 3) = when scanning for planets to trigger against at least this much hostile strength must be present. This is in base points of strength (1000 BSP = 1 ingame Strength).
    • "periodic_spawn_only_trigger_on_occupation" (DLC 3) = when scanning for planets to trigger against, they must not only have enemy strength on them but also be occupied by an enemy faction.
    • "periodic_spawn_only_trigger_against_player" (DLC 3) = when scaning for planets to trigger against only player factions are regarded. _NEW_.
    • "periodic_spawn_create_unit_of_type" (DLC 3) = creates units of a certain SpecialEntityType when spawning. Takes priority over "periodic_spawn_create_unit_with_tag", incompatible with wave and exo spawning.
    • "periodic_spawn_create_unit_with_tag" (DLC 3) = creates units with a certain tag when spawning. Overwritten by "periodic_spawn_create_unit_of_type", incompatible with wave and exo spawning.
    • "periodic_spawn_min_units_to_spawn" (DLC 3) = when creating units, spawns at least this many. _NEW_.
    • "periodic_spawn_max_units_to_spawn" (DLC 3) = when creating units, spawns at most this many. If set to less than the minimum unit count it's set to the minimum unit count. If both min and max are the same, exactly this many units will be spawned per triggering. _NEW_.
    • "periodic_spawn_unit_name_override_for_display" (DLC 3) = overwrites the SpecialEntityType and tag units are spawned by with this in tooltips and notifications. Only exists for clarity.
    • "Periodic_spawn_faction_for_unit" (DLC 3) = can spawn the unit for a specific AI sub-faction. Allowed are "Sentinels", "Hunter", "Warden", "Praetorian", "BorderAggression", "CPA" and "Wave". The latter 3 are _NEW_.
  • Internal code, including tooltips and notifications were updated to reflect these changes.
  • There is new fields on GameEntityTypeData: HasPeriodicSpawn, PeriodicallySpawnsUnit, PeriodicallySpawnsEvent. These can be used to quickly identify what the entity does.

• Added a new XML field "keep_damage_and_debuffs_on_transformation" which does just that: All damage to hull and shield as well as engine/weapon slow, paralysis and damage amplification will carry over. It defaults to false, which makes things work as they are now. When transforming the old entity type's XML flag being set to true or false counts, the not the entity type.
  • Affected transformations are: The mechanic AI Eyes or Gridlock/Deadlock use, time-based transformations such as warping-in structures, centerpieces tranformed via hacking points, necromancer transformations, Spire Telium transformations, Spire City flagship transformations, Neinzul Migrant wormhole transformations, Templar castle upgrades. Again: None of these are forced to have damage and debuffs carry over now, but the possibility to enable it via XML alone exists now.
  • Things that could be considered debuffs but aren't kept are time spent in a specific state-of-matter and death effects.
• Fixed a bit bad formating with entity tooltips relating to the description of multi-hitting beams and shots.
• Fixes and improvements to the periodic spawn mechanic:
  • Fixed the periodic_spawn_initial_delay and periodic_spawn_delay_between_spawns XML fields still being only bytes instead of shorts, limiting them to a maximum number of 128.
  • Fixed a bug where if the minimum strength to trigger on was set to 0 it'd always trigger - mostly against Outguard, unless more allied minor factions exist. In addition, when factions have the same amount of strength the raid engine or any other structure will spawn against the first faction with that amount of strength, which usually will be a player.
  • Fixed a bit of text overflow for Raid Engine notifications.
  • The mechanic will now work for all AI sub-factions as well, not just AI sentinels.
• When DLC 3 is installed the AI will now spawn 3 Brutal Guardian Lairs somewhere in the galaxy, at least 5 hops from the player homeworld(s). Currently they may not even spawn on maps such as X if the player is in the center.
  • These react to any player presence, ignoring any other faction type. When a player unit comes within 1 hop range a countdown of 5 minutes will begin. If the player retreats it will stop, and if need be continue again.
  • Once the 5-minute time has run out a Brutal Guardian will spawn for the Hunter Fleet of the AI owning the Lair. Currently 5 types exist:
    • Brutal Entrenching Guardian: If outnumbered on a planet changes to being an immobile, stronger variant with more firepower and shields. When transforming does not restore health and shields, and all debuffs will stay as well.
    • Brutal Warp Gate Guardian: Acts as a warp gate and Raid Engine in one. Only triggers against occupied planets, not simply all enemy presence. Especially dangerous if lurking as part of a Hunter fleetball right next door to a player.
    • Brutal Attrition Guardian: High attrition damage and shield-depleting ability combined into one terrifying ship.
    • Brutal Adaptive Guardian: Has adaptive multishot and high generalist DPS, which means it can equally strike against swarmers and big targets of all types alike.
    • Brutal Darkness Guardian: Heavily cloaked, very fast and able to attack while cloaked for almost 40 seconds with all guns active. Its light and heavy sabot weaponry strikes well against most things as big or bigger than turrets.
  • Once the spawning has started it will not stop until the Lair is destroyed. Killing it will inflict 15 AIP, but also grant 500 science and 15 hacking points. The Lair will also self-destruct if the AI command station of its planet dies.
• Outguard can no longer have the AI send waves against them. It makes no sense to do so anyway.

Beta 3.705 Blazing Collections And The New Spire

(Released October 6th, 2021)

Bugfixes

• Corrected some base game starting battlestation templates having incorrectly divided by five twice over turret counts. Small turret counts returned to their correct 25 count values.
  • Thanks to Mac for reporting.

• Fixed yet another exception that could happen in the faction windows when swapping between factions. It was kind of a timing/race condition thing with the UI refreshing.
  • Thanks to BoBy for reporting.

• Fixed another exception in long-range-planning for the scourge.
  • Thanks to BoBy for reporting.

• Since long before the great refactor, we've had occasional issues with fireteams getting strange preferred speeds set on them. I suspect it's some sort of integer overflow, or some other related issue. At any rate, there are now protections in place to keep from generating an absurdly high speed (anything higher than the highest speed available (unless everything is slower than 550, in which case it still accelerates to that), and also avoiding anything that is negative or zero. This should solve some strange errors that would occasionally come out of fireteams.
  • Thanks to BoBy for the most recent report.

• The serialization of all prior savegames has now been broken again, because I made a major structural change to how we save entities into the list. (Remember, we are still in that breaking-savegames timeframe, but that won't last much longer).
  • Essentially, the old approach was inefficient on the host in multiplayer, and for the player in general when saving to disk, and it occasionally was prone to cross-threading issues when it comes to units dying on a background thread while trying to be sent over the network or to disk on the foreground thread.
  • The new approach uses one extra bit per entity being serialized (though it saves 32 bits before serializing all of them), so it uses very slightly more space and network bandwidth, but it requires only one loop over the entity lists rather than two (this is a really good improvement), as well as being a lot more robust against those cross-threading issues. The old approach was kind of asking for trouble, even though it was still rare to actually HAVE trouble.
  • This is in relation to the "faction A on planet B had X entities to serialize but only Y were actually serialized" errors that you could get, among other ails.
  • Thanks to BoBy for reporting.

• Hardened EntityOrder.CalculateStrengthCountingData against cross-threading issues that could happen at stage 5000. Those should not be possible anymore.
  • Thanks to BoBy for reporting.

• Fixed a visual issue where a wrong markup tag was causing the entire planet tooltip text to be bold.

• Random Factions, a super cool feature, work once again! These have been broken since the great refactor, but it was not nearly as big a break as previously thought!

• Fixed a bug introduced in yesterday's build where if you loaded up savegames or started new games after playing for a while in a different one, you'd have perma-vision on random planets. At least it was a two-line fix, but this was an error on my part as I went through the performance improvement work.
  • Thanks to Badger for discovering.

• can_be_selected_by_any_player_even_if_not_owner=true" has been removed, as that is now universally true no matter what, anyway.
  • can_be_given_orders_by_any_player_even_if_not_owner="true" has replaced that. This is used mainly for the spire relic right now.
  • With some tweaks in this area, the "cannot give orders to spire relic" bug seems to be fixed. This was not something that was super reliable to reproduce, though, so it may still turn up again.
  • Thanks to Vinco for reporting.

Improved Handling Of Textboxes

• The way that we were listening for TextMeshPro actions when a textbox is highlighted has been stripped out. It was not working reliably.
  • Essentially, we were listening for the escape key or the enter key to either clear or submit things, and this was too flaky to keep using it. It made sending chats recently (which also means entering cheat codes) a real exercise in frustration.

• There is now a gets_sent_to_focused_textboxes="true" that can be added to any custom input (like open menu, open chat, whatever).
  • Things that have this one it will still not do their normal execution, but instead will be sent to the textbox that is currently open. The textbox gets to choose what to do with it, and then can also say "execute as normal" or "ignore it."
  • This gives us the flexibility to override things, and not have unexpected things happen, but at the same time to actually make these respond reliably.
  • This is now applied to OpenSystemMenu (Escape key), and Return (the Enter key). If we (or a modder) ever needs more than this to punch through, that's fine. I can't see what the use would be right now, though, as people are generally typing.8

• In general when we were trying to unfocus text fields, it was not always working fully. This is apparently a known unity bug with unity textmeshpro.
  • I implemented the hack workaround from here, and that fixed it: https://answers.unity.com/questions/1638173/removing-focus-from-a-tmp-inputfield.html

• Based on the above changes, plus adjustments all over the place:
  • Hitting enter in the textbox on the save menu after you've typed your text now saves the game right away, versus you having to click the button.
  • Hitting enter in the chat textbox now works reliably (omg this was rage-inducing for me before).
  • Hitting escape in the chat textbox clears whatever text is there as well as unfocusing. Hitting escape a second time closes the window.
  • In the galaxy map filter box, hitting enter leaves the text you have typed, but removes focus from the field. So you can then pan around or what have you. That keeping focus until you clicked was also hyper frustrating.
  • In the galaxy map filter box, hitting escape clears the text you have typed, and also removes focus from the field. This way you don't have to hit the little X button in there. Thanks to Badger for suggesting this long ago.
  • In the lobby, when you hit enter in the campaign name textbox, it now starts the game. Escape just clears focus.
  • In the quick start menu, when you hit enter in the campaign name textbox, it now starts the game. Escape just clears focus.
  • In the MP lobby, hitting enter in the chat box now works reliably, and hitting escape now clears you text.
  • Everything else seems to pretty much behave as you would expect.

• In the multiplayer lobby, if you try to send a chat message and nobody is there, it no longer gives you an error message about failing to send to anyone.
  • This is a longstanding gripe of mine, finally fixed. It was an overcautious bit of error-handling on my part.

Improving The Fallen Spire and the Necromancer

• Added two new special_entity_type options: MobileCustomFleetFlagship and CityCenter.
  • Removed becomes_centerpiece_of_custom_fleet_if_exists="true" and is_quasi_centerpice_for_stationary_structures_in_fleets="true"
  • Both the fallen spire and necromancer have been converted to use this.
  • A couple of spire-specific and necromancer-specific rollups have been removed as redundant, as well.

• Added a new PlayerCustomCity FleetCategory.

• A new CityBolstersFleetID has been added on fleets, only for PlayerCustomCity.

• The compound "city and mobile fleet" concept from the fallen spire has been removed from the game. That was such a mess, and made many things like "destroy distant fleet members" not work correctly.
  • The entire premise was just very not-helpful. There will instead now be separate fleets for the city and for the mobile forces.

• Added Pluralize.NET to our source code (it is MIT license), and added a new ArcenStrings.Pluralize() method, which efficiently wrappers it.
  • If you have a word that is singular that you want to make plural, this is how to do it -- don't just add "s" one the end, or what have you.
  • This aids in our dynamic text generation.

• We now have name_for_city_sockets="string" and name_for_city_center="string" for cities and city sockets, which are required for any unit that uses those features.
  • This allows us to properly talk about spire city sockets, and necropolis bone holes (wow that sounded wrong) or whatever we want to.

• "City Sockets" in Necropolises are now called "Necropolis Hexes," at least for now.
  • In spire cities, these are also now called "Spire Casings."
  • I specifically chose "city sockets" in the past because it was generic, but now we can be specific.

• At this point, things compile and the spire cities have been separated from their fleet portions, but not combined back in. This means fallen spire is effectively broken, but that won't last long.
  • Similarly, the necromancers spawn their base necropolis just fine, but their core fleet stuff has also not been split out properly yet. This breaks them, too.
  • I made enough changes that I was nervous about continuing to make more without first checking these in, though.

• Spire cities (and necropolises) can now be renamed if you wish. I'm not sure if that was possible before, but I think it was not.

• When you create a spire city, you now only get a city itself, not a half-city-half-fleet.
  • You will get up to three fleets (shared between all human players as they wish) depending on how many cities you have.
  • The first spire city gives you a first fleet. The first time you get any spire city to mark 3, you get another spire fleet. The first time you get any spire city to mark 5, you get a third spire fleet.
  • This starts you off with a more manageable and limited number of spire fleets, and keeps them separate from the cities (which work the same as ever), while letting you grow your fleets over time so that you can eventually cover many areas of the map.
  • Overall, thanks to this separation, the fleets for the cities and the actual fleets themselves work... way more like you'd expect. It's so much easier to deal with, now.
  • More changes are needed to adapt this whole thing into this new idiom, so stay tuned, but this is a good start.

• The city interface now has the start of where you can control which mobile spire fleet (or necromancer fleet) is being "bolstered" by the current spire city / necropolis. The idea is that each city-type-thing gives its units to one fleet or another, and that can be changed as you get more fleets.
  • The full functionality of being able to actually change that is not in yet, but the basic structure is, including auto-bolstering any new spire fleet from any cities that are not bolstering anything yet.
  • There's still more to do there, and the logic is per-faction, so I'm going to be having some hooks in there so that the necromancer can do whatever it needs to do, which is likely to be slightly different from the fallen spire. This is again something that is very mod-friendly now, so similar sorts of things can be set up in mods in the future. (So many possibilities, SirLimbo...!)

• The FleetCategory PlayerCustomMobile has been split into two parts: PlayerCustomCityFedMobile and PlayerCustomUnattachedMobile.

• For the city-fed fleets, there is now a FedHereFromCityFleetID on each fleet.

• The Necromancer is back to being more functional, with a Necrofleet being spawned with every Necropolis.
  • The term Necrofleet is obviously subject to change, and right now I'm not sure that all of the ships that should be spawning in the fleet are doing so -- they may still be spawning in the Necropolis itself.
  • But at any rate, it's progress.

• Removed the DoForFleetsParallel methods, because those are basically never going to pan out as a positive performance boost.

• Added a new DoForCityCenterFleets() and a new DoForCityFedMobileFleets(), and the underlying data structures needed to support making those run quickly.
  • Also made it vastly faster to look up fleets by ID, partially because in some areas we will be doing that a bit more (but honestly I should have done this a while ago, now that I look at it).
  • Also made it faster to loop over just player fleets.

• Necromaner ship attritioning and updating has been moved from DeepInfo to BaseInfo. This will be much better for multiplayer.

• More necromancer logic has been moved into BaseInfo. This is stuff that needed to be present for both the client and the host to work properly in MP.

• The necropolis mobile bits have been split out more, and the city should be feeding them. There are still some glitchy bits with it, but I think it's because I'm getting some extra fleets right now.

• More necromancer updates to try to get it to populate the mobile fleets from the city fleets. No dice quite yet.

• Success! The necromancer no longer creates double-fleets (meaning quadruple total) when it creates the major necropolises and the attached necrofleets.
  • Additionally, the skeleton and wight caps and so on happen on the necrofleet, fed by the necropolis bolstering them, rather than being created directly.
  • As far as getting back to prior functionality baseline of necromancer, this seems like we are pretty much there. More things will need to be adjusted in the coming weeks, but this is a great start.

• UniqueTypeDataDifferentiatorForDuplicates is now an int, rather than a byte. We're using this to delineate which ship lines are bolstered from other fleets, now, to make things easier.

• Added a new FleetQualifier string on fleets, which lets us identify certain fleets even if their fleet leader has died.

• In the fallen spire campaign, it now upgrades all of your flagships to battleships after you have your first mark 3 city. Once you have your first mark 6 city, they all become dreadnoughts.

• In the fallen spire campaign, all flagships of the spire are simply the mark level of the highest-mark city you have. This is simpler than the other system we had discussed on discord, and this faction is complex enough to play already.
  • This way you don't have to worry about spire flagship booster buildings, that's not a thing now.

• In the past, there used to be one fleet led by a spire flagship for each city you had. This is no longer the case.
  • You now start out with only one fleet. Once you have your first mark 3 city, then you get a second fleet. Once you get your first mark 4 city, you get a third fleet.

• In the fallen spire campaign, your cities properly bolster the fleets they are assigned to, and give more units in them at the mark level of the city.
  • There's still some functionality to add here, but the generalities are in place now.

• From the very start, each spire city now gives one spire frigate to the fleet it is bolstering.
  • This does make them a bit more powerful, but you are getting fewer flagships than before, and it's nice to have a bit of new ships from the start with every city.

Code Refactoring For Clarity

• Fleet.Membership has been changed into FleetMembership.
  • This is mostly for ease of file organization, but also ease of typing it.
  • In your mods, just do a simple search and replace that is case-sensitive and whole-word for "Fleet.Membership" to "FleetMembership"

• Civilian Industries (the mod) has been updated to function with the above change.

• The "PlayerPlanetaryDefense" FleetCategory has been renamed to "PlayerBattlestation"
  • This is way easier to understand in the code and know what you are looking at versus having to guess.

• General cleanup. Removed some deprecated code and data.

• SpecialEntityType: MobileCustomFleetFlagship has been changed to MobileCustomCityFedFleetFlagship.
  • MobileCustomStrikeCombatFleetFlagship has been changed to MobileCustomUnattachedFleetFlagship
  • These were set to be confusing as heck. The latter is not actually used right now (but is useful for mods), and the former is used for spire and necromancer stuff.
  • This separation also allows for me to start putting some specific sub-data on MobileCustomCityFedFleetFlagship fleets.

Collection Performance Improvements

• I have reimplemented a custom version of the List<> class from Microsoft's generics, and made various improvements to make it faster.
  • Overall it ranges from 3% to 10% faster on average, but it also no longer takes a poor-performing parameterless constructor. That was about 300% slower than a proper use case.
  • Sorting is also about 10% faster than before.

• A new test framework for testing how well various data structures work IN UNITY has been added.
  • Thanks to SirLimbo for originating the base versions of it.

• ArcenListPool has been removed, as we can do the same thing much better with ThreadStatic.
  • This was used only in the layouter rebuilder for the UI, and was a source of some slowness.
  • Got rid of TMP_ArcenListPool for the same reason.

• Removed Debugging_ArcenReadonlyList, because we simply did not even use it and it's not useful.
  • Oh, also Debugging_ArcenReadonlyArray.
  • Also RefPairCollection.

• The Queue<> class has been moved into ArcenUniversal. For the moment there were not really any performance improvements to make to this.
  • However, we may make some in the future, and because of the fact that we moved List<> into the codebase and we want to replace that, we need to also handle the other Generics collections.
  • So, same deal with Stack<>.
  • Oh, I did make the Clear() method a bit more efficient on both of them.

• Dictionary<> has also been moved into our codebase and made slightly more efficient.
  • Also one plus is that if you call in to get a key that doesn't exist, it gives the default value for that type, rather than throwing an exception. This is the same as our ArcenSparseLookup.

• ConcurrentDictionary has been pulled into our codebase, along with the other collections, so that it can use our own versions of the enumerators consistently, and so that if we need to make performance changes to it later we can.
  • At the moment I'm not super aware of any performance issues with these, and they're used only in certain circumstances anyhow. But being able to take control of our most central collections and correct errors that we do find is useful.
  • This of course means that ConcurrentQueue and ConcurrentBag also are making the jump in.
  • After looking at the code for ConcurrentQueue vs ConcurrentBag, I can see why there were such differences in performance that some programmers on the internet were reporting in the past.
    • These have completely different philosophies on how to build handle themselves and their data, and the fact that ConcurrentQueue was faster than ConcurrentBag in practical uses is probably not just a "whoops there was a small error there" sort of issue. Supposedly it was fixed so that Bag is now as fast or faster than Queue in the concurrent classes, but I've never seen any benchmarks, and I have not yet run any of my own.
    • From just looking at the code, however, I am now deeply suspicious of ConcurrentBag. It really throws off a lot of extra objects at the very least. It is notably simpler code (sort of) compared to Queue, but the thing about this simplicity is that it is missing some of the support data structures on Queue that just _happen_ to seem to make that likely to be faster.
    • In other words, I'm not convinced that Bag can ever be faster than Queue without a complete rewrite, or if it is then it will be highly context-specific. You would think that the fact that the Queue is ordered would make it perform better than the unordered Bag, but to me with a bit of a look through things, it looks like a lot more intensive thought went into Queue than Bag. Which is a shame.
    • We currently use some of both, but we are really going to need to test out performance in a real-world scenario and see if we can actually get an answer on which seems to be faster for those purposes, minus the noise of other things that fluctuate. I'm not convinced that any meaningful test-bench type tests would give us realistic expectations or results when it comes to the concurrent collections. It would be really easy to accidentally design tests that favor one or the other artificially.

• On ConcurrentDictionary, I have specifically avoided implementation of certain methods and properties, such as Values and Keys (the properties that give you IEnumerators or similar).
  • Those particular properties are handy in the abstract, but really are just kind of a convenience to make outer code the TINIEST bit more brief. But what happens is, under the hood, rather than doing a wonderful iteration over the internal buckets with a yield statement -- like the normal ConcurrentDictionary enumerator is able to do -- it has to pull things out into a separate list, and then enumerate that. Translation: lots of extra garbage being generated, and some performance loss. So, yeah, we can do without that feature.
  • In general with all of these collections, if there was something that is likely to shoot your in the foot with performance without you knowing it, then that's missing. If you're trying to do something and find you can't now... well, there's an easy mostly-equivalent still there, and it's going to perform much better than whatever it was you were trying to do.
  • Another notable example on all of these is CopyToArray() or similar. That's something that is rarely really useful in a runtime sense. It can be a handy shortcut when you're doing a debug dump or something... but just loop over things and use the extra performance during said debug dump (it will be slower than CopyTo or CopyFrom would have been, but that's okay because this is a debug dump). The thing I want to avoid is someone trying to do a CopyTo, thinking that's okay to do during runtime, because it's really not. There is a method to my madness! Often programmers make choices without understanding the hidden costs in the classes they are calling, and my goal is generally to help avoid that. Microsoft's goal is maximum flexibility, by contrast, and so you can shoot yourself in the foot all day as far as they are concerned. We have different goals, so provide different things.

• One side effect of a lot of these changes is that you can't declare "var" as the type of the thing you are doing in most foreach calls. The reason why is a bit complicated, but since this leads to more-readable code (that also performs faster in visual studio when you're editing, as a side bonus), I'm not going to complain or worry about it.

• Another side effect is that I can (and have) now put in base classes that let me control more things in the future, and which make my custom version of ObjectDumper now give a vastly better snapshot of game RAM when I do a dump of that sort.

• I swear, I have no idea where certain code was coming from, like TryDequeue on the regular Queue. That is in the Microsoft specifications, but not in their published source code. It must be added by some sort of other dll.
  • There have been some odd cases in the past for me where I had a Clear() method only some of the time for certain collections (this is weeks ago), probably based on which using directives I had imported. That was annoying as heck. All of the stuff is now directly implemented. If there wasn't an implementation, I made one.

• Side effect of some of these changes: the game itself may load more quickly now (getting to the main menu), and map generation may be a tad faster (probably not much), and savegame loading might be faster (closer to the speed of a second load rather than a first-load).
  • Depending on the specifics of your machine, it may make very little difference or it might be really noticeable.
  • Also during actual runtime, it should run more smoothly for basically everyone, but here again this is... cleaning up a lot of tiny paper cuts all over the body, essentially. It's not some giant wound or something. So seeing exactly how that has an effect in aggregate is always interesting.
  • On the MP client in particular (still not functional yet in the latest beta, but coming soon) there's some speedups with where I had been using the .Keys sub-listing of ConcurrentDictionary, having no idea that that performs terribly.

• I did not port over Hashset, so I have implemented RemoveWhere() onto dictionary, since that's something that StarKelp seems to find handy. (And yeah, that is handy.)

• If you have code of your own (for instance a mod), you'll want to remove the following references. You can do a global search and replace to make them blank:
  • using System.Collections.Generic;
  • using System.Collections.Concurrent;
  • using System.Collections; (this last one is optional -- you may need a couple of things from this, like IEnumerator, so you can leave it if you want).
  • Assuming that you use a using reference to Arcen.Universal, that's all you need now.
  • You can then expect to see some errors of the following kinds:
    • Collections that you didn't specify a starting capacity for asking you for one. Go ahead and set those up to be a little bit larger than you need, because RAM is cheaper than CPU on the scale we're talking about (small ram, big cpu to expand).
    • Any place you used var in a foreach will need to use the actual type name.
    • Any attempts to loop over the .Values list of a dictionary will also be busted and you should just loop over the dictionary itself.
    • Any places where you are using "IEnumerable<" should become "System.Collections.Generic.IEnumerable<" Easy global search and replace.

• Our regular dictionary now lazy-loads its internals, so that we can specify a desired size without actually having to do any work if it's not going to be used for a while. But at the same time, it doesn't initialize to a too-small value that then has performance issues during gameplay. The loading time after getting all this running was truly abysmal, double what it used to be with these changes, and RAM usage was also too high.
  • Most of that had to do with initializing a lot of lists and dictionaries to sizes that are sufficiently large for use, but which are very often only contextually-used, if used at all.
  • There are actually a ton of lists and dictionaries in our codebase that get initialized "just in case" of future need, and by lazy-loading those instead of aggressively front-loading them, we can save ram and performance and the end code never knows the difference.
  • This is the sort of cool thing that we can do now that we've taken control of our own collections, versus letting them be just part of the core .net framework.

• The KeyCollection and ValueCollection have been removed from the Dictionary class, which means that .Keys and .Values are also gone.
  • We did not use these super frequently, but they were a definite performance drain both when initializing a dictionary and also when making these calls. These essentially made the entire game a bit slower, and so now we just loop over the core, same as with the concurrent dictionary.
  • Also, this prevents programmers from making some... uh... questionable array accesses inside loops for things like serialization. Again a speed boost.
    • Actually this was happening in more than just serialization. So, yay, speed boosts in a variety of places, then.

• The Stack class now lazy-loads, giving it the same performance boost as dictionary now has.
  • Queue now, also.
  • And then the big one: list. Performance is back to what I would hope for all this, and is now finally better than the old standard.

Sparse Lookup Replacement

• Dictionaries, lists, and so forth all now support doing a foreach on them even when changes happen.
  • This means that you can remove from a dictionary you are foreaching over and not get an error. Similarly, if another thread touches the data, it no longer will error a foreach.

• Added a new ProtectedDictionary, which does much the same sort of thing that ProtectedArcenSparseLookup does, but for Dictionary.
  • Most things that used to use the ArcenSparseLookup are being converted to Dictionary, so this also needs to exist now.

• Ported ClearAndCopyFrom() from ArcenSparseLookup to Dictionary.

• ArcenSparseLookup has internal improvements from both our new List and Dictionary classes, but it's still stupidly slow because it does both things at once.
  • Essentially, the dictionary class now does everything that this one did, except faster, with one exception: sorting. So in the event you need that, you need this. Otherwise Dictionary is the way to go now.
  • With that in mind, this has been renamed from ArcenSparseLookup to InefficientSortableDictionary.
  • Unless you need sorting capabilities, please be sure to use Dictionary from now on!
  • If you are converting things over, you can do a global search for "ArcenSparseLookupPair" with "KeyValuePair".
    • Also replace "ArcenSparseLookupPair" with "KeyValuePair"
    • And replace "ArcenSparseLookup" with "Dictionary"
  • You will find some cases where previously we took advantage of the fact that ArcenSparseLookupPair was a reference type. The old code from ChooseWaveTarget() is a good example of this.
    • That code will no longer work, because as a value type, trying to assign into the KeyValuePair is readonly on purpose (it prevents programmer error). You would now instead need to assign to that in a constructor, or assign directly into the dictionary itself, depending on what you are doing. This is a very good thing, it's way more performant and also error-proof.
      • See the above method (check diffs if you want) as an example of how to deal with it. The code is just as legible now, if not more so, and it also doesn't throw any garbage onto the heap.

• Added a new extremely-efficient CopyToList() function on Dictionary, which works via a special hook I've set up with List<> to make that work really really well.
  • This is an early example of the sort of power we've now got to (either shoot ourselves in the foot if we make an error or) do really cool things that allow for better functionality and also improve performance while we're at it.

• Added a new GenericDictionaryBase, which defines some abstract methods that are normally implemented by dictionary.
  • This allows me to make ProtectedDictionary inherit from Dictionary, rather than wrapper it. This is notably more efficient, and also more functional.
  • This will be a nice little performance boost in general, and since I'm using abstract methods, it's just as fast. Note that I could have gotten the same effect by marking dictionary methods as virtual, but if you look at the language specs, that is slower.

• Okay, so a minor case study. One of the things that we needed to sort previously was EnergyConsumedPerFleet_ForUIOnly. This is on factions, and was an ArcenSparseLookup.
  • This is now just a dictionary. Previously, it was aggregating the data during sim AND sorting it (which it even noted was redundant, sigh). It no longer sorts this data until you actually view these details, in the UI subwindow where it was already re-sorting before.
  • But wait! We can't sort a dictionary... what's happening, then? Well... this is where we get into the magic and vastly better performance (and code that is also simple.
  • Before, the code looked like this: localFaction.EnergyConsumedPerFleet_ForUIOnly.Sort( ArcenSparseLookupPair <Fleet, EnergyUsageDataHolder> ). And then we'd iterate over the internal list (hyper inefficient, sigh) that the SparseLookup had.
  • Now we do the following in the UI code when we want to draw this in a sorted way:
    • Declare a List<KeyValuePair<Fleet, EnergyUsageDataHolder>> energyConsumedPerFleetForSorting, which is one line of code. This is very efficient.
    • Call this: localFaction.EnergyConsumedPerFleet_ForUIOnly.CopyToList( energyConsumedPerFleetForSorting, 0 );
    • At this point we have a list copy of all our dictionary data, and it was done using Array.Copy internally, and allocated nothing new. It's super duper efficient, and only runs when we actually need it.
    • Of course, now that we have a list, we can just sort it. And we can do this when we need to, but skip any other times.

• Okay, let's follow up the above case study. That is indeed not TOO bad, and you can do that still, but it's more code than I want to deal with. Generally there's a 1:1 relationship between the thing you want to sort and the dictionary, right? So let's build that into dictionary instead, a bit more directly but with that same degree of high efficiency.
  • The TLDR is that I don't feel like typing that much code each time I want to do a sort, and since I control the dictionary class now, I don't have to. Let's see how much shorter we can make the code while not incurring even a tiny bit more performance:
  • Define a holder working variable in the local method like this: List<KeyValuePair<Fleet, EnergyUsageDataHolder>> energyConsumedPerFleetSortedList = null;
  • Then assign into it in one call like this: energyConsumedPerFleetSortedList = localFaction.EnergyConsumedPerFleet_ForUIOnly.SortIntoList( yourSortHere )
  • Then just loop over your energyConsumedPerFleetSortedList, and ignore it other than that. Much more quick to type! Same efficiency as before.

• You know what this means, I think: goodbye InefficientSortableDictionary! There is absolutely no reason to have it or its variants now. The above case study for sorting is far more efficient.

• While I have been going through the dictionaries to fix these up (well, the ones that were arcen sparse lookups in a prior life), I also have fixed a number of places where we were doing extra allocations and hitting the GC. Basically just stuff from before we knew about ThreadStatic, or where somebody didn't bother to use it for whatever reason.

• Added a new SortedDictionary class, which mostly just uses dictionary EXCEPT that it is expects to be sorted and will enumerate over the internals of the sorted list rather than the dictionary itself.
  • This is used in very few places in our code, but this is the only way I could really think of to make sure that code remained readable and also high-performance and without being prone to bugs. My first couple of ideas on this were terrible, so I dropped back to this, which works very well.
  • This is mainly for the strong-against and weak-against code, and some other ship lister results, and I haven't tested it yet. If there's a bug, that's where it will be. It's also used for the "list of ships that will benefit from hacking a tech vault." Also the list of planets that have threat on them when you mouse over the threat tooltip. Oh, also the list of ships that will benefit from upgrading a fleet by science directly. Aaaand it's also used for the ships that benefit from regular techs, too. If those tooltips are broken or not sorted nicely, you'll know it's this.
  • Okay, have tested these out now, and they do work (I had a few fixes to do before they did).

• For the sake of efficiency and code brevity, I've added a SortIntoListWithoutSort() and GetRandomKeyValuePair( RandomGenerator Generator ) onto the dictionary classes.
  • This makes it easier to do some things that we used to only be able to do with the sparse lookup. It's less efficient than not doing these things on a dictionary, but it's more efficient than sparse lookup used to be.

Beta 3.704 Unexpected Turbo Sim Speed

(Released October 1st, 2021)

• In the lobby, the AIs now have a nicer display on the left sidebar of their difficulty and their type.

• There was some OLD logic in our "read strings from files in a folder" logic that said to look in an "Override" folder inside that folder if you wanted to, well, override it. That's been stripped out.
  • This logic instead now checks all the mod folders for the folder name in question, and if it exists then it pulls from all of those mods (if there are more than one with the same folder name) and ignores the main game.
  • Translation: this now allows for mods to be created that have fleet names or planet names. These can be additional sets, or they can be replacement sets for our existing "Standard" option or what have you.
  • Thanks to RocketAssistedPuffin and NR SirLimbo for reminding me that this was one hole in our modding toolkit.

• Added a new "League+" name set for planet names, created by Angel of death.

• The fleet management window has been updated slightly, to make room for modular unit controls.

• The start of modular ships -- aka, ship loadouts -- is in place now.
  • There is an is_modular="true" field, which can be assigned to any unit from DLC1 or DLC3, or any mod unit if you have DLC1 installed (loadouts/modularity are going to be considered a DLC1 feature -- a few other DLC1 features are coming, along with a later price increase on DLC1, to make all three DLCs equal in scope and value).
  • Right now the modular flag doesn't do much, but it's the gateway to the next things I'll be adding on those units.
  • Of note, I wound up looking at how I was thinking about changing things like forcefields to work via systems, and... definitely decided against that. I'm going to wind up with a split system for handling modules, one for weapons (easy), and one for "other random stuff" like extra health, or bubble forcefields, etc. I have a good design in mind, and it will be higher-performance and not require breaking existing saves in order to do it. All in all that should make it faster to implement, too.

• The fallen spire ships have all been split up so that they now have a base root unit for each class which is what gives the general stats and such to the imperial spire versions and regular and flagship versions, but no weapons.
  • Right now, the regular, flagship, and imperial spire versions all have the same weapons, but they have some other different stats. The imperial versions are used by NPCs and are marked as not-modular.
  • The regular and flagship versions are going to become modular soon, and will have potentially different modules than one another, so hence having a divergence from a new common base.

Bugfixes

• Fixed exceptions that could happen with CustomData_StartingAILayout, CustomData_StartingAIPlanetRatio, and CustomData_WormholeBorerIncomeModifier if the data for them was blank. Now it just uses the default value in those case. Not entirely sure how the data can be blank, but in some fashion or other it was just data that was initialized without the player actually making a choice about it.
  • Thanks to BoBy for reporting.

• Fixed an exception that could happen in Marauders stage2 with lazy-initialization of their external data.
  • Thanks to BoBy for reporting.

• Fixed an issue where the lobby and in-game factions screens were showing the same information in the tabs on the left for every faction beyond the first for any given type of faction. So for instance, each AI showed the info from the first, etc.
  • This actually finally explains how the attachedfaction used to be null in that area, which is nice to know I wasn't crazy. I couldn't see how it was null, but it was using something outdated that I forgot existed. Now it uses the expected central items.

• When I set up the AI data and subdata in as part of the great refactor, I accidentally wound up having two copies of the central AI type and difficulty. One on the main object (not needed), and one on the sentinels sub-object. Most things were reading from the latter, but the latter was also always set to the defaults rather than the values you chose.
  • Now there is only one set, and everything reads from that, and this works great.
  • Thanks to Arctic for reporting.

• At least since the great refactor, but maybe longer, the escape menu where it shows the list of factions has been a bit of a mess, with extra intensities showing and so forth. Fixed up those bits, and then also just made it look a bit nicer than it did in the past for the AI types in particular.
  • Thanks to Arctic for reporting.

• Fixed a nullref exception that could happen in SpecialFactionPlanning.GetNeedsToRun().
  • Thanks to BoBy for reporting.

• Fixed a cluster of issues that could result in the AI sub-types or sub-difficulties not being set properly when loading savegames in some cases.
  • This then fixes all sorts of other random issues that would show up in other spots if this happened. The central logic for these is also now set to error if they still remain null, so that we know what the true problem is rather than worrying with random other issues.

• I can't (yet) duplicate the error, but there was a pathfinder being used from two contexts incorrectly, and I've now put in a lot more code around making that not happen. It appears that this was caused during an exogalactic attack, but that was honestly the part that looked correct from the log -- there was some prior call from the AI sentinels to the wrong pathfinder, and that set things up for the later error. I now have it so that it will detect the FIRST wrong call, and not just when there are calls from two contexts. I've done a manual code review of all the affected areas, and can't see how it would have a problem, and I've done some testing for a while as well, so I guess we'll just have to wait for it to trip again. Next time will be more informative, or should be.
  • Thanks to whimsee and Vinco for reporting.

• HandleExogalacticAttacks has been updated so that it is now using threadstatic working collections so that it is fully threadsafe. An error in a log may have been due to that method being called by two factions at once, or it might have been related to the pathfinding issues. Either way, I made the method threadsafe, made it slightly more efficient, and then also put in extra logging spots so that we will know which it is if it ever has an issue again.
  • Thanks to Vinco for reporting.

• Fixed an exception in the scourge that could happen during long-range-planning if their warriors were not properly assigned their BaseInfo data yet. Honestly this seems risky from a threading point of view to have it handled this way, but apparently it worked prior to now. At any rate, now it should still work.

• Further fixes to the scourge. Their data is now initialized in Stage2 if it is missing, which should avoid any LRP issues or other cross-threading issues, and some followup exceptions that I saw.

• Improved the error detection when deserializing faction data.

• Previously, if you were trying to hack something but got an error about having too many hackers to choose from, there were the following problems, all now fixed:
  • It would choose one and hack anyway, which was potentially going to be quite a problem.
  • It would only show you this warning AFTER you clicked "yes I'm sure," rather than before, which is just plain frustrating.
  • Thanks to Vinco for reporting.

• Fixed a variety of errors that could happen with astro trains based on them not having their external data properly initialized. At least one or two of these were translation errors on my part during the great refactor.
  • Thanks to whimsee for reporting.

• The internal nomenclature for getting network-attached settings is now a lot more clear. It says NetworkAttached, specifically.
  • In order to trigger journal entries properly in multiplayer, the network-attached settings idiom is still really useful for our automation settings. At first it seemed superfluous, but it's really not.
  • Found and fixed a problem (maybe from the great refactor, not sure), where network-attached settings were not properly being sent. This affects singleplayer as well as multiplayer.
  • With that, watchmen frigates now auto-build again, and any other non-default automation settings also work properly.
  • Thanks to whimsee and Vinco for reporting.

• The timings window now shows us how many player accounts are in a game (way down below the threading info), and which one is the local account, if any.

• Found and fixed that pathfinder issue! It turns out it would only happen after you had saved and loaded a few times, or better yet had generated a couple of different maps. It was a pooling issue.
  • Thanks to whimsee and Vinco for reporting.

• Fixed a rather subtle bug that had to do with singleton cleanup on external faction data in general. Essentially, it was not being cleaned up when going back into a pool, but only on coming back out.
  • The notes in the code make it more clear why I now handle factions and squads in an opposite manner from one another, but at any rate factions have been adjusted so that there are no longer lingering effects of absent factions after you remove a faction.
  • The most visible of these was nomad planets, where if you added that faction and then removed it, you would see nomad planets in all your maps until you restarted the entire game program. The nomad faction was indeed gone, but its singleton was still there and so it was kind of half-alive.
    • There would have bene other factions (fallen spire, etc) that also had this sort of characteristic, but they no longer do.
  • Thanks to whimsee for reporting.

• Reworked the lobby faction tab so that it no longer has any chance of doing funky stuff when factions are removed or the faction you are viewing is changed. Previously it could throw exceptions in some very rare and unlucky cases. I don't think the factions window in-game (after starting a campaign) can do this, but if we see it then I'll make similar changes there.
  • Thanks to whimsee for reporting.

• Fixed an exception that could happen in nanocaust logic because I had switched one of their data structures from an ArcenSparseLookup to a Dictionary. Rather than trying to recode all that logic (bad idea), I just returned it to being an ArcenSparseLookup. This should solve the issue.
  • Thanks to nas1m for reporting.

• Fixed a post-refactor exception that could happen in QueueChatMessageOrCommand based on null being passed in for SFXItemInternalName instead of empty string.
  • The reason this was only happening post-refactor is that there used to be different message signatures for passing in a Context object, and the method call in the DZ thought it was passing null context, not null SFXItemInternalName. But since the change to no longer pass context, it wound up effectively passing null SFXItemInternalName.
  • I have just made it properly handle null SFXItemInternalName, so any other cases of that are now fine.
  • Thanks to Badger for reporting.

• Fix a DZ bug where the PerUnit data was not clearing all of its fields.
  • Thanks to Badger for fixing, and HarryT for reporting.

• Hardened all of the methods in AllegianceHelper against errors during game quit. There were a number of places where they were vulnerable to cross-threading exceptions during game teardown.
  • Thanks to Badger for reporting.

• The game is now a lot better about killing threads when it wants to go back to the main menu, or exit the game in general. Before it was possible that they could linger for a moment and throw errors.
  • This is one of those "how did that ever work without errors all the time on exit?" sort of situations. Nevertheless, it was still rare even now.
  • Thanks to Badger for reporting.

• Okay, took a much more aggressive approach with on-quit errors. The fact is, we simply have too many threads with too much error-handling to guarantee that none will error a we pull down the walls around them. And there's no guarantee that a mod will properly handle ThreadAbortExceptions, let alone anything else more specific. So let's handle this centrally!
  • When you quit the game or quit to the main menu, after it has done anything that we might care about (like saving an ironman save or something like that), it now gives a 3-second moratorium where it ignores any and all exceptions that are thrown. Those are all assumed to be spurious (because they are), and that means we don't have to worry about the fact that we can't halt a Parallel thread easily (that was one of many problems), or that certain code might not be set up properly for the abort codes.
  • This has no net negative effect on the game, because essentially this is just stuff that was running that did not need to be, and it's not going to hurt any data.
  • Thanks to Badger for reporting.

• The default planet naming style is now set to Standard.

• When you hit Reset To Defaults, previously it was not setting the planet naming style to the default, but now it does. Same with the planet count needing to be set to 80, and also with the map type needing to be set to the default (which is Realistic). Previously it was just randomizing the map type.
  • Thanks to Zweihand for reporting.

Simulation Performance Improvements

• Improved the efficiency of the strengthcounting background thread when it comes to looking at incoming waves and adding their strength to your totals.

• In general, improved the efficiency of strengthcounting by making it use a ton more background threads. This is one of those areas of the game that can be a real sim-speed limiter, it turns out.
  • In particular, with higher numbers of planets, this can peg even really new, high-speed high-core CPUs. I have it spreading out amongst CPU cores better now with more threads, but I still see it using too much, so I am going to make some further improvements.
  • This and metal flows planning are two areas where I should be able to make the simulation run a lot better for everyone, particularly in giant games, now that I can see that they are spiking up as problem points pretty frequently even on my Ryzen 7 4800H.

• MAJOR refactor of the strength counting logic to time-slice its work over the span of 10 frames (1 second).
  • As part of this, also discovered that too much Parallel looping over planets was a major performance problem inside this thread.
  • Even on an 8-core machine, this was spending more work getting those threads up and down than it was getting benefit out of the threads themselves.
  • Also, as part of this, I've discovered that the strength counting itself is one of those things that has a MAJOR impact on the game the more planets there are. I had wondered where all our performance was going with high numbers of planets in the past, and... well, here's the answer. This is one of the most longstanding mysteries of the game, solved.
    • Even now, with the awesome new time-slicing, the difference between me running this on 80 planets and 110 planets is "always buttery smooth" and "kind of choppy at times, but not so bad as before." This was with two AI factions and the fallen spire.
    • When I stick with 80 planets but add 10 AI factions and one each of most every other faction available in all 3 DLCs, plus two of a few of those factions, then the performance of this -- from second one onwards -- is abysmally low. I go from having something that takes about 26ms on average (that's lower than 100ms by a lot, so that's good), to something that takes 300ms on average (that makes for instant incredibly high sim lag) with spikes to 600ms. This is AFTER all my existing optimizations.
  • I think that what I'm going to do is actually spread this out even further, to time-slice this over 3 seconds rather than 1 second. My scenario with such an insane count of factions is not a reasonable one to play, but I have a very high-end CPU, and I would like for the base case to have a lot of extra wiggle room where it's not even coming close to performing poorly.

• More strength counting performance improvements:
  • The final step of copying the strength aggregation data from the working location to the final data spots that are used has been moved to the same background thread as the rest of the strength counting.
  • In my prior version, I had the planet strength counting split over two time slices. That is now split over 5.
  • In my prior version, I had the aggregation strength counting split over four time slices. That is now split over 20.
  • With this setup, I now get vastly better performance in my unreasonably huge test bed, and notably better performance in my regular use cases. However, there are some major downsides, so I'm going to redo this yet again from another angle.
    • Firstly, in the huge testbed case there are still some outlier spikes, which get worse as time goes on. So an initial sim speed that was better deteriorates over half a minute or so. I'm not entirely sure why that is, but I have some guesses.
    • Secondly, the human-facing data is only being updated every 3 seconds now, which really does not feel right. When I explore the map via a cheat, it taking 3 realtime seconds to appear feels very off-putting.
  • Don't worry, I have a plan. :)

• Okay! A whole new approach has been implemented for the strength counting thread. The short answer is that performance is through the roof.
  • Before, I was trying to time-slice at manual spots that I was personally selecting. That just wasn't working out very well, for a variety of reasons. We would have some slices that ran well over 100ms, and thus slowed down the sim, and others that were still a lot higher than I wanted, while others were almost no time at all. But... even worse, which parts are the most expensive is something that depends on the specific savegame (how many units, how full specific planets are, and many other things). So there's literally no way to make a generalized solution to this in a hand-designed fashion. It will always fail somewhere, making it completely impossible to solve that way. So...
  • The new solution that I implemented is ADAPTIVE time-slicing. Here's how it works: essentially, at certain juncture points it decides if it has been running too long, and if so then it terminates. Wherever it is, it doesn't matter. The next time it is run, it resumes right where it left off. To a non-programmer this might sound easy and obvious, but when you're talking about a linear function it is really not a common pattern to do something like this.
  • The metric that I wound up using was 40ms. If the current iteration is taking more than 40ms, then it stops for now. I want the remainder of that 100ms to be used for the rest of the sim. And honestly, since I can only check but so frequently how long it has been it will sometimes hit 60-70ms per slice like this, anyway. In a suitably rough location in a suitably huge save, it can get a bit over 100ms, but rarely. I've iteratively optimized that down so it doesn't happen much.
  • Overall this now means that the strength-counting data is slightly out of sync on clients and hosts, and it means that it is not up-to-the-frame accurate anymore (well, not always). In small savegames, if it can run all in one frame, it still will, of course. But for really huge savegames, it will take a bit longer.
  • I had a couple of test cases, and both of them improved dramatically.
    • One was a basic game with two AIs and the fallen spire. Under the hood, this meant 23 factions. I was testing on 110 planets, which earlier today was feeling a bit choppy even though it was close to being perfect sim speed. Now it's running without the chop, at full sim speed. It takes about 300ms (76ms of calculation-time, which is 40% of realtime) for the data to be updated now, versus forcing it to be updated within the initial 100ms span. Previously, this had meant that there was only 24ms of time to do ALL the rest of each sim step, but now there's lots more, so it runs well.
    • The second test case was a truly insane one, with 80 planets and 10 AIs and one of every faction along with two of a couple. Under the hood, this makes for 111 factions. Prior to this specific update, I was getting about 40% sim speed on my very nice CPU, and it was so choppy it was almost like a slideshow. Now I get well over 60fps, and almost perfect sim speed (95% or more), and it takes about 1.5 seconds for the strength counting data to be updated (670ms of calculation time, which is 40% of realtime).
  • This is... an incredibly dramatic improvement. A ton of game scenarios that were never possible before are now potentially playable, based on this early data. And since the code is adaptive, it should handle many cases such as super high unit counts or super high planet counts or whatever else. I'm going to test the latter of those next, just to see what it's like.

Options For Larger Maps

• The planet count max for all map types has been increased to 300 from a prior range of 120-160.
  • With 300 planets, just starting the game used to give very poor sim speed, but now that problem is solved. Framerate still takes a big hit even when you aren't looking at anything in particular, but it is north of 40fps for me on my dev rig, so it seems like an option worth having (with warnings in the lobby of likely performance issues if you don't have a very very nice machine).
    • I am reminded of the Supreme Commander maps that were at the largest possible scale, where they ran poorly on basically any machine that existed when the game released, but now they run fine. This is kind of like that, although it is playable on current machines if you don't have too many factions.
  • I tried with up to 500 planets, and it took 15 seconds to generate the map and then I was getting literally 20fps even though the sim speed was about 80%. That is still a titanic improvement from the past, where it would have been sub-10%. Nevertheless, 500 planets just feels way too laggy and likely always will given how the game is structured. I'm not quite sure where the new bottleneck is in the code, but the game just wasn't designed for that sort of planet count, anyway.
  • Planet counts higher than 170 and 220 now give you different warnings about possible performance issues if your machine is not very good. So you have the option for these mega maps, but hopefully also know what you're getting into.

Beta 3.703 Smooth Sim

(Released September 29th, 2021)

Improved Simulation Handling And Speed

• Making some changes to the threading model for AI War 2 to make things a bit more clear:
  • ProcessSimStep is now called ProcessArbitraryFrameOnMainThread, because that is FAR more clear and accurate. Even I was confused by the old name.
  • DoActualSimStep is, similarly, now called ProcessCoreLogicForArbitraryFrameOnMainThread.
  • Fixed a bit of logic that was running twice on the host and in single-player, eating up some extra processing time. Not much, but nice to clear that out (this happened as part of The Great Refactor).
    • Actually, this might have caused flickery lines, not sure.
  • OnClient_SecondsStoredUpForNextFrame is now OnClientOrHost_TimeStartedLastFrame.
  • All of the short-term sim context work is now handled on the main sim thread, rather than on a secondary worker thread managed by that.
  • These short-term threads are also now executed on the main sim thread via a Parallel call, which makes it so that they are able to happen much more efficiently in terms of how they get started and finish.
    • This is a MAJOR adjustment to the game, and likely to improve performance quite a lot, potentially on older machines the most (but really for anyone).
  • A whole bunch of other logic that I've hated for a long time relating to how the simulation kicks off threads and checks for gamecommands and so forth is a lot simpler now.
    • The idea is that we want to make sure there is minimal input lag in multiplayer, and that we're seeing as close to zero lag between "finished one thing" and "started another" as possible.
    • Previously, if you had a choppy main thread framerate for whatever reason (low powered graphics card and giant battle?), it could make for your sim speed being affected since that was being time-sliced over frames. That sort of time-slicing is no longer done, which leads to more stable simulation kickoffs.
  • This whole thing actually simplifies several areas that were really complicated before, and which were impossible to profile well, because they had noise from cross-frame data in them that made it impossible to know what was just general frame time, and what was actual processing time on these themselves.
    • This simplification should lead to some performance benefits in intense savegames in general, and in multiplayer basically all the time. Not that multiplayer is ready to go again yet since The Great Refactor, but this will be a big benefit for it once it is back up.

• In the thread timings details window (right-click the timer in the bottom left window), you can now see the last sim time in milliseconds, and the average sim times of the last 20 sim frames (2 seconds' worth).
  • If this is above 100ms, then you will have a slow sim because of the sim itself. If it's below 100ms, then if your sim is seeming slow, it's because of factors outside of the sim itself. That could be frame timings or other factors, but this is giving you the details on JUST the sim itself, which is useful for isolating what the problem is. In multiplayer, "sim speed" can be low because of network issues, for example, and this lets you see that.
  • Interestingly, I was seeing some performance ratios hitting down into the 80% range when looking at the traditional view of things, but I can at the same time see that the slowest actual sim frame time was 36ms. That is faster is actually needed in order to have a perfect 100% sim speed. So either the calculation is off, or something else is causing sim speed to stutter.
  • Until today's refactor of the central sim timings, I would have had no way to tell which. Now I can actually investigate it.

• Fixed an issue (that has been incredibly longstanding -- years and years) where after the lobby, your game would be acting like the sim steps were extra slow.
  • Also fixed a much more recent issue where after the lobby it would say there were a bunch of faction threads that had been inactive for a long while.
  • This sort of thing is trivial, but at the same time could be alarming to new players and is not the sort of thing we want people seeing right as they start a new game.

• When mousing over the clock in the bottom left and seeing the simulation speed tooltip, it could have a pretty wicked flicker. Fixed.
  • Additionally, when mousing over this or clicking into it, it now also shows you the framerate that we have calculated you are getting. Previously we only showed that on the main menu for whatever reason.

• The way that I have been calculating the simulation speed (for visual display) over the last years has been... cough... truly asinine.
  • It was some strange logic where it was kind of half calculating itself based on the time that it was called in the UI when the frame number had changed, rather than just doing a proper calculation from "frame number counts up to next time frame number counts up."
  • This has been corrected, and the timings are now vastly more accurate.

• Because of the combination of changes in this release, I now can see that I am often tripping the sim to start slightly slower than I should (about 95% instead of 100%), even while running at 120fps and with a sim load that is 30% of what is available, which is leading to some slight choppiness as well, probably because of how I am handling this between frame-timings.
  • Ironically, unity themselves had something along these lines themselves recently: https://blog.unity.com/technology/fixing-time-deltatime-in-unity-2020-2-for-smoother-gameplay-what-did-it-take
  • I can't blame my own frame timings being off on them, but the mistake that I made is very similar to theirs. I'll have to get that straightened out now, too.

• Well now! Here's another REALLY nice improvement, which will have a positive impact on multiplayer and single-player.
  • Essentially, I smoothed out the sim-steps by sampling what recent frame-times were (the most recent 30 only, no matter the framerate), and then taking the average of them. If the next sim step is not yet ready to start based on our target of "every 100ms," but 55% of our average recent frame time WOULD push us past the sim step start time... then I run the frame on this sim step anyhow.
  • This essentially works out so that we get between 99-102% sim speed on average, with a slight bias towards being a tiny bit fast (that's why 55% instead of 50%), whereas before it was averaging only 95% pretty much all the time.
    • Why the bias towards being fast? Well, ship movement gets slightly choppy when the sim is running slow. The UI layer is expecting things to be running at exactly 10fps on the sim, and is basing its linear interpolations on that being correct. Being slightly behind on a lerp will still look smooth, but being even a bit slow (well, more than 1% slow anyhow) introduces micro-stutters.
  • It has been quite noticeable when you watch ships move around, how they have micro-stutters in past versions, and it has always bugged me. That micro-stutter is now gone.
  • This also has positive effects for multiplayer, and really single-player also, in making the clock move at a consistent speed rather than slightly too slowly, or at slightly different rates on computers with different framerates. I believe that in the past the drop in sim speed, and the micro stutters, were worse on lower framerates. When the frame times are larger (more coarse), it's harder to get exactly to the 100ms interval I'm seeking.
    • At 30fps, every frame is 33.3ms long, for instance, so previously the sim steps would be off by a range of 1-33ms. In the new system, by splitting the difference the way I am, the divergence can only ever be 1-16ms at 30fps. This is still large, but it's now 16% of the total runtime instead of the (huge) 30% of the runtime of a sim step.
    • At 100fps, we get 10ms per frame, and before it ranged from 1-10ms off. Now it just ranges from 1-5ms off, which is much better.
  • TLDR: gameplay in general is now smoother visually, on all hardware and all framerates, and multiplayer will diverge less between machines of different power levels.

Bugfixes

• Quieted the error "Entity in CenterPlanetViewOnEntity null!", which could happen in some circumstances when clicking through wormholes. It's not an error that is actually worth talking about; we should just ignore it.

• Fixed an exception that would happen whenever you moused over wave notifications. This was another case of my efficiency-improvement code hitting a legacy case which errors on purpose to let us know. It was supposed to give us compiler-level errors, but unfortunately that didn't work as hoped, so now we have to find them in the runtime.
  • Thanks to CRCGamer for reporting.

• Fixed an oversight in the great refactor, where I forgot to actually clear the AIP info between loads of the game! So that meant that starting a new game after having been in a game with AIP would leave you with extra AIP and a strange AIP history. Loading saves was fine.
  • Thanks to Arctic for reporting.

• Fixed the bug (starting from just before the great refactor) where every time you load a savegame it would give you another 10 AIP. Yikes!
  • Thanks to Lord Of Nothing, Badger, Strategic Sage, Arctic, and HTL9000 for reporting.

• Fixed a pre-refactor bug that was introduced when I redid the notifications system. The spire debris notification always showed zero, and would trigger errors when clicked.
  • Thanks to Vinco for reporting.

Youtube Threading & Multiplayer Modding/Programming Guide

Coding With Chris: AI War 2's Threading & Multiplayer Part 1: Challenges

Coding With Chris: AI War 2's Threading & Multiplayer Part 2: Data Flow For Unusual Solutions

Modders: Do you need to watch all of the above?

1. Not one bit if you're not coding anything. If you're making xml mods, you can completely ignore the above.

2. If you already think you understand how RTS networking works, then just read the below images and then skip to about the 30-minute mark in the second video.

New Diagram For Modders/Programmers: AI War 2 Threading Model

New Diagram For Modders/Programmers: Multiplayer Data Transmission Model

Beta 3.702 Bugfixes And Mod Conversion

(Released September 28th, 2021)

Bugfixes

• Fixed a bug with all hacks where if the Fallen Spire was not enabled or installed, you'd get endless errors instead of the hacking screen.
  • Thanks to Arctic and UncleYar for reporting.

• The Zenith Architrave now does a lot more verbose logging of its externaldata in case anything is wrong with it.
  • From analyzing a specific savegame, it looked like there was something wrong with it, but now that does not seem to be the case.

• Fixed a minor nanocaust deserialization issue that would cause multiplayer clients to drop their constants and reload them every second or so. This was found during code review while looking for something else. Not a biggie, but could have been annoying.

• Fixed two deserialization errors that meant that any savegames with nanocaust in them could not be loaded.
  • Thanks to Magiik for reporting.

• Fixed a serialization issue in the dyson sphere faction that would throw warning errors, but not actually cause any problems.
  • Fixed a similar one in Nanocaust.
  • Thanks to Magiik for reporting.

• The scourge as a faction were throwing errors and shutting down in Stage2 if they were included in your save in certain circumstances.
  • The scourge have had their Stage2 instrumented more to give us more informative feedback when that happens.
  • It appears that some Scourge Spawners could sometimes be missing their externaldata. This is now fixed.
  • Thanks to Magiik for reporting.

• Okay, when I reorganized the entire savegame file, one of the things that I changed was how galaxy info was deserialized relative to the entities in it. Normally entities are units -- ships, structures, etc. But sometimes they are wormholes.
  • Long story short, I had an oversight where I was calculating which wormholes are linked to which other planets BEFORE we had loaded any wormholes out of the savegame, meaning that each planet thought it had no links to any others. That meant that your movement commands between planets would fail after you loaded a save, but not after you started a fresh game.
  • Fixed, and this also fixes existing saves if that matters.
  • Thanks to Daniexpert and Andriy Mykhaylyk for reporting.

Mod Updates

• Civilian Industries has been converted over to work in the new framework, although it has not yet been tested. There is a four and a half hour youtube video showing the entire process, with commentary and advice on a variety of coding things.
  • https://youtu.be/3Aw34XjO9nk

• The following mods have been hidden for now, since they are nonfunctional for the moment.
  • AMU (SirLimbo is actively working on this. You can follow him on the modding discord channel. This may be quite a while coming.
  • Devourer Chrysalis (Again, NR SirLimbo is working on this. It's more than 100 hours of work, so it may take a month or two).
  • Kaizer's Marauders (Same notes as the above two, SirLimbo is still working on it).
  • Macrophage Histiocytes (and spire integration) (These are by StarKelp, and I may have time to fix them someday, or he might get around to it, it's hard to be sure yet).
  • The Reprocessors (this is a CRZgatecrusher mod that needs updating, and I haven't heard from him yet).
  • AI Shield Generators (this is a cml mod, and this one is going to be retired, with the author's permission, as its functions are going to be built into the base game with consultation with the mod author).
  • This should solve any confusion about which mods do and do not work. All of the other mods that you see are working so far as we know, although of course with Civilian Industries that has not been tested at all yet, but I have done the conversion as part of this build.

Beta 3.701 Natural Object Order

(Released September 23rd, 2021)

• Made it so that saves from 3.700 are no longer valid to load, because they are SO broken there is nothing new we can learn from them now.

Bugfixes

• Apparently we had never really used bool toggles on factions prior to now, because there were some errors lingering in there from way on back. During TGR, I converted the "debug mode" options for various factions to all use those, and that then led to errors in the faction screens for them. Fallen Spire "Super Easy Mode" is one example of those.
  • This all works now!
  • Thanks to Vinco and Magiik for reporting.

• Adjusted things so that the "Skipped an extra mapgen call that was substantially the same as one that was run X seconds ago" won't be hit if there are changes in the number of factions compared to now versus then. Just in case. I am not positive this was a problem before, but it might have been, and now it can't be.

• The warning for things like "Roaming Enclave Migrant is 23 characters long, and needs a display_name_for_sidebar variant that is 18 characters are less." is now an outright error, rather than a quiet thing in the log. Since we are making sure that mods are updated and such anyhow, this is a good time to make sure that we don't have things like this sneaking through.
  • Fixed the two instances of this sort of thing that were previously in the codebase in DLC3.

• Fixed an issue where NaturalObject (no owner) faction was not always set to faction index 0 anymore. This was something I was already aware of, but wasn't sure if it would be a problem.
  • It turns out that this was a major problem, and would cause various ailments such as units not reacting to aggro and releasing units, to all sorts of other potentially unknown stuff. This will not fix existing 3.700 saves that had this problem, but any new saves will not have the problem.
  • Additionally, this was causing units to not respond to move commands properly between planets. This again isn't fixed in existing 3.700 saves, but any new saves no longer exhibit the problem.
  • Thanks to Vinco and Daniexpert for reporting.

• Fixed some general issues with GetControllingPlanetFaction() and similar methods where they could now return null at some points, rather than returning naturalobject when nobody owns a world.
  • Thanks to Daniexpert for reporting.

• Fixed an exception where if you tried to change the main AI color in the lobby, it would error.
  • Thanks to Magiik for reporting.

• Fixed a rather egregious and VERY longstanding bug where if an unclaimed unit (a capturable unit) was placed into a LazyLoadSquadWrapper, it would return null from that squad wrapper.
  • This was potentially never a real problem before, but we now use lazy load squad wrappers on fleet centerpieces, which meant that this bug now showed itself by deleting all capturable golems, arks, transports, and so on.
  • This bug is at least 3 years old, and who knows what it was affecting (maybe nothing, honestly), but it just had now found new expression. Glad to have that fixed.
  • Thanks to Vinco for reporting.

• Fixed exceptions that could happen in a number of places if you tried to get the wave info, because it was using the old ToString() method rather than the new efficient method that writes to a buffer without causing a lot of garbage collection.

• Fixed an exception that would happen in the AIP history window, because it was using the old ToString() method rather than the new efficient method that writes to a buffer without causing a lot of garbage collection.
  • Thanks to Daniexpert for reporting.

• Fixed a typo in the MarauderOutpostRaiderPerUnitBaseInfo that had it linked to the outguard class instead, which then led to errors. This will still be broken in old savegames, but is fixed in any new ones.
  • Thanks to StarKelp for reporting.

Youtube Deep Modding/Programming Guide

Okay, here's the modding guide for AI War 2 now: https://www.youtube.com/playlist?list=PLc2dK5ROW0WCjIHUBRJPp5oiJkTIMZupL

Modders: Do you need to watch all of the above? No, absolutely not. The timestamps do point out some interesting things that you may want to listen to, though, if a topic strikes your fancy. Even the regular code developers, like Badger, may find some of those sub topics rather edifying. You can click the timestamps in the video description, and it takes you right to where I am examining and discussing some particular piece of code. This jumps all over the place for different examples as appropriate, and does some work on AMU but not remotely the whole thing. It also shows some simple and complex faction data from the base game, and discusses both the theory behind these designs, as well as the practical uses of it. I feel like a professor trying to impart an entire textbook in a 3 hour lecture, but such is life. I will be available for questions and consultations. I want people to have a good experience with their transition to the better format, and also understand the sheer power and performance gained by doing the transition.

Beta 3.700 Bulk Of The Great Refactor

(Released September 22nd, 2021)

Known Issues

• This is a brand new version of the game in a whole lot of senses, so it is expect there are going to be a bunch of random errors that are not known. Please don't stress about it. If you just report them, most should be very quick to fix. With that said, here's the list of things we do know are not working at the moment:

• Tutorials are temporarily broken.

• Beacons for factions are temporarily removed. They will be re-implemented in a more efficient manner soon.

• Most mods are probably broken completely or partially.

• The kills and deaths data that you could get is no longer present. It was limited and stupid and boring, and I'm going to give you cool data, with graphs, instead.

• Random Factions are broken.

• All savegames prior to this version will no longer load in this new version, which is by design: we have completely reworked the savegame format to be more efficient.
  • I should warn you that I am going to break even NEW saves several more times prior to 4.0. This will be in pursuit of making the Fallen Spire and Necromancer factions feel better to play with, and to add ship loadouts (the AI War 2 version of modules) with minimal CPU overhead during gameplay, and a few other reasons like that. I will let you know when I'm done breaking saves, but until that point please consider any save as potentially invalidated by the next version coming out.

Random Features And Fixes

• Added a new is_immune_to_reconquest_waves="true", that can be used to make it so that a given faction won't have reconquest waves launched against them (other waves are fine).
  • This is also newly applied to both flavors of dyson sphere, which is new. This should let them get a lot less hassled.
  • Thanks to easytarget for inspiring this addition, and Badger for the notes on how to quickly add it.

• The "A Tale Of Two Stars" quickstart has been removed, because with the Dyson Sphere changes coming in this build, that scenario will no longer work. Likely there will be even more changes to Dyson Spheres over time, so it would probably just get a bit more busted with time.
  • Thanks to Puffin for reporting.

• For purposes of damages/penalties to damage output of weapons when under a friendly bubble forcefield, weapons now also consider themselves to be under a bubble forcefield if their parent unit is the one emitting said bubble forcefield.
  • For example, a Riot Cruiser that has a bubble forcefield and 10 guns on itself, if its bubble forcefields are up at all, its guns will work better unless it has the usual things that let it not cause that. Previously, it would only get the "under bubble forcefield" status on its weapons if some OTHER unit was providing said bubble shield.
  • Thanks to CRCGamer for reporting.

• Moved the mark level scaling data for Fleet Officers from being ZO only to also being applied to the base game to simplify balancing.
  • Additional revised offensive scaling of Fleet Officers to cap at 6x base damage instead of x7. This actually does cut a few of the more outrageously upgraded options by 50-70K DPS.

• Adjusted Combat Factory internal drone metal rate up from fixed 1000 to 1500 plus 250 per mark level.
  • Combat Factories have been pretty slow at replacing their drones for a while and unlike most of the support factories they have a much larger number they maintain.
• Adjusted Scrapper Factory internal drone metal rate to gain 250 per mark level.
  • Since this factory type essentially keeps a stock of knock-off autobombs as drones it just made sense to let it make replacements faster with higher mark levels.

• Adjusted Shieldwall Battlestation to be a much more compelling option overall.
  • It no longer gives units under its bubble shield a damage penalty.
  • Said shield also no longer shrinks when taking damage.
  • As it uses classic scaling and thus defenses only improve by up to 4x base value by Mark 7 it has been given an additional 25% base shield value.
  • A new mechanic where as long as its shield is up it takes reduced damage to said shield has been applied. Base 10% damage reduction with an additional 4% per mark level. This effectively means that at Mark 7 it has 34% damage reduction to shields.

• Adjusted Grenade Corvette strength bonus modifier slightly upwards to account for higher damage reliability with shot compression.

• Bumped up Harasser variant of MLRS Corvette base damage from 18 to 20. (Note that these ships can fire 8 shots of that damage per salvo)

• Previous adjustment downwards on target cap for Electric Bomber wasn't quite sufficient alone after shot compression made reliability higher. Further adjustment of base damage down from 900 to 750.

Quick Starts Data Overhaul

• Added a new ultra-micro format for "last lobby settings" and for quickstarts.
  • This isn't really so much for performance purposes as it is for forward-compatibility purposes.
  • I'm about to absolutely revamp the entire savegame format and how a lot of data is stored for basically every faction that exists, and I want quick starts to persist past something even that drastic.
  • Why the huge revamp of data? Primarily multiplayer, but honestly this is something that will likely yield performance benefits and added correctness in single-player, too.
  • We basically never do such huge revamps of data, because we absolutely loathe breaking savegame formats, but at the moment the time for half-measures is past. So we'll do it once, and make it really count.

• Starting to recreate all of the quickstarts. The first one recreated is now Helping Hands, which is identical in form and function to what it used to be, but now part of the micro-save format so that it will survive the upcoming purge night.

• All of the quickstarts from the base game have been updated to the new micro-save format that will be the only thing future-compatible after this build.

• Fixed a bug with Helping Hands 2 from The Spire Rises, where it was not working if you didn't also have DLC2 and DLC3 installed.
  • It is expected that there were other similar errors in some of the other quickstarts in DLC, but I am just fixing them all as I move them to the micro-save format, so I can't say for sure.

• All of the The Spire Rises quickstarts are now safely in the new micro-save format.

• All of the DLC2 quick starts are now fully recreated in the micro-save format.

Make Your Own Quickstarts!

• makequickstart, [folder name], [quickstart name]
  • Causes a new quickstart to be saved to the quickstarts folder in the new ultra-brief quickstarts format.
  • This quickstart format is required if you want to have future-proof data that does not break with future versions of the game.
  • Example usage:
    • cmd:makequickstart,1-Basic,Helping Hands
      • This would make a replacement for the basic helping hands quickstart.
  • Please note that you need to still add a .tooltip file next to any quickstart. You'll have to do that in your filesystem on your own, but it's just an ASCII text file with narrative information inside it about what the quickstart is about. Otherwise that tooltip will come up blank.
  • When you do this process, it also gives you a note telling you which dlc and mods are going to be required for someone to play that quick start. We had lots of cases in the past of accidentally requiring more than we meant to.

Bugfixes

• Improved the handling of DoForPlanetsParallel() so that it can't spam errors when you're exiting the game, or exiting to the main menu.

• Put in some extra debugging in DefinePlanetFactionDefenseTypesIfNeeded(), as it could throw exceptions in some cases.

• Fixed things up better so that any DLC can use features from any other DLC without incident, despite mods being required to have the parent DLC installed.
  • We originally had some cool chain lightning units in DLC3, which got removed after we introduced the restrictions, but we'd like for you to have those back!
  • There's also a detailed writeup on all this here: https://wiki.arcengames.com/index.php?title=AI_War_2:Features_From_DLCs

• Fixed what looked to be a bug in recent versions with outguard not actually spawning. This was based on code analysis, not testing, so let us know if something is off.

• Fixed a number of potential cross-threading issues with nomad planets using the main thread context on a background thread.
  • And some with wormhole borers, though fewer of those.

• Discovered and marked a bug where MP clients cannot cancel hacks in progress, but have not fixed it yet.

• Been meaning to do this for a while: multiplier_to_all_fleet_caps has been removed from the game.
  • It was previously on all turrets, set to 0.2. It was only there for historical savegame purposes, and messed with the unit cap math badly.
  • There are other modifiers in place about weapons strength or hull or shield health, and those are used in a variety of cases and we're leaving those intact and in place.
  • The ship cap one being removed means that if you have a mod that is seeding turrets, you'll want to remove this multiplier and then probably divide your number of seeded turrets by 5.
  • This should solve a number of longstanding but rare bugs that we keep seeing with the TSS and other turret-granters, but we'll see. It certainly won't hurt.

• Fixed typo in "ADVICE: Next Steps".
  • Thanks to Ben Parrish for reporting.

• Fixed a few grammatical errors in EncirclingSpider and IrefulMLRSCorvette descriptions.
  • Thanks to Fenrakk101 for reporting.

Rip And Tear For ExternalData

• All savegames and quickstarts older than 3.700 are now broken in the latest beta branch, and there is now a popup message explaining that any older saves you have should be finished on the older versions.
  • This is something that was not done lightly, and we're going to keep breaking beta-branch saves for at least the rest of this week as we refactor things to run in a high-performance, MP-friendly, modder-friendly, non-error-prone way. No more half measures.

• The StatisticsSet class, which was mostly unused except for some small bits of data that were stored in a way that was hard to get at, has been removed.
  • The actual data about offline achievements has not been in here for years, for example. As has the info about achievements won during a given specific save.

• Our ArcenExternalDataLookup class, ArcenExternalData class, ArcenExternalDataPatternTable class, ArcenExternalSubManagedData interface, and ArcenExternalSubSubUnmanagedData interface have all been removed. ArcenExternalDataPatternImplementationBase is also gone, and all the related stuff, really.
  • The generic "let's talk about external data attached anonymously to an object" concept that has plagued this game since our earliest versions is now going away.
  • For those who have coded factions or mods, please don't stress -- the vast bulk of your content will still work once I'm done rebuilding this in a more productive and direct way.
  • The problems with the old approach (of connecting the Core and External data parts together) were many, and reflect that it was originally kind of a prototype concept that never got refactored significantly enough because we didn't want to break savegames. I've spent the last year and a half trying half-measures to work with this stuff, but now it's time to go all the way.
    • Among problems with the old method: memory leaks, bad performance, inefficient data storage and transmission, no deliniation of data that we really consider central versus data that is just needed for host-only execution, and more.
    • There were also shifting and evolving standards in use over the years, leading to better and better code over the years, but still pulling along old bad code. Everything is getting a facelift out of this.
  • Last year and earlier this year you can see a number of cases of me trying to bring this to heel in a better fashion, but it's just crossing a line now where it's way too wild.
  • My one rule is generally "don't break legacy savegames," so you know this is serious business at this point. Time to get a lot done all at once to make this worth it.

• Old concept being removed: you can have "externaldata" which has any amount of sub-data on it, attached to various objects.
  • The precise nature of how that connection is forged is actually something that exactly zero people fully understood, and at any rate it was a very ineffective method for gathering this data. There were three levels of pooled wrappers in there, it was nasty.
  • The types of objects that could have externaldata were World, Planet, PlayerAccount (never needed that, though), Unit (this was used all the time, but then much less over time), and Faction (this is the really meaty one that tends to matter, but even this is problematic in the extreme for older factions in particular).
  • Please note that PlayerAccount is permanently losing an kind of external-data analogue, since that was never actually needed. The new network-enabled personal settings actually are much closer to what we had once thought we might do with these, and those are highly organized data that works really well.

• A LOT of code that was dedicated to maintaining old savegame compatibility has been removed.
  • This was thousands of lines of code that made the codebase that much harder to work with.
  • This alone is going to help performance for network transfers and savegame loads, given just how many versions there have been.

• Stripped out some of the excess of paranoid canaries that I had placed in game units a number of versions back. There are still some in there, but it's more efficient having fewer, and the reduced count will be sufficient for our purposes at this point.

• Previously, game units could only have a maximum of 15 systems, per their savegame format. This was very very sensible in a time before loadouts, but now this is being expanded to 63 instead.
  • This allows for us to have a great number of optional systems on a unit for loadouts. We still think 63 is an absurd number, but that's the sort of buffer zone we want to be in, really.

• Removed DoomData, which hasn't been used ever but was kept as an example bit of code for a long time.

• In our external code, right now we have some temporary classes as bases for things that will later be split off into other files and have their data style changed around.
  • Right now the goal is just getting things compiling, but having the external factions be braindead. The places where many things will change from the root up are marked with EXTERNALDATA_TODO for now.
  • It's going to be several days at minimum before we have a truly functional build again.
  • It's worth noting that a lot of giant gaps are there right now, like faction allegiances even being off once you load. This is because the config data doesn't have a pathway to the external data at the moment.
  • At the moment, however, you CAN load up the lobby and everything there is 100% functional. It doesn't actually kick off factions after the lobby is left, but that's not the point -- the point is nicely-siloed data that we understand the boundaries of.
  • At this point, after you load a quick start or click start from the lobby, it runs part of mapgen, but most of it is disabled. You then instantly win the game, and errors spam the world.
    • Here again, this is kind of the point... it's impossible for us to reliably have known prior to now what a client would or would not have on it in MP, and everything was very messy. These errors likely were the same ones that were going to show up on a client after getting a map from the host in the recent betas, but we would not have known until testing.
    • My work tomorrow is going to be centered around building in stable and clear data bridges between Core and External: this will have to be well-organized, easy to understand at glance, not prone to memory leakage, and equally as powerful as the tools that they are replacing.
    • The idea from that point on is to start getting the most-core data linked up, that stuff which is used by every faction, or by the human or AI factions in particular. And the idea will be to separate out that which is present for everyone, and that which only the host has, and that which the host automatically shares with the client as it is dirtied.
      • After that is done, and we have the basic factions running again, then it's a matter of going faction-by-faction and converting them to this new format. How much work that will be remains to be seen, but I anticipate being done with most of that this week.
      • To be clear, this should then have covered essentially all of the major things that multiplayer is currently lacking in data transmissions, and also have improved the performance of getting to said data in a single-player context as well.

• The old ShipStats information had data on how many ships of each sort you had killed and lost, by mark level. This was actually stored on every faction.
  • This is not very interesting data, and is very very hard to show. We're going to track and show much more interesting data, in a completely different way. So this is being removed.
  • This was the data that was "experimental" when you right-clicked the metal view. For the moment that's blank, but soon I'm going to have actual graphs.

• Adjusted all of the ExternalData serialization to no longer have the old data for deserialization logic. This was less to remove in the main game, but still several hundred lines of code.
  • The idea is making this cleaner and easier to read since we're in the process of breaking saves anyway, and it also makes it easier for me to move forward with splitting and cutting up the data.

• Removed some ultra old rally point code that hasn't been used since fleets were added.

• (Almost) all of our older code and folders and such which used to say "Mercenary" or "Merc" or similar have been updated to say Outguard instead.
  • This simply makes things more clear, particularly since we've been calling them both things in code for a few years now.
  • There are still a few references to mercenaries in a few files and filenames, but it's WAY lesser now, which is nice for avoiding confusion in the future in particular.

• Removed old features that had been retained only for really really old savegames:
  • ship_cap_multiplier_strikecraft, ship_cap_multiplier_turret, ship_cap_multiplier_frigate. These were all tied to the IGC in older versions.
    • There was a LOT of related code for this, which is all removed now, hooray.
  • The deprecated custom-flagships xml.
  • FCE xml and a bit of related code.
  • supercharges_strikecraft_ship_cap_of_rest_of_player_fleet and related code.

• Old logic for removing AI frigates has been removed. We no longer need to deal with those savegames.

• Removed the HostOnlyAggregatePlanning background thread, which I added a week or so ago but never wound up needing.

• Removed some FleetFormation logic and xml that had never been used and never been tested. It was an old prototype just clogging things up minorly.

Reconstructing ExternalData Links

• FactionCenterColor and FactionTrimColor have been removed as direct variables from the Faction class. So has FactionType Type and and SpecialFactionData SpecialFactionData.
  • In place of these, ConfigurationForFaction Config is now always permanently on the faction, and it has a direct link to this data.
  • Previously it was frustratingly common to have two copies of various pieces of configuration data, or even three, and we want to avoid that. Trying to keep that in sync serves no good purpose, versus having one place where each piece of data can be queried is very useful.
  • However, it's nice to have nice wrappers around all of these so that the same code will work as always have, so this is what we've done.
  • This may also fix a bug we've had for a while with the "subsequent faction colors not being set on factions of the same type" for some subfactions, but if not it at least makes that easier to fix in the future.

• The "MissingFaction" is set as the default entry for the factions table so that it will automatically be chosen in cases where a faction is, well, missing. Removes the need for some code.

• Added in some more wrapper methods for looking up custom field data for factions in a direct sense, rather than potentially getting stale data from one of the secondary places it is stored. We never want it to be the case that the UI is lying to you while having a secret separate value behind it, so this is part of that. We had accidental cases of that above-mentioned problem before.

• ArcenSimContextBase now has an abstract AmIAHostOnlySubContext on itself, and a ShouldThisMethodBeSkippedBecauseThisMachineIsClient() direct method.
  • This lays the groundwork for us removing some of the RNG divergences that were happening in multiplayer prior to now. This also serves as a guide for how we're going to be splitting our External dlls. Honestly it's making me review all the methods and make it really clear what can do what, which is pretty darn useful to have to do right in advance of that other surgery.
  • Added a new ArcenClientOrHostSimContextCore, and ArcenSimContext and ArcenSimContextBase are getting a TON of additions and changes. Way too many to note the details of in this format.
  • Added a new ArcenHostOnlySimContext that is for the host-only sections of the game.
  • QueueLogJournalEntryToSidebar and QueueChatMessageOrCommand no longer require contexts to be passed to them (they were not using it anyhow).
  • Lots of methods now either require a ArcenHostOnlySimContext, or a ArcenClientOrHostSimContextCore.
    • It is very rare to now require a root ArcenSimContext on a primary logic method, and this is part of how we're now controlling what methods run on the client or host (which we were already doing, but more haphazardly), AND controlling which random generator is used (this is the bit that is new, which will help stop that "units go over here and then zip back over on the MP client" bug).
    • That said, ArcenSimContext is absolutely still used on all the various worker methods that are called from a variety of contexts. Aka some code that might be used in any and all shared circumstances.
    • To make things even more clear, and prevent un-examined code from upgrading into this without examination, ArcenSimContext is now called ArcenSimContextAnyStatus to denote we really mean that.
  • ILongRangePlanningContext is now ILongRangePlanningHostContext.
  • ArcenSimPlanningContext is now ArcenSimPlanningCommonCoreContext_DoNotReferenceDirectly. It should never be used directly anymore, as it's ambiguous as to whether it's on the client or the host.
    • There are now intermediate ArcenHostOnlySimPlanningContext and ArcenClientOrHostSimPlanningContext classes that inherit from it. Mostly this is a formality, but it does add clarity and keeps consistency with the rest of the code.
  • A metric ton of code has been adjusted for all of this, in some cases removing things that were passing around some form of Context that had no business doing that. These are meant to stay in some reasonable sync, and sometimes even some UI choices could cause problems.
  • Because of the way inheritance works from classes in C# (single inheritance only, interfaces aside), there are a few methods that are duplicated across a couple of central classes.
  • THAT said, this leads to less duplicated code elsewhere in end-implementation classes, AND the inheritance tree of the "context" types makes it so that people literally can't call methods from improper spots.
    • If you're a modder and you find yourself trying to work around a method call that doesn't have a compatible context type, then either you just need to call yourCurrentContext.GetHostOnlyContext() and try that, or else give up and do something else because this would be a MP sync issue or other data invalidation issues would be caused. Or let me know, and I can always look into making something more permissive. But honestly other than people forgetting they need to call .GetHostOnlyContext(), I don't expect any troubles.
  • So what's up with calling .GetHostOnlyContext()? Isn't that kind of a waste of end-coder time? Actually... no! It's super important. The host-only context is used for parts of the code that only run on the host, and it has its own separate RNG that the shared clientAndHost context does not have. This means that all of what runs on the client and host has the same number of RNG calls, yielding a deterministic result. But peppered in there are some host-only calls, which previously were breaking sync but now just require a call to .GetHostOnlyContext() and sync is preserved while still keeping the same code structure you're used to.
    • This is round one of several rounds of fixes that could also be described as idiot-proofing (where all of us are equally idiots, and not meaning that in a pejorative way). The code is now demonstrably clear about when it cares what sort of caller is around, which is a big step forward for the next round of this, which will involve splitting things between dlls in order to separate out a new client-accessible layer of data from lots of host-only code that runs against it.
      • Put another way, this is getting around the problem of having to replace tooltip appenders with calls to the host, or doing all of the notification calculation on the host and then transmitting lots of complicated data. We're instead handling this in a much more tri-layered fashion, based in general on the MCV (Model-View-Controller) design pattern.

• DoOnSelfBuildingCompleteLogic has been moved from the scenario to the faction class, so that it's able to properly do its job. Before, this would have been a mighty big annoyance for any modders who made a new faction.
  • Same deal with DoOnInternalConstructionCompleteLogic, although nothing actually uses this one yet.

• The "ExternalCode" project, which is the main open-source AI meat of AI War 2, has been split into two parts.
  • One part remains in the existing dll, which is the most "permissive" part.
    • Anything in this dll is stuff that we're intending the client in multiplayer to have access to. (That is a statement of design intent, but I have a lot of work to do before that part is actually fulfilled.
    • Also, anything in this part can be freely examined by the UI, on the host or the client. I mean, thread safety aside, the UI is allowed to know about these things, and these things exist on every computer in multiplayer (again, talking about design intent, not current status).
  • The new part is called "AIWarExternalDeepProcessingCode," and is meant to be only things that the host has access to. Also, the UI cannot directly query these things.
    • Right now we have moved the faction processing logic, "on death effect" code, the main mapgen code, the "hacking is happening" code, and the "long term planning" code in general.
    • Please do bear in mind! Right now there's data mixed up in many of the files that were moved, along with the logic that was moved. Some of this data is meant to be host-only, but a lot of it is not. The next job will be splitting up all of these files and restructuring how the data is found, and that goes hand in hand with also re-teaching the Core code from AI War 2 how to reach these new split-up forms of data and code.
  • Please note that all mods will be required to follow this same double-dll format, which we will be enforcing by a quick scan of your mod code as it is first loaded. More on this will be available when this conversion process is complete.
    • The goal with this structure is to make things multiplayer-safe, and separate the data from the code a bit better, and separate the host-only code a bit better from "everybody runs it" code.

• Updated our build scripts so that when we build an external project from within visual studio, it will now properly set them into ReliableDLLStorage. Prior to this point, any compile-breaking changes needed to first be copied there by running the entire command-line build script.

• My external code loading framework is being put to new use in terms of helping ensure data-correctness.
  • For classes that will go into deep processing or the client-safe data processing, they will all inherit from either IExternalDataAndSimCode or IExternalDeepProccessingCode.
  • The game now checks to make sure two of these same things are never in the same dll, and additionally, that no dlls that have deep processing code are referenced by any classes with DataAndSimCode in them.
  • This is really important for preventing errors that would affect multiplayer clients, among a few other maluses, and so by doing this very quick check on startup we can make sure that all of our own code, as well as any mods, all conform to the standard that prevents the most common MP-only issues from happening.
  • Put another way: if you've just coded a mod, and it works for single player, you want to be pretty darn sure that it "just works" in multiplayer without having to go test it there. It's the same with any changes we make to the core game, or in DLC. All of this code restructuring, along with these checks, are how I ensure that is the case at a very low level. If MP has a problem that is unique to it, I want that to be something that's my central problem to solve, not something that faction designers or modders are chasing on their own stuff.

Restructuring The Upwards Link From Core To External, Part 1

• Previously, we had a thing called "ExternalDataPattern," and some of the DLC content had to be included in the base game or else beacon factions would mess up the ability to load quickstarts. And last-settings would get messed up if you disabled a DLC and went back into the lobby, too.
  • Thankfully, now we don't have this problem at all because of the new "micro save" format that is now used for those two purposes as of this build. Therefore, the new equivalents of this old sort of data are now able to be stored in each DLC properly (this has the same implications for mods, so this is a good thing for mods being usable for sure).

• Added the following two sets of data/code pairs:
  • ExternalDataAndSimCode, which is for clients to know about and the ui to be able to see and use.
  • ExternalDeepProccessingCode, which is only for hosts and single-player, and contains all of the deeper logic and working variables and whatever else. We don't want this to accidentally get sent to a client at any time.
  • It is worth noting that the above things can have 1:1 pairs (faction data in the sim code that the ui can also see, and then deep processing for said faction), but in many other cases we don't need a faction to be involved at all, or have several sub-parts to a faction (unit data, etc), or whatever other designs we want.
  • As part of the two above sets of data/code pairs, added the following three categories:
    • World. This is something attached to the world, and it can run whatever logic you want to, kind of like a faction would have a chance to but without the faction.
      • Internally, some of the logic that we have very confusingly on "Human" right now -- there can be many human factions, and we generally want to run this stuff once for all of them -- will move here. This is also for the things that are human-faction-type agnostic.
      • Externally in mods, we expect to see the FakeExecutorFaction from AMU move into this. We should never need to have a fake faction again, as there is now a clean and fully supported method for doing that same sort of thing here.
    • Faction. This is where the meat of the game logic lives for anything that isn't just the autonomic style of unit AI.
      • Internally and externally there will be some swapping back and forth of what is loaded where, and the SpecialFactionLogic class in general will be subsumed into this.
    • Squad. This is data that goes on an actual individual ship/structure for whatever reason. Some factions are designed where this is needed, others are not.
      • Having deep processing available for individual ships is now possible, although that would be a little bit on the strange side. We have a way of handling it with efficiency, but it's an oddity for sure if you structure it that way.
  • ExternalDataPattern used to be loaded very early on in the load process, and these new ones are loaded only slightly later. Other xml will be referring to these.
  • So far, each of these is just blank slates, but they'll all have unique methods on them, and be the vessel through which most communication happens between Core and External and ExternalDeepProcessing.
    • It is worth noting that these all implement time-based pools that are incredibly direct and also centrally handled, so we can avoid the memory weak worry that was everpresent with MP clients in the past.
  • Oh, another thing to note: all of these are individual object data holders that can also contain code. None of them are singletons.
    • So for instance, if a faction defines a FactionDataA class that goes with itself, then if there are 3 instances of that faction TYPE, there will also be three instances of this faction data. This is very useful, because you can therefore store faction-specific working data in the deep processing in particular, or use static variables if you want to share across factions. This is very similar to the ethos behind ISpecialFactionImplementation, which will be leaving us before too long.

• Experimented with a new method of serializing data for objects. The data is really really sparse, though, because of "all the things that can be on a ship," for example, usually only one or two actually are. It's much the same for faction and even world.
  • I spent a goodly while coming up with wrapper classes to manage arrays based on the length of how many external linkages there are of any given type (which is based on how many DLC and/or mods you have installed), and was going to have some sort of very high limit (so that if you run with 100 mods on, someday when there actually are 100 mods), you don't have a problem.
    • However, that has many many drawbacks, because we wind up iterating a ton of arrays that are almost universally empty, or have a single item or two in there. And there is a finite limit on the number of mods you could have, even if it is very high.
    • The other big drawback is just how long it will take to iterate all of that, and for squads we have issues of how timely we can get those data updates from the host to a client after there's some change. Also, we have some order of operations questions with things like fast blast data potentially arriving before the external data, or vice-versa. Either way, it's not a good model.
  • The solution will to take this "vertical" approach and make it horizontal, and instead serialize on-the-fly in a more centralized fashion. This keeps our amount of iteration very very low. The downside is that I then have to make it very fast to link back and forth between the data and the thing it is attached to, but that's not the worst problem in the world. I was already potentially going to have to find a row index and then index into an array and then cast to the proper type. Instead we're likely looking at a concurrent dictionary lookup, which is at least a single operation rather than multiple (admittedly cheaper) ones.
  • Well. At any rate, my code for the old vertical method is not complete and does not compile (I was in the middle of it when I had this realization). I'm checking it in in case it is ever needed in the future, though. Next step will be to convert it to the horizontal format.

• Some rather mind-bending code for horizontally serializing and deserializing all of the external data for any world data, faction data, and squad data, all at the central faction level, has now been put in place.
  • This differs from a lot of our serialization code in that it's very... reactive... to what it finds. Essentially what happens is it iterates over every possible piece of data that can be sent (hey, macrophage ships can sometimes have data attached -- are there any in this game? If so, send those that are there). It's clearing out stale data as it goes, and as-before it gives the details of the serialization itself to the actual External code classes.
  • One of the chief differences this time around is that we don't have to serialize any table or row names -- we are using central indices that are synced from the client to the host -- and we're also using smaller ranges of integers for the PKIDs for factions and squads based on the differential between the largest and smallest ID in the particular list at any given time.
  • This will require testing once we can run the game again, to make sure that I didn't make any typos, but it uses a lot of generics and type inheritance in order to minimize code, so whatever problems are there should be of the "fix once, don't think about it again" variety.
  • At the moment, what is in the codebase is kind of impressive, but also kind of baffling. Here we have six identical tables and classes, all serialized using some very complicated code that is very centralized... but there's nothing different about any of these classes or tables. They're all super identical.
  • Where the magic really is going to come from is the fact that we can have all manner of different methods on each of those distinct six classes, and so we can set, retrieve, and manipulate data without having to do any type casting at all.
  • This a much more elegant solution than the old solution we had, but it strays slightly close to "clever code" (the sort of thing that is showing off for no reason, and thus is super hard to maintain and nobody else can read it).
    • That said, the reasons for this solutions design are pretty clear: keeping the guard rails on, not mis-identifying data (often happened before during load issues), minimal steps to get from Core to External, the ability to have lots and lots of very direct methods that we can can directly call, and pooling that is NOT in the hands of the External programmers, but instead is controlled by Core. There's also a goal for having substantially less code, and substantially clearer code, on the External side. That will start becoming evident tomorrow as I start wiring this up to them. On the Core side it takes some extra complexity in order to create more simplicity on the External side.
  • That's the idea, anyhow. I have long considered myself a code architect in terms of programming sub-specialties, and I've done a few dozen refactors of major programs, ranging between languages and business software and games. This one has so far been, to my surprise, the most complicated one I've had to do. There's a lot of pleasure in that, but at the same time: wow complex.
  • The main good news from today is that there is really a small amount of code (all things considered) where I could have made mistakes. So once I start this up, after connecting to the External code where I CAN start this up, any errors should make themselves apparent immediately and then be gone forever.
    • The old way of doing things was something I was never able to wrap my brain completely around because it had three layers of indirection that also required bidirectional communication and reading of the code. In other words, to understand that code, you had to read the entire path of it from Universal, through Core, all the way to External, and then follow a further chain back down the other direction. That was... code that worked, but was very very hard to be sure was not secretly broken. It had some arrays of objects in the middle, hitting the GC in a way I didn't like, and it was particularly hard to tell what it was doing o =n MP clients.
    • But the new code? It flows one direction, you can follow it clearly, there's no GC hits in the middle, and despite the code being a little bit odd syntactically (generics really can look odd), it's fairly straightforward to follow.
  • Tomorrow will be the real test, where I start wiring up the various pieces of External code to these new pieces, get them compiling again, and then eventually transform it all into the simplest possible version of itself that will be as error-free as possible in MP and SP as we want. It won't all happen in one go, likely, and I expect several more major refactoring passes next week to clear up various perennial bugbears like linked factions, the strange combined fleets/cities of the fallen spire, the odd (and not architecturally wise) way we have "faction common data" and similar structural oddities that make some of the externaldata have a larger learning curve than we'd prefer. Hopefully we have a working SP game this weekend, and hopefully we have a refactor-complete game by next weekend. We'll see how quickly things move tomorrow.

Restructuring The Upwards Link From Core To External, Part 2

• Okay, all that "clever" code yesterday? I said in my release notes that it was pushing the edge of being too clever for its own good (when programmers of a certain experience start talking about how "clever" code is, it's not a compliment -- if a professor tells you your code is clever, that's a compliment, but if a colleague does, they are worried about ever having to maintain it. Smart and cool and elegant are always compliments.).
  • The code was indeed the simplest possible implementation with the least amount of code for doing the sort of many-to-many relationships that I was assuming some of our data might have. But thankfully, no actual data is structured that way.
  • The old code that we used to have was allowing for many-to-many relationships between many kinds of objects and data, but thankfully everyone was sensible in both internal and mod code, and instead what we wound up with was a lot of one-to-one code. THIS ship has THAT data and no other, etc.
  • So! After talking to the folks who have been working on the bigger mods and the external factions and verifying that my sudden suspicion was correct, that means it was time to strip out pretty much most of the work I did yesterday (sigh, but it was a good intellectual exercise I suppose) and implement something new.
  • This new approach treats all of the World data as being singletons (aka only one copy of that external object, ever, period), and it even queries them "hey, you want in on this?" to which they can contextually say yes or no.
    • This is super duper useful, because you could actually have some sort of helper code turn itself on for a while to manage some process in the game, and then relieve itself of duty and stop using any bandwidth or CPU cycles. Or it could start out as off, but turn itself on once some situation happens, etc. Handy!
    • In general, one of the larger problems with external data assignments like this is "when do they get attached, and how do we know to do it?" In the past, a lot of that was "when ships are created in externaldata, we'll assign it contextually then, and hope we miss no cases. If found we did, then backfill." We may keep to that, but will also try to formalize it a bit. But for the world, simply asking the objects "you want in, or you good sitting out for the moment?" is feasible as well as most performant in this instance.
  • Factions and squads are still pooled, and so a lot of what was done yesterday (the easy parts) still apply for these.

• Map generation code is now flagged as deep processing, so it will be disallowed from being in any dll with the base data and sim code.

• ISpecialFactionImplementation, which has for so many years been the absolute centerpiece of faction and AI code, has been removed from the game.
  • It's functions are now being split between base-data-and-sim faction code, and deep-processing faction code objects, which has been one of the chief aims from the start of this rework.
  • There is no longer an "Implementation" object on the faction at all. Instead, there is now a "BaseInfo" and a "DeepInfo."
    • These objects serialize data to disk (like externaldata used to), as well as contain whatever logic we want them to. That's not quite an MVC philosophy, but it's following a similar and useful OOP design.
      • It's worth noting that the deep info ONLY ever serializes to disk, and never across the network to MP clients. It's also worth noting that the UI should not be trying to get any data out of the deep data unless it's purely for literal debugging purposes.

• For the first time, squads themselves now also have a BaseInfo and a DeepInfo, which mirrors what you're used to with the factions.
  • This doesn't replace anything on the "big logic processor side," but will probably fold in things like "description appenders" later. However, this replaces the notion of ExternalData that used to exist.
  • It's worth noting that this external data is no longer a collection of things, unlike before. There's only a single item attached to each squad, or zero items attached to it (if it's not some special faction other than humans or the AI, it's probably zero items).
  • Before we had a many-to-many relationship, but that was hyper inefficient and something you are better off handling with data at the faction or even world. There are no known cases of any internal or mod code actually needing that, though.

• Added ParallelDoForAllActiveEntries() onto TimeBasedPool, which gives us a comparably-quick way of doing some action VERY completely against all live objects in that pool.
  • Quick is a relative term in this one, but it is multithreaded, at least. This shouldn't be used in most cases for during-game things, but rather is for times like "when we reload all the xml" or things like that.
  • We also added a ParallelDoForAllEntriesEverEvenInactive, since that actually seemed kind of warranted for the xml case.

• GameProgramIsAboutToReloadExternalXmlAndSimilar() is now a thing that exists on all the external objects, rather than being limited to the faction data.
  • This is mainly as a courtesy to mods that may have different structures from our code. We don't actually use it, but it's useful as a thing that can be called for those that do.

• ExternalDataExports is now functional again, and lets us dump out all of the new-style external data.

• Everything that was called "ExternalXDataAndSimCode" is now called "ExternalXBaseInfo"
  • This is a lot easier to type, and is way easier to read at a glance.

• Everything that was called "ExternalXDeepProccessingCode" is now called "ExternalXDeepInfo"
  • Again easier to type and read.

• There was also kind of an awkward convention between the tables and rows describing the external data, and the external data itself.
  • That has now been clarified into the tables and rows being called "InfoSources", while the actual things themselves are "Info." Logical and straightforward at a glance.

• Faction xml (aka from the SpecialFaction folder) no longer wants or accepts a dll_name and type_name.
  • Instead, it now requires a base_info_source (referring to an entry from ExternalFactionBaseInfoSourceTable).
  • And then also a deep_info_source (that refers to ExternalFactionDeepInfoSourceTable).

• Squad xml now has the OPTION of specifying base_info_source or deep_info_source as a way of automatically attaching these types of external data to all entities of the type in question.
  • This is actually a really clean way to handle things, and then the data can be populated later by the external code as desired. Nobody HAS to use this, but it can be a nice thing.
  • Do note that this IS inherited via copy_from, so if you are copying a ship that happens to have base_info_source set, then on any that are being copied for use without that, you need to set base_info_source="None" on any direct children to blank that back out.

• Actually, it looks like the DescriptionAppenders and the AlternativeMoveOrderHandlers, and the EntitySpecialPerSecondLogic all can just stay as they are.
  • They are being flagged as IExternalBaseInfo so that they have to go into that dll rather than the deep dll (otherwise these would all break on MP clients), but that's about it.
  • These are a great example of something that can just stay flexible (no need for a rigid structure here), but now have a proper place to live that also guarantees what data access they have access to. Previously, these things would maybe work or maybe not work on MP clients, depending on the status of things. But generally trending towards not working. Now they can be verified to function just by testing them in single player, since they're in the dll where clients also get access to all the info.

• At this point, the Core dll has had all of the changes made to it that are required in order to handle the upwards-facing communications that the Core dll has with External code.
  • This means that the work of actually translating over the External dlls can now begin, and some things can be redone in passes, but not all things have to be.

Restructuring External Base Info, Part 1

• Started the process of transitioning some data from the Deep dll into the Base dll, with the idea being to separate out the running of things from the actual viewing of data related to those things.
  • Also, of course, wanting to isolate the sort of background and working data off in Deep away from the UI and any temptation to try to show that data, which would be explosive on the MP client and just a bad idea on single player or the host.
  • Moving over HunterFleetBudgetType and its related stuff is straightforward enough, and matches with the sort of thing I have in mind. That's just a copy-paste job, and hey I'll make new files for these for easier location of them in the future.
  • Then I turned my attention to HunterFleetType, and immediately hit a hard stop: it has something on it called IHunterFleetTypeImplementation Implementation, which defines both data (Base type) and logic (Deep type).
    • This is a bit of a conundrum, but not a crisis. Ultimately this mirrors the same sort of setup that I have going on in Core, and I'll need to split these things out in some fashion.
    • Right now my brain is feeling quite tired, so maybe I'll wait a bit on that problem, but essentially this is where we get to the "dream within a dream" level of inception, and this is actually more important than it might look on the surface.
      • First of all, modders and general faction designers need to be able to create new data tables in a subordinate fashion like this, and there needs to be a really central way of handling it. I can't idiot-proof it like I can with things coming out of Core, specifically because as the "dreamer" of the first dream, the modder/faction-coder has full control over how they structure their inner dream within a dream. That's both neccessary, and something I don't want to take away.
      • This actually does get at the generalized need to formalize that as much as I can, however. I can't make it idiot-proof, but I can give the proper tools to standardize dreams within dreams (going with that metaphor), AND also to make sure that they serialize in a way that does not cause a memory leak on MP clients. It's this last part that is the most challenging, and if I could make that one part idiot-proof, that would make me very happy. That's the one bit of power (accidental memory leaks) I'd like to take away from the dreamers in the top level dream.
      • Further compounding this is that I might actually need to support more depth, aka dreams within dreams within dreams -- three levels, or even four, or some arbitrary recursive number. I sure hope no-one has structured their data like that, but I guess we'll be finding out. First, though, I'm going to take a break and focus on some other compiler issues that need to get sorted out anyway.
  • Overall when it comes to looking at this, I have factions and squads which presently have a very similar -- but slightly different way of serializing. The faction MUST always have both Base and Deep info, and it serializes with that in mind. The squad may or may not have Base or Deep in any combination or none at all, and it serializes with THAT in mind.
    • The world, meanwhile, can have a great many base and deep infos attached to itself, but they are all singletons and are also all loosely, voluntarily attached.
    • The question will mainly be if I can formalize these relationships in a generic and encapsulated format that can be attached to arbitrary objects. The huntertype seems to be a singleton processor-type class, and the other classes that I've run into are similar.

• Added a new ArcenLongTermIntermittentPlanningContextBase in the BaseInfo dll.
  • This is going to be used in some of our "dream within a dream" data management cases.

• Fireteams have been partially moved into the BaseInfo dll, but parts of their code is clearly DeepInfo code.
  • That has been commented-out for now, and will be grafted back on in the DeepInfo dll once we can compile BaseInfo again.
  • Additionally, FireteamUtility has been split into one part that is FireteamBaseUtility that can run anywhere, and then the other part that stays behind in DeepInfo.

• In Universal, split up the interfaces that contribute to the external BaseInfo and DeepInfo so that we can have ones that are singletons and ones that are multiples properly denoted.

• All thirty-seven "Decription appenders" have been moved from their files in the DeepInfo dll into their own individual files inside the BaseInfo dll.
  • There were two old description appenders (hacker and science) that were not used and thus have been removed.

• All of the IAlternativeMoveOrderHandlers have been moved from their files in the DeepInfo dll into their own individual files inside the BaseInfo dll. There was only one.
  • In recent versions of multiplayer, clients were not going to be able to order spire relics around. Nice that we're doing this overhaul, as that was not on my radar and would have been a huge pain...

• FrenzyFleets from the nanocaust have been removed, as fireteams have far far replaced those.

• FactionUtilityMethods has been partially moved into the BaseInfo dll, and it is now a singleton rather than a static class.
  • Thanks to it being a singleton, this means that we can also add extension methods in the DeepInfo dll and not have to worry about having two classes with similar names that we wonder which item is on which.

• The "SpecialForces" faction has been fully renamed to Warden in the code. It was old legacy naming from 2018 or prior, and getting rid of that makes things a lot more readable.

• EntitySimLogic has been split into EntitySimLogicBaseInfo and EntitySimLogicDeepInfo.

• Moved over the bulk of the data that should be in BaseInfo rather than in DeepInfo.
  • There's still more that needs to come over, but it's part of larger classes that will need to be split.
  • At the moment, I've still got a lot of things commented out or using fake connections to mimic the old style. It's not functional, but it's hitting the point where I can compile again and thus move towards getting it working.

• A variety of new methods have been added to FactionUtilityMethods to make checking for faction types easier and more reliable.

• Current status: lots of things are moved around to be in a better spot, but a lot more work needs to be done.
  • There's still a lot of temporary code linking to the old style of doing things so that we can compile the main ExternalData dll (the base info one). That actually does compile now! But it's not linked up yet, and won't function yet.
  • The new deep info dll doesn't compile yet, and will require a lot more splitting of data into multiple classes before it does.
  • There are still some things that need to move one direction or the other, and then other things that need to be moved around or restructured so that data is grouped properly and can be accessed properly.
  • This is... quite a bit more tangled-up than I had expected, to be honest. But this is a really super good thing to be doing, and if this past week is any guide, it should be less than a week of work left to get this functional again.

Reconnecting External BaseInfo to Core, Part 1

• There was a "does nothing, but helps things compile" set of code that we had in the ExternalData (BaseInfo) dll since making the intial rip and tear changes to ExternalData as a whole.
  • This code has now been removed, which creates hundreds of errors but allows us to start replacing those hooks with new ones. Doing a massive code shift like this is an iterative process, and sometimes you just need "somewhere to plug things in for a bit so you can work on another part," if you think of it like working on a physical piece of machinery.

• The base and deep infos on factions are automatically created just like those ISpecialFactionImplemenations used to be, so that's not something that you ever have to worry about creating as an end programmer/modder in this game.
  • Similarly, the ones for the world are created as singletons and are never really attached to the world. The world just asks your code "hey, you can see what's up, do you want to run?" And if you say yes, then the world will run your code and serialize it, etc.
  • On the other hand, squads are a bit different. Historically we have added things to them as we went, as we wanted to, but that doesn't mean we can't centralize some of it. There are two modes of operation for the squad, therefore:
    • If you specify the xml for it, then it will automatically add them when the squad is created. This is not a bad idea! But this does assume that every unit of that type EVER will have this external data, and there are potentially cases where that is not true.
    • For those sorts of conditional cases, you can instead manually call CreateExternalBaseInfo() or CreateExternalDeepInfo() on a squad, and pass in the type of data you want it to initialize for you.
      • If this is already initialized to something else, then it will throw an error, since that is in fact an error and you need to take a look at what's going on.
      • If, however, this is already initialized to the same type you just asked for, then it just returns the existing item it has. Basically if there are cases where we might accidentally call the same method here more than once with the same code on it, then this doesn't hurt anything with the extra calls.

• Okay, here's a description of the pattern for changing things over from the old format to the new. This is mainly for modders to know, but also anyone working on the codebase it's useful to see.
  • We previously stored Wormhole Invasion Data (which goes on a squad) like this:
    • WormholeInvasionDataExternal was the outer wrapper that inherited from ArcenExternalDataPatternImplementationBase_Squad.
      • Structurally, this was clunky code, and very rarely had actual data of its own in there (in terms of general ExternalData for whatever function). There's no real analogue for this anymore in the new system.
      • IF you were doing the extremely old-school thing (for our codebase) of storing multiple sub-objects in here, you will have to choose one to become your new primary BaseInfo, and all the others will be sub-objects of that. Easy-peasy.
    • ExtensionMethodsFor_WormholeInvasionDataExternal was an optional (but always-used) convention that made it easier to get the wormhole invasion data sub-ojects from externaldata.
      • Structurally this was also a bit of a pain, even though it wasn't much code. A small typo, and dealing with the annoying patternindex (which no longer exists) could really foul things up. We can safely remove this, also.
      • If you really want some extension methods for THAT easy of access to BaseInfo or DeepInfo in the new system... then honestly instead maybe think about just how frequently you're calling such methods, and instead call it fewer times and pass the result around. Player CPUs will thank you.
    • Finally we come to the actual meat, which in this case was WormholeInvasionData, which inherited from ArcenExternalSubManagedData (which no longer exists). This is what will become BaseInfo.
      • The reason that this is BaseInfo instead of DeepInfo is that this is information that both the client and the host might like to see, in terms of it affecting the UI for both of them. In the event that there are some parts that are truly host-only, and never get shown on the ui except for maybe debugging purposes, then consider moving it to DeepInfo in a separate class IF it's substantial enough data to actually care about (why tell the client about things it never cares about?).
      • This class is now going to inherit from ExternalSquadBaseInfo, since this is what it is (base info, squad-attached). Let's talk about what THAT means separately, in a moment.
  • In terms of translating things over, for this type of data this is all you need to know from a bird's view.

• So now let's talk about details of what it means to handle this ExternalSquadBaseInfo, starting with WormholeInvasionData -- which let's rename to WormholeInvasionBaseInfo.
  • Trust me, it's worth it to rename this just to save yourself confusion later. Remember to do a global search and replace from old name to new name in both the BaseInfo and DeepInfo dlls (the DeepInfo almost always acts on BaseInfo).
  • Now that we have this class inheriting from ExternalSquadBaseInfo instead of ArcenExternalSubManagedData, there are some immediate things to change. (This will be identical for DeepInfo, and for faction and world data.)
    • DeserializeIntoSelf and SerializeTo both no longer are passed a SerializationCommandType SerializationType, so you can just remove those and those methods will then be happy. You can probably just run a global replace in your BaseInfo dll for ", SerializationCommandType SerializationType" and replace it with nothing and it will save you time (we had 226 instances that were replaced in our own innternal BaseInfo code).
    • We need to implement the base method Cleanup, which gets called whenever an object is being pulled back out of the pool.
      • This may or may not be important to actually fill, depending on your class. If you were previously setting some values in the constructor, and your code always expects that to be the starting state, then you should probably move that initialization code to Cleanup and have your constructor also call Cleanup(). You'll note that this is what we've done with WormholeInvasionBaseInfo.
  • At this point we have translated things over such that they CAN connect up, but there isn't an actual linkage being created yet. It's time to address that.

• Since we did a global search and replace in both projects, we can search for WormholeInvasionBaseInfo in both the BaseInfo and DeepInfo dlls.
  • In our case, there are no other references to this in BaseInfo -- that's just where the data is stored. (This is temporary and wrong, actually! We're going to move the notifiers over to BaseInfo to save processing and data, so later that will actually be there).
  • At any rate, there are two ways we might want to use this data now. We might want to get a reference to it, to change it or just write about it.
    • Previously that would have been done with the GetWormholeInvasionBaseInfo() extension method, or something similar.
    • Now this is done by calling GetExternalBaseInfoAs<WormholeInvasionBaseInfo>() on the squad, which will give you back a WormholeInvasionBaseInfo that already exists or give you an error. For DeepInfo, you can call GetExternalDeepInfoAs<YourTypeHere>(), and it works the same.
    • It is worth noting that if the base or deep info here is null, then these will just return null, so your end code should be prepared to handle that if you think that might be a possibility. In the past, most of the time our code just returned a new instance of the data type, which... well, that could have hid a LOT of code errors. We have no way of knowing. If someone calls these things on the wrong kinds of unit/squad/structure, we actually do want this to error, because that is almost certainly a logic error that then would lead to strange behavior. It's possible we will find some "new" bugs that are actually existing ones that had been hiding, in the coming weeks. This in turn would likely fix a lot of other rare bugs that were downstream from this problem, if that's actually something that happens.
  • In the old code, after changing some data, you'd MAYBE have to call SetWormholeInvasionBaseInfo(). There is no longer any such analogue.
    • Technically in the old code, this was not required for this particular class. But if you were using certain kinds of data, in certain situations, it was VERY much required. So end-coders wound up just calling it uselessly after making any changes in general, and thankfully now that's not a thing anymore.
  • Lastly (though technically firstly), there's the creation of this data. This would, presumably, happen when the squad itself is actually created, and in practice that is what the code does.
    • This can be handled two ways.
      • Option 1: You can set it up so that all of the units with the "ExogalacticWormhole" tag have this data linked via the new xml tabs, since that's apparently the criteria we're using.
        • Pros: the xml will be nice and clear about what is linked, and there's no code path where you might accidentally forget to add it.
        • Cons: properly transferring the existing code to hit all the proper entities is slightly tedious, and also introduces the possiblity, of errors.
      • Option 2: You can take what was a call to GetWormholeInvasionBaseInfo( ExternalDataRetrieval.CreateIfNotFound ) and turn that into a call to CreateExternalBaseInfo<WormholeInvasionBaseInfo>( "WormholeInvasionBaseInfo" ).
        • Pros: this keeps the same structure we had before, so there's fewer errors of short term errors in conversion. Requires no xml work. Can be used to add BaseInfo data onto units in a conditional manner -- for instance, usually just when they belong to a specific faction.
        • Cons: in general, if you forget to do this on all possible code paths that are expected to lead to the result you want, then the game is going to have new errors popping up where it used to not have that. In the vast majority of internal cases, there's only one central method that creates these things in the first place, so that's usually less of a risk, but sometimes a consideration.
    • Before anyone asks: yes, you CAN handle it "both ways." It won't complain if it already has the type you're asking to create. It will just return the type that is already there.
    • Also worth a thought: there isn't any sort of initialization callback on these right now, so if there's linkages you need to make on setup, you're going to need to at the very least call the GetExternalBaseInfoAs<WormholeInvasionBaseInfo>() method and then assign some stuff. So this again creates as situation where you must be sure to always catch every possible code path that could generate this sort of entity, though we may work on solving that in a couple of ways in the very near future. Probably before this build is out.
  • Okay, now that the process is outlined, it's time for me to repeat this another few hundred times on the main game code.

• Previously, there was some information stored on every faction. To get at this data, you would have to call GetCommonExternal() on the faction object.
  • This data has moved onto the BaseInfo object for factions in general, in Core.
  • There is a new IPlanetPathfinder in Core, which lets us handle a lot of this centrally now.

• Made some updates to methods in FactionUtilityMethods to use a more efficient way of finding the highest AI difficulty in various situations.
  • Also got rid of some duplicative methods of this from MapGen_Base.

• SpecialFaction_Human had a lot of random stuff on it that is moving elsewhere:
  • The audio cue suppression stuff, which was complex but is only really used for the king unit being damaged, has been moved to nonsim in the base game.
    • This is local data for a client or a host, and we don't use this in such a way that it warrants nearly the intensity it is generating.
    • Similarly, the data from HostOnlyAudioTracking has moved onto the planet itself and is no longer host-only (that was an error on my part a week or two ago).
    • The function is identical to before, but it's now far less code.
  • GetMaxPossibleStrengthOfAllHumanFlagships() has moved to a new StrengthHelper class in Core.
  • GetPathingModeForLocalPlayer() has moved to a new PathingHelper in External (BaseInfo).
    • Similarly, a bunch of stuff like InnerFindPath_Raw(), InnerFindPath_RawSinglePathfinder(), FindPathFreshOrFromCache(), and FindPathFreshOrFromCache_ShortTermPlanningOnly() moved to that PathingHelper out of BaseFaction.

• Also previously, there used to be a ExternalData_MinorFactionCommon.Primitives, which you could get from most factions via calling Ex_MinorFactionCommon_GetPrimitives().
  • This is now gone as well, with SOME of its data wrapped into the BaseInfo root just like what we did with the CommonExternal stuff.
  • The rest of this data is currently sitting (as of this writing) in a Temp.Historical class, and will have to be properly distributed to various classes that actually need this information. Right now there's a lot of junk in there that is not used at all, and a bunch of stuff that is faction-specific in a way that doesn't need to be central (the HRF intensity is not needed on every faction in the game, etc).
  • One thing that is moving is the Intensity being directly on any faction. This should instead be per-faction where needed (AIs don't use this, they use AIDifficulty, same with Wardens and Hunter and PG, for instance, and many others use nothing at all).
    • Instead, when trying to quickly get the intensity for a faction in general, which before was Ex_MinorFactionCommon_GetPrimitives( ExternalDataRetrieval.CreateIfNotFound ).Intensity, the code that should be called instead is BaseInfo.GetDifficultyOrdinal_OrNegativeOneIfNotRelevant(). This was by far the most common usage of primitives.
  • Actually, for the moment nothing has moved data-wise from here! The Allegiance field would be the one to move, but for now I've implemented a string get-only property called Allegiance on BaseInfo, which is just a wrapper around calling directly into the config file on the attached faction. This is getting back to (from last week) only having one copy of certain data.
  • HasPlayerGainedIntel has been removed, as there is a HasBeenSeenByPlayer on the core faction object now. This was only used by the devourer golem for one notification, anyhow. That now uses DoOnFirstSightingOfFactionByPlayer(), which was added long after the devourer golem faction was coded.
  • TotalMetalMetabolized has also been moved to the roof faction object in Core.

• Got rid of an old GetFactionIntensity() method from Scenario_BaseScenario. The proper code is now BaseInfo.GetDifficultyOrdinal_OrNegativeOneIfNotRelevant().
  • There was a pretty large bug in there that returned -1 for the intensity of any faction that actually used the central intensity settings (aka not AI or its subfactions).
  • This was causing achievements to not fire for beating other factions in general, or at least it should have been doing that.

• The "SpawnAnimationData" and "DespawnAnimationData" data classes have been entirely removed.
  • This was vaguely like the despawn info, which the game DOES use, but it was not actually ever data that was used!
  • It's vaguely possible that a few parts of this were actually functional, but it's honestly very hard to tell. If that turns out to be the case, then we'll have to implement that as more centralized data. The way it was handled before was super confusing.

• The "InstigatorData" faction info is our first example of a faction that needs to be converted to have BaseInfo and DeepInfo.
  • The SpecialFaction_Instigators class would be the basis of the DeepInfo, while the InstigatorData that inherits from ArcenExternalSubManagedData is the basis of our BaseInfo, but really we're going to be pulling both code and data from both of these for our new setup.
  • First off, let's rename "InstigatorData" to instead be InstigatorFactionBaseInfo.
  • Next we need to implement the Cleanup() method in the exact same fashion that we had for the squad baseinfo.
  • Also, because there is some wrapper data also being saved, you must rename DeserializeIntoSelf to DeserializeFactionIntoSelf, and SerializeTo into SerializeFactionTo.
  • IF you have any usage of ModdableGameCommandExecution() on your old SpecialFaction class, then you should port that over.
  • Again, IF you have any usage of DoOnSpecialEvent_OnMainThread_ClientOrHost() on your old SpecialFaction class, then port that over here.
  • You must implement GetDifficultyOrdinal_OrNegativeOneIfNotRelevant() now, even if that's just to return a -1. Ideally you would return a number between 1 and 10, unless your faction has no concept of a difficulty level (or intensity) associated with itself.
    • This method no longer has a faction object passed in. Use the AttachedFaction property to know what faction you're talking about.
    • This method was widely incorrect in the prior version of the game, so this should fix bugs like the "OnLoad: AI faction: X had a difficulty of -1, which is not helpful for setting overlord difficulty level."
    • Instigators are a great example of one with nothing like that.
  • IF you need a nonstandard implementation of FindPathFreshOrFromCache(), then be sure to migrate that over from your old SpecialFaction class. We use this on the hunter, warden, and similar, but not much else.
  • IF you used DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost in your SpecialFaction class, then move that over. Otherwise ignore.
    • Same with DoOnPerSecondWhilePausedNonSimUpdates_OnMainThread if you use it.
    • Also DoOnLocalStartNonSimUpdates_OnMainThread if you use it.
    • And WriteTextToSecondLineOfLeftSidebarInLobby if you need something beyond the defaults.
    • And GetShouldAttackNormallyExcludedTarget() if you use it.
    • And finally, SetStartingFactionRelationships() if you deviate from the defaults, which is the most likely out of this list.
  • Whew, that's everything. A lot of stuff had to be adjusted in order to get this first one working, but now future ones are pretty straightforward.

• Removed GroupTargetSorting (it's some random ExternalData), since that is not in use. It was some very legacy vestigial code from pre-pivot.

• Status time: at this point "all" that's left for the BaseInfo dll is to translate the remaining faction and squad data over. If there's any world data we run into during that process, that will also need to be ported over.
  • After that, there's going to be a needed conversion for all of the SpecialFaction_Whatever data from DeepInfo, but that is going to be a way more direct item to tackle.
  • Also todo is to move notifications back into BaseInfo where the client can generate them all, and turn back on a lot of things that were disabled for being-able-to-compile purposes when the data was not all in the correct place.
    • Some of that will then also probably involve moving some data from DeepInfo classes to their BaseInfo counterparts.
  • This is probably another 2-3 days of work. After that I can run this for the first time and see how badly things are broken. Hopefully that is less than one day's further work of fixing, and then we're back to actually having beta releases. That would put us at Friday, if the general estimate here holds.
  • I did previously have a worry about data-within-data, but most of that is (knock on wood) not an issue. There are still some interfaces in a few spots where a Base class wants to call a Deep class, thanks to SurrogateTables (this is things like the Warden Difficulty, etc). I don't have a firm solution for that yet, but at worst I can use the surrogate table itself to manage that. It's also possible that this form of nesting might not be problematic in the first place.

Reconnecting External BaseInfo to Core, Part 2

• There are now GetExternalBaseInfoAs() and GetExternalDeepInfoAs methods on the faction, to mirror those that are on the squad.
  • This helps us have a more orderly error if something goes wrong, and it also allows for the code to look that much more consistent, too.
  • It's fine to talk about the BaseInfo or DeepInfo directly on a faction, and if you want to then cast that to a specific implementation type that can also be done. But if you need to move from the base implementation to the specific implementation, the most robust way that errors would be reported is by calling these methods.

• LazyLoadSquadWrapper and SquadWrapper have both been updated to hide more of their internals, and thus have fewer bits of confusion.
  • Because these are structs, there's definite risk of confusion on the part of some end programmers, but now there is not. There is also now a Clear() method on both.
  • Related, also removed the SquadPrimaryKeyID_Neg1ToPos serialization variants. People should instead use SquadPrimaryKeyID_PosDef0.
    • In all cases a squad ID of 0 means null, and anything less than 0 will be saved as 0.
  • There are REALLY some notable cases in the external data that need to be using these two structs, specifically because without that the usage of ship pooling might cause some very strange artifacts.

• Fixed up a whole lot of references in the UI to be properly matched to the new format.

• Faction BaseInfo now has a virtual AppendStateForDebugDisplay() method, which does a much more efficient version of what GetAstroTrainsStateForDisplay() and similar used to do.

• All of the external base and deep infos now have a AppendStateForInterfaceDisplay() virtual method on them, and they both seal the ToString() method, fill it with an error, and mark it as obsolete.
  • The idea is that we should never be calling ToString(), as that's confusing and wasteful on the GC. Instead, we pass in a buffer in an ordered fashion.
  • The fact that it is now sealed and obsolete will in both cases help us automatically convert older code to the newer format, since rather than it being invisible (that was one problem before) it is now a compiler error.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • AIReservesPerUnitData is now AIReservesPerUnitBaseInfo
  • NebulaGuardiansPerUnitData is now NebulaGuardiansPerUnitBaseInfo
  • AstroTrainsPerDepotData is now AstroTrainsPerDepotBaseInfo
    • AstroTrainData has been renamed to AstroTrainBehaviorType. If you didn't already know what this was (I did not), it was incomprehensible in the code until you looked in the xml.
    • Fixed a minor issue with TrainsSpawned being serialized twice.
  • AstroTrainsGuardData is now AstroTrainsPerTrainGuardUnitBaseInfo
    • This now uses a LazyLoadSquadWrapper and also uses better serialization. This should solve several rare issues, and is also slightly more efficient during gameplay.

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • AIReservesData is now AIReservesFactionBaseInfo
  • NebulaGuardiansGlobalData is now NebulaGuardiansFactionBaseInfo
    • Fixed some pooling issues.
  • AstroTrainsGlobalData is now AstroTrainsFactionBaseInfo
    • A variety of data has been ported over from the SpecialFaction, and part of that also involved decoupling some of the settings data from the loading of xml. It was messy and prone to potential errors the other way.

• ExoData is... an interesting challenge. This is technically first-level faction subdata, but just something that can be run from a variety of different facions.
  • I don't think that pooling these would give us any notable benefit, so instead these should really just become... a secondary data structure that the game doesn't manage at all, and that is a readonly variable on the faction classes in question. That keeps them implicitly pooled with those factions (there are X ExoData classes if there are X faction classes in pools that have ExoData on them as a readonly property (readonly means the root object can't be reassigned; the data inside it can be changed all day long).
  • In other words, the best way to think about this as if it were just directly data on the faction objects. But of course, we don't want to LITERALLY do that, because then we're repeating code, so instead we have a readonly object that has the fields insde it. Handy!
  • The actual TLDR of this is that it's a challenging concept for a moment, but then has a deceptively simple solution. Hooray.

• Added a new DoGeneralAggregationsPausedOrUnpaused() virtual method on faction BaseInfo.
  • The idea is that there's various lookup information we might need to use from the front-end and the DeepInfo, and this is data that we can simply aggregate from other existing data.
  • This should be done asap on load to make sure the UI is correct from the start. Therefore, we call it right after we call DoOnLocalStartNonSimUpdates_OnMainThread, which is another vritual method you may or may not have a use for.
  • This also needs to stay up to the second accurate even when the game is paued, so it gets called right BEFORE we call DoOnPerSecondWhilePausedNonSimUpdates_OnMainThread() (so that method can use its results if need be).
  • Finally, this also needs to be up-to-the-sim-step accurate when the game is running, so it gets called right BEFORE we call DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost() (so that method can use its results if need be).
  • Reworked AstroTrainsFactionBaseInfo to use this, and it should be a lot more efficient as well as providing a better pattern for other factions in the future. The code also is more brief.

• Added a new DoubleBufferedList<T> to ArcenUniversal, which is a lot like a SwizzleList<T>, except easier to use and based around List<> rather than an array of T[].

• Added GetFactionBaseInfoOrNullAs_Safe<T> and GetFactionDeepInfoOrNullAs_Safe<T> onto squads, for quick acquisition of the faction external info as its native type.
  • Fixed an issue with Dark Spire Vengeance Generator description appenders, where in multi-DS games they would likely all show the information for the first DS, or something else equally wrong. Now they properly show the info for the DS faction of the actual VG you hovered over.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • FallenSpirePerUnitData is now FallenSpirePerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • FallenSpireGlobalData is now FallenSpireFactionBaseInfo
    • Updated the way that FallenSpireDifficultyTable stores its rows by intensity so that we can look them up more efficiently.
    • All of the various lists of cities, debris, and so forth have been updated to use the new DoubleBufferedList<T>.
    • Additionally, rather than having a simple DarkSpireEnabled boolean, this faction now keeps a double-buffered list of all the dark spire factions.
    • And finally for now, SpireDebris_ForUI and SpireDebris_ForUI have both been removed, as the new double-buffered lists that controls the central SpireDebris and SpireCities is now sufficient for our purposes here.
  • DarkSpireData is now DarkSpireFactionBaseInfo
    • DarkSpirePerPlanet is the first example of external data within external data. This really should be pooled, but we can't do that centrally.
      • The methodology for this is going to be up to individual modders and faction designers. I don't have a central solution for this, but I can provide code to replicate. On the worst case, if you do nothing but Clear() the list that is incoming, you could just throw the data away and it might be a memory leak, or the GC might take care of it. Deviating from the approach outline below by too much can introduce an MP-client-only leak, so please just follow the approach below. ;)
      • The better case is to add a new pool that is just for your object type, and then use a ProtectedList or ProtectedArcenSparseLookup where you need to centrally store these. That way when you clear, it goes right back in the pool instantly.
        • To copy this code, simply copy the Pooling section from DarkSpirePerPlanet and, replace "DarkSpirePerPlanet" with the name of your class. Additionally, make sure it inherits from the same interfaces as DarkSpireData does.
        • Next, PerPlanet on DarkSpireFactionBaseInfo was previously a ArcenSparseLookup<Int16, DarkSpirePerPlanet>. It is now a ProtectedArcenSparseLookup<Int16, DarkSpirePerPlanet>. That means that whenever it has entries removed or cleared, it will pop them back in the pool.
        • Finally, Clear() calls need to be Clear( true ) for that list, and anything that would call new DarkSpirePerPlanet() should instead call DarkSpirePerPlanet.GetFromPoolOrCreate(). Congratulations, you now have a leak-proof, MP-friendly, pooled, high-performance piece of sub-data.
    • By the by -- there used to be a method called World.Instance.GetDarkSpireFactionBaseInfoExt_AndCacheAfter(). There's no such thing anymore. That was designed as if there could only ever be one dark spire faction, and would cause problems if there were more.
    • Similarly, some faction helper methods like GetDarkSpireFaction() are now renamed to GetFirstDarkSpireFaction(), and honestly potentially they should not be used at all.

• Vast amounts of the fireteam data has been brought back to life and converted over.
  • It has also been split into three files, for now. One which has text export bits, one which has DeepInfo bits that currently live in BaseInfo (but might move), and then the core stuff.
    • Actually, for the parts we could go ahead and move into Deep, those are now in a FireteamExtensionMethods class in the Deep dll. All but about 80 lines of the deep-type code were able to be moved there. I'll circle back around later. Please note that from the perspective of calling code in the deep dll, there aren't any real differences here (other than the things to do with strings and the one method split below).
  • Additionally, the UpdateNonSerializedFields() method has been split into two parts: one that is run on clients, and one that is not.
  • A bajillion string concatenations have been turned into double character buffers for the sake of efficiency. So has the ShipsByPlanet_ForUI field.

• Added TryGetExternalBaseInfoAs<T> and TryGetExternalDeepInfoAs<T> on squads, along with TryGetFactionBaseInfoOrNullAs_Safe<T> and TryGetFactionDeepInfoOrNullAs_Safe<T>.
  • The idea here is that in a few pieces of code, mostly some of the central Fireteam handling code, it says something like "hey if you've got some scourge data on you, then also do this." But in the event there is no scourge data on the target, or the target's data is some other type, we don't want to complain or have errors!

• On the World BaseInfo, added in DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost as an abstract method, and Safe_DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost as a wrapper method to call it.
  • The various things alongside this which let us see any error data, and clear any error data.
  • This will allow us to handle things like notifications to being a client-and-host activity.
  • While I was added it, went ahead and added the host-only equivalent per-sim-step items on the world DeepInfo. Namely, DoPerSimStepLogic_OnMainThread_NonSim__HostOnly and Safe_DoPerSimStepLogic_OnMainThread_NonSim__HostOnly.
    • This will allow for things like autosave handling, which is host-only.

• Added a new LocalPlayerWorldBaseInfo, which is our very first ExternalWorldBaseInfo.
  • The logic from SpecialFaction_Human that has to do with hovering over planets and finding paths between them has been moved into this.

• Split things up so that we can once again call the ArcenLongTermContinuousPlanningContext.RunAllContexts_ForUnpausedOnly() from the simulation, even though that's a host-only thing and thus moved.
  • Further added a new ArcenLongTermContinuousPlanningClientOrHostContext, and moved PerSecondNonSimPlanning back into BaseInfo and have it now descending from that.

• DoPerSecondNonSimUpdate_OnBackgroundNonSimThread_NonBlocking_HostOnly has moved from faction DeepInfo and is now DoPerSecondNonSimUpdate_OnBackgroundNonSimThread_NonBlocking_ClientOrHost on BaseInfo for factions.
  • Same thing with DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_HostOnly, and it becoming DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_ClientOrHost.

• The seven notifiers that were still left in the old SpecialFaction_Human class have now been moved into their own files in BaseInfo, and are filled from the LocalPlayerWorldBaseInfo world BaseInfo class.

• All of the code for generating intel tab entries (internally called objectives) has been moved back to being run on the client and the host.
  • Sigh. This was a lot of work to make host-only, and the host was going to have to periodically share that data with clients, but at least that no longer has to happen.
  • This is now non-serialized data, as it always had been in the past, and doesn't go in savegames or across the wire to clients.

• Added a new DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_ClientOrHost onto ExternalWorldBaseInfo, which we can now use to generate notifications that are not faction-specific.
  • This is really important for purposes of things like players who are not human empires (solo ark, necromancer, etc). Some notifications would be shared across human player types, others would not, and now we have the choice to actually set that up.
  • Also? If you're a modder and you want to add a notification, and it's not connected to a faction of your own, then this now gives you a way to do it. Previously there was not a way other than adding a fake invisible faction or something else wasteful (but required) like that.

• Added DoPerSecondLogic_Stage1Clearing_OnMainThreadAndPartOfSim_ClientAndHost and DoPerSecondLogic_Stage2Aggregating_OnMainThreadAndPartOfSim_ClientAndHost on the BaseInfo for factions.
  • We still have the host-only versions on the DeepInfo, and that won't change. The host-only version also has Stage0, Stage3, and Stage4. The client section only ever gets its version of Stage1 and Stage2.
  • A lot of factions do aggregations that need to happen in Stage2 on the client and the host, so simply moving those over to BaseInfo while still having the option of having things split is a good idea.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • ScourgePerUnitData is now ScourgePerUnitBaseInfo
  • AstroTrainsPerTrainData is now AstroTrainsPerTrainBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • ScourgeGlobalData is now ScourgeFactionBaseInfo

• A bunch more of the Astro Train faction data has been split over to BaseInfo, partially to go ahead and get some of the notifications relating to them able to function. This really makes things more legible and organized, I have to say. It's still a mess because it's not done, but there's no more wondering which variables are available when, which is really nice.

• Status update! Another day done, and... well, a lot of framework and structure has been implemented today, and SOME data has been transitioned over.
  • In a lot of respects, I was underestimating just how much additional framework work (additions to BaseInfo to make things convenient and quick, and make the cross-talk work) there was when I gave my three-day estimate yesterday.
  • At this point I don't really have a good sense as to how much more framework refactoring there is going to be, but I have a couple of the simpler factions to the point where their BaseInfo is done, and the squad-level data is easy. And the world-level data is also off to the races now, so that's also good.
  • The main problem is that I still have at least a full day of more work on BaseInfo before I can fully start on DeepInfo, although I am making many DeepInfo changes as I go to lessen the impact of that later. I can't imagine that DeepInfo will be less than a day, but it could be two. BaseInfo could be 1-3 more days on its own, which really really stinks.
  • On the other hand, the code is looking GOOD. I can actually understand what's going on, and aside from the truly most complicated factions, I have all of the general relationships figured out. The AI itself is one of the nastiest things remaining, but I plan to try and be as literal of a straight-port as I can, get that working, and then later potentially improve its structure further.
  • Even so, I haven't been able to run the game for something like two weeks now, so we'll see how many runtime errors have piled up once I even get everything to compile, which is somewhere between 2-5 days of work left before I hit THAT point. I had been allocating a day-ish to get it functional enough to share with players again, but I wonder if that's optimistic thinking. We shall see.
  • Another bit of good news is that many MP issues that I was going to have to tackle, some of whcih were multi-day problems that were going to be super error-prone, are just... no longer a worry. And as I've been doing this, I've also been doing a systematic code review and have fixed a number of bugs that were affecting SP at least some of the time. So this is paying dividends all over the place, it's just... really long.

Reconnecting External BaseInfo to Core, Part 3

• Added a new GetDoesThisImplementInterface() on ExternalData_AbsoluteBase, which means that all of the new BaseInfo and DeepInfo classes are able to use this.
  • This is something that allows for a new form of generalized expansion of code capabilities from just the external dlls, without having to update Core.
  • This is useful for purposes of mods in particular, letting them add discoverable methods without having to have access to Core (or without having to alter it if the modder does have access to it).
  • Internally, this method checks to see if the end type that it is (for example, some Faction's BaseInfo type) ALSO happens to inherit from some interface of your choosing. It caches the results, so it's very efficient.
    • Please note, simply calling the normal "BaseInfoClass is YourInterface" to see if that is true will not always return true for interfaces, particularly if they are not directly assigned on the most recent class, or if you are checking for an interface that is a parent or grandparent of the interface which is directly assigned to your class.
      • Confused by that? Doesn't really matter -- just use this method to avoid any funny business with "is" or "as" keywords. Then when you get a true back from "SomeExternalData.GetDoesThisImplementInterface( typeof(YourInterface) )", make sure and do "(YourInterface)SomeExternalData" to get a YourInterface, NOT "SomeExternalData as YourInterface" in order to properly get the interface.
      • If you're curious, then calling "(YourInterface)SomeExternalData" when "SomeExternalData.GetDoesThisImplementInterface( typeof(YourInterface) )" returns false will give you an InvalidCastException. Technically you could wrapper a try/catch around that and use that to direct your code, but this is rightly considered very bad code and also is comparably very very slow.
  • So what can we DO with this?
    • Well, let's take ExoData as an example. As mentioned in yesterday's notes, sometimes there is an ExoData that is now directly assigned to a class (like FallenSpireFactionBaseInfo). Let's break down how this works.
      • If you know that FallenSpireFactionBaseInfo has ExoData and you want to look at it, you can just call faction.GetExternalBaseInfoAs<FallenSpireFactionBaseInfo>() and then immediately access the ExoData. Well that's simple!
      • But what if you DON'T know what type the BaseInfo is, for instance if you're iterating over a list of BaseInfos for various other factions for whatever reason. Maybe they are from other mods, even. Who knows?
      • Well, in that case we need a generalized way to ask ANY BaseInfo "Do you have an ExoData?" In the past, we could just call faction.GetExoDataExt( ExternalDataRetrieval.ReturnNullIfNotFound ), and what we want is an equivalent to that. So that's what we're creating here.
    • First of all, we need a new interface. In the file where the ExoData class exists, I have defined a new interface called IExoDataHolder, and gave it just a single method, called GetExoData(). See code for details, but it's super duper simple.
    • Secondly, any class that has ExoData on it that I want to expose in this generic fashion should now inherit from IExoDataHolder. For the moment, that's just FallenSpireFactionBaseInfo, but later it will be two others once I port them in.
      • Now on thos classes, I have to implement the very very simple GetExoData() method that I designed in IExoDataHolder. This is one line of internal code, just returning this.exoData.
    • Now, back in some other code where I would have previously called faction.GetExoDataExt( ExternalDataRetrieval.ReturnNullIfNotFound ), I can call the following:
      • "if ( otherFaction.BaseInfo.GetDoesThisImplementInterface( typeof(IExoDataHolder) ) )"
      • then: "exodata = ((IExoDataHolder)otherFaction.BaseInfo).GetExoData();"
  • And that's it! It would have been possible for me to make this code slightly less verbose, BUT that would have been at the expense of flexibility for faction designers and mod designers.
    • With the current approach, you can implement ANY arbitrary data-holders on any subset of classes you choose, and Core never needs to be altered in order to do so. The power of that is really useful.
    • Please bear in mind that for most sub-data, we don't remotely need to make use of this. We don't always CARE what factions use what feature. But for cases where we want to loop over all the factions and ask "which of you can give me this kind of subdata?", this new pattern lets you do so.

• NanocaustPerUnitData has been retired from the game, since it was noted as deprecated in 2019.
  • ConstructorData for the nanocaust was not marked as deprecated, but it sure was unused, so it's also gone.

• Added a new AllegianceHelper static class, which has things like enemyThisFactionToAll and such moved into itself.
  • Fixed a couple of small bugs in here that could lead to strange allegiances. One of which was related to multiplayer allegiances with players beyond the first.
  • Beyond that, there was a lot of special-case type code that was looking for specific factions and making choices about that. Those instead need to be triggered by xml data or this is not future-proof and mod-friendly.
    • Added allegiance_should_always_be_neutral_to_all="true" as a new option for factions, and this is now on the nebula guardians and territory dispute factions.
    • Added allegiance_should_always_be_friendly_to_all="true" as a new option for factions, and this is now on the Zenith Trader.
      • Fixed several ways in which factions could become hostile to the Trader in particular.
    • Also added allegiance_factions_i_always_hate="Faction1,Faction2" and allegiance_factions_i_always_ally="Faction1,Faction2".
      • This lets us centrally define some relationships that should always be friendly or enemy, using the name (InternalName) field from the factions in question.
      • Key note: these faction names are not checked, so they can be from mods or dlc that are not installed without problem. But if you have a typo, it may be hard to notice for that same reason.
      • This is not only more flexible (for mods and otherwise), but it also prevents the AllegianceHelper from accidentally overwriting entries if things like MakeEveryoneAnEnemy or whatever are called.
      • At present this is used to make Nebula Guardians and Champions hate one another, and Necromancers and Templars to always hate one another as well.
  • Also moved SetDefaultStartingFactionRelationships() into there, so that the defaults can be seen in External code (it had recently moved to Core) as well as changed as-needed.
    • Bear in mind you can override the defaults at a faction level via a mod, but you can't do that over what is in this helper class for ALL factions. But you can certainly rewrite faction alliances however you want via a mod.
  • As part of this, made it so that we never can override (via larger faction alliance methods) team affiliations (for the minor faction teams of three colors) or always-hate or always-ally settings. There were definitely some cases where someone could break off previously.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • MacrophagePerSporeData is now MacrophagePerSporeBaseInfo
  • MacrophagePerHarvesterData is now MacrophagePerHarvesterBaseInfo
  • MacrophagePerTeliumData is now MacrophagePerTeliumBaseInfo
  • OutguardUnitData is now OutguardPerUnitBaseInfo
    • Made OutguardGroupDataTable serialize by index, and thus improved the data transmission of this.
  • MarauderOutpostRaiderSpawnData is now MarauderOutpostRaiderPerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • MacrophageGlobalData is now MacrophageFactionBaseInfo
  • NanocaustMgr is now NanocaustFactionBaseInfo
  • OutguardGlobalData is now OutguardFactionBaseInfo
  • DysonData is now DysonSphereFactionBaseInfo
    • DysonAntagonizerData has been removed, and merged back into DysonSphereFactionBaseInfo. There was always a 1:1 ratio of these existing, and there is no benefit to having them separated at this time, while there are several downsides.
  • HRFData is now HumanResistanceFighterFactionBaseInfo
  • MarauderData is now MarauderFactionBaseInfo
    • The marauder UpdateAllegiance has been updated to be more robust and use the AllegianceHelper.
    • Eight constants from the marauders xml were messily being saved into the savegame, which was mildly wasteful and moderately confusing. This is all cleaned up now.
  • RiskAnalyzerData is now RiskAnalyzerFactionBaseInfo, and is on the risk analyzer faction rather than on a human faction (that was super confusing and likely led to errors).
    • It is now our second example of an IExoDataHolder.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • DarkZenithPerUnitData is now DarkZenithPerUnitBaseInfo
  • ZenithMinersPerUnitData is now ZenithMinersPerUnitBaseInfo
  • ZenithArchitravePerUnitData is now ZenithArchitravePerUnitBaseInfo
  • TemplarPerUnitData is now TemplarPerUnitBaseInfo
  • TerritoryDisputePerUnitBuildData is now TerritoryDisputePerBuilderBaseInfo
  • SappersPerUnitData is now SappersPerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • GlobalCPALogic is now AICrossPlanetAttackerBaseInfo
  • ScourgeFactionBaseInfo is actually fully set up now, apparently I got distracted in the middle of that one.
  • DarkZenithGlobalData is now DarkZenithFactionBaseInfo
  • ZenithArchitraveGlobalData is now ZenithArchitraveFactionBaseInfo
  • ZenithMinersGlobalData is now ZenithMinersFactionBaseInfo
  • TemplarGlobalData is now TemplarFactionBaseInfo
  • NecromancerGlobalData is now NecromancerFactionBaseInfo
  • TerritoryDisputeTeamData is now TerritoryDisputeTeamFactionBaseInfo
  • NomadPlanetsGlobalData is now NomadPlanetsFactionBaseInfo
  • SappersGlobalData is now SappersFactionBaseInfo
  • MigrantFleetsData is now MigrantFleetsFactionBaseInfo

• Went ahead and removed the champions and nebula guardians factions and their related data.
  • The history of them is available in svn, but as has been discussed elsewhere, necromancers and templars replaced these.
  • At the moment, the time cost of translating the rest of these over was going to be a waste of several hours.
  • EnableChampionResponse is now EnableNecromancerResponse, although that setting is not yet hooked up to do anything, so it's commented-out for now.
  • A lot of the central code that was related to champions has been renamed to SoloArk, and converted slightly in preparation for that kind of faction later on.

• We're now to the point where the only faction info remaining in BaseInfo that has not been converted is stuff relating directly to the AIs themselves!
  • There will still be a flood of other things needing to be adjusted after that, but that's a milestone that is approaching (and a milestone that has been hit, as well).

• The concept of ProcessFactionConfigCustomFields has been removed from the game.
  • This was something that was previously running right after you left the lobby and translating the settings from your "soft settings" into permanent saved data on the actual factions.
  • As of a few days ago, I made it so that we're now pulling much more directly from the "soft settings" in the first place, which is useful for making sure that the game always has your latest settings even after making changes in the post-lobby settings areas.
  • It also lets us avoid having to save the data a second time for the factions, so that the "soft settings" location is instead the only place the data is saved -- which is good, because when you save the same thing two places you are at BEST wasting some bandwidth/disk space (less than a kilobyte, but still), and at worst you secretly have an accidental disconnect from what front-end shows and what the back-end is working from.
  • In place of this, there's a new convention -- not enforced -- of DoRefreshFromSettings() on factions, where we can get the settings on a small random interval, called from the chain of DoGeneralAggregationsPausedOrUnpaused(). This keeps the CPU usage down, things nice and responsive, and the settings super up to date, all at once. It's also easier code to read.

• We now generate a completely random FactionRandomSeed on each faction when we're starting the game. This allows us to do things like random AI types that are unpredictable, but still deterministic per-campaign.
  • Actually we wound up not using this right now, but it might be useful in the future, and it's very little data, so what the heck; let's keep it.

• Serializing a table entry no longer confusingly takes two string variables, one optional and one not, when deserializing while taking only one for serialization. It now just requires one for both, which is the same one.
  • Additionally, that FieldNameForErrors is no longer optional for these.

• Work is ongoing with getting the AI and its sub-factions set up. A lot of this is coming together quite well.
  • There were a few surprises, like the way that I need to cache the AI Type after randomly choosing it, but that's not something you can change once the game starts, anyway (or if it's adaptive, it changes itself in the one spot here). Differentiating this from the other settings was not hard. Now it's just a matter of doing the rest, carefully.
  • There's a solid chance that this will implicitly fix some bugs that were recently reported about things like "individually chosen sub-difficulties not working properly when universal difficulty is off." That was likely an artifact of the order of operations craziness that was part of the older way of using ProcessFactionConfigCustomFields. Once it's ported over, it should be order-of-operation-independent.

For Modders: How To Handle Lists Of Sub-Data In Multiplayer-Friendly Way

• Added a new extension method to List<> and ProtectedList<> that makes deserialization into an existing uneven list waaaay simpler.
  • This is what needs to happen, over and over again, on clients when they are getting certain kinds of external data.
  • PlannedWaves are one of the many that needs this sort of logic, so formalizing it so that it isn't prone to errors (and is easier to read and understand as end-coder) is a win.
  • Basically, you use this when:
    • You have a list of stuff that is variable length over time, and the client is going to be getting a deserialized copy of it pretty frequently... AND
    • You don't want to just clear that list (because what a waste on the one hand, or other code might be having references to objects in it)... AND
    • You do want to refill the list with the most up to date data in host-order, and then discard any leftover husks that were on the client.
  • Example 1:
    • Client has a list of 10 incoming waves. Host sends an update that includes 7 waves for whatever reason.
      • Result from this process: Client deserializes those 7 into its first 7 slots (overwriting whatever is there), and then discards the last 3. If it's a ProtectedList<>, then those 3 are headed back to the pool automatically.
  • Example 2:
    • Client has a list of 1 incoming wave. Host sends and update that includes 5.
    • Result: Client deserializes the first 1 into the existing one it has, then adds 4 more.
  • Example 3 is simply when the number of waves match, whether or not the data in them does, and it just overwrite-assigns each one as it goes.
  • The idea is to handle all of that logic without having to write all of that code every time (and to avoid the typos and confusion that are inhernt in copy-pasting such code all over the place).

• Okay! Time to handle the first of the sub-objects that are pooled properly. Let's look at PlannedWave as an example.
  • We're going to want to have it inherit from the following: PlannedWave : TimeBasedPoolable<PlannedWave>, IProtectedListable<PlannedWave>
    • This sets us up to let us have it go in a variety of kinds of pools, as well as also function inside "protected lists" (where when you clear the list, it goes back in the pool for you automatically).
  • There is then a "#region Pooling" section, and you basically should copy that whole bit over and change and references to PlannedWave to be whatever your class is (same as with the inheritance bit above, of course).
    • In this section there is a DoAnyBelatedCleanupWhenComingOutOfPool(), which you can usually leave blank, but essentially it's helpful if there are some object references that might briefly be in use still when this gets put back into the pool, that you don't want to cause nullref exceptions from.
    • In this section there is also a DoEarlyCleanupWhenGoingBackIntoPool(), which tends to happen on background threads instead of one that is more busy, so anything you need to blank out that you can, should probably go here. You want to make sure and reset EVERY variable and list and so on that is a sub part of this object in this.
      • In this example case, to keep things simple and also consistent, I've got a SetDefaults() method that is being set in the constructor as well as on the DoEarlyCleanupWhenGoingBackIntoPool(). This is a good pattern to keep.
        • It is SUPER important that absolutely every property or variable on your class (PlannedWave in this case) is reset to its default value in SetDefaults(). Otherwise the next wave will have some stale data and act strangely when this gets pulled back out of the pool.
    • Please also note that there's a private constructor in here, and then no other constructors anywhere else in the class! We don't want any way of creating new objects of this sort other than calling the static method GetFromPoolOrCreate(). Anything else we call would be bypassing the pool and causing at least a mild memory leak.
  • At this point we have a poolable object that can only be retrieved from a pool, but we're not doing any retrieval or ever putting them back. So now let's do that.
    • On AISentinelsFactionBaseInfo, I would normally have a List<PlannedWave> WaveList. Instead we're going to use a ProtectedList<PlannedWave> WaveList, and it pretty much works identically.
    • This is a case of this list being the truly master list. If it's in this list, it's meant to exist. If we take it out, it's meant to be gone back to the pool. So, based on that, we don't ever have to think about calling ReturnToPool() manually on the PlannedWaves. That's handled for us. Sweet!
      • On Clear and Remove from this list, just be sure to pass in true as the required parameter, and it will do that handling for you. In the event that you wanted to use a protected list without it automatically sending stuff to the pool on removal, you could pass false... but that is not advised.
    • In that case, that means that the only thing we need to change beyond this is wherever we were previously calling "new PlannedWave()". That simply changes to "PlannedWave.GetFromPoolOrCreate()", and boom we're pooling. And we're done.
  • In the case of PlannedWave, I am using a ConcurrentPool, by the way. This is something that can be written to from any thread, and it has some nice advantages. But there's not "time out" cooldown before waves are reused from the pool. Because of the nature of how waves are created, this is not a problem.
    • In the case of a piece of data where you might have lingering references for a while, you might be better served using a TimeBasedPool instead. That works exactly like ConcurrentPool except that you can specify a cooldown time that the objects must stay in the pool before they can come back out. 2 intervals of 30 seconds each are enough for all long range planning to complete for every faction in a super pessimistic case, if that's what you want to base it on. Otherwise just give it a couple of seconds of delay. At any rate, PlannedWave doesn't need that.

Reworking Subfaction Relationships And The AI and Beacons, Part 1

• Added a new AISentinelsFactionBaseInfoExtensions class, with GetAISentinelsBaseInfo() on it as an extension of the faction class.
  • The reason I'm doing this for the sentinels and not any other factions is because of all its many subfactions, and their constant need to get the info of their parent.
  • I'm also planning on storing all of the meaningful subfaction data of any note on the parent itself, to further tighten these things together.

• Removed GetPrecedingAIFaction() and GetNextFactionOfSpecialFactionData() from the Faction class in Core.
  • Now that I'm working on the AI and its subfactions, the current situation with how they are linked is just really untenable. It was unreliable before, and would not always properly link the factions together exactly as expected. Time to change that.

• Just like we have FactionIndicesIAmHostileTo and FactionIndicesIAmAlliedWith, factions now also have FactionIndicesOfMyChildren and a new FactionIndexOfMyParentIfIHaveOne.
  • These let us keep track, explicitly, of which subfactions are linked to what parent faction, in a bidirectional sense.
  • There is a new MakeChildOfOtherFaction( OtherFaction ) method on the faction class that makes this a one-line call to initialize.
  • You can then just call the new GetParentFactionOrNull() on factions to get a reference to the parent faction if there is one.
  • Part of the idea here is to make it much simpler for our AI factions and subfactions, sure, but also the other part of the idea is to make it so that mods can now set up compound-factions like this if they need to. Or us with other related factions if need be.

• General improvements to some of the serialization code have been made.
  • Among these, planets and factions no longer have a serialization gate calculated for them. They are simply now capped at 511 for planets, and 255 for factions. These limits were already well enforced elsewhere.
  • Fixed a minor new SpecialFactionData serialization issue from it getting mangled in all the ripping and tearing. It would have been fast to find later, but this saves time.

• Getting rid of some other methods that we no longer use and so just clutter things up:
  • On faction: GetDoesThisFactionHaveBaseInfoType(), GetDoesThisFactionHaveDeepInfoType(). That new FactionRandomSeed.
  • On squad: GetFactionBaseInfoType_Safe(), GetFactionDeepInfoType_Safe().

• Split the BaseInfo and DeepInfo serialization out of the main faction methods for serialization into new SerializeExternalFactionData() and DeserializeExternalFactionData().
  • This is actually a pretty huge deal, because now we'll send the externaldata at the end of any transmission, and at the end of any savegame.
  • This means that factions, squads, planets, etc, all already exist when externaldata starts getting loaded. Previously a lot of that was not loaded yet, so externaldata had to use indices during deserialization rather than looking at real objects. This will simplify code, and speed up some bits of code, and reduce coding errors.
  • Same for BaseInfo and DeepInfo on squads, which is now based from SerializeExternalSquadData() and DeserializeExternalSquadData().

• The multiplayer fast-blast sync code has now been updated to handle the BaseInfo from host squads needing to be synced over.
  • The faction BaseInfo sync in multiplayer via PeriodicFactionExternalSyncDataThatJustOverrides has been updated to the new format, which was very simple.

• Split SerializePlanetFactionEntities and DeserializePlanetFactionEntities out from the core PlanetFaction serialization, since that was rather confusing code the way it was before.
  • Also, CurrentAttackMultiplier and CurrentSpeedMultiplier are no longer serialized at all (it's recalculated every frame).
  • And the shipgroups are no longer serialized, even in super-brief form, except for AI factions (they're the only ones who use it).
  • Uh, also a lot of other stuff moved around. It's more efficiently organized, less likely to cause errors, and easier to extend.
    • The really big one is that this completes the move of all the externaldata to the very end of a network transmission or a savegame save. For factions, squads, and the world. That way the external data can always freely talk about the Core data without worrying if it's been loaded or not yet.

• For the sake of clarity, the factions that are auto-added-always-there-but-only-one have been moved into one file, and the "factions that are auto-added-per-AI" have been moved into another.

• On factions, auto_add_one_of_faction_if_missing_on_save_load="true" has been changed to should_always_have_exactly_one_of_these_in_every_game="true"
  • This has been removed from several sub-factions of the AI, and instead is just on the things like Outguard, AI Reserves, Instigators, and the zombie factions.

• There was then an actually_seeds_multiple_so_do_not_complain="true" that was an answer to some of the factions that had must_be_at_most_one="true" on them but were not supposed to complain.
  • Those factions were erroneously marked that way in the first place (because of a long history of evolution of the game, it's complicated), so that actually_seeds_multiple_so_do_not_complain="true" has been removed.

• auto_add_one_of_faction_per_ai_if_missing_on_save_load="true" and remove_if_more_than_one_per_ai_prior_to_this="true" have both been removed, as they were kind of funky.
  • Instead, added a new is_a_subfaction_of="OtherFactionName", which lets us define the relationships directly, and which causes them to get created one-per-such faction, call MakeChildOfOtherFaction() on said factions, and disappear if that faction is not present. All speaking here about on the lobby load, or savegame load.
  • The warden, hunter, PG, CPA, relentlesss wave, and border aggression are now marked as subfactions of the AI.
  • The territory dispute teams are now marked as subfactions of the parent territory dispute faction.

• takes_on_color_of_the_first_ai_faction="true" is being left alone -- that's useful for instigators and so on -- but takes_on_center_color_of_the_preceding_ai_faction="true" is being removed.
  • We now instead have a takes_on_center_color_of_the_parent_faction="true" as an option which can work for any faction that is a child of another. For now it's just used with the AI subfactions: warden, hunter, PG, CPA, relentlesss wave, and border aggression.

• is_considered_visible_part_of_the_ai_faction_for_what_can_own_purposes="true" is removed, as it was not actually used at this point.

• takes_on_current_general_mark_level_of_the_ai_factions="true" has been renamed to takes_on_highest_current_general_mark_level_of_any_ai_faction="true", because that's what it does.
  • This is now used by the instigators, AI reserves, astro trains, and risk analyzers, but is no longe used by the AI subfactions.

• Added a new takes_on_current_general_mark_level_of_parent_faction="true" that is now used by the AI subfactions.
  • Previously if you had many AIs of different strengths, their subfactions would all be the mark level of the strongest AI, which was wrong.

• The way that "beacon factions" work is being completely reworked under the hood.
  • The user experience was never great (people didn't realize they could configure said factions prior to hacking the beacons but that it got more limited after that).
  • And there was a nontrivial performance drain, and some bugs, from having lots of these. Macrophage picking up spire shards when not in the game except in beacon form, etc.
  • In the next few weeks I will re-code the hacks for the beacons so that you can configure all the aspects of the beacon faction as you are summoning them. The beacons will from now on belong to the NaturalObjects faction, however.
  • In the meantime, all of the stuff related to a faction being present but "needing to be awakened" is being removed.
    • MustBeAwakenedByPlayer, MustBeAwakenedAndIsAllied, ShouldBeRetainedIntoLobbyLoadEvenIfMustBeAwakened, and HasBeenAwakenedByPlayer.

• auto_remove_faction_on_lobby_load_and_mapgen_when_other_faction_present="FactionName" is also being removed, as we don't need it anymore.

• auto_add_one_of_faction_when_other_faction_present has been converted into auto_add_singleton_of_faction_when_any_count_of_other_faction_present, to be more clear.
  • This is still used on tamed and enraged macrophage, but is also now used by the antagonized dyson sphere. This is also still used by the templars in response to a necromancer.

• When you load up a savegame into the lobby, or old lobby settings, or a quickstart into the lobby or for generation, it now strips out all the following:
  • Any subfactions of other factions. Any that should always have exactly one per game. Any that are automatically added when some other faction is present.
  • This is mostly equivalent to what we did in the past, but the general idea is we never want to see things like a dormant tamed macrophage with no main macrophage present, or extra hunters due to having fewer AIs present now, etc.
  • The only time these things are added BACK is when a full generation of the map happens (aka moving from the lobby into the game, or from the quickstart into creation of the game proper). Then all the magic fills out the relationships and should have exactly the faction list as desired.
  • We had half a dozen idle factions clogging up every savegame in the past, and sometimes more, which was potentially a waste of bandwidth and performance. It was also a barrier to adding beacons for more factions, since there was such a performance hit with each one.
  • All of the various code paths that do these sorts of things now seem to function, which is awesome.

• Removed GetDefaultFactionConfigurations() from the Scenario definition, since now most of our code is much more xml driven.

• One thing that isn't fully xml-driven, but nonetheless is now part of the Expert campaign type, is that you must have a minimum of two AIs to face.
  • A lot of the complexity of the game is lost when you only have one AI, because you don't get that ramp-up in challenge after defeating the first one, and have to figure out how to still deal with the second one.

• The seeding options for a couple of DLC2 factions that were "Only From Beacon" have been removed.
  • These will get beacons in the near future, along with some other factions that never had beacons, but they will work the new way.
  • Worth noting that this didn't play well with the Enable Beacons setting, and so could get kind of funky in Expert mode because of that.

• Also stripped out more code that was based around the concept of a faction being present as a beacon but not enabled.
  • Later, the logic will be that the beacon is present as a NaturalObject, and the only time the faction will arrive is after a hack adds them (letting you clearly choose settings for it before you complete the hack, too).

• Added in a new HasDoneInvasionStyleAction to factions, which is false by default.
  • Previously, we had a HasBeenAwakenedByPlayer which was used one way sometimes (for beacons) and another way other times (for delayed invasions). That had led to various bugs in the lobby.
  • Now we just have HasDoneInvasionStyleAction, which is used for invasions or not at all. Most factions ignore this one bool and just go about life.
    • This is primarily used by the nanocaust, but in theory other factions could use it later.

• Simplified some of the PlanetFaction data. FInts are the largest data format that we have for casual use, and each PlanetFaction object had three of them (AIPLeftFromControlling, AIPLeftFromCommandStation, and AIPLeftFromWarpGate).
  • In an 80 planet game with 21 factions under the hood (remember, one AI is a compound faction that is 6 under the hood alone), that would have been 5,040 FInts. Those each require somewhere in the ballpark of 64 bits to store, which would be a total of 40 kilobytes in a savegame that is maybe 200-600kb in total in the early stretch (so, that's as high as 20% of our entire data storage in the early game, for context. Granted, things get adjusted around a lot to avoid this, but I still wanted to minimize what is on PlanetFactions, and the above is a good enough example to show why.
  • We no longer have AIPLeftFromControlling at all, since that could be inferred from the other two. The other two are now stored as integer values, and specifically AIPLeftFromCommandStation is serialized as a 17-bit format, and AIPLeftFromWarpGate as a 9-bit format.
    • Assuming no other changes (ha -- there already were improvements elsewhere, but let's ignore those for this example), this would be 5.74 kilobytes of data now, instead of 40.
    • This matters much less in savegames than it does in multiplayer, where we are frequently sending things like the planetfactions from the host to each client just to make sure that things are synced up.
  • The bandwidth use by the game prior to this whole rewrite of the game's structure was pretty darn alarming (though still lower than a poor-quality youtube video, for instance). But that bandwidth usage made higher player counts harder for people to do, and put an extra strain on people who had a lot of packet loss or were physically very far apart.
  • This is just one of those "low hanging fruit" ways to reduce transmission amounts, but the whole rewrite of all this is aimed at least partially in that direction (other than bandwidth reduction, the rewrite is aimed at code correctness, client functionality parity, and ease of coding for modders with the game having to send less self-correction data.

• Added TryGetExternalBaseInfoAs() and TryGetExternalDeepInfoAs() on faction, mirroring the versions that are on squad.
  • Added a new TryGetAISentinelsBaseInfo() in AISentinelsFactionBaseInfoExtensions as an extension on the faction class. This more or less mirrors the "get or return null" style.
  • Essentially, in your mods you can do a global search and replace for "GetSentinelsExternal( ExternalDataRetrieval.ReturnNullIfNotFound )" to "TryGetAISentinelsBaseInfo()"

• All of the AI subfactions are now going to have sub-objects that are shared on the central AI's object. This greatly simplifies how they can access each other's information, but lets us still keep things ordered. As with ExoData, these will be readonly subobjects that we can then edit at will without ever replacing them, and so they don't need to be pooled, etc.
  • AISentinelsExternalData is now AISentinelsBaseInfo. This is now a readonly sub-object of AISentinelsFactionBaseInfo, which makes it really convenient for any sentinels faction or subfaction to get at the data. I'll be following the same pattern with the other data of this sort.

• TimeBasedPoolable now does the thing where it seals and obsoletes ToString(), and has a virtual method called AppendStateForInterfaceDisplay().
  • We really want to avoid having classes that are calling ToString(), because that's pretty much always GC churn and a waste.
  • EntityOrder now uses AppendStateForInterfaceDisplay instead of ToString(). It's a case of not even knowing it was a problem until it was made a compiler error.
  • PlannedWave also has been updated in this way, as has EntitySystem.

• ArcenCharacterBuffers of all sorts now support passing in an ArcenPoint and having that serialize nicely.

• ExtragalacticBudget has been updated to be poolable in the same pattern that was implemented for PlannedWave.
  • Fixed an issue that would cause deserialization errors in ExtragalacticBudget on MP clients. This was a longstanding issue, looks like, and very rare and confusing if it happened.
    • The fix for this was switching to the new DeserializeUncertainNumberOfEntriesIntoExistingList(), which is a great example of just why this new thing is so darn useful.
  • ExtragalacticBudget is an interesting case, because it was very complex with multiple constructors. But those simply are turned into static Create() methods, and then suddenly the complexity is gone and the program flow is the same. There are many similar examples in Core, but this is the first one in external code.

Reworking The AI And The Last Of BaseInfo, Part 1

• AISentinelsBaseInfo is now AISentinelsCoreData, because that naming was HYPER confusing. Naming anything BaseInfo or DeepInfo when it's not actually that is a really bad idea, turns out.
  • The other sub-classes of AISentinelsFactionBaseInfo are now being set up as well. Bear in mind that these are part of the sentinels faction data, not part of the sub-data of the sub-factions. They do have their own sub-objects, though, to keep things organized but accessible.
    • AIWardenExternalData is now AIWardenCoreData
    • HunterFleetExternalData is now AIHunterCoreData
    • HunterFleetExternalData is now AIHunterCoreData
    • PraetorianGuardExternalData is now AIPraetorianGuardCoreData

• AICommonExternalData is now GlobalAIWorldBaseInfo, which is something that I am attaching to the world object, because it's thing that is truly global between all of the AI factions that might exist.
  • This is slightly more efficient from a data standpoint, and is simpler to use. Any faction can do this sort of thing, particularly if there are going to many of that faction.
  • This also declares a singleton, so it's really easy for any code anywhere to now just go GlobalAIWorldBaseInfo.Instance.AIProgress_Effective or whatever.

• AIPChange is now being treated like PlannedWave, with the new TimeBasedPoolable and IProtectedListable inheritance.
  • This is probably now the simplest example of this pattern in use, because it's a fairly low-data-and-methods class to begin with.
  • There was some logic in GetAIP that previously was setting the AIP to the highest version based off of any AIP. The coder had been note the code as being a bit hacky. I've move that over into GlobalAIWorldBaseInfo, and it calculates those things now on that method.

• Added a new EnumIndexedArray in Universal, so that we can get rid of all those class-specific versions and add more functionality to them.
  • In particular there is a new ClearTo method. But there's also a reference tracker, so that we'll know how many there are in the game if we do a data dump. And there's just no longer a bunch of duplicate code all over the place so much.
  • Then removed the following classes, converting them to instead use this new EnumIndexedArray:
    • Universal: ArcenEnumIndexedArray_PerformanceSegment, ArcenEnumIndexedArray_FourDirection
    • Core: ArcenEnumIndexedArray_CoreFunction, ArcenEnumIndexedArray_EntityLineHardcodedType, ArcenEnumIndexedArray_MetalFlowPurpose, ArcenEnumIndexedArray_SpecialEntityType, ArcenEnumIndexedArray_OtherSpecialEntityType, ArcenEnumIndexedArray_EntityRollupType, ArcenEnumIndexedArray_FactionType, ArcenEnumIndexedArray_PlanetFactionBooleanFlag, ArcenEnumIndexedArray_EntityOrderType, ArcenEnumIndexedArray_FactionStance, ArcenEnumIndexedArray_ResourceType, ArcenEnumIndexedArray_EntityBehaviorType
    • External: ArcenEnumIndexedArray_AIBudgetType, ArcenEnumIndexedArray_HunterFleetBudgetType, ArcenEnumIndexedArray_PraetorianGuardBudgetType, ArcenEnumIndexedArray_AIWardenBudgetType, ArcenEnumIndexedArray_Code, ArcenEnumIndexedArray_ClientConnectionStage
  • Tip for modders: you can use a simple global search and replace for each type you remove, in this fasion: "ArcenEnumIndexedArray_AIBudgetType<" with "EnumIndexedArray<AIBudgetType,"

• The external data for factions and squads now has a virtual method that you can override called DoAnyInitializationImmediatelyAfterSquadAssigned() and DoAnyInitializationImmediatelyAfterFactionAssigned(), respectively.
  • On AISentinelsFactionBaseInfo, I am now using this to call "this.WardenInfo.InitializePathfinders( this.AttachedFaction );" and the same for the other subfactions.

• It turns out that ExoData is the perfect model for how to handle the AntiMinorFactionWaveData, so that's the pattern I'm going with, including having a IAntiMinorFactionWaveDataHolder.
  • MarauderFactionBaseInfo and NanocaustFactionBaseInfo have been updated to hold an AntiMinorFactionWaveData WaveData object.

• And that's the last of converting old style ExternalData over to BaseInfo!
  • At the time of this writing, there are 195 errors still remaining (newly revealed now that all the types are converted over) in the BaseInfo dll that need to be resolved before I can start truly working on DeepInfo.
  • There's also 85 "//EXTERNALDATA_TODO" tags where I commented out some code in BaseInfo to let it compile at one point last week. I'll have to uncommented those and get that all compiling, and then BaseInfo will be done-enough-until-DeepInfo-forces-more-changes, but at least I'll be able to compile it.

• Uncommented out all 85 of the //EXTERNALDATA tag sections, and now I'm up to 385 reported errors in the BaseInfo dll. Time to get cracking on those!

• Removed the PreferredAIFactionIndex from faction external base data.
  • This has been made obsolete by FactionIndexOfMyParentIfIHaveOne on the Faction object itself.

• BaseAIFaction.ChangeAIP() should now be GlobalAIWorldBaseInfo.Instance.ChangeAIP(), which has been converted to the new format.

• SpecialFaction_CPALogic.cpaDelayTime is now AICrossPlanetAttackerBaseInfo.CPA_DELAY_TIME.

• Rather than ever using factionExternal.AIProgress_Effective, it should now always be GlobalAIWorldBaseInfo.Instance.AIProgress_Effective.
  • Same with AIPChangeHistory and things like that.

• GetPrecedingAIFaction() should now be GetParentFactionOrNull().

• I've improved the way that we're able to call in and get faction configuration data. You can have it error when you ask for data from a field that does not exist on that faction, or just return the default value.
  • Most of the time you'll want it to error, because that's probably a glitch in your code that will cause unexpected behavior. It's the sort of thing where a small typo can just wreck your whole day, but now it's actually going to tell you about it and therefore it's super quick to fix.
  • Then again, there are other times where we're scanning past lots of factions, and saying something like "hey, do you have an intensity? I'll show it if so. Got an allegiance field? I'll show that, too, if you got one." And if the answer is "no I don't have one," that's not an error at all. Those are the cases where you'd pass in false to having it error on missing data.

• Added some extension methods where you can call faction.CustomData_GetSpawningBooleanOne(), and a variety of others that start with CustomData_[Whatever]
  • Many references to Ex_MinorFactionCommon_GetPrimitives have been expunged, with most being converted using the above, but others just directly referencing some variable in a new place. Still more to go.
  • This also pulled out a ton of code from ProcessFactionConfigCustomFields() that was very specific to just that one method, and now that's more formalized and something that can be called from more than just that one place.
  • The complete difficulty settings for the AI and all of its subfactions now get pulled out properly, and should be far more accurate (there were some reports before of univesal difficulty not being handled properly when disabled)
  • Also as part of this, many of the generic names for things (like "SpawningBooleanOne") have been renamed to actual sensible things (like "EnableFimbulwinter").
    • Side note: it was a huge pain in the rear to even try to add new proper field names before, so this is not a criticism of faction coders in the past. It was an archtectural challenge that no longer exists.

• Added a new DevourerFactionBaseInfo, which tracks the TimeLastExisted for that faction, as well as some other info that was in the DeepInfo dll but needed to come out.
  • Also added a new LoneWandererFactionBaseInfo, which will act as the underlying basis for handling both the Devourer and the Zenith Trader.
  • Please note, this is actually a great class to look at if you're thinking of doing something similar. It has a TON of extra comments in there, mostly about how to properly use DoubleBufferedList<>.
  • I am going to start having a habit of putting "TEACHING_MOMENT:" in the code anytime there's an example class with a lot of things that are explained. That way coders can search for "TEACHING_MOMENT" and find anything that I (or someone else) has put in there.
    • There are two teaching moments in this particular file. One for double buffered lists, and the other for Sealed methods and abstract submethods.

• Updated my DoubleBufferList so that the swap and clear actions are two separate things.
  • The reason for this is that at swap-time, it's very likely that another thread has a hold of the expiring display list. So clearing that would cause an exception.
  • By the time we loop back around and are ready to construct a new list, any other threads are fairly certain to all be using the new display list, so clearing that other list won't bother them. There's a much larger gap between list constructions than there is any display-list-accesses, by design. This is a common feeder pattern.
  • It's something I had realized the other day, and had just put on my long list of to-dos, but at this point I need more of these to use for things like keeping track of the Devourers running around, etc.

Reworking The AI And The Last Of BaseInfo, Part 2

• A variety of code translations to the new format that are not worth detailing, and finished the work of getting rid of Ex_MinorFactionCommon_GetPrimitives.

• Fixed an issue where we had apparently wished for the Macrophage to have the same "Seed On Nomad If Possible" option that the Dyson Sphere had, but the setting was partially missing.

• The central AI BaseInfo now has direct references to all of its subfactions in the form of handy names like SubFac_CPA and SubFac_Warden and so on.
  • This doubles as a self-checker to make sure that an AI faction is in proper health with all the expected subfactions and no extra subfactions hanging around.
  • This then lets us get rid of a lot of things like GetCPALogicForThisAI(), happily.

• Added a new AISubFactionBaseInfo, which acts vaguely like the LoneWandererFactionBaseInfo class.
  • This is now the base class of all of the AI subfactions, and it makes the act of finding the parent faction and its data super trivial, and also ensures proper linkages or it has a centralized freakout about that.

• All of the various factions in the game and DLCs now have proper BaseInfo classes, and the xml is all linked up for BaseInfo and DeepInfo.
  • As part of this, the three types of zombie factions now share a common base class, like how the lone wanderers and the AI subfactions do.
  • Also, the Svikari have become a subclass of DarkZenith rather than using DarkZenith quite so directly.

• All of the ISpecialFaction classes from DeepInfo have now been renamed to their new proper names, and inherit from ExternalFactionDeepInfo.
  • This leaves a lot of other fixing to do, but it gets things in proper place at least. This is things like SpecialFaction_Human becoming HumanEmpireFactionDeepInfo.

• The above two items also involved splitting a number of files, and setting up some template empty files for various factions, etc.
  • This gets all of the major objects into place, but just not yet with the correct formatting and linkages and so on.
  • It was about an hour and a half just to get things organized like this, and would have taken a lot longer if I did it in small stages. There are still probably tens of hours left on this conversion, but having things working by next weekend is looking increasingly feasible.

• At this point, there were still 374 lines in BaseInfo and DeepInfo that referred to ExternalDataRetrieval, which no longer exists.
  • A targeted set of 64 global search and replace commands got all of those fixed up. (Yeah, it was odd to me that it was exactly 64, too.)
  • Total errors reported by the compiler in BaseInfo are now down to 224, but this was fixing up a lot of others that had not been reported in that number yet, or which are in DeepInfo. The IDE tells me there are 3269 errors in all at the moment, but that's both overcounting and undercounting at the same time. It's progress, anyhow!

Reworking The AI And The Last Of BaseInfo, Part 3

• All of the general cases where I could update "Implementation is" or "Implementation as" in a global search and replace way are now fixed (I didn't count how many it took, but a couple dozen operations to get about 140 items corrected). I'll get the rest as I go through files in more detail.
  • So this leaves me with 220 errors reported by the compiler for BaseInfo. I spent a lot of the morning doing payroll, so let's see if I can knock out the rest of BaseInfo by the end of today. That really would be nice.

• All of the BaseInfo objects for factions that are only allowed to have a single version now have a static Instance on them for quick access.
  • Worth noting: if those factions are not active in the current game, those instance variables will be null! This is by design.
  • These classes also now all inherit from IExternalBaseInfo_Singleton, just to make it a bit more clear again what the intent is.
  • This applies to: Dark Spire, Fallen Spire, Astro Trains, antagonized dyson sphere, HRF, Risk Analyzers, tamed and enraged macrophage, naturalobject, ai reserves, instigators, all three zombie factions, outguard, zenith miners, dark zenith (which has had its class split from its core, for later benefit because of svikari), nomad planets, territory dispute parent, necromancer, and templars.
  • Might as well have a consistent pattern, and this makes converting a lot of other code easier.

• All of the "WAVES_TODO" markers are happily able to be removed, as they will now work properly on clients without any further effort thanks to this general restructuring of data.

• Spire relics can now be moved around properly by multiplayer clients. That was something I hadn't even realized was broken by my changes in July, but it was -- a great example of why this much more intentional split, as time-consuming as it was, is so much better in the long term. I can clearly see in the code itself when something is wrong, without having to wait for testing an exception to pop up.

• The ClientSafeExternal namespace has been removed, and its contents returned to External, because the separation is now in between dlls rather than just namespaces.
  • This is something that is no longer needed at all, and was a half-measure where now a full measure is being taken.

• Marauder outpost descriptions will now have proper info after you load a save and before you unpause the game.
  • Right now I'm making dozens of code changes that are not really worth detailing since they are just tidying things from one format to the other, so instead I'm mentioning things that happen to get fixed along the way.

• Added a new DoubleBufferedDictionary, which works just like the DoubleBufferedList does. Turns out this is needed for various factions, marauders among them.

• Fixed up some code from UpdatePowerLevel() in a ton of factions to keep them from having values that flickered for a few microseconds at a time. It probably wasn't a problem, but why tempt fate.

• The scourge now use vastly less GC every frame in general, and also less when showing their details for the UI.
  • Several minor formatting issues were fixed with scourge being displayed on the UI, and additionally it now shows the list of spawners like it does for armories. Before it meant to, from the look of it, but it only actually showed the count.

• Added a new DoubleBufferedArray, which is now used by various factions, including the scourge.

• Ported over a huge amount of the scourge data and the logic for its data aggregation.

• Added a new DoubleBufferedValue, which is basically like the other double-buffered things but just for a single integer or similar. This is also now used on the scourge data.
  • There are many other ways to handle this same problem, the simplest of which is a working variable. But in looking at the scourge class, where there was a mix of local variables and naked assignments, it's easy to see that that's hard to maintain properly.

• At this point, down to 191 compiler errors reported in BaseInfo.
  • It's worth noting that the bug count in DeepInfo falls dramatically with each of these checkins, too, but those are not numbers I'll ever actually see as high as they are, since I can't attempt a true compile of DeepInfo until BaseInfo works.

• Macrophage turns out to be a really complicated class in terms of how the base, tamed, and enraged variants relate to one another.
  • There were a number of parsing things that I don't think were quite right, and probably led to things like the enraged version always having an effective intensity of 0.
    • I've adjusted things so that tamed still has the intensity of 10 that someone put in there (wow, nice), and the enraged version has the intensity to match the highest regular macrophage now.

• Also to support the macrophage, because they do this sort of cross-faction aggregation to fill what used to be called SporesPerPlanet, but which I'm now calling CrossFaction_SporesPerPlanet, I've had to make a number of new little things.
  • For one thing, the way that this data structure gets filled across faction lines is now vastly more efficient (blind Add calls and one-way caching, versus expensive Contains lookups).
  • This then required me to added a new Stage2APostAllFactionAggregating method, though, that lets me run the final SwitchConstructionToDisplay() on the DoubleBufferList array that is CrossFaction_SporesPerPlanet. That needs to happen once per cycle, after all of the other factions have done their stuff.

• While I was at it, I noticed that the central logic had not yet been set up to actually call the BaseInfo Stage1 and Stage2, they were just calling DeepInfo Stage1 and Stage2.
  • Now it calls both, although I'm unsure if I'll keep Stage1 and Stage2 on DeepInfo. I see pros and cons to that, but it will likely just remain, but be empty for all of our built-in factions. I could see some modders needing the feature.

• Added a DoubleBufferedArcenSparseLookup, because I know how fond some programmers are of the ArcenSparseLookup's extended capabilities.
  • I'd prefer not leaning on this too terribly much, because it comes at a bit of an extra cost compared to a regular Dictionary, but such is life. Sometimes we need those capabilities.
  • As an aside, I built some of the "freedom to call into keys that don't exist without error and get a default value back" capabilities into all of my DoubleBuffered structures. It's one of the favorite advantages of ArcenSparseLookup over Dictionary, and it's something you can use with DoubleBufferedDictionary without having to go all the way to DoubleBufferedArcenSparseLookup.
  • In my opinion, the only real good reason to go to DoubleBufferedArcenSparseLookup is if you need it to be a dictionary but also have the ability to sort, like ArcenSparseLookup has.
    • I almost switched the list of Telia to being a DoubleBufferedArcenSparseLookup because of how it is used in SporeDescriptionAppender, but then realized that I could accomplish what I wanted just by using a foreach rather than a for statement if I kept it DoubleBufferedDictionary. This will be a tad more performant, so that's what I went with.

• Also, I'm no longer keeping any concept of "Wilds" for macrophage. That seemed to be a mix of enraged and normal macrophage for some reason.
  • I now have a singleton for both tamed and enraged, and then a full list of the regular macrophage. They can be queried separately or in series. Sticking the enraged in with the regulars didn't sit right with me, it felt like a problem waiting to happen.
  • Unlike pretty much every other faction I've seen so far, Macrophage is having to have a bit more surgery to parts of its actual logic, partly because it's just that old, and partly because it's been maintained by a revolving list of programmers over the years. Please let me know what issues you find once we're back to being able to play on the new builds again.

• And with that I'm down to 161 stated compiler errors in BaseInfo. But once again, the day has had lots of progress on DeepInfo, so it's not going quite as slowly as it seems to by that error count number alone. My hope, knock on wood, is that DeepInfo will just fly by fast after BaseInfo is all finished up.

• A bunch of fallen spire stuff got moved over into BaseInfo. That was the easiest yet because it's more or less my code, most of it.

• Same with the Zenith Architrave, mainly because they're just a really modern faction in terms of their code.

• Also the same with the Migrant Fleets faction, which is very well done from what I see so far.
  • One thing I fixed was converting cachedMovementTargets from an ArcenSparseLookup into a ConcurrentDictionary, because both the UI and the long-range-planning threads use this, so a cross-threading conflict was otherwise just a matter of time. Now the existing logic will work fine.

• Down to 147 reported compiler errors in BaseInfo.

• Some more general fixes and translations, getting the error count lower.

• Fixed a variety of xml filenames that said BaseInfo instead of DeepInfo. It didn't hurt anything, but was confusing.

• Trying to get the host to trigger SharkB (which must happen in DeepInfo) in response to something that happens in a gamecommand that the client also needs to execute (in BaseInfo) is an early and interesting test of a pattern that is not very common thus far.
  • Overall most of the control code is in DeepInfo, and so we don't need to "call up" like this. Instead, deep just calls down and can see everything, which is the idea. But these exceptions do exist, and so having a formalized way of handling them has always been part of the plan.
  • In this particular case, we're getting our first DeepInfo world-attached object, which I'm calling GlobalGeneralDeepInfoCommandHandler. It's just for a variety of general things, over time.
  • Additionally, the ExternalData_World base object now has its first method of interest, which is called DoAfterEntityDeath(), and lets you pass in a string ActionCode to specify one or more types of things.
    • The flexibility of this is that we can have more than one thing handled in our central library, but also modders can set up their own objects of this sort and also call them. We don't need one object per type of on-death call, since that would be excessive.
      • It's important to know that these are called manually, from the bottom-up, when the sim code detects something it wants to react to. This is the same pattern SharkB was already using.
  • There's another "TEACHING_MOMENT" piece of code in there (I've added several of those today, as well as flagging some comments of Badger's that were teaching moments as being such so people can search for them).

• Imported a bunch of Dark Zenith logic (the stage 2 aggregation). This was more complex because it was having to do way more work since it did not have double-buffered collections in the past.
  • Some things also were able to be simplified because of the new architecture where you can actually get planets and squads and such immediately on deserialization, whereas before this version you would not. So for instance, serializedFimbulwinterEligibleTime is able to be ditched, and we just write straight to FimbulwinterEligibleTime, saving ourselves some hassle.
  • Same with OriginalPlanetIdxs going away, and just having OriginalPlanets, instead.

• Okay, that's it for today. Down to 134 errors in BaseInfo. It took a while to convert over the Dark Zenith, mainly because it's such a big complicated class. But thankfully, the newer logic (double buffers, mainly, but also the deserialization order) really make the code a lot more legible and cut down on extra work.
  • I'm sure it's going to be mildly frustrating to Badger that he didn't have those tools when he originally coded this faction, but I wouldn't have thought of adding them without seeing the same patterns over and over again and realizing I could simplify it based on that.

Reworking The AI And The Last Of BaseInfo, Part 4

• GameCommands used to be all in one giant monolithic file. They have now been split out into a variety of files, by loose purpose or association. This makes it a lot easier to find a specific GameCommand, or to look at several GameCommands that are related to one another.
  • As part of this, and because I needed to, two of the GameCommands (both dealing with changing the galaxy map, as part of ZO) have been moved to DeepInfo, where they will only be run by the host, and where they can access data and methods that only the host would have.

• The dyson antagonizer faction logic has been ported properly between base and deep info now.
  • Same with the the main dyson sphere faction, including wiht the ability to handle multiple spheres around the galaxy, now using the same pattern for that which macrophage are using.

• All the cheat codes work again (yeah that was a thing).

• All of the description appenders work again.

• The central faction processing code is now split out fully into DeepInfo, which is excellent. It's still kicked off from logic in BaseInfo, but the actual execution (only on the host) is in DeepInfo now.

• A number of faction methods that used to be FOR UI ONLY and very problematic if used otherwise are now using ThreadStatic internal markers on their lists to make themselves threadsafe. These are now labeled as being threadsafe, and can be used by the UI, main thread, LRP, whatever -- all at the same time, even!
  • The various notifications that used these ui-only versions of the methods have been updated to point to the new ones, and a TEACHING_MOMENT flag has been put in next to a comment about ThreadStatic at one point.

• The HostOnlyJournalsAndAutosaveAndSimilarHandler has been moved into DeepInfo.

• AIUtilityMethods has been moved from DeepInfo into BaseInfo.

• Fixed a likely bug where nanocaust intel entries have not been shown ever since frenzy fleets were removed (back around the time of fireteams).

• Imported some various things from deepinfo and sappers and such into BaseInfo that were going to move here soon enough anyhow.

• Okay! Well, BaseInfo now compiles, which means that I'm able to get going on DeepInfo properly.
  • There's a lot to translate over there, but I've been steadily doing that as I went, too, so we'll see how long this really takes.
    • There's a fair number of things which are kind of mindless search-and-replace things, like a lot of the ExternalData faction methods no longer have the faction being passed in. That's now handled as AttachedFaction as an embedded field in the class itself.
    • To trick the IDE into fixing over 800+ references that would otherwise have to be done by hand, I simply did a search and replace for the methods in question, which then made them have references to (typically) "Faction faction" that no longer exists. So I made it exist, in the base class. Then used the IDE's rename function (F2) to rename that to AttachedFaction, and presto. Then just delete that temporary variable you made. You can do the same in your mods, if you wish.
    • The reason for gettting rid of Faction as a property on this methods is that it creates ambiguity and the potential for errors. It's just cleaner not to have it.
  • At the moment, I have about 200 compiler errors in DeepInfo, and 2934 errors in visual studio, which is probably vaguely accurate. It's several thousand lines that need correcting or adjusting in some fashion. There's still over 240 references to SpecialFaction_[Whatever], and those will take a mixture of search and replace (when it's checking is or as again), and manual adjustments as I link up the base and deep infos of each faction in particular.
  • Based on the speed of things lately, and the somewhat unknown volume of code that has to be corrected, I think this will easily take me another 2-3 days. I worked through almost all of last weekend, and I hope not to do that again. The picture will become more clear tomorrow and the day after as I get further along.

• Okay, I couldn't quite leave it for the night. I got autosaves working there's a new AutosaveHandlerDeepInfo class in DeepInfo which replaces what used to be there.
  • Really, all the code is the same, but it just doesn't have any chance of going to a client, and isn't attached to any random factions. The translation of this was literally under five minutes, nice.

• Got a little over halfway through setting up quick-links between the DeepInfo objects and their BaseInfo counterparts. This includes setting up the DeepInfo objects as singletons where need be, for handy access.
  • As I was doing this, I also made various other corrections and shifts to code as I saw it, and in all the number of errors in DeepInfo are plummeting. There's still plenty there -- visual studio says 2400, which seems about correct at this point -- but the code is thinning out thanks to being better organized (data and data access in one place, logic in another place, and in consistent order and consistently organized in both).
  • I don't want to get ahead of myself, but after a few weeks of being overly optimistic on schedules (or is that just a terminal case for me, one could wonder), it actually looks like I was -- potentially, and majorly knock on wood -- overly pessimistic, this time.
  • THAT said, assuming I do come in ahead of what I was projecting, that's still just going to be leaving more time for me testing for runtime errors and fixing those. It's not like as soon as this compiles it's good to go. I've been coding for almost three weeks now on very long days and over weekends without testing any of it -- because I couldn't -- and if you're a programmer, you know how horrifying that is. All things considered I think I've been very clean with my work in this time, but I expect to find some major gaffes.
  • There's a vague, outside possibility I could have this compiling tomorrow. At the current pace I have observed, it's not outside of the realm of possibility. It's not hugely likely, either, but it's what I'm going to shoot for. I want those two days prior to the weekend for testing and fixing the runtime.

• Removed some probably-unused-and-unworking code for ships warping in and out over distances. As Badger notes, we have way cooler things to look at these days, and we haven't noticed this actually functioning for a few years now. It was taking up data space, though, and eating some CPU processing, so out it goes.

Moving Into DeepInfo, Part 1

• ModdableGameCommandExecution has been moved out of BaseInfo and into DeepInfo, after seeing a bit more how it is used and thinking about it.
  • Some nice comments have also been put in there, with two TEACHING_MOMENT entries: one about what the moddable game command things do, and another about how to get data from the host to clients in general.

• Thinking of trying to minimize multiplayer bandwidth, I decided to make a second Stage3, which does happen in BaseInfo and thus on clients.
  • We want to be relatively careful what logic we put in there, to not make MORE desyncs. But for simple logic in particular -- like zombies attritioning -- it's going to be way more of a desync if this DOESN'T happen on the client.
  • There's a TEACHING_MOMENT note in there for one of the zombie factions.

• I also changed my mind and am removing DoPerSecondLogic_Stage1Clearing_OnMainThreadAndPartOfSim_HostOnly and DoPerSecondLogic_Stage2Aggregating_OnMainThreadAndPartOfSim_HostOnly from the DeepInfo.
  • We just don't need it, and I don't think any mod that would need that is structured properly. These methods are specifically design around working on the data that is held in BaseInfo, so we don't need a DeepInfo version.

• Somewhat related, Stage0 -- aka DoPerSecondLogic_Stage0Clearing_OnMainThreadAndPartOfSim_OncePerFactionTypeEvenForFactionsNotInGame_HostOnly -- has been removed from the game.
  • That was simply not something that was ever needed at this point. We have better ways of handling similar things, and the new Stage2A is part of that.
  • Specifically, also a lot of the double-buffered lists and similar make this completely unneeded.

• The remaining 16 factions (of of 37) have now been fully linked up from DeepInfo to BaseInfo. I was feeling really happy and motivated last night and got the first 19 out of the joy of it.

• Down to 141 direct compiler errors at this point, and 1782 visual studio errors (which is likely to be pretty close to accurate). There's still a chance I can knock the rest of this out today, but if not then tomorrow is pretty much a certainty.

• Added a new ExternalFactionBaseInfoRoot, which now sits under all of the BaseInfos for factions.
  • It was becoming really clear that there were some commonalities that needed to become shared, plus a few things that needed to be able to happen on a schedule that is centrally controlled (to avoid some mapgen errors I suddenly realized were possible thanks to a comment by StarKelp in one of his faction files).
  • First of all, we now are ensuring that DoRefreshFromSettings() and DoGeneralAggregationsPausedOrUnpaused() get called from mapgen.
  • Secondly, something that was just an optional convention that I had been using (that being DoRefreshFromSettings() itself) is now a required part of faction base info. It allows for great flexibility without overtaxing the processor.
  • Next, the base implementations for GetRaidDesirability() and GetRaidTraversalDifficulty() now have somewhere appropriate to live.
  • Lastly (for now), this allows us to implement a proper GetFireteamById() rather than just GetFireteamBaseById(), mirroring how faction code used to work.

• The call to link the faction externaldata into the factions, and also to call DoAnyInitializationImmediatelyAfterFactionAssigned(), has been moved around substantially.
  • The idea is that we want more of the data available as we're deserializing savegames, but when it comes to mapgen we also need to be after-all-factions-but-before-actually-doing-any-mapgen.
  • This should save quite a few headaches for me from after I get this compiling again (I would have had to make these order of operation changes in general).

• GetFireteamById() has been moved over into BaseInfo where it belongs.

• All of the pathfinding logic has been adjusted to be where it should or call the new locations of relevance.

• Added a new AIRelentlessAndBorderAggressionFactionBaseInfoRoot, which is a common ancestor now to AIRelentlessWaveFactionBaseInfo and AIBorderAggressionFactionBaseInfo, mirroring the setup we have in DeepInfo.

• Stage2 logic was still hanging around in DeepInfo for the following factions and has been migrated over (along with a bunch of other data):
  • Cross Planet Attacker AI subfaction.
  • AI Relentless And Border Aggression root.
  • AI reserves
  • six more to go, at the moment, but AI reserves took... a lot.

• The AI Reserves faction no longer has dual lists for "sim" and "lrp." It just calculates their data with double-buffered lists, instead.
  • This also includes a new teaching note, and some code that has been adjusted to exist in DeepInfo only, along with some guidelines on why exactly it works this way versus some other way.
  • In other words: how do you know which place a variable or collection should go? This lays it out, with the example of DeepstrikeEligiblePlanets as a guide.

• Added a new SortThreadsafeCopyOfDisplayList() to double buffered lists, along with an optional TryClearThreadsafeSortedCopyOfDisplayList().
  • See, the problem with the double buffered lists is that normally you sort them ONCE right before you finish constructing them and flipping them active.
  • However, after they are the display list, any number of threads might be reading from them, so to sort then would not be threadsafe in the slightest.
  • To work around that, this method allows any thread to sort a copy of the display list, in a perfectly threadsafe way.
  • This will create some extra data in RAM, very small amounts, which you can clear with TryClearThreadsafeSortedCopyOfDisplayList() if you feel like it. Or it will just be cleared the next time you call SortThreadsafeCopyOfDisplayList() on the same list from the same thread. Either way.
  • This is another addition that has been made thanks to AI Reserves and some of the complicated things they want to do. Their list of eligible planets gets sorted over and over based on distance from other planets they are thinking about, but this is all on a super low priority background LRP thread, so it's fine... except for the need for thread safety.
  • Yep there's another TEACHING_MOMENT in the code about this.

• Split out some of the waves logic into a new WavesHelper static class that variable factions can use. AI Reserves needed it.

• "Set the preferred AI faction" logic has been removed from the remaining locations where it existed, since that is now obsolete and handled as parent/child factions. There were over a dozen copies of the same lengthy code all over the place.
  • That then left me with 21 references to PreferredAIFactionIndex, which have also now been converted over.

• Four instances of DoOnFirstSightingOfFactionByPlayer needed adjusting, and have gotten it.

• Rather than being a static class, WavesHelper is now a singleton.
  • This allows us to extend that singleton in DeepInfo, and add DeployComposition into there where only DeepInfo can reach.
  • We use the same pattern for FactionUtilityMethods, if you're curious.

• Necromancer stage 2 and various data has been ported to BaseInfo properly.
  • Good grief this was complicated. It provides a couple of TEACHING_MOMENT items, too.
  • This also shows one of the examples of how we no longer need to precalculate a string (and hit the heap when we do), by instead changing a ToString() method to one that takes in an ArcenDoubleCharacterBuffer and writes to that on-demand.
    • A lot of other classes also do this exact thing now, but SkeletonVariantPercentagesByFleet_ToString() and similar provide a great example of it in action.

• The necromancer faction is mostly working again in general, but it has uncovered a bit of a problem in my logic with hacking in particular.
  • Right now, hacks are only scheduled to run on the host, but there's a lot of info about "can we start a hack" and such that the client needs to be able to see. I'll have to deal with that.
  • The necromaner sacrifice unit hack in particular is also problematic, because it executes on all clients and the host (well, or just on the host), but is using local-to-one-machine data (selection info). That's going to require some refactoring on my part to figure out how to handle cases like this.
  • I have a similar problem coming up with the sub-types for the sentinels, warden, hunter, and PG. They essentially need some of their stuff in BaseInfo, and some other bits in DeepInfo. I mentioned this last week actually as a concern point that I didn't know how to solve at the time. At the moment, I have a fairly decent idea of a way to handle it, but I need to examine the code a bit more first.
  • I'm down to 2009 errors at this point. Not sure if I'll finish tomorrow or not, at this rate. Which really bites, but it's the nature of the beast, I guess. At the moment I'm fixing some things that were always going to work incorrectly in multiplayer anyhow, so it's not just architecture for the sake of nothing.

• Moving hacking back into BaseInfo. I'll build in hooks into DeepInfo as needed, but the vast majority of this code really needs to be run as part of the sim.
  • One thing that is interesting about this whole revamp process is that the way that data is available to the UI or where it needs to be is really becoming increasingly clearer, even to me. I knew in pretty great detail what I wanted to do, but there have still been many cases where my intuition was wrong. This is one of them.

• The necromancer "sacrifice" code and hacks has all been removed. That was going to be complicated, but Badger noted that was deprecated in favor of the Rift hacks. This saves a lot of time! But still helped to identify a problem with how I was handling hacking in general in DeepInfo, so was useful.

• Now that hacking has been moved back to BaseInfo, there were several hundred errors that needed to be fixed (they were going to need to be dealt with anyway, but now they prevent BaseInfo from compiling again until I fix them).
  • As part of this, moved around a lot of things that are not really worth detailing, and fixed a number of references that were still old-style. It's hundreds of lines of code, but there's nothing really surprising in this batch worth reporting in detail.

• TerritoryDisputeTeamFactionDeepInfo has had its Stage2 merged into BaseInfo properly. This was needed for some of the hacking stuff, but also just required in general.

• Added ConstructionContainsKey() to DoubleBufferedDictionary, and ConstructionHasKey() to DoubleBufferedArcenSparseLookup.
  • This is needed to support some of the territory dispute logic.
  • Ah, and it turns out we also need versions for the display dictionary, so those were also added.
  • And it turns out that we also need DisplayContains() and ConstructionContains() on the DoubleBufferedList, so that's also in now!

• Added a new table called ExternalDeepLink, which lets us define singletons in BaseInfo, which can then be initialized via xml, and then assign their reference back to BaseInfo from DeepInfo.
  • Essentially, this answers the question "how can a modder or faction designer add arbitrary code calls from BaseInfo to DeepInfo without modifying Core code?
  • I actually already use this sort of pattern all over the codebase, and have since 2018. Lots of external classes register themselves with more information than the raw table that loads them has access to.
    • This is kind of complicated to explain in a written note like this, so it will be much better to show examples. I'll have some teaching moments in there, too.
  • The first and simplest example is ExoGalacticDeepLinkRoot / ExoGalacticDeepLink.
    • ExoGalacticDeepLinkRoot is an abstract class defined in BaseInfo. It defines what methods it has, and has a static Instance.
    • Xml states to load ExoGalacticDeepLink, which is in DeepInfo. In its constructor it assigns itself to the Instance variable on ExoGalacticDeepLinkRoot.
    • After this, any BaseInfo code can just call ExoGalacticDeepLinkRoot.Instance.YourMethodHere(), and that will be directed straight to DeepInfo and do whatever it is you want it to.
  • Ultimately it's a really simple pattern, but it's just hard to explain in text like this.

• We now also have a DysonUtilityMethodsDeepLink, for supporting a number of hacks.

• Fixed an old and undiscovered pooling issue with ZenithArchitravePerUnitBaseInfo, where it would get into a funky state after being hacked.

• There's also now a NecromancerDeepLink. And a MacrophageDeepLink.
  • These are stupidly simple to add (just a couple of lines of code), and allow for full control of how you can talk from BaseInfo to DeepInfo. Really pleased with how this turned out.

• Stopping for the night. At this point, all of the hacking stuff should work now, and is back in BaseInfo but using DeepLink calls where needed to call into DeepInfo-only areas.
  • There are presently 92 compiler errors reported for DeepInfo, which is getting pleasantly low, and last I saw the visual studio error count it was just under 2000 now.
  • This is definitely a tense schedule, but there's a substantial chance I could get this done tomorrow -- to the point where I can start running it and finding all the runtime errors on Friday, anyhow.
  • We'll see how it goes. Honestly there is deceptively high numbers of things that need attention or conversion, but the new DeepLink tool solves a HUGE number of the problems that I was otherwise having trouble figuring out a graceful solution to.

Moving Into DeepInfo, Part 2

• BuildTerritoryCollection() was called a variety of times in the territory dispute faction classes.
  • This basically just made a lookup, by faction, of what territory each faction controlled. This sort of thing used to be required, but now we can access the factions so easily and they all have double-buffered territory lists already. So we can, happily, skip this and use data we already have.
  • Getting rid of this sort of thing (which previously existed because of either threading concerns or because of difficulty finding the correct other classes' data) is a major win of the new framework. Saves a lot of CPU, and in many cases also some minor RAM GC churn.
  • Note that I'm not particularly calling out the territory dispute faction anymore than I've meant to call out any other faction. They're all coded very well in the limitations of the old architecture. But at various points, various factions illustrate different benefits of the new architecture.
  • Added a new DoForAllTeams() method to TerritoryDisputeParentFactionBaseInfo that also helps offset the need for this method, and methods like BuildTeamCollection().
  • GetColonyShipTargets() and similar are also now a great example of how to reduce CPU usage and GC churn by using threadstatic lists. These are thus also threadsafe methods. Please see the teaching_moment related to threadstatic if you want more info.

• The "Stage4 Advanced Allegiance Code" that I added a while back to support the territory dispute faction has now been moved into BaseInfo.

• Added an extension method for DoFor() to Dictionary, giving it that handy same method that ArcenSparseLookup contains.

• The territory dispute factions (both the parent and the subfaction) are now fully ported over, including its notifications.
  • This was a very interesting use case, because it's very modern and written very well, but also has some unusual stuff like the sub-teams. It's probably pretty obvious, but the architecture has been continually expanding as I see what each faction author wrote in detail and how I need to improve my central tools for them to do their same work in less code.
  • There is a "StarKelp_Question: hey, I was unclear on this logic." note in there at one point, which StarKelp you should feel free to either correct or confirm whenever you like.

• Down to 63 compiler errors in DeepInfo, and 1818 errors reported errors in visual studio. I'm determined to get through this today so that it all at least compiles!

• Stage2 logic has now been moved over for the last four factions that needed that pulled out of DeepInfo and into BaseInfo:
  • Dark Spire (also got rid of some ForUI variants of lists, and some list variants "for all factions" when there can only be one dark spire).
    • This also has a great example of some of the old-style codebase stuff, where it had to have a count of spire cities for itself. That's been removed, and it just asks the spire, which is able to say that really clearly now.
    • Also removed the pointless FactionIndex, which was just referring to itself it turned out; that was actually hard to be sure of for a bit. This faction is _really_ old. At one point we probably legitimately needed that to know how the external data connected in to its faction.
  • Templars (also got rid of some ForUI variants of lists).
    • A teaching_moment has been added in there, along with a bit of logic shifting on how rifts are spawned, essentially working around a cross-threading error that would happen but rarely, and it explains how to notice it. The double-buffered lists make it a lot more obvious, since you should not be calling add to a construction list outside of the construction process, and you should NEVER be adding to a display list.
  • Zenith Architrave (removed many ForUI variants, also made serialization of territory and planets ever taken direct now rather than first sent to IDx).
    • Also got it so that their BaseInfo now properly cleans itself up before pooling. (When it was originally coded, it wasn't meant to be pooled.)
    • The way that territory is calculated, and a number of other things beyond that, have been simplified a lot, and hopefully not mangled in the process. Before there were some _severe_ order of operations constraints across multiple passes that these revisions aim to make just happen in order, 1, 2, 3, much simpler.
    • Gah, this one was really a cluster. This is new top prize for hardest faction to transition over. I don't think I messed anything up, but Badger will have to review it, sigh. The logic is probably easier to follow now, though, on the bright side.
  • Nanocaust (okay, this one was shockingly easy).
    • Got rid of yet a bit more really super old deprecated code, and repointed a few variables and it is what it is.

• Moved over more notififications logic from DeepInfo to BaseInfo, which is one of those really lovely things that lets us have functionality that works properly in multiplayer:
  • 2 from Dark Spire.
  • 2 from ZA.
  • There are still 10 more I need to grab.

• Many various other redirections from old code spots to new, nothing terribly exciting with most of that.

• Dark Spire faction is now fully compiler-error-free.
  • Same with nanocaust and Zenith Architrave, and a bunch of other factions.
  • We're down to 44 compilter errors now, and 1594 in visual studio.
  • The Zenith Architrave and the Territory Dispute factions were both way more challenging than I had expected, and the largest faction of them all, Dark Zenith, still needs a lot of translaton work, as does the AI. We'll see what I can do...

• AI Reserves notifier moved over from Deep to Base.
  • The last bits of the AI Reserves DeepInfo errors are also resolved.

• Many various other random small improvements all over the place. This is going to last well into tomorrow, sigh.

Moving Into DeepInfo, Part 3

• Many various cleanups and fixes to the new format.

• Added a new DoFor on GamEntity_Squad that works on a List<>, like the one that works on ArcenLessLinkedList<>.

• Added a new HasSimStepRunSinceGameLoad on BaseInfo for each faction, which is set after Stage4 has run on each of them.
  • This is really useful, because Nanocaust had some stuff that it really didn't want to do on long-range-planning threads (it would make wrong choices) until at least one sim frame had elapsed.
  • While looking at that, it became really obvious that this is a problem that other factions would now share, if they didn't already (most already probably did).
  • So now the long range planning threads are blocked from running at all until the first sim-step pass has happened since the game was last loaded (savegame load, quickstart, game start, whatever).
  • To be clear, this is a delay of about 100ms, so it's not really noticeable. But it will make it so that the initial run of long-range-planning is never acting strange because of a lack of data.

• Nanocaust data is now properly cleaned up when it goes back into the pool.
  • Also removed yet more old frenzy variables that are not used since the advent of fireteams.

• The nanocaust now uses double buffered lists in place of the linked lists or other formats.
  • Also, the nanocaust has some cross-faction lists, along the lines of what the dyson spheres do.

• The way that the nanocaust aggregates its data has been converted into the modern format. Well, the NEW modern format, but also the modern-since-DLC1 format, where Stage2 is used for aggregations.
  • Whoof, some of its logic in general, such as for attrition or for constructor upgrades, has been moved to baseinfo and converted a bit.
    • I had to go back and check the older versions of the code on some of this, because the constructor upgrade logic applies to non-constructors and that is how it was coded. I'm leaving it because that seems to be what has been the historical function even if that's not what the code names would seem to indicate was intended.
  • The nanocaust faction is now fully up and ready to run (or at least compile). Ultimately the surgery was not too bad, and things should be more clear than they were before when looking at that code.

• Moved the remaining notifications over from DeepInfo to BaseInfo, along with the code for generating them:
  • 5 from Fallen Spire.
  • 2 from Dark Zenith.
  • 1 from Nomad Planets (that's the last of any compile errors with nomad planets gone!)
  • 1 from Zenith Miners (and with about six other corrections, all compile errors with the ZM are gone also!)
  • None from sappers, but some outer scaffolding that was prepping for them has been moved.
  • At this point, notifications should again work properly on multiplayer clients, which was one of the sticking points that led to this whole overhaul in the first place.

• HasSimStepRunSinceGameLoad has been renamed to HasPerSecondSimRunSinceGameLoad.
  • This is already what it did, but it's now correctly named. It's important to note that the long range planning threads won't start for an entire second, not just 100ms. This is of no gameplay significance from a player point of view, but it keeps the data and the factions happen.
  • Both marauders and HRF had nanocaust-style "have I loaded yet?" checks to enforce this sort of thing in an ad-hoc manner. As suspected, this affected a lot more than just the nanocaust.

• FireteamRegiments have been moved out of the BaseInfo and back into DeepInfo.
  • After consultation with Badger, it became clear that this is purely host-only information, and not something that would need to be seen on the ui in a non-debugging fashion. Therefore, into DeepInfo it goes, to avoid confusion.

• A number of things on the Marauders faction that had been keeping an index of planets now keep the actual planets since that is now possible. Makes for some added simplicity.

• After quite a number of changes, marauders now fully compile.

• The HRF now also compile, with the changes to make them use planets directly rather than planet indices.

• IPraetorianGuardTypeImplementation, IHunterFleetTypeImplementation, and IAIWardenTypeImplementation have all been removed.
  • This was a really funky way of setting things up in general, and Puffin had already made a todo in the code stating that we should move this into xml long ago. As he noted, it took him forever to find these balance levers.
  • In the more recent short term, this whole design just really wasn't fitting with the new setup anyhow, so it was doubly time to adjust this stuff.
  • AISubFactionDeepInfo now also has been dismantled, since this was just a vehicle for the three above.
  • Xml for all of the above are now present, and you can change a lot of things very easily now that were previously very hard to do! As a modder, this can majorly let you change how the hunter, warden or praetorians act.
  • A lot of the code that was tied to this whole set of structures needs to be reconnected, but I have to get some other things in place first.

• Lots of other random minor fixes and rearrangements. That got rid of the last of the low-number compiler-blocking errors... revealing a further 1446 errors. That's about what visual studio had been predicting. Woo...

• Got rid of the last 5 references to "is SpecialFaction_", down from hundreds initially.
  • That left 8 more references to "as SpecialFaction_", also down from hundreds initially. Those are also gone now.

• The AI type implementations have been moved back into BaseInfo rather than DeepInfo (sigh), and split out into multiple files so that they are easier to keep track of.
  • Technically this stuff is only used from DeepInfo, but it's a real pain to try to sequester it there, and there's nothing about the data of this that would break any of the client/host rules, so... fine. BaseInfo it is.
  • The ability to add per-sim-step or per-second logic on AI types has been removed, incidentally. That was never used.

• A bunch of the AI Sentinels DeepInfo methods were static for some reason, potentially because we thought we might want to cross-call from other locations or something.
  • Thankfully we were not actually doing that (that would have been a nightmare), so these have been made regular methods and are already private, so they have quicker access to the data of the AI they are dealing with.

• The way that AI spending ratios are assigned to budgets no longer generates a bunch of garbage for the GC to take care of.

• DoOnAnyDeathLogic_HostOnly on factions has been renamed to DoOnAnyDeathLogic_MyFactionUnitsOnly_HostOnly, because honestly it was not clear if that was how it works or not.
  • There's another method called DoOnAnyDeathLogic_FromCentralLoop_NotJustMyOwnShips_HostOnly which runs on any unit, so it's vaguely implied that the above one is only for its own ships if you see that, but it's not clear enough for my taste.
  • Because of that, there's a number of cases of indirection (that existed prior to this rework) that I'm able to simplify away. Just working more directly with object references makes things faster, and makes the code more readable.

• Fixed what seems to have been a bit of a glitch with macrophage not getting any metal income from them killing enemies in some circumstances -- maybe if they did not have a spire telium?

• Zenith Trader now fully compiles.

• The way that the macrophage keeps track of its entities has been updated to a more modern interpretation.

• Added a new UniqueNameForFactionToAvoidThreadConflicts, which defines intentional collisions for "don't run these faction's LRP threads at the same time."
  • This allows us to ensure thread safety even amongst factions with inheritance chains, like the macrophage, lone wanderers, dark zenith, and so on.
  • We had something kinda like this in the past, but it wasn't as good and didn't cross types like this. It had made it so that rare LRP cross-threading issues were possible, that now are not.

• The macrophage code now compiles, and is a real showcase of how a complicated (but understandable) faction should work in the new code framework.
  • There are a number of teaching moments, and a number of threading things that get explored here.
  • There are also two "BADER_TODO / STARKELP_TODO" entries with questions where I was curious about the logic that was pre-existing, and woudl like more clarity myself.
  • This one is a great one to look at to understand the new format better, albeit with something that is on the more middle-complex side

• At this point, I'm down to 1124 compiler errors. This is taking MUCH longer than expected, as everything seems to have with this process, but it's worth it.
  • There are some things related to hunter, warden, and PG that are just commented out right now, which I'll be re-enabling later as I get their patterns fixed up to hook them back in.

Moving Into DeepInfo, Part 4

• Got rid of the visual studio solutions that had the base game and dlc xml in themselves separately. Instead, combined both into a single solution that has the separate projects from each of them in it. This makes searching and replacing a lot easier, and I don't know why I never did this before.

• The death effects have been fixed up in DeepInfo:
  • Zombificiation of all sorts.
  • Metabolization and Necromancy were already fine.
  • Nanocaustation.

• The AI Sentinels DeepInfo class is now down to 5 errors (from hundreds), and uses its new structure to the fullest. It should be much easier to read, and it doesn't have to do subfaction lookups and such nearly so often.
  • The remaining 5 errors are related to parts of subfactions that I have temporarily disabled, so I need to get those subfactions all fixed up and then those 5 will go away.
  • Total error count is now down to 1090.

• Set up a new AISubFactionCoreDataRoot to live in BaseInfo under the sub-data for Warden, Hunter, and PG, since those all have a certain type of data involved.
  • This makes it so that they can now reach back out and talk about their parent faction with ease, rather than having to have a faction passed in and then fumble around to find it on the faction like was the case in the old architecture.
  • This actually pulls in pretty much all of the code that used to live in IndependentFleet, and translates it into the new format. It is vastly more brief code now, and I put like five different TEACHING_MOMENTS in there.
    • In particular there's some great teaching moments about the threading model of the entire game, and the System.Threading.Interchanged class and how to use that, and cross-thread-safety concerns in general.
  • This section of code (and the three subfactions that ride upon it) was the one I was most worried about coming into the weekend, so I'm glad to see it looking so well.

• Warden logic that was previously on IAIWardenTypeImplementation has now been moved to AIWardenCoreData.
  • ReceiveDonation is a really straightforward copy-paste from where it used to live. Now it's in BaseInfo.
  • The rest of those other things now have migrated over to live on top of the framework that AISubFactionCoreDataRoot has built. It's wonderfully simple by comparison to what it used to be, and the code is also much closer together and doesn't have an interface in between the definition and the implementation.

• Added a new MapgenDeepLink, which lets me handle things like seeding the base-oriented warden secret ninja bases. See how handy this pattern is? It literally can be used for anything.

• The "HunterFleetHelper", which I had created only for a little time, has gone away. This is what was on IHunterFleetTypeImplementation at one point.
  • The translation across for this was a ten minute job, after spending hours getting the general framework up to make this and warden and such actually work. Nice.

• The Praetorian data went even faster than that, excellent.

• The SetSpecialTargetPlanet command has been removed from the game, because it existed just to send data to the client which was basically host-only (and which is set in a different threadsafe manner now from the host).
  • There's several teaching moments in the code related to this, but essentially it was a pointless game command that just increased complexity a bit without actually solving any issue in the way multiplayer and the faction logic actually turned out.
  • Speaking of, in this check-in the CurrentTargetPlanetIndex moved from being a core thing on every external faction's BaseInfo, to only being a thing that the hunter, warden, and Praetorian share. Nothing else uses it.

• The UpdateFireteams() method on factions has been removed, because it was possible for someone to unknowingly just directly set NumFireteams.
  • Numfireteams is now a property whose setter does the logic that UpdateFireteams() used to do.

• The UpdateFireteamSizes game command has also been removed.
  • On the surface, this might seem a bit like bad form for multiplayer. We are changing something sim-affecting, after all... kinda.
  • But the fireteam logic all pretty much just runs on the host, and -- again -- the faction data is transmitted very frequently to the client, anyway.
  • It's more economical in all senses to just directly set the NumFireteams on the faction directly from the external code that used to call the gamecommand.

• Those last errors in the AI sentinels faction code that I mentioned earlier today are now gone, and the hunter faction code is now also error-free.
  • Praetorian faction code is now zero errors, and so is the warden.

• Also, while I was at it, in the xml for the warden I took their default multipliers for how they gauge enemy strength and their own strength and moved those from being 1.5x enemy, 1.0 self, to instead being 1.0 enemy, 1.1 self.
  • This will make them considerably more willing to help out, even on planets where they may be outmatched. We've had off and on comments for a long time about the Warden being a bit too passive, and this will change that. If it's too much, we can tweak the xml, or make variants, or whatever else.
  • Since this is controlled by xml now, end users can experiment with it with just a text editor, or modders can make their own versions of the warden who act differently, etc. No code required.

• Stopping for the evening and my total remaining compiler error count is now 935. That should be pretty accurate, and is the first time we've been sub-1000 errors since starting this whole process a long while back. At the height it was over 7000.
  • I still have some pretty heft factions to deal with (Fallen Spire, Dark Zenith, Scourge, and Sappers are the big ones, with Dyson Sphere also clocking in a lot of errors right now as well). A few other factions have a handful of errors (Migrant Fleets, Astro Trains, etc), and some mapgen code also needs a few fixes. But it's definitely feeling like the downhill slope now.
  • The AI cluster of factions was the thing I was most worried about working on, and I commented about my worry about the Subfaction interfaces a couple of weeks ago when I ran into those and remembered them (they were my doing, I believe, but I had forgotten about them). As of today, that part is all done, which is a giant weight off!

• Fixed up how the AI Start Sizes for AI empires work in mapgen now that the code for getting their settings has been adjusted.

• The other handful of compiler errors in mapgen are also now fixed.

• A handful of errors in various factions that were otherwise okay are now fixed, leaving mainly the big factions to deal with.
  • Also two small errors in TargetListPlanning.
  • And nine in EntitySimLogicImplementation_DeepInfo.
  • Six from FireteamUtility.
  • Four from FireteamExtensionMethods.
  • Two in SpecialFactionPlanning.
  • One in ExternalFactionDeepInfoRoot.

• Okay, now actually it's time to deal with some Fireteam change that are required, because they give funky results on the clients.
  • TargetPlanetIdx is gone, just leaving the actual TargetPlanet. The only reason for TargetPlanetIdx was because of the order of serialization in the past.
  • Same deal with TargetID, LurkPlanetIdx,
  • Also, DoEarlyCleanupWhenGoingBackIntoPool() and InitializeToDefaults() have had their code merged to make them not duplicate one another.
  • Made a few of the serialized variables that are stored here more efficient, too, since there are a lot of fireteams being transmitted around.

• FireteamRequiredTarget has been made more efficient to serialize, and on the CPU. It requires fewer lookups during runtime, and has the data it needs on demand.
  • In order to fully make this work for the AgainstSquad, that now uses a LazyLoadSquadWrapper, but for the factions and such that's already handled.
  • This in turn prevents us from having to cache some information on the Fireteam itself that was just kind of copies of this.
    • Those fields on Fireteam are now instead properties that wrapper the FireteamRequiredTarget. This will actually work properly on MP clients without having to have them run UpdateNonSerializedFields(), which was a problem up until now.

• Whew, okay, fireteams are really problematic. They have a lot of host-only stuff that gets fed into the UI, and it's really hard to tell the parts apart.
  • I've improved things quite a bit so far, but there's nothing there that will keep the problem from recurring at the moment (it's a case of easy to shoot yourself in the foot when adding functionality to fireteams, right now).
  • I'm currently checking in what I have on this front so far (with 904 errors left, incidentally), and then seeing what I can do to actually split this out so that not all of the Fireteams functionality exists in BaseInfo. There are parts that frankly just need to exist in DeepInfo so that we don't do this foot-shooting thing.
  • Right now it's mainly a matter of the MP clients are going to be incredibly spotty in what they show properly in their tooltips about fireteams; the actual gameplay of them acting properly is fine for MP and SP.

• The hunter and warden types now properly load again from the settings in the lobby. This was just one of those things I had not ported over.
  • Vassal status now also loads properly, and centrally for all factions in an optional way, which is nice.

• The starting fleets and battlestations and such for human empires now load properly again. This again was a matter of porting over the existing stuff.

• And finally, the last of the lobby fields, the trim colors for AI subfactions, are also now ported over.
  • There's a slight chance they may be a bit janky either in the lobby or after, but I think it will work this way, and it's more efficient. I have a couple of lines commented out that will fix it (but do more CPU work) if it's actually a problem.

• Fixed a few more things here and there, Devourer and some exogalactic bits.

• The Scourge faction has been mostly updated, except for the parts where the LRP thread does some things that would violate the normal double-buffering pattern. I'll deal with those when I am feeling more fresh.
  • For now, got rid of several hundred errors in the Scourge for compilation, and now there are 4 left. One of those is related to some fireteams changes I still need to make (I have a great idea on how to do it now, but I'd like everything else cleaned up first so I can see my work more easily).
  • Overall error count now down to 613. That's really it for me tonight, now.

• The Sappers collection on the Sappers faction is now DoubleBufferedConcurrentList. All the others were fine without the concurrency.
  • Oops, no, actually FloweredCrystals also needed that.
  • Okay, the sappers are fully compiling and ready to go. That's us down to 73 compiler errors.

• Hjarnum on Dark Zenith is now DoubleBufferedConcurrentList. It seems to be the only one there that needs that pattern!
  • Sigh. That said, it needs to be sorted, which is incompatible. So we also now have a SortedHjarnum that we build at the same time and sort. This is apparently only used in the UI, which is handy at least.
  • Almost all the errors are out of the Dark Zenith faction now, but there are a few things that I still need to look at in the morning when I am more fresh. I was working until 1:30am last night, and then back at it at about 8:30am this morning, so I'm not the most fresh.
  • That said, the total compiler error count is now... 6.
  • Tomorrow morning I'll need to sort out these last issues with the DZ, and then get the fireteams split I have planned done (it shouldn't be too bad, knock on wood), and then correct a couple of vis-dll errors that should take under 30 minutes, and then I'm on to actually... runtime testing!
  • This whole process has taken enormously longer than I ever expected, but the end is finally in sight. If I don't have at least a compiling version tomorrow, then something is very very wrong. Honestly if there's not one by lunch I'll be shocked.

Moving Into DeepInfo, Part 5

• Removed the ConcurrentDictionaryExtensions class, and merged that into the general DictionaryExtensions class.
  • Added a new DoFor() extension to ConcurrentDictionary via extension, the same as we had previously added that to Dictionary<>.

• Astro Trains had a few enough number of errors in its file -- 63 -- to actually have them shown properly in visual studio.
  • Oof, but its BaseInfo was not using double buffered lists, and the way that it was aggregating train guards was... fraught. So let's fix all that up, and also serialize less while we're in the process.
    • AstroTrainsPerTrainBaseInfo now has a pair of double buffered collections, which is the first time we've used that on per-unit data, but it's actually a great way to store things in the new framework. I haven't needed this before now, but it's a very useful form to use.
    • We also no longer have DeployedGuardsPerTrainSim on the BaseInfo. That's now being handled on the actual trains, rather than via an indirect lookup. This solves a lot of code complexity and chances for orphaned data. It also uses less CPU to calculate in the new format, though in the old format of the game it would have been debatable.
    • This actually has turned into a rather fascinating study of nested data. It's quick and easy to understand and a pattern to be emulated. This is actually kind of a great showpiece for the power of the new system.
    • This turned out to be SO useful that I've put a teaching moment in there.
    • Improved the description appenders for trains to be a bit more informative now that they are able to be.
  • Okay, back to the astro trains deepinfo file, how many errors do we have now? 71, okay.
    • Well, we can safely get rid of DeployedGuardsPerTrainLRP, StationsLRP, and TrainDepotsLRP.
    • TimeLastDepotSpawned was not consistently called, so it has been moved into SpawnNewTrainDepot. Same with AstroTrainsPerDepotBaseInfo being created and attached to it.
    • initializeDepotData has been removed and made part of the depot creation process.
  • The guards code was really confusing because things were not named very clearly. The guards on the train are now called StoredGuardsInsideThisTrain and DeployedGuardsOfThisTrain, so that it's clear that's how this works.
  • The way that guards are stored for finding lost guards, and the way that they attrition, has been moved into BaseInfo (and thus runs on the client as well as the host, which is good), and is also a lot simpler to follow now. Same general logic, but it won't miss anything by accident.
  • StoredGuardsInsideThisTrain was an ArcenSparseLookup, which was a cross-threading problem waiting to happen, but it can't use the double-buffered style collections because that's not what it is (it's actual stored data). So this is now a ConcurrentDictionary, and there's a teaching moment in there explaining why.

• Silly me, I thought that Astro Trains were going to be a very easy faction. But it turns out there was quite a lot going on there that was really really out of date for the codebase, so... here we are.
  • Down to 551 compiler errors left.

• Okay, let's see what's going on in the Fallen Spire deep infor class. I see 85 errors at the moment, mainly so low because I've been in here before quite a bit, but not with my full attention.
  • Most of that melted like butter in under five minutes. There was a funky thing with the journal logging that wasn't very clear, but that was something that the svn history let me clear up and then document pretty easily.
  • Transceiver needed to be converted into a DoubleBufferedValue. From the look of this class, this was before I added that specific item (in the distant memory of last week).
  • Oh... wow, what an error on my part. Coding while tired, I guess. This was also one of the earlier factions I started the conversion on, and then I had set it aside. All the BaseInfo Stage2 stuff was in DoFactionGeneralAggregationsPausedOrUnpaused instead, which runs WAY too frequently. Talk about a performance drain. Glad that code never released.
    • I also untangled the double-buffered clears and switches from the logic that fills them, which is what all the other classes I've done more recently were like. It's amazing how that takes this from unreadable to really nicely legible.
  • Okay, even so with all the oddness in BaseInfo, this gets me down to two errors remaining in the DeepInfo class... and they are the same sort of nastiness that I've seen in several classes when working with my nice new double-buffered lists. They just... aren't sufficient for a lot of the code that exists in actual DeepInfo classes. I'm particularly unhappy with the butchering that I did to Zenith Architrave; I think I may revert back out that entire class to before my last pass on it, and redo it more nicely. But before that, I need a double-buffered collection that supports concurrency; one that is used for spire cities and debris (thus solving my last two errors here), and which then is also used on Zenith Architrave (thus returning its logic to what it used to be). What I had expected to be an edge case actually turned out to be a really common case, so new data collection it is.
  • I'm down to 468 errors, but I'm going to pause on forward-progress for a bit, add my new data structure, and then undo all of the likely damage I did to Zenith Architrave.

• Added a new Engine_Universal.DEFAULT_CONCURRENCY_LEVEL, which is set to 4. This should be passed into every ConcurrentDictionary in the program as the default concurrency level (and then choose yourself what the starting capacity for that dictionary should be).
  • Why does this matter? https://blog.getpaint.net/2017/06/30/concurrentdictionary-allocates-a-lot/
  • Engine_Universal.DEFAULT_CONCURRENCY_LEVEL is the default level of concurrency that we are likely to need in AI War 2, based on my designs for threading. If more threads than this are accessing a ConcurrentDictionary at once, then performance will suffer slightly, but not much.
  • The default concurrency level is actually 4x the logical processor count, which is... insanely too much, and wastes RAM and performance when it comes to high-CPU computers. I only have 8 phyical cores with hyperthreading for 16 logical processors, but that means that it is creating (for me) literally 16x more RAM usage and locks than are needed. In the future, as computers advance, the design of this program is such that it will still use plenty of processors, but not on an individual dictionary basis.
  • So we'd start seeing MAJOR performance degredation and memory spikes if 128 logical processors were ever a commonplace thing, etc. Now we won't. Yay futureproofing. And honestly, it helps for performance NOW, too.

• Added a new DoubleBufferedConcurrentList. This is a lot like DoubleBufferedList, but less efficient in some ways, and more powerful.
  • This one also needs to support the display list taking additions and working as a working list for background threads. This tends to be a case where something says "oh hey, let me look at what is there, and also let me add this one," and then the rest of that background thread needs to have access to thing that was added RIGHT NOW.
    • For example, this sort of thing could happen on Stage3 or in LRP or both, who knows or cares. Either of those is still happening while the UI is using the same DisplayList as the others.
  • There's no way to handle that without using a concurrent list, and you might think that would be enough to avoid flicker in the UI, but it's not. The flicker would come when the construction list is torn down and then rebuilt, since that is often on a (different) background thread.
    • So we need to always have persistent lists, and support ad-hoc additions to the display one as we go, while also supporting the common cycle of construct-clear, construct, flip-to-display.
  • Sidebar: Internally, we are using a ConcurrentDictionary to power this. What... the heck?
    • Well, there is no "list" class in the .NET Concurrent collections, because the idea of an ordered list is antithetical to cross-thread ad-hoc addition and removal. They have ConcurrentBag and ConcurrentQueue and a few other things. ConcurrentQueue is vaguely ordered, but the main point is that it supports the foreach operator, and also supports direct removal and additions. Throughout the codebase, I have preferred ConcurrentQueue to ConcurrentBag because of profiling work that various other programmers have done, demonstrating that Microsoft's implementation of ConcurrentQueue is slightly more performant. That said, neither of those classes -- nor ConcurrentStack -- support the Contains operation. There isn't any equivalent of a ConcurrentHashSet, either. The only collection that really supports all of what we want to do is ConcurrentDictionary, which has some performance drawbacks, but not too severe to be worth it.
    • If you CAN use DoubleBufferedList<>, then of course use that instead. That thing flies. Start with DoubleBufferedList<>, and if you find you need to graduate to DoubleBufferedConcurrentList<>, then do so.
  • Implementation notes:
    • You will notice that there is no GetDisplayList() like you are used to seeing all over the place with DoubleBufferedList<>. Instead, you must use DorFor on it to do the equivalent of a foreach loop (which you can break out of as-needed).
    • There IS a Contains method, but there is a (very very) low chance that it will return false when the answer is really true.
      • Because of that, there is an int TimesToRecheckIfSeemsLikeNotThere = 0 parameter to this method. If you REALLY need an accurate Contains, you can set that to something larger than 0, but be aware that has a peformance impact that is equal to calling the Contains methods however many extra times you say.
    • Since this is a dictionary under the hood, you can't put the same item in the list more than once. Or rather, if you do, they will just collapse down to the unique set.
      • This is actually useful, since typically adding something repeatedly is an error anyhow. We specifically have some things like AddPlanetIfNotAlreadyInList(), but you do NOT have to do that. That's implicit in this structure. For goodness say, definitely do NOT call "if not contains X, then add X," because you just wasted some CPU cycles for nothing in that case. Just add it.
    • Since we are specifically wanting to be able to modify the display list on the fly (that's why this variant of DoubleBufferedList<> exists at all!), there is of course also a new AddToDisplayList() method.
    • As a consequence of this being a concurrent collection, you can forget about any sorting methods. The normal DoubleBufferedList<> supports sorting, but this version does not. You'll have to feed the results of a DoFor operation into some local working list and then sort that if you really must combine sorting with this sort of list. But in general, I haven't seen any cases where that would be relevant.
    • You may have noticed that none of the DoubleBuffered collections suport IEnumerable (so you can't do foreach on them). This class is no different. The reason is because that would be a highly ambiguous call. Someone might do a foreach while thinking they are doing list construction, or they might do it while thinking they are looking at the display results. Lack of clarity like this is very bad, because a second programmer following behind you can't even tell with ease if you made a mistake or are doing what you mean to. So instead we have the DoFor variants.

• Went ahead and added a Display_DoFor onto DoubleBufferedList<>, so that it has the same style of functionality as the concurrent one from that angle.

• Added a new Display_GetRandomItem() on both DoubleBufferedList<> (just to save code, and it's very efficient), and to DoubleBufferedConcurrentList<> (this is not very efficient at all, but it works; and in most cases of relevance in our codebase, these lists are so small that it's plenty efficient ENOUGH).
  • Normally you can't index into a dictionary in the way we would want to in order to get a random return value, but our code would often need to do this on the LRP in particular with DoubleBufferedConcurrentLists, so now it's possible.

• For Astro Trains, they are now the pilot for DoubleBufferedConcurrentList, with TrainDepots and TrainsEnRoute now using them.
  • This was definitely a major diversion off to the side, but this keeps the code for the end product of the game as close as possible to what it was in the past, and thus lowers our risk of new logic errors or just incomprehensible code. It's time for me to completely redo the travesty of what I visited upon the Zenith Architrave a few days ago, now that I have this tool.

• Reverted the Zenith Architrave to svn revision 14224. Ugh I hate undoing work, but that was really a hash. This brings me back up from 468 errors to now 539 errors.
  • My goal this time is to use the double buffered concurrent lists to accomplish keeping the logic of the ZA intact, rather than trying to rewrite it and hash it up like I did on Thursday night. I had noted even in my checkin notes on Thursday that it was a hash, which is never what you want to see in your checkin notes.

• Fixed all of the non-controversial errors with the Zenith Architrave, which took only 20 minutes by my reckoning (I'm getting much faster!). I had temporary working variables to quiet certain errors related to the central lookups that are now going to need to change.
  • I am first checking in this version that has all those changes in place before I start changing the bits that need the new collection types. With the temporary variables back out of there, I'm back up to 75 errors from this file, so that's quite a lot related to them.
  • Also worth noting is that I'm not messing with the LRP collections at all. This class is complicated enough that they may as well just keep doing what they do. I don't really want Badger to have to come in here and review this code like I had asked him to before, and at this point I don't see a reason why he should have to.
  • 543 errors in total for now.

• Okay, got the next batch of ZA fixes in place. All of them except the WarpingInSpawners and the Castra are able to use the normal DoubleBufferedList. That's easy enough.
  • If I were to extend this to the LRP variables, which I think I did before, then that would cause some need to use the concurrent version more. But that would be a net negative in terms of performance, so I'll just leave LRP doing its own thing, giving more room for the sim and UI to share work/benefits with one another.
  • At this point there are 17 remaining ZA errors, and those will be solved by me converting the Castra and WarpingInSpawners to the concurrent double buffered list.
  • This time around, the flow of logic is preserved exactly as Badger wrote it, and there's no longer a need for a code review of this faction.

• WarpingInSpawners and Castra are both now DoubleBufferedConcurrentLists.
  • WarpingInSpawners previously was sorting itself after the initial creation of its list, but that's no longer a possibility with the new collection format.
    • The logic that was relying on this sorted list was kind of complicated, so it was easiest just to implement a secondary SortedWarpingInSpawners that we feed from the concurent list and then sort like we did before.
  • That's a wrap on ZA. They are now error-free, and no longer in need of a code review any more than any other part of the codebase now is.
  • Back down to 468 errors.

• Clearing "two" errors on the fallen spire took 35 minutes and generated tons of other things that had to be converted. But nevertheless, the fallen spire is now error-free:
  • Spire Cities and Spire Debris are now using a DoubleBufferedConcurrentList.
    • However, because spire cities in particular ALSO need to be sorted half the time (sigh), there is now a SortedSpireCities that is built for mostly UI purposes.

• GetScourgeStateForDisplay now includes information on fortresses. Previously it just had armories and spawners.

• The three kinds of "warping in" collections on scourge now all use DoubleBufferedConcurrentList. This was about a 10 minute job for "three errors."
  • Other than one fireteam-related error which I am intentionally leaving to the last (it's a structural thing that I need to be able to see in isolation to fix), Scourge are now error-free.
  • Total error count now 463.

• Fixed right about 60 errors that were remaining in the migrant fleets faction code (22 were obscured from the compiler until I fixed the ForEach they were in.
  • This was jolly easy and pleasant, nice. 425 errors remaining.

• Fixed what was reportedly originally 38 antagonized dyson sphere errors, but wow that was kind of a mess in general. They're all fixed up now, and probably for the first time ever we're not mixing up data of the antagonizer and the dyson spheres themselves, which makes the whole thing a lot easier to follow. Previously they had kinda-sorta shared an external data object, but not fully, and that made it a bit confusing to untangle.
  • Anyhow, I happened to also notice that AntagonizedDysonDescriptionAppender is not actually used on dyson spheres at any point from what I can tell. Maybe this was deprecated on purpose, but it would be functional if someone wants to attach that to a dyson sphere that's being antagonized.
  • Error total is now 394. Other than the fireteam stuff that will be at the very last (and some vis-layer stuff that is incredibly minor), this is down to mostly the Dark Zenith and the Sappers. Oh, and apparently the Dyson Sphere proper has 63 errors, too, I now see.

• Well, that was again pleasantly straightforward: dyson sphere faction is now error free. New compiler error count is 332. Time for the last two big guys.

• Decided to make the safe changes first, and hit low-hanging fruit, in case I make another mistake again like I did with the Zenith Architrave.
  • Sappers are down from 119 errors to 45, and what remains to be done with the rest of them is a lot more clear.
  • Dark Zenith are down from 211 errors to 71. It looks like Sappers and Hjarnum will need to be concurrent, but none of the other lists, which is excellent. Additionally, a couple of methods need some adjustment based on working lists, but that's not too bad. All told, this faction was less intimidating than it had seemed, thanks to judicious search and replace. Zenith Architrave was the more challenging one structurally, even though Dark Zenith is more complicated in general.
  • Total compiler errors are now down to 117, which is... insane! I was happy just to get to three digits today, after being in the high four digits for weeks. Now we're almost down to two digits!

Hunting Runtime Errors, Part 1

• The last of the non-fireteam compiler errors are now fixed. Time to get the last fireteams bits fixed up and that will be that!

• The text-export portions of fireteams are being merged back into the main file of Fireteams.
  • A new FireteamHonoraryDeepInfo object has been created, which exists as an immutable sub-part of every fireteam. On MP clients, this sub-part is ignored and unfilled.
  • There are detailed teaching moments that explain the thought process behind this, and how to mimic it for other data if you ever need to (as a main developer or a modder).
  • PreferredSpeed has been moved to be a serialized field, so that it can be shown on the UI on clients and hosts.
  • Removed some string construction that was happening overly-frequently on fireteams even on the host/singleplayer cases. It now only happens if you're actually looking at the debug menu in question.
  • For the first time since it has existed, our DeepInfo dll (AIWarExternalDeepProcessingCode) actually compiles!

• The 23 errors from the external vis dll are also all now fixed.
  • Running our complete build scripts to verify that everything compiles... yep! We are officially on to runtime errors now. Time to start up the game for the first time in three weeks (time for that to be possible for the first time in three weeks, internally).

• Fixed a bug in GetIsDescendentOfInterface() that was causing false positives.

• Corrected GameEntityTypeData to not require BaseInfo or DeepInfo for units. Most of them will have neither, and so far no units have needed DeepInfo.

• Fixed a couple of xml typos that were preventing proper parsing.

• For the sake of clarity, IArcenExternalClassBase has been renamed to IArcenExternalClassRoot.
  • We don't want this to get confused with BaseInfo, because that's not what it is (but it was sort of what it looked like).
  • ArcenCachedExternalTypeBase is now ArcenCachedExternalTypeRoot for the same reason.'

• Verified that my logic is working for checking if BaseInfo and DeepInfo (true DeepInfo, not honorary) are living together in the same dll.
  • As I had kind of expected, it's tripping on my own code at the moment, which mostly means I have some things miscategorized.
  • This exception checking is here to prevent accidental mixing of this data that is supposed to be separate, including dlls referencing dlls that they should not have access to.
    • Essentially this sets up a boundary where it tells programmers (modders, later, but for the moment myself) "hey, you're either arranging this wrong or you've tagged things wrong, but either way fix it because bad things would happen in multiplayer if you do not."

• Now that I have verified that the above works, a whole bunch of debugging output that was related to that has been turned back off.

• Fixed all the map definitions still pointing to the wrong dll from their xml.
  • Ditto for the TargetEvaluators.
  • Ditto the AIDefensePlacers.
  • And DeathEffects.
  • And GuardpostAndCommandPlacers.

• Fixed an issue where the xml tag multiplier_to_all_fleet_caps="0.2" was still being used on a variety of outguard units, which I don't think had any relevance for that at all.
  • Also some sappers turrets, and a ZA turret, etc.

• Then there was an unexpected multiplier_to_all_fleet_caps="1.34" on:
  • Golden Wasp Drones, which made their cap 536 instead of 400. Let's make that 540 just to look tidier.
  • Queen Bee Drone, normal and Dark Zenith versions, ditto.

• Milestone: I can get to the main menu without any errors at all! It took just over 1 hour from the point of being able to compile at all to get to that point. Now I'm going to break for lunch, but it will be interesting to see how long it takes to get the rest working. It's nice to see the main menu again!

• Fixed an issue with the Source not being properly linked to each of the four PrototypeObjectForGlobalLobbyClearingOnly objects for the new externaldata, which led to an immediate exception when trying to start a game.

• Fixed an one-line code error that broke the new sub-faction linkages for the AI. They needed to be after a loop, not inside it, and it was one curly brace too early.

• Fixed another one-line error that mixed up relentless wave linkages from DeepInfo to BaseInfo.

• Fixed an exception where Outguard mistakenly thought they were supposed to have an Intensity (they are not).

• Fixed a minor issue with order of operations with the lobby partial-map-generations not linking up parentages properly.
  • This was technically working by design, but this was from a few weeks ago before I knew the shape of things to come. It needed to be changed a bit in light of the final design.

• Fixed another issue with Allegieances, where every faction was expected to have one (but many invisible ones don't, in particular).

• Milestone: I can now open the lobby without errors. I took a long lunch and ran an errand, so this was about 1 hour of work since I got things working to the main menu.

• Fixed up how the faction external data was linked to its "attached object" (faction or squad).
  • Previously it did it way too late, and therefore some errors were resulting.

• Started tracking the way that faction external data is linked to its attached faction, because I am getting some strange nullref exceptions.
  • By assigning cross-thread-safe unique IDs, I can see that the version of the faction data that we have in the lobby for use is very different from the one that previously had itself assigned. Curious!
  • Stuff like this is central and breaks every faction, and is just part of the framework itself being tested out for the first time. This is the sort of thing I was hoping to see, and not a thousand little per-faction errors. I may yet see a tide of those later, but at any rate this current part is just a normal part of unit testing after coding for three weeks without being able to test any of it.
  • At this point, I can see that we have a minor memory leak of factions again, without having to even look the memory dumps and diff them (which I will do, later, when all of this is working better). Nice to find this out sooner than later.

• BaseInfo and DeepInfo on factions and squads are now carefully-managed and sent back to their pools more aggressively.
  • ...That did nothing, unfortunately, to help my actual problem. It does make the code easier to read, and it's hard to say beyond that.

• Fixed things up so that BaseInfo and DeepInfo actually get put back into their pools when they are supposed to.
  • This helps a little bit, but there's still a central leak with my pooling framework, which is a surprising thing to learn at this point. I thought I had established that as solid a while back.
  • Then again, back when I was establishing it as solid, I had noticed that it would increase to about twice what I expected, and then stop. So it was using extra memory, but not infinite endless memory. This may be related to what I'm seeing now, which I am going to fix because it causes problems for me on several levels.

• Added a new GetItemsWaitingInPoolRightNow() and GetItemsInTheForeverList() onto TimeBasedList, to help me see what's going on as the factions are pulling external data out of their pools (right now they create too many objects, and I want to know why; after that I'll still have to deal with the problem of improper item teardown, but one thing at at a time).

• Discovered that a variety of pools were not using their original entity (the "creator entity") for actual data storage. For the larger objects, where there are just one or two per type, this is very wasteful. Read: factions.
  • In the past, I've always just really cared about things like entity orders, or squads, or whatever else -- and the difference of literally one object is super stupidly negligible. For a faction, it's an extra 100% usage of RAM, by comparison.
  • These pool types now include the creator entity as a normal object to use, just to remove this confusion and slight waste (it can still do its job as an object factory no matter what else is happening): TimeBasedPool<>, BetweenMapGenPool<>, and ConcurrentPool<>. I think that's all the major ones of relevance.

• Immediately after that, started having some exceptions in Recalculate_PreSystemRecalculate() on initial game load (before getting to the main menu), which is curious but not completely out of character with some OLD issues I've seen off and on sine 2018. Let's get that fixed.
  • For now, Recalculate_PreSystemRecalculate is being instrumented.
  • Based on this, fixed an issue with AIWar2GalaxySettingTable.GetFauxFIntValueFromSettingByName_DefaultOne_DuringGame() and all similar methods where it would give an error if those were called prior to the game being fully initialized.
  • Based on this, also fixed an issue with GetResourceProductionBeforeAnyBonuses() where it would also error if called prior to the world being initialized.
  • Same with GetFrameSizeMultiplierAsDoubleNonSim().
  • Did a further bunch of hardening of a bunch of other places, several dozen; mostly windows.
  • Worth noting that none of this code was changed during this refactor. This is an old issue that is only triggered by certain rare circumstances that -- for whatever reason -- become common during this pooling shift.
    • On the whole, this is not the sort of issue I was expecting to chase today at all (something unrelated to the work of the last three weeks), but there's always the chance that this has been secretly breaking for a rare subset of players, so let's make sure it works.

• I think that the world is actually being set to null for some reason, possibly because of the world itself being pooled in a funky way...
  • So let's fix THAT: The three pool types noted above all now have a CreatorEntryIsAddedToThePool parameter. Now it's up to the calling code to decide how to handle it.
  • True on everything except: PlayerAccount, World (the first entry in that is a "dummy world!"), ConfigurationForFaction, EntitySystem, SpeedGroup, PlayerAccount_AIW2, and World_AIW2.
  • The game now loads just fine to the main menu again.
  • Also, the lobby faction tab no longer spams errors about a missing faction link! So I DID solve the problem I was originally chasing, but just in a really roundabout way.
    • This made me suspect that entering the lobby multiple times in a row, or otherwise serially regenerating map types would be a problem. But nope, those work great also.
    • Adding multiple AI factions works great, and adding extra human factions also works.
    • However, after adding two extra AI factions and then another human faction, I did get an error where the AI subfactions were not linked to their parent AI faction in the case of all three AIs. So I'll have to look into that, after I can duplicate it more.
    • Also, reading last-lobby-settings seems to be broken at the moment, but that's fine. Just adding that to my list.
    • Starting a new game from the lobby is also broken, but this is probably unrelated to anything I've done thus far; this is the first time I've tried clicking that button today, and the place it errors looks like a sensible place for a one-line typo.
  • So, I'm going to call this current line of investigation complete, and move on to the new list of errors that I have accrued to look into. (As kind of an aside, I was fully expecting to have serialization be broken in the new version, as that's really hard to do completely correctly en-masse. I built in tools a couple of years ago that makes it trivial to find those errors, generally, but it's not my top priority right this second).

• Milestone: I can click around the lobby, add factions and such, with very few errors (though not zero yet).

• The game now does a handy dump of all the faction basics when it runs into parentage issues.
  • See, back when I was originally designing the parentage stuff a few weeks ago, I had not wanted the subfactions clogging up the lobby for very good reason -- it's the reason I'm having some exceptions now, in fact; old stale sub-factions get left around. Having the subfactions earlier than later is actually better overall, I just need to have them clean themselves up between mapgens now, basically.

• Using the above, the game now properly removes and adds all the various factions and subfactions. No more errors on that front, or mis-parenting of factions, or duplicate factions, using any routes that I can tell.
  • I do see there is still an issue with null attachedfactions sometimes on factions added via the lobby, but hitting the map regeneration button fixes it. Still needs to be fixed to not happen at all.

• Made mapgen a bit quieter in the log when it's not having problems.

• Fixed an exception that would happen when it tried to add nomad planets if you did not have the nomad faction enabled.

• Fixed a minor issue where the game had a default of "Normal" for the AI Layout field, when that was not an option. The default is now Small Clusters, as intended, and "Normal" is also taken as meaning "Small Clusters" just to be on the safe side.

• Fixed another exception that would happen in the escape menu when trying to describe factions. It was being too strict and expecting all factions to have a NumberToSeed, which of course they do not.

• Updated the version number to 3.700, so that the game can actually TRY to load saves it makes (it won't try for anything lower than 3.7).

• Milestone: Game is nominally playable. Currently in single-player, you are not properly assigned to the first human faction, but you can start the game anyway. Once you are in, all the menus seem to work and you can assign yourself to that faction. Ships obey orders, fighting works, factions seem to be doing what they are supposed to, and there are no errors so far. Bear in mind this is just with the human and the AI and the auto-added factions, not with anything from DLCs yet, but that's still pretty awesome.

• Fixed an issue with "Super Easy Mode" on the Fallen Spire (debug mode) not being a proper bool. I had converted all the others, but this one was erroring because I missed that part of the xml.

• Fixed a couple of serialization issues with savegames, but I have a few more to fix still. Doesn't look like anything super major, knock on wood.
  • There is less data being sent now, partially because I am reading things from the Config for factions directly rather than sending/storing that multiple times, and that now fully works.
  • From the look of things, the "last stored settings" for the lobby also work again. Have not tested quickstarts yet.

• One of the benefits of breaking all the savegames is that I can change the order of things around to be more efficient/clear with how we load things. Previously I used that in order to put all of the ExternalData at the very end of the savegame, which makes deserialization a lot easier there.
  • Now I am switching fleets and factions, so that factions are deserialized first. In order to do some more-efficient conditional deserialization of fleets (sending less data based on the type of faction in question), we need to have the factions loaded first. Now we do!
  • As part of this, we no longer have to have a loadingFactionIndex on fleets. We can just directly load the faction.

• Similarly, I have now split it so that the planets and the galaxy are now loaded before fleets. However, the PlanetFactions (and all the units inside them) are not loaded until AFTER fleets.
  • This split makes even more things easier to load, which is really nice. When I originally designed the order of these things back in 2018 or so, the full scope of what their needs would be was not remotely clear.
  • We now no longer have to do the loadingPlanetIndex on fleets, and we can also set the PlanetFaction immediately, too.
  • PlanetFaction PFaction on fleets has been removed, as it was not actually ever directly needed and it was slow to calculate.

• On fleets, we no longer have a Centerpiece_DeserID. Also, Centerpiece is no longer a GameEntity_Squad -- it's now a LazyLoadSquadWrapper.

• Thanks to the changes above, we no longer need FinishPostFactionAndPlanetDeserializationLinkages() to run after fleets are deserialized. This is good for speed in general of loading savegames, but also for multiplayer CPU processing on the clients as they get fleet data.

• Because of the order of operations change (factions loading before fleets), I can no longer serialize the LooseFleet ID for a faction directly on the faction itself -- the fleets don't exist yet.
  • I _could_ go in and make some indiretion (store the fleetID, then later link it), but there is a much better solution.
  • We now just store a serialized IsLooseFleetForFaction bool on the fleet itself, and since it loads after the faction, it can just set itself as the LooseFleet with ease.
    • Fun fact: this takes one bit of data, rather than 32 bits. Also it's faster to do, because we don't need to do a lookup at any point; we already have a reference to the faction object.

• Necromancer data on the fleets and factions now only serializes for the necromancer faction.
  • Similarly, a bunch of player-specific faction data only serializes for players in general.

• PlanetFactionBooleanFlags are also now only sent for player planetfactions.

• In general cleaned up a few fields in serialization code, and put in a few more markers for telling where we are in the logs. These don't affect non-logged transmissions and saves.

• Fixed up some more code in the Templars, just correcting a few things that I mised and thus were not quite up to the latest standard.
  • There's still an exception in MarkUpCastles, but I'll deal with that tomorrow.

• There's also still at least a few issues with deserialization of savegames and network data, but it's getting closer to being done, and is more efficient, and is easier code to read. So that's progress!

Hunting Runtime Errors, Part 2

• Fixed up most of the core problems with savegame/network serialization that were just a consequence of me changing the up the format so much. I also... changed up the format even more, a bit. Just adding bits of extra efficiency on things that are sent more frequently. As one random example, for instance, we now use 38 bits to send the offset-coordinates of each step in a unit's guard-path rather than using 64 bits to do the same.
  • There's still something up with the serialization, but now it's to the point where it is at the external info and it's hitting an infinite loop on load. That shouldn't take too long to track down. From the serialization logs, it looks like most of the data that I would expect to be in there is actually being saved, which is nice. Right now the deserilization log itself cuts off in the middle of writing, however, which I can tell means there's a buffer that hasn't fully been flushed by the time it hits the place where the infinite loop is. So I'm going to work on making that buffer flush faster, which will take me right to the problem code.
  • At the moment, these are still the sort of structural problems that you fix-once-and-then-not-again, so I'm happy about that so far.

• The serialization logger now flushes its buffers to disk much more rapidly, so that if there is a problem that interrupts the program or throws it into a loop, the output isn't behind.

• Fixed two silly lines of code that were accidentally infinite-recursive, and caused deserialization of faction external data to not work. It didn't actually take that long to find, but there have been a lot of other secondary distractions and things today.

• Fixed the last of the deserialization errors that were new structural problems introduced in The Great Refactor.

• Milestone: The game can now save savegames, and then load them.
  • Reminder 1: older savegames from prior to The Great Refactor will not work in the new format. Trust me, it is absolutely worth it for all the improvements throwing out that compatibility has lent us. Much as I hate losing access to all those older saves.
  • Reminder 2: until we hit version 4.0, hopefully within a few weeks, I still reserve the right to continually break NEW savegames as well. There are some structural things with the Fallen Spire in particular that I need to redo, and that was too big of a thing to tack on DURING The Great Refactor. So those will be coming right on the tail of it.

• The debug setting "Write Deserialization Data Sizes" is no longer considered an advanced field.
  • When this option is on (it was added a little after DLC1 came out), it gives a detailed breakdown in WorldDeserializationDataSizes.txt of how much room each part of a savegame takes up, and what kind of data it is.
  • This has been adjusted today a fair bit to make sure it really captures things properly. Looking at the data, it's looking good! I see a few places where I can wring out a bit more efficiency, but honestly that was something that already happened a lot during The Great Refactor, and it shows.

• Discovered that I had put in the sort of funky working-list stuff in Templars for Rifts, not just in Zenith Architrave for their stuff.
  • Fixed up Templars to instead use the much-nicer and also much-less-likely-to-error DoubleBufferedConcurrentList.

• Fixed an exception in MarkUpCastles for Templars, where essentially the data for the castles was not attached onto them at the time of creation, but instead was randomly attached to them during this method.
  • There will likely be other places like this in the code, from all of us different programmers, and you can handle it in one of two different ways in any given case:
    • Option A: When you first create some entity (squad) that will later need some BaseInfo or DeepInfo of a certain point... just populate it then!
      • That's what the game mostly assumes you will do, but you also don't have to.
      • It's possible you might have some units that only need extra data SOME of the time, and in that case, that's a great reason to wait and not add it now, for instance.
    • Option B: When you need to do some sort of operation on an entity (squad), then just call CreateExternalBaseInfo<YourType>( "YourType" ), rather than calling GetExternalBaseInfoAs<YourType>() or TryGetExternalBaseInfoAs<YourType>().
      • This is also an acceptable choice, and it has the benefit of working really well with old savegames (which we have zero of at the moment, but in the future if you're trying to deal with code additions to things that now need data that didn't have it before, then this is the way to do it).
      • This approach does have the downside, though, that the Create() method has a bit more overhead than the Get() or TryGet() methods (Get and TryGet are equal in overhead, but the former errors when data is missing and the latter does not). The extra overhead of the Create() call is really just a matter of a table lookup, which means a dictionary call, and then one additional string comparison check. So it's not exactly massively slow or something.

• Fixed a longstanding minor bug in how battles were calculated for the galaxy map, with them overestimating the scale of your allies. This is for the little ripple thingies on planets.
  • This was found while fixing a really pointless warning message that could be sent to the log now when loading savegames post-Refactor.

• Now that my pooling works really well and extends fully into the externaldata realm, there are inevitably going to be some things that require some adjustment just because of having been missed and not previously being expected to pool.
  • One of those was pathfinders, which would throw exceptions after the second time you loaded a world. That's now fixed, and at the moment I'm not seeing anything else that has this sort of issue (Pathfinders were always kind of an odd case, like EntitySystems are an odd case, because those two categories of data retain attached to their parent object but still pool).

• When you are clicking the randomize button on the map screen, on solo play it now has a delay of only 0.2 seconds rather than 1 second. At the host in MP, only 1 second rather than 3. As a client, only 1.5 seconds instead of 5.
  • The idea is to prevent a bunch of regeneration requests from queuing up, which used to be a problem, but this all runs faster now and it was just annoyingly slow to let you click more, before.

• Fixed an issue (not sure how long this has been a thing) where when you hit reset to defaults in the lobby, it was not actually resetting the map tab items.
  • Specifically, it would reset the map type, but not any sub-options for the map type, and not the number of planets. This probably is new as of the current beta series, but not The Great Refactor, if I had to guess.

• Removed ControlledByPlayerAccounts_DuringGame from on the Faction class.
  • This was duplicative of something in the ConfigurationForFaction class, so we could already get that information a better way.

• Also removed/hid ControlledByPlayerAccounts_OnlyUseInLobby on the ConfigurationForFaction class, instead making this a private field that is only something that can be touched via properties and methods on its class:
  • bool IsFactionControlledByAnyPlayer tells you that.
  • bool IsFactionControlledByLocalPlayer tells you THAT.
  • int CountOfPlayersFactionControllingFaction gives you that answer.
  • byte GetPKIDOfControllingPlayerAtIndex( int Index ) has the expected answers.
  • PlayerAccount GetAccountOfControllingPlayerAtIndex( int Index ) again tells you the expected bit.
  • int GetPKIDOfFirstControllingPlayer() is predictable also.
  • PlayerAccount GetAccountOfFirstControllingPlayerOrNull(), ditto.
  • bool GetDoesPlayerControlThisFaction( byte PlayerPrimaryKeyID ) answers as you'd figure.
  • bool GetDoesPlayerControlThisFaction( PlayerAccount Player ) does too.
  • void RemovePlayerAccountFromControllingThisFaction( byte PlayerPrimaryKeyID ) is clear.
  • void ClearAllPlayerAccountsFromControllingThisFaction() is also.
  • void SetPlayerAccountInControlOfThisFaction( byte PlayerPrimaryKeyID ) is the last one, which does as you would expect.
    • Crucially, when things don't work as we expect, we can now insturment any and all of these and figure out WHY.
  • string GetNameOfPlayerOwnersOrEmptyString() is the actual last one, since the Faction class can't really do this anymore with much ease.
  • void SetPlayerAccountsToViewStartingPlanetAfterMapgen() is another "actual last" one, this time pulling in from mapgen.

• Okay... I just cannot get the faction ownership thing to work fully correctly. We have a lobby version and a long-term version, and the long-term version is right but the lobby version is wrong even after I set it to be right. I... my brain hurts. It's been hours staring at this.
  • I think that a further simplification to the lobby config settings is incoming. I will have to delay and do a release tomorrow. In the meantime I have further broken tutorials, but those will need some slight structural adjustments in general.
  • The only other really big bug that concerns me before a release is the factions external data not always having attached factions properly, but I think that can also be solved by simplifying the central configuration. This will need to be tomorrow, as I am likely to make matters worse rather than better if I try to fix it tonight.

Hunting Runtime Errors, Part 3

• Okay! After mulling this a lot last night, I have figured out that I hate the current design for our lobby configuration. It's my design, so I can say that. ;)
  • This is a design from 2018 or something along those lines, and it was in response to severe problems that I had when working with the lobby in AI War Classic in 2008-2009. It DOES solve the problem, but it created new ones. It's not the simplest-possible solution.
  • So here's the problem, and let's see if you think of the new solution before I tell you what it is. When framed like this, it's deceptively simple, but remember I'm framing it for you in a way that points toward the answer. Don't think too poorly of my past self, heh.
    • 1. Map generation takes time. How much time varies by the map and the computer, but it's a lot longer than a frame. It's something like 200ms at best, and 1.5 seconds on the super horrible end of things.
    • 2. During map generation, we need to have a copy of all the settings you chose from the lobby prior to that generation call, and those settings need to not change. So for instance, how many planets you asked for, any other map-related bits, anything do with factions that affects the map, etc.
    • 3. We obviously want to run map generation on a background worker thread, because we're not going have the game seem to freeze for 200ms - 1.5s while you wait for mapgen. For one that's just really annoying and bad form, but it also would halt all network communications during that time.
    • 4. Given that, players will then be trying to mess with the lobby settings, potentially, while the map is generating. 200ms is not enough time to do much, but 1.5s is.
      • One solution to this is to temporarily make all of the lobby controls read-only, which was my solution in AI War Classic, but that's really messy and requires a lot of extra code (a bit of code for every stinking field). It's very annoying.
        • Another limitation to THAT solution is that the client doesn't know that the host is in the process of map generation when using that solution, and so we can't lock all their controls. In AI War Classic, this is why most of the controls for map generation could not be edited on clients.
      • The second solution to this is to just have two copies of the data. There's one copy for display, which mapgen doesn't use. It's what players see, and they can edit it from anywhere at any time. When it's time to submit it to mapgen, it makes a copy of itself and we're off to the races.
        • This is the solution that AI War 2 has been using since about 2018, but the problem is -- well, now we've got two copies of the data. It's possible for them to get out of sync, which is what has been frustrating me for several hours yesterday.
        • The good news is that once you solve that problem once, it's solved until the next time you make a structural change to the entire program, but it's complicated enough that it will deter you from making structural changes to the program because of what a huge pain this is.
  • All right, so what are some potential solutions that are better than the above? This is what I've been thinking about since last night.
    • Firstly, rather than having a lobby-copy and a permanent-copy (which is really what I've been doing all this time in AI War 2), maybe the sole copy of the config just makes a copy of itself that gets passed around in mapgen.
      • I don't like this, though, for a couple of reasons. One, it's entirely possible for a mapgen programmer to still go reach the central data by accident, and thus have a threading error or worse. Two, it requires adjusting how all of the mapgen code passes data around. And three, it is a minor pooling nightmare, since we would want to make sure that all of this gets pooled properly, even when mapgen fails or exits early, etc. There are "valid" mapgen failure cases, such as "there's not enough room on this size of map for all the factions you chose," for instance.
    • Okay, when that's out... hey, you know what? Our mapgen code really is more like 200ms to 400ms at the moment. How much clicking CAN a player do during that time? One or two at most, generally.
      • This is the solution I'm going with: the UI is still responsive and listens to players during mapgen, but no GameCommands get processed during mapgen. They all get processed right after, instead. Problem solved, and a single copy of configuration now.
      • Background: when players click around and make changes in the lobby, they don't directly change data. Instead, all of these things are generating GameCommands, which are what then actually change the data. There's normally a 100ms lag on this, because networking framework and sim cycle.
      • The leap from a 100ms lag to 200-1500ms of lag in the event that a new map is actively generating is really not far at all. And realistically it's typically more like 200-400ms, as noted. This lets players continue to have a responsive UI, and I expect in almost all cases they won't even notice the difference at all. It keeps threading issues away from mapgen, lets mapgen continue to work as before, and removes the complexity of having two copies of data.

• Milestone able to properly start a game, and reset a game, with all of the proper settings and things working as expected. Editing faction and galaxy option data in the lobby and in-game both work as expected.
  • New problems: not put in charge of any faction after loading savegame; savegames from yesterday broken; quickstarts broken.
  • Reminder to self: tutorials still broken, check to see if there are still issues with random faction external datas in lobby not being properly attached to a faction (this is unknown, it might have been fixed by the work from this morning).
  • Observation: the lobby feels more responsive to me, but it's hard to be sure if that's just me imagining it.

• Fixed a single line in the deserialization code that was causing quickstarts to not load properly.
  • They still don't load fully properly after they initialize, but the data I need gets off disk now, and so the only thing left with them is making sure their faction boot up properly. The fact that the data on disk is good for actually getting that started now is the important thing.

• Milestone: Quickstarts once again fully work, including loading them into the lobby or starting them directly. They properly strip out and recreate the bits that need that sort of thing, and all seems to be well from what I can tell.

• Fixed a bug where savegames did not remember who was in control of each faction on load.

• Fixed a load exception for the astro trains faction, and another for tamed macrophage. Just one-liner translation type issues.
  • Got another one for the dark spire fixed.

• Fixed an issue that made it so that when you added a new faction via the lobby, it did not actually fully generate things such that they had their needed attached info.

• Fixed another one-liner that was the vassal flag for units being read wrong.

• Fixed another one-liner that made it so that the second time astro trains were refreshed on a game load, they'd freak out.

• WorkThreadOnly_StartNewGame and WorkThreadOnly_GenerateNewGame and DoFinalBitsAfterMapGenerationOrFixingSavegame are all now instrumented to report errors better.

• Discovered that Random Factions are broken at the moment, but that's okay; I will get that fixed after this build.

• Fixed an error in Dark Spire where HandleTerminiiAndEpistyles assumed that DarkZenithPerUnitBaseInfo would be initialized, but it was not always (this sort of thing was me being hopeful, but my fallback was explained yesterday).

• Fixed a pair of errors in the enraged and tamed macrophage trying to update their allegiances like the regular macrophage.

• Fixed a one-line bug (this was not new, I just happened to find it) where Neinzul Sappers would not properly get their intensity if you put them on a minor faction team.

• It looks like random factions actually DO work, after a fashion, but are invisible in the lobby for some reason. So more will need to go into looking at that.

• Added the following new option to the debug settings in personal settings, in Debug, Detailed Logs: Log Fleet Setups For Custom ARS Types
  • When this is on, during startup it will log to the debug log any information about AddedToFleet_MinorFactionStrikecraft and AddedToFleet_MinorFactionFrigate. If you are getting an error with some custom faction structure that lets you hack it like an ARS, this will let you see what is wrong with your xml.
  • As you may be able to guess, there's a data problem here at the moment. It looks related to the dark spire, and I have no idea if this is new, or if this is an old problem that actually just gets reported now. But it sure spams the game once it starts, at the moment.

• Added a new IsSvikari onto the dark zenith core BaseInfo. This is used to let the Svikari not try to find the SpawningOptions field.

• I had corrected the internal name of the Dark Spire to be DarkSpire instead of "Dark Spire", but I had not fully corrected that throughout the rest of the xml. That was leading to the flood of exceptions, which I could now easily find thanks to the new debug setting that I put in. Those are all now fixed.
  • Additionally, I'm not sure if I made the same fix to Fallen Spire or not, but either way the random factions had a wrong value for them, and I corrected that also now, too.

• Discovered that the Zenith Architrave also do not always proactively initialize their units' external data. Fixed so that it works around that.
  • Same for macrophage teliums. Same fix applied. I suspect there will be a variety of this sort of thing showing up for factions this week.
  • And some more for the Dark Zenith.
  • The old structure for the game kind of encouraged lazy-loading of externaldata, so... here we are. We can either fix them to proactively attach said data (that would be nice, but has not too much point), or we can set it up to lazy-load as-needed here, too (which is what I'm doing now).
    • Despite the pejorative-sounding name, there are sometimes benefits to lazy loading (sometimes EXTREME benefits), so this isn't really a criticism of anyone. It's just an observational about differences in architectural assumptions.

• Fixed a minor linkage issue between the territory dispute subfactions and their parent. One-line of code type thing (most of these are one line of code fixes).

• Fixed a variety of conversions in the Dark Zenith resources converter table that were using the dark-zenith-specific variant rather than the DarkZenithFactionBaseInfoRoot that lives underneath both the dark zenith and the Svikari. This led to exceptions on the Svikari. Easily fixed with some search and replacing.

• Fixed a really silly issue with purging dead fireteam units, where I meant to check IF they were scourge, not assume they HAD to be scourge.

• Milestone: with over 20 factions added, I can now run the game for over twenty minutes without any errors popping up. They are fighting, and doing what they do, and etc. All of the errors thus far have been of the sort of one-line translation things that are fixed in about 10 seconds. Writing the change log note takes as long as finding the error in the code and fixing it combined.
  • Factions tested so far: AI and all of the auto-added factions, human empire, AI risk analyzers, astro trains, dark spire, dark zenith, dark zenith svikari (multiple at once), devourer golem (multiple at once), macrophage (multiple at once), marauders, nanocaust, nomad planets, territory dispute, zenith architrave (multiple), zenith miners, and zenith trader.
  • Not yet tested in that way: necromancer, fallen spire, HRF, migrant fleets, sappers, scourge, zenith dyson sphere.

• SphereNoLongerAntagonized() has been instrumented for better error reporting.
  • Fixed what seems to have been two translation errors on my part (the worst ones I've seen yet, but they were still quick to fix, knock on wood) where all dyson spheres were being treated as if they were antagonized, and constantly trying to revert to normal.

• Milestone: I have now run with all the other factions that I mentioned before, PLUS the necromancer, the fallen spire, the HRF, migrant fleets, sappers, scourge, and zenith dyson sphere. None of the others had any errors. That is every faction in the base game and all of the DLCs, even the unreleased one.
  • Please do note I didn't actually start the spire quests or anything like that, so there may be errors once we get into that more. It's hard to do a quick test of fallen spire. But at any rate, the factions in general there all lied for 15 minutes without any errors now.

• The faction utility methods to find either a human king or AI king are a lot more explicit now, and will only throw errors if you say they should. Additionally, if you ask them to throw an error, that error will be very visible. This eliminates any chance of invisible log spam that slows down your game, while keeping the flexibility to error if we really need to (we pretty much never need to in the game itself, but some mod might).

Mod Updates

• In general, expect all code mods to be broken for a bit. All xml mods should mostly be fine, but it depends on the mod.

• AMU:
  • More functions for the ManagedFleetHandler:
    • It now has a list of tasks to do, with their own on-failure and on-flagship-death finisher actions, and settings such as whether flagships are carriers, unload ships to attack enemies, etc, etc...
    • The OnSuccess/Failure/FlagshipDeath extra logic virtual functions have been renamed to OnEvery[something], and fire on every [something].
    • Additionally there now is a function for every entity removal that is triggered by the ManagedFleetHandler.
    • Tasks will now be archived and can be looped for perpetually repeating orders with the matching OnSuccess or OnSuccess_Fallback action.
    • The ManagedFleetHandler now has default settings for all tasks, so to reduce the hassle of setting up new tasks. This comes with a bunch of variants on the constructor to set up tasks easier.
  • Fixed one more reference to The Neinzul Abyss being DLC 4 instead of 3 in AMU.
  • The AMU_BattleshipBase entity type and all entities that inherit from it now use large health bars (visual change only).
• Devourer Chrysalis:
  • Updated to work with the above changes.
  • This hopefully fixes all remaining cases where the Devourer Hunters warp out but don't refund their strength as budget.
• NEW MOD! More Frigates:
  • Adds several starships from AI War Classic to 2 in frigate form.
  • Adds two new starting fleets composed solely of these frigates.
• NEW MOD! Super Saiyan Overlord:
  • This mod replaces the default overlord with a new one that is equipped with new weapons and receives a massive stat buff.
  • The Overlord Phase 2 also now sends raiding fleets against the player as long as it is engaged in combat with the player's units.
  • This mod is meant to be the ultimate test of your skills.
• More Starting Fleets:
  • Moved several fleets into their respective DLCs in order to allow players to use the mod even without said DLC (they will be missing the fleets that uses ships from the DLCs).
  • Reworked/Buffed/Nerfed several fleets and removed some that were too similar to other fleets.
• More Starting Fleets Plus:
  • Due to lack of demand and technical support, said mod will be disabled and sunsetted.
  • May return at a later date but currently is too similar to More Starting Fleets without bringing anything unique to the table.

Prior Release Notes

AI War 2: Finalizing Multiplayer