AI War 2:The Great Refactor

Known Issues

• Any bugs or requests should go to our mantis bugtracker
  • If you need to submit log files, those can generally be found under your PlayerData folder in the folder your game is installed in. The most relevant one is called ArcenDebugLog.txt. You can send us the whole thing, or just strip out relevant parts.
  • In rare cases, mainly if your entire game crashes (that almost never happens), we will need your unity player log. That gets overwritten the next time you run the game after a crash, unlike the other log. These can be found here:
    • Windows: C:\Users\username\AppData\LocalLow\Arcen Games, LLC\AIWar2\Player.log
    • macOS: ~/Library/Logs/Arcen Games, LLC/AIWar2/Player.log
    • Linux: ~/.config/unity3d/Arcen Games, LLC/AIWar2/Player.log

• Multiplayer is in public beta, as noted below. There is a detailed multiplayer guide that we are working on building up.

• Feel free to join discussions on discord!

What Does Multiplayer Beta Mean?

Please see this link for details on multiplayer. This wound up taking up too much space in this document, so all of the multiplayer-relevant bits have been moved to the other page.

What's this phase all about?

We hit a point, while working on the beta branch, where we realized we just needed to really rework a lot of the underlying code to make multiplayer and modding and so on be as bug-free and compatible as we wanted for the future. So this was supposed to be a digression that would take a week or two at most, but wound up taking closer to four. And there are then after-effects for a while, on top of that. But the good news is that it dramatically improves... performance, multiplayer correctness, how easy or hard the game is to code to, etc.

The goal of this phase is to finish up reworking all of the things that we need to, for maximum long-term enjoyment of the game. This also means getting multiplayer absolutely completely finished and working. After that, we'll be shifting back into work on DLC3 (The Neinzul Abyss), which will be the final DLC for the game. The various remaining kickstarter items will also be taken care of, plus some other bits that we just always wanted to add for the game, and after that we'll be moving on to other projects aside from bugfixes and balance tweaks and modder-support throughout 2022.

The target release for DLC3 is in January of 2022, and that is when we'll also have the AI War 2 Complete Edition. After that, it will just be fixing material defects, and supporting modders. We expect it to have a long life if AI War Classic is anything to go by. The game is in a really good state and only getting better, and doing a refactor of this scale near the end of the project development cycle is a sign of our commitment to that longevity. Even if you aren't giving us more money in the future for this title, we want you to be able to enjoy it for years and years.

3.704

(Not yet released -- we're still working on it!)

• In the lobby, the AIs now have a nicer display on the left sidebar of their difficulty and their type.

Bugfixes

• Fixed exceptions that could happen with CustomData_StartingAILayout, CustomData_StartingAIPlanetRatio, and CustomData_WormholeBorerIncomeModifier if the data for them was blank. Now it just uses the default value in those case. Not entirely sure how the data can be blank, but in some fashion or other it was just data that was initialized without the player actually making a choice about it.
  • Thanks to BoBy for reporting.

• Fixed an exception that could happen in Marauders stage2 with lazy-initialization of their external data.
  • Thanks to BoBy for reporting.

• Fixed an issue where the lobby and in-game factions screens were showing the same information in the tabs on the left for every faction beyond the first for any given type of faction. So for instance, each AI showed the info from the first, etc.
  • This actually finally explains how the attachedfaction used to be null in that area, which is nice to know I wasn't crazy. I couldn't see how it was null, but it was using something outdated that I forgot existed. Now it uses the expected central items.

• When I set up the AI data and subdata in as part of the great refactor, I accidentally wound up having two copies of the central AI type and difficulty. One on the main object (not needed), and one on the sentinels sub-object. Most things were reading from the latter, but the latter was also always set to the defaults rather than the values you chose.
  • Now there is only one set, and everything reads from that, and this works great.
  • I made this change in a way that breaks savegames (I didn't have to, but it's slightly more efficient and I'm about to break savegames today anyway).
  • Thanks to Arctic for reporting.

• At least since the great refactor, but maybe longer, the escape menu where it shows the list of factions has been a bit of a mess, with extra intensities showing and so forth. Fixed up those bits, and then also just made it look a bit nicer than it did in the past for the AI types in particular.
  • Thanks to Arctic for reporting.

• Fixed a nullref exception that could happen in SpecialFactionPlanning.GetNeedsToRun().
  • Thanks to BoBy for reporting.

• Fixed a cluster of issues that could result in the AI sub-types or sub-difficulties not being set properly when loading savegames in some cases.
  • This then fixes all sorts of other random issues that would show up in other spots if this happened. The central logic for these is also now set to error if they still remain null, so that we know what the true problem is rather than worrying with random other issues.

• I can't (yet) duplicate the error, but there was a pathfinder being used from two contexts incorrectly, and I've now put in a lot more code around making that not happen. It appears that this was caused during an exogalactic attack, but that was honestly the part that looked correct from the log -- there was some prior call from the AI sentinels to the wrong pathfinder, and that set things up for the later error. I now have it so that it will detect the FIRST wrong call, and not just when there are calls from two contexts. I've done a manual code review of all the affected areas, and can't see how it would have a problem, and I've done some testing for a while as well, so I guess we'll just have to wait for it to trip again. Next time will be more informative, or should be.
  • Thanks to Vinco for reporting.

• HandleExogalacticAttacks has been updated so that it is now using threadstatic working collections so that it is fully threadsafe. An error in a log may have been due to that method being called by two factions at once, or it might have been related to the pathfinding issues. Either way, I made the method threadsafe, made it slightly more efficient, and then also put in extra logging spots so that we will know which it is if it ever has an issue again.
  • Thanks to Vinco for reporting.

• Fixed an exception in the scourge that could happen during long-range-planning if their warriors were not properly assigned their BaseInfo data yet. Honestly this seems risky from a threading point of view to have it handled this way, but apparently it worked prior to now. At any rate, now it should still work.

• Further fixes to the scourge. Their data is now initialized in Stage2 if it is missing, which should avoid any LRP issues or other cross-threading issues, and some followup exceptions that I saw.

Simulation Performance Improvements

• Improved the efficiency of the strengthcounting background thread when it comes to looking at incoming waves and adding their strength to your totals.

• In general, improved the efficiency of strengthcounting by making it use a ton more background threads. This is one of those areas of the game that can be a real sim-speed limiter, it turns out.
  • In particular, with higher numbers of planets, this can peg even really new, high-speed high-core CPUs. I have it spreading out amongst CPU cores better now with more threads, but I still see it using too much, so I am going to make some further improvements.
  • This and metal flows planning are two areas where I should be able to make the simulation run a lot better for everyone, particularly in giant games, now that I can see that they are spiking up as problem points pretty frequently even on my Ryzen 7 4800H.

• MAJOR refactor of the strength counting logic to time-slice its work over the span of 10 frames (1 second).
  • As part of this, also discovered that too much Parallel looping over planets was a major performance problem inside this thread.
  • Even on an 8-core machine, this was spending more work getting those threads up and down than it was getting benefit out of the threads themselves.
  • Also, as part of this, I've discovered that the strength counting itself is one of those things that has a MAJOR impact on the game the more planets there are. I had wondered where all our performance was going with high numbers of planets in the past, and... well, here's the answer. This is one of the most longstanding mysteries of the game, solved.
    • Even now, with the awesome new time-slicing, the difference between me running this on 80 planets and 110 planets is "always buttery smooth" and "kind of choppy at times, but not so bad as before." This was with two AI factions and the fallen spire.
    • When I stick with 80 planets but add 10 AI factions and one each of most every other faction available in all 3 DLCs, plus two of a few of those factions, then the performance of this -- from second one onwards -- is abysmally low. I go from having something that takes about 26ms on average (that's lower than 100ms by a lot, so that's good), to something that takes 300ms on average (that makes for instant incredibly high sim lag) with spikes to 600ms. This is AFTER all my existing optimizations.
  • I think that what I'm going to do is actually spread this out even further, to time-slice this over 3 seconds rather than 1 second. My scenario with such an insane count of factions is not a reasonable one to play, but I have a very high-end CPU, and I would like for the base case to have a lot of extra wiggle room where it's not even coming close to performing poorly.

• More strength counting performance improvements:
  • The final step of copying the strength aggregation data from the working location to the final data spots that are used has been moved to the same background thread as the rest of the strength counting.
  • In my prior version, I had the planet strength counting split over two time slices. That is now split over 5.
  • In my prior version, I had the aggregation strength counting split over four time slices. That is now split over 20.
  • With this setup, I now get vastly better performance in my unreasonably huge test bed, and notably better performance in my regular use cases. However, there are some major downsides, so I'm going to redo this yet again from another angle.
    • Firstly, in the huge testbed case there are still some outlier spikes, which get worse as time goes on. So an initial sim speed that was better deteriorates over half a minute or so. I'm not entirely sure why that is, but I have some guesses.
    • Secondly, the human-facing data is only being updated every 3 seconds now, which really does not feel right. When I explore the map via a cheat, it taking 3 realtime seconds to appear feels very off-putting.
  • Don't worry, I have a plan. :)

• Okay! A whole new approach has been implemented for the strength counting thread. The short answer is that performance is through the roof.
  • Before, I was trying to time-slice at manual spots that I was personally selecting. That just wasn't working out very well, for a variety of reasons. We would have some slices that ran well over 100ms, and thus slowed down the sim, and others that were still a lot higher than I wanted, while others were almost no time at all. But... even worse, which parts are the most expensive is something that depends on the specific savegame (how many units, how full specific planets are, and many other things). So there's literally no way to make a generalized solution to this in a hand-designed fashion. It will always fail somewhere, making it completely impossible to solve that way. So...
  • The new solution that I implemented is ADAPTIVE time-slicing. Here's how it works: essentially, at certain juncture points it decides if it has been running too long, and if so then it terminates. Wherever it is, it doesn't matter. The next time it is run, it resumes right where it left off. To a non-programmer this might sound easy and obvious, but when you're talking about a linear function it is really not a common pattern to do something like this.
  • The metric that I wound up using was 40ms. If the current iteration is taking more than 40ms, then it stops for now. I want the remainder of that 100ms to be used for the rest of the sim. And honestly, since I can only check but so frequently how long it has been it will sometimes hit 60-70ms per slice like this, anyway. In a suitably rough location in a suitably huge save, it can get a bit over 100ms, but rarely. I've iteratively optimized that down so it doesn't happen much.
  • Overall this now means that the strength-counting data is slightly out of sync on clients and hosts, and it means that it is not up-to-the-frame accurate anymore (well, not always). In small savegames, if it can run all in one frame, it still will, of course. But for really huge savegames, it will take a bit longer.
  • I had a couple of test cases, and both of them improved dramatically.
    • One was a basic game with two AIs and the fallen spire. Under the hood, this meant 23 factions. I was testing on 110 planets, which earlier today was feeling a bit choppy even though it was close to being perfect sim speed. Now it's running without the chop, at full sim speed. It takes about 300ms (76ms of calculation-time, which is 40% of realtime) for the data to be updated now, versus forcing it to be updated within the initial 100ms span. Previously, this had meant that there was only 24ms of time to do ALL the rest of each sim step, but now there's lots more, so it runs well.
    • The second test case was a truly insane one, with 80 planets and 10 AIs and one of every faction along with two of a couple. Under the hood, this makes for 111 factions. Prior to this specific update, I was getting about 40% sim speed on my very nice CPU, and it was so choppy it was almost like a slideshow. Now I get well over 60fps, and almost perfect sim speed (95% or more), and it takes about 1.5 seconds for the strength counting data to be updated (670ms of calculation time, which is 40% of realtime).
  • This is... an incredibly dramatic improvement. A ton of game scenarios that were never possible before are now potentially playable, based on this early data. And since the code is adaptive, it should handle many cases such as super high unit counts or super high planet counts or whatever else. I'm going to test the latter of those next, just to see what it's like.

Options For Larger Maps

• The planet count max for all map types has been increased to 300 from a prior range of 120-160.
  • With 300 planets, just starting the game used to give very poor sim speed, but now that problem is solved. Framerate still takes a big hit even when you aren't looking at anything in particular, but it is north of 40fps for me on my dev rig, so it seems like an option worth having (with warnings in the lobby of likely performance issues if you don't have a very very nice machine).
    • I am reminded of the Supreme Commander maps that were at the largest possible scale, where they ran poorly on basically any machine that existed when the game released, but now they run fine. This is kind of like that, although it is playable on current machines if you don't have too many factions.
  • I tried with up to 500 planets, and it took 15 seconds to generate the map and then I was getting literally 20fps even though the sim speed was about 80%. That is still a titanic improvement from the past, where it would have been sub-10%. Nevertheless, 500 planets just feels way too laggy and likely always will given how the game is structured. I'm not quite sure where the new bottleneck is in the code, but the game just wasn't designed for that sort of planet count, anyway.
  • Planet counts higher than 170 and 220 now give you different warnings about possible performance issues if your machine is not very good. So you have the option for these mega maps, but hopefully also know what you're getting into.

Beta 3.703 Smooth Sim

(Released September 29th, 2021)

Improved Simulation Handling And Speed

• Making some changes to the threading model for AI War 2 to make things a bit more clear:
  • ProcessSimStep is now called ProcessArbitraryFrameOnMainThread, because that is FAR more clear and accurate. Even I was confused by the old name.
  • DoActualSimStep is, similarly, now called ProcessCoreLogicForArbitraryFrameOnMainThread.
  • Fixed a bit of logic that was running twice on the host and in single-player, eating up some extra processing time. Not much, but nice to clear that out (this happened as part of The Great Refactor).
    • Actually, this might have caused flickery lines, not sure.
  • OnClient_SecondsStoredUpForNextFrame is now OnClientOrHost_TimeStartedLastFrame.
  • All of the short-term sim context work is now handled on the main sim thread, rather than on a secondary worker thread managed by that.
  • These short-term threads are also now executed on the main sim thread via a Parallel call, which makes it so that they are able to happen much more efficiently in terms of how they get started and finish.
    • This is a MAJOR adjustment to the game, and likely to improve performance quite a lot, potentially on older machines the most (but really for anyone).
  • A whole bunch of other logic that I've hated for a long time relating to how the simulation kicks off threads and checks for gamecommands and so forth is a lot simpler now.
    • The idea is that we want to make sure there is minimal input lag in multiplayer, and that we're seeing as close to zero lag between "finished one thing" and "started another" as possible.
    • Previously, if you had a choppy main thread framerate for whatever reason (low powered graphics card and giant battle?), it could make for your sim speed being affected since that was being time-sliced over frames. That sort of time-slicing is no longer done, which leads to more stable simulation kickoffs.
  • This whole thing actually simplifies several areas that were really complicated before, and which were impossible to profile well, because they had noise from cross-frame data in them that made it impossible to know what was just general frame time, and what was actual processing time on these themselves.
    • This simplification should lead to some performance benefits in intense savegames in general, and in multiplayer basically all the time. Not that multiplayer is ready to go again yet since The Great Refactor, but this will be a big benefit for it once it is back up.

• In the thread timings details window (right-click the timer in the bottom left window), you can now see the last sim time in milliseconds, and the average sim times of the last 20 sim frames (2 seconds' worth).
  • If this is above 100ms, then you will have a slow sim because of the sim itself. If it's below 100ms, then if your sim is seeming slow, it's because of factors outside of the sim itself. That could be frame timings or other factors, but this is giving you the details on JUST the sim itself, which is useful for isolating what the problem is. In multiplayer, "sim speed" can be low because of network issues, for example, and this lets you see that.
  • Interestingly, I was seeing some performance ratios hitting down into the 80% range when looking at the traditional view of things, but I can at the same time see that the slowest actual sim frame time was 36ms. That is faster is actually needed in order to have a perfect 100% sim speed. So either the calculation is off, or something else is causing sim speed to stutter.
  • Until today's refactor of the central sim timings, I would have had no way to tell which. Now I can actually investigate it.

• Fixed an issue (that has been incredibly longstanding -- years and years) where after the lobby, your game would be acting like the sim steps were extra slow.
  • Also fixed a much more recent issue where after the lobby it would say there were a bunch of faction threads that had been inactive for a long while.
  • This sort of thing is trivial, but at the same time could be alarming to new players and is not the sort of thing we want people seeing right as they start a new game.

• When mousing over the clock in the bottom left and seeing the simulation speed tooltip, it could have a pretty wicked flicker. Fixed.
  • Additionally, when mousing over this or clicking into it, it now also shows you the framerate that we have calculated you are getting. Previously we only showed that on the main menu for whatever reason.

• The way that I have been calculating the simulation speed (for visual display) over the last years has been... cough... truly asinine.
  • It was some strange logic where it was kind of half calculating itself based on the time that it was called in the UI when the frame number had changed, rather than just doing a proper calculation from "frame number counts up to next time frame number counts up."
  • This has been corrected, and the timings are now vastly more accurate.

• Because of the combination of changes in this release, I now can see that I am often tripping the sim to start slightly slower than I should (about 95% instead of 100%), even while running at 120fps and with a sim load that is 30% of what is available, which is leading to some slight choppiness as well, probably because of how I am handling this between frame-timings.
  • Ironically, unity themselves had something along these lines themselves recently: https://blog.unity.com/technology/fixing-time-deltatime-in-unity-2020-2-for-smoother-gameplay-what-did-it-take
  • I can't blame my own frame timings being off on them, but the mistake that I made is very similar to theirs. I'll have to get that straightened out now, too.

• Well now! Here's another REALLY nice improvement, which will have a positive impact on multiplayer and single-player.
  • Essentially, I smoothed out the sim-steps by sampling what recent frame-times were (the most recent 30 only, no matter the framerate), and then taking the average of them. If the next sim step is not yet ready to start based on our target of "every 100ms," but 55% of our average recent frame time WOULD push us past the sim step start time... then I run the frame on this sim step anyhow.
  • This essentially works out so that we get between 99-102% sim speed on average, with a slight bias towards being a tiny bit fast (that's why 55% instead of 50%), whereas before it was averaging only 95% pretty much all the time.
    • Why the bias towards being fast? Well, ship movement gets slightly choppy when the sim is running slow. The UI layer is expecting things to be running at exactly 10fps on the sim, and is basing its linear interpolations on that being correct. Being slightly behind on a lerp will still look smooth, but being even a bit slow (well, more than 1% slow anyhow) introduces micro-stutters.
  • It has been quite noticeable when you watch ships move around, how they have micro-stutters in past versions, and it has always bugged me. That micro-stutter is now gone.
  • This also has positive effects for multiplayer, and really single-player also, in making the clock move at a consistent speed rather than slightly too slowly, or at slightly different rates on computers with different framerates. I believe that in the past the drop in sim speed, and the micro stutters, were worse on lower framerates. When the frame times are larger (more coarse), it's harder to get exactly to the 100ms interval I'm seeking.
    • At 30fps, every frame is 33.3ms long, for instance, so previously the sim steps would be off by a range of 1-33ms. In the new system, by splitting the difference the way I am, the divergence can only ever be 1-16ms at 30fps. This is still large, but it's now 16% of the total runtime instead of the (huge) 30% of the runtime of a sim step.
    • At 100fps, we get 10ms per frame, and before it ranged from 1-10ms off. Now it just ranges from 1-5ms off, which is much better.
  • TLDR: gameplay in general is now smoother visually, on all hardware and all framerates, and multiplayer will diverge less between machines of different power levels.

Bugfixes

• Quieted the error "Entity in CenterPlanetViewOnEntity null!", which could happen in some circumstances when clicking through wormholes. It's not an error that is actually worth talking about; we should just ignore it.

• Fixed an exception that would happen whenever you moused over wave notifications. This was another case of my efficiency-improvement code hitting a legacy case which errors on purpose to let us know. It was supposed to give us compiler-level errors, but unfortunately that didn't work as hoped, so now we have to find them in the runtime.
  • Thanks to CRCGamer for reporting.

• Fixed an oversight in the great refactor, where I forgot to actually clear the AIP info between loads of the game! So that meant that starting a new game after having been in a game with AIP would leave you with extra AIP and a strange AIP history. Loading saves was fine.
  • Thanks to Arctic for reporting.

• Fixed the bug (starting from just before the great refactor) where every time you load a savegame it would give you another 10 AIP. Yikes!
  • Thanks to Lord Of Nothing, Badger, Strategic Sage, Arctic, and HTL9000 for reporting.

• Fixed a pre-refactor bug that was introduced when I redid the notifications system. The spire debris notification always showed zero, and would trigger errors when clicked.
  • Thanks to Vinco for reporting.

Youtube Threading & Multiplayer Modding/Programming Guide

Coding With Chris: AI War 2's Threading & Multiplayer Part 1: Challenges

Coding With Chris: AI War 2's Threading & Multiplayer Part 2: Data Flow For Unusual Solutions

Modders: Do you need to watch all of the above?

1. Not one bit if you're not coding anything. If you're making xml mods, you can completely ignore the above.

2. If you already think you understand how RTS networking works, then just read the below images and then skip to about the 30-minute mark in the second video.

New Diagram For Modders/Programmers: AI War 2 Threading Model

New Diagram For Modders/Programmers: Multiplayer Data Transmission Model

Beta 3.702 Bugfixes And Mod Conversion

(Released September 28th, 2021)

Bugfixes

• Fixed a bug with all hacks where if the Fallen Spire was not enabled or installed, you'd get endless errors instead of the hacking screen.
  • Thanks to Arctic and UncleYar for reporting.

• The Zenith Architrave now does a lot more verbose logging of its externaldata in case anything is wrong with it.
  • From analyzing a specific savegame, it looked like there was something wrong with it, but now that does not seem to be the case.

• Fixed a minor nanocaust deserialization issue that would cause multiplayer clients to drop their constants and reload them every second or so. This was found during code review while looking for something else. Not a biggie, but could have been annoying.

• Fixed two deserialization errors that meant that any savegames with nanocaust in them could not be loaded.
  • Thanks to Magiik for reporting.

• Fixed a serialization issue in the dyson sphere faction that would throw warning errors, but not actually cause any problems.
  • Fixed a similar one in Nanocaust.
  • Thanks to Magiik for reporting.

• The scourge as a faction were throwing errors and shutting down in Stage2 if they were included in your save in certain circumstances.
  • The scourge have had their Stage2 instrumented more to give us more informative feedback when that happens.
  • It appears that some Scourge Spawners could sometimes be missing their externaldata. This is now fixed.
  • Thanks to Magiik for reporting.

• Okay, when I reorganized the entire savegame file, one of the things that I changed was how galaxy info was deserialized relative to the entities in it. Normally entities are units -- ships, structures, etc. But sometimes they are wormholes.
  • Long story short, I had an oversight where I was calculating which wormholes are linked to which other planets BEFORE we had loaded any wormholes out of the savegame, meaning that each planet thought it had no links to any others. That meant that your movement commands between planets would fail after you loaded a save, but not after you started a fresh game.
  • Fixed, and this also fixes existing saves if that matters.
  • Thanks to Daniexpert and Andriy Mykhaylyk for reporting.

Mod Updates

• Civilian Industries has been converted over to work in the new framework, although it has not yet been tested. There is a four and a half hour youtube video showing the entire process, with commentary and advice on a variety of coding things.
  • https://youtu.be/3Aw34XjO9nk

• The following mods have been hidden for now, since they are nonfunctional for the moment.
  • AMU (SirLimbo is actively working on this. You can follow him on the modding discord channel. This may be quite a while coming.
  • Devourer Chrysalis (Again, NR SirLimbo is working on this. It's more than 100 hours of work, so it may take a month or two).
  • Kaizer's Marauders (Same notes as the above two, SirLimbo is still working on it).
  • Macrophage Histiocytes (and spire integration) (These are by StarKelp, and I may have time to fix them someday, or he might get around to it, it's hard to be sure yet).
  • The Reprocessors (this is a CRZgatecrusher mod that needs updating, and I haven't heard from him yet).
  • AI Shield Generators (this is a cml mod, and this one is going to be retired, with the author's permission, as its functions are going to be built into the base game with consultation with the mod author).
  • This should solve any confusion about which mods do and do not work. All of the other mods that you see are working so far as we know, although of course with Civilian Industries that has not been tested at all yet, but I have done the conversion as part of this build.

Beta 3.701 Natural Object Order

(Released September 23rd, 2021)

• Made it so that saves from 3.700 are no longer valid to load, because they are SO broken there is nothing new we can learn from them now.

Bugfixes

• Apparently we had never really used bool toggles on factions prior to now, because there were some errors lingering in there from way on back. During TGR, I converted the "debug mode" options for various factions to all use those, and that then led to errors in the faction screens for them. Fallen Spire "Super Easy Mode" is one example of those.
  • This all works now!
  • Thanks to Vinco and Magiik for reporting.

• Adjusted things so that the "Skipped an extra mapgen call that was substantially the same as one that was run X seconds ago" won't be hit if there are changes in the number of factions compared to now versus then. Just in case. I am not positive this was a problem before, but it might have been, and now it can't be.

• The warning for things like "Roaming Enclave Migrant is 23 characters long, and needs a display_name_for_sidebar variant that is 18 characters are less." is now an outright error, rather than a quiet thing in the log. Since we are making sure that mods are updated and such anyhow, this is a good time to make sure that we don't have things like this sneaking through.
  • Fixed the two instances of this sort of thing that were previously in the codebase in DLC3.

• Fixed an issue where NaturalObject (no owner) faction was not always set to faction index 0 anymore. This was something I was already aware of, but wasn't sure if it would be a problem.
  • It turns out that this was a major problem, and would cause various ailments such as units not reacting to aggro and releasing units, to all sorts of other potentially unknown stuff. This will not fix existing 3.700 saves that had this problem, but any new saves will not have the problem.
  • Additionally, this was causing units to not respond to move commands properly between planets. This again isn't fixed in existing 3.700 saves, but any new saves no longer exhibit the problem.
  • Thanks to Vinco and Daniexpert for reporting.

• Fixed some general issues with GetControllingPlanetFaction() and similar methods where they could now return null at some points, rather than returning naturalobject when nobody owns a world.
  • Thanks to Daniexpert for reporting.

• Fixed an exception where if you tried to change the main AI color in the lobby, it would error.
  • Thanks to Magiik for reporting.

• Fixed a rather egregious and VERY longstanding bug where if an unclaimed unit (a capturable unit) was placed into a LazyLoadSquadWrapper, it would return null from that squad wrapper.
  • This was potentially never a real problem before, but we now use lazy load squad wrappers on fleet centerpieces, which meant that this bug now showed itself by deleting all capturable golems, arks, transports, and so on.
  • This bug is at least 3 years old, and who knows what it was affecting (maybe nothing, honestly), but it just had now found new expression. Glad to have that fixed.
  • Thanks to Vinco for reporting.

• Fixed exceptions that could happen in a number of places if you tried to get the wave info, because it was using the old ToString() method rather than the new efficient method that writes to a buffer without causing a lot of garbage collection.

• Fixed an exception that would happen in the AIP history window, because it was using the old ToString() method rather than the new efficient method that writes to a buffer without causing a lot of garbage collection.
  • Thanks to Daniexpert for reporting.

• Fixed a typo in the MarauderOutpostRaiderPerUnitBaseInfo that had it linked to the outguard class instead, which then led to errors. This will still be broken in old savegames, but is fixed in any new ones.
  • Thanks to StarKelp for reporting.

Youtube Deep Modding/Programming Guide

Okay, here's the modding guide for AI War 2 now: https://www.youtube.com/playlist?list=PLc2dK5ROW0WCjIHUBRJPp5oiJkTIMZupL

Modders: Do you need to watch all of the above? No, absolutely not. The timestamps do point out some interesting things that you may want to listen to, though, if a topic strikes your fancy. Even the regular code developers, like Badger, may find some of those sub topics rather edifying. You can click the timestamps in the video description, and it takes you right to where I am examining and discussing some particular piece of code. This jumps all over the place for different examples as appropriate, and does some work on AMU but not remotely the whole thing. It also shows some simple and complex faction data from the base game, and discusses both the theory behind these designs, as well as the practical uses of it. I feel like a professor trying to impart an entire textbook in a 3 hour lecture, but such is life. I will be available for questions and consultations. I want people to have a good experience with their transition to the better format, and also understand the sheer power and performance gained by doing the transition.

Beta 3.700 Bulk Of The Great Refactor

(Released September 22nd, 2021)

Known Issues

• This is a brand new version of the game in a whole lot of senses, so it is expect there are going to be a bunch of random errors that are not known. Please don't stress about it. If you just report them, most should be very quick to fix. With that said, here's the list of things we do know are not working at the moment:

• Tutorials are temporarily broken.

• Beacons for factions are temporarily removed. They will be re-implemented in a more efficient manner soon.

• Most mods are probably broken completely or partially.

• The kills and deaths data that you could get is no longer present. It was limited and stupid and boring, and I'm going to give you cool data, with graphs, instead.

• Random Factions are broken.

• All savegames prior to this version will no longer load in this new version, which is by design: we have completely reworked the savegame format to be more efficient.
  • I should warn you that I am going to break even NEW saves several more times prior to 4.0. This will be in pursuit of making the Fallen Spire and Necromancer factions feel better to play with, and to add ship loadouts (the AI War 2 version of modules) with minimal CPU overhead during gameplay, and a few other reasons like that. I will let you know when I'm done breaking saves, but until that point please consider any save as potentially invalidated by the next version coming out.

Random Features And Fixes

• Added a new is_immune_to_reconquest_waves="true", that can be used to make it so that a given faction won't have reconquest waves launched against them (other waves are fine).
  • This is also newly applied to both flavors of dyson sphere, which is new. This should let them get a lot less hassled.
  • Thanks to easytarget for inspiring this addition, and Badger for the notes on how to quickly add it.

• The "A Tale Of Two Stars" quickstart has been removed, because with the Dyson Sphere changes coming in this build, that scenario will no longer work. Likely there will be even more changes to Dyson Spheres over time, so it would probably just get a bit more busted with time.
  • Thanks to Puffin for reporting.

• For purposes of damages/penalties to damage output of weapons when under a friendly bubble forcefield, weapons now also consider themselves to be under a bubble forcefield if their parent unit is the one emitting said bubble forcefield.
  • For example, a Riot Cruiser that has a bubble forcefield and 10 guns on itself, if its bubble forcefields are up at all, its guns will work better unless it has the usual things that let it not cause that. Previously, it would only get the "under bubble forcefield" status on its weapons if some OTHER unit was providing said bubble shield.
  • Thanks to CRCGamer for reporting.

• Moved the mark level scaling data for Fleet Officers from being ZO only to also being applied to the base game to simplify balancing.
  • Additional revised offensive scaling of Fleet Officers to cap at 6x base damage instead of x7. This actually does cut a few of the more outrageously upgraded options by 50-70K DPS.

• Adjusted Combat Factory internal drone metal rate up from fixed 1000 to 1500 plus 250 per mark level.
  • Combat Factories have been pretty slow at replacing their drones for a while and unlike most of the support factories they have a much larger number they maintain.
• Adjusted Scrapper Factory internal drone metal rate to gain 250 per mark level.
  • Since this factory type essentially keeps a stock of knock-off autobombs as drones it just made sense to let it make replacements faster with higher mark levels.

• Adjusted Shieldwall Battlestation to be a much more compelling option overall.
  • It no longer gives units under its bubble shield a damage penalty.
  • Said shield also no longer shrinks when taking damage.
  • As it uses classic scaling and thus defenses only improve by up to 4x base value by Mark 7 it has been given an additional 25% base shield value.
  • A new mechanic where as long as its shield is up it takes reduced damage to said shield has been applied. Base 10% damage reduction with an additional 4% per mark level. This effectively means that at Mark 7 it has 34% damage reduction to shields.

• Adjusted Grenade Corvette strength bonus modifier slightly upwards to account for higher damage reliability with shot compression.

• Bumped up Harasser variant of MLRS Corvette base damage from 18 to 20. (Note that these ships can fire 8 shots of that damage per salvo)

• Previous adjustment downwards on target cap for Electric Bomber wasn't quite sufficient alone after shot compression made reliability higher. Further adjustment of base damage down from 900 to 750.

Quick Starts Data Overhaul

• Added a new ultra-micro format for "last lobby settings" and for quickstarts.
  • This isn't really so much for performance purposes as it is for forward-compatibility purposes.
  • I'm about to absolutely revamp the entire savegame format and how a lot of data is stored for basically every faction that exists, and I want quick starts to persist past something even that drastic.
  • Why the huge revamp of data? Primarily multiplayer, but honestly this is something that will likely yield performance benefits and added correctness in single-player, too.
  • We basically never do such huge revamps of data, because we absolutely loathe breaking savegame formats, but at the moment the time for half-measures is past. So we'll do it once, and make it really count.

• Starting to recreate all of the quickstarts. The first one recreated is now Helping Hands, which is identical in form and function to what it used to be, but now part of the micro-save format so that it will survive the upcoming purge night.

• All of the quickstarts from the base game have been updated to the new micro-save format that will be the only thing future-compatible after this build.

• Fixed a bug with Helping Hands 2 from The Spire Rises, where it was not working if you didn't also have DLC2 and DLC3 installed.
  • It is expected that there were other similar errors in some of the other quickstarts in DLC, but I am just fixing them all as I move them to the micro-save format, so I can't say for sure.

• All of the The Spire Rises quickstarts are now safely in the new micro-save format.

• All of the DLC2 quick starts are now fully recreated in the micro-save format.

Make Your Own Quickstarts!

• makequickstart, [folder name], [quickstart name]
  • Causes a new quickstart to be saved to the quickstarts folder in the new ultra-brief quickstarts format.
  • This quickstart format is required if you want to have future-proof data that does not break with future versions of the game.
  • Example usage:
    • cmd:makequickstart,1-Basic,Helping Hands
      • This would make a replacement for the basic helping hands quickstart.
  • Please note that you need to still add a .tooltip file next to any quickstart. You'll have to do that in your filesystem on your own, but it's just an ASCII text file with narrative information inside it about what the quickstart is about. Otherwise that tooltip will come up blank.
  • When you do this process, it also gives you a note telling you which dlc and mods are going to be required for someone to play that quick start. We had lots of cases in the past of accidentally requiring more than we meant to.

Bugfixes

• Improved the handling of DoForPlanetsParallel() so that it can't spam errors when you're exiting the game, or exiting to the main menu.

• Put in some extra debugging in DefinePlanetFactionDefenseTypesIfNeeded(), as it could throw exceptions in some cases.

• Fixed things up better so that any DLC can use features from any other DLC without incident, despite mods being required to have the parent DLC installed.
  • We originally had some cool chain lightning units in DLC3, which got removed after we introduced the restrictions, but we'd like for you to have those back!
  • There's also a detailed writeup on all this here: https://wiki.arcengames.com/index.php?title=AI_War_2:Features_From_DLCs

• Fixed what looked to be a bug in recent versions with outguard not actually spawning. This was based on code analysis, not testing, so let us know if something is off.

• Fixed a number of potential cross-threading issues with nomad planets using the main thread context on a background thread.
  • And some with wormhole borers, though fewer of those.

• Discovered and marked a bug where MP clients cannot cancel hacks in progress, but have not fixed it yet.

• Been meaning to do this for a while: multiplier_to_all_fleet_caps has been removed from the game.
  • It was previously on all turrets, set to 0.2. It was only there for historical savegame purposes, and messed with the unit cap math badly.
  • There are other modifiers in place about weapons strength or hull or shield health, and those are used in a variety of cases and we're leaving those intact and in place.
  • The ship cap one being removed means that if you have a mod that is seeding turrets, you'll want to remove this multiplier and then probably divide your number of seeded turrets by 5.
  • This should solve a number of longstanding but rare bugs that we keep seeing with the TSS and other turret-granters, but we'll see. It certainly won't hurt.

• Fixed typo in "ADVICE: Next Steps".
  • Thanks to Ben Parrish for reporting.

• Fixed a few grammatical errors in EncirclingSpider and IrefulMLRSCorvette descriptions.
  • Thanks to Fenrakk101 for reporting.

Rip And Tear For ExternalData

• All savegames and quickstarts older than 3.700 are now broken in the latest beta branch, and there is now a popup message explaining that any older saves you have should be finished on the older versions.
  • This is something that was not done lightly, and we're going to keep breaking beta-branch saves for at least the rest of this week as we refactor things to run in a high-performance, MP-friendly, modder-friendly, non-error-prone way. No more half measures.

• The StatisticsSet class, which was mostly unused except for some small bits of data that were stored in a way that was hard to get at, has been removed.
  • The actual data about offline achievements has not been in here for years, for example. As has the info about achievements won during a given specific save.

• Our ArcenExternalDataLookup class, ArcenExternalData class, ArcenExternalDataPatternTable class, ArcenExternalSubManagedData interface, and ArcenExternalSubSubUnmanagedData interface have all been removed. ArcenExternalDataPatternImplementationBase is also gone, and all the related stuff, really.
  • The generic "let's talk about external data attached anonymously to an object" concept that has plagued this game since our earliest versions is now going away.
  • For those who have coded factions or mods, please don't stress -- the vast bulk of your content will still work once I'm done rebuilding this in a more productive and direct way.
  • The problems with the old approach (of connecting the Core and External data parts together) were many, and reflect that it was originally kind of a prototype concept that never got refactored significantly enough because we didn't want to break savegames. I've spent the last year and a half trying half-measures to work with this stuff, but now it's time to go all the way.
    • Among problems with the old method: memory leaks, bad performance, inefficient data storage and transmission, no deliniation of data that we really consider central versus data that is just needed for host-only execution, and more.
    • There were also shifting and evolving standards in use over the years, leading to better and better code over the years, but still pulling along old bad code. Everything is getting a facelift out of this.
  • Last year and earlier this year you can see a number of cases of me trying to bring this to heel in a better fashion, but it's just crossing a line now where it's way too wild.
  • My one rule is generally "don't break legacy savegames," so you know this is serious business at this point. Time to get a lot done all at once to make this worth it.

• Old concept being removed: you can have "externaldata" which has any amount of sub-data on it, attached to various objects.
  • The precise nature of how that connection is forged is actually something that exactly zero people fully understood, and at any rate it was a very ineffective method for gathering this data. There were three levels of pooled wrappers in there, it was nasty.
  • The types of objects that could have externaldata were World, Planet, PlayerAccount (never needed that, though), Unit (this was used all the time, but then much less over time), and Faction (this is the really meaty one that tends to matter, but even this is problematic in the extreme for older factions in particular).
  • Please note that PlayerAccount is permanently losing an kind of external-data analogue, since that was never actually needed. The new network-enabled personal settings actually are much closer to what we had once thought we might do with these, and those are highly organized data that works really well.

• A LOT of code that was dedicated to maintaining old savegame compatibility has been removed.
  • This was thousands of lines of code that made the codebase that much harder to work with.
  • This alone is going to help performance for network transfers and savegame loads, given just how many versions there have been.

• Stripped out some of the excess of paranoid canaries that I had placed in game units a number of versions back. There are still some in there, but it's more efficient having fewer, and the reduced count will be sufficient for our purposes at this point.

• Previously, game units could only have a maximum of 15 systems, per their savegame format. This was very very sensible in a time before loadouts, but now this is being expanded to 63 instead.
  • This allows for us to have a great number of optional systems on a unit for loadouts. We still think 63 is an absurd number, but that's the sort of buffer zone we want to be in, really.

• Removed DoomData, which hasn't been used ever but was kept as an example bit of code for a long time.

• In our external code, right now we have some temporary classes as bases for things that will later be split off into other files and have their data style changed around.
  • Right now the goal is just getting things compiling, but having the external factions be braindead. The places where many things will change from the root up are marked with EXTERNALDATA_TODO for now.
  • It's going to be several days at minimum before we have a truly functional build again.
  • It's worth noting that a lot of giant gaps are there right now, like faction allegiances even being off once you load. This is because the config data doesn't have a pathway to the external data at the moment.
  • At the moment, however, you CAN load up the lobby and everything there is 100% functional. It doesn't actually kick off factions after the lobby is left, but that's not the point -- the point is nicely-siloed data that we understand the boundaries of.
  • At this point, after you load a quick start or click start from the lobby, it runs part of mapgen, but most of it is disabled. You then instantly win the game, and errors spam the world.
    • Here again, this is kind of the point... it's impossible for us to reliably have known prior to now what a client would or would not have on it in MP, and everything was very messy. These errors likely were the same ones that were going to show up on a client after getting a map from the host in the recent betas, but we would not have known until testing.
    • My work tomorrow is going to be centered around building in stable and clear data bridges between Core and External: this will have to be well-organized, easy to understand at glance, not prone to memory leakage, and equally as powerful as the tools that they are replacing.
    • The idea from that point on is to start getting the most-core data linked up, that stuff which is used by every faction, or by the human or AI factions in particular. And the idea will be to separate out that which is present for everyone, and that which only the host has, and that which the host automatically shares with the client as it is dirtied.
      • After that is done, and we have the basic factions running again, then it's a matter of going faction-by-faction and converting them to this new format. How much work that will be remains to be seen, but I anticipate being done with most of that this week.
      • To be clear, this should then have covered essentially all of the major things that multiplayer is currently lacking in data transmissions, and also have improved the performance of getting to said data in a single-player context as well.

• The old ShipStats information had data on how many ships of each sort you had killed and lost, by mark level. This was actually stored on every faction.
  • This is not very interesting data, and is very very hard to show. We're going to track and show much more interesting data, in a completely different way. So this is being removed.
  • This was the data that was "experimental" when you right-clicked the metal view. For the moment that's blank, but soon I'm going to have actual graphs.

• Adjusted all of the ExternalData serialization to no longer have the old data for deserialization logic. This was less to remove in the main game, but still several hundred lines of code.
  • The idea is making this cleaner and easier to read since we're in the process of breaking saves anyway, and it also makes it easier for me to move forward with splitting and cutting up the data.

• Removed some ultra old rally point code that hasn't been used since fleets were added.

• (Almost) all of our older code and folders and such which used to say "Mercenary" or "Merc" or similar have been updated to say Outguard instead.
  • This simply makes things more clear, particularly since we've been calling them both things in code for a few years now.
  • There are still a few references to mercenaries in a few files and filenames, but it's WAY lesser now, which is nice for avoiding confusion in the future in particular.

• Removed old features that had been retained only for really really old savegames:
  • ship_cap_multiplier_strikecraft, ship_cap_multiplier_turret, ship_cap_multiplier_frigate. These were all tied to the IGC in older versions.
    • There was a LOT of related code for this, which is all removed now, hooray.
  • The deprecated custom-flagships xml.
  • FCE xml and a bit of related code.
  • supercharges_strikecraft_ship_cap_of_rest_of_player_fleet and related code.

• Old logic for removing AI frigates has been removed. We no longer need to deal with those savegames.

• Removed the HostOnlyAggregatePlanning background thread, which I added a week or so ago but never wound up needing.

• Removed some FleetFormation logic and xml that had never been used and never been tested. It was an old prototype just clogging things up minorly.

Reconstructing ExternalData Links

• FactionCenterColor and FactionTrimColor have been removed as direct variables from the Faction class. So has FactionType Type and and SpecialFactionData SpecialFactionData.
  • In place of these, ConfigurationForFaction Config is now always permanently on the faction, and it has a direct link to this data.
  • Previously it was frustratingly common to have two copies of various pieces of configuration data, or even three, and we want to avoid that. Trying to keep that in sync serves no good purpose, versus having one place where each piece of data can be queried is very useful.
  • However, it's nice to have nice wrappers around all of these so that the same code will work as always have, so this is what we've done.
  • This may also fix a bug we've had for a while with the "subsequent faction colors not being set on factions of the same type" for some subfactions, but if not it at least makes that easier to fix in the future.

• The "MissingFaction" is set as the default entry for the factions table so that it will automatically be chosen in cases where a faction is, well, missing. Removes the need for some code.

• Added in some more wrapper methods for looking up custom field data for factions in a direct sense, rather than potentially getting stale data from one of the secondary places it is stored. We never want it to be the case that the UI is lying to you while having a secret separate value behind it, so this is part of that. We had accidental cases of that above-mentioned problem before.

• ArcenSimContextBase now has an abstract AmIAHostOnlySubContext on itself, and a ShouldThisMethodBeSkippedBecauseThisMachineIsClient() direct method.
  • This lays the groundwork for us removing some of the RNG divergences that were happening in multiplayer prior to now. This also serves as a guide for how we're going to be splitting our External dlls. Honestly it's making me review all the methods and make it really clear what can do what, which is pretty darn useful to have to do right in advance of that other surgery.
  • Added a new ArcenClientOrHostSimContextCore, and ArcenSimContext and ArcenSimContextBase are getting a TON of additions and changes. Way too many to note the details of in this format.
  • Added a new ArcenHostOnlySimContext that is for the host-only sections of the game.
  • QueueLogJournalEntryToSidebar and QueueChatMessageOrCommand no longer require contexts to be passed to them (they were not using it anyhow).
  • Lots of methods now either require a ArcenHostOnlySimContext, or a ArcenClientOrHostSimContextCore.
    • It is very rare to now require a root ArcenSimContext on a primary logic method, and this is part of how we're now controlling what methods run on the client or host (which we were already doing, but more haphazardly), AND controlling which random generator is used (this is the bit that is new, which will help stop that "units go over here and then zip back over on the MP client" bug).
    • That said, ArcenSimContext is absolutely still used on all the various worker methods that are called from a variety of contexts. Aka some code that might be used in any and all shared circumstances.
    • To make things even more clear, and prevent un-examined code from upgrading into this without examination, ArcenSimContext is now called ArcenSimContextAnyStatus to denote we really mean that.
  • ILongRangePlanningContext is now ILongRangePlanningHostContext.
  • ArcenSimPlanningContext is now ArcenSimPlanningCommonCoreContext_DoNotReferenceDirectly. It should never be used directly anymore, as it's ambiguous as to whether it's on the client or the host.
    • There are now intermediate ArcenHostOnlySimPlanningContext and ArcenClientOrHostSimPlanningContext classes that inherit from it. Mostly this is a formality, but it does add clarity and keeps consistency with the rest of the code.
  • A metric ton of code has been adjusted for all of this, in some cases removing things that were passing around some form of Context that had no business doing that. These are meant to stay in some reasonable sync, and sometimes even some UI choices could cause problems.
  • Because of the way inheritance works from classes in C# (single inheritance only, interfaces aside), there are a few methods that are duplicated across a couple of central classes.
  • THAT said, this leads to less duplicated code elsewhere in end-implementation classes, AND the inheritance tree of the "context" types makes it so that people literally can't call methods from improper spots.
    • If you're a modder and you find yourself trying to work around a method call that doesn't have a compatible context type, then either you just need to call yourCurrentContext.GetHostOnlyContext() and try that, or else give up and do something else because this would be a MP sync issue or other data invalidation issues would be caused. Or let me know, and I can always look into making something more permissive. But honestly other than people forgetting they need to call .GetHostOnlyContext(), I don't expect any troubles.
  • So what's up with calling .GetHostOnlyContext()? Isn't that kind of a waste of end-coder time? Actually... no! It's super important. The host-only context is used for parts of the code that only run on the host, and it has its own separate RNG that the shared clientAndHost context does not have. This means that all of what runs on the client and host has the same number of RNG calls, yielding a deterministic result. But peppered in there are some host-only calls, which previously were breaking sync but now just require a call to .GetHostOnlyContext() and sync is preserved while still keeping the same code structure you're used to.
    • This is round one of several rounds of fixes that could also be described as idiot-proofing (where all of us are equally idiots, and not meaning that in a pejorative way). The code is now demonstrably clear about when it cares what sort of caller is around, which is a big step forward for the next round of this, which will involve splitting things between dlls in order to separate out a new client-accessible layer of data from lots of host-only code that runs against it.
      • Put another way, this is getting around the problem of having to replace tooltip appenders with calls to the host, or doing all of the notification calculation on the host and then transmitting lots of complicated data. We're instead handling this in a much more tri-layered fashion, based in general on the MCV (Model-View-Controller) design pattern.

• DoOnSelfBuildingCompleteLogic has been moved from the scenario to the faction class, so that it's able to properly do its job. Before, this would have been a mighty big annoyance for any modders who made a new faction.
  • Same deal with DoOnInternalConstructionCompleteLogic, although nothing actually uses this one yet.

• The "ExternalCode" project, which is the main open-source AI meat of AI War 2, has been split into two parts.
  • One part remains in the existing dll, which is the most "permissive" part.
    • Anything in this dll is stuff that we're intending the client in multiplayer to have access to. (That is a statement of design intent, but I have a lot of work to do before that part is actually fulfilled.
    • Also, anything in this part can be freely examined by the UI, on the host or the client. I mean, thread safety aside, the UI is allowed to know about these things, and these things exist on every computer in multiplayer (again, talking about design intent, not current status).
  • The new part is called "AIWarExternalDeepProcessingCode," and is meant to be only things that the host has access to. Also, the UI cannot directly query these things.
    • Right now we have moved the faction processing logic, "on death effect" code, the main mapgen code, the "hacking is happening" code, and the "long term planning" code in general.
    • Please do bear in mind! Right now there's data mixed up in many of the files that were moved, along with the logic that was moved. Some of this data is meant to be host-only, but a lot of it is not. The next job will be splitting up all of these files and restructuring how the data is found, and that goes hand in hand with also re-teaching the Core code from AI War 2 how to reach these new split-up forms of data and code.
  • Please note that all mods will be required to follow this same double-dll format, which we will be enforcing by a quick scan of your mod code as it is first loaded. More on this will be available when this conversion process is complete.
    • The goal with this structure is to make things multiplayer-safe, and separate the data from the code a bit better, and separate the host-only code a bit better from "everybody runs it" code.

• Updated our build scripts so that when we build an external project from within visual studio, it will now properly set them into ReliableDLLStorage. Prior to this point, any compile-breaking changes needed to first be copied there by running the entire command-line build script.

• My external code loading framework is being put to new use in terms of helping ensure data-correctness.
  • For classes that will go into deep processing or the client-safe data processing, they will all inherit from either IExternalDataAndSimCode or IExternalDeepProccessingCode.
  • The game now checks to make sure two of these same things are never in the same dll, and additionally, that no dlls that have deep processing code are referenced by any classes with DataAndSimCode in them.
  • This is really important for preventing errors that would affect multiplayer clients, among a few other maluses, and so by doing this very quick check on startup we can make sure that all of our own code, as well as any mods, all conform to the standard that prevents the most common MP-only issues from happening.
  • Put another way: if you've just coded a mod, and it works for single player, you want to be pretty darn sure that it "just works" in multiplayer without having to go test it there. It's the same with any changes we make to the core game, or in DLC. All of this code restructuring, along with these checks, are how I ensure that is the case at a very low level. If MP has a problem that is unique to it, I want that to be something that's my central problem to solve, not something that faction designers or modders are chasing on their own stuff.

Restructuring The Upwards Link From Core To External, Part 1

• Previously, we had a thing called "ExternalDataPattern," and some of the DLC content had to be included in the base game or else beacon factions would mess up the ability to load quickstarts. And last-settings would get messed up if you disabled a DLC and went back into the lobby, too.
  • Thankfully, now we don't have this problem at all because of the new "micro save" format that is now used for those two purposes as of this build. Therefore, the new equivalents of this old sort of data are now able to be stored in each DLC properly (this has the same implications for mods, so this is a good thing for mods being usable for sure).

• Added the following two sets of data/code pairs:
  • ExternalDataAndSimCode, which is for clients to know about and the ui to be able to see and use.
  • ExternalDeepProccessingCode, which is only for hosts and single-player, and contains all of the deeper logic and working variables and whatever else. We don't want this to accidentally get sent to a client at any time.
  • It is worth noting that the above things can have 1:1 pairs (faction data in the sim code that the ui can also see, and then deep processing for said faction), but in many other cases we don't need a faction to be involved at all, or have several sub-parts to a faction (unit data, etc), or whatever other designs we want.
  • As part of the two above sets of data/code pairs, added the following three categories:
    • World. This is something attached to the world, and it can run whatever logic you want to, kind of like a faction would have a chance to but without the faction.
      • Internally, some of the logic that we have very confusingly on "Human" right now -- there can be many human factions, and we generally want to run this stuff once for all of them -- will move here. This is also for the things that are human-faction-type agnostic.
      • Externally in mods, we expect to see the FakeExecutorFaction from AMU move into this. We should never need to have a fake faction again, as there is now a clean and fully supported method for doing that same sort of thing here.
    • Faction. This is where the meat of the game logic lives for anything that isn't just the autonomic style of unit AI.
      • Internally and externally there will be some swapping back and forth of what is loaded where, and the SpecialFactionLogic class in general will be subsumed into this.
    • Squad. This is data that goes on an actual individual ship/structure for whatever reason. Some factions are designed where this is needed, others are not.
      • Having deep processing available for individual ships is now possible, although that would be a little bit on the strange side. We have a way of handling it with efficiency, but it's an oddity for sure if you structure it that way.
  • ExternalDataPattern used to be loaded very early on in the load process, and these new ones are loaded only slightly later. Other xml will be referring to these.
  • So far, each of these is just blank slates, but they'll all have unique methods on them, and be the vessel through which most communication happens between Core and External and ExternalDeepProcessing.
    • It is worth noting that these all implement time-based pools that are incredibly direct and also centrally handled, so we can avoid the memory weak worry that was everpresent with MP clients in the past.
  • Oh, another thing to note: all of these are individual object data holders that can also contain code. None of them are singletons.
    • So for instance, if a faction defines a FactionDataA class that goes with itself, then if there are 3 instances of that faction TYPE, there will also be three instances of this faction data. This is very useful, because you can therefore store faction-specific working data in the deep processing in particular, or use static variables if you want to share across factions. This is very similar to the ethos behind ISpecialFactionImplementation, which will be leaving us before too long.

• Experimented with a new method of serializing data for objects. The data is really really sparse, though, because of "all the things that can be on a ship," for example, usually only one or two actually are. It's much the same for faction and even world.
  • I spent a goodly while coming up with wrapper classes to manage arrays based on the length of how many external linkages there are of any given type (which is based on how many DLC and/or mods you have installed), and was going to have some sort of very high limit (so that if you run with 100 mods on, someday when there actually are 100 mods), you don't have a problem.
    • However, that has many many drawbacks, because we wind up iterating a ton of arrays that are almost universally empty, or have a single item or two in there. And there is a finite limit on the number of mods you could have, even if it is very high.
    • The other big drawback is just how long it will take to iterate all of that, and for squads we have issues of how timely we can get those data updates from the host to a client after there's some change. Also, we have some order of operations questions with things like fast blast data potentially arriving before the external data, or vice-versa. Either way, it's not a good model.
  • The solution will to take this "vertical" approach and make it horizontal, and instead serialize on-the-fly in a more centralized fashion. This keeps our amount of iteration very very low. The downside is that I then have to make it very fast to link back and forth between the data and the thing it is attached to, but that's not the worst problem in the world. I was already potentially going to have to find a row index and then index into an array and then cast to the proper type. Instead we're likely looking at a concurrent dictionary lookup, which is at least a single operation rather than multiple (admittedly cheaper) ones.
  • Well. At any rate, my code for the old vertical method is not complete and does not compile (I was in the middle of it when I had this realization). I'm checking it in in case it is ever needed in the future, though. Next step will be to convert it to the horizontal format.

• Some rather mind-bending code for horizontally serializing and deserializing all of the external data for any world data, faction data, and squad data, all at the central faction level, has now been put in place.
  • This differs from a lot of our serialization code in that it's very... reactive... to what it finds. Essentially what happens is it iterates over every possible piece of data that can be sent (hey, macrophage ships can sometimes have data attached -- are there any in this game? If so, send those that are there). It's clearing out stale data as it goes, and as-before it gives the details of the serialization itself to the actual External code classes.
  • One of the chief differences this time around is that we don't have to serialize any table or row names -- we are using central indices that are synced from the client to the host -- and we're also using smaller ranges of integers for the PKIDs for factions and squads based on the differential between the largest and smallest ID in the particular list at any given time.
  • This will require testing once we can run the game again, to make sure that I didn't make any typos, but it uses a lot of generics and type inheritance in order to minimize code, so whatever problems are there should be of the "fix once, don't think about it again" variety.
  • At the moment, what is in the codebase is kind of impressive, but also kind of baffling. Here we have six identical tables and classes, all serialized using some very complicated code that is very centralized... but there's nothing different about any of these classes or tables. They're all super identical.
  • Where the magic really is going to come from is the fact that we can have all manner of different methods on each of those distinct six classes, and so we can set, retrieve, and manipulate data without having to do any type casting at all.
  • This a much more elegant solution than the old solution we had, but it strays slightly close to "clever code" (the sort of thing that is showing off for no reason, and thus is super hard to maintain and nobody else can read it).
    • That said, the reasons for this solutions design are pretty clear: keeping the guard rails on, not mis-identifying data (often happened before during load issues), minimal steps to get from Core to External, the ability to have lots and lots of very direct methods that we can can directly call, and pooling that is NOT in the hands of the External programmers, but instead is controlled by Core. There's also a goal for having substantially less code, and substantially clearer code, on the External side. That will start becoming evident tomorrow as I start wiring this up to them. On the Core side it takes some extra complexity in order to create more simplicity on the External side.
  • That's the idea, anyhow. I have long considered myself a code architect in terms of programming sub-specialties, and I've done a few dozen refactors of major programs, ranging between languages and business software and games. This one has so far been, to my surprise, the most complicated one I've had to do. There's a lot of pleasure in that, but at the same time: wow complex.
  • The main good news from today is that there is really a small amount of code (all things considered) where I could have made mistakes. So once I start this up, after connecting to the External code where I CAN start this up, any errors should make themselves apparent immediately and then be gone forever.
    • The old way of doing things was something I was never able to wrap my brain completely around because it had three layers of indirection that also required bidirectional communication and reading of the code. In other words, to understand that code, you had to read the entire path of it from Universal, through Core, all the way to External, and then follow a further chain back down the other direction. That was... code that worked, but was very very hard to be sure was not secretly broken. It had some arrays of objects in the middle, hitting the GC in a way I didn't like, and it was particularly hard to tell what it was doing o =n MP clients.
    • But the new code? It flows one direction, you can follow it clearly, there's no GC hits in the middle, and despite the code being a little bit odd syntactically (generics really can look odd), it's fairly straightforward to follow.
  • Tomorrow will be the real test, where I start wiring up the various pieces of External code to these new pieces, get them compiling again, and then eventually transform it all into the simplest possible version of itself that will be as error-free as possible in MP and SP as we want. It won't all happen in one go, likely, and I expect several more major refactoring passes next week to clear up various perennial bugbears like linked factions, the strange combined fleets/cities of the fallen spire, the odd (and not architecturally wise) way we have "faction common data" and similar structural oddities that make some of the externaldata have a larger learning curve than we'd prefer. Hopefully we have a working SP game this weekend, and hopefully we have a refactor-complete game by next weekend. We'll see how quickly things move tomorrow.

Restructuring The Upwards Link From Core To External, Part 2

• Okay, all that "clever" code yesterday? I said in my release notes that it was pushing the edge of being too clever for its own good (when programmers of a certain experience start talking about how "clever" code is, it's not a compliment -- if a professor tells you your code is clever, that's a compliment, but if a colleague does, they are worried about ever having to maintain it. Smart and cool and elegant are always compliments.).
  • The code was indeed the simplest possible implementation with the least amount of code for doing the sort of many-to-many relationships that I was assuming some of our data might have. But thankfully, no actual data is structured that way.
  • The old code that we used to have was allowing for many-to-many relationships between many kinds of objects and data, but thankfully everyone was sensible in both internal and mod code, and instead what we wound up with was a lot of one-to-one code. THIS ship has THAT data and no other, etc.
  • So! After talking to the folks who have been working on the bigger mods and the external factions and verifying that my sudden suspicion was correct, that means it was time to strip out pretty much most of the work I did yesterday (sigh, but it was a good intellectual exercise I suppose) and implement something new.
  • This new approach treats all of the World data as being singletons (aka only one copy of that external object, ever, period), and it even queries them "hey, you want in on this?" to which they can contextually say yes or no.
    • This is super duper useful, because you could actually have some sort of helper code turn itself on for a while to manage some process in the game, and then relieve itself of duty and stop using any bandwidth or CPU cycles. Or it could start out as off, but turn itself on once some situation happens, etc. Handy!
    • In general, one of the larger problems with external data assignments like this is "when do they get attached, and how do we know to do it?" In the past, a lot of that was "when ships are created in externaldata, we'll assign it contextually then, and hope we miss no cases. If found we did, then backfill." We may keep to that, but will also try to formalize it a bit. But for the world, simply asking the objects "you want in, or you good sitting out for the moment?" is feasible as well as most performant in this instance.
  • Factions and squads are still pooled, and so a lot of what was done yesterday (the easy parts) still apply for these.

• Map generation code is now flagged as deep processing, so it will be disallowed from being in any dll with the base data and sim code.

• ISpecialFactionImplementation, which has for so many years been the absolute centerpiece of faction and AI code, has been removed from the game.
  • It's functions are now being split between base-data-and-sim faction code, and deep-processing faction code objects, which has been one of the chief aims from the start of this rework.
  • There is no longer an "Implementation" object on the faction at all. Instead, there is now a "BaseInfo" and a "DeepInfo."
    • These objects serialize data to disk (like externaldata used to), as well as contain whatever logic we want them to. That's not quite an MVC philosophy, but it's following a similar and useful OOP design.
      • It's worth noting that the deep info ONLY ever serializes to disk, and never across the network to MP clients. It's also worth noting that the UI should not be trying to get any data out of the deep data unless it's purely for literal debugging purposes.

• For the first time, squads themselves now also have a BaseInfo and a DeepInfo, which mirrors what you're used to with the factions.
  • This doesn't replace anything on the "big logic processor side," but will probably fold in things like "description appenders" later. However, this replaces the notion of ExternalData that used to exist.
  • It's worth noting that this external data is no longer a collection of things, unlike before. There's only a single item attached to each squad, or zero items attached to it (if it's not some special faction other than humans or the AI, it's probably zero items).
  • Before we had a many-to-many relationship, but that was hyper inefficient and something you are better off handling with data at the faction or even world. There are no known cases of any internal or mod code actually needing that, though.

• Added ParallelDoForAllActiveEntries() onto TimeBasedPool, which gives us a comparably-quick way of doing some action VERY completely against all live objects in that pool.
  • Quick is a relative term in this one, but it is multithreaded, at least. This shouldn't be used in most cases for during-game things, but rather is for times like "when we reload all the xml" or things like that.
  • We also added a ParallelDoForAllEntriesEverEvenInactive, since that actually seemed kind of warranted for the xml case.

• GameProgramIsAboutToReloadExternalXmlAndSimilar() is now a thing that exists on all the external objects, rather than being limited to the faction data.
  • This is mainly as a courtesy to mods that may have different structures from our code. We don't actually use it, but it's useful as a thing that can be called for those that do.

• ExternalDataExports is now functional again, and lets us dump out all of the new-style external data.

• Everything that was called "ExternalXDataAndSimCode" is now called "ExternalXBaseInfo"
  • This is a lot easier to type, and is way easier to read at a glance.

• Everything that was called "ExternalXDeepProccessingCode" is now called "ExternalXDeepInfo"
  • Again easier to type and read.

• There was also kind of an awkward convention between the tables and rows describing the external data, and the external data itself.
  • That has now been clarified into the tables and rows being called "InfoSources", while the actual things themselves are "Info." Logical and straightforward at a glance.

• Faction xml (aka from the SpecialFaction folder) no longer wants or accepts a dll_name and type_name.
  • Instead, it now requires a base_info_source (referring to an entry from ExternalFactionBaseInfoSourceTable).
  • And then also a deep_info_source (that refers to ExternalFactionDeepInfoSourceTable).

• Squad xml now has the OPTION of specifying base_info_source or deep_info_source as a way of automatically attaching these types of external data to all entities of the type in question.
  • This is actually a really clean way to handle things, and then the data can be populated later by the external code as desired. Nobody HAS to use this, but it can be a nice thing.
  • Do note that this IS inherited via copy_from, so if you are copying a ship that happens to have base_info_source set, then on any that are being copied for use without that, you need to set base_info_source="None" on any direct children to blank that back out.

• Actually, it looks like the DescriptionAppenders and the AlternativeMoveOrderHandlers, and the EntitySpecialPerSecondLogic all can just stay as they are.
  • They are being flagged as IExternalBaseInfo so that they have to go into that dll rather than the deep dll (otherwise these would all break on MP clients), but that's about it.
  • These are a great example of something that can just stay flexible (no need for a rigid structure here), but now have a proper place to live that also guarantees what data access they have access to. Previously, these things would maybe work or maybe not work on MP clients, depending on the status of things. But generally trending towards not working. Now they can be verified to function just by testing them in single player, since they're in the dll where clients also get access to all the info.

• At this point, the Core dll has had all of the changes made to it that are required in order to handle the upwards-facing communications that the Core dll has with External code.
  • This means that the work of actually translating over the External dlls can now begin, and some things can be redone in passes, but not all things have to be.

Restructuring External Base Info, Part 1

• Started the process of transitioning some data from the Deep dll into the Base dll, with the idea being to separate out the running of things from the actual viewing of data related to those things.
  • Also, of course, wanting to isolate the sort of background and working data off in Deep away from the UI and any temptation to try to show that data, which would be explosive on the MP client and just a bad idea on single player or the host.
  • Moving over HunterFleetBudgetType and its related stuff is straightforward enough, and matches with the sort of thing I have in mind. That's just a copy-paste job, and hey I'll make new files for these for easier location of them in the future.
  • Then I turned my attention to HunterFleetType, and immediately hit a hard stop: it has something on it called IHunterFleetTypeImplementation Implementation, which defines both data (Base type) and logic (Deep type).
    • This is a bit of a conundrum, but not a crisis. Ultimately this mirrors the same sort of setup that I have going on in Core, and I'll need to split these things out in some fashion.
    • Right now my brain is feeling quite tired, so maybe I'll wait a bit on that problem, but essentially this is where we get to the "dream within a dream" level of inception, and this is actually more important than it might look on the surface.
      • First of all, modders and general faction designers need to be able to create new data tables in a subordinate fashion like this, and there needs to be a really central way of handling it. I can't idiot-proof it like I can with things coming out of Core, specifically because as the "dreamer" of the first dream, the modder/faction-coder has full control over how they structure their inner dream within a dream. That's both neccessary, and something I don't want to take away.
      • This actually does get at the generalized need to formalize that as much as I can, however. I can't make it idiot-proof, but I can give the proper tools to standardize dreams within dreams (going with that metaphor), AND also to make sure that they serialize in a way that does not cause a memory leak on MP clients. It's this last part that is the most challenging, and if I could make that one part idiot-proof, that would make me very happy. That's the one bit of power (accidental memory leaks) I'd like to take away from the dreamers in the top level dream.
      • Further compounding this is that I might actually need to support more depth, aka dreams within dreams within dreams -- three levels, or even four, or some arbitrary recursive number. I sure hope no-one has structured their data like that, but I guess we'll be finding out. First, though, I'm going to take a break and focus on some other compiler issues that need to get sorted out anyway.
  • Overall when it comes to looking at this, I have factions and squads which presently have a very similar -- but slightly different way of serializing. The faction MUST always have both Base and Deep info, and it serializes with that in mind. The squad may or may not have Base or Deep in any combination or none at all, and it serializes with THAT in mind.
    • The world, meanwhile, can have a great many base and deep infos attached to itself, but they are all singletons and are also all loosely, voluntarily attached.
    • The question will mainly be if I can formalize these relationships in a generic and encapsulated format that can be attached to arbitrary objects. The huntertype seems to be a singleton processor-type class, and the other classes that I've run into are similar.

• Added a new ArcenLongTermIntermittentPlanningContextBase in the BaseInfo dll.
  • This is going to be used in some of our "dream within a dream" data management cases.

• Fireteams have been partially moved into the BaseInfo dll, but parts of their code is clearly DeepInfo code.
  • That has been commented-out for now, and will be grafted back on in the DeepInfo dll once we can compile BaseInfo again.
  • Additionally, FireteamUtility has been split into one part that is FireteamBaseUtility that can run anywhere, and then the other part that stays behind in DeepInfo.

• In Universal, split up the interfaces that contribute to the external BaseInfo and DeepInfo so that we can have ones that are singletons and ones that are multiples properly denoted.

• All thirty-seven "Decription appenders" have been moved from their files in the DeepInfo dll into their own individual files inside the BaseInfo dll.
  • There were two old description appenders (hacker and science) that were not used and thus have been removed.

• All of the IAlternativeMoveOrderHandlers have been moved from their files in the DeepInfo dll into their own individual files inside the BaseInfo dll. There was only one.
  • In recent versions of multiplayer, clients were not going to be able to order spire relics around. Nice that we're doing this overhaul, as that was not on my radar and would have been a huge pain...

• FrenzyFleets from the nanocaust have been removed, as fireteams have far far replaced those.

• FactionUtilityMethods has been partially moved into the BaseInfo dll, and it is now a singleton rather than a static class.
  • Thanks to it being a singleton, this means that we can also add extension methods in the DeepInfo dll and not have to worry about having two classes with similar names that we wonder which item is on which.

• The "SpecialForces" faction has been fully renamed to Warden in the code. It was old legacy naming from 2018 or prior, and getting rid of that makes things a lot more readable.

• EntitySimLogic has been split into EntitySimLogicBaseInfo and EntitySimLogicDeepInfo.

• Moved over the bulk of the data that should be in BaseInfo rather than in DeepInfo.
  • There's still more that needs to come over, but it's part of larger classes that will need to be split.
  • At the moment, I've still got a lot of things commented out or using fake connections to mimic the old style. It's not functional, but it's hitting the point where I can compile again and thus move towards getting it working.

• A variety of new methods have been added to FactionUtilityMethods to make checking for faction types easier and more reliable.

• Current status: lots of things are moved around to be in a better spot, but a lot more work needs to be done.
  • There's still a lot of temporary code linking to the old style of doing things so that we can compile the main ExternalData dll (the base info one). That actually does compile now! But it's not linked up yet, and won't function yet.
  • The new deep info dll doesn't compile yet, and will require a lot more splitting of data into multiple classes before it does.
  • There are still some things that need to move one direction or the other, and then other things that need to be moved around or restructured so that data is grouped properly and can be accessed properly.
  • This is... quite a bit more tangled-up than I had expected, to be honest. But this is a really super good thing to be doing, and if this past week is any guide, it should be less than a week of work left to get this functional again.

Reconnecting External BaseInfo to Core, Part 1

• There was a "does nothing, but helps things compile" set of code that we had in the ExternalData (BaseInfo) dll since making the intial rip and tear changes to ExternalData as a whole.
  • This code has now been removed, which creates hundreds of errors but allows us to start replacing those hooks with new ones. Doing a massive code shift like this is an iterative process, and sometimes you just need "somewhere to plug things in for a bit so you can work on another part," if you think of it like working on a physical piece of machinery.

• The base and deep infos on factions are automatically created just like those ISpecialFactionImplemenations used to be, so that's not something that you ever have to worry about creating as an end programmer/modder in this game.
  • Similarly, the ones for the world are created as singletons and are never really attached to the world. The world just asks your code "hey, you can see what's up, do you want to run?" And if you say yes, then the world will run your code and serialize it, etc.
  • On the other hand, squads are a bit different. Historically we have added things to them as we went, as we wanted to, but that doesn't mean we can't centralize some of it. There are two modes of operation for the squad, therefore:
    • If you specify the xml for it, then it will automatically add them when the squad is created. This is not a bad idea! But this does assume that every unit of that type EVER will have this external data, and there are potentially cases where that is not true.
    • For those sorts of conditional cases, you can instead manually call CreateExternalBaseInfo() or CreateExternalDeepInfo() on a squad, and pass in the type of data you want it to initialize for you.
      • If this is already initialized to something else, then it will throw an error, since that is in fact an error and you need to take a look at what's going on.
      • If, however, this is already initialized to the same type you just asked for, then it just returns the existing item it has. Basically if there are cases where we might accidentally call the same method here more than once with the same code on it, then this doesn't hurt anything with the extra calls.

• Okay, here's a description of the pattern for changing things over from the old format to the new. This is mainly for modders to know, but also anyone working on the codebase it's useful to see.
  • We previously stored Wormhole Invasion Data (which goes on a squad) like this:
    • WormholeInvasionDataExternal was the outer wrapper that inherited from ArcenExternalDataPatternImplementationBase_Squad.
      • Structurally, this was clunky code, and very rarely had actual data of its own in there (in terms of general ExternalData for whatever function). There's no real analogue for this anymore in the new system.
      • IF you were doing the extremely old-school thing (for our codebase) of storing multiple sub-objects in here, you will have to choose one to become your new primary BaseInfo, and all the others will be sub-objects of that. Easy-peasy.
    • ExtensionMethodsFor_WormholeInvasionDataExternal was an optional (but always-used) convention that made it easier to get the wormhole invasion data sub-ojects from externaldata.
      • Structurally this was also a bit of a pain, even though it wasn't much code. A small typo, and dealing with the annoying patternindex (which no longer exists) could really foul things up. We can safely remove this, also.
      • If you really want some extension methods for THAT easy of access to BaseInfo or DeepInfo in the new system... then honestly instead maybe think about just how frequently you're calling such methods, and instead call it fewer times and pass the result around. Player CPUs will thank you.
    • Finally we come to the actual meat, which in this case was WormholeInvasionData, which inherited from ArcenExternalSubManagedData (which no longer exists). This is what will become BaseInfo.
      • The reason that this is BaseInfo instead of DeepInfo is that this is information that both the client and the host might like to see, in terms of it affecting the UI for both of them. In the event that there are some parts that are truly host-only, and never get shown on the ui except for maybe debugging purposes, then consider moving it to DeepInfo in a separate class IF it's substantial enough data to actually care about (why tell the client about things it never cares about?).
      • This class is now going to inherit from ExternalSquadBaseInfo, since this is what it is (base info, squad-attached). Let's talk about what THAT means separately, in a moment.
  • In terms of translating things over, for this type of data this is all you need to know from a bird's view.

• So now let's talk about details of what it means to handle this ExternalSquadBaseInfo, starting with WormholeInvasionData -- which let's rename to WormholeInvasionBaseInfo.
  • Trust me, it's worth it to rename this just to save yourself confusion later. Remember to do a global search and replace from old name to new name in both the BaseInfo and DeepInfo dlls (the DeepInfo almost always acts on BaseInfo).
  • Now that we have this class inheriting from ExternalSquadBaseInfo instead of ArcenExternalSubManagedData, there are some immediate things to change. (This will be identical for DeepInfo, and for faction and world data.)
    • DeserializeIntoSelf and SerializeTo both no longer are passed a SerializationCommandType SerializationType, so you can just remove those and those methods will then be happy. You can probably just run a global replace in your BaseInfo dll for ", SerializationCommandType SerializationType" and replace it with nothing and it will save you time (we had 226 instances that were replaced in our own innternal BaseInfo code).
    • We need to implement the base method Cleanup, which gets called whenever an object is being pulled back out of the pool.
      • This may or may not be important to actually fill, depending on your class. If you were previously setting some values in the constructor, and your code always expects that to be the starting state, then you should probably move that initialization code to Cleanup and have your constructor also call Cleanup(). You'll note that this is what we've done with WormholeInvasionBaseInfo.
  • At this point we have translated things over such that they CAN connect up, but there isn't an actual linkage being created yet. It's time to address that.

• Since we did a global search and replace in both projects, we can search for WormholeInvasionBaseInfo in both the BaseInfo and DeepInfo dlls.
  • In our case, there are no other references to this in BaseInfo -- that's just where the data is stored. (This is temporary and wrong, actually! We're going to move the notifiers over to BaseInfo to save processing and data, so later that will actually be there).
  • At any rate, there are two ways we might want to use this data now. We might want to get a reference to it, to change it or just write about it.
    • Previously that would have been done with the GetWormholeInvasionBaseInfo() extension method, or something similar.
    • Now this is done by calling GetExternalBaseInfoAs<WormholeInvasionBaseInfo>() on the squad, which will give you back a WormholeInvasionBaseInfo that already exists or give you an error. For DeepInfo, you can call GetExternalDeepInfoAs<YourTypeHere>(), and it works the same.
    • It is worth noting that if the base or deep info here is null, then these will just return null, so your end code should be prepared to handle that if you think that might be a possibility. In the past, most of the time our code just returned a new instance of the data type, which... well, that could have hid a LOT of code errors. We have no way of knowing. If someone calls these things on the wrong kinds of unit/squad/structure, we actually do want this to error, because that is almost certainly a logic error that then would lead to strange behavior. It's possible we will find some "new" bugs that are actually existing ones that had been hiding, in the coming weeks. This in turn would likely fix a lot of other rare bugs that were downstream from this problem, if that's actually something that happens.
  • In the old code, after changing some data, you'd MAYBE have to call SetWormholeInvasionBaseInfo(). There is no longer any such analogue.
    • Technically in the old code, this was not required for this particular class. But if you were using certain kinds of data, in certain situations, it was VERY much required. So end-coders wound up just calling it uselessly after making any changes in general, and thankfully now that's not a thing anymore.
  • Lastly (though technically firstly), there's the creation of this data. This would, presumably, happen when the squad itself is actually created, and in practice that is what the code does.
    • This can be handled two ways.
      • Option 1: You can set it up so that all of the units with the "ExogalacticWormhole" tag have this data linked via the new xml tabs, since that's apparently the criteria we're using.
        • Pros: the xml will be nice and clear about what is linked, and there's no code path where you might accidentally forget to add it.
        • Cons: properly transferring the existing code to hit all the proper entities is slightly tedious, and also introduces the possiblity, of errors.
      • Option 2: You can take what was a call to GetWormholeInvasionBaseInfo( ExternalDataRetrieval.CreateIfNotFound ) and turn that into a call to CreateExternalBaseInfo<WormholeInvasionBaseInfo>( "WormholeInvasionBaseInfo" ).
        • Pros: this keeps the same structure we had before, so there's fewer errors of short term errors in conversion. Requires no xml work. Can be used to add BaseInfo data onto units in a conditional manner -- for instance, usually just when they belong to a specific faction.
        • Cons: in general, if you forget to do this on all possible code paths that are expected to lead to the result you want, then the game is going to have new errors popping up where it used to not have that. In the vast majority of internal cases, there's only one central method that creates these things in the first place, so that's usually less of a risk, but sometimes a consideration.
    • Before anyone asks: yes, you CAN handle it "both ways." It won't complain if it already has the type you're asking to create. It will just return the type that is already there.
    • Also worth a thought: there isn't any sort of initialization callback on these right now, so if there's linkages you need to make on setup, you're going to need to at the very least call the GetExternalBaseInfoAs<WormholeInvasionBaseInfo>() method and then assign some stuff. So this again creates as situation where you must be sure to always catch every possible code path that could generate this sort of entity, though we may work on solving that in a couple of ways in the very near future. Probably before this build is out.
  • Okay, now that the process is outlined, it's time for me to repeat this another few hundred times on the main game code.

• Previously, there was some information stored on every faction. To get at this data, you would have to call GetCommonExternal() on the faction object.
  • This data has moved onto the BaseInfo object for factions in general, in Core.
  • There is a new IPlanetPathfinder in Core, which lets us handle a lot of this centrally now.

• Made some updates to methods in FactionUtilityMethods to use a more efficient way of finding the highest AI difficulty in various situations.
  • Also got rid of some duplicative methods of this from MapGen_Base.

• SpecialFaction_Human had a lot of random stuff on it that is moving elsewhere:
  • The audio cue suppression stuff, which was complex but is only really used for the king unit being damaged, has been moved to nonsim in the base game.
    • This is local data for a client or a host, and we don't use this in such a way that it warrants nearly the intensity it is generating.
    • Similarly, the data from HostOnlyAudioTracking has moved onto the planet itself and is no longer host-only (that was an error on my part a week or two ago).
    • The function is identical to before, but it's now far less code.
  • GetMaxPossibleStrengthOfAllHumanFlagships() has moved to a new StrengthHelper class in Core.
  • GetPathingModeForLocalPlayer() has moved to a new PathingHelper in External (BaseInfo).
    • Similarly, a bunch of stuff like InnerFindPath_Raw(), InnerFindPath_RawSinglePathfinder(), FindPathFreshOrFromCache(), and FindPathFreshOrFromCache_ShortTermPlanningOnly() moved to that PathingHelper out of BaseFaction.

• Also previously, there used to be a ExternalData_MinorFactionCommon.Primitives, which you could get from most factions via calling Ex_MinorFactionCommon_GetPrimitives().
  • This is now gone as well, with SOME of its data wrapped into the BaseInfo root just like what we did with the CommonExternal stuff.
  • The rest of this data is currently sitting (as of this writing) in a Temp.Historical class, and will have to be properly distributed to various classes that actually need this information. Right now there's a lot of junk in there that is not used at all, and a bunch of stuff that is faction-specific in a way that doesn't need to be central (the HRF intensity is not needed on every faction in the game, etc).
  • One thing that is moving is the Intensity being directly on any faction. This should instead be per-faction where needed (AIs don't use this, they use AIDifficulty, same with Wardens and Hunter and PG, for instance, and many others use nothing at all).
    • Instead, when trying to quickly get the intensity for a faction in general, which before was Ex_MinorFactionCommon_GetPrimitives( ExternalDataRetrieval.CreateIfNotFound ).Intensity, the code that should be called instead is BaseInfo.GetDifficultyOrdinal_OrNegativeOneIfNotRelevant(). This was by far the most common usage of primitives.
  • Actually, for the moment nothing has moved data-wise from here! The Allegiance field would be the one to move, but for now I've implemented a string get-only property called Allegiance on BaseInfo, which is just a wrapper around calling directly into the config file on the attached faction. This is getting back to (from last week) only having one copy of certain data.
  • HasPlayerGainedIntel has been removed, as there is a HasBeenSeenByPlayer on the core faction object now. This was only used by the devourer golem for one notification, anyhow. That now uses DoOnFirstSightingOfFactionByPlayer(), which was added long after the devourer golem faction was coded.
  • TotalMetalMetabolized has also been moved to the roof faction object in Core.

• Got rid of an old GetFactionIntensity() method from Scenario_BaseScenario. The proper code is now BaseInfo.GetDifficultyOrdinal_OrNegativeOneIfNotRelevant().
  • There was a pretty large bug in there that returned -1 for the intensity of any faction that actually used the central intensity settings (aka not AI or its subfactions).
  • This was causing achievements to not fire for beating other factions in general, or at least it should have been doing that.

• The "SpawnAnimationData" and "DespawnAnimationData" data classes have been entirely removed.
  • This was vaguely like the despawn info, which the game DOES use, but it was not actually ever data that was used!
  • It's vaguely possible that a few parts of this were actually functional, but it's honestly very hard to tell. If that turns out to be the case, then we'll have to implement that as more centralized data. The way it was handled before was super confusing.

• The "InstigatorData" faction info is our first example of a faction that needs to be converted to have BaseInfo and DeepInfo.
  • The SpecialFaction_Instigators class would be the basis of the DeepInfo, while the InstigatorData that inherits from ArcenExternalSubManagedData is the basis of our BaseInfo, but really we're going to be pulling both code and data from both of these for our new setup.
  • First off, let's rename "InstigatorData" to instead be InstigatorFactionBaseInfo.
  • Next we need to implement the Cleanup() method in the exact same fashion that we had for the squad baseinfo.
  • Also, because there is some wrapper data also being saved, you must rename DeserializeIntoSelf to DeserializeFactionIntoSelf, and SerializeTo into SerializeFactionTo.
  • IF you have any usage of ModdableGameCommandExecution() on your old SpecialFaction class, then you should port that over.
  • Again, IF you have any usage of DoOnSpecialEvent_OnMainThread_ClientOrHost() on your old SpecialFaction class, then port that over here.
  • You must implement GetDifficultyOrdinal_OrNegativeOneIfNotRelevant() now, even if that's just to return a -1. Ideally you would return a number between 1 and 10, unless your faction has no concept of a difficulty level (or intensity) associated with itself.
    • This method no longer has a faction object passed in. Use the AttachedFaction property to know what faction you're talking about.
    • This method was widely incorrect in the prior version of the game, so this should fix bugs like the "OnLoad: AI faction: X had a difficulty of -1, which is not helpful for setting overlord difficulty level."
    • Instigators are a great example of one with nothing like that.
  • IF you need a nonstandard implementation of FindPathFreshOrFromCache(), then be sure to migrate that over from your old SpecialFaction class. We use this on the hunter, warden, and similar, but not much else.
  • IF you used DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost in your SpecialFaction class, then move that over. Otherwise ignore.
    • Same with DoOnPerSecondWhilePausedNonSimUpdates_OnMainThread if you use it.
    • Also DoOnLocalStartNonSimUpdates_OnMainThread if you use it.
    • And WriteTextToSecondLineOfLeftSidebarInLobby if you need something beyond the defaults.
    • And GetShouldAttackNormallyExcludedTarget() if you use it.
    • And finally, SetStartingFactionRelationships() if you deviate from the defaults, which is the most likely out of this list.
  • Whew, that's everything. A lot of stuff had to be adjusted in order to get this first one working, but now future ones are pretty straightforward.

• Removed GroupTargetSorting (it's some random ExternalData), since that is not in use. It was some very legacy vestigial code from pre-pivot.

• Status time: at this point "all" that's left for the BaseInfo dll is to translate the remaining faction and squad data over. If there's any world data we run into during that process, that will also need to be ported over.
  • After that, there's going to be a needed conversion for all of the SpecialFaction_Whatever data from DeepInfo, but that is going to be a way more direct item to tackle.
  • Also todo is to move notifications back into BaseInfo where the client can generate them all, and turn back on a lot of things that were disabled for being-able-to-compile purposes when the data was not all in the correct place.
    • Some of that will then also probably involve moving some data from DeepInfo classes to their BaseInfo counterparts.
  • This is probably another 2-3 days of work. After that I can run this for the first time and see how badly things are broken. Hopefully that is less than one day's further work of fixing, and then we're back to actually having beta releases. That would put us at Friday, if the general estimate here holds.
  • I did previously have a worry about data-within-data, but most of that is (knock on wood) not an issue. There are still some interfaces in a few spots where a Base class wants to call a Deep class, thanks to SurrogateTables (this is things like the Warden Difficulty, etc). I don't have a firm solution for that yet, but at worst I can use the surrogate table itself to manage that. It's also possible that this form of nesting might not be problematic in the first place.

Reconnecting External BaseInfo to Core, Part 2

• There are now GetExternalBaseInfoAs() and GetExternalDeepInfoAs methods on the faction, to mirror those that are on the squad.
  • This helps us have a more orderly error if something goes wrong, and it also allows for the code to look that much more consistent, too.
  • It's fine to talk about the BaseInfo or DeepInfo directly on a faction, and if you want to then cast that to a specific implementation type that can also be done. But if you need to move from the base implementation to the specific implementation, the most robust way that errors would be reported is by calling these methods.

• LazyLoadSquadWrapper and SquadWrapper have both been updated to hide more of their internals, and thus have fewer bits of confusion.
  • Because these are structs, there's definite risk of confusion on the part of some end programmers, but now there is not. There is also now a Clear() method on both.
  • Related, also removed the SquadPrimaryKeyID_Neg1ToPos serialization variants. People should instead use SquadPrimaryKeyID_PosDef0.
    • In all cases a squad ID of 0 means null, and anything less than 0 will be saved as 0.
  • There are REALLY some notable cases in the external data that need to be using these two structs, specifically because without that the usage of ship pooling might cause some very strange artifacts.

• Fixed up a whole lot of references in the UI to be properly matched to the new format.

• Faction BaseInfo now has a virtual AppendStateForDebugDisplay() method, which does a much more efficient version of what GetAstroTrainsStateForDisplay() and similar used to do.

• All of the external base and deep infos now have a AppendStateForInterfaceDisplay() virtual method on them, and they both seal the ToString() method, fill it with an error, and mark it as obsolete.
  • The idea is that we should never be calling ToString(), as that's confusing and wasteful on the GC. Instead, we pass in a buffer in an ordered fashion.
  • The fact that it is now sealed and obsolete will in both cases help us automatically convert older code to the newer format, since rather than it being invisible (that was one problem before) it is now a compiler error.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • AIReservesPerUnitData is now AIReservesPerUnitBaseInfo
  • NebulaGuardiansPerUnitData is now NebulaGuardiansPerUnitBaseInfo
  • AstroTrainsPerDepotData is now AstroTrainsPerDepotBaseInfo
    • AstroTrainData has been renamed to AstroTrainBehaviorType. If you didn't already know what this was (I did not), it was incomprehensible in the code until you looked in the xml.
    • Fixed a minor issue with TrainsSpawned being serialized twice.
  • AstroTrainsGuardData is now AstroTrainsPerTrainGuardUnitBaseInfo
    • This now uses a LazyLoadSquadWrapper and also uses better serialization. This should solve several rare issues, and is also slightly more efficient during gameplay.

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • AIReservesData is now AIReservesFactionBaseInfo
  • NebulaGuardiansGlobalData is now NebulaGuardiansFactionBaseInfo
    • Fixed some pooling issues.
  • AstroTrainsGlobalData is now AstroTrainsFactionBaseInfo
    • A variety of data has been ported over from the SpecialFaction, and part of that also involved decoupling some of the settings data from the loading of xml. It was messy and prone to potential errors the other way.

• ExoData is... an interesting challenge. This is technically first-level faction subdata, but just something that can be run from a variety of different facions.
  • I don't think that pooling these would give us any notable benefit, so instead these should really just become... a secondary data structure that the game doesn't manage at all, and that is a readonly variable on the faction classes in question. That keeps them implicitly pooled with those factions (there are X ExoData classes if there are X faction classes in pools that have ExoData on them as a readonly property (readonly means the root object can't be reassigned; the data inside it can be changed all day long).
  • In other words, the best way to think about this as if it were just directly data on the faction objects. But of course, we don't want to LITERALLY do that, because then we're repeating code, so instead we have a readonly object that has the fields insde it. Handy!
  • The actual TLDR of this is that it's a challenging concept for a moment, but then has a deceptively simple solution. Hooray.

• Added a new DoGeneralAggregationsPausedOrUnpaused() virtual method on faction BaseInfo.
  • The idea is that there's various lookup information we might need to use from the front-end and the DeepInfo, and this is data that we can simply aggregate from other existing data.
  • This should be done asap on load to make sure the UI is correct from the start. Therefore, we call it right after we call DoOnLocalStartNonSimUpdates_OnMainThread, which is another vritual method you may or may not have a use for.
  • This also needs to stay up to the second accurate even when the game is paued, so it gets called right BEFORE we call DoOnPerSecondWhilePausedNonSimUpdates_OnMainThread() (so that method can use its results if need be).
  • Finally, this also needs to be up-to-the-sim-step accurate when the game is running, so it gets called right BEFORE we call DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost() (so that method can use its results if need be).
  • Reworked AstroTrainsFactionBaseInfo to use this, and it should be a lot more efficient as well as providing a better pattern for other factions in the future. The code also is more brief.

• Added a new DoubleBufferedList<T> to ArcenUniversal, which is a lot like a SwizzleList<T>, except easier to use and based around List<> rather than an array of T[].

• Added GetFactionBaseInfoOrNullAs_Safe<T> and GetFactionDeepInfoOrNullAs_Safe<T> onto squads, for quick acquisition of the faction external info as its native type.
  • Fixed an issue with Dark Spire Vengeance Generator description appenders, where in multi-DS games they would likely all show the information for the first DS, or something else equally wrong. Now they properly show the info for the DS faction of the actual VG you hovered over.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • FallenSpirePerUnitData is now FallenSpirePerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • FallenSpireGlobalData is now FallenSpireFactionBaseInfo
    • Updated the way that FallenSpireDifficultyTable stores its rows by intensity so that we can look them up more efficiently.
    • All of the various lists of cities, debris, and so forth have been updated to use the new DoubleBufferedList<T>.
    • Additionally, rather than having a simple DarkSpireEnabled boolean, this faction now keeps a double-buffered list of all the dark spire factions.
    • And finally for now, SpireDebris_ForUI and SpireDebris_ForUI have both been removed, as the new double-buffered lists that controls the central SpireDebris and SpireCities is now sufficient for our purposes here.
  • DarkSpireData is now DarkSpireFactionBaseInfo
    • DarkSpirePerPlanet is the first example of external data within external data. This really should be pooled, but we can't do that centrally.
      • The methodology for this is going to be up to individual modders and faction designers. I don't have a central solution for this, but I can provide code to replicate. On the worst case, if you do nothing but Clear() the list that is incoming, you could just throw the data away and it might be a memory leak, or the GC might take care of it. Deviating from the approach outline below by too much can introduce an MP-client-only leak, so please just follow the approach below. ;)
      • The better case is to add a new pool that is just for your object type, and then use a ProtectedList or ProtectedArcenSparseLookup where you need to centrally store these. That way when you clear, it goes right back in the pool instantly.
        • To copy this code, simply copy the Pooling section from DarkSpirePerPlanet and, replace "DarkSpirePerPlanet" with the name of your class. Additionally, make sure it inherits from the same interfaces as DarkSpireData does.
        • Next, PerPlanet on DarkSpireFactionBaseInfo was previously a ArcenSparseLookup<Int16, DarkSpirePerPlanet>. It is now a ProtectedArcenSparseLookup<Int16, DarkSpirePerPlanet>. That means that whenever it has entries removed or cleared, it will pop them back in the pool.
        • Finally, Clear() calls need to be Clear( true ) for that list, and anything that would call new DarkSpirePerPlanet() should instead call DarkSpirePerPlanet.GetFromPoolOrCreate(). Congratulations, you now have a leak-proof, MP-friendly, pooled, high-performance piece of sub-data.
    • By the by -- there used to be a method called World.Instance.GetDarkSpireFactionBaseInfoExt_AndCacheAfter(). There's no such thing anymore. That was designed as if there could only ever be one dark spire faction, and would cause problems if there were more.
    • Similarly, some faction helper methods like GetDarkSpireFaction() are now renamed to GetFirstDarkSpireFaction(), and honestly potentially they should not be used at all.

• Vast amounts of the fireteam data has been brought back to life and converted over.
  • It has also been split into three files, for now. One which has text export bits, one which has DeepInfo bits that currently live in BaseInfo (but might move), and then the core stuff.
    • Actually, for the parts we could go ahead and move into Deep, those are now in a FireteamExtensionMethods class in the Deep dll. All but about 80 lines of the deep-type code were able to be moved there. I'll circle back around later. Please note that from the perspective of calling code in the deep dll, there aren't any real differences here (other than the things to do with strings and the one method split below).
  • Additionally, the UpdateNonSerializedFields() method has been split into two parts: one that is run on clients, and one that is not.
  • A bajillion string concatenations have been turned into double character buffers for the sake of efficiency. So has the ShipsByPlanet_ForUI field.

• Added TryGetExternalBaseInfoAs<T> and TryGetExternalDeepInfoAs<T> on squads, along with TryGetFactionBaseInfoOrNullAs_Safe<T> and TryGetFactionDeepInfoOrNullAs_Safe<T>.
  • The idea here is that in a few pieces of code, mostly some of the central Fireteam handling code, it says something like "hey if you've got some scourge data on you, then also do this." But in the event there is no scourge data on the target, or the target's data is some other type, we don't want to complain or have errors!

• On the World BaseInfo, added in DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost as an abstract method, and Safe_DoPerSimStepLogic_OnMainThreadAndPartOfSim_ClientAndHost as a wrapper method to call it.
  • The various things alongside this which let us see any error data, and clear any error data.
  • This will allow us to handle things like notifications to being a client-and-host activity.
  • While I was added it, went ahead and added the host-only equivalent per-sim-step items on the world DeepInfo. Namely, DoPerSimStepLogic_OnMainThread_NonSim__HostOnly and Safe_DoPerSimStepLogic_OnMainThread_NonSim__HostOnly.
    • This will allow for things like autosave handling, which is host-only.

• Added a new LocalPlayerWorldBaseInfo, which is our very first ExternalWorldBaseInfo.
  • The logic from SpecialFaction_Human that has to do with hovering over planets and finding paths between them has been moved into this.

• Split things up so that we can once again call the ArcenLongTermContinuousPlanningContext.RunAllContexts_ForUnpausedOnly() from the simulation, even though that's a host-only thing and thus moved.
  • Further added a new ArcenLongTermContinuousPlanningClientOrHostContext, and moved PerSecondNonSimPlanning back into BaseInfo and have it now descending from that.

• DoPerSecondNonSimUpdate_OnBackgroundNonSimThread_NonBlocking_HostOnly has moved from faction DeepInfo and is now DoPerSecondNonSimUpdate_OnBackgroundNonSimThread_NonBlocking_ClientOrHost on BaseInfo for factions.
  • Same thing with DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_HostOnly, and it becoming DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_ClientOrHost.

• The seven notifiers that were still left in the old SpecialFaction_Human class have now been moved into their own files in BaseInfo, and are filled from the LocalPlayerWorldBaseInfo world BaseInfo class.

• All of the code for generating intel tab entries (internally called objectives) has been moved back to being run on the client and the host.
  • Sigh. This was a lot of work to make host-only, and the host was going to have to periodically share that data with clients, but at least that no longer has to happen.
  • This is now non-serialized data, as it always had been in the past, and doesn't go in savegames or across the wire to clients.

• Added a new DoPerSecondNonSimNotificationUpdates_OnBackgroundNonSimThread_NonBlocking_ClientOrHost onto ExternalWorldBaseInfo, which we can now use to generate notifications that are not faction-specific.
  • This is really important for purposes of things like players who are not human empires (solo ark, necromancer, etc). Some notifications would be shared across human player types, others would not, and now we have the choice to actually set that up.
  • Also? If you're a modder and you want to add a notification, and it's not connected to a faction of your own, then this now gives you a way to do it. Previously there was not a way other than adding a fake invisible faction or something else wasteful (but required) like that.

• Added DoPerSecondLogic_Stage1Clearing_OnMainThreadAndPartOfSim_ClientAndHost and DoPerSecondLogic_Stage2Aggregating_OnMainThreadAndPartOfSim_ClientAndHost on the BaseInfo for factions.
  • We still have the host-only versions on the DeepInfo, and that won't change. The host-only version also has Stage0, Stage3, and Stage4. The client section only ever gets its version of Stage1 and Stage2.
  • A lot of factions do aggregations that need to happen in Stage2 on the client and the host, so simply moving those over to BaseInfo while still having the option of having things split is a good idea.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • ScourgePerUnitData is now ScourgePerUnitBaseInfo
  • AstroTrainsPerTrainData is now AstroTrainsPerTrainBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • ScourgeGlobalData is now ScourgeFactionBaseInfo

• A bunch more of the Astro Train faction data has been split over to BaseInfo, partially to go ahead and get some of the notifications relating to them able to function. This really makes things more legible and organized, I have to say. It's still a mess because it's not done, but there's no more wondering which variables are available when, which is really nice.

• Status update! Another day done, and... well, a lot of framework and structure has been implemented today, and SOME data has been transitioned over.
  • In a lot of respects, I was underestimating just how much additional framework work (additions to BaseInfo to make things convenient and quick, and make the cross-talk work) there was when I gave my three-day estimate yesterday.
  • At this point I don't really have a good sense as to how much more framework refactoring there is going to be, but I have a couple of the simpler factions to the point where their BaseInfo is done, and the squad-level data is easy. And the world-level data is also off to the races now, so that's also good.
  • The main problem is that I still have at least a full day of more work on BaseInfo before I can fully start on DeepInfo, although I am making many DeepInfo changes as I go to lessen the impact of that later. I can't imagine that DeepInfo will be less than a day, but it could be two. BaseInfo could be 1-3 more days on its own, which really really stinks.
  • On the other hand, the code is looking GOOD. I can actually understand what's going on, and aside from the truly most complicated factions, I have all of the general relationships figured out. The AI itself is one of the nastiest things remaining, but I plan to try and be as literal of a straight-port as I can, get that working, and then later potentially improve its structure further.
  • Even so, I haven't been able to run the game for something like two weeks now, so we'll see how many runtime errors have piled up once I even get everything to compile, which is somewhere between 2-5 days of work left before I hit THAT point. I had been allocating a day-ish to get it functional enough to share with players again, but I wonder if that's optimistic thinking. We shall see.
  • Another bit of good news is that many MP issues that I was going to have to tackle, some of whcih were multi-day problems that were going to be super error-prone, are just... no longer a worry. And as I've been doing this, I've also been doing a systematic code review and have fixed a number of bugs that were affecting SP at least some of the time. So this is paying dividends all over the place, it's just... really long.

Reconnecting External BaseInfo to Core, Part 3

• Added a new GetDoesThisImplementInterface() on ExternalData_AbsoluteBase, which means that all of the new BaseInfo and DeepInfo classes are able to use this.
  • This is something that allows for a new form of generalized expansion of code capabilities from just the external dlls, without having to update Core.
  • This is useful for purposes of mods in particular, letting them add discoverable methods without having to have access to Core (or without having to alter it if the modder does have access to it).
  • Internally, this method checks to see if the end type that it is (for example, some Faction's BaseInfo type) ALSO happens to inherit from some interface of your choosing. It caches the results, so it's very efficient.
    • Please note, simply calling the normal "BaseInfoClass is YourInterface" to see if that is true will not always return true for interfaces, particularly if they are not directly assigned on the most recent class, or if you are checking for an interface that is a parent or grandparent of the interface which is directly assigned to your class.
      • Confused by that? Doesn't really matter -- just use this method to avoid any funny business with "is" or "as" keywords. Then when you get a true back from "SomeExternalData.GetDoesThisImplementInterface( typeof(YourInterface) )", make sure and do "(YourInterface)SomeExternalData" to get a YourInterface, NOT "SomeExternalData as YourInterface" in order to properly get the interface.
      • If you're curious, then calling "(YourInterface)SomeExternalData" when "SomeExternalData.GetDoesThisImplementInterface( typeof(YourInterface) )" returns false will give you an InvalidCastException. Technically you could wrapper a try/catch around that and use that to direct your code, but this is rightly considered very bad code and also is comparably very very slow.
  • So what can we DO with this?
    • Well, let's take ExoData as an example. As mentioned in yesterday's notes, sometimes there is an ExoData that is now directly assigned to a class (like FallenSpireFactionBaseInfo). Let's break down how this works.
      • If you know that FallenSpireFactionBaseInfo has ExoData and you want to look at it, you can just call faction.GetExternalBaseInfoAs<FallenSpireFactionBaseInfo>() and then immediately access the ExoData. Well that's simple!
      • But what if you DON'T know what type the BaseInfo is, for instance if you're iterating over a list of BaseInfos for various other factions for whatever reason. Maybe they are from other mods, even. Who knows?
      • Well, in that case we need a generalized way to ask ANY BaseInfo "Do you have an ExoData?" In the past, we could just call faction.GetExoDataExt( ExternalDataRetrieval.ReturnNullIfNotFound ), and what we want is an equivalent to that. So that's what we're creating here.
    • First of all, we need a new interface. In the file where the ExoData class exists, I have defined a new interface called IExoDataHolder, and gave it just a single method, called GetExoData(). See code for details, but it's super duper simple.
    • Secondly, any class that has ExoData on it that I want to expose in this generic fashion should now inherit from IExoDataHolder. For the moment, that's just FallenSpireFactionBaseInfo, but later it will be two others once I port them in.
      • Now on thos classes, I have to implement the very very simple GetExoData() method that I designed in IExoDataHolder. This is one line of internal code, just returning this.exoData.
    • Now, back in some other code where I would have previously called faction.GetExoDataExt( ExternalDataRetrieval.ReturnNullIfNotFound ), I can call the following:
      • "if ( otherFaction.BaseInfo.GetDoesThisImplementInterface( typeof(IExoDataHolder) ) )"
      • then: "exodata = ((IExoDataHolder)otherFaction.BaseInfo).GetExoData();"
  • And that's it! It would have been possible for me to make this code slightly less verbose, BUT that would have been at the expense of flexibility for faction designers and mod designers.
    • With the current approach, you can implement ANY arbitrary data-holders on any subset of classes you choose, and Core never needs to be altered in order to do so. The power of that is really useful.
    • Please bear in mind that for most sub-data, we don't remotely need to make use of this. We don't always CARE what factions use what feature. But for cases where we want to loop over all the factions and ask "which of you can give me this kind of subdata?", this new pattern lets you do so.

• NanocaustPerUnitData has been retired from the game, since it was noted as deprecated in 2019.
  • ConstructorData for the nanocaust was not marked as deprecated, but it sure was unused, so it's also gone.

• Added a new AllegianceHelper static class, which has things like enemyThisFactionToAll and such moved into itself.
  • Fixed a couple of small bugs in here that could lead to strange allegiances. One of which was related to multiplayer allegiances with players beyond the first.
  • Beyond that, there was a lot of special-case type code that was looking for specific factions and making choices about that. Those instead need to be triggered by xml data or this is not future-proof and mod-friendly.
    • Added allegiance_should_always_be_neutral_to_all="true" as a new option for factions, and this is now on the nebula guardians and territory dispute factions.
    • Added allegiance_should_always_be_friendly_to_all="true" as a new option for factions, and this is now on the Zenith Trader.
      • Fixed several ways in which factions could become hostile to the Trader in particular.
    • Also added allegiance_factions_i_always_hate="Faction1,Faction2" and allegiance_factions_i_always_ally="Faction1,Faction2".
      • This lets us centrally define some relationships that should always be friendly or enemy, using the name (InternalName) field from the factions in question.
      • Key note: these faction names are not checked, so they can be from mods or dlc that are not installed without problem. But if you have a typo, it may be hard to notice for that same reason.
      • This is not only more flexible (for mods and otherwise), but it also prevents the AllegianceHelper from accidentally overwriting entries if things like MakeEveryoneAnEnemy or whatever are called.
      • At present this is used to make Nebula Guardians and Champions hate one another, and Necromancers and Templars to always hate one another as well.
  • Also moved SetDefaultStartingFactionRelationships() into there, so that the defaults can be seen in External code (it had recently moved to Core) as well as changed as-needed.
    • Bear in mind you can override the defaults at a faction level via a mod, but you can't do that over what is in this helper class for ALL factions. But you can certainly rewrite faction alliances however you want via a mod.
  • As part of this, made it so that we never can override (via larger faction alliance methods) team affiliations (for the minor faction teams of three colors) or always-hate or always-ally settings. There were definitely some cases where someone could break off previously.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • MacrophagePerSporeData is now MacrophagePerSporeBaseInfo
  • MacrophagePerHarvesterData is now MacrophagePerHarvesterBaseInfo
  • MacrophagePerTeliumData is now MacrophagePerTeliumBaseInfo
  • OutguardUnitData is now OutguardPerUnitBaseInfo
    • Made OutguardGroupDataTable serialize by index, and thus improved the data transmission of this.
  • MarauderOutpostRaiderSpawnData is now MarauderOutpostRaiderPerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • MacrophageGlobalData is now MacrophageFactionBaseInfo
  • NanocaustMgr is now NanocaustFactionBaseInfo
  • OutguardGlobalData is now OutguardFactionBaseInfo
  • DysonData is now DysonSphereFactionBaseInfo
    • DysonAntagonizerData has been removed, and merged back into DysonSphereFactionBaseInfo. There was always a 1:1 ratio of these existing, and there is no benefit to having them separated at this time, while there are several downsides.
  • HRFData is now HumanResistanceFighterFactionBaseInfo
  • MarauderData is now MarauderFactionBaseInfo
    • The marauder UpdateAllegiance has been updated to be more robust and use the AllegianceHelper.
    • Eight constants from the marauders xml were messily being saved into the savegame, which was mildly wasteful and moderately confusing. This is all cleaned up now.
  • RiskAnalyzerData is now RiskAnalyzerFactionBaseInfo, and is on the risk analyzer faction rather than on a human faction (that was super confusing and likely led to errors).
    • It is now our second example of an IExoDataHolder.

• Updated the following squad data into the ExternalSquadBaseInfo format, usually with a rename on top of that:
  • DarkZenithPerUnitData is now DarkZenithPerUnitBaseInfo
  • ZenithMinersPerUnitData is now ZenithMinersPerUnitBaseInfo
  • ZenithArchitravePerUnitData is now ZenithArchitravePerUnitBaseInfo
  • TemplarPerUnitData is now TemplarPerUnitBaseInfo
  • TerritoryDisputePerUnitBuildData is now TerritoryDisputePerBuilderBaseInfo
  • SappersPerUnitData is now SappersPerUnitBaseInfo

• Updated the following faction data into the ExternalFactionBaseInfo format, usually with a rename and some movement of functions from the old SpecialFaction class.
  • GlobalCPALogic is now AICrossPlanetAttackerBaseInfo
  • ScourgeFactionBaseInfo is actually fully set up now, apparently I got distracted in the middle of that one.
  • DarkZenithGlobalData is now DarkZenithFactionBaseInfo
  • ZenithArchitraveGlobalData is now ZenithArchitraveFactionBaseInfo
  • ZenithMinersGlobalData is now ZenithMinersFactionBaseInfo
  • TemplarGlobalData is now TemplarFactionBaseInfo
  • NecromancerGlobalData is now NecromancerFactionBaseInfo
  • TerritoryDisputeTeamData is now TerritoryDisputeTeamFactionBaseInfo
  • NomadPlanetsGlobalData is now NomadPlanetsFactionBaseInfo
  • SappersGlobalData is now SappersFactionBaseInfo
  • MigrantFleetsData is now MigrantFleetsFactionBaseInfo

• Went ahead and removed the champions and nebula guardians factions and their related data.
  • The history of them is available in svn, but as has been discussed elsewhere, necromancers and templars replaced these.
  • At the moment, the time cost of translating the rest of these over was going to be a waste of several hours.
  • EnableChampionResponse is now EnableNecromancerResponse, although that setting is not yet hooked up to do anything, so it's commented-out for now.
  • A lot of the central code that was related to champions has been renamed to SoloArk, and converted slightly in preparation for that kind of faction later on.

• We're now to the point where the only faction info remaining in BaseInfo that has not been converted is stuff relating directly to the AIs themselves!
  • There will still be a flood of other things needing to be adjusted after that, but that's a milestone that is approaching (and a milestone that has been hit, as well).

• The concept of ProcessFactionConfigCustomFields has been removed from the game.
  • This was something that was previously running right after you left the lobby and translating the settings from your "soft settings" into permanent saved data on the actual factions.
  • As of a few days ago, I made it so that we're now pulling much more directly from the "soft settings" in the first place, which is useful for making sure that the game always has your latest settings even after making changes in the post-lobby settings areas.
  • It also lets us avoid having to save the data a second time for the factions, so that the "soft settings" location is instead the only place the data is saved -- which is good, because when you save the same thing two places you are at BEST wasting some bandwidth/disk space (less than a kilobyte, but still), and at worst you secretly have an accidental disconnect from what front-end shows and what the back-end is working from.
  • In place of this, there's a new convention -- not enforced -- of DoRefreshFromSettings() on factions, where we can get the settings on a small random interval, called from the chain of DoGeneralAggregationsPausedOrUnpaused(). This keeps the CPU usage down, things nice and responsive, and the settings super up to date, all at once. It's also easier code to read.

• We now generate a completely random FactionRandomSeed on each faction when we're starting the game. This allows us to do things like random AI types that are unpredictable, but still deterministic per-campaign.
  • Actually we wound up not using this right now, but it might be useful in the future, and it's very little data, so what the heck; let's keep it.

• Serializing a table entry no longer confusingly takes two string variables, one optional and one not, when deserializing while taking only one for serialization. It now just requires one for both, which is the same one.
  • Additionally, that FieldNameForErrors is no longer optional for these.

• Work is ongoing with getting the AI and its sub-factions set up. A lot of this is coming together quite well.
  • There were a few surprises, like the way that I need to cache the AI Type after randomly choosing it, but that's not something you can change once the game starts, anyway (or if it's adaptive, it changes itself in the one spot here). Differentiating this from the other settings was not hard. Now it's just a matter of doing the rest, carefully.
  • There's a solid chance that this will implicitly fix some bugs that were recently reported about things like "individually chosen sub-difficulties not working properly when universal difficulty is off." That was likely an artifact of the order of operations craziness that was part of the older way of using ProcessFactionConfigCustomFields. Once it's ported over, it should be order-of-operation-independent.

For Modders: How To Handle Lists Of Sub-Data In Multiplayer-Friendly Way

• Added a new extension method to List<> and ProtectedList<> that makes deserialization into an existing uneven list waaaay simpler.
  • This is what needs to happen, over and over again, on clients when they are getting certain kinds of external data.
  • PlannedWaves are one of the many that needs this sort of logic, so formalizing it so that it isn't prone to errors (and is easier to read and understand as end-coder) is a win.
  • Basically, you use this when:
    • You have a list of stuff that is variable length over time, and the client is going to be getting a deserialized copy of it pretty frequently... AND
    • You don't want to just clear that list (because what a waste on the one hand, or other code might be having references to objects in it)... AND
    • You do want to refill the list with the most up to date data in host-order, and then discard any leftover husks that were on the client.
  • Example 1:
    • Client has a list of 10 incoming waves. Host sends an update that includes 7 waves for whatever reason.
      • Result from this process: Client deserializes those 7 into its first 7 slots (overwriting whatever is there), and then discards the last 3. If it's a ProtectedList<>, then those 3 are headed back to the pool automatically.
  • Example 2:
    • Client has a list of 1 incoming wave. Host sends and update that includes 5.
    • Result: Client deserializes the first 1 into the existing one it has, then adds 4 more.
  • Example 3 is simply when the number of waves match, whether or not the data in them does, and it just overwrite-assigns each one as it goes.
  • The idea is to handle all of that logic without having to write all of that code every time (and to avoid the typos and confusion that are inhernt in copy-pasting such code all over the place).

• Okay! Time to handle the first of the sub-objects that are pooled properly. Let's look at PlannedWave as an example.
  • We're going to want to have it inherit from the following: PlannedWave : TimeBasedPoolable<PlannedWave>, IProtectedListable<PlannedWave>
    • This sets us up to let us have it go in a variety of kinds of pools, as well as also function inside "protected lists" (where when you clear the list, it goes back in the pool for you automatically).
  • There is then a "#region Pooling" section, and you basically should copy that whole bit over and change and references to PlannedWave to be whatever your class is (same as with the inheritance bit above, of course).
    • In this section there is a DoAnyBelatedCleanupWhenComingOutOfPool(), which you can usually leave blank, but essentially it's helpful if there are some object references that might briefly be in use still when this gets put back into the pool, that you don't want to cause nullref exceptions from.
    • In this section there is also a DoEarlyCleanupWhenGoingBackIntoPool(), which tends to happen on background threads instead of one that is more busy, so anything you need to blank out that you can, should probably go here. You want to make sure and reset EVERY variable and list and so on that is a sub part of this object in this.
      • In this example case, to keep things simple and also consistent, I've got a SetDefaults() method that is being set in the constructor as well as on the DoEarlyCleanupWhenGoingBackIntoPool(). This is a good pattern to keep.
        • It is SUPER important that absolutely every property or variable on your class (PlannedWave in this case) is reset to its default value in SetDefaults(). Otherwise the next wave will have some stale data and act strangely when this gets pulled back out of the pool.
    • Please also note that there's a private constructor in here, and then no other constructors anywhere else in the class! We don't want any way of creating new objects of this sort other than calling the static method GetFromPoolOrCreate(). Anything else we call would be bypassing the pool and causing at least a mild memory leak.
  • At this point we have a poolable object that can only be retrieved from a pool, but we're not doing any retrieval or ever putting them back. So now let's do that.
    • On AISentinelsFactionBaseInfo, I would normally have a List<PlannedWave> WaveList. Instead we're going to use a ProtectedList<PlannedWave> WaveList, and it pretty much works identically.
    • This is a case of this list being the truly master list. If it's in this list, it's meant to exist. If we take it out, it's meant to be gone back to the pool. So, based on that, we don't ever have to think about calling ReturnToPool() manually on the PlannedWaves. That's handled for us. Sweet!
      • On Clear and Remove from this list, just be sure to pass in true as the required parameter, and it will do that handling for you. In the event that you wanted to use a protected list without it automatically sending stuff to the pool on removal, you could pass false... but that is not advised.
    • In that case, that means that the only thing we need to change beyond this is wherever we were previously calling "new PlannedWave()". That simply changes to "PlannedWave.GetFromPoolOrCreate()", and boom we're pooling. And we're done.
  • In the case of PlannedWave, I am using a ConcurrentPool, by the way. This is something that can be written to from any thread, and it has some nice advantages. But there's not "time out" cooldown before waves are reused from the pool. Because of the nature of how waves are created, this is not a problem.
    • In the case of a piece of data where you might have lingering references for a while, you might be better served using a TimeBasedPool instead. That works exactly like ConcurrentPool except that you can specify a cooldown time that the objects must stay in the pool before they can come back out. 2 intervals of 30 seconds each are enough for all long range planning to complete for every faction in a super pessimistic case, if that's what you want to base it on. Otherwise just give it a couple of seconds of delay. At any rate, PlannedWave doesn't need that.

Reworking Subfaction Relationships And The AI and Beacons, Part 1

• Added a new AISentinelsFactionBaseInfoExtensions class, with GetAISentinelsBaseInfo() on it as an extension of the faction class.
  • The reason I'm doing this for the sentinels and not any other factions is because of all its many subfactions, and their constant need to get the info of their parent.
  • I'm also planning on storing all of the meaningful subfaction data of any note on the parent itself, to further tighten these things together.

• Removed GetPrecedingAIFaction() and GetNextFactionOfSpecialFactionData() from the Faction class in Core.
  • Now that I'm working on the AI and its subfactions, the current situation with how they are linked is just really untenable. It was unreliable before, and would not always properly link the factions together exactly as expected. Time to change that.

• Just like we have FactionIndicesIAmHostileTo and FactionIndicesIAmAlliedWith, factions now also have FactionIndicesOfMyChildren and a new FactionIndexOfMyParentIfIHaveOne.
  • These let us keep track, explicitly, of which subfactions are linked to what parent faction, in a bidirectional sense.
  • There is a new MakeChildOfOtherFaction( OtherFaction ) method on the faction class that makes this a one-line call to initialize.
  • You can then just call the new GetParentFactionOrNull() on factions to get a reference to the parent faction if there is one.
  • Part of the idea here is to make it much simpler for our AI factions and subfactions, sure, but also the other part of the idea is to make it so that mods can now set up compound-factions like this if they need to. Or us with other related factions if need be.

• General improvements to some of the serialization code have been made.
  • Among these, planets and factions no longer have a serialization gate calculated for them. They are simply now capped at 511 for planets, and 255 for factions. These limits were already well enforced elsewhere.
  • Fixed a minor new SpecialFactionData serialization issue from it getting mangled in all the ripping and tearing. It would have been fast to find later, but this saves time.

• Getting rid of some other methods that we no longer use and so just clutter things up:
  • On faction: GetDoesThisFactionHaveBaseInfoType(), GetDoesThisFactionHaveDeepInfoType(). That new FactionRandomSeed.
  • On squad: GetFactionBaseInfoType_Safe(), GetFactionDeepInfoType_Safe().

• Split the BaseInfo and DeepInfo serialization out of the main faction methods for serialization into new SerializeExternalFactionData() and DeserializeExternalFactionData().
  • This is actually a pretty huge deal, because now we'll send the externaldata at the end of any transmission, and at the end of any savegame.
  • This means that factions, squads, planets, etc, all already exist when externaldata starts getting loaded. Previously a lot of that was not loaded yet, so externaldata had to use indices during deserialization rather than looking at real objects. This will simplify code, and speed up some bits of code, and reduce coding errors.
  • Same for BaseInfo and DeepInfo on squads, which is now based from SerializeExternalSquadData() and DeserializeExternalSquadData().

• The multiplayer fast-blast sync code has now been updated to handle the BaseInfo from host squads needing to be synced over.
  • The faction BaseInfo sync in multiplayer via PeriodicFactionExternalSyncDataThatJustOverrides has been updated to the new format, which was very simple.

• Split SerializePlanetFactionEntities and DeserializePlanetFactionEntities out from the core PlanetFaction serialization, since that was rather confusing code the way it was before.
  • Also, CurrentAttackMultiplier and CurrentSpeedMultiplier are no longer serialized at all (it's recalculated every frame).
  • And the shipgroups are no longer serialized, even in super-brief form, except for AI factions (they're the only ones who use it).
  • Uh, also a lot of other stuff moved around. It's more efficiently organized, less likely to cause errors, and easier to extend.
    • The really big one is that this completes the move of all the externaldata to the very end of a network transmission or a savegame save. For factions, squads, and the world. That way the external data can always freely talk about the Core data without worrying if it's been loaded or not yet.

• For the sake of clarity, the factions that are auto-added-always-there-but-only-one have been moved into one file, and the "factions that are auto-added-per-AI" have been moved into another.

• On factions, auto_add_one_of_faction_if_missing_on_save_load="true" has been changed to should_always_have_exactly_one_of_these_in_every_game="true"
  • This has been removed from several sub-factions of the AI, and instead is just on the things like Outguard, AI Reserves, Instigators, and the zombie factions.

• There was then an actually_seeds_multiple_so_do_not_complain="true" that was an answer to some of the factions that had must_be_at_most_one="true" on them but were not supposed to complain.
  • Those factions were erroneously marked that way in the first place (because of a long history of evolution of the game, it's complicated), so that actually_seeds_multiple_so_do_not_complain="true" has been removed.

• auto_add_one_of_faction_per_ai_if_missing_on_save_load="true" and remove_if_more_than_one_per_ai_prior_to_this="true" have both been removed, as they were kind of funky.
  • Instead, added a new is_a_subfaction_of="OtherFactionName", which lets us define the relationships directly, and which causes them to get created one-per-such faction, call MakeChildOfOtherFaction() on said factions, and disappear if that faction is not present. All speaking here about on the lobby load, or savegame load.
  • The warden, hunter, PG, CPA, relentlesss wave, and border aggression are now marked as subfactions of the AI.
  • The territory dispute teams are now marked as subfactions of the parent territory dispute faction.

• takes_on_color_of_the_first_ai_faction="true" is being left alone -- that's useful for instigators and so on -- but takes_on_center_color_of_the_preceding_ai_faction="true" is being removed.
  • We now instead have a takes_on_center_color_of_the_parent_faction="true" as an option which can work for any faction that is a child of another. For now it's just used with the AI subfactions: warden, hunter, PG, CPA, relentlesss wave, and border aggression.

• is_considered_visible_part_of_the_ai_faction_for_what_can_own_purposes="true" is removed, as it was not actually used at this point.

• takes_on_current_general_mark_level_of_the_ai_factions="true" has been renamed to takes_on_highest_current_general_mark_level_of_any_ai_faction="true", because that's what it does.
  • This is now used by the instigators, AI reserves, astro trains, and risk analyzers, but is no longe used by the AI subfactions.

• Added a new takes_on_current_general_mark_level_of_parent_faction="true" that is now used by the AI subfactions.
  • Previously if you had many AIs of different strengths, their subfactions would all be the mark level of the strongest AI, which was wrong.

• The way that "beacon factions" work is being completely reworked under the hood.
  • The user experience was never great (people didn't realize they could configure said factions prior to hacking the beacons but that it got more limited after that).
  • And there was a nontrivial performance drain, and some bugs, from having lots of these. Macrophage picking up spire shards when not in the game except in beacon form, etc.
  • In the next few weeks I will re-code the hacks for the beacons so that you can configure all the aspects of the beacon faction as you are summoning them. The beacons will from now on belong to the NaturalObjects faction, however.
  • In the meantime, all of the stuff related to a faction being present but "needing to be awakened" is being removed.
    • MustBeAwakenedByPlayer, MustBeAwakenedAndIsAllied, ShouldBeRetainedIntoLobbyLoadEvenIfMustBeAwakened, and HasBeenAwakenedByPlayer.

• auto_remove_faction_on_lobby_load_and_mapgen_when_other_faction_present="FactionName" is also being removed, as we don't need it anymore.

• auto_add_one_of_faction_when_other_faction_present has been converted into auto_add_singleton_of_faction_when_any_count_of_other_faction_present, to be more clear.
  • This is still used on tamed and enraged macrophage, but is also now used by the antagonized dyson sphere. This is also still used by the templars in response to a necromancer.

• When you load up a savegame into the lobby, or old lobby settings, or a quickstart into the lobby or for generation, it now strips out all the following:
  • Any subfactions of other factions. Any that should always have exactly one per game. Any that are automatically added when some other faction is present.
  • This is mostly equivalent to what we did in the past, but the general idea is we never want to see things like a dormant tamed macrophage with no main macrophage present, or extra hunters due to having fewer AIs present now, etc.
  • The only time these things are added BACK is when a full generation of the map happens (aka moving from the lobby into the game, or from the quickstart into creation of the game proper). Then all the magic fills out the relationships and should have exactly the faction list as desired.
  • We had half a dozen idle factions clogging up every savegame in the past, and sometimes more, which was potentially a waste of bandwidth and performance. It was also a barrier to adding beacons for more factions, since there was such a performance hit with each one.
  • All of the various code paths that do these sorts of things now seem to function, which is awesome.

• Removed GetDefaultFactionConfigurations() from the Scenario definition, since now most of our code is much more xml driven.

• One thing that isn't fully xml-driven, but nonetheless is now part of the Expert campaign type, is that you must have a minimum of two AIs to face.
  • A lot of the complexity of the game is lost when you only have one AI, because you don't get that ramp-up in challenge after defeating the first one, and have to figure out how to still deal with the second one.

• The seeding options for a couple of DLC2 factions that were "Only From Beacon" have been removed.
  • These will get beacons in the near future, along with some other factions that never had beacons, but they will work the new way.
  • Worth noting that this didn't play well with the Enable Beacons setting, and so could get kind of funky in Expert mode because of that.

• Also stripped out more code that was based around the concept of a faction being present as a beacon but not enabled.
  • Later, the logic will be that the beacon is present as a NaturalObject, and the only time the faction will arrive is after a hack adds them (letting you clearly choose settings for it before you complete the hack, too).

• Added in a new HasDoneInvasionStyleAction to factions, which is false by default.
  • Previously, we had a HasBeenAwakenedByPlayer which was used one way sometimes (for beacons) and another way other times (for delayed invasions). That had led to various bugs in the lobby.
  • Now we just have HasDoneInvasionStyleAction, which is used for invasions or not at all. Most factions ignore this one bool and just go about life.
    • This is primarily used by the nanocaust, but in theory other factions could use it later.

• Simplified some of the PlanetFaction data. FInts are the largest data format that we have for casual use, and each PlanetFaction object had three of them (AIPLeftFromControlling, AIPLeftFromCommandStation, and AIPLeftFromWarpGate).
  • In an 80 planet game with 21 factions under the hood (remember, one AI is a compound faction that is 6 under the hood alone), that would have been 5,040 FInts. Those each require somewhere in the ballpark of 64 bits to store, which would be a total of 40 kilobytes in a savegame that is maybe 200-600kb in total in the early stretch (so, that's as high as 20% of our entire data storage in the early game, for context. Granted, things get adjusted around a lot to avoid this, but I still wanted to minimize what is on PlanetFactions, and the above is a good enough example to show why.
  • We no longer have AIPLeftFromControlling at all, since that could be inferred from the other two. The other two are now stored as integer values, and specifically AIPLeftFromCommandStation is serialized as a 17-bit format, and AIPLeftFromWarpGate as a 9-bit format.
    • Assuming no other changes (ha -- there already were improvements elsewhere, but let's ignore those for this example), this would be 5.74 kilobytes of data now, instead of 40.
    • This matters much less in savegames than it does in multiplayer, where we are frequently sending things like the planetfactions from the host to each client just to make sure that things are synced up.
  • The bandwidth use by the game prior to this whole rewrite of the game's structure was pretty darn alarming (though still lower than a poor-quality youtube video, for instance). But that bandwidth usage made higher player counts harder for people to do, and put an extra strain on people who had a lot of packet loss or were physically very far apart.
  • This is just one of those "low hanging fruit" ways to reduce transmission amounts, but the whole rewrite of all this is aimed at least partially in that direction (other than bandwidth reduction, the rewrite is aimed at code correctness, client functionality parity, and ease of coding for modders with the game having to send less self-correction data.

• Added TryGetExternalBaseInfoAs() and TryGetExternalDeepInfoAs() on faction, mirroring the versions that are on squad.
  • Added a new TryGetAISentinelsBaseInfo() in AISentinelsFactionBaseInfoExtensions as an extension on the faction class. This more or less mirrors the "get or return null" style.
  • Essentially, in your mods you can do a global search and replace for "GetSentinelsExternal( ExternalDataRetrieval.ReturnNullIfNotFound )" to "TryGetAISentinelsBaseInfo()"

• All of the AI subfactions are now going to have sub-objects that are shared on the central AI's object. This greatly simplifies how they can access each other's information, but lets us still keep things ordered. As with ExoData, these will be readonly subobjects that we can then edit at will without ever replacing them, and so they don't need to be pooled, etc.
  • AISentinelsExternalData is now AISentinelsBaseInfo. This is now a readonly sub-object of AISentinelsFactionBaseInfo, which makes it really convenient for any sentinels faction or subfaction to get at the data. I'll be following the same pattern with the other data of this sort.

• TimeBasedPoolable now does the thing where it seals and obsoletes ToString(), and has a virtual method called AppendStateForInterfaceDisplay().
  • We really want to avoid having classes that are calling ToString(), because that's pretty much always GC churn and a waste.
  • EntityOrder now uses AppendStateForInterfaceDisplay instead of ToString(). It's a case of not even knowing it was a problem until it was made a compiler error.
  • PlannedWave also has been updated in this way, as has EntitySystem.

• ArcenCharacterBuffers of all sorts now support passing in an ArcenPoint and having that serialize nicely.

• ExtragalacticBudget has been updated to be poolable in the same pattern that was implemented for PlannedWave.
  • Fixed an issue that would cause deserialization errors in ExtragalacticBudget on MP clients. This was a longstanding issue, looks like, and very rare and confusing if it happened.
    • The fix for this was switching to the new DeserializeUncertainNumberOfEntriesIntoExistingList(), which is a great example of just why this new thing is so darn useful.
  • ExtragalacticBudget is an interesting case, because it was very complex with multiple constructors. But those simply are turned into static Create() methods, and then suddenly the complexity is gone and the program flow is the same. There are many similar examples in Core, but this is the first one in external code.

Reworking The AI And The Last Of BaseInfo, Part 1

• AISentinelsBaseInfo is now AISentinelsCoreData, because that naming was HYPER confusing. Naming anything BaseInfo or DeepInfo when it's not actually that is a really bad idea, turns out.
  • The other sub-classes of AISentinelsFactionBaseInfo are now being set up as well. Bear in mind that these are part of the sentinels faction data, not part of the sub-data of the sub-factions. They do have their own sub-objects, though, to keep things organized but accessible.
    • AIWardenExternalData is now AIWardenCoreData
    • HunterFleetExternalData is now AIHunterCoreData
    • HunterFleetExternalData is now AIHunterCoreData
    • PraetorianGuardExternalData is now AIPraetorianGuardCoreData

• AICommonExternalData is now GlobalAIWorldBaseInfo, which is something that I am attaching to the world object, because it's thing that is truly global between all of the AI factions that might exist.
  • This is slightly more efficient from a data standpoint, and is simpler to use. Any faction can do this sort of thing, particularly if there are going to many of that faction.
  • This also declares a singleton, so it's really easy for any code anywhere to now just go GlobalAIWorldBaseInfo.Instance.AIProgress_Effective or whatever.

• AIPChange is now being treated like PlannedWave, with the new TimeBasedPoolable and IProtectedListable inheritance.
  • This is probably now the simplest example of this pattern in use, because it's a fairly low-data-and-methods class to begin with.
  • There was some logic in GetAIP that previously was setting the AIP to the highest version based off of any AIP. The coder had been note the code as being a bit hacky. I've move that over into GlobalAIWorldBaseInfo, and it calculates those things now on that method.

• Added a new EnumIndexedArray in Universal, so that we can get rid of all those class-specific versions and add more functionality to them.
  • In particular there is a new ClearTo method. But there's also a reference tracker, so that we'll know how many there are in the game if we do a data dump. And there's just no longer a bunch of duplicate code all over the place so much.
  • Then removed the following classes, converting them to instead use this new EnumIndexedArray:
    • Universal: ArcenEnumIndexedArray_PerformanceSegment, ArcenEnumIndexedArray_FourDirection
    • Core: ArcenEnumIndexedArray_CoreFunction, ArcenEnumIndexedArray_EntityLineHardcodedType, ArcenEnumIndexedArray_MetalFlowPurpose, ArcenEnumIndexedArray_SpecialEntityType, ArcenEnumIndexedArray_OtherSpecialEntityType, ArcenEnumIndexedArray_EntityRollupType, ArcenEnumIndexedArray_FactionType, ArcenEnumIndexedArray_PlanetFactionBooleanFlag, ArcenEnumIndexedArray_EntityOrderType, ArcenEnumIndexedArray_FactionStance, ArcenEnumIndexedArray_ResourceType, ArcenEnumIndexedArray_EntityBehaviorType
    • External: ArcenEnumIndexedArray_AIBudgetType, ArcenEnumIndexedArray_HunterFleetBudgetType, ArcenEnumIndexedArray_PraetorianGuardBudgetType, ArcenEnumIndexedArray_AIWardenBudgetType, ArcenEnumIndexedArray_Code, ArcenEnumIndexedArray_ClientConnectionStage
  • Tip for modders: you can use a simple global search and replace for each type you remove, in this fasion: "ArcenEnumIndexedArray_AIBudgetType<" with "EnumIndexedArray<AIBudgetType,"

• The external data for factions and squads now has a virtual method that you can override called DoAnyInitializationImmediatelyAfterSquadAssigned() and DoAnyInitializationImmediatelyAfterFactionAssigned(), respectively.
  • On AISentinelsFactionBaseInfo, I am now using this to call "this.WardenInfo.InitializePathfinders( this.AttachedFaction );" and the same for the other subfactions.

• It turns out that ExoData is the perfect model for how to handle the AntiMinorFactionWaveData, so that's the pattern I'm going with, including having a IAntiMinorFactionWaveDataHolder.
  • MarauderFactionBaseInfo and NanocaustFactionBaseInfo have been updated to hold an AntiMinorFactionWaveData WaveData object.

• And that's the last of converting old style ExternalData over to BaseInfo!
  • At the time of this writing, there are 195 errors still remaining (newly revealed now that all the types are converted over) in the BaseInfo dll that need to be resolved before I can start truly working on DeepInfo.
  • There's also 85 "//EXTERNALDATA_TODO" tags where I commented out some code in BaseInfo to let it compile at one point last week. I'll have to uncommented those and get that all compiling, and then BaseInfo will be done-enough-until-DeepInfo-forces-more-changes, but at least I'll be able to compile it.

• Uncommented out all 85 of the //EXTERNALDATA tag sections, and now I'm up to 385 reported errors in the BaseInfo dll. Time to get cracking on those!

• Removed the PreferredAIFactionIndex from faction external base data.
  • This has been made obsolete by FactionIndexOfMyParentIfIHaveOne on the Faction object itself.

• BaseAIFaction.ChangeAIP() should now be GlobalAIWorldBaseInfo.Instance.ChangeAIP(), which has been converted to the new format.

• SpecialFaction_CPALogic.cpaDelayTime is now AICrossPlanetAttackerBaseInfo.CPA_DELAY_TIME.

• Rather than ever using factionExternal.AIProgress_Effective, it should now always be GlobalAIWorldBaseInfo.Instance.AIProgress_Effective.
  • Same with AIPChangeHistory and things like that.

• GetPrecedingAIFaction() should now be GetParentFactionOrNull().

• I've improved the way that we're able to call in and get faction configuration data. You can have it error when you ask for data from a field that does not exist on that faction, or just return the default value.
  • Most of the time you'll want it to error, because that's probably a glitch in your code that will cause unexpected behavior. It's the sort of thing where a small typo can just wreck your whole day, but now it's actually going to tell you about it and therefore it's super quick to fix.
  • Then again, there are other times where we're scanning past lots of factions, and saying something like "hey, do you have an intensity? I'll show it if so. Got an allegiance field? I'll show that, too, if you got one." And if the answer is "no I don't have one," that's not an error at all. Those are the cases where you'd pass in false to having it error on missing data.

• Added some extension methods where you can call faction.CustomData_GetSpawningBooleanOne(), and a variety of others that start with CustomData_[Whatever]
  • Many references to Ex_MinorFactionCommon_GetPrimitives have been expunged, with most being converted using the above, but others just directly referencing some variable in a new place. Still more to go.
  • This also pulled out a ton of code from ProcessFactionConfigCustomFields() that was very specific to just that one method, and now that's more formalized and something that can be called from more than just that one place.
  • The complete difficulty settings for the AI and all of its subfactions now get pulled out properly, and should be far more accurate (there were some reports before of univesal difficulty not being handled properly when disabled)
  • Also as part of this, many of the generic names for things (like "SpawningBooleanOne") have been renamed to actual sensible things (like "EnableFimbulwinter").
    • Side note: it was a huge pain in the rear to even try to add new proper field names before, so this is not a criticism of faction coders in the past. It was an archtectural challenge that no longer exists.

• Added a new DevourerFactionBaseInfo, which tracks the TimeLastExisted for that faction, as well as some other info that was in the DeepInfo dll but needed to come out.
  • Also added a new LoneWandererFactionBaseInfo, which will act as the underlying basis for handling both the Devourer and the Zenith Trader.
  • Please note, this is actually a great class to look at if you're thinking of doing something similar. It has a TON of extra comments in there, mostly about how to properly use DoubleBufferedList<>.
  • I am going to start having a habit of putting "TEACHING_MOMENT:" in the code anytime there's an example class with a lot of things that are explained. That way coders can search for "TEACHING_MOMENT" and find anything that I (or someone else) has put in there.
    • There are two teaching moments in this particular file. One for double buffered lists, and the other for Sealed methods and abstract submethods.

• Updated my DoubleBufferList so that the swap and clear actions are two separate things.
  • The reason for this is that at swap-time, it's very likely that another thread has a hold of the expiring display list. So clearing that would cause an exception.
  • By the time we loop back around and are ready to construct a new list, any other threads are fairly certain to all be using the new display list, so clearing that other list won't bother them. There's a much larger gap between list constructions than there is any display-list-accesses, by design. This is a common feeder pattern.
  • It's something I had realized the other day, and had just put on my long list of to-dos, but at this point I need more of these to use for things like keeping track of the Devourers running around, etc.

Reworking The AI And The Last Of BaseInfo, Part 2

• A variety of code translations to the new format that are not worth detailing, and finished the work of getting rid of Ex_MinorFactionCommon_GetPrimitives.

• Fixed an issue where we had apparently wished for the Macrophage to have the same "Seed On Nomad If Possible" option that the Dyson Sphere had, but the setting was partially missing.

• The central AI BaseInfo now has direct references to all of its subfactions in the form of handy names like SubFac_CPA and SubFac_Warden and so on.
  • This doubles as a self-checker to make sure that an AI faction is in proper health with all the expected subfactions and no extra subfactions hanging around.
  • This then lets us get rid of a lot of things like GetCPALogicForThisAI(), happily.

• Added a new AISubFactionBaseInfo, which acts vaguely like the LoneWandererFactionBaseInfo class.
  • This is now the base class of all of the AI subfactions, and it makes the act of finding the parent faction and its data super trivial, and also ensures proper linkages or it has a centralized freakout about that.

• All of the various factions in the game and DLCs now have proper BaseInfo classes, and the xml is all linked up for BaseInfo and DeepInfo.
  • As part of this, the three types of zombie factions now share a common base class, like how the lone wanderers and the AI subfactions do.
  • Also, the Svikari have become a subclass of DarkZenith rather than using DarkZenith quite so directly.

• All of the ISpecialFaction classes from DeepInfo have now been renamed to their new proper names, and inherit from ExternalFactionDeepInfo.
  • This leaves a lot of other fixing to do, but it gets things in proper place at least. This is things like SpecialFaction_Human becoming HumanEmpireFactionDeepInfo.

• The above two items also involved splitting a number of files, and setting up some template empty files for various factions, etc.
  • This gets all of the major objects into place, but just not yet with the correct formatting and linkages and so on.
  • It was about an hour and a half just to get things organized like this, and would have taken a lot longer if I did it in small stages. There are still probably tens of hours left on this conversion, but having things working by next weekend is looking increasingly feasible.

• At this point, there were still 374 lines in BaseInfo and DeepInfo that referred to ExternalDataRetrieval, which no longer exists.
  • A targeted set of 64 global search and replace commands got all of those fixed up. (Yeah, it was odd to me that it was exactly 64, too.)
  • Total errors reported by the compiler in BaseInfo are now down to 224, but this was fixing up a lot of others that had not been reported in that number yet, or which are in DeepInfo. The IDE tells me there are 3269 errors in all at the moment, but that's both overcounting and undercounting at the same time. It's progress, anyhow!

Reworking The AI And The Last Of BaseInfo, Part 3

• All of the general cases where I could update "Implementation is" or "Implementation as" in a global search and replace way are now fixed (I didn't count how many it took, but a couple dozen operations to get about 140 items corrected). I'll get the rest as I go through files in more detail.
  • So this leaves me with 220 errors reported by the compiler for BaseInfo. I spent a lot of the morning doing payroll, so let's see if I can knock out the rest of BaseInfo by the end of today. That really would be nice.

• All of the BaseInfo objects for factions that are only allowed to have a single version now have a static Instance on them for quick access.
  • Worth noting: if those factions are not active in the current game, those instance variables will be null! This is by design.
  • These classes also now all inherit from IExternalBaseInfo_Singleton, just to make it a bit more clear again what the intent is.
  • This applies to: Dark Spire, Fallen Spire, Astro Trains, antagonized dyson sphere, HRF, Risk Analyzers, tamed and enraged macrophage, naturalobject, ai reserves, instigators, all three zombie factions, outguard, zenith miners, dark zenith (which has had its class split from its core, for later benefit because of svikari), nomad planets, territory dispute parent, necromancer, and templars.
  • Might as well have a consistent pattern, and this makes converting a lot of other code easier.

• All of the "WAVES_TODO" markers are happily able to be removed, as they will now work properly on clients without any further effort thanks to this general restructuring of data.

• Spire relics can now be moved around properly by multiplayer clients. That was something I hadn't even realized was broken by my changes in July, but it was -- a great example of why this much more intentional split, as time-consuming as it was, is so much better in the long term. I can clearly see in the code itself when something is wrong, without having to wait for testing an exception to pop up.

• The ClientSafeExternal namespace has been removed, and its contents returned to External, because the separation is now in between dlls rather than just namespaces.
  • This is something that is no longer needed at all, and was a half-measure where now a full measure is being taken.

• Marauder outpost descriptions will now have proper info after you load a save and before you unpause the game.
  • Right now I'm making dozens of code changes that are not really worth detailing since they are just tidying things from one format to the other, so instead I'm mentioning things that happen to get fixed along the way.

• Added a new DoubleBufferedDictionary, which works just like the DoubleBufferedList does. Turns out this is needed for various factions, marauders among them.

• Fixed up some code from UpdatePowerLevel() in a ton of factions to keep them from having values that flickered for a few microseconds at a time. It probably wasn't a problem, but why tempt fate.

• The scourge now use vastly less GC every frame in general, and also less when showing their details for the UI.
  • Several minor formatting issues were fixed with scourge being displayed on the UI, and additionally it now shows the list of spawners like it does for armories. Before it meant to, from the look of it, but it only actually showed the count.

• Added a new DoubleBufferedArray, which is now used by various factions, including the scourge.

• Ported over a huge amount of the scourge data and the logic for its data aggregation.

• Added a new DoubleBufferedValue, which is basically like the other double-buffered things but just for a single integer or similar. This is also now used on the scourge data.
  • There are many other ways to handle this same problem, the simplest of which is a working variable. But in looking at the scourge class, where there was a mix of local variables and naked assignments, it's easy to see that that's hard to maintain properly.

• At this point, down to 191 compiler errors reported in BaseInfo.
  • It's worth noting that the bug count in DeepInfo falls dramatically with each of these checkins, too, but those are not numbers I'll ever actually see as high as they are, since I can't attempt a true compile of DeepInfo until BaseInfo works.

• Macrophage turns out to be a really complicated class in terms of how the base, tamed, and enraged variants relate to one another.
  • There were a number of parsing things that I don't think were quite right, and probably led to things like the enraged version always having an effective intensity of 0.
    • I've adjusted things so that tamed still has the intensity of 10 that someone put in there (wow, nice), and the enraged version has the intensity to match the highest regular macrophage now.

• Also to support the macrophage, because they do this sort of cross-faction aggregation to fill what used to be called SporesPerPlanet, but which I'm now calling CrossFaction_SporesPerPlanet, I've had to make a number of new little things.
  • For one thing, the way that this data structure gets filled across faction lines is now vastly more efficient (blind Add calls and one-way caching, versus expensive Contains lookups).
  • This then required me to added a new Stage2APostAllFactionAggregating method, though, that lets me run the final SwitchConstructionToDisplay() on the DoubleBufferList array that is CrossFaction_SporesPerPlanet. That needs to happen once per cycle, after all of the other factions have done their stuff.

• While I was at it, I noticed that the central logic had not yet been set up to actually call the BaseInfo Stage1 and Stage2, they were just calling DeepInfo Stage1 and Stage2.
  • Now it calls both, although I'm unsure if I'll keep Stage1 and Stage2 on DeepInfo. I see pros and cons to that, but it will likely just remain, but be empty for all of our built-in factions. I could see some modders needing the feature.

• Added a DoubleBufferedArcenSparseLookup, because I know how fond some programmers are of the ArcenSparseLookup's extended capabilities.
  • I'd prefer not leaning on this too terribly much, because it comes at a bit of an extra cost compared to a regular Dictionary, but such is life. Sometimes we need those capabilities.
  • As an aside, I built some of the "freedom to call into keys that don't exist without error and get a default value back" capabilities into all of my DoubleBuffered structures. It's one of the favorite advantages of ArcenSparseLookup over Dictionary, and it's something you can use with DoubleBufferedDictionary without having to go all the way to DoubleBufferedArcenSparseLookup.
  • In my opinion, the only real good reason to go to DoubleBufferedArcenSparseLookup is if you need it to be a dictionary but also have the ability to sort, like ArcenSparseLookup has.
    • I almost switched the list of Telia to being a DoubleBufferedArcenSparseLookup because of how it is used in SporeDescriptionAppender, but then realized that I could accomplish what I wanted just by using a foreach rather than a for statement if I kept it DoubleBufferedDictionary. This will be a tad more performant, so that's what I went with.

• Also, I'm no longer keeping any concept of "Wilds" for macrophage. That seemed to be a mix of enraged and normal macrophage for some reason.
  • I now have a singleton for both tamed and enraged, and then a full list of the regular macrophage. They can be queried separately or in series. Sticking the enraged in with the regulars didn't sit right with me, it felt like a problem waiting to happen.
  • Unlike pretty much every other faction I've seen so far, Macrophage is having to have a bit more surgery to parts of its actual logic, partly because it's just that old, and partly because it's been maintained by a revolving list of programmers over the years. Please let me know what issues you find once we're back to being able to play on the new builds again.

• And with that I'm down to 161 stated compiler errors in BaseInfo. But once again, the day has had lots of progress on DeepInfo, so it's not going quite as slowly as it seems to by that error count number alone. My hope, knock on wood, is that DeepInfo will just fly by fast after BaseInfo is all finished up.

• A bunch of fallen spire stuff got moved over into BaseInfo. That was the easiest yet because it's more or less my code, most of it.

• Same with the Zenith Architrave, mainly because they're just a really modern faction in terms of their code.

• Also the same with the Migrant Fleets faction, which is very well done from what I see so far.
  • One thing I fixed was converting cachedMovementTargets from an ArcenSparseLookup into a ConcurrentDictionary, because both the UI and the long-range-planning threads use this, so a cross-threading conflict was otherwise just a matter of time. Now the existing logic will work fine.

• Down to 147 reported compiler errors in BaseInfo.

• Some more general fixes and translations, getting the error count lower.

• Fixed a variety of xml filenames that said BaseInfo instead of DeepInfo. It didn't hurt anything, but was confusing.

• Trying to get the host to trigger SharkB (which must happen in DeepInfo) in response to something that happens in a gamecommand that the client also needs to execute (in BaseInfo) is an early and interesting test of a pattern that is not very common thus far.
  • Overall most of the control code is in DeepInfo, and so we don't need to "call up" like this. Instead, deep just calls down and can see everything, which is the idea. But these exceptions do exist, and so having a formalized way of handling them has always been part of the plan.
  • In this particular case, we're getting our first DeepInfo world-attached object, which I'm calling GlobalGeneralDeepInfoCommandHandler. It's just for a variety of general things, over time.
  • Additionally, the ExternalData_World base object now has its first method of interest, which is called DoAfterEntityDeath(), and lets you pass in a string ActionCode to specify one or more types of things.
    • The flexibility of this is that we can have more than one thing handled in our central library, but also modders can set up their own objects of this sort and also call them. We don't need one object per type of on-death call, since that would be excessive.
      • It's important to know that these are called manually, from the bottom-up, when the sim code detects something it wants to react to. This is the same pattern SharkB was already using.
  • There's another "TEACHING_MOMENT" piece of code in there (I've added several of those today, as well as flagging some comments of Badger's that were teaching moments as being such so people can search for them).

• Imported a bunch of Dark Zenith logic (the stage 2 aggregation). This was more complex because it was having to do way more work since it did not have double-buffered collections in the past.
  • Some things also were able to be simplified because of the new architecture where you can actually get planets and squads and such immediately on deserialization, whereas before this version you would not. So for instance, serializedFimbulwinterEligibleTime is able to be ditched, and we just write straight to FimbulwinterEligibleTime, saving ourselves some hassle.
  • Same with OriginalPlanetIdxs going away, and just having OriginalPlanets, instead.

• Okay, that's it for today. Down to 134 errors in BaseInfo. It took a while to convert over the Dark Zenith, mainly because it's such a big complicated class. But thankfully, the newer logic (double buffers, mainly, but also the deserialization order) really make the code a lot more legible and cut down on extra work.
  • I'm sure it's going to be mildly frustrating to Badger that he didn't have those tools when he originally coded this faction, but I wouldn't have thought of adding them without seeing the same patterns over and over again and realizing I could simplify it based on that.

Reworking The AI And The Last Of BaseInfo, Part 4

• GameCommands used to be all in one giant monolithic file. They have now been split out into a variety of files, by loose purpose or association. This makes it a lot easier to find a specific GameCommand, or to look at several GameCommands that are related to one another.
  • As part of this, and because I needed to, two of the GameCommands (both dealing with changing the galaxy map, as part of ZO) have been moved to DeepInfo, where they will only be run by the host, and where they can access data and methods that only the host would have.

• The dyson antagonizer faction logic has been ported properly between base and deep info now.
  • Same with the the main dyson sphere faction, including wiht the ability to handle multiple spheres around the galaxy, now using the same pattern for that which macrophage are using.

• All the cheat codes work again (yeah that was a thing).

• All of the description appenders work again.

• The central faction processing code is now split out fully into DeepInfo, which is excellent. It's still kicked off from logic in BaseInfo, but the actual execution (only on the host) is in DeepInfo now.

• A number of faction methods that used to be FOR UI ONLY and very problematic if used otherwise are now using ThreadStatic internal markers on their lists to make themselves threadsafe. These are now labeled as being threadsafe, and can be used by the UI, main thread, LRP, whatever -- all at the same time, even!
  • The various notifications that used these ui-only versions of the methods have been updated to point to the new ones, and a TEACHING_MOMENT flag has been put in next to a comment about ThreadStatic at one point.

• The HostOnlyJournalsAndAutosaveAndSimilarHandler has been moved into DeepInfo.

• AIUtilityMethods has been moved from DeepInfo into BaseInfo.

• Fixed a likely bug where nanocaust intel entries have not been shown ever since frenzy fleets were removed (back around the time of fireteams).

• Imported some various things from deepinfo and sappers and such into BaseInfo that were going to move here soon enough anyhow.

• Okay! Well, BaseInfo now compiles, which means that I'm able to get going on DeepInfo properly.
  • There's a lot to translate over there, but I've been steadily doing that as I went, too, so we'll see how long this really takes.
    • There's a fair number of things which are kind of mindless search-and-replace things, like a lot of the ExternalData faction methods no longer have the faction being passed in. That's now handled as AttachedFaction as an embedded field in the class itself.
    • To trick the IDE into fixing over 800+ references that would otherwise have to be done by hand, I simply did a search and replace for the methods in question, which then made them have references to (typically) "Faction faction" that no longer exists. So I made it exist, in the base class. Then used the IDE's rename function (F2) to rename that to AttachedFaction, and presto. Then just delete that temporary variable you made. You can do the same in your mods, if you wish.
    • The reason for gettting rid of Faction as a property on this methods is that it creates ambiguity and the potential for errors. It's just cleaner not to have it.
  • At the moment, I have about 200 compiler errors in DeepInfo, and 2934 errors in visual studio, which is probably vaguely accurate. It's several thousand lines that need correcting or adjusting in some fashion. There's still over 240 references to SpecialFaction_[Whatever], and those will take a mixture of search and replace (when it's checking is or as again), and manual adjustments as I link up the base and deep infos of each faction in particular.
  • Based on the speed of things lately, and the somewhat unknown volume of code that has to be corrected, I think this will easily take me another 2-3 days. I worked through almost all of last weekend, and I hope not to do that again. The picture will become more clear tomorrow and the day after as I get further along.

• Okay, I couldn't quite leave it for the night. I got autosaves working there's a new AutosaveHandlerDeepInfo class in DeepInfo which replaces what used to be there.
  • Really, all the code is the same, but it just doesn't have any chance of going to a client, and isn't attached to any random factions. The translation of this was literally under five minutes, nice.

• Got a little over halfway through setting up quick-links between the DeepInfo objects and their BaseInfo counterparts. This includes setting up the DeepInfo objects as singletons where need be, for handy access.
  • As I was doing this, I also made various other corrections and shifts to code as I saw it, and in all the number of errors in DeepInfo are plummeting. There's still plenty there -- visual studio says 2400, which seems about correct at this point -- but the code is thinning out thanks to being better organized (data and data access in one place, logic in another place, and in consistent order and consistently organized in both).
  • I don't want to get ahead of myself, but after a few weeks of being overly optimistic on schedules (or is that just a terminal case for me, one could wonder), it actually looks like I was -- potentially, and majorly knock on wood -- overly pessimistic, this time.
  • THAT said, assuming I do come in ahead of what I was projecting, that's still just going to be leaving more time for me testing for runtime errors and fixing those. It's not like as soon as this compiles it's good to go. I've been coding for almost three weeks now on very long days and over weekends without testing any of it -- because I couldn't -- and if you're a programmer, you know how horrifying that is. All things considered I think I've been very clean with my work in this time, but I expect to find some major gaffes.
  • There's a vague, outside possibility I could have this compiling tomorrow. At the current pace I have observed, it's not outside of the realm of possibility. It's not hugely likely, either, but it's what I'm going to shoot for. I want those two days prior to the weekend for testing and fixing the runtime.

• Removed some probably-unused-and-unworking code for ships warping in and out over distances. As Badger notes, we have way cooler things to look at these days, and we haven't noticed this actually functioning for a few years now. It was taking up data space, though, and eating some CPU processing, so out it goes.

Moving Into DeepInfo, Part 1

• ModdableGameCommandExecution has been moved out of BaseInfo and into DeepInfo, after seeing a bit more how it is used and thinking about it.
  • Some nice comments have also been put in there, with two TEACHING_MOMENT entries: one about what the moddable game command things do, and another about how to get data from the host to clients in general.

• Thinking of trying to minimize multiplayer bandwidth, I decided to make a second Stage3, which does happen in BaseInfo and thus on clients.
  • We want to be relatively careful what logic we put in there, to not make MORE desyncs. But for simple logic in particular -- like zombies attritioning -- it's going to be way more of a desync if this DOESN'T happen on the client.
  • There's a TEACHING_MOMENT note in there for one of the zombie factions.

• I also changed my mind and am removing DoPerSecondLogic_Stage1Clearing_OnMainThreadAndPartOfSim_HostOnly and DoPerSecondLogic_Stage2Aggregating_OnMainThreadAndPartOfSim_HostOnly from the DeepInfo.
  • We just don't need it, and I don't think any mod that would need that is structured properly. These methods are specifically design around working on the data that is held in BaseInfo, so we don't need a DeepInfo version.

• Somewhat related, Stage0 -- aka DoPerSecondLogic_Stage0Clearing_OnMainThreadAndPartOfSim_OncePerFactionTypeEvenForFactionsNotInGame_HostOnly -- has been removed from the game.
  • That was simply not something that was ever needed at this point. We have better ways of handling similar things, and the new Stage2A is part of that.
  • Specifically, also a lot of the double-buffered lists and similar make this completely unneeded.

• The remaining 16 factions (of of 37) have now been fully linked up from DeepInfo to BaseInfo. I was feeling really happy and motivated last night and got the first 19 out of the joy of it.

• Down to 141 direct compiler errors at this point, and 1782 visual studio errors (which is likely to be pretty close to accurate). There's still a chance I can knock the rest of this out today, but if not then tomorrow is pretty much a certainty.

• Added a new ExternalFactionBaseInfoRoot, which now sits under all of the BaseInfos for factions.
  • It was becoming really clear that there were some commonalities that needed to become shared, plus a few things that needed to be able to happen on a schedule that is centrally controlled (to avoid some mapgen errors I suddenly realized were possible thanks to a comment by StarKelp in one of his faction files).
  • First of all, we now are ensuring that DoRefreshFromSettings() and DoGeneralAggregationsPausedOrUnpaused() get called from mapgen.
  • Secondly, something that was just an optional convention that I had been using (that being DoRefreshFromSettings() itself) is now a required part of faction base info. It allows for great flexibility without overtaxing the processor.
  • Next, the base implementations for GetRaidDesirability() and GetRaidTraversalDifficulty() now have somewhere appropriate to live.
  • Lastly (for now), this allows us to implement a proper GetFireteamById() rather than just GetFireteamBaseById(), mirroring how faction code used to work.

• The call to link the faction externaldata into the factions, and also to call DoAnyInitializationImmediatelyAfterFactionAssigned(), has been moved around substantially.
  • The idea is that we want more of the data available as we're deserializing savegames, but when it comes to mapgen we also need to be after-all-factions-but-before-actually-doing-any-mapgen.
  • This should save quite a few headaches for me from after I get this compiling again (I would have had to make these order of operation changes in general).

• GetFireteamById() has been moved over into BaseInfo where it belongs.

• All of the pathfinding logic has been adjusted to be where it should or call the new locations of relevance.

• Added a new AIRelentlessAndBorderAggressionFactionBaseInfoRoot, which is a common ancestor now to AIRelentlessWaveFactionBaseInfo and AIBorderAggressionFactionBaseInfo, mirroring the setup we have in DeepInfo.

• Stage2 logic was still hanging around in DeepInfo for the following factions and has been migrated over (along with a bunch of other data):
  • Cross Planet Attacker AI subfaction.
  • AI Relentless And Border Aggression root.
  • AI reserves
  • six more to go, at the moment, but AI reserves took... a lot.

• The AI Reserves faction no longer has dual lists for "sim" and "lrp." It just calculates their data with double-buffered lists, instead.
  • This also includes a new teaching note, and some code that has been adjusted to exist in DeepInfo only, along with some guidelines on why exactly it works this way versus some other way.
  • In other words: how do you know which place a variable or collection should go? This lays it out, with the example of DeepstrikeEligiblePlanets as a guide.

• Added a new SortThreadsafeCopyOfDisplayList() to double buffered lists, along with an optional TryClearThreadsafeSortedCopyOfDisplayList().
  • See, the problem with the double buffered lists is that normally you sort them ONCE right before you finish constructing them and flipping them active.
  • However, after they are the display list, any number of threads might be reading from them, so to sort then would not be threadsafe in the slightest.
  • To work around that, this method allows any thread to sort a copy of the display list, in a perfectly threadsafe way.
  • This will create some extra data in RAM, very small amounts, which you can clear with TryClearThreadsafeSortedCopyOfDisplayList() if you feel like it. Or it will just be cleared the next time you call SortThreadsafeCopyOfDisplayList() on the same list from the same thread. Either way.
  • This is another addition that has been made thanks to AI Reserves and some of the complicated things they want to do. Their list of eligible planets gets sorted over and over based on distance from other planets they are thinking about, but this is all on a super low priority background LRP thread, so it's fine... except for the need for thread safety.
  • Yep there's another TEACHING_MOMENT in the code about this.

• Split out some of the waves logic into a new WavesHelper static class that variable factions can use. AI Reserves needed it.

• "Set the preferred AI faction" logic has been removed from the remaining locations where it existed, since that is now obsolete and handled as parent/child factions. There were over a dozen copies of the same lengthy code all over the place.
  • That then left me with 21 references to PreferredAIFactionIndex, which have also now been converted over.

• Four instances of DoOnFirstSightingOfFactionByPlayer needed adjusting, and have gotten it.

• Rather than being a static class, WavesHelper is now a singleton.
  • This allows us to extend that singleton in DeepInfo, and add DeployComposition into there where only DeepInfo can reach.
  • We use the same pattern for FactionUtilityMethods, if you're curious.

• Necromancer stage 2 and various data has been ported to BaseInfo properly.
  • Good grief this was complicated. It provides a couple of TEACHING_MOMENT items, too.
  • This also shows one of the examples of how we no longer need to precalculate a string (and hit the heap when we do), by instead changing a ToString() method to one that takes in an ArcenDoubleCharacterBuffer and writes to that on-demand.
    • A lot of other classes also do this exact thing now, but SkeletonVariantPercentagesByFleet_ToString() and similar provide a great example of it in action.

• The necromancer faction is mostly working again in general, but it has uncovered a bit of a problem in my logic with hacking in particular.
  • Right now, hacks are only scheduled to run on the host, but there's a lot of info about "can we start a hack" and such that the client needs to be able to see. I'll have to deal with that.
  • The necromaner sacrifice unit hack in particular is also problematic, because it executes on all clients and the host (well, or just on the host), but is using local-to-one-machine data (selection info). That's going to require some refactoring on my part to figure out how to handle cases like this.
  • I have a similar problem coming up with the sub-types for the sentinels, warden, hunter, and PG. They essentially need some of their stuff in BaseInfo, and some other bits in DeepInfo. I mentioned this last week actually as a concern point that I didn't know how to solve at the time. At the moment, I have a fairly decent idea of a way to handle it, but I need to examine the code a bit more first.
  • I'm down to 2009 errors at this point. Not sure if I'll finish tomorrow or not, at this rate. Which really bites, but it's the nature of the beast, I guess. At the moment I'm fixing some things that were always going to work incorrectly in multiplayer anyhow, so it's not just architecture for the sake of nothing.

• Moving hacking back into BaseInfo. I'll build in hooks into DeepInfo as needed, but the vast majority of this code really needs to be run as part of the sim.
  • One thing that is interesting about this whole revamp process is that the way that data is available to the UI or where it needs to be is really becoming increasingly clearer, even to me. I knew in pretty great detail what I wanted to do, but there have still been many cases where my intuition was wrong. This is one of them.

• The necromancer "sacrifice" code and hacks has all been removed. That was going to be complicated, but Badger noted that was deprecated in favor of the Rift hacks. This saves a lot of time! But still helped to identify a problem with how I was handling hacking in general in DeepInfo, so was useful.

• Now that hacking has been moved back to BaseInfo, there were several hundred errors that needed to be fixed (they were going to need to be dealt with anyway, but now they prevent BaseInfo from compiling again until I fix them).
  • As part of this, moved around a lot of things that are not really worth detailing, and fixed a number of references that were still old-style. It's hundreds of lines of code, but there's nothing really surprising in this batch worth reporting in detail.

• TerritoryDisputeTeamFactionDeepInfo has had its Stage2 merged into BaseInfo properly. This was needed for some of the hacking stuff, but also just required in general.

• Added ConstructionContainsKey() to DoubleBufferedDictionary, and ConstructionHasKey() to DoubleBufferedArcenSparseLookup.
  • This is needed to support some of the territory dispute logic.
  • Ah, and it turns out we also need versions for the display dictionary, so those were also added.
  • And it turns out that we also need DisplayContains() and ConstructionContains() on the DoubleBufferedList, so that's also in now!

• Added a new table called ExternalDeepLink, which lets us define singletons in BaseInfo, which can then be initialized via xml, and then assign their reference back to BaseInfo from DeepInfo.
  • Essentially, this answers the question "how can a modder or faction designer add arbitrary code calls from BaseInfo to DeepInfo without modifying Core code?
  • I actually already use this sort of pattern all over the codebase, and have since 2018. Lots of external classes register themselves with more information than the raw table that loads them has access to.
    • This is kind of complicated to explain in a written note like this, so it will be much better to show examples. I'll have some teaching moments in there, too.
  • The first and simplest example is ExoGalacticDeepLinkRoot / ExoGalacticDeepLink.
    • ExoGalacticDeepLinkRoot is an abstract class defined in BaseInfo. It defines what methods it has, and has a static Instance.
    • Xml states to load ExoGalacticDeepLink, which is in DeepInfo. In its constructor it assigns itself to the Instance variable on ExoGalacticDeepLinkRoot.
    • After this, any BaseInfo code can just call ExoGalacticDeepLinkRoot.Instance.YourMethodHere(), and that will be directed straight to DeepInfo and do whatever it is you want it to.
  • Ultimately it's a really simple pattern, but it's just hard to explain in text like this.

• We now also have a DysonUtilityMethodsDeepLink, for supporting a number of hacks.

• Fixed an old and undiscovered pooling issue with ZenithArchitravePerUnitBaseInfo, where it would get into a funky state after being hacked.

• There's also now a NecromancerDeepLink. And a MacrophageDeepLink.
  • These are stupidly simple to add (just a couple of lines of code), and allow for full control of how you can talk from BaseInfo to DeepInfo. Really pleased with how this turned out.

• Stopping for the night. At this point, all of the hacking stuff should work now, and is back in BaseInfo but using DeepLink calls where needed to call into DeepInfo-only areas.
  • There are presently 92 compiler errors reported for DeepInfo, which is getting pleasantly low, and last I saw the visual studio error count it was just under 2000 now.
  • This is definitely a tense schedule, but there's a substantial chance I could get this done tomorrow -- to the point where I can start running it and finding all the runtime errors on Friday, anyhow.
  • We'll see how it goes. Honestly there is deceptively high numbers of things that need attention or conversion, but the new DeepLink tool solves a HUGE number of the problems that I was otherwise having trouble figuring out a graceful solution to.

Moving Into DeepInfo, Part 2

• BuildTerritoryCollection() was called a variety of times in the territory dispute faction classes.
  • This basically just made a lookup, by faction, of what territory each faction controlled. This sort of thing used to be required, but now we can access the factions so easily and they all have double-buffered territory lists already. So we can, happily, skip this and use data we already have.
  • Getting rid of this sort of thing (which previously existed because of either threading concerns or because of difficulty finding the correct other classes' data) is a major win of the new framework. Saves a lot of CPU, and in many cases also some minor RAM GC churn.
  • Note that I'm not particularly calling out the territory dispute faction anymore than I've meant to call out any other faction. They're all coded very well in the limitations of the old architecture. But at various points, various factions illustrate different benefits of the new architecture.
  • Added a new DoForAllTeams() method to TerritoryDisputeParentFactionBaseInfo that also helps offset the need for this method, and methods like BuildTeamCollection().
  • GetColonyShipTargets() and similar are also now a great example of how to reduce CPU usage and GC churn by using threadstatic lists. These are thus also threadsafe methods. Please see the teaching_moment related to threadstatic if you want more info.

• The "Stage4 Advanced Allegiance Code" that I added a while back to support the territory dispute faction has now been moved into BaseInfo.

• Added an extension method for DoFor() to Dictionary, giving it that handy same method that ArcenSparseLookup contains.

• The territory dispute factions (both the parent and the subfaction) are now fully ported over, including its notifications.
  • This was a very interesting use case, because it's very modern and written very well, but also has some unusual stuff like the sub-teams. It's probably pretty obvious, but the architecture has been continually expanding as I see what each faction author wrote in detail and how I need to improve my central tools for them to do their same work in less code.
  • There is a "StarKelp_Question: hey, I was unclear on this logic." note in there at one point, which StarKelp you should feel free to either correct or confirm whenever you like.

• Down to 63 compiler errors in DeepInfo, and 1818 errors reported errors in visual studio. I'm determined to get through this today so that it all at least compiles!

• Stage2 logic has now been moved over for the last four factions that needed that pulled out of DeepInfo and into BaseInfo:
  • Dark Spire (also got rid of some ForUI variants of lists, and some list variants "for all factions" when there can only be one dark spire).
    • This also has a great example of some of the old-style codebase stuff, where it had to have a count of spire cities for itself. That's been removed, and it just asks the spire, which is able to say that really clearly now.
    • Also removed the pointless FactionIndex, which was just referring to itself it turned out; that was actually hard to be sure of for a bit. This faction is _really_ old. At one point we probably legitimately needed that to know how the external data connected in to its faction.
  • Templars (also got rid of some ForUI variants of lists).
    • A teaching_moment has been added in there, along with a bit of logic shifting on how rifts are spawned, essentially working around a cross-threading error that would happen but rarely, and it explains how to notice it. The double-buffered lists make it a lot more obvious, since you should not be calling add to a construction list outside of the construction process, and you should NEVER be adding to a display list.
  • Zenith Architrave (removed many ForUI variants, also made serialization of territory and planets ever taken direct now rather than first sent to IDx).
    • Also got it so that their BaseInfo now properly cleans itself up before pooling. (When it was originally coded, it wasn't meant to be pooled.)
    • The way that territory is calculated, and a number of other things beyond that, have been simplified a lot, and hopefully not mangled in the process. Before there were some _severe_ order of operations constraints across multiple passes that these revisions aim to make just happen in order, 1, 2, 3, much simpler.
    • Gah, this one was really a cluster. This is new top prize for hardest faction to transition over. I don't think I messed anything up, but Badger will have to review it, sigh. The logic is probably easier to follow now, though, on the bright side.
  • Nanocaust (okay, this one was shockingly easy).
    • Got rid of yet a bit more really super old deprecated code, and repointed a few variables and it is what it is.

• Moved over more notififications logic from DeepInfo to BaseInfo, which is one of those really lovely things that lets us have functionality that works properly in multiplayer:
  • 2 from Dark Spire.
  • 2 from ZA.
  • There are still 10 more I need to grab.

• Many various other redirections from old code spots to new, nothing terribly exciting with most of that.

• Dark Spire faction is now fully compiler-error-free.
  • Same with nanocaust and Zenith Architrave, and a bunch of other factions.
  • We're down to 44 compilter errors now, and 1594 in visual studio.
  • The Zenith Architrave and the Territory Dispute factions were both way more challenging than I had expected, and the largest faction of them all, Dark Zenith, still needs a lot of translaton work, as does the AI. We'll see what I can do...

• AI Reserves notifier moved over from Deep to Base.
  • The last bits of the AI Reserves DeepInfo errors are also resolved.

• Many various other random small improvements all over the place. This is going to last well into tomorrow, sigh.

Moving Into DeepInfo, Part 3

• Many various cleanups and fixes to the new format.

• Added a new DoFor on GamEntity_Squad that works on a List<>, like the one that works on ArcenLessLinkedList<>.

• Added a new HasSimStepRunSinceGameLoad on BaseInfo for each faction, which is set after Stage4 has run on each of them.
  • This is really useful, because Nanocaust had some stuff that it really didn't want to do on long-range-planning threads (it would make wrong choices) until at least one sim frame had elapsed.
  • While looking at that, it became really obvious that this is a problem that other factions would now share, if they didn't already (most already probably did).
  • So now the long range planning threads are blocked from running at all until the first sim-step pass has happened since the game was last loaded (savegame load, quickstart, game start, whatever).
  • To be clear, this is a delay of about 100ms, so it's not really noticeable. But it will make it so that the initial run of long-range-planning is never acting strange because of a lack of data.

• Nanocaust data is now properly cleaned up when it goes back into the pool.
  • Also removed yet more old frenzy variables that are not used since the advent of fireteams.

• The nanocaust now uses double buffered lists in place of the linked lists or other formats.
  • Also, the nanocaust has some cross-faction lists, along the lines of what the dyson spheres do.

• The way that the nanocaust aggregates its data has been converted into the modern format. Well, the NEW modern format, but also the modern-since-DLC1 format, where Stage2 is used for aggregations.
  • Whoof, some of its logic in general, such as for attrition or for constructor upgrades, has been moved to baseinfo and converted a bit.
    • I had to go back and check the older versions of the code on some of this, because the constructor upgrade logic applies to non-constructors and that is how it was coded. I'm leaving it because that seems to be what has been the historical function even if that's not what the code names would seem to indicate was intended.
  • The nanocaust faction is now fully up and ready to run (or at least compile). Ultimately the surgery was not too bad, and things should be more clear than they were before when looking at that code.

• Moved the remaining notifications over from DeepInfo to BaseInfo, along with the code for generating them:
  • 5 from Fallen Spire.
  • 2 from Dark Zenith.
  • 1 from Nomad Planets (that's the last of any compile errors with nomad planets gone!)
  • 1 from Zenith Miners (and with about six other corrections, all compile errors with the ZM are gone also!)
  • None from sappers, but some outer scaffolding that was prepping for them has been moved.
  • At this point, notifications should again work properly on multiplayer clients, which was one of the sticking points that led to this whole overhaul in the first place.

• HasSimStepRunSinceGameLoad has been renamed to HasPerSecondSimRunSinceGameLoad.
  • This is already what it did, but it's now correctly named. It's important to note that the long range planning threads won't start for an entire second, not just 100ms. This is of no gameplay significance from a player point of view, but it keeps the data and the factions happen.
  • Both marauders and HRF had nanocaust-style "have I loaded yet?" checks to enforce this sort of thing in an ad-hoc manner. As suspected, this affected a lot more than just the nanocaust.

• FireteamRegiments have been moved out of the BaseInfo and back into DeepInfo.
  • After consultation with Badger, it became clear that this is purely host-only information, and not something that would need to be seen on the ui in a non-debugging fashion. Therefore, into DeepInfo it goes, to avoid confusion.

• A number of things on the Marauders faction that had been keeping an index of planets now keep the actual planets since that is now possible. Makes for some added simplicity.

• After quite a number of changes, marauders now fully compile.

• The HRF now also compile, with the changes to make them use planets directly rather than planet indices.

• IPraetorianGuardTypeImplementation, IHunterFleetTypeImplementation, and IAIWardenTypeImplementation have all been removed.
  • This was a really funky way of setting things up in general, and Puffin had already made a todo in the code stating that we should move this into xml long ago. As he noted, it took him forever to find these balance levers.
  • In the more recent short term, this whole design just really wasn't fitting with the new setup anyhow, so it was doubly time to adjust this stuff.
  • AISubFactionDeepInfo now also has been dismantled, since this was just a vehicle for the three above.
  • Xml for all of the above are now present, and you can change a lot of things very easily now that were previously very hard to do! As a modder, this can majorly let you change how the hunter, warden or praetorians act.
  • A lot of the code that was tied to this whole set of structures needs to be reconnected, but I have to get some other things in place first.

• Lots of other random minor fixes and rearrangements. That got rid of the last of the low-number compiler-blocking errors... revealing a further 1446 errors. That's about what visual studio had been predicting. Woo...

• Got rid of the last 5 references to "is SpecialFaction_", down from hundreds initially.
  • That left 8 more references to "as SpecialFaction_", also down from hundreds initially. Those are also gone now.

• The AI type implementations have been moved back into BaseInfo rather than DeepInfo (sigh), and split out into multiple files so that they are easier to keep track of.
  • Technically this stuff is only used from DeepInfo, but it's a real pain to try to sequester it there, and there's nothing about the data of this that would break any of the client/host rules, so... fine. BaseInfo it is.
  • The ability to add per-sim-step or per-second logic on AI types has been removed, incidentally. That was never used.

• A bunch of the AI Sentinels DeepInfo methods were static for some reason, potentially because we thought we might want to cross-call from other locations or something.
  • Thankfully we were not actually doing that (that would have been a nightmare), so these have been made regular methods and are already private, so they have quicker access to the data of the AI they are dealing with.

• The way that AI spending ratios are assigned to budgets no longer generates a bunch of garbage for the GC to take care of.

• DoOnAnyDeathLogic_HostOnly on factions has been renamed to DoOnAnyDeathLogic_MyFactionUnitsOnly_HostOnly, because honestly it was not clear if that was how it works or not.
  • There's another method called DoOnAnyDeathLogic_FromCentralLoop_NotJustMyOwnShips_HostOnly which runs on any unit, so it's vaguely implied that the above one is only for its own ships if you see that, but it's not clear enough for my taste.
  • Because of that, there's a number of cases of indirection (that existed prior to this rework) that I'm able to simplify away. Just working more directly with object references makes things faster, and makes the code more readable.

• Fixed what seems to have been a bit of a glitch with macrophage not getting any metal income from them killing enemies in some circumstances -- maybe if they did not have a spire telium?

• Zenith Trader now fully compiles.

• The way that the macrophage keeps track of its entities has been updated to a more modern interpretation.

• Added a new UniqueNameForFactionToAvoidThreadConflicts, which defines intentional collisions for "don't run these faction's LRP threads at the same time."
  • This allows us to ensure thread safety even amongst factions with inheritance chains, like the macrophage, lone wanderers, dark zenith, and so on.
  • We had something kinda like this in the past, but it wasn't as good and didn't cross types like this. It had made it so that rare LRP cross-threading issues were possible, that now are not.

• The macrophage code now compiles, and is a real showcase of how a complicated (but understandable) faction should work in the new code framework.
  • There are a number of teaching moments, and a number of threading things that get explored here.
  • There are also two "BADER_TODO / STARKELP_TODO" entries with questions where I was curious about the logic that was pre-existing, and woudl like more clarity myself.
  • This one is a great one to look at to understand the new format better, albeit with something that is on the more middle-complex side

• At this point, I'm down to 1124 compiler errors. This is taking MUCH longer than expected, as everything seems to have with this process, but it's worth it.
  • There are some things related to hunter, warden, and PG that are just commented out right now, which I'll be re-enabling later as I get their patterns fixed up to hook them back in.

Moving Into DeepInfo, Part 4

• Got rid of the visual studio solutions that had the base game and dlc xml in themselves separately. Instead, combined both into a single solution that has the separate projects from each of them in it. This makes searching and replacing a lot easier, and I don't know why I never did this before.

• The death effects have been fixed up in DeepInfo:
  • Zombificiation of all sorts.
  • Metabolization and Necromancy were already fine.
  • Nanocaustation.

• The AI Sentinels DeepInfo class is now down to 5 errors (from hundreds), and uses its new structure to the fullest. It should be much easier to read, and it doesn't have to do subfaction lookups and such nearly so often.
  • The remaining 5 errors are related to parts of subfactions that I have temporarily disabled, so I need to get those subfactions all fixed up and then those 5 will go away.
  • Total error count is now down to 1090.

• Set up a new AISubFactionCoreDataRoot to live in BaseInfo under the sub-data for Warden, Hunter, and PG, since those all have a certain type of data involved.
  • This makes it so that they can now reach back out and talk about their parent faction with ease, rather than having to have a faction passed in and then fumble around to find it on the faction like was the case in the old architecture.
  • This actually pulls in pretty much all of the code that used to live in IndependentFleet, and translates it into the new format. It is vastly more brief code now, and I put like five different TEACHING_MOMENTS in there.
    • In particular there's some great teaching moments about the threading model of the entire game, and the System.Threading.Interchanged class and how to use that, and cross-thread-safety concerns in general.
  • This section of code (and the three subfactions that ride upon it) was the one I was most worried about coming into the weekend, so I'm glad to see it looking so well.

• Warden logic that was previously on IAIWardenTypeImplementation has now been moved to AIWardenCoreData.
  • ReceiveDonation is a really straightforward copy-paste from where it used to live. Now it's in BaseInfo.
  • The rest of those other things now have migrated over to live on top of the framework that AISubFactionCoreDataRoot has built. It's wonderfully simple by comparison to what it used to be, and the code is also much closer together and doesn't have an interface in between the definition and the implementation.

• Added a new MapgenDeepLink, which lets me handle things like seeding the base-oriented warden secret ninja bases. See how handy this pattern is? It literally can be used for anything.

• The "HunterFleetHelper", which I had created only for a little time, has gone away. This is what was on IHunterFleetTypeImplementation at one point.
  • The translation across for this was a ten minute job, after spending hours getting the general framework up to make this and warden and such actually work. Nice.

• The Praetorian data went even faster than that, excellent.

• The SetSpecialTargetPlanet command has been removed from the game, because it existed just to send data to the client which was basically host-only (and which is set in a different threadsafe manner now from the host).
  • There's several teaching moments in the code related to this, but essentially it was a pointless game command that just increased complexity a bit without actually solving any issue in the way multiplayer and the faction logic actually turned out.
  • Speaking of, in this check-in the CurrentTargetPlanetIndex moved from being a core thing on every external faction's BaseInfo, to only being a thing that the hunter, warden, and Praetorian share. Nothing else uses it.

• The UpdateFireteams() method on factions has been removed, because it was possible for someone to unknowingly just directly set NumFireteams.
  • Numfireteams is now a property whose setter does the logic that UpdateFireteams() used to do.

• The UpdateFireteamSizes game command has also been removed.
  • On the surface, this might seem a bit like bad form for multiplayer. We are changing something sim-affecting, after all... kinda.
  • But the fireteam logic all pretty much just runs on the host, and -- again -- the faction data is transmitted very frequently to the client, anyway.
  • It's more economical in all senses to just directly set the NumFireteams on the faction directly from the external code that used to call the gamecommand.

• Those last errors in the AI sentinels faction code that I mentioned earlier today are now gone, and the hunter faction code is now also error-free.
  • Praetorian faction code is now zero errors, and so is the warden.

• Also, while I was at it, in the xml for the warden I took their default multipliers for how they gauge enemy strength and their own strength and moved those from being 1.5x enemy, 1.0 self, to instead being 1.0 enemy, 1.1 self.
  • This will make them considerably more willing to help out, even on planets where they may be outmatched. We've had off and on comments for a long time about the Warden being a bit too passive, and this will change that. If it's too much, we can tweak the xml, or make variants, or whatever else.
  • Since this is controlled by xml now, end users can experiment with it with just a text editor, or modders can make their own versions of the warden who act differently, etc. No code required.

• Stopping for the evening and my total remaining compiler error count is now 935. That should be pretty accurate, and is the first time we've been sub-1000 errors since starting this whole process a long while back. At the height it was over 7000.
  • I still have some pretty heft factions to deal with (Fallen Spire, Dark Zenith, Scourge, and Sappers are the big ones, with Dyson Sphere also clocking in a lot of errors right now as well). A few other factions have a handful of errors (Migrant Fleets, Astro Trains, etc), and some mapgen code also needs a few fixes. But it's definitely feeling like the downhill slope now.
  • The AI cluster of factions was the thing I was most worried about working on, and I commented about my worry about the Subfaction interfaces a couple of weeks ago when I ran into those and remembered them (they were my doing, I believe, but I had forgotten about them). As of today, that part is all done, which is a giant weight off!

• Fixed up how the AI Start Sizes for AI empires work in mapgen now that the code for getting their settings has been adjusted.

• The other handful of compiler errors in mapgen are also now fixed.

• A handful of errors in various factions that were otherwise okay are now fixed, leaving mainly the big factions to deal with.
  • Also two small errors in TargetListPlanning.
  • And nine in EntitySimLogicImplementation_DeepInfo.
  • Six from FireteamUtility.
  • Four from FireteamExtensionMethods.
  • Two in SpecialFactionPlanning.
  • One in ExternalFactionDeepInfoRoot.

• Okay, now actually it's time to deal with some Fireteam change that are required, because they give funky results on the clients.
  • TargetPlanetIdx is gone, just leaving the actual TargetPlanet. The only reason for TargetPlanetIdx was because of the order of serialization in the past.
  • Same deal with TargetID, LurkPlanetIdx,
  • Also, DoEarlyCleanupWhenGoingBackIntoPool() and InitializeToDefaults() have had their code merged to make them not duplicate one another.
  • Made a few of the serialized variables that are stored here more efficient, too, since there are a lot of fireteams being transmitted around.

• FireteamRequiredTarget has been made more efficient to serialize, and on the CPU. It requires fewer lookups during runtime, and has the data it needs on demand.
  • In order to fully make this work for the AgainstSquad, that now uses a LazyLoadSquadWrapper, but for the factions and such that's already handled.
  • This in turn prevents us from having to cache some information on the Fireteam itself that was just kind of copies of this.
    • Those fields on Fireteam are now instead properties that wrapper the FireteamRequiredTarget. This will actually work properly on MP clients without having to have them run UpdateNonSerializedFields(), which was a problem up until now.

• Whew, okay, fireteams are really problematic. They have a lot of host-only stuff that gets fed into the UI, and it's really hard to tell the parts apart.
  • I've improved things quite a bit so far, but there's nothing there that will keep the problem from recurring at the moment (it's a case of easy to shoot yourself in the foot when adding functionality to fireteams, right now).
  • I'm currently checking in what I have on this front so far (with 904 errors left, incidentally), and then seeing what I can do to actually split this out so that not all of the Fireteams functionality exists in BaseInfo. There are parts that frankly just need to exist in DeepInfo so that we don't do this foot-shooting thing.
  • Right now it's mainly a matter of the MP clients are going to be incredibly spotty in what they show properly in their tooltips about fireteams; the actual gameplay of them acting properly is fine for MP and SP.

• The hunter and warden types now properly load again from the settings in the lobby. This was just one of those things I had not ported over.
  • Vassal status now also loads properly, and centrally for all factions in an optional way, which is nice.

• The starting fleets and battlestations and such for human empires now load properly again. This again was a matter of porting over the existing stuff.

• And finally, the last of the lobby fields, the trim colors for AI subfactions, are also now ported over.
  • There's a slight chance they may be a bit janky either in the lobby or after, but I think it will work this way, and it's more efficient. I have a couple of lines commented out that will fix it (but do more CPU work) if it's actually a problem.

• Fixed a few more things here and there, Devourer and some exogalactic bits.

• The Scourge faction has been mostly updated, except for the parts where the LRP thread does some things that would violate the normal double-buffering pattern. I'll deal with those when I am feeling more fresh.
  • For now, got rid of several hundred errors in the Scourge for compilation, and now there are 4 left. One of those is related to some fireteams changes I still need to make (I have a great idea on how to do it now, but I'd like everything else cleaned up first so I can see my work more easily).
  • Overall error count now down to 613. That's really it for me tonight, now.

• The Sappers collection on the Sappers faction is now DoubleBufferedConcurrentList. All the others were fine without the concurrency.
  • Oops, no, actually FloweredCrystals also needed that.
  • Okay, the sappers are fully compiling and ready to go. That's us down to 73 compiler errors.

• Hjarnum on Dark Zenith is now DoubleBufferedConcurrentList. It seems to be the only one there that needs that pattern!
  • Sigh. That said, it needs to be sorted, which is incompatible. So we also now have a SortedHjarnum that we build at the same time and sort. This is apparently only used in the UI, which is handy at least.
  • Almost all the errors are out of the Dark Zenith faction now, but there are a few things that I still need to look at in the morning when I am more fresh. I was working until 1:30am last night, and then back at it at about 8:30am this morning, so I'm not the most fresh.
  • That said, the total compiler error count is now... 6.
  • Tomorrow morning I'll need to sort out these last issues with the DZ, and then get the fireteams split I have planned done (it shouldn't be too bad, knock on wood), and then correct a couple of vis-dll errors that should take under 30 minutes, and then I'm on to actually... runtime testing!
  • This whole process has taken enormously longer than I ever expected, but the end is finally in sight. If I don't have at least a compiling version tomorrow, then something is very very wrong. Honestly if there's not one by lunch I'll be shocked.

Moving Into DeepInfo, Part 5

• Removed the ConcurrentDictionaryExtensions class, and merged that into the general DictionaryExtensions class.
  • Added a new DoFor() extension to ConcurrentDictionary via extension, the same as we had previously added that to Dictionary<>.

• Astro Trains had a few enough number of errors in its file -- 63 -- to actually have them shown properly in visual studio.
  • Oof, but its BaseInfo was not using double buffered lists, and the way that it was aggregating train guards was... fraught. So let's fix all that up, and also serialize less while we're in the process.
    • AstroTrainsPerTrainBaseInfo now has a pair of double buffered collections, which is the first time we've used that on per-unit data, but it's actually a great way to store things in the new framework. I haven't needed this before now, but it's a very useful form to use.
    • We also no longer have DeployedGuardsPerTrainSim on the BaseInfo. That's now being handled on the actual trains, rather than via an indirect lookup. This solves a lot of code complexity and chances for orphaned data. It also uses less CPU to calculate in the new format, though in the old format of the game it would have been debatable.
    • This actually has turned into a rather fascinating study of nested data. It's quick and easy to understand and a pattern to be emulated. This is actually kind of a great showpiece for the power of the new system.
    • This turned out to be SO useful that I've put a teaching moment in there.
    • Improved the description appenders for trains to be a bit more informative now that they are able to be.
  • Okay, back to the astro trains deepinfo file, how many errors do we have now? 71, okay.
    • Well, we can safely get rid of DeployedGuardsPerTrainLRP, StationsLRP, and TrainDepotsLRP.
    • TimeLastDepotSpawned was not consistently called, so it has been moved into SpawnNewTrainDepot. Same with AstroTrainsPerDepotBaseInfo being created and attached to it.
    • initializeDepotData has been removed and made part of the depot creation process.
  • The guards code was really confusing because things were not named very clearly. The guards on the train are now called StoredGuardsInsideThisTrain and DeployedGuardsOfThisTrain, so that it's clear that's how this works.
  • The way that guards are stored for finding lost guards, and the way that they attrition, has been moved into BaseInfo (and thus runs on the client as well as the host, which is good), and is also a lot simpler to follow now. Same general logic, but it won't miss anything by accident.
  • StoredGuardsInsideThisTrain was an ArcenSparseLookup, which was a cross-threading problem waiting to happen, but it can't use the double-buffered style collections because that's not what it is (it's actual stored data). So this is now a ConcurrentDictionary, and there's a teaching moment in there explaining why.

• Silly me, I thought that Astro Trains were going to be a very easy faction. But it turns out there was quite a lot going on there that was really really out of date for the codebase, so... here we are.
  • Down to 551 compiler errors left.

• Okay, let's see what's going on in the Fallen Spire deep infor class. I see 85 errors at the moment, mainly so low because I've been in here before quite a bit, but not with my full attention.
  • Most of that melted like butter in under five minutes. There was a funky thing with the journal logging that wasn't very clear, but that was something that the svn history let me clear up and then document pretty easily.
  • Transceiver needed to be converted into a DoubleBufferedValue. From the look of this class, this was before I added that specific item (in the distant memory of last week).
  • Oh... wow, what an error on my part. Coding while tired, I guess. This was also one of the earlier factions I started the conversion on, and then I had set it aside. All the BaseInfo Stage2 stuff was in DoFactionGeneralAggregationsPausedOrUnpaused instead, which runs WAY too frequently. Talk about a performance drain. Glad that code never released.
    • I also untangled the double-buffered clears and switches from the logic that fills them, which is what all the other classes I've done more recently were like. It's amazing how that takes this from unreadable to really nicely legible.
  • Okay, even so with all the oddness in BaseInfo, this gets me down to two errors remaining in the DeepInfo class... and they are the same sort of nastiness that I've seen in several classes when working with my nice new double-buffered lists. They just... aren't sufficient for a lot of the code that exists in actual DeepInfo classes. I'm particularly unhappy with the butchering that I did to Zenith Architrave; I think I may revert back out that entire class to before my last pass on it, and redo it more nicely. But before that, I need a double-buffered collection that supports concurrency; one that is used for spire cities and debris (thus solving my last two errors here), and which then is also used on Zenith Architrave (thus returning its logic to what it used to be). What I had expected to be an edge case actually turned out to be a really common case, so new data collection it is.
  • I'm down to 468 errors, but I'm going to pause on forward-progress for a bit, add my new data structure, and then undo all of the likely damage I did to Zenith Architrave.

• Added a new Engine_Universal.DEFAULT_CONCURRENCY_LEVEL, which is set to 4. This should be passed into every ConcurrentDictionary in the program as the default concurrency level (and then choose yourself what the starting capacity for that dictionary should be).
  • Why does this matter? https://blog.getpaint.net/2017/06/30/concurrentdictionary-allocates-a-lot/
  • Engine_Universal.DEFAULT_CONCURRENCY_LEVEL is the default level of concurrency that we are likely to need in AI War 2, based on my designs for threading. If more threads than this are accessing a ConcurrentDictionary at once, then performance will suffer slightly, but not much.
  • The default concurrency level is actually 4x the logical processor count, which is... insanely too much, and wastes RAM and performance when it comes to high-CPU computers. I only have 8 phyical cores with hyperthreading for 16 logical processors, but that means that it is creating (for me) literally 16x more RAM usage and locks than are needed. In the future, as computers advance, the design of this program is such that it will still use plenty of processors, but not on an individual dictionary basis.
  • So we'd start seeing MAJOR performance degredation and memory spikes if 128 logical processors were ever a commonplace thing, etc. Now we won't. Yay futureproofing. And honestly, it helps for performance NOW, too.

• Added a new DoubleBufferedConcurrentList. This is a lot like DoubleBufferedList, but less efficient in some ways, and more powerful.
  • This one also needs to support the display list taking additions and working as a working list for background threads. This tends to be a case where something says "oh hey, let me look at what is there, and also let me add this one," and then the rest of that background thread needs to have access to thing that was added RIGHT NOW.
    • For example, this sort of thing could happen on Stage3 or in LRP or both, who knows or cares. Either of those is still happening while the UI is using the same DisplayList as the others.
  • There's no way to handle that without using a concurrent list, and you might think that would be enough to avoid flicker in the UI, but it's not. The flicker would come when the construction list is torn down and then rebuilt, since that is often on a (different) background thread.
    • So we need to always have persistent lists, and support ad-hoc additions to the display one as we go, while also supporting the common cycle of construct-clear, construct, flip-to-display.
  • Sidebar: Internally, we are using a ConcurrentDictionary to power this. What... the heck?
    • Well, there is no "list" class in the .NET Concurrent collections, because the idea of an ordered list is antithetical to cross-thread ad-hoc addition and removal. They have ConcurrentBag and ConcurrentQueue and a few other things. ConcurrentQueue is vaguely ordered, but the main point is that it supports the foreach operator, and also supports direct removal and additions. Throughout the codebase, I have preferred ConcurrentQueue to ConcurrentBag because of profiling work that various other programmers have done, demonstrating that Microsoft's implementation of ConcurrentQueue is slightly more performant. That said, neither of those classes -- nor ConcurrentStack -- support the Contains operation. There isn't any equivalent of a ConcurrentHashSet, either. The only collection that really supports all of what we want to do is ConcurrentDictionary, which has some performance drawbacks, but not too severe to be worth it.
    • If you CAN use DoubleBufferedList<>, then of course use that instead. That thing flies. Start with DoubleBufferedList<>, and if you find you need to graduate to DoubleBufferedConcurrentList<>, then do so.
  • Implementation notes:
    • You will notice that there is no GetDisplayList() like you are used to seeing all over the place with DoubleBufferedList<>. Instead, you must use DorFor on it to do the equivalent of a foreach loop (which you can break out of as-needed).
    • There IS a Contains method, but there is a (very very) low chance that it will return false when the answer is really true.
      • Because of that, there is an int TimesToRecheckIfSeemsLikeNotThere = 0 parameter to this method. If you REALLY need an accurate Contains, you can set that to something larger than 0, but be aware that has a peformance impact that is equal to calling the Contains methods however many extra times you say.
    • Since this is a dictionary under the hood, you can't put the same item in the list more than once. Or rather, if you do, they will just collapse down to the unique set.
      • This is actually useful, since typically adding something repeatedly is an error anyhow. We specifically have some things like AddPlanetIfNotAlreadyInList(), but you do NOT have to do that. That's implicit in this structure. For goodness say, definitely do NOT call "if not contains X, then add X," because you just wasted some CPU cycles for nothing in that case. Just add it.
    • Since we are specifically wanting to be able to modify the display list on the fly (that's why this variant of DoubleBufferedList<> exists at all!), there is of course also a new AddToDisplayList() method.
    • As a consequence of this being a concurrent collection, you can forget about any sorting methods. The normal DoubleBufferedList<> supports sorting, but this version does not. You'll have to feed the results of a DoFor operation into some local working list and then sort that if you really must combine sorting with this sort of list. But in general, I haven't seen any cases where that would be relevant.
    • You may have noticed that none of the DoubleBuffered collections suport IEnumerable (so you can't do foreach on them). This class is no different. The reason is because that would be a highly ambiguous call. Someone might do a foreach while thinking they are doing list construction, or they might do it while thinking they are looking at the display results. Lack of clarity like this is very bad, because a second programmer following behind you can't even tell with ease if you made a mistake or are doing what you mean to. So instead we have the DoFor variants.

• Went ahead and added a Display_DoFor onto DoubleBufferedList<>, so that it has the same style of functionality as the concurrent one from that angle.

• Added a new Display_GetRandomItem() on both DoubleBufferedList<> (just to save code, and it's very efficient), and to DoubleBufferedConcurrentList<> (this is not very efficient at all, but it works; and in most cases of relevance in our codebase, these lists are so small that it's plenty efficient ENOUGH).
  • Normally you can't index into a dictionary in the way we would want to in order to get a random return value, but our code would often need to do this on the LRP in particular with DoubleBufferedConcurrentLists, so now it's possible.

• For Astro Trains, they are now the pilot for DoubleBufferedConcurrentList, with TrainDepots and TrainsEnRoute now using them.
  • This was definitely a major diversion off to the side, but this keeps the code for the end product of the game as close as possible to what it was in the past, and thus lowers our risk of new logic errors or just incomprehensible code. It's time for me to completely redo the travesty of what I visited upon the Zenith Architrave a few days ago, now that I have this tool.

• Reverted the Zenith Architrave to svn revision 14224. Ugh I hate undoing work, but that was really a hash. This brings me back up from 468 errors to now 539 errors.
  • My goal this time is to use the double buffered concurrent lists to accomplish keeping the logic of the ZA intact, rather than trying to rewrite it and hash it up like I did on Thursday night. I had noted even in my checkin notes on Thursday that it was a hash, which is never what you want to see in your checkin notes.

• Fixed all of the non-controversial errors with the Zenith Architrave, which took only 20 minutes by my reckoning (I'm getting much faster!). I had temporary working variables to quiet certain errors related to the central lookups that are now going to need to change.
  • I am first checking in this version that has all those changes in place before I start changing the bits that need the new collection types. With the temporary variables back out of there, I'm back up to 75 errors from this file, so that's quite a lot related to them.
  • Also worth noting is that I'm not messing with the LRP collections at all. This class is complicated enough that they may as well just keep doing what they do. I don't really want Badger to have to come in here and review this code like I had asked him to before, and at this point I don't see a reason why he should have to.
  • 543 errors in total for now.

• Okay, got the next batch of ZA fixes in place. All of them except the WarpingInSpawners and the Castra are able to use the normal DoubleBufferedList. That's easy enough.
  • If I were to extend this to the LRP variables, which I think I did before, then that would cause some need to use the concurrent version more. But that would be a net negative in terms of performance, so I'll just leave LRP doing its own thing, giving more room for the sim and UI to share work/benefits with one another.
  • At this point there are 17 remaining ZA errors, and those will be solved by me converting the Castra and WarpingInSpawners to the concurrent double buffered list.
  • This time around, the flow of logic is preserved exactly as Badger wrote it, and there's no longer a need for a code review of this faction.

• WarpingInSpawners and Castra are both now DoubleBufferedConcurrentLists.
  • WarpingInSpawners previously was sorting itself after the initial creation of its list, but that's no longer a possibility with the new collection format.
    • The logic that was relying on this sorted list was kind of complicated, so it was easiest just to implement a secondary SortedWarpingInSpawners that we feed from the concurent list and then sort like we did before.
  • That's a wrap on ZA. They are now error-free, and no longer in need of a code review any more than any other part of the codebase now is.
  • Back down to 468 errors.

• Clearing "two" errors on the fallen spire took 35 minutes and generated tons of other things that had to be converted. But nevertheless, the fallen spire is now error-free:
  • Spire Cities and Spire Debris are now using a DoubleBufferedConcurrentList.
    • However, because spire cities in particular ALSO need to be sorted half the time (sigh), there is now a SortedSpireCities that is built for mostly UI purposes.

• GetScourgeStateForDisplay now includes information on fortresses. Previously it just had armories and spawners.

• The three kinds of "warping in" collections on scourge now all use DoubleBufferedConcurrentList. This was about a 10 minute job for "three errors."
  • Other than one fireteam-related error which I am intentionally leaving to the last (it's a structural thing that I need to be able to see in isolation to fix), Scourge are now error-free.
  • Total error count now 463.

• Fixed right about 60 errors that were remaining in the migrant fleets faction code (22 were obscured from the compiler until I fixed the ForEach they were in.
  • This was jolly easy and pleasant, nice. 425 errors remaining.

• Fixed what was reportedly originally 38 antagonized dyson sphere errors, but wow that was kind of a mess in general. They're all fixed up now, and probably for the first time ever we're not mixing up data of the antagonizer and the dyson spheres themselves, which makes the whole thing a lot easier to follow. Previously they had kinda-sorta shared an external data object, but not fully, and that made it a bit confusing to untangle.
  • Anyhow, I happened to also notice that AntagonizedDysonDescriptionAppender is not actually used on dyson spheres at any point from what I can tell. Maybe this was deprecated on purpose, but it would be functional if someone wants to attach that to a dyson sphere that's being antagonized.
  • Error total is now 394. Other than the fireteam stuff that will be at the very last (and some vis-layer stuff that is incredibly minor), this is down to mostly the Dark Zenith and the Sappers. Oh, and apparently the Dyson Sphere proper has 63 errors, too, I now see.

• Well, that was again pleasantly straightforward: dyson sphere faction is now error free. New compiler error count is 332. Time for the last two big guys.

• Decided to make the safe changes first, and hit low-hanging fruit, in case I make another mistake again like I did with the Zenith Architrave.
  • Sappers are down from 119 errors to 45, and what remains to be done with the rest of them is a lot more clear.
  • Dark Zenith are down from 211 errors to 71. It looks like Sappers and Hjarnum will need to be concurrent, but none of the other lists, which is excellent. Additionally, a couple of methods need some adjustment based on working lists, but that's not too bad. All told, this faction was less intimidating than it had seemed, thanks to judicious search and replace. Zenith Architrave was the more challenging one structurally, even though Dark Zenith is more complicated in general.
  • Total compiler errors are now down to 117, which is... insane! I was happy just to get to three digits today, after being in the high four digits for weeks. Now we're almost down to two digits!

Hunting Runtime Errors, Part 1

• The last of the non-fireteam compiler errors are now fixed. Time to get the last fireteams bits fixed up and that will be that!

• The text-export portions of fireteams are being merged back into the main file of Fireteams.
  • A new FireteamHonoraryDeepInfo object has been created, which exists as an immutable sub-part of every fireteam. On MP clients, this sub-part is ignored and unfilled.
  • There are detailed teaching moments that explain the thought process behind this, and how to mimic it for other data if you ever need to (as a main developer or a modder).
  • PreferredSpeed has been moved to be a serialized field, so that it can be shown on the UI on clients and hosts.
  • Removed some string construction that was happening overly-frequently on fireteams even on the host/singleplayer cases. It now only happens if you're actually looking at the debug menu in question.
  • For the first time since it has existed, our DeepInfo dll (AIWarExternalDeepProcessingCode) actually compiles!

• The 23 errors from the external vis dll are also all now fixed.
  • Running our complete build scripts to verify that everything compiles... yep! We are officially on to runtime errors now. Time to start up the game for the first time in three weeks (time for that to be possible for the first time in three weeks, internally).

• Fixed a bug in GetIsDescendentOfInterface() that was causing false positives.

• Corrected GameEntityTypeData to not require BaseInfo or DeepInfo for units. Most of them will have neither, and so far no units have needed DeepInfo.

• Fixed a couple of xml typos that were preventing proper parsing.

• For the sake of clarity, IArcenExternalClassBase has been renamed to IArcenExternalClassRoot.
  • We don't want this to get confused with BaseInfo, because that's not what it is (but it was sort of what it looked like).
  • ArcenCachedExternalTypeBase is now ArcenCachedExternalTypeRoot for the same reason.'

• Verified that my logic is working for checking if BaseInfo and DeepInfo (true DeepInfo, not honorary) are living together in the same dll.
  • As I had kind of expected, it's tripping on my own code at the moment, which mostly means I have some things miscategorized.
  • This exception checking is here to prevent accidental mixing of this data that is supposed to be separate, including dlls referencing dlls that they should not have access to.
    • Essentially this sets up a boundary where it tells programmers (modders, later, but for the moment myself) "hey, you're either arranging this wrong or you've tagged things wrong, but either way fix it because bad things would happen in multiplayer if you do not."

• Now that I have verified that the above works, a whole bunch of debugging output that was related to that has been turned back off.

• Fixed all the map definitions still pointing to the wrong dll from their xml.
  • Ditto for the TargetEvaluators.
  • Ditto the AIDefensePlacers.
  • And DeathEffects.
  • And GuardpostAndCommandPlacers.

• Fixed an issue where the xml tag multiplier_to_all_fleet_caps="0.2" was still being used on a variety of outguard units, which I don't think had any relevance for that at all.
  • Also some sappers turrets, and a ZA turret, etc.

• Then there was an unexpected multiplier_to_all_fleet_caps="1.34" on:
  • Golden Wasp Drones, which made their cap 536 instead of 400. Let's make that 540 just to look tidier.
  • Queen Bee Drone, normal and Dark Zenith versions, ditto.

• Milestone: I can get to the main menu without any errors at all! It took just over 1 hour from the point of being able to compile at all to get to that point. Now I'm going to break for lunch, but it will be interesting to see how long it takes to get the rest working. It's nice to see the main menu again!

• Fixed an issue with the Source not being properly linked to each of the four PrototypeObjectForGlobalLobbyClearingOnly objects for the new externaldata, which led to an immediate exception when trying to start a game.

• Fixed an one-line code error that broke the new sub-faction linkages for the AI. They needed to be after a loop, not inside it, and it was one curly brace too early.

• Fixed another one-line error that mixed up relentless wave linkages from DeepInfo to BaseInfo.

• Fixed an exception where Outguard mistakenly thought they were supposed to have an Intensity (they are not).

• Fixed a minor issue with order of operations with the lobby partial-map-generations not linking up parentages properly.
  • This was technically working by design, but this was from a few weeks ago before I knew the shape of things to come. It needed to be changed a bit in light of the final design.

• Fixed another issue with Allegieances, where every faction was expected to have one (but many invisible ones don't, in particular).

• Milestone: I can now open the lobby without errors. I took a long lunch and ran an errand, so this was about 1 hour of work since I got things working to the main menu.

• Fixed up how the faction external data was linked to its "attached object" (faction or squad).
  • Previously it did it way too late, and therefore some errors were resulting.

• Started tracking the way that faction external data is linked to its attached faction, because I am getting some strange nullref exceptions.
  • By assigning cross-thread-safe unique IDs, I can see that the version of the faction data that we have in the lobby for use is very different from the one that previously had itself assigned. Curious!
  • Stuff like this is central and breaks every faction, and is just part of the framework itself being tested out for the first time. This is the sort of thing I was hoping to see, and not a thousand little per-faction errors. I may yet see a tide of those later, but at any rate this current part is just a normal part of unit testing after coding for three weeks without being able to test any of it.
  • At this point, I can see that we have a minor memory leak of factions again, without having to even look the memory dumps and diff them (which I will do, later, when all of this is working better). Nice to find this out sooner than later.

• BaseInfo and DeepInfo on factions and squads are now carefully-managed and sent back to their pools more aggressively.
  • ...That did nothing, unfortunately, to help my actual problem. It does make the code easier to read, and it's hard to say beyond that.

• Fixed things up so that BaseInfo and DeepInfo actually get put back into their pools when they are supposed to.
  • This helps a little bit, but there's still a central leak with my pooling framework, which is a surprising thing to learn at this point. I thought I had established that as solid a while back.
  • Then again, back when I was establishing it as solid, I had noticed that it would increase to about twice what I expected, and then stop. So it was using extra memory, but not infinite endless memory. This may be related to what I'm seeing now, which I am going to fix because it causes problems for me on several levels.

• Added a new GetItemsWaitingInPoolRightNow() and GetItemsInTheForeverList() onto TimeBasedList, to help me see what's going on as the factions are pulling external data out of their pools (right now they create too many objects, and I want to know why; after that I'll still have to deal with the problem of improper item teardown, but one thing at at a time).

• Discovered that a variety of pools were not using their original entity (the "creator entity") for actual data storage. For the larger objects, where there are just one or two per type, this is very wasteful. Read: factions.
  • In the past, I've always just really cared about things like entity orders, or squads, or whatever else -- and the difference of literally one object is super stupidly negligible. For a faction, it's an extra 100% usage of RAM, by comparison.
  • These pool types now include the creator entity as a normal object to use, just to remove this confusion and slight waste (it can still do its job as an object factory no matter what else is happening): TimeBasedPool<>, BetweenMapGenPool<>, and ConcurrentPool<>. I think that's all the major ones of relevance.

• Immediately after that, started having some exceptions in Recalculate_PreSystemRecalculate() on initial game load (before getting to the main menu), which is curious but not completely out of character with some OLD issues I've seen off and on sine 2018. Let's get that fixed.
  • For now, Recalculate_PreSystemRecalculate is being instrumented.
  • Based on this, fixed an issue with AIWar2GalaxySettingTable.GetFauxFIntValueFromSettingByName_DefaultOne_DuringGame() and all similar methods where it would give an error if those were called prior to the game being fully initialized.
  • Based on this, also fixed an issue with GetResourceProductionBeforeAnyBonuses() where it would also error if called prior to the world being initialized.
  • Same with GetFrameSizeMultiplierAsDoubleNonSim().
  • Did a further bunch of hardening of a bunch of other places, several dozen; mostly windows.
  • Worth noting that none of this code was changed during this refactor. This is an old issue that is only triggered by certain rare circumstances that -- for whatever reason -- become common during this pooling shift.
    • On the whole, this is not the sort of issue I was expecting to chase today at all (something unrelated to the work of the last three weeks), but there's always the chance that this has been secretly breaking for a rare subset of players, so let's make sure it works.

• I think that the world is actually being set to null for some reason, possibly because of the world itself being pooled in a funky way...
  • So let's fix THAT: The three pool types noted above all now have a CreatorEntryIsAddedToThePool parameter. Now it's up to the calling code to decide how to handle it.
  • True on everything except: PlayerAccount, World (the first entry in that is a "dummy world!"), ConfigurationForFaction, EntitySystem, SpeedGroup, PlayerAccount_AIW2, and World_AIW2.
  • The game now loads just fine to the main menu again.
  • Also, the lobby faction tab no longer spams errors about a missing faction link! So I DID solve the problem I was originally chasing, but just in a really roundabout way.
    • This made me suspect that entering the lobby multiple times in a row, or otherwise serially regenerating map types would be a problem. But nope, those work great also.
    • Adding multiple AI factions works great, and adding extra human factions also works.
    • However, after adding two extra AI factions and then another human faction, I did get an error where the AI subfactions were not linked to their parent AI faction in the case of all three AIs. So I'll have to look into that, after I can duplicate it more.
    • Also, reading last-lobby-settings seems to be broken at the moment, but that's fine. Just adding that to my list.
    • Starting a new game from the lobby is also broken, but this is probably unrelated to anything I've done thus far; this is the first time I've tried clicking that button today, and the place it errors looks like a sensible place for a one-line typo.
  • So, I'm going to call this current line of investigation complete, and move on to the new list of errors that I have accrued to look into. (As kind of an aside, I was fully expecting to have serialization be broken in the new version, as that's really hard to do completely correctly en-masse. I built in tools a couple of years ago that makes it trivial to find those errors, generally, but it's not my top priority right this second).

• Milestone: I can click around the lobby, add factions and such, with very few errors (though not zero yet).

• The game now does a handy dump of all the faction basics when it runs into parentage issues.
  • See, back when I was originally designing the parentage stuff a few weeks ago, I had not wanted the subfactions clogging up the lobby for very good reason -- it's the reason I'm having some exceptions now, in fact; old stale sub-factions get left around. Having the subfactions earlier than later is actually better overall, I just need to have them clean themselves up between mapgens now, basically.

• Using the above, the game now properly removes and adds all the various factions and subfactions. No more errors on that front, or mis-parenting of factions, or duplicate factions, using any routes that I can tell.
  • I do see there is still an issue with null attachedfactions sometimes on factions added via the lobby, but hitting the map regeneration button fixes it. Still needs to be fixed to not happen at all.

• Made mapgen a bit quieter in the log when it's not having problems.

• Fixed an exception that would happen when it tried to add nomad planets if you did not have the nomad faction enabled.

• Fixed a minor issue where the game had a default of "Normal" for the AI Layout field, when that was not an option. The default is now Small Clusters, as intended, and "Normal" is also taken as meaning "Small Clusters" just to be on the safe side.

• Fixed another exception that would happen in the escape menu when trying to describe factions. It was being too strict and expecting all factions to have a NumberToSeed, which of course they do not.

• Updated the version number to 3.700, so that the game can actually TRY to load saves it makes (it won't try for anything lower than 3.7).

• Milestone: Game is nominally playable. Currently in single-player, you are not properly assigned to the first human faction, but you can start the game anyway. Once you are in, all the menus seem to work and you can assign yourself to that faction. Ships obey orders, fighting works, factions seem to be doing what they are supposed to, and there are no errors so far. Bear in mind this is just with the human and the AI and the auto-added factions, not with anything from DLCs yet, but that's still pretty awesome.

• Fixed an issue with "Super Easy Mode" on the Fallen Spire (debug mode) not being a proper bool. I had converted all the others, but this one was erroring because I missed that part of the xml.

• Fixed a couple of serialization issues with savegames, but I have a few more to fix still. Doesn't look like anything super major, knock on wood.
  • There is less data being sent now, partially because I am reading things from the Config for factions directly rather than sending/storing that multiple times, and that now fully works.
  • From the look of things, the "last stored settings" for the lobby also work again. Have not tested quickstarts yet.

• One of the benefits of breaking all the savegames is that I can change the order of things around to be more efficient/clear with how we load things. Previously I used that in order to put all of the ExternalData at the very end of the savegame, which makes deserialization a lot easier there.
  • Now I am switching fleets and factions, so that factions are deserialized first. In order to do some more-efficient conditional deserialization of fleets (sending less data based on the type of faction in question), we need to have the factions loaded first. Now we do!
  • As part of this, we no longer have to have a loadingFactionIndex on fleets. We can just directly load the faction.

• Similarly, I have now split it so that the planets and the galaxy are now loaded before fleets. However, the PlanetFactions (and all the units inside them) are not loaded until AFTER fleets.
  • This split makes even more things easier to load, which is really nice. When I originally designed the order of these things back in 2018 or so, the full scope of what their needs would be was not remotely clear.
  • We now no longer have to do the loadingPlanetIndex on fleets, and we can also set the PlanetFaction immediately, too.
  • PlanetFaction PFaction on fleets has been removed, as it was not actually ever directly needed and it was slow to calculate.

• On fleets, we no longer have a Centerpiece_DeserID. Also, Centerpiece is no longer a GameEntity_Squad -- it's now a LazyLoadSquadWrapper.

• Thanks to the changes above, we no longer need FinishPostFactionAndPlanetDeserializationLinkages() to run after fleets are deserialized. This is good for speed in general of loading savegames, but also for multiplayer CPU processing on the clients as they get fleet data.

• Because of the order of operations change (factions loading before fleets), I can no longer serialize the LooseFleet ID for a faction directly on the faction itself -- the fleets don't exist yet.
  • I _could_ go in and make some indiretion (store the fleetID, then later link it), but there is a much better solution.
  • We now just store a serialized IsLooseFleetForFaction bool on the fleet itself, and since it loads after the faction, it can just set itself as the LooseFleet with ease.
    • Fun fact: this takes one bit of data, rather than 32 bits. Also it's faster to do, because we don't need to do a lookup at any point; we already have a reference to the faction object.

• Necromancer data on the fleets and factions now only serializes for the necromancer faction.
  • Similarly, a bunch of player-specific faction data only serializes for players in general.

• PlanetFactionBooleanFlags are also now only sent for player planetfactions.

• In general cleaned up a few fields in serialization code, and put in a few more markers for telling where we are in the logs. These don't affect non-logged transmissions and saves.

• Fixed up some more code in the Templars, just correcting a few things that I mised and thus were not quite up to the latest standard.
  • There's still an exception in MarkUpCastles, but I'll deal with that tomorrow.

• There's also still at least a few issues with deserialization of savegames and network data, but it's getting closer to being done, and is more efficient, and is easier code to read. So that's progress!

Hunting Runtime Errors, Part 2

• Fixed up most of the core problems with savegame/network serialization that were just a consequence of me changing the up the format so much. I also... changed up the format even more, a bit. Just adding bits of extra efficiency on things that are sent more frequently. As one random example, for instance, we now use 38 bits to send the offset-coordinates of each step in a unit's guard-path rather than using 64 bits to do the same.
  • There's still something up with the serialization, but now it's to the point where it is at the external info and it's hitting an infinite loop on load. That shouldn't take too long to track down. From the serialization logs, it looks like most of the data that I would expect to be in there is actually being saved, which is nice. Right now the deserilization log itself cuts off in the middle of writing, however, which I can tell means there's a buffer that hasn't fully been flushed by the time it hits the place where the infinite loop is. So I'm going to work on making that buffer flush faster, which will take me right to the problem code.
  • At the moment, these are still the sort of structural problems that you fix-once-and-then-not-again, so I'm happy about that so far.

• The serialization logger now flushes its buffers to disk much more rapidly, so that if there is a problem that interrupts the program or throws it into a loop, the output isn't behind.

• Fixed two silly lines of code that were accidentally infinite-recursive, and caused deserialization of faction external data to not work. It didn't actually take that long to find, but there have been a lot of other secondary distractions and things today.

• Fixed the last of the deserialization errors that were new structural problems introduced in The Great Refactor.

• Milestone: The game can now save savegames, and then load them.
  • Reminder 1: older savegames from prior to The Great Refactor will not work in the new format. Trust me, it is absolutely worth it for all the improvements throwing out that compatibility has lent us. Much as I hate losing access to all those older saves.
  • Reminder 2: until we hit version 4.0, hopefully within a few weeks, I still reserve the right to continually break NEW savegames as well. There are some structural things with the Fallen Spire in particular that I need to redo, and that was too big of a thing to tack on DURING The Great Refactor. So those will be coming right on the tail of it.

• The debug setting "Write Deserialization Data Sizes" is no longer considered an advanced field.
  • When this option is on (it was added a little after DLC1 came out), it gives a detailed breakdown in WorldDeserializationDataSizes.txt of how much room each part of a savegame takes up, and what kind of data it is.
  • This has been adjusted today a fair bit to make sure it really captures things properly. Looking at the data, it's looking good! I see a few places where I can wring out a bit more efficiency, but honestly that was something that already happened a lot during The Great Refactor, and it shows.

• Discovered that I had put in the sort of funky working-list stuff in Templars for Rifts, not just in Zenith Architrave for their stuff.
  • Fixed up Templars to instead use the much-nicer and also much-less-likely-to-error DoubleBufferedConcurrentList.

• Fixed an exception in MarkUpCastles for Templars, where essentially the data for the castles was not attached onto them at the time of creation, but instead was randomly attached to them during this method.
  • There will likely be other places like this in the code, from all of us different programmers, and you can handle it in one of two different ways in any given case:
    • Option A: When you first create some entity (squad) that will later need some BaseInfo or DeepInfo of a certain point... just populate it then!
      • That's what the game mostly assumes you will do, but you also don't have to.
      • It's possible you might have some units that only need extra data SOME of the time, and in that case, that's a great reason to wait and not add it now, for instance.
    • Option B: When you need to do some sort of operation on an entity (squad), then just call CreateExternalBaseInfo<YourType>( "YourType" ), rather than calling GetExternalBaseInfoAs<YourType>() or TryGetExternalBaseInfoAs<YourType>().
      • This is also an acceptable choice, and it has the benefit of working really well with old savegames (which we have zero of at the moment, but in the future if you're trying to deal with code additions to things that now need data that didn't have it before, then this is the way to do it).
      • This approach does have the downside, though, that the Create() method has a bit more overhead than the Get() or TryGet() methods (Get and TryGet are equal in overhead, but the former errors when data is missing and the latter does not). The extra overhead of the Create() call is really just a matter of a table lookup, which means a dictionary call, and then one additional string comparison check. So it's not exactly massively slow or something.

• Fixed a longstanding minor bug in how battles were calculated for the galaxy map, with them overestimating the scale of your allies. This is for the little ripple thingies on planets.
  • This was found while fixing a really pointless warning message that could be sent to the log now when loading savegames post-Refactor.

• Now that my pooling works really well and extends fully into the externaldata realm, there are inevitably going to be some things that require some adjustment just because of having been missed and not previously being expected to pool.
  • One of those was pathfinders, which would throw exceptions after the second time you loaded a world. That's now fixed, and at the moment I'm not seeing anything else that has this sort of issue (Pathfinders were always kind of an odd case, like EntitySystems are an odd case, because those two categories of data retain attached to their parent object but still pool).

• When you are clicking the randomize button on the map screen, on solo play it now has a delay of only 0.2 seconds rather than 1 second. At the host in MP, only 1 second rather than 3. As a client, only 1.5 seconds instead of 5.
  • The idea is to prevent a bunch of regeneration requests from queuing up, which used to be a problem, but this all runs faster now and it was just annoyingly slow to let you click more, before.

• Fixed an issue (not sure how long this has been a thing) where when you hit reset to defaults in the lobby, it was not actually resetting the map tab items.
  • Specifically, it would reset the map type, but not any sub-options for the map type, and not the number of planets. This probably is new as of the current beta series, but not The Great Refactor, if I had to guess.

• Removed ControlledByPlayerAccounts_DuringGame from on the Faction class.
  • This was duplicative of something in the ConfigurationForFaction class, so we could already get that information a better way.

• Also removed/hid ControlledByPlayerAccounts_OnlyUseInLobby on the ConfigurationForFaction class, instead making this a private field that is only something that can be touched via properties and methods on its class:
  • bool IsFactionControlledByAnyPlayer tells you that.
  • bool IsFactionControlledByLocalPlayer tells you THAT.
  • int CountOfPlayersFactionControllingFaction gives you that answer.
  • byte GetPKIDOfControllingPlayerAtIndex( int Index ) has the expected answers.
  • PlayerAccount GetAccountOfControllingPlayerAtIndex( int Index ) again tells you the expected bit.
  • int GetPKIDOfFirstControllingPlayer() is predictable also.
  • PlayerAccount GetAccountOfFirstControllingPlayerOrNull(), ditto.
  • bool GetDoesPlayerControlThisFaction( byte PlayerPrimaryKeyID ) answers as you'd figure.
  • bool GetDoesPlayerControlThisFaction( PlayerAccount Player ) does too.
  • void RemovePlayerAccountFromControllingThisFaction( byte PlayerPrimaryKeyID ) is clear.
  • void ClearAllPlayerAccountsFromControllingThisFaction() is also.
  • void SetPlayerAccountInControlOfThisFaction( byte PlayerPrimaryKeyID ) is the last one, which does as you would expect.
    • Crucially, when things don't work as we expect, we can now insturment any and all of these and figure out WHY.
  • string GetNameOfPlayerOwnersOrEmptyString() is the actual last one, since the Faction class can't really do this anymore with much ease.
  • void SetPlayerAccountsToViewStartingPlanetAfterMapgen() is another "actual last" one, this time pulling in from mapgen.

• Okay... I just cannot get the faction ownership thing to work fully correctly. We have a lobby version and a long-term version, and the long-term version is right but the lobby version is wrong even after I set it to be right. I... my brain hurts. It's been hours staring at this.
  • I think that a further simplification to the lobby config settings is incoming. I will have to delay and do a release tomorrow. In the meantime I have further broken tutorials, but those will need some slight structural adjustments in general.
  • The only other really big bug that concerns me before a release is the factions external data not always having attached factions properly, but I think that can also be solved by simplifying the central configuration. This will need to be tomorrow, as I am likely to make matters worse rather than better if I try to fix it tonight.

Hunting Runtime Errors, Part 3

• Okay! After mulling this a lot last night, I have figured out that I hate the current design for our lobby configuration. It's my design, so I can say that. ;)
  • This is a design from 2018 or something along those lines, and it was in response to severe problems that I had when working with the lobby in AI War Classic in 2008-2009. It DOES solve the problem, but it created new ones. It's not the simplest-possible solution.
  • So here's the problem, and let's see if you think of the new solution before I tell you what it is. When framed like this, it's deceptively simple, but remember I'm framing it for you in a way that points toward the answer. Don't think too poorly of my past self, heh.
    • 1. Map generation takes time. How much time varies by the map and the computer, but it's a lot longer than a frame. It's something like 200ms at best, and 1.5 seconds on the super horrible end of things.
    • 2. During map generation, we need to have a copy of all the settings you chose from the lobby prior to that generation call, and those settings need to not change. So for instance, how many planets you asked for, any other map-related bits, anything do with factions that affects the map, etc.
    • 3. We obviously want to run map generation on a background worker thread, because we're not going have the game seem to freeze for 200ms - 1.5s while you wait for mapgen. For one that's just really annoying and bad form, but it also would halt all network communications during that time.
    • 4. Given that, players will then be trying to mess with the lobby settings, potentially, while the map is generating. 200ms is not enough time to do much, but 1.5s is.
      • One solution to this is to temporarily make all of the lobby controls read-only, which was my solution in AI War Classic, but that's really messy and requires a lot of extra code (a bit of code for every stinking field). It's very annoying.
        • Another limitation to THAT solution is that the client doesn't know that the host is in the process of map generation when using that solution, and so we can't lock all their controls. In AI War Classic, this is why most of the controls for map generation could not be edited on clients.
      • The second solution to this is to just have two copies of the data. There's one copy for display, which mapgen doesn't use. It's what players see, and they can edit it from anywhere at any time. When it's time to submit it to mapgen, it makes a copy of itself and we're off to the races.
        • This is the solution that AI War 2 has been using since about 2018, but the problem is -- well, now we've got two copies of the data. It's possible for them to get out of sync, which is what has been frustrating me for several hours yesterday.
        • The good news is that once you solve that problem once, it's solved until the next time you make a structural change to the entire program, but it's complicated enough that it will deter you from making structural changes to the program because of what a huge pain this is.
  • All right, so what are some potential solutions that are better than the above? This is what I've been thinking about since last night.
    • Firstly, rather than having a lobby-copy and a permanent-copy (which is really what I've been doing all this time in AI War 2), maybe the sole copy of the config just makes a copy of itself that gets passed around in mapgen.
      • I don't like this, though, for a couple of reasons. One, it's entirely possible for a mapgen programmer to still go reach the central data by accident, and thus have a threading error or worse. Two, it requires adjusting how all of the mapgen code passes data around. And three, it is a minor pooling nightmare, since we would want to make sure that all of this gets pooled properly, even when mapgen fails or exits early, etc. There are "valid" mapgen failure cases, such as "there's not enough room on this size of map for all the factions you chose," for instance.
    • Okay, when that's out... hey, you know what? Our mapgen code really is more like 200ms to 400ms at the moment. How much clicking CAN a player do during that time? One or two at most, generally.
      • This is the solution I'm going with: the UI is still responsive and listens to players during mapgen, but no GameCommands get processed during mapgen. They all get processed right after, instead. Problem solved, and a single copy of configuration now.
      • Background: when players click around and make changes in the lobby, they don't directly change data. Instead, all of these things are generating GameCommands, which are what then actually change the data. There's normally a 100ms lag on this, because networking framework and sim cycle.
      • The leap from a 100ms lag to 200-1500ms of lag in the event that a new map is actively generating is really not far at all. And realistically it's typically more like 200-400ms, as noted. This lets players continue to have a responsive UI, and I expect in almost all cases they won't even notice the difference at all. It keeps threading issues away from mapgen, lets mapgen continue to work as before, and removes the complexity of having two copies of data.

• Milestone able to properly start a game, and reset a game, with all of the proper settings and things working as expected. Editing faction and galaxy option data in the lobby and in-game both work as expected.
  • New problems: not put in charge of any faction after loading savegame; savegames from yesterday broken; quickstarts broken.
  • Reminder to self: tutorials still broken, check to see if there are still issues with random faction external datas in lobby not being properly attached to a faction (this is unknown, it might have been fixed by the work from this morning).
  • Observation: the lobby feels more responsive to me, but it's hard to be sure if that's just me imagining it.

• Fixed a single line in the deserialization code that was causing quickstarts to not load properly.
  • They still don't load fully properly after they initialize, but the data I need gets off disk now, and so the only thing left with them is making sure their faction boot up properly. The fact that the data on disk is good for actually getting that started now is the important thing.

• Milestone: Quickstarts once again fully work, including loading them into the lobby or starting them directly. They properly strip out and recreate the bits that need that sort of thing, and all seems to be well from what I can tell.

• Fixed a bug where savegames did not remember who was in control of each faction on load.

• Fixed a load exception for the astro trains faction, and another for tamed macrophage. Just one-liner translation type issues.
  • Got another one for the dark spire fixed.

• Fixed an issue that made it so that when you added a new faction via the lobby, it did not actually fully generate things such that they had their needed attached info.

• Fixed another one-liner that was the vassal flag for units being read wrong.

• Fixed another one-liner that made it so that the second time astro trains were refreshed on a game load, they'd freak out.

• WorkThreadOnly_StartNewGame and WorkThreadOnly_GenerateNewGame and DoFinalBitsAfterMapGenerationOrFixingSavegame are all now instrumented to report errors better.

• Discovered that Random Factions are broken at the moment, but that's okay; I will get that fixed after this build.

• Fixed an error in Dark Spire where HandleTerminiiAndEpistyles assumed that DarkZenithPerUnitBaseInfo would be initialized, but it was not always (this sort of thing was me being hopeful, but my fallback was explained yesterday).

• Fixed a pair of errors in the enraged and tamed macrophage trying to update their allegiances like the regular macrophage.

• Fixed a one-line bug (this was not new, I just happened to find it) where Neinzul Sappers would not properly get their intensity if you put them on a minor faction team.

• It looks like random factions actually DO work, after a fashion, but are invisible in the lobby for some reason. So more will need to go into looking at that.

• Added the following new option to the debug settings in personal settings, in Debug, Detailed Logs: Log Fleet Setups For Custom ARS Types
  • When this is on, during startup it will log to the debug log any information about AddedToFleet_MinorFactionStrikecraft and AddedToFleet_MinorFactionFrigate. If you are getting an error with some custom faction structure that lets you hack it like an ARS, this will let you see what is wrong with your xml.
  • As you may be able to guess, there's a data problem here at the moment. It looks related to the dark spire, and I have no idea if this is new, or if this is an old problem that actually just gets reported now. But it sure spams the game once it starts, at the moment.

• Added a new IsSvikari onto the dark zenith core BaseInfo. This is used to let the Svikari not try to find the SpawningOptions field.

• I had corrected the internal name of the Dark Spire to be DarkSpire instead of "Dark Spire", but I had not fully corrected that throughout the rest of the xml. That was leading to the flood of exceptions, which I could now easily find thanks to the new debug setting that I put in. Those are all now fixed.
  • Additionally, I'm not sure if I made the same fix to Fallen Spire or not, but either way the random factions had a wrong value for them, and I corrected that also now, too.

• Discovered that the Zenith Architrave also do not always proactively initialize their units' external data. Fixed so that it works around that.
  • Same for macrophage teliums. Same fix applied. I suspect there will be a variety of this sort of thing showing up for factions this week.
  • And some more for the Dark Zenith.
  • The old structure for the game kind of encouraged lazy-loading of externaldata, so... here we are. We can either fix them to proactively attach said data (that would be nice, but has not too much point), or we can set it up to lazy-load as-needed here, too (which is what I'm doing now).
    • Despite the pejorative-sounding name, there are sometimes benefits to lazy loading (sometimes EXTREME benefits), so this isn't really a criticism of anyone. It's just an observational about differences in architectural assumptions.

• Fixed a minor linkage issue between the territory dispute subfactions and their parent. One-line of code type thing (most of these are one line of code fixes).

• Fixed a variety of conversions in the Dark Zenith resources converter table that were using the dark-zenith-specific variant rather than the DarkZenithFactionBaseInfoRoot that lives underneath both the dark zenith and the Svikari. This led to exceptions on the Svikari. Easily fixed with some search and replacing.

• Fixed a really silly issue with purging dead fireteam units, where I meant to check IF they were scourge, not assume they HAD to be scourge.

• Milestone: with over 20 factions added, I can now run the game for over twenty minutes without any errors popping up. They are fighting, and doing what they do, and etc. All of the errors thus far have been of the sort of one-line translation things that are fixed in about 10 seconds. Writing the change log note takes as long as finding the error in the code and fixing it combined.
  • Factions tested so far: AI and all of the auto-added factions, human empire, AI risk analyzers, astro trains, dark spire, dark zenith, dark zenith svikari (multiple at once), devourer golem (multiple at once), macrophage (multiple at once), marauders, nanocaust, nomad planets, territory dispute, zenith architrave (multiple), zenith miners, and zenith trader.
  • Not yet tested in that way: necromancer, fallen spire, HRF, migrant fleets, sappers, scourge, zenith dyson sphere.

• SphereNoLongerAntagonized() has been instrumented for better error reporting.
  • Fixed what seems to have been two translation errors on my part (the worst ones I've seen yet, but they were still quick to fix, knock on wood) where all dyson spheres were being treated as if they were antagonized, and constantly trying to revert to normal.

• Milestone: I have now run with all the other factions that I mentioned before, PLUS the necromancer, the fallen spire, the HRF, migrant fleets, sappers, scourge, and zenith dyson sphere. None of the others had any errors. That is every faction in the base game and all of the DLCs, even the unreleased one.
  • Please do note I didn't actually start the spire quests or anything like that, so there may be errors once we get into that more. It's hard to do a quick test of fallen spire. But at any rate, the factions in general there all lied for 15 minutes without any errors now.

• The faction utility methods to find either a human king or AI king are a lot more explicit now, and will only throw errors if you say they should. Additionally, if you ask them to throw an error, that error will be very visible. This eliminates any chance of invisible log spam that slows down your game, while keeping the flexibility to error if we really need to (we pretty much never need to in the game itself, but some mod might).

Mod Updates

• In general, expect all code mods to be broken for a bit. All xml mods should mostly be fine, but it depends on the mod.

• AMU:
  • More functions for the ManagedFleetHandler:
    • It now has a list of tasks to do, with their own on-failure and on-flagship-death finisher actions, and settings such as whether flagships are carriers, unload ships to attack enemies, etc, etc...
    • The OnSuccess/Failure/FlagshipDeath extra logic virtual functions have been renamed to OnEvery[something], and fire on every [something].
    • Additionally there now is a function for every entity removal that is triggered by the ManagedFleetHandler.
    • Tasks will now be archived and can be looped for perpetually repeating orders with the matching OnSuccess or OnSuccess_Fallback action.
    • The ManagedFleetHandler now has default settings for all tasks, so to reduce the hassle of setting up new tasks. This comes with a bunch of variants on the constructor to set up tasks easier.
  • Fixed one more reference to The Neinzul Abyss being DLC 4 instead of 3 in AMU.
  • The AMU_BattleshipBase entity type and all entities that inherit from it now use large health bars (visual change only).
• Devourer Chrysalis:
  • Updated to work with the above changes.
  • This hopefully fixes all remaining cases where the Devourer Hunters warp out but don't refund their strength as budget.
• NEW MOD! More Frigates:
  • Adds several starships from AI War Classic to 2 in frigate form.
  • Adds two new starting fleets composed solely of these frigates.
• NEW MOD! Super Saiyan Overlord:
  • This mod replaces the default overlord with a new one that is equipped with new weapons and receives a massive stat buff.
  • The Overlord Phase 2 also now sends raiding fleets against the player as long as it is engaged in combat with the player's units.
  • This mod is meant to be the ultimate test of your skills.
• More Starting Fleets:
  • Moved several fleets into their respective DLCs in order to allow players to use the mod even without said DLC (they will be missing the fleets that uses ships from the DLCs).
  • Reworked/Buffed/Nerfed several fleets and removed some that were too similar to other fleets.
• More Starting Fleets Plus:
  • Due to lack of demand and technical support, said mod will be disabled and sunsetted.
  • May return at a later date but currently is too similar to More Starting Fleets without bringing anything unique to the table.

Prior Release Notes

AI War 2: Finalizing Multiplayer