Rework startup/upgrade/bootstrap process to prune modules before core upgrade

[labkey-adam]

Rationale

Adding SNPRC modules to the TeamCity PostgreSQL distributions resulted in memory leaks since their SND dependency is not yet present.

Currently, modules with missing dependencies are removed after the core module is upgraded. The removed modules show up as leaked, since several code paths end up holding onto them. For example, core bootstrap() creates key containers (/, /home, /shared), which causes us to enumerate the active modules, which loads file-based webparts, which creates file watcher listeners that hold onto their owning modules. While webparts are the culprit here, it's likely that other resources could cause similar leaks.

My solution is to rework the startup/upgrade/bootstrap process to prune modules with missing dependencies before core upgrade occurs. So now we prune for database compatibility and then immediately prune for module dependencies. As a result, core upgrade and init() implementations that enumerate active modules work on the pruned list. A side benefit is that we log messages about all pruned modules (regardless of reason) together. Module initialization still happens after core upgrade, and any module that fails initialization still gets pruned along with its dependents. A module leak will likely happen in this case, but I think it's rare for init() to throw, plus it's always treated as a module startup error.

I considered initializing all modules before core upgrade, which is appealing in theory. However, that would have meant any operation that touches the database (role registration, property retrieval & update, etc.) would need to move from init() into startup() since core schemas could be empty or in a bad state when init() is invoked. This would have changed our rules and forced updates to 25+ modules that we manage (plus potentially others that we don't).

Related Pull Requests

• Allow registering multiple InvoicedItemsProcessingService impls wnprc-modules#917

[labkey-adam]

With the exception of the TNPRC_EHR postgres suite, which we expect to have failures, the EHR suites on this branch are now running green, or nearly so: https://teamcity.labkey.org/project/LabKey_Trunk_Premium_Ehr?branch=pg_support&mode=builds

[labkey-martyp]

I did not test this out but looks ok. Just a couple questions.

[labkey-martyp]

Do we want to keep the behavior that dev mode won't treat a ModuleDependencyException as an error? Looks like that was the logic previously.

[labkey-adam]

Sure, I'm back to separate removeModule() calls due to the differing verbs, so it was easy to restore that behavior.

[labkey-martyp]

Does the message need to be updated since we're not even trying to initialize the dependency module in some cases?

[labkey-adam]

Sure, I now pass in the verb to use. In most cases, it's "load" (because we don't know why the module didn't load), but in the initialize() loop, it's "initialize" because we know that's the only possibility.

I also pass in a verb to removeModule() to clarify what we attempted ("load" or "initialize").