NUTCH-3154 Implement integration testing framework for Nutch IndexWriter plugins using Testcontainers

[lewismc]

This is a proof of concept for NUTCH-3154.
I really like this model as the @Testcontainers(disabledWithoutDocker = true) annotation will simply skip the test(s) if no local container environment is present. More info at https://java.testcontainers.org/quickstart/junit_5_quickstart/#4-additional-attributes

The new target is test-indexer-integration. This will be triggered in CI if changes to src/plugin/indexer-*/** are detected.

My proposal is as follows

indexer plugin	action
indexer-cloudsearch	DO NOT implement see Transition from Amazon CloudSearch to Amazon OpenSearch Service. Likely we should @Deprecate this plugin.
indexer-csv	DO NOT implement, for testing only
indexer-dummy	DO NOT implement, for testing only
indexer-elastic	implement per this PR
indexer-kafka	implement with Kafka module
indexer-opensearch-1x	DO NOT implement. Likely we should @Deprecate this plugin.
indexer-rabbit	implement with RabbitMQ module
indexer-solr	implement with Solr module

Thanks for any peer review

[lewismc]

Looks like the ubuntu CI run was successful with the new target executing in 1m 19s.

[lewismc]

I am happy to add the Solr, RabbitMQ and Kafka implementations to this PR. If we decide to @Deprecate cloudsearch and opensearch-1x plugins I would propose opening a new issue PR.

[lewismc]

PR updated. i think this is a cleaner implementation. The proposed indexer integration tests reuse the protocol-test pattern (abstract base in core, plugins extend and plug in specifics), but use integration tests (real backends), a separate Ant target, and more configuration via the interface.
I also intentionally chose to NOT include integration tests in the existing ant test execution to retain existing behavior. The CI will take care of integration testing if indexer plugin changes are detected.

[sebastian-nagel]

+1

That's an important contribution. Manually testing the indexer plugins which send docs to an indexer service was a burden.

I also intentionally chose to NOT include integration tests in the existing ant test execution to retain existing behavior.

Definitely. Also because the integration tests can take quite long - 9 minutes on my laptop.

The CI will take care of integration testing if indexer plugin changes are detected.

Perfect!