Skip to content

AO3-7284 Configurable indexer batch size#5589

Open
not-varram wants to merge 1 commit intootwcode:masterfrom
not-varram:AO3-7284
Open

AO3-7284 Configurable indexer batch size#5589
not-varram wants to merge 1 commit intootwcode:masterfrom
not-varram:AO3-7284

Conversation

@not-varram
Copy link
Contributor

Pull Request Checklist

Issue

https://otwarchive.atlassian.net/browse/AO3-7284

Purpose

Make Search::Indexer batch size configurable via config/config.yml instead of hardcoded.

Testing Instructions

  1. Set SEARCH_INDEXER_BATCH_SIZE in config/config.yml (default is 1000).
  2. Run:
    • docker compose run --rm test bundle exec rspec spec/models/search/indexer_spec.rb spec/lib/tasks/search_tasks.rake_spec.rb
  3. Optional functional check:
    • docker compose run --rm test bundle exec rake search:reindex_collections
  4. Confirm all commands pass.

Credit

varram (he/him)

@Bilka2 Bilka2 changed the title AO3-7284-Configurable-indexer-batch-size AO3-7284 Configurable indexer batch size Feb 17, 2026
Copy link
Contributor

@Bilka2 Bilka2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, I have a few comments

# Job size used for the HitCountUpdateJob class:
HIT_COUNT_JOB_SIZE: 1000

# Batch size used by Search::Indexer when reading records from the database:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Batch size used by Search::Indexer when reading records from the database:
# Batch size used by Indexer when reading records from the database:

end

def self.batch_size
ArchiveConfig.SEARCH_INDEXER_BATCH_SIZE || 1000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is always set since it's in the config, so there's no reason to have a default here

let(:indexables) { instance_double(ActiveRecord::Relation, count: 2) }

before do
stub_const("TestIndexer", Class.new(Indexer))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't normally stub like this, could you please work with less stubbing/mocking here? It would be better to actually make records and then index those, e.g. make a few works and then call the method on the WorkIndexer. Then the only thing you have to stub/mock is the AsyncIndexer, so you're interacting a lot less with the innards of the class you're testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants