Enhance Docker API to expose full arun_many config-list support

[Codinator007]

Summary

The Docker API already calls arun_many() internally for multi-URL requests, but the current /crawl contract only accepts a single crawler_config per request.

This makes the Docker service less expressive than the SDK, where arun_many() supports:

• a single CrawlerRunConfig, or
• a list of CrawlerRunConfig objects with url_matcher patterns

Why this matters

For production orchestrators, the Docker API is the cleanest architecture because it keeps:

• the app as orchestration only
• Crawl4AI as the actual crawler runtime
• browser/playwright dependencies out of the app image

But today this forces a tradeoff:

• use the SDK directly for full control, or
• use the Docker API with reduced batch/config flexibility

Requested enhancement

Please extend the official Docker API so it can expose the richer arun_many() capability from the SDK, ideally by supporting one of these forms:

1. crawler_configs as a list of serialized CrawlerRunConfig objects
2. support for URL-specific config matching equivalent to SDK config=[...] with url_matcher

Desired behavior

• multi-URL synchronous crawling without webhooks
• per-result success/failure isolation
• support for mixed URL configs in one request
• optional dispatcher settings if possible

Current observation

The current Docker API implementation loads:

• BrowserConfig.load(browser_config)
• CrawlerRunConfig.load(crawler_config)
  and then calls arun_many() when len(urls) > 1, but only with a single config object.

Benefit

This would make the official Docker API much more viable for real production systems that want clean service separation without losing SDK-level batch control.

[hafezparast]

Hi @Codinator007 — great suggestion! A PR has been submitted: #1852.

What was added: The /crawl endpoint now accepts an optional crawler_configs field — a list of serialized CrawlerRunConfig dicts. When provided with multiple URLs, each config is deserialized and passed as a list to arun_many(), enabling per-URL configuration.

Example request:

{
  "urls": ["https://example.com", "https://other.com"],
  "crawler_configs": [
    {"type": "CrawlerRunConfig", "params": {"word_count_threshold": 5}},
    {"type": "CrawlerRunConfig", "params": {"cache_mode": {"type": "CacheMode", "params": "bypass"}}}
  ]
}

The existing crawler_config (singular) still works as before — fully backward compatible.

How to test before it's merged

pip install git+https://github.com/unclecode/crawl4ai.git@feat/maysam-arun-many-config-list-1837

Then rebuild your Docker image and test with the multi-config payload above.

Could you please check the PR and let us know if this matches your use case? Your feedback helps us merge with confidence and ensures we've addressed your specific scenario correctly.

@unclecode @ntohidi