Improve error handling and test robustness across CLI and crawler

[YusukeHirao]

Summary

This PR improves error handling patterns and test reliability across the codebase by introducing proper error aggregation, better exception handling in async operations, and more robust test assertions.

closes: #16

Key Changes

Error Handling Improvements

• CLI Error Propagation: Modified crawl and pipeline commands to throw CrawlAggregateError instead of calling process.exit(1), allowing callers to handle errors programmatically. The error is caught at the CLI entry point and exits there.
• New CrawlAggregateError Class: Created a custom error class that wraps multiple errors from crawl operations, providing structured error information to callers.
• Lighthouse Error Normalization: Added toError() utility function to safely convert any thrown value (including non-Error types) into an Error instance, preserving stack traces for actual Errors.
• Lighthouse Plugin Robustness: Updated the Lighthouse plugin to wrap non-Error exceptions into Error objects and return error results instead of propagating unexpected throws.

Test Improvements

• Archive Tests: Wrapped test cleanup in try-finally blocks to ensure mocks are restored and resources are cleaned up even when assertions fail.
• New Test Coverage: Added comprehensive tests for toError() utility covering Error instances, strings, null, undefined, and Error subclasses.
• Test Assertions: Replaced conditional assertions with explicit expect().toBeDefined() checks in e2e tests, making test failures more obvious and preventing silent test passes.
• WriteQueue Tests: Added test for drain() on empty queue to verify immediate resolution.

CLI Enhancements

• Silent Mode Support: Added --silent flag support to the pipeline command to suppress console output during execution, useful for programmatic usage.
• Documentation: Updated JSDoc comments to reflect that errors now propagate as exceptions rather than causing process exit.

Code Quality

• WriteQueue Signature: Updated enqueue() to accept both async and synchronous operations (Promise<T> | T), improving flexibility.
• Documentation: Added TODO comment in pipeline.ts noting flag definition duplication across commands and the need for manual synchronization.

Implementation Details

• Error handling now follows a consistent pattern: operations throw exceptions, which are caught at appropriate boundaries (CLI entry points) where process.exit() is called.
• The toError() utility ensures that even unexpected non-Error throws are properly converted to Error instances with meaningful messages.
• Test cleanup is now guaranteed via try-finally blocks, preventing resource leaks and mock pollution between tests.

https://claude.ai/code/session_01Gz7eS3SnV1N6LakV8b7CxM