Skip to content

feat(mcp): single-env onboarding, --help prerequisites, and error classification#1175

Open
iamcxa wants to merge 15 commits intomainfrom
fix/drc-2796-mcp-help-prerequisites
Open

feat(mcp): single-env onboarding, --help prerequisites, and error classification#1175
iamcxa wants to merge 15 commits intomainfrom
fix/drc-2796-mcp-help-prerequisites

Conversation

@iamcxa
Copy link
Contributor

@iamcxa iamcxa commented Mar 4, 2026

Summary

  • DRC-2794: Fix MCP server crash when target-base/ is missing — no longer throws raw FileNotFoundError
  • DRC-2795: Single-env onboarding mode — when target-base/ is missing, uses target/ for both environments with _warning in diff responses and description hints
  • DRC-2796: --help prerequisites section documenting required dbt artifacts
  • DRC-2754: Error classification for MCP server — shared indicator constants, classified errors → logger.warning() + Sentry metrics, unclassified → logger.error() + traceback

Reviewer Guide

Quick mapping from issue → code area. Start here if you're reviewing file-by-file.

Issue What Key files / functions
DRC-2794 Don't crash on missing target-base/ recce/cli.pymcp_server() detects missing dir, sets single_env=True
DRC-2795 Diff tools add _warning in single-env mode recce/mcp_server.py_maybe_add_warning(), diff tool descriptions
DRC-2796 --help shows prerequisites recce/cli.pymcp_server() epilog text
DRC-2754 Central error classification recce/mcp_server.py_classify_db_error() in call_tool; recce/tasks/rowcount.py → shared indicator constants

Design notes for reviewers:

  • _maybe_add_warning() only applies to 3 diff tools (row_count_diff, query_diff, profile_diff) — lineage_diff and schema_diff work without base env, so no warning needed
  • Error classification priority: permission_denied > table_not_found > syntax_error (first match wins) — this matches severity order
  • Central handler re-raises after logging — MCP SDK requires raised exceptions to set isError=True
  • run.py:schema_diff_should_be_approved() try/except is intentional — ensures check creation even on error

Changes

Single-env onboarding (DRC-2794 + DRC-2795)

  • CLI detects missing target-base/ and sets single_env=True (fixes DRC-2794 crash)
  • _maybe_add_warning() adds _warning field to diff tool results
  • Diff tool descriptions include conditional single-env note
  • All tools remain available (no tool restriction in single-env mode)

Help text prerequisites (DRC-2796)

  • recce mcp-server --help shows prerequisites section before examples
  • Documents required dbt artifact directories and generation commands

Error classification (DRC-2754)

  • Shared error indicator constants in recce/tasks/rowcount.py
  • _classify_db_error() in MCP central handler: permission_denied > table_not_found > syntax_error
  • Classified errors → logger.warning() + sentry_metrics.count(); unclassified → logger.error()
  • Applied to run.py schema_diff_should_be_approved() and sqlmesh path

Test coverage

  • 25 E2E tests (real DuckDB via DbtTestHelper) — Layer 1 (direct) + Layer 2 (anyio MCP protocol)
  • Unit tests for error classification, call_tool handler, single-env mode
  • Coverage for edge cases: missing base env, removed/modified columns, run_check error paths
  • _query_row_count dbt path permission_denied and table_not_found branches

Docs

  • Project-level MCP developer skill (.claude/skills/recce-mcp-dev/)
  • MCP E2E verification skill (.claude/skills/recce-mcp-e2e/)
  • AGENTS.md: add coverage command and skills directory

Demo

https://www.loom.com/share/91c92e56501d460d95b421e6a2ce1649

DRC-2795 single-env onboarding + MCP E2E auto test

Test plan

  • python -m pytest tests/test_mcp_server.py tests/test_mcp_e2e.py -v — 71 tests pass
  • python -m pytest tests/test_run.py tests/test_summary.py tests/test_rowcount_sqlmesh.py — 40 tests pass
  • make flake8 — clean
  • Codecov patch coverage gaps resolved (12 → 0 missing patch lines)

Closes DRC-2794, DRC-2795, DRC-2796, DRC-2754

🤖 Generated with Claude Code

@codecov
Copy link

codecov bot commented Mar 4, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
recce/cli.py 52.09% <100.00%> (+2.32%) ⬆️
recce/mcp_server.py 77.27% <100.00%> (+24.86%) ⬆️
recce/run.py 28.69% <100.00%> (+3.15%) ⬆️
recce/summary.py 66.05% <100.00%> (+0.72%) ⬆️
recce/tasks/rowcount.py 90.04% <100.00%> (+16.28%) ⬆️
tests/test_cli.py 100.00% <100.00%> (ø)
tests/test_mcp_e2e.py 100.00% <100.00%> (ø)
tests/test_mcp_server.py 100.00% <100.00%> (ø)
tests/test_rowcount_sqlmesh.py 100.00% <100.00%> (ø)
tests/test_run.py 100.00% <100.00%> (ø)
... and 1 more

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@iamcxa iamcxa requested a review from Copilot March 4, 2026 15:34
@iamcxa iamcxa marked this pull request as ready for review March 4, 2026 15:35
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds “single-environment onboarding mode” to the Recce MCP server to allow starting when target-base/ artifacts are missing, and expands test coverage around MCP tools and DB error classification.

Changes:

  • Add single_env mode to MCP server, including _warning injection into diff tool responses and tool description hints.
  • Centralize DB error classification indicators and extend rowcount/sqlmesh error handling + summary rendering (N/A).
  • Add new MCP E2E test suite (direct tool calls + full protocol) and CLI tests for single-env auto-detection.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
recce/mcp_server.py Adds single_env flag, warning injection, and centralized error classification/logging.
recce/cli.py Documents prerequisites in --help and auto-enables single-env mode when target-base/ missing.
recce/tasks/rowcount.py Extracts shared DB error indicator lists; improves sqlmesh error classification behavior.
recce/summary.py Returns "N/A" when row count is unavailable due to known statuses (table missing/permission/etc.).
recce/run.py Improves logging/handling for schema-diff auto-approval when expected DB errors occur.
tests/test_mcp_e2e.py Adds E2E tests for MCP tools (DuckDB) and protocol wiring.
tests/test_mcp_server.py Adds unit tests for single-env warnings, tool descriptions, and DB error classification.
tests/test_cli.py Adds CLI tests ensuring single-env auto-detection + messaging.
tests/test_summary.py Adds coverage for new "N/A" behavior under specific meta statuses.
tests/test_rowcount_sqlmesh.py Adds sqlmesh rowcount error classification tests.
pyproject.toml Pins sentry-sdk minimum version.
Makefile Adds an mcp target (currently with a hard-coded local path).
Comments suppressed due to low confidence (3)

recce/mcp_server.py:41

  • sentry_sdk is imported but never used in this module (only sentry_metrics is referenced). This will fail flake8 with an unused import (F401); remove the unused import or use it explicitly.
try:
    import sentry_sdk
    from sentry_sdk import metrics as sentry_metrics
except ImportError:
    sentry_sdk = None
    sentry_metrics = None

recce/tasks/rowcount.py:337

  • _classify_sqlmesh_error re-raises unknown exceptions via raise error, which discards the original traceback and makes debugging harder. Prefer re-raising with raise from the original except block (or preserve the traceback with raise error.with_traceback(...)).
        # Unknown error — re-raise
        raise error

Makefile:155

  • The mcp make target hard-codes a developer-specific absolute path (/Users/kent/...). This will break for other contributors and in CI, and it also leaks a local filesystem path into the repo. Please remove this target or refactor it to use a configurable env var (with a safe default) and document it outside the tracked Makefile if it’s only for local use.
JAFFLE_SHOP_DIR := /Users/kent/Project/recce/jaffle_shop_golden

mcp:
	@dotenv -f $(JAFFLE_SHOP_DIR)/.env run python -m recce.cli mcp-server --sse \
	--project-dir $(JAFFLE_SHOP_DIR) \
	--profiles-dir $(JAFFLE_SHOP_DIR) \
	--target-path $(JAFFLE_SHOP_DIR)/target \
	--target-base-path $(JAFFLE_SHOP_DIR)/target-base \
	--config $(JAFFLE_SHOP_DIR)/recce.yml \
	--port 8000

iamcxa and others added 4 commits March 4, 2026 23:54
When target-base/ directory is missing, MCP server now starts in
single-env mode instead of crashing. Diff tools include _warning
fields and tool descriptions note single-env limitations.

Closes DRC-2794
Closes DRC-2795
Closes DRC-2796

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
- Unit tests for single-env passthrough, descriptions, CLI detection
- E2E tests with DuckDB-backed fixtures (Layer 1 + Layer 2)
- Skip MCP CLI tests when mcp package not installed

DRC-2795, DRC-2796

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
Classify expected DB errors (table_not_found, permission_denied,
syntax_error) as warnings with Sentry metrics instead of error events.
Consolidate error indicator constants in rowcount.py as single source
of truth. Fix schema_diff, sqlmesh, and summary error handling.

- Central handler: classify → warning+metrics / error → re-raise
- Remove duplicate logging from tool handlers
- Upgrade sentry-sdk >= 2.44.0 for metrics API

Closes DRC-2754

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
- Priority-order test: permission_denied wins over table_not_found
- Sqlmesh error classification tests
- Summary N/A detection and non-dict meta defensive tests

DRC-2754

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
@iamcxa iamcxa force-pushed the fix/drc-2796-mcp-help-prerequisites branch from 9980380 to 8e2dc61 Compare March 4, 2026 15:56
iamcxa and others added 3 commits March 5, 2026 00:04
- Use Union[] instead of PEP 604 syntax for Python 3.9 compatibility
- Remove unused sentry_sdk import (only sentry_metrics is used)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
…ication

- Add TestSchemaDiffShouldBeApproved: cover error classification in run.py
- Add TestCallToolHandler: test call_tool error handler via MCP Server internals
  including sentry_metrics emission and classified/unclassified error logging

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
Add 7 tests covering uncovered patch lines:
- lineage_diff missing base env, schema_diff removed/modified columns
- run_check missing check_id and RecceException path
- _query_row_count dbt path permission_denied and table_not_found

Remove dead hasattr branches in test_mcp_e2e.py, add pragma: no cover
to sentry import fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
@iamcxa iamcxa requested a review from wcchang1115 March 4, 2026 17:05
@iamcxa iamcxa self-assigned this Mar 4, 2026
Add project-level skill for MCP server development guidance.
Update AGENTS.md with coverage command and skills directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
iamcxa added a commit to DataRecce/recce-docs that referenced this pull request Mar 4, 2026
Update prerequisites, server modes, and troubleshooting to reflect the
new single-env onboarding mode where the MCP server starts without
target-base/ artifacts. Depends on DataRecce/recce#1175 (DRC-2795).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@iamcxa iamcxa changed the title feat(mcp): add single-env onboarding mode for MCP server feat(mcp): single-env onboarding, error classification, and test coverage Mar 4, 2026
@iamcxa iamcxa changed the title feat(mcp): single-env onboarding, error classification, and test coverage feat(mcp): single-env onboarding mode, error classification, and comprehensive test suite Mar 4, 2026
@iamcxa iamcxa changed the title feat(mcp): single-env onboarding mode, error classification, and comprehensive test suite feat(mcp): single-env onboarding, --help prerequisites, and error classification Mar 4, 2026
iamcxa and others added 2 commits March 5, 2026 02:44
Adds a project-level skill for full-stack verification of all 8 MCP
tools against a real dbt project. Includes a reusable Python test
template that auto-discovers models and validates both full mode
(8 tools) and single-env mode (5 warning checks).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
Remove local JAFFLE_SHOP_DIR and mcp target that were accidentally
included in the PR.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
Copy link
Contributor Author

@iamcxa iamcxa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review: inline comments explaining design rationale for each DRC issue area.

iamcxa and others added 3 commits March 5, 2026 15:55
…o dev skill

- Restructure Testing section into Three Layers table (Unit/Integration/Smoke)
- Add Test Coverage Gap Analysis: proactive scan across all three layers
- Add E2E Verification Gate: interactive prompt to invoke /recce-mcp-e2e
- Both gates share consistent exclusion rules (test-only, docs, imports)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
Use MCP protocol's `instructions` field to inform agents about
single-environment mode during the initialize handshake, so they
know the server state before calling any tools.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
- Remove overly broad "CATALOG ERROR" from TABLE_NOT_FOUND_INDICATORS;
  "DOES NOT EXIST" already catches DuckDB missing-table errors
- Fix _classify_sqlmesh_error docstring to document raise behavior
- Preserve exception chain: raise ValueError(str(e)) from e
- Fix misleading "synchronously" comment on run_in_executor call
- Clarify SKILL.md: priority order enforced by _classify_db_error(),
  sentry_metrics guarded by sentry_sdk availability
- Add reference comment to unavailable_statuses in summary.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
iamcxa added a commit to DataRecce/recce-docs that referenced this pull request Mar 6, 2026
Update prerequisites, server modes, and troubleshooting to reflect the
new single-env onboarding mode where the MCP server starts without
target-base/ artifacts. Depends on DataRecce/recce#1175 (DRC-2795).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Collaborator

@wcchang1115 wcchang1115 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi KC,
I feel that the recce-mcp-dev skill is a bit overfit for the row count development case.
And there are some minor comments for your further check, thanks!

- Rename _maybe_add_warning → _maybe_add_single_env_warning for clarity
- Show unavailable status reason in N/A display: "N/A (table_not_found)"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Kent <iamcxa@gmail.com>
@iamcxa
Copy link
Contributor Author

iamcxa commented Mar 8, 2026

All review feedback addressed — please see individual thread replies for details.

@wcchang1115 @claude

Copy link
Collaborator

@wcchang1115 wcchang1115 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants