diff --git a/docs/2-getting-started/connect-git.md b/docs/2-getting-started/connect-git.md new file mode 100644 index 0000000..a702c60 --- /dev/null +++ b/docs/2-getting-started/connect-git.md @@ -0,0 +1,85 @@ +# Connect Your Repository + +**Goal:** Connect your GitHub or GitLab repository to Recce Cloud for automated PR data review. + +Recce Cloud supports GitHub and GitLab. Using a different provider? Contact us at support@reccehq.com. + +## Prerequisites + +- [x] Recce Cloud account (free trial at cloud.reccehq.com) +- [x] Repository admin access (required to authorize app installation) +- [x] dbt project in the repository + +## How It Works + +When you connect a Git provider, Recce Cloud maps your setup: + +| Git Provider | Recce Cloud | +|--------------|-------------| +| Organization | Organization | +| Repository | Project | + +Every Recce Cloud account starts with one organization and one project. When you connect your Git provider, you select which organization and repository to link. + +**Monorepo support:** If you have multiple dbt projects in one repository, you can create multiple Recce Cloud projects that connect to the same repo. + + +## Connect GitHub + +### 1. Authorize the Recce GitHub App + +Navigate to Settings → Git Provider in Recce Cloud. Click **Connect GitHub**. + +**Expected result:** GitHub authorization page opens. + +### 2. Select Organization and Repository + +Choose which GitHub organization to connect. This becomes your Recce Cloud organization. + +Then select the repository containing your dbt project. This becomes your Recce Cloud project. + +**Expected result:** Repository connected. Your Recce Cloud project is ready to use. + +![alt text](../assets/images/2-getting-started/connect-github.png){: .shadow} + +## Connect GitLab + +GitLab uses Personal Access Tokens (PAT) instead of OAuth. + +### 1. Create a Personal Access Token + +In GitLab: User Settings → Access Tokens → Add new token. + +**Required scopes:** + +- `api` - Full access (required for PR comments) +- `read_api` - Read-only alternative (limited functionality) + +**Expected result:** Token string displayed (copy immediately). + +### 2. Add Token to Recce Cloud + +Navigate to Settings → Git Provider. Select GitLab, paste token. + +## Verify Success + +In Recce Cloud, navigate to your repository. You should see: + +- Connection status: "Connected" +- Organization Project is linked to a git repository + +![alt text](../assets/images/2-getting-started/connect-gitlab.png){: .shadow} +![alt text](../assets/images/2-getting-started/org-projects.png){: .shadow} + +## Troubleshooting + +| Issue | Solution | +| --- | --- | +| Repository not found | Ensure proper permissions are granted (GitLab: token access, GitHub: app authorized) | +| Invalid token (GitLab) | Generate new token with `api` scope | +| Cannot post PR comments (GitLab) | Regenerate token with `api` scope instead of `read_api` | + +## Next Steps + +- [Connect Data Warehouse](connect-to-warehouse.md) +- [Add Recce to CI/CD](../7-cicd/ci-cd-getting-started.md) diff --git a/docs/2-getting-started/connect-to-warehouse.md b/docs/2-getting-started/connect-to-warehouse.md new file mode 100644 index 0000000..ef36911 --- /dev/null +++ b/docs/2-getting-started/connect-to-warehouse.md @@ -0,0 +1,107 @@ +# Connect Data Warehouse + +**Goal:** Connect your data warehouse to Recce Cloud to enable data diffing on PRs. + +Recce Cloud supports **[Snowflake](#connect-snowflake), [Databricks](#connect-databricks), [BigQuery](#connect-bigquery), and [Redshift](connect-redshift)**. Using a different warehouse? Contact us at support@reccehq.com. + +## Prerequisites + +- [x] Warehouse credentials with read access +- [x] Network access configured (IP whitelisting if required) + +## Security + +Recce Cloud queries your warehouse directly to compare Base and Current environments. Recce encrypts and stores credentials securely. Read-only access is sufficient for all data diffing features. + +## Connect Snowflake + +### Option 1: Username/Password + +| Field | Description | Example | +|-------|-------------|---------| +| Account | Snowflake account identifier | `xxxxxx.us-central1.gcp` | +| Username | Database username | `MY_USER` | +| Password | Database password | `my_password` | +| Role | Role with read access | `ANALYST_ROLE` | +| Warehouse | Compute warehouse name | `WH_LOAD` | + +### Option 2: Key Pair Authentication + +| Field | Description | Example | +|-------|-------------|---------| +| Account | Snowflake account identifier | `xxxxxx.us-central1.gcp` | +| Username | Service account username | `MY_USER` | +| Private Key | PEM-formatted private key | `-----BEGIN RSA PRIVATE KEY-----...` | +| Passphrase | Key passphrase (if encrypted) | `my_passphrase` | +| Role | Role with read access | `ANALYST_ROLE` | +| Warehouse | Compute warehouse name | `WH_LOAD` | + +## Connect Databricks + +### Option 1: Personal Access Token + +| Field | Description | Example | +|-------|-------------|---------| +| Host | Workspace URL | `adb-1234567890123456.7.azuredatabricks.net` | +| HTTP Path | SQL warehouse path | `/sql/1.0/warehouses/abc123def456` | +| Token | Personal access token | `dapiXXXXXXXXXXXXXXXXXXXXXXX` | +| Catalog | Unity Catalog name (optional) | `my_catalog` | + +### Option 2: OAuth (M2M) + +| Field | Description | Example | +|-------|-------------|---------| +| Host | Workspace URL | `adb-1234567890123456.7.azuredatabricks.net` | +| HTTP Path | SQL warehouse path | `/sql/1.0/warehouses/abc123def456` | +| Client ID | Service principal client ID | `12345678-1234-1234-1234-123456789012` | +| Client Secret | Service principal secret | `dose1234567890abcdef` | +| Catalog | Unity Catalog name (optional) | `my_catalog` | + + +> **Note**: OAuth M2M is auto-enabled in Databricks accounts. For setup details, see [dbt Databricks setup](https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup#oauth-machine-to-machine-m2m-authentication). + +## Connect BigQuery + +| Field | Description | Example | +|-------|-------------|---------| +| Project | GCP project ID | `my-gcp-project-123456` | +| Service Account JSON | Full JSON key file contents | `{"type": "service_account", ...}` | + + +> **Note**: For authentication, we currently provide support for service account JSON only. More details [here](https://docs.getdbt.com/docs/core/connect-data-platform/bigquery-setup#service-account-json). + +## Connect Redshift + +| Field | Description | Example | +|-------|-------------|---------| +| Host | Cluster endpoint | `my-cluster.abc123xyz.us-west-2.redshift.amazonaws.com` | +| Port | Database port | `5439` (Default) | +| Database | Database name | `analytics_db` | +| Username | Database user | `admin_user` | +| Password | Database password | `my_password` | + + +> **Note**: We currently support Database (Password-based authentication) only. More details [here](https://docs.getdbt.com/docs/core/connect-data-platform/redshift-setup#authentication-parameters). + +## Save Connection + +After entering your connection details, click **Save**. Recce Cloud runs a connection test automatically and displays "Connected" on success. + +## Verify Success + +Navigate to Organization Settings in Recce Cloud. Your data warehouse should appear. + +![alt text](../assets/images/2-getting-started/connect-dw.png){: .shadow} + +## Troubleshooting + +| Issue | Solution | +| --- | --- | +| Connection refused | Whitelist Recce Cloud IP ranges in your network configuration | +| Authentication failed | Verify credentials and regenerate if expired | +| Permission denied on table | Grant SELECT permissions on target schemas | + +## Next Steps + +- [Add Recce to CI/CD](../7-cicd/setup-ci.md) +- [Run Your First Data Diff](../5-data-diffing/row-count-diff.md) diff --git a/docs/2-getting-started/gitlab-pat-guide.md b/docs/2-getting-started/gitlab-pat-guide.md deleted file mode 100644 index d707744..0000000 --- a/docs/2-getting-started/gitlab-pat-guide.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: GitLab Personal Access Token ---- - -# GitLab Personal Access Token Setup - -To integrate Recce with your GitLab project, you'll need to create a Personal Access Token (PAT) with appropriate -permissions. - -## Token Scope Requirements - -Recce supports two different permission levels depending on your needs: - -| Scope | Permissions | Features Available | -|------------|----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------| -| `api` | Full API access (read and write) | • View and track merge requests
• **Receive generated summaries and notes on MRs from Recce**
• Full integration capabilities | -| `read_api` | Read-only API access | • View and track merge requests
• **Cannot receive generated summaries and notes on MRs from Recce** | - -!!!warning "Important: Choose the Right Scope" - If you want Recce to automatically post validation summaries and notes directly to your merge requests, you **must** use - the `api` scope. The `read_api` scope only allows Recce to read merge request information but cannot write comments or - summaries back to GitLab. - -## How to Create a Personal Access Token - -Follow these steps to create a Personal Access Token in GitLab: - -1. **Navigate to [Personal Access Token Settings in GitLab](https://gitlab.com/-/user_settings/personal_access_tokens)** - -2. **Create New Token** - - Click **Add new token** button - - Enter a descriptive **Token name** (e.g., "Recce Integration") - - Set an **Expiration date** - -3. **Select Scopes** - - Choose one of the following based on your needs: - - **Option A: Full Integration (Recommended)**
- - ✅ `api` scope
- - This enables Recce to post validation summaries and notes to your merge requests - - **Option B: Read-Only Access**
- - ✅ `read_api` scope
- - ⚠️ You will **not** receive generated PR summaries and notes on your MRs from Recce - -4. **Generate Token** - - Click **Create personal access token** - - **Important**: Copy the token immediately - you won't be able to see it again! - -5. **Save Token Securely** - - Store the token in a secure location - - -## Using Your Token with Recce - -Once you have your Personal Access Token: - -1. Navigate to Recce settings -2. Select GitLab integration -3. Paste your Personal Access Token -4. Complete the connection setup - -## Troubleshooting - -**Token not working?** - -- Verify you've selected the correct scope (`api` or `read_api`) -- Check that the token hasn't expired -- Ensure you have appropriate project permissions (Maintainer or Owner role) - -**Not receiving summaries on merge requests?** - -- Verify your token uses the `api` scope (not just `read_api`) -- Check that Recce has write permissions to your project - -**Still having issues?** - -- Please reach out to us on [our Discord](https://discord.gg/HUUx9fyphJ) or via email at `help@reccehq.com` - -## Related Documentation - -- [Connect Git Provider](./start-free-with-cloud.md#2-connect-git-provider) -- [CI/CD Getting Started](../7-cicd/ci-cd-getting-started.md) diff --git a/docs/2-getting-started/start-free-with-cloud.md b/docs/2-getting-started/start-free-with-cloud.md index 4edc0d0..8c23d3d 100644 --- a/docs/2-getting-started/start-free-with-cloud.md +++ b/docs/2-getting-started/start-free-with-cloud.md @@ -60,7 +60,7 @@ You can use the Recce Web Agent to help automate your setup. Currently it handle The agent covers common setups and continues to expand coverage. If your setup isn't supported yet, the agent directs you to the Setup Guide below for manual configuration. Need help? Contact us at support@reccehq.com. -**Coming soon**: The agent will guide you through steps 1–3, including warehouse connection, Git connection, and CI/CD configuration. +The agent will guide you through steps 1–3, including Git connection, data warehouse connection, and CI/CD configuration. --- @@ -70,20 +70,20 @@ This guide explains each onboarding step in detail. First, go to [cloud.reccehq.com](https://cloud.reccehq.com) and create your free account. -### 1. Connect Data Warehouse - -1. Select your data warehouse (e.g. Snowflake) -2. Provide your read-only warehouse credentials - -> **Note**: This guide uses Snowflake. For supported warehouses, see [Connect to Warehouse](../5-data-diffing/connect-to-warehouse.md). - -### 2. Connect Git Provider +### 1. Connect Git Provider 1. Click **Connect GitHub** 2. Authorize the Recce app installation 3. Select the repositories you want to connect -> **Note**: This guide uses GitHub. For GitLab setup, see [GitLab Personal Access Token](gitlab-pat-guide.md). +> **Note**: This guide uses GitHub. For supported Git providers, see [Connect to Git](connect-git.md) + +### 2. Connect Data Warehouse + +1. Select your data warehouse (e.g. Snowflake) +2. Provide your read-only warehouse credentials + +> **Note**: This guide uses Snowflake. For supported warehouses, see [Connect to Warehouse](connect-to-warehouse.md). ### 3. Add Recce to CI/CD diff --git a/docs/5-data-diffing/connect-to-warehouse.md b/docs/5-data-diffing/connect-to-warehouse.md deleted file mode 100644 index aec4903..0000000 --- a/docs/5-data-diffing/connect-to-warehouse.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: Connect to Warehouse ---- - -## Recce OSS -Recce OSS supports all warehouses that dbt supports. It uses the same configuration as dbt, simply use your existing dbt profiles to connect to your warehouse. No additional setup required. - -## Recce Cloud -If you use Recce Cloud, here are the warehouse connection settings. We currently support: - -- [Snowflake](#snowflake) -- [Databricks](#databricks) -- [BigQuery](#bigquery) -- [Redshift](#redshift) - -Others are coming in future releases - -### Security - -Recce Cloud protects all warehouse connection config (such as passwords, tokens, and private keys) using envelope encryption with AWS KMS. Credentials are encrypted at rest using AES-256, with encryption keys managed by AWS KMS. Decrypted credentials exist only in memory during connection establishment and are never written to disk. AWS KMS keys rotate automatically every 365 days to maintain security best practices. - -### Snowflake -We support two authentication methods for Snowflake: - -- **User & Password**: Traditional username and password authentication -- **Key Pair**: More secure authentication using RSA key pairs - -#### Common Fields - -| Field | Description | Examples | -| ----------- | ----------------------------------------- | ------------------------ | -| `account` | The Snowflake account to connect to | `xxxxxx.us-central1.gcp` | -| `database` | The default database to connect to | `MY_DB` | -| `schema` | The default schema to connect to | `PUBLIC` | -| `warehouse` | The warehouse to use when running queries | `WH_LOAD` | - -#### User & Password Authentication - -| Field | Description | Examples | -| ---------- | ------------------------- | -------- | -| `user` | The user to log in as | `MY_USER` | -| `password` | The password for the user | `MY_PASS` | - -#### Key Pair Authentication - -| Field | Description | Required | -| ------------------------ | ------------------------------------------------------------------------------- | -------- | -| `user` | The user to log in as | Yes | -| `private_key` | Your RSA private key in PEM format or Base64-encoded DER format | Yes | -| `private_key_passphrase` | Passphrase for the private key (only required if your private key is encrypted) | No | - -For more information on setting up key pair authentication, refer to [Snowflake's key pair authentication documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth). - - -### Databricks -We support two authentication methods for Databricks: - -- **Token-based**: Require Personal Access Token (PAT) -- **OAuth Client-based (M2M)**: Auto-enabled in every account with expected settings -More details [here](https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup#examples)). - -#### Common Fields - -| Field | Description | Examples | -| ----------- | --------------------------------------------------------------------------------------------- | ----------------------------- | -| `host` | The hostname of your cluster | `YOURORG.databrickshost.com` | -| `http_path` | The HTTP path to your SQL Warehouse or all-purpose cluster | `/SQL/YOUR/HTTP/PATH` | -| `catalog` | The catalog used to connect to the warehouse. This is optional if you are using Unity Catalog | `MY_CATALOG` | -| `schema` | The default schema to connect to | `MY_SCHEMA` | - -#### Token-based Authentication - -| Field | Description | Examples | -| ---------- | -------------------------------------------------------- | ----------------------------- | -| `token` | The Personal Access Token (PAT) to connect to Databricks | `dapiXXXXXXXXXXXXXXXXXXXXXXX` | - -#### OAuth Client-based Authentication (M2M) - -| Field | Description | Required | -| ----------------------| ------------------------------------------------------------------------------- | -------- | -| `auth_type` | The type of authorization needed to connect to Databricks. Default as `oauth` | Yes | -| `client_id` | The client ID for your Databricks OAuth application | Yes | -| `client_secret` | The client secret for your Databricks OAuth application | Yes | - -For more information on setting up OAuth Client-based authentication, refer to [Databricks's Oauth Client-based authentication documentation]((https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup?tokenoauth=oauth#examples)). - - -### BigQuery - -!!!info "Important" - For authentication, we currently provide support for service account JSON only. More details [here](https://docs.getdbt.com/docs/core/connect-data-platform/bigquery-setup#service-account-json). - -| Field | Description | Examples | -| -------------- | ------------------------------------------------------------------------------------------ | ----------------------------- | -| `project` | The GCP project to connect to | `GCP_PROJECT_ID` | -| `dataset` | The default BigQuery dataset to connect to | `DBT_DATASET_NAME` | -| `keyfile_json` | The [keyfile](https://docs.getdbt.com/guides/bigquery?step=7#generate-bigquery-credentials) generated from your GCP service account to connect to BigQuery | `keyfile_json: type: xxx project_id: xxx private_key_id: xxx ...` | - - -### Redshift - -!!!info "Important" - We currently support Database (Password-based authentication) only. More details [here](https://docs.getdbt.com/docs/core/connect-data-platform/redshift-setup#authentication-parameters). - -| Field | Description | Examples | -| ----------- | --------------------------------------------------------------------------------------------- | ----------------------------- | -| `host` | Host of your cluster | `hostname.region.redshift.amazonaws.com` | -| `user` | Account username to log into your cluster | `MY_USER` | -| `password` | Password for authentication | `MY_PASS` | -| `dbname` | The default database to connect to | `MY_DB` | -| `schema` | The default schema to connect to | `MY_SCHEMA` | -| `port` | Port for your Redshift environment | `5439`| - diff --git a/docs/7-cicd/ci-cd-getting-started.md b/docs/7-cicd/ci-cd-getting-started.md index eda3981..4b6df0d 100644 --- a/docs/7-cicd/ci-cd-getting-started.md +++ b/docs/7-cicd/ci-cd-getting-started.md @@ -57,7 +57,7 @@ Before setting up, ensure you have: - [x] **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/) - [x] **Repository connected** to Recce Cloud - [Connect Git Provider](../2-getting-started/start-free-with-cloud.md#2-connect-git-provider) - - For GitLab: [Create a Personal Access Token](../2-getting-started/gitlab-pat-guide.md) if not already done + - For GitLab: [Create a Personal Access Token](../2-getting-started/connect-git.md#connect-gitlab) if not already done - [x] **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your project ## Setup Steps diff --git a/docs/7-cicd/setup-cd.md b/docs/7-cicd/setup-cd.md index 075d0d5..2425212 100644 --- a/docs/7-cicd/setup-cd.md +++ b/docs/7-cicd/setup-cd.md @@ -293,7 +293,7 @@ prod-build: 1. Verify your repository is connected in [Recce Cloud settings](https://cloud.reccehq.com/settings) 2. **For GitHub**: Ensure `GITHUB_TOKEN` is passed explicitly to the upload step and the job has `contents: read` permission 3. **For GitLab**: Verify project has GitLab integration configured - - Check that you've created a [Personal Access Token](../2-getting-started/gitlab-pat-guide.md) + - Check that you've created a [Personal Access Token](../2-getting-started/connect-git.md#connect-gitlab) - Ensure the token has appropriate scope (`api` or `read_api`) - Verify the project is connected in Recce Cloud settings diff --git a/docs/assets/images/2-getting-started/connect-dw.png b/docs/assets/images/2-getting-started/connect-dw.png new file mode 100644 index 0000000..aa05e9e Binary files /dev/null and b/docs/assets/images/2-getting-started/connect-dw.png differ diff --git a/docs/assets/images/2-getting-started/connect-github.png b/docs/assets/images/2-getting-started/connect-github.png new file mode 100644 index 0000000..5fc4ba0 Binary files /dev/null and b/docs/assets/images/2-getting-started/connect-github.png differ diff --git a/docs/assets/images/2-getting-started/connect-gitlab.png b/docs/assets/images/2-getting-started/connect-gitlab.png new file mode 100644 index 0000000..75ace08 Binary files /dev/null and b/docs/assets/images/2-getting-started/connect-gitlab.png differ diff --git a/docs/assets/images/2-getting-started/org-projects.png b/docs/assets/images/2-getting-started/org-projects.png new file mode 100644 index 0000000..c127ef3 Binary files /dev/null and b/docs/assets/images/2-getting-started/org-projects.png differ diff --git a/docs/index.md b/docs/index.md index 7344ae3..afd7590 100644 --- a/docs/index.md +++ b/docs/index.md @@ -61,7 +61,7 @@ With CI/CD: Yes, you can run Recce locally for dev sessions. But CI/CD unlocks the full value: automatic validation on every PR without manual setup. **What data platforms does Recce support?** -Recce works with data warehouses like Snowflake, BigQuery, Redshift, and Databricks. See [Connect to Warehouse](5-data-diffing/connect-to-warehouse/) for setup. +Recce works with data warehouses like Snowflake, BigQuery, Redshift, and Databricks. See [Connect to Warehouse](2-getting-started/connect-to-warehouse/) for setup. ## Related - Interactive Demo: [Try the Data Review Agent](https://reccehq.com/demo/) diff --git a/mkdocs.yml b/mkdocs.yml index bee8774..6565e4b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -54,6 +54,8 @@ nav: - Getting Started: - 2-getting-started/oss-vs-cloud.md - 2-getting-started/start-free-with-cloud.md + - 2-getting-started/connect-git.md + - 2-getting-started/connect-to-warehouse.md #- 2-getting-started/cloud-5min-tutorial.md - 2-getting-started/installation.md - Claude Plugin: 2-getting-started/claude-plugin.md @@ -69,7 +71,6 @@ nav: - 4-downstream-impacts/breaking-change-analysis.md #- 4-downstream-impacts/transformation-types.md - Data Diffing: - - 5-data-diffing/connect-to-warehouse.md - 5-data-diffing/row-count-diff.md - 5-data-diffing/profile-diff.md - 5-data-diffing/value-diff.md @@ -155,6 +156,10 @@ theme: plugins: - search + - redirects: + redirect_maps: + '5-data-diffing/connect-to-warehouse.md': '2-getting-started/connect-to-warehouse.md' + '2-getting-started/gitlab-pat-guide.md': '2-getting-started/connect-git.md' - glightbox: skip_classes: - skip-glightbox