Draft
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
@prisma-next/runtime-executor
@prisma-next/sql-runtime
@prisma-next/extension-paradedb
@prisma-next/extension-pgvector
@prisma-next/postgres
@prisma-next/sql-orm-client
@prisma-next/contract-authoring
@prisma-next/contract-ts
@prisma-next/ids
@prisma-next/psl-parser
@prisma-next/psl-printer
@prisma-next/cli
@prisma-next/emitter
@prisma-next/eslint-plugin
@prisma-next/migration-tools
@prisma-next/vite-plugin-contract-emit
@prisma-next/sql-contract
@prisma-next/sql-errors
@prisma-next/sql-operations
@prisma-next/sql-schema-ir
@prisma-next/sql-contract-psl
@prisma-next/sql-contract-ts
@prisma-next/sql-contract-emitter
@prisma-next/family-sql
@prisma-next/sql-kysely-lane
@prisma-next/sql-lane-query-builder
@prisma-next/sql-relational-core
@prisma-next/sql-builder-new
@prisma-next/sql-lane
@prisma-next/target-postgres
@prisma-next/adapter-postgres
@prisma-next/driver-postgres
@prisma-next/core-control-plane
@prisma-next/core-execution-plane
@prisma-next/config
@prisma-next/contract
@prisma-next/operations
@prisma-next/plan
@prisma-next/utils
commit: |
Spec covers the code-first data migration model for prisma-next's graph-based migration system: authoring surface (defineMigration with required check/run), execution model (class-based op interleaving, three transaction modes), planner detection and scaffolding, invariant- aware routing, and 18 schema evolution scenarios walked through against the design. Includes 8 key decision records documenting alternatives considered and rationale.
Four milestones: authoring/runner (M1), planner detection/scaffolding (M2), graph integration/routing (M3), close-out (M4). Prerequisite: ref format refactor to carry invariants. All acceptance criteria from the spec mapped to tests.
…ift, clean up strikethroughs - Key Decisions section now precedes Requirements for better reading flow - Content hash drift detection descoped to non-goals with rationale (manifest stale after scaffold edit, ledger per-db, workflow friction) - D8 reframed around the three properties that matter (scaffold, context, prevent accidental no-ops) rather than the throw mechanism - Removed all strikethroughs in favor of proper non-goals documentation
migration new collapses into the data migrations feature — users write
raw SQL via readSql('migration.sql') or directly in run(db). No
separate structural SQL authoring surface needed. Post-apply schema
verification is the safety net.
…sections Requirements now describe problems to solve (R1-R14), not implementation details. Solution section addresses each requirement with back-references. Added primary run(db) code example, documented data_migration as ops.json operation, readSql as SQL-first alternative.
Moves Key Decisions after Solution so the reader understands the problem and approach before encountering design rationale. Merges R4/R9 (both about preventing accidental skipping). Moves Other Considerations and References to the end.
Four non-functional requirements were dropped during restructure: target-independent op partitioning, in-process execution, parameterized query enforcement, and idempotency guidance. Added back as constraints at the top of the Solution section.
…ization model Data migrations are now TypeScript-authored but SQL-executed. The TS is evaluated at verification time to produce SQL ASTs, which are serialized into ops.json. At apply time, only serialized SQL runs — no TypeScript is loaded. Integrates with existing Draft → Attested → Applied lifecycle. Reworks D1, D3, authoring surface, and acceptance criteria to match.
- Summary/Description: reflect serialized queries, not executable code - R1: "express" not "run arbitrary code" - R2: acknowledge S16/S17 are out of scope, not "no scenario impossible" - R4: failure at verification time (draft state), not apply time - Serialize JSON ASTs into ops.json (target-agnostic), not raw SQL - Target adapter renders ASTs to SQL at apply time - Fix all stale db/check(db) references to client/check(client) - Fix D4/D6 references to match new model - Update Security section: no TS at apply time - Remove stale "Typed db interface" non-goal - Add OQ-5 (query builder expressiveness) and OQ-6 (check result interpretation) - Fix unmanaged mode description (serialized SQL, not user-controlled batching)
Check returns a query AST (empty result = already applied), false (always run, for seeding/idempotent cases), or true (always skip). Resolves OQ-6.
…tion Check executes between phase 1 and phase 3: before run (to skip if done) and after run (to validate before constraints tighten). Produces meaningful diagnostics instead of cryptic DB constraint errors.
Complete rewrite reflecting R0 pivot: TypeScript-authored, JSON AST- serialized, SQL-executed. Adds P2 (query builder validation) prereq. Updates P1 for refs directory structure and non-optional invariants. All tasks updated to reflect verify-time serialization, no TS at apply time, recording query builder client, and draft state model.
readSql descoped — query builder is sole v1 authoring surface. Document that the AST model supports a future raw_sql node type (opaque SQL string, target-specific) for readSql and raw expressions. This enables the same AST format to work across SQL and document database targets. Removes R10 (SQL-first), renumbers R10-R12, removes readSql plan tasks.
Scenarios now include solution fit analysis for each case, noting query builder expressiveness needs (OQ-5), detection gaps, and out-of- scope scenarios (S16 fully, S8/S17 partially). Removed stale readSql reference from OQ-5. All R-numbers consistent across spec.
Query builder has known DML gaps (expressions in SET, INSERT...SELECT, mutation joins, AST JSON serialization) but these are independent of the data migration infrastructure. As the QB is extended, data migrations gain expressiveness automatically. Updated scenarios with solution fit analysis and OQ-5 gap notes.
Draft state: op entry has source field pointing to data-migration.ts with null check/run. Attested state: verify evaluates TS, fills in serialized ASTs. Source field stays for traceability but isn't part of edgeId. Clear draft detection: null check/run = not serialized.
Explains why the data migration mechanism works for MongoDB and document databases: the contract represents the application's data model (not the DB schema), schema evolution is semantically equivalent across targets, and the data migration becomes the primary surface (not supplement) for schemaless targets. Target-specific work is limited to the query builder client.
Quick-read table mapping all 18 scenarios to auto-detection, execution model support, and known gaps. 12 fully supported, 2 with manual authoring workarounds, 2 with detection gaps, 2 out of scope.
Breaking change: refs move from migrations/refs.json to migrations/refs/
with individual <name>.json files containing { hash, invariants }.
Core module:
- New RefEntry type: { hash: string; invariants: string[] }
- readRef(refsDir, name) for single-ref reads
- writeRef(refsDir, name, entry) for atomic per-ref writes
- deleteRef(refsDir, name) with empty parent cleanup
- readRefs(refsDir) reads all refs from directory recursively
- resolveRef returns RefEntry instead of string
- Removed writeRefs (single-file bulk write)
CLI:
- migration-ref set/get/delete/list use per-file operations
- migration-apply uses readRef for single-ref lookup
- migration-status uses readRefs with RefEntry mapping
- All help text updated from refs.json to refs/
Fixtures: 13 migration-fixtures converted from refs.json to refs/
Major spec revision incorporating feedback from data-migrations-response: - All migrations (structural + data) authored as TS operation chains - Planner outputs migration.ts with operation builder calls, not ops.json directly - Data transforms are operations in the chain, not a separate data-migration.ts - Strategies encapsulate common patterns (columnSplit, etc.) — future DX - Planner manages operation ordering directly (no class-based partitioning) - Eliminates OQ-1 (constraint classification gap) — planner knows where to put them - D4 reworked: planner-managed ordering replaces class-based partition - D6 (class-based partition) removed, replaced by planner-managed ordering - Transaction model becomes composable annotations on operations/groups - migration.ts replaces data-migration.ts as the authoring file - New OQs: operation builder API design, planner TS output transition
Covers createTable, dropTable, addColumn, dropColumn, alterColumnType, setNotNull, dropNotNull, setDefault, dropDefault, addPrimaryKey, addUnique, addForeignKey, dropConstraint, createIndex, dropIndex, createType, dataTransform, transaction, noTransaction. Each maps to the corresponding planner operation with correct operationClass.
5 milestones: operation builders + serialization (M1), runner data transform execution (M2), planner detection/scaffolding/migration-new (M3), graph integration + routing (M4), close-out (M5). P1 (refs) marked done. All tasks reflect unified migration.ts authoring, operation builder API, planner-managed ordering.
Adds 'data' to MigrationOperationClass union. DataTransformOperation extends MigrationPlanOperation with name (invariant identity), source (TS file path), check (query AST | boolean | null), and run (query AST[] | null). SerializedQueryNode is a typed placeholder for the query builder's JSON serialization format (kind discriminator + data).
…ract The fallback built SQL from descriptor strings without proper referential action mapping. Since the destination contract always reflects the desired end state (user updates contract before authoring migration), the FK must be present. If not, it's a workflow error — throw with a clear message.
…or-contract join Descriptors reference contract elements by name, not by value. The resolver looks up StorageColumn/StorageTable/ForeignKey from the destination contract — the single source of truth. fromContract was never used by any resolve function and has been removed from OperationResolverContext.
Descriptors no longer carry type strings, defaults, constraint names, or FK references. They are pure references: table + column name for columns, table + columns for constraints. The resolver looks up all definitions from the destination contract (StorageColumn, StorageTable, ForeignKey, PrimaryKey, UniqueConstraint, Index). This makes the contract the single source of truth — no duplication between descriptor and contract.
Scaffolds a migration package with migration.ts for manual authoring. Derives from/to hashes from graph state and emitted contract (same logic as migration plan). Writes empty ops.json + migration.ts scaffold. Package stays draft (migrationId: null) — user fills in migration.ts, then runs migration verify to resolve and attest. Supports --name (directory slug) and --from (explicit starting hash).
Journey test: migration new → fill migration.ts with descriptors + dataTransform (raw_sql) → verify → apply → verify data is correct. Tests the full authoring pipeline end-to-end with a real database. Also: - Add runMigrationNew helper to journey test helpers - Add migration-new to CLI build config and package exports - Add migration-builders to target-postgres build config - Allow 'data' operation class in migration apply policy
…tatus - migration verify: --dir is now optional. Without it, scans all packages in migrations dir. Draft packages get migration.ts evaluated and attested. Reports per-package results. - runMigrationVerify helper now uses runCommand (with --config) - E2E test updated to use verify without --dir - Plan updated with done items, new milestones for verify hardening, draft visibility, planner detection, and graph integration - Issue triage: draft migration visibility (high severity)
Reverts verify scan-all-packages behavior. Keeps required --dir (original behavior) plus migration.ts evaluation additions. Scan-all mode deferred to future work. Plan updated to reflect reduced scope and remaining milestones.
…ition Attestation hashes for integrity but doesn't verify ops transform fromContract to toContract. Planner-generated ops are correct by construction; user-authored ops (migration new) are only validated at apply time against a live database. Static check would require applyOpsToSchema simulation which doesn't exist.
Previously migration.ts was only evaluated when the package was in draft state (migrationId: null). This meant editing migration.ts after attestation had no effect — verify would report "verified" with stale ops. Now verify always evaluates migration.ts when present, writes fresh ops.json, then verifies/attests. If the ops changed, the hash mismatches and verify re-attests automatically.
Node import() resolves relative to the calling module, not cwd. Using resolve() ensures the path is absolute regardless of how packageDir was constructed. Also adds plan note for verify display fix.
…e states
The add-nullable → backfill → setNotNull pattern requires adding a
column as nullable even when the contract says NOT NULL. addColumn now
accepts optional { nullable: boolean } overrides. The resolver applies
overrides on top of the contract-looked-up StorageColumn. Documented
in spec as a design note.
Spec now shows both the intended callback API (check/run receive typed client, return ASTs) and the current v1 stopgap (raw_sql nodes directly). Plan adds milestone for slotting in the existing typed client once available — no new client design needed, just wiring and serialization.
Detailed analysis of current planner, what it does, what's a data migration workaround (temp defaults), and how to replace it with an issue-driven descriptor emitter. Prerequisite: augment SchemaIssue with missing constraint kinds. Removes ~500 lines of planner code, temp default strategy, identity values. Resolver already handles all needed operations.
Patterns are functions with a type signature: take issues + context, return either match (modified issues + ops) or no_match. Planner chains matchers before default issue handling. v1 matcher: NOT NULL backfill. Future: column rename, column split, etc.
New planner alongside the existing one. Takes schema issues and emits MigrationOpDescriptor[] instead of SqlMigrationPlanOperation[]. - Pattern matchers consume recognized issues before default handling - v1 matcher: notNullBackfillMatcher (NOT NULL without default → addColumn(nullable) + dataTransform placeholder + setNotNull) - No verifier changes needed — existing _mismatch issues with undefined actual = missing constraint - Issues sorted by dependency order for correct op sequencing - Descriptors merged: drops → tables → columns → pattern ops → alters → constraints
Each issue maps to Result<descriptors, conflict>. Unhandled issues (constraint mismatches with different columns, missing types/deps) are explicit conflicts with actionable messages pointing to migration new. No silent dropping. Also notes SchemaIssue should be a discriminated union to eliminate defensive null checks.
migration plan now uses planWithDescriptors when available. Always writes migration.ts. If no data migration: evaluates, resolves, writes ops, attests. If data migration: writes draft with placeholder. Falls back to old planner if planWithDescriptors not available. Tests will fail — the descriptor planner produces different ops than the old planner (no temp defaults, different ordering, different precheck/postcheck structure). Needs investigation.
… ops for new tables The descriptor planner now emits addForeignKey, createIndex, and addUnique descriptors alongside createTable for new tables (the verifier skips constraint checks for missing tables, so the planner must derive them from the contract). scaffoldMigrationTs serializes descriptors as executable builder calls instead of comments, so the plan→evaluate→resolve round-trip produces real ops. Adds OperationDescriptor framework-level type and builders registry object on operation-descriptors. Removes old planner fallback from migration plan command.
… issue kinds Enum verifier now emits enum_values_added and enum_values_changed instead of generic type_values_mismatch, matching the diff already computed by determineEnumDiff. Descriptor planner maps type_missing to createEnumType (with codec check) and dependency_missing to createDependency. Enum value changes are deferred as conflicts. Resolver generates CREATE TYPE AS ENUM SQL directly for createEnumType (no hook delegation). Dependencies resolve via install ops from framework components. resolveOperation now returns arrays, resolveOperations uses flatMap. SchemaIssue gains dependencyId field. Old planner reconciliation updated for new issue kinds.
The resolver needs codec hooks for parameterized type expansion (e.g. character(36)). Without frameworkComponents, the resolver can't look up expandNativeType hooks and throws.
The contract JSON omits nullable when false (compact serialization). The matcher was checking `nullable !== false` which skipped columns where nullable was undefined (absent). Changed to `nullable === true` so only explicitly nullable columns are skipped.
Non-enum codec hooks may still emit type_values_mismatch. Without a case in mapIssue, the switch would fall through silently. Added as a conflict until per-codec type alteration is supported.
…uite typeChangeMatcher: safe widenings (int4→int8) emit alterColumnType, unsafe changes emit dataTransform with TODO USING expression. nullableTighteningMatcher: nullable→NOT NULL emits dataTransform (user handles existing NULLs) + setNotNull. Adds 40+ scenario tests covering additive, reconciliation, drops, alters, types, deps, ordering, and data-safety gaps. 7 known failures: 5 enum/type (verifier path), 1 dependency (verifier path), 1 ordering (drops before pattern ops in column split scenario). Also fixes typo in typeChangeMatcher (toType read from wrong column) and adds using field to alterColumnType descriptor.
These descriptor kinds were missing from Phase 4 filters in planDescriptors, causing them to be silently dropped from the output. Added depOps category (deps + types first) to the ordering. Fixes 3 scenario tests (enum type, dependency, types-before-tables). Adds pgvector dependency ordering test. 4 remaining failures: enum value changes (deferred), unknown codec (deferred), column split ordering (drops before pattern ops).
saevarb
force-pushed
the
feat/data-migrations
branch
from
b1321a5 to
8c0255a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.