ENG-3233: Optimize Cypress CI with shared build and better sharding#7792
ENG-3233: Optimize Cypress CI with shared build and better sharding#7792gilluminate merged 12 commits intomainfrom
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
d123096 to
a5fc612
Compare
- Add a dedicated build job that runs npm ci + next build once and shares the artifact across all matrix jobs, eliminating ~5 min of redundant build work per shard - Replace test-count sharding heuristic with file-size-based bin-packing, which better correlates with actual execution time - Increase parallel groups from 5 to 7 to reduce wall time per shard - Run build and prepare-matrix in parallel since they're independent Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Cypress binary lives at ~/.cache/Cypress, not in node_modules, so it wasn't included in the shared build artifact. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use actions/cache keyed on package-lock.json hash to persist node_modules and ~/.cache/Cypress between workflow runs. The build artifact now only carries .next and fides-js/dist, which are the only outputs that change per-commit. On cache hit the build job skips npm ci entirely. Matrix jobs use a read-only cache restore (fail-on-cache-miss since the build job always runs first and populates the cache). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
a5fc612 to
c3c732d
Compare
There was a problem hiding this comment.
Code Review
Overall this is a solid CI optimization. The architectural approach — centralizing the expensive npm ci + next build into a single build job, sharing output via artifacts, and running build + prepare-matrix in parallel — is well-reasoned and the implementation is clean. The digests.cy.ts fix (hoisting cy.assumeRole into beforeEach) is the right correction.
Three things worth a look:
-
Video artifact upload is now a no-op (line 196–200): With
video: falseincypress.config.ts, the on-failure artifact upload step will always produce an empty archive. The config comment tells developers to re-enable video for debugging, but they'd also need to remember to re-enable this upload step — or the step should be removed/guarded until video is turned back on. -
Admin-UI-Cypressifcondition is asymmetric (line 146): Themerge_groupexclusion is present onbuildandprepare-matrixbut missing here. Behavior is correct (the job is auto-skipped when its dependencies are skipped), but the inconsistency is worth cleaning up for readability. -
Tarball contents /
admin-ui/public/libverification (line 135): Bothadmin-ui/public/libandfides-js/distare included. Worth confirming that extracting them in the matrix runners (intoclients/) correctly satisfies all runtime path expectations ofnpm run start, particularly whether the FidesJS assets need to be present inpublic/libat serve time or are already embedded in the Next.js build output.
|
@claude Good catch. Commented out the video artifact upload step with a note, since |
- add merge_group exclusion to Admin-UI-Cypress job for consistency - comment out video artifact upload step (no-op with video: false) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
64b39f7 to
78af8c1
Compare
78af8c1 to
3412cac
Compare
fde95a2 to
351c5c9
Compare
Ticket ENG-3233
Description Of Changes
Optimize the Admin UI Cypress CI workflow to reduce wall-time and total compute. Each of the 5 parallel matrix jobs was independently running
npm ci(~1m20s) andnext build(~4min), duplicating ~6 min of build work per job (~30 min total wasted compute). The actual Cypress test execution was only 4-7 min per shard.Code Changes
buildjob that runsnpm ci+ builds once, compresses artifacts into a tarball, and uploads for matrix jobs to download - eliminates redundant build workbuildandprepare-matrixin parallel since they have no dependency on each other, shaving ~30s of serial waitit(/test(occurrences) with file-size-based bin-packing, which better correlates with actual execution time since heavier tests have more intercepts, fixtures, and setup codebuildjobSteps to Confirm
buildjob completes and uploads thecypress-buildartifactnpm ciornext buildstepsmain(target: ~8 min vs current ~13 min)Pre-Merge Checklist
CHANGELOG.mdupdatedmaindowngrade()migration is correct and works