Visual Regression

Part of Mobile DevTools — open-source tools for mobile engineering teams.

Automated visual regression testing for mobile apps. Capture screenshots from two builds, pixel-diff them, and catch UI changes before they ship.

Baseline Build          Current Build
     |                       |
     v                       v
 [Revyl Device]         [Revyl Device]
     |                       |
     v                       v
 screenshots/            screenshots/
 baseline/               current/
     |                       |
     +----------+------------+
                |
           pixel diff
                |
                v
         report.html
         report.md

How It Works

1. Capture -- Boots cloud devices via Revyl CLI, navigates through a configurable list of screens, and saves a screenshot of each one.
2. Diff -- Compares each screenshot pair pixel-by-pixel, highlights changed regions, and calculates a similarity percentage.
3. Report -- Generates an HTML report with side-by-side comparisons and a Markdown summary for PR comments.
4. Gate -- Exits with code 1 if any screen differs by more than the configured threshold (default 0.1%).

Quick Start

Prerequisites

• A Revyl account with an API key
• Python 3.10+
• A mobile app build (APK for Android, .app bundle for iOS)

1. Set secrets

export REVYL_API_KEY="your-api-key"
export REVYL_ORG_ID="your-org-id"

For GitHub Actions, add these as repository secrets along with your app binary.

2. Install dependencies

pip install -r requirements.txt

3. Create your app and upload a build

# Create an app on Revyl
revyl app create --name "MyApp" --platform android --json

# Upload your APK
revyl build upload --skip-build --platform android --app "$APP_ID" --file app.apk --json --yes

4. Define your screens

Edit screens.yaml to list the screens you want to compare. Use natural language to describe tap targets -- Revyl's AI resolves them to coordinates.

screens:
  - name: "home"
    description: "Main home screen"
    steps: []

  - name: "settings"
    description: "Settings page"
    steps:
      - action: tap
        target: "Settings icon"
    reset: true

5. Capture and compare

# Capture baseline screenshots
python scripts/capture.py \
  --platform android \
  --app-id $BASELINE_APP_ID \
  --output-dir ./baseline \
  --screens ./screens.yaml

# Capture current build screenshots
python scripts/capture.py \
  --platform android \
  --app-id $CURRENT_APP_ID \
  --output-dir ./current \
  --screens ./screens.yaml

# Compare
python scripts/diff.py \
  --baseline ./baseline \
  --current ./current \
  --output ./report \
  --threshold 0.1

6. Review the report

Open report/report.html in a browser for the full visual report, or check report/report.md for a text summary.

Examples

Both examples compare builds of Bug Bazaar, a demo e-commerce app, against the same green baseline.

---

Example 1: Structural Layout Changes (5–15% diff)

Layout shifts with no color changes — the kind of regression that's invisible in code review but immediately obvious in a pixel diff.

What changed:

• Product detail: image area shrunk, price/name order swapped, buy button moved from sticky footer to inline
• Search: "Popular Specimens" and "Trending Searches" sections swapped, trending changed from horizontal pills to vertical list
• Account: stats moved above profile card, changed from 3 horizontal boxes to full-width vertical rows

  [FAIL] account                         7.1243% diff  (21,031 px)
  [FAIL] product_detail                  15.2835% diff  (45,117 px)
  [FAIL] search                          5.7930% diff  (17,101 px)

Product Detail — Image box shrunk, price moved above name, buy button moved from sticky footer into scroll content:

Baseline	Current	Diff Overlay

Search — "Popular Specimens" and "Trending Searches" sections swapped; trending changed from horizontal pills to a vertical list:

Baseline	Current	Diff Overlay

Account — Stats moved above profile card and changed from 3 small horizontal boxes to full-width vertical rows:

Baseline	Current	Diff Overlay

---

Example 2: Theme + Text Changes (0.2–15% diff)

A rebrand that changes the primary color from green to purple and updates copy — same layout, different paint.

  [FAIL] account                         0.7395% diff  (2,183 px)
  [FAIL] cart                            1.7009% diff  (5,021 px)
  [FAIL] filter_results                  14.8276% diff  (43,771 px)
  [FAIL] product_detail                  0.1792% diff  (529 px)
  [FAIL] search                          0.1687% diff  (498 px)
  [FAIL] shop_home                       14.6274% diff  (43,180 px)

Shop Home — Hero banner changed from green to purple, text changed from "Rare Species Collection" to "Premium Bug Marketplace":

Baseline	Current	Diff Overlay

Product Detail — "ADD TO CART" button changed to "BUY NOW":

Baseline	Current	Diff Overlay

---

How to Read the Diff Overlay

Changed pixels are highlighted in red on top of the current screenshot. Anti-aliased pixels (font rendering, subpixel edges) are shown in yellow and excluded from the diff count. The status bar (clock, battery) is masked out automatically so it never triggers false positives.

GitHub Actions

The included workflow (.github/workflows/visual-regression.yml) runs automatically on pull requests:

1. On push to main: captures baseline screenshots and stores them as an artifact.
2. On pull request: captures current screenshots, diffs against the baseline, and posts a summary comment on the PR.

Required Secrets

Secret	Description
REVYL_API_KEY	Your Revyl API key
REVYL_ORG_ID	Your Revyl organization ID
REVYL_APP_ID	The Revyl app ID for your mobile app

Customization

Adjusting the Threshold

The --threshold flag on diff.py controls the maximum percentage of pixels that can differ before a screen is marked as failing.

# Strict: catch even tiny rendering differences
python scripts/diff.py --baseline ./baseline --current ./current --threshold 0.01

# Lenient: allow dynamic content variation
python scripts/diff.py --baseline ./baseline --current ./current --threshold 5.0

Adding Screens

Edit screens.yaml to add more screens. Each screen needs:

• A unique name (used as the screenshot filename)
• Navigation steps to reach the screen from the app's starting state
• Optional reset: true to return to the home screen after capture

Supported Actions

Action	Fields	Description
tap	target	Tap a UI element described in natural language
type	target, text	Type text into a field
swipe	target, direction	Swipe on an element (up/down/left/right)
go-home	--	Press the device home button
wait	duration	Pause for a number of seconds

Platform Support

Both Android and iOS are supported. Set --platform android or --platform ios on capture.py.

Project Structure

visual-regression/
  scripts/
    capture.py      # Screenshot capture via Revyl CLI
    diff.py         # Pixel diffing and report generation
  screens.yaml      # Screen navigation config
  requirements.txt  # Python dependencies
  .github/
    workflows/
      visual-regression.yml  # CI workflow

Built With

• Revyl CLI -- Cloud device provisioning and AI-grounded mobile interaction
• pixelmatch -- Anti-aliasing aware pixel comparison (same algorithm as Playwright/Storybook)
• Pillow -- Image processing
• Claude Code Action -- AI agent for CI automation