Skip to content

Improve build-test loop for AI agents#25310

Open
mokagio wants to merge 16 commits intotrunkfrom
ainfra-1963-refine-claudemd-in-wordpress-ios-with-build-test-loop-2
Open

Improve build-test loop for AI agents#25310
mokagio wants to merge 16 commits intotrunkfrom
ainfra-1963-refine-claudemd-in-wordpress-ios-with-build-test-loop-2

Conversation

@mokagio
Copy link
Contributor

@mokagio mokagio commented Feb 26, 2026

Description

Supersedes #25242 to address @kean 's experience trying to run tests.

The core difference is that AGENTS.md instructs to prioritize Xcode's MCP.

image

Testing instructions

I retried all of Alex's experiments.

Test WordPressData

Run into an issue. The MCP couldn't find the target/scheme to run

Screenshot 2026-02-27 at 9 29 42 AM

Successfully fell back to Fastlane, but no luck there either. Claude then discovered the scheme and used xcodebuild:

image

Action Item:

  • Set up straightforward target-scheme-xctestplan for WordPressData

Test WordPressShared

Again, the first MCP call failed

image

Claude then opted for xcodebuild given it worked before. It deprioritized the instruction in AGENTS.md. I let it run to see how it worked. Unfortunately, it kept running into linker issues.

image

Retried with explicit instruction to exhaust all MCP options first. Got stuck on this:

image

Cancelled and looked at Xcode on my own. Test run fine. I did a few more iterations of this test and figured out the issue is with how SwiftPM autogenerates the schemes Xcode uses for packages.

I worked around this by creating a new scheme for WordPressShared in the project container so that it won't be accidentally modified:

image

However, despited having the scheme in place Claude did something funky here running the tests one at a time, but at least it worked and it was relatively fast.

Screenshot 2026-02-27 at 10 54 12 AM image

Make a small change to WordPressShared that would result in an error

I changed a test so that it would fail. MCP test failed to detect it

image

Only picked up the failure when instructed

image

Run all tests (without reverting the failing test)

image image

I haven't investigated the expected failure or not run tests.

Run all tests (after reverting the intentional failure)

image

@dangermattic
Copy link
Collaborator

1 Message
📖 This PR is still a Draft: some checks will be skipped.

Generated by 🚫 Danger

@wpmobilebot
Copy link
Contributor

wpmobilebot commented Feb 26, 2026

App Icon📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.
App NameJetpack
ConfigurationRelease-Alpha
Build Number31211
VersionPR #25310
Bundle IDcom.jetpack.alpha
Commiteb00410
Installation URL3ajhu4c7cdr3g
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot
Copy link
Contributor

wpmobilebot commented Feb 26, 2026

App Icon📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.
App NameWordPress
ConfigurationRelease-Alpha
Build Number31211
VersionPR #25310
Bundle IDorg.wordpress.alpha
Commiteb00410
Installation URL2nuhn32k9jrf8
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot
Copy link
Contributor

wpmobilebot commented Feb 26, 2026

🤖 Build Failure Analysis

This build has failures. Claude has analyzed them - check the build annotations for details.

mokagio and others added 14 commits February 27, 2026 09:20
CI handles builds and tests via Fastlane;
the `xcodebuild`-wrapping Rake tasks were unused.

---

Generated with the help of Claude Code, https://claude.ai/code

Co-Authored-By: Claude Code Opus 4.6 <noreply@anthropic.com>
Skips CI prerequisites (env files, signing, toolkit checks) and
reuses DerivedData for incremental builds. Supports only_testing,
scheme, device, and clean options.

---

Generate with the help of Claude Code, https://code.claude.com

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---

Generate with the help of Claude Code, https://code.claude.com

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The file was in a `WordPressUITests` directory with no matching
SPM target, so it was never compiled or run.
Updated its import from `WordPress` to `WordPressUI`.

---

Generate with the help of Claude Code, https://code.claude.com

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applies the same test-bundle workaround already used elsewhere
in WordPressUI so the storyboard loads in SPM test context.

---

Generate with the help of Claude Code, https://code.claude.com

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test calendar used `.eastern` but the service used
`.current`, so date interval keys never aligned.

---

Generate with the help of Claude Code, https://code.claude.com

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test crashes when DesignSystem is built as a dynamic
library, which happens because WordPressAuthenticator
(also in the plan) depends on it transitively.

---

Generate with the help of Claude Code, https://code.claude.com

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use `Bundle.designSystemBundle` instead of `.module` so
resources resolve when Xcode builds the module as a
dynamic library (same workaround as WordPressUI).

---

Generate with the help of Claude Code, https://code.claude.com

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prefer Xcode MCP when available, fall back to Fastlane.

---

Generated with the help of Claude Code, https://claude.ai/code

Co-Authored-By: Claude Code Opus 4.6 <noreply@anthropic.com>
When `clean:false` (the default), packages are already resolved
from a previous build.
Passing `-skipPackageResolving` via `skip_package_dependencies_resolution`
avoids redundant resolution on incremental test runs.

---

Generated with the help of Claude Code, https://claude.ai/code

Co-Authored-By: Claude Code Opus 4.6 <noreply@anthropic.com>
@mokagio mokagio force-pushed the ainfra-1963-refine-claudemd-in-wordpress-ios-with-build-test-loop-2 branch from 5e7bd4f to 918c849 Compare February 26, 2026 22:22
This way, we can properly configure it and SwiftPM won't clobber it from
the .swiftpm folder at its own discretion.
@sonarqubecloud
Copy link

@mokagio mokagio marked this pull request as ready for review February 27, 2026 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants