Switch nightly upgrade job to nightly-4.22-upgrade-from-stable-4.22 variant#985
Switch nightly upgrade job to nightly-4.22-upgrade-from-stable-4.22 variant#985rioliu-rh wants to merge 1 commit intoopenshift:mainfrom
Conversation
…ariant The stable-4.22-upgrade-from-stable-4.22 job was incorrectly used for nightly upgrade testing. It sets OPENSHIFT_INSTALL_EXPERIMENTAL_DISABLE_IMAGE_POLICY to empty string, causing unsigned nightly images to fail signature verification during upgrade. Point to the new dedicated nightly variant.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
WalkthroughA single line in the release job configuration file is updated, changing one upgrade task's Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~2 minutes ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@_releases/ocp-4.22-test-jobs-amd64.json`:
- Line 20: Confirm the referenced job variant exists before merging: verify the
openshift/release PR that introduces the "nightly-4.22-upgrade-from-stable-4.22"
variant has been merged to main and that the Prow configuration exposes the
variant, then either (a) proceed with this change if the job
"periodic-ci-openshift-openshift-tests-private-release-4.22-automated-release-nightly-4.22-upgrade-from-stable-4.22-aws-ipi-f999"
resolves successfully in Prow job listings, or (b) revert this replacement and
keep the previous stable variant until the openshift/release change is merged
and visible; check the openshift/release PR and the Prow job config to confirm
the variant name "nightly-4.22-upgrade-from-stable-4.22" is present.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: a1aa3824-cdab-40ce-880f-68025ecfdc6b
📒 Files selected for processing (1)
_releases/ocp-4.22-test-jobs-amd64.json
| }, | ||
| { | ||
| "prowJob": "periodic-ci-openshift-openshift-tests-private-release-4.22-automated-release-stable-4.22-upgrade-from-stable-4.22-aws-ipi-f999", | ||
| "prowJob": "periodic-ci-openshift-openshift-tests-private-release-4.22-automated-release-nightly-4.22-upgrade-from-stable-4.22-aws-ipi-f999", |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Check if the new nightly upgrade job variant exists in openshift/release
# Search for the new job definition in openshift/release repository
gh api repos/openshift/release/contents/ci-operator/jobs/openshift/openshift-tests-private \
--jq '.[].name' 2>/dev/null | \
grep -E 'release-4.22.*nightly-4.22-upgrade-from-stable-4.22' || \
echo "Job definition not found in main branch"
# Also check recent PRs that might add this job
gh pr list --repo openshift/release \
--search "nightly-4.22-upgrade-from-stable-4.22 in:title,body" \
--state all \
--limit 5 \
--json number,title,state,mergedAtRepository: openshift/release-tests
Length of output: 305
🌐 Web query:
nightly-4.22-upgrade-from-stable-4.22 openshift job variant
💡 Result:
The "nightly-4.22-upgrade-from-stable-4.22" appears to be a specific job variant name in OpenShift's Prow CI system for testing upgrades within the 4.22 release stream, such as from a stable 4.22 payload to a nightly 4.22 payload. However, no direct matches for this exact variant were found in search results or job histories. Similar existing job variants include: - nightly-4.22-upgrade-from-stable-4.21-e2e-metal-ovn-single-node-rt-upgrade (e.g., result 1) - periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-aws-upgrade-ovn-single-node (e.g., result 12) - ci-4.22-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade (e.g., result 8, 60) - nightly-4.22-upgrade-from-stable-4.21-e2e-metal-ipi-upgrade-ovn-ipv6 (e.g., result 13, 59) These jobs test upgrades from stable 4.21 to nightly 4.22 on platforms like AWS, metal, with OVN networking, single-node, etc. The pattern follows OpenShift CI naming: periodic-ci-[org]-[repo]-[branch]-[variant]-[test], where "nightly-4.22-upgrade-from-stable-4.21-[platform/test]" is common for release validation (e.g., results 3, 16 show upgrades from 4.21 in release tags). OpenShift CI configurations are defined in https://github.com/openshift/release/tree/main/ci-operator/config/openshift/release (result 58), with variants like "nightly-4.22" (result 15). A same-version upgrade job (stable-to-nightly 4.22) may exist for micro-updates or analysis but was not explicitly located. For exact config, check https://config.ci.openshift.org or Prow job history at https://prow.ci.openshift.org/job-history/gs/test-platform-results/jobs?org=openshift-release&repo=main&variant=nightly-4.22-upgrade-from-stable-4.22.
Citations:
- 1: https://prow.ci.openshift.org/job-history/test-platform-results/logs/openshift-origin-30608-nightly-4.22-upgrade-from-stable-4.21-e2e-metal-ovn-single-node-rt-upgrade
- 2: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-analysis-all/2029530144976146432
- 3: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasetag/4.22.0-0.nightly-2026-03-23-022245
- 4: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade/2031714043349176320
- 5: https://medium.com/@adarsh.kumar.yadav1/understanding-the-openshift-release-repository-from-beginner-to-advanced-a634ac5759a5
- 6: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-aws-upgrade-ovn-single-node/2026423404709548032
- 7: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-metal-ipi-upgrade-ovn-ipv6/2026953067173253120
- 8: https://config.ci.openshift.org/config?branch=master&org=openshift&repo=release&variant=nightly-4.10
- 9: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasetag/4.22.0-0.nightly-2026-03-02-153725
- 10: https://github.com/openshift/release/tree/main/ci-operator/config
- 11: https://prow.ci.openshift.org/job-history/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-metal-ipi-upgrade-ovn-ipv6
- 12: https://prow.ci.openshift.org/job-history/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade
🌐 Web query:
openshift/release PR nightly-4.22-upgrade-from-stable-4.22
💡 Result:
No pull request titled or numbered "nightly-4.22-upgrade-from-stable-4.22" exists in the openshift/release repository. The string appears to reference a Prow CI job name pattern used for testing upgrades in OpenShift 4.22 nightly releases, such as "periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade". These jobs test upgrades from stable-4.21 to nightly 4.22, but no exact match for "from-stable-4.22" was found, likely because intra-version (nightly to stable 4.22) upgrade tests follow different naming or are not standard for the same minor version. OpenShift 4.22 is in active development with ongoing nightly builds as of 2026-04-03, visible in release tags like 4.22.0-0.nightly-2026-03-23-022245 showing upgrade tests from prior 4.22 nightlies and 4.21 stables.
Citations:
- 1: https://github.com/openshift/release/pulls
- 2: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-analysis-all/2029530144976146432
- 3: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasetag/4.22.0-0.nightly-2026-03-23-022245
- 4: OCPBUGS-78771: Branch Sync release-4.22 to release-4.21 [03-11-2026] ovn-kubernetes#3050
- 5: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-aws-upgrade-ovn-single-node/2026423404709548032
- 6: https://prow.ci.openshift.org/job-history/test-platform-results/logs/openshift-origin-30608-nightly-4.22-upgrade-from-stable-4.21-e2e-metal-ovn-single-node-rt-upgrade
- 7: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-metal-ipi-upgrade-ovn-ipv6/2026953067173253120
- 8: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.22.0-0.nightly/release/4.22.0-0.nightly-2026-02-23-223725
🌐 Web query:
openshift-tests-private nightly 4.22 upgrade job
💡 Result:
The "openshift-tests-private nightly 4.22 upgrade job" refers to OpenShift CI nightly testing jobs for version 4.22 that incorporate private or QE (Quality Engineering) extended tests from the openshift-tests-private suite. openshift-tests-private is an internal/private repository containing additional E2E test cases used in OpenShift CI, often migrated to component repositories like cloud-credential-operator for extension testing. No single job is explicitly named "openshift-tests-private nightly 4.22 upgrade job". Instead, relevant nightly upgrade jobs for 4.22 include: - periodic-ci-openshift-release-main-nightly-4.22-upgrade-analysis-all #2029530144976146432: Analyzes upgrade test results across configurations. View logs/artifacts: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-analysis-all/2029530144976146432 - periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-aws-upgrade-ovn-single-node #2026423404709548032: Upgrade test from 4.21 to 4.22 on AWS single-node OVN. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-aws-upgrade-ovn-single-node/2026423404709548032 - periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-vsphere-ovn-upgrade #2026952825568759808: CI upgrade test on vSphere OVN. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-vsphere-ovn-upgrade/2026952825568759808 These jobs run extended tests, including those originating from openshift-tests-private (e.g., CCO tests migrated in openshift/cloud-credential-operator#961). For full job history, search Prow: https://prow.ci.openshift.org/?job=.*4\.22.*upgrade.* or OpenShift CI search: https://search.ci.openshift.org/. Recent 4.22 nightly payloads (e.g., 4.22.0-0.okd-scos-nightly-2026-04-01-024551) show mixed upgrade test results, with some AWS upgrades failing after retries.
Citations:
- 1: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-analysis-all/2029530144976146432
- 2: https://origin-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.22.0-0.okd-scos-nightly/release/4.22.0-0.okd-scos-nightly-2026-04-01-024551
- 3: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-vsphere-ovn-upgrade/2026952825568759808
- 4: CCO-767: test-ext: Migrate openshift-tests-private E2E cases cloud-credential-operator#961
- 5: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-metal-ipi-upgrade-ovn-ipv6/2030446781250146304
- 6: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.21-e2e-aws-upgrade-ovn-single-node/2026423404709548032
- 7: https://docs.ci.openshift.org/internals/artifacts/
- 8: https://prow.ci.openshift.org/
Verify the new job variant exists before merging.
The PR description states this change depends on an openshift/release PR that adds the new nightly-4.22-upgrade-from-stable-4.22 job variant. This specific job variant was not found in publicly available OpenShift CI Prow job histories or recent pull requests. Merging this configuration change before confirming the job definition exists in openshift/release will cause job execution failures.
Confirm that:
- The openshift/release PR adding
nightly-4.22-upgrade-from-stable-4.22has been merged to the main branch - The job variant is visible in the Prow job configuration
The change itself logically improves consistency by using a nightly-specific upgrade variant in the nightly array, but the dependency must be resolved first.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@_releases/ocp-4.22-test-jobs-amd64.json` at line 20, Confirm the referenced
job variant exists before merging: verify the openshift/release PR that
introduces the "nightly-4.22-upgrade-from-stable-4.22" variant has been merged
to main and that the Prow configuration exposes the variant, then either (a)
proceed with this change if the job
"periodic-ci-openshift-openshift-tests-private-release-4.22-automated-release-nightly-4.22-upgrade-from-stable-4.22-aws-ipi-f999"
resolves successfully in Prow job listings, or (b) revert this replacement and
keep the previous stable variant until the openshift/release change is merged
and visible; check the openshift/release PR and the Prow job config to confirm
the variant name "nightly-4.22-upgrade-from-stable-4.22" is present.
|
@rioliu-rh: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Summary
Update
_releases/ocp-4.22-test-jobs-amd64.jsonto use the new dedicated nightly upgrade variant job instead of the stable-to-stable job for nightly upgrade testing.Problem
The nightly section was referencing
automated-release-stable-4.22-upgrade-from-stable-4.22-aws-ipi-f999, which setsOPENSHIFT_INSTALL_EXPERIMENTAL_DISABLE_IMAGE_POLICY: "". This caused upgrade failures when testing stable-to-nightly paths because nightly images are unsigned and the image policy enforcement blocked theopenshift-config-operatorpod from rolling out.Changes
periodic-ci-openshift-openshift-tests-private-release-4.22-automated-release-stable-4.22-upgrade-from-stable-4.22-aws-ipi-f999withperiodic-ci-openshift-openshift-tests-private-release-4.22-automated-release-nightly-4.22-upgrade-from-stable-4.22-aws-ipi-f999in the nightly sectionDependency
Depends on openshift/release PR openshift/release#77382 adding the new variant.
Jira
https://redhat.atlassian.net/browse/OCPERT-352