Skip to content

Spark: Fix wap branch metadata delete on the wrong branch#15592

Open
yingjianwu98 wants to merge 3 commits intoapache:mainfrom
yingjianwu98:yingjianw/fix_spark_wap_branch_metadata_delete
Open

Spark: Fix wap branch metadata delete on the wrong branch#15592
yingjianwu98 wants to merge 3 commits intoapache:mainfrom
yingjianwu98:yingjianw/fix_spark_wap_branch_metadata_delete

Conversation

@yingjianwu98
Copy link
Contributor

@yingjianwu98 yingjianwu98 commented Mar 12, 2026

SparkTable.deleteWhere() and canDeleteWhere() do not resolve the WAP (Write-Audit-Publish) branch from the session configuration. When a user sets spark.wap.branch and performs a metadata-level delete, both the scan and the commit incorrectly target the main branch instead of the WAP branch.

FordeleteWhere(), I think it's a recent regression from this PR. The new change doesn't resolve to the right branch in the new extension rule for Spark 4.1. I don't see an easy way to resolve this issues in the resolveBranch Rule as I believe OptimizeMetadataOnlyDeleteFromTable optimizer rule will discard ResolveBranch rule and then trigger deleteWhere() which currently doesn't respect the wap branch if specified.

For canDeleteWhere(), this is an existing issue across all spark version and I have been working on a fix here
#15512

This causes two issues:

  • canDeleteWhere() scans the wrong branch's files, so it cannot correctly determine whether the delete can be resolved at the metadata level
  • deleteWhere() commits the delete to main instead of the WAP branch, silently dropping the delete

Fix:

  • Use determineReadBranch() in canDeleteWhere() to scan the correct branch
  • Use determineWriteBranch() in deleteWhere() to commit to the correct branch

@github-actions github-actions bot added the spark label Mar 12, 2026
@yingjianwu98 yingjianwu98 changed the title Yingjianw/fix spark wap branch metadata delete Spark: Fix wap branch metadata delete on the wrong branch Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant