Skip to content

PR automation testing (2)#8

Closed
chanwoo7 wants to merge 10 commits intomainfrom
ci/pr-ai-description-v2
Closed

PR automation testing (2)#8
chanwoo7 wants to merge 10 commits intomainfrom
ci/pr-ai-description-v2

Conversation

@chanwoo7
Copy link
Member

@chanwoo7 chanwoo7 commented Mar 14, 2026

PR Summary

GitHub Actions 기반 PR 설명 자동생성 도구를 추가합니다. OpenAI(langsmith 래퍼 포함)를 사용해 변경된 코드 diff를 요약하고 검증된 JSON 스키마로 PR 본문에 삽입하거나 PR 제목/라벨/담당자를 업데이트하는 스크립트와 유닛 테스트, dry-run 도구 및 CodeRabbit 설정을 포함합니다. 민감 정보 마스킹, diff 절단, 라벨 필터링 등 안전장치가 구현되어 있으며 CI 워크플로우와 package.json 의존성 업데이트가 포함됩니다.

  • PR diff 수집(compare API / fallback git), 민감정보 마스킹, byte-limited diff 생성 로직 추가
  • OpenAI 호출(requestOpenAiSummary) 및 JSON 스키마 검증(validateAiSummaryJson) 구현
  • PR 본문에 삽입할 마커 블록(renderSummaryBlock / upsertSummaryBlock) 및 제목/라벨/담당자 적용 로직 추가
  • 단위 테스트(.github/scripts/tests/**)와 dry-run(.github/scripts/pr-ai-dry-run.mjs) 도구 포함
  • .coderabbit.yaml 에 코드리뷰/파일별 가이드라인 추가
  • CI 워크플로우(.github/workflows/pr-ai-description.yml) 추가 및 package.json에 langsmith/openai 의존성 추가

Changes

8 files changed

File Changes
.coderabbit.yaml CodeRabbit 설정 파일을 추가했습니다. 리뷰 프로필, 경로별 리뷰 지침(path_instructions)과 사전 병합 검사(pre_merge_checks) 등을 정의하여 NestJS/GraphQL/Prisma/테스트/워크플로우에 대한 리뷰 규칙을 포함합니다.
.github/scripts/__tests__/pr-ai-description-lib.spec.mjs pr-ai-description-lib.mjs의 유닛 테스트를 추가했습니다. 파일 필터링, diff 절단, JSON 스키마 검증, 마커 블록 삽입/교체, 라벨 필터링 등의 동작을 검증합니다.
.github/scripts/__tests__/pr-ai-description.spec.mjs pr-ai-description.mjs의 유닛 테스트를 추가했습니다. OpenAI 응답 처리, 에러 변환, PR 본문 갱신 로직, 최신 PR 재조회 실패 시 fallback 동작 등을 테스트합니다.
.github/scripts/pr-ai-description-lib.mjs PR 생성/업데이트에 필요한 도우미 라이브러리를 추가했습니다. 파일 glob 처리, diff 직렬화/절단, 민감정보 마스킹, JSON 스키마 정의 및 검증, 마커 블록 렌더링/업서트, 라벨 필터링 등 핵심 유틸을 구현합니다.
.github/scripts/pr-ai-description.mjs 메인 실행 스크립트를 추가했습니다. GitHub API/로컬 git으로 diff 수집(compare API 우선), OpenAI 호출 및 응답 검증, PR 본문/제목/라벨/담당자 적용, 에러/타임아웃 처리 및 step summary 기록 로직을 포함합니다.
.github/scripts/pr-ai-dry-run.mjs 실제 PR 업데이트 없이 동일한 흐름으로 요약을 생성하여 콘솔에 출력하는 dry-run 도구를 추가했습니다(로컬/PR 모드 모두 지원).
.github/workflows/pr-ai-description.yml Pull Request 이벤트에 반응하는 GitHub Actions 워크플로우를 추가했습니다. 체크아웃, Node 설치, 의존성 설치, 유닛 테스트, AI PR 설명 생성 스텝을 포함하며 필요한 시크릿/변수를 사용하도록 구성되어 있습니다.
package.json 개발 의존성에 langsmith와 openai를 추가했습니다(버전: langsmith ^0.5.7, openai ^6.27.0).

Impact

  • PR 생성/동기화 시 자동으로 AI 요약을 생성하여 PR 본문에 삽입하거나 제목/라벨/담당자를 업데이트합니다(권한 및 시크릿 필요).
  • CI 실행 시간이 증가할 수 있으며 OpenAI 호출 비용 및 Langsmith 추적 사용이 발생합니다.
  • 레포 라벨 목록과 비교해 AI가 제안한 라벨만 적용하므로 라벨 정책에 영향이 있습니다.
  • 코드/로그에 민감 정보가 노출되지 않도록 마스킹 로직이 추가되었으나, 워크플로우와 스크립트의 시크릿 사용 설정을 정확히 해야 합니다.

Checklist

  • 워크플로우(.github/workflows/pr-ai-description.yml)의 GITHUB_TOKEN 및 다른 env 값들이 올바르게 설정되어 있는지(형식/중복 중괄호 등 문법 오류 없음) 확인
  • pr-ai-description.mjs의 createGitHubRequest에서 Authorization 헤더가 githubToken을 사용하도록 'Authorization: Bearer ${githubToken}' 또는 적절한 형식으로 구현되어 있는지 확인
  • 코드에 남아있는 [REDACTED] 플레이스홀더가 실제 시크릿/환경변수로 대체되었는지 확인(커밋된 시크릿이 없는지 검증)
  • 유닛 테스트(.github/scripts/tests)를 로컬 및 CI에서 실행하여 모두 통과하는지 확인
  • OpenAI 및 Langsmith 관련 의존성(package.json)이 설치되었는지와 lockfile(예: yarn.lock/pnpm-lock.json)이 업데이트되었는지 확인
  • 민감정보 마스킹(maskSensitiveContent) 로직이 프로젝트 특성에 맞게 충분히 커버되는지 검토(Authorization, Bearer, password/secret/token 등)
  • upsertSummaryBlock 동작(기존 CodeRabbit summary 보존/교체) 및 shouldApplyTitle 로직(ai-title-lock 레이블 처리)을 실제 PR 시나리오로 검증
  • compare API 실패 시 git fallback 흐름이 정상 동작하는지(권한/페치/깊이 옵션 등) dry-run으로 확인

Dependencies

  • langsmith ^0.5.7
  • openai ^6.27.0

@coderabbitai
Copy link

coderabbitai bot commented Mar 14, 2026

📝 Walkthrough

Walkthrough

This PR introduces an automated AI-powered GitHub PR description feature. It adds OpenAI integration via GitHub Actions to generate PR summaries, labels, and titles from code diffs. The implementation includes configuration files, utility libraries for diff processing and validation, test suites, a dry-run script, and a GitHub Actions workflow that orchestrates the entire pipeline.

Changes

Cohort / File(s) Summary
Configuration & Dependencies
.coderabbit.yaml, package.json
Adds CodeRabbit review automation configuration with path-specific review instructions (NestJS, GraphQL, migrations, CI/CD, tests) and introduces langsmith and openai npm dependencies.
Test Suites
.github/scripts/__tests__/pr-ai-description-lib.spec.mjs, .github/scripts/__tests__/pr-ai-description.spec.mjs
Comprehensive test coverage for PR AI utilities and workflows, validating diff filtering, truncation, schema validation, PR body updates, error handling, and OpenAI integration scenarios.
Core Implementation
.github/scripts/pr-ai-description-lib.mjs
Utility library providing diff normalization, exclusion-glob matching, diff size-aware truncation, sensitive content masking, AI response schema validation, PR summary block rendering, and label/title handling.
Workflow Orchestration
.github/scripts/pr-ai-description.mjs, .github/scripts/pr-ai-dry-run.mjs
Main workflow script orchestrating PR data retrieval, diff collection/masking, OpenAI prompt generation, response handling, and autonomous PR updates (assignees, labels, title/body); dry-run utility for testing both PR-based and branch-based flows.
GitHub Actions
.github/workflows/pr-ai-description.yml
Workflow definition triggering on PR events (opened, reopened, synchronize, ready_for_review), running tests and invoking the AI description generation script with OpenAI and Langsmith configuration.

Sequence Diagram(s)

sequenceDiagram
    participant GHA as GitHub Actions
    participant GH as GitHub API
    participant OAI as OpenAI API
    participant PR as Pull Request
    
    GHA->>GH: Fetch PR metadata & repository labels
    GHA->>GH: Retrieve diff (via compare endpoint or git)
    GHA->>GHA: Filter & normalize diff files
    GHA->>GHA: Mask sensitive content in diff
    GHA->>OAI: Send prompt with diff & repository labels
    OAI-->>GHA: Return JSON summary (title, labels, changes)
    GHA->>GHA: Validate response against schema
    GHA->>GHA: Render summary block & determine label/title changes
    GHA->>GH: Update PR body (insert/replace summary block)
    GHA->>GH: Apply labels (filtered against known labels)
    GHA->>GH: Update PR title (if applicable)
    GHA->>GHA: Write step summary with results
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested labels

⚙️ Setting, ✅ Test, ⭐️ Feature

Poem

🐰 Hoppy news for automation,
A rabbit built with dedication,
OpenAI now writes PR tales,
While masked secrets never fail,
GitHub Actions hops along—
The future's diff-erently strong! 🚀

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 5.56% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive No pull request description was provided by the author, making it impossible to assess whether the description relates to the changeset. Add a pull request description that explains the purpose, scope, and key changes introduced by this PR.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The PR title clearly summarizes the main change: adding a CI workflow and scripts for automated PR AI description generation.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch ci/pr-ai-description-v2
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions bot added ⚙️ Setting 개발환경 세팅 ✅ Test 테스트 관련 labels Mar 14, 2026
@github-actions github-actions bot changed the title Ci/pr ai description v2 ci: PR AI 설명 생성 워크플로우 및 스크립트 추가 Mar 14, 2026
@coderabbitai coderabbitai bot added the ⭐️ Feature 기능 개발 label Mar 14, 2026
@chanwoo7 chanwoo7 added 🌏 Deploy 배포 관련 and removed ⭐️ Feature 기능 개발 labels Mar 14, 2026
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/scripts/pr-ai-description-lib.mjs:
- Around line 401-416: In maskSensitiveContent expand the redaction regexes to
cover common env-var and multiline secret shapes: add case-insensitive patterns
for VAR-style keys (e.g.,
/^[A-Z0-9_+-]*?(?:API[_-]?KEY|OPENAI[_-]?API[_-]?KEY|CLIENT[_-]?SECRET|REFRESH[_-]?TOKEN|PRIVATE[_-]?KEY|SECRET|TOKEN|PASSWORD|APIKEY)\s*[:=]\s*.+$/im)
and key/value forms with underscores/hyphens and mixed case, and add a multiline
PEM block matcher that redacts anything between "-----BEGIN [A-Z ]+-----" and
"-----END [A-Z ]+-----" (with s/m flags). Apply these new replace calls in
maskSensitiveContent (keep existing Authorization/Bearer handling) and add
regression tests that assert redaction for env-var lines like
"OPENAI_API_KEY=...", "client_secret=...", "refresh_token=...", "PRIVATE_KEY"
PEM blocks, and multiline secrets to prevent regressions.
- Around line 536-585: The renderSummaryBlock function currently writes
model-provided fields directly into the PR block; sanitize all AI-sourced
strings (e.g., summary.summary, summary.summaryBullets items, each
summary.changes[].description and .file, summary.impact items, summary.checklist
items, and arrays passed to renderOptionalSection like
summary.breakingChanges/relatedIssues/dependencies) by stripping any occurrences
of MARKER_START and MARKER_END, collapsing newlines into single spaces, trimming
whitespace, and normalizing/escaping pipe characters (e.g., replace '\n' with '
', remove marker substrings, and then escape '|' as '\|') before rendering so
upsertSummaryBlock cannot be prematurely terminated or inject multiline table
cells.

In @.github/scripts/pr-ai-description.mjs:
- Around line 216-237: The current loadCurrentPullRequest silently falls back to
the webhook snapshot on fetch failure which lets stale title/body be used for
PATCHes; change loadCurrentPullRequest to return null (or throw a specific
error) when fetchPullRequest fails for write-related flows so callers can detect
failure and skip applying title/body updates; update the caller applyPrUpdates
(and any other call sites such as the similar block around lines 992-1005) to
check for a null result from loadCurrentPullRequest before calling
buildPrUpdatePayload and skip the title/body patch when null, preserving other
safe updates and logging a clear warning.
- Around line 428-449: The prompt currently instructs the model to emit
"changes" for each changed file even when buildLimitedDiff() (check
diff-truncation-meta) may have omitted files, which causes hallucinated per-file
entries; update the construction that uses prMeta/repositoryLabels/diffText so
the prompt: 1) checks diff-truncation-meta (or a flag in prMeta) and, if omitted
files are indicated, explicitly tells the model to only describe files present
in diffText and not to invent any entries for missing files, 2) instructs the
model to add a top-level note (e.g., "partial_diff": true) or an explicit
sentence in the output when the diff is truncated, and 3) preserves original
identifiers like prMeta and diffText in the prompt while emphasizing "do NOT
fabricate descriptions for files not shown." Ensure the change targets the
string-building code that concatenates prMeta/repositoryLabels/diffText (the
array being .join('\n')) so the prompt text is updated accordingly.

In @.github/workflows/pr-ai-description.yml:
- Around line 39-51: The "Generate AI PR description" step (run: node
.github/scripts/pr-ai-description.mjs) will fail for Dependabot because
Dependabot PRs have a read-only GITHUB_TOKEN and no Actions secrets; update the
workflow to skip or use a read-only fallback when github.actor ==
'dependabot[bot]': add an if condition to the job/step (e.g., if: github.actor
!= 'dependabot[bot]') or modify the step to detect missing OPENAI_API_KEY/GITHUB
secrets and short-circuit (or run a non-writing readonly mode) so the script in
.github/scripts/pr-ai-description.mjs does not attempt to access OPENAI_API_KEY
or write labels/title/body when invoked by dependabot.
- Around line 23-51: The workflow currently checks out and runs PR branch code
(the "Generate AI PR description" step invoking
.github/scripts/pr-ai-description.mjs) while passing secrets (OPENAI_API_KEY,
LANGSMITH_API_KEY, GITHUB_TOKEN) — split this into two workflows: keep a safe
unprivileged pull_request workflow that does not run checked-out PR scripts or
expose secrets (remove the checkout/run of pr-ai-description.mjs, limit
permissions, run only tests/lint), and create a separate privileged workflow
triggered only on pull_request_target or workflow_run that checks out the
trusted base branch (use github.event.pull_request.base.sha when checking out)
and runs .github/scripts/pr-ai-description.mjs with the required secrets and
write-scoped GITHUB_TOKEN; ensure the privileged workflow explicitly sets
minimal permissions and avoids checking out untrusted head code.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2b3a2ef7-1e33-4ff2-8207-3352fc168185

📥 Commits

Reviewing files that changed from the base of the PR and between 64808a2 and 3cbc7f2.

⛔ Files ignored due to path filters (1)
  • yarn.lock is excluded by !**/yarn.lock, !**/*.lock
📒 Files selected for processing (8)
  • .coderabbit.yaml
  • .github/scripts/__tests__/pr-ai-description-lib.spec.mjs
  • .github/scripts/__tests__/pr-ai-description.spec.mjs
  • .github/scripts/pr-ai-description-lib.mjs
  • .github/scripts/pr-ai-description.mjs
  • .github/scripts/pr-ai-dry-run.mjs
  • .github/workflows/pr-ai-description.yml
  • package.json

Comment on lines +401 to +416
export function maskSensitiveContent(text) {
if (typeof text !== 'string' || text.length === 0) {
return '';
}

return text
.replaceAll(/(Authorization\s*[:=]\s*)([^\n\r]+)/gi, '$1[REDACTED]')
.replaceAll(/Bearer\s+[A-Za-z0-9\-._~+/]+=*/gi, 'Bearer [REDACTED]')
.replaceAll(
/(\"?(?:password|secret|token|apiKey)\"?\s*[:=]\s*)\"([^\"\n\r]*)\"/gi,
'$1"[REDACTED]"',
)
.replaceAll(
/(\"?(?:password|secret|token|apiKey)\"?\s*[:=]\s*)([^\s,\n\r]+)/gi,
'$1[REDACTED]',
);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

The redaction pass misses common secret shapes.

These regexes only catch bare password|secret|token|apiKey keys plus Authorization/Bearer patterns. Values like OPENAI_API_KEY=..., client_secret=..., refresh_token=..., PRIVATE_KEY=..., or PEM blocks survive unchanged and are then sent in the masked diff payload. Please broaden the matcher before any network call and add regression cases for env-var and multiline-key formats.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/scripts/pr-ai-description-lib.mjs around lines 401 - 416, In
maskSensitiveContent expand the redaction regexes to cover common env-var and
multiline secret shapes: add case-insensitive patterns for VAR-style keys (e.g.,
/^[A-Z0-9_+-]*?(?:API[_-]?KEY|OPENAI[_-]?API[_-]?KEY|CLIENT[_-]?SECRET|REFRESH[_-]?TOKEN|PRIVATE[_-]?KEY|SECRET|TOKEN|PASSWORD|APIKEY)\s*[:=]\s*.+$/im)
and key/value forms with underscores/hyphens and mixed case, and add a multiline
PEM block matcher that redacts anything between "-----BEGIN [A-Z ]+-----" and
"-----END [A-Z ]+-----" (with s/m flags). Apply these new replace calls in
maskSensitiveContent (keep existing Authorization/Bearer handling) and add
regression tests that assert redaction for env-var lines like
"OPENAI_API_KEY=...", "client_secret=...", "refresh_token=...", "PRIVATE_KEY"
PEM blocks, and multiline secrets to prevent regressions.

Comment on lines +536 to +585
export function renderSummaryBlock(summary) {
const lines = [MARKER_START];

// Summary
lines.push('### PR Summary');
lines.push(summary.summary);
if (summary.summaryBullets.length > 0) {
lines.push('');
for (const bullet of summary.summaryBullets) {
lines.push(`- ${bullet}`);
}
}

// Changes (테이블)
lines.push('');
lines.push('### Changes');
lines.push(`> ${summary.changes.length} files changed`);
lines.push('');
lines.push('| File | Changes |');
lines.push('|------|---------|');
for (const change of summary.changes) {
const escapedDesc = change.description.replaceAll('|', '\\|');
lines.push(`| \`${change.file}\` | ${escapedDesc} |`);
}

// Impact
if (summary.impact.length > 0) {
lines.push('');
lines.push('### Impact');
for (const item of summary.impact) {
lines.push(`- ${item}`);
}
}

// Checklist (체크박스)
if (summary.checklist.length > 0) {
lines.push('');
lines.push('### Checklist');
for (const item of summary.checklist) {
lines.push(`- [ ] ${item}`);
}
}

// 선택적 섹션들
lines.push(...renderOptionalSection('Breaking Changes', summary.breakingChanges));
lines.push(...renderOptionalSection('Related Issues', summary.relatedIssues));
lines.push(...renderOptionalSection('Dependencies', summary.dependencies));

lines.push(MARKER_END);
return lines.join('\n');
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Sanitize model text before embedding your own markers.

renderSummaryBlock() writes model-provided strings directly into the block. If the model echoes <!-- pr-ai-summary:end --> or multiline table content, the next upsertSummaryBlock() can terminate early and leave stale tail content in the PR body. Strip MARKER_START/MARKER_END from all AI fields and normalize table cells to single-line text before rendering.

💡 Suggested guard
+function sanitizeSummaryText(value) {
+  return String(value)
+    .replaceAll(MARKER_START, '[pr-ai-summary:start]')
+    .replaceAll(MARKER_END, '[pr-ai-summary:end]');
+}
...
-  lines.push(summary.summary);
+  lines.push(sanitizeSummaryText(summary.summary));
...
-    const escapedDesc = change.description.replaceAll('|', '\\|');
-    lines.push(`| \`${change.file}\` | ${escapedDesc} |`);
+    const escapedFile = sanitizeSummaryText(change.file)
+      .replaceAll(/\r?\n/g, ' ')
+      .replaceAll('`', '\\`')
+      .replaceAll('|', '\\|');
+    const escapedDesc = sanitizeSummaryText(change.description)
+      .replaceAll(/\r?\n/g, ' ')
+      .replaceAll('|', '\\|');
+    lines.push(`| \`${escapedFile}\` | ${escapedDesc} |`);
🧰 Tools
🪛 ESLint

[error] 580-580: Replace ...renderOptionalSection('Breaking·Changes',·summary.breakingChanges) with ⏎····...renderOptionalSection('Breaking·Changes',·summary.breakingChanges),⏎··

(prettier/prettier)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/scripts/pr-ai-description-lib.mjs around lines 536 - 585, The
renderSummaryBlock function currently writes model-provided fields directly into
the PR block; sanitize all AI-sourced strings (e.g., summary.summary,
summary.summaryBullets items, each summary.changes[].description and .file,
summary.impact items, summary.checklist items, and arrays passed to
renderOptionalSection like summary.breakingChanges/relatedIssues/dependencies)
by stripping any occurrences of MARKER_START and MARKER_END, collapsing newlines
into single spaces, trimming whitespace, and normalizing/escaping pipe
characters (e.g., replace '\n' with ' ', remove marker substrings, and then
escape '|' as '\|') before rendering so upsertSummaryBlock cannot be prematurely
terminated or inject multiline table cells.

Comment on lines +216 to +237
export async function loadCurrentPullRequest({
githubRequest,
owner,
repo,
number,
fallbackPullRequest,
onWarn = logWarn,
}) {
try {
const latestPullRequest = await fetchPullRequest(githubRequest, owner, repo, number);

if (latestPullRequest && typeof latestPullRequest === 'object') {
return latestPullRequest;
}
} catch (error) {
onWarn('failed to fetch latest pull request. fallback to event payload.', {
status: error?.status,
message: error?.message ?? 'unknown-error',
});
}

return fallbackPullRequest;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Stale event payload should not become the source of truth for a PATCH.

When the reload fails, loadCurrentPullRequest() returns the webhook snapshot. applyPrUpdates() then feeds that stale body/title into buildPrUpdatePayload() and can overwrite edits that were made after the workflow started. For write paths, return null/an error here and skip the title/body update instead of patching from the fallback snapshot.

💡 Safer write path
 export async function loadCurrentPullRequest({
   githubRequest,
   owner,
   repo,
   number,
   fallbackPullRequest,
   onWarn = logWarn,
 }) {
   try {
     const latestPullRequest = await fetchPullRequest(githubRequest, owner, repo, number);

     if (latestPullRequest && typeof latestPullRequest === 'object') {
       return latestPullRequest;
     }
   } catch (error) {
     onWarn('failed to fetch latest pull request. fallback to event payload.', {
       status: error?.status,
       message: error?.message ?? 'unknown-error',
     });
   }

-  return fallbackPullRequest;
+  return null;
 }
           const latestPullRequest = await loadCurrentPullRequest({
             githubRequest,
             owner,
             repo,
             number: prNumber,
             fallbackPullRequest: pullRequest,
           });
+          if (!latestPullRequest) {
+            await writeStepSummary(
+              '- PR body/title update skipped because the latest PR body could not be reloaded.',
+            );
+            return {
+              assigneesAdded,
+              labelsAdded,
+              unknownLabelsIgnoredCount,
+              titleUpdated: false,
+              bodyUpdated: false,
+            };
+          }
           const { updatedBody, nextTitle, bodyUpdated } = buildPrUpdatePayload({
             currentPullRequest: latestPullRequest,
             fallbackPullRequest: pullRequest,
             block,
             applyTitle,

Also applies to: 992-1005

🧰 Tools
🪛 ESLint

[error] 225-225: Replace githubRequest,·owner,·repo,·number with ⏎······githubRequest,⏎······owner,⏎······repo,⏎······number,⏎····

(prettier/prettier)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/scripts/pr-ai-description.mjs around lines 216 - 237, The current
loadCurrentPullRequest silently falls back to the webhook snapshot on fetch
failure which lets stale title/body be used for PATCHes; change
loadCurrentPullRequest to return null (or throw a specific error) when
fetchPullRequest fails for write-related flows so callers can detect failure and
skip applying title/body updates; update the caller applyPrUpdates (and any
other call sites such as the similar block around lines 992-1005) to check for a
null result from loadCurrentPullRequest before calling buildPrUpdatePayload and
skip the title/body patch when null, preserving other safe updates and logging a
clear warning.

Comment on lines +428 to +449
return [
'다음 Pull Request 정보를 기반으로 한국어 PR 요약 JSON을 생성하세요.',
'',
'규칙:',
'- title은 conventional commits 형식의 prefix를 포함하세요 (feat:, fix:, refactor:, docs:, style:, perf:, test:, build:, ci:, chore:, revert: 중 택1). 예: "feat: 사용자 인증 기능 추가"',
'- summary는 PR 전체를 요약하는 한국어 텍스트를 작성하세요.',
'- summaryBullets는 핵심 변경사항을 간결한 한국어 리스트 항목으로 작성하세요. 불필요하면 빈 배열로 두세요.',
'- changes는 변경된 각 파일별로 { file, description } 형태로 작성하세요. file은 파일 경로 원문, description은 해당 파일의 변경 내용을 한국어 문장으로 작성하세요.',
'- impact는 이 PR이 시스템에 미치는 영향을 자유 형식 한국어 텍스트 리스트로 작성하세요. 영향이 없으면 빈 배열로 두세요.',
'- checklist는 리뷰어가 꼭 확인해야 할 항목들을 한국어로 작성하세요. 애매하거나 추상적인 항목은 피하고, 객관적이고 명확한 행동 지침을 간결하게 작성하세요.',
'- breakingChanges는 하위 호환성을 깨는 변경사항을 작성하세요. 없으면 빈 배열로 두세요.',
'- relatedIssues는 관련 이슈/PR을 "#번호 설명" 형태로 작성하세요. 없으면 빈 배열로 두세요.',
'- dependencies는 추가/제거/업데이트된 패키지 의존성을 작성하세요. 없으면 빈 배열로 두세요.',
'- labels는 아래 제공된 레포 라벨 목록에서만 선택하세요. 최대 3개까지 선택할 수 있으나, 꼭 필요한 경우가 아니면 1-2개로 제한하는 것을 권장합니다.',
'- 코드 식별자/파일 경로/에러 메시지는 원문을 유지하세요.',
'',
`PR Meta:\n${JSON.stringify(prMeta, null, 2)}`,
'',
`Repository Labels:\n${JSON.stringify(repositoryLabels, null, 2)}`,
'',
`Diff:\n${diffText}`,
].join('\n');
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don't ask for per-file descriptions when the diff may be truncated.

buildLimitedDiff() can omit files, but this prompt still tells the model to emit changes for "each changed file". Once diff-truncation-meta says some files were omitted, the model has no source for those descriptions and will hallucinate entries that then get written back into the PR body.

💡 Suggested prompt tweak
-    '- changes는 변경된 각 파일별로 { file, description } 형태로 작성하세요. file은 파일 경로 원문, description은 해당 파일의 변경 내용을 한국어 문장으로 작성하세요.',
+    '- changes는 Diff 섹션에 실제로 포함된 파일만 대상으로 { file, description } 형태로 작성하세요.',
+    '- diff-truncation-meta가 있으면 omittedFiles에 해당하는 파일 내용은 추측하지 말고, 필요하면 summary/impact에서 일부 파일이 생략되었다고만 명시하세요.',
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/scripts/pr-ai-description.mjs around lines 428 - 449, The prompt
currently instructs the model to emit "changes" for each changed file even when
buildLimitedDiff() (check diff-truncation-meta) may have omitted files, which
causes hallucinated per-file entries; update the construction that uses
prMeta/repositoryLabels/diffText so the prompt: 1) checks diff-truncation-meta
(or a flag in prMeta) and, if omitted files are indicated, explicitly tells the
model to only describe files present in diffText and not to invent any entries
for missing files, 2) instructs the model to add a top-level note (e.g.,
"partial_diff": true) or an explicit sentence in the output when the diff is
truncated, and 3) preserves original identifiers like prMeta and diffText in the
prompt while emphasizing "do NOT fabricate descriptions for files not shown."
Ensure the change targets the string-building code that concatenates
prMeta/repositoryLabels/diffText (the array being .join('\n')) so the prompt
text is updated accordingly.

Comment on lines +23 to +51
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 1

- name: Setup Node.js (24.x)
uses: actions/setup-node@v4
with:
node-version: '24.x'

- name: Install dependencies
run: yarn install --immutable

- name: Run PR AI helper unit tests
run: node --test .github/scripts/__tests__/*.spec.mjs

- name: Generate AI PR description
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_MODEL: ${{ vars.OPENAI_MODEL }}
PR_AI_MAX_DIFF_BYTES: ${{ vars.PR_AI_MAX_DIFF_BYTES }}
PR_AI_MAX_FILES: ${{ vars.PR_AI_MAX_FILES }}
LANGSMITH_TRACING: ${{ vars.LANGSMITH_TRACING }}
LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
LANGSMITH_ENDPOINT: ${{ vars.LANGSMITH_ENDPOINT }}
LANGSMITH_PROJECT: ${{ vars.LANGSMITH_PROJECT }}
LANGSMITH_WORKSPACE_ID: ${{ vars.LANGSMITH_WORKSPACE_ID }}
run: node .github/scripts/pr-ai-description.mjs
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Don't execute PR-branch code in the secret-bearing job.

This job checks out the PR branch, runs repository code, and then invokes .github/scripts/pr-ai-description.mjs with OPENAI_API_KEY, LANGSMITH_API_KEY, and write scopes on the job token. A same-repo PR can change the checked-out scripts/tests or mutate the workspace before the final step, which gives that PR a path to exfiltrate the referenced secrets or abuse the token. GitHub's runner security docs explicitly call out pull_request from a branch within the repository as a case where attackers can steal repository secrets or use the job's GITHUB_TOKEN. Split this into an unprivileged pull_request test workflow and a separate privileged workflow that only runs trusted base-branch code. (docs.github.com)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/pr-ai-description.yml around lines 23 - 51, The workflow
currently checks out and runs PR branch code (the "Generate AI PR description"
step invoking .github/scripts/pr-ai-description.mjs) while passing secrets
(OPENAI_API_KEY, LANGSMITH_API_KEY, GITHUB_TOKEN) — split this into two
workflows: keep a safe unprivileged pull_request workflow that does not run
checked-out PR scripts or expose secrets (remove the checkout/run of
pr-ai-description.mjs, limit permissions, run only tests/lint), and create a
separate privileged workflow triggered only on pull_request_target or
workflow_run that checks out the trusted base branch (use
github.event.pull_request.base.sha when checking out) and runs
.github/scripts/pr-ai-description.mjs with the required secrets and write-scoped
GITHUB_TOKEN; ensure the privileged workflow explicitly sets minimal permissions
and avoids checking out untrusted head code.

Comment on lines +39 to +51
- name: Generate AI PR description
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_MODEL: ${{ vars.OPENAI_MODEL }}
PR_AI_MAX_DIFF_BYTES: ${{ vars.PR_AI_MAX_DIFF_BYTES }}
PR_AI_MAX_FILES: ${{ vars.PR_AI_MAX_FILES }}
LANGSMITH_TRACING: ${{ vars.LANGSMITH_TRACING }}
LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
LANGSMITH_ENDPOINT: ${{ vars.LANGSMITH_ENDPOINT }}
LANGSMITH_PROJECT: ${{ vars.LANGSMITH_PROJECT }}
LANGSMITH_WORKSPACE_ID: ${{ vars.LANGSMITH_WORKSPACE_ID }}
run: node .github/scripts/pr-ai-description.mjs
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Dependabot PRs will fail here.

pull_request runs triggered by Dependabot get a read-only GITHUB_TOKEN and no Actions secrets. This step requires OPENAI_API_KEY, and the script later tries to write labels/title/body back to the PR, so Dependabot updates will fail instead of being skipped. Add an explicit github.actor != 'dependabot[bot]' guard or a read-only fallback path. (docs.github.com)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/pr-ai-description.yml around lines 39 - 51, The "Generate
AI PR description" step (run: node .github/scripts/pr-ai-description.mjs) will
fail for Dependabot because Dependabot PRs have a read-only GITHUB_TOKEN and no
Actions secrets; update the workflow to skip or use a read-only fallback when
github.actor == 'dependabot[bot]': add an if condition to the job/step (e.g.,
if: github.actor != 'dependabot[bot]') or modify the step to detect missing
OPENAI_API_KEY/GITHUB secrets and short-circuit (or run a non-writing readonly
mode) so the script in .github/scripts/pr-ai-description.mjs does not attempt to
access OPENAI_API_KEY or write labels/title/body when invoked by dependabot.

@chanwoo7 chanwoo7 closed this Mar 15, 2026
@chanwoo7 chanwoo7 changed the title ci: PR AI 설명 생성 워크플로우 및 스크립트 추가 PR automation testing (2) Mar 15, 2026
@chanwoo7 chanwoo7 deleted the ci/pr-ai-description-v2 branch March 15, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🌏 Deploy 배포 관련 ⚙️ Setting 개발환경 세팅 ✅ Test 테스트 관련

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant