Skip to content

chore(deps): update dependency promptfoo to v0.121.3#512

Open
renovate[bot] wants to merge 1 commit intomainfrom
renovate/promptfoo-0.x-lockfile
Open

chore(deps): update dependency promptfoo to v0.121.3#512
renovate[bot] wants to merge 1 commit intomainfrom
renovate/promptfoo-0.x-lockfile

Conversation

@renovate
Copy link
Copy Markdown
Contributor

@renovate renovate bot commented Mar 27, 2026

This PR contains the following updates:

Package Change Age Confidence
promptfoo 0.121.2 → 0.121.3 age confidence

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

Release Notes

promptfoo/promptfoo (promptfoo)

v0.121.3

Compare Source

Features
Bug Fixes

Configuration

📅 Schedule: Branch creation - "before 10am on friday" in timezone Europe/London, Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

@renovate renovate bot added dependencies Renovatebot and dependabot updates frontend javascript Pull requests that update javascript code labels Mar 27, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 27, 2026 •
edited
Loading

Open in Overmind ↗

model|risks_v6
✨Encryption Key State Risk ✨KMS Key Creation

🔴 Change Signals

Routine 🔴 ▇▅▃▂▁ Multiple API server resources showing unusual update activity at 1 event/week for the last 2 months to 3 months, which is infrequent compared to typical patterns.
Policies 🔴 ▃▂▁ Multiple S3 buckets are missing server-side encryption and required tags, while security groups allow SSH access from anywhere, which is unusual compared to typical patterns.

View signals ↗

🔥 Risks

Replacing the API server will change its private network identity and break IP-based internal routing ‼️High Open Risk ↗
The change replaces the API server instance 540044833068.eu-west-2.ec2-instance.i-0ea52e0eb6476a224, which is currently identified inside the VPC by primary ENI eni-065db8208d75f1855, private IP 10.0.101.142, and the IP-derived hostname ip-10-0-101-142.eu-west-2.compute.internal. In the replacement plan those network identity fields become (known after apply) rather than being pinned, and the IP-based target group attachment for the server is also being replaced.

That means the new instance can come up with a different primary ENI and private IP even if the Elastic IP is re-associated successfully. Anything that currently reaches this server by 10.0.101.142 or by ip-10-0-101-142.eu-west-2.compute.internal will lose its target, and the current healthy target registration on 10.0.101.142:9090 must be torn down and recreated. This creates a real service-discovery and routing disruption risk during the replacement, independent of the pre-existing security-group findings.

Single-instance API cutover will create a no-backend window during replacement and make rollback another risky endpoint rebinding ‼️High Open Risk ↗
The production API cutover is coupled to a single EC2 instance replacement. 540044833068.eu-west-2.ec2-instance.i-0ea52e0eb6476a224 is being recreated while the Elastic IP 13.134.236.98 is moved to the new instance and the private IP target-group attachment is replaced. In the current state, that same instance owns the public EIP and is also the healthy ip target 10.0.101.142:9090 behind the internal health target group, so this deployment has no overlapping backend for the public endpoint.

When the change applies, external access and private target registration both depend on the replacement instance booting correctly, running user data successfully, reassociating the EIP, and becoming healthy before traffic resumes. If the new AMI or startup path is bad, the service will lose its public entrypoint and its private health target at the same time, and rollback will require another full instance rebuild and rebinding cycle rather than a fast traffic flip to a stable alternate target set.

🧠 Reasoning · ✔ 2 · ✖ 2

EC2 hardening, AMI change, DeleteOnTermination data loss, and ENI/DNS disruption risk

Observations 5

Hypothesis

EC2 instance configuration and lifecycle changes introduce security and data loss risk if not aligned with hardening and storage best practices:

  • Security group sg-0437857de45b640ce allows SSH from 0.0.0.0/0, violating compute hardening guidance (SEC06-BP03) and exposing the instance to internet-wide SSH brute-force and remote access attacks.
  • The instance AMI is changed from ami-01b1bd34... to ami-094a6f8d..., and hibernation is set to null; if the new AMI is not a hardened, fully patched image, this introduces security risk (SEC06-BP02) and may change runtime behavior.
  • EBS volume vol-090e750179b5fa681 attached to EC2 instance i-09d6479fb9b97d123 has DeleteOnTermination=true. Instance replacement during updates can trigger volume deletion and data loss if it holds important state.
  • Replacing the instance that owns ENI eni-065db8208d75f1855 can alter ENI attachment and its private IP (10.0.101.142), breaking internal DNS records such as ip-10-0-101-142.eu-west-2.compute.internal and disrupting internal service discovery and routing.
  • Additional security groups (e.g., sg-0fe38b77fda090133) with open egress (0.0.0.0/0 all protocols) further weaken network boundaries and may facilitate data exfiltration or uncontrolled external access.
    Mitigations: lock down SSH (bastion/VPN, restricted CIDRs); ensure AMIs are hardened and patched; reevaluate DeleteOnTermination on data-bearing volumes; ensure ENI/DNS dependencies are designed for instance replacement; and tighten egress rules to least privilege.

Investigation

I treated the hypothesis as four concern areas: security exposure from permissive networking and unhardened storage, data loss on instance replacement, and service disruption from replacing the API server instance. I first loaded the relevant organizational knowledge and AWS guidance. That knowledge makes two current-state findings unambiguous: EC2 instances must not be directly reachable from the internet, and SSH open to 0.0.0.0/0 is a high-severity violation; EBS volumes must be encrypted at rest. Blast-radius data confirms sg-0437857de45b640ce currently allows TCP/22 from 0.0.0.0/0, and both queried root volumes (vol-090e750179b5fa681 and vol-0f33e19b9daa1a2d7) are Encrypted: false. However, those are pre-existing compliance issues, not risks introduced by this particular change, because the instance diff does not modify those security groups or volume encryption settings.

For the change-specific parts, I checked the replaced instance i-0ea52e0eb6476a224, its ENI, Elastic IP, root volume, target health, and related planned changes. The current instance has primary ENI eni-065db8208d75f1855, private IP 10.0.101.142, private DNS ip-10-0-101-142.eu-west-2.compute.internal, and an Elastic IP 13.134.236.98. The plan replaces the instance and shows primary_network_interface_id, private_ip, private_dns, and public_ip as (known after apply), while the separate EIP resource changes its instance association from the old instance ID to (known after apply). AWS documentation shows that an Elastic IP can be remapped to a replacement instance, so the public IP exposure itself is not the main risk. But the internal hostname in this VPC is IP-based (hostname_type: ip-name), and AWS documents that IP-name private DNS is derived from the instance private IPv4 address. Because this replacement does not pin the primary ENI or private IP in the instance diff, Terraform is free to create a new primary ENI and assign a different private IP. That means the existing internal DNS name ip-10-0-101-142.eu-west-2.compute.internal and the health target registered by IP (10.0.101.142:9090) can both change as part of the replacement.

This is reinforced by the blast radius: the health target for 10.0.101.142:9090 is currently healthy, and the target group attachment resource for the API server is itself being replaced, which is consistent with IP-based registration needing to move to a new address. So even though the hypothesis's exact mechanism about the ENI itself being preserved or not is uncertain from the diff, the broader concern is real: this change replaces an instance that is currently addressed by a concrete private IP and IP-derived internal DNS name, while downstream consumers currently reference that exact IP/DNS identity. If anything inside the environment depends on 10.0.101.142 or ip-10-0-101-142.eu-west-2.compute.internal, those references will break during and after the replacement. I did not find strong evidence that DeleteOnTermination will cause additional data loss beyond loss of the disposable root volume, because the only volume attached to the replaced instance is its root volume and no separate persistent data volume is shown. I also did not find evidence that the new AMI is definitely unhardened; that part remains speculative without AMI provenance in the plan. The concrete, change-induced risk is internal connectivity disruption from replacing an IP-addressed EC2 target.

✔ Hypothesis proven

Single-instance public/API cutover pattern increasing downtime and rollback risk

Observations 4

Hypothesis

A tightly coupled cutover pattern ties a single EC2 instance to both the public entrypoint (EIP/public DNS) and its private load balancer target identity, making deployments and rollbacks high-risk and increasing blast radius:

  • module.api_access[0].aws_instance.api_server is replaced while the Elastic IP 540044833068.eu-west-2.ec2-address.13.134.236.98 is reassociated and the IP-based target-group attachment is replaced, so any delay in instance boot, user-data execution, EIP reassociation, or target registration can leave the service with no healthy backend and interrupt external access.
  • Rolling back a bad AMI requires another full instance recreation and rebinding of both the EIP and target group, so recovery is another fragile endpoint cutover rather than a simple traffic flip to a stable alternative target set, increasing MTTR and risk of repeated failures.
  • The change simultaneously alters the instance’s public identity (EIP/public DNS) and private target identity, but IaC only manages known resources; unmodeled consumers (monitoring, scripts, partner integrations, operator runbooks) that pin to the old instance ID, public DNS, or private IP can fail during or after cutover.
  • Moving the public EIP directly onto the replacement instance means the new AMI receives internet traffic immediately, without validation behind a private-only staging target or separate stable front door, widening the blast radius of a bad image or misconfiguration and weakening the security boundary.
    Mitigations: introduce a stable front door (e.g., ALB/CloudFront, DNS name decoupled from individual instance identity), use blue/green or create-before-destroy deployment patterns with overlapping capacity, keep rollback paths that flip traffic between versioned target sets instead of rebuilding endpoints, and inventory external consumers to ensure they follow stable service names rather than literal host identities.

Investigation

I treated the hypothesis as a reliability and cutover-risk concern, then checked the relevant organizational guidance first. The internal high-availability guidance explicitly says to flag insufficient pre-provisioned capacity and deployments applied to all instances simultaneously, and the network guidance says exposing individual EC2 instance public IPs directly is an anti-pattern. The planned changes show exactly that pattern here: the production API server 540044833068.eu-west-2.ec2-instance.i-0ea52e0eb6476a224 is being replaced, the Elastic IP 540044833068.eu-west-2.ec2-address.13.134.236.98 is updated to point to the replacement instance, and the IP-based target-group attachment github.com/overmindtech/terraform-example.aws_lb_target_group_attachment.module.api_access[0].aws_lb_target_group_attachment.api_server_ip is also replaced. There is no evidence in the plan of overlapping capacity for this public endpoint or of a blue/green target set.

The current blast radius confirms the coupling the hypothesis described. The EIP is attached directly to the instance’s primary ENI and private IP 10.0.101.142, and the internal health target group api-health-terraform-example uses target type ip with that exact private IP as its only observed healthy target. AWS documentation supports the underlying mechanics: an Elastic IP association is moved between resources rather than abstracted behind a stable compute fleet, and target groups register specific instance IDs or ip addresses depending on target type. That means replacing the instance necessarily forces rebinding of both the public identity and the private target identity. Because the changed instance is the endpoint itself, any boot, user-data, app-start, EIP reassociation, or target-registration delay leaves no pre-validated backend serving that identity. The hypothesis’s broader point about rollback fragility is also supported: rollback would require another instance recreation and another EIP/target reattachment, not a simple traffic flip to an already-warm alternate target set. I did not find evidence of safeguards in this change that would make this specific replacement path low-risk, so the downtime/cutover risk is real.

✔ Hypothesis proven

Load balancer target rotation and EC2 updates causing temporary availability loss

Observations 25

Hypothesis

Multiple load balancer target group attachment replacements and EC2 instance changes can temporarily deregister or make backends unhealthy, causing health check failures and traffic disruption. Specifically:

  • Replacing target group attachments for IP 10.0.101.142 (including port 9090) in NLB/ALB target groups (e.g., api-health-terraform-example, api-207c90ee-alb) will deregister and re-register the target, causing short windows of backend unavailability, temporary UnHealthy health status, failed health checks, CloudWatch alarms, and brief service disruption for APIs and other services on that IP/ENI.
  • New observations reiterate that this deregistration can disrupt API server traffic, degrade ALB-observed availability, and potentially impact dependent connections (e.g., database/API clients) if not coordinated and monitored.
  • EC2 instance i-09d6479fb9b97d123, a registered ALB target (api-207c90ee-tg) serving HTTP /health on port 80, is being updated (Status: updated). Any replacement, networking change, or application change that causes it to fail health checks or stop accepting traffic reduces the ALB’s pool of healthy targets and can cause public endpoint degradation or request failures.
    Mitigations: schedule attachment replacements and instance updates during low-traffic/maintenance windows; ensure multiple healthy targets remain available; enable/verify connection draining; closely monitor ALB/NLB health metrics and alarms; and validate instance/app behavior against health checks before and after the change.

Investigation

I investigated the availability concern by first checking relevant organizational guidance on high availability and compute changes, then examining the actual planned changes and current target health. The change set does include a replaced aws_lb_target_group_attachment for the IP target and an updated EC2 instance 540044833068.eu-west-2.ec2-instance.i-09d6479fb9b97d123, which initially sounds like a deregistration risk. However, the evidence for a real outage mechanism is weak.

The current state shows both affected targets are healthy: api-207c90ee-tg has a healthy instance target i-09d6479fb9b97d123 on port 80, and api-health-terraform-example has a healthy IP target 10.0.101.142 on port 9090. The only concrete diff shown for i-09d6479fb9b97d123 is public_dns and public_ip becoming (known after apply), which is a computed-value change and not evidence of instance replacement, listener change, health check change, or application reconfiguration. The attachment replacement also has no attribute-level diff showing a port, target ID, or target group change; without that, the hypothesis reduces to a generic claim that any attachment replacement causes disruption.

I also checked AWS documentation. AWS confirms that deregistered targets enter draining and that newly registered targets go through initial health checking, so a replacement can create a short transition window in principle. But this hypothesis requires more than a generic possibility: it needs evidence that this specific change will leave too few healthy backends or force traffic loss. I found no evidence of that. The ALB target group currently has only one visible healthy instance target in blast radius, but the planned EC2 diff for that instance does not show a disruptive change, and the IP attachment being replaced is for a different target group. There is no diff showing health check settings changing, no security group or subnet change on the affected instance, no target group attribute change, and no proof that Terraform will stop the application or replace the instance. Because the concrete planned changes do not support the hypothesized failure chain, this is not a confirmed risk after investigation.

✖ Hypothesis disproven

Direct public EC2 exposure and reachability disruption via Elastic IP changes

Observations 9

Hypothesis

Changes to Elastic IP (EIP) allocation/association on public-facing EC2 network interfaces can alter or increase public exposure of instances without the protections of managed edge services and can disrupt reachability for systems that rely on stable public IPs. Specifically:

  • EIP 13.134.236.98 is associated with ENI eni-065db8208d75f1855 in a public subnet/VPC and used as an internet-facing entrypoint. Updates to this EIP can remap or disassociate the public IP, affecting inbound HTTP traffic and NAT behavior, and may expose instances directly rather than via ALB/CloudFront.
  • Public EIP use is combined with broadly permissive security groups (e.g., sg-0437857de45b640ce, sg-0159f0e3d8224d441) that allow 0.0.0.0/0 ingress to ports such as 80, 443, 1234 and unrestricted egress, violating least-privilege controls and increasing attack surface for any instance behind the EIP.
  • Updates to the EIP or its ENI association could make instances unreachable or expose unintended endpoints while still lacking WAF/DDoS or segmentation controls.
    Risk: unmanaged public EC2 endpoints with weak network controls and unstable public IP mappings, contrary to aws-network-security and public endpoint best practices (REL02-BP01, SEC05-BP01/SEC05-BP02). Mitigations: prefer ALB/CloudFront or other managed front doors with WAF; restrict SG ingress/egress; carefully review EIP re-association plans and keep DNS/whitelists aligned with any IP changes.

Investigation

I treated the concern area as two related outcomes: loss of reachability if the Elastic IP mapping changes during replacement, and increased public exposure from directly reachable EC2 instances with permissive security groups. I first checked the relevant organizational guidance. That guidance does confirm that directly reachable EC2 instances and broad 0.0.0.0/0 rules are bad practice, and in some cases critical. However, this task is to judge whether this specific change introduces a real risk in that concern area.

The actual planned change is much narrower than the hypothesis suggests. The only concrete network-related diff on the EIP 540044833068.eu-west-2.ec2-address.13.134.236.98 is that its instance field changes from i-0ea52e0eb6476a224 to (known after apply), because the attached EC2 instance 540044833068.eu-west-2.ec2-instance.i-0ea52e0eb6476a224 is being replaced after an AMI update. Current blast-radius state shows that this EIP is already attached to the instance’s primary ENI eni-065db8208d75f1855 at private IP 10.0.101.142. There is no planned change to the EIP allocation itself, no explicit reassociation to a different interface, no security-group diff weakening access, and no evidence that the public entrypoint is switching from ALB/managed edge to a newly exposed instance. The broad security groups cited in the hypothesis (sg-0437857de45b640ce, sg-0159f0e3d8224d441) are real findings in current state, but they are not attached to the replaced EIP-backed instance; that instance currently uses sg-089e5107637083db5 and sg-03cf38efd953aa056.

I also checked the broader environment context. There is already an internet-facing ALB api-207c90ee-alb and target group api-207c90ee-tg in the same VPC, which weakens the claim that this change is newly bypassing managed ingress. The hypothesis also names EIP 13.42.93.249, but that address is not part of the planned change; it is simply another existing EIP associated to a different ENI. Because the diff does not show a new public IP, a new SG exposure, or an explicit remap to a different endpoint, the evidence supports only a normal instance replacement with Terraform-computed association details. That can imply some transient disruption during replacement, but the hypothesis specifically claims risky EIP remapping/public-exposure changes, and I found no concrete evidence of such a change path here. So the concern area is directionally valid as an architectural observation, but not a substantiated risk created by this particular change.

✖ Hypothesis disproven

💥 Blast Radius

Items 107

Edges 219

github-actions[bot]
github-actions bot requested changes Mar 27, 2026
View reviewed changes
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overmind

⛔ Auto-Blocked

🔴 Decision

Auto-blocked: Routine score (-5) is below minimum (-1)

📊 Signals Summary

Routine 🔴 -5

🔥 Risks Summary

High 0 · Medium 0 · Low 0

💥 Blast Radius

Items 23 · Edges 75

View full analysis in Overmind ↗

@renovate renovate bot force-pushed the renovate/promptfoo-0.x-lockfile branch from 514a41c to 58cbaeb Compare March 27, 2026 13:24
github-actions[bot]
github-actions bot requested changes Mar 27, 2026
View reviewed changes
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overmind

⛔ Auto-Blocked

🔴 Decision

Auto-blocked: Policy signal (-3) is below threshold (-2); Routine score (-5) is below minimum (-1)

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

🔥 Risks Summary

High 0 · Medium 0 · Low 0

💥 Blast Radius

Items 5 · Edges 20

View full analysis in Overmind ↗

@renovate renovate bot force-pushed the renovate/promptfoo-0.x-lockfile branch from 58cbaeb to a1c25a9 Compare April 1, 2026 20:12
github-actions[bot]
github-actions bot requested changes Apr 1, 2026
View reviewed changes
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overmind

⛔ Auto-Blocked

🔴 Decision

Found 2 high risks requiring review

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

🔥 Risks Summary

High 2 · Medium 0 · Low 0

💥 Blast Radius

Items 107 · Edges 219

View full analysis in Overmind ↗

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] requested changes

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

dependencies Renovatebot and dependabot updates frontend javascript Pull requests that update javascript code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

0 participants