Enable k8s client cache for Secrets and ConfigMaps in the single namespace mode

[tfujiwar]

WHAT

• Allow the controller to watch Secrets / ConfigMaps in the single namespace mode.
• Enable the k8s API client cache for Secrets / ConfigMaps in the single namespace mode.

WHY

To reduce latency of runner pod creation (#3276).

The EphemeralRunnerReconciler retrieves a JITConfig Secret for every reconciliation once the Secret is created. It can be a performance bottleneck because the k8s API client cache is disabled for Secrets.

The cache has been disabled for Secrets because it requires cluster-wide list/watch permissions in the default mode. But in the single namespace mode, we can narrow down the permissions only to the single namespace and the controller namespace, which would be acceptable.

This change is aligned with ADR 2023-04-11: Limit Permissions for Service Accounts in Actions-Runner-Controller.

In this mode, you will end up with a manager Role that has all Get/List/Create/Delete/Update/Patch/Watch permissions on resources we need, and a RoleBinding to bind the Role with the controller ServiceAccount in the watched single namespace and the controller namespace

TESTING

We have tested the change with a controller that manages 300-400 EphemeralRunners at peak times. We confirmed that the duration of getting k8s Secrets from the EphemeralRunnerReconciler has decreased due to the caching.

We also confirmed that the number of Pending EphemeralRunners has decreased because they are reconciled faster.

Before

After

[Copilot]

Pull Request Overview

This PR enables caching of Secrets and ConfigMaps in the Kubernetes API client when the controller operates in single namespace mode, improving performance by reducing reconciliation latency for EphemeralRunners.

Key Changes:

• Conditionally enables client cache for Secrets/ConfigMaps based on watch mode
• Adds list/watch permissions for Secrets and ConfigMaps in single namespace RBAC roles
• Updates test assertions to reflect the additional RBAC rules

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
main.go	Moves cache disable configuration into a conditional block that only applies when not in single namespace mode
charts/gha-runner-scale-set-controller/tests/template_test.go	Updates expected rule counts to account for new Secret and ConfigMap permissions
charts/gha-runner-scale-set-controller/templates/manager_single_namespace_watch_role.yaml	Adds list/watch permissions for Secrets and ConfigMaps in the watched namespace
charts/gha-runner-scale-set-controller/templates/manager_single_namespace_controller_role.yaml	Adds list/watch permissions for Secrets and ConfigMaps in the controller namespace

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Okabe-Junya]

@nikola-jokic @rentziass

Friendly reminder, can you take a look when you have time?
I can take over this PR. Thanks in advance!