GitHub App for Automated AI Security Testing (looking for community feedback)

[ksiabani]

Hi everyone 👋

We’d love to get your thoughts on an idea we’re exploring: a GitHub App that helps projects like Company Research Agent stay secure as they grow.

What it would do

1. Scan & detect risks against the OWASP LLM Top 10 and OWASP Agentic Top 10.
2. Map the agentic system: generate a graph of agents, prompts, and connected tools.
3. Simulate & attack: run contextual adversarial prompts/conversations in a sandbox.
4. Record findings: share attack transcripts for any successful exploits.
5. Suggest mitigations: prompt-hardening changes.
6. Reference standards: tie recommendations back to OWASP guidelines.

Additional notes

• It would run periodically (e.g., weekly) to keep results fresh.
• It would always open a PR with findings + proposed fixes.
• It would be free for OSS projects and stay that way.

Why this could help Company Research Agent

• Reduce the burden on maintainers by getting ready-to-review security PRs.
• Give the community a transparent view of security posture with reproducible reports and fixes.
• Keep the project resilient against evolving AI threats without requiring extra setup.
• Build even more trust with users by showing security is continuously tested and improved.
• Comply with recommendations from major LLM providers (OpenAI, Google, Microsoft, etc.) for thorough testing of GenAI apps, including Red Teaming (see OpenAI’s guidance).

👉 Would this be useful for the Company Research Agent community?
👉 Are there specific risks or areas you’d like the app to cover?

We’d love to co-design this with you. Your input will shape whether and how we bring this to life here.