fix(redteam): add missing plugins to webui #5546
Conversation
- Add missing plugins to risk categories in metadata.ts: - cca, reasoning-dos, system-prompt-override, rag-document-exfiltration, rag-poisoning to Security & Access Control - off-topic, unverifiable-claims to Brand category - Add comprehensive test to catch future missing plugins - Ensures all defined plugins appear in the red team setup UI 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
|
⏩ No test execution environment matched (57ac382) View output ↗ |
mldangelo
changed the title
📝 WalkthroughWalkthrough
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Pre-merge checks (3 passed)✅ Passed checks (3 passed)
✨ Finishing Touches
🧪 Generate unit tests
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
src/redteam/constants/metadata.ts (1)
408-525: Avoid cross-category duplicates to keep categoryMapReverse deterministicSome plugins (e.g., hijacking) appear in multiple top-level categories. Because categoryMapReverse assigns last-wins, UI/logic depending on a single category per plugin may be inconsistent. Consider enforcing single ownership per plugin or add a test to prevent cross-category duplicates.
Suggested test addition is included below in comments for test/redteam/constants/metadata.test.ts.
Also applies to: 538-546
test/redteam/constants/metadata.test.ts (1)
78-83: Add a guard against cross-category duplicatesYou already test duplicates within a category. Add a test to ensure the same plugin isn’t assigned to multiple top-level categories (prevents last-wins in categoryMapReverse).
Apply this diff right after the current duplicate-within-category test:
it('should not include duplicate plugin ids within a category', () => { Object.entries(riskCategories).forEach(([category, plugins]) => { const uniquePlugins = new Set(plugins); expect(uniquePlugins.size).toBe(plugins.length); }); }); + + it('should not assign the same plugin id to multiple top-level categories', () => { + const seen = new Map<Plugin, string>(); + const dupes: string[] = []; + Object.entries(riskCategories).forEach(([cat, plugins]) => { + plugins.forEach((p) => { + const prev = seen.get(p); + if (prev && prev !== cat) { + dupes.push(`${p} in ${prev} and ${cat}`); + } + seen.set(p, cat); + }); + }); + expect(dupes).toEqual([]); + });
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
src/redteam/constants/metadata.ts(3 hunks)test/redteam/constants/metadata.test.ts(2 hunks)
🧰 Additional context used
📓 Path-based instructions (7)
**/*.{test,spec}.{js,ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/gh-cli-workflow.mdc)
Avoid disabling or skipping tests unless absolutely necessary and documented
Files:
test/redteam/constants/metadata.test.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/gh-cli-workflow.mdc)
Prefer not to introduce new TypeScript types; use existing interfaces whenever possible
Use TypeScript with strict type checking
Files:
test/redteam/constants/metadata.test.tssrc/redteam/constants/metadata.ts
test/**/*.{test,spec}.ts
📄 CodeRabbit inference engine (.cursor/rules/jest.mdc)
test/**/*.{test,spec}.ts: Mock as few functions as possible to keep tests realistic
Never increase the function timeout - fix the test instead
Organize tests in descriptivedescribeanditblocks
Prefer assertions on entire objects rather than individual keys when writing expectations
Clean up after tests to prevent side effects (e.g., useafterEach(() => { jest.resetAllMocks(); }))
Run tests with--randomizeflag to ensure your mocks setup and teardown don't affect other tests
Use Jest's mocking utilities rather than complex custom mocks
Prefer shallow mocking over deep mocking
Mock external dependencies but not the code being tested
Reset mocks between tests to prevent test pollution
For database tests, use in-memory instances or proper test fixtures
Test both success and error cases for each provider
Mock API responses to avoid external dependencies in tests
Validate that provider options are properly passed to the underlying service
Test error handling and edge cases (rate limits, timeouts, etc.)
Ensure provider caching behaves as expected
Always include both--coverageand--randomizeflags when running tests
Run tests in a single pass (no watch mode for CI)
Ensure all tests are independent and can run in any order
Clean up any test data or mocks after each test
Files:
test/redteam/constants/metadata.test.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Follow consistent import order (Biome will handle import sorting)
Use consistent curly braces for all control statements
Prefer const over let; avoid var
Use object shorthand syntax whenever possible
Use async/await for asynchronous code
Use consistent error handling with proper type checks
Files:
test/redteam/constants/metadata.test.tssrc/redteam/constants/metadata.ts
test/**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
test/**/*.{ts,tsx,js,jsx}: Follow Jest best practices with describe/it blocks
Write tests that cover both success and error cases
Files:
test/redteam/constants/metadata.test.ts
test/**
📄 CodeRabbit inference engine (CLAUDE.md)
Place tests under test/
Files:
test/redteam/constants/metadata.test.ts
src/**
📄 CodeRabbit inference engine (CLAUDE.md)
Place core production code under src/
Files:
src/redteam/constants/metadata.ts
🧬 Code graph analysis (1)
test/redteam/constants/metadata.test.ts (3)
src/redteam/constants/plugins.ts (8)
Plugin(327-335)BASE_PLUGINS(243-249)ADDITIONAL_PLUGINS(252-304)BIAS_PLUGINS(215-215)PII_PLUGINS(213-213)MEDICAL_PLUGINS(217-224)FINANCIAL_PLUGINS(226-237)HARM_PLUGINS(204-210)src/redteam/constants/metadata.ts (1)
riskCategories(408-525)src/matchers.ts (1)
fail(168-185)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (20)
- GitHub Check: Redteam
- GitHub Check: Run Integration Tests
- GitHub Check: webui tests
- GitHub Check: Build Docs
- GitHub Check: Redteam Custom Enterprise Server
- GitHub Check: Test on Node 24.x and ubuntu-latest
- GitHub Check: Share Test
- GitHub Check: Test on Node 24.x and macOS-latest
- GitHub Check: Test on Node 22.x and windows-latest
- GitHub Check: Test on Node 20.x and windows-latest
- GitHub Check: Test on Node 22.x and ubuntu-latest
- GitHub Check: Test on Node 20.x and macOS-latest
- GitHub Check: Build on Node 24.x
- GitHub Check: Generate Assets
- GitHub Check: Test on Node 24.x and windows-latest
- GitHub Check: Test on Node 20.x and ubuntu-latest
- GitHub Check: Build on Node 20.x
- GitHub Check: Build on Node 22.x
- GitHub Check: Style Check
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (2)
src/redteam/constants/metadata.ts (1)
414-425: Added missing plugins to riskCategories — LGTMThe new entries (cca, reasoning-dos, system-prompt-override, rag-document-exfiltration, rag-poisoning, off-topic, unverifiable-claims) are correctly placed and consistent with display names, descriptions, aliases, and severities defined above.
Also applies to: 438-440, 487-493
test/redteam/constants/metadata.test.ts (1)
20-28: Imports for plugin sources — LGTMBrings in all plugin constant sources needed for coverage validation.
| if (missingPlugins.length > 0) { | ||
| fail( | ||
| `The following plugins are defined but missing from risk categories: ${missingPlugins.join( | ||
| ', ', | ||
| )}. Please add them to the appropriate category in riskCategories object in metadata.ts`, | ||
| ); | ||
| } | ||
|
|
||
| expect(missingPlugins).toEqual([]); |
There was a problem hiding this comment.
Do not call undefined/global fail; throw to fail the test
fail is neither imported nor guaranteed to exist under jest-circus, and the custom fail in src/matchers returns an object (no throw). Replace with a thrown Error.
Apply this diff:
- if (missingPlugins.length > 0) {
- fail(
- `The following plugins are defined but missing from risk categories: ${missingPlugins.join(
- ', ',
- )}. Please add them to the appropriate category in riskCategories object in metadata.ts`,
- );
- }
+ if (missingPlugins.length > 0) {
+ throw new Error(
+ `The following plugins are defined but missing from risk categories: ${missingPlugins.join(
+ ', ',
+ )}. Please add them to the appropriate category in riskCategories in metadata.ts`,
+ );
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if (missingPlugins.length > 0) { | |
| fail( | |
| `The following plugins are defined but missing from risk categories: ${missingPlugins.join( | |
| ', ', | |
| )}. Please add them to the appropriate category in riskCategories object in metadata.ts`, | |
| ); | |
| } | |
| expect(missingPlugins).toEqual([]); | |
| if (missingPlugins.length > 0) { | |
| throw new Error( | |
| `The following plugins are defined but missing from risk categories: ${missingPlugins.join( | |
| ', ', | |
| )}. Please add them to the appropriate category in riskCategories in metadata.ts`, | |
| ); | |
| } | |
| expect(missingPlugins).toEqual([]); |
🤖 Prompt for AI Agents
In test/redteam/constants/metadata.test.ts around lines 128 to 136, the code
calls a global fail(...) which may be undefined under jest-circus and the
project’s custom fail does not throw; replace the fail(...) call with throwing
an Error (e.g., throw new Error(`The following plugins are defined but missing
from risk categories: ${missingPlugins.join(', ')}. Please add them to the
appropriate category in riskCategories object in metadata.ts`)) so the test
actually fails; keep or remove the subsequent expect(missingPlugins).toEqual([])
as desired, but ensure the thrown Error replaces the fail call.
Summary
This PR fixes missing plugins in the red team setup UI and adds comprehensive test coverage to prevent future regressions.
Problem: Several plugins were defined in the system but missing from the
riskCategoriesobject inmetadata.ts, causing them to not appear in the red team setup UI at/redteam/setup#3.Missing plugins found and fixed:
off-topic→ Added to Brand categorycca(Context Compliance Attack) → Added to Security & Access Controlreasoning-dos→ Added to Security & Access Controlsystem-prompt-override→ Added to Security & Access Controlrag-document-exfiltration→ Added to Security & Access Controlrag-poisoning→ Added to Security & Access Controlunverifiable-claims→ Added to BrandChanges
src/redteam/constants/metadata.tstest/redteam/constants/metadata.test.tsto catch future missing pluginsTest plan
The new test will fail if any plugins are defined but missing from risk categories, preventing this issue from happening again.
🤖 Generated with Claude Code