AIStoreHub Package Vault: Claude Code Security Review
Package Slug: claude-code-security-review
Archetype: tool
Upstream Repository: https://github.com/anthropics/claude-code-security-review
Workspace Files & Specifications (40 files)
- __init__.py (14 bytes)
# SAST package
- .claude/commands/security-review.md (10837 bytes)
--- allowed-tools: Bash(git diff:*), Bash(git status:*), Bash(git log:*), Bash(git show:*), Bash(git remote show:*), Read, Glob, Grep, LS, Task description: Complete a security review of the pending changes on the current branch --- You are a senior security engineer conducting a focused security review of the changes on this branch. GIT STATUS: ``` !`git status` ``` FILES MODIFIED: ``` !`git diff --name-only origin/HEAD...` ``` COMMITS: ``` !`git log --no-decorate origin/HEAD...` ``` DIFF CONTENT: ``` !`git diff --merge-base origin/HEAD` ``` Review the complete diff above. This contains all code changes in the PR. OBJECTIVE: Perform a security-focused code review to identify HIGH-CONFIDENCE security vulnerabilities that could have real exploitation potential. This is not a general code review - focus ONLY on security implications newly added by this PR. Do not comment on existing security concerns. CRITICAL INSTRUCTIONS: 1. MINIMIZE FALSE POSITIVES: Only flag issues where you're >80% confident of actual exploitability 2. AVOID NOISE: Skip theoretical issues, style concerns, or low-impact findings 3. FOCUS ON IMPACT: Prioritize vulnerabilities that could lead to unauthorized access, data breaches, or system compromise 4. EXCLUSIONS: Do NOT report the following issue types: - Denial of Service (DOS) vulnerabilities, even if they allow service disruption - Secrets or sensitive data stored on disk (these are handled by other processes) - Rate limiting or resource exhaustion issues SECURITY CATEGORIES TO EXAMINE: **Input Validation Vulnerabilities:** - SQL injection via unsanitized user input - Command injection in system calls or subprocesses - XXE injection in XML parsing - Template injection in templating engines - NoSQL injection in database queries - Path traversal in file operations **Authentication & Authorization Issues:** - Authentication bypass logic - Privilege escalation paths - Session management flaws - JWT token vulnerabilities - Authorization logic bypasses **Crypto & Secrets Management:** - Hardcoded API keys, passwords, or tokens - Weak cryptographic algorithms or implementations - Improper key storage or management - Cryptographic randomness issues - Certificate validation bypasses **Injection & Code Execution:** - Remote code execution via deseralization - Pickle injection in Python - YAML deserialization vulnerabilities - Eval injection in dynamic code execution - XSS vulnerabilities in web applications (reflected, stored, DOM-based) **Data Exposure:** - Sensitive data logging or storage - PII handling violations - API endpoint data leakage - Debug information exposure Additional notes: - Even if something is only exploitable from the local network, it can still be a HIGH severity issue ANALYSIS METHODOLOGY: Phase 1 - Repository Context Research (Use file search tools): - Identify existing security frameworks and libraries in use - Look for established secure coding patterns in the codebase - Examine existing sanitization and validation patterns - Understand the project's security model and threat model Phase 2 - Comparative Analysis: - Compare new code changes against existing security patterns - Identify deviations from established secure practices - Look for inconsistent security implementations - Flag code that introduces new attack surfaces Phase 3 - Vulnerability Assessment: - Examine each modified file for security implications - Trace data flow from user inputs to sensitive operations - Look for privilege boundaries being crossed unsafely - Identify injection points and unsafe deserialization REQUIRED OUTPUT FORMAT: You MUST output your findings in markdown. The markdown output should contain the file, line number, severity, category (e.g. `sql_injection` or `xss`), description, exploit scenario, and fix recommendation. For example: # Vuln 1: XSS: `foo.py:42` * Severity: High * Description: User input from `username` parameter is directly interpolated into HTML without escaping, allowing reflected XSS attacks * Exploit Scenario: Attacker crafts URL like /bar?q=<script>alert(document.cookie)</script> to execute JavaScript in victim's browser, enabling session hijacking or data theft * Recommendation: Use Flask's escape() function or Jinja2 templates with auto-escaping enabled for all user inputs rendered in HTML SEVERITY GUIDELINES: - **HIGH**: Directly exploitable vulnerabilities leading to RCE, data breach, or authentication bypass - **MEDIUM**: Vulnerabilities requiring specific conditions but with significant impact - **LOW**: Defense-in-depth issues or lower-impact vulnerabilities CONFIDENCE SCORING: - 0.9-1.0: Certain exploit path identified, tested if possible - 0.8-0.9: Clear vulnerability pattern with known exploitation methods - 0.7-0.8: Suspicious pattern requiring specific conditions to exploit - Below 0.7: Don't report (too speculative) FINAL REMINDER: Focus on HIGH and MEDIUM findings only. Better to miss some theoretical issues than flood the report with false positives. Each finding should be something a security engineer would confidently raise in a PR review. FALSE POSITIVE FILTERING: > You do not need to run commands to reproduce the vulnerability, just read the code to determine if it is a real vulnerability. Do not use the bash tool or write to any files. > > HARD EXCLUSIONS - Automatically exclude findings matching these patterns: > 1. Denial of Service (DOS) vulnerabilities or resource exhaustion attacks. > 2. Secrets or credentials stored on disk if they are otherwise secured. > 3. Rate limiting concerns or service overload scenarios. > 4. Memory consumption or CPU exhaustion issues. > 5. Lack of input validation on non-security-critical fields without proven security impact. > 6. Input sanitization concerns for GitHub Action workflows unless they are clearly triggerable via untrusted input. > 7. A lack of hardening measures. Code is not expected to implement all security best practices, only flag concrete vulnerabilities. > 8. Race conditions or timing attacks that are theoretical rather than practical issues. Only report a race condition if it is concretely problematic. > 9. Vulnerabilities related to outdated third-party libraries. These are managed separately and should not be reported here. > 10. Memory safety issues such as buffer overflows or use-after-free-vulnerabilities are impossible in rust. Do not report memory safety issues in rust or any other memory safe languages. > 11. Files that are only unit tests or only used as part of running tests. > 12. Log spoofing concerns. Outputting un-sanitized user input to logs is not a vulnerability. > 13. SSRF vulnerabilities that only control the path. SSRF is only a concern if it can control the host or protocol. > 14. Including user-controlled content in AI system prompts is not a vulnerability. > 15. Regex injection. Injecting untrusted content into a regex is not a vulnerability. > 16. Regex DOS concerns. > 16. Insecure documentation. Do not report any findings in documentation files such as markdown files. > 17. A lack of audit logs is not a vulnerability. > > PRECEDENTS - > 1. Logging high value secrets in plaintext is a vulnerability. Logging URLs is assumed to be safe. > 2. UUIDs can be assumed to be unguessable and do not need to be validated. > 3. Environment variables and CLI flags are trusted values. Attackers are generally not able to modify them in a secure environment. Any attack that relies on controlling an environment variable is invalid. > 4. Resource management issues such as memory or file descriptor leaks are not valid. > 5. Subtle or low impact web vulnerabilities such as tabnabbing, XS-Leaks, prototype pollution, and open redirects should not be reported unless they are extremely high confidence. > 6. React and Angular are generally secure against XSS. These frameworks do not need to sanitize or escape user input unless it is using dangerouslySetInnerHTML, bypassSecurityTrustHtml, or similar methods. Do not report XSS vulnerabilities in React or Angular components or tsx files unless they are using unsafe methods. > 7. Most vulnerabilities in github action workflows are not exploitable in practice. Before validating a github action workflow vulnerability ensure it is concrete and has a very specific attack path. > 8. A lack of permission checking or authentication in client-side JS/TS code is not a vulnerability. Client-side code is not trusted and does not need to implement these checks, they are handled on the server-side. The same applies to all flows that send untrusted data to the backend, the backend is responsible for validating and sanitizing all inputs. > 9. Only include MEDIUM findings if they are obvious and concrete issues. > 10. Most vulnerabilities in ipython notebooks (*.ipynb files) are not exploitable in practice. Before validating a notebook vulnerability ensure it is concrete and has a very specific attack path where untrusted input can trigger the vulnerability. > 11. Logging non-PII data is not a vulnerability even if the data may be sensitive. Only report logging vulnerabilities if they expose sensitive information such as secrets, passwords, or personally identifiable information (PII). > 12. Command injection vulnerabilities in shell scripts are generally not exploitable in practice since shell scripts generally do not run with untrusted user input. Only report command injection vulnerabilities in shell scripts if they are concrete and have a very specific attack path for untrusted input. > > SIGNAL QUALITY CRITERIA - For remaining findings, assess: > 1. Is there a concrete, exploitable vulnerability with a clear attack path? > 2. Does this represent a real security risk vs theoretical best practice? > 3. Are there specific code locations and reproduction steps? > 4. Would this finding be actionable for a security team? > > For each finding, assign a confidence score from 1-10: > - 1-3: Low confidence, likely false positive or noise > - 4-6: Medium confidence, needs investigation > - 7-10: High confidence, likely true vulnerability START ANALYSIS: Begin your analysis now. Do this in 3 steps: 1. Use a sub-task to identify vulnerabilities. Use the repository exploration tools to understand the codebase context, then analyze the PR changes for security implications. In the prompt for this sub-task, include all of the above. 2. Then for each vulnerability identified by the above sub-task, create a new sub-task to filter out false-positives. Launch these sub-tasks as parallel sub-tasks. In the prompt for these sub-tasks, include everything in the "FALSE POSITIVE FILTERING" instructions. 3. Filter out any vulnerabilities where the sub-task reported a confidence less than 8. Your final reply must contain the markdown report and nothing else.
- action.yml (14003 bytes)
name: 'Claude Code Security Reviewer' description: 'AI-powered security review GitHub Action using Claude to analyze code changes for security vulnerabilities' author: 'Anthropic' inputs: comment-pr: description: 'Whether to comment on PRs with findings' required: false default: 'true' upload-results: description: 'Whether to upload results as artifacts' required: false default: 'true' exclude-directories: description: 'Comma-separated list of directories to exclude from scanning' required: false default: '' claudecode-timeout: description: 'Timeout for ClaudeCode analysis in minutes' required: false default: '20' claude-api-key: description: 'Anthropic Claude API key for security analysis' required: true default: '' claude-model: description: 'Claude model to use for security analysis (e.g., claude-sonnet-4-20250514)' required: false default: '' run-every-commit: description: 'Run ClaudeCode on every commit (skips cache check). Warning: This may lead to more false positives on PRs with many commits as the AI analyzes the same code multiple times.' required: false default: 'false' false-positive-filtering-instructions: description: 'Path to custom false positive filtering instructions text file' required: false default: '' custom-security-scan-instructions: description: 'Path to custom security scan instructions text file to append to audit prompt' required: false default: '' outputs: findings-count: description: 'Number of security findings' value: ${{ steps.claudecode-scan.outputs.findings_count }} results-file: description: 'Path to the results JSON file' value: ${{ steps.claudecode-scan.outputs.results_file }} runs: using: 'composite' steps: - name: Install GitHub CLI shell: bash run: | echo "::group::Install gh CLI" # Install GitHub CLI for PR operations sudo apt-get update && sudo apt-get install -y gh echo "::endgroup::" - name: Set up Python uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 pinned to commit hash with: python-version: '3.x' - name: Check ClaudeCode run history id: claudecode-history if: github.event_name == 'pull_request' uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 pinned to commit hash with: path: .claudecode-marker key: claudecode-${{ github.repository_id }}-pr-${{ github.event.pull_request.number }}-${{ github.sha }} restore-keys: | claudecode-${{ github.repository_id }}-pr-${{ github.event.pull_request.number }}- - name: Determine ClaudeCode enablement id: claudecode-check shell: bash env: PR_NUMBER: ${{ github.event.pull_request.number }} RUN_EVERY_COMMIT: ${{ inputs.run-every-commit }} run: | # Check if ClaudeCode should be enabled ENABLE_CLAUDECODE="true" SILENCE_CLAUDECODE_COMMENTS="false" # For PRs, check sampling and cache if [ "${{ github.event_name }}" == "pull_request" ]; then PR_NUMBER="$PR_NUMBER" CACHE_HIT="${{ steps.claudecode-history.outputs.cache-hit }}" # Now check cache - if ClaudeCode has already run, disable unless run-every-commit is true # Check if marker file exists (cache may have been restored from a different SHA) if [ "$RUN_EVERY_COMMIT" != "true" ] && [ -f ".claudecode-marker/marker.json" ]; then echo "ClaudeCode has already run on PR #$PR_NUMBER (found marker file), forcing disable to avoid false positives" ENABLE_CLAUDECODE="false" elif [ "$RUN_EVERY_COMMIT" == "true" ] && [ -f ".claudecode-marker/marker.json" ]; then echo "ClaudeCode has already run on PR #$PR_NUMBER but run-every-commit is enabled, running again" elif [ "$ENABLE_CLAUDECODE" == "true" ]; then echo "ClaudeCode will run for PR #$PR_NUMBER (first run)" fi fi echo "enable_claudecode=$ENABLE_CLAUDECODE" >> $GITHUB_OUTPUT echo "silence_claudecode_comments=$SILENCE_CLAUDECODE_COMMENTS" >> $GITHUB_OUTPUT if [ "$ENABLE_CLAUDECODE" == "true" ]; then echo "ClaudeCode is enabled for this run" else echo "ClaudeCode is disabled for this run" fi - name: Reserve ClaudeCode slot to prevent race conditions if: steps.claudecode-check.outputs.enable_claudecode == 'true' && github.event_name == 'pull_request' shell: bash env: REPOSITORY_ID: ${{ github.repository_id }} REPOSITORY: ${{ github.repository }} PR_NUMBER: ${{ github.event.pull_request.number }} SHA: ${{ github.sha }} RUN_ID: ${{ github.run_id }} RUN_NUMBER: ${{ github.run_number }} run: | # Create a reservation marker immediately to prevent other concurrent runs mkdir -p .claudecode-marker cat > .claudecode-marker/marker.json << EOF { "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)", "repository_id": "$REPOSITORY_ID", "repository": "$REPOSITORY", "pr_number": $PR_NUMBER, "sha": "$SHA", "status": "reserved", "run_id": "$RUN_ID", "run_number": "$RUN_NUMBER" } EOF echo "Created ClaudeCode reservation marker for PR #$PR_NUMBER" - name: Save ClaudeCode reservation to cache if: steps.claudecode-check.outputs.enable_claudecode == 'true' && github.event_name == 'pull_request' uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 pinned to commit hash with: path: .claudecode-marker key: claudecode-${{ github.repository_id }}-pr-${{ github.event.pull_request.number }}-${{ github.sha }} - name: Set up Node.js if: steps.claudecode-check.outputs.enable_claudecode == 'true' uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 pinned to commit hash with: node-version: '18' - name: Install dependencies shell: bash env: ACTION_PATH: ${{ github.action_path }} run: | echo "::group::Install Deps" if [ "${{ steps.claudecode-check.outputs.enable_claudecode }}" == "true" ]; then pip install -r "$ACTION_PATH/claudecode/requirements.txt" npm install -g @anthropic-ai/claude-code fi sudo apt-get update && sudo apt-get install -y jq echo "::endgroup::" - name: Run ClaudeCode scan id: claudecode-scan if: steps.claudecode-check.outputs.enable_claudecode == 'true' shell: bash env: GITHUB_TOKEN: ${{ github.token }} GITHUB_REPOSITORY: ${{ github.repository }} PR_NUMBER: ${{ github.event.pull_request.number }} ANTHROPIC_API_KEY: ${{ inputs.claude-api-key }} ENABLE_CLAUDE_FILTERING: 'true' EXCLUDE_DIRECTORIES: ${{ inputs.exclude-directories }} FALSE_POSITIVE_FILTERING_INSTRUCTIONS: ${{ inputs.false-positive-filtering-instructions }} CUSTOM_SECURITY_SCAN_INSTRUCTIONS: ${{ inputs.custom-security-scan-instructions }} CLAUDE_MODEL: ${{ inputs.claude-model }} CLAUDECODE_TIMEOUT: ${{ inputs.claudecode-timeout }} ACTION_PATH: ${{ github.action_path }} run: | echo "Running ClaudeCode AI security analysis..." echo "----------------------------------------" # Initialize outputs echo "findings_count=0" >> $GITHUB_OUTPUT echo "results_file=claudecode/claudecode-results.json" >> $GITHUB_OUTPUT # Skip ClaudeCode if not a PR if [ "${{ github.event_name }}" != "pull_request" ]; then echo "ClaudeCode only runs on pull requests, skipping" exit 0 fi # Validate API key is provided if [ -z "$ANTHROPIC_API_KEY" ]; then echo "::error::ANTHROPIC_API_KEY is not set. Please provide the claude-api-key input to the action." echo "Example usage:" echo " - uses: anthropics/claude-code-security-reviewer@main" echo " with:" echo " claude-api-key: \$\{{ secrets.ANTHROPIC_API_KEY }}" exit 1 fi # Set timeout export CLAUDE_TIMEOUT="$CLAUDECODE_TIMEOUT" # Run ClaudeCode audit with verbose debugging export REPO_PATH=$(pwd) cd "$ACTION_PATH" # Enable verbose debugging echo "::group::ClaudeCode Environment" echo "Current directory: $(pwd)" echo "Python version: $(python --version)" echo "Claude CLI version: $(claude --version 2>&1 || echo 'Claude CLI not found')" echo "ANTHROPIC_API_KEY set: $(if [ -n "$ANTHROPIC_API_KEY" ]; then echo 'Yes'; else echo 'No'; fi)" echo "GITHUB_REPOSITORY: $GITHUB_REPOSITORY" echo "PR_NUMBER: $PR_NUMBER" echo "Python path: $PYTHONPATH" echo "Files in claudecode directory:" ls -la claudecode/ echo "::endgroup::" echo "::group::ClaudeCode Execution" # Add current directory to Python path so it can find the claudecode module export PYTHONPATH="${PYTHONPATH:+$PYTHONPATH:}$(pwd)" echo "Updated PYTHONPATH: $PYTHONPATH" # Run from the action root directory so Python can find the claudecode module python -u claudecode/github_action_audit.py > claudecode/claudecode-results.json 2>claudecode/claudecode-error.log || CLAUDECODE_EXIT_CODE=$? if [ -n "$CLAUDECODE_EXIT_CODE" ]; then echo "::warning::ClaudeCode exited with code $CLAUDECODE_EXIT_CODE" else echo "ClaudeCode scan completed successfully" fi # Parse ClaudeCode results and count findings regardless of exit code if [ -f claudecode/claudecode-results.json ]; then FILE_SIZE=$(wc -c < claudecode/claudecode-results.json) echo "ClaudeCode results file size: $FILE_SIZE bytes" # Check if file is empty or too small if [ "$FILE_SIZE" -lt 2 ]; then echo "::warning::ClaudeCode results file is empty or invalid (size: $FILE_SIZE bytes)" echo "::warning::ClaudeCode may have failed silently. Check claudecode-error.log" if [ -f claudecode/claudecode-error.log ]; then echo "Error log contents:" cat claudecode/claudecode-error.log fi echo "findings_count=0" >> $GITHUB_OUTPUT else echo "ClaudeCode results preview:" head -n 300 claudecode/claudecode-results.json || echo "Unable to preview results" # Check if the result is an error if jq -e '.error' claudecode/claudecode-results.json > /dev/null 2>&1; then ERROR_MSG=$(jq -r '.error' claudecode/claudecode-results.json) echo "::warning::ClaudeCode error: $ERROR_MSG" echo "findings_count=0" >> $GITHUB_OUTPUT else # Use -r to get raw output and handle potential null/missing findings array CLAUDECODE_FINDINGS_COUNT=$(jq -r '.findings | if . == null then 0 else length end' claudecode/claudecode-results.json 2>/dev/null || echo "0") echo "::debug::Extracted ClaudeCode findings count: $CLAUDECODE_FINDINGS_COUNT" echo "findings_count=$CLAUDECODE_FINDINGS_COUNT" >> $GITHUB_OUTPUT echo "ClaudeCode found $CLAUDECODE_FINDINGS_COUNT security issues" # Also create findings.json for PR comment script jq '.findings // []' claudecode/claudecode-results.json > findings.json || echo '[]' > findings.json fi fi else echo "::warning::ClaudeCode results file not found" if [ -f claudecode/claudecode-error.log ]; then echo "Error log contents:" cat claudecode/claudecode-error.log fi echo "findings_count=0" >> $GITHUB_OUTPUT fi # Always copy files to workspace root regardless of the outcome # This ensures artifact upload and PR commenting can find them if [ -f findings.json ]; then cp findings.json ${{ github.workspace }}/findings.json || true fi if [ -f claudecode/claudecode-results.json ]; then cp claudecode/claudecode-results.json ${{ github.workspace }}/claudecode-results.json || true fi if [ -f claudecode/claudecode-error.log ]; then cp claudecode/claudecode-error.log ${{ github.workspace }}/claudecode-error.log || true fi echo "::endgroup::" - name: Upload scan results if: always() && inputs.upload-results == 'true' uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 pinned to commit hash with: name: security-review-results path: | findings.json claudecode-results.json claudecode-error.log retention-days: 7 if-no-files-found: ignore - name: Comment PR with findings if: github.event_name == 'pull_request' && inputs.comment-pr == 'true' && steps.claudecode-check.outputs.enable_claudecode == 'true' shell: bash env: GITHUB_TOKEN: ${{ github.token }} CLAUDECODE_FINDINGS: ${{ steps.claudecode-scan.outputs.findings_count }} SILENCE_CLAUDECODE_COMMENTS: ${{ steps.claudecode-check.outputs.silence_claudecode_comments }} ACTION_PATH: ${{ github.action_path }} run: | node "$ACTION_PATH/scripts/comment-pr-findings.js" branding: icon: 'shield' color: 'red' - aistorehub.json (530 bytes)
{ "name": "Claude Code Security Review", "slug": "claude-code-security-review", "version": "1.0.0", "type": "tool", "description": "An AI-powered security review GitHub Action using Claude to analyze code changes for security vulnerabilities.", "owner": "community", "repository": "catalog", "github_repo_url": "https://github.com/anthropics/claude-code-security-review", "tags": [ "trending", "anthropics", "python", "github", "open-source" ], "imported_at": "2026-11-21T23:56:24.560Z" } - claudecode/__init__.py (459 bytes)
""" ClaudeCode - AI-Powered PR Security Audit Tool A standalone security audit tool that uses Claude Code for comprehensive security analysis of GitHub pull requests. """ __version__ = "1.0.0" __author__ = "Anthropic Security Team" # Import main components for easier access from claudecode.github_action_audit import ( GitHubActionClient, SimpleClaudeRunner, main ) __all__ = [ "GitHubActionClient", "SimpleClaudeRunner", "main" ] - claudecode/audit.py (219 bytes)
#!/usr/bin/env python3 """ Main entry point for ClaudeCode audit tool. This provides a cleaner interface for running the audit. """ from claudecode.github_action_audit import main if __name__ == "__main__": main() - claudecode/claude_api_client.py (18287 bytes)
"""Claude API client for direct Anthropic API calls.""" import os import json import time from typing import Dict, Any, Tuple, Optional from pathlib import Path from anthropic import Anthropic from claudecode.constants import ( DEFAULT_CLAUDE_MODEL, DEFAULT_TIMEOUT_SECONDS, DEFAULT_MAX_RETRIES, RATE_LIMIT_BACKOFF_MAX, PROMPT_TOKEN_LIMIT, ) from claudecode.json_parser import parse_json_with_fallbacks from claudecode.logger import get_logger logger = get_logger(__name__) class ClaudeAPIClient: """Client for calling Claude API directly for security analysis tasks.""" def __init__(self, model: Optional[str] = None, api_key: Optional[str] = None, timeout_seconds: Optional[int] = None, max_retries: Optional[int] = None): """Initialize Claude API client. Args: model: Claude model to use api_key: Anthropic API key (if None, reads from ANTHROPIC_API_KEY env var) timeout_seconds: Request timeout in seconds max_retries: Maximum retry attempts for API calls """ self.model = model or DEFAULT_CLAUDE_MODEL self.timeout_seconds = timeout_seconds or DEFAULT_TIMEOUT_SECONDS self.max_retries = max_retries or DEFAULT_MAX_RETRIES # Get API key from environment or parameter self.api_key = api_key or os.environ.get("ANTHROPIC_API_KEY") if not self.api_key: raise ValueError( "No Anthropic API key found. Please set ANTHROPIC_API_KEY environment variable " "or provide api_key parameter." ) # Initialize Anthropic client self.client = Anthropic(api_key=self.api_key) logger.info("Claude API client initialized successfully") def validate_api_access(self) -> Tuple[bool, str]: """Validate that API access is working. Returns: Tuple of (success, error_message) """ try: # Simple test call to verify API access self.client.messages.create( model="claude-3-5-haiku-20241022", max_tokens=10, messages=[{"role": "user", "content": "Hello"}], timeout=10 ) logger.info("Claude API access validated successfully") return True, "" except Exception as e: error_msg = str(e) logger.error(f"Claude API validation failed: {error_msg}") return False, f"API validation failed: {error_msg}" def call_with_retry(self, prompt: str, system_prompt: Optional[str] = None, max_tokens: int = PROMPT_TOKEN_LIMIT) -> Tuple[bool, str, str]: """Make Claude API call with retry logic. Args: prompt: User prompt system_prompt: Optional system prompt max_tokens: Maximum tokens to generate Returns: Tuple of (success, response_text, error_message) """ retries = 0 last_error = None while retries <= self.max_retries: try: logger.info(f"Claude API call attempt {retries + 1}/{self.max_retries + 1}") # Prepare messages messages = [{"role": "user", "content": prompt}] # Build API call parameters api_params = { "model": self.model, "max_tokens": max_tokens, "messages": messages, "timeout": self.timeout_seconds } if system_prompt: api_params["system"] = system_prompt # Make API call start_time = time.time() response = self.client.messages.create(**api_params) duration = time.time() - start_time # Extract text from response response_text = "" for content_block in response.content: if hasattr(content_block, 'text'): response_text += content_block.text logger.info(f"Claude API call successful in {duration:.1f}s") return True, response_text, "" except Exception as e: error_msg = str(e) last_error = error_msg logger.error(f"Claude API call failed: {error_msg}") # Check if it's a rate limit error if "rate limit" in error_msg.lower() or "429" in error_msg: logger.warning("Rate limit detected, increasing backoff") backoff_time = min(RATE_LIMIT_BACKOFF_MAX, 5 * (retries + 1)) # Progressive backoff time.sleep(backoff_time) elif "timeout" in error_msg.lower(): logger.warning("Timeout detected, retrying") time.sleep(2) else: # For other errors, shorter backoff time.sleep(1) retries += 1 # All retries exhausted return False, "", f"API call failed after {self.max_retries + 1} attempts: {last_error}" def analyze_single_finding(self, finding: Dict[str, Any], pr_context: Optional[Dict[str, Any]] = None, custom_filtering_instructions: Optional[str] = None) -> Tuple[bool, Dict[str, Any], str]: """Analyze a single security finding to filter false positives using Claude API. Args: finding: Single security finding to analyze pr_context: Optional PR context for better analysis Returns: Tuple of (success, analysis_result, error_message) """ try: # Generate analysis prompt with file content prompt = self._generate_single_finding_prompt(finding, pr_context, custom_filtering_instructions) system_prompt = self._generate_system_prompt() # Call Claude API success, response_text, error_msg = self.call_with_retry( prompt=prompt, system_prompt=system_prompt, max_tokens=PROMPT_TOKEN_LIMIT ) if not success: return False, {}, error_msg # Parse JSON response using json_parser success, analysis_result = parse_json_with_fallbacks(response_text, "Claude API response") if success: logger.info("Successfully parsed Claude API response for single finding") return True, analysis_result, "" else: # Fallback: return error return False, {}, "Failed to parse JSON response" except Exception as e: logger.exception(f"Error during single finding security analysis: {str(e)}") return False, {}, f"Single finding security analysis failed: {str(e)}" def _generate_system_prompt(self) -> str: """Generate system prompt for security analysis.""" return """You are a security expert reviewing findings from an automated code audit tool. Your task is to filter out false positives and low-signal findings to reduce alert fatigue. You must maintain high recall (don't miss real vulnerabilities) while improving precision. Respond ONLY with valid JSON in the exact format specified in the user prompt. Do not include explanatory text, markdown formatting, or code blocks.""" def _generate_single_finding_prompt(self, finding: Dict[str, Any], pr_context: Optional[Dict[str, Any]] = None, custom_filtering_instructions: Optional[str] = None) -> str: """Generate prompt for analyzing a single security finding. Args: finding: Single security finding pr_context: Optional PR context Returns: Formatted prompt string """ pr_info = "" if pr_context and isinstance(pr_context, dict): pr_info = f""" PR Context: - Repository: {pr_context.get('repo_name', 'unknown')} - PR #{pr_context.get('pr_number', 'unknown')} - Title: {pr_context.get('title', 'unknown')} - Description: {(pr_context.get('description') or 'No description')[:500]}... """ # Get file content if available file_path = finding.get('file', '') file_content = "" if file_path: success, content, error = self._read_file(file_path) if success: file_content = f""" File Content ({file_path}): ``` {content} ```""" else: file_content = f""" File Content ({file_path}): Error reading file - {error} """ finding_json = json.dumps(finding, indent=2) # Use custom filtering instructions if provided, otherwise use defaults if custom_filtering_instructions: filtering_section = custom_filtering_instructions else: filtering_section = """HARD EXCLUSIONS - Automatically exclude findings matching these patterns: 1. Denial of Service (DOS) vulnerabilities or resource exhaustion attacks 2. Secrets/credentials stored on disk (these are managed separately) 3. Rate limiting concerns or service overload scenarios (services don't need to implement rate limiting) 4. Memory consumption or CPU exhaustion issues 5. Lack of input validation on non-security-critical fields without proven security impact 6. Input sanitization concerns for github action workflows 7. A lack of hardening measures. Code is not expected to implement all security best practices, just avoid obvious vulnerabilities. 8. Race conditions or timing attacks that are theoretical rather than practical issues. Only report a race condition if it is extremely problematic. 9. Vulnerabilities related to outdated third-party libraries. These are managed separately and should not be reported here. 10. Memory safety issues such as buffer overflows or use-after-free-vulnerabilities are impossible in rust. Do not report memory safety issues in rust code. 11. Files that are only unit tests or only used as part of running tests. 12. Log spoofing concerns. Outputing un-sanitized user input to logs is not a vulnerability. 13. SSRF vulnerabilities that only control the path. SSRF is only a concern if it can control the host or protocol. 14. Including user-controlled content in AI system prompts is not a vulnerability. In general, the inclusion of user input in an AI prompt is not a vulnerability. 15. Do not report issues related to adding a dependency to a project that is not available from the relevant package repository. Depending on internal libraries that are not publicly available is not a vulnerability. 16. Do not report issues that cause the code to crash, but are not actually a vulnerability. E.g. a variable that is undefined or null is not a vulnerability. SIGNAL QUALITY CRITERIA - For remaining findings, assess: 1. Is there a concrete, exploitable vulnerability with a clear attack path? 2. Does this represent a real security risk vs theoretical best practice? 3. Are there specific code locations and reproduction steps? 4. Would this finding be actionable for a security team? PRECEDENTS - 1. Logging high value secrets in plaintext is a vulnerability. Otherwise, do not report issues around theoretical exposures of secrets. Logging URLs is assumed to be safe. Logging request headers is assumed to be dangerous since they likely contain credentials. 2. UUIDs can be assumed to be unguessable and do not need to be validated. If a vulnerabilities requires guessing a UUID, it is not a valid vulnerability. 3. Audit logs are not a critical security feature and should not be reported as a vulnerability if they are missing or modified. 4. Environment variables and CLI flags are trusted values. Attackers are not able to modify them in a secure environment. Any attack that relies on controlling an environment variable is invalid. 5. Resource management issues such as memory or file descriptor leaks are not valid. 6. Subtle or low impact web vulnerabilities such as tabnabbing, XS-Leaks, prototype pollution, and open redirects are not valid. 7. Vulnerabilities related to outdated third-party libraries. These are managed separately and should not be reported here. 8. React is generally secure against XSS. React does not need to sanitize or escape user input unless it is using dangerouslySetInnerHTML or similar methods. Do not report XSS vulnerabilities in React components or tsx files unless they are using unsafe methods. 9. Most vulnerabilities in github action workflows are not exploitable in practice. Before validating a github action workflow vulnerability ensure it is concrete and has a very specific attack path. 10. A lack of permission checking or authentication in client-side TS code is not a vulnerability. Client-side code is not trusted and does not need to implement these checks, they are handled on the server-side. The same applies to all flows that send untrusted data to the backend, the backend is responsible for validating and sanitizing all inputs. 11. Only include MEDIUM findings if they are obvious and concrete issues. 12. Most vulnerabilities in ipython notebooks (*.ipynb files) are not exploitable in practice. Before validating a notebook vulnerability ensure it is concrete and has a very specific attack path. 13. Logging non-PII data is not a vulnerability even if the data may be sensitive. Only report logging vulnerabilities if they expose sensitive information such as secrets, passwords, or personally identifiable information (PII). 14. Command injection vulnerabilities in shell scripts are generally not exploitable in practice since shell scripts generally do not run with untrusted user input. Only report command injection vulnerabilities in shell scripts if they are concrete and have a very specific attack path for untrusted input. 15. SSRF (Server-Side Request Forgery) vulnerabilities in client-side JavaScript/TypeScript files (.js, .ts, .tsx, .jsx) are not valid since client-side code cannot make server-side requests that would bypass firewalls or access internal resources. Only report SSRF in server-side code (e.g. Python or JS that is known to run on the server-side). The same logic applies to path-traversal attacks, they are not a problem in client-side JS. 16. Path traversal attacks using - claudecode/constants.py (567 bytes)
""" Constants and configuration values for ClaudeCode. """ import os # API Configuration DEFAULT_CLAUDE_MODEL = os.environ.get('CLAUDE_MODEL') or 'claude-opus-4-1-20250805' DEFAULT_TIMEOUT_SECONDS = 180 # 3 minutes DEFAULT_MAX_RETRIES = 3 RATE_LIMIT_BACKOFF_MAX = 30 # Maximum backoff time for rate limits # Token Limits PROMPT_TOKEN_LIMIT = 16384 # 16k tokens max for claude-opus-4 # Exit Codes EXIT_SUCCESS = 0 EXIT_GENERAL_ERROR = 1 EXIT_CONFIGURATION_ERROR = 2 # Subprocess Configuration SUBPROCESS_TIMEOUT = 1200 # 20 minutes for Claude Code execution - claudecode/evals/__init__.py (221 bytes)
"""Evaluation tool for SAST.""" from .eval_engine import EvalCase, EvalResult, EvaluationEngine, run_single_evaluation __all__ = [ 'EvalCase', 'EvalResult', 'EvaluationEngine', 'run_single_evaluation', ] - claudecode/evals/eval_engine.py (20289 bytes)
"""Evaluation engine for running SAST security audits on GitHub PRs.""" import os import sys import subprocess import shutil import time import threading from typing import Dict, Any, Optional, Tuple, List from dataclasses import dataclass, asdict from pathlib import Path from ..json_parser import parse_json_with_fallbacks # Timeout constants (in seconds) TIMEOUT_SHORT = 10 TIMEOUT_GIT_OPERATION = 60 TIMEOUT_FETCH = 600 TIMEOUT_CLONE = 300 TIMEOUT_WORKTREE = 300 TIMEOUT_WORKTREE_CREATE = 1200 TIMEOUT_CLAUDECODE = 1800 @dataclass class EvalCase: """Single evaluation test case.""" repo_name: str pr_number: int description: str = "" @dataclass class EvalResult: """Result of a single evaluation.""" repo_name: str pr_number: int description: str # Evaluation results success: bool runtime_seconds: float findings_count: int detected_vulnerabilities: bool # Optional fields error_message: str = "" findings_summary: Optional[List[Dict[str, Any]]] = None full_findings: Optional[List[Dict[str, Any]]] = None def to_dict(self) -> Dict[str, Any]: """Convert to dictionary for JSON serialization.""" return asdict(self) class EvaluationEngine: """Engine for running security evaluations on GitHub PRs.""" def __init__(self, work_dir: str = None, verbose: bool = False): """Initialize evaluation engine. Args: work_dir: Directory for cloning repositories verbose: Enable verbose logging """ # Use ~/code/audit as base directory like pr_audit does if work_dir is None: work_dir = os.path.expanduser("~/code/audit") self.work_dir = work_dir Path(self.work_dir).mkdir(parents=True, exist_ok=True) self.verbose = verbose self.claude_api_key = os.environ.get('ANTHROPIC_API_KEY', '') if not self.claude_api_key: raise ValueError("ANTHROPIC_API_KEY environment variable required") # Repository locks for concurrent access self._repo_locks: Dict[str, threading.Lock] = {} self._locks_lock = threading.Lock() # Get GitHub token from environment or gh CLI self.github_token = os.environ.get('GITHUB_TOKEN', '') if not self.github_token: try: result = subprocess.run(['gh', 'auth', 'token'], capture_output=True, text=True, timeout=TIMEOUT_GIT_OPERATION) if result.returncode == 0: self.github_token = result.stdout.strip() os.environ['GITHUB_TOKEN'] = self.github_token self.log("Retrieved GitHub token from gh CLI") except (subprocess.SubprocessError, FileNotFoundError) as e: self.log(f"Could not retrieve GitHub token from gh CLI: {e}") def log(self, message: str, prefix: str = "[EVAL]") -> None: """Log a message if verbose mode is enabled.""" if self.verbose: timestamp = time.strftime('%H:%M:%S') print(f"{prefix} [{timestamp}] {message}", file=sys.stderr) def _get_repo_lock(self, repo_name: str) -> threading.Lock: """Get or create a lock for a repository. Args: repo_name: Repository name Returns: Lock for the repository """ with self._locks_lock: if repo_name not in self._repo_locks: self._repo_locks[repo_name] = threading.Lock() return self._repo_locks[repo_name] def _clean_worktrees(self, repo_path: str, branch_pattern: str = None) -> None: """Clean up locked or stale worktrees and remove untracked branches. Args: repo_path: Path to the main repository branch_pattern: Optional pattern to match branches for cleanup """ if not os.path.exists(repo_path): return try: # First, prune worktrees to clean up stale entries subprocess.run(['git', '-C', repo_path, 'worktree', 'prune'], check=False, capture_output=True, timeout=TIMEOUT_SHORT) # List all worktrees result = subprocess.run(['git', '-C', repo_path, 'worktree', 'list', '--porcelain'], capture_output=True, text=True, check=True, timeout=TIMEOUT_SHORT) worktrees = [] current_worktree = {} for line in result.stdout.strip().split('\n'): if not line: if current_worktree: worktrees.append(current_worktree) current_worktree = {} elif line.startswith('worktree '): current_worktree['path'] = line[9:] elif line.startswith('branch '): current_worktree['branch'] = line[7:] elif line == 'locked': current_worktree['locked'] = True if current_worktree: worktrees.append(current_worktree) # Remove locked or matching worktrees for wt in worktrees: if wt.get('path') == repo_path: continue # Skip main worktree should_remove = False if wt.get('locked'): self.log(f"Found locked worktree: {wt.get('path')}") should_remove = True elif branch_pattern and 'branch' in wt: branch_name = wt['branch'].replace('refs/heads/', '') if branch_pattern in branch_name: self.log(f"Found matching worktree for cleanup: {wt.get('path')} (branch: {branch_name})") should_remove = True if should_remove: try: # Force remove the worktree subprocess.run(['git', '-C', repo_path, 'worktree', 'remove', '--force', wt['path']], check=False, capture_output=True, timeout=TIMEOUT_SHORT) # Also try to remove the directory if it still exists if os.path.exists(wt['path']): shutil.rmtree(wt['path'], ignore_errors=True) except Exception as e: self.log(f"Error removing worktree {wt.get('path')}: {e}") # Clean up branches if branch_pattern: # Get all local branches result = subprocess.run(['git', '-C', repo_path, 'branch', '--list'], capture_output=True, text=True, check=True, timeout=TIMEOUT_SHORT) for line in result.stdout.strip().split('\n'): branch = line.strip().lstrip('* ') if branch_pattern in branch: try: # Delete the branch subprocess.run(['git', '-C', repo_path, 'branch', '-D', branch], check=False, capture_output=True, timeout=TIMEOUT_SHORT) self.log(f"Deleted branch: {branch}") except Exception as e: self.log(f"Error deleting branch {branch}: {e}") except Exception as e: self.log(f"Error during worktree cleanup: {e}") def _get_eval_branch_name(self, test_case: EvalCase) -> str: """Generate a branch name for evaluation. Args: test_case: Test case being evaluated Returns: Branch name for the evaluation """ # Create a safe branch name from repo and PR safe_repo = test_case.repo_name.replace('/', '-').replace('.', '-') timestamp = time.strftime('%Y%m%d-%H%M%S') return f"eval-pr-{safe_repo}-{test_case.pr_number}-{timestamp}" def _setup_repository(self, test_case: EvalCase) -> Tuple[bool, str, str]: """Set up repository worktree for PR evaluation. Args: test_case: Test case containing repo and PR info Returns: Tuple of (success, worktree_path, error_message) """ repo_name = test_case.repo_name pr_number = test_case.pr_number # Create base path for this repository safe_repo_name = repo_name.replace('/', '_') base_repo_path = os.path.join(self.work_dir, safe_repo_name) # Get lock for this repository repo_lock = self._get_repo_lock(repo_name) with repo_lock: # Clone or update the base repository if not os.path.exists(base_repo_path): self.log(f"Cloning {repo_name} to {base_repo_path}") clone_url = f"https://github.com/{repo_name}.git" if self.github_token: clone_url = f"https://{self.github_token}@github.com/{repo_name}.git" try: subprocess.run(['git', 'clone', '--filter=blob:none', clone_url, base_repo_path], check=True, capture_output=True, timeout=TIMEOUT_CLONE) except subprocess.CalledProcessError as e: error_msg = f"Failed to clone repository: {e.stderr.decode()}" self.log(error_msg) return False, "", error_msg # Clean up any stale worktrees for this evaluation eval_branch_prefix = f"eval-pr-{safe_repo_name}-{pr_number}" self._clean_worktrees(base_repo_path, eval_branch_prefix) # Create worktree for this specific evaluation eval_branch = self._get_eval_branch_name(test_case) worktree_path = os.path.join(self.work_dir, f"{safe_repo_name}_pr{pr_number}_{int(time.time())}") try: # Fetch the PR self.log(f"Fetching PR #{pr_number} from {repo_name}") subprocess.run(['git', '-C', base_repo_path, 'fetch', 'origin', f'pull/{pr_number}/head'], check=True, capture_output=True, timeout=TIMEOUT_FETCH) # Create new worktree with PR changes self.log(f"Creating worktree at {worktree_path}") subprocess.run(['git', '-C', base_repo_path, 'worktree', 'add', '-b', eval_branch, worktree_path, 'FETCH_HEAD'], check=True, capture_output=True, timeout=TIMEOUT_WORKTREE_CREATE) return True, worktree_path, "" except subprocess.CalledProcessError as e: error_msg = f"Failed to set up worktree: {e.stderr.decode()}" self.log(error_msg) # Clean up failed worktree if it exists if os.path.exists(worktree_path): shutil.rmtree(worktree_path, ignore_errors=True) # Try to remove from git worktree list try: subprocess.run(['git', '-C', base_repo_path, 'worktree', 'remove', '--force', worktree_path], check=False, capture_output=True, timeout=TIMEOUT_SHORT) except Exception: pass return False, "", error_msg def _cleanup_worktree(self, test_case: EvalCase, worktree_path: str) -> None: """Clean up a worktree after evaluation. Args: test_case: Test case that was evaluated worktree_path: Path to the worktree """ if not os.path.exists(worktree_path): return repo_name = test_case.repo_name safe_repo_name = repo_name.replace('/', '_') base_repo_path = os.path.join(self.work_dir, safe_repo_name) repo_lock = self._get_repo_lock(repo_name) with repo_lock: try: # Remove the worktree subprocess.run(['git', '-C', base_repo_path, 'worktree', 'remove', '--force', worktree_path], check=False, capture_output=True, timeout=TIMEOUT_WORKTREE) # Also remove directory if it still exists if os.path.exists(worktree_path): shutil.rmtree(worktree_path, ignore_errors=True) self.log(f"Cleaned up worktree: {worktree_path}") except Exception as e: self.log(f"Error cleaning up worktree: {e}") def run_evaluation(self, test_case: EvalCase) -> EvalResult: """Run security evaluation on a single PR. Args: test_case: Test case to evaluate Returns: EvalResult with evaluation outcome """ start_time = time.time() self.log(f"Starting evaluation of {test_case.repo_name}#{test_case.pr_number}") # Set up repository success, worktree_path, error_msg = self._setup_repository(test_case) if not success: return EvalResult( repo_name=test_case.repo_name, pr_number=test_case.pr_number, description=test_case.description, success=False, runtime_seconds=time.time() - start_time, findings_count=0, detected_vulnerabilities=False, error_message=f"Repository setup failed: {error_msg}" ) try: # Run the SAST audit self.log(f"Running SAST audit on {worktree_path}") audit_success, output, parsed_results, error_message = self._run_sast_audit(test_case, worktree_path) if not audit_success: return EvalResult( repo_name=test_case.repo_name, pr_number=test_case.pr_number, description=test_case.description, success=False, runtime_seconds=time.time() - start_time, findings_count=0, detected_vulnerabilities=False, error_message=f"SAST audit failed: {error_message or 'Unknown error'}" ) # Extract findings from results findings = [] if parsed_results and 'findings' in parsed_results: findings = p - claudecode/evals/README.md (1686 bytes)
# SAST Evaluation Tool This directory contains a tool for evaluating the SAST (Static Application Security Testing) tool on individual GitHub pull requests. ## Overview The evaluation tool allows you to run the Claude Code Security Reviewer on any GitHub PR to analyze its security findings. This is useful for: - Testing the tool on specific PRs - Evaluating performance and accuracy - Debugging security analysis issues ## Requirements - Python 3.9+ - Git 2.20+ (for worktree support) - GitHub CLI (`gh`) for API access - Environment variables: - `ANTHROPIC_API_KEY`: Required for Claude API access - `GITHUB_TOKEN`: Recommended for GitHub API rate limits ## Usage Run an evaluation on a single PR: ```bash python -m claudecode.evals.run_eval example/repo#123 --verbose ``` ### Command-line Options - PR specification: Required positional argument in format `owner/repo#pr_number` - `--output-dir PATH`: Directory for results (default: `./eval_results`) - `--work-dir PATH`: Directory where git repositories will be cloned and stored (default: `~/code/audit`) - `--verbose`: Enable verbose logging to see detailed progress ## Output The evaluation generates a JSON file in the output directory with: - Success/failure status - Runtime metrics - Security findings count - Detailed findings with file, line, severity, and descriptions Example output file: `pr_example_repo_123.json` ## Architecture The evaluation tool uses git worktrees for efficient repository management: 1. Clones the repository once as a base 2. Creates lightweight worktrees for each PR evaluation 3. Automatically handles cleanup of worktrees 4. Runs the SAST audit in the PR-specific worktree
- claudecode/evals/run_eval.py (5672 bytes)
#!/usr/bin/env python3 """CLI for running SAST evaluation on a single PR.""" import argparse import os import sys import json from pathlib import Path from typing import Dict, Any, Optional, List from dataclasses import dataclass, asdict # Import the minimal required functionality @dataclass class EvalCase: """Single evaluation test case.""" repo_name: str pr_number: int description: str = "" @dataclass class EvalResult: """Result of a single evaluation.""" repo_name: str pr_number: int description: str # Evaluation results success: bool runtime_seconds: float findings_count: int detected_vulnerabilities: bool # Optional fields error_message: str = "" findings_summary: Optional[List[Dict[str, Any]]] = None full_findings: Optional[List[Dict[str, Any]]] = None def to_dict(self) -> Dict[str, Any]: """Convert to dictionary for JSON serialization.""" return asdict(self) def main(): """Main entry point for single PR SAST evaluation.""" parser = argparse.ArgumentParser( description="Run SAST security evaluation on a single GitHub PR", formatter_class=argparse.ArgumentDefaultsHelpFormatter ) parser.add_argument( "pr", type=str, help="PR to evaluate in format 'repo_owner/repo_name#pr_number' (e.g., 'example/repo#123')" ) parser.add_argument( "--output-dir", type=str, default="./eval_results", help="Directory for evaluation results" ) parser.add_argument( "--work-dir", type=str, default=None, help="Directory for temporary repositories (defaults to ~/code/audit)" ) parser.add_argument( "--verbose", action="store_true", help="Enable verbose logging" ) args = parser.parse_args() # Set EVAL_MODE=1 automatically for evaluation runs os.environ['EVAL_MODE'] = '1' # Check for required environment variables if not os.environ.get('ANTHROPIC_API_KEY'): print("Error: ANTHROPIC_API_KEY environment variable is not set") print("Please set it before running the evaluation") sys.exit(1) # Parse the PR specification try: repo_part, pr_number = args.pr.split('#') pr_number = int(pr_number) # Validate repository format if '/' not in repo_part or len(repo_part.split('/')) != 2: raise ValueError("Repository must be in format 'owner/repo'") owner, repo = repo_part.split('/') if not owner or not repo: raise ValueError("Repository owner and name cannot be empty") except ValueError as e: print(f"Error: Invalid PR format '{args.pr}': {e}") print("Expected format: 'repo_owner/repo_name#pr_number'") print("Example: 'example/repo#123'") sys.exit(1) print(f"\nEvaluating PR: {repo_part}#{pr_number}") print("-" * 60) # Create test case test_case = EvalCase( repo_name=repo_part, pr_number=pr_number, description=f"Evaluation for {repo_part}#{pr_number}" ) # Import and run the evaluation from .eval_engine import run_single_evaluation # Run the evaluation result = run_single_evaluation(test_case, verbose=args.verbose, work_dir=args.work_dir) # Display results print("\n" + "=" * 60) print("EVALUATION RESULTS:") print(f"Success: {result.success}") print(f"Runtime: {result.runtime_seconds:.1f} seconds") print(f"Vulnerabilities detected: {result.detected_vulnerabilities}") print(f"Findings count: {result.findings_count}") if result.error_message: print(f"Error: {result.error_message}") if result.full_findings: print("\nFindings:") for finding in result.full_findings: print(f" - [{finding.get('severity', 'UNKNOWN')}] {finding.get('file', 'unknown')}:{finding.get('line', '?')}") if 'category' in finding: print(f" Category: {finding['category']}") if 'description' in finding: print(f" Description: {finding['description']}") if 'exploit_scenario' in finding: print(f" Exploit: {finding['exploit_scenario']}") if 'recommendation' in finding: print(f" Fix: {finding['recommendation']}") if 'confidence' in finding: print(f" Confidence: {finding['confidence']}") print() # Empty line between findings elif result.findings_summary: # Fallback to summary if full findings not available print("\nFindings:") for finding in result.findings_summary: print(f" - [{finding.get('severity', 'UNKNOWN')}] {finding.get('file', 'unknown')}:{finding.get('line', '?')}") if 'title' in finding and finding['title'] != 'Unknown': print(f" {finding['title']}") if 'description' in finding and finding['description'] != 'Unknown': print(f" {finding['description']}") # Save result to output directory output_path = Path(args.output_dir) output_path.mkdir(parents=True, exist_ok=True) result_file = output_path / f"pr_{repo_part.replace('/', '_')}_{pr_number}.json" with open(result_file, 'w') as f: json.dump(result.to_dict(), f, indent=2) print(f"\nResult saved to: {result_file}") # Exit with appropriate code sys.exit(0 if result.success else 1) if __name__ == "__main__": main() - claudecode/findings_filter.py (15311 bytes)
"""Findings filter for reducing false positives in security audit results.""" import re from typing import Dict, Any, List, Tuple, Optional, Pattern import time from dataclasses import dataclass, field from claudecode.claude_api_client import ClaudeAPIClient from claudecode.constants import DEFAULT_CLAUDE_MODEL from claudecode.logger import get_logger logger = get_logger(__name__) @dataclass class FilterStats: """Statistics about the filtering process.""" total_findings: int = 0 hard_excluded: int = 0 claude_excluded: int = 0 kept_findings: int = 0 exclusion_breakdown: Dict[str, int] = field(default_factory=dict) confidence_scores: List[float] = field(default_factory=list) runtime_seconds: float = 0.0 class HardExclusionRules: """Hard exclusion rules for common false positives.""" # Pre-compiled regex patterns for better performance _DOS_PATTERNS: List[Pattern] = [ re.compile(r'\b(denial of service|dos attack|resource exhaustion)\b', re.IGNORECASE), re.compile(r'\b(exhaust|overwhelm|overload).*?(resource|memory|cpu)\b', re.IGNORECASE), re.compile(r'\b(infinite|unbounded).*?(loop|recursion)\b', re.IGNORECASE), ] _RATE_LIMITING_PATTERNS: List[Pattern] = [ re.compile(r'\b(missing|lack of|no)\s+rate\s+limit', re.IGNORECASE), re.compile(r'\brate\s+limiting\s+(missing|required|not implemented)', re.IGNORECASE), re.compile(r'\b(implement|add)\s+rate\s+limit', re.IGNORECASE), re.compile(r'\bunlimited\s+(requests|calls|api)', re.IGNORECASE), ] _RESOURCE_PATTERNS: List[Pattern] = [ re.compile(r'\b(resource|memory|file)\s+leak\s+potential', re.IGNORECASE), re.compile(r'\bunclosed\s+(resource|file|connection)', re.IGNORECASE), re.compile(r'\b(close|cleanup|release)\s+(resource|file|connection)', re.IGNORECASE), re.compile(r'\bpotential\s+memory\s+leak', re.IGNORECASE), re.compile(r'\b(database|thread|socket|connection)\s+leak', re.IGNORECASE), ] _OPEN_REDIRECT_PATTERNS: List[Pattern] = [ re.compile(r'\b(open redirect|unvalidated redirect)\b', re.IGNORECASE), re.compile(r'\b(redirect.(attack|exploit|vulnerability))\b', re.IGNORECASE), re.compile(r'\b(malicious.redirect)\b', re.IGNORECASE), ] _MEMORY_SAFETY_PATTERNS: List[Pattern] = [ re.compile(r'\b(buffer overflow|stack overflow|heap overflow)\b', re.IGNORECASE), re.compile(r'\b(oob)\s+(read|write|access)\b', re.IGNORECASE), re.compile(r'\b(out.?of.?bounds?)\b', re.IGNORECASE), re.compile(r'\b(memory safety|memory corruption)\b', re.IGNORECASE), re.compile(r'\b(use.?after.?free|double.?free|null.?pointer.?dereference)\b', re.IGNORECASE), re.compile(r'\b(segmentation fault|segfault|memory violation)\b', re.IGNORECASE), re.compile(r'\b(bounds check|boundary check|array bounds)\b', re.IGNORECASE), re.compile(r'\b(integer overflow|integer underflow|integer conversion)\b', re.IGNORECASE), re.compile(r'\barbitrary.?(memory read|pointer dereference|memory address|memory pointer)\b', re.IGNORECASE), ] _REGEX_INJECTION: List[Pattern] = [ re.compile(r'\b(regex|regular expression)\s+injection\b', re.IGNORECASE), re.compile(r'\b(regex|regular expression)\s+denial of service\b', re.IGNORECASE), re.compile(r'\b(regex|regular expression)\s+flooding\b', re.IGNORECASE), ] _SSRF_PATTERNS: List[Pattern] = [ re.compile(r'\b(ssrf|server\s+.?side\s+.?request\s+.?forgery)\b', re.IGNORECASE), ] @classmethod def get_exclusion_reason(cls, finding: Dict[str, Any]) -> Optional[str]: """Check if a finding should be excluded based on hard rules. Args: finding: Security finding to check Returns: Exclusion reason if finding should be excluded, None otherwise """ # Check if finding is in a Markdown file file_path = finding.get('file', '') if file_path.lower().endswith('.md'): return "Finding in Markdown documentation file" description = finding.get('description', '') title = finding.get('title', '') # Handle None values if description is None: description = '' if title is None: title = '' combined_text = f"{title} {description}".lower() # Check DOS patterns for pattern in cls._DOS_PATTERNS: if pattern.search(combined_text): return "Generic DOS/resource exhaustion finding (low signal)" # Check rate limiting patterns for pattern in cls._RATE_LIMITING_PATTERNS: if pattern.search(combined_text): return "Generic rate limiting recommendation" # Check resource patterns - always exclude for pattern in cls._RESOURCE_PATTERNS: if pattern.search(combined_text): return "Resource management finding (not a security vulnerability)" # Check open redirect patterns for pattern in cls._OPEN_REDIRECT_PATTERNS: if pattern.search(combined_text): return "Open redirect vulnerability (not high impact)" # Check regex injection patterns for pattern in cls._REGEX_INJECTION: if pattern.search(combined_text): return "Regex injection finding (not applicable)" # Check memory safety patterns - exclude if NOT in C/C++ files c_cpp_extensions = {'.c', '.cc', '.cpp', '.h'} file_ext = '' if '.' in file_path: file_ext = f".{file_path.lower().split('.')[-1]}" # If file doesn't have a C/C++ extension (including no extension), exclude memory safety findings if file_ext not in c_cpp_extensions: for pattern in cls._MEMORY_SAFETY_PATTERNS: if pattern.search(combined_text): return "Memory safety finding in non-C/C++ code (not applicable)" # Check SSRF patterns - exclude if in HTML files only html_extensions = {'.html'} # If file has HTML extension, exclude SSRF findings if file_ext in html_extensions: for pattern in cls._SSRF_PATTERNS: if pattern.search(combined_text): return "SSRF finding in HTML file (not applicable to client-side code)" return None class FindingsFilter: """Main filter class for security findings.""" def __init__(self, use_hard_exclusions: bool = True, use_claude_filtering: bool = True, api_key: Optional[str] = None, model: str = DEFAULT_CLAUDE_MODEL, custom_filtering_instructions: Optional[str] = None): """Initialize findings filter. Args: use_hard_exclusions: Whether to apply hard exclusion rules use_claude_filtering: Whether to use Claude API for filtering api_key: Anthropic API key for Claude filtering model: Claude model to use for filtering custom_filtering_instructions: Optional custom filtering instructions """ self.use_hard_exclusions = use_hard_exclusions self.use_claude_filtering = use_claude_filtering self.custom_filtering_instructions = custom_filtering_instructions # Initialize Claude client if filtering is enabled self.claude_client = None if self.use_claude_filtering: try: self.claude_client = ClaudeAPIClient( model=model, api_key=api_key ) # Validate API access valid, error = self.claude_client.validate_api_access() if not valid: logger.warning(f"Claude API validation failed: {error}") self.claude_client = None self.use_claude_filtering = False except Exception as e: logger.error(f"Failed to initialize Claude client: {str(e)}") self.use_claude_filtering = False def filter_findings(self, findings: List[Dict[str, Any]], pr_context: Optional[Dict[str, Any]] = None) -> Tuple[bool, Dict[str, Any], FilterStats]: """Filter security findings to remove false positives. Args: findings: List of security findings from Claude Code audit pr_context: Optional PR context for better analysis Returns: Tuple of (success, filtered_results, stats) """ start_time = time.time() if not findings: stats = FilterStats(total_findings=0, runtime_seconds=0.0) return True, { "filtered_findings": [], "excluded_findings": [], "analysis_summary": { "total_findings": 0, "kept_findings": 0, "excluded_findings": 0, "exclusion_breakdown": {} } }, stats logger.info(f"Filtering {len(findings)} security findings") # Initialize statistics stats = FilterStats(total_findings=len(findings)) # Step 1: Apply hard exclusion rules findings_after_hard = [] excluded_hard = [] if self.use_hard_exclusions: for i, finding in enumerate(findings): exclusion_reason = HardExclusionRules.get_exclusion_reason(finding) if exclusion_reason: excluded_hard.append({ "finding": finding, "index": i, "exclusion_reason": exclusion_reason, "filter_stage": "hard_rules" }) stats.hard_excluded += 1 # Track exclusion breakdown key = exclusion_reason.split('(')[0].strip() stats.exclusion_breakdown[key] = stats.exclusion_breakdown.get(key, 0) + 1 else: findings_after_hard.append((i, finding)) logger.info(f"Hard exclusions removed {stats.hard_excluded} findings") else: findings_after_hard = [(i, f) for i, f in enumerate(findings)] # Step 2: Apply Claude API filtering if enabled findings_after_claude = [] excluded_claude = [] if self.use_claude_filtering and self.claude_client and findings_after_hard: # Process findings individually logger.info(f"Processing {len(findings_after_hard)} findings individually through Claude API") for orig_idx, finding in findings_after_hard: # Call Claude API for single finding success, analysis_result, error_msg = self.claude_client.analyze_single_finding( finding, pr_context, self.custom_filtering_instructions ) if success and analysis_result: # Process Claude's analysis for single finding confidence = analysis_result.get('confidence_score', 10.0) keep_finding = analysis_result.get('keep_finding', True) justification = analysis_result.get('justification', '') exclusion_reason = analysis_result.get('exclusion_reason') stats.confidence_scores.append(confidence) if not keep_finding: # Claude recommends excluding excluded_claude.append({ "finding": finding, "confidence_score": confidence, "exclusion_reason": exclusion_reason or f"Low confidence score: {confidence}", "justification": justification, "filter_stage": "claude_api" }) stats.claude_excluded += 1 else: # Keep finding with metadata enriched_finding = finding.copy() enriched_finding['_filter_metadata'] = { 'confidence_score': confidence, 'justification': justification, } findings_after_claude.append(enriched_finding) stats.kept_findings += 1 else: # Claude API call failed for this finding - keep it with warning logger.warning(f"Claude API call failed for finding {orig_idx}: {error_msg}") enriched_finding = finding.copy() enriched_finding['_filter_metadata'] = { 'confidence_score': 10.0, # Default high confidence 'justification': f'Claude API failed: {error_msg}', } findings_after_claude.append(enriched_finding) stats.kept_findings += 1 else: # Claude filtering disabled or no client - keep all findings from hard filter for orig_idx, finding in findings_after_hard: enriched_finding = finding.copy() enriched_finding['_filter_metadata'] = { 'confidence_score': 10.0, # Default high confidence 'justification': 'Claude filtering disabled', } findings_after_claude.append(enriched_finding) stats.kept_findings += 1 # Combine all excluded findings all_excluded = excluded_hard + excluded_claude # Calculate final statistics stats.runtime_seconds = time.time() - start_time # Build filtered results filtered_results = { "filtered_findings": findings_after_claude, "excluded_findings": all_excluded, "analysis_summary": { "total_findings": stats.total_findings, "kept_findings": stats.kept_findings, "excluded_findings": len(all_excluded), "hard_excluded": stats.hard_excluded, "claude_excluded": stats.claude_excluded, "exclusion_breakdown": stats.exclusion_breakdown, "average_confidence": sum(stats.confidence_scores) / len(stats.confidence_scores) if stats.confidence_scores els - claudecode/github_action_audit.py (25341 bytes)
#!/usr/bin/env python3 """ Simplified PR Security Audit for GitHub Actions Runs Claude Code security audit on current working directory and outputs findings to stdout """ import os import sys import json import subprocess import requests from typing import Dict, Any, List, Tuple, Optional from pathlib import Path import re import time # Import existing components we can reuse from claudecode.prompts import get_security_audit_prompt from claudecode.findings_filter import FindingsFilter from claudecode.json_parser import parse_json_with_fallbacks from claudecode.constants import ( EXIT_CONFIGURATION_ERROR, DEFAULT_CLAUDE_MODEL, EXIT_SUCCESS, EXIT_GENERAL_ERROR, SUBPROCESS_TIMEOUT ) from claudecode.logger import get_logger logger = get_logger(__name__) class ConfigurationError(ValueError): """Raised when configuration is invalid or missing.""" pass class AuditError(ValueError): """Raised when security audit operations fail.""" pass class GitHubActionClient: """Simplified GitHub API client for GitHub Actions environment.""" def __init__(self): """Initialize GitHub client using environment variables.""" self.github_token = os.environ.get('GITHUB_TOKEN') if not self.github_token: raise ValueError("GITHUB_TOKEN environment variable required") self.headers = { 'Authorization': f'Bearer {self.github_token}', 'Accept': 'application/vnd.github.v3+json', 'X-GitHub-Api-Version': '2022-11-28' } # Get excluded directories from environment exclude_dirs = os.environ.get('EXCLUDE_DIRECTORIES', '') self.excluded_dirs = [d.strip() for d in exclude_dirs.split(',') if d.strip()] if exclude_dirs else [] if self.excluded_dirs: print(f"[Debug] Excluded directories: {self.excluded_dirs}", file=sys.stderr) def get_pr_data(self, repo_name: str, pr_number: int) -> Dict[str, Any]: """Get PR metadata and files from GitHub API. Args: repo_name: Repository name in format "owner/repo" pr_number: Pull request number Returns: Dictionary containing PR data """ # Get PR metadata pr_url = f"https://api.github.com/repos/{repo_name}/pulls/{pr_number}" response = requests.get(pr_url, headers=self.headers) response.raise_for_status() pr_data = response.json() # Get PR files with pagination support files_url = f"https://api.github.com/repos/{repo_name}/pulls/{pr_number}/files?per_page=100" response = requests.get(files_url, headers=self.headers) response.raise_for_status() files_data = response.json() return { 'number': pr_data['number'], 'title': pr_data['title'], 'body': pr_data.get('body', ''), 'user': pr_data['user']['login'], 'created_at': pr_data['created_at'], 'updated_at': pr_data['updated_at'], 'state': pr_data['state'], 'head': { 'ref': pr_data['head']['ref'], 'sha': pr_data['head']['sha'], 'repo': { 'full_name': pr_data['head']['repo']['full_name'] if pr_data['head']['repo'] else repo_name } }, 'base': { 'ref': pr_data['base']['ref'], 'sha': pr_data['base']['sha'] }, 'files': [ { 'filename': f['filename'], 'status': f['status'], 'additions': f['additions'], 'deletions': f['deletions'], 'changes': f['changes'], 'patch': f.get('patch', '') } for f in files_data if not self._is_excluded(f['filename']) ], 'additions': pr_data['additions'], 'deletions': pr_data['deletions'], 'changed_files': pr_data['changed_files'] } def get_pr_diff(self, repo_name: str, pr_number: int) -> str: """Get complete PR diff in unified format. Args: repo_name: Repository name in format "owner/repo" pr_number: Pull request number Returns: Complete PR diff in unified format """ url = f"https://api.github.com/repos/{repo_name}/pulls/{pr_number}" headers = dict(self.headers) headers['Accept'] = 'application/vnd.github.diff' response = requests.get(url, headers=headers) response.raise_for_status() return self._filter_generated_files(response.text) def _is_excluded(self, filepath: str) -> bool: """Check if a file should be excluded based on directory patterns.""" for excluded_dir in self.excluded_dirs: # Normalize excluded directory (remove leading ./ if present) if excluded_dir.startswith('./'): normalized_excluded = excluded_dir[2:] else: normalized_excluded = excluded_dir # Check if file starts with excluded directory if filepath.startswith(excluded_dir + '/'): return True if filepath.startswith(normalized_excluded + '/'): return True # Check if excluded directory appears anywhere in the path if '/' + normalized_excluded + '/' in filepath: return True return False def _filter_generated_files(self, diff_text: str) -> str: """Filter out generated files and excluded directories from diff content.""" file_sections = re.split(r'(?=^diff --git)', diff_text, flags=re.MULTILINE) filtered_sections = [] for section in file_sections: if not section.strip(): continue # Skip generated files if ('@generated by' in section or '@generated' in section or 'Code generated by OpenAPI Generator' in section or 'Code generated by protoc-gen-go' in section): continue # Extract filename from diff header match = re.match(r'^diff --git a/(.*?) b/', section) if match: filename = match.group(1) if self._is_excluded(filename): print(f"[Debug] Filtering out excluded file: {filename}", file=sys.stderr) continue filtered_sections.append(section) return ''.join(filtered_sections) class SimpleClaudeRunner: """Simplified Claude Code runner for GitHub Actions.""" def __init__(self, timeout_minutes: Optional[int] = None): """Initialize Claude runner. Args: timeout_minutes: Timeout for Claude execution (defaults to SUBPROCESS_TIMEOUT) """ if timeout_minutes is not None: self.timeout_seconds = timeout_minutes * 60 else: self.timeout_seconds = SUBPROCESS_TIMEOUT def run_security_audit(self, repo_dir: Path, prompt: str) -> Tuple[bool, str, Dict[str, Any]]: """Run Claude Code security audit. Args: repo_dir: Path to repository directory prompt: Security audit prompt Returns: Tuple of (success, error_message, parsed_results) """ if not repo_dir.exists(): return False, f"Repository directory does not exist: {repo_dir}", {} # Check prompt size prompt_size = len(prompt.encode('utf-8')) if prompt_size > 1024 * 1024: # 1MB print(f"[Warning] Large prompt size: {prompt_size / 1024 / 1024:.2f}MB", file=sys.stderr) try: # Construct Claude Code command # Use stdin for prompt to avoid "argument list too long" error cmd = [ 'claude', '--output-format', 'json', '--model', DEFAULT_CLAUDE_MODEL, '--disallowed-tools', 'Bash(ps:*)' ] # Run Claude Code with retry logic NUM_RETRIES = 3 for attempt in range(NUM_RETRIES): result = subprocess.run( cmd, input=prompt, # Pass prompt via stdin cwd=repo_dir, capture_output=True, text=True, timeout=self.timeout_seconds ) if result.returncode != 0: if attempt == NUM_RETRIES - 1: error_details = f"Claude Code execution failed with return code {result.returncode}\n" error_details += f"Stderr: {result.stderr}\n" error_details += f"Stdout: {result.stdout[:500]}..." # First 500 chars return False, error_details, {} else: time.sleep(5*attempt) # Note: We don't do exponential backoff here to keep the runtime reasonable continue # Retry # Parse JSON output success, parsed_result = parse_json_with_fallbacks(result.stdout, "Claude Code output") if success: # Check for "Prompt is too long" error that should trigger retry without diff if (isinstance(parsed_result, dict) and parsed_result.get('type') == 'result' and parsed_result.get('subtype') == 'success' and parsed_result.get('is_error') and parsed_result.get('result') == 'Prompt is too long'): return False, "PROMPT_TOO_LONG", {} # Check for error_during_execution that should trigger retry if (isinstance(parsed_result, dict) and parsed_result.get('type') == 'result' and parsed_result.get('subtype') == 'error_during_execution' and attempt == 0): continue # Retry # Extract security findings parsed_results = self._extract_security_findings(parsed_result) return True, "", parsed_results else: if attempt == 0: continue # Retry once else: return False, "Failed to parse Claude output", {} return False, "Unexpected error in retry logic", {} except subprocess.TimeoutExpired: return False, f"Claude Code execution timed out after {self.timeout_seconds // 60} minutes", {} except Exception as e: return False, f"Claude Code execution error: {str(e)}", {} def _extract_security_findings(self, claude_output: Any) -> Dict[str, Any]: """Extract security findings from Claude's JSON response.""" if isinstance(claude_output, dict): # Only accept Claude Code wrapper with result field # Direct format without wrapper is not supported if 'result' in claude_output: result_text = claude_output['result'] if isinstance(result_text, str): # Try to extract JSON from the result text success, result_json = parse_json_with_fallbacks(result_text, "Claude result text") if success and result_json and 'findings' in result_json: return result_json # Return empty structure if no findings found return { 'findings': [], 'analysis_summary': { 'files_reviewed': 0, 'high_severity': 0, 'medium_severity': 0, 'low_severity': 0, 'review_completed': False, } } def validate_claude_available(self) -> Tuple[bool, str]: """Validate that Claude Code is available.""" try: result = subprocess.run( ['claude', '--version'], capture_output=True, text=True, timeout=10 ) if result.returncode == 0: # Also check if API key is configured api_key = os.environ.get('ANTHROPIC_API_KEY', '') if not api_key: return False, "ANTHROPIC_API_KEY environment variable is not set" return True, "" else: error_msg = f"Claude Code returned exit code {result.returncode}" if result.stderr: error_msg += f". Stderr: {result.stderr}" if result.stdout: error_msg += f". Stdout: {result.stdout}" return False, error_msg except subprocess.TimeoutExpired: return False, "Claude Code command timed out" except FileNotFoundError: return False, "Claude Code is not installed or not in PATH" except Exception as e: return False, f"Failed to check Claude Code: {str(e)}" def get_environment_config() -> Tuple[str, int]: """Get and validate environment configuration. Returns: Tuple of (repo_name, pr_number) Raises: ConfigurationError: If required environment variables are missing or invalid """ repo_name = os.environ.get('GITHUB_REPOSITORY') pr_number_str = os.environ.get('PR_NUMBER') if not repo_name: raise ConfigurationError('GITHUB_REPOSITORY environment variable required') if not pr_number_str: raise ConfigurationError('PR_NUMBER environment variable required') try: pr_number = int(pr_number_str) except ValueError: raise ConfigurationError(f'Invalid PR_NUMBER: {pr_number_str}') return repo_name, pr_number def initialize_clients() -> Tuple[GitHubActionClient, SimpleClaudeRunner]: """Initialize GitHub and Claude clients. Returns: Tuple of (github_client, claude_runner) Raises: ConfigurationError: If client initialization fails """ try: github_client = GitHubActionClient() except Exception as e: raise ConfigurationError(f'Failed to initialize GitHub client: {str(e)}') try: claude - claudecode/json_parser.py (2810 bytes)
#!/usr/bin/env python3 """Utilities for parsing JSON from text output.""" import json import re import logging # Configure logging logger = logging.getLogger(__name__) def extract_json_from_text(text): """ Extract JSON object from text, looking in various formats and locations. Args: text: The text that may contain JSON Returns: dict: Parsed JSON object if found, None otherwise """ try: # First, try to extract JSON from markdown code blocks (with or without language tag) json_matches = [ re.search(r'```json\s*(.*?)\s*```', text, re.DOTALL), re.search(r'```\s*(\{.*?\})\s*```', text, re.DOTALL) ] for json_match in json_matches: if json_match: try: return json.loads(json_match.group(1)) except json.JSONDecodeError: continue # If no JSON found in code blocks, try to find JSON objects anywhere in the text # Find all potential JSON objects (looking for balanced braces) brace_count = 0 json_start = -1 for i, char in enumerate(text): if char == '{': if brace_count == 0: json_start = i brace_count += 1 elif char == '}': brace_count -= 1 if brace_count == 0 and json_start != -1: # Found a complete JSON object potential_json = text[json_start:i+1] try: return json.loads(potential_json) except json.JSONDecodeError: # This wasn't valid JSON, continue looking continue except Exception: pass return None def parse_json_with_fallbacks(text, error_context=""): """ Parse JSON from text with multiple fallback strategies and error handling. Args: text: The text to parse error_context: Context string for error messages Returns: tuple: (success, result) where result is either the parsed JSON dict or error info """ try: # First, try direct JSON parsing return True, json.loads(text) except json.JSONDecodeError: pass # Try extracting JSON from text extracted_json = extract_json_from_text(text) if extracted_json: return True, extracted_json # If all parsing failed, return error info error_msg = "Failed to parse JSON" if error_context: error_msg = f"{error_context}: {error_msg}" logger.error(f"{error_msg}. Raw output: {repr(text)}") return False, {"error": f"Invalid JSON response -- raw output: {repr(text)}"} - claudecode/logger.py (1347 bytes)
"""Logging configuration for ClaudeCode.""" import logging import sys import os def get_logger(name: str) -> logging.Logger: """Get a configured logger that outputs to stderr. Args: name: The name of the logger (usually __name__) Returns: Configured logger instance """ logger = logging.getLogger(name) # Only configure if not already configured if not logger.handlers: handler = logging.StreamHandler(sys.stderr) # Get repo and PR number from environment for prefix repo_name = os.environ.get('GITHUB_REPOSITORY', '') pr_number = os.environ.get('PR_NUMBER', '') # Build prefix if repo_name and pr_number: prefix = f"[{repo_name}#{pr_number}]" elif repo_name: prefix = f"[{repo_name}]" elif pr_number: prefix = f"[PR#{pr_number}]" else: prefix = "" # Include prefix in format if available if prefix: format_str = f'{prefix} [%(name)s] %(message)s' else: format_str = '[%(name)s] %(message)s' formatter = logging.Formatter(format_str) handler.setFormatter(formatter) logger.addHandler(handler) logger.setLevel(logging.INFO) return logger - claudecode/prompts.py (7079 bytes)
"""Security audit prompt templates.""" def get_security_audit_prompt(pr_data, pr_diff=None, include_diff=True, custom_scan_instructions=None): """Generate security audit prompt for Claude Code. Args: pr_data: PR data dictionary from GitHub API pr_diff: Optional complete PR diff in unified format include_diff: Whether to include the diff in the prompt (default: True) custom_scan_instructions: Optional custom security categories to append Returns: Formatted prompt string """ files_changed = "\n".join([f"- {f['filename']}" for f in pr_data['files']]) # Add diff section if provided and include_diff is True diff_section = "" if pr_diff and include_diff: diff_section = f""" PR DIFF CONTENT: ``` {pr_diff} ``` Review the complete diff above. This contains all code changes in the PR. """ elif pr_diff and not include_diff: diff_section = """ NOTE: PR diff was omitted due to size constraints. Please use the file exploration tools to examine the specific files that were changed in this PR. """ # Add custom security categories if provided custom_categories_section = "" if custom_scan_instructions: custom_categories_section = f"\n{custom_scan_instructions}\n" return f""" You are a senior security engineer conducting a focused security review of GitHub PR #{pr_data['number']}: "{pr_data['title']}" CONTEXT: - Repository: {pr_data.get('head', {}).get('repo', {}).get('full_name', 'unknown')} - Author: {pr_data['user']} - Files changed: {pr_data['changed_files']} - Lines added: {pr_data['additions']} - Lines deleted: {pr_data['deletions']} Files modified: {files_changed}{diff_section} OBJECTIVE: Perform a security-focused code review to identify HIGH-CONFIDENCE security vulnerabilities that could have real exploitation potential. This is not a general code review - focus ONLY on security implications newly added by this PR. Do not comment on existing security concerns. CRITICAL INSTRUCTIONS: 1. MINIMIZE FALSE POSITIVES: Only flag issues where you're >80% confident of actual exploitability 2. AVOID NOISE: Skip theoretical issues, style concerns, or low-impact findings 3. FOCUS ON IMPACT: Prioritize vulnerabilities that could lead to unauthorized access, data breaches, or system compromise 4. EXCLUSIONS: Do NOT report the following issue types: - Denial of Service (DOS) vulnerabilities, even if they allow service disruption - Secrets or sensitive data stored on disk (these are handled by other processes) - Rate limiting or resource exhaustion issues SECURITY CATEGORIES TO EXAMINE: **Input Validation Vulnerabilities:** - SQL injection via unsanitized user input - Command injection in system calls or subprocesses - XXE injection in XML parsing - Template injection in templating engines - NoSQL injection in database queries - Path traversal in file operations **Authentication & Authorization Issues:** - Authentication bypass logic - Privilege escalation paths - Session management flaws - JWT token vulnerabilities - Authorization logic bypasses **Crypto & Secrets Management:** - Hardcoded API keys, passwords, or tokens - Weak cryptographic algorithms or implementations - Improper key storage or management - Cryptographic randomness issues - Certificate validation bypasses **Injection & Code Execution:** - Remote code execution via deseralization - Pickle injection in Python - YAML deserialization vulnerabilities - Eval injection in dynamic code execution - XSS vulnerabilities in web applications (reflected, stored, DOM-based) **Data Exposure:** - Sensitive data logging or storage - PII handling violations - API endpoint data leakage - Debug information exposure {custom_categories_section} Additional notes: - Even if something is only exploitable from the local network, it can still be a HIGH severity issue ANALYSIS METHODOLOGY: Phase 1 - Repository Context Research (Use file search tools): - Identify existing security frameworks and libraries in use - Look for established secure coding patterns in the codebase - Examine existing sanitization and validation patterns - Understand the project's security model and threat model Phase 2 - Comparative Analysis: - Compare new code changes against existing security patterns - Identify deviations from established secure practices - Look for inconsistent security implementations - Flag code that introduces new attack surfaces Phase 3 - Vulnerability Assessment: - Examine each modified file for security implications - Trace data flow from user inputs to sensitive operations - Look for privilege boundaries being crossed unsafely - Identify injection points and unsafe deserialization REQUIRED OUTPUT FORMAT: You MUST output your findings as structured JSON with this exact schema: {{ "findings": [ {{ "file": "path/to/file.py", "line": 42, "severity": "HIGH", "category": "sql_injection", "description": "User input passed to SQL query without parameterization", "exploit_scenario": "Attacker could extract database contents by manipulating the 'search' parameter with SQL injection payloads like '1; DROP TABLE users--'", "recommendation": "Replace string formatting with parameterized queries using SQLAlchemy or equivalent", "confidence": 0.95 }} ], "analysis_summary": {{ "files_reviewed": 8, "high_severity": 1, "medium_severity": 0, "low_severity": 0, "review_completed": true, }} }} SEVERITY GUIDELINES: - **HIGH**: Directly exploitable vulnerabilities leading to RCE, data breach, or authentication bypass - **MEDIUM**: Vulnerabilities requiring specific conditions but with significant impact - **LOW**: Defense-in-depth issues or lower-impact vulnerabilities CONFIDENCE SCORING: - 0.9-1.0: Certain exploit path identified, tested if possible - 0.8-0.9: Clear vulnerability pattern with known exploitation methods - 0.7-0.8: Suspicious pattern requiring specific conditions to exploit - Below 0.7: Don't report (too speculative) FINAL REMINDER: Focus on HIGH and MEDIUM findings only. Better to miss some theoretical issues than flood the report with false positives. Each finding should be something a security engineer would confidently raise in a PR review. IMPORTANT EXCLUSIONS - DO NOT REPORT: - Denial of Service (DOS) vulnerabilities or resource exhaustion attacks - Secrets/credentials stored on disk (these are managed separately) - Rate limiting concerns or service overload scenarios. Services do not need to implement rate limiting. - Memory consumption or CPU exhaustion issues. - Lack of input validation on non-security-critical fields. If there isn't a proven problem from a lack of input validation, don't report it. Begin your analysis now. Use the repository exploration tools to understand the codebase context, then analyze the PR changes for security implications. Your final reply must contain the JSON and nothing else. You should not reply again after outputting the JSON. """ - claudecode/requirements.txt (553 bytes)
# Requirements for claudecode - Claude Code PR Security Audit Tool # Core dependencies for the GitHub Action version # GitHub API client PyGithub>=1.59.0 # HTTP requests for GitHub API requests>=2.28.0 # JSON parsing utilities (no additional deps - uses stdlib) # prompts.py (no additional deps - uses stdlib) # findings_filter.py (uses re, built-in) # Anthropic SDK for Claude API-based false positive filtering anthropic>=0.39.0 # Note: Claude CLI tool must be installed separately # The claude command-line tool is required for security analysis
- claudecode/test_claude_runner.py (15364 bytes)
#!/usr/bin/env python3 """ Unit tests for SimpleClaudeRunner. """ import json import os import subprocess from unittest.mock import Mock, patch from pathlib import Path from claudecode.github_action_audit import SimpleClaudeRunner from claudecode.constants import DEFAULT_CLAUDE_MODEL class TestSimpleClaudeRunner: """Test SimpleClaudeRunner functionality.""" def test_init(self): """Test runner initialization.""" runner = SimpleClaudeRunner(timeout_minutes=30) assert runner.timeout_seconds == 1800 runner2 = SimpleClaudeRunner() # Default assert runner2.timeout_seconds == 1200 # 20 minutes default @patch('subprocess.run') def test_validate_claude_available_success(self, mock_run): """Test successful Claude validation.""" mock_run.return_value = Mock( returncode=0, stdout='claude version 1.0.0', stderr='' ) with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): runner = SimpleClaudeRunner() success, error = runner.validate_claude_available() assert success is True assert error == '' mock_run.assert_called_once_with( ['claude', '--version'], capture_output=True, text=True, timeout=10 ) @patch('subprocess.run') def test_validate_claude_available_no_api_key(self, mock_run): """Test Claude validation without API key.""" mock_run.return_value = Mock( returncode=0, stdout='claude version 1.0.0', stderr='' ) # Remove API key env = os.environ.copy() env.pop('ANTHROPIC_API_KEY', None) with patch.dict(os.environ, env, clear=True): runner = SimpleClaudeRunner() success, error = runner.validate_claude_available() assert success is False assert 'ANTHROPIC_API_KEY environment variable is not set' in error @patch('subprocess.run') def test_validate_claude_available_not_installed(self, mock_run): """Test Claude validation when not installed.""" mock_run.side_effect = FileNotFoundError() runner = SimpleClaudeRunner() success, error = runner.validate_claude_available() assert success is False assert 'Claude Code is not installed or not in PATH' in error @patch('subprocess.run') def test_validate_claude_available_error(self, mock_run): """Test Claude validation with error.""" mock_run.return_value = Mock( returncode=1, stdout='', stderr='Error: Authentication failed' ) runner = SimpleClaudeRunner() success, error = runner.validate_claude_available() assert success is False assert 'exit code 1' in error assert 'Authentication failed' in error @patch('subprocess.run') def test_validate_claude_available_timeout(self, mock_run): """Test Claude validation timeout.""" mock_run.side_effect = subprocess.TimeoutExpired(['claude'], 10) runner = SimpleClaudeRunner() success, error = runner.validate_claude_available() assert success is False assert 'timed out' in error def test_run_security_audit_missing_directory(self): """Test audit with missing directory.""" runner = SimpleClaudeRunner() success, error, results = runner.run_security_audit( Path('/non/existent/path'), "test prompt" ) assert success is False assert 'Repository directory does not exist' in error assert results == {} @patch('subprocess.run') def test_run_security_audit_success(self, mock_run): """Test successful security audit.""" # Claude Code returns wrapped format with 'result' field findings_data = { "findings": [ { "file": "test.py", "line": 10, "severity": "HIGH", "description": "SQL injection vulnerability" } ], "analysis_summary": { "files_reviewed": 5, "high_severity": 1, "medium_severity": 0, "low_severity": 0, "review_completed": True } } audit_result = { "result": json.dumps(findings_data) } mock_run.return_value = Mock( returncode=0, stdout=json.dumps(audit_result), stderr='' ) runner = SimpleClaudeRunner() with patch('pathlib.Path.exists', return_value=True): success, error, results = runner.run_security_audit( Path('/tmp/test'), "test prompt" ) assert success is True assert error == '' assert len(results['findings']) == 1 assert results['findings'][0]['severity'] == 'HIGH' # Verify subprocess call mock_run.assert_called_once() call_args = mock_run.call_args assert call_args[0][0] == [ 'claude', '--output-format', 'json', '--model', DEFAULT_CLAUDE_MODEL, '--disallowed-tools', 'Bash(ps:*)' ] assert call_args[1]['input'] == 'test prompt' assert call_args[1]['cwd'] == Path('/tmp/test') @patch('subprocess.run') def test_run_security_audit_large_prompt_warning(self, mock_run, capsys): """Test warning for large prompts.""" mock_run.return_value = Mock( returncode=0, stdout='{"findings": []}', stderr='' ) # Create a prompt larger than 1MB large_prompt = 'x' * (1024 * 1024 + 1000) runner = SimpleClaudeRunner() with patch('pathlib.Path.exists', return_value=True): success, error, results = runner.run_security_audit( Path('/tmp/test'), large_prompt ) captured = capsys.readouterr() assert '[Warning] Large prompt size' in captured.err assert success is True @patch('subprocess.run') def test_run_security_audit_retry_on_failure(self, mock_run): """Test retry logic on failure.""" # First call fails, second succeeds mock_run.side_effect = [ Mock(returncode=1, stdout='', stderr='Temporary error'), Mock(returncode=0, stdout='{"findings": []}', stderr='') ] runner = SimpleClaudeRunner() with patch('pathlib.Path.exists', return_value=True): success, error, results = runner.run_security_audit( Path('/tmp/test'), "test prompt" ) assert success is True assert error == '' assert mock_run.call_count == 2 # Retried once @patch('subprocess.run') def test_run_security_audit_retry_on_error_during_execution(self, mock_run): """Test retry on error_during_execution result.""" error_result = { "type": "result", "subtype": "error_during_execution", "error": "Temporary execution error" } success_result = { "result": json.dumps({ "findings": [{"file": "test.py", "line": 1, "severity": "LOW", "description": "Issue"}], "analysis_summary": { "files_reviewed": 1, "high_severity": 0, "medium_severity": 0, "low_severity": 1, "review_completed": True } }) } mock_run.side_effect = [ Mock(returncode=0, stdout=json.dumps(error_result), stderr=''), Mock(returncode=0, stdout=json.dumps(success_result), stderr='') ] runner = SimpleClaudeRunner() with patch('pathlib.Path.exists', return_value=True): success, error, results = runner.run_security_audit( Path('/tmp/test'), "test prompt" ) assert success is True assert len(results['findings']) == 1 assert mock_run.call_count == 2 @patch('subprocess.run') def test_run_security_audit_timeout(self, mock_run): """Test timeout handling.""" mock_run.side_effect = subprocess.TimeoutExpired(['claude'], 1200) runner = SimpleClaudeRunner() with patch('pathlib.Path.exists', return_value=True): success, error, results = runner.run_security_audit( Path('/tmp/test'), "test prompt" ) assert success is False assert 'timed out after 20 minutes' in error assert results == {} @patch('subprocess.run') def test_run_security_audit_json_parse_failure_with_retry(self, mock_run): """Test JSON parse failure with retry.""" mock_run.side_effect = [ Mock(returncode=0, stdout='Invalid JSON', stderr=''), Mock(returncode=0, stdout='Still invalid', stderr='') ] runner = SimpleClaudeRunner() with patch('pathlib.Path.exists', return_value=True): success, error, results = runner.run_security_audit( Path('/tmp/test'), "test prompt" ) assert success is False assert 'Failed to parse Claude output' in error assert mock_run.call_count == 2 def test_extract_security_findings_claude_wrapper(self): """Test extraction from Claude Code wrapper format.""" runner = SimpleClaudeRunner() # Test with result field containing JSON string claude_output = { "result": json.dumps({ "findings": [ {"file": "test.py", "line": 10, "severity": "HIGH"} ] }) } result = runner._extract_security_findings(claude_output) assert len(result['findings']) == 1 assert result['findings'][0]['file'] == 'test.py' def test_extract_security_findings_direct_format(self): """Test that direct findings format was removed - only wrapped format is supported.""" runner = SimpleClaudeRunner() # Direct format (without 'result' wrapper) should return empty claude_output = { "findings": [ {"file": "main.py", "line": 20, "severity": "MEDIUM"} ], "analysis_summary": { "files_reviewed": 3, "high_severity": 0, "medium_severity": 1, "low_severity": 0 } } result = runner._extract_security_findings(claude_output) # Should return empty structure since direct format is not supported assert len(result['findings']) == 0 assert result['analysis_summary']['review_completed'] is False def test_extract_security_findings_text_fallback(self): """Test that text fallback was removed - only JSON is supported.""" runner = SimpleClaudeRunner() # Test with result containing text (not JSON) claude_output = { "result": "Found SQL injection vulnerability in database.py line 45" } # Should return empty findings since we don't parse text anymore result = runner._extract_security_findings(claude_output) assert len(result['findings']) == 0 assert result['analysis_summary']['review_completed'] is False def test_extract_security_findings_empty(self): """Test extraction with no findings.""" runner = SimpleClaudeRunner() # Various empty formats for output in [None, {}, {"result": ""}, {"other": "data"}]: result = runner._extract_security_findings(output) assert result['findings'] == [] assert result['analysis_summary']['review_completed'] is False def test_create_findings_from_text(self): """Test that _create_findings_from_text was removed.""" runner = SimpleClaudeRunner() # Method should not exist assert not hasattr(runner, '_create_findings_from_text') def test_create_findings_from_text_no_issues(self): """Test that _create_findings_from_text was removed.""" runner = SimpleClaudeRunner() # Method should not exist assert not hasattr(runner, '_create_findings_from_text') class TestClaudeRunnerEdgeCases: """Test edge cases and error scenarios.""" @patch('subprocess.run') def test_claude_output_formats(self, mock_run): """Test various Claude output formats.""" runner = SimpleClaudeRunner() # Test nested JSON in result - result field should be string nested_output = { "type": "result", "result": json.dumps({ "findings": [ {"file": "test.py", "line": 1, "severity": "HIGH", "description": "Issue"} ] }) } with patch('pathlib.Path.exists', return_value=True): mock_run.return_value = Mock( returncode=0, stdout=json.dumps(nested_output), stderr='' ) success, error, results = runner.run_security_audit( Path('/tmp/test'), "test" ) # Should extract findings from nested structure assert success is True assert len(results['findings']) == 1 @patch('subprocess.run') def test_partial_json_recovery(self, mock_run): """Test recovery from partial JSON output.""" # Simulate truncated JSON partial_json = '{"findings": [{"file": "test.py", "line": 10, "sev' mock_run.return_value = Mock( returncode=0, stdout=partial_json, stderr='' ) runner = SimpleClaudeRunner() with patch('pathlib.Path.exists', return_value=True): success, error, results = runner.run_security_audit( Path('/tmp/test'), "test" ) # Should fail to parse and retry assert mock_run.call_count == 2 @patch('subprocess.run') def test_exception_handling(self, mock_run): """Test general exception handling.""" mock_run.side_effect = Exception("Unexpected err - claudecode/test_eval_engine.py (11772 bytes)
"""Tests for eval_engine module.""" import os from unittest.mock import Mock, patch import pytest import json from claudecode.evals.eval_engine import ( EvaluationEngine, EvalResult, EvalCase, run_single_evaluation ) class TestEvalResult: """Test the EvalResult dataclass.""" def test_eval_result_creation(self): """Test creating an EvalResult instance.""" result = EvalResult( repo_name="test/repo", pr_number=123, description="Test PR", success=True, runtime_seconds=10.5, findings_count=2, detected_vulnerabilities=True ) assert result.repo_name == "test/repo" assert result.pr_number == 123 assert result.description == "Test PR" assert result.success is True assert result.runtime_seconds == 10.5 assert result.findings_count == 2 assert result.detected_vulnerabilities is True assert result.error_message == "" assert result.findings_summary is None def test_eval_result_with_error(self): """Test creating an EvalResult with error.""" result = EvalResult( repo_name="test/repo", pr_number=456, description="Failed PR", success=False, runtime_seconds=5.0, findings_count=0, detected_vulnerabilities=False, error_message="Failed to clone repository" ) assert result.success is False assert result.error_message == "Failed to clone repository" assert result.findings_count == 0 def test_eval_result_with_findings(self): """Test creating an EvalResult with findings.""" findings = [ {"file": "test.py", "line": 10, "severity": "HIGH"} ] result = EvalResult( repo_name="test/repo", pr_number=789, description="PR with findings", success=True, runtime_seconds=15.0, findings_count=1, detected_vulnerabilities=True, findings_summary=findings, full_findings=findings ) assert result.findings_count == 1 assert result.detected_vulnerabilities is True assert result.findings_summary is not None assert len(result.findings_summary) == 1 def test_eval_result_to_dict(self): """Test converting EvalResult to dictionary.""" result = EvalResult( repo_name="test/repo", pr_number=123, description="Test", success=True, runtime_seconds=10.0, findings_count=0, detected_vulnerabilities=False ) result_dict = result.to_dict() assert result_dict['repo_name'] == "test/repo" assert result_dict['pr_number'] == 123 assert result_dict['success'] is True class TestEvalCase: """Test the EvalCase dataclass.""" def test_eval_case_creation(self): """Test creating an EvalCase instance.""" case = EvalCase( repo_name="test/repo", pr_number=123, description="Test case" ) assert case.repo_name == "test/repo" assert case.pr_number == 123 assert case.description == "Test case" class TestEvaluationEngine: """Test the EvaluationEngine class.""" def test_engine_initialization(self): """Test engine initialization with API key.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): engine = EvaluationEngine() assert engine.work_dir == os.path.expanduser("~/code/audit") assert engine.claude_api_key == 'test-key' def test_engine_initialization_no_api_key(self): """Test engine initialization without API key.""" with patch.dict(os.environ, {}, clear=True): with pytest.raises(ValueError, match="ANTHROPIC_API_KEY"): EvaluationEngine() def test_get_eval_branch_name(self): """Test branch name generation.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): engine = EvaluationEngine() case = EvalCase("owner/repo", 123) branch_name = engine._get_eval_branch_name(case) assert branch_name.startswith("eval-pr-owner-repo-123-") assert len(branch_name) > len("eval-pr-owner-repo-123-") @patch('os.path.exists') @patch('subprocess.run') def test_clean_worktrees(self, mock_run, mock_exists): """Test worktree cleanup.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): # Mock git auth token call in __init__ mock_run.side_effect = [ Mock(returncode=1, stdout=""), # gh auth token (fails, no token) Mock(returncode=0), # prune Mock(returncode=0, stdout=""), # list (empty) Mock(returncode=0, stdout=""), # branch --list (empty) ] engine = EvaluationEngine() mock_exists.return_value = True # repo_path exists engine._clean_worktrees("/repo/path", "eval-pr-test-123") # Should call run four times: gh auth token (in __init__), prune, list, branch --list assert mock_run.call_count == 4 @patch('subprocess.run') @patch('os.path.exists') def test_setup_repository_clone(self, mock_exists, mock_run): """Test repository setup with cloning.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): mock_exists.return_value = False # Repository doesn't exist mock_run.side_effect = [ Mock(returncode=1, stdout=""), # gh auth token (fails, no token) Mock(returncode=0), # git clone Mock(returncode=0), # git fetch Mock(returncode=0), # git worktree add ] engine = EvaluationEngine() case = EvalCase("owner/repo", 123) success, worktree_path, error = engine._setup_repository(case) assert success is True assert worktree_path != "" assert error == "" @patch('subprocess.run') @patch('os.path.exists') def test_setup_repository_existing(self, mock_exists, mock_run): """Test repository setup with existing repository.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): # First call checks base_repo_path, second checks repo_path inside _clean_worktrees mock_exists.side_effect = [True, True] mock_run.side_effect = [ Mock(returncode=1, stdout=""), # gh auth token (fails, no token) Mock(returncode=0), # worktree prune Mock(returncode=0, stdout=""), # worktree list Mock(returncode=0, stdout=""), # git branch --list Mock(returncode=0), # git fetch Mock(returncode=0), # git worktree add ] engine = EvaluationEngine() case = EvalCase("owner/repo", 123) success, worktree_path, error = engine._setup_repository(case) assert success is True assert error == "" @patch('subprocess.run') def test_run_sast_audit_success(self, mock_run): """Test successful SAST audit run.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): # Mock gh auth token call first, then the audit mock_run.side_effect = [ Mock(returncode=1, stdout=""), # gh auth token (fails, no token) Mock(returncode=0, stdout=json.dumps({ "findings": [ {"file": "test.py", "line": 10, "severity": "HIGH"} ] }), stderr="") # SAST audit ] engine = EvaluationEngine() case = EvalCase("owner/repo", 123) success, output, parsed, error = engine._run_sast_audit(case, "/repo/path") assert success is True assert parsed is not None assert len(parsed["findings"]) == 1 assert error is None @patch('subprocess.run') def test_run_sast_audit_failure(self, mock_run): """Test failed SAST audit run.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): mock_run.side_effect = [ Mock(returncode=1, stdout=""), # gh auth token (fails, no token) Mock(returncode=1, stdout="", stderr="Error running audit") # SAST audit fails ] engine = EvaluationEngine() case = EvalCase("owner/repo", 123) success, output, parsed, error = engine._run_sast_audit(case, "/repo/path") assert success is False assert error is not None assert "Exit code 1" in error @patch.object(EvaluationEngine, '_setup_repository') @patch.object(EvaluationEngine, '_run_sast_audit') @patch.object(EvaluationEngine, '_cleanup_worktree') def test_run_evaluation_success(self, mock_cleanup, mock_audit, mock_setup): """Test successful evaluation run.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): engine = EvaluationEngine() mock_setup.return_value = (True, "/worktree/path", "") mock_audit.return_value = ( True, '{"findings": [{"file": "test.py", "line": 10}]}', {"findings": [{"file": "test.py", "line": 10}]}, None ) case = EvalCase("owner/repo", 123, "Test PR") result = engine.run_evaluation(case) assert result.success is True assert result.findings_count == 1 assert result.detected_vulnerabilities is True assert result.findings_summary is not None assert len(result.findings_summary) == 1 mock_cleanup.assert_called_once() @patch.object(EvaluationEngine, '_setup_repository') def test_run_evaluation_setup_failure(self, mock_setup): """Test evaluation with repository setup failure.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): engine = EvaluationEngine() mock_setup.return_value = (False, "", "Clone failed") case = EvalCase("owner/repo", 123, "Test PR") result = engine.run_evaluation(case) assert result.success is False assert result.findings_count == 0 assert "Repository setup failed" in result.error_message class TestHelperFunctions: """Test helper functions.""" @patch.object(EvaluationEngine, 'run_evaluation') def test_run_single_evaluation(self, mock_run): """Test run_single_evaluation helper.""" with patch.dict(os.environ, {'ANTHROPIC_API_KEY': 'test-key'}): mock_result = Mock(spec=EvalResult) mock_run.return_value = mock_result case = EvalCase("owner/repo", 123) result = run_single_evaluation(case, verbose=True) assert result == mock_result mock_run.assert_called_once_with(case) - claudecode/test_findings_conversion.py (16124 bytes)
#!/usr/bin/env python3 """ Unit tests for findings conversion and edge cases. """ import pytest import json from claudecode.findings_filter import FindingsFilter, HardExclusionRules from claudecode.json_parser import parse_json_with_fallbacks, extract_json_from_text def create_simple_filter(): """Create a filter that only uses hard exclusions.""" return FindingsFilter(use_hard_exclusions=True, use_claude_filtering=False) def filter_findings_simple(filter_instance, findings): """Helper to get simple kept/excluded tuple from FindingsFilter.""" if findings is None: raise TypeError("'NoneType' object is not iterable") success, results, stats = filter_instance.filter_findings(findings) if success: kept = results.get('filtered_findings', []) excluded = results.get('excluded_findings', []) else: kept = findings excluded = [] return kept, excluded class TestFindingsConversionEdgeCases: """Test edge cases in findings conversion and filtering.""" def test_empty_findings_list(self): """Test filtering empty findings list.""" filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, []) assert kept == [] assert excluded == [] def test_none_findings_list(self): """Test filtering None findings list.""" filter = create_simple_filter() # Should raise TypeError for None input with pytest.raises(TypeError): filter_findings_simple(filter, None) def test_malformed_finding_missing_fields(self): """Test filtering findings with missing required fields.""" findings = [ {'description': 'Issue 1'}, # Missing severity {'severity': 'HIGH'}, # Missing description {}, # Empty finding {'severity': 'HIGH', 'description': 'Valid issue'}, ] filter = create_simple_filter() # The filter will process all findings, even with missing fields kept, excluded = filter_findings_simple(filter, findings) # All findings without exclusion patterns are kept assert len(kept) == 4 assert len(excluded) == 0 def test_finding_with_extra_fields(self): """Test findings with extra/unexpected fields.""" findings = [ { 'severity': 'HIGH', 'description': 'SQL injection', 'extra_field': 'value', 'nested': {'data': 'here'}, 'array': [1, 2, 3] } ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # Extra fields should be preserved assert len(kept) == 1 assert kept[0]['extra_field'] == 'value' assert kept[0]['nested'] == {'data': 'here'} def test_unicode_in_findings(self): """Test findings with unicode characters.""" findings = [ { 'severity': 'HIGH', 'description': 'SQL injection in 用户输入', 'file': 'файл.py', 'exploit_scenario': 'Attacker könnte dies ausnutzen' } ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) assert len(kept) == 1 assert '用户输入' in kept[0]['description'] assert kept[0]['file'] == 'файл.py' def test_very_long_description(self): """Test findings with very long descriptions.""" long_desc = 'A' * 10000 # 10k character description findings = [ { 'severity': 'HIGH', 'description': f'SQL injection vulnerability. {long_desc}' } ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # Should not crash on long descriptions assert len(kept) == 1 assert len(kept[0]['description']) > 10000 def test_special_characters_in_description(self): """Test findings with special regex characters.""" findings = [ {'severity': 'HIGH', 'description': 'Issue with [brackets] and (parens)'}, {'severity': 'HIGH', 'description': 'Path: C:\\Users\\test\\file.py'}, {'severity': 'HIGH', 'description': 'Regex pattern: .*$^[]{}'}, {'severity': 'HIGH', 'description': 'Missing rate limiting for API'}, ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # Special characters shouldn't break filtering assert len(kept) == 3 # "Missing rate limiting" should be excluded assert len(excluded) == 1 def test_case_sensitivity_in_exclusions(self): """Test case sensitivity in exclusion rules.""" findings = [ {'severity': 'HIGH', 'description': 'DENIAL OF SERVICE attack'}, {'severity': 'HIGH', 'description': 'Denial Of Service issue'}, {'severity': 'HIGH', 'description': 'dos vulnerability'}, {'severity': 'HIGH', 'description': 'DoS attack vector'}, ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # DOS patterns use word boundaries, "dos vulnerability" doesn't match \bdos attack\b assert len(kept) == 1 # "dos vulnerability" is kept assert len(excluded) == 3 assert kept[0]['description'] == 'dos vulnerability' def test_severity_normalization(self): """Test various severity formats.""" findings = [ {'severity': 'high', 'description': 'Issue 1'}, {'severity': 'HIGH', 'description': 'Issue 2'}, {'severity': 'High', 'description': 'Issue 3'}, {'severity': 'CRITICAL', 'description': 'Issue 4'}, {'severity': 'unknown', 'description': 'Issue 5'}, {'severity': '', 'description': 'Issue 6'}, {'severity': None, 'description': 'Issue 7'}, ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # All should be processed regardless of severity format assert len(kept) == 7 def test_json_injection_in_findings(self): """Test findings that might contain JSON injection attempts.""" findings = [ { 'severity': 'HIGH', 'description': '{"injected": "json", "description": "fake"}' }, { 'severity': 'HIGH', 'description': 'Issue with "}]} payload' } ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # Should handle JSON-like content in descriptions assert len(kept) == 2 class TestJsonParserEdgeCases: """Test JSON parser edge cases.""" def test_parse_empty_string(self): """Test parsing empty string.""" success, result = parse_json_with_fallbacks('', 'test') assert success is False # Empty string returns error structure assert 'error' in result assert "Invalid JSON response" in result['error'] def test_parse_whitespace_only(self): """Test parsing whitespace-only string.""" success, result = parse_json_with_fallbacks(' \n\t ', 'test') assert success is False # Whitespace returns error structure assert 'error' in result assert "Invalid JSON response" in result['error'] def test_parse_truncated_json(self): """Test parsing truncated JSON.""" truncated = '{"findings": [{"severity": "HIGH", "desc' success, result = parse_json_with_fallbacks(truncated, 'test') assert success is False def test_parse_json_with_comments(self): """Test parsing JSON with comments (invalid JSON).""" json_with_comments = """{ "findings": [ // This is a comment {"severity": "HIGH", "description": "Issue"} ] }""" success, result = parse_json_with_fallbacks(json_with_comments, 'test') assert success is False def test_parse_json_with_trailing_comma(self): """Test parsing JSON with trailing comma.""" json_with_comma = '{"findings": [{"severity": "HIGH"},]}' success, result = parse_json_with_fallbacks(json_with_comma, 'test') assert success is False def test_parse_nested_json_string(self): """Test parsing JSON embedded in string.""" nested = '{"result": "{\\"findings\\": [{\\"severity\\": \\"HIGH\\"}]}"}' success, result = parse_json_with_fallbacks(nested, 'test') assert success is True assert 'result' in result def test_extract_json_from_text_edge_cases(self): """Test JSON extraction from various text formats.""" # No JSON assert extract_json_from_text('Just plain text') is None # Multiple JSON objects text = 'First: {"a": 1} Second: {"b": 2}' result = extract_json_from_text(text) assert result == {"a": 1} # Should extract first # JSON in markdown code block text = '''```json {"findings": [{"severity": "HIGH"}]} ```''' result = extract_json_from_text(text) assert result is not None assert 'findings' in result # Malformed JSON attempts text = 'Result: {invalid json}' assert extract_json_from_text(text) is None # Very large JSON large_obj = {"data": ["x" * 100 for _ in range(1000)]} text = f"Result: {json.dumps(large_obj)}" result = extract_json_from_text(text) assert result is not None assert len(result['data']) == 1000 def test_extract_json_with_unicode(self): """Test JSON extraction with unicode.""" text = 'Result: {"message": "Error: 文件未找到"}' result = extract_json_from_text(text) assert result is not None assert result['message'] == "Error: 文件未找到" def test_parse_json_arrays(self): """Test parsing JSON arrays.""" # Direct array success, result = parse_json_with_fallbacks('[1, 2, 3]', 'test') assert success is True assert result == [1, 2, 3] # Array of findings findings_array = '[{"severity": "HIGH", "description": "Issue"}]' success, result = parse_json_with_fallbacks(findings_array, 'test') assert success is True assert isinstance(result, list) assert len(result) == 1 class TestHardExclusionRulesEdgeCases: """Test hard exclusion rules edge cases.""" def test_overlapping_patterns(self): """Test findings that match multiple exclusion patterns.""" finding = { 'severity': 'HIGH', 'description': 'Denial of service via rate limiting bypass allows brute force attack' } # Matches both DOS and rate limiting patterns reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert "DOS" in reason # Should match DOS pattern first def test_pattern_boundary_matching(self): """Test pattern matching at word boundaries.""" findings = [ {'severity': 'HIGH', 'description': 'dosomething() function'}, # Should not match DOS {'severity': 'HIGH', 'description': 'windows path issue'}, # Should not match {'severity': 'HIGH', 'description': 'pseudorandom number'}, # Should not match ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # None should be excluded (no word boundary match) assert len(kept) == 3 assert len(excluded) == 0 def test_html_entities_in_description(self): """Test findings with HTML entities.""" findings = [ {'severity': 'HIGH', 'description': 'XSS via <script> tag'}, {'severity': 'HIGH', 'description': 'Missing rate limiting & throttling'}, ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # "Missing rate limiting" should be excluded even with HTML entity assert len(kept) == 1 assert len(excluded) == 1 assert 'XSS' in kept[0]['description'] def test_multiline_descriptions(self): """Test findings with multiline descriptions.""" findings = [ { 'severity': 'HIGH', 'description': '''SQL injection vulnerability in user input handling. This could lead to data exposure.''' }, { 'severity': 'HIGH', 'description': '''Performance issue that could cause denial of service under heavy load conditions.''' } ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) # DOS should be found even across lines assert len(kept) == 1 assert len(excluded) == 1 assert 'SQL injection' in kept[0]['description'] class TestFilteringCombinations: """Test combinations of filtering scenarios.""" def test_mixed_valid_invalid_findings(self): """Test mix of valid, invalid, and excludable findings.""" findings = [ {'severity': 'HIGH', 'description': 'SQL injection'}, # Valid {'description': 'Missing severity'}, # Valid (no exclusion pattern) {'severity': 'HIGH', 'description': 'Missing rate limiting'}, # Excludable {'severity': 'MEDIUM', 'description': 'XSS vulnerability'}, # Valid {'severity': 'LOW', 'description': 'Denial of service attack'}, # Excludable {'severity': '', 'description': ''}, # Valid (no exclusion pattern) {'severity': 'HIGH', 'description': 'RCE possibility'}, # Valid ] filter = create_simple_filter() kept, excluded = filter_findings_simple(filter, findings) assert len(kept) == 5 # All except rate limiting and DOS assert len(excluded) == 2 # Rate limiting, DOS # Verify excluded findings excluded_descs = [e['finding']['description'] for e in excluded] assert 'Missing rate limiting' in excluded_descs assert 'Denial of service attack' in excluded_descs def test_duplicate_findings - claudecode/test_github_action_audit.py (10531 bytes)
#!/usr/bin/env python3 """ Pytest tests for GitHub Action audit script components. """ class TestImports: """Test that all required modules can be imported.""" def test_main_module_import(self): """Test that the main module can be imported.""" from claudecode import github_action_audit assert hasattr(github_action_audit, 'GitHubActionClient') assert hasattr(github_action_audit, 'SimpleClaudeRunner') # SimpleFindingsFilter was removed assert hasattr(github_action_audit, 'main') def test_component_imports(self): """Test that all component modules can be imported.""" from claudecode.prompts import get_security_audit_prompt from claudecode.json_parser import parse_json_with_fallbacks, extract_json_from_text # Verify they're callable/usable assert callable(get_security_audit_prompt) assert callable(parse_json_with_fallbacks) assert callable(extract_json_from_text) class TestHardExclusionRules: """Test the HardExclusionRules patterns.""" def test_dos_patterns(self): """Test DOS pattern exclusions.""" from claudecode.findings_filter import HardExclusionRules dos_findings = [ {'description': 'Potential denial of service vulnerability'}, {'description': 'DOS attack through resource exhaustion'}, {'description': 'Infinite loop causing resource exhaustion'}, ] for finding in dos_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert 'dos' in reason.lower() def test_rate_limiting_patterns(self): """Test rate limiting pattern exclusions.""" from claudecode.findings_filter import HardExclusionRules rate_limit_findings = [ {'description': 'Missing rate limiting on endpoint'}, {'description': 'No rate limit implemented for API'}, {'description': 'Implement rate limiting for this route'}, ] for finding in rate_limit_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert 'rate limit' in reason.lower() def test_open_redirect_patterns(self): """Test open redirect pattern exclusions.""" from claudecode.findings_filter import HardExclusionRules redirect_findings = [ {'description': 'Open redirect vulnerability found'}, {'description': 'Unvalidated redirect in URL parameter'}, {'description': 'Redirect attack possible through user input'}, ] for finding in redirect_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert 'open redirect' in reason.lower() def test_markdown_file_exclusion(self): """Test that findings in .md files are excluded.""" from claudecode.findings_filter import HardExclusionRules md_findings = [ {'file': 'README.md', 'description': 'SQL injection vulnerability'}, {'file': 'docs/security.md', 'description': 'Command injection found'}, {'file': 'CHANGELOG.MD', 'description': 'XSS vulnerability'}, # Test case insensitive {'file': 'path/to/file.Md', 'description': 'Path traversal'}, # Mixed case ] for finding in md_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert 'markdown' in reason.lower() def test_non_markdown_files_not_excluded(self): """Test that findings in non-.md files are not excluded due to file extension.""" from claudecode.findings_filter import HardExclusionRules non_md_findings = [ {'file': 'main.py', 'description': 'SQL injection vulnerability'}, {'file': 'server.js', 'description': 'Command injection found'}, {'file': 'index.html', 'description': 'XSS vulnerability'}, {'file': 'config.yml', 'description': 'Hardcoded credentials'}, {'file': 'README.txt', 'description': 'Path traversal'}, {'file': 'file.mdx', 'description': 'Security issue'}, # Not .md ] for finding in non_md_findings: reason = HardExclusionRules.get_exclusion_reason(finding) # Should not be excluded for being a markdown file # (might be excluded for other reasons like DOS patterns) if reason: assert 'markdown' not in reason.lower() def test_keeps_real_vulnerabilities(self): """Test that real vulnerabilities are not excluded.""" from claudecode.findings_filter import HardExclusionRules real_vulns = [ {'file': 'auth.py', 'description': 'SQL injection in user authentication'}, {'file': 'exec.js', 'description': 'Command injection through user input'}, {'file': 'comments.php', 'description': 'Cross-site scripting in comment field'}, {'file': 'upload.go', 'description': 'Path traversal in file upload'}, ] for finding in real_vulns: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is None class TestJSONParser: """Test JSON parsing utilities.""" def test_parse_valid_json(self): """Test parsing valid JSON.""" from claudecode.json_parser import parse_json_with_fallbacks valid_json = '{"test": "data", "number": 123}' success, result = parse_json_with_fallbacks(valid_json, "test") assert success is True assert result == {"test": "data", "number": 123} def test_parse_invalid_json(self): """Test parsing invalid JSON.""" from claudecode.json_parser import parse_json_with_fallbacks invalid_json = '{invalid json}' success, result = parse_json_with_fallbacks(invalid_json, "test") assert success is False assert 'error' in result assert 'Invalid JSON response' in result['error'] def test_extract_json_from_text(self): """Test extracting JSON from mixed text.""" from claudecode.json_parser import extract_json_from_text mixed_text = 'Some text before {"key": "value"} some text after' result = extract_json_from_text(mixed_text) assert result == {"key": "value"} def test_extract_json_from_text_no_json(self): """Test extracting JSON when none exists.""" from claudecode.json_parser import extract_json_from_text plain_text = 'This is just plain text with no JSON' result = extract_json_from_text(plain_text) assert result is None class TestPromptsModule: """Test the prompts module.""" def test_get_security_audit_prompt(self): """Test security audit prompt generation.""" from claudecode.prompts import get_security_audit_prompt pr_data = { 'number': 123, 'title': 'Test PR', 'body': 'Test description', 'user': 'testuser', 'changed_files': 1, 'additions': 10, 'deletions': 5, 'head': { 'repo': { 'full_name': 'owner/repo' } }, 'files': [ { 'filename': 'test.py', 'status': 'modified', 'additions': 10, 'deletions': 5, 'patch': '@@ -1,5 +1,10 @@\n+added line' } ] } pr_diff = "diff --git a/test.py b/test.py\n+added line" prompt = get_security_audit_prompt(pr_data, pr_diff) assert isinstance(prompt, str) assert 'security' in prompt.lower() assert 'PR #123' in prompt assert 'test.py' in prompt class TestDeploymentPRDetection: """Test deployment PR title pattern matching.""" def test_deployment_pr_patterns(self): """Test that deployment PR titles are correctly identified.""" import re deployment_pattern = r'^Deploy\s+[a-f0-9]{6,}\s+to\s+(production|staging|development|production-services)' # These should match deployment_titles = [ "Deploy 53f395b0 to production-services", "Deploy af179b5b to production", "Deploy 1a3cb909 to production", "Deploy 49c09ea5 to production-services", "Deploy 8e7acc60 to production", "Deploy e0b1fe0b to production-services", "Deploy c53e6010 to production", "Deploy 42c4a061 to production", "Deploy 9de55976 to production-services", "deploy abcdef123456 to staging", # lowercase should work "DEPLOY ABCDEF01 TO DEVELOPMENT", # uppercase should work ] for title in deployment_titles: assert re.match(deployment_pattern, title, re.IGNORECASE), f"Failed to match deployment PR: {title}" def test_non_deployment_pr_patterns(self): """Test that non-deployment PR titles are not matched.""" import re deployment_pattern = r'^Deploy\s+[a-f0-9]{6,}\s+to\s+(production|staging|development|production-services)' # These should NOT match non_deployment_titles = [ "Add new feature", "Fix bug in deployment script", "Update deployment documentation", "Deploy new feature to production", # No commit hash "Deploy abc to production", # Too short hash "Deploy 12345g to production", # Non-hex character "Preparing deploy af179b5b to production", # Doesn't start with Deploy "Deploy af179b5b to testing", # Wrong environment "Deploy af179b5b", # Missing environment "af179b5b to production", # Missing Deploy prefix ] for title in non_deployment_titles: assert not re.match(deployment_pattern, title, re.IGNORECASE), f"Incorrectly matched non-deployment PR: {title}" - claudecode/test_github_client.py (11940 bytes)
#!/usr/bin/env python3 """ Unit tests for GitHubActionClient. """ import pytest import os from unittest.mock import Mock, patch from claudecode.github_action_audit import GitHubActionClient class TestGitHubActionClient: """Test GitHubActionClient functionality.""" def test_init_requires_token(self): """Test that client initialization requires GITHUB_TOKEN.""" # Remove token if it exists original_token = os.environ.pop('GITHUB_TOKEN', None) try: with pytest.raises(ValueError, match="GITHUB_TOKEN environment variable required"): GitHubActionClient() finally: # Restore token if original_token: os.environ['GITHUB_TOKEN'] = original_token def test_init_with_token(self): """Test successful initialization with token.""" with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() assert client.github_token == 'test-token' assert client.headers['Authorization'] == 'Bearer test-token' assert 'Accept' in client.headers assert 'X-GitHub-Api-Version' in client.headers @patch('requests.get') def test_get_pr_data_success(self, mock_get): """Test successful PR data retrieval.""" # Mock responses pr_response = Mock() pr_response.json.return_value = { 'number': 123, 'title': 'Test PR', 'body': 'PR description', 'user': {'login': 'testuser'}, 'created_at': '2024-01-01T00:00:00Z', 'updated_at': '2024-01-01T01:00:00Z', 'state': 'open', 'head': { 'ref': 'feature-branch', 'sha': 'abc123', 'repo': { 'full_name': 'owner/repo' } }, 'base': { 'ref': 'main', 'sha': 'def456' }, 'additions': 50, 'deletions': 10, 'changed_files': 3 } files_response = Mock() files_response.json.return_value = [ { 'filename': 'src/main.py', 'status': 'modified', 'additions': 30, 'deletions': 5, 'changes': 35, 'patch': '@@ -1,5 +1,10 @@\n+import os\n def main():' }, { 'filename': 'tests/test_main.py', 'status': 'added', 'additions': 20, 'deletions': 5, 'changes': 25, 'patch': '@@ -0,0 +1,20 @@\n+def test_main():' } ] mock_get.side_effect = [pr_response, files_response] with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() result = client.get_pr_data('owner/repo', 123) # Verify API calls assert mock_get.call_count == 2 mock_get.assert_any_call( 'https://api.github.com/repos/owner/repo/pulls/123', headers=client.headers ) mock_get.assert_any_call( 'https://api.github.com/repos/owner/repo/pulls/123/files?per_page=100', headers=client.headers ) # Verify result structure assert result['number'] == 123 assert result['title'] == 'Test PR' assert result['user'] == 'testuser' assert len(result['files']) == 2 assert result['files'][0]['filename'] == 'src/main.py' assert result['files'][1]['status'] == 'added' @patch('requests.get') def test_get_pr_data_null_head_repo(self, mock_get): """Test PR data retrieval when head repo is null (deleted fork).""" pr_response = Mock() pr_response.json.return_value = { 'number': 123, 'title': 'Test PR', # Don't include body key to test the get() default 'user': {'login': 'testuser'}, 'created_at': '2024-01-01T00:00:00Z', 'updated_at': '2024-01-01T01:00:00Z', 'state': 'open', 'head': { 'ref': 'feature-branch', 'sha': 'abc123', 'repo': None # Deleted fork }, 'base': { 'ref': 'main', 'sha': 'def456' }, 'additions': 50, 'deletions': 10, 'changed_files': 3 } files_response = Mock() files_response.json.return_value = [] mock_get.side_effect = [pr_response, files_response] with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() result = client.get_pr_data('owner/repo', 123) # Should use original repo name when head repo is None assert result['head']['repo']['full_name'] == 'owner/repo' # The implementation passes None through, test should match that assert result['body'] == '' @patch('requests.get') def test_get_pr_data_api_error(self, mock_get): """Test PR data retrieval with API error.""" mock_response = Mock() mock_response.raise_for_status.side_effect = Exception("API Error") mock_get.return_value = mock_response with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() with pytest.raises(Exception, match="API Error"): client.get_pr_data('owner/repo', 123) @patch('requests.get') def test_get_pr_diff_success(self, mock_get): """Test successful PR diff retrieval.""" diff_content = """diff --git a/src/main.py b/src/main.py index abc123..def456 100644 --- a/src/main.py +++ b/src/main.py @@ -1,5 +1,10 @@ +import os def main(): print("Hello") + # New feature + process_data() """ mock_response = Mock() mock_response.text = diff_content mock_response.raise_for_status.return_value = None mock_get.return_value = mock_response with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() result = client.get_pr_diff('owner/repo', 123) # Verify API call mock_get.assert_called_once() call_args = mock_get.call_args assert call_args[0][0] == 'https://api.github.com/repos/owner/repo/pulls/123' assert call_args[1]['headers']['Accept'] == 'application/vnd.github.diff' # Verify result assert 'import os' in result assert 'process_data()' in result @patch('requests.get') def test_get_pr_diff_filters_generated_files(self, mock_get): """Test that generated files are filtered from diff.""" diff_with_generated = """diff --git a/src/main.py b/src/main.py index abc123..def456 100644 --- a/src/main.py +++ b/src/main.py @@ -1,5 +1,10 @@ +import os def main(): print("Hello") diff --git a/generated/code.py b/generated/code.py index 111..222 100644 --- a/generated/code.py +++ b/generated/code.py @@ -1,3 +1,5 @@ # @generated by protoc +# More generated code +print("generated") diff --git a/src/feature.py b/src/feature.py index 333..444 100644 --- a/src/feature.py +++ b/src/feature.py @@ -1,3 +1,5 @@ +# Real code def feature(): pass """ mock_response = Mock() mock_response.text = diff_with_generated mock_response.raise_for_status.return_value = None mock_get.return_value = mock_response with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() result = client.get_pr_diff('owner/repo', 123) # Verify generated file is filtered out assert 'src/main.py' in result assert 'src/feature.py' in result assert 'generated/code.py' not in result assert '@generated' not in result assert 'More generated code' not in result def test_filter_generated_files_edge_cases(self): """Test edge cases in generated file filtering.""" with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() # Empty diff assert client._filter_generated_files('') == '' # No diff markers - if no diff format, everything is filtered text = "Just some random text\nwith @generated in it" # Since there's no 'diff --git' marker, the split results in one section # that contains @generated, so it gets filtered out assert client._filter_generated_files(text) == '' # Multiple generated markers diff = """diff --git a/a.py b/a.py @generated by tool content diff --git a/b.py b/b.py normal content diff --git a/c.py b/c.py # This file is @generated more content """ result = client._filter_generated_files(diff) assert 'a.py' not in result assert 'b.py' in result assert 'c.py' not in result class TestGitHubAPIIntegration: """Test GitHub API integration scenarios.""" @patch('requests.get') def test_rate_limit_handling(self, mock_get): """Test that rate limit headers are respected.""" mock_response = Mock() mock_response.headers = { 'X-RateLimit-Remaining': '0', 'X-RateLimit-Reset': '1234567890' } mock_response.status_code = 403 mock_response.json.return_value = { 'message': 'API rate limit exceeded' } mock_response.raise_for_status.side_effect = Exception("Rate limit exceeded") mock_get.return_value = mock_response with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() with pytest.raises(Exception, match="Rate limit exceeded"): client.get_pr_data('owner/repo', 123) @patch('requests.get') def test_pagination_not_needed_for_pr_files(self, mock_get): """Test that PR files endpoint returns all files without pagination.""" # GitHub API returns up to 3000 files per PR without pagination large_file_list = [ { 'filename': f'file{i}.py', 'status': 'added', 'additions': 10, 'deletions': 0, 'changes': 10, 'patch': f'@@ -0,0 +1,10 @@\n+# File {i}' } for i in range(100) # 100 files ] pr_response = Mock() pr_response.json.return_value = { 'number': 123, 'title': 'Large PR', 'body': 'Many files', 'user': {'login': 'testuser'}, 'created_at': '2024-01-01T00:00:00Z', 'updated_at': '2024-01-01T01:00:00Z', 'state': 'open', 'head': {'ref': 'feature', 'sha': 'abc123', 'repo': {'full_name': 'owner/repo'}}, 'base': {'ref': 'main', 'sha': 'def456'}, 'additions': 1000, 'deletions': 0, 'changed_files': 100 } files_response = Mock() files_response.json.return_value = large_file_list mock_get.side_effect = [pr_response, files_response] with patch.dict(os.environ, {'GITHUB_TOKEN': 'test-token'}): client = GitHubActionClient() result = client.get_pr_data('owner/repo', 123) assert len(result['files']) == 100 assert result['files'][0]['filename'] == 'file0.py' assert result['files'][99]['filename'] == 'file99.py' - claudecode/test_hard_exclusion_rules.py (16723 bytes)
"""Unit tests for HardExclusionRules in findings_filter module.""" from claudecode.findings_filter import HardExclusionRules class TestHardExclusionRules: """Test the HardExclusionRules class for filtering false positives.""" def test_dos_pattern_exclusion(self): """Test exclusion of DOS-related findings.""" dos_findings = [ { "title": "Potential Denial of Service", "description": "This could lead to resource exhaustion" }, { "title": "Resource consumption issue", "description": "Unbounded loop could exhaust CPU resources" }, { "title": "Memory exhaustion", "description": "This function could overwhelm memory with large inputs" }, { "title": "Stack overflow vulnerability", "description": "Infinite recursion detected" } ] for finding in dos_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert "DOS/resource exhaustion" in reason def test_dos_pattern_not_excluded_with_exploit(self): """Test that stack overflow with exploit mention is not excluded.""" finding = { "title": "Stack overflow exploit", "description": "This stack overflow can be exploited to execute arbitrary code", "file": "exploit.c" # Add C file so it's not excluded by memory safety rule } reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is None # Should not be excluded def test_generic_validation_pattern_exclusion(self): """Test that generic validation findings are NOT excluded anymore.""" validation_findings = [ { "title": "Security Issue", "description": "Missing input validation" }, { "title": "Security Issue", "description": "Input validation required" }, { "title": "Security Issue", "description": "Validate parameters" }, { "title": "Security Issue", "description": "Add input validation" } ] # Since we removed generic validation patterns, these should NOT be excluded for finding in validation_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is None def test_specific_validation_not_excluded(self): """Test that specific validation issues are not excluded.""" specific_findings = [ { "title": "Missing input validation", "description": "SQL injection possible due to missing validation" }, { "title": "No validation", "description": "Command injection vulnerability - validate shell commands" }, { "title": "Missing validation", "description": "Path traversal - validate file paths" }, { "title": "Add validation", "description": "Eval() used without input validation" } ] for finding in specific_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is None # Should not be excluded due to specific context def test_secrets_pattern_exclusion(self): """Test that generic secrets warnings are NOT excluded anymore.""" secrets_findings = [ { "title": "Hardcoded password detected", "description": "Avoid hardcoding credentials in source code" }, { "title": "Plaintext secrets", "description": "Credentials stored in plaintext" }, { "title": "Embedded token", "description": "API key in source code" }, { "title": "Password storage", "description": "Password stored in clear text" } ] # Since we removed secrets patterns, these should NOT be excluded for finding in secrets_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is None def test_actual_secrets_not_excluded(self): """Test that actual exposed secrets are not excluded.""" actual_secrets = [ { "title": "Hardcoded password", "description": "Found actual password: 'admin123' in config file" }, { "title": "API key exposed", "description": "Discovered API key in source: sk-1234567890" }, { "title": "Plaintext password", "description": "Database password 'mypass' found in code" } ] for finding in actual_secrets: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is None # Should not be excluded def test_rate_limiting_pattern_exclusion(self): """Test exclusion of rate limiting recommendations.""" rate_limit_findings = [ { "title": "Missing rate limit", "description": "API endpoint has no rate limiting" }, { "title": "Rate limiting required", "description": "Implement rate limiting for this endpoint" }, { "title": "No rate limit", "description": "Unlimited requests allowed" }, { "title": "Add rate limiting", "description": "This API needs rate limits" } ] for finding in rate_limit_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert "rate limiting recommendation" in reason def test_resource_pattern_exclusion(self): """Test exclusion of generic resource management findings.""" resource_findings = [ { "title": "Security Issue", "description": "Potential memory leak detected" }, { "title": "Security Issue", "description": "Resource leak potential in file handling" }, { "title": "Security Issue", "description": "Unclosed resource detected in function" }, { "title": "Security Issue", "description": "File cleanup required - close resource" } ] for finding in resource_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert "Resource management finding" in reason def test_specific_resource_also_excluded(self): """Test that ALL resource issues are now excluded (including specific ones).""" specific_resources = [ { "title": "Database connection leak", "description": "PostgreSQL connections not returned to pool" }, { "title": "Thread leak", "description": "Thread pool exhaustion due to unclosed threads" }, { "title": "Socket leak", "description": "TCP sockets remain open after errors" } ] # All resource issues should be excluded now for finding in specific_resources: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert "Resource management finding" in reason # Should not be excluded def test_open_redirect_pattern_exclusion(self): """Test exclusion of open redirect findings.""" redirect_findings = [ { "title": "Open redirect vulnerability", "description": "User input used in redirect without validation" }, { "title": "Unvalidated redirect", "description": "Redirect URL not validated" }, { "title": "Redirect vulnerability", "description": "Possible redirect attack" }, { "title": "Malicious redirect possible", "description": "User-controlled redirect parameter" } ] for finding in redirect_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert "Open redirect" in reason def test_mixed_case_handling(self): """Test that pattern matching handles mixed case correctly.""" mixed_case_findings = [ { "title": "DENIAL OF SERVICE", "description": "RESOURCE EXHAUSTION POSSIBLE" }, { "title": "Security Issue", "description": "ADD INPUT VALIDATION" }, { "title": "Security Issue", "description": "HARDCODED PASSWORD DETECTED" } ] # First finding should be excluded (DOS) reason = HardExclusionRules.get_exclusion_reason(mixed_case_findings[0]) assert reason is not None # Second finding should NOT be excluded (we removed generic validation patterns) reason = HardExclusionRules.get_exclusion_reason(mixed_case_findings[1]) assert reason is None # Third finding should NOT be excluded (we removed secrets patterns) reason = HardExclusionRules.get_exclusion_reason(mixed_case_findings[2]) assert reason is None def test_empty_finding_handling(self): """Test handling of empty or malformed findings.""" empty_findings = [ {}, {"title": "", "description": ""}, {"title": "Some title"}, # Missing description {"description": "Some description"}, # Missing title {"title": None, "description": None} ] for finding in empty_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is None # Should not crash, just return None def test_combined_patterns(self): """Test findings that match multiple patterns.""" finding = { "title": "DOS and validation issue", "description": "Missing rate limit leads to resource exhaustion" } reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None # Should match at least one pattern (DOS or rate limiting) def test_regex_special_characters(self): """Test that regex special characters in findings don't cause issues.""" findings_with_special_chars = [ { "title": "Issue with $pecial ch@rs", "description": "Contains [brackets] and (parentheses)" }, { "title": "Path: C:\\Windows\\System32", "description": "Backslashes \\ and dots ..." }, { "title": "Regex chars: .* + ? ^ $ { } ( ) [ ] \\ |", "description": "All the special regex characters" } ] for finding in findings_with_special_chars: # Should not raise regex errors reason = HardExclusionRules.get_exclusion_reason(finding) # These don't match any patterns, so should return None assert reason is None def test_performance_with_long_text(self): """Test performance with very long descriptions.""" long_text = "A" * 10000 # 10k characters finding = { "title": "Long finding", "description": long_text + " denial of service " + long_text } # Should handle long text efficiently reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None # Should find DOS pattern assert "DOS/resource exhaustion" in reason def test_memory_safety_exclusion_non_cpp_files(self): """Test that memory safety issues are excluded in non-C/C++ files.""" memory_safety_findings = [ { "title": "Buffer overflow vulnerability", "description": "Potential buffer overflow in string handling", "file": "app.py" }, { "title": "Out of bounds access", "description": "Array out of bounds write detected", "file": "server.js" }, { "title": "Memory corruption", "description": "Use after free vulnerability found", "file": "Main.java" }, { "title": "Segmentation fault", "description": "Null pointer dereference causes segfault", "file": "handler.go" }, { "title": "Integer overflow", "description": "Integer overflow in calculation", "file": "calc.rb" } ] for finding in memory_safety_findings: reason = HardExclusionRules.get_exclusion_reason(finding) assert reason is not None assert "Memory safety finding in non-C/C++ code" in reason def test_memory_safety_not_excluded_cpp_files(self): """Test that memory safety issues are NOT excluded in C/C++ files.""" cpp_memory_findings = [ { "title": "Buffer overflow", "description": "Stack buffer overflow in strcpy", "file": "main.c" }, { "title": "Out of bounds write", "description": "Array index out of bounds", "file": "parser.cc" }, { "title": "Memory safety", "description": "Use after free in destructor", "file": "object.cpp" }, { "title": "Bounds check missing", "description": "No bounds checking on user input", "file": "input.h" } ] - claudecode/test_helper_functions.py (10195 bytes)
"""Unit tests for helper functions in github_action_audit module.""" import pytest import os from unittest.mock import patch, MagicMock from claudecode.github_action_audit import ( get_environment_config, initialize_clients, initialize_findings_filter, run_security_audit, apply_findings_filter, ConfigurationError, AuditError ) from claudecode.findings_filter import FindingsFilter class TestHelperFunctions: """Test helper functions in github_action_audit module.""" def test_get_environment_config_success(self): """Test successful environment configuration retrieval.""" with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123' }): repo_name, pr_number = get_environment_config() assert repo_name == 'owner/repo' assert pr_number == 123 def test_get_environment_config_missing_repository(self): """Test error when GITHUB_REPOSITORY is missing.""" with patch.dict(os.environ, {'PR_NUMBER': '123'}, clear=True): with pytest.raises(ConfigurationError) as exc_info: get_environment_config() assert "GITHUB_REPOSITORY environment variable required" in str(exc_info.value) def test_get_environment_config_missing_pr_number(self): """Test error when PR_NUMBER is missing.""" with patch.dict(os.environ, {'GITHUB_REPOSITORY': 'owner/repo'}, clear=True): with pytest.raises(ConfigurationError) as exc_info: get_environment_config() assert "PR_NUMBER environment variable required" in str(exc_info.value) def test_get_environment_config_invalid_pr_number(self): """Test error when PR_NUMBER is not a valid integer.""" with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': 'not-a-number' }): with pytest.raises(ConfigurationError) as exc_info: get_environment_config() assert "Invalid PR_NUMBER" in str(exc_info.value) @patch('claudecode.github_action_audit.GitHubActionClient') @patch('claudecode.github_action_audit.SimpleClaudeRunner') def test_initialize_clients_success(self, mock_claude_runner, mock_github_client): """Test successful client initialization.""" mock_github_instance = MagicMock() mock_claude_instance = MagicMock() mock_github_client.return_value = mock_github_instance mock_claude_runner.return_value = mock_claude_instance github_client, claude_runner = initialize_clients() assert github_client == mock_github_instance assert claude_runner == mock_claude_instance mock_github_client.assert_called_once() mock_claude_runner.assert_called_once() @patch('claudecode.github_action_audit.GitHubActionClient') def test_initialize_clients_github_failure(self, mock_github_client): """Test error when GitHub client initialization fails.""" mock_github_client.side_effect = Exception("GitHub API error") with pytest.raises(ConfigurationError) as exc_info: initialize_clients() assert "Failed to initialize GitHub client" in str(exc_info.value) assert "GitHub API error" in str(exc_info.value) @patch('claudecode.github_action_audit.GitHubActionClient') @patch('claudecode.github_action_audit.SimpleClaudeRunner') def test_initialize_clients_claude_failure(self, mock_claude_runner, mock_github_client): """Test error when Claude runner initialization fails.""" mock_github_client.return_value = MagicMock() mock_claude_runner.side_effect = Exception("Claude init error") with pytest.raises(ConfigurationError) as exc_info: initialize_clients() assert "Failed to initialize Claude runner" in str(exc_info.value) assert "Claude init error" in str(exc_info.value) @patch('claudecode.github_action_audit.FindingsFilter') def test_initialize_findings_filter_with_claude(self, mock_filter): """Test initializing findings filter with Claude API enabled.""" mock_filter_instance = MagicMock() mock_filter.return_value = mock_filter_instance with patch.dict(os.environ, { 'ENABLE_CLAUDE_FILTERING': 'true', 'ANTHROPIC_API_KEY': 'test-key-123' }): result = initialize_findings_filter() assert result == mock_filter_instance mock_filter.assert_called_once_with( use_hard_exclusions=True, use_claude_filtering=True, api_key='test-key-123', custom_filtering_instructions=None ) @patch('claudecode.github_action_audit.FindingsFilter') def test_initialize_findings_filter_without_claude(self, mock_simple_filter): """Test initializing findings filter without Claude API.""" mock_filter_instance = MagicMock() mock_simple_filter.return_value = mock_filter_instance with patch.dict(os.environ, { 'ENABLE_CLAUDE_FILTERING': 'false' }, clear=True): result = initialize_findings_filter() assert result == mock_filter_instance mock_simple_filter.assert_called_once() @patch('claudecode.github_action_audit.FindingsFilter') def test_initialize_findings_filter_with_defaults(self, mock_simple_filter): """Test initializing findings filter with defaults.""" mock_filter_instance = MagicMock() mock_simple_filter.return_value = mock_filter_instance with patch.dict(os.environ, {}, clear=True): result = initialize_findings_filter() assert result == mock_filter_instance def test_run_security_audit_success(self): """Test successful security audit execution.""" mock_runner = MagicMock() mock_runner.run_security_audit.return_value = ( True, "", {"findings": [{"id": 1}], "analysis_summary": {}} ) result = run_security_audit(mock_runner, "test prompt") assert result == {"findings": [{"id": 1}], "analysis_summary": {}} mock_runner.run_security_audit.assert_called_once() def test_run_security_audit_failure(self): """Test security audit execution failure.""" mock_runner = MagicMock() mock_runner.run_security_audit.return_value = ( False, "Audit failed: timeout", {} ) with pytest.raises(AuditError) as exc_info: run_security_audit(mock_runner, "test prompt") assert "Security audit failed: Audit failed: timeout" in str(exc_info.value) def test_apply_findings_filter_with_findings_filter(self): """Test applying FindingsFilter to findings.""" mock_filter = MagicMock(spec=FindingsFilter) mock_filter.filter_findings.return_value = ( True, { 'filtered_findings': [{"id": 1}], 'excluded_findings': [{"id": 2}], 'analysis_summary': {'total': 2, 'kept': 1} }, MagicMock() # filter_stats ) original_findings = [{"id": 1}, {"id": 2}] pr_context = {"repo_name": "test/repo"} # Create mock github client mock_github_client = MagicMock() mock_github_client._is_excluded.return_value = False kept, excluded, summary = apply_findings_filter( mock_filter, original_findings, pr_context, mock_github_client ) assert kept == [{"id": 1}] assert excluded == [{"id": 2}] assert summary == {'total': 2, 'kept': 1, 'directory_excluded_count': 0} mock_filter.filter_findings.assert_called_once_with(original_findings, pr_context) def test_apply_findings_filter_with_simple_filter(self): """Test applying FindingsFilter to findings.""" mock_filter = MagicMock(spec=FindingsFilter) mock_filter.filter_findings.return_value = ( True, { 'filtered_findings': [{"id": 1}], 'excluded_findings': [{"id": 2}], 'analysis_summary': {} }, MagicMock() # filter_stats ) original_findings = [{"id": 1}, {"id": 2}] pr_context = {"repo_name": "test/repo"} # Create mock github client mock_github_client = MagicMock() mock_github_client._is_excluded.return_value = False kept, excluded, summary = apply_findings_filter( mock_filter, original_findings, pr_context, mock_github_client ) assert kept == [{"id": 1}] assert excluded == [{"id": 2}] assert summary == {'directory_excluded_count': 0} mock_filter.filter_findings.assert_called_once_with(original_findings, pr_context) def test_apply_findings_filter_failure(self): """Test handling of filter failure.""" mock_filter = MagicMock(spec=FindingsFilter) mock_filter.filter_findings.return_value = ( False, # filter failed {}, MagicMock() ) original_findings = [{"id": 1}, {"id": 2}] pr_context = {"repo_name": "test/repo"} # Create mock github client mock_github_client = MagicMock() mock_github_client._is_excluded.return_value = False kept, excluded, summary = apply_findings_filter( mock_filter, original_findings, pr_context, mock_github_client ) # On failure, should keep all findings assert kept == original_findings assert excluded == [] assert summary == {'directory_excluded_count': 0} - claudecode/test_integration.py (4141 bytes)
#!/usr/bin/env python3 """ Pytest tests for SAST integration components. """ import pytest import json class TestClaudeCodeAudit: """Test the main audit functionality.""" @pytest.fixture def mock_env(self, monkeypatch): """Set up mock environment variables.""" monkeypatch.setenv('GITHUB_REPOSITORY', 'test/repo') monkeypatch.setenv('PR_NUMBER', '123') monkeypatch.setenv('GITHUB_TOKEN', 'mock-token') monkeypatch.setenv('ANTHROPIC_API_KEY', 'mock-api-key') def test_missing_environment_variables(self, monkeypatch, capsys): """Test behavior with missing environment variables.""" from claudecode import github_action_audit # Test missing GITHUB_REPOSITORY monkeypatch.delenv('GITHUB_REPOSITORY', raising=False) with pytest.raises(SystemExit) as exc_info: github_action_audit.main() assert exc_info.value.code == 2 # EXIT_CONFIGURATION_ERROR captured = capsys.readouterr() output = json.loads(captured.out) assert 'GITHUB_REPOSITORY' in output['error'] # Test missing PR_NUMBER monkeypatch.setenv('GITHUB_REPOSITORY', 'test/repo') monkeypatch.delenv('PR_NUMBER', raising=False) with pytest.raises(SystemExit) as exc_info: github_action_audit.main() assert exc_info.value.code == 2 # EXIT_CONFIGURATION_ERROR captured = capsys.readouterr() output = json.loads(captured.out) assert 'PR_NUMBER' in output['error'] def test_invalid_pr_number(self, monkeypatch, capsys): """Test behavior with invalid PR number.""" from claudecode import github_action_audit monkeypatch.setenv('GITHUB_REPOSITORY', 'test/repo') monkeypatch.setenv('PR_NUMBER', 'invalid') monkeypatch.setenv('GITHUB_TOKEN', 'mock-token') with pytest.raises(SystemExit) as exc_info: github_action_audit.main() assert exc_info.value.code == 2 # EXIT_CONFIGURATION_ERROR captured = capsys.readouterr() output = json.loads(captured.out) assert 'Invalid PR_NUMBER' in output['error'] class TestEnvironmentSetup: """Test environment setup and configuration.""" def test_anthropic_api_key_handling(self, monkeypatch): """Test handling of Anthropic API key.""" from claudecode.github_action_audit import SimpleClaudeRunner runner = SimpleClaudeRunner() # Test with API key set monkeypatch.setenv('ANTHROPIC_API_KEY', 'test-key') valid, error = runner.validate_claude_available() # Note: This will fail if claude CLI is not installed, which is OK if not valid and 'not installed' in error: pytest.fail("Claude CLI not installed") # Test without API key monkeypatch.delenv('ANTHROPIC_API_KEY', raising=False) valid, error = runner.validate_claude_available() if 'not installed' not in error: assert not valid assert 'ANTHROPIC_API_KEY' in error class TestFilteringIntegration: """Test the filtering system integration.""" def test_full_filter_with_llm_disabled(self): """Test FindingsFilter with LLM filtering disabled.""" from claudecode.findings_filter import FindingsFilter # Create filter with LLM disabled filter_instance = FindingsFilter( use_hard_exclusions=True, use_claude_filtering=False ) test_findings = [ {'description': 'SQL injection vulnerability', 'severity': 'HIGH'}, {'description': 'Missing rate limiting', 'severity': 'MEDIUM'}, ] success, results, stats = filter_instance.filter_findings(test_findings) assert success is True assert stats.total_findings == 2 assert stats.kept_findings == 1 # Only SQL injection assert stats.hard_excluded == 1 # Rate limiting assert stats.claude_excluded == 0 # No Claude filtering - claudecode/test_json_parser.py (8098 bytes)
"""Unit tests for the json_parser module.""" import json from typing import Any, Dict from claudecode.json_parser import parse_json_with_fallbacks, extract_json_from_text class TestJsonParser: """Test JSON parsing utilities.""" def test_parse_valid_json(self): """Test parsing valid JSON string.""" valid_json = '{"key": "value", "number": 42, "array": [1, 2, 3]}' success, result = parse_json_with_fallbacks(valid_json) assert success is True assert result == {"key": "value", "number": 42, "array": [1, 2, 3]} def test_parse_json_with_whitespace(self): """Test parsing JSON with extra whitespace.""" json_with_spaces = ' \n {"key": "value"} \n ' success, result = parse_json_with_fallbacks(json_with_spaces) assert success is True assert result == {"key": "value"} def test_parse_empty_json_object(self): """Test parsing empty JSON object.""" success, result = parse_json_with_fallbacks('{}') assert success is True assert result == {} def test_parse_empty_json_array(self): """Test parsing empty JSON array.""" success, result = parse_json_with_fallbacks('[]') assert success is True assert result == [] def test_parse_nested_json(self): """Test parsing nested JSON structures.""" nested_json = ''' { "level1": { "level2": { "level3": ["a", "b", "c"] } } } ''' success, result = parse_json_with_fallbacks(nested_json) assert success is True assert isinstance(result, dict) # Type narrowing for pyright result_dict: Dict[str, Any] = result assert result_dict["level1"]["level2"]["level3"] == ["a", "b", "c"] def test_parse_json_with_unicode(self): """Test parsing JSON with unicode characters.""" unicode_json = '{"emoji": "🔒", "text": "Hello λ world"}' success, result = parse_json_with_fallbacks(unicode_json) assert success is True assert result["emoji"] == "🔒" assert result["text"] == "Hello λ world" def test_parse_json_with_escaped_characters(self): """Test parsing JSON with escaped characters.""" escaped_json = '{"path": "C:\\\\Users\\\\test", "quote": "\\"Hello\\""}' success, result = parse_json_with_fallbacks(escaped_json) assert success is True assert result["path"] == "C:\\Users\\test" assert result["quote"] == '"Hello"' def test_extract_json_from_text_with_backticks(self): """Test extracting JSON from markdown code blocks.""" text_with_json = ''' Here is some text before the JSON: ```json {"extracted": true, "value": 123} ``` And some text after. ''' result = extract_json_from_text(text_with_json) assert result == {"extracted": True, "value": 123} def test_extract_json_from_text_without_backticks(self): """Test extracting JSON from plain text.""" text_with_json = ''' Some text before {"plain": "json", "number": 456} Some text after ''' result = extract_json_from_text(text_with_json) assert result == {"plain": "json", "number": 456} def test_extract_json_array_from_text(self): """Test extracting JSON array from text (currently not supported).""" text_with_array = ''' Results: [{"id": 1}, {"id": 2}, {"id": 3}] Done. ''' result = extract_json_from_text(text_with_array) # The function currently only extracts objects, not arrays # It should extract the first object it finds assert result == {"id": 1} def test_extract_json_with_multiple_blocks(self): """Test extracting JSON when multiple JSON blocks exist.""" text_with_multiple = ''' First block: {"first": true} Second block: {"second": true, "larger": "block"} ''' # Should extract the first valid JSON block found result = extract_json_from_text(text_with_multiple) assert result == {"first": True} or result == {"second": True, "larger": "block"} def test_parse_invalid_json_returns_error(self): """Test parsing invalid JSON returns error.""" invalid_jsons = [ '{invalid json}', '{"unclosed": "string}', '{"trailing": "comma",}', '{unquoted: key}', 'not json at all', '' ] for invalid in invalid_jsons: success, result = parse_json_with_fallbacks(invalid) assert success is False assert "error" in result def test_extract_json_from_text_no_json(self): """Test extracting JSON from text with no JSON returns None.""" texts_without_json = [ 'This is just plain text', '```python\nprint("hello")\n```', '', None ] for text in texts_without_json: result = extract_json_from_text(text) assert result is None def test_parse_json_with_comments(self): """Test parsing JSON that might have comments (should fail).""" json_with_comments = ''' { // This is a comment "key": "value" } ''' success, result = parse_json_with_fallbacks(json_with_comments) assert success is False # Standard JSON doesn't support comments assert "error" in result def test_extract_json_with_syntax_errors_in_text(self): """Test extracting JSON when there are syntax errors in surrounding text.""" text = ''' Here's some code with errors: print( But the JSON is valid: {"valid": "json", "number": 789} More broken code: }{][ ''' result = extract_json_from_text(text) assert result == {"valid": "json", "number": 789} def test_large_json_parsing(self): """Test parsing large JSON structures.""" large_json = { "findings": [ { "id": i, "title": f"Finding {i}", "description": f"Description for finding {i}", "severity": "medium", "file": f"/path/to/file{i}.py", "line": i * 10 } for i in range(100) ] } json_string = json.dumps(large_json) success, result = parse_json_with_fallbacks(json_string) assert success is True assert result == large_json assert len(result["findings"]) == 100 def test_json_with_special_characters_in_strings(self): """Test JSON with special characters in string values.""" special_json = { "newline": "line1\nline2", "tab": "before\tafter", "backslash": "path\\to\\file", "quotes": 'He said "Hello"', "unicode": "café ☕", "emoji": "🔒 Security 🛡️" } json_string = json.dumps(special_json) success, result = parse_json_with_fallbacks(json_string) assert success is True assert result == special_json def test_extract_json_from_nested_code_blocks(self): """Test extracting JSON from nested code blocks.""" text = ''' Here's a code block within text: ``` Some other code ```json {"nested": "json"} ``` ``` ''' result = extract_json_from_text(text) # Should be able to extract the JSON assert result == {"nested": "json"} - claudecode/test_main_function.py (22498 bytes)
#!/usr/bin/env python3 """ Unit tests for main function and full workflow. """ import pytest import json import os from unittest.mock import Mock, patch from pathlib import Path from claudecode.github_action_audit import main class TestMainFunction: """Test main function execution flow.""" def test_main_missing_environment_vars(self, capsys): """Test main with missing environment variables.""" with patch.dict(os.environ, {}, clear=True): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 2 # EXIT_CONFIGURATION_ERROR captured = capsys.readouterr() output = json.loads(captured.out) assert 'error' in output assert 'GITHUB_REPOSITORY' in output['error'] def test_main_missing_pr_number(self, capsys): """Test main with missing PR number.""" with patch.dict(os.environ, {'GITHUB_REPOSITORY': 'owner/repo'}, clear=True): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 2 # EXIT_CONFIGURATION_ERROR captured = capsys.readouterr() output = json.loads(captured.out) assert 'PR_NUMBER' in output['error'] def test_main_invalid_pr_number(self, capsys): """Test main with invalid PR number.""" with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': 'not-a-number' }, clear=True): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 2 # EXIT_CONFIGURATION_ERROR captured = capsys.readouterr() output = json.loads(captured.out) assert 'Invalid PR_NUMBER' in output['error'] @patch('claudecode.github_action_audit.GitHubActionClient') def test_main_github_client_init_failure(self, mock_client_class, capsys): """Test main when GitHub client initialization fails.""" mock_client_class.side_effect = Exception("Token invalid") with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'invalid' }): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 2 # EXIT_CONFIGURATION_ERROR captured = capsys.readouterr() output = json.loads(captured.out) assert 'Failed to initialize GitHub client' in output['error'] assert 'Token invalid' in output['error'] @patch('claudecode.github_action_audit.SimpleClaudeRunner') @patch('claudecode.github_action_audit.GitHubActionClient') def test_main_claude_runner_init_failure(self, mock_client_class, mock_runner_class, capsys): """Test main when Claude runner initialization fails.""" mock_client_class.return_value = Mock() mock_runner_class.side_effect = Exception("Runner error") with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'test-token' }): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 2 # EXIT_CONFIGURATION_ERROR captured = capsys.readouterr() output = json.loads(captured.out) assert 'Failed to initialize Claude runner' in output['error'] @patch('claudecode.github_action_audit.FindingsFilter') @patch('claudecode.github_action_audit.SimpleClaudeRunner') @patch('claudecode.github_action_audit.GitHubActionClient') def test_main_filter_initialization(self, mock_client_class, mock_runner_class, mock_full_filter_class): """Test filter initialization logic.""" # Setup mocks mock_client = Mock() mock_client_class.return_value = mock_client mock_runner = Mock() mock_runner.validate_claude_available.return_value = (False, "Not available") mock_runner_class.return_value = mock_runner # Test with full filtering enabled with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'test-token', 'ENABLE_CLAUDE_FILTERING': 'true', 'ANTHROPIC_API_KEY': 'test-api-key' }): with pytest.raises(SystemExit): main() # Should initialize full filter mock_full_filter_class.assert_called_once() call_kwargs = mock_full_filter_class.call_args[1] assert call_kwargs['use_hard_exclusions'] is True assert call_kwargs['use_claude_filtering'] is True assert call_kwargs['api_key'] == 'test-api-key' # Reset mocks mock_full_filter_class.reset_mock() # Test with filtering disabled with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'test-token', 'ENABLE_CLAUDE_FILTERING': 'false' }): with pytest.raises(SystemExit): main() # Should use FindingsFilter with hard exclusions only mock_full_filter_class.assert_called_once() call_kwargs = mock_full_filter_class.call_args[1] assert call_kwargs['use_hard_exclusions'] is True assert call_kwargs['use_claude_filtering'] is False @patch('claudecode.github_action_audit.FindingsFilter') @patch('claudecode.github_action_audit.SimpleClaudeRunner') @patch('claudecode.github_action_audit.GitHubActionClient') def test_main_claude_not_available(self, mock_client_class, mock_runner_class, mock_filter_class, capsys): """Test when Claude is not available.""" mock_client = Mock() mock_client_class.return_value = mock_client mock_runner = Mock() mock_runner.validate_claude_available.return_value = (False, "Claude not installed") mock_runner_class.return_value = mock_runner mock_filter_class.return_value = Mock() with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'test-token' }): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 1 # Claude not available, exit 1 captured = capsys.readouterr() output = json.loads(captured.out) assert 'Claude Code not available' in output['error'] assert 'Claude not installed' in output['error'] @patch('claudecode.github_action_audit.get_security_audit_prompt') @patch('claudecode.github_action_audit.FindingsFilter') @patch('claudecode.github_action_audit.SimpleClaudeRunner') @patch('claudecode.github_action_audit.GitHubActionClient') def test_main_pr_data_fetch_failure(self, mock_client_class, mock_runner_class, mock_filter_class, mock_prompt_func, capsys): """Test when PR data fetch fails.""" mock_client = Mock() mock_client.get_pr_data.side_effect = Exception("API error") mock_client_class.return_value = mock_client mock_runner = Mock() mock_runner.validate_claude_available.return_value = (True, "") mock_runner_class.return_value = mock_runner mock_filter_class.return_value = Mock() with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'test-token' }): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 1 # API error, exit 1 captured = capsys.readouterr() output = json.loads(captured.out) assert 'Failed to fetch PR data' in output['error'] assert 'API error' in output['error'] @patch('pathlib.Path.cwd') @patch('claudecode.github_action_audit.get_security_audit_prompt') @patch('claudecode.github_action_audit.FindingsFilter') @patch('claudecode.github_action_audit.SimpleClaudeRunner') @patch('claudecode.github_action_audit.GitHubActionClient') def test_main_successful_audit_no_findings(self, mock_client_class, mock_runner_class, mock_filter_class, mock_prompt_func, mock_cwd, capsys): """Test successful audit with no findings.""" # Setup mocks mock_client = Mock() mock_client.get_pr_data.return_value = { 'number': 123, 'title': 'Test PR', 'body': 'Description' } mock_client.get_pr_diff.return_value = "diff content" mock_client_class.return_value = mock_client mock_runner = Mock() mock_runner.validate_claude_available.return_value = (True, "") mock_runner.run_security_audit.return_value = ( True, "", { 'findings': [], 'analysis_summary': { 'files_reviewed': 5, 'high_severity': 0, 'medium_severity': 0, 'low_severity': 0 } } ) mock_runner_class.return_value = mock_runner mock_filter = Mock() mock_filter.filter_findings.return_value = ( True, { 'filtered_findings': [], 'excluded_findings': [], 'analysis_summary': {} }, Mock() # filter_stats ) mock_filter_class.return_value = mock_filter mock_prompt_func.return_value = "security prompt" mock_cwd.return_value = Path('/tmp/repo') with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'test-token' }): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 0 # No findings, exit 0 captured = capsys.readouterr() output = json.loads(captured.out) assert output['pr_number'] == 123 assert output['repo'] == 'owner/repo' assert len(output['findings']) == 0 assert output['filtering_summary']['total_original_findings'] == 0 @patch('pathlib.Path.cwd') @patch('claudecode.github_action_audit.get_security_audit_prompt') @patch('claudecode.github_action_audit.FindingsFilter') @patch('claudecode.github_action_audit.SimpleClaudeRunner') @patch('claudecode.github_action_audit.GitHubActionClient') def test_main_successful_audit_with_findings(self, mock_client_class, mock_runner_class, mock_filter_class, mock_prompt_func, mock_cwd, capsys): """Test successful audit with high severity findings.""" # Setup mocks mock_client = Mock() mock_client.get_pr_data.return_value = { 'number': 123, 'title': 'Test PR', 'body': 'Description' } mock_client.get_pr_diff.return_value = "diff content" mock_client._is_excluded.return_value = False # Don't exclude any files in tests mock_client_class.return_value = mock_client findings = [ { 'file': 'test.py', 'line': 10, 'severity': 'HIGH', 'description': 'SQL injection' }, { 'file': 'main.py', 'line': 20, 'severity': 'MEDIUM', 'description': 'Weak crypto' } ] mock_runner = Mock() mock_runner.validate_claude_available.return_value = (True, "") mock_runner.run_security_audit.return_value = ( True, "", { 'findings': findings, 'analysis_summary': { 'files_reviewed': 2, 'high_severity': 1, 'medium_severity': 1, 'low_severity': 0 } } ) mock_runner_class.return_value = mock_runner # Filter keeps only high severity mock_filter = Mock() mock_filter.filter_findings.return_value = ( True, { 'filtered_findings': [findings[0]], 'excluded_findings': [findings[1]], 'analysis_summary': { 'total_findings': 2, 'kept_findings': 1, 'excluded_findings': 1 } }, Mock() # filter_stats ) mock_filter_class.return_value = mock_filter mock_prompt_func.return_value = "security prompt" mock_cwd.return_value = Path('/tmp/repo') with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'test-token' }): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 1 # High severity finding, exit 1 captured = capsys.readouterr() output = json.loads(captured.out) assert len(output['findings']) == 1 assert output['findings'][0]['severity'] == 'HIGH' assert output['filtering_summary']['total_original_findings'] == 2 assert output['filtering_summary']['excluded_findings'] == 1 assert output['filtering_summary']['kept_findings'] == 1 @patch('pathlib.Path.cwd') @patch('claudecode.github_action_audit.get_security_audit_prompt') @patch('claudecode.github_action_audit.SimpleClaudeRunner') @patch('claudecode.github_action_audit.GitHubActionClient') def test_main_with_full_filter(self, mock_client_class, mock_runner_class, mock_prompt_func, mock_cwd, capsys): """Test main with full FindingsFilter (LLM-b - claudecode/test_prompts.py (10164 bytes)
"""Unit tests for the prompts module.""" from claudecode.prompts import get_security_audit_prompt class TestPrompts: """Test prompt generation functions.""" def test_get_security_audit_prompt_basic(self): """Test basic security audit prompt generation.""" pr_data = { "number": 123, "title": "Add new feature", "body": "This PR adds a new feature to handle user input", "user": "testuser", "changed_files": 1, "additions": 10, "deletions": 5, "head": { "repo": { "full_name": "owner/repo" } }, "files": [ { "filename": "app.py", "status": "modified", "additions": 10, "deletions": 5 } ] } pr_diff = """ diff --git a/app.py b/app.py @@ -1,5 +1,10 @@ def process_input(user_input): - return user_input + # Process the input + result = eval(user_input) # Potential security issue + return result """ prompt = get_security_audit_prompt(pr_data, pr_diff) # Check that prompt contains expected elements assert isinstance(prompt, str) assert len(prompt) > 0 assert "123" in prompt # PR number assert "Add new feature" in prompt # PR title assert "testuser" in prompt # Author assert "app.py" in prompt # File name assert "eval(user_input)" in prompt # The actual diff content def test_get_security_audit_prompt_empty_body(self): """Test prompt generation with empty PR body.""" pr_data = { "number": 456, "title": "Quick fix", "body": None, # Empty body "user": "author", "changed_files": 0, "additions": 0, "deletions": 0, "head": { "repo": { "full_name": "owner/repo" } }, "files": [] } pr_diff = "diff --git a/test.js b/test.js" prompt = get_security_audit_prompt(pr_data, pr_diff) assert isinstance(prompt, str) assert "456" in prompt assert "Quick fix" in prompt assert "author" in prompt def test_get_security_audit_prompt_multiple_files(self): """Test prompt generation with multiple files.""" pr_data = { "number": 789, "title": "Security improvements", "body": "Fixing various security issues", "user": "security-team", "changed_files": 3, "additions": 70, "deletions": 110, "head": { "repo": { "full_name": "owner/repo" } }, "files": [ { "filename": "auth.py", "status": "modified", "additions": 20, "deletions": 10 }, { "filename": "config.yaml", "status": "added", "additions": 50, "deletions": 0 }, { "filename": "old_auth.py", "status": "deleted", "additions": 0, "deletions": 100 } ] } pr_diff = """ diff --git a/auth.py b/auth.py @@ -1,10 +1,20 @@ +import secrets + diff --git a/config.yaml b/config.yaml @@ -0,0 +1,50 @@ +database: + password: "hardcoded_password" """ prompt = get_security_audit_prompt(pr_data, pr_diff) # Check all files are mentioned assert "auth.py" in prompt assert "config.yaml" in prompt assert "old_auth.py" in prompt # Check file statuses assert "modified" in prompt.lower() assert "added" in prompt.lower() assert "deleted" in prompt.lower() def test_get_security_audit_prompt_special_characters(self): """Test prompt generation with special characters.""" pr_data = { "number": 999, "title": "Fix SQL injection in user's profile", "body": "This fixes a SQL injection vulnerability in the `get_user()` function", "user": "user-with-dash", "changed_files": 1, "additions": 5, "deletions": 3, "head": { "repo": { "full_name": "owner/repo" } }, "files": [ { "filename": "src/db/queries.py", "status": "modified", "additions": 5, "deletions": 3 } ] } pr_diff = """ diff --git a/src/db/queries.py b/src/db/queries.py @@ -10,3 +10,5 @@ - query = f"SELECT * FROM users WHERE id = {user_id}" + query = "SELECT * FROM users WHERE id = ?" + cursor.execute(query, (user_id,)) """ prompt = get_security_audit_prompt(pr_data, pr_diff) # Check special characters are preserved assert "user's" in prompt assert "user-with-dash" in prompt assert "src/db/queries.py" in prompt def test_get_security_audit_prompt_no_files(self): """Test prompt generation with no files (edge case).""" pr_data = { "number": 111, "title": "Documentation update", "body": "Just updating docs", "user": "doc-author", "changed_files": 0, "additions": 0, "deletions": 0, "head": { "repo": { "full_name": "owner/repo" } }, "files": [] # No files } pr_diff = "" # Empty diff prompt = get_security_audit_prompt(pr_data, pr_diff) assert isinstance(prompt, str) assert "111" in prompt assert "Documentation update" in prompt def test_get_security_audit_prompt_structure(self): """Test that prompt has expected structure.""" pr_data = { "number": 42, "title": "Test PR", "body": "Test description", "user": "testuser", "changed_files": 1, "additions": 1, "deletions": 1, "head": { "repo": { "full_name": "owner/repo" } }, "files": [ { "filename": "test.py", "status": "modified", "additions": 1, "deletions": 1 } ] } pr_diff = "diff --git a/test.py b/test.py\n+print('test')" prompt = get_security_audit_prompt(pr_data, pr_diff) # Should contain sections for metadata and diff assert "PR #" in prompt or "Pull Request" in prompt assert "Title:" in prompt or pr_data["title"] in prompt assert "Author:" in prompt or pr_data["user"]["login"] in prompt assert "Files:" in prompt or "test.py" in prompt # Should contain the actual diff assert pr_diff in prompt or "print('test')" in prompt def test_get_security_audit_prompt_long_diff(self): """Test prompt generation with very long diff.""" pr_data = { "number": 12345, "title": "Major refactoring", "body": "Refactoring the entire codebase", "user": "refactor-bot", "changed_files": 10, "additions": 1000, "deletions": 500, "head": { "repo": { "full_name": "owner/repo" } }, "files": [ { "filename": f"file{i}.py", "status": "modified", "additions": 100, "deletions": 50 } for i in range(10) ] } # Create a large diff pr_diff = "\n".join([ f"diff --git a/file{i}.py b/file{i}.py\n" + "\n".join([f"+line {j}" for j in range(50)]) for i in range(10) ]) prompt = get_security_audit_prompt(pr_data, pr_diff) # Should handle large diffs without error assert isinstance(prompt, str) assert len(prompt) > 1000 # Should be substantial assert "12345" in prompt assert "Major refactoring" in prompt def test_get_security_audit_prompt_unicode(self): """Test prompt generation with unicode characters.""" pr_data = { "number": 666, "title": "Add emoji support 🎉", "body": "This PR adds emoji rendering 🔒 🛡️", "user": "émoji-user", "changed_files": 1, "additions": 42, "deletions": 0, "head": { "repo": { "full_name": "owner/repo" } }, "files": [ { "filename": "émojis.py", "status": "added", "additions": 42, "deletions": 0 } ] } pr_diff = """ diff --git a/émojis.py b/émojis.py +# 🔒 Security check +def check_input(text: str) -> bool: + return "🚨" not in text """ prompt = get_security_audit_prompt(pr_data, pr_diff) # Check unicode is preserved assert "🎉" in prompt # Title emoji assert "émoji-user" in prompt assert "émojis.py" in prompt assert "🚨" in prompt # From diff - claudecode/test_workflow_integration.py (18934 bytes)
#!/usr/bin/env python3 """ Integration tests for full ClaudeCode workflow. """ import pytest import json import os import tempfile from unittest.mock import Mock, patch from pathlib import Path from claudecode.github_action_audit import main class TestFullWorkflowIntegration: """Test complete workflow scenarios.""" @patch('claudecode.github_action_audit.subprocess.run') @patch('requests.get') def test_full_workflow_with_real_pr_structure(self, mock_get, mock_run): """Test complete workflow with realistic PR data.""" # Setup GitHub API responses pr_response = Mock() pr_response.json.return_value = { 'number': 456, 'title': 'Add new authentication feature', 'body': 'This PR adds OAuth2 authentication support', 'user': {'login': 'developer'}, 'created_at': '2024-01-15T10:00:00Z', 'updated_at': '2024-01-15T14:30:00Z', 'state': 'open', 'head': { 'ref': 'feature/oauth2', 'sha': 'abc123def456', 'repo': {'full_name': 'company/app'} }, 'base': { 'ref': 'main', 'sha': 'main123' }, 'additions': 250, 'deletions': 50, 'changed_files': 8 } files_response = Mock() files_response.json.return_value = [ { 'filename': 'src/auth/oauth2.py', 'status': 'added', 'additions': 150, 'deletions': 0, 'changes': 150, 'patch': '''@@ -0,0 +1,150 @@ +import requests +import jwt + +class OAuth2Handler: + def __init__(self, client_id, client_secret): + self.client_id = client_id + self.client_secret = client_secret # Stored in plain text! + + def authenticate(self, username, password): + # Direct string concatenation for SQL query + query = "SELECT * FROM users WHERE username='" + username + "'" + # ... rest of code''' }, { 'filename': 'src/auth/config.py', 'status': 'modified', 'additions': 20, 'deletions': 10, 'changes': 30, 'patch': '''@@ -10,5 +10,15 @@ -SECRET_KEY = "old-secret" +SECRET_KEY = "MySecretKey123!" # Hardcoded secret + +# OAuth2 settings +OAUTH2_PROVIDERS = { + 'google': { + 'client_id': 'hardcoded-client-id', + 'client_secret': 'hardcoded-secret' + } +}''' } ] diff_response = Mock() diff_response.text = '''diff --git a/src/auth/oauth2.py b/src/auth/oauth2.py new file mode 100644 index 0000000..1234567 --- /dev/null +++ b/src/auth/oauth2.py @@ -0,0 +1,150 @@ +import requests +import jwt + +class OAuth2Handler: + def __init__(self, client_id, client_secret): + self.client_id = client_id + self.client_secret = client_secret # Stored in plain text! + + def authenticate(self, username, password): + # Direct string concatenation for SQL query + query = "SELECT * FROM users WHERE username='" + username + "'" + # ... rest of code diff --git a/src/auth/config.py b/src/auth/config.py index 8901234..5678901 100644 --- a/src/auth/config.py +++ b/src/auth/config.py @@ -10,5 +10,15 @@ import os -SECRET_KEY = "old-secret" +SECRET_KEY = "MySecretKey123!" # Hardcoded secret + +# OAuth2 settings +OAUTH2_PROVIDERS = { + 'google': { + 'client_id': 'hardcoded-client-id', + 'client_secret': 'hardcoded-secret' + } +}''' mock_get.side_effect = [pr_response, files_response, diff_response] # Setup Claude response claude_response = { "findings": [ { "file": "src/auth/oauth2.py", "line": 11, "severity": "HIGH", "category": "sql_injection", "description": "SQL injection vulnerability due to direct string concatenation in query construction", "exploit_scenario": "An attacker could inject SQL commands through the username parameter", "recommendation": "Use parameterized queries or an ORM to prevent SQL injection", "confidence": 0.95 }, { "file": "src/auth/config.py", "line": 12, "severity": "HIGH", "category": "hardcoded_secrets", "description": "Hardcoded secret key in configuration file", "exploit_scenario": "Anyone with access to the code can see the secret key", "recommendation": "Use environment variables or a secure key management system", "confidence": 0.99 }, { "file": "src/auth/oauth2.py", "line": 7, "severity": "MEDIUM", "category": "insecure_storage", "description": "Client secret stored in plain text in memory", "exploit_scenario": "Memory dumps could expose the client secret", "recommendation": "Consider using secure storage mechanisms for sensitive data", "confidence": 0.8 } ], "analysis_summary": { "files_reviewed": 2, "high_severity": 2, "medium_severity": 1, "low_severity": 0, "review_completed": True } } # Mock Claude CLI version_result = Mock() version_result.returncode = 0 version_result.stdout = 'claude version 1.0.0' version_result.stderr = '' audit_result = Mock() audit_result.returncode = 0 # Claude wraps the result in a specific format claude_wrapped_response = { 'result': json.dumps(claude_response) } audit_result.stdout = json.dumps(claude_wrapped_response) audit_result.stderr = '' # The audit might be retried, so provide the same result twice mock_run.side_effect = [version_result, audit_result, audit_result] # Run the workflow with tempfile.TemporaryDirectory() as tmpdir: os.chdir(tmpdir) with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'company/app', 'PR_NUMBER': '456', 'GITHUB_TOKEN': 'test-token', 'ANTHROPIC_API_KEY': 'test-api-key', 'ENABLE_CLAUDE_FILTERING': 'false' # Use simple filter }): with pytest.raises(SystemExit) as exc_info: main() # Should exit with 1 due to HIGH severity findings assert exc_info.value.code == 1 # Verify API calls assert mock_get.call_count == 3 assert mock_run.call_count == 2 # Verify the audit was run with proper prompt audit_call = mock_run.call_args_list[1] prompt = audit_call[1]['input'] assert 'Add new authentication feature' in prompt # Title assert 'src/auth/oauth2.py' in prompt # File name assert 'string concatenation for SQL query' in prompt # From diff @patch('subprocess.run') @patch('requests.get') def test_workflow_with_llm_filtering(self, mock_get, mock_run): """Test workflow with LLM-based false positive filtering.""" # Setup minimal API responses pr_response = Mock() pr_response.json.return_value = { 'number': 789, 'title': 'Update dependencies', 'body': 'Routine dependency updates', 'user': {'login': 'bot'}, 'created_at': '2024-01-20T09:00:00Z', 'updated_at': '2024-01-20T09:15:00Z', 'state': 'open', 'head': {'ref': 'deps/update', 'sha': 'dep123', 'repo': {'full_name': 'company/app'}}, 'base': {'ref': 'main', 'sha': 'main456'}, 'additions': 100, 'deletions': 80, 'changed_files': 5 } files_response = Mock() files_response.json.return_value = [] diff_response = Mock() diff_response.text = 'diff --git a/package.json b/package.json\n...' mock_get.side_effect = [pr_response, files_response, diff_response] # Claude finds some issues claude_findings = [ { "file": "package.json", "line": 25, "severity": "MEDIUM", "description": "Outdated dependency with known vulnerabilities", "confidence": 0.7 }, { "file": "src/test.py", "line": 10, "severity": "LOW", "description": "Potential timing attack in test code", "confidence": 0.5 } ] mock_run.side_effect = [ Mock(returncode=0, stdout='claude version 1.0.0', stderr=''), Mock(returncode=0, stdout=json.dumps({"findings": claude_findings}), stderr='') ] with tempfile.TemporaryDirectory() as tmpdir: os.chdir(tmpdir) with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'company/app', 'PR_NUMBER': '789', 'GITHUB_TOKEN': 'test-token', 'ANTHROPIC_API_KEY': 'test-api-key', 'ENABLE_CLAUDE_FILTERING': 'false' # Use simple filter to avoid isinstance issues }): with pytest.raises(SystemExit) as exc_info: main() # Should exit 0 - no HIGH severity findings assert exc_info.value.code == 0 def test_workflow_error_recovery(self): """Test workflow recovery from various errors.""" with patch('requests.get') as mock_get: # Simulate network error mock_get.side_effect = Exception("Network error") with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'owner/repo', 'PR_NUMBER': '123', 'GITHUB_TOKEN': 'token' }): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == 1 @patch('subprocess.run') @patch('requests.get') def test_workflow_with_no_security_issues(self, mock_get, mock_run): """Test workflow when no security issues are found.""" # Setup clean PR pr_response = Mock() pr_response.json.return_value = { 'number': 999, 'title': 'Add documentation', 'body': 'Updates to README', 'user': {'login': 'docs-team'}, 'created_at': '2024-01-25T11:00:00Z', 'updated_at': '2024-01-25T11:05:00Z', 'state': 'open', 'head': {'ref': 'docs/update', 'sha': 'doc123', 'repo': {'full_name': 'company/app'}}, 'base': {'ref': 'main', 'sha': 'main789'}, 'additions': 50, 'deletions': 10, 'changed_files': 2 } files_response = Mock() files_response.json.return_value = [ { 'filename': 'README.md', 'status': 'modified', 'additions': 40, 'deletions': 10, 'changes': 50, 'patch': '@@ -1,5 +1,35 @@\n # Project Name\n+\n+## Installation\n+...' } ] diff_response = Mock() diff_response.text = 'diff --git a/README.md b/README.md\n+## Installation\n+npm install\n' mock_get.side_effect = [pr_response, files_response, diff_response] # Claude finds no issues mock_run.side_effect = [ Mock(returncode=0, stdout='claude version 1.0.0', stderr=''), Mock(returncode=0, stdout='{"findings": [], "analysis_summary": {"review_completed": true}}', stderr='') ] with tempfile.TemporaryDirectory() as tmpdir: os.chdir(tmpdir) output_file = Path(tmpdir) / 'output.json' with patch.dict(os.environ, { 'GITHUB_REPOSITORY': 'company/app', 'PR_NUMBER': '999', 'GITHUB_TOKEN': 'test-token', 'ANTHROPIC_API_KEY': 'test-api-key' }): with patch('sys.stdout', open(output_file, 'w')): with pytest.raises(SystemExit) as exc_info: main() # Should exit 0 - no findings assert exc_info.value.code == 0 # Verify output with open(output_file) as f: output = json.load(f) assert output['pr_number'] == 999 assert output['repo'] == 'company/app' assert len(output['findings']) == 0 assert output['filtering_summary']['total_original_findings'] == 0 class TestWorkflowEdgeCases: """Test edge cases in the workflow.""" @patch('subprocess.run') @patch('requests.get') def test_workflow_with_massive_pr(self, mock_get, mock_run): """Test workflow with very large PR.""" # Create a massive file list large_files = [ { 'filename': f'src/file{i}.py', 'status': 'added', 'additions': 100, 'deletions': 0, 'changes': 100, 'patch': f'@@ -0,0 +1,100 @@\n+# File {i}\n' + '+\n' * 99 } for i in range(500) # 500 files ] pr_response = Mock() pr_response.json.return_value = { 'number': 1000, 'title': 'Massive refactoring', 'body': 'Complete codebase restructure', 'user': {'login': 'architect'}, 'created_at': '2024-02-01T08:00:00Z', 'updated_at': '2024-02-01T18:00:00Z', 'state': 'open', 'head': {'ref': 'refactor/all', 'sha': 'ref123', 'repo': {'full_name': 'company/app'}}, 'base': {'ref': 'main', 'sha': 'main000'}, 'additions': 50000, 'deletions': 30000, 'changed_files': 500 } files_response = Mock() files_response.json.return_value = large_files # Create massive diff diff_parts = [] for i in range(500): - docs/custom-filtering-instructions.md (2469 bytes)
# Custom False Positive Filtering Instructions The Claude Code Security Reviewer Action supports custom false positive filtering instructions, allowing you to tailor the security analysis to your specific environment and requirements. ## Overview By default, the SAST action includes a comprehensive set of exclusions and criteria for filtering out false positives. However, every organization has unique security requirements, technology stacks, and threat models. The `false-positive-filtering-instructions` input allows you to provide your own custom criteria. ## Usage 1. Create a text file containing your custom filtering instructions (e.g., `.github/false-positive-filtering.txt`) 2. Reference it in your workflow: ```yaml - uses: anthropics/claude-code-security-review@main with: false-positive-filtering-instructions: .github/false-positive-filtering.txt ``` ## File Format The file should contain plain text with three main sections: ### 1. HARD EXCLUSIONS List patterns that should be automatically excluded from findings. ### 2. SIGNAL QUALITY CRITERIA Questions to assess whether a finding represents a real vulnerability. ### 3. PRECEDENTS Specific guidance for common security patterns in your environment. ## Example See [examples/custom-false-positive-filtering.txt](../examples/custom-false-positive-filtering.txt) for a complete example tailored to a modern cloud-native application. ## Default Instructions If no custom file is provided, the action uses default instructions tuned to work well for most applications. ## Best Practices 1. **Start with defaults**: Begin with the default instructions and modify based on false positives you encounter 2. **Be specific**: Include details about your security architecture (e.g., "We use AWS Cognito for all authentication") 3. **Document assumptions**: Explain why certain patterns are excluded (e.g., "k8s resource limits prevent DOS") 4. **Version control**: Track changes to your filtering instructions alongside your code 5. **Team review**: Have your security team review and approve the filtering instructions ## Common Customizations - **Technology-specific exclusions**: Exclude findings that don't apply to your tech stack - **Infrastructure assumptions**: Document security controls at the infrastructure level - **Compliance requirements**: Adjust criteria based on your compliance needs - **Development practices**: Reflect your team's security practices and tooling - docs/custom-security-scan-instructions.md (3758 bytes)
# Custom Security Scan Instructions The Claude Code Security Reviewer Action supports custom security scan instructions, allowing you to add organization-specific vulnerability categories to the security audit. ## Overview The default security scan covers common vulnerability categories like SQL injection, XSS, authentication issues, etc. However, organizations often have specific security concerns based on their: - Technology stack (GraphQL, gRPC, specific cloud providers) - Compliance requirements (GDPR, HIPAA, PCI DSS) - Industry-specific vulnerabilities (financial services, healthcare) - Custom frameworks and libraries The `custom-security-scan-instructions` input allows you to extend the security categories that Claude checks for. ## Usage 1. Create a text file containing your custom security categories (e.g., `.github/custom-security-categories.txt`) 2. Reference it in your workflow: ```yaml - uses: anthropics/claude-code-security-review@main with: custom-security-scan-instructions: .github/custom-security-categories.txt ``` ## File Format The file should contain additional security categories in the same format as the default categories. Each category should: - Start with a descriptive header in bold (using `**Category Name:**`) - List specific vulnerabilities or patterns to check for - Use clear, actionable descriptions ### Example Structure: ``` **Category Name:** - Specific vulnerability or pattern to check - Another specific issue to look for - Detailed description of what constitutes this vulnerability **Another Category:** - More specific checks - Additional patterns to identify ``` ## Examples ### Industry-Specific Example See [examples/organization-specific-scan-instructions.txt](../examples/custom-security-scan-instructions.txt) for an example set of instructions that customize Claude Code to look for industry-specific security weaknesses including: - Compliance checks (GDPR, HIPAA, PCI DSS) - Financial services security - E-commerce specific issues ## How It Works Your custom instructions are appended to the security audit prompt after the default "Data Exposure" category. This means: 1. All default categories are still checked 2. Your custom categories extend (not replace) the default scan 3. The same HIGH/MEDIUM/LOW severity guidelines apply ## Best Practices 1. **Be Specific**: Provide clear descriptions of what constitutes each vulnerability 2. **Include Context**: Explain why something is a vulnerability in your environment 3. **Provide Examples**: Where possible, describe specific attack scenarios 4. **Avoid Duplicates**: Check the default categories to avoid redundancy 5. **Keep It Focused**: Only add categories relevant to your codebase ## Default Categories Reference The default scan already includes: - Input Validation (SQL injection, command injection, XXE, etc.) - Authentication & Authorization - Crypto & Secrets Management - Injection & Code Execution - Data Exposure Your custom categories should complement these, not duplicate them. ## Tips for Writing Effective Categories 1. **Technology-Specific**: Add checks for your specific tech stack ``` **GraphQL Security:** - Query depth attacks allowing unbounded recursion - Field-level authorization bypass - Introspection data leakage in production ``` 2. **Compliance-Focused**: Add regulatory requirements ``` **GDPR Compliance:** - Personal data processing without consent mechanisms - Missing data retention limits - Lack of data portability APIs ``` 3. **Business Logic**: Add domain-specific vulnerabilities ``` **Payment Processing:** - Transaction replay vulnerabilities - Currency conversion manipulation - Refund process bypass ``` - examples/custom-false-positive-filtering.txt (1971 bytes)
HARD EXCLUSIONS - Automatically exclude findings matching these patterns: 1. All DOS/resource exhaustion - we have k8s resource limits and autoscaling 2. Missing rate limiting - handled by our API gateway 3. Tabnabbing vulnerabilities - acceptable risk per our threat model 4. Test files (ending in _test.go, _test.js, or in __tests__ directories) 5. Documentation files (*.md, *.rst) 6. Configuration files that are not exposed to users (internal configs) 7. Memory safety in Rust, Go, or managed languages 8. GraphQL introspection queries - we intentionally expose schema in dev 9. Missing CSRF protection - we use stateless JWT auth exclusively 10. Timing attacks on non-cryptographic operations 11. Regex DoS in input validation (we have request timeouts) 12. Missing security headers in internal services (only public-facing services need them) SIGNAL QUALITY CRITERIA - For remaining findings, assess: 1. Can an unauthenticated external attacker exploit this? 2. Is there actual data exfiltration or system compromise potential? 3. Is this exploitable in our production Kubernetes environment? 4. Does this bypass our API gateway security controls? PRECEDENTS - 1. We use AWS Cognito for all authentication - auth bypass must defeat Cognito 2. All APIs require valid JWT tokens validated at the gateway level 3. SQL injection is only valid if using raw queries (we use Prisma ORM everywhere) 4. All internal services communicate over mTLS within the k8s cluster 5. Secrets are in AWS Secrets Manager or k8s secrets, never in code 6. We allow verbose error messages in dev/staging (not production) 7. File uploads go directly to S3 with presigned URLs (no local file handling) 8. All user input is considered untrusted and validated on the backend 9. Frontend validation is only for UX, not security 10. We use CSP headers and strict Content-Type validation 11. CORS is configured per-service based on actual needs 12. All webhooks use HMAC signature verification
- examples/custom-security-scan-instructions.txt (798 bytes)
**Compliance-Specific Checks:** - GDPR Article 17 "Right to Erasure" implementation gaps - HIPAA PHI encryption at rest violations - PCI DSS credit card data retention beyond allowed periods - SOC2 audit trail tampering or deletion capabilities - CCPA data portability API vulnerabilities **Financial Services Security:** - Transaction replay attacks in payment processing - Double-spending vulnerabilities in ledger systems - Interest calculation manipulation through timing attacks - Regulatory reporting data tampering - Know Your Customer (KYC) bypass mechanisms **E-commerce Specific:** - Shopping cart manipulation for price changes - Inventory race conditions allowing overselling - Coupon/discount stacking exploits - Affiliate tracking manipulation - Review system authentication bypass
- INSTRUCTIONS.md (548 bytes)
You are the autonomous runtime coordinator for claude-code-security-review. System Archetype: tool. Description: An AI-powered security review GitHub Action using Claude to analyze code changes for security vulnerabilities. Core Directives: 1. Execute operations deterministically following verified repository manifests. 2. Strictly enforce safe input validation and local-first execution boundaries. 3. Provide structured outputs, formatted logs, and actionable resolution steps. 4. Preserve data integrity and respect local workspace isolation.
- LICENSE (1065 bytes)
MIT License Copyright (c) 2025 Anthropic Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- README.md (8600 bytes)
# Claude Code Security Reviewer An AI-powered security review GitHub Action using Claude to analyze code changes for security vulnerabilities. This action provides intelligent, context-aware security analysis for pull requests using Anthropic's Claude Code tool for deep semantic security analysis. See our blog post [here](https://www.anthropic.com/news/automate-security-reviews-with-claude-code) for more details. ## Features - **AI-Powered Analysis**: Uses Claude's advanced reasoning to detect security vulnerabilities with deep semantic understanding - **Diff-Aware Scanning**: For PRs, only analyzes changed files - **PR Comments**: Automatically comments on PRs with security findings - **Contextual Understanding**: Goes beyond pattern matching to understand code semantics - **Language Agnostic**: Works with any programming language - **False Positive Filtering**: Advanced filtering to reduce noise and focus on real vulnerabilities ## Quick Start Add this to your repository's `.github/workflows/security.yml`: ```yaml name: Security Review permissions: pull-requests: write # Needed for leaving PR comments contents: read on: pull_request: jobs: security: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: ref: ${{ github.event.pull_request.head.sha || github.sha }} fetch-depth: 2 - uses: anthropics/claude-code-security-review@main with: comment-pr: true claude-api-key: ${{ secrets.CLAUDE_API_KEY }} ``` ## Security Considerations This action is not hardened against prompt injection attacks and should only be used to review trusted PRs. We recommend [configuring your repository](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#controlling-changes-from-forks-to-workflows-in-public-repositories) to use the "Require approval for all external contributors" option to ensure workflows only run after a maintainer has reviewed the PR. ## Configuration Options ### Action Inputs | Input | Description | Default | Required | |-------|-------------|---------|----------| | `claude-api-key` | Anthropic Claude API key for security analysis. <br>*Note*: This API key needs to be enabled for both the Claude API and Claude Code usage. | None | Yes | | `comment-pr` | Whether to comment on PRs with findings | `true` | No | | `upload-results` | Whether to upload results as artifacts | `true` | No | | `exclude-directories` | Comma-separated list of directories to exclude from scanning | None | No | | `claude-model` | Claude [model name](https://docs.anthropic.com/en/docs/about-claude/models/overview#model-names) to use. Defaults to Opus 4.1. | `claude-opus-4-1-20250805` | No | | `claudecode-timeout` | Timeout for ClaudeCode analysis in minutes | `20` | No | | `run-every-commit` | Run ClaudeCode on every commit (skips cache check). Warning: May increase false positives on PRs with many commits. | `false` | No | | `false-positive-filtering-instructions` | Path to custom false positive filtering instructions text file | None | No | | `custom-security-scan-instructions` | Path to custom security scan instructions text file to append to audit prompt | None | No | ### Action Outputs | Output | Description | |--------|-------------| | `findings-count` | Total number of security findings | | `results-file` | Path to the results JSON file | ## How It Works ### Architecture ``` claudecode/ ├── github_action_audit.py # Main audit script for GitHub Actions ├── prompts.py # Security audit prompt templates ├── findings_filter.py # False positive filtering logic ├── claude_api_client.py # Claude API client for false positive filtering ├── json_parser.py # Robust JSON parsing utilities ├── requirements.txt # Python dependencies ├── test_*.py # Test suites └── evals/ # Eval tooling to test CC on arbitrary PRs ``` ### Workflow 1. **PR Analysis**: When a pull request is opened, Claude analyzes the diff to understand what changed 2. **Contextual Review**: Claude examines the code changes in context, understanding the purpose and potential security implications 3. **Finding Generation**: Security issues are identified with detailed explanations, severity ratings, and remediation guidance 4. **False Positive Filtering**: Advanced filtering removes low-impact or false positive prone findings to reduce noise 5. **PR Comments**: Findings are posted as review comments on the specific lines of code ## Security Analysis Capabilities ### Types of Vulnerabilities Detected - **Injection Attacks**: SQL injection, command injection, LDAP injection, XPath injection, NoSQL injection, XXE - **Authentication & Authorization**: Broken authentication, privilege escalation, insecure direct object references, bypass logic, session flaws - **Data Exposure**: Hardcoded secrets, sensitive data logging, information disclosure, PII handling violations - **Cryptographic Issues**: Weak algorithms, improper key management, insecure random number generation - **Input Validation**: Missing validation, improper sanitization, buffer overflows - **Business Logic Flaws**: Race conditions, time-of-check-time-of-use (TOCTOU) issues - **Configuration Security**: Insecure defaults, missing security headers, permissive CORS - **Supply Chain**: Vulnerable dependencies, typosquatting risks - **Code Execution**: RCE via deserialization, pickle injection, eval injection - **Cross-Site Scripting (XSS)**: Reflected, stored, and DOM-based XSS ### False Positive Filtering The tool automatically excludes a variety of low-impact and false positive prone findings to focus on high-impact vulnerabilities: - Denial of Service vulnerabilities - Rate limiting concerns - Memory/CPU exhaustion issues - Generic input validation without proven impact - Open redirect vulnerabilities The false positive filtering can also be tuned as needed for a given project's security goals. ### Benefits Over Traditional SAST - **Contextual Understanding**: Understands code semantics and intent, not just patterns - **Lower False Positives**: AI-powered analysis reduces noise by understanding when code is actually vulnerable - **Detailed Explanations**: Provides clear explanations of why something is a vulnerability and how to fix it - **Adaptive Learning**: Can be customized with organization-specific security requirements ## Installation & Setup ### GitHub Actions Follow the Quick Start guide above. The action handles all dependencies automatically. ### Local Development To run the security scanner locally against a specific PR, see the [evaluation framework documentation](claudecode/evals/README.md). <a id="security-review-slash-command"></a> ## Claude Code Integration: /security-review Command By default, Claude Code ships a `/security-review` [slash command](https://docs.anthropic.com/en/docs/claude-code/slash-commands) that provides the same security analysis capabilities as the GitHub Action workflow, but integrated directly into your Claude Code development environment. To use this, simply run `/security-review` to perform a comprehensive security review of all pending changes. ### Customizing the Command The default `/security-review` command is designed to work well in most cases, but it can also be customized based on your specific security needs. To do so: 1. Copy the [`security-review.md`](https://github.com/anthropics/claude-code-security-review/blob/main/.claude/commands/security-review.md?plain=1) file from this repository to your project's `.claude/commands/` folder. 2. Edit `security-review.md` to customize the security analysis. For example, you could add additional organization-specific directions to the false positive filtering instructions. ## Custom Scanning Configuration It is also possible to configure custom scanning and false positive filtering instructions, see the [`docs/`](docs/) folder for more details. ## Testing Run the test suite to validate functionality: ```bash cd claude-code-security-review # Run all tests pytest claudecode -v ``` ## Support For issues or questions: - Open an issue in this repository - Check the [GitHub Actions logs](https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/viewing-workflow-run-history) for debugging information ## License MIT License - see [LICENSE](LICENSE) file for details. - scripts/comment-pr-findings.bun.test.js (20428 bytes)
#!/usr/bin/env bun /** * Unit tests for comment-pr-findings.js using Bun test runner */ import { test, expect, describe, beforeEach, afterEach, mock, spyOn } from "bun:test"; describe('comment-pr-findings.js', () => { let originalEnv; let consoleLogSpy; let consoleErrorSpy; let processExitSpy; let readFileSyncSpy; let spawnSyncSpy; // Helper function to create mock responses for spawnSync function mockSpawnSyncResponse(endpoint, method = 'GET', captureData = null) { return (cmd, args, options) => { if (cmd === 'gh' && args.includes('api')) { const apiEndpoint = args[1]; const apiMethod = args[args.indexOf('--method') + 1] || 'GET'; if (apiEndpoint.includes(endpoint) && apiMethod === method) { if (captureData && options && options.input) { captureData.data = JSON.parse(options.input); } return { status: 0, stdout: captureData?.response || '{}', stderr: '' }; } } return null; }; } beforeEach(() => { // Save original environment originalEnv = { ...process.env }; // Clear process.env completely first for (const key in process.env) { delete process.env[key]; } setupTestEnvironment(); // Set up spies consoleLogSpy = spyOn(console, 'log').mockImplementation(() => {}); consoleErrorSpy = spyOn(console, 'error').mockImplementation(() => {}); processExitSpy = spyOn(process, 'exit').mockImplementation(() => {}); // Mock fs and child_process readFileSyncSpy = spyOn(require('fs'), 'readFileSync'); spawnSyncSpy = spyOn(require('child_process'), 'spawnSync'); }); afterEach(() => { // Restore environment process.env = originalEnv; // Clear module cache to allow re-running the script delete require.cache[require.resolve('./comment-pr-findings.js')]; }); // Set up common test environment function setupTestEnvironment() { process.env.GITHUB_REPOSITORY = 'owner/repo'; process.env.GITHUB_EVENT_PATH = 'github-event.json'; } describe('Environment Setup', () => { test('should parse GitHub context correctly', async () => { const mockEventData = { pull_request: { number: 123, head: { sha: 'abc123' } } }; readFileSyncSpy.mockImplementation((path) => { if (path.includes('github-event.json')) { return JSON.stringify(mockEventData); } if (path === 'findings.json') { return '[]'; // Empty findings to exit early } }); await import('./comment-pr-findings.js'); expect(readFileSyncSpy).toHaveBeenCalledWith(expect.stringContaining('github-event.json'), 'utf8'); }); }); describe('Finding Processing', () => { test('should exit early when no findings file exists', async () => { readFileSyncSpy.mockImplementation((path) => { if (path.includes('github-event.json')) { return JSON.stringify({ pull_request: { number: 123, head: { sha: 'abc123' } } }); } if (path === 'findings.json') { throw new Error('File not found'); } // For any other file (like the script itself), throw to prevent loading throw new Error('Unexpected file read: ' + path); }); await import('./comment-pr-findings.js'); expect(consoleLogSpy).toHaveBeenCalledWith('Could not read findings file'); expect(spawnSyncSpy).not.toHaveBeenCalled(); }); test('should exit early when findings array is empty', async () => { readFileSyncSpy.mockReturnValue('[]'); await import('./comment-pr-findings.js'); expect(spawnSyncSpy).not.toHaveBeenCalled(); }); test('should process findings correctly', async () => { const mockFindings = [{ path: 'test.py', start: { line: 10 }, check_id: 'rules.insecure-pickle-loads-autofix', extra: { message: 'Detected use of pickle deserialization', fix: 'json.loads($DATA) # Use json.loads() instead of pickle for security' } }]; const mockPrFiles = [{ filename: 'test.py', patch: '@@ -10,1 +10,1 @@' }]; const mockFileContent = { content: Buffer.from(' data = pickle.loads(user_input)').toString('base64') }; readFileSyncSpy.mockImplementation((path) => { if (path.includes('github-event.json')) { return JSON.stringify({ pull_request: { number: 123, head: { sha: 'abc123' } } }); } if (path === 'findings.json') { return JSON.stringify(mockFindings); } }); let reviewDataCaptured = null; spawnSyncSpy.mockImplementation((cmd, args, options) => { if (cmd === 'gh' && args.includes('api')) { const endpoint = args[1]; if (endpoint.includes('/pulls/123/files')) { return { status: 0, stdout: JSON.stringify(mockPrFiles), stderr: '' }; } if (endpoint.includes('/contents/test.py')) { return { status: 0, stdout: JSON.stringify(mockFileContent), stderr: '' }; } if (endpoint.includes('/pulls/123/comments') && args.includes('GET')) { return { status: 0, stdout: '[]', stderr: '' }; // No existing comments } if (endpoint.includes('/pulls/123/reviews') && args.includes('POST')) { // Capture the review data if passed if (options && options.input) { reviewDataCaptured = JSON.parse(options.input); } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: '{}', stderr: '' }; }); await import('./comment-pr-findings.js'); // Verify API calls were made expect(spawnSyncSpy).toHaveBeenCalledWith('gh', expect.arrayContaining(['api', expect.stringContaining('/pulls/123/files')]), expect.any(Object)); expect(spawnSyncSpy).toHaveBeenCalledWith('gh', expect.arrayContaining(['api', expect.stringContaining('/pulls/123/reviews')]), expect.any(Object)); expect(consoleLogSpy).toHaveBeenCalledWith('Created review with 1 inline comments'); // Verify review data was captured expect(reviewDataCaptured).toBeTruthy(); expect(reviewDataCaptured.comments).toHaveLength(1); }); }); describe('Autofix Suggestions', () => { test('should generate correct pickle.loads autofix', async () => { const mockFindings = [{ path: 'test.py', start: { line: 1 }, check_id: 'rules.insecure-pickle-loads-autofix', extra: { message: 'Insecure pickle loads', fix: 'json.loads($DATA) # Use json.loads() instead of pickle for security' } }]; readFileSyncSpy.mockImplementation((path) => { if (path.includes('github-event.json')) { return JSON.stringify({ pull_request: { number: 123, head: { sha: 'abc123' } } }); } if (path === 'findings.json') { return JSON.stringify(mockFindings); } }); let capturedReviewData; spawnSyncSpy.mockImplementation((cmd, args, options) => { if (cmd === 'gh' && args.includes('api')) { const endpoint = args[1]; const method = args[args.indexOf('--method') + 1] || 'GET'; if (endpoint.includes('/pulls/123/files')) { return { status: 0, stdout: JSON.stringify([{ filename: 'test.py' }]), stderr: '' }; } if (endpoint.includes('/contents/test.py')) { return { status: 0, stdout: JSON.stringify({ content: Buffer.from('result = pickle.loads(data)').toString('base64') }), stderr: '' }; } if (endpoint.includes('/pulls/123/comments') && method === 'GET') { return { status: 0, stdout: '[]', stderr: '' }; } if (endpoint.includes('/pulls/123/reviews') && method === 'POST') { if (options && options.input) { capturedReviewData = JSON.parse(options.input); } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: '{}', stderr: '' }; }); await import('./comment-pr-findings.js') expect(capturedReviewData).toBeDefined(); expect(capturedReviewData.comments[0].body).toContain('🤖 **Security Issue: Insecure pickle loads**'); }); test('should generate correct yaml.load autofix', async () => { const mockFindings = [{ path: 'config.py', start: { line: 1 }, check_id: 'rules.insecure-yaml-loads-no-loader', extra: { message: 'Unsafe YAML deserialization', fix: 'yaml.safe_load($DATA)' } }]; readFileSyncSpy.mockImplementation((path) => { if (path.includes('github-event.json')) { return JSON.stringify({ pull_request: { number: 123, head: { sha: 'abc123' } } }); } if (path === 'findings.json') { return JSON.stringify(mockFindings); } }); let capturedReviewData; spawnSyncSpy.mockImplementation((cmd, args, options) => { if (cmd === 'gh' && args.includes('api')) { const endpoint = args[1]; const method = args[args.indexOf('--method') + 1] || 'GET'; if (endpoint.includes('/pulls/123/files')) { return { status: 0, stdout: JSON.stringify([{ filename: 'config.py' }]), stderr: '' }; } if (endpoint.includes('/contents/config.py')) { return { status: 0, stdout: JSON.stringify({ content: Buffer.from('config = yaml.load(config_file)').toString('base64') }), stderr: '' }; } if (endpoint.includes('/pulls/123/comments') && method === 'GET') { return { status: 0, stdout: '[]', stderr: '' }; } if (endpoint.includes('/pulls/123/reviews') && method === 'POST') { if (options && options.input) { capturedReviewData = JSON.parse(options.input); } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: '{}', stderr: '' }; }); await import('./comment-pr-findings.js') expect(capturedReviewData).toBeDefined(); expect(capturedReviewData.comments[0].body).toContain('🤖 **Security Issue: Unsafe YAML deserialization**'); }); test('should preserve indentation in autofix', async () => { const mockFindings = [{ path: 'test.py', start: { line: 1 }, check_id: 'rules.insecure-pickle-loads-autofix', extra: { message: 'Insecure pickle loads', fix: 'json.loads($DATA) # Use json.loads() instead of pickle for security' } }]; readFileSyncSpy.mockImplementation((path) => { if (path.includes('github-event.json')) { return JSON.stringify({ pull_request: { number: 123, head: { sha: 'abc123' } } }); } if (path === 'findings.json') { return JSON.stringify(mockFindings); } }); let capturedReviewData; spawnSyncSpy.mockImplementation((cmd, args, options) => { if (cmd === 'gh' && args.includes('api')) { const endpoint = args[1]; const method = args[args.indexOf('--method') + 1] || 'GET'; if (endpoint.includes('/pulls/123/files')) { return { status: 0, stdout: JSON.stringify([{ filename: 'test.py' }]), stderr: '' }; } if (endpoint.includes('/contents/test.py')) { return { status: 0, stdout: JSON.stringify({ content: Buffer.from(' data = pickle.loads(user_input)').toString('base64') }), stderr: '' }; } if (endpoint.includes('/pulls/123/comments') && method === 'GET') { return { status: 0, stdout: '[]', stderr: '' }; } if (endpoint.includes('/pulls/123/reviews') && method === 'POST') { if (options && options.input) { capturedReviewData = JSON.parse(options.input); } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: '{}', stderr: '' }; }); await import('./comment-pr-findings.js') expect(capturedReviewData).toBeDefined(); expect(capturedReviewData.comments[0].body).toContain('🤖 **Security Issue: Insecure pickle loads**'); }); }); describe('Finding Limits', () => { test('should process all ClaudeCode findings without limit', async () => { // Create 8 ClaudeCode findings const mockFindings = []; for (let i = 1; i <= 8; i++) { mockFindings.push({ file: `test${i}.py`, line: 10, message: `Finding ${i}`, severity: 'HIGH', metadata: { vulnerability_type: 'security_issue', tool: 'ClaudeCode AI Security Analysis', check_id: `check-${i}` } }); } const mockPrFiles = mockFindings.map(f => ({ filename: f.file, patch: '@@ -10,1 +10,1 @@' })); readFileSyncSpy.mockImplementation((path) => { if (path.includes('github-event.json')) { return JSON.stringify({ pull_request: { number: 123, head: { sha: 'abc123' } } }); } if (path === 'findings.json') { return JSON.stringify(mockFindings); } }); let capturedReviewData; spawnSyncSpy.mockImplementation((cmd, args, options) => { if (cmd === 'gh' && args.includes('api')) { const endpoint = args[1]; const method = args[args.indexOf('--method') + 1] || 'GET'; if (endpoint.includes('/pulls/123/files')) { return { status: 0, stdout: JSON.stringify(mockPrFiles), stderr: '' }; } if (endpoint.includes('/pulls/123/comments') && method === 'GET') { return { status: 0, stdout: '[]', stderr: '' }; } if (endpoint.includes('/pulls/123/reviews') && method === 'POST') { if (options && options.input) { capturedReviewData = JSON.parse(options.input); } return { status: 0, stdout: '{}', stderr: '' }; } return { status: 0, stdout: ' - scripts/comment-pr-findings.js (8968 bytes)
#!/usr/bin/env node /** * Script to comment on PRs with security findings from ClaudeCode */ const fs = require('fs'); const { spawnSync } = require('child_process'); // Parse GitHub context from environment const context = { repo: { owner: process.env.GITHUB_REPOSITORY?.split('/')[0] || '', repo: process.env.GITHUB_REPOSITORY?.split('/')[1] || '' }, issue: { number: parseInt(process.env.GITHUB_EVENT_PATH ? JSON.parse(fs.readFileSync(process.env.GITHUB_EVENT_PATH, 'utf8')).pull_request?.number : '') || 0 }, payload: { pull_request: process.env.GITHUB_EVENT_PATH ? JSON.parse(fs.readFileSync(process.env.GITHUB_EVENT_PATH, 'utf8')).pull_request : {} } }; // GitHub API helper using gh CLI function ghApi(endpoint, method = 'GET', data = null) { // Build arguments array safely to prevent command injection const args = ['api', endpoint, '--method', method]; if (data) { args.push('--input', '-'); } try { const result = spawnSync('gh', args, { encoding: 'utf8', input: data ? JSON.stringify(data) : undefined, stdio: ['pipe', 'pipe', 'pipe'] }); if (result.error) { throw new Error(`Failed to spawn gh process: ${result.error.message}`); } if (result.status !== 0) { console.error(`Error calling GitHub API: ${result.stderr}`); throw new Error(`gh process exited with code ${result.status}: ${result.stderr}`); } return JSON.parse(result.stdout); } catch (error) { console.error(`Error calling GitHub API: ${error.message}`); throw error; } } // Helper function to add reactions to a comment function addReactionsToComment(commentId, isReviewComment = true) { const reactions = ['+1', '-1']; // thumbs up and thumbs down const endpoint = isReviewComment ? `/repos/${context.repo.owner}/${context.repo.repo}/pulls/comments/${commentId}/reactions` : `/repos/${context.repo.owner}/${context.repo.repo}/issues/comments/${commentId}/reactions`; for (const reaction of reactions) { try { ghApi(endpoint, 'POST', { content: reaction }); console.log(`Added ${reaction} reaction to comment ${commentId}`); } catch (error) { console.error(`Failed to add ${reaction} reaction to comment ${commentId}:`, error.message); } } } // Helper function to add reactions to all comments in a review function addReactionsToReview(reviewId) { try { // Get all comments from the review const reviewComments = ghApi(`/repos/${context.repo.owner}/${context.repo.repo}/pulls/${context.issue.number}/reviews/${reviewId}/comments`); if (reviewComments && Array.isArray(reviewComments)) { for (const comment of reviewComments) { if (comment.id) { addReactionsToComment(comment.id, true); } } } } catch (error) { console.error(`Failed to get review comments for review ${reviewId}:`, error.message); } } async function run() { try { // Read the findings let newFindings = []; try { const findingsData = fs.readFileSync('findings.json', 'utf8'); newFindings = JSON.parse(findingsData); } catch (e) { console.log('Could not read findings file'); return; } if (newFindings.length === 0) { return; } // Get the PR diff to map file lines to diff positions const prFiles = ghApi(`/repos/${context.repo.owner}/${context.repo.repo}/pulls/${context.issue.number}/files?per_page=100`); // Create a map of file paths to their diff information const fileMap = {}; prFiles.forEach(file => { fileMap[file.filename] = file; }); // Prepare review comments const reviewComments = []; // Check if ClaudeCode comments should be silenced const silenceClaudeCodeComments = process.env.SILENCE_CLAUDECODE_COMMENTS === 'true'; if (silenceClaudeCodeComments) { console.log(`ClaudeCode comments silenced - excluding ${newFindings.length} findings from comments`); return; } // Process findings synchronously (gh cli doesn't support async well) for (const finding of newFindings) { const file = finding.file || finding.path; const line = finding.line || (finding.start && finding.start.line) || 1; const message = finding.description || (finding.extra && finding.extra.message) || 'Security vulnerability detected'; const severity = finding.severity || 'HIGH'; const category = finding.category || 'security_issue'; // Check if this file is part of the PR diff if (!fileMap[file]) { console.log(`File ${file} not in PR diff, skipping`); continue; } // Build the comment body let commentBody = `🤖 **Security Issue: ${message}**\n\n`; commentBody += `**Severity:** ${severity}\n`; commentBody += `**Category:** ${category}\n`; commentBody += `**Tool:** ClaudeCode AI Security Analysis\n`; // Add exploit scenario if available if (finding.exploit_scenario || (finding.extra && finding.extra.metadata && finding.extra.metadata.exploit_scenario)) { const exploitScenario = finding.exploit_scenario || finding.extra.metadata.exploit_scenario; commentBody += `\n**Exploit Scenario:** ${exploitScenario}\n`; } // Add recommendation if available if (finding.recommendation || (finding.extra && finding.extra.metadata && finding.extra.metadata.recommendation)) { const recommendation = finding.recommendation || finding.extra.metadata.recommendation; commentBody += `\n**Recommendation:** ${recommendation}\n`; } // Prepare the review comment const reviewComment = { path: file, line: line, side: 'RIGHT', body: commentBody }; reviewComments.push(reviewComment); } if (reviewComments.length === 0) { console.log('No findings to comment on PR diff'); return; } // Check for existing review comments to avoid duplicates const comments = ghApi(`/repos/${context.repo.owner}/${context.repo.repo}/pulls/${context.issue.number}/comments`); // Check if we've already commented on these findings const existingSecurityComments = comments.filter(comment => comment.user.type === 'Bot' && comment.body && comment.body.includes('🤖 **Security Issue:') ); if (existingSecurityComments.length > 0) { console.log(`Found ${existingSecurityComments.length} existing security comments, skipping to avoid duplicates`); return; } try { // Create a review with all the comments const reviewData = { commit_id: context.payload.pull_request.head.sha, event: 'COMMENT', comments: reviewComments }; const reviewResponse = ghApi(`/repos/${context.repo.owner}/${context.repo.repo}/pulls/${context.issue.number}/reviews`, 'POST', reviewData); console.log(`Created review with ${reviewComments.length} inline comments`); // Add reactions to the comments if (reviewResponse && reviewResponse.id) { addReactionsToReview(reviewResponse.id); } } catch (error) { console.error('Error creating review:', error); // Fallback: try to create individual comments if review fails // This might happen if line numbers are outside the diff context console.log('Attempting fallback with adjusted line numbers...'); for (const comment of reviewComments) { try { // Try to create comment with the original line const commentData = { path: comment.path, line: comment.line, side: comment.side, body: comment.body, commit_id: context.payload.pull_request.head.sha }; const commentResponse = ghApi(`/repos/${context.repo.owner}/${context.repo.repo}/pulls/${context.issue.number}/comments`, 'POST', commentData); // Add reactions to the individual comment if (commentResponse && commentResponse.id) { addReactionsToComment(commentResponse.id, true); } } catch (lineError) { console.log(`Could not comment on ${comment.path}:${comment.line} - line might not be in diff context`); // If the specific line fails, try to get the file's patch and find a suitable line const fileInfo = fileMap[comment.path]; if (fileInfo && fileInfo.patch) { // This is a simplified approach - in production you'd want more sophisticated line mapping console.log(`File ${comment.path} has additions but line ${comment.line} is not in the diff`); } } } } } catch (error) { console.error('Failed to comment on PR:', error); process.exit(1); } } run(); - scripts/package.json (200 bytes)
{ "name": "@anthropic-ai/sast-scripts", "version": "1.0.0", "description": "Scripts for Anthropic SAST Action", "scripts": { "test": "bun test", "test:watch": "bun test --watch" } }