Phase 3: Reviewing Everything for Flaws (Quality & Security Assurance)

Automated verification is the cornerstone of enterprise safety. No agent-generated output should reach production without passing multi-layered verification. This principle applies doubly to AI-generated code, which introduces a category of failure modes that do not exist in purely human-written codebases.

Agent-Generated Code Risks #

AI-generated code fails differently than human-written code. Understanding these failure modes is essential for designing effective review pipelines:

• Plausible but incorrect logic: Agents produce code that compiles, passes linting, and looks reasonable but contains subtle logical errors—off-by-one errors in boundary conditions, incorrect operator precedence, or inverted boolean logic. These bugs pass casual review because the code "looks right."
• Hallucinated APIs and libraries: Agents occasionally call functions or reference libraries that do not exist, especially when working with less common frameworks or recent API versions not well-represented in training data.
• Security pattern violations: Agents may reintroduce security anti-patterns that were previously fixed if the reasoning for the original fix is not documented in an ADR or code comment.
• Style drift: Without strict rules files, agents gradually drift from project conventions—introducing new naming patterns, different error handling styles, or inconsistent import ordering that fragments the codebase.
• Over-engineering: Agents have a tendency toward verbose, abstraction-heavy implementations. A simple three-line utility function may become a class hierarchy with interfaces, factories, and configuration objects. Skills like ponytail exist specifically to counteract this tendency.

Automated Edge-Case Generation #

Beyond standard unit tests, QA agents must actively generate adversarial test vectors to uncover latent bugs before deployment. This includes:

• Boundary & Payload Stressing: Injecting empty strings, massive payloads (10MB+ request bodies), overflow numbers (Number.MAX_SAFE_INTEGER + 1), and unexpected character encodings (UTF-8 multibyte, null-byte injection, RTL override characters).

• Concurrency & State Collisions: Stressing database transactions with concurrent requests to flag potential race conditions, deadlocks, or lock contention. This is particularly important for agent-generated code, which often assumes sequential execution.

• Failure Mode Simulation: Mocking downstream API outages, timeout latency (30s+ response times), partial response truncation, and network dropouts to verify application fault tolerance and graceful degradation.

Two-Tiered Review Architecture #

To balance speed and thoroughness, code review must occur across two distinct tiers:

• Tier 1: Automated Mechanical Review (Agent-Driven): Specialized review agents automatically inspect every diff for mechanical defects: off-by-one errors, unhandled null pointers, missing try/catch blocks, unclosed resource handles, type mismatches, and syntax inconsistencies. Tools in this tier include:

  • BugBot — Automated PR review agent that catches low-level bugs and security regressions in diffs.
  • CodeQL (GitHub) — Semantic code analysis that queries code as data to find vulnerability patterns.
  • Semgrep — Lightweight static analysis with custom rule authoring for project-specific anti-patterns.
  • ESLint / TypeScript strict mode — Catches type errors, unused variables, and style violations at the syntax level.

• Tier 2: Architectural & Business Review (Human-Driven): Human engineers review PRs focusing exclusively on strategic questions: Does this solve the core business problem? Is the architectural direction sound? Will this be maintainable long-term? Does the agent's implementation match the intent of the original ticket? This tier cannot be automated—it requires understanding organizational priorities, user empathy, and long-term maintainability judgment that models do not possess.

Security & Vulnerability Auditing #

Security checks are non-negotiable and integrated directly into the automated pipeline:

• Static Application Security Testing (SAST): Tools like Semgrep and CodeQL scan every commit for common security antipatterns (SQL injection, XSS, insecure deserialization, path traversal).

• Dependency & Secret Scanning: Tools like Snyk, Trivy, and Dependabot scan third-party dependencies for known CVEs and block any commit containing accidental secrets, private keys, or API tokens. Gitleaks provides fast, pre-commit secret detection.

• Least-Privilege Verification: Audit tools verify that agents operating within CI/CD loops hold zero read/write access to production environments or secrets. Agents should have the minimum permissions required to complete their assigned task and nothing more.

Continuous Workflow Critiques #

Engineering teams must establish a monthly workflow retrospective specifically focused on agent performance. If agents consistently ship repetitive bugs, the response is not to fix bugs piecemeal—it is to identify the systemic cause and update the system:

• Repetitive type errors? Strengthen the rules file with explicit type conventions and enable stricter linting.
• Agents ignoring architectural boundaries? Add explicit module dependency rules and forbidden import patterns.
• Security regressions? Document the reasoning in ADRs and add Semgrep rules to catch the specific pattern.
• Over-engineered implementations? Add ponytail-style simplicity constraints to the rules file.

The goal is a continuously improving feedback loop where every agent failure makes the system permanently better, not a whack-a-mole game where the same issues recur sprint after sprint.