How to review AI-generated pull requests

Review an agent pull request in a fixed order. Let the machines check the code first, then compare the pull request with the other open ones, then read the tests, the security checks and the intent. The second step is the one that no single diff can show you.

Niklas Femerstrand, founder · 21 Oct 2026 · Updated 22 Oct 2026

The order that works

1. Machine checks. CI, a code scanner and a dependency check.
2. The other open pull requests. Which of them touch the same code, and does the pair still pass your tests?
3. Changes to tests and CI. Deleted tests, skipped tests, fewer assertions.
4. Security controls. Login, permission and input checks around the change.
5. Intent. Does the diff do what the task asked, and only that?
6. Instruction files. Changes to AGENTS.md, CLAUDE.md and similar.
7. Maintainability. Names, structure and duplicated code.

Steps one to four are cheap to check and expensive to miss. Steps five to seven need a person who knows the code. The rest of this guide goes through each step with commands you can run.

Agent pull requests come in groups

A person usually has one or two pull requests open. Agents often open several at once. Xu, Subramanian and Karthik looked at 33,596 agent pull requests in 2,807 GitHub repositories. Of those, 79.4% were open at the same time as at least one other agent pull request in the same repository (Xu et al., arXiv 2607.04697).

We replayed 733 of those overlapping pairs ourselves, one pair per repository, and merged each pair with Git. 179 pairs (24.4%) had a textual conflict. When two different agents had made the pull requests, the rate was 43.3%. So when you review one agent pull request, there is a good chance another open one changes the same code.

Your CI does not show this. It tests each pull request alone against the base branch. A green check on both pull requests tells you nothing about the two together.

Step one, the machine checks

GitHub's own guide to reviewing AI code starts here, with tests and static analysis before a person reads anything (GitHub Docs, Review AI-generated code). Run CI, a code scanner such as CodeQL and a dependency check on every agent pull request, and wait for them before you read.

gh pr checks 73          # every check and its state
gh pr diff 73 --name-only   # the files it touches

Run the tests in a clean environment when you can. In our replay, 218 of the clean pairs had a pull request that already failed its own tests in a clean container. Most of those suites needed a database, a browser or network access that the container didn't have. Even so, find out before you read whether a pull request passes only on the agent's machine.

Step two, the other open pull requests

Here, a single diff cannot show you the problem. The clearest case in our replay is in firecrawl-mcp-server. Devin opened #72 and #73 with the same title, "Implement maxAge fast scraping parameter". Each one adds a maxAge key to the input schema of the scrape tool in src/index.ts. #72 puts it after timeout, #73 after location, about a hundred lines further down.

Read alone, each diff is correct. Each one passes npm ci && npm test. Git merges them without a conflict. The merged object then has two maxAge keys, and the TypeScript build stops.

src/index.ts(198,7): error TS1117: An object literal cannot have
multiple properties with the same name.

Two questions catch this kind of problem. First, does another open pull request do the same task? In firecrawl, one agent opened two pull requests for one feature. Second, does another open pull request change code that this one uses? The GitHub CLI answers both quickly.

# open pull requests and the files each one changes
gh pr list --state open --json number,title,files \
  --jq '.[] | "\(.number) \(.title)\n  \([.files[].path] | join(" "))"'

# do two of them still merge? exit status 0 clean, 1 conflict
git fetch origin pull/72/head:pr72 pull/73/head:pr73
git merge-tree --write-tree --name-only pr72 pr73

A clean merge-tree result doesn't mean the pair works. In firecrawl it was clean. To know, you must build and test the merge. The gitta check does this for every open pair, on every push.

The gitta check on pull requests

The check posts one comment on each pull request and keeps it up to date. It reports three things.

For GitHub, add one workflow file. The push trigger checks every open pull request again when main moves, so a pull request that fit yesterday is checked against what merged today.

# .github/workflows/gitta.yml
name: Gitta
on:
  pull_request: { types: [opened, synchronize, reopened] }
  push: { branches: [main] }
permissions: { contents: read, pull-requests: write }
concurrency: { group: gitta-pr-check, cancel-in-progress: false }
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: gitta-io/gitta/gitta-action@v1
        with:
          gate: "npm ci && npm test"
          mode: ${{ github.event_name == 'push' && 'all' || 'current' }}

When two open pull requests break together, only the newer one fails the check. The older one passes and its comment names the clash. After the older one merges, the newer one is checked again against the new base. Two incompatible pull requests never block each other. GitLab and Azure Repos are coming next. See Pull requests and CI.

On firecrawl, the check without a gate warned both edited: src/index.ts assign SCRAPE_TOOL (#72 + #73). With the gate it reported the break and showed the TS1117 error.

How much to trust a warning

Treat the three results differently. "Breaks together" means that your own tests failed on the merged code, so you must act. "Potentially affected" and "both edited" point at code where two changes meet. Read that part first, and expect it to be correct most of the time.

Our replay gives numbers for this. We could test 44 clean pairs end to end, and one of them broke. In the first run, the check without a gate also warned on 13 of the 43 pairs that worked. In most of those, both pull requests edited a changelog, a CI workflow or a test function. These overlaps now show as info, which never fails the check. With the gate on, the check failed on no working pair in one run. In the other run it failed on one flaky test.

Step three, changes to tests and CI

An agent that can't make a test pass sometimes changes the test. Look for deleted test files, new skip markers, removed assertions and relaxed CI settings. Git can show these in seconds.

# test files the pull request deleted or changed
git diff --stat origin/main...pr73 -- '*test*' '*spec*'
# lines with assertions that it removed
git diff origin/main...pr73 -- '*test*' | grep -E '^-.*(assert|expect)'
# skip markers that it added
git diff origin/main...pr73 | grep -E '^\+.*(skip|xfail|only\()'

gitta warns the agent about weakened checks while it works. When the work is merged, gitta reports the weakened checks in the merge output and the CI log and records them in the audit trail. In CI the check stops such a merge by default.

Step four, security controls

Look at the login, permission and input checks around the change. Read security sensitive code more closely. The OpenHands review checklist says to look for a check that is missing or only assumed (OpenHands, code review checklist). Also look at what other open pull requests do to those checks. A control removed in one pull request can leave code from another pull request unprotected.

The gitta comment has a Security line for this case. In this run on microblog, one pull request makes profile pages public and removes the login check from the user view. Another adds a short profile link whose view calls user.

- **Security** with public-profiles: public-profiles removed the login check
  (`@login_required`) from `app.main.routes.user`, which short-links' new
  code in app/main/routes.py calls. That code may now run without a logged-in user.

gitta also checks the lines that an agent adds for well-known risks. These include SQL or shell commands built from text and variables, certificate checks that are turned off, and secrets in the code. The agent hears about each one at its next step, while the code is still in front of it. To mark your own security checks, list them in a committed .gitta-security file, and the CI check reads it too.

Step five, the intent

Now read the task, then the diff. Agents sometimes solve a nearby problem, or do more than the task asked. Be careful when a small task has a large diff. Ask the agent to split work that goes beyond the task, and close pull requests that duplicate another one. In firecrawl, one of the two maxAge pull requests was redundant from the start.

Step six, instruction files

A change to AGENTS.md, CLAUDE.md or a similar file changes how every agent in the repository works from then on. An agent should change them only when a person asked for it. gitta flags these changes in the agent's warnings and in gitta status. Review them like a change to your CI setup.

Step seven, maintainability

Names, structure, duplicated helpers and dead code stay with the people who will own the code. Before you approve a new helper, search the code base for one that already does the same job.

Where each check shows up

A person still reads the code and decides. The gitta comment shows that person all the open agent work together, which no single diff can do. Set it up in Pull requests and CI.

The same 733 pull request pairs show why a clean merge can still break, in why git worktrees don't prevent agent conflicts.