Playwright Parser - Flakiness Detective

Overview

The Playwright Parser extracts structured information from Playwright test failures, including error details, test metadata, and execution results. It implements a hybrid approach: works perfectly without any configuration, but provides enhanced features when you add optional annotations.

🎯 Key Features

✅ Zero Config - Works immediately without any setup
🎯 Rich Error Parsing - Extracts matcher, locator, expected/actual values
🔄 Hybrid Approach - Auto-generates test IDs when annotations are missing
📊 Optional Metadata - Add annotations for enhanced tracking
🏭 Production Tested - Battle-tested with 79 passing tests

Installation

# npm
npm install @flakiness-detective/playwright-parser

# pnpm
pnpm add @flakiness-detective/playwright-parser

# yarn
yarn add @flakiness-detective/playwright-parser

Quick Start

Use the parser in your custom Playwright reporter or analysis tool. It works immediately without any configuration:

import { parsePlaywrightTest } from '@flakiness-detective/playwright-parser';
import type { TestCase, TestResult } from '@playwright/test/reporter';

// In your custom reporter
const parsed = parsePlaywrightTest(test, result);

console.log(parsed.error.matcher);    // 'toBeVisible'
console.log(parsed.error.locator);    // 'button[name="Submit"]'
console.log(parsed.error.expected);   // 'visible'
console.log(parsed.error.actual);     // 'hidden'
console.log(parsed.metadata.testCaseId); // 'auto-a3f7b9c4' (auto-generated!)

Level Up with Annotations

For better tracking across test renames and historical analysis, add annotations:

test('user can login', async ({ page }) => {
  // Add testCaseId for stable tracking
  test.info().annotations.push({
    type: 'testCaseId',
    description: 'TC-AUTH-001'
  });
  
  // Optional: group by user journey
  test.info().annotations.push({
    type: 'journeyId',
    description: 'authentication'
  });
  
  // Optional: organize into test suites
  test.info().annotations.push({
    type: 'testSuiteName',
    description: 'Login Tests'
  });
  
  // Your test code...
});

Why Annotations?

Without annotations (still works!)

✅ Pattern detection across similar errors
✅ Clustering by locator, matcher, etc.
✅ Basic flakiness detection
⚠️ Test renames break history

With annotations (10x better!)

✅ All the above PLUS:
✅ Track test history across renames
✅ Temporal pattern detection
✅ Journey-level insights
✅ Long-term stability trends

Complete Example

Here's a full example of using the parser in a custom reporter:

import { parsePlaywrightTest } from '@flakiness-detective/playwright-parser';
import type { TestCase, TestResult, Reporter } from '@playwright/test/reporter';

class FlakinessReporter implements Reporter {
  onTestEnd(test: TestCase, result: TestResult) {
    if (result.status !== 'failed' && result.status !== 'timedOut') {
      return; // Only process failures
    }

    const parsed = parsePlaywrightTest(test, result, {
      // Optional: enable strict mode (require annotations)
      strict: false,
      
      // Optional: customize auto-ID strategy
      autoIdStrategy: 'hash', // 'hash' | 'uuid' | 'none'
      
      // Optional: warn users about missing annotations
      warnOnMissingAnnotations: true,
      
      // Optional: extract custom annotations
      customAnnotationTypes: ['priority', 'owner', 'tags']
    });

    // Send to your flakiness detection system
    this.sendToAnalytics({
      testId: parsed.metadata.testCaseId,
      journey: parsed.metadata.journeyId,
      error: parsed.error,
      result: parsed.result,
      timestamp: new Date()
    });
  }
}

export default FlakinessReporter;

API Reference

parsePlaywrightTest(test, result, options?)

Main parsing function that extracts all information from a failed test.

Parameters:

test: TestCase - The Playwright test case
result: TestResult - The test result (must be 'failed' or 'timedOut')
options?: ParserOptions - Optional configuration

Returns:

ParsedTestFailure - Complete parsed test failure object

ParserOptions

interface ParserOptions {
  // Strict mode: throw error if required annotations missing
  strict?: boolean; // Default: false
  
  // Warn when annotations are missing (educates users)
  warnOnMissingAnnotations?: boolean; // Default: true
  
  // Strategy for auto-generating testCaseId
  autoIdStrategy?: 'hash' | 'uuid' | 'none'; // Default: 'hash'
  
  // Custom annotation types to extract
  customAnnotationTypes?: string[];
}

ParsedTestFailure

interface ParsedTestFailure {
  // Structured error details
  error: PlaywrightErrorDetails;
  
  // Test metadata (annotations + auto-generated)
  metadata: TestMetadata;
  
  // Test information
  test: TestInfo;
  
  // Execution result
  result: TestResultInfo;
}

Use Cases

1. Flakiness Detection System

Collect structured failure data for AI-powered analysis:

const parsed = parsePlaywrightTest(test, result);

await sendToFlakinessDetector({
  testId: parsed.metadata.testCaseId,
  journey: parsed.metadata.journeyId,
  errorSignature: {
    matcher: parsed.error.matcher,
    locator: parsed.error.locator,
    message: parsed.error.message
  },
  timestamp: new Date()
});

2. Database Storage

Store structured failure data for historical analysis:

await db.collection('test_failures').add({
  testCaseId: parsed.metadata.testCaseId,
  testTitle: parsed.test.title,
  errorMessage: parsed.error.message,
  locator: parsed.error.locator,
  matcher: parsed.error.matcher,
  expected: parsed.error.expected,
  actual: parsed.error.actual,
  timestamp: new Date()
});

3. CI/CD Integration

Generate detailed failure reports in your CI pipeline:

class CIReporter implements Reporter {
  failures = [];
  
  onTestEnd(test: TestCase, result: TestResult) {
    if (result.status === 'failed') {
      const parsed = parsePlaywrightTest(test, result);
      this.failures.push(parsed);
    }
  }
  
  onEnd() {
    console.log(`📊 Failure Report (${this.failures.length} failures)`);
    
    for (const failure of this.failures) {
      console.log(`❌ ${failure.test.title}`);
      console.log(`   ID: ${failure.metadata.testCaseId}`);
      console.log(`   Matcher: ${failure.error.matcher}`);
      console.log(`   Locator: ${failure.error.locator}`);
    }
  }
}

Advanced Configuration

Strict Mode

Enforce annotations for production systems:

const parsed = parsePlaywrightTest(test, result, {
  strict: true,           // Throws error if testCaseId missing
  autoIdStrategy: 'none'  // Disable auto-generation
});

Custom ID Strategy

// Deterministic hash (recommended)
parsePlaywrightTest(test, result, { autoIdStrategy: 'hash' });

// Random UUID (for one-off analysis)
parsePlaywrightTest(test, result, { autoIdStrategy: 'uuid' });

// No auto-generation (annotations required)
parsePlaywrightTest(test, result, { autoIdStrategy: 'none' });

Integration with Flakiness Detective

This parser works seamlessly with the @flakiness-detective/core package for AI-powered flakiness detection:

import { FlakinessDetective } from '@flakiness-detective/core';
import { createDataAdapter, createEmbeddingProvider } from '@flakiness-detective/adapters';
import { parsePlaywrightTest } from '@flakiness-detective/playwright-parser';

// Parse and store failures in your reporter
class MyReporter implements Reporter {
  async onTestEnd(test: TestCase, result: TestResult) {
    if (result.status === 'failed') {
      const parsed = parsePlaywrightTest(test, result);
      await adapter.saveFailure(parsed);
    }
  }
}

// Later: Run flakiness detection
const detective = new FlakinessDetective(adapter, embeddingProvider);
const clusters = await detective.detect();

💡 Pro Tip: Start Simple

The parser works great without annotations. Start using it immediately, see the value, then add annotations incrementally to unlock advanced features. This progressive enhancement approach means you get value in 30 seconds, not 30 hours.

@flakiness-detective/playwright-parser