Building End-to-End Automation Testing with Playwright, AI, Video Recording, and Screenshot Comparison

In the fast-evolving world of software development, ensuring the quality and reliability of web applications is paramount. Manual testing can be time-consuming and error-prone, which is where automation testing shines. In this blog, we’ll explore how to create a robust end-to-end (E2E) testing framework using Playwright, enhance it with an AI tool for smarter test assertions, record videos of test runs, and implement screenshot comparison to validate application pages visually. Let’s dive in!

Why Playwright?

Playwright, developed by Microsoft, is a powerful open-source automation library that supports testing across Chromium, Firefox, and WebKit browsers with a single API. It’s fast, reliable, and designed for modern web applications with features like auto-waiting, network interception, and built-in tools for visual testing. Here’s why it’s a great choice:

• Cross-browser support: Test on Chrome, Firefox, and Safari effortlessly.
• Headless and headed modes: Debug visually or run silently in CI/CD pipelines.
• Rich ecosystem: Built-in support for screenshots, video recording, and more.

What We’ll Build

Our goal is to create an E2E testing pipeline that:

1. Automates navigation and interactions on a web app using Playwright.
2. Leverages an AI tool to enhance test logic (e.g., dynamic assertions).
3. Captures videos of test execution for debugging.
4. Takes screenshots of key pages and compares them to baseline images for visual regression testing.

Prerequisites

Before we start, ensure you have:

• Node.js (v16 or higher) installed.
• A sample web application to test (e.g., a local app or a public site like https://example.com).
• Basic knowledge of JavaScript/TypeScript and testing concepts.

Install Playwright by running:

npm init playwright@latest

This sets up Playwright with a default configuration file (playwright.config.js).

Step 1: Setting Up Playwright with Video Recording

Let’s configure Playwright to record videos of test runs. Open playwright.config.js and update it as follows:

module.exports = {
  use: {
    browserName: 'chromium', // or 'firefox', 'webkit'
    headless: false, // Set to true in CI/CD
    screenshot: 'only-on-failure', // Capture screenshots on failure
    video: 'retain-on-failure', // Record video, keep if test fails
  },
  retries: 1, // Retry failed tests once
  outputDir: 'test-results/', // Store videos and screenshots here
};

Now, write a simple test to verify video recording works:

const { test, expect } = require('@playwright/test');
 
test('Basic navigation with video', async ({ page }) => {
  await page.goto('https://example.com');
  await expect(page).toHaveTitle(/Example Domain/);
  await page.click('a'); // Click a link (if applicable)
});

Run the test:

npx playwright test

After execution, check the test-results/ folder. If the test fails, you’ll find a video file (e.g., .webm) showing the browser actions—perfect for debugging!

Step 2: Adding Screenshot Capture

Playwright makes it easy to take screenshots. Let’s modify our test to capture a screenshot of the page:

test('Capture screenshot of page', async ({ page }) => {
  await page.goto('https://example.com');
  await page.screenshot({ path: 'screenshots/example-page.png', fullPage: true });
  await expect(page).toHaveTitle(/Example Domain/);
});

Create a screenshots/ folder in your project root to store these images. Run the test again, and you’ll see example-page.png in the folder.

Step 3: Implementing Screenshot Comparison

For visual regression testing, we’ll compare screenshots against baseline images. Playwright doesn’t natively handle pixel-by-pixel comparison beyond its toMatchSnapshot method, so let’s use the pixelmatch library for precision.

Install pixelmatch and pngjs:

npm install pixelmatch pngjs

Here’s a test that compares a new screenshot to a baseline:

const { test, expect } = require('@playwright/test');
const { PNG } = require('pngjs');
const pixelmatch = require('pixelmatch');
const fs = require('fs');
 
test('Compare screenshots for visual regression', async ({ page }) => {
  await page.goto('https://example.com');
  const newScreenshotPath = 'screenshots/new-example.png';
  await page.screenshot({ path: newScreenshotPath, fullPage: true });
  const baseline = PNG.sync.read(fs.readFileSync('screenshots/baseline-example.png'));
  const newImage = PNG.sync.read(fs.readFileSync(newScreenshotPath));
  const { width, height } = baseline;
  const diff = new PNG({ width, height });
  const mismatchedPixels = pixelmatch(
    baseline.data,
    newImage.data,
    diff.data,
    width,
    height,
    { threshold: 0.1 }
  );
  fs.writeFileSync('screenshots/diff-example.png', PNG.sync.write(diff));
  expect(mismatchedPixels).toBeLessThan(100);
});

Run the test:

npx playwright test

Step 4: Running in CI/CD

To integrate into a CI/CD pipeline (e.g., GitHub Actions):

name: E2E Tests
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with: { node-version: '16' }
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
      - uses: actions/upload-artifact@v3
        if: failure()
        with:
          name: test-results
          path: test-results/

Conclusion

By combining Playwright for automation, video recording for debugging, screenshot comparison for visual validation, and an AI tool for intelligent assertions, you’ve built a powerful E2E testing framework. This setup catches functional bugs, visual regressions, and even subtle content issues—all while providing rich artifacts for analysis.

Happy testing! 🎉