Part 2 – Comparing run time images

In the previous article on Playwright Test image comparisons and validation we focused on the .toHaveScreenshot() method. This method allows us to visually compare how elements of our application look now against previously captured baseline “golden” reference images.

I finished that post with the question: “What if we want to capture a screenshot at run time, do something, then capture a new screenshot and compare the two?”.

It’s now time to answer that. So let’s continue our look at Playwright Test image comparison.

The answer isn’t quite as straight forward as you might hope, but at the same time it’s not impossible to work through. .toHaveScreenShot() expects to find the golden reference image on disc, ready to compare against a freshly captured run time image. Playwright can automatically derive the reference image file name, or you can specify it. In either case that file must ultimately exist on disc when .toHaveScreenShot()executes – or Playwright will fail the test and create it.

In memory capture and comparison

Capturing screenshots in memory as buffer objects is straightforward:

const originalImage = await page.locator('#textInput').screenshot(); 

Playwright Test does not have a supported way of comparing two image buffers – although it does look like one is planned. So in the meantime, and as a first step let’s just try capturing and comparing buffers using .toEqual().

test('Comparing images at runtime', async({page})=>{
  await page.goto('https://www.edgewordstraining.co.uk/webdriver2/docs/forms.html');
  await page.locator('#textInput').fill('Hello World'); //Set initial state
  
  //Capture screenshot of initial state in to memory buffer
  const originalImage = await page.locator('#textInput').screenshot(); 
  
  await page.locator('#textInput').fill('Hello World.'); //Change state slightly
  //Capture new screenshot with updated state
  const newImage = await page.locator('#textInput').screenshot(); 
  
  expect(originalImage).toEqual(newImage); //Compare buffers
});

The test passes – as hoped – if the state is kept constant i.e. the captured images should and do match.

When the images do not match (a full stop has been added after “Hello World”) the test fails:

The report however has some significant problems. Instead of expected and actual images it contains just a diff of the bytes that differed between the buffers. For larger images (or whole page screenshots) this will be very long.

Also, there’s no expected and actual images shown, and no diff image to highlight the variation between the images. Nor is there a way of controlling how similar the images have to be to be considered a match (as is possible with .toHaveScreenshot() ). Every pixel must be a perfect match or the comparison will fail.

Writing to the report

To mitigate the missing report image issues we could choose to manually attach the expected and actual images to the report using testInfo:

.toEqual() can only verify the images match exactly. There is no ability to perform fuzzy matching, we cannot set allowable pixel ratio differences or allow for colour variations. Nor can we generate a diff image.  These things are achievable with third party libraries, but Playwright already does them when using .toHaveScreenshot(). I’d prefer to avoid unnecessary extra dependencies if possible.

Digging in to Playwright Tets’s internals

User d2vid on StackOverflow suggests an approach (which I’ve adapted and expanded upon) leveraging some internals of Playwright Test:

import {test, expect} from '@playwright/test';
// using internal Playwright Test methods to perform comparison
import { getComparator } from './../node_modules/playwright-core/lib/utils/comparators';

test('using comparator', async({page}, testInfo)=>{
  await page.goto("https://www.edgewordstraining.co.uk/webdriver2/docs/forms.html");

  await page.locator('#textInput').fill("Hello World"); //Set initial state
  const beforeImage =  await page.locator('#textInput').screenshot();

  await page.locator('#textInput').fill("Hello world"); //Alter state
  const afterImage = await page.locator('#textInput').screenshot();

  const comparator = getComparator('image/png');
  //Perform image comparison - can use options object to handle allowable variation
  const result = await comparator(beforeImage,afterImage,{maxDiffPixels: 150});
  if(result!==null){ //If there is no difference (or found difference is tolerated) result will be null
    const diffImage = result.diff; //Get diff image buffer
    //Attach images to report
    await testInfo.attach('Expected image', {body: beforeImage, contentType: 'image/png'});
    await testInfo.attach('Actual image', {body: afterImage, contentType: 'image/png'});
    await testInfo.attach('Diff image', {body: diffImage, contentType: 'image/png'});
  }
  
  expect(result).toBeNull(); 

});

This results in the test faiing (as it should!) with the error…

…and with screenshots in the report:

Sadly there’s no slider comparison tool. I feel nervous about using internal functions because Playwright might not include these methods in future versions, or their functionality could change. Is there a safer way to achieve something similar?

Meeting expectations

Returning to .toHaveScreenshot(), this method wants the expected screenshot to be present on disc – so why not just put it there before calling .toHaveScreenshot()? This way we will get the usual reporting benefits and ability to handle expected image variations “for free”.

Using .screenshot() if we’re not capturing the resulting image in to a variable we can pass an options object with a path property to write the file to the disc:

await page.locator('#textInput').screenshot({path: 'filename.png'});

Remember from part 1 that .toHaveScreenshot() looks for files in the test snapshot directory (not in the project root folder) and the expected file name includes the OS and browser name (as different browser and OS combinations will result in slightly different page renderings). TestInfo will give us most of the runtime information we need, and browserName can be used to discover with WebBrowser is currently executing our test:

test("compare runtime images", async ({page, browserName}, testInfo)=>{
    await page.goto('https://www.edgewordstraining.co.uk/webdriver2/docs/forms.html');

    await page.locator('#textInput').fill('Hello World'); //Set initial state

    //screenshots will need to:
    //  *be saved in to the test snapshot directory
    //  *with an identifiable name
    //  *and include the browser name and OS used
    await page.locator('#textInput')
      .screenshot({path: `${testInfo.snapshotDir}/textbox-${browserName}-${testInfo.snapshotSuffix}.png`});
    
    await page.locator('#textInput').fill("Hello world"); //Change element text

    //Recapture screenshot, compare to previous (on disk) version.
    await expect(page.locator('#textInput')).toHaveScreenshot('textbox.png',{maxDiffPixels: 170});
});

As promised the (failing) report is a lot more readable and useable:

Summary

Capturing and comparing screenshots at run time isn’t (yet) straightforward with Playwright Test. In this article I presented two approaches capturing run time screenshots and comparing them:

• Leverage Playwright Test’s internal code to perform the comparison in memory, taking additional steps to write the results to the report.
• Manually capture the initial screenshot to the disc using .screenshot() ensuring that the filename used matches the future expectations of Playwright Tests .toHaveScreenshot() method.