We learned so much about Playwright traces while adding the Playwright's tracingcapabilities to NeetoPlaydash. We builtNeetoPlaydash and it is the most affordable Playwright Dashboard. NeetoPlaydashcollects, monitors and debugs Playwright test reports.

What are Playwright traces?

Playwright traces are compressed archives (.zip files) that contain:

• Complete DOM snapshots before, during, and after each action.
• Screenshots of the browser state at every step.
• Network activity, including all HTTP requests and responses.
• Console logs from both the browser and your test.
• Action timeline showing what your test did and when.
• Source code mapping actions back to your test files.
• Metadata about the test environment, browser, and configuration.

Think of traces as a time machine for test execution - we can pause at anymoment and see exactly what the browser looked like, what network calls were inflight, and what our code was doing.

What is the Trace Viewer?

The Trace Viewer is an application used to view and analyze the informationcollected in trace files. Traces are information collected into a zip fileduring test execution. The Trace Viewer makes sense of and displays thisinformation in an interactive interface.

We can view traces in two ways:

• Locally - Using the Trace Viewer app that ships with Playwright (via thePlaywright CLI).
• Online - By visiting trace.playwright.devand uploading our trace file.

Both methods provide the same powerful debugging experience, allowing us tonavigate through our test execution, inspect DOM snapshots, analyze networkrequests, and debug failures.

Understanding the Trace Viewer interface

One can download asample trace fileto follow along with this blog. We can also view it directly in theTrace Viewer.

The Trace Viewer interface is divided into several key areas that work togetherto help us debug your tests.

1. Action Timeline (Top)

A visual filmstrip showing screenshots of our test execution over time. We can:

• Hover to see magnified previews.
• Click to jump to specific moments.
• Double-click an action to focus on its time range.
• Drag to select a range of actions for filtering.

2. The Actions Tab: Our test execution timeline

The Actions tab is typically our starting point. Each action shows:

• Duration anomalies - Actions taking unusually long suggest performanceissues or waiting problems.
• Locator information - Verify that we are targeting the correct elements.
• Action sequence - Ensure actions execute in the expected order.
• Red/failed actions - These are our primary debugging targets.

Hover over each action to see the DOM highlight change in real-time. This helpsverify that Playwright is interacting with the correct element.

Example:

 page.goto("https://app.neetocal.com") - 1.2s page.getByTestId("login-button").click() - 0.3s page.getByTestId("email-input").fill("test@example.com") - 30s TIMEOUT

Here, the fill action timed out. Select the action and use the Before andAfter tabs above the main snapshot viewing area to see why the elementwasn't available.

3. Metadata Tab: Environment context

The Metadata section provides high-level contextual information about thetest execution environment and run characteristics. This information helps inunderstanding when, how, and under what conditions the trace was recorded.

• Time: Displays the start time, end time, and total duration of the test.
• Browser: Displays the browser (e.g., Chromium, Firefox, WebKit), platform,and user agent used during the test run.
• Config: Shows the Playwright configuration applied for the run. Includesrelevant settings such as test options, retries, timeouts, and project-leveloverrides.
• Viewport: Specifies the viewport dimensions used during execution. Helpsdiagnose layout, responsiveness, and visual issues tied to screen size.
• Counts (Metrics): Summarizes key execution metrics captured in the trace,such as:
  • Pages - Pages captured during the trace.
  • Actions - Actions performed during the test.
  • Events - Runtime events logged during execution.

This is particularly useful when:

• Tests pass locally but fail in CI.
• Tests behave differently across browsers.
• Issues are viewport-specific (responsive design bugs).

4. Main Content Area (Center)

The Main Content Area is the primary visual workspace of the Trace Viewer.It displays detailed, interactive views of the application state for theselected action, enabling precise inspection and debugging.

Pick Locator

Allows us to interactively select elements directly from the snapshot.Automatically generates the corresponding Playwright locator, helping validateselectors and improve test reliability.

Snapshots: Time-Travel Debugging

Snapshots capture the complete DOM state at three critical moments:

• Action - The exact moment of interaction (showing the precise clickcoordinates or input position with a red dot).
• Before - State when the action was called.
• After - State after the action is completed.

Using snapshots effectively:

• Compare Before/After to see what changed.
• Inspect element visibility - Was the element actually visible andinteractable?
• Check for overlays - Are modals, loading spinners, or other elementsblocking interaction?
• Verify element state - Is the button disabled? Is the input read-only?

Debugging technique: When a click fails, examine the highlighted clickposition in the Action snapshot. If it's not where we expect, we may have:

• Multiple elements matching your locator (strict mode violation).
• An element that moved during Playwright's auto-wait.
• An element obscured by another element (z-index issues).

Open Snapshot in a New Tab

Opens the current DOM snapshot in a separate browser tab. Useful for deepinspection, side-by-side comparison, or analyzing complex layouts without losingtrace context.

5. Tab Bar (Bottom/Right)

Locator Tab

This is useful to get the locator of any element. Click the Locator button andhover over any component in the snapshot; the locator for that element willappear in the code space below the button. The reverse is also possible - typinga locator in the code space will highlight the corresponding element in the maincontent area, making it easy to verify locators and test selectors.

Call Tab: Action details

The Call tab provides granular information about each action:

• Function signature - The exact Playwright method called.
• Parameters - Arguments passed to the function.
• Locator string - How Playwright found the element.
• Strict mode - Whether strict mode was enforced.
• Timeout - Maximum wait time configured.
• Return value - The resolved value of the Playwright function call.

Example:

await expect(page.getByTestId("publish-btn")).toBeDisabled({ timeout: 10_000 });

The corresponding details would show the function signature, parameters, andexecution result.

Log Tab: Playwright's internal actions

The Log tab reveals what Playwright does behind the scenes. ContainsPlaywright-generated internal logs for the selected action. Provides insightinto retries, waiting behavior, timeouts, and internal decision-making.

 waiting for getByTestId('submit-button'). locator resolved to <button>Submit</button>. scrolling element into view if needed. waiting for element to be visible. waiting for element to be enabled. waiting for element to be stable. waiting for element to receive pointer events. performing click action. click action completed.

Why this matters: Understanding Playwright's auto-wait mechanism helps us infollowing areas:

• Identify which wait condition failed.
• Optimize your locators.
• Add appropriate waits when auto-wait isn't sufficient.
• Debug flaky tests caused by race conditions.

Errors Tab: Failure analysis

When tests fail, the Errors tab is our first stop. It shows:

• Error messages with stack traces.
• Playwright's failure reason.
• Timeout information.
• Expected vs. actual states (for assertions).

The timeline also highlights errors with a red vertical line, making it easyto see when things went wrong. Lists errors and exceptions associated with theaction or test step. Includes failure messages, stack traces, and error types toquickly identify what went wrong.

Console Tab: Browser and test logs

The Console tab shows all console output, including:

• console.log, console.error, console.warn from our application.
• Browser warnings and errors.
• Test framework logs.
• Playwright's internal logs.

Visual indicators:

• Different icons distinguish between browser console logs and test logs.
• Error messages are highlighted in red.
• Warnings appear in yellow.

Filtering console logs: Double-click an action in the sidebar to filterconsole logs to only that action's timeframe. This is crucial when dealing withverbose applications.

Common patterns to look for:

// React errorsWarning: Can't perform a React state update on an unmounted component// Network errorsFailed to load resource: the server responded with a status of 404// Application errorsUncaught TypeError: Cannot read property 'id' of undefined// CORS issuesAccess to fetch at 'https://api.example.com' has been blocked by CORS policy

The Network Tab: API and resource investigation

The Network tab is invaluable for debugging issues related to:

• API failures.
• Slow page loads.
• Missing resources.
• Authentication problems.

Key columns:

• Method (GET, POST, PUT, DELETE, etc.)
• URL (full request path)
• Status (200, 404, 500, etc.)
• Content Type (application/json, text/html, etc.)
• Duration (request time)
• Size (response size)

Filtering network requests:

• Use the timeline to select a specific action range.
• The network tab automatically filters to show only requests during thatperiod.

What to investigate:

 GET /api/auth/session - 200 - 45ms - application/json POST /api/bookings/create - 500 - 2.1s - application/json GET /api/user/profile - 200 - 89ms - application/json

The 500 error on the booking creation is our culprit. Click on it to see:

• Request headers - Is authentication included (CSRF token)?
• Request body - Is the payload correct?
• Response headers - Any CORS issues?
• Response body - What error message did the server return?

Source Tab: Connecting traces to code

The Source tab displays our test code and highlights the exact linecorresponding to the selected action. This is crucial for:

• Understanding what your test was trying to do.
• Verifying locators and test logic.
• Jumping between test code and execution results.

Workflow:

1. Click an action in the sidebar.
2. Source tab automatically shows the relevant code line.
3. Review the locator, expected behavior, and assertions.
4. Cross-reference with the DOM snapshot to verify assumptions.

Attachments Tab: Visual regression and screenshots

For tests using visual comparisons or custom attachments, this tab shows:

• Screenshot comparisons (expected, actual, diff)
• Image slider to overlay images and spot differences
• Custom attachments added via test.attach()
• Screenshots & Video recordings (if configured)

Visual regression workflow:

1. Navigate to the Attachments tab.
2. View the diff image, highlighting differences in red.
3. Use the slider to compare expected vs. actual.
4. Determine if changes are legitimate or bugs.

Advanced debugging techniques

1. Time-Range filtering for complex tests

For long-running tests with many actions:

1. Click a starting point on the timeline.
2. Drag to an ending point.
3. All tabs (Actions, Network, Console) filter to this range.
4. Focus your investigation on the relevant portion.

2. Network request correlation

When debugging API-dependent tests:

1. Find the failing action.
2. Check the Network tab for API calls during that action.
3. Verify request/response timing.
4. Ensure data contracts match expectations.

Example investigation:

Test: User creates a booking.Action: page.click("button[type=submit]").Network: POST /api/bookings/create - 400 Bad Request.Response: {"error": "Invalid time slot"}.Conclusion: Form validation passed, but the API rejected the data.Next step: Check whether the time slot selection logic has a bug.

3. Console error causation

Browser console errors often precede test failures:

1. Review the Console tab chronologically.
2. Look for errors before the failing action.
3. JavaScript errors may prevent event handlers from working.
4. Network errors may leave the UI in an invalid state.

4. Locator strategy validation

When the element isn't found:

1. Check the Before snapshot - is the element present?
2. Review the Call tab - is the locator correct?
3. Use browser DevTools on the snapshot to test alternative locators

5. Race condition detection

Flaky tests often have race conditions:

1. Compare traces from passed vs. failed runs.
2. Look for timing differences in the Network tab.
3. Check if elements appear/disappear between snapshots.

Common debugging scenarios

Scenario 1: "Element Not Found" errors

Trace analysis steps:

1. Navigate to the failing click/fill action.
2. Examine the Before snapshot - is the element in the DOM?
3. Check the Console for JavaScript errors that might prevent rendering.
4. Review Network tab - did the page load completely?
5. Verify the locator in the Call tab matches the element you expect.

Possible causes:

• Element hasn't rendered yet (needs waitForSelector).
• Wrong locator (typo, dynamic attributes).
• Element is in a different frame/iframe.
• Previous action failed, leaving UI in an unexpected state.

Scenario 2: "Timeout Waiting for Element" errors

Trace analysis steps:

1. Check element visibility in the Before snapshot.
2. Look at the element's CSS properties (display, opacity, visibility).
3. Check for z-index issues or overlapping elements.
4. Review the Network tab for slow API responses blocking the UI.
5. Check the Console for loading state indicators.

Possible causes:

• CSS hides the element.
• Loading spinner still active.
• API call hasn't completed.
• Modal or overlay blocking interaction.
• Element removed and re-added (Playwright lost reference).

Scenario 3: API failures causing test failures

Trace analysis steps:

1. Filter the Network tab to the action's timeframe.
2. Find failed API requests (4xx, 5xx status codes).
3. Inspect request payload - is test data valid?
4. Check the response body for error details.
5. Verify authentication headers are present.

Possible causes:

• Test data doesn't match API validation rules.
• Authentication token expired (X-CSRF token).
• Database state inconsistent (previous test didn't clean up).
• API endpoint changed (version mismatch).

Scenario 4: Tests pass locally but fail in CI

Trace analysis steps:

Run the test locally as well and open the traces for both runs (local and CI):

1. Compare the Metadata tab - check browser, viewport, timezone differences
2. Look for timing differences in action durations
3. Check for environment-specific console errors
4. Compare Network tab - are API endpoints different?

Possible causes:

• Timezone-dependent test data.
• Slower CI environment (needs longer timeouts).
• Different environment variables.
• Some other tests are affecting the test (very rare case).

Scenario 5: Flaky tests (Intermittent failures)

Trace analysis steps:

1. Compare traces from multiple runs (passed and failed).
2. Look for timing variations in Network requests.
3. Check for race conditions between actions.
4. Review auto-wait logs for differences in element stability.
5. Look for animations or transitions affecting element states.

Possible causes:

• Race conditions between UI updates and test actions.
• Async operations without proper waits.
• Animations not completing before interaction.
• Network request order is non-deterministic.
• Shared test state between test runs.

Best practices for trace analysis

1. Start with the error

Always begin at the point of failure. The Errors tab and red timeline markersguide you directly there.

2. Trace backwards

After identifying the error, trace back through the actions to determine itsroot cause, which may be linked to an event that occurred several steps earlier.

3. Compare known-good traces

If you have a passing trace, compare it side-by-side with the failing trace tospot differences quickly.

4. Use timeline filtering liberally

Don't drown in information. Filter the timeline to focus on relevant actions andreduce noise.

5. Correlate across tabs

True debugging power comes from correlating information across tabs:

• Action timing + Network requests + Console logs = complete picture

6. Document your findings

When you identify the root cause, document it:

• Add comments to your test code.
• Update test data or fixtures.
• Fix race conditions with proper waits.
• Report application bugs with trace evidence.

7. Configure appropriate trace collection

In your playwright.config.ts:

export default defineConfig({  use: {    // Capture trace only on first retry (recommended for CI)    trace: "on-first-retry",    // Or retain traces only for failures    // trace: 'retain-on-failure',    // For local debugging, enable traces for all tests    // trace: 'on',  },  // Enable retries to capture traces on failures  retries: process.env.CI ? 2 : 0,});

Trace Viewer keyboard shortcuts

Speed up your analysis with these shortcuts:

• Arrow keys - Navigate between actions.
• Esc - Clear selection/filtering.
• Ctrl/Cmd + F - Search within trace.

Accessing traces in NeetoPlaydash

NeetoPlaydash is a test management platformthat integrates seamlessly with Playwright's tracing capabilities. When a testruns on CI, Playwright automatically generates trace files. To access them:

1. Navigate to NeetoPlaydash - Open your projects dashboard.
2. Select a Test Run - Choose the test execution you want to investigate fora particular project.
3. Open Test Details Pane - View the details of a specific test.
4. Click "Open Trace" - This button launches the Trace Viewer with yourtest's trace file.

The trace opens in your browser at trace.playwright.dev or locally via thePlaywright CLI, providing a complete, interactive debugging experience withoutrequiring manual file downloads.

Further resources

• Playwright Trace Viewer Documentation
• Playwright Best Practices
• Debugging Playwright Tests
• Online Trace Viewer
• Learn QA Automation using Playwright Course

We at BigBinary also switched from Cypress to Playwright in late 2023. In thisarticle, we will see some critical reasons for this change in trends and ourpersonal views on why we think Playwright is the superior JavaScript testingframework.

<div style="width:100%;max-width:600px;margin:auto;display:grid;grid-template-columns:auto auto;gap:2rem;align-items:end;"><figure style="display:flex;flex-direction:column;align-items:center;gap:0.5rem;"><imgwidth="1000"height="600"alt="Cypress weekly downloads early 2024"src="/blog/images/images_used_in_blog/2024/why-we-switched-from-cypress-to-playwright/cypress-weekly-downloads-early-2024.png"/><figcaption style="font-size:x-small;">Cypress weekly downloads - Early 2024</figcaption></figure><figure style="display:flex;flex-direction:column;align-items:center;gap:0.5rem;"><imgwidth="1000"height="600"alt="Playwright weekly downloads early 2024"src="/blog/images/images_used_in_blog/2024/why-we-switched-from-cypress-to-playwright/playwright-weekly-downloads-early-2024.png"/><figcaption style="font-size:x-small;">Playwright weekly downloads - Early 2024</figcaption></figure><figure style="display:flex;flex-direction:column;align-items:center;gap:0.5rem;"><imgwidth="1000"height="600"alt="Cypress weekly downloads September 2024"src="/blog/images/images_used_in_blog/2024/why-we-switched-from-cypress-to-playwright/cypress-weekly-downloads-september-2024.png"/><figcaption style="font-size:x-small;">Cypress weekly downloads - September 2024</figcaption></figure><figure style="display:flex;flex-direction:column;align-items:center;gap:0.5rem;"><imgwidth="1200"height="800"alt="Playwright weekly downloads September 2024"src="/blog/images/images_used_in_blog/2024/why-we-switched-from-cypress-to-playwright/playwright-weekly-downloads-september-2024.png"/><figcaption style="font-size:x-small;">Playwright weekly downloads - September 2024</figcaption></figure></div>

Why we chose Cypress initially

At BigBinary, we are building a number of products atNeeto. When the number of products in our product suitegrew and the complexity of each one increased, we needed an automated end-to-endsolution to ensure our applications were stable since manual testing was nolonger viable.

When the discussion about choosing the e2e testing framework began in mid-2020,a few names emerged, including top players like Selenium and Cypress, and newplayers like Playwright. We chose Cypress owing to its popularity andsimplicity.

We were satisfied with its overall performance and easy learning curve. We choseCypress as our primary e2e testing framewor,k and wrote extensive e2e tests forthe entire application suite. While things were smooth sailing initially, wesoon encountered many issues with Cypress.

Why we decided to switch to Playwright

In late August 2023, Cypress released version 13, a major upgrade to theframework that brought along many new features. As Cypress users, we wereoverjoyed. But the excitement quickly turned to frustration when we realizedthat, along with the latest features, Cypress had introduced a few changes thatwere not so open-source in nature.

It's a known fact that Cypress Cloud is a veryexpensive platform. A few third-party providers likeCurrents.dev and Testomatprovided similar services at much more affordable costs. However, Cypressversion 13 blocked all third-party reporters. The main reason offered by theCypress team was that Cypress Cloud was their primary source of income and thatthey had to block the third-party tools to survive in the market. They laterrevised this explanation with other arguments on how these third-party reportersmisused the Cypress name for personal gain due to public backlash.

We had switched to Currents.dev ourselves a few months prior to the event andwere affected by this change. At that point, we had two options: switch toCypress Cloud and incur the additional cost, or stay with Currents.dev but getlocked into an older version of Cypress permanently.

Both of these choices were unacceptable to us. This was the final nudge weneeded to switch to a new framework. We were already dissatisfied with manyissues with Cypress, so we took advantage of the opportunity to research thebest e2e testing framework available and switch to that. We compared all thepopular frameworks available and observed how they solved our pain points withCypress. That is when we fell in love with Playwright.

In our comparison, Playwright was the fastest framework in terms of rawperformance and had the highest adoption rate compared to all the otherframeworks. It is an open-source framework maintained by Microsoft. Thearchitecture would enable us to automate more scenarios that were deemedunautomatable using Cypress. We were thrilled to learn that Playwright would fixmost of the issues we faced with Cypress.

Features locked behind a paywall and control over third-party software

While Cypress supports parallelism and orchestration, it is blocked behind apaywall with a subscription to Cypress Cloud. This means these features, whichcan easily be implemented with the base Cypress package, are only accessiblethrough an external package.

While Cypress provides APIs for reporters and orchestration, it deliberatelyblocks popular third-party tools and services. That's not a good open sourcepractice. The combination of both makes Cypress an incomplete tool withoutsubscribing to the expensive Cypress Cloud plans, even though the tool isconsidered free and open-source.

At the same time, Playwright is an entirely open framework in which anyone cancreate and publish third-party reporters. It comes in-built with features suchas parallelization, sharding and orchestration without needing third-party toolsand services. The Playwright team goes a step further by showcasing the popularthird-party reporters on their official documentation.

Performance

Cypress is the slowest of the e2e testing frameworks available in JS. Here isthe list of the most popular frameworks in decreasing order of performance.

<div style="width:100%;display:flex;justify-content:center;"><imgalt="Speed comparison of popular JS testing frameworks"src="/blog/images/images_used_in_blog/2024/why-we-switched-from-cypress-to-playwright/speed-comparison-of-testing-frameworks.png"/></div>

<br />

Let's compare the performance when the same scenario is implemented in Cypressand Playwright. The scenario is to visit the Neeto homepageand verify the page title.

// Cypresscy.visit("https://neeto.com");cy.title().should("eq", "Neeto: Get things done");

// Playwrightawait page.goto("https://neeto.com");expect(await page.title()).toBe("Neeto: Get things done");

The results speak for themselves. While Cypress took 16.09 seconds to finishthe execution, Playwright took only 1.82 seconds. This is an improvement of88.68%! Here, the execution time combines the time taken for setup and thetime to complete the test. This is the actual time that matters because this isthe time an engineer has to wait until they see the final test result.

<div><br /><figure style="display:flex;flex-direction:column;align-items:center;gap:0.5rem;"><imgalt="Cypress weekly downloads early 2024"src="/blog/images/images_used_in_blog/2024/why-we-switched-from-cypress-to-playwright/cypress-execution-time.png"/><figcaption style="font-size:small;">Cypress execution</figcaption></figure><br /><figure style="display:flex;flex-direction:column;align-items:center;gap:0.5rem;"><imgalt="Cypress weekly downloads early 2024"src="/blog/images/images_used_in_blog/2024/why-we-switched-from-cypress-to-playwright/playwright-execution-time.png"/><figcaption style="font-size:small;">Playwright execution</figcaption></figure><br /></div>

This shows how much of a performance gain switching to Playwright gave us. If welook at a more practical example, our authentication flows through Cypress, andPlaywright gives a much better idea of the time saved. The authentication flow,which consistently took around 2 minutes in Cypress, is completed in under20 seconds using Playwright.

Playwright's out-of-the-box support for parallelism and sharding can havemultiplicative effect in time savings. If we provide a process-based parallelismof 4 and shard the tests in 4 machines, then 16 tests are run concurrently,which reduces the execution time dramatically.

Implementing these additional configurations reduced the total test duration forone of our products from 2 hours and 27 minutes to just 16 minutes. This89.12% of time saving, directly translating to CI cost savings.

Memory issues

Cypress follows a split architecture. This means that Cypress executes the testswith a NodeJS process, which orchestrates the tests in the browser where thetests are executed. This also means that the browser execution environmentlimits the memory available for tests. Due to this, we have faced crashes inbetween tests multiple times. At one point, the crashes became so frequent thatwe had to invest a lot of time and energy into finding a solution because notest executions were running to completion. We have written a detailed blog onthis topic, which can be foundhere.

Playwright fixes these issues because it handles the test execution in NodeJSservice and communicates with the browsers usingCDP sessions. This meansthat the memory management can be done on the NodeJS application, while thebrowser only has to worry about handling the actual web application we'retesting.

Architecture prone to flakiness

Many of Cypress's features are closely tied to its architecture. For example,one of the popular features in Cypress is itsretry mechanismandchaining.However, these features do not always go hand-in-hand.

Let's consider this snippet of Cypress code.

cy.get(".inactive-field").click().type("Oliver Smith");

While this code looks syntactically correct, it will make the test flaky. Thisis because, in Cypress, only queries are retried, not commands. In the exampleabove, consider that the class name of the field is updated to active-fieldwhen we click on it. This means that the cy.get query locates the field andthe click command work fine, but the chain fails at the type command. Thisis because an element with the class name .inactive-field no longer exists inthe DOM tree.

With the Cypress retry mechanisms, one would think that the whole chain would beretried from fetching the element. However, the chain was completed successfullyuntil the click action. So, only the type action will be retried and willcause the whole chain to fail. To avoid this issue, we must rewrite the testsafter splitting the chain.

cy.get(".inactive-field").click();cy.get(".inactive-field").type("Oliver Smith");

While this works without issues, the syntactical sugar that Cypress provides bychaining the commands is no longer usable. Now, let's observe the Playwrightcode for the same.

await page.locator(".inactive-field").click();await page.locator(".inactive-field").type("Oliver Smith");

It looks pretty similar. This is because Playwright is designed to reduceflakiness as much as possible. To achieve that goal, it prevents the user fromimplementing anti-patterns that can lead to flaky results.

Misleading simplicity

Cypress is well known for its simplicity and natural syntax, which even the mostnon-technical person can learn. The code samples in the official documentation(which is still one of the best documentation for any framework) make it seemlike a walk in the park. But you soon realize that the examples are for verystraightforward application scenarios that we seldom encounter when working onlarge projects. When you start automating complex scenarios, things soon getcomplicated.

While performing simple tasks such as clicking on a button or asserting a textis extremely simple, doing something more moderately complex, such as storingthe text contents of a button in a variable, becomes highly complex. This isbecause Cypress architecture works by enqueuing the asynchronous commands. Thismeans that there are no return values for the commands, and the only way toretrieve values from Cypress commands is through a combination of closures andaliases. Let's consider a scenario where we have to verify that the sum of therandomly generated numbers on the screen is the same as the value shown on thepage.

<div><br/><figure style="display:flex;flex-direction:column;align-items:center;gap:0.5rem;"><img width="666" alt="Sample scenarios of the sum application" src="/blog/images/images_used_in_blog/2024/why-we-switched-from-cypress-to-playwright/sample-application-that-adds-two-numbers.png"><figcaption style="font-size:small;">A sample application that adds two random numbers</figcaption></figure><br/></div>

Let's see the difference in code when automating this scenario in Cypress andPlaywright.

// Cypress// Considering all elements have proper data-cy labelscy.get('[data-cy="generate-new-numbers-button"]').click();cy.get('[data-cy="first-number"]').as("firstNumber");cy.get('[data-cy="second-number"]').as("secondNumber");cy.get('[data-cy="sum"]').as("sum");cy.get("@firstNumber").invoke("text").then(parseInt).as("num1");cy.get("@secondNumber").invoke("text").then(parseInt).as("num2");cy.get("@sum").invoke("text").then(parseInt).as("displayedSum");// Use the aliases to perform the assertioncy.get("@num1").then(num1 => {  cy.get("@num2").then(num2 => {    cy.get("@displayedSum").then(displayedSum => {      const expectedSum = num1 + num2;      expect(displayedSum).to.equal(expectedSum);    });  });});

We can see how complicated the code becomes when the scenario is just slightlycomplex. Meanwhile, the Playwright code will look like this.

// Playwright// Considering all elements have proper data-cy labels and the default test-id-attribute is data-cyawait page.getByTestId("generate-new-numbers-button").click();const firstNumber = await page.getByTestId("first-number").innerText();const secondNumber = await page.getByTestId("second-number").innerText();const sum = await page.getByTestId("sum").innerText();expect(parseInt(firstNumber) + parseInt(secondNumber)).toBe(parseInt(sum));

We can see from the code above, how easily we can implement the same logic inPlaywright.

Cost of maintenance for Cypress vs. Playwright tests

Cypress is an easy-to-learn framework. This simplicity is due to the abstractionof the most commonly used functionalities into Cypress commands. However, thisis a double-edged sword. The abstraction of logic into commands means thatcustomization is complicated in Cypress.

One of Cypress's significant drawbacks is its reliance on HTML tags andattributes to locate an element. While this makes sense from a programmingstandpoint, the end user is concerned about the roles of the page element(button, heading, etc.) and not how they have been implemented. For the samereason, the text, appearance and functionality of the application are bound toremain consistent throughout the various iterations, while the attributesthemselves are prone to changes.

This ultimately means the developers must keep fixing/rewriting the Cypresstests for minor UI updates. Cypress is also the slowest of all the e2e testingframeworks in JavaScript, resulting in longer CI runtimes and costs. Thesecombined make the cost of maintaining Cypress tests exceptionally high.

Besides this, Cypress has many features locked behind its Cypress Cloudplatform, which is very expensive, consideringthe fact that all it does is to collect the test results. This is an additionalcost to bear over the already expensive costs of maintaining the Cypress tests.Given these factors, the cost of preserving Cypress tests can quickly outweighthe benefits of its simplicity and ease of use.

Playwright solves all of these issues. It has many built-in reporters and anexcellent API for creating custom reporters, so many third-party reporters areavailable. We can even build our custom reporter to save even more costs.

Browser support and support for mobile viewport

Since Cypress tests run directly on the browsers, only a few are supported.Until recently, it did not even support WebKit browsers,even though Safari has a considerable marketshare. Even at the time of writing this article, Cypress's WebKit support isstill in beta. Even if it supports the required browsers, there is still theconstraint that only one browser can be used during an execution.

Playwright fixes all these issues with minimal effort from our end. It hascomplete support for WebKit browsers and conveniently provides a set of presetsfor the browsers, user agents and viewport of the most popular devices in themarket, including mobile devices. Furthermore, Playwright allows us to executethe same test in different browsers concurrently with the help of projects.These configurations give us the confidence that a passing Playwright test meansthe features will work fine for all users.

Support for multiple tabs and browsers

While most of the features of a web application can be tested within a singletab, there are a few cases where multiple tabs or browsers become necessary. Onesuch scenario we encountered while writing tests forNeetoChat, a real-time chat application. To testNeetoChat we need to open two screens - one for the sender and the other for thereceiver.

Cypress lacks the support for multiple tabs, so the only way to test thesescenarios was to do this long and complicated process:

1. Login as the sender
2. Send a message
3. Logout
4. Login as the receiver
5. Verify the message
6. Send a reply
7. Logout
8. Login as the sender
9. Verify the response.

We can see the tedious steps that we need to perform for a relatively simplescenario. This becomes even more tedious if we configure sessions in Cypressbecause we need to invalidate them each time we log out so that we can log in asa different user.

On the other hand, Playwright provides support for multiple tabs and multiplebrowsers. This means we can log in as the sender from one tab and the receiverfrom another, making the scenario more straightforward and effective.Additionally, we could identify whether the messages were being delivered inreal time because there is no delay in the user switching between the messageposting and verification processes. Playwright also supports browser contexts,which isolate the events between two browser instances, aiding in test isolationduring parallel test execution.

Lack of necessary tools

Cypress depends on plugins for many necessary tools. These are features we havecome to expect from any modern testing framework. Let's examine a few such toolsand how Playwright handles them natively.

<table><tr><td>Feature</td><td>Cypress plugin</td><td>Playwright implementation</td></tr><tr><td>Waiting until a particular event completes on a page</td><td><a href="https://github.com/NoriSte/cypress-wait-until">cypress-wait-until</a></td><td>Playwright offers a variety of APIs which pause the tests until a triggerevent like<a href="https://playwright.dev/docs/api/class-page#page-wait-for-url">waitForURL</a>,<a href="https://playwright.dev/docs/api/class-page#page-wait-for-request">waitForRequest</a>,<a href="https://playwright.dev/docs/api/class-locator#locator-wait-for">waitFor</a>etc.</td></tr><tr><td>Adding steps blocks in tests to logically group commands</td><td><a href="https://github.com/filiphric/cypress-plugin-steps">cypress-plugin-steps</a></td><td><a href="https://playwright.dev/docs/api/class-test#test-step">test.step</a></td></tr><tr><td>Ability to interact with iframes</td><td><a href="https://gitlab.com/kgroat/cypress-iframe">cypress-iframe</a></td><td><a href="https://playwright.dev/docs/api/class-framelocator">frameLocator</a></td></tr><tr><td>Filtering tests based on the titles or tags</td><td><a href="https://github.com/cypress-io/cypress/tree/develop/npm/grep">@cypress/grep</a></td><td><a href="https://playwright.dev/docs/api/class-fullproject#full-project-grep">Playwright grep</a></td></tr></table>

We must consider that Playwright includes all these tools out of the box whilestill being more performant than Cypress. As we add such plugins in Cypress, thepackage size also increases.

Random errors during tests due to Cypress's iFrame Execution Model

As discussed already, Cypress tests are executed inside a browser. They work byrunning Cypress as the main page and running the application which is beingtested as an iframe within the page. This can lead to a lot of unexpected errorsduring the test execution.

One of the most commonly encountered errors is related to security issues withcookies. When the tested application uses cookies, it might throw random errorsduring the Cypress execution depending on the configuration. This is because theSameSiteconfiguration of the cookies might block it from being shared with the parentCypress application. This can lead to them not be sent during API requestscausing authentication failures and also cause difficulties withsession configuration.

Additional benefits of using Playwright

More resistance to test failures due to minor text changes

At BigBinary, we follow a behavioral testing pattern where we verify thefeatures, and not minor details like texts and styles. We made this decision toensure that the tests don't fail due to minor changes in the application. Whilethis is our go-to testing style, there are still some cases where we cannotavoid testing the texts on the page (for example, the error message shown whiletesting negative test cases). This required our tests to be frequently updatedwhile working with Cypress whenever a minor text change was made in theapplication.

When we switched to Playwright, we got excited about how much we could customizeit according to our needs. This customization is available because, under thehood, it's still a Node.js application. We already usei18next to serve the texts on our application. Wefigured that the tests should use the same translation file. The translationkeys remain consistent even when the texts are updated.

This minor change brought a huge difference in our test stability. The averagenumber of tests we had to update each week decreased from 22 to 2. Thatis a lot of time saved, which we could effectively use to expand our testcoverage instead of wasting it fixing the existing suite.

Ability to build our own in-house reporter

When working with Cypress, we had to switch between many reporting tools,including Cypress Cloud, Currents.dev, and many other third-party tools. Whilethey all had benefits and drawbacks, we couldn't find one that addressed all ourneeds. This is where the excellent reporter APIs offered by Playwright allowedus to write our own Playwright reporter -NeetoPlaydash.

We currently use NeetoPlaydash for all our reporting needs and can customize itaccording to our requirements. Most importantly, we reduced the monthlyreporting tool costs by 77% ($405 to $90 per month). We also didn't have toworry about exhausting the monthly test limits of third-party reporters,allowing us to run our tests more frequently, thus improving the stability ofour applications.

More coverage on tests relating to third-party integrations

In Neeto products, we have support for third-party integrations. For example, inNeetoCal we can have integrations forGoogle Calendar, Zoom,Microsoft teams and a lot more third-partyapplications. Most of these applications have bot detection algorithmsimplemented in place to ensure that their platforms are not misused by badactors. This also meant that we had to consider the integration features to beunautomatable when tested using Cypress.

Playwright does things differently. Since it's a Node.js application, itsupports all the packages available for the platform. Because of its widesupport, the community has developed a lot of tools for Playwright. We tookadvantage of these tools and plugins and were able to bypass the bot-detectionalgorithms that prevented us from testing the third-party integrations. Thisallowed us to test these integrations in our products effectively and ensurethat the application ran smoothly with the help of automation tests.

Conclusion

We strongly believe that migrating to Playwright is one of the best decisions wehave ever made. We did not know what we were missing out on until we decided totake the leap and migrate. We got better performance, less flakiness and morecoverage from our test suites. The cost and time saved helped us to effectivelydivert resources to things that actually matter and let the tests do testinginstead of using additional resources to maintain the tests themselves.

Taming flakiness in Cypress

In Cypress, the cy.wait() command is used to pause the test execution. Let'sexplore how Cypress handles flakiness with the cy.intercept() and cy.wait()commands.

Let's consider an example of an online shopping application where a new order iscreated when we click the submit button.

cy.intercept("/orders/*").as("fetchOrder");cy.get("[data-cy='submit']").click();cy.wait("@fetchOrder");

Let's understand what the above example is trying to achieve line by line.

cy.intercept("/orders/\*").as("fetchOrder"): Sets up a network interception.It intercepts any network request that matches the pattern /orders/ and givesit a unique alias fetchOrder. This allows us to capture and control thenetwork request for further testing.

cy.get("[data-cy='submit']").click(): Locates an HTML element with theattribute data-cy set to submit and simulates a click on it.

cy.wait("@fetchOrder"): Instructs Cypress to wait until the interceptednetwork request with the alias fetchOrder is completed before proceeding withthe test.

The cy.wait() command involves two distinct phases of waiting.

Phase 1: The command waits for a matching request to be sent from thebrowser. In the provided example, the wait command pauses execution until arequest with the URL pattern /orders/ is initiated by the browser. Thiswaiting period continues until a matching request is found. If the command failsto identify such a request within the configured request timeout, a timeouterror message is triggered. Upon successfully detecting the matching request,the second phase kicks in.

Phase 2: In this phase, the command waits until the server responds. If theanticipated response fails to arrive within the configured response timeout, atimeout error is thrown. In the above example, the wait command in this phasewill wait for the response of the request aliased as fetchOrders.

The dual-layered waiting mechanism, as explained above, significantly contributesto the reliability of tests. It ensures a synchronized interaction between UIactions and server responses, facilitating more robust and dependable testscenarios.

Managing multiple responses

Consider a situation where a user adds a product to the cart thus initiating twoconcurrent requests. The first request adds the product to the cart, while thesecond request fetches the updated list of orders. To ensure the synchronizationof these asynchronous actions, we must wait for both requests to be successfullycompleted before continuing with the test execution.

Cypress provides the times property in the cy.intercept() options, offeringcontrol over how many times a request with a particular pattern should beintercepted.

cy.intercept({ url: "/orders/*", times: 2 }).as("fetchOrders");cy.get("[data-cy='submit']").click();cy.wait(["@fetchOrders", "@fetchOrders"]);

Let's decode the above example line by line.

cy.intercept({ url: "/orders/\*", times: 2 }).as("fetchOrders"): Specifiesthat the interception should match requests with a pattern /orders/ and limitthe interception to exactly two occurrences.

cy.get("[data-cy='submit']").click(): Locates an HTML element with theattribute data-cy set to submit and simulates a click on it.

cy.wait(["@fetchOrders", "@fetchOrders"]): Ensures that the test waits untilthe two intercepted requests with the alias fetchOrders are completed beforemoving on to the next steps.

Taming Flakiness in Playwright

Playwright offers page methods like waitForRequest and waitForResponse toaddress synchronization challenges between UI actions and API responses. Boththese methods return a promise which is resolved when an API with a matchingpattern is found and throws an error if it exceeds the configured timeout.

Let's consider the same example of an online shopping application where a neworder is created when we click the submit button.

await page.getByRole("button", { name: "Submit" }).click();await page.waitForResponse(response => response.url().includes("/orders/"));

In the above example, page.waitForResponse waits for a network response thatmatches with the URL pattern /orders/ after clicking the submit button.

Even though the above example seems simple, there is a chance for flakinesshere. That is because the API might respond before Playwright starts waiting forit. It might happen for two reasons:

1. API is very fast.
2. External factors delay the test script.

Such situations could lead to timeouts and test failures.

To address the above issue, it's important to coordinate the promises so thatthe waitForResponse command runs at the same time as UI actions. The followingexample illustrates this approach.

const fetchOrder = page.waitForResponse(response =>  response.url().includes("/orders/"));await page.getByRole("button", { name: "Submit" }).click();await fetchOrder;

In the above example, the page starts watching for the responses matching thespecific URL pattern, /orders/, before clicking the submit button. ThewaitForResponse command returns a promise, which we have saved into thevariable fetchOrder. After performing the click action in the following line,we wait for the promise stored in fetchOrder to resolve. When it resolves, itsignifies that the response has been received. This enables us to move on to thenext assertion without facing any reliability issues.

Managing Multiple Responses

Let's consider a scenario similar to the one explained in Cypress, where we haveto manage multiple responses, one to add a product and another to fetch theupdated list of products.

To wait for the completion of 2 requests from the same URL pattern, consider thefollowing approach.

const fetchOrders = Promise.all(  [...new Array(2)].map(    page.waitForResponse(response => response.url().includes("/orders/"))  ));await page.getByRole("button", { name: "Submit" }).click();await fetchOrders;

In the above example, we start waiting for two responses with the pattern/orders/ using Promise.all. The flaw in the above code is that when both thewaitForResponse methods run in parallel, and they end up tracking the exact sameAPI request. In simpler terms, it's like waiting for just one request, as bothof them wait for the completion of the same API.

To solve the above problem, it's important to improve the code by keeping trackof the resolved APIs. Let's see how to achieve the same.

const trackedResponses = [];const fetchOrders = Promise.all(  [...new Array(2)].map(() =>    page.waitForResponse(response => {      const requestId = response.headers()?.["x-request-id"];      if (        response.url().includes("/orders/") &&        !trackedResponses.includes(requestId)      ) {        trackedResponses.push(requestId);        return true;      }      return false;    })  ));await page.getByRole("button", { name: "Submit" }).click();await fetchOrders;

In the above example, we have initialized a new variable trackedResponses withan empty array, intended to store unique identifiers (request IDs) of resolvedAPIs. It checks if the URL includes the substring /orders/ and also whetherthe request ID has not already been tracked in trackedResponses array. If bothconditions are satisfied, it adds the request ID to trackedResponses array andreturns true, indicating that we should wait for the response. This approachprevents the monitoring of the same response more than once.

Conclusion

By understanding and implementing these synchronization techniques in Cypressand Playwright, we can significantly enhance the robustness and reliability ofend-to-end tests, ultimately contributing to a more stable and trustworthytesting suite.

References

cy.intercept

cy.wait

page.waitForRequest

page.waitForResponse