We detected that the Chromium Renderer process just crashed.This is the equivalent of seeing the 'sad face' when Chrome dies.This can happen for a number of different reasons:- You wrote an endless loop and you must fix your own code- You are running Docker (there is an easy fix for this: see link below)- You are running lots of tests on a memory intense application.    - Try enabling experimentalMemoryManagement in your config file.    - Try lowering numTestsKeptInMemory in your config file.- You are running in a memory starved VM environment.    - Try enabling experimentalMemoryManagement in your config file.    - Try lowering numTestsKeptInMemory in your config file.- There are problems with your GPU / GPU drivers- There are browser bugs in ChromiumYou can learn more, including how to fix Docker here:https://on.cypress.io/renderer-process-crashed

The occurrence of the crashes was rare initially. But as our test suitesexpanded the crash frequency increased as well. The crashes were so frequent ata point in time that none of our tests would run to completion. Neither thesolutions mentioned in the official documentation nor the suggestions in thecommunity discussions (like enabling experimentalMemoryManagement) wereeffective. This led us to investigate this problem.

About our CI setup

We used to run Cypress onCircleCIon a medium Docker resource class. This resource class allocates 4GB of memoryto the process. Later on, we moved to our home-grown CI solution,NeetoCI for running our Cypress tests whichgave us much more control over the test environment.

The investigation setup

Since the errors were caused because Cypress ran out of memory, we started bylooking into the resource utilization on the VM environment. We noticed thatnone of the crashed runs used more than 50% of the allotted memory. The memorystarvation while using only a portion of the allocated resources, meant thatCypress was not utilizing the full memory.

We couldn't reproduce this issue reliably, so we attempted to simulate the errorby creating a high memory usage scenario. For the simulation, we created a dummytest that takes the following steps.

1. Visit a page.
2. Get an element.
3. Save the element in the memory as a new alias.
4. Repeat steps 1-3 infinitely until the browser crashes.
5. During each iteration, log the iteration number to know how many iterationswere completed successfully before the crash.

For logging the iteration number, we used theCypress task - log is illustratedin the official documentation. The iteration number provided us with anadditional metric to compare the performance of the solutions we tried. The codefor implementing the investigation setup can be seen below.

const saveButtonAsAlias = iteration => {  cy.get(".button").as(`button-${iteration}`);  saveButtonAsAlias(iteration + 1);  cy.task("log", iteration);};it("dummy test", () => {  cy.visit("/");  saveButtonAsAlias(1);});

The above code will save the same button component as different aliases in thememory thus simulating a high memory usage test environment. On executing thistest we saw that the memory usage peaked at about 1GB - 1.5GB in a 4GB dockerenvironment before the browser crashed.

Solutions

1. Using an alternate browser

Even though Google Chrome is the most popularbrowser in the market, it's far from being the most memory-efficient. So wetested out with other chromium-based browsers available for Cypress andconcluded that Microsoft Edge ran thetests in a much more memory-efficient manner. While running the dummy test, weobserved the memory usage by each of the browsers and compared the results.

Google Chrome ran the tests faster and crashed first when memory was starved.Microsoft Edge ran the tests at a similar pace initially, but when the memorywas almost used up completely, the tests slowed dow,n and the browser startedrigorous garbage collection. The memory usage was increasing at a gradual rateand more iterations were completed successfully, as compared to Chrome, beforethe browser crashed. The table below shows the runtime comparison between GoogleChrome and Microsoft Edge (higher runtime is better).

<table><tr><td> Attempt </td><td> Google Chrome runtime before crash </td><td> Microsoft Edge runtime before crash </td></tr><tr><td>1</td><td>0:45</td><td>0:59</td></tr><tr><td>2</td><td>0:46</td><td>1:00</td></tr><tr><td>3</td><td>0:45</td><td>1:01</td></tr></table>

While switching the browser improved the completion rate of the runs, it stilldidn't solve the issue completely. This led us to look for further enhancements.

2. Increasing the max-old-space-size

The most unusual behaviour we noticed in resource utilization was that Cypressdid not use the entire allocated memory before crashing. To understand whyCypress behaves like this we need to have a basic understanding of itsarchitecture which can be seen below.

Cypress works as two different processes. The NodeJS application and the browseron which the tests run. When executing cypress run and cypress opencommands, we start the NodeJS application. This NodeJS application goes throughour tests and configuration and loads them into our preferred browser where theyare executed.

The split architecture of Cypress means that the memory allocation for theNodeJS process and the Chromium browser are different. This is why the totalmemory usage by the NodeJS process doesn't give us proper insights into why theChromium process crashed and was starved of memory. To analyze the browsermemory usage we used the browserPerformance APIs.

We found that the Cypress tests were allocated only about 500MB of memorydespite the test environment having 4GB of memory. So the solution was toincrease the heap memory allocated to the chromium renderer. Themax-old-space-sizecommand-line flag is used to set the V8 engine's maximum old memory limit. Whenthe memory usage approaches this limit, garbage collection begins in an effortto free up memory. So by manually increasing the max-old-space-size for thechromium renderer, we can increase the heap memory allocated to it.

If it were a node application, the process of increasing themax-old-space-size would be as simple as executing the Cypress commandlike-wise:

NODE_OPTIONS=--max-old-space-size=3500 yarn cypress run

But because of the split architecture, executing the above command onlyincreases the max-old-space-size for the NodeJS application and not the actualCypress tests running in the Chromium browser. To increase themax-old-space-size for the Chromium renderer we need to make use of theBrowser launch APIsprovided by Cypress.

// cypress.config.jsconst { defineConfig } = require("cypress");module.exports = defineConfig({  // setupNodeEvents can be defined in either  // the e2e or component configuration  e2e: {    setupNodeEvents(on, config) {      on("before:browser:launch", (browser = {}, launchOptions) => {        launchOptions.args.push("--js-flags=--max-old-space-size=3500");        return launchOptions;      });    },  },});

In the configuration above, we can see that we have passed in the--max-old-space-size command line flag within the --js-flags Chromium flag.This is because Chromium expects NodeJS options using the --js-flags commandline switch. The above configuration increases the maximum usable heap size ofthe Cypress tests to 3500MB.

Depending on the available memory on the test environment, we can increase ordecrease the max-old-space-size value. The benchmarking results we receivedafter making this configuration change showed a significant improvement in theperformance. The table below documents the runtime comparison between thedefault max-old-space-size and max-old-space-size set to 3500 MB (higherruntime is better).

<table><tr><td> Attempt </td><td>Runtime before crash with default <code>max-old-space-size</code></td><td>Runtime before crash with <code>max-old-space-size=3500</code></td></tr><tr><td>1</td><td>0:44</td><td>2:22</td></tr><tr><td>2</td><td>0:45</td><td>2:20</td></tr><tr><td>3</td><td>0:45</td><td>2:21</td></tr></table>

The benchmark above shows the improvement in performance after increasing themax-old-space-size in the Google Chrome browser. By switching the browser toMicrosoft Edge we got even better results. The table below shows the runtimecomparison between the default max-old-space-size and max-old-space-size setto 3500 MB in each of these browsers (higher runtime is better).

<table><tr><td> Attempt </td><td colspan="2">Runtime before crash with default <code>max-old-space-size</code></td><td colspan="2">Runtime before crash with <code>max-old-space-size=3500</code></td></tr><tr><td></td><td>Google Chrome</td><td>Microsoft Edge</td><td>Google Chrome</td><td>Microsoft Edge</td></tr><tr><td>1</td><td>0:44</td><td>0:59</td><td>2:22</td><td>2:40</td></tr><tr><td>2</td><td>0:45</td><td>1:00</td><td>2:20</td><td>2:38</td></tr><tr><td>1</td><td>0:45</td><td>1:01</td><td>2:21</td><td>2:37</td></tr></table>

Additional tips to reduce memory usage in Cypress

1. Chromium browsers sandbox the pages which increases the memory usage. Sincewe're running the Cypress tests on trusted sites, we can enable the--no-sandbox flag to reduce memory consumption.

2. When running Cypress tests in headless mode, we can disable the WebGLgraphics on the rendered pages to avoid additional memory usage by passingthe --disable-gl-drawing-for-tests flag.

3. When running tests on low-resource machines, using hardware acceleration canimpact performance. To avoid this we can pass the --disable-gpu flag.

// cypress.config.jsconst { defineConfig } = require("cypress");module.exports = defineConfig({  // setupNodeEvents can be defined in either  // the e2e or component configuration  e2e: {    setupNodeEvents(on, config) {      on("before:browser:launch", (browser, launchOptions) => {        if (["chrome", "edge"].includes(browser.name)) {          if (browser.isHeadless) {            launchOptions.args.push("--no-sandbox");            launchOptions.args.push("--disable-gl-drawing-for-tests");            launchOptions.args.push("--disable-gpu");          }          launchOptions.args.push("--js-flags=--max-old-space-size=3500");        }        return launchOptions;      });    },  },});

Conclusion

Since Cypress tests are executed inside the browser all the constraints of abrowser environment apply to them, including the memory constraints. The defaultconfigurations in the browsers are targeted to run on the most number ofsystems. When facing memory starvation issues during complex and long-runningtests, we should configure Cypress according to the resources available in theenvironment in which our tests are running to achieve peak performance.Increasing the available memory for the browser by manually setting anappropriate max-old-space-size value and choosing a memory-efficient browserwill make sure that Cypress will be able to run smoothly in most of thescenarios.

References

• Chromium renderer crash GitHub issue
• Memory management glossary
• Increasing node memory size
• Cypress docker images
• Chromium command-line switches
• Docker resource constraints

Taming flakiness in Cypress

In Cypress, the cy.wait() command is used to pause the test execution. Let'sexplore how Cypress handles flakiness with the cy.intercept() and cy.wait()commands.

Let's consider an example of an online shopping application where a new order iscreated when we click the submit button.

cy.intercept("/orders/*").as("fetchOrder");cy.get("[data-cy='submit']").click();cy.wait("@fetchOrder");

Let's understand what the above example is trying to achieve line by line.

cy.intercept("/orders/\*").as("fetchOrder"): Sets up a network interception.It intercepts any network request that matches the pattern /orders/ and givesit a unique alias fetchOrder. This allows us to capture and control thenetwork request for further testing.

cy.get("[data-cy='submit']").click(): Locates an HTML element with theattribute data-cy set to submit and simulates a click on it.

cy.wait("@fetchOrder"): Instructs Cypress to wait until the interceptednetwork request with the alias fetchOrder is completed before proceeding withthe test.

The cy.wait() command involves two distinct phases of waiting.

Phase 1: The command waits for a matching request to be sent from thebrowser. In the provided example, the wait command pauses execution until arequest with the URL pattern /orders/ is initiated by the browser. Thiswaiting period continues until a matching request is found. If the command failsto identify such a request within the configured request timeout, a timeouterror message is triggered. Upon successfully detecting the matching request,the second phase kicks in.

Phase 2: In this phase, the command waits until the server responds. If theanticipated response fails to arrive within the configured response timeout, atimeout error is thrown. In the above example, the wait command in this phasewill wait for the response of the request aliased as fetchOrders.

The dual-layered waiting mechanism, as explained above, significantly contributesto the reliability of tests. It ensures a synchronized interaction between UIactions and server responses, facilitating more robust and dependable testscenarios.

Managing multiple responses

Consider a situation where a user adds a product to the cart thus initiating twoconcurrent requests. The first request adds the product to the cart, while thesecond request fetches the updated list of orders. To ensure the synchronizationof these asynchronous actions, we must wait for both requests to be successfullycompleted before continuing with the test execution.

Cypress provides the times property in the cy.intercept() options, offeringcontrol over how many times a request with a particular pattern should beintercepted.

cy.intercept({ url: "/orders/*", times: 2 }).as("fetchOrders");cy.get("[data-cy='submit']").click();cy.wait(["@fetchOrders", "@fetchOrders"]);

Let's decode the above example line by line.

cy.intercept({ url: "/orders/\*", times: 2 }).as("fetchOrders"): Specifiesthat the interception should match requests with a pattern /orders/ and limitthe interception to exactly two occurrences.

cy.get("[data-cy='submit']").click(): Locates an HTML element with theattribute data-cy set to submit and simulates a click on it.

cy.wait(["@fetchOrders", "@fetchOrders"]): Ensures that the test waits untilthe two intercepted requests with the alias fetchOrders are completed beforemoving on to the next steps.

Taming Flakiness in Playwright

Playwright offers page methods like waitForRequest and waitForResponse toaddress synchronization challenges between UI actions and API responses. Boththese methods return a promise which is resolved when an API with a matchingpattern is found and throws an error if it exceeds the configured timeout.

Let's consider the same example of an online shopping application where a neworder is created when we click the submit button.

await page.getByRole("button", { name: "Submit" }).click();await page.waitForResponse(response => response.url().includes("/orders/"));

In the above example, page.waitForResponse waits for a network response thatmatches with the URL pattern /orders/ after clicking the submit button.

Even though the above example seems simple, there is a chance for flakinesshere. That is because the API might respond before Playwright starts waiting forit. It might happen for two reasons:

1. API is very fast.
2. External factors delay the test script.

Such situations could lead to timeouts and test failures.

To address the above issue, it's important to coordinate the promises so thatthe waitForResponse command runs at the same time as UI actions. The followingexample illustrates this approach.

const fetchOrder = page.waitForResponse(response =>  response.url().includes("/orders/"));await page.getByRole("button", { name: "Submit" }).click();await fetchOrder;

In the above example, the page starts watching for the responses matching thespecific URL pattern, /orders/, before clicking the submit button. ThewaitForResponse command returns a promise, which we have saved into thevariable fetchOrder. After performing the click action in the following line,we wait for the promise stored in fetchOrder to resolve. When it resolves, itsignifies that the response has been received. This enables us to move on to thenext assertion without facing any reliability issues.

Managing Multiple Responses

Let's consider a scenario similar to the one explained in Cypress, where we haveto manage multiple responses, one to add a product and another to fetch theupdated list of products.

To wait for the completion of 2 requests from the same URL pattern, consider thefollowing approach.

const fetchOrders = Promise.all(  [...new Array(2)].map(    page.waitForResponse(response => response.url().includes("/orders/"))  ));await page.getByRole("button", { name: "Submit" }).click();await fetchOrders;

In the above example, we start waiting for two responses with the pattern/orders/ using Promise.all. The flaw in the above code is that when both thewaitForResponse methods run in parallel, and they end up tracking the exact sameAPI request. In simpler terms, it's like waiting for just one request, as bothof them wait for the completion of the same API.

To solve the above problem, it's important to improve the code by keeping trackof the resolved APIs. Let's see how to achieve the same.

const trackedResponses = [];const fetchOrders = Promise.all(  [...new Array(2)].map(() =>    page.waitForResponse(response => {      const requestId = response.headers()?.["x-request-id"];      if (        response.url().includes("/orders/") &&        !trackedResponses.includes(requestId)      ) {        trackedResponses.push(requestId);        return true;      }      return false;    })  ));await page.getByRole("button", { name: "Submit" }).click();await fetchOrders;

In the above example, we have initialized a new variable trackedResponses withan empty array, intended to store unique identifiers (request IDs) of resolvedAPIs. It checks if the URL includes the substring /orders/ and also whetherthe request ID has not already been tracked in trackedResponses array. If bothconditions are satisfied, it adds the request ID to trackedResponses array andreturns true, indicating that we should wait for the response. This approachprevents the monitoring of the same response more than once.

Conclusion

By understanding and implementing these synchronization techniques in Cypressand Playwright, we can significantly enhance the robustness and reliability ofend-to-end tests, ultimately contributing to a more stable and trustworthytesting suite.

References

cy.intercept

cy.wait

page.waitForRequest

page.waitForResponse