How to run parallel Cypress tests for free

# TL;DR

If you want to run cypress tests in parallel for free, use the --spec command line attribute with the cypress run command and pass it a different directory from each machine containing the tests to run. For example you could set an env variable of DIRECTORY, and start the different tests in different ci/cd agents: cypress run --env host=$HOST --spec "tests/e2e/specs/${DIRECTORY}/*.spec.js"

# Cypress tests in the CI/CD pipelines

Sometime ago, my team and I started writing Cypress UI tests for a repository that held all the components for our static landing pages (used as server side rendered components). As more components got added to the repo, the tests grew with them. We test for different viewport sizes, and so the number of tests increased quite significantly.

At the beginning each developer would run the Cypress tests locally with the Cypress UI. It quickly became apparent that we needed a solution to run them in our build Jenkins CI pipeline, so that is what we did. Fortunately the tests only took about 5 minutes to complete.

I also added these Cypress tests to our deployment pipelines (post-deployment) in order to have even more confidence that what was deployed was working as intended. At work, we have 2 environments, staging and production. So I added a way of running these tests with the cypress/included docker image for the build and deployment pipelines.

For the build pipeline, I would start our app containing all our Vue.js components on a page, in a custom docker image based off of the alpine node docker image. Putting both the cypress/included container and our app container in the same network, one would test the other with little network latency. The reason why I didn't run cypress inside the same container where the Vue app, is because of Cypress limitations running on non official docker containers. As for the post-deployment cypress tests, the same Cypress docker container would instead test against the real staging or prod environments.

This solution worked well for a while, until the tests became too overwhelming for the cypress container. The Cypress electron browser started crashing, and tests were timing out. We added all sorts of recommended settings to remedy these issues, such as increasing the --shm-size or adding this to the plugins/index.js file:

on('before:browser:launch', (browser = {}, launchOptions) => {
  if (browser.family === 'chromium' && browser.name !== 'electron') {
    launchOptions.args.push('--disable-dev-shm-usage')
  }
  return launchOptions
})

These changes made an improvement for a little while, but tests were still taking too long to finish. It got to a point were if other CI jobs where being run in the Jenkins Agents, it would cause the Cypress tests to fail, due to the heavy resource usage on our CI instances. If say a build running the Cypress tests were happening at the same time a deployment was running its Cypress tests, the tests would break everywhere.

Moreover, the tests got to a point where they were taking 50 minutes post depolyment, and about 25 minutes on a build. The post-deployment Cypress tests always took longer due to the increased network latency (since it was requesting the real hosted pages).

# Cypress dashboard (paid for service)

In order to fix this, we looked into Cypress dashboard which offers parallelisation for a monthly fee. So we decided to go ahead with it, and got the $99/month sprout dashboard service. Once this was all wired in, we noticed that the quota we had was too little. Our test suite for our components repo had around 500 test runs. This would allow us to run the test suite in Jenkins about 50 times a month.

We knew this might be a problem but we decided to keep using it, and if we hit the quota we would go up a tier. Long story short, we added the Cypress tests to our feature CI pipeline too, so we ended up hitting the quota in no time. Another issue we found, is that if a Jenkins pipeline is aborted forcefully, the cypress tests in the Cypress Dashboard service would hang forever. We had to log in to the dashboard to check for any runninng tests, and then kill them.

Whatsmore, we did not even care about the test recording that Cypress dashboard offers. I mean, it is nice to see the video (which we disabled to make tests faster) and screenshots to see where a test failed, but we found that by just looking at the CI pipeline console log, it would tell us where the issue was. All we needed to do was to run that failing test locally, to find out more information in order to fix it.

The Cypress Dashboard might be nice for a QA or someone with no coding experience to look at and monitor where things are going wrong. But this was not needed for our use case. All we wanted the service for was to allow us to run our tests in parallel (to make them faster). Unfortunately Cypress did not offer this for free, so we gave up and paid.

# But what if we try passing a directory to test?

The way to run Cypress tests in parallel with Cypress dashboard is to add the API key to your cypress run command (with the --parallel argument) and then the test id to the cypress.json file. Then you start the cypress run command on a bunch of instances/computers, and their backend does the rest. Their service will load balance the tests, and each machine will run some of tests. Their API talks to the cypress/included container and tells which spec file to run next, and to run only a subset of the total spec files. If a machine finishes a test, their service will give hand over a new test to that machine. This means they all work together and no machine is ever left idle.

We tried doing this with 6 Jenkins agents with 2 cypress/included containers per agent. This reduced our test times to about 10 minutes on the build pipeline (cypress/included container testing against our components app docker container. Less network latency as they are in the same docker network), and about 15 minutes in the deployment pipeline (post-deployment).

The idea came when I noticed that the cypress run command can take a --spec command line attribute. This meant I could do things like this: cypress run --spec ./e2e/tests/group0/*.spec.js. Then I thought, why not grab all the spec.js files with all the Cypress tests, and separate them into different directories where they could be grouped together? This is what I came up with:

├── group0
│   ├── AtolText.spec.js
│   ├── HolidaysGadgetExtra.spec.js
│   ...
├── group1
│   ├── CmaInfo.spec.js
│   ├── HolidaysGadgetComplex.spec.js
│   ...
├── group2
│   ├── Blog.spec.js
│   ├── Breadcrumbs.spec.js
│   ...
├── group3
│   ├── AccordionContent.spec.js
│   ├── CookieNotice.spec.js
│   ...
├── group4
│   ├── CarHireGadget.spec.js
│   ├── Column.spec.js
│   ...
└── group5
    ├── Carousel.spec.js
    ├── Header.spec.js
    ...

Then using the Jenkins CI, I used this if statement (in startCypress.sh) to tell the cypress run command to go and look in a directory to test those files.

echo "About to test environment '$STAGE' on '$HOST'"
if [ -z "$PIPELINE" ]; then
    # When running locally we do not need to parallelise.
    cypress run --env host=$HOST
else
    # Only pipelines can run on parallel.
    cypress run --env host=$HOST --spec "tests/e2e/specs/${DIRECTORY}/*.spec.js"
fi

One of the disadvantages was that there was no automatic load balancing, but I tried grouping the tests in a way that they would last around the same time. After a few tweaks, the groups were running all in between 6 and 8 minutes in the feature and build pipelines, and in about 10 to 15 minutes in the post-deployment test runs. Some groups would finish before others, but the difference between them was just only about 1 minute from each other. (Still better than 50 minutes, and for free)

The way we made Jenkins run the Cypress tests in parallel in our pipelines is similar to this. This is an extract of the deployment pipeline, but the feature and build pipelines look very similar. (Notice where we call the startCypress.sh, we set the DIRECTORY environment variable for the group of tests to test.)

stage('Automation tests') {
    failFast true
    parallel {
        stage('Cypress 0') {
            agent { label "agent-0" }
            steps {
                unstash "node_modules"
                ansiColor('xterm') {
                    sh "docker run --rm --ipc=host --name cypress.tsm.components.deploy.local.0 --shm-size=1g --mount src=${WORKSPACE},target=/e2e/,type=bind -e DIRECTORY=group0 -e PIPELINE=deploy -e STAGE=${STAGE} --entrypoint ./e2e/bin/startCypress cypress/included:4.0.2"
                }
            }
        }
        stage('Cypress 1') {
            agent { label "agent-0" }
            steps {
                unstash "node_modules"
                ansiColor('xterm') {
                    sh "docker run --rm --ipc=host --name cypress.tsm.components.deploy.local.1 --shm-size=1g --mount src=${WORKSPACE},target=/e2e/,type=bind -e DIRECTORY=group1 -e PIPELINE=deploy -e STAGE=${STAGE} --entrypoint ./e2e/bin/startCypress cypress/included:4.0.2"
                }
            }
        }
        stage('Cypress 2') {
            agent { label "agent-1" }
            steps {
                unstash "node_modules"
                ansiColor('xterm') {
                    sh "docker run --rm --ipc=host --name cypress.tsm.components.deploy.local.2 --shm-size=1g --mount src=${WORKSPACE},target=/e2e/,type=bind -e DIRECTORY=group2 -e PIPELINE=deploy -e STAGE=${STAGE} --entrypoint ./e2e/bin/startCypress cypress/included:4.0.2"
                }
            }
        }
        stage('Cypress 3') {
            agent { label "agent-1" }
            steps {
                unstash "node_modules"
                ansiColor('xterm') {
                    sh "docker run --rm --ipc=host --name cypress.tsm.components.deploy.local.3 --shm-size=1g --mount src=${WORKSPACE},target=/e2e/,type=bind -e DIRECTORY=group3 -e PIPELINE=deploy -e STAGE=${STAGE} --entrypoint ./e2e/bin/startCypress cypress/included:4.0.2"
                }
            }
        }
        stage('Cypress 4') {
            agent { label "agent-2" }
            steps {
                unstash "node_modules"
                ansiColor('xterm') {
                    sh "docker run --rm --ipc=host --name cypress.tsm.components.deploy.local.4 --shm-size=1g --mount src=${WORKSPACE},target=/e2e/,type=bind -e DIRECTORY=group4 -e PIPELINE=deploy -e STAGE=${STAGE} --entrypoint ./e2e/bin/startCypress cypress/included:4.0.2"
                }
            }
        }
        stage('Cypress 5') {
            agent { label "agent-2" }
            steps {
                unstash "node_modules"
                ansiColor('xterm') {
                    sh "docker run --rm --ipc=host --name cypress.tsm.components.deploy.local.5 --shm-size=1g --mount src=${WORKSPACE},target=/e2e/,type=bind -e DIRECTORY=group5 -e PIPELINE=deploy -e STAGE=${STAGE} --entrypoint ./e2e/bin/startCypress cypress/included:4.0.2"
                }
            }
        }
    }
}

I have stripped away the rest of the steps and only left the relevant Cypress parallel steps for clarity.

There possibly is a neater way to generate all these very similar looking stages (remove repetition) in the Jenkins CI/CD pipeline file (with a loop), but I haven't been successful at making this work yet.

You can see these parallelised steps in more detail in the following screenshot:

# Wrap up

Our Cypress tests now run in parallel for free and they are faster than running with Cypress dashboard with their parallel solution. This is because we don't have the back and forth between the Cypress dashboard API service (doing all the load balancing of the .spec.js files between the different machines, and uploading video or screenshots).

If you don't care about the Cypress dashboard recording your tests (with videos and screenshots) and you are after making them faster with parallelisation for free, then this is a good alternative you could try.