This presentation was created in Spectacle
A ReactJS-based Presentation Library.
If you've worked with Angular, you have probably seen Protractor as the end-to-end (E2E) testing solution provided when you ng new a project. If you have ever worked with Protractor or other Selenium based solutions, you have likely been frustrated with the experience.
Our team chose to go towards greener, more JS based pastures by opting to use Cypress instead of Protractor. Was the grass greener on the other side? We sure think so! Here's been our experience.
Note: E2E testing can sometimes be referred to as UI testing - because the UI is the entry point at one end. I may use these terms interchangeably but will make a distinction between our use case later in this article
Cypress is "fast, easy and reliable testing for anything that runs in a browser". It is batteries included with baked in features and perks. Cypress has a strong user community and great documentation.
It also has limitations. Most notably, Cypress currently only supports Chrome. It also comes with inherent concerns that all shiny and new open source projects have—what if the next hot thing comes out and maintainers jump ship? Not likely, and the same thing could happen with Protractor as we have seen with tslint
Cypress has a super generous free tier. You can npm install cypress and get started writing tests in no time. Then hop into the test runner and go back in time to see what happened at each point in your tests.
Free tier also provides the capability to take screenshots, videos, and output reports. Not to mention all the baked in test development tools like cypress commands, traffic control, aliasing, and dropping into jQuery attributes or Mocha assertions as needed.
The benefits of the paid tier of Cypress are pretty straightforward: a dope, magical dashboard with a history of your test runs, including those video and screenshots mentioned above. We figured out how to extract these into our pipeline.
End-to-end testing has a fraught history of being flaky and unreliable. Software always has a chance of misfiring and crashing. Additionally, running software in a browser with HTTP requests increases the odds that something will go wrong.
Sometimes requests don't come through. Sometimes it takes 0.3 seconds for that element to appear on the DOM. Sometimes it never appears. As test suites grow, the likelihood of all tests passing in a single run diminishes. We can try to wrangle instability in Selenium-based solutions with waits, but then sometimes things never return. Then we have to do error handing and Ack! I'm going home
Cypress mitigates a lot of these headache points with:
- Auto retry capability
- Sweet debugging capabilities
- Traffic control for flaky API calls
- Wait helpers for DOM elements and XHR requests
Many development teams opt not to automate end-to-end tests. Besides the flakiness discussed above, it's another test suite the team needs to manage. On our project, we were already using unit tests in Jest on the frontend, unit tests in MSTest on the back end, and SpecFlow to acceptance test our API.
So what is the value of adding E2E tests? Just like any testing, it can help inform the development process and can help catch defects quickly after development. We've also found doing E2E tests has tightened the communication loop between development and QA.
Possibly the most important factor is that your E2E test suite serves as a regression test suite. As you add more and more features, it's unreasonable to expect QA to manually regression test old features to make sure things haven't changed. Their efforts can stay focused on creating test cases for new features and exploratory testing.
Cypress accesses your application through the browser, therefore it is most suitable for end-to-end acceptance tests or UI tests. If you have interest, Cypress can be used to unit test your components in Angular with some home-brewing. Considering the UI access point, Cypress is not appropriate for API testing.
On our team, challenges with TestBed were a strong motivation to try Cypress. TestBed is the way in Angular to set up the modules, providers, and child components necessary to shallow mount a component. It is necessary to do any UI testing on a unit test level.
We have opted not to do this 🧐
describe('Shallow Mount', () => {
beforeEach(async(() => {
TestBed.configureTestingModule({
declarations: [DiscardContractAddComponent],
schemas: [NO_ERRORS_SCHEMA],
imports: [MatDialogModule, RouterTestingModule],
providers: [AuthTokenService, MatDialog]
}).compileComponents();
}));
beforeEach(() => {
authTokenService = TestBed.get(AuthTokenService);
authTokenService.isAuthorized = jest.fn().mockReturnValue(true);
matDialog = TestBed.get(MatDialog);
matDialog.open = jest.fn();
fixture = TestBed.createComponent(DiscardContractAddComponent);
component = fixture.componentInstance;
});
it('can do fixture/template tests', () => {
const addButton = fixture.debugElement.nativeElement.querySelector(
'.add-discard-contract'
);
addButton.click();
expect(matDialog.open).toHaveBeenCalled();
});
});The configureTestingModule method proved to be an ongoing challenge for us. Any additional material component or service injected meant additional imports or providers required. Also, extracting the services from the testbed to assign them as variables to spy on was a challenging paradigm to get into the groove of.
We stopped UI testing in Jest and stuck to controller unit testing.
We unit test typescript functionality only in Jest 😎
describe('Controller', () => {
beforeEach(() => {
matDialog = new MatDialog(null, null);
matDialog.open = jest.fn();
const router: Router = jest.genMockFromModule('@angular/router');
authTokenService = new AuthTokenService(router);
authTokenService.isAuthorized = jest.fn();
component = new DiscardContractAddComponent(matDialog, authTokenService);
});
it('can trigger and test things on the typescript level', () => {
component.addClicked();
expect(component).toBeTruthy();
});
});Focusing on controller logic has simplified our Jest tests. We offload UI testing to E2E tests in Cypress.
We use Cypress to test our UI 🥳
describe('Cypress', () => {
beforeEach(() => {
cy.login();
cy.visit('http://localhost:4200/discard');
});
it('tests things on the UI', () => {
cy.get('.add-discard-contract').click();
cy.get('mat-dialog-container').should('exist');
cy.get('.market-field input').type('ADM');
cy.contains('mat-option', 'ADM Altamont').click();
.
.
.
cy.get('.submit-button').click();
cy.get('mat-dialog-container').should('not.exist');
});
});We've found that writing E2E tests has bolstered communication on our team, particularly tightening our dev-to-QA feedback loop.
At the beginning of each story, developers and testing come together to whiteboard a story around acceptance criteria. If needed, developers will have a separate technical implementation whiteboarding session.
When the devs are close to wrapping up feature work they meet again with QA to discuss what was implemented. At this point, test cases have been written, are refined, and the developers and QA can decide which test cases should be automated via Cypress (or SpecFlow if the test is API/DB related).
Even if test cases are automated, QA still rigorously manually tests around them. They also check the effectiveness of the Cypress tests against their manual testing for a given case. Then when new futures are produced, Cypress can act as our regression test suite and QA can focus on smoke and exploratory testing, rather than redoing tedious manual regression testing.
In this section, I will refer to a build or build artifact as the code that is going to be released to an environment. Another term for releasing is deploying. The deploy process involves releasing a build to an environment. Both a build and a release have a possibility of failing. We have five distinct environments we deploy our builds to: DEV, TEST, AUTO, STAGE, and PROD. FIX ME In short, we run Cypress tests in our deployed AUTO environment. AUTO is required to be passing (green) in order to go to stage.
As part of our build process (before deploy/release), we run integration tests. These include unit tests in Jest on the frontend, unit tests in MSTest on the back end, and SpecFlow to acceptance test our API. If these fail, the build does not release to any of our environments.
After building, we deploy to DEV and AUTO environments concurrently. If the DEV release passes, the build also goes to TEST. These releases stand up resources and deploy our new code.
The AUTO release does the same as DEV and AUTO with an additional step. Once the new code has been deployed to the environment, we trigger an additional build that runs all of our Cypress E2E tests against the AUTO environment. If the tests fail, the release fails (red) and rolls back to the previous build. If the tests pass, the release passes (green).
For tests that fail, we capture screen-shots. Regardless of tests passing or failing, videos are captured of each test. Both of these are saved to the build artifact that runs the Cypress tests.
The initial setup in Azure was done by a teammate and we received further help from our DevOps colleague to save the screen-shots and videos as a build artifact.
Given our CI/CD and QA workflows, we needed to figure out how to integrate feature flags into our Cypress testing flow. What we ended up agreeing on as a dev team is:
- We start with feature flag on in DEV/Local for initial feature development
- We submit a PR to merge the feature with the flag off in AUTO
- All Cypress tests should pass in AUTO without the new feature
- We update Cypress tests locally with the feature flag on
- When the Cypress PR merges we turn the feature flag on in AUTO
- The Cypress tests pass in AUTO with the new feature 🎉
Between steps 3-5 we have our testing discussions with QA and QA can begin testing the new feature by toggling the feature flag separately in our TEST environment.
Cypress has brought a lot to our team and our process. It has brought our team closer in communication around feature expectations. It has provided us with a tool to catch defects as they are being generated. Possibly most importantly, it has created a regression testing suite that helps us feel assured that we have not negatively affected any previous features when implementing new ones.
Cypress also offers a breadth of helpful features allowing developers to dive into E2E testing without having to deal with the minutia of creating a Page Object Model and writing complex waiting helpers. Lastly, the value of the interactive test runner, screen-shot and video output makes this tool highly valuable to non-developer members of our team.
I have written another blog (see below) with more information about technical nuances of Cypress.
I'll be honest, working with Azure AD, EasyAuth, and ADAL has been challenging. However, integrating auth with any end-to-end testing solution poses common challenges. We chose to follow an auth flow as close to our PROD auth flow as possible, rather than building a backdoor.
To do this in Cypress, we needed to catch the call to Single Sign-On (SSO) attempting to make an auth request. This is easier said than done and there are many forums about achieving this SSO flow. Our solution is provided here with confidential information redacted as UPERCASE_UNDERCORE_VARIABLES:
function login(secret: string, id: string) {
cy.request({
method: 'POST',
url: SSO_TOKEN,
form: true,
body: {
grant_type: 'client_credentials',
client_id: id,
client_secret: secret,
resource: RESOURCE_ID
}
}).then(response => {
const adalToken = response.body.access_token;
window.localStorage.setItem(
'adal.access.token.keyede' + RESOURCE_ID,
adalToken
);
window.localStorage.setItem('adal.idtoken', adalToken);
});
}
export function loginWriteAll() {
login(WRITE_ALL_SECRET, WRITE_ALL_ID);
}
export function loginReadAll() {
login(READ_ALL_SECRET, READ_ALL_ID);
}
export function loginReporting() {
login(REPORTING_SECRET, REPORTING_ID);
}What you see above is three cypress commands that can be used to login our Cypress test at each authorization level that we have in our application. We do this before each test:
describe('writeAll role', () => {
beforeEach(() => {
cy.loginWriteAll();
cy.visit(guiUrl);
});
... tests ...
});
describe('readAll role', () => {
beforeEach(() => {
cy.loginReadAll();
cy.visit(guiUrl);
});
... tests ...
});
describe('reporting role', () => {
beforeEach(() => {
cy.loginReporting();
cy.visit(guiMbUrl);
});
... tests ...
});For more information on setting up this flow, see this write up from our teammate.
E2E testing means testing your entire application from the browser through your API to your database. UI testing focuses on specific feature implementation on the UI. We decided to take advantage of traffic control in Cypress to test UI separately from our E2E processes. This is how we setup the folder structure for cypress:
▾ cypress/
▸ fixtures/
▾ integration/
▾ e2e/
- cbot-price-table.e2e.ts
- market-..is-table.e2e.ts
- market-..is-table.e2e.ts
- market-..is-table.e2e.ts
- payment..is-table.e2e.ts
- price-quote-table.e2e.ts
▾ ui/
- cbot-price-table.ui.ts
- market-basis-table.ui.ts
- market-..sis-table.ui.ts
- market-..sis-table.ui.ts
- payment..sis-table.ui.ts
- price-quote-table.ui.ts
- side-nav.ts
▸ plugins/
▾ support/
- commands.ts
- index.js
- README.md
- cypress.json
- tsconfig.json
Our E2E tests were written as expected with requests hitting our API. This required us to create a seed project to setup the data in our AUTO database to be accurate to our Prod environment. This is easier said than done and each team should figure out which database seeding process works best for them sooner than later (perhaps speaking from experience 😅)
In our UI tests, we created lists of mock data. We then caught API requests using cy.server() and cy.route() (thank you, Cypress!) and returned the mocked data instead of letting the API requests go through.
The example below shows how we checked our UI messages stating "No ___ POMBs Available". POMBs is a type of transactional business model we use. There can either be "Pending" or "Active"—sometimes none of one type or the other. You'll see we hardcoded the messages we expected to see (or not see), mocking the data (data redacted), and how we caught our API calls and returned mocked data. Finally, we assert that we see the "No Active POMBs" on the active tab when no active POMBs are returned form our API.
beforeEach(() => {
cy.loginWriteAll();
cy.server({});
});
const pendingPombTitle = 'No Pending Payment Option Market Basis Results';
const pendingPombMessage = 'All POMB records for this basis date are Active.';
const activePombTitle = 'No Active Payment Option Market Basis Results';
const activePombMessage = 'All POMB records for this basis date are Pending.';
const mockedPendingPombs = [
{ 'POMB object 1' }, ..., { 'POMB object N' }
];
const mockActivePombs = [
{ 'POMB object 1' }, ..., { 'POMB object N' }
];
it('should have a "no Active POMBS" message if there are no Active POMBs but some Pending POMBs', () => {
cy.route({
method: 'Get',
url: apiPombUrl,
response: [
{
paymentOptionMarketBasisListViewModels: [],
status: 'Active'
},
{
paymentOptionMarketBasisListViewModels: mockedPendingPombs,
status: 'Pending'
}
]
});
cy.visitAndWaitXhr(guiPombUrl);
cy.checkNoDataMessageHidden(pendingPombTitle, pendingPombMessage);
cy.checkNoDataMessageHidden(activePombTitle, activePombMessage);
cy.contains('.mat-tab-label', 'Active').click();
cy.checkNoDataMessageHidden(pendingPombTitle, pendingPombMessage);
cy.checkNoDataMessage(activePombTitle, activePombMessage);
});