Purpose

This memo outlines the commitment to improving automated testing across Dispel in Q1 and Q2. The goal is to increase confidence in our releases, improve developer efficiency, and ensure critical customer flows remain functional as we scale.

Background

We’ve been moving in this direction for 2–3 years, and I want to acknowledge the progress we’ve made. That said, our automated testing still isn’t where it needs to be, and it’s limiting us. We’ve done an admirable job adding tests while delivering features; in Q1 and Q2 we’ll supplement that with dedicated projects to strengthen our testing foundation.

The point of testing is to increase confidence that changes have not introduced new issues and that critical customer flows remain functional. For example, a common test is making sure customers can still log in — a critical piece of functionality that requires our website, our API, our database, and our caching layer to all be functional.

Key Benefits

• Safer Releases: Our customers use us for critical functionality, meaning that we have a higher bar to clear than many other software products when it comes to having releases which do no harm. Having good testing is a tried and true way to achieve this.
• Developer efficiency & Software Performance: Having automated testing helps us move faster. Our current MTTM (Mean Time to Merge, meaning the time it takes from a PR being ready for review to it actually being merged) is 3-4 days on average. If we had confidence the logic didn’t break anything, then this should come down to 1-2 days.

Key Points

Imagine a scenario where we had just two tests for the API endpoint we use when figuring out which facilities a user should be able to see in our UI’s.

1. TEST 1 = An ADMIN sees facilities 1, 2, and 3
2. TEST 2 = A USER is a member of facility 1, therefore they can see it. They do not see facility 2 and 3

It is discovered that the endpoint has major scaling issues when querying for 500+ facilities. Having those two tests makes it more straightforward to improve the endpoint’s performance while still being confident that the logic is intact. And there is nothing stopping us from adding more tests for this endpoint, nailing down every permutation of the logic we expect it to have, and adding a performance test to ensure we aren’t accidentally making it slower. In summary, the tests for this endpoint could very well help us find issues ahead of time, find performance issues ahead of time, prevent new issues from being introduced and help us resolve and guard for issues which escape us more quickly.

Having these types of tests means we can improve our scaling to match what ABB and other future customers expect from us.

Here is a list representing the direction we are committing to:

• Write and Run

  • Everyone¹ should understand our test suites and be comfortable writing test for them as well as running them.
  • Running the tests should feel fun. It should be a natural step of development to run tests because they are fast and don’t cause your dev machine to be unusable while they are running.
  • Engineers are expected to run and validate new features in their dev environment, as well as a regression that it doesn’t effect other parts of the application.
  • Writing and maintaining tests should be a shared responsibility in engineering. It should not be QA’s job to maintain an entire suite of tests that must pass before a new feature can be released. We all benefit from having confidence that things are working properly on an ongoing basis.
  • We will transition to engineers being responsible for our E2E tests.
  • Bugs and flaky tests are the responsibility of the engineer to fix.

• Test Types and Type Checking

  • We should take better advantage of different types of tests to increase our confidence in code, correctness, performance (some tests should fail if they are too slow!), and user experience.
  • Accelerate TypeScript adoption for CAPI and CWEB, and add more Python Type Annotations and a type checker to Orchestrator.

• Environments

  • We should have environments that can test for performance.
  • We should have environments that can test for a particular scale of customer before releasing code to that customer.

Next Steps

Let’s enable our scaling efforts, improve our release confidence, and have a great year.

Signature:

This memo was signed via git commit. Verify authenticity using git log --show-signature