Chances are, if you’re not a programmer, you have never heard about canary testing. That’s okay, however you’re probably wondering what it is, and why is it called canary testing?
The term derives from the “canary in the coal mine” phrase. You see, coal miners used to bring a caged canary into the coal mine with them, as canaries have a lower tolerance for toxic gas.
Therefore, workers knew that if the canary died, they should get out of the mine. In terms of software development, when creating new software, the goal is to detect and resolve issues as quickly as possible, before they become a problem for users.
What Is Canary Testing?
Canary testing is the method of trying out new functionalities and features with hopefully minimal negative impact on the user’s end. This type of testing is normally used with a small group of users, for which the application is presented with small changes. This way, the majority of users will still be using the current version of the software, while the selected, smaller group of users will be using a slightly upgraded version.
The purpose of this exercise is to allow the majority of people using the software to continue working with it, without any issues, while pushing changes for only a small percentage of users.
This is all done while keeping both the existing and the new version of the application running simultaneously. Then, if the canary testing doesn’t return any errors, you can continue to expand it to larger percentages of users.
Readers Also Enjoy: What Is Rapid Application Development? 4 Phases of RAD Methodology – DevriX
How to Perform a Canary Test?
The first step in a canary test is to launch a set of back-end servers or containers to run the new code. Once there are new users, your load balancer will send a percentage of them to the “canary cluster”.
The DevOps specialists then monitor the servers to identify potential issues, like the compute load or higher rates of I/O. And, keep in mind that, even if serious issues occur, it’s easy to revert things back to as they were, since the process does not affect everyone using the software.
Otherwise, you can easily implement and automate canary testing by using tools like Spinnaker to designate the amount of users using the new code.
Said percentage is typically around 5%, and once the assigned DevOps team establishes there are no issues, they can steadily increase that percentage, until everyone is using the new code version of the application.
What’s more, canary testing is beneficial in situations where development or staging environments do not fully match the production environment. Using a small test group of users can reveal any issues that were not detected during the staging or development environments.
Readers Also Enjoy: A/B Testing with WordPress: The Definitive Guide – DevriX
Benefits and Drawbacks of Canary Testing
Let’s define the good and the bad when it comes to testing, using the canary method.
Benefits of Canary Testing
- Real-world testing. Testing in closed, internal environments is great, however, being able to test software on real users is what takes you one step further. With canary deployments, you can do exactly that – test your application on a small-scale audience, without the risk of releasing a completely new application.
- Better risk management. Gradually releasing new features and making sure they work properly, greatly reduces any significant errors and the total cost of any failures.
By using the canary testing method, you basically, eliminate the need to roll back major updates or rewrite large amounts of code.
- More business-oriented. Need to present business data to managers and marketing/sales people? With canary, you can very quickly view results. For example, if you want to test a new software feature – you can deploy a new feature for a small group of users, and then wait to see if the feature goes down well or not You can then use this data to drive up engagement for your product.
Drawbacks of Canary Testing
- Software issues. There is still untested code, so, naturally, there are risks involved. In case even a few users are experiencing significant problems with a specific feature, it’s best to skip canary testing in favor of more strict internal testing.
- Larger expenses. If you intend on using a load balancer to partition users, you will need additional infrastructure and more administration. This means you will have to create two separate production environments and backends to execute the test. So, two app servers, two code lines, two web servers, and a networking infrastructure to maintain.
- Bad user experience. Some users might not be too keen on the idea of being used as test subjects. If you want to be open about what’s going on, you can let users know they’re being used as “canaries”, through an “early adoption” program or something similar.
When Not to Use Canary Testing?
Canary testing is beneficial in a lot of cases, however, it’s not a surefire answer to everything (that’s 42).
Here’s when you should definitely not use the canary approach:
- In environments that are not suitable for continuous deployment.
- When dealing with critical software like life-supporting systems or nuclear reactors.
- Financial systems where failure can result in a substantial economic impact.
When you cannot update software remotely.
How to Do a Canary Deployment?
Now, let’s take a look at the necessary steps you need to take, in order to perform a canary deployment.
- Prepare for deployment on the staging server.
- Exclude canary nodes through load balancing.
- Deploy the new version to canary nodes.
- Test the new version with automated test scripts.
- Connect canary nodes to traffic with load balancing.
- Roll out the update to the remaining production nodes.
In practice, the deployment begins with the preparation of the staging server. Check the configuration files, testing scripts, build artifacts, and so on. After that, you can either run automated or manual tests, or leave the server live and see if everything runs well, or if it crashes.
When to Push Canary to Production?
Once you’ve tested your canary update on 5 or 10% of users, and everything is working correctly, you can proceed to double or triple the percentage of test users.
Then, test again, gather feedback from metrics, and once you’ve assessed everything is working fine, once more, you can go on to update 70% of the production environment.
Finally, repeat the testing procedure, and if everything is good to go, roll out the new version of the software to 100% of users, and, of course, continue monitoring.
Companies That Have Implemented Canary Deployment
To show you that canary deployment is a popular method of implementation, we will take a look at some companies that use canary deployment.
The streaming service giant has been using automated canary analysis with Kayenta since 2018. This enables extremely fast updates to Netflix’s production environment. The company states that this deployment method has increased the productivity of developers while reducing the risk of making changes in their production environment.
This social media platform has also been a long-time fan of canary deployment. Their co-founder and CTO, Mike Krieger, shared that canary releases ensure that bugs don’t do too much damage and only affect a small group of people.
Did you know that Google Chrome has a special version, meant for developers? That’s right, Google Chrome Canary allows you to test the new web platform APIs and see what the future of the browser would look like before anyone else.
Microsoft uses its own staff to canary test all their product updates. The process has been in place since Windows Vista and Microsoft Azure. The Windows 10 upgrade was also done using canary updates of their staff’s operating systems.
Canary testing is a great way of introducing new application versions quickly, in real-world testing environments. Not only that, but it allows for better risk management and more trackable data.
Despite canary testing having so many benefits, it’s not always the solution to every situation. Make sure to do your research before you start a canary deployment, otherwise, it could backfire on you.
Marketing Assistance for Your Business
One of our core specialties as a part of our full-service suite of services is the Inbound Marketing package. If you need expert assistance in keyword generation and SEO, you should get in touch with us. We are a professional WordPress development company profiling in business growth for your next big challenge.