Continuous Deployment – How To Use It Optimally

May 13 2019

Continuous deployment (CD) is the practice of minimizing the time between a code change and production deployment via automation.

One may ask: Is it a good practice?

And the short answer is: When done right, it is very powerful. When done wrong, it can be catastrophic.

 

So how can one tell if a team can or should practice CD? In this post, I'll try to answer this question.


Motivation

First off, why even bother with implementing CD?

Small versions = small risks.

  • Small changes are usually safe. They affect only a smaller fraction of the system.

  • In case something goes wrong, minor changes are easy to rollback.

Better money/value ratio.

  • Automation – This allows a team to spend more time on productive tasks instead of mind-numbing manual processes.

  • Fast user feedback – With this, a team spends less time on things that users don't want.

Continuous Integration & Delivery

Continuous integration and delivery are the building blocks of CD.

A team must master both before attempting CD.

Continuous integration means that code changes are built, tested and merged into the codebase HEAD (i.e. Git master branch) immediately.

Continuous delivery means that software versions are built, tested & packed immediately. This process ensures that there is always a stable version ready for deployment.

In other words:

  • Continuous integration – Ensures that we have a stable version from the developers POV.

  • Continuous delivery – Ensures that we have a stable version from the QA POV.

Merging the two creates a process in which every code change is packed into a new stable version.

A team that uses these practices is in pretty good shape. We should strive for them whether we are interested in CD or not.

There are a number of key factors to take into account in order to build software that can automate both integration and delivery.

But Beware: Stay out of Integration Hell

“Integration hell” is a state where the same code behaves differently in different environments.

For example, an app that is built on one developer’s local machine can easily fail when it runs on other machines.

 

Such pitfalls can be avoided by making sure that:

  1. The whole development team works within the same environment.

  2. The staging environment mirrors the production one.

Architecture, Control & Test Coverage

Obviously, empty test suites always pass, and code that has never been validated works. Therefore:

  1. Code should be easy to test, meaning that it is modular (i.e. divided into standalone modules/libs/micro-services).

  2. Every code change must have sufficient code coverage.

It should be easy to control the software execution during both deployment and run time.

  1. Code should be controlled by flags (on/off switches while under development with environment variables or configuration files).

  2. Be aware of run time side effects (i.e. what would happen if we rollback DB scheme changes).

A code change that does not meet the above standards will suffer from regression issues.

 

Pipeline Simplicity

Continuous integration & delivery entails creating some kind of a pipeline implementation, for example:

  1. Merge code change with HEAD.

  2. Build a version.

  3. Run test suites.

  4. Deploy it to a staging environment.

  5. Run e2e/integration tests.

  6. Run stress/load tests.

This pipeline allows us to understand whether a version is stable or not.

Pipelines should be easy to use, and every developer must be able to run the pipeline locally (at least the lion's share of its steps. Stress testing locally, however, doesn't contribute much).

Pipelines should give quick feedback.

A developer should know whether the change is legit, or is at least on the right track, within a matter of a few minutes.

The pipeline is sacred ground

It must always be consistent & robust.

A problem with the pipeline must be fixed with the highest priority.

Agility

As mentioned above, CD is about fast pacing.

If agility is not part of a team’s DNA, it can never hope to succeed with CD.

Moreover, a non-agile team that tries to implement CD will most likely get bogged down.

In this case, using CD would cause a lot of unnecessary noise in the form of bugs and friction among team members.

A well-oiled team should be small, independent, responsive, with full e2e responsibility & ownership over its work.

 

An efficient team should also allow and encourage free communication, be open to criticism, and must be able to translate this feedback into concrete steps toward self-improvement.

If a developer must wait days for QA & PM approval for every change, then his or her team is not agile.

If a developer makes a mistake and explains it by saying "We always did it that way," then the team is not agile.


Moving from Continuous Integration & Delivery to Continuous Deployment

Once we have automated a continuous integration & delivery pipeline, it’s a short step to CD.

We just need to expand the pipeline from deployment to production.

 

But before doing so, here is one final consideration: Lo and behold, there is no such thing as bug-free software.

With manually deployed versions of software, a team watches the development process more closely, which means that failures or problems are often detected early on. With CD, we can deploy versions dozens of times each day, all without any human oversight.

If you use CD, at some point, it will push a bad version to production.

Monitoring

 

The team must be able to monitor software performance in real-time.

If or when things go awry, problems should be detected and fixed ASAP.

Because CD allows every developer to deploy to production, every developer must know how to monitor the software.

A team must have on-call troubleshooters that can react to failures 24/7.

User Perception

The greatest advantage of CD is lighting-quick feedback.

When UX is bad or faulty, and users report these problems, the latter will not be very pleased with our new feature. Therefore:

  1. There must be a simple way for users to offer feedback.

  2. The team should heed and process feedback ASAP.

But do users appreciate this QA role we have thrust upon them?

"Users as QA" is a luxury enterprise that only behemoths like Facebook or WhatsApp can afford. They have millions of users, and can make sure changes affect only a subset of them before reaching the whole world (via A/B testing, Green/Blue, canary versions, for example).

Worst-case scenario: Some of their captive users are annoyed for while. It’s not ideal, but it’s a  calculated risk.

However, if I have only 100 users and there are five market competitors, then relying on the good patience of users is a massive and unnecessary risk. In this case, waiting until the product is more mature is a better tack.

 


Summary

CD is not easy to achieve.

While there are many programs and services that can assist, CD is not a tool that a team can setup, fire and just forget about.

It is an automation option that encapsulates the team ethic and culture. Therefore, when using it, the team must remain highly responsive to problems while being fluid in their communications.  

Additional readings

Guy Y.
Software Developer
Back to Blog