Learn How To Design A Resilient Production Environment

Resilient Production

Resilient Production
To gain the full benefits of Docker, applications need to be moved from development and test environments to production environments. Concerns related to security and lack of expertise in container management in production environments are some of the challenges faced when migrating Docker applications to a production environment. Applications that are natively developed using Docker are easier to migrate than applications that are not natively Docker.

Some of the operational aspects that are affected by use of containers are listed below.

  • Backing up procedures
  • Rolling out a change from a development environment to a production environments
  • Debugging and reporting of issues that arise in a production environments
  • Monitoring of deployed applications
  • Managing the different application versions in a production environments
  • Recovering and continuing with operations after a disaster
  • Planning for required resources in a deployed environments

To create an environment, that is resilient to unforeseen events and is easily supportable a well thought out approach needs to be used. The following section will discuss practices that can be put in place to avoid pitfalls that destabilize a production environment.

Only one code base should be maintained
All instances of running applications should originate from one code repository. In the production environment, a Docker image ought to be built from a single repository, which holds all the code required for the container functionality. This approach simplifies code rebuild and ensures all the required components are in the repository. The key point here is that to build a fully functional application the different pieces of code functionality come from one repository as opposed to different repositories. This does not restrict the use of dependencies or artifacts from other repositories, however clear documentation needs to be put in place to show shipped code. Docker is not strong at simplifying dependency management when applications are being built from multiple repositories and its repeatability is poor.

All the required dependencies need to be clearly documented
The assumption that the required application dependencies will be provided by different means, like an install, needs to be avoided. Application dependencies should be included in the code and the build process relied on to include the dependencies. By using this approach the application is expected to run properly without relying on external processes to provide dependencies. Including all dependencies is important especially when using containers because a container is isolated from the host operating system. The container process can only access the resources that are within its filesystem.

In the Dockerfile and any configuration files, for example package.json, all internal dependencies the application needs to run should be defined. This will guarantee an application will run correctly in different environments. By defining all dependencies, there are advantages of a reliable, repeatable and secure deployment process.

Avoid hard coding configuration settings into the code, instead place them in environment variables
Using environment variables simplifies the process of deploying the same code to different target environments such as development and production without tweaking any container aspects. By using environment variables, we have cleaner code because information such as that required to connect to databases which is likely to change is placed in environment variables.

Design a backup process that is able to handle lack of access to required resources
Services that are provided by third parties as well as databases are prone to outages. Therefore, applications need to be designed in such a way that they can handle resource outages and continue operating as expected in the event of a resource outage. Although, Docker does not directly support handling resource outages it is very important that robust services are put in place. Carefully handling dependencies is important because containers can only access resources within their file systems so disk availability cannot be relied on. Ensuring high availability of containers differs from high availability of virtual machines. For high availability, containers rely on scaling horizontally and rolling deployment while virtual machines rely on processes being live migrated.

The build and run steps need to be managed separately
The three important steps are building the code, including required configurations when releasing the code and finally deploying code. By having distinct steps, it is possible to work on an individual step without affecting the entire work flow. Feedback from previous steps is quickly available to help improve the next step. Docker supports separation of build and run steps by having a clear way of moving an image from build to production.

Applications need to be executed as stateless processes
To avoid loss of data when applications are deployed, shared data should be implemented using a stateful store. Keeping critical state in memory process or in the container needs to be avoided. It is important to have in mind applications running in containers are short lived. The ability to create and destroy containers quickly provides the flexibility to quickly recover from an outage and rapidly deploy applications. To minimize the impact of stopping containers it is important to develop applications that only need to maintain state for the time period they will processing requests. To maintain state, a remotely accessed store such PostgreSQL or Amazon S3 can be used.

Access to services needs to be through ports
Addressing of applications needs to be through a port that is specific to the application. Direct port binding instead of an external daemon should be used. This guarantees when talking to the port you are communicating with the application.

When scaling out rely on the process model
When designing applications it is important to factor in how they will scale and operate concurrently. When an instance resource is increased it is not simple to scale down the resources. Addition and removal of instances as resource needs change is a more flexible way of infrastructure management. When horizontal scaling is used by starting a container on a separate server requires least effort.

In this post, we noted the importance of being aware of challenges that are likely to arise when moving from a development to production environment. We discussed the different practices that can be put in place to mitigate challenges in a production environment.


Please enter your comment!
Please enter your name here