I found an old writeup I had done for the deployment process I helped setup at a large health organization a few years ago. It worked really well for deploying an architecture consisting of apache/tomcat based multi-instance applications which integrated with multiple back-ends and provided a portal style interface to the end user. We were able to know exactly what version of what software was deployed in which environment and tomcat instance, when it was deployed and by whom and track it back to the source that was used to build it. We could also deploy at will anytime of the day due to the stateless architecture of the applications. All automated via Puppet (except for staging/production which was not fully baked when I moved on from the consulting gig).
Core Deploy specifies the model and process whereby a software artifact from a binary repository built by software developers is propagated through various environments in an automated fashion until it runs in production to meet the user requirements. Some of the benefits of this controlled process include:
Creating packaged build artifacts which can be deployed in various environments in an automated and repeatable fashion.
Creating a 'pipeline' for software development which includes steps such as quality assurance testing, performance testing, etc. The subsequent “Release Rhythm” section touches on this in more detail.
Providing some level of auditing to track software artifacts as they move between environments.
Initially, the software development organization has to agree on the environments that the artifacts progress through. This may consist of some of the following in order of progression:
Shared Dev – enables developers to test their built artifact to ensure that it will deploy correctly and integrate with any back-end services which they may have mocked out in their local development.
Shared QA – provides an environment for the Quality Assurance department to validate the software meets the functional requirements.
Performance – an environment to ensure that the software meets performance requirements.
Stage – basically production but which is not accessible to end users. Provides the ability to do final production validation of software artifacts before they run in production. This environment can also enable stateless automated deployments to production when paired with a load balancer.
Production – end users use the software from this environment.
Once the environments have been defined and the order of the progression of the built artifacts defined between them, the organization has to select tools to support the automated deployment of the artifacts between the environments.
Initial deployments will most likely be manual to help flesh out the deployment process. The manual steps can then be automated to provide as close to a push button experience as possible.
The deployment process specifies guidelines for who requests deployments for the environments above, how to request deployments, the notification process while the deployment occurs and some typical steps in a deployment.
Developers request deployments into the “Shared Dev” environment. Deployments to the “Shared QA” and “Performance” environments are requested by QA testers. Once software artifacts are tested and ready to be release into “Stage” and then “Production”, the Product Owner and/or the Project Manager request the deployment.
It is important to have a standard deployment request process in place. This can be something as simple as an email template or a file which contains the necessary information that needs to be filled out or something more sophisticated. Paired with archival software such as Source Code Management software, this mechanism can be used for auditing deployment requests.
A robust notification process is necessary to disseminate information as a deployment occurs. This will generally consist of emails which are sent out to the software development organization before, during and after the deployment to indicate progress. Note that this may result in a lot of emails so it would be beneficial to use filters or some other feature of email clients to archive them. The notification process should be incorporated into the deployment process in an automated fashion.
Sometimes deployments will fail for various reasons ( bugs, performance issues, unanticipated software interactions, etc). In such cases it is extremely important to have a rollback strategy which ensures that software is restored to the previous working version as soon as possible. This is extremely important if any issues are found in production, especially after some time.
This may consist of rolling back the application to a previous version and/or rolling back schema changes in a database. It could also involve a coordinated rollback if multiple services or applications have dependencies between them and need to updated or rolled back together.
Environment Configuration Changes
Most software applications have some mechanism to externalize configuration information. It is recommended that such configuration information be maintained in a Source Code Management system to enable auditing. Requests to modify configuration information should follow the same deployment process as updating application code. The configuration changes should be tested via existing automated test suites.