DevOps Approach for Analytics Company

The client’s product bridges the gap between Operational Technology (OT) and the IT world. The product enables companies to easily integrate OTs such as Internet-enabled devices into IT solutions such as ERP platforms. Through a series of processes, data is organized into consumable formats that downstream systems can use. QBurst worked with the client in the role of DevOps—a new hybrid role for tech leads that enables them to control changes to product master repositories, deploy to a wider range of environments, and gain broader access to help operations diagnose production issues.

• A continuous integration pipeline to replace the manual development of their real-time analytics platform.
• A highly agile, frictionless environment for development, integration, testing, and release.

QBurst was engaged to help in environment setup, continuous integration and delivery, monitoring, technical support, and disaster recovery. We designed a system on the cloud-based Amazon EC2 environment, where horizontal deployment could be achieved with scalability and resilience.

Environment Setup

There are three main environments in the setup:

• Developer environment: Built, deployed, tested, and destroyed by developer
• UAT/QA environment: Code built and deployed using continuous integration tools; server troubleshooting and log analysis
• Production environment: Services running on Docker containers; resources at multiple availability zones

We leveraged embedded CloudFormation stacks to model the infrastructure in layers and quickly deploy stacks to any one of the Amazon Web Service (AWS) datacenters around the world that was closer to the client centers. This allowed developers to run and destroy their own isolated stacks and also version control our templates, using them to manage changes to stacks.

Continuous Integration and Delivery

QBurst built and deployed the system using Docker, a platform that swiftly assembles applications, eliminates friction when moving through different environments, tests, and deploys the code into production as fast as possible.

• Whenever a code change is pushed to GitHub, it triggers a build.
• Jenkins, called via a post-commit hook, builds and tests the code.
• If passed, a Docker image is built using the bundled Dockerfile with a simple command.
• Our CI server pushes the image into a private Docker registry.
• Jenkin runs an Ansible playbook to deliver/deploy the container to relevant servers.

Monitoring, Tech Support, and Disaster Recovery

Our 24/7 support team monitors the infrastructure and application, proactively fixing issues related to service disruption. Continual troubleshooting helps to identify bottlenecks and improve performance of the infrastructure. Resolution time based on Service Level Agreement (SLA) is maintained. Backup and restore scenario implemented: in the event of a disaster, resources are quickly launched in AWS to ensure business continuity.

• Published a set of application platforms to allow developers to code in the platform and language of their choice.
• Integrated the application platforms with Dev’s standard repositories and existing continuous integration and configuration management tools.
• Used complete automation to avoid human error and ensure the same build, metadata, and content are used from development to production.
• Enabled developers to create environments and push new builds at a button press to easily test integration and operation of their projects.
• Established policies for application release automation and defined user roles and rights within applications.
• Allowed for concurrent development of multiple projects without risk of regression or merge errors.

• Docker
• Jenkins
• Ansible
• AWS
• CloudFormation
• CloudWatch
• Nagios
• Slack

• Time taken for environment setup reduced from one week to a few hours.
• Operations teams got over 80% of their time back with automation.
• Tech leads got 54% back essentially doubling time on keyboard developing code.
• Docker containers provided isolation of services and upgrading/downgrading of application versions was easier.
• Improved deployment frequency leading to faster time-to-market.
• Lower failure rate and less complex problems to fix.
• Faster recovery with quick resolution of problems.