About the Client
A North American cable company with more than 32 million customers needed to modernise its cloud-based infrastructure following a series of business acquisitions and a merger.
Sourced Group an Amdocs Company (Sourced), previously supported the telco’s early cloud adoption through enhanced use of Amazon Web Services (AWS). This time, we unified disparate technologies and transformed release engineering. The following were achieved through this engagement:
- Two platforms and microservices architectures were consolidated into one.
- Deployment time reduced by 75%.
- Improvement in strategic positioning for innovation and technology delivery.
Challenge: Merger and Acquisitions Amplified Technical Debt
Following a series of acquisitions and a substantial merger, our client’s priority was to consolidate multiple technical stacks. It was using a range of data centres and technologies, with customers accessing diverse, inconsistent applications and services depending on their geographic location. The overarching goal was to streamline the customer experience.
Initially, the client started working towards a new architecture while Sourced focused on strategies to evolve the microservices architecture. For three years we collaborated on the design and implementation of an integrated self-service experience across multiple acquired and merged companies. We started with a federated identity solution and worked towards a unified customer experience, reducing call volume by 10% each year.
In late 2019, a reorganisation of the client’s business provided an opportunity to critically review the release process and pinpoint improvement possibilities. We turned our attention to the platform and operations space to assess and subsequently help lead a multi-year modernisation effort. Our aim was to eliminate technical debt with a clear set of objectives: (i) to double deployment frequency, (ii) improve release quality, and (iii) reduce the operational burden.
Solution: ‘Everything as Code’ with GitOps
Most of the client’s technical debt was related to a lack of automation across infrastructure, platform and release processes. This was resolved with an Everything as Code approach.
Together with the client we devised and implemented three streams of work. The first focused on platform automation, release process transformation, CI/CD pipeline refactoring, and DevOps tooling migration. Next, we focused on modernising the microservices database. Finally, we looked at legacy microservices modernisation.
Stream One
To improve infrastructure and platform processes, we automated AWS and Kubernetes resource lifecycles. Using tools such as Terraform, Ansible, and Kubernetes Operations (KOPS) we empowered developers to spin up and tear down stacks as needed. Release plans for more than 200 microservices were also created in code, with all platform, application, and configuration components included. They are maintained in Git, and GitOps manages the product release lifecycle.
CI/CD pipelines were designed and rebuilt as well. We streamlined the packaging pipelines so that all code is built, tested, released and published to the client’s repository, allowing full control and visibility of the entire process. All DevOps tooling was migrated from Jenkins, Bitbucket, and Nexus to newer tools including Gitlab CI/CD and Artifactory.
Stream Two
The database modernisation effort encompassed persistent data and ephemeral or cached data. Cassandra had been the client’s database of choice for storing persistent information since the initial move to AWS. However, operations and maintenance had become a burden and an alternative option was needed. Alongside this, several different platform solutions had been used for ephemeral data, but older deprecating caching solutions and services were still in use, resulting in technical debt.
To resolve these database issues, we planned and implemented a strategy to consolidate to one persistent data store and one ephemeral data store. Over a two-year period, we successfully migrated all service codebases to AWS-managed Cassandra and AWS-managed Redis, as appropriate.
Stream Three
This stream addressed application debt and modernised all legacy services not yet retired or migrated to the target Kubernetes containerised architecture. Services that were no longer maintained but still accessed by customers needed to migrate to an appropriate microservice. At the same time, services without a domain in the new environment needed containerisation, removal of old dependencies, and migration to the new platform.
This modernisation program required extensive migration of elements related to service discovery, configuration management and certificate management. We also updated application dependencies for containerisation compatibility, choosing newer application frameworks like Spring Boot and GraphQL when a complete rewrite was necessary.
Outcome: Deployment Time Slashed, Operations Improved, Developers Empowered
With our support, the client consolidated two platform and microservices architectures into one. This eliminated years of technical debt and reduced the operational overhead of custom database solutions to the elegant simplicity of AWS managed services.
Today, the client maintains a two-week deployment frequency with features that touch up to 200 microservices and configurations. The ability to rebuild any environment, including production, has led to a 75% reduction in deployment time. Every release now follows immutability principles, and all infrastructure is written as code. The client has also eliminated environmental drift by enforcing environment as code policies.
Additional achievements include successful deployment of major releases via blue/green deployments across four key products. The blue/green approach has reduced operational overhead, decreased Mean Time to Resolution (MTTR), and increased observability for every stack component. More than 26 deployments have been made, without any rollbacks being required. We have also increased lead time by 30% by promoting changes across environments (dev to prod) in a consistent manner with a single-pipeline release lifecycle.
Finally, we have improved the developer experience by creating consistent, easy-to-use CI/CD pipelines to propagate code across environments and into releases. We have moved control of cluster ingress, egress, pod-to-pod network routing, and certificate and key management to the platform via a single service mesh control plane allowing developers to focus on business logic.
Together, these changes put the client in a strong position to accelerate innovation and technology delivery.
