An Energy Company Modernizes Real-Time Data Transmission with AWS and GFT
When Advantages turn into Results
Simplified Operations and Governance
It standardized streaming services on AWS to improve platform management and strengthen data governance
Modernization Using Native AWS Technologies
Migrated from Confluent Kafka to Amazon MSK, simplifying the architecture and operations using native AWS services
A Scalable Foundation for Growth
It built a scalable and resilient platform, ready to support future business and data growth
Challenge
Modernize the streaming platform without disrupting business operations
The company needed to reduce the operating costs of its event streaming platform without compromising scalability, resilience, real-time data processing capabilities, business continuity, or rollback options. The environment included approximately 300 topics, 581 partitions, Schema Registry, Kafka Connect, Debezium CDC, S3 Sink connectors, and multiple producer and consumer microservices. The migration had to minimize the impact on critical services during the transition from Confluent Cloud to Amazon MSK Serverless.
Engagement
Migration to an AWS-native architecture
GFT collaborated with the company to design and implement a migration strategy from Confluent Cloud to Amazon MSK. The target architecture replaced the Kafka brokers with Amazon MSK, migrated the Confluent Schema Registry to the AWS Glue Schema Registry, and moved the Kafka Connect workloads to MSK Connect. To mitigate migration risks, GFT designed a coexistence phase during which both environments ran in parallel, using Kafka MirrorMaker 2.0 on Amazon EC2 to replicate data between Confluent and Amazon MSK and ensure the ability to roll back if necessary.
Benefit
A more efficient, scalable, and future-ready platform
The project provided the company with an event streaming platform that is better aligned with the native AWS ecosystem and its global cloud adoption strategy. The migration simplified operations by consolidating streaming services within the AWS environment, preserved critical capabilities such as real-time data processing, change data capture (CDC), schema management, and event-driven communication, and increased scalability through a serverless architecture designed to meet future business needs.
The managed environment includes approximately 300 Kafka topics, 581 partitions, Schema Registry, Kafka Connect workloads, Debezium CDC, S3 Sink connectors, and numerous producer and consumer microservices. In addition, replacing a low-utilization Confluent deployment with Amazon MSK Serverless reduced the platform’s monthly costs by approximately 40%.
Got Questions? We’re Happy to Help.
