by Diana Dima
Context
A well-established telecom company undergoes a digital transformation process. A part of a monolith legacy application, in use since early 2000 was about to be decommissioned and replaced by a newer, opensource application. Telecom market is one of the most regulated in Romania, and the competition among players is high. So, there was little room for errors.
Challenges
preserving the same user experience
implement against the clock, as important market milestones were coming
high expected quality
Solution
Implementation of Apache Kafka Streams with Java, in AWS Cloud and monitor with Kibana and Grafana. High availability was provided through Kubernetes.
Main Kafka advantages
faster handling
real time handling of data
open-source that can be access for free
About our client
A well established telecom company that provides mobile and broadband services. By number of customers, it is one of the largest telcos in the world, with over 400 million connections and customers across 26 countries.
The challenge
The client was already undergoing a digital transformation processes to decommission legacy monolith apps in usage since early 2000 and replacing them with more modern, reliable, and flexible solutions. The business needs were clear regarding user experience and timely implementation. So, the challenge was to combine new microservices architecture with the legacy apps making sure the errors are kept at a low level, control downtime while providing the same user experience to the end customer in a highly regulated and competitive telecom market.
The solution
Based on the business needs, it was important to identify a simple and easy-to-integrate solution. After thorough consideration, we implemented Apache Kafka Streams to replace the legacy component and orchestrate the information flow. Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. The solution is highly regarded within the IT community.
Some of its best advantages to competitor solutions such as IBM MQ, Active MQ are real time data handling and the fact that it is an open-source tool.
Kafka is way faster than ActiveMQ. It can handle millions of messages per sec. ActiveMQ supports both message queues and publishes/subscribe messaging systems. On the other hand, Kafka is based on publish/subscribe, but does have certain advantages of message-queues. Apache Kafka is designed to enable the streaming of real time data feeds and it's an open-source tool that users can access for free. IBM MQ is a traditional message queue system that allows multiple subscribers to pull messages from the end of the queue.
In our case, the main benefit brought was the data reliability that it provided. At any point in time one can reuse information from other offsets when one system is down. In case of unprocessed information, Kafka can reprocess the information from any point earlier in time. We implemented retry mechanisms, which automatically reprocesses information left unprocessed. Also, when reentry failed, we received a dead letter list and a support engineer was able to manually investigate and resend the messages.
We managed to synchronize relational data bases of the legacy product, developed since 2003, built on monolith architecture to Amazon data bases, based on the new AWS Cloud. We started by decommissioning the legacy systems piece by piece (JMSQ) and replace it with the new solution. In total there were 4 interconnected systems that had to process each other’s messages, and had to be orchestrated so as not to impact the final user.
The data flow of information was beautifully processed through Kafka with more than 400 mil daily messages with no downtime registered. Moreover, during important campaigns, the spikes registered up to 200k messages per minute. We knew it was very good at processing short messages in real time, on multithreading streams and send them to their destination, but the speed and quality were still impressive. The technical design brought together clusters of producers (the ones that store the information and send it further unprocessed) with a cluster of consumers (the ones that process and normalize the information received from the other cluster). The consumer is a spring boot app, deployed in cloud with very specific function; one is responsible for bringing billing information, user information, alerts and so on. One consumer can connect with one or more different producers at the same time to finally send the needed information further to user requesting it from self-portal. The challenge for us was to understand business flows and which consumer should consume what information from which producer and in what order. The stakes were high, and we had to monitor and debug in real time through Grafana and Kibana. We were able have a clear imagine of how and what kind of information flows through the app. We captured consumer data such as logs and error messaging flows, put it in a file and investigate and analyze it qualitatively in real time. The rate of success vs error flows was intensely monitored for these asynchronous events. To ensure high availability we used Kubernetes.
The results
We successfully implemented the solution together with the client, while preserving the same experience for the end customers. As a result, the end customer was able to navigate between newer and older applications without noticing the difference, with their requests processed by the synchronization system deployed.
The speed and quality of the implementation were impressive. It all took 4 months and after a very successful stress test it was dimed a success case study for our client.
About us
At IT Smart Systems, we value fresh ideas, passion and above all our ITSS Community! While working on cutting-edge technologies and on-the-rise domains, you’ll discover new interests and put your career on hyper growth mode.
For two decades, we’ve worked with a steady stream of the world’s biggest players in their industry to become one of the most reliable Romanian Digital Transformation Enablers.
Our expertise lies in
Taking care of your 24×7 available touchpoints, whether it’s Omnichannel, eCommerce, or Self-care.
Modernised Digital Identity Solutions, ready for the Cloud, bringing that frictionless yet secure authentication process to life.
Integrating. Everything. Anywhere. Speed you up in bringing new products and services into the market, with highly integrated application integration and business process management.
Also pioneering for the open banking sector in Romania where, in Alliance with Smart Fintech, we continuously innovate open banking solutions and successful implementations to #keepITsimple.
Diana Dima
HR Business Partner
With ITSS for 7 years, Diana is passionate about people and believes that HR is the engine of any organisation. Always determined to acknowledge the networking behind every organisational culture, Diana has a great flair for technical projects and a deep understanding of internal processes.