Member-only story
AWS MSK(managed streaming for kafka) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data.
I worked on building kafka cluster on on-premise servers as well as ec2 instances, and AWS MSK really helped to relieve so many manul steps wrt to kafka cluster setup and simplified monitoring(CloudWatch) and maintainability(rolling software/hardware upgrades)
But one problem with AWS MSK as of Aug 2021, is once we scale up EBS storage volume per broker by using manual or auto-scaling, we cannot scale back down (say due to reduced workload ) and due to which we will be billed more for unused broker storage.
One solution is to create a new cluster and use a mirror maker or some other tool using say spark jobs that transfer data from topics of existing cluster to new cluster, which is a tedious process with lot of moving parts waiting to fail.
Raised a couple of support tickets with aws and got an update they may consider it in the future roadmap and if it is added, it will definitely helps organizations to dynamically scale up or scale down EBS storage per broker.
