As a data engineer, while developing spark jobs and performing operations, you will encounter a situation where your spark code that is running on executors(like using map or foreach) might need to perform below operations Call aservice to enrich the row of the given dataset. Store or retrieve something from…

A vulnerability( CVE-2021–44228) in Apache Log4j, a widely used logging package for Java has been found( first reported to Apache on November 24 2021 and was patched with version 2.15.0 of Log4j on December 9 2021 To know more about logging frameworks/wrappers in java refer What is the difference between…

One of the most important concepts in programming languages(like go, java, .e.t.c) or distributed computing is the difference between concurrency and parallelism. Both used to speed up a computation using their own methods, so let’s dive into them. Let’s say we need to prepare pasta with sauce (made of tomato…

What is data → Data is a representation of some aspect of the real world. We can classify data as structured or unstructured or semi-structured based on how it is organized. Customers mostly select structured or un structured or semi structured not based on their data, but on the applications…

AWS MSK(managed streaming for kafka) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. I worked on building kafka cluster on on-premise servers as well as ec2 instances, and AWS MSK really helped to relieve so many manul…

Amazon Simple Storage Service (Amazon S3) is a cost-efficient and highly scalable persistent or temporary object storage that most of the organizations consider using to store regular or big data. Before Nov 2014, whenever objects were created/deleted/e.t.c there was no notification system to transmit the events. To detect those events…

If you are not worked on “clearcase” then you are really a fortunate soul. “ClearCase” is one of the complex repository management software and I always used to face issues due to rebase and other operations to commit code and the errors not able to resolve and some times I…

To process continuous streams of data from sources HDFS directories, TCP sockets, Kafka, Flume, Twitter e.t.c spark has two methodologies. Spark streaming in general works on something called as a micro batch. …

Kafka best practices edition → How to design Kafka message key and why it is important in determining application performance? What is a Kafka Message: A record or unit of data within Kafka. Each message has a key and a value, and optionally headers.The key is commonly used for data about the message and the value is the body of the message Message Key → Can be null or contain…