Please read my previous article https://sprinkle-twinkles.medium.com/how-jmx-metrics-from-spark-applications-will-help-to-configure-driver-executor-memory-correctly-560d5863d0af article to get context around the need and use of exposing spark jmx metrics . Below are steps we need follow to expose spark jmx metrics and export them to datadog. Step one: Configure spark job to expose jmx metrics. This can be done…

What are jmx metrics → Java Management Extensions (JMX) is a specification for monitoring and managing Java applications. Monitoring and Instrumentation There are several ways to monitor Spark applications: web UIs, metrics, and external instrumentation. Every…spark.apache.org JMX metrics we are interested in generally are like below Spark driver jvm metrics like heap and non heap used, usage and committed stats

When a spark application invoke an action, such as collect() or take() on your DataFrame or Dataset, the action will create a job. Below is the Cheat sheet to remember. One job results one or more stages One stage results one or more tasks one task operates on one partition …

Compaction → Process of converting small files to large file(s) (consolidation of files) and clean up of the smaller files. Generally, compaction jobs run in the background and most of the big data processing applications support manual and automatic compactions. What is the issue with small files → Having so…

As a data engineer, while developing spark jobs and performing operations, you will encounter a situation where your spark code that is running on executors(like using map or foreach) might need to perform below operations Call aservice to enrich the row of the given dataset. Store or retrieve something from…

A vulnerability( CVE-2021–44228) in Apache Log4j, a widely used logging package for Java has been found( first reported to Apache on November 24 2021 and was patched with version 2.15.0 of Log4j on December 9 2021 To know more about logging frameworks/wrappers in java refer What is the difference between…

One of the most important concepts in programming languages(like go, java, .e.t.c) or distributed computing is the difference between concurrency and parallelism. Both used to speed up a computation using their own methods, so let’s dive into them. Let’s say we need to prepare pasta with sauce (made of tomato…

What is data → Data is a representation of some aspect of the real world. We can classify data as structured or unstructured or semi-structured based on how it is organized. Customers mostly select structured or un structured or semi structured not based on their data, but on the applications…

AWS MSK(managed streaming for kafka) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. I worked on building kafka cluster on on-premise servers as well as ec2 instances, and AWS MSK really helped to relieve so many manul…