In which scenarios need to use mapPartitions or foreachPartitionin in spark (Simple question that can gauge your knowledge in spark programming)

  1. Call aservice to enrich the row of the given dataset.
  2. Store or retrieve something from a data store (like database/redis) e.t.c

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aditya

Aditya

177 Followers

Principal data engineer → Distributed Threat hunting security platform | aws certified solutions architect | gssp-java | Chicago-IL