“Lambda Architecture” in big data with real-world scenarios(Why every big data system design consider using it)

Lambda architecture identifies itself with big data, not to be confused with AWS lambda which is just a function(or piece of code) invoked on an event on a source(like s3, sqs e.t.c) or lambda expressions in java.

Big data is the most celebrated word in the last decade, thanks to the gigantic explosion of data and the requirement to make sense of this data to drive business

What is batch processing → It’s a processing methodology where data allowed to accumulate for a specific ( period like 30 or 60 or 180 min). Once data accumulated it will be processed using big data processing frameworks/tools like a spark to generate meaningful insights for that period.

E.g -1 : In the retail world customer orders allowed to accumulate in the system for say 12 hours and then a spark job will run to analyze the trends for a given customer and updates his personalized view or feed buying patterns to some machine learning model for training e.t.c

E.g -2:In security space, security events like logins, process launches data e.t.c will be accumulated for say 3 hours, and then a spark or google cloud dataflow will go through the events and determine any malicious process launch or anomalous sign-ins…