The Lambda Architecture Simplified

A Brief History of the Lambda Architecture - The surest sign you have invented something worthwhile is when several other people invent it too. That means the creative pressure that gave birth to the idea is more general than your particular situation.  Even when faced with the same pressures, people will approach an idea in different ways. When Jay Kreps was developing Kafka at LinkedIn, he called it The Log. Facebook (being Facebook) created several independent implementations of “stream-oriented processing”, including Puma and TailerSwift. Twitter has the adorably named Summingbird. The jargon we seem to be converging on for these kinds of systems is the Lambda Architecture.

Lambda Origin - In his book, Big Data: Principles and best practices of scalable real-time data systems, Nathan Marz coined the term ‘Lambda Architecture’ to describe a generic, scalable and fault-tolerant data processing architecture based on his experience in working on distributed systems at Backtype and Twitter.

Lambda in a Nutshell -  The gist of the Lambda Architecture is to model everything that goes on in a complex computing system as an ordered, immutable log of events. Processing the data (say, totaling up the number of website visitors) is completed as a series of transformations that output to new tables or streams. It is important to keep the input unchanged. By breaking data processing into independent pieces, each with a defined input and output, you get closer to the ideal of purely functional programming. Writing and testing each piece is made simpler and parallelization can be automated. Parts of the dataflow can be replayed (say, when code changes or machines fail) and toyed together with other flows.