Stream processing systems have become increasingly popular in the last decade. It found its usage in a variety of real time systems across the industry. There are mainly two ways to implementing a stream processing system — Continuous Streaming and Micro-batches. Continuous streaming The likes of Apache Storm and Apache Flink implements…

ACID is an acronym for Atomicity, Consistency, Isolation and Durability. ACID compliant databases provides Atomicity, if it has the ability to group multiple statements together and execute it as if it were executing a single atomic statement. There can only be two outcomes for such an execution, either all the…

In Part 1 and Part 2 of the series, I talked about all the cool concepts used in Amazon Dynamo. The key aspect of the system is the decentralised nature of its architecture. This enables it to be incrementally scalable. …

In Part 1 of this series, we saw the various techniques that Amazon Dynamo employs to achieve high availability for write requests. In this post, we will deep dive into the different failure handling techniques used by Dynamo. Quorum Failure In Part 1, we saw how we can configure the values of…

I happened to read the Amazon Dynamo paper recently. I have read it before but not nearly as intently. Its architecture manages to bring in multiple computer science concepts together and make it work in tandem to achieve the desired goals of the system. This will be a three part…

In the Internet, a lot of content is consumed on a daily basis. For global services like Google, Facebook, Netflix, etc serving this data to the end users in a seamless manner becomes a priority. To design a web service to satisfy these requirements, we need to be able to…

As the name suggests, Relation Databases are used to store entities and the relationships between them in a normalised manner. Addition to this, they also provide transactional support and indexing. There are many use cases in the industry where we would want to use the same set of relational data…

SSTables and LSM-Trees are techniques which are used to efficiently store and retrieve data. In this post, we will limit our discussion to data retrieval based on a primary key. Before diving deeper into these techniques, let us understand a simplified storage engine, where the keys are stored in memory…

To understand the need for consistent hashing, we need to first understand the need for data partitioning. Dataset, which is small enough, could be stored in a single physical machine. Every read request will be served by this one machine. This single machine is the ‘master’. For fault tolerance, let’s…