In this article, we are going to look at the sharding feature in MongoDB. The phenomenon of distributing data across multiple machines or servers is known as sharding. It is a vital and beneficial feature of MongoDB as it supports deployments with very large data sets and high throughput operations across multiple servers.
Sharding feature in MongoDB
MongoDB, a NoSQL document database system is well equipped with the Sharding mechanism to deal with the increasing data size without compromising its performance or throughput. Sharding feature enables MongoDB to store data records i.e. documents across multiple servers in order to meet the data growth demand over the period of time. As data size increases, a single machine may not be able to meet the demand of data storage along read and write throughput or performance. Such issues could be solved with horizontal scaling.
MongoDB achieves horizontal scaling through a technique known as “sharding”. It is a phenomenon to add more machines to write or store data across these machines in order to support the data growth through a key-value system with satisfactory read and write throughput.
Types of Scaling
There are two procedures to address the growth of a system. They are vertical scaling and horizontal scaling.
Vertical Scaling: – A single server can be made powerful enough to handle the given workload by installing a powerful CPU which increases its capacity, adding more RAM (i.e. Random Access Memory), or increasing hard disk storage capacity. But, every hardware architecture has some limitations in terms of technology support that may restrict a single machine for being competent enough to deal with a given workload without compromising read and write throughput. Therefore, with the help of vertical scaling, we can reach up to a particular maximum depending on the available hardware configurations.
Horizontal Scaling: – In horizontal scaling, we can have multiple machines in order to divide the system dataset as well the load over those machines which collectively increase the overall capacity as required. An individual machine may not have the overall capacity to be high, but each machine handles a subset of the overall workload in order to provide a better efficiency as compared to a single high-capacity server. Here, we can expand the overall capacity of the deployment by adding more servers as required, which could be of lower overall cost as compared to a single machine high-end hardware. Again, there is a limit to add number machines due to increased infrastructure complexity and deployment maintenance. Horizontal scaling is achieved through “sharding”.
Sharding in MongoDB
Below is the demonstration of the sharded cluster in MongoDB.
In the above diagram, there are three main components. Each of the component is described below.
Shards: – Shard is also known as replica set. It is used to store data. Shards together provide high availability and data consistency across the cluster. It should be noted that in the production environment, each shard represents a separate replica set.
Config Servers: − Config servers in the cluster are meant to store the metadata associated with the cluster. Metadata basically stores a mapping of the cluster’s data set to the shards. Where target queries with shard key are fired then the query router uses this metadata to target operations to specific shards. It should be noted that in the production environment, sharded clusters have exactly 3 config servers.
Query Routers: − Query routers are the mongodb instances which directly interface with the client applications. These routers direct the operations to the appropriate shard in order to read or write the data set in the cluster. The query router processes the incoming request from client application and targets the operations to required shards in the cluster which returns the result to the client. A single shard cluster may contain one or more query router in order to divide the client request load. Usually, a shard cluster have many query routers.
Advantages of Sharding
Reads / Writes Workload: – Each machine in a sharded cluster is known as shard and the read and write workload is distributed across these shards which allow each shard to process only a subset of cluster operations (i.e. division of labour). Therefore, the overall read and write workloads can be scaled up horizontally across the cluster as we add more shards. We can directly target the query to a particular shard or a shard set by including the shard key or a compound shard key prefix in the query that acts as an address. The efficiency of such targeted operations is more than broadcasting the query to every shard in the shard cluster.
Storage Capacity: – The memory on local disk is not big enough to handle large dataset. Therefore in Sharding, the data is not stored on a single machine but it is distributed across the shards in the shard cluster which allows each shard to contain a subset of the data out of the total cluster data. As a result if the volume of data set grows, the addition of the extra shards increase the overall storage capacity of the shard cluster.
High Availability: – A sharded cluster is composed of multiple machines or shards. Therefore, when a particular shard is unavailable in a shard cluster, we can still continue to perform our partial read / write operations through the available shards active in the shard cluster. It ensures availability of the subset of data during the downtime of a shard, where the reads or writes operations are directed at the available shards in the shard cluster.
Vertical scaling is too expensive: – Vertical scaling is expensive when we compare a single machine to a shard in term of read and write throughput comparison.
Conclusion: –
Sharding is the phenomenon to achieve horizontal scaling in MongoDB to meet the increasing data size and maintaining constant read and write throughput. But before, we move to a sharded environment the sharded cluster infrastructure should be careful planned and reviewed for its complexity and maintenance. There should be careful consideration while choosing the shard key, to ensure the shard cluster performance and efficiency. Sharding has certain operational requirements and restrictions which should be taken into consideration during the planning phase.
