Skip to content
SparkTime, home
Menu
Theme
Tag · 2 pieces

#sharding

Related tags

Tagged #sharding

GuideTechnology

17 min read

Design a Web Search Engine: Crawling, Inverted Indexes and Ranking

An educational design for searching a billion web pages: a polite, distributed crawler with a URL frontier and robots.txt, deduplication, building an inverted index with batch processing, document-partitioned shards and replicas, scatter-gather query serving with result caching and tail-latency control, ranking from relevance and link signals, freshness tiers, failures and safety.

GuideTechnology

17 min read

Database Sharding and Partitioning: How It Works and When You Need It

When one database can no longer hold the data or absorb the writes, you split it. Learn vertical and horizontal partitioning, how to choose a partition key, range, hash, consistent-hashing and directory-based sharding, rebalancing with logical shards, hot partitions, cross-shard queries and secondary indexes, and why you should shard as late as possible.