Redis Cluster sharding, L1+L2 cache layering, event-driven invalidation across services, and the stampede-prevention tricks that kept reads alive at peak hours.
It was a Tuesday morning at the creator economy platform I spent the last few years at. Community feeds were the hottest read path on the system, sitting in front of a multi-terabyte Aurora writer and a handful of reader replicas. Around 10:14 a.m. PT, p99 on /communities/:id/posts climbed from ~120 ms to >8 s in four minutes. Replica lag was at 14 minutes and still growing. I wasn’t on call that week but the Slack thread tagged me anyway because I owned the Aurora layer.
We unstuck that one in the database (a rogue ANALYZE was starving WAL emission, killing it drained the lag in ~6 minutes). But the postmortem made one thing very loud. The read path had no shared cache worth the name. Every service was talking to Postgres for things that hadn’t changed in hours. So we sat down and actually designed the caching layer.
This is roughly what I’d do again, give or take.
The thing is, distributed caching isn’t really about Redis. It’s about deciding where your truth lives, where your copies live, and who’s allowed to invalidate them. Get that wrong and you spend the next quarter chasing ghosts.
The shape I keep coming back to in microservices:
L1 absorbs the truly hot keys (the same posts feed pulled a hundred times in two seconds). L2 absorbs the warm tail and keeps services from beating up Postgres. Kafka is the only thing allowed to invalidate cross-service.
// packages/cache/src/multi-level-cache.ts
import { LRUCache } from 'lru-cache';
import type { RedisClusterType } from 'redis';
type Loader<T> = () => Promise<T>;
export class MultiLevelCache {
private readonly l1 = new LRUCache<string, { v: unknown; e: number }>({
max: 5_000,
ttl: 2_000,
});
constructor(
private readonly l2: RedisClusterType,
private readonly opts: { l2TtlMs: number; negativeTtlMs: number },
) {}
async get<T>(key: string, loader: Loader<T>): Promise<T> {
const local = this.l1.get(key);
if (local && local.e > Date.now()) return local.v as T;
const raw = await this.l2.get(key);
if (raw !== null) {
const parsed = JSON.parse(raw) as { v: T; e: number };
this.l1.set(key, parsed);
return parsed.v;
}
const fresh = await loader();
const envelope = { v: fresh, e: Date.now() + this.opts.l2TtlMs };
await this.l2.set(key, JSON.stringify(envelope), { PX: this.opts.l2TtlMs });
this.l1.set(key, envelope);
return fresh;
}
}Two details that I will die on. The L1 TTL is short on purpose. A couple of seconds is enough to absorb a burst, short enough that you don’t need to invalidate L1 from another pod, because it expires before anyone notices. And the envelope carries its own expiry, so you can distinguish “I have a value” from “I have a negative cache entry”, which matters more than you’d think.
Redis Cluster shards by slot. If you want a pipeline or a MGET to be fast, the keys have to live on the same slot. The trick is the {} hash tag.
const postsKey = (communityId: string, postId: string) =>
`posts:{community:${communityId}}:${postId}`;
const counterKey = (communityId: string) =>
`counters:{community:${communityId}}:posts`;Both keys hash on community:${communityId}, so they land on the same shard. You can MGET all posts for a community in one round trip. Without the hash tag you’d cross-slot and Redis Cluster would refuse the multi-key op. I learned that one the hard way during a load test where a single tenant’s feed was making 40 round trips per render.
Cross-service invalidation needs a contract. TTL is not a contract. TTL is hope.
The pattern I lean on: the service that owns the entity publishes a domain event on Kafka. Every other service that caches a derived view subscribes and invalidates. The event carries the entity id, the version, and an occurred_at. Consumers use the version to ignore out-of-order deletes (the kind of thing that happens whenever a Kafka consumer rebalances mid-flight).
// services/posts/src/events/post-updated.publisher.ts
export async function publishPostUpdated(
producer: KafkaProducer,
post: { id: string; communityId: string; version: number },
) {
await producer.send({
topic: 'community.posts.v1',
messages: [
{
key: post.communityId,
headers: { 'content-type': 'application/json' },
value: JSON.stringify({
type: 'post.updated',
postId: post.id,
communityId: post.communityId,
version: post.version,
occurredAt: new Date().toISOString(),
}),
},
],
});
}Keying messages by communityId keeps related events on the same partition, which keeps order on the consumer side. Order matters because a late post.deleted after a post.created will resurrect a post you thought was gone.
On the consumer side, the invalidation handler is boring on purpose.
// services/feeds/src/consumers/posts.consumer.ts
@Controller()
export class PostsConsumer {
constructor(private readonly cache: MultiLevelCache, private readonly versions: VersionStore) {}
@EventPattern('community.posts.v1')
async onPostEvent(@Payload() msg: PostEvent) {
const known = await this.versions.get(msg.postId);
if (known !== null && known >= msg.version) return;
await this.cache.delete(`posts:{community:${msg.communityId}}:${msg.postId}`);
await this.cache.delete(`feed:{community:${msg.communityId}}:page:1`);
await this.versions.set(msg.postId, msg.version);
}
}The version check makes the handler idempotent and order-resilient. Without it, a redelivery from a consumer rebalance will happily reorder your invalidations.
Cache stampedes happen when a hot key expires and a thousand requests all try to repopulate it at once. Two tools that actually work:
// packages/cache/src/stampede.ts
export async function fetchWithStampedeGuard<T>(
redis: RedisClusterType,
key: string,
ttlMs: number,
loader: () => Promise<T>,
): Promise<T> {
const cached = await redis.get(key);
if (cached) {
const { v, e } = JSON.parse(cached) as { v: T; e: number };
const remaining = e - Date.now();
const beta = 1.0;
const shouldRefresh = -Math.log(Math.random()) * beta * 1000 >= remaining;
if (!shouldRefresh) return v;
}
const lockKey = `${key}:lock`;
const acquired = await redis.set(lockKey, '1', { NX: true, PX: 5_000 });
if (!acquired) {
if (cached) return (JSON.parse(cached) as { v: T }).v;
await new Promise((r) => setTimeout(r, 50));
return fetchWithStampedeGuard(redis, key, ttlMs, loader);
}
try {
const v = await loader();
await redis.set(
key,
JSON.stringify({ v, e: Date.now() + ttlMs }),
{ PX: ttlMs },
);
return v;
} finally {
await redis.del(lockKey).catch(() => undefined);
}
}Beta = 1.0 is a reasonable default. Crank it higher for keys that are expensive to compute. The point is, you’re spreading the refresh load across a window instead of letting it all land at once.
Thanks for reading. If you’ve got thoughts, send them my way.