A matching engine pairs supply with demand under tight latency budgets — riders to drivers, orders to inventory, jobs to workers. The hard part isn't the matching logic; it's keeping p99 latency under 200ms while the candidate set grows.
1. Shard the index, not just the data
A single global index becomes a contention point fast. Partition by a natural key — geo-cell, region, or category — so each lookup only scans the relevant shard. This turns an O(N) scan into O(N/shards) and lets you scale horizontally.
2. Two-tier caching
Pair a process-local LRU with a shared Redis layer. The local cache absorbs hot reads with zero network hops; Redis keeps shards consistent across nodes. In practice this cut our lookup time by ~60% under sustained load.
// local LRU first, fall back to Redis
Candidate c = localCache.getIfPresent(key);
if (c == null) {
c = redis.get(key);
localCache.put(key, c);
}3. Backpressure beats overload
When request rate spikes, shed or queue rather than letting the engine thrash. A sliding-window rate limit at the edge protects tail latency for everyone else.
Optimize for the p99, not the average. Users feel the slow requests, not the fast ones.
Combine sharded indices, two-tier caching, and edge backpressure and sub-200ms matching at scale stops being a fire drill and becomes the steady state.