Lançado · em melhoria
Guia de Hashing consistente · 5/6
Por enquanto, este capítulo está disponível apenas em inglês.
Every problem below reuses hash64 and HashRing from the implementation chapter (sorted points plus the owner map from position to node). Try each one yourself before reading the solution.
Given a ring, compute what fraction of the full 2^64 space each node owns. Vary the vnode count over 1, 10, 100 and 1000 and see how far the largest and smallest shares drift from the average.
Approach: each point owns everything after the previous point up to itself. Walk the sorted points and add (p - previous) mod 2^64 to the owner's total. The predecessor of index 0 is the last point, and Python's pts[-1] handles that wrap for free. A ring with a single point gives a difference of 0, so treat it separately.
RING_SIZE = 1 << 64
def ownership(ring: HashRing) -> dict[str, float]:
pts = ring.points
if len(pts) == 1:
return {ring.owner[pts[0]]: 1.0}
share: dict[str, int] = {}
for i, p in enumerate(pts):
arc = (p - pts[i - 1]) % RING_SIZE # i == 0 wraps to the last point
node = ring.owner[p]
share[node] = share.get(node, 0) + arc
return {node: s / RING_SIZE for node, s in share.items()}
for v in (1, 10, 100, 1000):
r = HashRing(vnodes=v)
for n in range(10):
r.add(f"node-{n}")
s = ownership(r)
print(v, round(max(s.values()) * 10, 2), round(min(s.values()) * 10, 2))Multiplying by the node count normalizes the average to 1. As vnodes increase, both extremes approach 1.
Each node belongs to an availability zone. Pick R replicas for a key that are distinct physical nodes in distinct zones. If not enough qualify, return as many as you found.
Approach: walk clockwise from the key one point at a time, skipping nodes already chosen and zones already used. Stop after one full lap.
import bisect
def zone_replicas(ring: HashRing, zone_of: dict[str, str], key: str, n: int) -> list[str]:
pts = ring.points
start = bisect.bisect_left(pts, hash64(key))
chosen: list[str] = []
zones: set[str] = set()
for step in range(len(pts)):
node = ring.owner[pts[(start + step) % len(pts)]]
if node in chosen or zone_of[node] in zones:
continue
chosen.append(node)
zones.add(zone_of[node])
if len(chosen) == n:
break
return chosen
zone_of = {"a1": "az-1", "a2": "az-1", "b1": "az-2", "b2": "az-2", "c1": "az-3"}
ring = HashRing(vnodes=50)
for node in zone_of:
ring.add(node)
print(zone_replicas(ring, zone_of, "order:1001", 3)) # one node from each zoneBefore adding a node, list which hash ranges it takes over and from which existing node. Return tuples where each range excludes start and includes end.
(start, end, previous_owner)Approach: in the ring after the addition, each point p of the new node takes over the range from its predecessor up to p. The previous owner is whoever owned p in the ring before the addition. The rule still holds when several new points sit next to each other.
def migration_plan(ring: HashRing, new_node: str) -> list[tuple[int, int, str]]:
old_points = list(ring.points)
old_owner = dict(ring.owner)
ring.add(new_node)
pts = ring.points
plan = []
for i, p in enumerate(pts):
if ring.owner[p] != new_node:
continue
j = bisect.bisect_left(old_points, p) % len(old_points)
plan.append((pts[i - 1], p, old_owner[old_points[j]]))
return plan
def in_range(h: int, start: int, end: int) -> bool:
if start < end:
return start < h <= end
return h > start or h <= end # the range wraps past 0To verify, look up a few thousand keys before and after the change and check that every key that changed owner falls inside a planned range with the matching previous owner, and that no other key moved.
Nodes have capacity weights. Build a rendezvous hash that assigns keys in proportion to weight and moves only the removed node's keys when one node leaves.
Approach: for each node, turn a hash of node and key into a uniform number u between 0 and 1, and pick the node with the largest score -w / ln(u). That score wins with probability proportional to w. Using the top 53 bits plus 0.5 keeps u strictly between 0 and 1.
import math
from collections import Counter
def hrw(key: str, weights: dict[str, float]) -> str:
best, best_score = "", -math.inf
for node, w in weights.items():
u = ((hash64(f"{node}:{key}") >> 11) + 0.5) / (1 << 53) # 0 < u < 1
score = -w / math.log(u)
if score > best_score:
best, best_score = node, score
return best
weights = {"small": 1.0, "medium": 2.0, "large": 4.0}
keys = [f"obj:{i}" for i in range(70_000)]
first = {k: hrw(k, weights) for k in keys}
print(Counter(first.values())) # roughly 10000 / 20000 / 40000
del weights["medium"]
moved = [k for k in keys if hrw(k, weights) != first[k]]
print(all(first[k] == "medium" for k in moved)) # True-w / ln(u) score gives weighted rendezvous hashing with minimal movement.
0 comentários
Fazer login · Faça login para deixar um comentário.
Seja o primeiro a comentar.