Cross-Region Consistency

The Interview Question

"Our app is deployed in US, EU, and Asia regions. A user updates their profile in EU, then immediately views it from a US server and sees stale data. How do we achieve strong consistency across regions without killing performance?"

Asked at: Google, Netflix, Meta, any global company

Time to solve: 35-40 minutes

Difficulty: ⭐⭐⭐⭐⭐ (Staff/Principal)

---

Clarifying Questions to Ask

1. "What's the acceptable latency?" → Strong consistency costs ~100-200ms cross-region
2. "What's the data update frequency?" → Hot data vs cold data
3. "Is stale read ever acceptable?" → Profile vs shopping cart vs payment
4. "What's the consistency window tolerance?" → Seconds vs minutes
5. "Are users typically stationary or mobile?" → Session affinity helps

---

The Problem Visualized

---

Solution Approaches

Approach 1: Session Affinity (Sticky Sessions)

Route user requests to the same region consistently:

class RegionAwareRouter:
    def route_request(self, request):
        user_id = request.user_id
        
        # Option 1: Hash-based region assignment
        primary_region = self.get_primary_region(user_id)
        
        # Option 2: Use user's home region
        user = self.user_cache.get(user_id)
        primary_region = user.home_region if user else self.detect_region(request.ip)
        
        # Route to primary region for writes AND reads
        return self.forward_to_region(request, primary_region)
    
    def get_primary_region(self, user_id: str) -> str:
        # Consistent hashing to assign user to region
        hash_value = hash(user_id) % 3
        return ['us-east', 'eu-west', 'ap-south'][hash_value]

Pros: Simple, no cross-region coordination
Cons: Latency if user travels, doesn't help with truly global data

Approach 2: Read-Your-Writes Consistency

After a write, ensure subsequent reads see that write:

class ReadYourWritesConsistency:
    def __init__(self):
        self.write_tokens = {}  # {user_id: {region: version}}
    
    def write(self, user_id: str, data: dict, region: str):
        # Write to local region
        version = self.db.write(user_id, data)
        
        # Store write token for this user
        self.write_tokens.setdefault(user_id, {})[region] = version
        
        # Return token to client (via cookie or header)
        return WriteToken(region=region, version=version)
    
    def read(self, user_id: str, write_token: WriteToken, region: str):
        if write_token and write_token.region != region:
            # User wrote to different region
            # Wait for replication or forward to write region
            
            local_version = self.db.get_version(user_id)
            
            if local_version < write_token.version:
                # Stale! Options:
                # 1. Wait for replication
                # 2. Forward to write region
                # 3. Return stale with warning
                
                return self.forward_read_to_region(
                    user_id, 
                    write_token.region
                )
        
        return self.db.read(user_id)

Client-side implementation:

// Frontend stores write token
class APIClient {
  async updateProfile(data) {
    const response = await fetch('/api/profile', {
      method: 'PUT',
      body: JSON.stringify(data),
    });
    
    // Store write token from response
    const writeToken = response.headers.get('X-Write-Token');
    localStorage.setItem('write-token', writeToken);
  }
  
  async getProfile() {
    const writeToken = localStorage.getItem('write-token');
    
    const response = await fetch('/api/profile', {
      headers: {
        'X-Write-Token': writeToken || '',
      },
    });
    
    return response.json();
  }
}

Approach 3: Synchronous Cross-Region Writes (Paxos/Raft)

Strong consistency via consensus:

class CrossRegionConsensus:
    """
    Use distributed consensus for strongly consistent writes.
    Examples: CockroachDB, Spanner, YugabyteDB
    """
    
    def write(self, key: str, value: dict):
        # Leader proposes write
        leader = self.get_leader(key)
        
        # Wait for majority of regions to acknowledge
        acks = self.propose_to_all_regions(key, value)
        
        if len(acks) >= self.quorum_size:
            # Majority agreed - commit
            self.commit_all_regions(key, value)
            return True
        else:
            # Consensus failed - abort
            raise ConsensusError("Failed to achieve quorum")
    
    def read(self, key: str, consistency: str = 'strong'):
        if consistency == 'strong':
            # Read from leader to ensure latest
            leader = self.get_leader(key)
            return leader.read(key)
        else:
            # Read from local replica (eventual consistency)
            return self.local_db.read(key)

Spanner-style TrueTime:

class SpannerStyle:
    """
    Google Spanner approach:
    - GPS + atomic clocks for synchronized time
    - Commit timestamps guarantee ordering
    """
    
    def write(self, key: str, value: dict):
        # Get globally synchronized timestamp
        commit_time = self.true_time.now()
        
        # Wait for uncertainty interval
        # (ensures no other region can have earlier timestamp)
        self.true_time.wait_until_certain(commit_time)
        
        # Write with timestamp
        self.db.write(key, value, timestamp=commit_time)
        
        return commit_time
    
    def read(self, key: str, at_timestamp: int = None):
        if at_timestamp:
            # Point-in-time read
            return self.db.read_at(key, at_timestamp)
        else:
            # Current read - wait for any pending writes
            return self.db.read_latest(key)

Approach 4: CRDTs for Eventual Consistency

For data that can be merged automatically:

from collections import Counter

class CRDTCounter:
    """
    Grow-only counter that merges across regions.
    No conflicts possible.
    """
    def __init__(self):
        self.counts = {}  # {region: count}
    
    def increment(self, region: str, amount: int = 1):
        self.counts[region] = self.counts.get(region, 0) + amount
    
    def merge(self, other: 'CRDTCounter'):
        for region, count in other.counts.items():
            self.counts[region] = max(
                self.counts.get(region, 0),
                count
            )
    
    def value(self) -> int:
        return sum(self.counts.values())

class CRDTLastWriterWins:
    """
    Last-writer-wins register for simple values.
    Uses timestamps for conflict resolution.
    """
    def __init__(self):
        self.value = None
        self.timestamp = 0
    
    def set(self, value, timestamp: int):
        if timestamp > self.timestamp:
            self.value = value
            self.timestamp = timestamp
    
    def merge(self, other: 'CRDTLastWriterWins'):
        if other.timestamp > self.timestamp:
            self.value = other.value
            self.timestamp = other.timestamp

Approach 5: Hybrid Approach (Per-Data-Type)

Different consistency for different data:

class HybridConsistencyManager:
    def __init__(self):
        self.strong_consistency_tables = {
            'payments', 'orders', 'inventory'
        }
        self.eventual_consistency_tables = {
            'user_preferences', 'analytics', 'notifications'
        }
    
    def write(self, table: str, data: dict, region: str):
        if table in self.strong_consistency_tables:
            # Use synchronous cross-region write
            return self.consensus_write(table, data)
        else:
            # Write locally, replicate async
            return self.local_write(table, data, region)
    
    def read(self, table: str, key: str, region: str, write_token=None):
        if table in self.strong_consistency_tables:
            # Always read latest (may have cross-region latency)
            return self.consensus_read(table, key)
        else:
            # Read-your-writes for user-facing data
            if write_token:
                return self.read_your_writes(table, key, write_token, region)
            else:
                return self.local_read(table, key, region)

---

Trade-offs Comparison

Approach	Consistency	Latency (write)	Latency (read)	Complexity
Session affinity	Session-level	Low	Low	Low
Read-your-writes	Per-user	Low	Low (usually)	Medium
Synchronous consensus	Strong	High (100-200ms)	Low-Medium	High
CRDTs	Eventual (no conflicts)	Low	Low	Medium
Hybrid	Varies	Varies	Varies	High

---

Real-World Example: User Profile Update

class GlobalUserService:
    """
    Hybrid approach for user profiles:
    - Critical fields (email, payment): Strong consistency
    - Preferences (theme, language): Eventual consistency
    - Activity (last seen): CRDT
    """
    
    def update_email(self, user_id: str, new_email: str):
        # Strong consistency - use Spanner/CockroachDB
        with self.global_db.transaction() as txn:
            # Verify email not taken globally
            existing = txn.query(
                "SELECT 1 FROM users WHERE email = ?", new_email
            )
            if existing:
                raise EmailTakenError()
            
            txn.execute(
                "UPDATE users SET email = ? WHERE id = ?",
                new_email, user_id
            )
        # Transaction commits synchronously across regions
    
    def update_preferences(self, user_id: str, prefs: dict, region: str):
        # Eventual consistency - write locally
        version = self.local_db.write(user_id, prefs)
        
        # Return token for read-your-writes
        return WriteToken(region=region, version=version)
    
    def record_activity(self, user_id: str, region: str):
        # CRDT - no conflicts
        activity = self.activity_crdt.get(user_id)
        activity.set_last_seen(region, time.time())
        self.activity_crdt.save(user_id, activity)

---

Architecture Diagram

---

Key Takeaways

1. No silver bullet - Strong consistency costs latency
2. Know your data - Not all data needs strong consistency
3. Read-your-writes is usually enough - Most users care about their own data
4. CRDTs for counters/flags - Automatic merge, no conflicts
5. Use managed solutions - Spanner, CockroachDB, Cosmos DB
6. Measure user impact - Sometimes eventual consistency is fine

---

CAP Theorem reminder: You can have at most 2 of: Consistency, Availability, Partition tolerance. In global systems, partitions WILL happen, so choose between C and A.