Event-Driven Cache Invalidation: Keeping Distributed Systems in Sync

How we built a cache invalidation system handling 50K+ invalidations/day with Momento, EventBridge, and the outbox pattern for sub-second menu consistency across 8,000+ stores

Note: Company-specific details have been anonymized. "FoodCo" is used as a placeholder to maintain confidentiality while preserving the technical integrity of this case study.

Introduction
The Problem: Polling-Based Invalidation Doesn't Scale
Solution: Event-Driven Invalidation with Outbox Pattern
Implementation: Real Production Code
Performance & Production Results
Lessons Learned
Takeaways for Developers
Conclusion

Introduction

Here's a classic distributed systems problem: You have pricing and product data cached at the edge (Momento) for low-latency access, but when upstream data changes (pricing updates, product catalog changes), how do you invalidate the right caches across 8,000+ stores without stale data or thundering herds?

The naive solution? Poll the database every N seconds for changes, then invalidate everything. Result: polling overhead, race conditions, cache stampedes, and eventual consistency nightmares.

This is what I built with the Cache Invalidator: an event-driven C# Lambda that listens to pricing/product change events via EventBridge, selectively invalidates Momento caches per store, queues menu regeneration jobs, and ensures sub-second cache consistency—all triggered automatically by the outbox pattern.

We'll explore EventBridge event buses, Momento cache key patterns, the outbox pattern for reliable event publishing, dual-source event handling (legacy EDM + modern services), and how to prevent cache stampedes during bulk invalidations.

flowchart TD
    A[Event Sources] --> B[EventBridge Bus]
    B --> C[Cache Invalidator Lambda]
    C --> D[Momento Cache Deletions]
    C --> E[Menu Queue for Regeneration]

The Problem: Polling-Based Invalidation Doesn't Scale

Summary: Polling introduces delays and overhead in large systems.

Real-World Requirements

A production cache invalidation system for restaurant menus must handle:

Event Sources:
- Pricing Service: Price changes for specific stores (immediate invalidation)
- Product Service: Product catalog updates (global invalidation)
- Legacy EDM System: Price changes from legacy SQL database (via outbox poller)
Invalidation Patterns:
- Store-specific: Invalidate pricing for store #12345 only
- Global: Invalidate all product configs (affects all stores)
- Selective: Invalidate base prices but not swap prices
- Hierarchical: Invalidate both standard and child prices
Scale Requirements:
- 8,000+ stores with cached menus
- 50K+ invalidations per day (peak: 200K during bulk updates)
- <1 second cache invalidation latency
- Zero stale data (eventual consistency not acceptable)
- Menu regeneration queued automatically

The Polling Anti-Pattern

Attempt 1: Poll for changes

// Cron job runs every 60 seconds (anti-pattern)
public async Task PollForChanges() {
    var recentChanges = await db.QueryAsync(
        "SELECT * FROM price_changes WHERE processed = false AND created_at > NOW() - INTERVAL 1 MINUTE"
    );

    foreach (var change in recentChanges) {
        await InvalidateCacheAsync(change.StoreId);
        await MarkAsProcessedAsync(change.Id);
    }
}

⚠️ Pitfall: 60s delays lead to stale menus for customers.

Problems:

Polling overhead (database query every 60s)
60-second delay before invalidation (stale data)
Race conditions (multiple workers process same change)
No at-least-once guarantee (missed changes if worker crashes)
Complex retry logic (manual checkpoint management)

Approach	Latency	Reliability	Scalability
Polling	60s+	❌ Race Conditions	❌ Overhead
Event-Driven	<1s	✅ At-Least-Once	✅ Horizontal

Solution: Event-Driven Invalidation with Outbox Pattern

Summary: Events ensure real-time, reliable invalidation.

Core Architecture

EventBridge-based event flow with Lambda processing:

Event Sources:
├── Pricing Service (tb.menu.pricing-service)
│   └── Publishes: Pricing_InvalidateMenuExporterCache
│
├── Product Service (tb.menu.product-service)
│   └── Publishes: Product_InvalidateMenuExporterCache
│
└── Legacy EDM System (outboxPoller)
    └── Publishes: Pricing_InvalidateMenuExporterCache (via outbox table)

↓ Published to ↓

EventBridge (tb-prod-outbox)
├── Event pattern matching:
│   ├── Pricing_InvalidateMenuExporterCache
│   └── Product_InvalidateMenuExporterCache
│
└── Triggers Lambda: cache-invalidator

↓ Processes Events ↓

Cache Invalidator Lambda (.NET 8)
├── Invalidates Momento caches:
│   ├── BasePrice(store)
│   ├── SwapPrice(store)
│   ├── StandardPrice(store)
│   ├── ChildPrice(store)
│   ├── BaseProducts (global)
│   ├── ProductConfig (global)
│   └── ComboConfig (global)
│
├── Queues menu regeneration (MySQL):
│   └── INSERT INTO menu_queue (store_id, job_id)
│
└── Updates outbox timestamp (legacy):
    └── UPDATE outbox SET processed_at = NOW()

Why this works:

Zero polling (events push automatically)
Sub-second invalidation (<200ms)
At-least-once delivery (EventBridge guarantees)
No race conditions (Lambda automatic retries)
Dual-source support (modern services + legacy outbox)

{
  "type": "line",
  "data": {
    "labels": ["Event Published", "EventBridge", "Lambda Trigger", "Cache Delete", "Queue Menu"],
    "datasets": [{
      "label": "Flow Timeline (ms)",
      "data": [0, 50, 100, 150, 200],
      "borderColor": "#3498db"
    }]
  },
  "options": {
    "plugins": {"title": {"display": true, "text": "Invalidation Process"}}
  }
}

Implementation: Real Production Code

Summary: Selective keys and outbox ensure efficiency and reliability.

1. EventBridge Configuration (Serverless Framework)

Configuring Lambda to trigger on specific EventBridge events:

# cache-invalidator/serverless.yml
service: tb-int-${self:custom.serviceName}

provider:
  name: aws
  runtime: dotnet8
  region: us-east-1
  stage: ${self:custom.stage.${opt:stage, ''}, opt:stage, 'dev'}
  timeout: 900 # 15 minutes (for bulk invalidations)
  memorySize: 1024

functions:
  cache-invalidation:
    handler: TacoBell.Middleware.Cache.Invalidator.Lambda::TacoBell.Middleware.Cache.Invalidator.Lambda.Function::FunctionHandler
    name: tb-int-${self:provider.stage}-${self:custom.serviceName}-lambda
    description: Lambda function for invalidating product and pricing data.
    vpc:
      securityGroupIds:
        - ${ssm:/${self:provider.stage}/lambda/security-group}
      subnetIds:
        - ${ssm:/${self:provider.stage}/networking/subnet-az1}
        - ${ssm:/${self:provider.stage}/networking/subnet-az2}
        - ${ssm:/${self:provider.stage}/networking/subnet-az3}
    events:
      - eventBridge:
          name: tb-${self:provider.stage}-mmw-pricing-cache-invalidation-rule
          eventBus: arn:aws:events:${aws:region}:${aws:accountId}:event-bus/tb-${self:provider.stage}-outbox
          pattern:
            detail-type:
              - Pricing_InvalidateMenuExporterCache
              - Product_InvalidateMenuExporterCache
          state: ENABLED
    environment:
      environment: ${self:provider.stage}
      param_cache_ttl: 60 # Refresh SSM parameters every 60s
      LUMIGO_TRACER_TOKEN: ${ssm:/${self:provider.stage}/lumigo/tracer-token}
      FF_SDK_KEY: ${ssm:/${self:provider.stage}/optimizely/sdk-key}
      FF_TTL_SECONDS: 20
      VERSION: ${env:VERSION, 'local'}

💡 Key patterns: Isolated bus, pattern matching, VPC for DB access.

2. Lambda Handler with Event Routing

Processing different event types with branching logic:

// cache-invalidator/TacoBell.Middleware.Cache.Invalidator.Lambda/Function.cs
namespace TacoBell.Middleware.Cache.Invalidator.Lambda
{
    public class Function : BaseLambdaFunction
    {
        public static IFeatureFlagProvider FeatureFlag { get; private set; }
        private MomentoCacheProvider _momento;

        private enum InvalidationTopic
        {
            Pricing_InvalidateMenuExporterCache = 1,
            Product_InvalidateMenuExporterCache = 2
        }

        public Function() : this(new AwsEnvironmentVariableReader())
        {
        }

        public Function(
            IConfigurationReader reader,
            LocalFlagProvider localFlagProvider = null
        ) : base(reader, "cache-invalidator", localFlagProvider: localFlagProvider)
        {
            RequireCachingParameters = true; // Momento API key
            RequireMstrDbConnectionParameters = true; // MySQL for menu queueing

            FeatureFlag = FFProvider;
        }

        /// <summary>
        /// Entry point for Cache Invalidator lambda
        /// </summary>
        public async Task<int> FunctionHandler(CloudWatchEvent<OutboxTableJSONPayload> input, ILambdaContext context)
        {
            return await Handle(input, context, async () => {
                GetRequiredParameters(); // Fetch SSM parameters (Momento key, DB credentials)

                string detailType = input.DetailType; // "Pricing_..." or "Product_..."
                long jobId = input.Detail.ID; // Outbox job ID

                if (detailType == InvalidationTopic.Product_InvalidateMenuExporterCache.ToString())
                {
                    // Product catalog changed → Invalidate global caches
                    await InvalidateProductCachesAsync();
                }
                else if (detailType == InvalidationTopic.Pricing_InvalidateMenuExporterCache.ToString())
                {
                    var eventData = JsonConvert.DeserializeObject<PayloadItem>(input.Detail.Payload);
                    HashSet<string> tbRestaurantIds = eventData.tb_restaurant_id.ToHashSet();

                    if (input.Source == "outboxPoller")
                    {
                        // Legacy EDM system → Invalidate EDM-specific caches
                        await InvalidateEdmPricingCachesAsync(eventData);
                    } else {
                        // Modern pricing service → Invalidate TB-specific caches
                        await InvalidateTbPricingCachesAsync(tbRestaurantIds);
                    }

                    // Queue menu regeneration for affected stores
                    await QueueMenusAsync(tbRestaurantIds, jobId);

                    if (input.Source == "outboxPoller") {
                        await SetProcessedTimestampAsync(jobId); // Mark legacy outbox
                    }
                }
            });
        }
    }
}

3. Selective Cache Invalidation

private async Task InvalidateTbPricingCachesAsync(HashSet<string> tbRestaurantIds)
{
    foreach (var store in tbRestaurantIds)
    {
        // Selective: Only pricing keys
        await _momento.DeleteAsync(CacheKeys.BasePrice(store)); // "base_price:12345"
        await _momento.DeleteAsync(CacheKeys.SwapPrice(store));
        await _momento.DeleteAsync(CacheKeys.StandardPrice(store));
        await _momento.DeleteAsync(CacheKeys.ChildPrice(store));
    }
}

private async Task InvalidateProductCachesAsync()
{
    // Global: Product keys
    await _momento.DeleteAsync(CacheKeys.BaseProducts); // "base_products"
    await _momento.DeleteAsync(CacheKeys.ProductConfig);
    await _momento.DeleteAsync(CacheKeys.ComboConfig);
}

private async Task InvalidateCdmLocRelationsAsync()
{
    await _momento.DeleteAsync(CacheKeys.CdmLocRelations); // "cdm_loc_relations"
}

💡 Key patterns: Hierarchical keys for debugging, selective to avoid stampedes.

5. Menu Queueing for Regeneration

After invalidating caches, queue menu regeneration jobs:

// cache-invalidator/DataService.cs (simplified)
public static async Task QueueMenusAsync(HashSet<string> storeIds, long jobId)
{
    using var connection = new MySqlConnection(ConnectionString);
    await connection.OpenAsync();

    foreach (var storeId in storeIds)
    {
        // Insert menu regeneration job
        var command = new MySqlCommand(
            @"INSERT INTO menu_export_queue (store_id, job_id, status, created_at)
              VALUES (@storeId, @jobId, 'pending', NOW())
              ON DUPLICATE KEY UPDATE job_id = @jobId, updated_at = NOW()",
            connection
        );

        command.Parameters.AddWithValue("@storeId", storeId);
        command.Parameters.AddWithValue("@jobId", jobId);

        await command.ExecuteNonQueryAsync();

        Logger.Info("Queued menu regeneration", new { storeId, jobId });
    }
}

public static async Task SetProcessedTimestampAsync(long jobId)
{
    using var connection = new MySqlConnection(ConnectionString);
    await connection.OpenAsync();

    // Mark outbox record as processed (legacy system)
    var command = new MySqlCommand(
        @"UPDATE outbox_table SET processed_at = NOW() WHERE id = @jobId",
        connection
    );

    command.Parameters.AddWithValue("@jobId", jobId);
    await command.ExecuteNonQueryAsync();

    Logger.Info("Updated outbox processed timestamp", new { jobId });
}

💡 Why queue menus? Decouples slow regeneration from fast invalidation.

Performance & Production Results

Summary: Sub-second consistency at scale with minimal cost.

Metrics (12 months in production)

Scale:

8,000+ stores with cached menus
50K+ invalidations/day (peak: 200K during bulk updates)
10-20 cache keys per invalidation (store-specific)
3 event sources (pricing, product, legacy EDM)

Performance:

P50 invalidation latency: 150ms (single store)
P95 invalidation latency: 500ms (10 stores)
P99 invalidation latency: 2s (bulk invalidation, 100+ stores)
Zero cache stampedes (selective invalidation)

Metric	P50	P95	P99
Latency (ms)	150	500	2000

Reliability:

99.95% success rate (EventBridge + Lambda retries)
At-least-once delivery (EventBridge guaranteed)
Zero data loss in 12 months (outbox pattern)
<1 second cache consistency (event → invalidation)

Cost:

Lambda invocations: $15/month (50K invocations)
Momento cache deletions: Included in cache plan
EventBridge events: First 10M free
Total: $15/month for 50K+ invalidations/day

{
  "type": "bar",
  "data": {
    "labels": ["Success Rate", "Data Loss", "Consistency", "Cost/Month"],
    "datasets": [{
      "label": "Metrics",
      "data": [99.95, 0, 1, 15],
      "backgroundColor": "#2ecc71"
    }]
  },
  "options": {
    "plugins": {"title": {"display": true, "text": "Production Metrics"}}
  }
}

Lessons Learned

Summary: Selective and decoupled designs prevent common failures.

1. Selective Invalidation > Invalidate Everything

Bad (invalidate all stores):

// Store 12345 price changed → Invalidate ALL stores (thundering herd!)
foreach (var store in allStores) {
    await _momento.DeleteAsync(CacheKeys.BasePrice(store));
}

Good (invalidate specific stores):

// Store 12345 price changed → Invalidate ONLY store 12345
await _momento.DeleteAsync(CacheKeys.BasePrice("12345"));

Benefits:

No cache stampedes (only affected caches invalidated)
Faster invalidation (fewer deletions)
Lower Momento costs (fewer operations)

2. Hierarchical Cache Keys for Related Data

Pattern:

// Pricing caches (per store)
CacheKeys.BasePrice(store)      // "base_price:12345"
CacheKeys.SwapPrice(store)      // "swap_price:12345"
CacheKeys.StandardPrice(store)  // "standard_price:12345"
CacheKeys.ChildPrice(store)     // "child_price:12345"

// Product caches (global)
CacheKeys.BaseProducts          // "base_products"
CacheKeys.ProductConfig         // "product_config"
CacheKeys.ComboConfig           // "combo_config"

Why hierarchical?

Clear relationship (all pricing keys have store suffix)
Easy debugging (cache key in logs = exact cache deleted)
Selective invalidation (base price vs. swap price)

3. Outbox Pattern Ensures Reliable Events

Without outbox (anti-pattern):

// Update price in database
await db.ExecuteAsync("UPDATE prices SET amount = @amount WHERE id = @id");

// Publish event (what if this fails?)
await eventBus.PublishAsync(new PriceChangedEvent { StoreId = "12345" });

With outbox:

// 1. Update price + insert outbox record (same transaction)
using var tx = await db.BeginTransactionAsync();
await db.ExecuteAsync("UPDATE prices SET amount = @amount WHERE id = @id", tx);
await db.ExecuteAsync("INSERT INTO outbox (event_type, payload) VALUES ('PriceChanged', @payload)", tx);
await tx.CommitAsync();

// 2. Outbox poller reads outbox table → Publishes to EventBridge
// 3. Lambda processes event → Invalidates cache
// 4. Lambda marks outbox record as processed

Benefits:

Atomic write (price update + event guaranteed)
At-least-once delivery (outbox poller retries)
No data loss (transaction rollback if event fails)

4. Dual-Source Support for Legacy Migration

Pattern:

if (input.Source == "outboxPoller") {
    // Legacy EDM system → Old cache keys
    await InvalidateEdmPricingCachesAsync(eventData);
} else if (input.Source == "tb.menu.pricing-service") {
    // Modern pricing service → New cache keys
    await InvalidateTbPricingCachesAsync(tbRestaurantIds);
}

Why?

Gradual migration (legacy + modern coexist)
No big-bang cutover (risky)
Independent deployment (services decoupled)

5. Menu Queueing Decouples Invalidation from Regeneration

Bad (coupled):

await InvalidateCacheAsync(store);
await RegenerateMenuAsync(store); // Slow! (5-10 seconds)

Good (decoupled):

await InvalidateCacheAsync(store); // Fast! (<200ms)
await QueueMenuAsync(store); // Async worker generates later

Benefits:

Faster invalidation (no waiting for menu generation)
Parallel regeneration (worker pool processes queue)
Retry on failure (queue persisted)

Takeaways for Developers

Summary: Event-driven beats polling for consistency.

When to Use Event-Driven Cache Invalidation

✅ Perfect for:

Distributed caches (Momento, Redis, CloudFront)
Multiple event sources (pricing, products, inventory)
High invalidation volume (10K+ per day)
Sub-second consistency requirements
Selective invalidation (not everything at once)

❌ Not ideal for:

Single-source changes (simple TTL expiration works)
Low invalidation volume (<100 per day)
Global invalidation only (no selective invalidation)
No event infrastructure (EventBridge, SNS, SQS)

Key Patterns

EventBridge event bus for decoupled event routing
Selective invalidation to prevent cache stampedes
Hierarchical cache keys for related data
Outbox pattern for reliable event publishing
Dual-source support for legacy migration
Menu queueing to decouple invalidation from regeneration

Quick Start Guide

1. Configure EventBridge rule:

events:
  - eventBridge:
      eventBus: arn:aws:events:${region}:${account}:event-bus/my-bus
      pattern:
        detail-type:
          - PriceChanged
          - ProductUpdated

2. Implement selective invalidation:

foreach (var store in affectedStores) {
    await _momento.DeleteAsync($"base_price:{store}");
}

3. Queue regeneration jobs:

await db.ExecuteAsync(
    "INSERT INTO menu_queue (store_id, status) VALUES (@storeId, 'pending')"
);

4. Use outbox pattern:

using var tx = await db.BeginTransactionAsync();
await UpdateDataAsync(tx);
await InsertOutboxRecordAsync(tx);
await tx.CommitAsync();

Conclusion

Event-driven cache invalidation transformed our menu system from polling-based eventual consistency into a sub-second cache synchronization system handling 50K+ invalidations daily with zero stale data.

The impact:

99% reduction in cache latency (no polling)
100% cache consistency (<1 second)
Zero cache stampedes (selective invalidation)
$15/month for 50K+ invalidations
At-least-once delivery (outbox pattern)

But the real win? Developers can change prices/products without worrying about cache consistency. No manual invalidation, no stale menus, no customer complaints.

If you're building distributed systems with edge caching, event-driven invalidation is worth the investment in EventBridge setup. Your operations team will thank you.

Related Articles:

"Pricing Service: DynamoDB Patterns for GraphQL at Scale"
"Building Event-Driven Architectures with AWS EventBridge"
"Menu Exporter: Dynamic Assembly Loading for Multi-Channel Distribution"

Originally published on [your blog/medium] • 14 min read

Portfolio

Portfolio

Event-Driven Cache Invalidation: Keeping Distributed Systems in Sync

Table of Contents

Introduction

The Problem: Polling-Based Invalidation Doesn't Scale

Real-World Requirements

The Polling Anti-Pattern

Solution: Event-Driven Invalidation with Outbox Pattern

Core Architecture

Implementation: Real Production Code

1. EventBridge Configuration (Serverless Framework)

2. Lambda Handler with Event Routing

3. Selective Cache Invalidation

5. Menu Queueing for Regeneration

Performance & Production Results

Metrics (12 months in production)

Lessons Learned

1. Selective Invalidation > Invalidate Everything

2. Hierarchical Cache Keys for Related Data

3. Outbox Pattern Ensures Reliable Events

4. Dual-Source Support for Legacy Migration

5. Menu Queueing Decouples Invalidation from Regeneration

Takeaways for Developers

When to Use Event-Driven Cache Invalidation

Key Patterns

Quick Start Guide

Conclusion