Skip to content

Cloudflare Based Multi Machine Sync

Jump to bottom
Henry edited this page Sep 20, 2025 · 2 revisions

Cloudflare-Based Multi-Machine Memory Sync

New in v6.13.7: Complete solution for bidirectional memory synchronization across multiple machines using Cloudflare as a central hub.

🎯 Problem Statement

Common Scenarios

  1. Central Server Failures: Narrowbox or dedicated server hardware failures
  2. Multi-Machine Development: Working across laptop, desktop, and remote machines
  3. Team Collaboration: Sharing memory context across team members
  4. Disaster Recovery: Need for robust backup and sync strategies

Before v6.13.7

  • Single Machine Limitation: sqlite_vec and ChromaDB backends were machine-specific
  • Manual Export/Import: Required manual JSON export/import for memory transfer
  • No Real-time Sync: Changes on one machine didn't appear on others
  • Central Server Dependency: Relied on dedicated hardware for HTTP-based sharing

🚀 Solution Architecture

Cloudflare as Central Hub

┌─────────────────────┐                ┌─────────────────────┐
│    MacBook Pro      │                │     M2 MacBook      │
│                     │                │                     │
│  ┌─────────────────┐│                │┌─────────────────┐  │
│  │ Claude Desktop  ││    ↕ SYNC ↕    ││ Claude Desktop  │  │
│  │                 ││                ││                 │  │
│  │ MCP Memory      ││                ││ MCP Memory      │  │
│  │ Service         ││                ││ Service         │  │
│  └─────────────────┘│                │└─────────────────┘  │
│           ↕         │                │          ↕          │
│  ┌─────────────────┐│                │┌─────────────────┐  │
│  │ sqlite_vec      ││                ││ sqlite_vec      │  │
│  │ (Local Backup)  ││                ││ (Local Backup)  │  │
│  └─────────────────┘│                │└─────────────────┘  │
└───────────┬─────────┘                └─────────┬───────────┘
            │                                    │
            └─────────────────┬──────────────────┘
                              ↕
              ┌─────────────────────────────────┐
              │         CLOUDFLARE              │
              │                                 │
              │  ┌─────────────────────────────┐│
              │  │       D1 Database           ││
              │  │   (Metadata & Content)      ││
              │  └─────────────────────────────┘│
              │                                 │
              │  ┌─────────────────────────────┐│
              │  │     Vectorize Index         ││
              │  │  (768-dim Embeddings)       ││
              │  └─────────────────────────────┘│
              │                                 │
              │  ┌─────────────────────────────┐│
              │  │      Workers AI             ││
              │  │ (@cf/baai/bge-base-en-v1.5) ││
              │  └─────────────────────────────┘│
              │                                 │
              │  ┌─────────────────────────────┐│
              │  │     R2 Storage              ││
              │  │   (Large Content)           ││
              │  └─────────────────────────────┘│
              └─────────────────────────────────┘

Key Benefits

  • ✅ Real-time Sync: Changes appear instantly across all machines
  • ✅ Global CDN: Fast access from anywhere in the world
  • ✅ Automatic Scaling: No server management required
  • ✅ Cost Effective: Pay-per-use pricing model
  • ✅ Backup Strategy: Local sqlite_vec files provide fallback
  • ✅ Team Collaboration: Easy sharing across team members

🛠️ Implementation Guide

Prerequisites

  1. Cloudflare Account with Workers/D1/Vectorize access
  2. API Token with permissions:
    • Vectorize:Edit
    • D1:Edit
    • Workers AI:Read
    • R2:Edit (optional)
  3. MCP Memory Service v6.13.7+ on all machines

Step 1: Cloudflare Resource Setup

# Install Wrangler CLI
npm install -g wrangler

# Login to Cloudflare
wrangler auth login

# Create Vectorize index
wrangler vectorize create mcp-memory-index \
  --dimensions=768 \
  --metric=cosine

# Create D1 database
wrangler d1 create mcp-memory-db

# Optional: Create R2 bucket for large content
wrangler r2 bucket create mcp-memory-content

Step 2: Export Existing Memories

From your primary machine (the one with existing memories):

# Using built-in export command
memory export /tmp/memories_export.json

# Or using export script
python scripts/sync/export_memories.py

Step 3: Migration Script

Create a migration script to handle the Cloudflare import:

#!/usr/bin/env python3
"""Migrate memories to Cloudflare with proper ID handling"""

import asyncio
import json
import sys
from pathlib import Path

# Add project to Python path
sys.path.insert(0, str(Path(__file__).parent.parent))

from src.mcp_memory_service.storage.cloudflare import CloudflareStorage
from src.mcp_memory_service.models.memory import Memory

async def migrate_to_cloudflare(export_file: str):
    # Initialize Cloudflare storage
    storage = CloudflareStorage(
        api_token="your-api-token",
        account_id="your-account-id",
        vectorize_index="mcp-memory-index",
        d1_database_id="your-d1-database-id"
    )

    await storage.initialize()

    # Load exported memories
    with open(export_file, 'r') as f:
        data = json.load(f)

    memories = data['memories']
    print(f"Migrating {len(memories)} memories to Cloudflare...")

    success_count = 0
    for i, mem_data in enumerate(memories):
        try:
            memory = Memory(
                content=mem_data['content'],
                content_hash=mem_data['content_hash'],
                tags=mem_data.get('tags', []),
                memory_type=mem_data.get('memory_type'),
                metadata=mem_data.get('metadata', {})
            )

            success, message = await storage.store(memory)
            if success:
                success_count += 1
            else:
                print(f"Failed to store memory {i}: {message}")

            if (i + 1) % 50 == 0:
                print(f"Progress: {i+1}/{len(memories)} ({success_count} successful)")

        except Exception as e:
            print(f"Error processing memory {i}: {e}")

    print(f"Migration complete: {success_count}/{len(memories)} successful")
    await storage.close()

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python migrate_to_cloudflare.py <export_file.json>")
        sys.exit(1)

    asyncio.run(migrate_to_cloudflare(sys.argv[1]))

Step 4: Configure Each Machine

Update Claude Desktop configuration on each machine:

Location:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
  • Windows: %APPDATA%\Claude\claude_desktop_config.json
  • Linux: ~/.config/claude-desktop/claude_desktop_config.json

Configuration:

{
  "mcpServers": {
    "memory": {
      "command": "/path/to/memory",
      "args": ["server"],
      "env": {
        "MCP_MEMORY_STORAGE_BACKEND": "cloudflare",
        "MCP_MEMORY_SQLITE_PATH": "/path/to/local/backup/sqlite_vec.db",
        "CLOUDFLARE_API_TOKEN": "your-cloudflare-api-token",
        "CLOUDFLARE_ACCOUNT_ID": "your-cloudflare-account-id",
        "CLOUDFLARE_D1_DATABASE_ID": "your-d1-database-id",
        "CLOUDFLARE_VECTORIZE_INDEX": "mcp-memory-index"
      }
    }
  }
}

Step 5: Verification Testing

Create verification scripts to test bidirectional sync:

#!/usr/bin/env python3
"""Test bidirectional sync between machines"""

import asyncio
from datetime import datetime
from mcp_memory_service.storage.cloudflare import CloudflareStorage
from mcp_memory_service.models.memory import Memory
from mcp_memory_service.utils.hashing import generate_content_hash

async def test_bidirectional_sync():
    storage = CloudflareStorage(
        api_token="your-api-token",
        account_id="your-account-id",
        vectorize_index="mcp-memory-index",
        d1_database_id="your-d1-database-id"
    )

    await storage.initialize()

    # Test 1: Store a memory from this machine
    machine_id = "Machine-A"  # Change for each machine
    test_content = f"Sync test from {machine_id} at {datetime.now()}"
    test_memory = Memory(
        content=test_content,
        content_hash=generate_content_hash(test_content),
        tags=["sync-test", machine_id.lower()],
        memory_type="test"
    )

    print(f"Storing test memory from {machine_id}...")
    success, message = await storage.store(test_memory)

    if success:
        print(f"✅ Memory stored: {message}")
    else:
        print(f"❌ Failed to store: {message}")
        return

    # Test 2: Retrieve memories from other machines
    print("\\nSearching for memories from other machines...")
    results = await storage.retrieve("sync test", n_results=10)

    print(f"Found {len(results)} sync test memories:")
    for result in results:
        print(f"  - {result.memory.content}")

    # Test 3: Check total memory count
    stats = await storage.get_stats()
    print(f"\\nTotal memories in Cloudflare: {stats['total_memories']}")

    await storage.close()

if __name__ == "__main__":
    asyncio.run(test_bidirectional_sync())

🔧 Troubleshooting

Common Issues

1. Vector ID Length Error (Fixed in v6.13.7)

Error:

HTTP 400: {"errors":[{"code":40008,"message":"id too long; max is 64 bytes, got 68 bytes"}]}

Solution:

  • Upgrade to MCP Memory Service v6.13.7 or later
  • The fix removes the "mem_" prefix from vector IDs

2. Environment Variables Not Loading

Problem: Memories not syncing between machines

Debugging:

# Check environment variables
echo $CLOUDFLARE_API_TOKEN
echo $CLOUDFLARE_ACCOUNT_ID
echo $MCP_MEMORY_STORAGE_BACKEND

# Test Cloudflare connectivity
memory status

Solution:

  • Verify environment variables are set correctly
  • Restart Claude Desktop after configuration changes
  • Check for typos in configuration file

3. API Token Permissions

Error:

HTTP 403: Insufficient permissions

Solution:

  • Verify API token has all required permissions:
    • Vectorize:Edit
    • D1:Edit
    • Workers AI:Read
    • R2:Edit (if using R2)

4. Resource Not Found

Error:

HTTP 404: Vectorize index 'mcp-memory-index' not found

Solution:

  • Verify resource names match exactly
  • Check resources were created in correct Cloudflare account
  • Confirm account ID is correct

Verification Commands

# Check memory service status
memory status

# Test memory storage
echo "Test memory from $(hostname)" | memory store --tags "sync-test"

# Test memory retrieval
memory retrieve "sync test"

# Check memory count
memory status | grep "Total memories"

📊 Performance Characteristics

Typical Latency

  • Memory Storage: ~200-400ms (including embedding generation)
  • Memory Retrieval: ~100-200ms for semantic search
  • Cross-machine Sync: Immediate (real-time)
  • Global Access: <100ms from most locations

Scaling Limits

  • D1 Database: 100k reads/sec, 1k writes/sec
  • Vectorize: 30k operations/minute
  • Workers AI: 1k requests/minute (embedding generation)
  • R2 Storage: Virtually unlimited

Cost Estimation

For typical usage (1000 memories, 100 operations/day):

  • D1: ~$0.01/month
  • Vectorize: ~$0.05/month
  • Workers AI: ~$0.10/month
  • R2: ~$0.02/month
  • Total: ~$0.18/month

🔒 Security Considerations

API Token Security

  • Store API tokens securely (environment variables, not code)
  • Use minimum required permissions
  • Rotate tokens periodically
  • Consider separate tokens per machine

Data Privacy

  • Cloudflare processes data in compliance with GDPR/CCPA
  • Data is encrypted in transit and at rest
  • Consider data residency requirements
  • Review Cloudflare's privacy policies

Access Control

  • API tokens provide account-level access
  • Consider Cloudflare Access for additional security
  • Monitor usage through Cloudflare dashboard
  • Set up alerts for unusual activity

🚀 Advanced Configurations

Team Collaboration Setup

{
  "mcpServers": {
    "team-memory": {
      "command": "/path/to/memory",
      "args": ["server"],
      "env": {
        "MCP_MEMORY_STORAGE_BACKEND": "cloudflare",
        "CLOUDFLARE_API_TOKEN": "team-shared-token",
        "CLOUDFLARE_ACCOUNT_ID": "team-account-id",
        "CLOUDFLARE_D1_DATABASE_ID": "team-memory-db",
        "CLOUDFLARE_VECTORIZE_INDEX": "team-memory-index"
      }
    }
  }
}

Development vs Production

Use different Cloudflare resources for different environments:

# Development
export CLOUDFLARE_D1_DATABASE_ID="dev-memory-db"
export CLOUDFLARE_VECTORIZE_INDEX="dev-memory-index"

# Production
export CLOUDFLARE_D1_DATABASE_ID="prod-memory-db"
export CLOUDFLARE_VECTORIZE_INDEX="prod-memory-index"

Backup Automation

#!/usr/bin/env python3
"""Automated backup from Cloudflare to local storage"""

import asyncio
import json
from datetime import datetime
from pathlib import Path

async def backup_cloudflare_to_local():
    # Implementation for regular backups
    # Export from Cloudflare -> Save to local sqlite_vec
    pass

if __name__ == "__main__":
    asyncio.run(backup_cloudflare_to_local())

📚 Related Documentation

🎯 Success Stories

Case Study: Failed Narrowbox Recovery

Scenario: Development team's narrowbox server (10.0.1.30) failed due to power issues

Challenge:

  • 978 memories stored locally on M2 MacBook
  • Old MacBook Pro configured for narrowbox access
  • Need immediate restoration of cross-machine access

Solution Implemented:

  1. Exported memories from M2 MacBook using scripts/sync/export_memories.py
  2. Created Cloudflare D1 database and Vectorize index
  3. Migrated all memories using custom import script with v6.13.7 ID fix
  4. Updated Claude Desktop configurations on both machines
  5. Verified bidirectional sync with test memories

Results:

  • ✅ 1062 memories successfully synced (978 original + test memories)
  • ✅ Zero data loss
  • ✅ Bidirectional sync working perfectly
  • ✅ 15-minute implementation time
  • ✅ More reliable than previous narrowbox setup

Lessons Learned:

  • Cloudflare provides better reliability than single-point-of-failure servers
  • v6.13.7 ID fix was crucial for success
  • Dual storage strategy (Cloudflare + local backup) provides best of both worlds

Note: This guide is based on real-world implementation and testing. All examples have been verified to work with MCP Memory Service v6.13.7.

Clone this wiki locally