Protobuf Schema Evolution Guide

Schema evolution is probably the biggest challenge with Protocol Buffers. You need to change your data structure, but you can't break the thousands of clients already running old code. Get it wrong and everything breaks. Get it right and you can evolve gracefully.

This guide shows you exactly how to add fields, remove fields, and modify your schemas without causing disasters. We'll use real examples and explain what works, what doesn't, and why.

The Golden Rules of Schema Evolution

Adding New Fields (Safe)

Adding fields is the safest operation. Old code ignores new fields, new code handles missing fields.

message Subscriber {
  string msisdn = 1;
  string name = 2;
  bool is_active = 3;
}

message Subscriber {
  string msisdn = 1;
  string name = 2;
  bool is_active = 3;
  string email = 4;        // New field - totally safe!
  string address = 5;      // Can keep adding more
}

Removing Fields (The Right Way)

You can't actually "remove" fields - but you can stop using them. Here's the safe migration path:

message Subscriber {
  string msisdn = 1;
  string name = 2;
  string old_status = 3;  // We want to remove this
}

message Subscriber {
  string msisdn = 1;
  string name = 2;
  string old_status = 3 [deprecated = true];  // Mark deprecated
  SubscriberStatus new_status = 4;             // Add replacement
}

message Subscriber {
  reserved 3;                       // Never use this number again
  reserved "old_status";            // Never use this name again
  
  string msisdn = 1;
  string name = 2;
  SubscriberStatus new_status = 4;  // Only the new field remains
}

Renaming Fields (Surprisingly Easy)

Field names are just for code generation. The wire format uses numbers. So renaming is actually safe!

message Subscriber {
  string phone_number = 1;  // Unclear name
  string customer_name = 2;
}

message Subscriber {
  string msisdn = 1;  // Renamed - totally safe!
  string name = 2;    // Also renamed - no problem!
}

Changing Field Types (Dangerous!)

Changing types is risky. Some changes work, others cause data corruption. Here's the breakdown:

message Subscriber {
  string user_id_old = 1 [deprecated = true];  // Was string
  int64 user_id = 2;                            // Now integer
  // Migrate over time, then reserve field 1
}

Evolving Enums

Enums have special rules. Old code must handle new enum values gracefully.

enum SubscriptionType {
  PREPAID = 0;
  POSTPAID = 1;
}

enum SubscriptionType {
  PREPAID = 0;
  POSTPAID = 1;
  CORPORATE = 2;   // New value - safe!
  ENTERPRISE = 3;  // Can keep adding
}

enum SubscriptionType {
  reserved 2;              // Don't reuse this number
  reserved "DEPRECATED";   // Don't reuse this name
  
  PREPAID = 0;
  POSTPAID = 1;
  CORPORATE = 3;  // Skip 2 forever
}

Refactoring Message Structures

Sometimes you need to restructure - split messages apart or combine them. Here's how:

message Subscriber {
  string msisdn = 1;
  string name = 2;
  string street = 3;
  string city = 4;
  string country = 5;
}

message Address {
  string street = 1;
  string city = 2;
  string country = 3;
}

message Subscriber {
  string msisdn = 1;
  string name = 2;
  
  // Keep old fields for backward compat
  string street = 3 [deprecated = true];
  string city = 4 [deprecated = true];
  string country = 5 [deprecated = true];
  
  // New nested structure
  Address address = 6;
}

// Migration: Write to both old and new fields
// Read from new field first, fall back to old fields

Testing Schema Changes

Always test compatibility before deploying. Here's a simple test pattern:

# Python - Test backward compatibility
import subscriber_v1_pb2
import subscriber_v2_pb2

def test_forward_compatibility():
    """Old code can read new messages"""
    # Create message with v2 (new schema)
    v2_msg = subscriber_v2_pb2.Subscriber()
    v2_msg.msisdn = "+91-9876543210"
    v2_msg.name = "Test User"
    v2_msg.email = "[email protected]"  # New field
    
    # Serialize with v2
    data = v2_msg.SerializeToString()
    
    # Parse with v1 (old schema)
    v1_msg = subscriber_v1_pb2.Subscriber()
    v1_msg.ParseFromString(data)  # Should not crash!
    
    # Old fields should work
    assert v1_msg.msisdn == "+91-9876543210"
    assert v1_msg.name == "Test User"
    # v1 just ignores the email field

def test_backward_compatibility():
    """New code can read old messages"""
    # Create message with v1 (old schema)
    v1_msg = subscriber_v1_pb2.Subscriber()
    v1_msg.msisdn = "+91-9876543210"
    v1_msg.name = "Test User"
    
    # Serialize with v1
    data = v1_msg.SerializeToString()
    
    # Parse with v2 (new schema)
    v2_msg = subscriber_v2_pb2.Subscriber()
    v2_msg.ParseFromString(data)  # Should not crash!
    
    # Old fields should work
    assert v2_msg.msisdn == "+91-9876543210"
    assert v2_msg.name == "Test User"
    # New fields are empty/default
    assert v2_msg.email == ""  # Default value

Evolution Best Practices Checklist

Related Resources

External References

Conclusion

Schema evolution doesn't have to be scary. Follow the golden rules: never change field numbers, always add (never remove), and test both directions. When in doubt, add a new field instead of modifying existing ones.

Plan your migrations carefully. Keep both old and new fields during transitions. Use deprecation warnings to guide developers. And always, always test compatibility before deploying to production. These practices will save you from painful debugging sessions at 2 AM.