Schema Registry

In a large-scale system, your data format will inevitably change. A new field is added, an old field is renamed. If the Producer changes its format but the Consumer isn’t ready, the consumer will crash. This is known as Topic Poisoning. The Schema Registry prevents this by enforcing a formal data contract.

1. How it Works

Instead of sending the entire schema with every message (which is wasteful), the producer sends a small Schema ID.

  1. Registering: The Producer sends the schema to the Schema Registry.
  2. ID Mapping: The Registry gives the Producer a unique ID for that schema.
  3. Sending: The Producer sends the data (e.g., in Avro format) prepended with the Schema ID.
  4. Retrieving: The Consumer sees the ID, fetches the corresponding schema from the Registry, and uses it to deserialize the data.

2. Supported Formats

  • Avro: The most common format in the Kafka ecosystem. It is compact and supports evolution.
  • Protobuf: Google’s format, widely used for high-performance microservices.
  • JSON Schema: Allows you to use standard JSON while still having a schema for validation.

3. The “Poison Pill” Problem

Without a registry, a developer might accidentally send a String into a topic that everyone expects to be an Integer.

  • The Registry Solution: If a producer tries to send a message that doesn’t match the registered schema, the Registry (or the client library) will reject the write before it ever reaches Kafka.

4. Interactive: Schema Validator

Try to send data that doesn’t match the schema.

Schema: { "id": INT, "name": STRING }
Input Data
Waiting for data...

5. Summary

The Schema Registry is the “Librarian” of your data ecosystem. It ensures that your services can communicate reliably over time, providing the foundation for Evolution without the risk of breaking production downstream.