NestJS Kafka

Connecting Nest to Kafka

Use Transport.KAFKA with broker addresses and a client/consumer group id. Nest maps message and event patterns onto Kafka topics depending on configuration.

const app = await NestFactory.createMicroservice(AppModule, {
  transport: Transport.KAFKA,
  options: {
    client: { brokers: ['localhost:9092'], clientId: 'nest-consumer' },
    consumer: { groupId: 'nest-group' },
  },
});

Consumer groupId controls how partitions are balanced across scaled Nest instances.

Producing With ClientProxy

ClientsModule.register([{
  name: 'KAFKA_SERVICE',
  transport: Transport.KAFKA,
  options: { client: { brokers: ['localhost:9092'] }, consumer: { groupId: 'nest-producer' } },
}])

• Kafka shines for event sourcing, audit logs, and multi-consumer streams.
• Plan partition keys so related messages stay ordered when needed.
• Monitor consumer lag—falling behind is a first-class Kafka operational concern.
• Serialization (JSON vs Avro/Protobuf) should be an intentional contract.

Kafka Nest Cheatsheet

Operational basics for Nest + Kafka.

Ordering Guarantees

Kafka orders messages within a partition. Choose keys (like userId) intentionally when order matters for an aggregate.

Poison Messages

A repeatedly failing message can block a partition consumer. Use dead-letter topics and careful error handling strategies.

Common Mistakes

• Using a single partition for everything and losing parallelism.
• Changing consumer group ids accidentally and reprocessing history unexpectedly.
• Ignoring schema evolution for event payloads.
• Treating Kafka like a simple Redis queue without lag monitoring.

Key Takeaways

• Nest supports Kafka via Transport.KAFKA.
• Consumer groups scale consumption across instances.
• Partition keys control ordering of related events.
• Operate Kafka with lag and poison-message strategies in mind.