Producer

Optimal LingerMs Setting

A commonly adjusted configuration property on the producer is LingerMs - the minimum time between batches of messages being sent to the cluster. Larger values allow for more batching, which increases throughput. Smaller values may reduce latency. The tradeoff is complicated somewhat as throughput increases though because a smaller LingerMs will result in more broker requests, putting greater load on the server, which in turn may reduce throughput.

A good general purpose setting is 5 (the default is 0.5).

Disabling Nagle

If your throughput is very low, and you really care about latency, you should probably set SocketNagleDisable to true.

How can I ensure delivery of messages in the order specified?

Set EnableIdempotence to true. This has very little overhead - there's not much downside to having this on by default

Before the idempotent producer was available, you could achieve this by setting max.in.flight to 1 (at the expense of reduced throughput).

Messages are sent in the same order as ProduceAsync are called. However, in case of failure (temporary network failure for example), Confluent.Kafka will try to resend the data, which can cause reordering (note that this is not happening frequently, only in case of failure).

Important notes

• If librdkafka can't send the message or there was an error, ProduceAsync still work, but the Message received will have a non-null Error
• If you active set delivery.report.only.error or call ProduceAsync with disableDeliveryCallback, librdkafka will not call callbacks on successfull produced messages. You must not use the Task ProduceAsync in this case, as the task will never complete (it will fail-fast, event ith). You should not use a deliveryhandler which await a callback for every message sent.
• At current time, ProduceAsync may throw exception in some case (message too big for example). This may change in a future release (no throw but returns a message with Error instead)
• When a message fail to be reported, there are two ways to be informed of this:
  • The ProduceAsync fail (fail-fast : the message wasn't even sent to the broker)
  • The Message passed to IDeliveryHandler has a Message.Error different from Error.NoError