Maximum Message Size

Kafka imposes a default size limit for individual messages sent to a topic. By default, messages cannot exceed 1048588 bytes in size.

However, this limit can be adjusted in the docker-compose file by modifying the following properties to accommodate larger message sizes. In this example, the maximum size is set to 20000000 bytes; adjust this value based on your needs.

In the broker service, add:

KAFKA_MESSAGE_MAX_BYTES: 20000000
KAFKA_MAX_REQUEST_SIZE: 20000000
KAFKA_PRODUCER_MAX_REQUEST_SIZE: 20000000
KAFKA_REPLICA_FETCH_MAX_BYTES: 20000000

In the connect service, add :

CONNECT_PRODUCER_MAX_REQUEST_SIZE: 20000000
CONNECT_CONSUMER_MAX_PARTITION_FETCH_BYTES: 20000000

In the ksqldb-server service, add :

KSQL_OPTS: "-Dmax.request.size=20000000"

Note: Adjusting message sizes beyond the default may negatively impact Kafka performance. Large messages often require a dedicated Kafka cluster, separate from clusters handling regular data, to manage these configurations efficiently.

Important: The connector relies on the max.message.bytes configuration of the output Kafka topic to determine the maximum allowable message size for aggregation results.

If an aggregation exceeds the size set by max.message.bytes, the connector will divide the aggregation into multiple messages. For instance, if max.message.bytes is set to 1000000 bytes, and the aggregation size is 1500000 bytes, the connector will split the aggregation, sending two messages: one of approximately 900000 bytes (leaving a buffer) and another of 600000 bytes.