Fix docs on how dynamic topic can be set on kafka

[khushijain21]

Summary

The docs were incorrect on how to set dynamic topic in kafka . See filebeat-kafka-output for correct example on how topic should be set. Updated the same here.

[github-actions]

✅ Vale Linting Results

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[github-actions]

🔍 Preview links for changed docs

• reference/fleet/kafka-output-settings.md
• reference/fleet/kafka-output.md

[vishaangelova]

Hi @khushijain21, the file you are modifying is for setting up the Kafka output in a standalone Elastic Agent’s configuration. There is also another file which is for setting up the Kafka output in the Fleet UI (fleet/kafka-output-setting.md, in which the same syntax is used as for the standalone agent).

Some time ago, I updated both files in #3048 and #3356 to add information about setting dynamic topic values. At the time, I remember getting to the conclusion that there are small difference between how the Beats configurations and Elastic Agent configurations are handling dynamic values (see Variables and conditions in input configurations), hence the difference in the docs for the Kafka output configuration for Filebeat and for the standalone Elastic Agent.

Beats configuration example:

topic: '%{[fields.log_topic]}'

fields:
  log_topic: '%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}'

Elastic Agent configuration example:

topic: '${fields.kafka_topic}'

fields: 
  kafka_topic: '${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}'

(A different custom field name is used for the example in the Elastic Agent docs.)

I might be wrong and maybe this was a mistake on my side, but I’d like to get some clarification first.

@cmacknz is this something you can help with? Thank you!

[khushijain21]

${var} represents variable substituiton where var is the name of a variable defined by a provider.

But %{} is different - this is used to dynamically set value based on the event.

topic in Kafka is same as pipeline in elasticsearch

---

When this config is used topic: '${data_stream.type}'
we encounter below error in standalone elastic-agent

applying new policy: generating component model: rendering outputs failed: rendering output \"kafka-output\" failed: no matching vars: ${data_stream.type}

What I would have expected it is that events are correctly routed based on data stream type

[vishaangelova]

Hmm, I see… would that also be the case if the same syntax is used in the Fleet UI for a Fleet-managed agent, do you know? If yes, then we’ll also have to update the fleet/kafka-output-setting.md doc.

[khushijain21]

Yes I think this would be same syntax. See PR elastic/beats#40415 where similar syntax in fleet is used. Ill update the docs

[vishaangelova]

If the default is 10s (as your update for the standalone configuration suggests), maybe lines 194 (and maybe 197?) should also be updated?

[vishaangelova]

Suggested change

	The number of partitions created is set automatically by the Kafka broker based on the list of topics. Records are then published to partitions either randomly, in round-robin order, or according to a calculated hash. Default is hash partitioner
	The number of partitions created is set automatically by the Kafka broker based on the list of topics. Records are then published to partitions either randomly, in round-robin order, or according to a calculated hash. The default is hash partitioner.