Kafka Consumer

This node allows to consume messages from a Kafka cluster for given topics and to store them in a table.
Note: Offsets are committed at the end of the nodes execution as well as whenever the partitions become reassigned.

Options

Settings

Client ID

An id string to pass to the Kafka cluster when making requests. This string can be included as a logical application name in server-side request logging, thus making it easier to track the source of requests beyond ip/port.

Group ID

A unique string that identifies the consumer group this consumer belongs to.

Topic(s)

A pattern or list of topics the consumer subscribes to.

A pattern has to be a regular expression as defined here. The topics list has to be entered in the form topic1,topic2,....

Topic is pattern

If this option is selected, the value of topics will be treated as a pattern instead of a list.

Append message info columns

If this option is selected, additional columns are added to the output table containing detailed information about the message/row. More precisely the following columns are added: Topic, Partition, Offset, Message creation date.

Convert message to JSON

If this option is selected, the consumed messages are converted from String to JSON.

Max number of messages per poll

The maximum number of messages polled in a single request.

Poll timeout (ms)

The time, in milliseconds, spent waiting for messages in a poll request.

Stop Criterion

Determines how to stop the execution of this node.

• Stop when number of consumed messages exceeds This node will stop once no more messages are available, but at the latest after the specified number of messages were consumed.
• Stop when message timestamp exceeds Consumes all messages created up until the selected date & time. If the selected date & time is in the future, the execution stops once a poll request returns no messages.
  Selecting <Use execution date & time> consumes all messages up until the date & time the node is executed. The <Current time> button sets the date & time to the current local time.

Advanced Settings

Advanced Settings

Allows to specify additional consumer related options, e.g., poll interval and session timeout. For further details on the individual options see the tooltip or consult the official Kafka documentation.

Views

This node has no views

Workflows

• 01_MDF_meets_Apache_KafkaKNIME Hub
• 3. Model Deployment with kafkaKNIME Hub
• 4. Retrain ModelKNIME Hub
• Kafka_demoKNIME Hub
• Kafka_integrationKNIME Hub

Links

• No links available

Developers