0 ×

Kafka Consumer

StreamableApache Kafka integration version 4.2.0.v202006230509 by KNIME AG, Zurich, Switzerland

This node allows to consume messages from a Kafka cluster for given topics and to store them in a table.
Note: Offsets are committed at the end of the nodes execution as well as whenever the partitions become reassigned.

Options

Settings

Client ID
An id string to pass to the Kafka cluster when making requests. This string can be included as a logical application name in server-side request logging, thus making it easier to track the source of requests beyond ip/port.
Group ID
A unique string that identifies the consumer group this consumer belongs to.
Topic(s)

A pattern or list of topics the consumer subscribes to.

A pattern has to be a regular expression as defined here. The topics list has to be entered in the form topic1,topic2,....

Topic is pattern
If this option is selected, the value of topics will be treated as a pattern instead of a list.
Append message info columns
If this option is selected, additional columns are added to the output table containing detailed information about the message/row. More precisely the following columns are added: Topic, Partition, Offset, Message creation date.
Convert message to JSON
If this option is selected, the consumed messages are converted from String to JSON.
Max number of messages per poll
The maximum number of messages polled in a single request.
Poll timeout (ms)
The time, in milliseconds, spent waiting for messages in a poll request.
Stop Criterion
Determines how to stop the execution of this node.
  • Stop when number of consumed messages exceeds This node will stop once no more messages are available, but at the latest after the specified number of messages were consumed.
  • Stop when message timestamp exceeds Consumes all messages created up until the selected date & time. If the selected date & time is in the future, the execution stops once a poll request returns no messages.
    Selecting <Use execution date & time> consumes all messages up until the date & time the node is executed. The <Current time> button sets the date & time to the current local time.

Advanced Settings

Advanced Settings
Allows to specify additional consumer related options, e.g., poll interval and session timeout. For further details on the individual options see the tooltip or consult the official Kafka documentation.

Input Ports

Icon
Connection details used to create the Kafka Consumer

Output Ports

Icon
Data table storing all the consumed messages

Best Friends (Incoming)

Best Friends (Outgoing)

Workflows

Installation

To use this node in KNIME, install KNIME Extension for Apache Kafka (Preview) from the following update site:

KNIME 4.2

A zipped version of the software site can be downloaded here. Read our FAQs to get instructions about how to install nodes from a zipped update site.

Wait a sec! You want to explore and install nodes even faster? We highly recommend our NodePit for KNIME extension for your KNIME Analytics Platform.

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.