Parquet Reader

Options

Settings

Read from

Select a file system which stores the data you want to read. There are four default file system options to choose from:

• Local File System: Allows you to select a file/folder from your local system.
• Mountpoint: Allows you to read from a mountpoint. When selected, a new drop-down menu appears to choose the mountpoint. Unconnected mountpoints are greyed out but can still be selected (note that browsing is disabled in this case). Go to the KNIME Explorer and connect to the mountpoint to enable browsing. A mountpoint is displayed in red if it was previously selected but is no longer available. You won't be able to save the dialog as long as you don't select a valid i.e. known mountpoint.
• Relative to: Allows you to choose whether to resolve the path relative to the current mountpoint, current workflow or the current workflow's data area. When selected a new drop-down menu appears to choose which of the three options to use.
• Custom/KNIME URL: Allows to specify a URL (e.g. file://, http:// or knime:// protocol). When selected, a spinner appears that allows you to specify the desired connection and read timeout in milliseconds. In case it takes longer to connect to the host / read the file, the node fails to execute. Browsing is disabled for this option.

To read from other file systems, click on ... in the bottom left corner of the node icon followed by Add File System Connection port. Afterwards, connect the desired file system connector node to the newly added input port. The file system connection will then be shown in the drop-down menu. It is greyed out if the file system is not connected in which case you have to (re)execute the connector node first. Note: The default file systems listed above can't be selected if a file system is provided via the input port.

Mode

Select whether you want to read a single file or multiple files in a folder. When reading files in a folder, you can set filters to specify which files and subfolders to include (see below).

Filter options

Only displayed if the mode Files in folder is selected. Allows to specify which files should be included according to their file extension and/or name. It is also possible to include hidden files. The folder filter options enable you to specify which folders should be included based on their name and hidden status. Note that the folders themselves will not be included, only the files they contain.

Include subfolders

If this option is checked, the node will include all files from subfolders that satisfy the specified filter options. If left unchecked, only the files in the selected folder will be included and all files from subfolders are ignored.

File, Folder or URL

Enter a URL when reading from Custom/KNIME URL, otherwise enter a path to a file or folder. The required syntax of a path depends on the chosen file system, such as "C:\path\to\file" (Local File System on Windows) or "/path/to/file" (Local File System on Linux/MacOS and Mountpoint). For file systems connected via input port, the node description of the respective connector node describes the required path format. You can also choose a previously selected file/folder from the drop-down list, or select a location from the "Browse..." dialog. Note that browsing is disabled in some cases:

• Custom/KNIME URL: Browsing is always disabled.
• Mountpoint: Browsing is disabled if the selected mountpoint isn't connected. Go to the KNIME Explorer and connect to the mountpoint to enable browsing.
• File systems provided via input port: Browsing is disabled if the connector node hasn't been executed since the workflow has been opened. (Re)execute the connector node to enable browsing.

The location can be exposed as or automatically set via a path flow variable.

Path column

If checked, the node will append a path column with the provided name to the output table. This column contains for each row which file it was read from. The node will fail if adding the column with the provided name causes a name collision with any of the columns in the read table.

Transformation

Transformations

This tab displays every column as a row in a table that allows modifying the structure of the output table. It supports reordering, filtering and renaming columns. It is also possible to change the type of the columns. Reordering is done via drag-and-drop. Just drag a column to the position it should have in the output table. Whether and where to add unknown columns during execution is specified via the special row <any unknown new column>. Note that the positions of columns are reset in the dialog if a new file or folder is selected.

Reset order

Resets the order of columns to the order in the input file/folder.

Reset filter

Clicking this button will reset the filters i.e. all columns will be included.

Reset names

Resets the names to the names that are read from file or created if the file/folder doesn't contain column names.

Reset types

Resets the output types to the default types guessed from the input file/folder.

Reset all

Resets all transformations.

Enforce types

Controls how columns whose type changes are dealt with. If selected, we attempt to map to the KNIME type you configured and fail if that's not possible. If unselected, the KNIME type corresponding to the new type is used.

Take columns from

Only enabled in "Files in folder" mode. Specifies which set of columns are considered for the output table.

• Union: Any column that is part of any input file is considered. If a file is missing a column, it's filled up with missing values.
• Intersection: Only columns that appear in all files are considered for the output table.

NOTE: This setting has special implications if you are controlling the input location with a flow variable. If Intersection is selected any column that moves into the intersection during execution will be considered to be an unknown new column, even if it was previously part of the union of columns. It is also important to note that the transformation matching during execution is based on name. That means if there was a column [A, Integer] during configuration in the dialog and this column becomes [A, String] during execution, then the stored transformation is applied to it. For filtering, ordering and renaming, this is straight forward. For type mapping the following is done: If there is an alternative converter to the specified KNIME type, then this converter is used, otherwise we default to the default KNIME type for the new column. In our example we might have specified that [A, Integer] should be mapped to Long. For the changed column [A, String] there is no converter to Long, so we default back to String and A becomes a String column in the output table.

Advanced

When schema in file has changed

Specifies the node behavior if the content of the configured file/folder changes between executions, i.e., columns are added/removed to/from the file(s) or their types change. The following options are available:

• Fail: If set, the node fails if the column names or types in the file have changed.
• Ignore: If set, the node tries to ignore the changes and outputs a table with the old table specification. This is not recommended, as it may lead to invalid data in the resulting table. The "Use new schema" option should be used instead.
• Use new schema: If set, the node will compute a new table specification for the current schema of the file at the time when the node is executed. Note that the node will not output a table specification before execution and that it will not apply transformations, therefore the transformation tab is disabled.

Fail if specs differ

If checked, the node will fail if multiple files are read via the "Files in folder" option and not all files have the same table structure.

Views

This node has no views

Workflows

• 04 Model Building on Big DataKNIME Hub
• 04 Model Building on Big DataKNIME Hub
• 04 Model Building on Big DataKNIME Hub
• 04 Model Building on Big Data - SolutionKNIME Hub
• 04 Model Building on Big Data - SolutionKNIME Hub
• Show all 9 workflows

Links

• No links available

Developers