HDFS Connector

This node connects to a Hadoop Distributed File System using HDFS, WebHDFS or HTTPFS. The resulting output port allows downstream nodes to access the files of the remote file system, e.g. to read or write, or to perform other file system operations (browse/list files, copy, move, ...).

Path syntax: Paths for HDFS are specified with a UNIX-like syntax, /myfolder/myfile. An absolute for HDFS consists of:

  1. A leading slash ("/").
  2. Followed by the path to the file ("myfolder/myfile" in the above example).

SSL: This node uses the JVM SSL settings.

Options

Protocol
HDFS protocol to use.
  • HDFS: Native HDFS protocol (default port: 8020)
  • WebHDFS: WebHDFS REST API (default port: 9870)
  • WebHDFS with SSL: WebHDFS REST API with SSL encryption (default port: 9871)
  • HttpFS: HttpFS REST API (default port: 14000)
  • HttpFS with SSL: HttpFS REST API with SSL encryption (default port: 14000)
Host
Address of HDFS name node or WebHDFS/HTTPFS node.
Custom Port
Choose whether to use the default port for the selected protocol or specify a custom port.
Custom port
The custom port number to connect to. Must be between 1 and 65535. Note: The WebHDFS default ports are the Hadoop 3.x default ports. The default WebHDFS port on Hadoop 2.x is 50070 and 50470 with SSL.
Authentication method
Specify the authentication method to use.
  • Username: Pseudo/Simple authentication using a given username
  • Kerberos: Kerberos ticket based authentication
Username
The username to use for Pseudo/Simple authentication. This username will be used to access the HDFS file system.
Working directory
Specify the working directory of the resulting file system connection. The working directory must be specified as an absolute path (starting with "/"). A working directory allows downstream nodes to access files/folders using relative paths, i.e. paths that do not have a leading slash. The default working directory is the root "/".

Input Ports

This node has no input ports

Output Ports

Icon
HDFS File System Connection.

Views

This node has no views

Workflows

Links

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.