Generic S3 Connector

This node connects to services that provide an S3-compatible API endpoint, for example MinIO. The resulting output port allows downstream nodes to access the data behind the endpoint as a file system, e.g. to read or write files and folders, or to perform other file system operations (browse/list files, copy, move, ...). If you want to connect to Amazon S3 on AWS, please use the Amazon S3 Connector node instead.

Path syntax: Paths for this file system are specified with a UNIX-like syntax, /mybucket/myfolder/myfile. An absolute consists of:

  1. A leading slash ("/").
  2. Followed by the name of a bucket ("mybucket" in the above example), followed by a slash.
  3. Followed by the name of an object within the bucket ("myfolder/myfile" in the above example).

URI formats: When you apply the Path to URI node to paths coming from this connector, you can create URIs with the following formats:

  1. Presigned https:// URLs which contain credentials, that allow to access files for a certain amount of time (see AWS documentation).
  2. s3:// URLs to access the S3-compatible endpoint with the aws command line interface, or from inside Hadoop environments.



http(s) URL of the S3-compatible service endpoint.
The authentication to use:
  • Anonymous: Use anonymous credentials to make anonymous requests to the S3-compatible endpoint.
  • Access Key ID and Secret Access Key: Use an access key ID and secret to authenticate. Check Use credentials to select a credentials flow variable to supply the ID and secret.
  • Default Credential Provider Chain: Supply credentials using environment variables or a credentials configuration file. For further details see the S3 documentation.
Working directory
Specifies the working directory using the path syntax explained above. The working directory must be specified as an absolute path. A working directory allows downstream nodes to access files/folders using relative paths, i.e. paths that do not have a leading slash. If not specified, the default working directory is "/".
Normalize paths
Determines if the path normalization should be applied. Path normalization eliminates redundant components of a path like, e.g. /a/../b/./c" can be normalized to "/b/c". When these redundant components like "../" or "." are part of an existing object, then normalization must be deactivated in order to access them properly.


Use path-style requests
If chosen, buckets will be accessed by appending their name to the path of the endpoint URL. Otherwise, they will be accessed by prepending their name as a subdomain in the URL hostname. The correct choice depends on how your S3-compatible endpoint is set up, but in most cases you will want to use path-style requests. For further explanation see the S3 documentation.
Optional region to set on the client. Might be empty, depending on how your S3-compatible endpoint is set up.
Connection timeout in seconds
The amount of time to wait when initially establishing a connection before giving up and timing out. For further details see the S3 documentation.
Read/write timeout in seconds
The socket read/write timeout. For further details see the S3 documentation.
Server-side encryption (SSE)
If selected, all data written to the endpoint will be encrypted with Server-side encryption (SSE) using SSE-S3, SSE-KMS or SSE-C.
Use default managed key
If SSE-KMS is selected as the SSE method, then this option specifies whether or not to encrypt data with the default managed CMK.
KMS key id
If SSE-KMS is selected as the SSE method and the default managed CMK should not be used, then this option allows to choose the KMS key with which to encrypt data written to the endpoint. Click on List keys to fetch the list of available keys (requires permissions kms:ListKeys, kms:DescribeKey and optionally kms:ListAliases).
Customer key
If SSE-C is selected as the SSE method it is necessary to provide an encryption key. There are multiple ways the key could be provided. The base64 encoded key could be entered directly or provided via credentials variable using the encoded key as password (username can be anything or left empty). The third way is to select a file that contains exactly 32 bytes (256 bit) that should be used as key.

Input Ports

A file system connection to read the customer key, when SSE-C encryption mode is enabled.

Output Ports

Generic S3 File System Connection


This node has no views




You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.