Icon

01_​Amazon_​S3_​Remote_​File_​Example

Amazon S3 Remote File Example

In order to connect to the service, Amazon S3 credentials are required.

Folders created on root level are created as buckets on S3. The virtual file system functionality is fully supported by the file handling nodes.
In the first step a bucket and a directory inside that bucket is created. After that a file is uploaded to the directory inside the bucket (path: /examplebucket***/exampledirectory). Finally all the files and folders inside the example bucket are listed. The result is a list of URIs which can be used to download files/directories.

In the second step we download the iris.csv which we uploaded in the step before. This saves the file to disk; this file can then be read using a csv reader. Since it might be unnecessary to cache the file to disk, the Amazon S3 File Picker provides the functionality to create URLs from which files can be read directly. This works with all file readeres that support reading files from a URL. These URLs have an expiration date which can be set in the dialog.

Stream reading also becomes useful in conjunction with the KNIME Streaming Executor as the (possibly large) data is read from the remote side and is immediately processed without being downloaded to the local machine.

After having seen how we upload and download files and how we use the File Picker to read files directly we can delete our example bucket from S3.

This workflow demonstrates how to use the S3 Connection. It demonstrates bucket creation, folder creation, file upload, file download, filelisting, and file deletion. It also shows how to use the S3 File Picker, for reading files directly without downloading them first. Connection to S3 serviceCreation of a bucket and a directory where a file is uploaded. Download of the previous uploaded file.The file can be aso be read from an URL Delete our example bucket from S3 Stream reading of the fle without beingdownloaded to the local machine. Create a bucketwith a unique nameConvert the flow variableto URI to delete bucketCreate directoryinside the created bucketList the all files/foldersin the created examplebucketrecursivelyUpload iris.csvto the createdexampledirectory insidethe examplebucketList data folder(includes adult.data)Convert URL (String)column to URIDownloadadult.dataRead the downloadedcsv from diskRead the csv directlyfrom S3Delete the bucketUsing the pathwe create a url fromwhich the csv readercan read directlySpecify your credentials Create Directory Table Creator Create UniqueBucket Name Create Directory List Remote Files Upload List Files String to URI Download Extract folder URIas flow variable CSV Reader Extract file URIas flow variable Extract csv path ons3 as flow variable CSV Reader Delete Files Amazon S3File Picker Stream it in InteractiveTable (local) Merge Variables Amazon S3Connection This workflow demonstrates how to use the S3 Connection. It demonstrates bucket creation, folder creation, file upload, file download, filelisting, and file deletion. It also shows how to use the S3 File Picker, for reading files directly without downloading them first. Connection to S3 serviceCreation of a bucket and a directory where a file is uploaded. Download of the previous uploaded file.The file can be aso be read from an URL Delete our example bucket from S3 Stream reading of the fle without beingdownloaded to the local machine. Create a bucketwith a unique nameConvert the flow variableto URI to delete bucketCreate directoryinside the created bucketList the all files/foldersin the created examplebucketrecursivelyUpload iris.csvto the createdexampledirectory insidethe examplebucketList data folder(includes adult.data)Convert URL (String)column to URIDownloadadult.dataRead the downloadedcsv from diskRead the csv directlyfrom S3Delete the bucketUsing the pathwe create a url fromwhich the csv readercan read directlySpecify your credentialsCreate Directory Table Creator Create UniqueBucket Name Create Directory List Remote Files Upload List Files String to URI Download Extract folder URIas flow variable CSV Reader Extract file URIas flow variable Extract csv path ons3 as flow variable CSV Reader Delete Files Amazon S3File Picker Stream it in InteractiveTable (local) Merge Variables Amazon S3Connection

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: oole
Created at: 2016-08-26
On NodePit since: 2024-03-22
Last update: 2024-11-21
Created with KNIME version: v4.1.1
Tags: Amazon S3CloudremotestreamingURL

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!