Icon

09_​AzureExample

Working with Azure services

This workflow demonstrates how to connect to various Azure services such as HDInsight clusters, Azure Blob Storage, and AzureSQL from within KNIME Analytics Platform.

The Azure Blob Store Connection node connects KNIME Analytics Platform with your Azure Blob Storage and allows you to work with your files using the file handling nodes. The Azure Blob Store File Picker node creates a pre-signed URL that can be used in the reader nodes in KNIME to read directly from Azure Blob Storage or that can be shared with other users to access the dedicated files without the need for authentication.

The Create Spark Context (Livy) node can be used to set up a Spark context in your Azure HDInsight cluster. For more details see link to the documentation. Once a context is created you can use all the existing Spark nodes to visually assemble your Spark analysis flow.

Additionally, it is also possible to connect to Hive within your HDInsight cluster via the Hive Connector node.

The Microsoft SQL Server Connector node provides a DB Connection which can be used with the existing DB nodes to visually assemble queries that are executed within your AzureSQL. To upload large amounts of data into AzureSQL use the DB Loader node.

Establish a connection to HDInsight cluster and submit Spark jobs Establish a connection to Hive on HDInsight cluster This workflow demonstrates how to connect to various Azure Services.For more information see the workflow metadata. Find it here: View -> Description Connect toAzure Blob Storage Working with data located on Azure Blob Storage Working with AzureSQL create a SparkContext via Livygeneraterandom data select a tablegenerate test datacreate an emptytableload table contentinto the new tablecreate connection to Azure SQLload table contentinto the new tablecreate connectionto Hive inHDInsight clustercreate an emptytablelist files in a selected remote foldergenerate a signed URLfor a specific objectImport the csv objectinto KNIMECreate SparkContext (Livy) Azure Blob StoreConnection Table to Spark Data Generator Spark to Parquet Parquet to Spark Spark Column Rename Spark to Table DB Table Selector Data Generator DB Table Creator DB Loader Microsoft SQLServer Connector DB Loader Hive Connector DB Table Creator List Remote Files Azure Blob StoreFile Picker CSV Reader Establish a connection to HDInsight cluster and submit Spark jobs Establish a connection to Hive on HDInsight cluster This workflow demonstrates how to connect to various Azure Services.For more information see the workflow metadata. Find it here: View -> Description Connect toAzure Blob Storage Working with data located on Azure Blob Storage Working with AzureSQL create a SparkContext via Livygeneraterandom dataselect a tablegenerate test datacreate an emptytableload table contentinto the new tablecreate connection to Azure SQLload table contentinto the new tablecreate connectionto Hive inHDInsight clustercreate an emptytablelist files in a selected remote foldergenerate a signed URLfor a specific objectImport the csv objectinto KNIMECreate SparkContext (Livy) Azure Blob StoreConnection Table to Spark Data Generator Spark to Parquet Parquet to Spark Spark Column Rename Spark to Table DB Table Selector Data Generator DB Table Creator DB Loader Microsoft SQLServer Connector DB Loader Hive Connector DB Table Creator List Remote Files Azure Blob StoreFile Picker CSV Reader

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: dewi
Created at: 2020-08-26
On NodePit since: 2020-10-22
Last update: 2024-04-15
Created with KNIME version: v4.2.2
Tags: azuremicrosofthdinsightazuresqlhiveazure blob storesparkbig datalivy

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!