Databricks Unity File System

<p>This workflow demonstrates KNIME's capability to connect with <strong>Databricks Unity Volumes</strong>, part of the Unity Catalog framework. It enables users to read and write files from and to Databricks Unity Volumes.</p><p>The use case presented here involves writing once-a-month Excel files containing daily weather information for different locations as Parquet files into the Databricks Unity Volume. Then, the data is read, and a simple linear regression model is applied to Spark.</p><p>For more information about Databricks Unity Catalog and Databricks Unity Volumes, please refer to the "External resources" links.</p><p>You can download the workflow and run it on your local machine using the latest version of the KNIME Analytics Platform. For optimal performance, it is recommended that you use the latest version of KNIME AP.</p><p></p><p>Workflow Requirements</p><p>To run the workflow locally, you will need:</p><ol><li><p>A Databricks account</p></li><li><p>An existing Databricks cluster</p></li></ol><p>Workflow Details</p><ol><li><p><strong>Connecting to Databricks Unity Volume</strong></p><ul><li><p>First, we connect to the Databricks Unity Volume, where we want to read and write files via the Databricks Unity File System Connector.</p></li></ul></li><li><p><strong>Writing Data to Unity Volume</strong></p><ul><li><p>The use case involves taking thirty generated Excel files with synthetic weather information from 1000 locations and writing them into the Databricks Unity Volume as Parquet files.</p></li></ul></li><li><p><strong>Creating a Spark Context</strong></p><ul><li><p>We create a Spark context using the Create Databricks Environment node and read the previously generated Parquet files with the Parquet to Spark node, creating a DataFrame in Spark.</p></li></ul></li><li><p><strong>Data Manipulation and Modeling</strong></p><ul><li><p>We manipulate the data in the Spark context using the KNIME Extension for Apache Spark nodes. This operation includes filtering missing values, splitting and normalizing the data frame, and applying a linear regression model.</p></li></ul></li><li><p><strong>Model Evaluation</strong></p><ul><li><p>Finally, we use the Spark Numeric Score node to visualize the linear regression performance and capacity to predict rainfall based on the selected features and shut down the Spark context.</p></li></ul></li></ol>