12_Incremental_processing_Parquet_file
In this workflow, we will use the NYC taxi dataset to show case a continous preprocessing and publishing of event data. Instead of the Group Loop Start node this workflow could executed once per week in order to preprocess and publish all data that has arrived during the week. The result is written as a separate Parquet file within the same folder for each run. To ensure the uniquness of the file for each run we use the year and week of each run as file prefix that is set via flow variable.
Since the folder stays the same and Parquet is reading all files within the same folder independent of their file name, this folder can be exposed as external table (e.g. in Hive or Impala) to power further analysis processes.
URL: KNIME File Handling Guide https://docs.knime.com/latest/analytics_platform_file_handling_guide/index.html
Nodes
- Google Authenticator2 ×
- Microsoft Authenticator2 ×
- Amazon Authentication1 ×
- Amazon S3 Connector1 ×
- Azure Data Lake Storage Gen2 Connector1 ×
- Show all 24 nodes
Extensions
- FeatureKNIME Amazon Cloud Connectors
- FeatureKNIME Azure Cloud Connectors
- FeatureKNIME Base nodes
- FeatureKNIME Basic File System Connectors
- FeatureKNIME Big Data Connectors
- Show all 14 modules
Links
- No links available