Icon

SQLite to Spark - KNIME Forum (76342)

Transfer Data between SQLite and Spark

URL: Forum entry https://forum.knime.com/t/sqlite-bad-value-for-type-bigdecimal/76342/6?u=mlauber71
URL: HUB: School of Hive - with KNIME's local Big Data environment (SQL for Big Data) https://hub.knime.com/-/spaces/-/~1q-JwD0cwmuEkWpy/current-state/
URL: KNIME, Databases and SQL https://medium.com/p/273e27c9702a

Transfer Data between SQLite and Spark

(https://forum.knime.com/t/sqlite-bad-value-for-type-bigdecimal/76342/6?u=mlauber71)

KNIME, Databases and SQL

https://medium.com/p/273e27c9702a

=> deletes the whole local big data folder/big_data decide in the configuration if you want sub-folders or parent foldersif you encouter any problems, closeKNIME and delete all data from the folder/big_data/
local big data context create
Path to String (Variable)
1st create (empty)hive tablemy_database_003
DB Table Creator
strftime('%Y-%m-%d %H:%M:%S', "Zoned Date Time")
DB Query
knime://knime.workflow/data/my_database.sqlite
SQLite Connector
exclude date and timevariables
DB Column Filter
my_database_003
Spark to Table
exclude date and timevariables
DB Column Filter
DB to Spark
v_sqlite_db
Create File/Folder Variables
1st create (empty)hive tablemy_database_001
DB Table Creator
Persist Spark DataFrame/RDD
remove table if it exists to have a clean start"my_database_001"
DB Table Deleter
Spark to Table
- Eastern US
Time Zone Modifier
DB Reader
Long_StringLong_Double
Column Renamer
, "StringCol"|| ";" AS "StringCol_Appended"=> experiment with different settings
DB Query
- London
Time Zone Modifier
remove table if it exists to have a clean startmy_database_005
DB Table Deleter
Column Resorter
DB to Spark
my_database_001
DB Table Selector
write Table:my_database_005
Spark to Hive
DB Reader
SELECT * FROM my_database_003
DB Table Selector
my_database_001
DB Writer
Spark to Table
remove table if it exists to have a clean startmy_database_003
DB Table Deleter
Spark to Table
my_database_005
DB Table Selector
upload fromKNIME to Spark
Table to Spark
DB Connection Extractor
TO_DATE(NOW()) as todayCAST(date_format(today, 'yyyyMMdd') AS INT) AS today_int??? why TODAY() is yesterday ...
Spark SQL Query
Persist Spark DataFrame/RDD
Hive to Spark
Spark to Table
test_data_all_types
Test Data Generator
DB Reader
SecondsUNIX -> Date & Time
UNIX Timestamp to Date&Time
load data into hive table
DB Loader
prepare_data

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: m_lauber
Created at: 2024-01-03
On NodePit since: 2024-01-04
Last update: 2025-11-04
Created with KNIME version: v5.8.0
Tags: SparkSQLite

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!