_knime_and_python_meta_collection

Meta Collection about KNIME and Python

KNIME and Python - just good friends
(https://kni.me/w/AvjrddXKOIoZYLV3)

First: KNIME (4.6+) now has a bundled Python version you can install just like any other extension (https://www.knime.com/whats-new-in-knime-46#bundled-python). For a lot of basic Python functions this is just fine.

Then: if you follow the official KNIME and Python guide and everything works out - fine you are done :-):

KNIME Python Integration Installation Guide
https://docs.knime.com/latest/python_installation_guide/index.html

--------------------------------
I have compiled informations from this collection into two articles you might want to check out. I will let the rest of this text-workflow stand as is for now

KNIME and Python — Setting up and managing Conda environments
https://medium.com/p/2ac217792539

KNIME and Python — Setting up Deep Learning Environments for Keras and TensorFlow
https://medium.com/p/4b66003858f4

--------------------------------

If you want to read about more technical details and use your own Python environments you can continue here:

KNIME as a platform is well suited to use all kinds of tools so also the popular Python. In order for KNIME and Python to work together you will have to have these things (besides a working KNIME version of course):

TL;DR:
A - a running Python environment using a compatible version (3.6 - 3.9 as of now)
B - install the KNIME Python extension(s) on your machine
C - set up KNIME to be able to communicate with Python (tell KNIME where conda is)
D - understand how the KNIME nodes use Python so you can interact with them

Python is not one program but a (sometimes quite complex) suite of packages and versions and not all of them work in accordance. To further complicate things KNIME might require a special set of packages which might differ depending on what tasks you want to do and what operating system you use (Windows, Mac, Linux). KNIME also has different families of nodes to use Python so this is evolving. You will also be able to write nodes in Python.

Please note the KNIME Python Deep Learning integrations will require very special packages (and versions), again depending on your operating system! More on that further down.

A - get Python with Miniforge ---------------------------

In order to get Python I would strongly suggest you use Miniforge (if you are familiar with it you can also use the larger suite Anaconda). Other Python integrations might also work but Miniforge should be easiest to maintain and is directly supported by KNIME. Also Miniforge as of today would make sure you follow the license regulations of Anaconda for commericial use (also you should therfor just use the "conda-forge" (https://anaconda.org/conda-forge) repository as in the following examples).

Yes I know working with conda environments sounds a little bit scary in the beginning but I would strongly recommend you spend maybe a few hours understanding the basics of it - it will greatly increase your understanding of Python and your power to get things done.

Also: it is completely normal to use different environments (in parallel) and also to edit, delete and re-install them if needed. KNIME would support you in that with the "Conda Environment Propagation" (https://kni.me/n/7HrPteTMsla4bZml).

Install Miniforge:
https://github.com/conda-forge/miniforge#download

You should have a prompt on MacOS or Windows (also Linux) and you can now manage your environments with the "conda" command
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

You can see a screenshot and example here:
https://forum.knime.com/t/python-problems-with-input-column-values-missing-values/38842/11?u=mlauber71

Now for example if you would download this example workflow to your local knime-workspace:
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_python_read_arff_file~mp7Pv607rD0biVI5
You will have a subfolder /data/ that contains two YAML files with configurations. You can now use the prompt and create new environments:

for MacOSX it could look somthing like this (you will have to adapt the path obviously)
(base) > conda env create -f="/Users/my_user/knime-workspace/kn_example_python_read_arff_file/data/py39_knime_macos.yaml"

For Windows it could be like this:
(base) > conda env create -f="C:\Users\12345678\knime-workspace\kn_example_python_read_arff_file\data\py39_knime_windows.yaml"

You might find additional settings used in a YAML file for special occasions here:
https://docs.knime.com/latest/python_installation_guide/index.html#conda-env-manual

Or you can found the YML / YAML files used directly on your local machine:
https://forum.knime.com/t/python-environment-creation-problem/41577/6?u=mlauber71

you can then activate yor environment like this (at the start there is the "base" environment set):
(base) > conda activate py39_knime_macos

You will then be able to install additional packages either thru conda-forge or special repositories (h2oai in this case). I would recommend to stay within the conda-forge repository (https://anaconda.org/conda-forge/repo):

(py39_knime_macos) > conda install -c h2oai h2o
(py39_knime_macos) > conda install -c conda-forge pyarrow

If you changed your YAML file with additional packages you can also make an update using the existing one:
(py39_knime_windows) > conda env update -f="C:\Users\12345678\py39_knime_windows.yaml" --prune
(py39_knime_macos) > conda env update-f="/Users/my_user/py39_knime_macos.yaml" --prune

Conda should make sure that your package combinations are OK and compatible with the main Python version and in combination of each other. KNIME sets some restrictions for compatibility, so you might want to check for that in the official guide if you experience any problems.

If a package is not available on conda-forge you can also use PIP to install additional packages like:
(py39_knime_macos) > pip install auto-sklearn

(base) > conda env list
Shows you what environments you currently have. More conda commands are covered here:
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

B - install the KNIME Python extension -------------------------------------

You will have to install the KNIME extension(s) for Python. I would suggest you use the latest version (currently in labs status) as well.

KNIME Conda Integration
https://kni.me/e/u7pbuYAqZDLXU3VA

KNIME Python Integration (Labs)
https://kni.me/e/xZE5dJbnuaKIpeyR

"From KNIME Analytics Platform, go to File → Install KNIME Extensions and search for Python Integration. The KNIME Python Integration extension should appear in the list. You can then select the extension and proceed through the installation wizard."
https://docs.knime.com/latest/python_installation_guide/index.html#installing-extension

You can read about the new integration (or you can just use it as it is)
https://forum.knime.com/t/data-passing-to-python-node-still-slow/38333/3?u=mlauber71

Columnar Backend integration and new internal data format should benefit the Python to KNIME data exchange performance
https://www.knime.com/blog/python-integration-for-fast-data-transfer
https://www.knime.com/blog/improved-performance-with-new-table-backend

Check out how to use the new Python integration (more practical examples under "D"):
https://docs.knime.com/latest/python_installation_guide/index.html#usage-examples

C - set up KNIME to use Python environments ------------

Under the preferences you can tell KNIME where to find Miniforge (or Ananconda) set up the default environment for Python (3) and cofigure which data transfer system to use. Here you can let KNIME create an environment (you can modify that later).

For every Python node you can define a special set of Python versions and packages with the "Conda Environment Propagation" (https://kni.me/n/7HrPteTMsla4bZml) to make sure you have the right ones available. A blog about managing the Conda environment is here:
https://www.knime.com/blog/how-to-manage-python-environments-conda-and-knime

This might be especially interesting if you use DeepLearning integrations. Keras and TensorFlow 1 & 2 need a special combination of packages in order to work with KNIME. I have prepared YAML files in these examples:
https://forum.knime.com/t/attributeerror-module-keras-optimizers-has-no-attribute-adam/38461/3?u=mlauber71

A summary about installing deep learning environments for KNIME can be found here:
https://forum.knime.com/t/tensorflow-keras-installation-conflict/48670/4?u=mlauber71

KNIME, Python and TensorFlow 2
https://kni.me/w/0P1TaKHmebS4g8zv

KNIME, Python and Keras (and TensorFlow 1)
https://kni.me/w/8pLvXVGL1hj7lXDh

You might have to try a few things and might have to read up about KNIME and deep learning integration (there should also be links to YAML files)
https://docs.knime.com/latest/deep_learning_installation_guide/index.html#_introduction

D - use Python nodes in KNIME ------------------

This workflow demonstrates how to use several different ports with KNIME and Python "Iris 2021"
https://kni.me/w/P6lVZYHup7PejexZ

KNIME and Parquet and SQLite
https://kni.me/w/G1kX4pbOlCeq56cH

KNIME and Python and ARFF
https://kni.me/w/mp7Pv607rD0biVI5

You can serach for additional examples how to use KNIME and Python
https://hub.knime.com/search?type=Workflow&tag=knime,python&sort=best

--- additional hints ------------------------------------------

You can see the used YML, YAML files from KNIME - and you can also export your current environment as a new YAML file to store it:
https://forum.knime.com/t/python-environment-creation-problem/41577/6?u=mlauber71

conda env export --from-history | grep -v "prefix" > environment.yml
conda env export --from-history | findstr -v "prefix" > environment.yml

KNIME and Jupyter notebooks - and the use of .PY files/modules
https://kni.me/w/BQEN92dR6W0cUM8R
https://forum.knime.com/t/unicodeencodeerror-charmap-codec-cant-encode-characters/39199/3?u=mlauber71

Please be aware how Pandas index and KNIME RowIDs are handled (check out this example):
https://kni.me/w/1D47JLruXBjh8eRr

You can also install a working R version via conda-forge packages and Conda Environment Propagation
https://kni.me/w/119aEydIH0oCdht8
https://forum.knime.com/t/data-separate-and-melting/45710/9?u=mlauber71

KNIME and Python can also be used to make more complex operations with Excel files:
https://hub.knime.com/search?type=Workflow&tag=openpyxl,python&sort=best

Nodes

No nodes found

Extensions

No modules found

_​knime_​and_​python_​meta_​collection

Nodes

Extensions

Links

Download

_knime_and_python_meta_collection