Icon

Leverage open-source, local LLMs for vision and embeddings via Ollama

This workflow shows how to leverage an LLM for vision and embeddings available via Ollama in KNIME.

URL: Download Ollama https://ollama.com/download
URL: Llava model (vision) card on Ollama https://ollama.com/library/llava
URL: Nomic-embed-text model (embeddings) card on Ollama https://ollama.com/library/nomic-embed-text

How to leverage open-source, local LLMs via Ollama


For Vision and Embeddings models
Set up

  • Visit https://ollama.com/download, download Ollama on your machine, and install it.


  • Identify the model you're interest in: https://ollama.com/library.


  • Run the model command in the wrapper: ollama run <model_name>


  • Make sure Ollama is running as a Background process on your machine.

Embeddings: Create numeric representations


Vision: Describe content in images


Request body
Container Input (JSON)
Binary Object to Base64
Texts to embed
Table Creator
...and transformto binary object
Read images
String to JSON
Parse output
JSON Path
Send request to model:nomic-embed-text
POST Request
Column Filter
Parse output
JSON Path
Send request to model:llava:7b(set time out very high)
POST Request
Column Appender
Request body
Container Input (JSON)
Assign text to embedin prompt
String Manipulation
Assign prompt
String Manipulation
Missing Value
Assign base64encoded image
String Manipulation
Parse prompt
JSON Path
Column Filter
Column Appender
Missing Value

Nodes

Extensions

Links