Icon

Finnovation Flows Extension Examples

Leverage open-source, local LLMs for vision and embeddings via Ollama

This workflow shows how to leverage an LLM for vision and embeddings available via Ollama in KNIME.

URL: Download Ollama https://ollama.com/download
URL: Llava model (vision) card on Ollama https://ollama.com/library/llava
URL: Nomic-embed-text model (embeddings) card on Ollama https://ollama.com/library/nomic-embed-text

Vision Model Prompter

Finnovation Flows Example Workflows

Example 1: Vision: Describe content in images - simple use of Vision Model Prompter with OpenAI Model


Pre-process Image Data and then convert to Base64 Format

Example 2: Vision: Describe content in images - get Structured Output using Vision Model Prompter with OpenAI Model


Example 3: Vision: Describe content in images - simple use of locally hosted model via Ollam (llava-phi3)


Insert Model Name: "llava-phi3:latest"

Open Advanced Settings and set Base URL to http://localhost:11434/v1


Note: local Models likely will not adhere to any schema provided. The standard response will be returned instead

Hover over Node and add Schema Port using "+" sign in bottom left


Create Field Structure in Table Creator and convert to Schema using Schema Generator

Connect Credentials variable to Vision Model Prompter or any node in the same branch before.

Select User Message Column and Image Column

Example 3: Vision: Describe content in images - simple use of locally hosted model via Ollam (llava-phi3)


Insert Model Name: e.g."claude-3-5-sonnet-20240620"

Change Radio Button to Anthropic

Note: Anthropic Models do not work with Structured Output (at least for this node)

Structured Output Prompter

Important: Add your API keys to the Credentials Configuration and Credential Widget Nodes

Connect Credentials variable to Vision Model Prompter or any node in the same branch before.

Select User Message Column

Create Field Structure in Table Creator and convert to Schema using Schema Generator

Structured Output Prompter
Path to images
List Files/Folders
Vision Model Prompter
Binary Object to Base64
Path to String
Read images
Image Reader (Table)
Get Images
Ungroup
PNGs to Binary Objects
To PNG
ImgPlus to PNG Images
Anthropic API Key
Credentials Widget
Schema Generator
Add User Message
String Manipulation
Set OpenAI API Keyas Password
Credentials Configuration
Define Fields (see Description of Schema Generator)
Table Creator
Example Data
Table Creator
Define Fields (see Description of Schema Generator)
Table Creator
Vision Model Prompter
Add User Message
String Manipulation
Add User Message
String Manipulation
Vision Model Prompter
Add User Message
String Manipulation
Vision Model Prompter
Schema Generator

Nodes

Extensions

Links