Icon

Leverage open-source, local LLMs via Ollama

How to leverage open-source, local LLMs via Ollama

This workflow shows how to leverage (i.e., authenticate, connect and prompt) an LLM (e.g., llama 3-instruct) available via Ollama in KNIME.

This approach is suitable for chat, instruct and code models.

____

Why do we use the OpenAI nodes to connect and prompt LLMs via Ollama?

Since Feb 2024, Ollama has built-in compatibility with the OpenAI Chat Completions API, making it possible to use more tooling and applications with Ollama locally. For this reason, we can conveniently use the OpenAI nodes of the KNIME AI Extension (and in particular the OpenAI Chat Model Connector node) to connect to the LLM of choice and send POST requests to the URL of a local host. In this way, we are freed from the burden of having to assemble and properly format the request body, and parse the resulting JSON output.

URL: Download Ollama https://ollama.com/download
URL: Llama 3 model card on Ollama https://ollama.com/library/llama3:instruct
URL: OpenAI compatibility with Ollama https://ollama.com/blog/openai-compatibility

How to leverage open-source, local LLMs via Ollama


For Chat, Instruct and Code models
1. Authenticate

  • Input dummy credentials ("password") just for the OpenAI client.


  • Point the OpenAI Authenticator node to Ollama server by providing the local host in "OpenAI base URL" (advanced settings).

2. Connect

  • Type the model name, and pass it on to the OpenAI Chat Model Connector as a flow variable in the "Specific Model ID" (advanced settings).

3. Prompt

  • Prompt the model as usual.

0. Set up

  • Visit https://ollama.com/download, download Ollama on your machine, and install it.


  • Identify the model you're interest in: https://ollama.com/library.


  • Run the model command in the wrapper. For example: ollama run llama3:instruct


  • Make sure Ollama is running as a Background process on your machine.

Why do we use the OpenAI nodes to connect and prompt LLMs via Ollama?

Since Feb 2024, Ollama has built-in compatibility with the OpenAI Chat Completions API, making it possible to use more tooling and applications with Ollama locally. For this reason, we can conveniently use the OpenAI nodes of the KNIME AI Extension (and in particular the OpenAI Chat Model Connector node) to connect to the LLM of choice and send POST requests to the URL of a local host. In this way, we are freed from the burden of having to assemble and properly format the request body, and parse the resulting JSON output.


Read more about OpenAI compatibility with Ollama: https://ollama.com/blog/openai-compatibility

Set a dummy credentialfor the OpenAI client
Credentials Configuration
Point to Ollama serverby providing the URL of a local host:http://localhost:11434/v1
OpenAI Authenticator
Ollama is compatible with theOpenAI Chat Completions API.Hence, we need the Chat Connector node.
OpenAI Chat Model Connector
Model name:llama3:instruct
String Configuration
LLM Prompter
Prompt engineering
Table Creator

Nodes

Extensions

Links