Icon

CrewAI - AI-Agents scan your CSVs and write SQL using Ollama

KNIME and CrewAI - use an AI-Agent system to scan your CSV files and let Ollama / Llama3 write the SQL code

KNIME and CrewAI - use an AI-Agent system to scan your CSV files and let Ollama / Llama3 write the SQL code

The agents will 'discuss' among themselvesm use the documents provided and come back with a (hopefully) perfect soltion to your task based on the instructions you gave

---
Adapted from:

Integrating Agent Frameworks into Low Code Tools: Making CrewAI work in KNIME
https://medium.com/p/a7fecb042915
https://hub.knime.com/lowcodeaiftw/spaces/Public/LLMs/CrewAI~lgdow45tWuhhb4fT/

by https://medium.com/@martindausa


Prerequisites:Ideally, the KNIME 4.7 (or higher) Version Installed including Python Integration Extension.Custom Python Environment set up including key KNIME Packages and CrewAI package as well as Ollama (also pullthe LLM models you want to use).cf. "py_knime_crewai.yml" in the /data/ folderDefine Agents:Agent NameAgent GoalAgent BackstoryAgent Model (currently Llamaa3 and Mistral, vou can add more)Define Tasks with Documents:Task NameTask DescriptionAssign Agent to perform the TaskThe Parameter with the document path is preassigned (Do not change)Flexibility to inject certain parameters into the task (ParameterName: ParameterValue)Further Hints:Using an agent system seem attractive but will provide you with an additional set of challenges (especially whenwrapping it in KNIME nodes to meke it easier to access).Currently we try to get back a JSON file that should either stand alone of be enclosed in backticks (```) that you mighthave to process further.One option would b eto further explore the "expected_output" setting (currently empty) in order to get the system to giveback a -well- defined output. Define Agents and their models Define the Tasks - in this case we have tasks that would have a document as aparameter. And although we also provide the document in the code it seems theagents also do want to have the document within the task itself.{path_to_document} Concatenate Agent and Task OutputsNote:Add new Agents / Tasks by copying respective components. Make sure to add additionalinput ports to Concat nodes and connect new Agents/tasks accordingly Kickoff your Agents - you can check the progress and the results in the log folder../data/chat/ollama_crewai_rag_<...> Adapted from:Integrating Agent Frameworks into Low Code Tools: Making CrewAI work in KNIMEhttps://medium.com/p/a7fecb042915by https://medium.com/@martindausa KNIME and CrewAI - use an AI-Agent system to scan your CSV files and let Ollama / Llama3 write the SQL codeThe agents will 'discuss' among themselvesm use the documents provided and come back with a (hopefully) perfect soltion to your task based on the instructions you gave Maybe download the whole LLM workflow group in order to get all the folders and documents(https://hub.knime.com/mlauber71/spaces/LLM_Space/~17k4zAECNryrZw1X/) Make sure you have the Ollama models pulled locally and they are runningollama pull mxbai-embed-largeollama pull mistral:instructollama pull llama3:instructollama pull llama3.1ollama pull codellamaollama run mistral:instructollama run llama3:instructollama run llama3.1 work in progress A Task without access to documents Convert the 3 "Northwind" CSV files into a local H2 database The code is not (yet) there to be run with H2 although often it will represent usable SQL code.More work on prompting needs to be done Propagate Python environmentfor KNIME on MacOSX (Apple Scilicon)OR Windowswith Miniforge / Minicondaconfigure how to handle the environmentdefault = just check the namesExpert for DatabasesSQL-ExpertCreate a structure from database "Northwind"=> provide the link to the document as Parameter (again){path_to_document} collect AgentsTask DetailsTaskParameterDetailsCrewAI using localOllama / Llama3 / Mistralcurrent timescan for CSV-FilesnorthwindURIFile Pathollama_crewai_rag_olama_crewai_locate and create/data/ folderwith absolute pathslist_files=> if you want just one filepath_to_documentconvert to stringLimit number of documents!!! => 3 (you can change that)path_to_document~#([\s\S]*?)~#Match $MATCHINDEX: => extract the SQL query from the answerWrite a SQL query=> provide the link to the document as Parameter (again){path_to_document} var_document_folderbank_01=> try to avoid special characers and blanks in paths and file namesvar_document_folderFile Path escapedpath_to_documentlist_fileskeep as Listlist_filescooperationmax iterationsTask DetailsTaskParameterDetailsregexReplace($value$, "```", "~#")Responseknime://knime.workflow/../data/db_northwind.mv.dbcreate tableSTARTread theCSV fileENDNode 1801knime://knime.workflow/../data/db_northwind.mv.dbSHUTDOWN COMPACT;Write a SQL query=> provide the link to the document as Parameter (again){path_to_document} regexReplace($Response$, "```sql", "~#")ResponseNode 1809File Path escaped=> not sure which escaping will be betterdepending on the operating systemconda_crewai_knime4 Define Agent Define Agent Define Taskwith Document Concatenate Concatenate Concatenate Python Script Create Date&TimeRange List Files/Folders Path to URI URL to File Path GroupBy create filename to collect Table Writer Collect LocalMetadata Java EditVariable (simple) Column Rename Row Filter Table Rowto Variable Regex Extractor Agents Tasks TaskDetails Define Task Java EditVariable (simple) String to Path(Variable) Java Snippet Column Expressions Column Rename Table Rowto Variable Merge Variables Basic Settings Java Snippet Java Snippet String Manipulation H2 Connector DB Writer Chunk Loop Start Table Rowto Variable CSV Reader Variable Loop End Table Rowto Variable H2 Connector DB SQL Executor Define Taskwith Document String Manipulation DB Query Reader Column Rename Prerequisites:Ideally, the KNIME 4.7 (or higher) Version Installed including Python Integration Extension.Custom Python Environment set up including key KNIME Packages and CrewAI package as well as Ollama (also pullthe LLM models you want to use).cf. "py_knime_crewai.yml" in the /data/ folderDefine Agents:Agent NameAgent GoalAgent BackstoryAgent Model (currently Llamaa3 and Mistral, vou can add more)Define Tasks with Documents:Task NameTask DescriptionAssign Agent to perform the TaskThe Parameter with the document path is preassigned (Do not change)Flexibility to inject certain parameters into the task (ParameterName: ParameterValue)Further Hints:Using an agent system seem attractive but will provide you with an additional set of challenges (especially whenwrapping it in KNIME nodes to meke it easier to access).Currently we try to get back a JSON file that should either stand alone of be enclosed in backticks (```) that you mighthave to process further.One option would b eto further explore the "expected_output" setting (currently empty) in order to get the system to giveback a -well- defined output. Define Agents and their models Define the Tasks - in this case we have tasks that would have a document as aparameter. And although we also provide the document in the code it seems theagents also do want to have the document within the task itself.{path_to_document} Concatenate Agent and Task OutputsNote:Add new Agents / Tasks by copying respective components. Make sure to add additionalinput ports to Concat nodes and connect new Agents/tasks accordingly Kickoff your Agents - you can check the progress and the results in the log folder../data/chat/ollama_crewai_rag_<...> Adapted from:Integrating Agent Frameworks into Low Code Tools: Making CrewAI work in KNIMEhttps://medium.com/p/a7fecb042915by https://medium.com/@martindausa KNIME and CrewAI - use an AI-Agent system to scan your CSV files and let Ollama / Llama3 write the SQL codeThe agents will 'discuss' among themselvesm use the documents provided and come back with a (hopefully) perfect soltion to your task based on the instructions you gave Maybe download the whole LLM workflow group in order to get all the folders and documents(https://hub.knime.com/mlauber71/spaces/LLM_Space/~17k4zAECNryrZw1X/) Make sure you have the Ollama models pulled locally and they are runningollama pull mxbai-embed-largeollama pull mistral:instructollama pull llama3:instructollama pull llama3.1ollama pull codellamaollama run mistral:instructollama run llama3:instructollama run llama3.1 work in progress A Task without access to documents Convert the 3 "Northwind" CSV files into a local H2 database The code is not (yet) there to be run with H2 although often it will represent usable SQL code.More work on prompting needs to be done Propagate Python environmentfor KNIME on MacOSX (Apple Scilicon)OR Windowswith Miniforge / Minicondaconfigure how to handle the environmentdefault = just check the namesExpert for DatabasesSQL-ExpertCreate a structure from database "Northwind"=> provide the link to the document as Parameter (again){path_to_document} collect AgentsTask DetailsTaskParameterDetailsCrewAI using localOllama / Llama3 / Mistralcurrent timescan for CSV-FilesnorthwindURIFile Pathollama_crewai_rag_olama_crewai_locate and create/data/ folderwith absolute pathslist_files=> if you want just one filepath_to_documentconvert to stringLimit number of documents!!! => 3 (you can change that)path_to_document~#([\s\S]*?)~#Match $MATCHINDEX: => extract the SQL query from the answerWrite a SQL query=> provide the link to the document as Parameter (again){path_to_document} var_document_folderbank_01=> try to avoid special characers and blanks in paths and file namesvar_document_folderFile Path escapedpath_to_documentlist_fileskeep as Listlist_filescooperationmax iterationsTask DetailsTaskParameterDetailsregexReplace($value$, "```", "~#")Responseknime://knime.workflow/../data/db_northwind.mv.dbcreate tableSTARTread theCSV fileENDNode 1801knime://knime.workflow/../data/db_northwind.mv.dbSHUTDOWN COMPACT;Write a SQL query=> provide the link to the document as Parameter (again){path_to_document} regexReplace($Response$, "```sql", "~#")ResponseNode 1809File Path escaped=> not sure which escaping will be betterdepending on the operating systemconda_crewai_knime4 Define Agent Define Agent Define Taskwith Document Concatenate Concatenate Concatenate Python Script Create Date&TimeRange List Files/Folders Path to URI URL to File Path GroupBy create filename to collect Table Writer Collect LocalMetadata Java EditVariable (simple) Column Rename Row Filter Table Rowto Variable Regex Extractor Agents Tasks TaskDetails Define Task Java EditVariable (simple) String to Path(Variable) Java Snippet Column Expressions Column Rename Table Rowto Variable Merge Variables Basic Settings Java Snippet Java Snippet String Manipulation H2 Connector DB Writer Chunk Loop Start Table Rowto Variable CSV Reader Variable Loop End Table Rowto Variable H2 Connector DB SQL Executor Define Taskwith Document String Manipulation DB Query Reader Column Rename

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: marti
Created at: 2024-01-09
On NodePit since: 2024-08-20
Last update: 2024-11-21
Created with KNIME version: v4.7.8
Tags: CrewAIAgentsLLMLLama3Ollama

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!