Icon

01_​caption_​preprocessing

01_caption_preprocessing
KNIME Deep Learning - Simple Image CaptioningIn this series of workflows, we want to demonstrate how to caption images using KNIME Deep Learning - Keras Integration. In image captioning the goalis to create a small textual description from just raw image data without additional information.This example is based on the COCO dataset (see http://cocodataset.org/#home), specifically on a subset of the 2014 data. This subset is defined in thefile coco_annotations_sample.csv located in the data folder of this workflow group. The 2014 data can be downloaded here: http://images.cocodataset.org/zips/train2014.zipThe workflows in this series need to be ran in order as they create data required for the execution of later workflows.Please note: The workflow series is heavily inspired by the great blog-post of Harshall Lamba (see https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-describe-pictures-c88a46a311b8)1. Workflow 01 Caption Preprocessing: In this workflow we use simple textprocessing techniques to reduce the complexity of the captions used fortraining. This will limit the words which our network is able to predict, i.e. makes the task a bit easier. Also, we will add special start and end tokens to thecleaned captions. As output, this workflow will write a table containing the vocabulary and a table containing the processed captions to the data folder.2. Workflow 02 Calculate InceptionV3 Features3. Workflow 03 Create GLOVE Vector Dictionary4. Workflow 04 Train Model5. Workflow 05 InferenceIn order to run the example, please make sure you have the following KNIME extensions installed:- KNIME Deep Learning - Keras Integration (Labs)- KNIME Image Processing (Community Contributions Trusted)- KNIME Image Processing - Deep Learning Extension (Community Contributions Trusted)- KNIME Streaming Execution (Labs)- KNIME TextprocessingYou also need a local Python installation that includes Keras. Please refer to https://www.knime.com/deeplearning#keras for installationrecommendations and further information. Read rawimage captionsConvert captionsto documents in orderto apply textprocessingWrite cleaned captions to data dirWrite vocabularyto data dirHandle "black-and-white"special caseConvert documents back to Stringand add special start + end tokens CSV Reader Strings To Document Unpivoting DuplicateRow Filter Column Filter Table Writer Column Filter Table Writer String Replacer Clean Captions Documents to String Create Vocabulary KNIME Deep Learning - Simple Image CaptioningIn this series of workflows, we want to demonstrate how to caption images using KNIME Deep Learning - Keras Integration. In image captioning the goalis to create a small textual description from just raw image data without additional information.This example is based on the COCO dataset (see http://cocodataset.org/#home), specifically on a subset of the 2014 data. This subset is defined in thefile coco_annotations_sample.csv located in the data folder of this workflow group. The 2014 data can be downloaded here: http://images.cocodataset.org/zips/train2014.zipThe workflows in this series need to be ran in order as they create data required for the execution of later workflows.Please note: The workflow series is heavily inspired by the great blog-post of Harshall Lamba (see https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-describe-pictures-c88a46a311b8)1. Workflow 01 Caption Preprocessing: In this workflow we use simple textprocessing techniques to reduce the complexity of the captions used fortraining. This will limit the words which our network is able to predict, i.e. makes the task a bit easier. Also, we will add special start and end tokens to thecleaned captions. As output, this workflow will write a table containing the vocabulary and a table containing the processed captions to the data folder.2. Workflow 02 Calculate InceptionV3 Features3. Workflow 03 Create GLOVE Vector Dictionary4. Workflow 04 Train Model5. Workflow 05 InferenceIn order to run the example, please make sure you have the following KNIME extensions installed:- KNIME Deep Learning - Keras Integration (Labs)- KNIME Image Processing (Community Contributions Trusted)- KNIME Image Processing - Deep Learning Extension (Community Contributions Trusted)- KNIME Streaming Execution (Labs)- KNIME TextprocessingYou also need a local Python installation that includes Keras. Please refer to https://www.knime.com/deeplearning#keras for installationrecommendations and further information. Read rawimage captionsConvert captionsto documents in orderto apply textprocessingWrite cleaned captions to data dirWrite vocabularyto data dirHandle "black-and-white"special caseConvert documents back to Stringand add special start + end tokens CSV Reader Strings To Document Unpivoting DuplicateRow Filter Column Filter Table Writer Column Filter Table Writer String Replacer Clean Captions Documents to String Create Vocabulary

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: dave
Created at: 2020-03-31
On NodePit since: 2020-05-15
Last update: 2024-04-18
Created with KNIME version: v4.1.2
Tags: image processingimage analysisimage captioningdeep learningneural networksGLOVEtransfer learningword embedding

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!