Icon

05_​inference

01_caption_preprocessing
KNIME Deep Learning - Simple Image CaptioningIn this series of workflows, we want to demonstrate how to caption images using KNIME Deep Learning - Keras Integration. In image captioning the goalis to create a small textual description from just raw image data without additional information.This example is based on the COCO dataset (see http://cocodataset.org/#home), specifically on a subset of the 2014 data. This subset is defined in thefile coco_annotations_sample.csv located in the data folder of this workflow group. The 2014 data can be downloaded here: http://images.cocodataset.org/zips/train2014.zipThe workflows in this series need to be ran in order as they create data required for the execution of later workflows.Please note: The workflow series is heavily inspired by the great blog-post of Harshall Lamba (see https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-describe-pictures-c88a46a311b8)1. Workflow 01 Caption Preprocessing2. Workflow 02 Calculate InceptionV3 Features3. Workflow 03 Create GLOVE Vector Dictionary4. Workflow 04 Train Model5. Workflow 05 Inference: In the last workflow of this series, we predict the captions of the images in our test sample. This can be done using a similariterative approach, meaning predicting the captions word by word. First, we create a seed by creating a partial caption only containing the special starttoken. This can then be input into the network to predict the first word of the caption. Using the predicted word, a new partial caption is created, and theprocess is repeated until the special end toke is predicted. In KNIME, this is implemented in a Recursive Loop. In each loop iteration we check if the endtoken has been predicted. If this is the case, we exclude the corresponding row from further iteration as the full caption has already been predicted. Allother rows are used for the next loop iteration. This is done until the end token has been predicted for all examples.In order to run the example, please make sure you have the following KNIME extensions installed:- KNIME Deep Learning - Keras Integration (Labs)- KNIME Image Processing (Community Contributions Trusted)- KNIME Image Processing - Deep Learning Extension (Community Contributions Trusted)- KNIME Streaming Execution (Labs)- KNIME TextprocessingYou also need a local Python installation that includes Keras. Please refer to https://www.knime.com/deeplearning#keras for installationrecommendations and further information. Read modelfor preditionRead imagefeature tableRead preprocessedcaptionsRead vocabularyExtract imagefile namesRead testdata tableTop: startseqBottom: endseqPredict word by worduntil end token is predictedMap predicted indicesback to wordsView predicted captions and imagesModel Reader Table Reader Table Reader Table Reader Column Expressions Joiner Max Sequence Length Image Reader(Table) Table Reader Extract Vocab Index Prepare Test Datafor Prediction IterativePrediction Format Prediction Image Viewer KNIME Deep Learning - Simple Image CaptioningIn this series of workflows, we want to demonstrate how to caption images using KNIME Deep Learning - Keras Integration. In image captioning the goalis to create a small textual description from just raw image data without additional information.This example is based on the COCO dataset (see http://cocodataset.org/#home), specifically on a subset of the 2014 data. This subset is defined in thefile coco_annotations_sample.csv located in the data folder of this workflow group. The 2014 data can be downloaded here: http://images.cocodataset.org/zips/train2014.zipThe workflows in this series need to be ran in order as they create data required for the execution of later workflows.Please note: The workflow series is heavily inspired by the great blog-post of Harshall Lamba (see https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-describe-pictures-c88a46a311b8)1. Workflow 01 Caption Preprocessing2. Workflow 02 Calculate InceptionV3 Features3. Workflow 03 Create GLOVE Vector Dictionary4. Workflow 04 Train Model5. Workflow 05 Inference: In the last workflow of this series, we predict the captions of the images in our test sample. This can be done using a similariterative approach, meaning predicting the captions word by word. First, we create a seed by creating a partial caption only containing the special starttoken. This can then be input into the network to predict the first word of the caption. Using the predicted word, a new partial caption is created, and theprocess is repeated until the special end toke is predicted. In KNIME, this is implemented in a Recursive Loop. In each loop iteration we check if the endtoken has been predicted. If this is the case, we exclude the corresponding row from further iteration as the full caption has already been predicted. Allother rows are used for the next loop iteration. This is done until the end token has been predicted for all examples.In order to run the example, please make sure you have the following KNIME extensions installed:- KNIME Deep Learning - Keras Integration (Labs)- KNIME Image Processing (Community Contributions Trusted)- KNIME Image Processing - Deep Learning Extension (Community Contributions Trusted)- KNIME Streaming Execution (Labs)- KNIME TextprocessingYou also need a local Python installation that includes Keras. Please refer to https://www.knime.com/deeplearning#keras for installationrecommendations and further information. Read modelfor preditionRead imagefeature tableRead preprocessedcaptionsRead vocabularyExtract imagefile namesRead testdata tableTop: startseqBottom: endseqPredict word by worduntil end token is predictedMap predicted indicesback to wordsView predicted captions and imagesModel Reader Table Reader Table Reader Table Reader Column Expressions Joiner Max Sequence Length Image Reader(Table) Table Reader Extract Vocab Index Prepare Test Datafor Prediction IterativePrediction Format Prediction Image Viewer

Nodes

Extensions

Links