Icon

04_​train_​model

01_caption_preprocessing
KNIME Deep Learning - Simple Image CaptioningIn this series of workflows, we want to demonstrate how to caption images using KNIME Deep Learning - Keras Integration. In image captioning the goalis to create a small textual description from just raw image data without additional information.This example is based on the COCO dataset (see http://cocodataset.org/#home), specifically on a subset of the 2014 data. This subset is defined in thefile coco_annotations_sample.csv located in the data folder of this workflow group. The 2014 data can be downloaded here: http://images.cocodataset.org/zips/train2014.zipThe workflows in this series need to be ran in order as they create data required for the execution of later workflows.Please note: The workflow series is heavily inspired by the great blog-post of Harshall Lamba (see https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-describe-pictures-c88a46a311b8)1. Workflow 01 Caption Preprocessing2. Workflow 02 Calculate InceptionV3 Features3. Workflow 03 Create GLOVE Vector Dictionary4. Workflow 04 Train Model: After we cleaned the training captions and pre-calculated image-/word- features, the caption network can be trained. In thisexample, the task of image captioning is modelled as an iterative approach predicting the captions word-by-word. For this, we input the network with animage, a partial caption, and let it predict the next word in the caption. This also means we model the task as a word classification (using our vocabularyas possible classes). Before we can start training, we have to bring the data into this iterative format. For each image/caption pair, we create severaltraining examples using all possible partial sentences until all words in the caption have been used as target word once.We use a simple network with two input branches for training. The first branch contains some dense layers to further process the image feature vector. Thesecond branch contains an embedding layer to map our encoded caption to GLOVE vectors. This is achieved by setting the parameters of the embeddingvector to the previously created Python dictionary and making the layer untrainable. As output, the workflow writes the trained model to disk.5. Workflow 05 InferenceIn order to run the example, please make sure you have the following KNIME extensions installed:- KNIME Deep Learning - Keras Integration (Labs)- KNIME Image Processing (Community Contributions Trusted)- KNIME Image Processing - Deep Learning Extension (Community Contributions Trusted)- KNIME Streaming Execution (Labs)- KNIME TextprocessingYou also need a local Python installation that includes Keras. Please refer to https://www.knime.com/deeplearning#keras for installationrecommendations and further information. Read vocabularyRead preprocessedcaptionsCreate actual trainingdata for each exampleRead imagefeature tableExtract imagefile namesSet weights of embeddinglayer to pretrainedGLOVE vectorsReplace words withtheir vocalbulary indexCreate GLOVEvector dictionary pathSplit smalltest sampleWrite testdata tableTable Reader Table Reader Python Script (1⇒1) Table Reader Column Expressions Joiner Build Network DL PythonNetwork Editor Keras NetworkLearner Create CollectionColumn Shuffle Calculate MaxSequence Length Word/Vocab Mapping GLOVE Path Partitioning Model Writer Table Writer KNIME Deep Learning - Simple Image CaptioningIn this series of workflows, we want to demonstrate how to caption images using KNIME Deep Learning - Keras Integration. In image captioning the goalis to create a small textual description from just raw image data without additional information.This example is based on the COCO dataset (see http://cocodataset.org/#home), specifically on a subset of the 2014 data. This subset is defined in thefile coco_annotations_sample.csv located in the data folder of this workflow group. The 2014 data can be downloaded here: http://images.cocodataset.org/zips/train2014.zipThe workflows in this series need to be ran in order as they create data required for the execution of later workflows.Please note: The workflow series is heavily inspired by the great blog-post of Harshall Lamba (see https://towardsdatascience.com/image-captioning-with-keras-teaching-computers-to-describe-pictures-c88a46a311b8)1. Workflow 01 Caption Preprocessing2. Workflow 02 Calculate InceptionV3 Features3. Workflow 03 Create GLOVE Vector Dictionary4. Workflow 04 Train Model: After we cleaned the training captions and pre-calculated image-/word- features, the caption network can be trained. In thisexample, the task of image captioning is modelled as an iterative approach predicting the captions word-by-word. For this, we input the network with animage, a partial caption, and let it predict the next word in the caption. This also means we model the task as a word classification (using our vocabularyas possible classes). Before we can start training, we have to bring the data into this iterative format. For each image/caption pair, we create severaltraining examples using all possible partial sentences until all words in the caption have been used as target word once.We use a simple network with two input branches for training. The first branch contains some dense layers to further process the image feature vector. Thesecond branch contains an embedding layer to map our encoded caption to GLOVE vectors. This is achieved by setting the parameters of the embeddingvector to the previously created Python dictionary and making the layer untrainable. As output, the workflow writes the trained model to disk.5. Workflow 05 InferenceIn order to run the example, please make sure you have the following KNIME extensions installed:- KNIME Deep Learning - Keras Integration (Labs)- KNIME Image Processing (Community Contributions Trusted)- KNIME Image Processing - Deep Learning Extension (Community Contributions Trusted)- KNIME Streaming Execution (Labs)- KNIME TextprocessingYou also need a local Python installation that includes Keras. Please refer to https://www.knime.com/deeplearning#keras for installationrecommendations and further information. Read vocabularyRead preprocessedcaptionsCreate actual trainingdata for each exampleRead imagefeature tableExtract imagefile namesSet weights of embeddinglayer to pretrainedGLOVE vectorsReplace words withtheir vocalbulary indexCreate GLOVEvector dictionary pathSplit smalltest sampleWrite testdata tableTable Reader Table Reader Python Script (1⇒1) Table Reader Column Expressions Joiner Build Network DL PythonNetwork Editor Keras NetworkLearner Create CollectionColumn Shuffle Calculate MaxSequence Length Word/Vocab Mapping GLOVE Path Partitioning Model Writer Table Writer

Nodes

Extensions

Links