Icon

Deploy of Face Detection MTCNN and Google Cloud Vision API

This workflow can be run by itself, but it is in truth complemetary to a main one, which needs to be executed at least once before this one, at least until the lexicon text mining module and the following table writer, in order to save in the working directory the necessary .table file. The main workflow is present in the same directory of this workflow on Knime Hub. This workflow also needs jpg posters named with the respective movie IMDb ID, but for obvious reasons they are not made available publicly.

This workflow aims at counting the number of faces in the posters, and uses the Google Cloud Vision API to perform color analysis and label detection on the same images.

For the purpose of subsequent analysis, the dominant color is extracted from each poster, while the most popular and meaningful labels among all the posters are selected and encoded in the model with dummy variables, for each observation.

A valid service account JSON key for Google Cloud (Vision) is needed to make this workflow run.

FACE RECOGNITION FOR THE POSTERSWe use MTCNN, a pretrained convolutional neural network (exploiting Tensorflow) available as a Py library, to recognize faces. We need to stress that for storageconstraints and most importantly time constraints (downloading one more time all the posters would take more than a day), we were not able to download the full sizeposters from the URL. The extreme downscaling we did makes it very difficult to recognize faces, especially in old and drawn posters. Thus, this section should be takenas demonstrative. There is no reason why this should not work with appropriate time and resources. SETTING UP WHAT WE NEED FOR GOOGLE CLOUD VISION APIWe reuse the workflow given to us with the challenge presentation, using our JSON key for the API and replacing a node in the data pre-processing metanode (a Container Input (JSON) that obviously interfered with the workflow call). We instead put a JSON reader and .json filein the workflow folder to be read by it. We were forced to split the process because of an unexpected error that would occur while performing the loop over the whole set of posters. COLOR FEATURE EXTRACTION: DOMINANT COLOR AND COLORFULNESSThe Cloud Vision API allows us to extract insightful features about the colors of the movies posters. More precisely, each movie poster will have different colors. These colors will be encoded in in RGB values and HEX, together with the percentageof that color in the movie poster. Then, to leave everything more interpretable and to add the color features in the models, we keep only the dominant color, namely the one with higher percentage in each poster. Furthermore, we also create the Colorfulness feature (more info inthe metanode). LABEL DETECTION AND PROCESSING For each movie poster we use the Cloud Vision API to extract the label(s) feature. In this way, we obtain a column for each label detected in the whole dataset (as binary variable) and each movie poster would have more labelsand the presence of them in a particular poster is given by the "1" under the corresponding column.We then process these new features obtained by selecting only the labels present in more movie posters (here we considered the top 5, but could be higher). The top5 selection is done in the Label(s) Processing metanodeand the result obtained is used as a reference table for the Reference Column Filter node. At the end, we will have the dataset with only the most frequent labels. Upload .json file withauthentication keysAnalize with GoogleCloud Vision APIin Chunks of50 imagesCollect datafrom all iterationsWe list all the posters we downloadedWe inner join the two tables(some images could not havebeen downloaded forwhatever reason, and this is the why we do this join)We append an IMDb ID column (eliminating the jpg extension and the rest of the path) to get an identifier through which we can perform a join of the 10'000 obs. with the related images pathsWe use MTCNN (a pretrained CNN that exploits TF and is available as Py library) to perform face recognitionKnime Mountpoint URLAbsolute path for Knime MountpointWe select only the path columnuse the Labels Processing table asa reference table to selectonly top5 columnsjoin selected labels back with other columns Dominant Colorjoining the selectedfetures of images:color and labelswhole dataset with all features neededfor the modelsprepare the dataset to be joined back with the one with all variables(.jpg removed)remove %sort percentagepercent as a numberjoin image featuresWe read image_analysis.table, a table produced by the main workflow, which needs to be executed once, at least partially (see the description)This node maysuperseed the previoustable reader (only in casethis workflow is called by another)Output port forexternal workflowsNode 846Analize with GoogleCloud Vision APIin Chunks of50 imagesCollect datafrom all iterationsCollect datafrom all iterationsAnalize with GoogleCloud Vision APIin Chunks of50 imagesPOST Request Conda EnvironmentPropagation Image properties AuthenticationGoogle Vision API Data preprocessing Keys upload Label detection Chunk Loop Start Loop End(deprecated) Label presence List Files/Folders Joiner Python Script Python Script Table Creator URL to FilePath (Variable) Table Rowto Variable Column Filter String to Path ReferenceColumn Filter Joiner GroupBy Labels Processing Joiner Joiner Cell Splitter Column Filter String Manipulation Sorter String To Number Joiner Colorfulness Statistics Table Reader ContainerInput (Table) ContainerOutput (Table) Row Filter Row Splitter Chunk Loop Start Loop End(deprecated) POST Request AuthenticationGoogle Vision API AuthenticationGoogle Vision API POST Request Row Splitter Loop End(deprecated) Concatenate Concatenate Chunk Loop Start FACE RECOGNITION FOR THE POSTERSWe use MTCNN, a pretrained convolutional neural network (exploiting Tensorflow) available as a Py library, to recognize faces. We need to stress that for storageconstraints and most importantly time constraints (downloading one more time all the posters would take more than a day), we were not able to download the full sizeposters from the URL. The extreme downscaling we did makes it very difficult to recognize faces, especially in old and drawn posters. Thus, this section should be takenas demonstrative. There is no reason why this should not work with appropriate time and resources. SETTING UP WHAT WE NEED FOR GOOGLE CLOUD VISION APIWe reuse the workflow given to us with the challenge presentation, using our JSON key for the API and replacing a node in the data pre-processing metanode (a Container Input (JSON) that obviously interfered with the workflow call). We instead put a JSON reader and .json filein the workflow folder to be read by it. We were forced to split the process because of an unexpected error that would occur while performing the loop over the whole set of posters. COLOR FEATURE EXTRACTION: DOMINANT COLOR AND COLORFULNESSThe Cloud Vision API allows us to extract insightful features about the colors of the movies posters. More precisely, each movie poster will have different colors. These colors will be encoded in in RGB values and HEX, together with the percentageof that color in the movie poster. Then, to leave everything more interpretable and to add the color features in the models, we keep only the dominant color, namely the one with higher percentage in each poster. Furthermore, we also create the Colorfulness feature (more info inthe metanode). LABEL DETECTION AND PROCESSING For each movie poster we use the Cloud Vision API to extract the label(s) feature. In this way, we obtain a column for each label detected in the whole dataset (as binary variable) and each movie poster would have more labelsand the presence of them in a particular poster is given by the "1" under the corresponding column.We then process these new features obtained by selecting only the labels present in more movie posters (here we considered the top 5, but could be higher). The top5 selection is done in the Label(s) Processing metanodeand the result obtained is used as a reference table for the Reference Column Filter node. At the end, we will have the dataset with only the most frequent labels. Upload .json file withauthentication keysAnalize with GoogleCloud Vision APIin Chunks of50 imagesCollect datafrom all iterationsWe list all the posters we downloadedWe inner join the two tables(some images could not havebeen downloaded forwhatever reason, and this is the why we do this join)We append an IMDb ID column (eliminating the jpg extension and the rest of the path) to get an identifier through which we can perform a join of the 10'000 obs. with the related images pathsWe use MTCNN (a pretrained CNN that exploits TF and is available as Py library) to perform face recognitionKnime Mountpoint URLAbsolute path for Knime MountpointWe select only the path columnuse the Labels Processing table asa reference table to selectonly top5 columnsjoin selected labels back with other columns Dominant Colorjoining the selectedfetures of images:color and labelswhole dataset with all features neededfor the modelsprepare the dataset to be joined back with the one with all variables(.jpg removed)remove %sort percentagepercent as a numberjoin image featuresWe read image_analysis.table, a table produced by the main workflow, which needs to be executed once, at least partially (see the description)This node maysuperseed the previoustable reader (only in casethis workflow is called by another)Output port forexternal workflowsNode 846Analize with GoogleCloud Vision APIin Chunks of50 imagesCollect datafrom all iterationsCollect datafrom all iterationsAnalize with GoogleCloud Vision APIin Chunks of50 imagesPOST Request Conda EnvironmentPropagation Image properties AuthenticationGoogle Vision API Data preprocessing Keys upload Label detection Chunk Loop Start Loop End(deprecated) Label presence List Files/Folders Joiner Python Script Python Script Table Creator URL to FilePath (Variable) Table Rowto Variable Column Filter String to Path ReferenceColumn Filter Joiner GroupBy Labels Processing Joiner Joiner Cell Splitter Column Filter String Manipulation Sorter String To Number Joiner Colorfulness Statistics Table Reader ContainerInput (Table) ContainerOutput (Table) Row Filter Row Splitter Chunk Loop Start Loop End(deprecated) POST Request AuthenticationGoogle Vision API AuthenticationGoogle Vision API POST Request Row Splitter Loop End(deprecated) Concatenate Concatenate Chunk Loop Start

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: simonedigregorio
Created at: 2021-12-04
On NodePit since: 2022-01-19
Last update: 2024-04-28
Created with KNIME version: v4.5.1
Tags: Google Cloud VisionColor AnalysisFace DetectionMTCNNMovie Posters

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!