0 ×

17_​TopicExtraction_​with_​the_​ElbowMethod

Workflow

Topic Extraction

This workflow shows how to extract topics from text documents using the Topic Extractor node.

It reads textual data from a table (or, alternatively, the data can be fetched directly from news websites using the RSS Feed Reader node) and converts them into documents. The documents are then preprocessed, i.e. tagged, filtered, lemmatized, etc. After that, the Topic Extractor node can be applied to the preprocessed documents. However, the node requires users to input the number of topics that should be extracted beforehand.

There exist already a couple of methods to determine the best number of topics, we would use the "Elbow Method" in this workflow. The method basically runs k-means clustering on the input documents for a range of values of the number of clusters (e.g, from 1 to 20), and for each value calculates the within-cluster sum of squared errors (SSE), which is the sum of the distances of each data point in a cluster to its cluster center. Then, the SSE value for each number of clusters is plotted in a Scatter Plot. The best number of clusters should be the one where there is a drop of the SSE value, giving an angle in the plot.
Note that the Elbow method doesn't always work for all data sets. If there is not a clear elbow to be found in the plot, try using a different approach, like the Silhouette Coefficient.

After finding out the optimal number of clusters/topics for the documents, the Topic Extractor node can be executed and a tag cloud is created to visualize the topics' terms.

topicextractionclusteringk-Meansmachine learningdata miningNLPNatural Language Processing
Data Reading Preprocessing Elbow Method Transformation Extract topics and create the tag cloud This workflow shows how to extract topics from text documents using the Topic Extractor node, and how to determine an optimal number of topics using the Elbow method. Keep only thecreated documentsConnect this output tableto the first "Preprocessing"meta nodeExtract topics fromdocumentsRead news docsfrom tableCount theoccurances of eachtermCreating one tag cloudfor each topic20 iterationsk: 1-20kNews sitesRSS feed URLsExtract the RSS feedsCreate bit vectorsfor documentsReduce thedims of featurespace Column Filter Topic Extractor(Parallel LDA) k-Means Table Reader Loop End Calculate sum ofsquared errors Term Count Tag Clouds Counting Loop Start Java Edit Variable Table Creator Find Elbow RSS Feed Reader Document Vector Scatter Plot PCA Variable toTable Column Preprocessing Data Reading Preprocessing Elbow Method Transformation Extract topics and create the tag cloud This workflow shows how to extract topics from text documents using the Topic Extractor node, and how to determine an optimal number of topics using the Elbow method. Keep only thecreated documentsConnect this output tableto the first "Preprocessing"meta nodeExtract topics fromdocumentsRead news docsfrom tableCount theoccurances of eachtermCreating one tag cloudfor each topic20 iterationsk: 1-20kNews sitesRSS feed URLsExtract the RSS feedsCreate bit vectorsfor documentsReduce thedims of featurespace Column Filter Topic Extractor(Parallel LDA) k-Means Table Reader Loop End Calculate sum ofsquared errors Term Count Tag Clouds Counting Loop Start Java Edit Variable Table Creator Find Elbow RSS Feed Reader Document Vector Scatter Plot PCA Variable toTable Column Preprocessing

Download

Get this workflow from the following link: Download

Resources

Nodes

17_​TopicExtraction_​with_​the_​ElbowMethod consists of the following 73 nodes(s):

Plugins

17_​TopicExtraction_​with_​the_​ElbowMethod contains nodes provided by the following 5 plugin(s):