Icon

Topic Modeling - Goodness of Fit

This workflow demonstrates how to compute the Chi-Square Statistics to address the goodness of fit in topic modeling. It tests the multinomial assumptions behind the LDA model and examines whether the observed and estimated word vectors are statistically indistinguishable.


In this workflow, the objective function we aim to optimize is the number of topics to extract.

Lewis, C. M., & Grossetti, F. "A Statistical Approach for Optimal Topic Model Identification" (Journal of Machine Learning, 23(58), 1−20, 2022).

URL: A Statistical Approach for Optimal Topic Model Identification https://www.jmlr.org/papers/volume23/19-297/19-297.pdf
URL: Say hi to Chi-square (χ²) for Optimal Topic Modeling (Medium Article) https://medium.com/low-code-for-advanced-data-science/say-hi-to-chi-square-%CF%87%C2%B2-for-optimal-topic-modeling-1cf1d55dc7fa

Politics customremove text data and metadata alreadyadded to document column typeLDAiteration: topic-modelcollectoptimization dataiteration: topic-model500PlotArticle news$Number of terms$>10Optimizationparameters(long execution times..)PlotLDAiteration: topic-modelcollectoptimization dataPOS tagging, lemmatization, stop word, number, ... filteringNode 2018Optimizationparametersiteration: topic-modelCSV Reader Strings to Document Preprocessing Column Filter Topic Extractor(Parallel LDA) Math Formula Loop End Math Formula Table Row toVariable Loop Start Row Sampling Column Renamer Column Filter Line Plot Table Row toVariable Loop Start Column Renamer CSV Reader Rule-basedRow Filter Column Filter Table Creator Line Plot Column Filter Topic Extractor(Parallel LDA) Math Formula Loop End Preprocessing Strings to Document Table Creator Math Formula Topic Model Goodnessof Fit (χ²) Topic Model Goodnessof Fit (χ²) Politics customremove text data and metadata alreadyadded to document column typeLDAiteration: topic-modelcollectoptimization dataiteration: topic-model500PlotArticle news$Number of terms$>10Optimizationparameters(long execution times..)PlotLDAiteration: topic-modelcollectoptimization dataPOS tagging, lemmatization, stop word, number, ... filteringNode 2018Optimizationparametersiteration: topic-modelCSV Reader Strings to Document Preprocessing Column Filter Topic Extractor(Parallel LDA) Math Formula Loop End Math Formula Table Row toVariable Loop Start Row Sampling Column Renamer Column Filter Line Plot Table Row toVariable Loop Start Column Renamer CSV Reader Rule-basedRow Filter Column Filter Table Creator Line Plot Column Filter Topic Extractor(Parallel LDA) Math Formula Loop End Preprocessing Strings to Document Table Creator Math Formula Topic Model Goodnessof Fit (χ²) Topic Model Goodnessof Fit (χ²)

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: Francesco
Created at: 2024-03-24
On NodePit since: 2024-04-10
Last update: 2024-11-23
Created with KNIME version: v5.2.3
Tags: LDATopic ModelingChi-SquaredGoodness of FitNLP

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!