Icon

Topic Modeling - Optimizing Alpha and Beta

This workflow demonstrates how to compute the Chi-Square Statistics to address the goodness of fit in topic modeling. It tests the multinomial assumptions behind the LDA model and examines whether the observed and estimated word vectors are statistically indistinguishable.


In this workflow, the objective functions we aim to optimize are alpha and beta.

Lewis, C. M., & Grossetti, F. "A Statistical Approach for Optimal Topic Model Identification" (Journal of Machine Learning, 23(58), 1−20, 2022).

URL: A Statistical Approach for Optimal Topic Model Identification https://www.jmlr.org/papers/volume23/19-297/19-297.pdf
URL: Say hi to Chi-square (χ²) for Optimal Topic Modeling https://medium.com/low-code-for-advanced-data-science/say-hi-to-chi-square-%CF%87%C2%B2-for-optimal-topic-modeling-1cf1d55dc7fa

Article NewsShort textsNode 17 customremove text data and metadata alreadyadded to document column typePOS tagging, lemmatization, stop word, number, ... filteringNode 298$Number of terms$>10Node 1948500Node 2001alphabetaAlpha domain(long execution time)Beta domain(long execution time)Combine parametersloop 1Collect optimization resultsNode 2018betaalphaCollect optimization resultsloop 2Exclude raw Chi-squareχ2 Chi-squareSurface plotχ2 Chi-squareExclude raw Chi-squareSurface plotBeta domainAlpha domainCSV Reader Strings to Document Preprocessing Column Filter Preprocessing Strings to Document Rule-basedRow Filter Column Filter CSV Reader Row Sampling Topic Extractor(Parallel LDA) Math Formula Math Formula Table Creator Table Creator Cross Joiner Table Row toVariable Loop Start Loop End Topic Extractor(Parallel LDA) Math Formula Math Formula Loop End Table Row toVariable Loop Start Column Filter Column Renamer Python View Column Renamer Column Filter Python View Table Creator Table Creator Topic Model Goodnessof Fit (χ²) Topic Model Goodnessof Fit (χ²) Article NewsShort textsNode 17 customremove text data and metadata alreadyadded to document column typePOS tagging, lemmatization, stop word, number, ... filteringNode 298$Number of terms$>10Node 1948500Node 2001alphabetaAlpha domain(long execution time)Beta domain(long execution time)Combine parametersloop 1Collect optimization resultsNode 2018betaalphaCollect optimization resultsloop 2Exclude raw Chi-squareχ2 Chi-squareSurface plotχ2 Chi-squareExclude raw Chi-squareSurface plotBeta domainAlpha domainCSV Reader Strings to Document Preprocessing Column Filter Preprocessing Strings to Document Rule-basedRow Filter Column Filter CSV Reader Row Sampling Topic Extractor(Parallel LDA) Math Formula Math Formula Table Creator Table Creator Cross Joiner Table Row toVariable Loop Start Loop End Topic Extractor(Parallel LDA) Math Formula Math Formula Loop End Table Row toVariable Loop Start Column Filter Column Renamer Python View Column Renamer Column Filter Python View Table Creator Table Creator Topic Model Goodnessof Fit (χ²) Topic Model Goodnessof Fit (χ²)

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: Francesco
Created at: 2024-03-12
On NodePit since: 2024-04-10
Last update: 2026-01-22
Created with KNIME version: v5.2.3
Tags: LDATopic ModelingNLPAlphaBetaLathent Dirichlet AllocationGoodness of Fit

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!