Icon

TextKleaner

This workflow is designed to help you prepare a textual dataset for a bag-of-words style computational analysis. It assumes that you already have your data in a tabular form - that is, a CSV or KNIME table containing a column of plain text documents along with metadata columns.

The workflow performs four types of operations to prepare your text for analysis. First, it scrubs your text, removing or replacing various characters to ensure that the text is formatted cleanly and consistently. Second, the workflow provides various ways to find and exclude documents that are irrelevant to your study. Third, it helps you to find and remove duplicated text, both in the form of highly similar documents and 'boilerplating' that is repeated at the start of documents. Finally, it allows you to enrich your data by tagging names and ngrams, and to refine your data by filtering out terms that are rare or uninformative, and by standardising plurals and other word variants.

While the loading and scrubbing of texts must be performed first, there is some flexibility around the remaining steps. The filtering operations in Step 2 are entirely optional, and can be performed in any order, although there are benefits to detecting duplicates before filtering documents by topic. Duplicate detection and boilerplate removal (Step 3) are also optional, but are highly recommended if you plan to tag ngrams in Step 4 or use topic modelling in your analysis. Duplicate detection should be performed after document filtering, but boilerplate removal can be performed at any stage before Step 4, and indeed may improve the results of the 'Filter by topic' operation. Tagging and filtering (Step 4) must be run last, as it will convert your documents from plain text strings into a tokenised format for subsequent analysis.

Except for duplicate detection (which saves information in a separate table), each operation in Steps 2 and 3 will overwrite the input data with the filtered data. The excluded documents are saved in a separate file, and can be reviewed or restored at any stage.

1. Load data and scrub textSelect your dataset (in either CSV orKNIME's table format), create uniquedocument IDs, and scrub the text tomake it ready for further processing. 3. Find duplicates andremove boilerplatingDetecting duplicated documents ishighly recommended if you plan to tagngrams or use topic modelling. 4. Tag and filter termsTag names and ngrams, standardiseplurals and other variants, and removeterms that are rare or uninformative. Theoutputs will be suitable for topicmodelling or term frequency analyses. TextKleanerA Knime workflow for preparing textual datasets for topic modelling and other types of analysis.Created by Angus Veitch, January 2020. Version 0.1.1. 2. Filter documents byrelevance, terms or topicsFind and remove unwanted documentsfrom your dataset. The excludeddocuments are saved in a separate file,which you can review at any stage. Tag Ngrams Filter andstandardise terms Load data Scrub text Create Document IDs Removeboilerplate text Filter bykey phrases Detect duplicates Filter byrelevance score Filter by topic Review and rescueexcluded docs Save processeddocuments Tag named entities 1. Load data and scrub textSelect your dataset (in either CSV orKNIME's table format), create uniquedocument IDs, and scrub the text tomake it ready for further processing. 3. Find duplicates andremove boilerplatingDetecting duplicated documents ishighly recommended if you plan to tagngrams or use topic modelling. 4. Tag and filter termsTag names and ngrams, standardiseplurals and other variants, and removeterms that are rare or uninformative. Theoutputs will be suitable for topicmodelling or term frequency analyses. TextKleanerA Knime workflow for preparing textual datasets for topic modelling and other types of analysis.Created by Angus Veitch, January 2020. Version 0.1.1. 2. Filter documents byrelevance, terms or topicsFind and remove unwanted documentsfrom your dataset. The excludeddocuments are saved in a separate file,which you can review at any stage. Tag Ngrams Filter andstandardise terms Load data Scrub text Create Document IDs Removeboilerplate text Filter bykey phrases Detect duplicates Filter byrelevance score Filter by topic Review and rescueexcluded docs Save processeddocuments Tag named entities

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: angus
Created at: 2020-12-11
On NodePit since: 2023-05-15
Last update: 2024-11-21
Created with KNIME version: v4.3.0
Tags: text analysistext preprocessingnamed entity taggingngramsduplicate detectionboilerplate removal

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!