Icon

Challenge 37 - Text Deduplication

Challenge 37 - Text Deduplication
Description: You are asked to read Swedish textual data from a PDF using the Tika Parser. You then notice that much of the text is duplicated, which could be an encoding issue with the PDF itself. Consequently, you decide to to deduplicate the text. In this challenge, do your best to remove excessive amountsof duplicated text using as few nodes as possible. In most cases like this, you are not aiming for perfect removalof text, but instead are aiming for a cost effective approach which eliminates a large chunk of the duplication. Hint: Our solution consists of 5 nodes, but the 5th node may be unnecessary depending on your workflow. Challenge 37: Text Deduplication READ DATA DEDUPLICATE DATA Read PDFSplit each row by "\n"Columns toRowApply variousFilters(missing, spaces, descriptions)Apply anotherspaces filterEliminate DuplicationsTika Parser Cell Splitter Transpose Rule-basedRow Filter Row Filter GroupBy Description: You are asked to read Swedish textual data from a PDF using the Tika Parser. You then notice that much of the text is duplicated, which could be an encoding issue with the PDF itself. Consequently, you decide to to deduplicate the text. In this challenge, do your best to remove excessive amountsof duplicated text using as few nodes as possible. In most cases like this, you are not aiming for perfect removalof text, but instead are aiming for a cost effective approach which eliminates a large chunk of the duplication. Hint: Our solution consists of 5 nodes, but the 5th node may be unnecessary depending on your workflow. Challenge 37: Text Deduplication READ DATA DEDUPLICATE DATA Read PDFSplit each row by "\n"Columns toRowApply variousFilters(missing, spaces, descriptions)Apply anotherspaces filterEliminate DuplicationsTika Parser Cell Splitter Transpose Rule-basedRow Filter Row Filter GroupBy

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: USER
Created at: 2022-10-08
On NodePit since: 2022-10-09
Last update: 2024-11-21
Created with KNIME version: v4.5.0
Tags: justknimeit-37

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!