Icon

justknimeit-37 - Text Deduplication

justknimeit-37 - Text Deduplication
Challenge 37: Text DeduplicationLevel: EasyDescription: You are asked to read Swedish textual data from a PDFusing the Tika Parser. You then notice that much of the text isduplicated, which could be an encoding issue with the PDF itself.Consequently, you decide to to deduplicate the text. In thischallenge, do your best to remove excessive amounts of duplicatedtext using as few nodes as possible. In most cases like this, you arenot aiming for perfect removal of text, but instead are aiming for acost effective approach which eliminates a large chunk of theduplication. Hint: Our solution consists of 5 nodes, but the 5th nodemay be unnecessary depending on your workflow. PDF readerSplits new line into columnsTransposesRemoves row duplicates Tika Parser Cell Splitter Transpose DuplicateRow Filter Challenge 37: Text DeduplicationLevel: EasyDescription: You are asked to read Swedish textual data from a PDFusing the Tika Parser. You then notice that much of the text isduplicated, which could be an encoding issue with the PDF itself.Consequently, you decide to to deduplicate the text. In thischallenge, do your best to remove excessive amounts of duplicatedtext using as few nodes as possible. In most cases like this, you arenot aiming for perfect removal of text, but instead are aiming for acost effective approach which eliminates a large chunk of theduplication. Hint: Our solution consists of 5 nodes, but the 5th nodemay be unnecessary depending on your workflow. PDF readerSplits new line into columnsTransposesRemoves row duplicates Tika Parser Cell Splitter Transpose DuplicateRow Filter

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: Andrew
Created at: 2022-10-06
On NodePit since: 2022-10-07
Last update: 2024-12-24
Created with KNIME version: v4.6.3
Tags: justknimeitjustknimeit-37tika parserpdfdeduplicationjustKNIMEit-37

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!