Note: We recommend to replace this node with the Web Page Content Extractor which does not use KNIME’s Textprocessing cells as output format and is thus more flexible.
This node provides different content extraction algorithms, which allow extracting textual content from web pages, discarding irrelevant elements like navigation, ads, footers, headers, etc. The extractors are generally optimized for typical news article and blog post web pages. Currently, three algorithms are provided:
Readability
A port of the JavaScript browser bookmarklet “Readability” by Arc90 -- a great tool for extracting content from HTML pages.
“Readability […] takes a crack at wiping out all that junk so you can have a more enjoyable reading experience. […] its
success rate is pretty respectable (we’d guess over 90% of web sites are handled properly)”. Readability operates on the
document’s DOM tree. Basically, it assigns all elements a score for their contents. Metrics for the scoring are length of their
text content, number of commas and link density. Also, class and id names are taken into consideration; for
example, elements with class name sidebar contain unlikely actual content in contrast to elements with class
article. Website,
JavaScript Source.
Palladian
The Palladian content extractor extracts clean sentences from (English) texts. That is, short phrases are not included in the output. Consider Readability for general content. The main difference is that this class also finds sentences in comment sections of web pages.
Options
Popular Predecessors
- HTTP Retriever14 %
- HTTP Result Data Extractor11 %
- String To XML7 %
- Table Creator4 %
- GET Request4 %
- Column Filter4 %
- Column To XML4 %
File Reader3 %- HTML Node to Text3 %
- Clean HTML Retriever3 %
- Row Filter2 %
NekoHtmlParser2 %XML Reader2 %- URL Extractor2 %
Joiner2 %- Column Rename2 %
- Cell Splitter2 %
CSV Reader1 %- Concatenate1 %
Excel Reader (XLS)1 %- XML To JSON1 %
- Web Searcher1 %
Content Extractor1 %- Unzip Files (legacy)1 %
- RSS Feed Reader1 %
Line Reader1 %String To Number< 1 %Strings To Document< 1 %- Feed Parser< 1 %
- Start WebDriver< 1 %
Table Reader< 1 %Extract Outer/InnerHTML< 1 %- Catch Errors (Data Ports)< 1 %
- Strings To Document< 1 %
- Corpus Creator< 1 %
List Files< 1 %- Flat File Document Parser< 1 %
Date Extractor< 1 %- Dataset Reader< 1 %
- Java Snippet< 1 %
Loop End< 1 %Cell Splitter< 1 %- RowID< 1 %
- String Manipulation< 1 %
- Rule-based Row Filter< 1 %
- Rule-based Row Splitter< 1 %
- String To Term< 1 %
- Twitter Search< 1 %
- XPath< 1 %
- HTML Parser< 1 %
- List Remote Files (legacy)< 1 %
- String to URI< 1 %
- Database Connection Table Reader (legacy)< 1 %
- Database Reader (legacy)< 1 %
- Chunk Loop Start< 1 %
- Row Splitter< 1 %
- Column Splitter< 1 %
- Rule Engine< 1 %
- Parallel Chunk Start< 1 %
- PMML To Cell< 1 %
Excel Reader (XLS)< 1 %Excel Reader (XLS)< 1 %- Tika Parser URL Input< 1 %
- N Chars Filter< 1 %
- PDF Parser< 1 %
Document Data Extractor< 1 %- Document Data Extractor< 1 %
- Sentence Extractor< 1 %
- Column Filter< 1 %
- Table View< 1 %
- Table to JSON< 1 %
- Columns to JSON< 1 %
- JSON To XML< 1 %
- Compare Segments< 1 %
- Webpage Retriever< 1 %
Table to HTML String< 1 %- Text Classifier Predictor< 1 %
- AP Calculator< 1 %
- TF-IDF Similarity< 1 %
- URL Domain Extractor< 1 %
HTTP Retriever< 1 %- Form Encoded HTTP Entity Creator< 1 %
- Multipart Encoded HTTP Entity Creator< 1 %
- Find Elements< 1 %
Views
- This node has no views
Workflows
Links
- No links available
Developers
You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.