This node provides different content extraction algorithms, which allow extracting textual content from web pages, discarding irrelevant elements like navigation, ads, footers, headers, etc. The extractors are generally optimized for typical news article and blog post web pages.
The node expects an “XML Document” cell input which contains the HTML page from which to extract the content. To properly parse a web page, use Palladian’s HTML Parser node.
Currently, the following algorithms are provided:
Readability
A port of the JavaScript browser bookmarklet “Readability” by Arc90
-- a great tool for extracting content from HTML pages. “Readability
[…] takes a crack at wiping out all that junk so you can have a more
enjoyable reading experience. […] its success rate is pretty respectable
(we’d guess over 90% of web sites are handled properly)”. Readability
operates on the document’s DOM tree. Basically, it assigns all elements a
score for their contents. Metrics for the scoring are length of their
text content, number of commas and link density. Also, class
and id names are taken into consideration; for example, elements
with class name sidebar contain unlikely actual content in
contrast to elements with class article.
Website,
JavaScript Source.
Palladian
The Palladian content extractor extracts clean sentences from (English) texts. That is, short phrases are not included in the output. Consider Readability for general content. The main difference is that this class also finds sentences in comment sections of web pages.
Options
Popular Successors
- Column Filter15 %
- XPath11 %
- Strings to Document6 %
Loop End3 %- Document Data Extractor3 %
Regex Extractor2 %- HTML Node to Text2 %
- GET Request2 %
- Web Searcher2 %
- String Replacer2 %
- Table View (JavaScript)2 %
- HTTP Result Data Extractor2 %
- Loop End2 %
- Google Sheets Writer2 %
- XML to JSON2 %
Location Extractor2 %- String Manipulation2 %
- HTTP Retriever2 %
- HTML Parser2 %
- CSV Writer2 %
- Table Manipulator2 %
- Missing Value2 %
- Cell Splitter2 %
- Web Page Content Extractor2 %
- String to XML1 %
- Rule-based Row Filter1 %
- Java Snippet (simple)1 %
- Column Filter (legacy)1 %
- Document Viewer< 1 %
- Tile View (JavaScript)< 1 %
- Data to Report (BIRT)< 1 %
- Find Elements< 1 %
- String to JSON< 1 %
- Set To List< 1 %
- Row Filter< 1 %
- URL Resolver< 1 %
Column Rename< 1 %- Table Transposer< 1 %
- Feed Discovery< 1 %
- Save File Locally< 1 %
CSV Writer< 1 %Table Writer< 1 %Joiner< 1 %- RowID< 1 %
- Rule-based Row Splitter< 1 %
- Interactive Table (legacy)< 1 %
Excel Writer (XLS)< 1 %- OpenNLP NE Tagger< 1 %
- Bag Of Words Creator< 1 %
- Columns to JSON< 1 %
- Corpus Creator< 1 %
- URL Domain Extractor< 1 %
- Palladian NER< 1 %
OAuth< 1 %Table Extractor< 1 %- Inject Variables (Data)< 1 %
- Table Writer< 1 %
- Chunk Loop Start< 1 %
- Concatenate< 1 %
- Constant Value Column< 1 %
- Topic Extractor (Parallel LDA)< 1 %
- Tika Language Detector< 1 %
- Line Plot (JavaScript)< 1 %
- JSON Path< 1 %
- JSON to Table< 1 %
- Modify Time< 1 %
- Column to XML< 1 %
XML Writer< 1 %- Text Classifier Learner< 1 %
- Multipart Encoded HTTP Entity Creator< 1 %
- Extract Attribute< 1 %
- Take Full Page Screenshot< 1 %
Views
- This node has no views
Workflows
HTML_READER_Want_to_open_in_Excel_fileKNIME Forum- kn_forum_44466_tahitian_web_extractKNIME Forum
- Meister_NLPNodePit Space
- OpenAI_Create_EmbeddingsNodePit
Links
- Scraping reference cites from google scholar with seleniumby qqilihq on 2021-10-26forum.knime.com/p/137924
- Wrong encoding in web text scrapingby qqilihq on 2022-06-23forum.knime.com/p/161247
- Any implementation of Pytorch library “newsletter3k”?by qqilihq on 2021-02-15forum.knime.com/p/115850
- Web scrapingby denisfi on 2023-06-29forum.knime.com/p/227456
- How to exclude links from the “Comments” section when scraping or analyzing scraped pagesby qqilihq on 2020-05-13forum.knime.com/p/90924
Developers
You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.