This node can be used to retrieve webpages by issuing HTTP GET requests
and parsing the
requested HTML webpages. For parsing,
jsoup
is used as library which implements the
WHATWG HTML5
specification. The parsed HTML will be cleaned by removing comments
and, optionally, replacing relative URLs by absolute ones.
By default, the output table will contain a column with the parsed HTML converted into XHTML. However, you can specify to get the parsed HTML as string output instead.
The node allows you to either send a request to a fixed URL (which is specified in the dialog) or to a list of URLs provided by an optional input table. Every URL will result in one request which in turn will result in one row in the output table. You can define custom request headers in the dialog.
The node supports several authentication methods, e.g. BASIC and DIGEST. Other authentication methods may be provided by additional extensions.
Cookies can be send to the server via the Request Header tab by setting the "Cookie" header. In order to receive cookies, set the "Extract cookies" option. Any cookies sent by the server are then extracted and appended as a List Cell in the output.
By default, the output table will contain a column with the parsed HTML converted into XHTML. However, you can specify to get the parsed HTML as string output instead.
The node allows you to either send a request to a fixed URL (which is specified in the dialog) or to a list of URLs provided by an optional input table. Every URL will result in one request which in turn will result in one row in the output table. You can define custom request headers in the dialog.
The node supports several authentication methods, e.g. BASIC and DIGEST. Other authentication methods may be provided by additional extensions.
Cookies can be send to the server via the Request Header tab by setting the "Cookie" header. In order to receive cookies, set the "Extract cookies" option. Any cookies sent by the server are then extracted and appended as a List Cell in the output.
The node supports the Credential port as input (see dynamic input ports). If the port is added, it must supply a Credential that can be embedded into the HTTP Authorization header, and all request done by the node will use the Credential from the port, regardless of other node settings. The OAuth2 Authenticator nodes provide such a Credential for example.
Options
General Settings
Authentication
Proxy
Error Handling
Request Headers
Input Ports
- Optional data table containing the variable parameters of the requests.
…- A Credential, that can be embedded into the HTTP Authorization header. If this port is added, then all request done by the node will always use the Credential from the port, regardless of other node settings. The OAuth2 Authenticator nodes provide such a Credential for example.
Popular Predecessors
- Table Creator6 %
- GET Request6 %
Recursive Loop Start4 %- String Manipulation3 %
- Rule-based Row Filter3 %
- XPath2 %
- Try (Data Ports)2 %
Table Row To Variable Loop Start2 %- Table Row to Variable Loop Start2 %
Row Filter (deprecated)2 %- String Manipulation (Variable)2 %
Counting Loop Start1 %- Java Snippet1 %
- Chunk Loop Start1 %
- Empty Table Creator1 %
- JSON Path1 %
- Excel Reader1 %
- RSS Feed Reader1 %
- Constant Value Column1 %
- XML Reader1 %
- Table Row to Variable< 1 %
- Generic Loop Start< 1 %
- PATCH Request< 1 %
- File Reader< 1 %
- Variable Expressions (legacy)< 1 %
- String to URI< 1 %
- Try (Variable Ports)< 1 %
- String to Date&Time< 1 %
- Table Reader< 1 %
Loop End< 1 %- Python Script (legacy)< 1 %
- Table Manipulator< 1 %
- JSON Reader< 1 %
- String Manipulation (Multi Column)< 1 %
- Load Text Files< 1 %
CSV Reader< 1 %Generic Loop Start< 1 %- Group Loop Start< 1 %
- Column Filter< 1 %
- Row Sampling< 1 %
- POST Request< 1 %
- Webpage Retriever< 1 %
- Variable to Table Row< 1 %
- GroupBy< 1 %
- Parallel Chunk Start< 1 %
- List Files/Folders< 1 %
- String to Path< 1 %
- Variable Creator< 1 %
- IF Switch< 1 %
- Rule-based Row Splitter< 1 %
- Line Reader< 1 %
- Wait...< 1 %
Variable to Table Row< 1 %Line Reader< 1 %- Column List Loop Start< 1 %
- Concatenate< 1 %
- Cross Joiner< 1 %
Row Splitter (deprecated)< 1 %Column Rename< 1 %- String Replacer< 1 %
- Rule Engine< 1 %
- Column Expressions (legacy)< 1 %
- Table View (JavaScript)< 1 %
- HTTP Retriever< 1 %
- URL to File Path< 1 %
- CSV Reader< 1 %
- Loop End< 1 %
- Counting Loop Start< 1 %
- Recursive Loop Start< 1 %
- Duplicate Row Filter< 1 %
- Partitioning< 1 %
- Ungroup< 1 %
- HTTP(S) Connector< 1 %
- Path to URI< 1 %
- String Configuration< 1 %
- PUT Request< 1 %
- Merge Variables< 1 %
- Table Editor (JavaScript)< 1 %
- XML Writer< 1 %
- Numeric Scorer< 1 %
- Java Edit Variable< 1 %
- Column Combiner< 1 %
- Joiner< 1 %
Table Row to Variable< 1 %- DELETE Request< 1 %
- DB Reader< 1 %
List Files< 1 %- MultiLayerPerceptron Predictor< 1 %
- Missing Value< 1 %
JSON Reader< 1 %- HTML Parser< 1 %
- Generic Run-for-time Loop Start< 1 %
- Column Aggregator< 1 %
- String Splitter (Regex)< 1 %
- External Tool< 1 %
Table Column to Variable< 1 %File Reader< 1 %Table Reader< 1 %- Top k Row Filter< 1 %
- TimeDelay Loop Start< 1 %
Excel Reader (XLS)< 1 %- File Reader (Complex Format)< 1 %
Recursive Loop Start (2 ports)< 1 %- Generic Web Service Client< 1 %
- Credentials Configuration< 1 %
- JSON Writer< 1 %
- Window Loop Start< 1 %
- Extract Attribute< 1 %
- Load text-based files< 1 %
- URL Encode< 1 %
Python Snippet< 1 %- Download (legacy)< 1 %
- Read Images< 1 %
Recursive Loop End< 1 %- Row Filter (Labs)< 1 %
- Sorter< 1 %
- Table Transposer< 1 %
- Explorer Browser< 1 %
- Math Formula< 1 %
- JPython Function (Legacy)< 1 %
- Geo-Coordinate Row Filter< 1 %
- Credentials Input (legacy)< 1 %
- Generic JavaScript View (JavaScript)< 1 %
- Double Widget< 1 %
- JSON to Table< 1 %
XML Reader< 1 %- Wait< 1 %
Flow Variable End IF< 1 %- Table Row To Variable Run-for-time Loop Start< 1 %
- Catch Errors (Data Ports)< 1 %
- Table Column to Variable< 1 %
- Interval Loop Start< 1 %
Interval Loop Start< 1 %- Nominal Value Row Filter< 1 %
- Reference Row Filter< 1 %
- Column Splitter< 1 %
- Unpivot< 1 %
- Data Generator< 1 %
- SDF Reader< 1 %
- Math Formula (Variable)< 1 %
- Path to String< 1 %
- Transfer Files< 1 %
- Google Sheets Connector< 1 %
- Google Sheets Reader< 1 %
- Boolean Configuration< 1 %
- String to JSON< 1 %
Python Script (1⇒1)< 1 %- Python Source (legacy)< 1 %
- Create Date&Time Range< 1 %
- String to XML< 1 %
- Table to HTML String< 1 %
- OPSIN< 1 %
- URL Normalizer< 1 %
- URL Resolver< 1 %
- Reset on Load< 1 %
Views
- This node has no views
Workflows
02_From_Links_to_Data_DemoKNIME Hub- 02_From_Links_to_Data_DemoKNIME Hub
- 02_From_Links_to_Data_DemoKNIME Hub
- 02_From_Links_to_Data_DemoKNIME Hub
- 02_From_Links_to_Data_DemoKNIME Hub
- Show all 154 workflows
Links
- KNIME E-Learning Course: Sending a GET Request to a REST servicewww.knime.com/knime-introductory-course/chapter2/section3/sending-get-request-to-rest-service
- Data capture from one of the private Big Commerce URL(https) to export it into excelby knimediger on 2021-09-16forum.knime.com/p/134752
- Scraping subcategories from main categoriesby armingrudd on 2019-12-07forum.knime.com/p/76393
- Download information from a web page without using selenium or paladinby AnotherFraudUser on 2024-04-27forum.knime.com/p/255723
- Trying to learn some webscrapping but get stuck with an “execute failure” in the first stepby denisfi on 2023-06-29forum.knime.com/p/227494
Developers
You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.