Webpage Retriever

This node can be used to retrieve webpages by issuing HTTP GET requests and parsing the requested HTML webpages. For parsing, jsoup is used as library which implements the WHATWG HTML5 specification. The parsed HTML will be cleaned by removing comments and, optionally, replacing relative URLs by absolute ones.

By default, the output table will contain a column with the parsed HTML converted into XHTML. However, you can specify to get the parsed HTML as string output instead.

The node allows you to either send a request to a fixed URL (which is specified in the dialog) or to a list of URLs provided by an optional input table. Every URL will result in one request which in turn will result in one row in the output table. You can define custom request headers in the dialog.

The node supports several authentication methods, e.g. BASIC and DIGEST. Other authentication methods may be provided by additional extensions.

Cookies can be send to the server via the Request Header tab by setting the "Cookie" header. In order to receive cookies, set the "Extract cookies" option. Any cookies sent by the server are then extracted and appended as a List Cell in the output.

Options

General Settings

URL

Select a constant URL or a column from the input table that contains the URLs that you want to request and parse.

Delay (ms)

Here you specify a delay between two consecutive requests, e.g. in order to avoid overloading the web service.

Concurrency

Number of concurrent requests

Ignore hostname mismatches

If checked, the node trusts the server's SSL certificate even if it was generated for a different host.

Trust all certificates

If checked, the node trusts all certificates regardless of their origin or expiration date.

Follow redirects

If checked, the node will follow redirects (HTTP status code 3xx ).

Send large data in chunks

Specifies whether HTTP Chunked Transfer Encoding is allowed to be used by the node. If enabled, messages with a large body size are being sent to the server in a series of chunks.

Use asynchronous HTTP client implementation

If checked, the node will use an asynchronous HTTP client implementation to perform the request (dedicated implementation of the web service client library, Apache CXF). If unchecked, a default client implementation is used, which is part of the Java Runtime Library. The latter tends to be a little faster but has certain limitations, e.g. proxy authentication is not supported. If unsure, leave unchecked. The asynchronous client cannot be used in conjunction with digest server authentication.

Timeout (s)

Timeout for a single request in seconds.

Output column name

The name of the created output column.

Output as XML

If checked, the output will be an XML column containing the parsed HTML converted into XHMTL. Otherwise, the output will be a String column containing the parsed HTML.

Replace relative URLs with absolute URLs

If checked, relative URLs in the HTML will be replaced by the absolute ones. This may simplify further processing.

Extract cookies

If checked, the cookies sent by the server are extracted from the response and appended as a list column. A missing value is appended if the server doesn't send cookies.

Cookie column name

The name of the column containing a list of cookies in the output table.

Authentication

Type

The authentication type, e.g. no authentication, BASIC, or DIGEST

Use credentials

Uses the selected credentials instead of username and password provided in the dialog.

Username

The username used for authentication.

Password

The corresponding password used for authentication.

Request Headers

Merge

If you click this button the request headers from the selected template on the left are merged with the already existing header definitions in the table below.

Replace

If you click this button the request headers from the selected template on the left will replace any existing header definitions in the table below.

Header key

The HTTP header key, e.g. Accept or X-Custom-Key. Note that some header keys such as Origin are silently ignored by default for security reasons. You can configure KNIME AP to allow any header key by setting the sun.net.http.allowRestrictedHeaders system property in the knime.ini configuration file to true.

Header value

The value for the header which can be a constant value or a reference to a flow variable, a column, a credential name, or a credential password (see the kind option).

Header kind

The kind of the value specified, which is either a constant value or a reference to a flow variable, a column, a credential name, or a credential password

Error Handling

Fail on connection problems (e.g. timeout, certificate errors, …)

This option describes what should happen if there was a problem establishing the connection to the server. The node either fails in execution or outputs status code and error message in the corresponding row.

Server-side errors (HTTP 5XX)

These options describe what should happen if a response with a 5XX status code is received. These status codes usually describe errors on the server side.

Client-side errors (HTTP 4XX)

These options describe what should happen if a response with a 4XX status code is received. These status codes usually describe client-side errors such as an incorrect web address.

Fail node execution or output missing value

This option describes what should happen after a request has failed. The node either fails in execution or outputs a missing value in the row of the output table that corresponds to this request. This option can be set separately for server- and client-side errors.

Retry on error

Specifies whether the node should retry a request if the initial request received a response indicating a server-side error.

Number of retries

The maximum number of retries to perform for server errors (count does not include the initial request).

Retry delay

The delay to apply between the first request and the first retry. For each subsequent retry, the delay is doubled.

Rate-limiting error (HTTP 429)

This status code can be returned by a server to indicate that the rate of incoming requests has been too high.

Pause execution

In case of a rate-limiting error, wait for the set amount of time before retrying the request. Note that this delay is static and does not increase with subsequent attempts, neither does it count as retries for server-side errors.

Output additional column with error cause

If enabled, each output row corresponding to a request will contain an additional cell that, in case the request has failed, will provide a description of the error cause. If the request was successful, the cell will contain a missing value.

Views

This node has no views

Workflows

• 02_From_Links_to_Data_DemoKNIME Hub
• 02_From_Links_to_Data_DemoKNIME Hub
• 02_From_Links_to_Data_DemoKNIME Hub
• 02-download_and_analyze_raw_fileKNIME Hub
• 06_NY_Times_RSS_Feed_Tag_CloudKNIME Hub
• Show all 73 workflows

Links

• KNIME E-Learning Course: Sending a GET Request to a REST service

  www.knime.com/knime-introductory-course/chapter2/section3/sending-get-request-to-rest-service
• Data capture from one of the private Big Commerce URL(https) to export it into excelby knimediger on 2021-09-16

  forum.knime.com/p/134752
• Scraping subcategories from main categoriesby armingrudd on 2019-12-07

  forum.knime.com/p/76393

Developers