URL Normalizer

Go to Product

This node allows to transform URLs to a canonicalized representation, e. g. for matching them in data mining scenarios. It is not intended, to produce equally working URLs in all cases. The following steps are performed by this node:

  • Lower case URL
  • Add http:// protocol, if not present
  • Transform https:// to http:// protocol
  • Remove session IDs from URLs
  • Normalize relative path components (e. g. "..")
  • Remove trailing slashes from URLs
  • Remove “index.htm*” part from URLs
  • Sort query parameters alphabetically

Options

URL
The column which holds the URLs to process.

Input Ports

Icon
Table which contains a column with URLs to process.

Output Ports

Icon
Input table with appended column for canonicalized URLs.

Popular Predecessors

  • URL Extractor26 %
  • Empty Table Creator10 %
  • Table Creator8 %
  • URL Resolver5 %
  • Column Filter4 %
  • URL Domain Extractor4 %
  • Row Filter3 %
  • String Manipulation3 %
  • Excel Reader3 %
  • Chunk Loop Start2 %
  • Form Encoded HTTP Entity Creator2 %
  • Row Splitter2 %
  • GroupBy2 %
  • XPath2 %
  • URL Normalizer2 %
  • HTTP Result Data Extractor2 %
  • Webpage Retriever2 %
  • String to URI< 1 %
  • File Reader< 1 %
  • List Files< 1 %
  • Java Snippet< 1 %
  • Joiner< 1 %
  • Parallel Chunk End< 1 %
  • XLS Reader< 1 %
  • Java Snippet (simple)< 1 %
  • Google Analytics Query< 1 %
  • String to JSON< 1 %
  • URL To Audio< 1 %
  • CSV Reader< 1 %
  • Line Reader< 1 %
  • Variable Condition Loop End< 1 %
  • Cross Joiner< 1 %
  • Column Rename< 1 %
  • String Replacer< 1 %
  • Rule-based Row Filter< 1 %
  • Rule-based Row Splitter< 1 %
  • Time Field Extractor (legacy)< 1 %
  • PalladianNer< 1 %
  • HTML Parser< 1 %
  • URL to File Path< 1 %
  • Missing Value< 1 %

Views

This node has no views

Workflows

  • No workflows found

Links

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.