Icon

exorbyte_​Inter_​Duplicate_​Discovery

This workflow models a duplicate discovery process between a user’s dataset and a reference dataset stored in a MatchMaker index for person data.

The MM sample index is located in a central exorbyte server and the workflow can be executed directly after download. There is no need for further configuration.

This workflow uses the exorbyte matchmaker server extension, which can be installed from the KNIME Partner Extensions Update Site.

exorbyte Inter-Duplicate DiscoveryWorkflow descriptionThis workflow models a duplicate discovery process between a user’s dataset and areference dataset stored in a MatchMaker index.Workflow requirementsexorbyte matchmaker server extension 2.0.xKNIME Analytics Platform >= 4.6, recommended 4.7 Convert relative paths of the rule files to absolute file paths Step 1. Read the dataChoose the data.csv data file from theKNIME workflow data directory. Step 2. MatchEach record from the data is used as aquery to be matched against the MM index.Similar records are returned as the matchresults. The Unmatched Records Table(Output Port 1) displays the data recordsthat did not have a match in the MM index. Step 3. AnnotateAssign the Match Relation, known as thereason why two records are considered amatch. The assignation is done based onthe hit quality of certain fields (Single FieldQuality) and the rule definitions in the rulefiles. Step 4. ClusterFilter the match results based on the Match Relation to include only good relations.The records are then grouped into clusters using the Match Relations. Each clusterhas a unique ID, which is added to each record to identify the cluster they belong to. read the data fileperform fuzzy match against the MM index on the serverfind inter-duplicatesannotate recordswith Match RelationsUsed variables: Relation Rules Definition File:relation_rule_file_pathRelation Levels Definition File:relation_level_file_pathinclude onlygood Match Relationsrule files' relative pathsto the current workflowrelation_ruleto absolute pathrelation_levelto absolute pathadd CLUSTERIDto the dataCSV Reader Matcher MM7.x DedupClusterer MatchAnnotator Nominal ValueRow Filter Table Creator Table Rowto Variable URL to FilePath (Variable) URL to FilePath (Variable) Joiner exorbyte Inter-Duplicate DiscoveryWorkflow descriptionThis workflow models a duplicate discovery process between a user’s dataset and areference dataset stored in a MatchMaker index.Workflow requirementsexorbyte matchmaker server extension 2.0.xKNIME Analytics Platform >= 4.6, recommended 4.7 Convert relative paths of the rule files to absolute file paths Step 1. Read the dataChoose the data.csv data file from theKNIME workflow data directory. Step 2. MatchEach record from the data is used as aquery to be matched against the MM index.Similar records are returned as the matchresults. The Unmatched Records Table(Output Port 1) displays the data recordsthat did not have a match in the MM index. Step 3. AnnotateAssign the Match Relation, known as thereason why two records are considered amatch. The assignation is done based onthe hit quality of certain fields (Single FieldQuality) and the rule definitions in the rulefiles. Step 4. ClusterFilter the match results based on the Match Relation to include only good relations.The records are then grouped into clusters using the Match Relations. Each clusterhas a unique ID, which is added to each record to identify the cluster they belong to. read the data fileperform fuzzy match against the MM index on the serverfind inter-duplicatesannotate recordswith Match RelationsUsed variables: Relation Rules Definition File:relation_rule_file_pathRelation Levels Definition File:relation_level_file_pathinclude onlygood Match Relationsrule files' relative pathsto the current workflowrelation_ruleto absolute pathrelation_levelto absolute pathadd CLUSTERIDto the dataCSV Reader Matcher MM7.x DedupClusterer MatchAnnotator Nominal ValueRow Filter Table Creator Table Rowto Variable URL to FilePath (Variable) URL to FilePath (Variable) Joiner

Nodes

Extensions

Links