Spark Association Rule (Apply)

This node applies association rules created by a Spark rule learner*. The rules are applied to a given column with item sets (transactions). A rule matches an item set, if the item set contains all antecedent items of the rule. The consequent items of the matching rules are then added to the output set of predicted items if they are not already included in the input items collection (the output set contains only new items).

To avoid memory and computations explosions, the number of used association rules can be limited. A warning will be added to the node if the limit is reached and only the first n rules are used (this means randomly chosen on a unsorted set of rules).

The Spark Association Rule Learner Node can be used to generate the association rules. The Spark GroupBy or Spark SQL Node can be used to generate a collection of items as input.

Rules with missing values in the selected antecedent or consequent column and item rows with missing values in the selected item column are removed.

This node requires at least Apache Spark 2.0.


(*) RULE LEARNER is a registered trademark of Minitab, LLC and is used with Minitab’s permission.

Options

Antecedent Column
Collection column from the first input table, which holds the antecedent item sets of the association rules.
Consequent Column
Column from the first input table, which holds the consequent items (one item per row) of the association rules.
Rule limit
Optional: Maximum number of association rules to use.
Item Column
Collection column from the second input table, which holds the transaction item sets to apply the association rules on.
Output Column
Name of the resulting output column, where each row holds a collection of the consequent items of the matching rules.

Input Ports

Icon
Spark DataFrame with association rules
Icon
Spark DataFrame with item collection column

Output Ports

Icon
Spark DataFrame with output column that holds the items which were predicted according to the association rules

Views

This node has no views

Workflows

Links

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.