Tree Ensemble Learner

Learns an ensemble of decision trees (such as random forest* variants). Typically, each tree is built with a different set of rows (records) and/or columns (attributes). See the options for Data Sampling and Attribute Sampling for more details. The attributes can also be provided as bit (fingerprint), byte, or double vector. The output model describes an ensemble of decision tree models and is applied in the corresponding predictor node using the selected aggregation mode to aggregate the votes of the individual decision trees.

The following configuration settings learn a model that is similar to the random forest™ classifier described by Leo Breiman and Adele Cutler:

Tree Options - Split Criterion: Gini Index
Tree Options - Limit number of levels (tree depth): unlimited
Tree Options - Minimum node size: unlimited
Ensemble Configuration - Number of models: Arbitrary (arguably, random forest does not overfit)
Ensemble Configuration - Data Sampling: Use all rows (fraction = 1) but choose sampling with replacement (bootstrapping)
Ensemble Configuration - Attribute Sampling: Sample using a different set of attributes for each tree node split; usually square root of number of attributes - but this can vary

Experiments have shown that results on different datasets are very similar to the random forest implementation available in R.

Decision tree construction takes place in main memory (all data and all models are kept in memory).

The missing value handling corresponds to the method described here. The basic idea is that for each split to try and send the missing values in every possible direction and the one yielding the best results (i.e. largest gain) is then used. If no missing values are present during training, the direction of the split that the most records are following is chosen as the direction for missing values during testing.

The tree ensemble nodes now also support binary splits for nominal columns. Depending on the kind of problem (two- or multi-class), different algorithms are implemented to enable the efficient calculation of splits.

For two-class classification problems the method described in section 9.4 of "Classification and Regression Trees" by Breiman et al. (1984) is used.
For multi-class classification problems the method described in "Partitioning Nominal Attributes in Decision Trees" by Coppersmith et al. (1999) is used.

(*) RANDOM FORESTS is a registered trademark of Minitab, LLC and is used with Minitab’s permission.

Options

Attribute Selection

Target Column

Select the column containing the value to be learned. Rows with missing values in this column are ignored during the learning process.

Attribute Selection

Select the attributes on which the model should be learned. You can choose from two modes.

Fingerprint attribute Uses a fingerprint/vector (bit, byte and double are possible) column to learn the model by treating each entry of the vector as a separate attribute (e.g. a bit vector of length 1024 is expanded into 1024 binary attributes). The node requires all vectors to be of the same length.

Column attributes Uses ordinary columns in your table (e.g. String, Double, Integer, etc.) as attributes to learn the model on. The dialog allows you to select the columns manually (by moving them to the right panel) or via a wildcard/regex selection (all columns whose names match the wildcard/regex are used for learning). In case of manual selection, the behavior for new columns (i.e. that are not available at the time you configure the node) can be specified as either Enforce exclusion (new columns are excluded and therefore not used for learning) or Enforce inclusion (new columns are included and therefore used for learning).

Ignore columns without domain information

If selected, nominal columns with no domain information are ignored (as they likely have too many possible values anyway).

Enable Hightlighting (#patterns to store)

If selected, the node stores the selected number of rows and allows highlighting them in the node view.

Save target distribution in tree nodes (memory expensive - only important for tree view and PMML export)

If selected, the model stores the distribution of the target category values in each tree node. Storing the class distribution may increase memory consumption considerably and we therefore recommend disabling it if your use-case doesn't require it. Class distribution is only needed if

You want to see the class distribution for each tree node in the node view.
You want to export individual decision trees to PMML.
You want to use soft-voting (i.e. aggregation of probability distributions instead of votes) in the predictor node.

Tree Options

Split Criterion: Choose the split criterion here. Gini is usually a good choice and is used in "Classification and Regression Trees" (Breiman et al, 1984) and the original random forest algorithm (as described by Breiman et al, 2001); information gain is used in C4.5; the information gain ratio normalizes the standard information gain by the split entropy to overcome any unfair preference for nominal splits with many child nodes.
Use mid points splits (only for numeric attributes): For numerical splits, uses the middle point between two class boundaries. If unselected the split attribute value is the smaller value with "<=" relationship.
Use binary splits for nominal columns: If selected, nominal columns also produce binary splits instead of multiway splits in which each nominal value corresponds to one child node.
Limit number of levels (tree depth): Number of tree levels to be learned. For instance, a value of 1 would only split the (single) root node resulting in a decision stump.
Minimum split node size: Minimum number of records in a decision tree node so that another split is attempted. Note, this option does not make any implications on the minimum number of records in a terminal node. If enabled, this number needs to be at least twice as large as the minimum child node size (as otherwise for binary splits one of the two children would have less records than specified).
Minimum child node size: Minimum number of records in child nodes. It can be at most half of the minimum split node size (see above). Note, this parameter is currently ignored for nominal columns if binary nominal splits are disabled.
Use fixed root attribute: If selected, the chosen column is used as the root split attribute in all decision trees -- even if the column is not in the attribute sample.

Ensemble Configuration

Number of models

The number of decision trees to be learned. For most datasets, a value between 100 and 500 yields good results, however the optimal number is data dependent and should thus be subject to hyperparameter tuning.

Data Sampling (Rows)

The sampling of the data rows for each individual tree: If disabled each tree learner gets the full dataset, otherwise each tree is learned with a different data sample. A data fraction of 1 (=100%) chosen "with replacement" is called bootstrapping (used in random forests). For sufficiently large datasets this bootstrap sample contains about 2/3 different data rows from the input, some of which are replicated multiple times. Rows that are not used in the training of a tree are called out-of-bag (see below).

Data Sampling Mode

The sampling mode decides how the rows are sampled. In the random mode, the rows are sampled from the whole dataset i.e. each row has exactly the same probability as in the sample. In case of equal size sampling, first a sample from the minority class is drawn and then the same number of rows as in the minority sample are drawn from all other classes i.e. each class is represented with the same number of rows in the sample. If stratified sampling is selected, the same fraction of rows is drawn from each class i.e. the class distribution in the sample is approximately the same as in the full set of rows.

Attribute Sampling (Columns)

Defines the sampling of attributes to learn an individual tree. This can either be a function based on the number of attributes (linear fraction or square root) or some absolute value. The latter can be used in conjunction with flow variables to inject some other value derived from the number of attributes (e.g. Breiman suggests starting with the square root of number of attributes but also to try to double/half that number).

Attribute Selection

Use the same set of attributes for each tree means that the attributes are sampled once for each tree and this sample is then used to construct the tree.

Use a different set of attributes for each tree node samples a different set of candidate attributes in each of the tree nodes from which the optimal one is chosen to perform the split. This is the option used in random forests.

Use static random seed

Choose a seed to get reproducible results.

Input Ports

: The data to be learned from. It must contain at least one nominal target column and either a fingerprint (bit/byte/double vector) column or another numeric or nominal column.

Output Ports

: The input data with the out-of-bag predictions, i.e. for each input row the majority vote of all models that did not use the row during their training. If the entire data was used to train the individual models then this output will contain the input data with missing values in the prediction columns. The appended columns are equivalent to the columns appended by the corresponding predictor node. There is one additional column model count, which contains the number of models used for voting (number of models not using the row throughout the learning.) The out-of-bag predictions can be used to get an estimate of the generalization error of the tree ensemble by feeding them into the Scorer node.
: A statistics table on the attributes used in the different trees. Each row represents one training attribute with these statistics: #splits (level x) as the number of models, which use the attribute as split on level x (with level 0 as root split); #candidates (level x) is the number of times an attribute was in the attribute sample for level x (in a random forest setup these samples differ from node to node). If no attribute sampling is used #candidates is the number of models. Note, these numbers are uncorrected, i.e. if an attribute is selected on level 0 but is also in the candidate set of level 1 (but is not split on level 1 because it has been split one level up), the #candidate number still counts the attribute as a candidate.
: The trained model.

Popular Predecessors

Popular Successors

Views

Tree Views: A decision tree viewer for all the trained models. Use the spinner to iterate through the different models.

Workflows

Developers

You want to see the source code for this node? Click the following button and we’ll use our super-powers to find it for you.

Installation

To use this node in KNIME, install the extension KNIME Ensemble Learning Wrappers from the below update site following our NodePit Product and Node Installation Guide:

v5.3

A zipped version of the software site can be downloaded here.

Plugin provider: KNIME AG, Zurich, Switzerland

Plugin version: 5.3.0.v202401171201

On NodePit since: 2024-07-09

Last update: 2024-11-21

KNIME versions: Since v3.6

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!