Gradient Boosted Decision Trees Classification Learner

Gradient boosting algorithm that also supports categorical data. Calls catboost::catboost.train() from package 'catboost'.

Dictionary

This Learner can be instantiated via lrn():

lrn("classif.catboost")

Meta Information

• Task type: “classif”

• Predict Types: “response”, “prob”

• Feature Types: “numeric”, “factor”, “ordered”

• Required Packages: mlr3, mlr3extralearners, catboost

Parameters

Id	Type	Default	Levels	Range
loss_function_twoclass	character	Logloss	Logloss, CrossEntropy	-
loss_function_multiclass	character	MultiClass	MultiClass, MultiClassOneVsAll	-
learning_rate	numeric	0.03		\([0.001, 1]\)
random_seed	integer	0		\([0, \infty)\)
l2_leaf_reg	numeric	3		\([0, \infty)\)
bootstrap_type	character	-	Bayesian, Bernoulli, MVS, Poisson, No	-
bagging_temperature	numeric	1		\([0, \infty)\)
subsample	numeric	-		\([0, 1]\)
sampling_frequency	character	PerTreeLevel	PerTree, PerTreeLevel	-
sampling_unit	character	Object	Object, Group	-
mvs_reg	numeric	-		\([0, \infty)\)
random_strength	numeric	1		\([0, \infty)\)
depth	integer	6		\([1, 16]\)
grow_policy	character	SymmetricTree	SymmetricTree, Depthwise, Lossguide	-
min_data_in_leaf	integer	1		\([1, \infty)\)
max_leaves	integer	31		\([1, \infty)\)
ignored_features	untyped	NULL		-
one_hot_max_size	untyped	FALSE		-
has_time	logical	FALSE	TRUE, FALSE	-
rsm	numeric	1		\([0.001, 1]\)
nan_mode	character	Min	Min, Max	-
fold_permutation_block	integer	-		\([1, 256]\)
leaf_estimation_method	character	-	Newton, Gradient, Exact	-
leaf_estimation_iterations	integer	-		\([1, \infty)\)
leaf_estimation_backtracking	character	AnyImprovement	No, AnyImprovement, Armijo	-
fold_len_multiplier	numeric	2		\([1.001, \infty)\)
approx_on_full_history	logical	TRUE	TRUE, FALSE	-
class_weights	untyped	-		-
auto_class_weights	character	None	None, Balanced, SqrtBalanced	-
boosting_type	character	-	Ordered, Plain	-
boost_from_average	logical	-	TRUE, FALSE	-
langevin	logical	FALSE	TRUE, FALSE	-
diffusion_temperature	numeric	10000		\([0, \infty)\)
score_function	character	Cosine	Cosine, L2, NewtonCosine, NewtonL2	-
monotone_constraints	untyped	-		-
feature_weights	untyped	-		-
first_feature_use_penalties	untyped	-		-
penalties_coefficient	numeric	1		\([0, \infty)\)
per_object_feature_penalties	untyped	-		-
model_shrink_rate	numeric	-		\((-\infty, \infty)\)
model_shrink_mode	character	-	Constant, Decreasing	-
target_border	numeric	-		\((-\infty, \infty)\)
border_count	integer	-		\([1, 65535]\)
feature_border_type	character	GreedyLogSum	Median, Uniform, UniformAndQuantiles, MaxLogSum, MinEntropy, GreedyLogSum	-
per_float_feature_quantization	untyped	-		-
classes_count	integer	-		\([1, \infty)\)
thread_count	integer	1		\([-1, \infty)\)
task_type	character	CPU	CPU, GPU	-
devices	untyped	-		-
logging_level	character	Silent	Silent, Verbose, Info, Debug	-
metric_period	integer	1		\([1, \infty)\)
train_dir	untyped	"catboost_info"		-
model_size_reg	numeric	0.5		\([0, 1]\)
allow_writing_files	logical	FALSE	TRUE, FALSE	-
save_snapshot	logical	FALSE	TRUE, FALSE	-
snapshot_file	untyped	-		-
snapshot_interval	integer	600		\([1, \infty)\)
simple_ctr	untyped	-		-
combinations_ctr	untyped	-		-
ctr_target_border_count	integer	-		\([1, 255]\)
counter_calc_method	character	Full	SkipTest, Full	-
max_ctr_complexity	integer	-		\([1, \infty)\)
ctr_leaf_count_limit	integer	-		\([1, \infty)\)
store_all_simple_ctr	logical	FALSE	TRUE, FALSE	-
final_ctr_computation_mode	character	Default	Default, Skip	-
verbose	logical	FALSE	TRUE, FALSE	-
ntree_start	integer	0		\([0, \infty)\)
ntree_end	integer	0		\([0, \infty)\)
early_stopping_rounds	integer	-		\([1, \infty)\)
eval_metric	untyped	-		-
use_best_model	logical	-	TRUE, FALSE	-
iterations	integer	1000		\([1, \infty)\)

Installation

See https://catboost.ai/en/docs/concepts/r-installation.

Initial parameter values

• logging_level:

  • Actual default: "Verbose"

  • Adjusted default: "Silent"

  • Reason for change: consistent with other mlr3 learners

• thread_count:

  • Actual default: -1

  • Adjusted default: 1

  • Reason for change: consistent with other mlr3 learners

• allow_writing_files:

  • Actual default: TRUE

  • Adjusted default: FALSE

  • Reason for change: consistent with other mlr3 learners

• save_snapshot:

  • Actual default: TRUE

  • Adjusted default: FALSE

  • Reason for change: consistent with other mlr3 learners

Early stopping

Early stopping can be used to find the optimal number of boosting rounds. Set early_stopping_rounds to an integer value to monitor the performance of the model on the validation set while training. For information on how to configure the validation set, see the Validation section of mlr3::Learner.

References

Dorogush, Veronika A, Ershov, Vasily, Gulin, Andrey (2018). “CatBoost: gradient boosting with categorical features support.” arXiv preprint arXiv:1810.11363.

See also

• Dictionary of Learners: mlr3::mlr_learners.

• as.data.table(mlr_learners) for a table of available Learners in the running session (depending on the loaded packages).

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-learners

• mlr3learners for a selection of recommended learners.

• mlr3cluster for unsupervised clustering learners.

• mlr3pipelines to combine learners with pre- and postprocessing steps.

• mlr3tuning for tuning of hyperparameters, mlr3tuningspaces for established default tuning spaces.

Author

sumny

Super classes

mlr3::Learner -> mlr3::LearnerClassif -> LearnerClassifCatboost

Active bindings

internal_valid_scores

The validation scores for the eval_metric and the loss_function, evaluated on the internal validation data with all tree_count trees, i.e. the trees that are also used for prediction.

best_valid_scores

The validation scores of the best iteration. Because use_best_model truncates the model to the best iteration, these are identical to $internal_valid_scores whenever early stopping is activated. If early stopping is not activated or use_best_model is FALSE, no best iteration is tracked and this is an empty list.

internal_tuned_values

Returns the early stopped iterations if early_stopping_rounds was set during training.

validate

How to construct the internal validation data. This parameter can be either NULL, a ratio, "test", or "predefined".

Methods

Public methods

• LearnerClassifCatboost$new()

• LearnerClassifCatboost$importance()

• LearnerClassifCatboost$clone()

Inherited methods

• mlr3::Learner$base_learner()
• mlr3::Learner$configure()
• mlr3::Learner$encapsulate()
• mlr3::Learner$format()
• mlr3::Learner$help()
• mlr3::Learner$predict()
• mlr3::Learner$predict_newdata()
• mlr3::Learner$print()
• mlr3::Learner$reset()
• mlr3::Learner$selected_features()
• mlr3::Learner$train()
• mlr3::LearnerClassif$predict_newdata_fast()

---

LearnerClassifCatboost$new()

Create a LearnerClassifCatboost object.

Usage

LearnerClassifCatboost$new()

---

LearnerClassifCatboost$importance()

The importance scores are calculated using catboost::catboost.get_feature_importance(), setting type = "FeatureImportance", returned for 'all'.

Usage

LearnerClassifCatboost$importance()

Returns

Named numeric().

---

LearnerClassifCatboost$clone()

The objects of this class are cloneable with this method.

Usage

LearnerClassifCatboost$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# Define the Learner
learner = lrn("classif.catboost", iterations = 10)
print(learner)
#> 
#> ── <LearnerClassifCatboost> (classif.catboost): Gradient Boosting ──────────────
#> • Model: -
#> • Parameters: loss_function_twoclass=Logloss,
#> loss_function_multiclass=MultiClass, thread_count=1, logging_level=Silent,
#> allow_writing_files=FALSE, save_snapshot=FALSE, iterations=10
#> • Validate: NULL
#> • Packages: mlr3, mlr3extralearners, and catboost
#> • Predict Types: [response] and prob
#> • Feature Types: numeric, factor, and ordered
#> • Encapsulation: none (fallback: -)
#> • Properties: importance, internal_tuning, missings, multiclass, twoclass,
#> validation, and weights
#> • Other settings: use_weights = 'use', predict_raw = 'FALSE'

# Define a Task
task = tsk("sonar")

# Create train and test set
ids = partition(task)

# Train the learner on the training ids
learner$train(task, row_ids = ids$train)

print(learner$model)
#> CatBoost model (10 trees)
#> Loss function: Logloss
#> Fit to 60 feature(s)
print(learner$importance())
#>         V12         V11         V46          V9         V48         V21 
#> 11.49152221 10.78857196  6.31969120  5.84219219  5.68614964  5.57396103 
#>         V18         V20         V26         V17         V54         V31 
#>  4.36536490  3.85386194  3.84513251  3.57234627  3.14497696  2.89574837 
#>         V45         V56          V7         V27          V4         V34 
#>  2.23677405  2.11968156  2.07182082  2.03854748  1.91579186  1.82419792 
#>         V35         V28         V57         V10         V24         V25 
#>  1.80924830  1.55053762  1.54689708  1.53743438  1.42686485  1.37303232 
#>         V36         V44          V1         V38         V51         V52 
#>  1.34900681  1.13860276  1.03929495  0.98994659  0.91556734  0.81698551 
#>         V47         V39         V19         V40         V42         V16 
#>  0.76625719  0.75735394  0.65473684  0.61869119  0.50886149  0.49226478 
#>         V23         V30         V15         V50         V14         V13 
#>  0.39780682  0.38343831  0.17169726  0.11006819  0.05907258  0.00000000 
#>          V2         V22         V29          V3         V32         V33 
#>  0.00000000  0.00000000  0.00000000  0.00000000  0.00000000  0.00000000 
#>         V37         V41         V43         V49          V5         V53 
#>  0.00000000  0.00000000  0.00000000  0.00000000  0.00000000  0.00000000 
#>         V55         V58         V59          V6         V60          V8 
#>  0.00000000  0.00000000  0.00000000  0.00000000  0.00000000  0.00000000 

# Make predictions for the test rows
predictions = learner$predict(task, row_ids = ids$test)

# Score the predictions
predictions$score()
#> classif.ce 
#>  0.2608696