Climbing trees 1: what are decision trees?
a year ago
- #algorithms
- #machine-learning
- #decision-trees
- Decision trees are foundational in machine learning, used for both classification and regression tasks.
- They work by splitting data into regions based on feature values, making them interpretable but prone to overfitting.
- Key types include classification trees (for categorical outcomes) and regression trees (for continuous outcomes).
- Popular algorithms include ID3, C4.5, and CART, with CART supporting both classification and regression.
- Decision trees use objective functions like Gini impurity, entropy, or squared loss to optimize splits.
- They struggle with non-hierarchical relationships (e.g., XOR, additive structures) due to axis-parallel splits.
- The 'staircase effect' describes their limitation in modeling smooth or oblique decision boundaries.
- Pros: interpretability, scalability, minimal data prep, handling mixed data types, and non-linear relationships.
- Cons: overfitting, instability, lack of smooth predictions, and difficulty with global dependencies.
- Decision trees do not extrapolate beyond training data bounds, which can be a pro or con depending on context.
- Ensemble methods like bagging and boosting (e.g., random forests, gradient boosting) enhance their performance.