Machine Learning: Functional Trees

The generalization ability of a learning algorithm depends on the appropriateness of its representation language to express a generalization of the examples for the given task. Different learning algorithms employ different representations, search heuristics, evaluation functions, and search spaces. It is now commonly accepted that each algorithm has its own selective superiority (Brodley, 1995); each one is best for some but not all tasks. The design of algorithms that explore multiple representation languages and explore different search spaces has an intuitive appeal.

In the context of supervised learning problems it is useful to distinguish between classification problems and regression problems. In the former the target variable takes values in a finite and pre-defined set of un-ordered values, and the usual goal is to minimize a 0-1 loss function. In the latter the target variable is ordered and takes values in a subset of R. The usual goal is to minimize a squared error loss function. Mainly due to the differences in the type of the target variable successful techniques in one class of problems are not directly applicable to the other class of problems.