Bootstrapping (machine learning)

Bootstrap aggregating, also called bagging, is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression. It also reduces variance and helps to avoid overfitting. Although it is usually applied to decision tree methods, it can be used with any type of method. Bagging is a special case of the model averaging approach.

Bagging (Bootstrap aggregating) was proposed by Leo Breiman in 1994 to improve classification by combining classifications of randomly generated training sets. See Breiman, 1994. Technical Report No. 421.

Given a standard training set ${\displaystyle D}$ of size n, bagging generates m new training sets ${\displaystyle D_{i}}$, each of size n′, by sampling from D uniformly and with replacement. By sampling with replacement, some observations may be repeated in each ${\displaystyle D_{i}}$. If n′=n, then for large n the set ${\displaystyle D_{i}}$ is expected to have the fraction (1 - 1/e) (≈63.2%) of the unique examples of D, the rest being duplicates. This kind of sample is known as a bootstrap sample. The m models are fitted using the above m bootstrap samples and combined by averaging the output (for regression) or voting (for classification).

...
Wikipedia