Supporting information and data for the paper "A probabilistic classifier ensemble weighting scheme based on cross validated accuracy estimates"

Our hypothesis is that building ensembles of small sets of strong classifiers constructed with different learning algorithms is, on average, the best approach to classification for real world problems. We propose a simple mechanism for building small heterogeneous ensembles based on exponentially weighting the probability estimates of the base classifiers with an estimate of the accuracy formed through cross-validation on the train data. We demonstrate through extensive experimentation that, given the same small set of base classifiers, this method has measurable benefits over commonly used alternative weighting, selection or meta classifier approaches to heterogeneous ensembles. We also show how an ensemble of five well known, fast classifiers can produce an ensemble that is not significantly worse than large homogeneous ensembles and tuned individual classifiers on datasets from the UCI archive. We provide evidence that the performance of the Cross-validation Accuracy Weighted Probabilistic Ensemble (CAWPE) generalises to a completely separate set of datasets, the UCR time series classification archive, and we also demonstrate that our ensemble technique can significantly improve the state-of-the-art classifier for this problem domain. We investigate the performance in more detail, and find that the improvement is most marked with problems with smaller train sets. We perform a sensitivity analysis and an ablation study to demonstrate the robustness of the ensemble and the significant contribution of each design element of the classifier. We conclude that it is, on average, better to ensemble strong classifiers with a weighting scheme rather than perform extensive tuning and that CAWPE is a sensible starting point for combining classifiers.

The full, up to date codebase can be found here, however a stable branch at the time of final submission with accompanying summary results can be found here

However, our codebase is of course being constantly updated, including the methods of results storage and analysis. A snapshot of the code when results were initially created is available here.

Generating data resamples is deterministic and can be reproduced using the method (Experiments.sampleDataset(directory,datasetName,foldID), using the UCI data or alternatively all folds can be downloaded (3.5 GB).

In the course of experiments we have generated gigabytes of prediction information and results. These are available in spreadsheets and in raw format (9 GB).

Results summaries and analysis leading to each figure can also be downloaded.