Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Accelerating difficulty estimation for conformal regression forests
Department of Computer and Systems Sciences, Stockholm University, Stockholm, Sweden.
Department of Information Technology, University of Borås, Borås, Sweden.
Department of Information Technology, University of Borås, Borås, Sweden.
Jönköping University, School of Engineering, JTH, Computer Science and Informatics. Jönköping University, School of Engineering, JTH. Research area Computer Science and Informatics.ORCID iD: 0000-0003-0412-6199
2017 (English)In: Annals of Mathematics and Artificial Intelligence, ISSN 1012-2443, E-ISSN 1573-7470, Vol. 81, no 1-2, 125-144 p.Article in journal (Refereed) Published
Abstract [en]

The conformal prediction framework allows for specifying the probability of making incorrect predictions by a user-provided confidence level. In addition to a learning algorithm, the framework requires a real-valued function, called nonconformity measure, to be specified. The nonconformity measure does not affect the error rate, but the resulting efficiency, i.e., the size of output prediction regions, may vary substantially. A recent large-scale empirical evaluation of conformal regression approaches showed that using random forests as the learning algorithm together with a nonconformity measure based on out-of-bag errors normalized using a nearest-neighbor-based difficulty estimate, resulted in state-of-the-art performance with respect to efficiency. However, the nearest-neighbor procedure incurs a significant computational cost. In this study, a more straightforward nonconformity measure is investigated, where the difficulty estimate employed for normalization is based on the variance of the predictions made by the trees in a forest. A large-scale empirical evaluation is presented, showing that both the nearest-neighbor-based and the variance-based measures significantly outperform a standard (non-normalized) nonconformity measure, while no significant difference in efficiency between the two normalized approaches is observed. The evaluation moreover shows that the computational cost of the variance-based measure is several orders of magnitude lower than when employing the nearest-neighbor-based nonconformity measure. The use of out-of-bag instances for calibration does, however, result in nonconformity scores that are distributed differently from those obtained from test instances, questioning the validity of the approach. An adjustment of the variance-based measure is presented, which is shown to be valid and also to have a significant positive effect on the efficiency. For conformal regression forests, the variance-based nonconformity measure is hence a computationally efficient and theoretically well-founded alternative to the nearest-neighbor procedure. 

Place, publisher, year, edition, pages
Springer, 2017. Vol. 81, no 1-2, 125-144 p.
Keyword [en]
Conformal prediction, Nonconformity measures, Random forests, Regression
National Category
Probability Theory and Statistics
Identifiers
URN: urn:nbn:se:hj:diva-35193DOI: 10.1007/s10472-017-9539-9ISI: 000407425000008Scopus ID: 2-s2.0-85014124316OAI: oai:DiVA.org:hj-35193DiVA: diva2:1081270
Available from: 2017-03-13 Created: 2017-03-13 Last updated: 2017-12-08Bibliographically approved

Open Access in DiVA

Fulltext(1109 kB)17 downloads
File information
File name FULLTEXT01.pdfFile size 1109 kBChecksum SHA-512
4060b93d24f2cbbadde183cf667bbcc17f74917ccfc9c38fc30be3e05aebacfad55482d14f6c3565617f0100ab09c50d1671bbb0de2dfb37b4381097395b1900
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records BETA

Löfström, TuveJohansson, Ulf

Search in DiVA

By author/editor
Löfström, TuveJohansson, Ulf
By organisation
JTH, Computer Science and InformaticsJTH. Research area Computer Science and Informatics
In the same journal
Annals of Mathematics and Artificial Intelligence
Probability Theory and Statistics

Search outside of DiVA

GoogleGoogle Scholar
Total: 17 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 189 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf