Change search
Link to record
Permanent link

Direct link
BETA
Publications (10 of 39) Show all publications
Linusson, H., Johansson, U., Boström, H. & Löfström, T. (2018). Classification with reject option using conformal prediction. In: Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part I. Paper presented at 22nd Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD 2018; Melbourne; Australia; 3 June 2018 through 6 June 2018 (pp. 94-105). Springer
Open this publication in new window or tab >>Classification with reject option using conformal prediction
2018 (English)In: Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part I, Springer, 2018, p. 94-105Conference paper, Published paper (Refereed)
Abstract [en]

In this paper, we propose a practically useful means of interpreting the predictions produced by a conformal classifier. The proposed interpretation leads to a classifier with a reject option, that allows the user to limit the number of erroneous predictions made on the test set, without any need to reveal the true labels of the test objects. The method described in this paper works by estimating the cumulative error count on a set of predictions provided by a conformal classifier, ordered by their confidence. Given a test set and a user-specified parameter k, the proposed classification procedure outputs the largest possible amount of predictions containing on average at most k errors, while refusing to make predictions for test objects where it is too uncertain. We conduct an empirical evaluation using benchmark datasets, and show that we are able to provide accurate estimates for the error rate on the test set. 

Place, publisher, year, edition, pages
Springer, 2018
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 10937
Keywords
Data mining, Errors, Forecasting, Testing, Uncertainty analysis, Benchmark datasets, Classification procedure, Conformal predictions, Cumulative errors, Empirical evaluations, Error rate, Test object, Test sets, Classification (of information)
National Category
Computer Sciences
Identifiers
urn:nbn:se:hj:diva-41260 (URN)10.1007/978-3-319-93034-3_8 (DOI)000443224400008 ()2-s2.0-85049360232 (Scopus ID)9783319930336 (ISBN)
Conference
22nd Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD 2018; Melbourne; Australia; 3 June 2018 through 6 June 2018
Funder
Knowledge Foundation
Available from: 2018-08-27 Created: 2018-08-27 Last updated: 2018-09-20Bibliographically approved
Johansson, U., Löfström, T., Linusson, H. & Boström, H. (2018). Efficient Venn Predictors using Random Forests. Machine Learning, 1-16
Open this publication in new window or tab >>Efficient Venn Predictors using Random Forests
2018 (English)In: Machine Learning, ISSN 0885-6125, E-ISSN 1573-0565, p. 1-16Article in journal (Refereed) Epub ahead of print
Place, publisher, year, edition, pages
Springer, 2018
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:hj:diva-41127 (URN)10.1007/s10994-018-5753-x (DOI)XYZ ()2-s2.0-85052523706 (Scopus ID)
Available from: 2018-08-13 Created: 2018-08-13 Last updated: 2019-02-07
Johansson, U., Linusson, H., Löfström, T. & Boström, H. (2018). Interpretable regression trees using conformal prediction. Expert systems with applications, 97, 394-404
Open this publication in new window or tab >>Interpretable regression trees using conformal prediction
2018 (English)In: Expert systems with applications, ISSN 0957-4174, E-ISSN 1873-6793, Vol. 97, p. 394-404Article in journal (Refereed) Published
Abstract [en]

A key property of conformal predictors is that they are valid, i.e., their error rate on novel data is bounded by a preset level of confidence. For regression, this is achieved by turning the point predictions of the underlying model into prediction intervals. Thus, the most important performance metric for evaluating conformal regressors is not the error rate, but the size of the prediction intervals, where models generating smaller (more informative) intervals are said to be more efficient. State-of-the-art conformal regressors typically utilize two separate predictive models: the underlying model providing the center point of each prediction interval, and a normalization model used to scale each prediction interval according to the estimated level of difficulty for each test instance. When using a regression tree as the underlying model, this approach may cause test instances falling into a specific leaf to receive different prediction intervals. This clearly deteriorates the interpretability of a conformal regression tree compared to a standard regression tree, since the path from the root to a leaf can no longer be translated into a rule explaining all predictions in that leaf. In fact, the model cannot even be interpreted on its own, i.e., without reference to the corresponding normalization model. Current practice effectively presents two options for constructing conformal regression trees: to employ a (global) normalization model, and thereby sacrifice interpretability; or to avoid normalization, and thereby sacrifice both efficiency and individualized predictions. In this paper, two additional approaches are considered, both employing local normalization: the first approach estimates the difficulty by the standard deviation of the target values in each leaf, while the second approach employs Mondrian conformal prediction, which results in regression trees where each rule (path from root node to leaf node) is independently valid. An empirical evaluation shows that the first approach is as efficient as current state-of-the-art approaches, thus eliminating the efficiency vs. interpretability trade-off present in existing methods. Moreover, it is shown that if a validity guarantee is required for each single rule, as provided by the Mondrian approach, a penalty with respect to efficiency has to be paid, but it is only substantial at very high confidence levels.

Place, publisher, year, edition, pages
Elsevier, 2018
Keywords
Conformal prediction, Interpretability, Predictive regression, Regression trees, Economic and social effects, Efficiency, Forestry, Query processing, Regression analysis, Conformal predictions, Conformal predictors, Empirical evaluations, Level of difficulties, State-of-the-art approach, Forecasting
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:hj:diva-38624 (URN)10.1016/j.eswa.2017.12.041 (DOI)000425074100030 ()2-s2.0-85040125577 (Scopus ID)
Available from: 2018-01-22 Created: 2018-01-22 Last updated: 2018-09-20Bibliographically approved
Boström, H., Linusson, H., Löfström, T. & Johansson, U. (2017). Accelerating difficulty estimation for conformal regression forests. Annals of Mathematics and Artificial Intelligence, 81(1-2), 125-144
Open this publication in new window or tab >>Accelerating difficulty estimation for conformal regression forests
2017 (English)In: Annals of Mathematics and Artificial Intelligence, ISSN 1012-2443, E-ISSN 1573-7470, Vol. 81, no 1-2, p. 125-144Article in journal (Refereed) Published
Abstract [en]

The conformal prediction framework allows for specifying the probability of making incorrect predictions by a user-provided confidence level. In addition to a learning algorithm, the framework requires a real-valued function, called nonconformity measure, to be specified. The nonconformity measure does not affect the error rate, but the resulting efficiency, i.e., the size of output prediction regions, may vary substantially. A recent large-scale empirical evaluation of conformal regression approaches showed that using random forests as the learning algorithm together with a nonconformity measure based on out-of-bag errors normalized using a nearest-neighbor-based difficulty estimate, resulted in state-of-the-art performance with respect to efficiency. However, the nearest-neighbor procedure incurs a significant computational cost. In this study, a more straightforward nonconformity measure is investigated, where the difficulty estimate employed for normalization is based on the variance of the predictions made by the trees in a forest. A large-scale empirical evaluation is presented, showing that both the nearest-neighbor-based and the variance-based measures significantly outperform a standard (non-normalized) nonconformity measure, while no significant difference in efficiency between the two normalized approaches is observed. The evaluation moreover shows that the computational cost of the variance-based measure is several orders of magnitude lower than when employing the nearest-neighbor-based nonconformity measure. The use of out-of-bag instances for calibration does, however, result in nonconformity scores that are distributed differently from those obtained from test instances, questioning the validity of the approach. An adjustment of the variance-based measure is presented, which is shown to be valid and also to have a significant positive effect on the efficiency. For conformal regression forests, the variance-based nonconformity measure is hence a computationally efficient and theoretically well-founded alternative to the nearest-neighbor procedure. 

Place, publisher, year, edition, pages
Springer, 2017
Keywords
Conformal prediction, Nonconformity measures, Random forests, Regression
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:hj:diva-35193 (URN)10.1007/s10472-017-9539-9 (DOI)000407425000008 ()2-s2.0-85014124316 (Scopus ID)
Available from: 2017-03-13 Created: 2017-03-13 Last updated: 2018-09-20Bibliographically approved
Johansson, U., Linusson, H., Löfström, T. & Boström, H. (2017). Model-agnostic nonconformity functions for conformal classification. In: Proceedings of the International Joint Conference on Neural Networks: . Paper presented at 2017 International Joint Conference on Neural Networks, IJCNN 2017, 14 May 2017 through 19 May 2017 (pp. 2072-2079). IEEE
Open this publication in new window or tab >>Model-agnostic nonconformity functions for conformal classification
2017 (English)In: Proceedings of the International Joint Conference on Neural Networks, IEEE, 2017, p. 2072-2079Conference paper, Published paper (Refereed)
Abstract [en]

A conformai predictor outputs prediction regions, for classification label sets. The key property of all conformai predictors is that they are valid, i.e., their error rate on novel data is bounded by a preset significance level. Thus, the key performance metric for evaluating conformal predictors is the size of the output prediction regions, where smaller (more informative) prediction regions are said to be more efficient. All conformal predictions rely on nonconformity functions, measuring the strangeness of an input-output pair, and the efficiency depends critically on the quality of the chosen nonconformity function. In this paper, three model-agnostic nonconformity functions, based on well-known loss functions, are evaluated with regard to how they affect efficiency. In the experimentation on 21 publicly available multi-class data sets, both single neural networks and ensembles of neural networks are used as underlying models for conformal classifiers. The results show that the choice of nonconformity function has a major impact on the efficiency, but also that different nonconformity functions should be used depending on the exact efficiency metric. For a high fraction of single-label predictions, a margin-based nonconformity function is the best option, while a nonconformity function based on the hinge loss obtained the smallest label sets on average.

Place, publisher, year, edition, pages
IEEE, 2017
Keywords
Classification, Conformal prediction, Neural networks, Efficiency, Forecasting, Classification labels, Conformal predictions, Conformal predictors, Label predictions, Loss functions, Performance metrices, Significance levels, Single neural, Classification (of information)
National Category
Computer Sciences
Identifiers
urn:nbn:se:hj:diva-38112 (URN)10.1109/IJCNN.2017.7966105 (DOI)000426968702043 ()2-s2.0-85031028048 (Scopus ID)9781509061815 (ISBN)
Conference
2017 International Joint Conference on Neural Networks, IJCNN 2017, 14 May 2017 through 19 May 2017
Available from: 2017-12-08 Created: 2017-12-08 Last updated: 2018-09-20Bibliographically approved
König, R., Johansson, U., Riveiro, M. & Brattberg, P. (2017). Modeling golf player skill using machine learning. In: Machine Learning and Knowledge Extraction: . Paper presented at 1st IFIP TC 5, WG 8.4, 8.9, 12.9 International Cross-Domain Conference on Machine Learning and Knowledge Extraction, CD-MAKE 2017; Reggio; Italy; 29 August 2017 through 1 September 2017 (pp. 275-294). Springer
Open this publication in new window or tab >>Modeling golf player skill using machine learning
2017 (English)In: Machine Learning and Knowledge Extraction, Springer, 2017, p. 275-294Conference paper, Published paper (Refereed)
Abstract [en]

In this study we apply machine learning techniques to Modeling Golf Player Skill using a dataset consisting of 277 golfers. The dataset includes 28 quantitative metrics, related to the club head at impact and ball flight, captured using a Doppler-radar. For modeling, cost-sensitive decision trees and random forest are used to discern between less skilled players and very good ones, i.e., Hackers and Pros. The results show that both random forest and decision trees achieve high predictive accuracy, with regards to true positive rate, accuracy and area under the ROC-curve. A detailed interpretation of the decision trees shows that they concur with modern swing theory, e.g., consistency is very important, while face angle, club path and dynamic loft are the most important evaluated swing factors, when discerning between Hackers and Pros. Most of the Hackers could be identified by a rather large deviation in one of these values compared to the Pros. Hackers, which had less variation in these aspects of the swing, could instead be identified by a steeper swing plane and a lower club speed. The importance of the swing plane is an interesting finding, since it was not expected and is not easy to explain. © 2017, IFIP International Federation for Information Processing.

Place, publisher, year, edition, pages
Springer, 2017
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 10410
Keywords
Classification, Decision trees, Golf, Machine learning, Swing analysis, Artificial intelligence, Classification (of information), Decision theory, Doppler radar, Extraction, Forestry, Personal computing, Sports, Area under the ROC curve, Large deviations, Machine learning techniques, Predictive accuracy, Quantitative metrics, True positive rates, Learning systems
National Category
Computer Sciences
Identifiers
urn:nbn:se:hj:diva-38113 (URN)10.1007/978-3-319-66808-6_19 (DOI)2-s2.0-85029009266 (Scopus ID)9783319668079 (ISBN)
Conference
1st IFIP TC 5, WG 8.4, 8.9, 12.9 International Cross-Domain Conference on Machine Learning and Knowledge Extraction, CD-MAKE 2017; Reggio; Italy; 29 August 2017 through 1 September 2017
Available from: 2017-12-08 Created: 2017-12-08 Last updated: 2018-09-20Bibliographically approved
Linusson, H., Norinder, U., Boström, H., Johansson, U. & Löfström, T. (2017). On the calibration of aggregated conformal predictors. In: Alex Gammerman, Vladimir Vovk, Zhiyuan Luo, and Harris Papadopoulos (Ed.), Proceedings of Machine Learning Research: Volume 60: Conformal and Probabilistic Prediction and Applications, 13-16 June 2017, Stockholm, Sweden. Paper presented at The 6th Symposium on Conformal and Probabilistic Prediction with Applications, (COPA 2017), 13-16 June, 2017, Stockholm, Sweden (pp. 154-173). Machine Learning Research
Open this publication in new window or tab >>On the calibration of aggregated conformal predictors
Show others...
2017 (English)In: Proceedings of Machine Learning Research: Volume 60: Conformal and Probabilistic Prediction and Applications, 13-16 June 2017, Stockholm, Sweden / [ed] Alex Gammerman, Vladimir Vovk, Zhiyuan Luo, and Harris Papadopoulos, Machine Learning Research , 2017, p. 154-173Conference paper, Published paper (Refereed)
Abstract [en]

Conformal prediction is a learning framework that produces models that associate with each of their predictions a measure of statistically valid confidence. These models are typically constructed on top of traditional machine learning algorithms. An important result of conformal prediction theory is that the models produced are provably valid under relatively weak assumptions—in particular, their validity is independent of the specific underlying learning algorithm on which they are based. Since validity is automatic, much research on conformal predictors has been focused on improving their informational and computational efficiency. As part of the efforts in constructing efficient conformal predictors, aggregated conformal predictors were developed, drawing inspiration from the field of classification and regression ensembles. Unlike early definitions of conformal prediction procedures, the validity of aggregated conformal predictors is not fully understood—while it has been shown that they might attain empirical exact validity under certain circumstances, their theoretical validity is conditional on additional assumptions that require further clarification. In this paper, we show why validity is not automatic for aggregated conformal predictors, and provide a revised definition of aggregated conformal predictors that gains approximate validity conditional on properties of the underlying learning algorithm.

Place, publisher, year, edition, pages
Machine Learning Research, 2017
Keywords
Confidence Predictions, Conformal Prediction, Classification, Ensembles
National Category
Computer Sciences
Identifiers
urn:nbn:se:hj:diva-38123 (URN)
Conference
The 6th Symposium on Conformal and Probabilistic Prediction with Applications, (COPA 2017), 13-16 June, 2017, Stockholm, Sweden
Available from: 2017-12-08 Created: 2017-12-08 Last updated: 2018-09-20Bibliographically approved
Ahlberg, E., Winiwarter, S., Boström, H., Linusson, H., Löfström, T., Norinder, U., . . . Carlsson, L. (2017). Using conformal prediction to prioritize compound synthesis in drug discovery. In: Alex Gammerman, Vladimir Vovk, Zhiyuan Luo, and Harris Papadopoulos (Ed.), Proceedings of Machine Learning Research: Volume 60: Conformal and Probabilistic Prediction and Applications, 13-16 June 2017, Stockholm, Sweden. Paper presented at The 6th Symposium on Conformal and Probabilistic Prediction with Applications, (COPA 2017), 13-16 June, 2017, Stockholm, Sweden (pp. 174-184). Machine Learning Research
Open this publication in new window or tab >>Using conformal prediction to prioritize compound synthesis in drug discovery
Show others...
2017 (English)In: Proceedings of Machine Learning Research: Volume 60: Conformal and Probabilistic Prediction and Applications, 13-16 June 2017, Stockholm, Sweden / [ed] Alex Gammerman, Vladimir Vovk, Zhiyuan Luo, and Harris Papadopoulos, Machine Learning Research , 2017, p. 174-184Conference paper, Published paper (Refereed)
Abstract [en]

The choice of how much money and resources to spend to understand certain problems is of high interest in many areas. This work illustrates how computational models can be more tightly coupled with experiments to generate decision data at lower cost without reducing the quality of the decision. Several different strategies are explored to illustrate the trade off between lowering costs and quality in decisions.

AUC is used as a performance metric and the number of objects that can be learnt from is constrained. Some of the strategies described reach AUC values over 0.9 and outperforms strategies that are more random. The strategies that use conformal predictor p-values show varying results, although some are top performing.

The application studied is taken from the drug discovery process. In the early stages of this process compounds, that potentially could become marketed drugs, are being routinely tested in experimental assays to understand the distribution and interactions in humans.

Place, publisher, year, edition, pages
Machine Learning Research, 2017
Keywords
Drug discovery, Conformal Prediction, ADME properties, Decision support
National Category
Computer Sciences
Identifiers
urn:nbn:se:hj:diva-38124 (URN)
Conference
The 6th Symposium on Conformal and Probabilistic Prediction with Applications, (COPA 2017), 13-16 June, 2017, Stockholm, Sweden
Available from: 2017-12-08 Created: 2017-12-08 Last updated: 2018-09-20Bibliographically approved
Johansson, U., Sundström, M., Håkan, S., Rickard, K. & Jenny, B. (2016). Dataanalys för ökad kundförståelse. Stockholm: Handelsrådet
Open this publication in new window or tab >>Dataanalys för ökad kundförståelse
Show others...
2016 (Swedish)Report (Other (popular science, discussion, etc.))
Place, publisher, year, edition, pages
Stockholm: Handelsrådet, 2016. p. 66
National Category
Business Administration
Identifiers
urn:nbn:se:hj:diva-38075 (URN)
Available from: 2017-03-31 Created: 2017-12-06Bibliographically approved
Boström, H., Linusson, H., Löfström, T. & Johansson, U. (2016). Evaluation of a variance-based nonconformity measure for regression forests. In: Conformal and Probabilistic Prediction with Applications: . Paper presented at 5th International Symposium on Conformal and Probabilistic Prediction with Applications, COPA 2016; Madrid; Spain; 20 April 2016 through 22 April 2016 (pp. 75-89). Springer
Open this publication in new window or tab >>Evaluation of a variance-based nonconformity measure for regression forests
2016 (English)In: Conformal and Probabilistic Prediction with Applications, Springer, 2016, p. 75-89Conference paper, Published paper (Refereed)
Abstract [en]

In a previous large-scale empirical evaluation of conformal regression approaches, random forests using out-of-bag instances for calibration together with a k-nearest neighbor-based nonconformity measure, was shown to obtain state-of-the-art performance with respect to efficiency, i.e., average size of prediction regions. However, the use of the nearest-neighbor procedure not only requires that all training data have to be retained in conjunction with the underlying model, but also that a significant computational overhead is incurred, during both training and testing. In this study, a more straightforward nonconformity measure is investigated, where the difficulty estimate employed for normalization is based on the variance of the predictions made by the trees in a forest. A large-scale empirical evaluation is presented, showing that both the nearest-neighbor-based and the variance-based measures significantly outperform a standard (non-normalized) nonconformity measure, while no significant difference in efficiency between the two normalized approaches is observed. Moreover, the evaluation shows that state-of-theart performance is achieved by the variance-based measure at a computational cost that is several orders of magnitude lower than when employing the nearest-neighbor-based nonconformity measure. 

Place, publisher, year, edition, pages
Springer, 2016
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 9653
Keywords
Conformal prediction, Nonconformity measures, Random forests, Regression, Decision trees, Efficiency, Nearest neighbor search, Regression analysis, Computational overheads, Conformal predictions, Empirical evaluations, State-of-the-art performance, Training and testing, Forecasting
National Category
Computer Sciences
Identifiers
urn:nbn:se:hj:diva-38115 (URN)10.1007/978-3-319-33395-3_6 (DOI)2-s2.0-84964088557 (Scopus ID)9783319333946 (ISBN)
Conference
5th International Symposium on Conformal and Probabilistic Prediction with Applications, COPA 2016; Madrid; Spain; 20 April 2016 through 22 April 2016
Available from: 2017-12-08 Created: 2017-12-08 Last updated: 2018-01-13Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0003-0412-6199

Search in DiVA

Show all publications