Conformal prediction has seen many applications in pharmaceutical science, being able to calibrate outputsof machine learning models and producing valid prediction intervals. We here present the open source softwareCPSign that is a complete implementation of conformal prediction for cheminformatics modeling. CPSign implements inductive and transductive conformal prediction for classifcation and regression, and probabilistic predictionwith the Venn-ABERS methodology. The main chemical representation is signatures but other types of descriptorsare also supported. The main modeling methodology is support vector machines (SVMs), but additional modelingmethods are supported via an extension mechanism, e.g. DeepLearning4J models. We also describe features for visualizing results from conformal models including calibration and efciency plots, as well as features to publish predictive models as REST services. We compare CPSign against other common cheminformatics modeling approachesincluding random forest, and a directed message-passing neural network. The results show that CPSign producesrobust predictive performance with comparative predictive efciency, with superior runtime and lower hardwarerequirements compared to neural network based models. CPSign has been used in several studies and is in production-use in multiple organizations. The ability to work directly with chemical input fles, perform descriptor calculationand modeling with SVM in the conformal prediction framework, with a single software package having a low footprint and fast execution time makes CPSign a convenient and yet fexible package for training, deploying, and predicting on chemical data. CPSign can be downloaded from GitHub at https://github.com/arosbio/cpsign.

Scientifc contribution 

CPSign provides a single software that allows users to perform data preprocessing, modeling and make predictionsdirectly on chemical structures, using conformal and probabilistic prediction. Building and evaluating new modelscan be achieved at a high abstraction level, without sacrifcing fexibility and predictive performance—showcasedwith a method evaluation against contemporary modeling approaches, where CPSign performs on par with a stateof-the-art deep learning based model.

We apply Conformal Predictive Distribution Systems (CPDS) and a non-exchangeable version of the traditional Conformal Prediction (NECP) method to short-term wind speed forecasting to generate probabilistic forecasts. These are compared to the more traditional Quantile Regression Forest (QRF) method. A short-term forecast is available from a few hours before the forecasted time period and is only extended a couple days into the future. The methods are supplied ensemble forecasts as input and additionally the Conformal methods are supplied with post-processed point forecasts for generating the probability distributions. In the NECP case we propose a method of producing probability distributions by creating sequentially larger prediction intervals. The methods are compared through a teaching schedule, to mimic a real-world setting. For each model update in the teaching schedule a grid-search approach is applied to select each method’s optimal hyperparameters, respectively. The methods are tested out of the box with tweaks to few hyperparameters. We also introduce a normalized nonconformity score and use it with the conformal method that handles data that violates the exchangeability assumption. The resulting probability distributions are compared to actual wind measurements through Continuous Ranked Probability Scores (CRPS) as well as their validity and efficiency of certain prediction intervals. Our results suggest that the conformal based methods, with the pre-trained underlying model, produce slightly more conservative but more efficient probability distributions than QRF at a lower computational cost. We further propose how the conformal-based methods could be improved for the application to real-world scenarios.

The success of revenue management models depends to a large extent on the quality of historical data used to forecast future bookings. Several theoretical models and best practices of handing historical data have been developed over the years, that all rely on assumptions about underlying distribution and seasonality in the historical data. In this paper, we describe a novel method that compares the fingerprints of the departure to optimise and selects historical departures without making assumptions on data distribution or seasonality. By evaluating the method at the departure level and using the Nemenyi rank test, we show the method’s application in the ferry transportation business and discuss its advantages.