Title:  Evaluate Clinical Prediction Models by Net Monetary Benefit 

Description:  Estimates when and where a modelguided treatment strategy may outperform a treatall or treatnone approach by Monte Carlo simulation and evaluation of the Net Monetary Benefit. Details can be viewed in Parsons et al. (2023) <doi:10.21105/joss.05328>. 
Authors:  Rex Parsons [aut, cre] , Robin Blythe [aut] , Adrian Barnett [aut] , Emi Tanaka [rev] (Emi Tanaka reviewed predictNMB for rOpenSci, see <https://github.com/ropensci/softwarereview/issues/566>.), Tinula Kariyawasam [rev] (Tinula Kariyawasam reviewed predictNMB for rOpenSci, see <https://github.com/ropensci/softwarereview/issues/566>.), Susanna Cramb [ctb] , Steven McPhail [ctb] 
Maintainer:  Rex Parsons <[email protected]> 
License:  GPL (>= 3) 
Version:  0.2.1.9000 
Built:  20241104 06:21:52 UTC 
Source:  https://github.com/ropensci/predictNMB 
Create plots of from screened predictNMB simulations.
## S3 method for class 'predictNMBscreen' autoplot( object, x_axis_var = NULL, constants = list(), what = c("nmb", "inb", "cutpoints", "qalys", "costs"), inb_ref_col = NA, plot_range = TRUE, plot_conf_level = TRUE, plot_line = TRUE, plot_alpha = 0.5, dodge_width = 0, conf.level = 0.95, methods_order = NULL, rename_vector, ... )
## S3 method for class 'predictNMBscreen' autoplot( object, x_axis_var = NULL, constants = list(), what = c("nmb", "inb", "cutpoints", "qalys", "costs"), inb_ref_col = NA, plot_range = TRUE, plot_conf_level = TRUE, plot_line = TRUE, plot_alpha = 0.5, dodge_width = 0, conf.level = 0.95, methods_order = NULL, rename_vector, ... )
object 
A 
x_axis_var 
The desired screened factor to be displayed along the x axis. For example, if the simulation screen was used with many values for event rate, this could be "event_rate". Defaults to the first detected, varied input. 
constants 
Named vector. If multiple inputs were screened in this object, this argument can be used to modify the selected values for all those except the input that's varying along the xaxis. See the summarising methods vignette. 
what 
What to summarise: one of "nmb", "inb", "cutpoints", "qalys" or "costs". Defaults to "nmb". 
inb_ref_col 
Which cutpoint method to use as the reference strategy
when calculating the incremental net monetary benefit.
See 
plot_range 

plot_conf_level 

plot_line 

plot_alpha 
Alpha value (transparency) of all plot elements. Defaults to 0.5. 
dodge_width 
The dodge width of plot elements. Can be used to avoid excessive overlap between methods. Defaults to 0. 
conf.level 
The confidence level of the interval. Defaults to 0.95 (coloured area of distribution represents 95% CIs). 
methods_order 
The order (left to right) to display the cutpoint methods. 
rename_vector 
A named vector for renaming the methods in the summary. The values of the vector are the default names and the names given are the desired names in the output. 
... 
Additional (unused) arguments. 
This plot method works with predictNMBscreen
objects that are
created using screen_simulation_inputs()
. Can be used to visualise
distributions from many different simulations and assign a varying input
to the xaxis of the plot.
Returns a ggplot
object.
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_screen_obj < screen_simulation_inputs( n_sims = 50, n_valid = 10000, sim_auc = seq(0.7, 0.9, 0.1), event_rate = c(0.1, 0.2, 0.3), fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb, cutpoint_methods = c("all", "none", "youden", "value_optimising") ) autoplot(sim_screen_obj) autoplot( sim_screen_obj, x_axis_var = "event_rate", constants = c(sim_auc = 0.8), dodge_width = 0.02, rename_vector = c( "ValueOptimising" = "value_optimising", "TreatNone" = "none", "TreatAll" = "all", "Youden Index" = "youden" ) )
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_screen_obj < screen_simulation_inputs( n_sims = 50, n_valid = 10000, sim_auc = seq(0.7, 0.9, 0.1), event_rate = c(0.1, 0.2, 0.3), fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb, cutpoint_methods = c("all", "none", "youden", "value_optimising") ) autoplot(sim_screen_obj) autoplot( sim_screen_obj, x_axis_var = "event_rate", constants = c(sim_auc = 0.8), dodge_width = 0.02, rename_vector = c( "ValueOptimising" = "value_optimising", "TreatNone" = "none", "TreatAll" = "all", "Youden Index" = "youden" ) )
Create plots of from predictNMB simulations.
## S3 method for class 'predictNMBsim' autoplot( object, what = c("nmb", "inb", "cutpoints", "qalys", "costs"), inb_ref_col = NA, conf.level = 0.95, methods_order = NULL, n_bins = 40, label_wrap_width = 12, fill_cols = c("grey50", "#ADD8E6"), median_line_size = 2, median_line_alpha = 0.5, median_line_col = "black", rename_vector, ... )
## S3 method for class 'predictNMBsim' autoplot( object, what = c("nmb", "inb", "cutpoints", "qalys", "costs"), inb_ref_col = NA, conf.level = 0.95, methods_order = NULL, n_bins = 40, label_wrap_width = 12, fill_cols = c("grey50", "#ADD8E6"), median_line_size = 2, median_line_alpha = 0.5, median_line_col = "black", rename_vector, ... )
object 
A 
what 
What to summarise: one of "nmb", "inb", "cutpoints", "qalys" or "costs". Defaults to "nmb". 
inb_ref_col 
Which cutpoint method to use as the reference strategy
when calculating the incremental net monetary benefit.
See 
conf.level 
The confidence level of the interval. Defaults to 0.95 (coloured area of distribution represents 95% CIs). 
methods_order 
The order (left to right) to display the cutpoint methods. 
n_bins 
The number of bins used when constructing histograms. Defaults to 40. 
label_wrap_width 
The number of characters in facet labels at which the label is wrapped. Default is 12. 
fill_cols 
Vector containing the colours used for fill aesthetic of
histograms. The first colour represents the area outside of the confidence
region, second colour shows the confidence region. Defaults to

median_line_size 
Size of line used to represent the median of distribution. Defaults to 2. 
median_line_alpha 
Alpha (transparency) for line used to represent the median of distribution. Defaults to 0.5. 
median_line_col 
Colour of line used to represent the median of
distribution. Defaults to 
rename_vector 
A named vector for renaming the methods in the summary. The values of the vector are the default names and the names given are the desired names in the output. 
... 
Additional (unused) arguments. 
This plot method works with predictNMBsim
objects that are created
using do_nmb_sim()
. Can be used to visualise distributions from
simulations for different cutpoint methods.
Returns a ggplot
object.
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb, cutpoint_methods = c("all", "none", "youden", "value_optimising") ) autoplot( sim_obj, rename_vector = c( "Value Optimising" = "value_optimising", "Treat None" = "none", "Treat All" = "all", "Youden Index" = "youden" ) ) + theme_sim()
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb, cutpoint_methods = c("all", "none", "youden", "value_optimising") ) autoplot( sim_obj, rename_vector = c( "Value Optimising" = "value_optimising", "Treat None" = "none", "Treat All" = "all", "Youden Index" = "youden" ) ) + theme_sim()
Create a costeffectiveness plot.
ce_plot( object, ref_col, wtp, show_wtp = TRUE, methods_order = NULL, rename_vector, shape = 21, wtp_linetype = "dashed", add_prop_ce = FALSE, ... )
ce_plot( object, ref_col, wtp, show_wtp = TRUE, methods_order = NULL, rename_vector, shape = 21, wtp_linetype = "dashed", add_prop_ce = FALSE, ... )
object 
A 
ref_col 
Which cutpoint method to use as the reference strategy when calculating the incremental net monetary benefit. Often sensible to use a "all" or "none" approach for this. 
wtp 
A 
show_wtp 
A 
methods_order 
The order (within the legend) to display the cutpoint methods. 
rename_vector 
A named vector for renaming the methods in the summary. The values of the vector are the default names and the names given are the desired names in the output. 
shape 
The 
wtp_linetype 
The 
add_prop_ce 
Whether to append the proportion of simulations for that
method which were costeffective (beneath the WTP threshold)
to their labels in the legend. Only applicable when 
... 
Additional (unused) arguments. 
This plot method works with predictNMBsim
objects that are created
using do_nmb_sim()
. Can be used to visualise the simulations on a
costeffectiveness plot (costs vs effectiveness)
Returns a ggplot
object.
get_nmb_evaluation < get_nmb_sampler( qalys_lost = function() rnorm(1, 0.33, 0.03), wtp = 28000, high_risk_group_treatment_effect = function() exp(rnorm(n = 1, mean = log(0.58), sd = 0.43)), high_risk_group_treatment_cost = function() rnorm(n = 1, mean = 161, sd = 49) ) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb_evaluation, fx_nmb_evaluation = get_nmb_evaluation ) ce_plot(sim_obj, ref_col = "all")
get_nmb_evaluation < get_nmb_sampler( qalys_lost = function() rnorm(1, 0.33, 0.03), wtp = 28000, high_risk_group_treatment_effect = function() exp(rnorm(n = 1, mean = log(0.58), sd = 0.43)), high_risk_group_treatment_cost = function() rnorm(n = 1, mean = 161, sd = 49) ) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb_evaluation, fx_nmb_evaluation = get_nmb_evaluation ) ce_plot(sim_obj, ref_col = "all")
Create a costeffectiveness plot.
## S3 method for class 'predictNMBsim' ce_plot( object, ref_col, wtp, show_wtp = TRUE, methods_order = NULL, rename_vector, shape = 21, wtp_linetype = "dashed", add_prop_ce = FALSE, ... )
## S3 method for class 'predictNMBsim' ce_plot( object, ref_col, wtp, show_wtp = TRUE, methods_order = NULL, rename_vector, shape = 21, wtp_linetype = "dashed", add_prop_ce = FALSE, ... )
object 
A 
ref_col 
Which cutpoint method to use as the reference strategy when calculating the incremental net monetary benefit. Often sensible to use a "all" or "none" approach for this. 
wtp 
A 
show_wtp 
A 
methods_order 
The order (within the legend) to display the cutpoint methods. 
rename_vector 
A named vector for renaming the methods in the summary. The values of the vector are the default names and the names given are the desired names in the output. 
shape 
The 
wtp_linetype 
The 
add_prop_ce 
Whether to append the proportion of simulations for that
method which were costeffective (beneath the WTP threshold)
to their labels in the legend. Only applicable when 
... 
Additional (unused) arguments. 
This plot method works with predictNMBsim
objects that are created
using do_nmb_sim()
. Can be used to visualise the simulations on a
costeffectiveness plot (costs vs effectiveness)
Returns a ggplot
object.
get_nmb_evaluation < get_nmb_sampler( qalys_lost = function() rnorm(1, 0.33, 0.03), wtp = 28000, high_risk_group_treatment_effect = function() exp(rnorm(n = 1, mean = log(0.58), sd = 0.43)), high_risk_group_treatment_cost = function() rnorm(n = 1, mean = 161, sd = 49) ) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb_evaluation, fx_nmb_evaluation = get_nmb_evaluation ) ce_plot(sim_obj, ref_col = "all")
get_nmb_evaluation < get_nmb_sampler( qalys_lost = function() rnorm(1, 0.33, 0.03), wtp = 28000, high_risk_group_treatment_effect = function() exp(rnorm(n = 1, mean = log(0.58), sd = 0.43)), high_risk_group_treatment_cost = function() rnorm(n = 1, mean = 161, sd = 49) ) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb_evaluation, fx_nmb_evaluation = get_nmb_evaluation ) ce_plot(sim_obj, ref_col = "all")
Do the predictNMB simulation, evaluating the net monetary benefit (NMB) of the simulated model.
do_nmb_sim( sample_size, n_sims, n_valid, sim_auc, event_rate, cutpoint_methods = get_inbuilt_cutpoint_methods(), fx_nmb_training, fx_nmb_evaluation, meet_min_events = TRUE, min_events = NA, show_progress = FALSE, cl = NULL )
do_nmb_sim( sample_size, n_sims, n_valid, sim_auc, event_rate, cutpoint_methods = get_inbuilt_cutpoint_methods(), fx_nmb_training, fx_nmb_evaluation, meet_min_events = TRUE, min_events = NA, show_progress = FALSE, cl = NULL )
sample_size 
Sample size of training set. If missing, a sample size calculation will be performed and the calculated size will be used. 
n_sims 
Number of simulations to run. 
n_valid 
Sample size for evaluation set. 
sim_auc 
Simulated model discrimination (AUC). 
event_rate 
Simulated event rate of the binary outcome being predicted. Also known as prevalence. 
cutpoint_methods 
A value or vector of cutpoint methods to include. Defaults to use the inbuilt methods:
Userdefined cutpoint methods can be used by passing the name of a function that takes the following arguments:
See 
fx_nmb_training 
Function or 
fx_nmb_evaluation 
Function or 
meet_min_events 
Whether or not to incrementally add samples until the
expected number of events ( 
min_events 
The minimum number of events to include in the training
sample. If less than this number are included in sample of size

show_progress 
Logical. Whether to display a progress bar. Requires the

cl 
A cluster made using 
This function runs a simulation for a given set of inputs that represent a
healthcare setting using modelguided interventions.
The arguments fx_nmb_training
and fx_nmb_evaluation
should be functions that capture the treatment being used, its costs and
effectiveness, and the costs of the outcome being treated/prevented.
Both of these are functions that return a named vector of NMB values when
called and are used for obtaining and evaluating cutpoints, respectively.
For example, the following function returns the appropriately named vector.
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4)
There is a helper function, get_nmb_sampler()
, to help you
create these.
Returns a predictNMBsim
object.
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb )
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb )
Evaluates a cutpoint by returning the mean treatment cost per sample.
evaluate_cutpoint_cost(predicted, actual, pt, nmb)
evaluate_cutpoint_cost(predicted, actual, pt, nmb)
predicted 
A vector of predicted probabilities. 
actual 
A vector of actual outcomes. 
pt 
The probability threshold to be evaluated. 
nmb 
A named vector containing NMB assigned to each classification and the treatment costs. 
Returns a numeric
value representing the mean cost for that
cutpoint and data.
evaluate_cutpoint_cost( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), pt = 0.1, nmb = c( "qalys_lost" = 5, "low_risk_group_treatment_cost" = 0, "high_risk_group_treatment_cost" = 1, "low_risk_group_treatment_effect" = 0, "high_risk_group_treatment_effect" = 0.3, "outcome_cost" = 10 ) )
evaluate_cutpoint_cost( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), pt = 0.1, nmb = c( "qalys_lost" = 5, "low_risk_group_treatment_cost" = 0, "high_risk_group_treatment_cost" = 1, "low_risk_group_treatment_effect" = 0, "high_risk_group_treatment_effect" = 0.3, "outcome_cost" = 10 ) )
Evaluates a cutpoint by returning the mean NMB per sample.
evaluate_cutpoint_nmb(predicted, actual, pt, nmb)
evaluate_cutpoint_nmb(predicted, actual, pt, nmb)
predicted 
A vector of predicted probabilities. 
actual 
A vector of actual outcomes. 
pt 
The probability threshold to be evaluated. 
nmb 
A named vector containing NMB assigned to each classification. 
Returns a numeric
value representing the NMB for that
cutpoint and data.
evaluate_cutpoint_nmb( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), pt = 0.1, nmb = c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) )
evaluate_cutpoint_nmb( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), pt = 0.1, nmb = c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) )
Evaluates a cutpoint by returning the mean QALYs lost per sample.
evaluate_cutpoint_qalys(predicted, actual, pt, nmb)
evaluate_cutpoint_qalys(predicted, actual, pt, nmb)
predicted 
A vector of predicted probabilities. 
actual 
A vector of actual outcomes. 
pt 
The probability threshold to be evaluated. 
nmb 
A named vector containing NMB assigned to each classification and the treatment effects and QALYS lost due to the event of interest. 
Returns a numeric
value representing the mean QALYs for that
cutpoint and data.
evaluate_cutpoint_qalys( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), pt = 0.1, nmb = c( "qalys_lost" = 5, "low_risk_group_treatment_effect" = 0, "high_risk_group_treatment_effect" = 0.5 ) )
evaluate_cutpoint_qalys( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), pt = 0.1, nmb = c( "qalys_lost" = 5, "low_risk_group_treatment_effect" = 0, "high_risk_group_treatment_effect" = 0.5 ) )
Get a cutpoint using the methods inbuilt to predictNMB
get_inbuilt_cutpoint(predicted, actual, nmb, method)
get_inbuilt_cutpoint(predicted, actual, nmb, method)
predicted 
A vector of predicted probabilities 
actual 
A vector of actual outcomes 
nmb 
A named vector containing NMB assigned to each classification 
method 
A cutpoint selection method to be used methods that can be used as the method argument 
Returns a selected cutpoint (numeric).
## get the list of available methods: get_inbuilt_cutpoint_methods() ## get the cutpoint that maximises the Youden index for a given set of ## probabilities and outcomes get_inbuilt_cutpoint( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), method = "youden" )
## get the list of available methods: get_inbuilt_cutpoint_methods() ## get the cutpoint that maximises the Youden index for a given set of ## probabilities and outcomes get_inbuilt_cutpoint( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), method = "youden" )
Get a vector of all the inbuilt cutpoint methods
get_inbuilt_cutpoint_methods()
get_inbuilt_cutpoint_methods()
Returns a vector cutpoint methods that can be used in
do_nmb_sim()
.
get_inbuilt_cutpoint_methods()
get_inbuilt_cutpoint_methods()
do_nmb_sim()
or
screen_simulation_inputs()
Make a NMB sampler for use in do_nmb_sim()
or
screen_simulation_inputs()
get_nmb_sampler( outcome_cost, wtp, qalys_lost, high_risk_group_treatment_effect, high_risk_group_treatment_cost, low_risk_group_treatment_effect = 0, low_risk_group_treatment_cost = 0, use_expected_values = FALSE, nboot = 10000 )
get_nmb_sampler( outcome_cost, wtp, qalys_lost, high_risk_group_treatment_effect, high_risk_group_treatment_cost, low_risk_group_treatment_effect = 0, low_risk_group_treatment_cost = 0, use_expected_values = FALSE, nboot = 10000 )
outcome_cost 
The cost of the outcome. Must be provided if 
wtp 
Willingnesstopay. 
qalys_lost 
Qualityadjusted life years (QALYs) lost due to healthcare event being predicted. 
high_risk_group_treatment_effect 
The effect of the treatment provided to patients given high risk prediction. Can be a number of a function. Provide a function to incorporate uncertainty. 
high_risk_group_treatment_cost 
The cost of the treatment provided to patients given high risk prediction. Can be a number of a function. Provide a function to incorporate uncertainty. 
low_risk_group_treatment_effect 
The effect of the treatment provided to patients given low risk prediction. Can be a number of a function. Provide a function to incorporate uncertainty. Defaults to 0 (no treatment). 
low_risk_group_treatment_cost 
The cost of the treatment provided to patients given low risk prediction. Can be a number of a function. Provide a function to incorporate uncertainty. Defaults to 0 (no treatment). 
use_expected_values 
Logical. If 
nboot 
The number of samples to use when creating a function that returns the expected values. Defaults to 10000. 
Returns a NMBsampler
object.
get_nmb_training < get_nmb_sampler( outcome_cost = 100, high_risk_group_treatment_effect = function() rbeta(1, 1, 2), high_risk_group_treatment_cost = 10, use_expected_values = TRUE ) get_nmb_evaluation < get_nmb_sampler( outcome_cost = 100, high_risk_group_treatment_effect = function() rbeta(1, 1, 2), high_risk_group_treatment_cost = 10 ) get_nmb_training() get_nmb_training() get_nmb_training() get_nmb_evaluation() get_nmb_evaluation() get_nmb_evaluation()
get_nmb_training < get_nmb_sampler( outcome_cost = 100, high_risk_group_treatment_effect = function() rbeta(1, 1, 2), high_risk_group_treatment_cost = 10, use_expected_values = TRUE ) get_nmb_evaluation < get_nmb_sampler( outcome_cost = 100, high_risk_group_treatment_effect = function() rbeta(1, 1, 2), high_risk_group_treatment_cost = 10 ) get_nmb_training() get_nmb_training() get_nmb_training() get_nmb_evaluation() get_nmb_evaluation() get_nmb_evaluation()
Samples data for a prediction model with a specified AUC and prevalence.
get_sample(auc, n_samples, prevalence, min_events = 0)
get_sample(auc, n_samples, prevalence, min_events = 0)
auc 
The Area Under the (receiver operating characteristic) Curve. 
n_samples 
Number of samples to draw. 
prevalence 
Prevalence or event rate of the binary outcome as a proportion (0.1 = 10%). 
min_events 
Minimum number of events required in the sample. 
Returns a data.frame
.
get_sample(0.7, 1000, 0.1)
get_sample(0.7, 1000, 0.1)
Gets probability thresholds given predicted probabilities, outcomes and NMB.
get_thresholds(predicted, actual, nmb, cutpoint_methods = NULL)
get_thresholds(predicted, actual, nmb, cutpoint_methods = NULL)
predicted 
A vector of predicted probabilities. 
actual 
A vector of actual outcomes. 
nmb 
A named vector containing NMB assigned to true positives, true negatives, false positives and false negatives 
cutpoint_methods 
Which cutpoint method(s) to return. The default (NULL) uses all the inbuilt methods. 
Returns a list
.
# get thresholds using default (all inbuilt) cutpoint methods get_thresholds( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), nmb = c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) ) # get cutpoints using userdefined functions # These functions must take the \code{predicted} and \code{actual} # as arguments. They can also take \code{nmb} (named vector containing NMB # with values for TP, FP, TN, FN). fx_roc01 < function(predicted, actual, ...) { cutpointr::cutpointr( x = predicted, class = actual, method = cutpointr::minimize_metric, metric = cutpointr::roc01, silent = TRUE )[["optimal_cutpoint"]] } fx_sum_sens_spec < function(predicted, actual, ...) { cutpointr::cutpointr( x = predicted, class = actual, method = cutpointr::maximize_metric, metric = cutpointr::sum_sens_spec, silent = TRUE )[["optimal_cutpoint"]] } get_thresholds( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), cutpoint_methods = c("fx_roc01", "fx_sum_sens_spec"), nmb = c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) ) # get a combination of cutpoints from both userdefined functions and # inbuilt methods get_thresholds( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), cutpoint_methods = c( "fx_roc01", "fx_sum_sens_spec", "youden", "all", "none" ), nmb = c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) )
# get thresholds using default (all inbuilt) cutpoint methods get_thresholds( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), nmb = c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) ) # get cutpoints using userdefined functions # These functions must take the \code{predicted} and \code{actual} # as arguments. They can also take \code{nmb} (named vector containing NMB # with values for TP, FP, TN, FN). fx_roc01 < function(predicted, actual, ...) { cutpointr::cutpointr( x = predicted, class = actual, method = cutpointr::minimize_metric, metric = cutpointr::roc01, silent = TRUE )[["optimal_cutpoint"]] } fx_sum_sens_spec < function(predicted, actual, ...) { cutpointr::cutpointr( x = predicted, class = actual, method = cutpointr::maximize_metric, metric = cutpointr::sum_sens_spec, silent = TRUE )[["optimal_cutpoint"]] } get_thresholds( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), cutpoint_methods = c("fx_roc01", "fx_sum_sens_spec"), nmb = c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) ) # get a combination of cutpoints from both userdefined functions and # inbuilt methods get_thresholds( predicted = runif(1000), actual = sample(c(0, 1), size = 1000, replace = TRUE), cutpoint_methods = c( "fx_roc01", "fx_sum_sens_spec", "youden", "all", "none" ), nmb = c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) )
Print a summary of a predictNMBscreen object
## S3 method for class 'predictNMBscreen' print(x, ...)
## S3 method for class 'predictNMBscreen' print(x, ...)
x 
A 
... 
Optional, ignored arguments. 
print(x)
returns x
invisibly.
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_screen_obj < screen_simulation_inputs( n_sims = 50, n_valid = 10000, sim_auc = seq(0.7, 0.9, 0.1), event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb ) print(sim_screen_obj)
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_screen_obj < screen_simulation_inputs( n_sims = 50, n_valid = 10000, sim_auc = seq(0.7, 0.9, 0.1), event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb ) print(sim_screen_obj)
Print a summary of a predictNMBsim object
## S3 method for class 'predictNMBsim' print(x, ...)
## S3 method for class 'predictNMBsim' print(x, ...)
x 
A 
... 
Optional, ignored arguments. 
print(x)
returns x
invisibly.
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb ) print(sim_obj)
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb ) print(sim_obj)
do_nmb_sim()
Runs do_nmb_sim()
with a range of inputs.
screen_simulation_inputs( sample_size, n_sims, n_valid, sim_auc, event_rate, cutpoint_methods = get_inbuilt_cutpoint_methods(), fx_nmb_training, fx_nmb_evaluation, pair_nmb_train_and_evaluation_functions = FALSE, meet_min_events = TRUE, min_events = NA, show_progress = FALSE, cl = NULL )
screen_simulation_inputs( sample_size, n_sims, n_valid, sim_auc, event_rate, cutpoint_methods = get_inbuilt_cutpoint_methods(), fx_nmb_training, fx_nmb_evaluation, pair_nmb_train_and_evaluation_functions = FALSE, meet_min_events = TRUE, min_events = NA, show_progress = FALSE, cl = NULL )
sample_size 
A value (or vector of values): Sample size of training set. If missing, a sample size calculation will be performed and the calculated size will be used. 
n_sims 
A value (or vector of values): Number of simulations to run. 
n_valid 
A value (or vector of values): Sample size for evaluation set. 
sim_auc 
A value (or vector of values): Simulated model discrimination (AUC). 
event_rate 
A value (or vector of values): simulated event rate of the binary outcome being predicted. 
cutpoint_methods 
cutpoint methods to include. Defaults to use the
inbuilt methods. This doesn't change across calls to 
fx_nmb_training 
A function or 
fx_nmb_evaluation 
A function or 
pair_nmb_train_and_evaluation_functions 

meet_min_events 
Whether or not to incrementally add samples until the
expected number of events ( 
min_events 
A value: the minimum number of events to include in the
training sample. If less than this number are included in sample of size

show_progress 
Logical. Whether to display a progress bar. 
cl 
A cluster made using 
Returns a predictNMBscreen
object.
# Screen for optimal cutpoints given increasing values of # model discrimination (sim_auc) get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_screen_obj < screen_simulation_inputs( n_sims = 50, n_valid = 10000, sim_auc = seq(0.7, 0.9, 0.1), event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb )
# Screen for optimal cutpoints given increasing values of # model discrimination (sim_auc) get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_screen_obj < screen_simulation_inputs( n_sims = 50, n_valid = 10000, sim_auc = seq(0.7, 0.9, 0.1), event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb )
predictNMBscreen
objects.Create table summaries of predictNMBscreen
objects.
## S3 method for class 'predictNMBscreen' summary( object, what = c("nmb", "inb", "cutpoints"), inb_ref_col = NULL, agg_functions = list(median = function(x) { round(stats::median(x), digits = 2) }, `95% CI` = function(x) { paste0(round(stats::quantile(x, probs = c(0.025, 0.975)), digits = 1), collapse = " to ") }), rename_vector, show_full_inputs = FALSE, ... )
## S3 method for class 'predictNMBscreen' summary( object, what = c("nmb", "inb", "cutpoints"), inb_ref_col = NULL, agg_functions = list(median = function(x) { round(stats::median(x), digits = 2) }, `95% CI` = function(x) { paste0(round(stats::quantile(x, probs = c(0.025, 0.975)), digits = 1), collapse = " to ") }), rename_vector, show_full_inputs = FALSE, ... )
object 
A 
what 
What to summarise: one of "nmb", "inb" or "cutpoints". Defaults to "nmb". 
inb_ref_col 
Which cutpoint method to use as the reference strategy
when calculating the incremental net monetary benefit. See 
agg_functions 
A named list of functions to use to aggregate the selected values. Defaults to the median and 95% interval. 
rename_vector 
A named vector for renaming the methods in the summary. The values of the vector are the default names and the names given are the desired names in the output. 
show_full_inputs 
A logical. Whether or not to include the inputs used for simulation alongside aggregations. 
... 
Additional, ignored arguments. 
Table summaries will be based on the what
argument.
Using "nmb" returns the simulated values for NMB, with no reference group;
"inb" returns the difference between simulated values for NMB and a set
strategy defined by inb_ref_col
; "cutpoints" returns the cutpoints
selected (0, 1).
Returns a tibble
.
# perform screen with increasing values of model discimination (sim_auc) get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_screen_obj < screen_simulation_inputs( n_sims = 50, n_valid = 10000, sim_auc = seq(0.7, 0.9, 0.1), event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb, cutpoint_methods = c("all", "none", "youden", "value_optimising") ) summary( sim_screen_obj, rename_vector = c( "Value_Optimising" = "value_optimising", "Treat_None" = "none", "Treat_All" = "all", "Youden_Index" = "youden" ) )
# perform screen with increasing values of model discimination (sim_auc) get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_screen_obj < screen_simulation_inputs( n_sims = 50, n_valid = 10000, sim_auc = seq(0.7, 0.9, 0.1), event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb, cutpoint_methods = c("all", "none", "youden", "value_optimising") ) summary( sim_screen_obj, rename_vector = c( "Value_Optimising" = "value_optimising", "Treat_None" = "none", "Treat_All" = "all", "Youden_Index" = "youden" ) )
predictNMBsim
objects.Create table summaries of predictNMBsim
objects.
## S3 method for class 'predictNMBsim' summary( object, what = c("nmb", "inb", "cutpoints"), inb_ref_col = NULL, agg_functions = list(median = function(x) { round(stats::median(x), digits = 2) }, `95% CI` = function(x) { paste0(round(stats::quantile(x, probs = c(0.025, 0.975)), digits = 1), collapse = " to ") }), rename_vector, ... )
## S3 method for class 'predictNMBsim' summary( object, what = c("nmb", "inb", "cutpoints"), inb_ref_col = NULL, agg_functions = list(median = function(x) { round(stats::median(x), digits = 2) }, `95% CI` = function(x) { paste0(round(stats::quantile(x, probs = c(0.025, 0.975)), digits = 1), collapse = " to ") }), rename_vector, ... )
object 
A 
what 
What to summarise: one of "nmb", "inb" or "cutpoints". Defaults to "nmb". 
inb_ref_col 
Which cutpoint method to use as the reference strategy
when calculating the incremental net monetary benefit. See 
agg_functions 
A named list of functions to use to aggregate the selected values. Defaults to the median and 95% interval. 
rename_vector 
A named vector for renaming the methods in the summary. The values of the vector are the default names and the names given are the desired names in the output. 
... 
Additional, ignored arguments. 
Table summaries will be based on the what
argument.
Using "nmb" returns the simulated values for NMB, with no reference group;
"inb" returns the difference between simulated values for NMB and a set
strategy defined by inb_ref_col
; "cutpoints" returns the cutpoints
selected (0, 1).
Returns a tibble
.
# perform simulation with do_nmb_sim() get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb, cutpoint_methods = c("all", "none", "youden", "value_optimising") ) summary( sim_obj, rename_vector = c( "Value_Optimising" = "value_optimising", "Treat_None" = "none", "Treat_All" = "all", "Youden_Index" = "youden" ) )
# perform simulation with do_nmb_sim() get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb, cutpoint_methods = c("all", "none", "youden", "value_optimising") ) summary( sim_obj, rename_vector = c( "Value_Optimising" = "value_optimising", "Treat_None" = "none", "Treat_All" = "all", "Youden_Index" = "youden" ) )
ggplot2
theme that reduces clutter in an autoplot()
of a predictNMBsim
object.Returns a ggplot2
theme that reduces clutter in an autoplot()
of a predictNMBsim
object.
theme_sim()
theme_sim()
Returns a ggplot2
theme.
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb ) autoplot(sim_obj) + theme_sim()
get_nmb < function() c("TP" = 3, "TN" = 0, "FP" = 1, "FN" = 4) sim_obj < do_nmb_sim( sample_size = 200, n_sims = 50, n_valid = 10000, sim_auc = 0.7, event_rate = 0.1, fx_nmb_training = get_nmb, fx_nmb_evaluation = get_nmb ) autoplot(sim_obj) + theme_sim()