experiments.base_example.BaseExample

class BaseExample(spec)

Bases: object

__init__(spec)

Base class for running experiments

Parameters:

spec – specification object created using the Seldonian Engine.

__repr__()

Return repr(self).

Methods

run(n_trials, data_fracs, results_dir, perf_eval_fn, n_workers=1, datagen_method='resample', baselines=[], model_label_dict={}, include_fairlearn_models=False, fairlearn_kwargs={}, performance_label='performance', performance_yscale='linear', plot_savename=None, plot_save_format='pdf', include_legend=True, plot_fontsize=12, legend_fontsize=8, verbose=False)

Run the experiments for this example. Runs any baseline models included in baselines parameter first. Then produces the three plots.

Parameters:

• n_trials – The number of trials for the experiments

• data_fracs – The data fractions for the experiments

• results_dir – Directory for saving results files

• perf_eval_fn – Performance evaluation function

• n_workers – Number of parallel processors to use

• datagen_method – Method for generating the trial data

• baselines – List of baseline models to include

• model_label_dict – Dictionary mapping model names (see model.model_name) to display name in the 3 plots legend.

• include_fairlearn_models (Bool) – Whether to include fairlearn baseline models

• performance_label (str) – Label to use on the performance plot (left-most plot)

• performance_yscale (str) – How to scale the y-axis on the performance plot. Options are “linear” and “log”

• plot_savename – If provided, the filepath where the three plots will be saved

• plot_save_format – “pdf” or “png”

• include_legend (bool) – Whether to include legend on the 3 plots