experiments.experiments.BaselineExperiment

class BaselineExperiment(baseline_model, results_dir)

Bases: Experiment

__init__(baseline_model, results_dir)

Class for running baseline experiments to compare against Seldonian Experiments

Parameters:

  • model_name (str) – The string name of the baseline model, e.g ‘logistic_regression’

  • results_dir (str) – Parent directory for saving any experimental results

__repr__()

Return repr(self).

Methods

aggregate_results(**kwargs)

Group together the data in each trial file into a single CSV file.

evaluate_constraint_functions(solution, constraint_eval_fns, constraint_eval_kwargs)

Helper function to evaluate the constraint functions to determine whether the baseline solution was safe on ground truth

Parameters:

  • solution (numpy ndarray) – The weights of the model found during model training in a given trial

  • constraint_eval_fns (List(function)) – List of functions to use to evaluate each constraint. An empty list (default) results in using the parse trees to evaluate the constraints

  • constraint_eval_kwargs (dict) – keyword arguments to pass to each constraint function in constraint_eval_fns

Returns:

a vector of g values (expected values) for the constraints

Return type:

np.ndarray

run_baseline_trial(data_frac, trial_i, **kwargs)

Run a trial of the baseline model. Currently only supports supervised learning experiments.

Parameters:

  • data_frac (float) – Fraction of overall dataset size to use

  • trial_i (int) – The index of the trial

run_experiment(**kwargs)

Run the baseline experiment

write_trial_result(data, colnames, trial_dir, verbose=False)

Write out the results from a single trial to a file.

Parameters:

  • data (List) – The information to save

  • colnames (List(str)) – Names of the items in the data list. These will comprise the header of the saved file

  • trial_dir (str) – The directory in which to save the file

  • verbose (bool) – if True, prints out saved filename