experiments.experiments.SeldonianExperiment

class SeldonianExperiment(model_name, results_dir)

Bases: Experiment

__init__(model_name, results_dir)

Class for running Seldonian experiments

Parameters:

  • model_name (str) – The string name of the Seldonian model, only option is currently: ‘qsa’ (quasi-Seldonian algorithm)

  • results_dir (str) – Parent directory for saving any experimental results

__repr__()

Return repr(self).

Methods

aggregate_results(**kwargs)

Group together the data in each trial file into a single CSV file.

evaluate_constraint_functions(solution, constraint_eval_fns, constraint_eval_kwargs)

Helper function for run_QSA_trial() to evaluate the constraint functions to determine whether the solution from the QSA was safe on a ground truth held out dataset.

Parameters:

  • solution (numpy ndarray) – The weights of the model found during candidate selection in a given trial

  • constraint_eval_fns (List(function)) – List of functions to use to evaluate each constraint. An empty list results in using the parse tree to evaluate the constraints

  • constraint_eval_kwargs (dict) – keyword arguments to pass to each constraint function in constraint_eval_fns

Returns:

a vector of g values (expected values) for the constraints

Return type:

np.ndarray

run_QSA_trial(data_frac, trial_i, **kwargs)

Run a trial of the quasi-Seldonian algorithm (QSA)

Parameters:

  • data_frac (float) – Fraction of overall dataset size to use

  • trial_i (int) – The index of the trial

run_experiment(**kwargs)

Run the Seldonian experiment

run_trials_par(args_list, shared_namespace)

Wrapper function that is run as a parallel process. Runs all the trials provided in args_list on a single core .

Parameters:

  • args_list – list of (data_frac,trial_i) pairs

  • shared_namespace – The namespace shared across multiple cores containing the keyword arguments that are the same for all trials.

write_trial_result(data, colnames, trial_dir, verbose=False)

Write out the results from a single trial to a file.

Parameters:

  • data (List) – The information to save

  • colnames (List(str)) – Names of the items in the data list. These will comprise the header of the saved file

  • trial_dir (str) – The directory in which to save the file

  • verbose (bool) – if True, prints out saved filename