experiments.experiments.FairlearnExperiment

class FairlearnExperiment(results_dir, fairlearn_epsilon_constraint)

Bases: Experiment

__init__(results_dir, fairlearn_epsilon_constraint)

Class for running Fairlearn experiments

Parameters:

  • results_dir (str) – Parent directory for saving any experimental results

  • fairlearn_epsilon_constraint (float) – The value of epsilon (the threshold) to use in the constraint to the Fairlearn model

__repr__()

Return repr(self).

Methods

aggregate_results(**kwargs)

Group together the data in each trial file into a single CSV file.

evaluate_constraint_function(y_pred, test_labels, fairlearn_constraint_name, epsilon_eval, eval_method='native', sensitive_features=[])

Evaluate the constraint function using the Fairlearn predictions

Parameters:

  • y_pred (1D array) – Predicted class labels, same shape as test_labels

  • test_labels (1D array) – True class labels

  • fairlearn_constraint_name (str) – The name of the constraint

  • epsilon_eval (float) – The threshold in the constraint to use for evaluation

  • eval_method (str, defaults to 'native') – The method for evaluating the constraint, two options: ‘native’ or ‘two-groups’

  • sensitive_features (List(str)) – List of column names that are considered sensitive in the Fairlearn dataset

get_fairlearn_predictions(mitigator, X_test_fairlearn)

Get the predicted labels from the fairlearn mitigator. The mitigator consists of potentially more than one predictor. For each predictor with non-zero weight, we figure out how many points to predict based on the weight of that predictor. Weights are normalized to 1 across all predictors.

Parameters:

  • mitigator – The Fairlearn mitigator

  • X_test_fairlearn – The test features from which to predict the labels

Return y_pred:

Array of predicted class labels

run_experiment(**kwargs)

Run the Fairlearn experiment

run_fairlearn_trial(data_frac, trial_i, **kwargs)

Run a Fairlearn trial

Parameters:

  • data_frac (float) – Fraction of overall dataset size to use

  • trial_i (int) – The index of the trial

write_trial_result(data, colnames, trial_dir, verbose=False)

Write out the results from a single trial to a file.

Parameters:

  • data (List) – The information to save

  • colnames (List(str)) – Names of the items in the data list. These will comprise the header of the saved file

  • trial_dir (str) – The directory in which to save the file

  • verbose (bool) – if True, prints out saved filename