Seldonian Experiments documentation

Welcome to the documentation for the Seldonian Experiments library. This library is designed to help you evaluate the performance and safety of Seldonian algorithms. The source code is hosted on GitHub.

This library makes heavy use of the Seldonian Engine library, the core library for running Seldonian algorithms.

If you are completely new to this library, see the Overview.

Table of contents

• Overview
  • Seldonian experiments
  • Three Plots
  • Plot generator
    • Spec object
    • n_trials
    • data_fracs
    • datagen_method
    • n_workers
    • perf_eval_fn
    • perf_eval_kwargs
    • constraint_eval_fns
    • constraint_eval_kwargs
    • results_dir
  • Baselines
  • Files generated in an experiment
• API
  • experiments
    • experiments.base_example
      • experiments.base_example.BaseExample
        • BaseExample
          • BaseExample.__init__()
          • BaseExample.__repr__()
          • BaseExample.run()
    • experiments.baselines
      • experiments.baselines.baselines
        • experiments.baselines.baselines.RLExperimentBaseline
          • RLExperimentBaseline
            • RLExperimentBaseline.__init__()
            • RLExperimentBaseline.__repr__()
            • RLExperimentBaseline.set_new_params()
            • RLExperimentBaseline.train()
        • experiments.baselines.baselines.SupervisedExperimentBaseline
          • SupervisedExperimentBaseline
            • SupervisedExperimentBaseline.__init__()
            • SupervisedExperimentBaseline.__repr__()
            • SupervisedExperimentBaseline.predict()
            • SupervisedExperimentBaseline.train()
      • experiments.baselines.decision_tree
        • experiments.baselines.decision_tree.DecisionTreeClassifierBaseline
          • DecisionTreeClassifierBaseline
            • DecisionTreeClassifierBaseline.__init__()
            • DecisionTreeClassifierBaseline.__repr__()
            • DecisionTreeClassifierBaseline.predict()
            • DecisionTreeClassifierBaseline.train()
      • experiments.baselines.decision_tree_leaf_tuning
        • experiments.baselines.decision_tree_leaf_tuning.DecisionTreeClassifierLeafTuningBaseline
          • DecisionTreeClassifierLeafTuningBaseline
            • DecisionTreeClassifierLeafTuningBaseline.__init__()
            • DecisionTreeClassifierLeafTuningBaseline.__repr__()
            • DecisionTreeClassifierLeafTuningBaseline.adam()
            • DecisionTreeClassifierLeafTuningBaseline.fit()
            • DecisionTreeClassifierLeafTuningBaseline.forward_pass()
            • DecisionTreeClassifierLeafTuningBaseline.get_jacobian()
            • DecisionTreeClassifierLeafTuningBaseline.get_leaf_node_probs()
            • DecisionTreeClassifierLeafTuningBaseline.predict()
            • DecisionTreeClassifierLeafTuningBaseline.set_leaf_node_values()
            • DecisionTreeClassifierLeafTuningBaseline.train()
            • DecisionTreeClassifierLeafTuningBaseline.wrapped_primary_objective()
      • experiments.baselines.diabetes_US_baseline
        • experiments.baselines.diabetes_US_baseline.RLDiabetesUSAgentBaseline
          • RLDiabetesUSAgentBaseline
            • RLDiabetesUSAgentBaseline.__init__()
            • RLDiabetesUSAgentBaseline.__repr__()
            • RLDiabetesUSAgentBaseline.primary_objective_fn()
            • RLDiabetesUSAgentBaseline.set_new_params()
            • RLDiabetesUSAgentBaseline.train()
      • experiments.baselines.fitted_Q
        • experiments.baselines.fitted_Q.ApproximateTabularFittedQBaseline
          • ApproximateTabularFittedQBaseline
            • ApproximateTabularFittedQBaseline.__init__()
            • ApproximateTabularFittedQBaseline.__repr__()
            • ApproximateTabularFittedQBaseline.get_max_q()
            • ApproximateTabularFittedQBaseline.get_next_obs()
            • ApproximateTabularFittedQBaseline.get_probs_from_observations_and_actions()
            • ApproximateTabularFittedQBaseline.get_regressor_weights()
            • ApproximateTabularFittedQBaseline.get_target()
            • ApproximateTabularFittedQBaseline.instantiate_regressor()
            • ApproximateTabularFittedQBaseline.make_X()
            • ApproximateTabularFittedQBaseline.make_regression_dataset()
            • ApproximateTabularFittedQBaseline.make_y()
            • ApproximateTabularFittedQBaseline.one_hot_encode()
            • ApproximateTabularFittedQBaseline.set_new_params()
            • ApproximateTabularFittedQBaseline.set_q_table()
            • ApproximateTabularFittedQBaseline.stopping_criteria_met()
            • ApproximateTabularFittedQBaseline.update_Q_weights()
        • experiments.baselines.fitted_Q.BaseFittedQBaseline
          • BaseFittedQBaseline
            • BaseFittedQBaseline.__init__()
            • BaseFittedQBaseline.__repr__()
            • BaseFittedQBaseline.get_max_q()
            • BaseFittedQBaseline.get_next_obs()
            • BaseFittedQBaseline.get_probs_from_observations_and_actions()
            • BaseFittedQBaseline.get_target()
            • BaseFittedQBaseline.instantiate_regressor()
            • BaseFittedQBaseline.make_X()
            • BaseFittedQBaseline.make_regression_dataset()
            • BaseFittedQBaseline.make_y()
            • BaseFittedQBaseline.set_new_params()
            • BaseFittedQBaseline.stopping_criteria_met()
            • BaseFittedQBaseline.update_Q_weights()
        • experiments.baselines.fitted_Q.ExactTabularFittedQBaseline
          • ExactTabularFittedQBaseline
            • ExactTabularFittedQBaseline.__init__()
            • ExactTabularFittedQBaseline.__repr__()
            • ExactTabularFittedQBaseline.get_max_q()
            • ExactTabularFittedQBaseline.get_next_obs()
            • ExactTabularFittedQBaseline.get_probs_from_observations_and_actions()
            • ExactTabularFittedQBaseline.get_regressor_weights()
            • ExactTabularFittedQBaseline.get_target()
            • ExactTabularFittedQBaseline.instantiate_regressor()
            • ExactTabularFittedQBaseline.make_X()
            • ExactTabularFittedQBaseline.make_regression_dataset()
            • ExactTabularFittedQBaseline.make_y()
            • ExactTabularFittedQBaseline.one_hot_encode()
            • ExactTabularFittedQBaseline.set_new_params()
            • ExactTabularFittedQBaseline.set_q_table()
            • ExactTabularFittedQBaseline.stopping_criteria_met()
            • ExactTabularFittedQBaseline.update_Q_weights()
      • experiments.baselines.linear_regression
        • experiments.baselines.linear_regression.LinearRegressionBaseline
          • LinearRegressionBaseline
            • LinearRegressionBaseline.__init__()
            • LinearRegressionBaseline.__repr__()
            • LinearRegressionBaseline.fit()
            • LinearRegressionBaseline.predict()
            • LinearRegressionBaseline.train()
      • experiments.baselines.logistic_regression
        • experiments.baselines.logistic_regression.BinaryLogisticRegressionBaseline
          • BinaryLogisticRegressionBaseline
            • BinaryLogisticRegressionBaseline.__init__()
            • BinaryLogisticRegressionBaseline.__repr__()
            • BinaryLogisticRegressionBaseline.fit()
            • BinaryLogisticRegressionBaseline.predict()
            • BinaryLogisticRegressionBaseline.train()
      • experiments.baselines.random_classifiers
        • experiments.baselines.random_classifiers.UniformRandomClassifierBaseline
          • UniformRandomClassifierBaseline
            • UniformRandomClassifierBaseline.__init__()
            • UniformRandomClassifierBaseline.__repr__()
            • UniformRandomClassifierBaseline.predict()
            • UniformRandomClassifierBaseline.train()
        • experiments.baselines.random_classifiers.WeightedRandomClassifierBaseline
          • WeightedRandomClassifierBaseline
            • WeightedRandomClassifierBaseline.__init__()
            • WeightedRandomClassifierBaseline.__repr__()
            • WeightedRandomClassifierBaseline.predict()
            • WeightedRandomClassifierBaseline.train()
      • experiments.baselines.random_forest
        • experiments.baselines.random_forest.RandomForestClassifierBaseline
          • RandomForestClassifierBaseline
            • RandomForestClassifierBaseline.__init__()
            • RandomForestClassifierBaseline.__repr__()
            • RandomForestClassifierBaseline.predict()
            • RandomForestClassifierBaseline.train()
    • experiments.experiment_utils
      • experiments.experiment_utils.batch_predictions
        • batch_predictions()
      • experiments.experiment_utils.batch_predictions_custom_regime
        • batch_predictions_custom_regime()
      • experiments.experiment_utils.generate_behavior_policy_episodes
        • generate_behavior_policy_episodes()
      • experiments.experiment_utils.generate_episodes_and_calc_J
        • generate_episodes_and_calc_J()
      • experiments.experiment_utils.has_failed
        • has_failed()
      • experiments.experiment_utils.load_regenerated_episodes
        • load_regenerated_episodes()
      • experiments.experiment_utils.load_resampled_datasets
        • load_resampled_datasets()
      • experiments.experiment_utils.make_batch_epoch_dict_fixedniter
        • make_batch_epoch_dict_fixedniter()
      • experiments.experiment_utils.make_batch_epoch_dict_min_sample_repeat
        • make_batch_epoch_dict_min_sample_repeat()
      • experiments.experiment_utils.prep_custom_data
        • prep_custom_data()
      • experiments.experiment_utils.prep_data_for_fairlearn
        • prep_data_for_fairlearn()
      • experiments.experiment_utils.prep_feat_labels
        • prep_feat_labels()
      • experiments.experiment_utils.prep_feat_labels_for_baseline
        • prep_feat_labels_for_baseline()
      • experiments.experiment_utils.setup_SA_spec_for_exp
        • setup_SA_spec_for_exp()
      • experiments.experiment_utils.supervised_initial_solution_fn
        • supervised_initial_solution_fn()
      • experiments.experiment_utils.trial_arg_chunker
        • trial_arg_chunker()
    • experiments.experiments
      • experiments.experiments.BaselineExperiment
        • BaselineExperiment
          • BaselineExperiment.__init__()
          • BaselineExperiment.__repr__()
          • BaselineExperiment.aggregate_results()
          • BaselineExperiment.evaluate_constraint_functions()
          • BaselineExperiment.run_baseline_trial()
          • BaselineExperiment.run_experiment()
          • BaselineExperiment.write_trial_result()
      • experiments.experiments.Experiment
        • Experiment
          • Experiment.__init__()
          • Experiment.__repr__()
          • Experiment.aggregate_results()
          • Experiment.write_trial_result()
      • experiments.experiments.FairlearnExperiment
        • FairlearnExperiment
          • FairlearnExperiment.__init__()
          • FairlearnExperiment.__repr__()
          • FairlearnExperiment.aggregate_results()
          • FairlearnExperiment.evaluate_constraint_function()
          • FairlearnExperiment.get_fairlearn_predictions()
          • FairlearnExperiment.run_experiment()
          • FairlearnExperiment.run_fairlearn_trial()
          • FairlearnExperiment.write_trial_result()
      • experiments.experiments.SeldonianExperiment
        • SeldonianExperiment
          • SeldonianExperiment.__init__()
          • SeldonianExperiment.__repr__()
          • SeldonianExperiment.aggregate_results()
          • SeldonianExperiment.evaluate_constraint_functions()
          • SeldonianExperiment.run_QSA_trial()
          • SeldonianExperiment.run_experiment()
          • SeldonianExperiment.run_trials_par()
          • SeldonianExperiment.write_trial_result()
    • experiments.generate_plots
      • experiments.generate_plots.CustomPlotGenerator
        • CustomPlotGenerator
          • CustomPlotGenerator.__init__()
          • CustomPlotGenerator.__repr__()
          • CustomPlotGenerator.generate_resampled_datasets()
          • CustomPlotGenerator.generate_trial_datasets()
          • CustomPlotGenerator.make_plots()
          • CustomPlotGenerator.run_seldonian_experiment()
          • CustomPlotGenerator.validate_constraint_eval_kwargs()
      • experiments.generate_plots.PlotGenerator
        • PlotGenerator
          • PlotGenerator.__init__()
          • PlotGenerator.__repr__()
          • PlotGenerator.make_plots()
          • PlotGenerator.validate_constraint_eval_kwargs()
      • experiments.generate_plots.RLPlotGenerator
        • RLPlotGenerator
          • RLPlotGenerator.__init__()
          • RLPlotGenerator.__repr__()
          • RLPlotGenerator.make_plots()
          • RLPlotGenerator.plot_importance_weights()
          • RLPlotGenerator.run_baseline_experiment()
          • RLPlotGenerator.run_seldonian_experiment()
          • RLPlotGenerator.validate_constraint_eval_kwargs()
      • experiments.generate_plots.SupervisedPlotGenerator
        • SupervisedPlotGenerator
          • SupervisedPlotGenerator.__init__()
          • SupervisedPlotGenerator.__repr__()
          • SupervisedPlotGenerator.generate_resampled_datasets()
          • SupervisedPlotGenerator.generate_trial_datasets()
          • SupervisedPlotGenerator.make_plots()
          • SupervisedPlotGenerator.run_baseline_experiment()
          • SupervisedPlotGenerator.run_fairlearn_experiment()
          • SupervisedPlotGenerator.run_headless_seldonian_experiment()
          • SupervisedPlotGenerator.run_seldonian_experiment()
          • SupervisedPlotGenerator.validate_constraint_eval_kwargs()
    • experiments.headless_example
      • experiments.headless_example.HeadlessExample
        • HeadlessExample
          • HeadlessExample.__init__()
          • HeadlessExample.__repr__()
          • HeadlessExample.run()
    • experiments.perf_eval_funcs
      • experiments.perf_eval_funcs.MSE
        • MSE()
      • experiments.perf_eval_funcs.binary_logistic_loss
        • binary_logistic_loss()
      • experiments.perf_eval_funcs.deterministic_accuracy
        • deterministic_accuracy()
      • experiments.perf_eval_funcs.multiclass_accuracy
        • multiclass_accuracy()
      • experiments.perf_eval_funcs.multiclass_logistic_loss
        • multiclass_logistic_loss()
      • experiments.perf_eval_funcs.probabilistic_accuracy
        • probabilistic_accuracy()