fairness metrics

Quantitative measures of whether an AI system treats different groups equitably. Key metrics include demographic parity, equalized odds, and predictive parity.

Syntax

ai-ethics

demographic_parity: P(y_hat=1 | A=0) = P(y_hat=1 | A=1)

Example

ai-ethics

# Fairness evaluation:
from fairlearn.metrics import MetricFrame
from sklearn.metrics import accuracy_score, selection_rate

mf = MetricFrame(
    metrics={"accuracy": accuracy_score, "selection_rate": selection_rate},
    y_true=y_test,
    y_pred=y_pred,
    sensitive_features=test_data["gender"]
)

print(mf.by_group)
# gender  accuracy  selection_rate
# female  0.82      0.31
# male    0.91      0.67  <- disparity!