Skip to content
LearnMLByDoing
Free

6.5 Responsible AI, fairness and generative AI security

30 minProintermediateFairMLOWASPAIP-C01AIF-C01AI-103GCP-PMLE

Two problems every deployed AI system has to answer for: whether its decisions treat groups fairly, which turns out to be a choice between criteria that cannot all hold at once, and whether its generative parts can be steered by text they were only meant to read.

01What responsible AI covers

Responsible AI is the engineering discipline of making a model's behaviour defensible to the people it affects. The cloud vendors' frameworks differ in wording but cover the same ground: fairness (similar people are treated similarly, and groups are not systematically disadvantaged), transparency and explainability (people can find out what the system is and why it decided), privacy (personal data is collected, used and exposed only as agreed), safety and security (the system resists misuse and does not cause harm), and accountability (a named owner, documentation, review and a way to contest a decision).

This lesson makes two of them concrete. The viewport's fairness view shows one credit model scoring applicants from two groups. Each lane is one group: the front ribbon is applicants who would repay (y=1y=1), the back ribbon those who would not (y=0y=0), drawn over the score ss. The model is equally good for both groups (the same two score distributions, AUC 0.921), but the groups' base rates differ: πA=0.5\tok{pi}{\pi_A}=0.5 and πB=0.3\pi_B=0.3. Each lane has its own threshold plane, tA\tok{tA}{t_A} and tB\tok{tB}{t_B}, both at 0.5. Colours follow lesson 2.8: emerald TP, amber FN, rose FP, grey TN.

On the right, four bars show the gap between the groups in four rates, and the panel behind them traces each gap while the thresholds sweep from 0 to 1 together. The security view shows the other half: a retrieval-augmented assistant whose prompt is assembled from trusted and untrusted text, and the guardrails that decide whether an attacker's sentence turns into an action.

Switch the viewport to the prompt pipeline with every guardrail off.

Fairness and Machine Learning, Barocas, Hardt & Narayanan, Ch. 1, Introduction

Developing AI Apps and Agents on Azure (study guide), Microsoft, Responsible AI principles

02Group fairness metrics from the confusion matrix

Start with people you can count. Twenty applicants from each group: in group A, 10 of the 20 would repay; in group B, 6 of 20. That difference is the base rate, 0.5 against 0.3, and it is a fact about the populations (history, income, how the label was recorded), not about the model. Drag either threshold in the widget, or use the buttons. Both thresholds are shared with the 3D lanes.

Loading figure
With same t = 0.5: A selects 10 of 20 (TPR 8/10=0.8008/10=0.800, FPR 2/10=0.2002/10=0.200, PPV 8/10=0.8008/10=0.800); B selects 7 (TPR 5/6=0.8335/6=0.833, FPR 2/14=0.1432/14=0.143, PPV 5/7=0.7145/7=0.714). Equal selection lowers tBt_B to 0.44: B also selects 10, but now every qualified B applicant is selected (TPR 1.000) and PPV falls to 6/10=0.6006/10=0.600. Equal PPV raises tBt_B to 0.68: B's PPV is 4/5=0.8004/5=0.800 like A's, but B selects only 5 and TPR drops to 4/6=0.6674/6=0.667.

Each group has its own confusion matrix, and every group fairness criterion asks that some ratio of it be equal across groups. Barocas, Hardt and Narayanan group them into three families: independence (the decision ignores the group), separation (given the true outcome, the decision ignores the group) and sufficiency (given the decision or score, the outcome ignores the group).

demographic parity:P(y^=1∣A)=P(y^=1∣B)SRA=SRBequal opportunity:P(y^=1∣y=1,A)=P(y^=1∣y=1,B)TPRA=TPRBequalised odds:equal opportunity and P(y^=1∣y=0,g) equal+ FPRA=FPRBpredictive parity:P(y=1∣y^=1,A)=P(y=1∣y^=1,B)PPVA=PPVB\begin{aligned}&\text{demographic parity:} && P(\hat y{=}1\mid A)=P(\hat y{=}1\mid B) && \tok{sr}{\mathrm{SR}}_A=\mathrm{SR}_B\\&\text{equal opportunity:} && P(\hat y{=}1\mid y{=}1,A)=P(\hat y{=}1\mid y{=}1,B) && \tok{tpr}{\mathrm{TPR}}_A=\mathrm{TPR}_B\\&\text{equalised odds:} && \text{equal opportunity and } P(\hat y{=}1\mid y{=}0,g)\text{ equal} && +\ \tok{fpr}{\mathrm{FPR}}_A=\mathrm{FPR}_B\\&\text{predictive parity:} && P(y{=}1\mid\hat y{=}1,A)=P(y{=}1\mid\hat y{=}1,B) && \tok{ppv}{\mathrm{PPV}}_A=\mathrm{PPV}_B\end{aligned}
Independence, separation (two forms) and sufficiency for a thresholded decision. Calibration within groups, P(y = 1 | s, g) = s, is the score-level form of sufficiency.
per-group confusion matrices[,,]8 values

Every criterion compares a ratio of cells across the first axis. A fairness report is this tensor plus the base rates; an overall accuracy averages it away.

The viewport draws the same thing for the full population, at tA=tB=0.5t_A=t_B=0.5. Because the score separates the classes equally well in both lanes, one threshold gives both groups the same TPR (0.841) and FPR (0.159): the emerald and rose bars are flat. But A selects 0.500 of its applicants and B 0.363, a parity gap of +0.137+0.137, and among the selected, 0.841 of A repay against 0.694 of B, a PPV gap of +0.147+0.147. The same score means a different repayment rate in each group, because the groups' base rates differ.

Lower tBt_B to 0.374, the threshold at which group B's selection rate matches A's 0.500, and hold the sweep.

Fairness and Machine Learning, Barocas, Hardt & Narayanan, Ch. 3, Classification (independence, separation, sufficiency)

03Why the criteria conflict, and how to choose

The viewport now enforces demographic parity: you set tAt_A and the scene solves for the tBt_B that equalises selection. Watch the sweep: the cyan curve in the panel stays at zero for every tAt_A, while the PPV curve (white) climbs as high as 0.285 and only sinks toward zero at the far right, where almost nobody is selected and both PPVs approach 1. That is not a quirk of this model. Whenever base rates differ and the classifier is not perfect, the criteria of the previous step cannot all hold at once.

Chouldechova's identity (2017)1/4
  1. 1
    PPV=π TPRπ TPR+(1−π) FPR\mathrm{PPV}=\frac{\pi\,\mathrm{TPR}}{\pi\,\mathrm{TPR}+(1-\pi)\,\mathrm{FPR}}

    Bayes' rule on one group's confusion matrix: the selected are true positives (π TPR\pi\,\mathrm{TPR} of the group) plus false positives ((1−π) FPR(1-\pi)\,\mathrm{FPR}).

Kleinberg, Mullainathan and Raghavan (2016) proved the score-level version: a risk score cannot be calibrated within groups and also balanced for the positive class and for the negative class (equal average score among those with y=1y=1, and among those with y=0y=0) unless base rates are equal or prediction is perfect. Both results are arithmetic, not opinion. Try the three criteria on the same tA=0.5t_A=0.5:

Equalise TPR across groups at tA=0.5t_A=0.5 and hold.

Now equalise PPV instead, still at tA=0.5t_A=0.5.

Keep predictive parity enforced and set the base-rate gap to 0 (πA=πB=0.4\pi_A=\pi_B=0.4).

Mitigation can happen at three stages. Pre-processing changes the data: reweighting or resampling so groups and outcomes are balanced, fixing label bias, removing proxies. In-processing changes training: a fairness constraint or penalty in the loss, or adversarial debiasing. Post-processing changes the decision: group-specific thresholds like this lab's, as in Hardt, Price and Srebro's equal-opportunity post-processing. Post-processing needs the group attribute at decision time, which some jurisdictions restrict, so it is a legal question as much as a technical one.

Fairness and Machine Learning, Barocas, Hardt & Narayanan, Ch. 3, Relationships between criteria

The rest of this lesson is in Pro

  • 04Explainability and documentation
  • 05Generative AI threats: injection, exfiltration and unsafe tools
  • 06Layered defences and governance
  • Python challenge: Measure fairness gaps and redact PII
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between demographic parity, equal opportunity and equalized odds?
Demographic parity asks that each group is selected at the same rate, regardless of who is qualified. Equal opportunity asks that qualified people are selected at the same rate in every group, so it compares true positive rates. Equalized odds adds the same requirement for unqualified people, so both the true positive rate and the false positive rate must match. When groups have different base rates, demographic parity and the other two usually pull the thresholds in opposite directions.
Why can a model not be fair by every definition at once?
Because the fairness metrics are linked by arithmetic. For a binary decision the false positive rate equals the base-rate odds times the odds of a wrong selection times the true positive rate, so if two groups have different base rates, a classifier that is not perfect cannot have equal positive predictive value, equal true positive rate and equal false positive rate at the same time. Kleinberg and colleagues and Chouldechova proved versions of this in 2016 and 2017. In practice you choose the criterion that matches the harm you most need to avoid and document the trade-off.
How do you prevent prompt injection in an LLM application?
You cannot prevent it completely, because the model reads instructions and data through the same channel, so you layer controls. Screen inputs and retrieved content with an injection and content filter, mark untrusted text clearly in the prompt, give the model only the tools and permissions its task needs and require human confirmation for actions with side effects, and filter outputs for links, markup and personal data before they are rendered or passed on. Then red-team the system, evaluate it like a regression test suite and monitor tool calls and blocked requests in production.

Sources

  • Fairness and Machine Learning, Barocas, Hardt & Narayanan
  • OWASP Top 10 for LLM Applications, OWASP Foundation
  • AWS Certified Generative AI Developer – Professional exam guide, Amazon Web Services
  • AWS Certified AI Practitioner exam guide, Amazon Web Services
  • Developing AI Apps and Agents on Azure (study guide), Microsoft
  • Professional Machine Learning Engineer exam guide, Google Cloud