3.4 Describe Methods to Evaluate FM Performance
Targeted AIF-C01 practice for this exam objective, with an explanation for every answer option.
- Certification
- AIF-C01
- Domain
- Applications of Foundation Models
- Questions
- 6
Practice AIF-C01 3.4 Describe Methods to Evaluate FM Performance questions with exam-style scenarios, detailed explanations, and option-by-option rationale for this topic area.
Example question from this objective
Each option is explained, so the reasoning behind the distractors is part of the answer.
An LLM judge consistently favors verbose answers even when concise answers satisfy the rubric. What is the best response?
- Raise the judge model's temperature so its scores vary less predictably.
- Revise the rubric and examples, test for position and style bias, and retain human review for disputed cases.
- Switch from pointwise scoring to pairwise comparison, leaving the rubric unchanged.
- Normalize the judge's scores after the fact by dividing each score by the answer's length.
Raise the judge model's temperature so its scores vary less predictably. — Temperature controls sampling randomness, so raising it adds variance to the scores rather than removing a systematic preference. A noisier biased judge is harder to evaluate, not fairer.
Revise the rubric and examples, test for position and style bias, and retain human review for disputed cases. — Explicit criteria, bias tests, and human adjudication reduce systematic evaluator distortion.
Switch from pointwise scoring to pairwise comparison, leaving the rubric unchanged. — Pairwise comparison removes some absolute-scale drift, but a judge that rewards verbosity will keep choosing the longer answer in each pair. Without changing the rubric the bias survives the change of format.
Normalize the judge's scores after the fact by dividing each score by the answer's length. — Length normalisation hides the symptom in the aggregate score but leaves the judge still preferring verbose answers on each individual comparison, and it penalises answers that are genuinely long because the task requires it.