An associate evaluates a visual question-answering pipeline on a benchmark of 600 images with five questions about each image, 3,000 questions in total. Accuracy is 71%. The associate computes a 95% confidence interval by bootstrap resampling the 3,000 question-level scores and gets a narrow interval. A reviewer notes that questions about the same image tend to be answered right or wrong together. How should the interval be computed instead?
- A.
Keep resampling individual questions, since 3,000 separate scores are plenty for the bootstrap to give an accurate interval
- B.
Resample whole images with all five of their questions in each bootstrap draw, then recompute accuracy per draw
- C.
Replace the bootstrap with a normal-approximation interval computed from the 3,000 question-level scores
- D.
Raise the number of bootstrap iterations from 10,000 to 100,000 so that the interval becomes more accurate
Show answer
Answer: B
When scores are correlated within images, bootstrap the images, not the individual questions.
- A. More questions do not help when they are not independent; question-level resampling treats 3,000 correlated scores as 3,000 separate pieces of evidence.
- B. Resampling the image, the unit that makes answers correlated, carries the within-image dependence into every draw, so the interval reflects it.
- C. A normal approximation on question-level scores makes the same independence assumption, so it is just as narrow.
- D. More iterations reduce Monte Carlo noise in the bootstrap itself but do not change what is being resampled, so the bias stays.