Found a fun little quiz somewhere, which I thought some of my readers might like to take. My aim is not to embarrass people who may get some answers wrong – in testing, the vast majority of all respondents (including researchers who reported substantial experience) were found to make mistakes. My hypothesis is that my readers are rather more intelligent than average 🙂 Please answer in comments but work out your answers before reading what others have said, so as not to be unduly influenced by them.
I will summarise and explain the quiz when enough have answered…
A researcher undertakes an experiment and reports “the 95% confidence interval for the mean ranges from 0.1 to 0.4”
Please mark each of the statements below as “true” or “false”. False means that the statement does not follow logically from the quoted result. Also note that all, several, or none of the statements may be correct:
1. The probability that the true mean is greater than 0 is at least 95%.
2. The probability that the true mean equals 0 is smaller than 5%.
3. The “null hypothesis” that the true mean equals 0 is likely to be incorrect.
4. There is a 95% probability that the true mean lies between 0.1 and 0.4.
5. We can be 95% confident that the true mean lies between 0.1 and 0.4.
6. If we were to repeat the experiment over and over, then 95% of the time the true mean falls between 0.1 and 0.4.
BlueSkiesResearch.org.uk 
Ok, I’ll bite:
1. The probability that the true mean is greater than 0 is at least 95%.
True.
2. The probability that the true mean equals 0 is smaller than 5%.
True.
3. The “null hypothesis” that the true mean equals 0 is likely to be incorrect.
Might not be a proper “null hypothesis”, and “likely” must be defined as 95% probabiliy.
4. There is a 95% probability that the true mean lies between 0.1 and 0.4.
True
5. We can be 95% confident that the true mean lies between 0.1 and 0.4.
True.
6. If we were to repeat the experiment over and over, then 95% of the time the true mean falls between 0.1 and 0.4.
False. The true mean might be 0.1 or 0.4, and then a large number of repeating the experiment would have only 50% of estimated means between 0.1 and 0.4.
I suspect I’ll learn something by answering.
I think I agree with Phil (I would probably have got 6 wrong without him).
Incidentally, you’ve also posted this at http://julesandjames.blogspot.com/2019/05/blueskiesresearchorguk-how-confident.html so will get divergent comments.
2 could be false, if you allow improbable distributions: if the mean were known to be in [0.1, 0.4] with 95%, and only otherwise 0, then it would be 5%, not less than.
Not sure who/how many read one blog vs the other hence double post. Will collate all answers anyway.
Oh, hold on, what does “6. If we were to repeat the experiment over and over, then 95% of the time the true mean falls between 0.1 and 0.4” mean? Obviously the true underlying mean doesn’t actually change. By “true mean” do you mean the researchers best estimate (in which case PH’s answer applies), or the underlying thing (in which case it is inside the range either 0% of the time or 100% of the time, so could be 0%, so 6 would be false, but for a different reason)?
Hey don’t shoot the messenger. These aren’t my questions. But I can refer you to the instructions at the top.
Pingback: How confident should you have been about confidence intervals? | BlueSkiesResearch.org.uk