The question, A young paraplegic Brazilian, assisted by a neurorobotic exoskeleton, will take the ceremonial first kick at the 2014 World Cup in São Paulo, Brazil. How far will the ball travel? has resolved as “Between 1-3 meters”. Our Brier score was 0.326, better than uniform.
There were 275 forecasts made by 53 unique users. Did you make a forecast? If so, login and check your dashboard.
Also, check out the epic comment stream on this popular SciCast question.
Analysis of the Predictions
Daily forecast snapshot. Note the confident but incorrect edit made by one forecaster just prior to the question locking.
Our raw market Brier Score1 (before smoothing or adjustments) was 0.82 0.33, which is better than the 0.88 0.41 that a uniform distribution would have achieved.
We apologize: we previously calculated the score as if the options had no natural order. But of course 0m < 1-3m < 3-5m < …, so we should have used the Ordered Brier.
Here is a histogram of the Brier scores following each individual edit. The strong spike just below 0.1 is presumably @ted’s persistent edits, which turned out to be correct.
The Brier Score (Brier 1950) measures the accuracy of probabilistic predictions. Lower is better. For dates, counts, and other ordered questions, we use the related Ranked Probability Score (Gneiting & Raftery 2007). Both range from 0 to 2 and we report them together as Briers. On a binary question, a 50-50 forecast yields a Brier of 0.5.
Comments
There were 120 participants’ comments. Here are a few:
Thanks to everyone who took the time to weigh in on this question. We’ve decided that we’re going to settle the “Between 1-3 meters” option 100%. That is, literally, “the total horizontal length the ball travels from the point of the kicked until it comes to a stop” which is how the question was written. As @DouglasM pointed out, it appears that an extrapolation of the available evidence would produce the same result, which will hopefully make this outcome easier to swallow for anyone losing out on points. Thanks for making this question one of the most heavily forecasted questions on SciCast to date, and for helping us reach a decision during this unexpected turn of events!
I’m having way too much fun with this discussion, but it probably has to come to an end, or at least a judgment, at some point. After thinking some more, it seems that the best choice is to be very literal and call it 1-3 meters based on the distance the ball actually traveled before being stopped and picked up. Since it appears that an “if only it had rolled” extrapolation will produce the same result, the point is moot..
I would like to note that this is the way soccer kickoffs almost always work, as I understand them. The kicker kicks the ball and then the ball stops when it reaches a referee or another player. I have never seen a soccer kickoff where someone kicks it and everyone crowds around until it comes to a complete halt. There’s always a target that you kick it to (either player or ref) and that target stops the ball when the ball arrives. I think the kick here was totally normal and if some forecasters made bets without realizing how soccer kickoffs work, that’s unfortunate, but not worth voiding a question over. Voiding this question would set a bad precedent, as now any time someone doesn’t agree with a question in the future they can complain it was unclear. The role of the fine print is to avoid these misunderstandings and the fine print is clear here - it’s the distance the ball rolls before coming to a stop. The question should resolve 1-3 meters.
@Prieur: “Distance will be the total horizontal length the ball travels from the point of the kicked until it comes to a stop.” Was the ball kicked? Did it stop? Yes and yes. “A young paraplegic Brazilian” - was he young, paraplegic, and Brazilian? “assisted by a neurorobotic exoskeleton” - the exoskeleton is obvious. “will take the ceremonial first kick” - he did. “2014 World Cup in São Paulo, Brazil” - the location and the event are correct. In each of your examples one of these is violated, so it makes sense to void the question in your hypothetical example. The way it came out to be, I don’t see any contradiction to the terms of the question. Unless the kicker wasn’t Brazilian, of course…
Did you like this post? Follow us on Twitter.
1. The Ordered Brier is officially known as RPS.
The real story here of course is @ted’s high-risk betting strategy that rocketed him to the top of the leaderboard in one swell foop. Read the comments stream. Note that this strategy is only for risk-seekers. @ted was well aware it’s a high-variance strategy — he deliberately chose not to maximize expected utility, but the chances of a quick “win”. Note that to have any hope of working, have good reason to think you have an “edge” over other forecasters.
By the way, I think that the 0.82 is probably including the last incorrect after-the-fact but before-the-market-closed edit. I’ll ask Ken to re-examine how the mainline forecasting scored.
I’d still like to know how many of the forecasters on that question watched the video available on their facebook page. i think it might be the video you get on https://www.facebook.com/pages/Miguel-Nicolelis/207736459237008 by going down to may 22nd. certainly that one is nearly identical to what i watched.
i had almost no points in the question until i saw that, and then i (weakly) started putting them into the very low end with ted. i can’t imagine how anyone watching that thought the operator was going to wind up and put any force into the ball. i feel ted was pretty safe.
Actually, I knew the whole time that I was making positive expectation bets. If I was truly seeking high variance, I would have bet on only one option instead of hedging among 2-3. I only said I was seeking high variance so that other bettors would be less careful in betting against me.
The Brier scores were incorrect, but for a different reason — the question was marked as an “unordered” multi-choice, so the wrong Brier formula was used. The actual Brier was 0.33, better than the 0.41 for the corresponding uniform forecast. The post has been updated.
@ted: Touché. I mistakenly thought you went all in for a single option. Shoulda’ checked.