The question, What percent of managed honey bee colonies in the US will be lost during the 2013-2014 winter? has resolved as “Between 21% and 25%,” with a Brier score of 0.31, notably worse than the baseline of 0.21. There were 252 forecasts by 73 users.
For more information on the resolution, view this post on Bee Informed.
Analysis of the Predictions
Our raw market Brier Score1 (before smoothing or adjustments) was 0.31. The baseline uniform forecast did better, achieving a Brier of only 0.21, using the ordered Brier (aka Ranked Probability Score) as appropriate to this question. Participant assessment was that we put too much emphasis on the Midwest state reports, without considering how much of the US bee population winters in California, which had far lower mortality.
The Brier Score (Brier 1950) is a measurement of the accuracy of probabilistic predictions. As a distance metric, a lower score is better than a higher score. The market Brier Score ranges from 0 to 2, and is the sum of the squared differences between the individual forecasts and the outcome weighted by how long the forecasts were on the market. On a binary (Yes/No) question, simply guessing 50% all the time yields a “no courage” score of 0.5.
There were 252 forecasts made by 73 unique users.
Comments
There were 20 participants’ comments. Here are a few:
My understanding (from a beekeeper in NY state I know) is that despite the importance of bee pollination services to CA agriculture, the vast majority of the beekeepers in question winter their bees in other locations and truck them in. If anyone has actual data, I’m certainly curious.
“Neglecting California” is sort of right. The issue is that we’re forecasting based on what we read in the news. Only extreme cases are reported in the news. We didn’t know whether lack of information from the non-Midwest was because there was extreme news that I couldn’t identify, or if it was because everything was okay everywhere else, so there was nothing bad to report.
I’m happy to hear the bees are doing better than expected, but hope someone has time to dive into the report and post-mortem our forecasting. Was it simply that we neglected California?
Check out these related forecasting questions that are still active on the market.
Did you like this post? Follow us on Twitter.
1. The Ordered Brier is officially known as RPS.
In this case unfortunately we did worse than a uniform distribution. The Ordered Brier (RPS) for 6 options resolving at the 3rd option is 0.21. #chastened