We have started adding “Accuracy” numbers to emails. For example:
What does that mean? The short answer is that it’s a transform of the familiar Brier score, which we have mentioned in several blog posts. Where the Brier measures your error (low is good), Accuracy measures your success (high is good). This is more intuitive … except when it’s not.
Users who have LIKED this post:
Prediction market performance can be assessed using a variety of methods. Recently, SciCast researchers have been taking a closer look at the market accuracy, which is measured in a variety of ways. A commonly used scoring rule is the Brier score that functions much like squared error between the forecasts and the outcomes on questions.
The Great Lakes questions have resolved.
Lake Ontario
What will be the peak mean ice concentration in Lake Ontario during the 2013/2014 winter? has resolved as “56% or greater”. Data from NOAA coastal watch indicates Lake Ontario ice concentration hit 61.52%.
Analysis of the Predictions
Our raw market Brier Score (before smoothing or adjustments) was .75.
The question, Will bitcoin be accepted at online retailer, Amazon, by March 31, 2014? has resolved “False/No” because bitcoin has NOT been accepted by online retailer, Amazon. However, for information on how to use bitcoin to shop on Amazon (via digital gift card purchase) view this Forbes article.
Analysis of the Predictions
Our raw market Brier Score1 (before smoothing or adjustments) was 0.02.
The Brier Score (Brier 1950) is a measurement of the accuracy of probabilistic predictions. As a distance metric, a lower score is better than a higher score. The market Brier Score ranges from 0 to 2, and is the sum of the squared differences between the individual forecasts and the outcome weighted by how long the forecasts were on the market. On a binary (Yes/No) question, simply guessing 50% all the time yields a “no courage” score of 0.5.
There were 214 forecasts made by 86 unique users.
There were 12 participants’ comments. Here are a few:
Apple recently moved to remove iOS apps that handle bitcoin, apparently to avoid legal issues related to bitcoin’s current ambiguous legality. I would not think Amazon would move in the opposite direction by March 31.
More woes for Bitcoin: Bitcoin Trading Technology In Question As Currency Dives After Glitch
It is still too early for bitcoin. It will take at least another 4 or 5 years for a company of size or a country to accept bitcoin. Changes in currency, let alone international, having in many cases required more than decades to make decisions regarding changes which result in acceptance. How many years did it take the Euro?
I don’t see the Euro as a great comparison point. At this stage, Bitcoin isn’t trying to replace another currency, even if that’s what some of its more ardent supporters see as its endgame. For this question to resolve as true, there doesn’t need to be any massive shift in public policy. Amazon doesn’t need to stop accepting dollars. It just needs to do what overstock.com did.
Check out these related forecasting questions that are still active on the market.
Did you like this post? Follow us on Twitter.
1. The Ordered Brier is officially known as RPS.