Resolution Guidance

This page states SciCast’s general resolution policy, definitions of common words, and guidelines for specific question types.  SciCast administrators work hard to ensure that all questions are clear and resolvable, and we require authors to list resolution sources where available. However, the future is uncertain, and even paragraph-long legal specifications can be overcome by events.  The historical invalidation rate is around 5%.

Table of Contents

 

General Advice

All other things being equal, the following principles apply:

  • The event trumps the source. Therefore when the named source is unavailable or unsuitable, the judge will look for alternate sources. In cases where equivalent sources disagree the judge will seek direct confirmation or solicit independent expertise.
  • The question text matters. The Fine Print and Resolution Criteria are meant to clarify the  question text, not replace it.
  • Ask before investing.  If you are unsure how the question will be resolved, notify the author or judge with a comment flagged for Admin attention.
  • Pay attention. Subscribe to comments and resolution emails.  The resolution process provides a 48 hour discussion period.  Respond promptly if you disagree with the proposed resolution. Final resolutions are final.  (Though points appeals can be made to [email protected].)

 

Question Statement Phrasing

Dates

All questions contain a “use-by date” — a fixed date when the event or quantity is to be measured. Examples include: whether an event happens BY a certain date, BEFORE a certain date, ON a date, or BETWEEN two dates. Ordered multiple-choice questions often ask WHEN something will happen, within a definite date range. Please use the following definitions when referring to dates:

  • BY: at or prior to the end (23:59:59 pm ET) of the indicated date.  “…by January 1, 2016?” → Event must have occurred on or before January 1, 2016
  • BEFORE: prior to the end (23:59:59 ET) of the previous day.  “…before January 1,2016?” → Event must have occurred on or before December 31, 2015
  • BETWEEN: includes the two dates listed, and all dates in between.  “…Between January 1, 2015 and January 31, 2015?” → Event must occur at any point in JAN

Numerical Ranges

When bin ranges overlap, BETWEEN represents a half-open interval including the lower value; this can also be expressed as [a, b) or a ≤ x < b.  This idiom is often seen when forecasting continuous values using multiple-choice questions. For example,  if the reported value was 17.0% efficiency, then Option C would be correct here:

A) Less than 15% efficiency
B) 15% to 17% efficiency
C) 17% to 19% efficiency…

Similarly, if the values in a multiple choice question are large compared to the unit size, they are often treated as continuous values, so which Option includes 1.3M users?

A) 1M - 1.3M users
B) 1.3M - 1.6M users
C) 1.6M - 1.9M users

The dash signifies “BETWEEN“, so Option B would be correct in this case.  This style prevents us from having to write, “B) 1.3M - 1,599,999″ users.

As noted above, dates use non-overlapping bins because that best matches common usage.  Some questions with small count ranges will also use non-overlapping bins.
 

Scaled Questions

For Scaled (Continuous) questions, instead of estimating the chance of a particular outcome, you are asked to forecast the actual expected value. Forecasts moving the estimate towards the actual value will be rewarded. Those moving it away will be penalized. As with probability questions, moving towards the extremes is progressively more expensive. (We have merely rescaled the usual 0%-100% range and customized the interface.)

Should the actual value fall outside the question range, the question will resolve at the nearest extreme. Note that it would take infinite points to move the forecast exactly to that extreme, just as it would take infinite points to move a binary question exactly to 0% or 100%. This is because moving it all the way to the extreme means you assign 0% chance of the question resolving in the interior — an extreme position.

To maximize your own expected gain (on a single question), you should move the forecast towards your own estimate of the expected value. So if the question is how many heads will be observed on 100 tosses of a fair coin, you should move the estimate towards 50. Note that you should also do that if the tosses are going to be with either a double-headed or double-tailed coin, but you don’t know which. Of course your actual gain will be larger if you extremized correctly, but your expected gain will drop quickly as you depart from your true belief.

0