Tag Archives: SciCast

Google Analytics statistics for SciCast, as of May 22, 2015.

SciCast Final Report Released

The final SciCast annual report has been released!  See the “About” or “Project Data” menus above, or go directly to the SciCast Final Report download page.

Exeutive Summary (excerpts)

Registration and Activity

SciCast has seen over 11,000 registrations, and over 129,000 forecasts. Google Analytics reports over 76K unique IP addresses (suggesting 8 per registered user), and 1.3M pageviews. The average session duration was 5 minutes.

Continue reading

3+

Users who have LIKED this post:

  • avatar
Color_Logo

SciCast Final Report (Public)

The SciCast 2015 Annual Report has been approved for public release. The report focuses on Y4 activities, but also includes a complete publication and presentation list for all four years.  Please click “Download SciCast Final Report”  to get the PDF.  You may also be interested in the SciCast anonymized dataset.

Here are two paragraphs from the Executive Summary:

We report on the fourth and final year of a large project at George Mason University developing and testing combinatorial prediction markets for aggregating expertise. For the first two years, we developed and ran the DAGGRE project on geopolitical forecasting. On May 26, 2013, renamed ourselves SciCast, engaged Inkling Markets to redesign our website front-end and handle both outreach and question management, re-engineered the system architecture and refactored key methods to scale up by 10x – 100x, engaged Tuuyi to develop a recommender service to guide people through the large number of questions, and pursued several engineering and algorithm improvements including smaller and faster asset data structures, backup approximate inference, and an arc-pricing model and dynamic junction-tree recompilation that allowed users to create their own arcs. Inkling built a crowdsourced question writing platform called Spark. The SciCast public site (scicast.org) launched on November 30, 2013, and began substantial recruiting in early January, 2014.

As of May 22, 2015, SciCast has published 1,275 valid questions and created 494 links among 655 questions. Of these, 624 questions are open now, of which 344 are linked (see Figure 1). SciCast has an average Brier score of 0.267 overall (0.240 on binary questions), beating the uniform distribution 85% of the time, by about 48%. It is also 18-23% more accurate than the available baseline: an unweighted average of its own “Safe Mode” estimates, even though those estimates are informed by the market. It beats that ULinOP about 7/10 times.

You are welcome to cite this annual report.  Please also cite our Collective Intelligence 2014 paper and/or our International Journal of Forecasting 2015 paper (if it gets published — under review now).

Sincerely,

Charles Twardy and the SciCast team

1+

Users who have LIKED this post:

  • avatar
%SafeMode Trades vs #Trades,

Analysis of SciCast Safe Mode Usage

by Kellen Leister

Note: The following post has been revised slightly to use a power law (x^k) instead of a logarithmic function (k ln x). This is because the logarithmic function could go negative, which we know the percentage of trades cannot do. Revisions are in red.

Given the ease of switching back and forth between Safe Mode and Power Mode, we were interested to observe the relative use of each mode.  (For a description of the different modes, see the Appendix.) Continue reading

2+

SciCast Accuracy Calculations

We have started adding “Accuracy” numbers to emails. For example:

  • Your average accuracy on this question was 83.
  • SciCast’s average accuracy on this question was 90.

What does that mean?  The short answer is that it’s a transform of the familiar Brier score, which we have mentioned in several blog posts.  Where the Brier measures your error (low is good), Accuracy measures your success (high is good). This is more intuitive … except when it’s not.

Continue reading

2+

Users who have LIKED this post:

  • avatar

New features: recurring forecasts and crowdsourced spam protection

Mark as Spam

Over the past 3 weeks we have more than tripled the number of users on SciCast. Hooray! But with growth comes growing pains, one of which is an increased amount of spam in our comments. Best as they try, certain people just can’t resist writing complete nonsense in to a discussion thread. So we’ve introduced a way for the better citizens of SciCast to mark a comment as spam:

Screen-Shot-2014-07-08-at-10.56

A comment must be marked as spam a certain number of times by multiple people. If it is, it will be hidden from the comment thread. So please do your duty and if you see something that’s clearly spammy, mark it. If you’re unsure, ask us about it.

Another change to commenting is the requirement to have verified your email address before you can comment. Currently we do NOT require an email address to sign up and begin participating in SciCast. But a quick survey of “spam” discussion items revealed that many of those comments were tracing back to people who did not provide an email address as part of their registration. We still aren’t requiring a valid email address to register, but we are to make a comment. We hope this too cuts down on spammy comments.

Recurring Forecasts

An even more substantive change we’re excited about is “recurring forecasts.” If you ask SciCast to, we will make a forecast for you once a day in a given question based on an initial forecast. So if you use Power Mode to raise the chance to 75%, you can spend a certain number of points to raise it to 75% whenever the once-a-day check finds it below that threshold.  If you used Safe Mode, we’ll simply make that same forecast for you for 7 days in a row (or for however many days you specify) using the usual Safe Mode rules.

Here’s an example:

Let’s say I made a forecast that it’s “Unlikely (20%-40%)” that “The same machine will hold the #1 rank on the Top500 and Graph500 lists in August 2014.” Once I’ve completed the forecast, I can ask SciCast to make the same forecast for me once a day for a week, by checking the box.

Screen Shot 2014-07-08 at 10.56.43 PM

If I’m forecasting in power mode, each edit can be much bigger (and more expensive), so instead of #days, I specify a total budget.  The power-mode forecast could use it all in one day, or it could last for months, depending on how much each edit costs:

Screen Shot 2014-07-08 at 10.57.10 PM

The values “1 week” and “300 points” are just defaults.  I can use “My Dashboard” for more control.  Here “My Dashboard -> My Activity -> My Questions” shows me the most recent forecast I made on each question, and offers me a customizable way to make it recur.

Screen Shot 2014-07-08 at 10.57.55 PM

In the figure above, my last Adelgid forecast used Power Mode, so it offers me the option to make “>78%” a recurring edit, for however many points I want.  My last Graph500 forecast used Safe Mode, so it offers me the option of a recurring Safe Mode forecast, for a specifiable #days (currently 1-7 only).

Recurring forecasts are tracked in your forecast history and marked as such so you can keep track of what was the original forecast and what was recurring.  Use “My Dashboard -> My Activities -> My Recurring” to see and cancel them.My Recurring

Next Steps

Please let us know how you like these new features. We’re continuing to work on a better commenting system and even more powerful and efficient forecasting tools, and this release is an important first step down that path.

 

0

Want to Win an Amazon Gift Card? These SciCast Forecasters Did!

Fortune Friday Amazon Gift Card Winners on 6/6/14

Congratulations, winners! The following SciCast forecasters won Fortune Friday Amazon gift cards! Learn more.

Winners, please let us know your Twitter handle if you have one, and we’ll tweet to you directly when you win. Be sure to share your news with friends!

Miku
Disinterest
benthinkin
dilettante
bw
dvasya
Salemicus
william.quixote
scicast33
Numberwang
SandyC
averagefruit
spaf
BK8324
poopstix
ejh444
arjunr
seachelle53
MarketwannaBE
Geoff
pigmoose

0

Badge Wednesday Winners this Week!

Badge Winners 6/4/14

Congratulations, winners! The following SciCast forecasters won badge merit points. Learn more.

Winners, please let us know your Twitter handle if you have one, and we’ll tweet to you directly when you win. Be sure to share your news with friends!

DouglasM
benthinkin
dilettante
william.quixote
sflicht
Dlurker
PrieurDeLaCoteD’Or
kristin_segura
danutile
giedriusbl
rmo
ted
TheOnePercent
CVGBs
jkominek
Disinterest

0

Badge Wednesday Winners!

Winners 5/28/14

Congratulations, winners! The following SciCast forecasters won the second round of prizes: badge merit points. Learn more.

Winners, please let us know your Twitter handle if you have one, and we’ll tweet to you directly when you win. Be sure to share your news with friends!

Disinterest
jmkina
dilettante
rfuoco
MarketwannaBE
baileyharper
jonmckay93
MemberFDIC
pigmoose
MethodicalMan
bmth94
wbutton
abhayg
leabk
sflicht
e
dvasya
astrojacq
peace0831
PrieurDeLaCoteD’Or
Geoff
chewitt
mmachin
poopstix
Saint_Fiasco
quetzyl
mev
spaf
KRH
HAL9000

 

 

0