SciCast is running a new special! The most accurate forecasters during the special will receive Amazon Gift Cards:
• The top 15 participants will win $2250* to spend at Amazon.com
• The other 135 of the top 150 participants will win $225 to spend at Amazon.com
Participants will be ranked according to their total expected and realized points from their forecasts during the special.
Be sure to use SciCast from November 7 through March 6!
*Winners will receive an IRS 1099 Form for tax purposes.
Wow, we’re talking real money here! Do you know if this is the largest real-money prize pool ever allocated for a single (play-money) prediction market contest? Hypermind seems to have $17K available, which is the largest I’d seen before. Can you share any of the plans you have to publicize this contest? I suspect you’re less bullish on banner ads now
I suggest /r/futurology on Reddit as one potential recruiting avenue.
I don’t know if it’s the largest, but it’s definitely designed to be competitive with sites like CrowdMed and Hypermind. Thanks for your suggestion.
So it appears that this contest allows automated trading within “the approved scripting API & terms.” Where exactly is all of this spelled out? I know that in the past, the “crazy bets” on popular questions have been quickly scooped up by a particular forecaster. I know that this forecaster has generously shared some code on Github, but deciphering and customizing it will take quite a bit of effort. Building something comparable to his automated system would be a huge task.
But I suspect that all 15 of the “big winners” will have fully-automated trading systems. The rules don’t seem to preclude “buying/selling” such a system; do you have a problem with a developer “sharing” such a system at $50 a pop? That might prove even more lucrative than the potential winnings. Is the frequency of hits against the API closely monitored and regulated? I wish that SciCast had chosen to spend some of this cash on developing/providing better tools for the masses, rather than providing easy winnings for the programming savvy. Now that there’s real money involved, perhaps it makes sense for SciCast to provide a page for authorized third-party vendors. Otherwise, SciCast risks scaring off desirable but less-committed (or programming savvy) sci/tech experts. They aren’t likely to stick around in an auto-trade circus to which they aren’t invited.
FWIW, the battle between John Henry and the steam engine is not yet over. @dvasya, @prieur, and I all still seem to be making the vast majority of our trades manually, and along with the rest of the leaderboard I think we’re holding our own quite nicely against @jkominek’s system. You’re presumably concerned that a new ad campaign will bring in a lot of dumb money and only @jkominek will be well-placed to profit from this. That’s fair, but I think the concern that new *smart* forecasters will be scared away by the presence of bot trading is somewhat overblown.
Since scicast now records what trades are automated, you could turn that suspicion into a meta-question for us.
The API was pretty much already there, it’s how the AJAX of the website operated. While I’m sure some funds were spent on it, almost no changes were made from what I’d discovered and was using before the release, except for documentation so that it was easier to use.
I’ll be keeping my program free, and while someone could make additions and charge for them, that’ll just tell me what features are desirable, and indicate that there is actual interest and demand for updates to the program.
Hey, @sflicht, “overblown” is what I’m all about. Of course an auto-trade is only as good as the “trade” being made, which obviously requires forecasting insight. The problem is that in any possible situation, an instant trade is always better than a slow trade, and the auto-sflicht will always beat manual-sflicht in the end (all other things being equal). The meat-based leaders may be holding their own, but they are also making the overwhelming majority of the trades here. The total volume is a fraction of what it was in July. And much of your recent gain is also likely a legacy of trades made long before the entry of the bot. And I suspect that @jkominek is far more conservative than what you’ll see during the contest. My point is that the autobots will have a tremendous advantage when trade volume increases. In fact, I would advise a newbie with a bot to simply set it to emulate each of @sflicht’s most recent trades, and adjust the program’s parameters each time you make a manual trade. With any influx of newbie forecasters, that bot will crush you in points.
I have more available than a (very small) army of n00bs
Fair point re: available points. My guess is that there are maybe >5 forecasters with over 5k available points, and <3 with more than 10k (maybe substantially more) available. So I grant that this small elite pool has a huge advantage in the contest. I would be more inclined to complain about that than to celebrate it, but I don’t pretend to understand what the organizers’ are trying to accomplish with any of this. I would much rather see the $$ go to you, @sflicht, than the Internet “marketers!” But in the end, I would rather see a greater chunk go to me! I think I’m pretty much guaranteed $225, and pretty much locked out of the $2250. So I will continue to find fault and complain…
I looked at trying to analyze trades to figure out what to “believe” on questions where I didn’t have any expertise.
My limited attempts didn’t pan out very well, and I don’t think there is an easy way of automatically converting what a good user is doing into what you should do. The problem that comes to mind immediately, is that if somebody starts trying to back points out of a question, it takes a decent amount of insight to realize that’s what is going on.
And even keeping the automated trader going takes some effort (or at least mental strain). I’ve got to kind of keep an eye on what it says it is doing, to make sure it isn’t fighting against somebody who knows the outcome of a question. (I rather like questions that won’t resolve until a fixed date, because I don’t have to worry about those.)
This is in danger of becoming unwieldy, so I posted a response to this on the SciCast Users at Google: https://groups.google.com/forum/#!forum/scicasters
Oops, when I used “>” and “3” in the above post to indicate “greater than three,” it somehow turned into a heart. I really wish SciCast could afford a comment system that allows editing. But enough complaining for now.
It’s a lovely heart.
We are updating the blog and making several improvements to it. We should be able to allow for simple comment editing.
Pingback: Overcoming Bias : SciCast Pays HUGE
When I log into SciCast, the banner ad for the contest indicates that only certain questions (those marked with a gold icon) qualify as part of this contest. The rules seem to indicate that the contest is based on total portfolio value increase across the contest duration. Which is it?
Robin Hanson’s post: “Added: At any one time about half the questions will be eligible for this contest. We of course hope to compare accuracy between eligible and ineligible questions.”
Only specified questions. The rules define portfolio later: “The value of a Portfolio is equal to the number of points expected to be gained or lost by an entrant over all his or her forecasts on specified questions during the specified times in the prediction market.” The qualifying questions may change monthly. The active ones will always be marked by the Au icon.
In the “Question List” screen, select Topics -> “Prize Eligible…”. Here’s a link that will directly select the Nov.7 - Dec.6 prize-eligible questions: https://scicast.org/#!/questions/?routeSort=routeSelectedCategories=19
Because the marked questions will be changing monthly (and because the contest ends in months), I worry that we’re not going to be predicting reality, but predicting consensus. My incentive is *not* to predict whether Event X will happen in 2016; my incentive is to predict whether the market thinks in 2015 whether Event X will happen in 2016. And of course you’ll say that markets are efficient and this should be corrected, but I worry about the inefficiencies:
(1) In particular, what stops someone with an account balance of 15,000 betting heavily on the last hour of the month, right before a question switches from Au to not-Au?
(2) When we get paid thousands of dollars for predicting consensus but get paid zero for predicting reality, I have a hard time believing that reality will act as an anchor for the long-term contracts.
Ted, the value of trades will be assessed in March, when over half of the questions will have resolved. Questions were selected from among those expected to resolve soonest. On questions that still haven’t resolved, the market consensus in March will be different from what it was at a switch from Au to not. If someone spends 15,000 on Dec 6 23:59:59, their competitors can still react. We’ll also be selecting a random time in March for the expected value calculations on forecasts on those unresolved questions.
Ah, thanks.
I think i may have already seen some illegal cheating on the question asking how many games Carsen will win in the World Chess Championship. It looks like there was flagrant point dumping by the accounts zhangqiang and challquist on the evening of Nov 9. I hope cheating does not become a big problem with this contest. Some forms would be quite hard to monitor.
Looks like it was all fine. Sorry to have commented.
there are other things to keep an eye on. it looks like recycler (currently number one on the prize leaderboard) might be operating 3 accounts: https://scicast.org/#!/questions/670/comments/power
there was also the whole Chambers/Disinterest point dump; and Chambers is still active. (_that_ was flagrant, as it had two folks moving one or two questions back and forth between 1% and 99% or so. this challquist/zhangqiang thing remained within the realm of plausible.)
the prizes this time are large enough i think it is worth making sure there is a bit of admin attention on these issues.
Thanks for the notes. We will indeed be paying closer attention. The previous Chambers/Disinterest actions would be disqualified this time around.
I think readers need some clarification on how the questions are selected each month. We selected questions for the competition that are most likely resolve before the contest ends. One already locked before most people even knew it had the Au symbol on it.
There are only two sets of questions that we’ve tried to make very similar in their characteristics. Question sets will alternate for prize eligibility each month over four months, so we’ll reveal the next set of questions a few days before December 7. Those questions will be eligible again on Feb 7, and the questions in this first month will be eligible again on January 7. We might or might not add more questions to the sets; we reserve the right to spontaneously add short-term questions (those likely to resolve before mid-March) to the sets as the contest continues.
Since question sets are alternating, then we know exactly when and which questions will turn Au. What’s to stop me from making a crazy 1,000 point bet at 11:59pm and then making the profitable opposite bet at 12:00am? It seems like people can use this discontinuity in reward payout to transform regular points into Au points.
As you note, your timing has to be very good to prevent other people (or bots) from profiting. It’s also risky as those tactics are disallowed by contest rules, and pretty obvious in review. That said, (a) we may randomize or fuzz the crossover boundary to make it harder, and (b) we’re already adding a coefficient in our regression model to account for for precisely that tactic. It’s most likely to evidence the second time a question set appears.
Pingback: Accuracy Contest: First round questions | The Official SciCast Blog
Anecdotally, I would say this experiment already has very interesting results, just a couple of days in! We’re seeing much more aggressive trading in the tails of the distribution. A majority of the prize-eligible questions are already outside the [0.05, 0.95] range, many with a long time until resolution. Since previous blog posts have highlighted significant miscalibration in the tails, this seems like a pretty rational strategy. Of course, pushing the price close to 0 or 1 incurs large liquidity costs. During the general competition, a major incentive for maintaining liquidity is that being short on available incurs large opportunity costs — when a new question is posted there is often a lot of low-risk profit to be had before many forecasters have had a chance to research the topic. In this case, people seem to be trading as if the prize question pool is essentially fixed. I believe it was noted elsewhere that Scicast reserves the right to add new short-term questions to the prize pool, but as of now it’s unclear whether you’ll exercise this right. I worry that this change in the opportunity costs might confound your analysis of the incentive effect of the cash bonus for accuracy. One possibility (although it may be too late to change your experimental design at this stage) would be to release the questions for the December prize pool in tranches, adding new ones at daily or weekly intervals. You can then regress trading behavior against the number of days left in the period to get a proxy for the potentially-confounding liquidity effect I allude to above, and hence assess its magnitude in comparison to the treatment (= in the prize pool) effect itself.
@sflicht: Good comments. While I expect we will keep the December release the same as the November release, we’ll at least think about trading off matched treatment for measuring the liquidity confound.
@sflicht, please keep an eye on this possible effect. Even if the overall design doesn’t change, we will add a controlling variable for the liquidity issue to our model when we analyze the results. The rationale for potentially adding questions is to maintain the number of questions and average anticipated duration on questions over time. These would not be completely new questions, and we’d add the same number to each group of questions.
Pingback: Accuracy Contest: Second round questions | The Official SciCast Blog
Pingback: Active Contest Questions Have Changed | The Official SciCast Blog
Pingback: Accuracy contest: February questions are active! | The Official SciCast Blog
Pingback: Additional Questions Added to Accuracy Contest | The Official SciCast Blog
Pingback: SciCast Accuracy Incentives Contest Has Ended | The Official SciCast Blog
Pingback: Survivor: Future! (And Accuracy Contest Winners Will Be Announced Soon) | The Official SciCast Blog
Pingback: Announcing the Winners of the Accuracy Contest! | The Official SciCast Blog