The SciCast Accuracy Incentives contest has come to an end. The final accuracy leaderboard will be selected at a random time in the next few weeks. This will allow more questions to resolve as well as randomize the selection time to reduce the impact of attempts to inflate scores on unresolved contest questions.
Read more about the contest.
seemed like there were a number of questions resolving at the end of month included, so hopefully you’ll wait until early april.
The official contest rules say the time will be selected randomly between March 6 and April 1. So, unless they are doing something fishy, there is a >50% chance everything will be over by March 20.
As someone in the money, I hope it ends sooner rather than later. It’s a hassle to be constantly logging in to make sure no one has attacked my longshot questions and damaged my score by a thousand points. I know that attacks are supposed to not work because the market has an incentive to correct them, but when getting 100:1 or 50:1 odds, it takes a huge amount of market liquidity to repel an attacker on highly certain questions. I suppose if someone really did make a bad faith attack on prices, their trades would be reversed by the admins, but I’m not very confident in that.
I am reasonably confident that they would respond reasonably to manipulation. Overall I’ve been impressed by the degree of honesty in the contest. (I still feel bad about impugning Faber’s motives while he was debugging his bot. :-P)
I agree that they would have good intentions and be able to fix things in a clear cut case. However, I can imagine a huge number of ‘blurry’ cases where manipulation intent was impossible to prove (maybe another user was competitively attacking my long position or maybe they just have a confident opinion that’s legitimately different than mine, in which case no foul). And with these blurry cases where intent cannot be proven, I think they’d have to stick to a policy of ‘innocent until proven guilty.’ And in such a scenario, I might end up out $2,250. (I admit this is unlikely and probably won’t happen. But I am an anxious loss-averse irrational human, as much as I try to suppress it.)
Looking forward to reading the analysis of the results.
Any sense of the likelihood of further incentives studies?
I suggest the following: give each user a new category of points (I’ll call them Au-points) and introduce a new set of Au-questions that can only be traded with Au-points. Allow users to convert points to Au-points using a nonlinear conversion function (to alleviate the advantage to users with many points, and to minimize the incentives to spam new accounts), and make Au-points directly redeemable for Amazon money. For example, using a marginal exchange rate of 1000 * x for the x’th point converted, N points translates into roughly 1000 ln(N) Au-points. So a newb’s 5K points yields ~8500 Au-points, while @dvasya’s 50K yields ~10800 if he converts them all. If Au-points are redeemable for Amazon gift cards at a rate of ~ 1000 Au-points / USD, this would be about $8.50 subsidy per new user, so assuming an optimistic 2K new users from ad campaigns, this is still an affordable $17K from the grant. If all 10K current users were to convert all their points, then (neglecting any LMSR gains and losses from existing questions) if we assume the false but simplifying fact that each user has 5K points, that would be another $85K, which is probably not affordable. (You’re giving away 15 * $2250 + 135 * $225 =~ $64K for the contest that just ended, so I assume that’s a practical upper bound for the funding commitment.) However, (a) it seems very unlikely that all 10K users would cash out, (b) even the top of the leaderboard doesn’t necessarily have 50K available to convert, and (c) it would be straightforward to impose minimum-number-of-trade requirements, or otherwise restrict Amazon $ withdrawals. (For example, only allow cashing Au-points to $ in multiples of $5. Then a new user who doesn’t make any bets on Au-questions can only walk away with $5, and I think the expected cost of the incentive program would be close to the $64K for the current contest.)
I think a scheme like this would leverage the existing points economy, not unduly favor scammers or those on top of the leaderboard, and give people real skin in the game. The main difference from the previous contest would be, if anything, *less* advantage for top forecasters. It seems to take good Scicasters around a year to 10x their points, so assuming similar performance on Au-questions, it would be quite challenging for @dvasya to walk away with as much as $1K in Amazon money over just a few months, starting from the $10.80 baseline. This might make the incentives too weak.
Alternately you could try a reward system based only on points akin to Hypermind’s:
Score(user) = Sum_(contest questions) sqrt(profit on question) - sqrt(loss on question)
Reward(user_i) = ($TotalReward) * (1 if Score(user_i) > 0 else 0) * Score(user_i) / Sum_(user_j s.t. Score(user_j) > 0) Score(user_j).
A final suggestion: Robin tweeted mysteriously about having a meal with Elon Musk at GMU and later something that suggested to me he got invited to a Falcon 9 launch. Maybe he could convince Elon to subsidize some of the SpaceX- and Tesla-related questions, just as Scott Sumner got Gabe Newell from Valve to subsidize his NGDP market on Hypermind?
Users who have LIKED this comment:
I like the idea of an incentive scheme with a linear relationship between profits and the Amazon incentive. However, I do worry that if every new user got ~$10 free, this would be posted to the front page of SlickDeals or something and a million users would sign up to claim their free $10.
Users who have LIKED this comment:
I started thinking about this possibility and wondered if the bad outcome of accounts just for claiming $10 could be mitigated by some sort of ‘must use account X times’ or some other more constructive measurement.
This reminded me of an XKCD comic that referenced evolving AIs to beat captchas and comment filters by making teaching them to make substantive comments. So SciCast offers this sort of thing ‘tricking’ people into learning to become more agile in their thinking and better able to predict future outcomes. Win-win.
Users who have LIKED this comment:
Still analyzing… The preliminary result suggests the Accuracy Contest doubled accuracy: Brier scores seem to have improved from ~0.3 to ~0.15 when questions were active. We haven’t controlled for other factors yet, but so far it looks impressive.
Your incentive scheme is worth some attention — though we’d have to work around @Ted’s objection. The next incentives are likely to be simple ones for survey completion tie up some research loose ends. Accuracy incentives seem to work, but we’re nearing the end of our (4 years!) of IARPA funding, so we need to be strategic: our best prospect for transitioning a public market may be in cybersecurity — we might spend the majority of our incentives budget on a dedicated cybersecurity site. I’m also thinking about crowdfunding and question bounties.
RE: cyber, I can certainly see why this is a sexy topic and good to focus on from a funding point of view. But FWIW the cybersecurity questions on Scicast don’t strike me as particularly successful or informative. The interesting questions in this domain might be closer to the geopolitics arena (attribution, role of non-state actors, political policy regarding encryption backdoors, etc.) rather than the technical arena. Specifically, technical questions on computer security require a high degree of domain specific knowledge to forecast well, and accurate data sources for resolution seem hard to come by.
Is there a public program assessment for IARPA’s ACE programs? Any sense of whether these are likely to be renewed?
Users who have LIKED this comment:
The ACE program ends this summer — two teams had their full 4 years. IARPA’s mission is to fund high-risk research programs; although it works closely with potential transition partners and helps to make connections, it’s mission does not cover operational use.
GJP has been very successful and will likely transition in some way if they have not already done so. We are working on the possible application to cybersecurity and a possible government transition.
I agree with your assessment of most SciCast cybersecurity questions — historically most were interesting for forecasting research, but not useful for decision-making. We now have a dozen weightier questions from Dan Geer, but most haven’t attracted much interest yet perhaps in part because SciCast is not full of cybersecurity experts — arguably a dedicated cybersecurity market would fare better there. There are many policy questions we cannot ask while funded by IARPA. But there are also a number of operational questions that practicing CISOs could help write — we’re working on that.