Q: Expected value? So, if I’m betting that A happens, shouldn’t I just make P(A)=99%?
A: If you knew exactly when we would calculate the expected value, this would be a profitable strategy. But (a) we will randomize, and (b) we have selected questions many questions are still likely to resolve, so there is some risk.
It’s true that if you can maintain P(A)=99% throughout, the payout will be nearly as much as if A happened, but if P(A) is not really that high, your putting it at 99% gives others huge leverage to reduce it to their expected benefit. It will cost them ~1.5 points to reduce it to 98% and cost you 100 points to put it back.
Q: You’re only paying for combo edits? What if I just want to adjust the marginal probability?
A: If you want your adjustment of A to count for the contest, then edit both A assuming B and A assuming “not B” to the same value. Example: if you think A should have a 70% chance, express it by assuming B and setting A to 70% then assuming not-B and setting A to 70%.
With the revised contest rules, all your edits count, so if you want a marginal edit, make it. Just be sure 25% of your edits are conditional edits. (Also, the previous advice was only correct if A and B were not previously correlated. Better advice would have been to make same-cost edits in each condition.)
Q: Isn’t that just more work for the same result?
A: Yes, but our goal is to put combo edits on equal footing with flat edits, and measure the effect on mutual information in the network. That’s still our goal, but as noted, the revised rules let 3/4 of your edits be marginal.
Q: What if there aren’t any assumptions?
A: Add the best ones you can find, or suggest / help create a more useful one! We really want to find all the useful links.
Q: How do we know how well we are doing?
A: We will add a Combo Contest leaderboard by 1-May. If you are eager enough you could calculate it via the datamart.
Users who have LIKED this post:
So an obvious winning move would be to trade a sure question to 1% with a non-conditional trade and then trade it back up to 99% assuming B and assuming not-B. I cannot find any clause in the rules forbidding this (this is surely not “losing behavior”), but since it would make the contest meaningless, I assume that it is not allowed?
A variation of this: it is my understanding that setting up the right question-links will cause a question’s probabilities to be changed as an effect of trades on other questions (shadow-trades). So you could perform a non-eligible trade on one question to setup a eligible trade on another question moving it back to the status quo. If my understanding is correct, you could keep on doing this indefinitely.
Actually, I believe that was possible in the previous accuracy contest, too, but I did not bring it up then, since there was pretty few conditional trades made in that contest.
How are you going to handle this?
This might actually be non-theoretical for me, since I cannot rule out that my bot wouldn’t perform something like this as a consequence of its internal rule-engine. I believe that I have ensured that it wouldn’t keep doing it indefinitely, but it might run through a few iterations of back and forth before it stopped (if the conditions were just right and its beliefs were slightly miscalibrated).
Ouch.
You’re right, these are both viable and we’re going to have to alter the contest rules. That’s embarrassing. We were trying both to increase combo edits and to use actually resolving questions, and that’s just not going to work. Best solution so far is to revert to counting all questions — we’ll announce a revision. Other suggestions welcome.
Thanks for posting. Clearly we should have run the rules past y’all first.
Thanks to @Faber for pointing out flaws in the original contest rules. We’ve had to make two key changes:
* All edits will count
* But since we want to encourage combo edits, to be eligible some proportion of the your edits will have to be combo.
Clarify “some proportion”?
25%. See revised post: http://blog.scicast.org/2015/04/23/announcing-the-combo-edits-contest/
Pingback: Updates to the Combo Edits Contest! | The Official SciCast Blog
Thank you for changing the rules. I hope you are able to get the same quality of data from the new contest.
A minor nitpick: In the official rules you still have the sentence “Remember that in this contest, only conditional edits count, and the conditional edits must have the contest questions as a target” under the heading “PORTFOLIO VALUATION AND SETTLEMENT”. Otherwise the contest rules seems fair and straightforward with even less possibility for gaming than the previous contest.
Thanks for promptly pointing out the flaw. By including all questions and all edits, we eliminate the ratchet mechanisms you pointed out. We’ll fix the remaining nit in the rules.
By including all questions we’ve had to swap the crisp metric of actual resolution for the mushier Expected Value. Expectation can be manipulated in the short term, but the combination of randomized resolution time and a diverse market should greatly reduce any incentive to do so.