The second round of questions has been selected for the new accuracy contest. Forecasts on these questions from December 7, 2014, through January 6, 2015, have their market scores calculated and added to a person’s “portfolio.” The best portfolios at a time shortly after March 7, 2015, will win big prizes.
Here is the list of questions’ identification numbers:
43, 46, 47, 84, 102, 130, 139, 145, 172, 183, 188, 189, 190, 192, 194, 199, 205, 207, 209, 212, 213, 217, 220, 226, 254, 255, 257, 307, 334, 337, 341, 342, 343, 349, 352, 359, 360, 361, 370, 372, 375, 382, 393, 399, 404, 405, 410, 413, 417, 422, 426, 445, 459, 465, 466, 469, 511, 530, 533, 551, 552, 554, 559, 560, 562, 566, 580, 582, 584, 587, 616, 623, 624, 628, 632, 640, 649, 656, 659, 664, 666, 673, 681, 703, 705, 708, 717, 718, 723, 727, 732, 733, 748, 761, 776, 787, 789, 792, 794, 802, 803, 817, 835, 837, 838, 845, 846, 848, 857, 859, 860, 861, 862, 865, 866, 867, 877, 878, 880, 894, 896, 898, 899, 901, 906, 907, 908, 917, 924, 925, 928, 932, 934, 937, 939, 947, 948, 949, 951, 952, 953, 954, 956, 967, 968, 969, 971, 972, 973, 974, 975, 982, 983, 984, 986, 1008, 1009, 1012, 1013, 1015, 1025, 1027, 1029, 1031, 1032, 1036, 1037, 1038, 1043
You can reach a particular question by typing “https://scicast.org/#!/questions/<NUMBER>/trades” into your browser while replacing <NUMBER> with the identification number for the question that you care about.
To see all the questions in one place on SciCast.org, filter the questions. You can most easily find the prize questions for the first round by logging in and clicking on “QUESTIONS.” On the left side under “TOPICS” check “Prize-Eligible Dec & Feb.“
Today is 12/7 but these questions don’t have Au symbols next to them. Are they live yet? I worry my trades aren’t being counted.
It would be nice if we could get a definitive answer on the exact time that the switch-over between question sets occur. My best guess is that the switch-over occurred at 2014-12-07T12:00:00-05 / 2014-12-07T17:00:00Z. Is this correct? Will the next switches also occur at noon EST?
Also note that the Nov/Jan questions are still marked with the gold symbol.
Now on 12/8 the questions do have Au symbols. Will my trades yesterday be counted? I hope so.
(I was trading under the assumption that they were live, since the blog post said they’d be live starting on Dec 7 and the fine print from the contest was using midnight Eastern time as the cutoffs for the beginning and end of the contest. If they were not actually live when they were supposed to be live, then my trades may damage your data on the effects of incentives by being counted in the wrong category.)
My concern is exactly the opposite of Ted’s. I would be incredibly frustrated if you were to reverse your earlier statements and arbitrarily decide to retroactively count forecasts that were not made under the “AU” symbol. I wasted a huge amount of my Sunday continually checking SciCast to see whether the questions were “turned on.” I was disappointed that they never were, as I missed the boat today (cursed redeye travel!) and now see that the good “easy” positions for the new question set have already been gobbled up. To retroactively give points to people that made forecasts yesterday, when I consciously restrained myself from doing so based on your guidance, would be to rub salt in the wounds. Failing to activate the questions before midnight was bad enough; please don’t make it worse by changing your rules midstream.
Do the Au symbols mean the questions are currently being rewarded? The Nov/Jan questions have Au symbols next to them but do not now count for the contest, right? I am not sure how to interpret the Au symbols.
I see both of our concerns, but to me the issue is less clear cut. The text said the period started on Dec 7. The symbols showed the period starting on Dec 8. But then again, the symbols also show the Nov contest continuing to run, so it’s not clear what the symbols mean or whether they a trustworthy indicator of the contest being ‘on.’ I understand your concern that your time on Sunday will have been wasted if my trades count, but my time will have also been wasted if my trades end up not counting. Either way, the ambiguity of the contest will hurt someone.
I don’t know how SciCast could resolve this fairly without going with the symbol switch time. They would otherwise have to retroactively pick a Dec 7 time and argue that it was a “super-secret” switch, which doesn’t cut it legally for a contest like this. When money is at stake, it’s important for them to get this stuff right. Must have been another technical snafu I suppose.
I agree that the use of the AU symbol is confusing; it is incredibly frustrating when I edit a question from the “Recent Activity” flow and am unsure whether it is a “November” or “December” question. They really need some indicator beyond searching and filtering.
@ctwardy explicitly stated in multiple places that the “switch” would occur at some random, unannounced time on Dec. 7. So I waited and waited and waited. It must have finally happened sometime after midnight and before 6 AM or so.
Hmm, where did @ctwardy say that? (Honestly curious, I don’t doubt you.)
In general, I think it’s bad policy to expect contest participants to keep up to date on all the comment threads. I wish these things were announced in the blog posts.
@ted, I didn’t recall the randomized changeover either, but Charles reminded me about it in this thread: https://groups.google.com/forum/#!topic/scicasters/4WzeusDcOiI.
Thanks for the link Sam.
Going forward I hope the blog posts are clearer about when the transition takes place and how to tell if a question is active or not. I guess I sabotaged myself a few hundred points by making the questions where I had an edge more accurate and leaving alone the questions where I had no edge.
Anyway, for the SciCast researchers trying to see the effects of incentives, please manually include my Dec 7 edits as ones made under the (perceived) presence of incentives. I don’t want to mess up your data.
(Speaking of incentives, with such a flat payout structure (the only slope being at x=15 and x=150), people like me who have resigned themselves to finish outside of the top 15 but feel no threat of dropping below 150 will generally be very unaffected by the incentives.)
@ted — LOL. It’s not even clear to me that the strategy you intended to follow is utility maximizing. In fact, here’s a game theory question: if *everyone* withholds their edge until the change-over, is that strategy vector a nash equilibrium?
(A key point here is that the translation from in-game utility [points] to real world utility [$] is only sensitive to your *rank* on the prize leaderboard not your actual score there. Although your real world utility function also has a time value term, and if your prize leaderboard score were to get sufficiently high you would be assured of the maximum payout and thus you could subsequently forego further forecasting. But my utility function also assigns positive value to forecast, so I’d forecast anyway. Plus everyone’s utility function assigns positive value to bragging rights. And beating bots.)
@ted. Regarding incentives, I suspect it’s a matter of baby steps with the funding agencies. I’m pretty confident that one day we will see a genuine, real-money version of Robin’s terrorism futures markets go live, and also a (legal for US traders) version of Scott’s NGDP futures. If there were a scaled continuous question about it, I’d be long at 2020 but short at 2050.
In truth, I didn’t have a plan (utility-maximizing or otherwise). I just showed up on the site, noticed it was Dec 7, and then spent more time than usual making edits because I thought a new wave of questions had hit. This was not a premeditated strategy on my part, but I would have been disappointed nonetheless to discover that (a) my time was wasted or (b) my time was counterproductive and even worse than wasted. That’s where I was coming from.
@ted, my impression is that your concern that particular forecasts will be misinterpreted as “incenitivized” or “not-incentivized” is misplaced. It appears that both of our expectations regarding “switch days” were a bit misguided. Read the full discussion that @sflicht linked above.
The way it works: on the 6th of the month, one question set is incentivized. On the 8th, a different set is incentivized. On the 7th, we do not know which of the sets is incentivized at any particular point in time. The day begins with Set A, at some point it changes to Set B, but the time of the shift is not defined or indicated. It is a mystery to discourage any of the sort of grubby strategizing you and I use to “maximize advantage,” which @ctwardy indicates that he considers “gaming.” In short, we are supposed to forecast to the best of our ability without regard to the very incentives we thought were a subject of study.
Perhaps the 7th of the month is actually intended to be a study of our behavior under conditions of extreme uncertainty? In any case, don’t expect clarity or clear rules at the margins. The vagueness appears to be a feature, not a bug, and too many questions are annoyances that indicate we are just in it for unseemly gain (which is simultaneously the point and a violation of an odd concern about “gaming”). I understood the reasons for not pre-announcing the time of the shift, but this really makes no sense to me.
In any case, depending on the time where they place the December 7 “line of mystery,” you seem to stand a good chance of cracking the Top 15. To tell the truth, had I understood these secret rules yesterday, I still don’t know exactly how I would have forecast. I certainly wouldn’t have foolishly waited for the irrelevant icons to change.
Ah, seems like we both misinterpreted the rules then and both felt some unnecessary anxiety. Hopefully going forward the rules are clearer.
Now that I know about it, I do like the idea of a random, undisclosed changeover time. It seems less susceptible to grubbing than the public changeovers that I can think of.
Maybe I missed it, but is there a reason the Nov & Jan questions still have the “Au” symbol? Bets placed now on these questions do not count towards the contest scoring, correct?
@Phoenix is correct. Admin said in the post on the list of Nov questions and also in a comment somewhere else that the time for the switch on Nov 7 could be random. The motivation was that people ought to be able to know the questions in advance to research them but not to trade in a way that takes advantage of the exact moment of the switch.
It appears that users “frenchlumber” and “samsa” are engaged in irregular betting behavior. For questions 422, 628, 624, 183, and possible others, “frenchlumber” has moved the forecast from near certainty in one direction to near certainty in the the other followed immediately by “samsa” moving the forecast back. I would appreciate if administrators would look into this more closely.