We’re thrilled to announce that everyone who participated in the accuracy contest will receive an invitation to be a guest star on Survivor: Future! Can you outwit the other castaways with your prediction power?
April Fools!
Actually, we wanted to let you know there will be a small delay in issuing the final accuracy awards while we formally close the contest questions that resolved by today’s contest deadline. We’ve checked and the actual resolutions still make a big difference to some of the final leaderboard payouts.
We have continued to refine our fraud analysis, and disqualified a few more accounts. If this happened to you and you think it was a mistake, drop us a line and let us know.
In the meantime, we’re inviting you to help us with our research by answering some surveys. You might even win something in exchange.
Thank you for your patience. Stay tuned…
How are you calculating the final scores?
If you are following the procedure from the rules document (portfolio value averaged over a series of random times between March 7th and April 1st), I can’t see how resolving any questions after April 1st would influence the result?
Arguably, the contest should have ended at Midnight between March 31 - April 1. But even conceding “April 1 at 12:00 AM ET” to actually include all of April 1, the sponsors now seem to have greatly exceeded the scope of their rules. I truly hope they don’t plan to resolve all of the flu questions, or wait for Windows 10 to be released, before wrapping it up.
@Faber: It also says we’ll use the resolved value on questions that resolved. Unfortunately it has taken a long time to finalize the values of questions that resolved by 1-April. That’s nearly done (for those where it can be done). We will use the actual resolutions for those questions and the averaged portfolio value on the remaining questions. I’m sorry for the delay — I expected we would be able to do this much faster. However, the contest is about accuracy, and I owe it to everyone — including my sponsor — to reward actual accuracy as much as possible.
The good news is that the Top 15 has been relatively stable throughout subsequent resolutions. However, there continues to be movement around 15th place, and there is a big difference in payout between 15th and 16th.
A lot of “resolved” questions have been in their 48-hour “limbo” period for a long time (like a week). Not a big deal, but a little strange.
@sflicht, thanks for your comment. Right now we have 94 questions that are in provisional or resolved - 93 of these should be finally resolved sometime today (give or take a couple of days depending on user comments).
We have 13 questions we are monitoring until the resolution source publishes a scheduled report, all of which are at least expected to be available within the next couple weeks.
So this means you aren’t going to be awarding prizes for at least another couple weeks?
I have very little confidence that we’ll have resolution sources for 566, 130, 620, 551, 333 within a month. For 1027 we have to wait until Apple’s earnings which won’t be for 1-2 weeks and the question still won’t necessarily resolve. For 551 it is not at all clear the relevant data will be published ever (ITU never published a “World in 2012″ report, for example). Wanting to wait to resolve things is a fine principle, but it can be taken too far.
On March 9, you announced that the “final accuracy leaderboard will be selected at a random time in the next few weeks.” On April 1, this evolved into “a small delay in issuing the final accuracy awards.” @ctwardy explained that the common understanding that “portfolio value averaged over a series of random times between March 7th and April 1st” would be supplemented with post-April 1 resolutions to resolvable questions. And now you indicate that there will be a very long wait because you want to resolve questions that are not yet resolvable? This seems rather outrageous. Why didn’t you simply clarify these intentions in the first place, rather than setting and encouraging other expectations?
@himmel.ca, @sflicht, @Market Snodsbury: Of the still-pending-at-1-April questions, payout depends only on the ones that were resolved as of today (the 92 of 93 expected to do so). We’re recalculating the leaderboard now. Final scores and payouts should be announced this week or early next week.
We’re monitoring the others (of course) but not holding payout for them, because as you have all observed, that could take an indeterminate amount of time. I apologize — the email I sent Jessie conflated two items.
Thanks for the clarification, Charles. BTW, I don’t begrudge you wanting to bend over backwards to please your sponsors — it’s free money after all! I think we all appreciate that the contest was run relatively smoothly and fairly.
I am still not sure that I understand how the final leaderboard will be calculated. There are some interactions between random change-over times, resolved/unresolved questions and conditional trades that are not clear.
Here is my best guess. Can you confirm if it is right or let me know where it is wrong?
A series of random times in the period 2015-03-08T00:00.000-5 to 2015-04-01T00:00.000-5 will be generated (how many? will this be the same times for all participants?). The score will be sampled at each of these times and averaged for the final result.
For each sample a random cutoff-time will be selected between each of 2014-12-07T00:00.000-5 and 2014-12-08T00:00.000-5, 2015-01-07T00:00.000-5 and 2015-01-08T00:00.000-5, 2015-02-07T00:00.000-5 and 2015-02-08T00:00.000-5 and 2015-03-07T00:00.000-5 and 2015-01-08T00:00.000-5 (again: will these be the same for all participants?).
When calculating the sample-score we consider all trades made after 2014-11-07T00:00.000-5 (the start was supposedly not random). The previously mentioned cutoff-times will be used to decide which trades are eligible, depending on which category they were placed in.
If the trade is non-conditional and the question has been resolved at the time of the calculation, we use the resolution as the probabilities in the log-score rule.
If the trade is non-conditional and the question has not been resolved, we use the probabilities at the selected sample-time.
If the trade is conditional, and both questions have been resolved, we use both resolutions.
If the trade is conditional, and neither questions have been resolved, we use both probabilities at the sample-time.
If the trade is conditional, and one question has been resolved, but the other has not, we use the resolution and the probability at the sample-time together (does this give a meaningful value? how does this interact with shadow trades?)
Will you publish enough data to allow us to calculate the final scores ourselves?
@Faber: Basically yes. I’ve written a page-long algorithm and am sending it around to have those responsible double-check before posting. We will post it with the announcements if not before.
I’ve approved the final winners list. That and the detailed calculation procedure will be announced on the blog this week. The site leaderboard has been updated to reflect the final calculations, averaging three leaderboards but accounting for resolved questions.
Now that the leaderboard is finalized, what is the procedure moving forward for awarding prizes? When will prizes be distributed?