As the number of open questions on SciCast increases, some users are finding their points stretched too thin. We want our users to make forecasts when they have the interest and knowledge. To make this possible, we’re giving every registered user an extra 4000 points, and starting out all new users with 5000 points.
This change will not immediately affect any person’s rank on the leaderboard, though it might offer new opportunities to move up or down in the rankings. If you have run out of points or are extremely short on points, you now have the assets you need to make more forecasts. Please use these new assets and opportunities wisely. The overall accuracy of SciCast’s market forecasts is improving, and we all want to see a steady continuation of that trend.
Did you like this post? Follow us on Twitter.
NOTE: More points are exciting, but be careful. If you cannot answer the following questions, you should probably stay in Safe Mode:
Safe Mode makes conservative edits in the direction of your declared belief, and should keep you from running out of points. If you run out of points, your opinions have negligible effect. Power Mode gives you the power to make big changes to the current estimate, but can use up your points in a hurry. If you use it, be conservative. Either way, revisit your questions to update your estimate, or simply to re-assert it.
To learn more, please review one or more of these:
With 4K points I think Safe Mode can potentially be more dangerous than Power, because 3% of 4000 is a full 120 points which moves the probability by a factor of two.
On another note, I wonder how well thought-through the decision to hand out points was. Maybe a much better option would have been to provide everyone with a nice interface allowing them to see all their assets at once and see where they could free up some points. And if someone doesn’t have assets left to free up then, this being a market, shouldn’t participants be allowed to go broke?
@dvasya: Thanks for the catch. We had meant to reduce the percentage used in the Safe Mode heuristic. That has now been set to 1%, which will give new (5000-point) forecasters 50 points per edit, at first. After about 160 independent edits they will be down to 1000 points, and consequently using less than 10 points per edit. After about 230 edits, they will use fewer than 5 points per edit. (Backing off previous edits will reclaim points, as will resolutions other than worst-case.)
@dvasya: On your second point: 5000 points was always our desired target (see initial promotional materials) because of the anticipated question load (5-10 times that of DAGGRE), but we had planned to divide it into an initial tranche, incentives for completing the user profile, and weekly manna. But a recent program review by our sponsor made increasing activity the #1 priority, an experiment is about to start preventing any such changes for 8-10 weeks, so we decided to bite the bullet and do it all now, accepting the short-term instability in favor of increased activity.
Better UX for point management: yes, definitely needed, but that costs a lot more in time and money.
Going broke: yes and no. In a points market, providing a regular infusion of points (e.g. ‘manna’) improves liquidity and allows people to learn and recover from mistakes, rather than just leaving (or proliferating new accounts).
Is this exogenous “gift” of extra points going to be a pain in the butt for analyzing the scicast accuracy results?
I suppose if people were perfectly rational Kelly rule bettors it would make no difference. Now it seems like your studies will be confounded by “liquidity” effects that make comparing the pre-gift and post-gift datapoints difficult. (I’m thinking of the econ results of Mullainathan et al about predictable irrationality under conditions of scarcity.) Of course, it’s interesting fodder for publishing behavioral econ results! But OTOH the problem of the interpretability of the pred market data remains…
@sflicht: We expect short-term instability from the points infusion while everyone adjusts to new effect size and total influence. We’re trading that for anticipated greater activity: a program review made it clear we need to focus on activity. Only 33 questions have resolved with enough activity to analyze, and they are doing only moderately well: better than uniform probability, but not the gain we saw in DAGGRE. One possibility is low activity, and given other constraints, we opted for a single exogenous shock, partly because it gets it over with ahead of controlled experiments.
To match the extra points, Safe Mode has been changed to use a smaller fraction of your points on a single edit. Instead of using max 4% of your points, it will use max 1%.
Pingback: Update to Extra Points Given | The Official SciCast Blog