Because SciCast strives for continual improvement and also needs even more participants and forecasts than in previous years, we have been exploring the effectiveness of incentives, particularly monetary incentives, for increasing the quality of participation in a prediction market. This post is the first of a five-part series to summarize the first two incentives studies as we start a third. (Parts of this series of posts are based on previous technical reports unavailable to the public.) This first post lays out the goals, hypotheses, and background of the incentives studies. If you’ve been participating in SciCast for a while, you might better understand some of your own experiences after reading this.
Background & Motivation
We think that monetary incentives increase participation but are unsure of whether they increase forecasting accuracy. A possible risk of these studies is that participation incentives paradoxically discourage participation, particularly after they are withdrawn. We’ve directly assessed this risk by contrasting incentive systems with a lack of any incentive system in an ABAB experimental design.
The greater risk is that extrinsic rewards could be counterproductive for SciCast’s research agenda and goal to predict the future of science and technology. The value of extrinsic rewards is a topic of debate among researchers. In Daniel Pink’s 2009 review of decades of research on human motivation, he claims that the best approach to improving work performance is not to provide external rewards like money but internal rewards like a sense of purpose. Adding some types of external rewards to an inherently motivating task, which we hope that forecasting with us is, can reduce time spent on task and quality of performance.
Pink provides examples of successful businesses built on competition without monetary rewards. In particular, he points to the proliferation of online games with play money, points, and other rewards that have little to no value outside the games. These rewards serve other functions, such as prompting people to set personal goals and helping them to track their progress.
External rewards are not always detrimental to performance, even when they are detrimental to intrinsic motivation. Pink acknowledges that his advice is most pertinent to intellectual tasks and less pertinent to others. Other authors (Ederer and Manso 2013, McGraw 1978) have shown that pay for performance typically effects repetition of previously successful work strategies but hinders attempts at new strategies, suggesting that less innovative types of work may benefit from monetary incentives.
Because we believe the accuracy of forecasts requires creative and deliberative thought, external rewards tied directly to a person’s accuracy could backfire. However, forecasting is a prerequisite to accurate forecasting and should be encouraged until the SciCast prediction market is mature. Because making forecasts – accurate or inaccurate – does not always require creative and deliberative thought, external rewards for the number of forecasts should result in an increase in forecasting, ultimately leading to greater accuracy. Our first goal has been to increase the frequency of valuable activities of forecasters, especially forecasting. Our second goal has been to increase the accuracy of forecasts.
Goals of the first two experiments
We developed a multi-incentive approach to increasing the number of forecasts in a prediction market, and such an approach is not new. Professionals who have developed prediction markets often advocate a mixture of market features to motivate participation. For example, Émile Servan-Schreiber, the founder of Newsfutures, one of the earliest providers of prediction markets to private companies, emphasized “rewards, recognition, and relevance” as the drivers of participation. Rewards in this case mean external incentives with easily quantified value, such as cash, a paid holiday, and material prizes. Recognition refers to incentives that acknowledge the quality of forecasts but do not offer external value. Relevance refers to how participation relates to a participant’s career, such as teaching new applicable skills and providing timely insights on product development. Our incentives fall mainly into the categories of “rewards” and “recognition.”
We were trying to discount the simple hypothesis that incentives have no effect and find evidence that incentives increase activity in a prediction market. We had no clear hypotheses on which types of incentives or incentive systems increase activity more than others. The research literature showed mixed results for various forms of incentives. We hoped to see incentives increase forecasting accuracy, but stuck with the hypothesis that incentives have no effect on accuracy.
To be continued…
Watch for Parts 2 through 5 of this series to come. Part 2 will explain most of the methodology of the first two studies.