An experiment on prediction markets in science

Abstract

Prediction markets are powerful forecasting tools. They have the potential to aggregate private information, to generate and disseminate a consensus among the market participants, and to provide incentives for information acquisition. These market functionalities can be very valuable for scientific research. Here, we report an experiment that examines the compatibility of prediction markets with the current practice of scientific publication. We investigated three settings. In the first setting, different pieces of information were disclosed to the public during the experiment. In the second setting, participants received private information. In the third setting, each piece of information was private at first, but was subsequently disclosed to the public. An automated, subsidizing market maker provided additional incentives for trading and mitigated liquidity problems. We find that the third setting combines the advantages of the first and second settings. Market performance was as good as in the setting with public information, and better than in the setting with private information. In contrast to the first setting, participants could benefit from information advantages. Thus the publication of information does not detract from the functionality of prediction markets. We conclude that for integrating prediction markets into the practice of scientific research it is of advantage to use subsidizing market makers, and to keep markets aligned with current publication practice.

Figure 1. Experimental Setup.

(A) In each market, participants traded claims representing six mutually exclusive hypotheses in seven consecutive trading rounds. At the outset of a trading round, information about the hypotheses in the form an error-prone test result, was disclosed either to all participants (Setting 1), or to an individual participant drawn at random (Settings 2 and 3). In Setting 3, the information was first private but then disclosed to the public. Each trading round took approximately three minutes. Trading was suspended for about two minutes between the trading rounds when novel information was distributed, and in the middle of each trading round in Setting 3, when private information was made public. After seven such trading rounds, the outcome was judged and the traders' accounts were settled. An entire session, including instructions and training on the market interface, took approximately 1 hr. (B) We used six different, randomly generated information histories (A–F), each of which differed in the tests and results given to the participants. The information history used in the market shown in Fig 2 is, for example, given by: OZ false, VO false, VZ false, OV true, ZV false, ZO true, VO true. Other information histories are given in the Methods. Each of the six information histories was used once in each of the three settings, giving in total 18 markets. Each trader participated in only one market, which means that the markets were not affected by differential learning effects.

Figure 2. Example Market.

(A)–(F) Prices of the six contracts over the course of one experiment session in Setting 1. Black dots indicate the trades in a share. Black lines show the theoretical market maker prices for infinitely small trades. The green line shows the correct pricing. Between the round (orange bars), trading is suspended, and new information is disclosed to the participants. (G) Mispricing. To quantify mispricing, we calculate the Kullback-Leibler divergence between the probability distribution implied by the market prices and the correct Bayesian posterior. Mispricing is typically generated when new information is released, and is subsequently reduced by the traders (see for example trading round 1 and 2). As it becomes more and more likely that the first hypothesis (ZOV) is the correct one, the traders in this session start overshooting in their estimate for the probability of this first hypothesis. This generates increasing mispricing in rounds 3–5. In round 6, the disclosed information reduces mispricing. This means that in this round, reality (i.e. correct prices) catches up with the market forecast. Prices at the end of the last round give a good estimate of the correct probabilities. (H) Net wealth of the seven traders in the market. The participants start with a cash position of 100,000, equivalent to USD 10. Over the course of the experiment they invest into the shares and thereby change the pricing of the shares and their own net wealth. (I) Final payoffs. After the last round, the hypotheses are judged: for each share of the correct hypotheses, participants receive a payoff of 100. This payoff is added to their cash position and determines the payoff. In this session all participants end up with a net gain. This illustrates the non-zero sum nature of trading due to the subsidizing market maker.

Figure 3. Mispricing after the final trading round.

Market efficiency, measured as the mispricing at the end of the final trading round, was considerably higher in the setting with public information (Setting 1) than in the setting with private information (Setting 2). The setting with private information that was later made public (Setting 3) performed similar to the market with public information only (Setting 1). In the setting with private information, prices in three markets gave better forecasts than average trader belief. In the three remaining markets, average trader belief would have given better forecasts. Mispricing is shown as Kullback-Leibler distance. Predictions in well-functioning markets typically fall within a 5–10% margin around the correct probabilities. In the non-functioning markets (A2, C2, E2) prices differed from the correct probabilities by as much as 50–80%. In other words, predictions in these markets were far away from the correct probabilities and gave, for example, probabilities of 10% instead of 90% for the correct hypothesis.