almanis is scientifically validated
The culmination of this work is a superior human-machine hybrid forecasting mechanism which facilitates earlier discovery of consensus-changing information.
almanis is designed to absorb prevailing LLM intelligence seamlessly into its hybrid forecasts, in real-time.
-
Background: The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for likely accuracy in real-time with machine learning.
Methods: We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n = 1822) and Next Generation Social Science (NGS2) platform (n = 103) were utilised.
Findings: A 43-feature model predicted accurate forecasters, those with top quintile relative Brier accuracy, with subsequent replication in two out-of-sample datasets (pboth <1 × 10-9). Trades graded by this model as having higher accuracy scores than others produced a greater AUC temporal gain in the overall market after vs before trade. Accuracy score-weighted forecasts had higher accuracy than market forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial % AUC gains of 13.2%, p = 1.35 × 10-14 and 13.8%, p = 0.003 in two out-of-sample datasets. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for market models, p = 0.007. This net classification benefit was replicated in the separate Almanis B dataset, p = 2.4 × 10-7.
Interpretation: Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. This could provide improved anticipation of and thus response to emerging risks.
Funding: This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources, Australia, to SlowVoice Pty Ltd. (IR 2101990) and Fellowship (GNT 1110200) and Investigator grant (GNT 1197234) to A-L Ponsonby by the National Health and Medical Research Council of Australia.
-
Prediction markets are a successful information aggregation structure, however the exact mechanism by which private information is incorporated into the price remains poorly understood. We introduce a novel method based on the “Kyle model” to identify traders who contribute valuable information to the market price. Applied to a large field prediction market dataset, we identify traders whose trades have positive informational price impact. In contrast to others, these traders realize profit (on average) in excess of a theoretical expected informed lower bound. Results are replicated on other field prediction market datasets, providing strong evidence in favor of the Kyle model.