Machines now forecast short yes-or-no questions about as well as professionals, and still lose to the betting markets

What a forecasting test is

A forecasting question is a yes-or-no question with a date. Will the Federal Reserve raise rates at its October meeting? Will a ceasefire hold through December? A forecaster gives a probability, the date passes, and the answer is known. The forecaster is scored on how close the probability was to what happened.

The usual score is the Brier score. It is zero for a forecaster who says 100% on everything that happens and 0% on everything that does not. A forecaster who says 50% on every question scores 0.25. Lower is better, and anything near 0.25 is a coin flip.

Two organisations run public tests of machines against people. Metaculus is a forecasting website with a paid team of professional forecasters, called Pros. The Forecasting Research Institute (FRI), a non-profit research group, runs ForecastBench, a rolling test that compares machines with a panel of superforecasters, people chosen for a record of accurate forecasts.

The results matter beyond the contest. Insurers and governments are being sold these systems, and a buyer needs to know what a machine's probability is worth. Bridgewater, one of the largest hedge funds, built its own system and published the test.

Five seasons of bots against professionals

Metaculus has run six seasons since July 2024. In each, bots forecast questions that open for one to two hours, so no person can tune a bot between the opening and the close. The Pros forecast about 100 of the same questions with two to three days each. Figure 1 shows how far behind the bots finished in each season that had a comparison.

The Pros won the first four seasons by between 8.9 and 20.03 points on Metaculus's scoring scale. In three of the four the margin was clear beyond statistical doubt. The authors never called it a trend, and wrote each time that the ranges were too wide to say.

In the Spring 2026 season the bot team trailed by 1.25 points over 99 shared questions. That gap is inside the range of chance. The report says the result "should not be confused with a tie": the Pros won 64% of the questions, and the bots' wins were larger when they came.

The human edge moved over the two years. In the first season bots said 29% on average on questions that happened 12% of the time. By the fourth season the problem was telling a yes from a no: the Pros separated the two groups of questions by 44 points, the best bot by 26. By Spring 2026 nothing measurable remained on yes-or-no questions. Questions with several possible answers were the bots' worst type in every season that had them.

The second test says the same, with a warning

ForecastBench asks 500 new questions every other Sunday, half from prediction markets and half from published data series such as unemployment or conflict counts. Its human baselines were collected once, in July 2024, from 39 superforecasters and 500 members of the public. The live board therefore compares machines with people who answered different questions, through a statistical adjustment for difficulty.

In its first paper the superforecasters scored 0.096 and the best machine 0.122 with the crowd forecast shown to it, and 0.136 without. On July 16th, 2026, FRI wrote that several systems had "likely reached parity" with the superforecasters. It listed its caveats: the human numbers are two years old, and professional forecasters now use AI tools themselves.

Bridgewater's system, called AIA, scored 0.1076 against the superforecasters' 0.1110 on 498 ForecastBench questions. The report says the difference is not statistically significant. Its method: ten separate searches and forecasts of each question, with a supervisor that settles disagreements by searching again. A fixed correction then pushes every probability away from 50%.

Ranks on these boards do not last. Lightning Rod, a start-up, titled a post on April 2nd, 2026 "How We Built the #1 AI Forecaster". By October 28th the same system sat second from last on that board, run by the University of Chicago under the name Prophet Arena. A rank without a date and the size of the field says little.

Where the markets stand

A prediction market is a betting exchange where people trade on an outcome, and the price is the crowd's probability. Polymarket and Kalshi are the largest. No machine has yet beaten a busy market's price on the Brier score, and the companies that tried say so in their own papers.

Lightning Rod trained a model on past market questions and tested it on 1,265 new ones. It scored 0.190, and the market price at the same moment scored 0.151, as Figure 2 shows. In a second test of 251 live questions over one month, the model scored 0.199 and the market 0.170. Bridgewater's AIA scored 0.1258 against the market's 0.1106 on more than sixteen hundred busy markets.

Mantic, a London start-up, measured where a machine's edge lies. In a backtest of 817 events between February and August 2026, its forecast was closer to the outcome than Polymarket's price on 84% of markets at the moment they opened. The share fell to 72% once a thousand dollars had been traded and 62% at ten thousand. Past a hundred thousand dollars there was no significant difference. The edge is at the thin start of a market, before the money arrives.

Prophet Arena found a pattern in timing. Machines are ahead of the market a long time before an event, and the market overtakes them as the date nears.

What makes a machine better

The season reports and the company papers agree on the order of what helps, and it is a different order from the sales pitch.

• The model itself. Each new generation of language model forecasts better than the last. A plain one-prompt bot placed near the top of the first Metaculus season and near the bottom two seasons later, as better models arrived.
• What the model reads. Bots that drew on more research sources did better in two seasons. Which search provider they used never mattered in a controlled comparison.
• Averaging. Running the same system five to ten times and taking the average helps, and more than that does not. FutureSearch, a research company, found that averaging five runs cut the spread of its forecasts by two-thirds, and that most of the spread came from the final judgment rather than the research.
• A correction afterwards. Machines sit too close to 50% when they are right. A standard statistical correction improved the bots' scores by 0.016 in a Metaculus study, which warned that it hurts a bot that is already well calibrated.
• Prompts. Changing the wording of the instructions moved scores by hundredths at most. An automated prompt optimizer that looked good in testing did nothing when run live.

Where machines still fail

The same weaknesses appear across the tests.

• Telling a yes from a no. The longest-lived finding: the best bots' forecasts on questions that came true were spread almost evenly, with an average near 50%.
• The ends of the scale. Models rarely say under 5%, and lose most where the crowd was below that.
• Questions with several answers, and questions that need a number rather than a yes or no.
• Reading the news. A September 2026 study of 1,613 question-and-news pairs found small open models moved in the direction the human crowd moved about a third of the time. More news made them worse. No frontier model was tested.
• Combinations. ForecastBench dropped its questions of the form "will both A and B happen" because models did badly on them. Bridgewater lists conditional forecasting as needing significant work.

How the tests themselves go wrong

A model trained on the internet may already know how a past question ended. Telling it to pretend the date is earlier does not work. A January 2026 study found the instruction closed only 48% of the gap between questions a model had seen resolve and ones it had not, and least on politics.

Questions collected after the fact lean toward things that happened. In one widely cited dataset at least 3.8% of the questions had already resolved before they were forecast.

Several of the parity results were reached with the market price in the model's prompt. On ForecastBench's market questions, Bridgewater notes, the best model would have scored as well by copying the price it was shown.

The field has also learned to distrust its own headlines. A 2025 Metaculus post titled "Bots Are Closing The Gap" reported a result that was not statistically significant, and the author conceded the point in the comments.

What this means for anyone buying a forecast

A probability from a machine on a short, well-posed yes-or-no question is now about as good as a professional's, and a busy market's price is better than both. That covers a narrow kind of question.

Nobody has published a scored forecast of scenarios: which several futures are possible, and what would tell them apart in time. Bridgewater lists it as future work. Metaculus's July 2026 review lists it as a product idea. No headline board shows accuracy by how far ahead the forecast was made.

Vektora publishes scenario maps rather than single probabilities, with dated signposts scored in public, and its own tests have found the faults the field reports. Its single-number prices on signposts have been no better than chance so far, and its scenario ranges run narrower than they should. It says so in its method notes.

The next Metaculus season report is the next test of whether the Spring result holds.