Decision support, not a forecast: how a Vektora brief is made and judged
Contents
Vektora makes written assessments, which we call briefs. They are for people who must commit money, capacity or a position before a political or policy outcome is known. A company deciding whether to keep a factory open through an election is one example. An insurer deciding what to charge for cover against a change in the law is another. A forecast would give such a reader one probability and be scored on whether the event happened. A brief does something else, which we call decision support1.
Decision support means this. The brief lays out the outcomes each force in play could lead to, on the scale the question resolves on, such as a vote share or a tariff rate. It names dated public facts, which we call signposts, that would show the real world turning toward one of those outcomes while there is still time to act. It keeps a record of what was known when, so the reader can show a board or a regulator how they reasoned1.
Because the reader's own move is one of the forces in play, no outcome carries a probability. What we check in public is whether the real result fell inside the outcomes the brief laid out, and whether the signposts warned in time and pointed the way things went. We run those checks on replays of events that have already settled and on every public brief, and we show every miss2,3.
The replays say the outcomes hold the result and the signposts beat chance by a clear margin. They also say three things against us. The outcomes are drawn about half as wide as they should be. Our own probability on a yes-or-no question is no better than a coin. Nothing yet shows a brief changed a decision in time. Those are printed here beside the rest3,4.
A brief is made for someone whose choice can change the outcome, so it carries no odds.
Most published forecasting, by people or by machines, stands outside the event and gives one probability. The field's own tests now put the best computer systems level with professional human forecasters on short yes-or-no questions, and behind a prediction market on all of them5. For a leader deciding what to do, that probability is a footnote.
Picture a chief risk officer at an insurer deciding whether to raise prices on a line of business, or a strategy team deciding whether to pull capacity out of a market. Each is one of the actors in the question. Their move is a force. A brief that showed them the worst outcome early enough to avoid it has done its job when that outcome never comes. A forecast scored on that result would call it a miss.
That is why no outcome on a Vektora map carries a probability. The outcomes are given equal weight by stated choice. An earlier version of the method asked a model to weigh each force by how likely it was, and every weight came back close to equal, so the step was dropped. Judging the odds belongs to the reader, who knows what they are about to do1.
The same reason sets a limit on what a replay can prove. A replay has no reader in it, and a past case is someone else's decision. The checks below therefore say whether the outcomes were plausible and whether the signposts fired in time. They do not say the map was accurate, and we do not call them that.
This stance could become an excuse: if the future is not settled until the reader acts, every miss has a story. The checks are therefore run in code against where the real result landed, with no exemption for a reader's move, and the replays, which have no reader, are published with every miss1.
Every term here is defined once, so nothing on the page assumes you have met us before.
| Word | What it means here |
|---|---|
| Brief | A written assessment of one question for one reader, such as "where will the US tariff on heating equipment stand on July 1st?". It is the product Vektora makes. |
| Decision support | Help with a decision, as opposed to a prediction of an event. It shows what could happen and what to watch for, and leaves the judgment of odds to the person deciding. |
| Forecast | A statement of what will happen, usually as a probability, scored afterwards on whether it did. |
| Scale | The measure a question resolves on: a vote share in percent, a policy rate, a tariff rate, or a set of named levels such as low, medium and high. |
| Force | Something that could move the answer: an election result, a court ruling, a supplier's decision, the reader's own move. |
| Outcome | Where the question would land if one force acted, with a range for how far it could run. One outcome is drawn for each force. |
| The map | All of a brief's outcomes set side by side on the scale. Reality is later checked against it. |
| Signpost | A dated public fact, written as one sentence that is true or false on a given day, chosen because its answer shows which outcomes the real world is moving toward. |
| Source | A page on the public web that a figure or quote in a brief is taken from. We store a copy of every source page at the time it was read. |
| Replay | A test of the method on an event that has already settled. The sources are frozen at a date weeks before, the method runs as usual, and its outcomes and signposts are checked against what happened. |
| Settled | Known for certain. A question settles when its official result is published. A signpost settles when a cited source answers it. |
| The record | Every brief made in public. A brief enters it by rule on the day it is made and cannot be taken out, so the record cannot be curated afterwards. |
| The scorecard | The public page that lists every signpost on every public brief, with its outcome once it settles, misses included. |
| Chance | What a coin toss would get. For signposts, the share that would point the right way if their answers were random, worked out from each signpost's own split of the outcomes. |
| Base rate | How often something happens in general. Always guessing the base rate is the simplest possible forecast, and the one a method must beat to mean anything. |
| Prediction market | A website where people bet money on whether an event will happen. Its price is a probability set by the bets, and it is the hardest benchmark a forecaster can face. |
| Interval | The range a measured figure could reasonably fall in, given how few cases it rests on. A result whose interval includes zero may be nothing. |
One outcome for each force that could move the answer, the past cases beside them, and dated signposts between.
Every brief starts from one question with a scale it resolves on: a vote share, a policy rate, a tariff rate, a court's ruling. For each force that could move the answer, the method plays out the world that follows if that force acts and says where on the scale it leaves the question, with a range. Those outcomes, side by side, are the map1.
One map, before and after its question settled
A replay of the US tariff on imported heating and cooling equipment, made as of May 15th, 2026. Each line is one outcome: the dot is its central rate, the line its range. The rate on July 1st, 2026 was 15%.
Source: Vektora, October 27th, 20262
From the worked example on How we test, which also shows how two signposts that settled on June 15th narrowed the map to the outcomes at 15%2.
- The map. Each outcome is drawn from the question's own sources and states where it lands and how wide it runs. On a numeric scale a worst and a best plausible case are drawn beside the map, anchored on the furthest past case the method can verify. That way the scale reaches past what a model would imagine on its own1.
- The outside view. Past cases are searched with no date limit, each checked against its page, and counted beside the map. They are never blended into it: a blend would need a weight nobody can justify. With fewer than 5 verified cases, no base rate is shown at all1.
- The signposts. Dated public facts, each with one sentence that is true if and only if it comes true, a source to check and a date. Each says which outcomes the reader is in if it happens and which if not. Each is set to settle at least a week before the decision is due1.
- The record. Every number a brief prints is tagged as sourced, derived or inferred. A sourced number is checked in code against the stored page it cites, and a brief with a number that traces to nothing is not published. Every version of a brief is kept and dated1.
Honesty here is a list of refusals, each one checkable against the number it hides.
It is easy to call a method honest. It is harder to show it, so this is the list of what a Vektora brief will not print, and where the fact behind each refusal sits.
- No odds on the outcomes. Equal weights by stated choice, for the reason in the first section. The one test of a weighting step hedged every weight back toward equal and was dropped1.
- No headline probability on a signpost. Our own probability on a yes-or-no event, with or without the map, is scored privately. It turned out to be no better than always guessing the base rate. The brief therefore prints how many outcomes a signpost happens in, and no percentage3,1.
- No number that does not trace. A brief with one figure that traces to neither a stored page nor a calculation we show is held back, and the build names the figure1.
- No base rate from thin evidence. Fewer than 5 verified past cases means none is shown. A forecast never counts as a past case, and one case is counted once however many angles describe it1.
- No signpost priced on a sentence the method did not write. A signpost whose test sentence had to be stitched together comes back unpriced, and the brief says so1.
- No miss withheld. Every replay result that went against the method is published with its interval, under a pre-registration where one existed. The sections below carry them3,6.
- No silent correction. A pinned scale, a widened scale, a swapped pole, a scale downgraded to named levels, a corrected reading of the measurement: each is said on the brief's method card with its cause1.
- No number without its noise. A signpost's price on a fresh run of the same question moved 7 points at the median and 29 points for one signpost in ten, and the scorecard prints that beside the prices. A map's movement between versions counts as a move only past a noise line measured on reruns with no news in between1.
Once the question settles, the map and each signpost are checked in code, and width is never counted as success.
"One outcome matched reality" is a test that cannot fail. Spread enough outcomes across a scale and one is always near whatever happens. So a brief is judged on three things about the map and three about each signpost, and a map that only covers by being wide is marked as wide1.
| What is checked | The test |
|---|---|
| The map held the result | The real result fell inside the span of the outcomes' own stated ranges, on the scale the brief was first set on. On a scale of named levels, a result past the end is not held. |
| The signposts narrowed it to the right part | The outcomes whose answers matched every settled signpost held the result. This is the test a single interval cannot take. |
| The map was sharp | The outcomes' centres span at most half the scale. Reported on the brief, never scored: a decision brief does not fail by being wide. |
| A signpost warned in time | It settles at least a week before the decision is due, or before the question's own date when no decision date was given. |
| A signpost splits the outcomes | At least two outcomes fall on each side of it, so its result tells the reader which part of the map they are entering. |
| A signpost pointed the right way | Once the question settles, the real result landed nearer the outcomes the signpost pointed to than the others. |
A signpost's own outcome is recorded only when two independent readings agree and each cites a source. An outcome neither can settle is left out rather than guessed, which is why some signposts stay unsettled. The worked example on How we test walks one brief through the three map tests2,1.
The maps hold the result and the signposts beat chance by a clear margin, on replays with no reader in them.
A replay takes an event that has already settled and freezes the sources at a date weeks before. It checks that none of the computer models in the method already knows the answer, then runs the method as usual. The counts below come from the replays made between October 17th and October 28th, 2026. Every one of those events settled after the date the models' training data ends, so none could have been memorised3.
Signposts that pointed the way reality went, against chance
Of 146 settled signposts across 26 replayed events. Chance is worked out from each signpost's own split of the outcomes and where the result landed.
Source: Vektora, October 28th, 20267
The lift over chance is 0.242 per signpost, with a bootstrap interval of 0.17 to 0.31 over events, and it was positive on 23 of 26 events7.
Maps that held the result
Source: Vektora, October 30th, 20263
Replays with a numeric result and at least six usable outcomes. On the 58 replays dated after the models' training cutoff, 53 held the result3.
The same event run twice gives a similar map and different signposts. On three events each run twice, all six maps held the result, and the narrowing test flipped on two of the three pairs because no signpost sentence repeated between runs8. The map and the pointing rate are therefore the figures to compare across batches, and the narrowing count is noisy. In the latest full week, 26 of 27 maps held the result9.
A forecaster who publishes a range and says it will hold the result nine times in ten, and is right about that, holds the result at that rate by definition. On holding alone, then, a good range and a map are level. The map does what a range cannot: it names the force behind each outcome, and the signposts read those forces. On the first batch of ten events, 21 of 29 settled signposts pointed the right way, a result that chance gives about once in 80 tries10.
The outcomes are drawn too narrow, our own probability is a coin, and a third of written signposts never settle.
These are measured on the same replays, and the method is changed only after a paired replay shows the change helps.
Where the real result fell inside each map
Each replay's result placed on its own map, from the calm end to the severe end, in five bins. A well-sized map would put about the same count in every bin.
Source: Vektora, October 30th, 20263
Over 72 replays, 19 results fell beyond the 5th or 95th percentile of their map, 26% against the 10% a well-sized map would give. The outcomes are about half as wide as they should be, on both sides, by a factor near 2 to 2.53.
- The outcomes are too narrow. The map holds the result because the outcomes' ranges, taken together, are wide, even when each one is narrow. Read as one distribution, the way a forecaster would read it, the map is overconfident on both sides. The brief prints only the first of those facts today. Whether to widen each outcome by a disclosed factor or to print the measurement on the page is an open decision3.
- Our own probability is a coin. Probabilities are scored with a standard rule where lower is better and a perfect record scores zero. Across 372 settled signposts, the probability the method gives a yes-or-no event scores 0.247, against 0.248 for always guessing the base rate: no better. We also priced 198 questions that had a prediction market, on the same day, four weeks before each settled. The market scored 0.070 and we scored 0.162, level with a fixed base rate of 0.165. Our gap of 0.092 per question is clear of zero, which means it is not a fluke of the sample. That is the reason no price is printed on a brief3,6.
- A third of written signposts never settle. In the replays behind the pointing figure, 78 of 233 written signposts were ambiguous or had no source that settled them. If every one of those had pointed wrong, the rate would be at chance. The real rate is somewhere between, and the settled count is printed beside the rate for that reason7.
- Nothing can be said about horizon. 359 of 372 settled signposts give under four weeks of warning before their question settles, because the replays are made two to four weeks out. How the map behaves months ahead has not been measured3.
- One model family. The outcomes, their grouping, the readings that settle a signpost and the no-map price all come from one vendor's models. The price without the map isolates what the map adds. Nothing isolates the model1.
Four claims this page does not make, and the measurement that would earn each.
- That a brief changed what a decision maker did, in time. Every figure above is a measure of the map and the signposts without a reader in them. Earned by logging, on every client brief, what was decided and when, on which version of the map, and which signposts had settled by then. Then count decisions taken before a warning signpost fired against those taken after. Kept private, counted in public.
- That a live brief holds the result at the replay rate. The public record is young, and no live brief has yet been replayed as of its own launch date to show that a replay and a live run agree. Earned by settled public briefs in the dozens, and a replay of each settled one against its live version.
- That the map is the right width. The measurement above says the opposite. Earned by a decision between widening the outcomes and printing the measurement, then a paired replay on a fixed set of settled events.
- That a third party has scored us. Our price could be entered in a public benchmark, labelled as the footnote it is. Earned by entering it and waiting the time the benchmark requires before a rank shows.
Every change carries a version, so one method is never scored as another.
The public record stays on the first version of the method so it can be judged as one method. Later versions are scored apart from it. Each change below was made after a replay showed the fault it fixes1.
Every public brief is scored by rule, none can be removed, and the first signposts settle from November 2026.
A brief enters the public record on the day it is made on our live system, by rule, so the record cannot be curated afterwards. One brief outside our political and policy focus was withdrawn by the owner's decision, and the scorecard says so1.
Made under version 1, with 52 signposts registered and none settled yet11.
Every settled signpost, its outcome and whether it pointed the right way is on the scorecard, misses included, with the measured noise beside any price. Briefs made for a client are scored the same way and never appear there1,11.
- Vektora methodology. Vektora, October 30th, 2026
- How we test. Vektora, October 27th, 2026
- Tier 1 tests: what the stored data says. Vektora, October 30th, 2026
- The case for Vektora's method, argued to survive cross-examination. Vektora, October 31st, 2026
- AI forecasting: the state of the field, October 2026. Vektora, October 30th, 2026
- Vektora's price against the market on 200 questions. Vektora, October 29th, 2026
- Signpost tuning, step 0: pointing against chance on 42 replays. Vektora, October 28th, 2026
- Run-to-run noise, and the first method-5 proof. Vektora, October 19th, 2026
- Weekly backtests: October 28th, 2026. Vektora, October 28th, 2026
- Testing suite: the first batch of ten. Vektora, October 18th, 2026
- The public scorecard (data). Vektora, October 31st, 2026