Findings
Three questions, written down before any model was built. Every number on this page comes straight from the pipeline's results file.
Can a cheap model read central bank tone?
- Language model vs people
- 65.3%
- deepseek-v4.1-flash, zero-shot · weighted F1 0.65
- Local model vs people
- 59.3%
- bge-small + logistic regression · weighted F1 0.60
- Two models agree, outside the Fed
- 88.7%
- 150 sentences, 20 banks · Cohen's κ 0.82
- Local model matches both
- 74.4%
- on the 133 sentences where they agree
No people have labeled other central banks' speeches, so outside the Fed a second model from a different family (glm-5.3) labeled the same sentences. That shows the labels are consistent across models, not that they are right.
Does tone move with policy?
Tone against the coming rate change, by central bank
Rank correlation over 5,481 bank-months. Above zero: a more hawkish tone came before rate rises.
Tone before each kind of decision
Mean tone over the three months before, with 95% intervals.
Leads, or follows?
Rank correlation of tone with each, across all bank-months.
- Rate change in the past three months
- +0.23
- Latest inflation
- +0.25
- Rate change in the next three months
- +0.20
- Next change, once the past change is known
- +0.09
Does tone predict the next decision?
Test results, 1,197 bank-months from 2020
Trained on 1999–2016, chosen on 2017–2019, tested once. Lower log loss is better.
| Model | Features | Log loss | Moves called |
|---|---|---|---|
| logistic | Past rates and inflation | 0.591 | 9.1% |
| plus speech tone | 0.588 | 11.8% | |
| boosted treeschosen | Past rates and inflation | 0.524 | 18.6% |
| plus speech tone | 0.539 | 15.6% |
Placebo test, 100 shuffles
Each bank's tone shuffled across months: same values, no timing. Bars count how much each shuffle improved test log loss; the orange bar marks where real tone falls: among the shuffles, on the worse side, not beyond them.