“General capability - and capabilities generally - have no good y-axis” by Gregory Lewis🔸

“General capability - and capabilities generally - have no good y-axis” by Gregory Lewis🔸

BLUF:

  • To determine whether AI is ‘improving exponentially’, ‘hitting the wall’, or any other claim which involves a quantity or magnitude (e.g. ‘This model was a big leap/small increment’). We need a good y-axis: an interval scale of AI capability which means +1 unit always represents the same degree of ‘how much better’, in the same way +1 degree Celsius is always the same amount of ‘how much hotter’.
  • Yet there is no good y-axis for AI capability. All our measures are of something related-to but clearly not identical-with it, thus ‘true’ AI capability can be a funhouse-mirror reflection of whatever was measured. Specifically:
    • Benchmark score: One small step in benchmark score can be a giant leap in capability, or the opposite, or whatever else. (My 6/10 vs. your 4/10 ≠ I’m 50% better at maths than you).
    • Elo et al: Can give a real y-axis in terms of winning chances, but doesn’t translate outside of beating others. (Going from 50% to 73% to 88% chance to get a higher score than you on a maths test ≠ gaining 0 → 1 → 2 units of maths ability over you)
    • Epoch Capabilities Index: Analogous to IQ, so [...]

---

Outline:

(03:05) Introduction

(04:26) Both a poor reflection and a dark glass

(07:51) Human benchmarking also has a y-axis problem

(12:27) A metrological elegy

(12:49) The base case: benchmarks (cf. exams)

(13:30) Elo et al.

(16:25) (And maybe not quite 'game ability ≡ winning games', after all?)

(18:28) ECI (cf. IQ)

(20:56) Intervals Rarely True

(25:01) Measure endogeneity

(31:12) Forking IRT

(34:45) (Dimensions of being, and beating, a bat)

(39:40) Prediction (cf. chronometry)

(41:58) Time horizons

(44:34) Human "capability" is also exponential in time horizon

(47:38) Intuitive/interpretative prelude

(52:56) Time horizons and ECI share an axis kink

(57:13) Perhaps money, as a measure, stinks the least

(01:01:07) Finale: AI as normal epistemics

(01:06:48) Acknowledgements

The original text contained 33 footnotes which were omitted from this narration.

---

First published:
August 4th, 2026

Source:
https://forum.effectivealtruism.org/posts/CQvdadxjCpd7i7kjA/general-capability-and-capabilities-generally-have-no-good-y

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(250)

“What if the third wave is a puddle?” by Morgan Fairless

“What if the third wave is a puddle?” by Morgan Fairless

The world of private philanthropy may be in for a period of large growth. News of Coefficient Giving increasing their funding for GiveWell to one billion in the near-term, and the general growth of th...

18 Aug 9min

“What would make us scale or stop? NOVAH’s pre-commitment before seeing the RCT results” by I.J.J., AlexisAt

“What would make us scale or stop? NOVAH’s pre-commitment before seeing the RCT results” by I.J.J., AlexisAt

TL;DR NOVAH (No Violence At Home) was incubated by Charity Entrepreneurship (now Ambitious Impact) in 2024 to test a promising idea: preventing intimate partner violence through edutainment, in our ca...

17 Aug 15min

“Never born, then maybe died twice” by NickLaing

“Never born, then maybe died twice” by NickLaing

“Your balls are baked aren’t they, mate”. He would not have been bornThanks Lyndon, that describes the situation. Like almost 1 in 10 modern men, my sperm weren’t up to the task - at least not the old...

1 Aug 10min

“Announcing: Cluelessness Critiques Competition” by Toby Tremlett🔹, Will Aldred

“Announcing: Cluelessness Critiques Competition” by Toby Tremlett🔹, Will Aldred

TL;DR We’re hosting an essay competition to elicit responses to Anthony DiGiovanni's sequence, ‘The challenge of unawareness for impartial altruist action guidance’—submit your entries by August 14th...

30 Juli 13min

“Spiro: an update 2.5 years on and a fundraising ask for expansion” by Habiba Banu

“Spiro: an update 2.5 years on and a fundraising ask for expansion” by Habiba Banu

Summary Back in November 2023 I posted here to launch Spiro and raise our first 198 thousand dollars. Two and a half years later this is an update and a fundraiser for the next step. The short version...

26 Juli 17min

“Let’s taboo the V-word” by lincolnq

“Let’s taboo the V-word” by lincolnq

“How long have you been v*g*n?” This is one of the most common icebreakers at animal protection events. It's a baseline assumption, and it mostly holds true: if you’re out advocating for animals not t...

14 Juli 12min

“Giving What We Can’s first  YouTube Video is out now!” by JustinPortela

“Giving What We Can’s first YouTube Video is out now!” by JustinPortela

Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job. My channel is using that same cinematic, high-production value beauty to talk about e...

9 Juli 1min

Populärt inom Samhälle & Kultur

podme-dokumentar
gynning-berg
aftonbladet-krim
p3-dokumentar
en-mork-historia
aftonbladet-daily
creepypodden-med-jack-werner
killradet
rss-expressen-dok
hor-har
flashback-forever
rss-schyffert-sundin-ar-det-har-nat
kod-katastrof
svenska-fall
mardromsgasten
p3-historia
rss-vad-fan-hande
p1-dokumentar
rss-mer-an-bara-morsa
rss-sanning-konsekvens