2016 ESPAI Survey Analysis

1. Overview & Survey Methodology

Randomization Blocks

Data Cleaning Notes

The following cleaning steps were applied (see cleaning_log.md for full details):

• No literal year values found — all values already in years-from-survey format.
• No year values above the 1e8 cap.
• Monotonicity violations found in 2 complete triplets.
• 2 descending triplets found across 2 respondents in year and probability framings (values set to NaN).

Caveat — sum-to-100 triplets: 29 triplets across 27 respondents have values summing to exactly 100 (e.g., 10%, 30%, 60% at the three time horizons). This may indicate respondents who treated the probabilities as shares that must total 100%, rather than independent cumulative probabilities at each time horizon. Following standard methodology, sum-to-100 triplets from respondents with descending errors were also removed (2 total triplets NaN'd).

3.1 When Will 39 AI Milestones Be Feasible?

Corresponds to Figure 1 in Grace et al. (2024). Dots show the 50% probability year from the mixture CDF; horizontal lines show the 10%–90% range. The x-axis is capped at 2100; milestones whose 50% or 90% year falls beyond the cap are labeled with an arrow showing the true year. Includes 39 tasks, 4 occupations, HLMI, and FAOL.

Combined year-framing and probability-framing responses using gamma mixture CDF aggregation (matching the 2023 report methodology). A gamma CDF is fitted to each respondent's data, then all CDFs are averaged pointwise to produce a mixture CDF from which percentiles are read. Sorted by 50% probability year (earliest first). The HLMI row uses the dedicated direct-HLMI blocks (hb_a_*/hb_b_*); it does not pool the final-occupation prediction block.

Year values are years from survey date (2016), with calendar year in parentheses. "10% prob year" = year at which the mixture CDF reaches 10%. "90% prob year" = 90%. Dashes indicate insufficient data or an aggregate CDF that does not reach the target percentile within the 1e8-year "never/infinity" sentinel bound.

3.2 HLMI & Full Automation of Labor Timing

High-Level Machine Intelligence (HLMI)

HLMI is defined as machines that can accomplish every task better and cheaper than human workers.

N (gamma fits): 252 (0 failed fits)

This HLMI estimate uses only the dedicated direct-HLMI question blocks (hb_a_* and hb_b_*). It does not pool the separate final-occupation prediction block (hj_*_final_pred).

Full Automation of Labor (FAOL)

All occupations fully automatable — machines carry out every task better and more cheaply than humans.

N (gamma fits): 92 (0 failed fits)

Gap Between HLMI and FAOL

The consistent gap between HLMI and FAOL predictions reflects researchers' view that achieving human-level AI capability differs from fully automating all occupations.

Methodology Note

Gamma mixture CDF method: Following the methodology of Grace et al. (2024), a gamma CDF is fitted to each respondent's three data points (whether year-framing or probability-framing). All individual CDFs are then averaged pointwise to produce a mixture CDF, from which percentiles are read off. This approach handles extrapolation naturally (pessimistic respondents contribute heavy-tailed CDFs rather than being dropped) and treats both question framings symmetrically. Respondents were randomly assigned to either a "year framing" (provide years for 10%/50%/90% probability) or a "probability framing" (provide probabilities at fixed time horizons: 10, 20, and 40 years for HLMI; 10, 20, and 50 years for FAOL). See compare_methods.py for a comparison of this method with linear interpolation.

Corresponds to Figure 3 in Grace et al. (2024). Thin lines show individual respondent gamma CDFs (random subset of 200); thick line is the mixture (average) CDF. The dashed line marks 50% probability.

3.2.4 High-Level Machine Intelligence Timing by Experience

This table splits respondents at the median years in field, then refits the direct High-Level Machine Intelligence (HLMI) timing model within each experience group.

The count values are not all respondents in each experience group. They count respondents who both answered the years-in-field question and gave enough direct HLMI timing answers to fit a gamma CDF. The separate occupation-automation prediction block is not pooled into this HLMI comparison. Only 107 respondents received the experience question due to randomization. Geographic region and citation count data are not available in the anonymized dataset.

3.2.5 Do Participants Agree on HLMI Timing?

"How much do you think your views on when HLMI will be achieved differ from those of the typical AI researcher?" (N=115)

3.3 Framing Effects

Respondents were randomly assigned to one of two question framings for the same underlying question. The "year framing" asked how many years until a given probability, while the "probability framing" asked for the probability at a fixed time horizon.

HLMI 50% Probability Year by Framing

A consistent framing effect has been observed across survey waves: the year-framing tends to produce earlier (shorter-timeline) predictions than the probability-framing. This is a documented cognitive bias in probability elicitation.

3.4 Perceived Rates of Progress

Respondents were asked whether AI progress was faster in the first or second half of their career. (N=107)

Corresponds to Figure 4 in Grace et al. (2024).

How Far Along Is AI Progress? (Outside-view slider)

Respondents were shown a slider with three anchors: A = where progress was when they started working in their AI area; B = where it is now; and C = where it would need to be for AI software to have roughly human-level abilities at the tasks they study. They were asked "What fraction of the distance between where progress was when you started working in the area (A) and where it would need to be to attain human-level abilities in the area (C) have we come so far (B)?" on a 0–100 scale. This complements the year-based timing questions by anchoring an estimate to each respondent's own career window — so it is robust to disagreement about absolute calendar dates.

IQR (2016): 5–25.

3.5 What Causes AI Progress?

Respondents estimated how much less AI progress there would have been with half as much of each input. Higher values = more important to progress. Values are percentage less progress (0-100%).

Corresponds to Figure 5 in Grace et al. (2024). Red dots are means; box shows IQR with median line.

3.6 Will There Be an Intelligence Explosion?

Probability Estimates (numeric, 0-100%)

Is the Feedback Loop Argument Broadly Correct? (N=224)

"If AI does nearly all R&D, improvements in AI will accelerate progress including further AI progress. This could cause progress to become >10x faster within 5 years."

Corresponds to Figure 6 in Grace et al. (2024).

4.2 How Good or Bad Will HLMI Be?

Respondents assigned probabilities to five outcome categories (summing to 100%). N=345

Corresponds to Figure 11 in Grace et al. (2024), "Thousands of AI Authors on the Future of AI."

Mean and Median Probabilities

Individual Respondent Views

Each vertical slice below is one respondent's probability allocation, sorted from most optimistic (left) to most pessimistic (right). The chart reveals the full diversity of expert opinion.

Corresponds to Figure 10 in Grace et al. (2024).

The same respondents sorted by how much probability they assign to "extremely bad (e.g. human extinction)" outcomes. The growing black band on the right shows those assigning the highest probability to extremely bad outcomes.

No direct equivalent in Grace et al. (2024); novel visualization of the same Section 4.2 data.

No direct equivalent in Grace et al. (2024); novel visualization of the same Section 4.2 data.

Individual Response Profiles

Each small bar below represents one respondent's five-category probability distribution, sampled evenly from the most optimistic to the most pessimistic.

No direct equivalent in Grace et al. (2024); novel visualization of the same Section 4.2 data.

Extreme Outcome Analysis

4.4 Are Future AI-Risk Concerns Due to Misunderstandings of AI Research?

"To what extent do you think people's concerns about future risks from AI are due to misunderstandings of AI research?" (N=115)

No direct figure equivalent in Grace et al. (2024); this data is discussed in Section 4.4 of that paper.

4.5 Rates of 5-Year Global AI Progress Prompting Most Optimism for Humanity

This question was not included in this survey year.

4.6 How Much Should AI Safety Research Be Prioritized?

"How much should society prioritize AI safety research, relative to how much it is currently prioritized?" (N=162)

Corresponds to Figure 14 in Grace et al. (2024).

4.7 The Alignment Problem

Corresponds to Figure 15 in Grace et al. (2024).

Do you think this argument points at an important problem? (N=149)

How valuable is it to work on this problem today, compared to other problems in AI? (N=149)

How hard do you think this problem is compared to other problems in AI? (N=147)

Appendix: Survey Flow

Survey-flow / randomization diagram modeled on Figure 16 of Grace et al., Thousands of AI Authors on the Future of AI. Each box is a question block. Percentages and n's are realized fill rates: the realized number of respondents who answered each block, divided by the total N. The total N counts everyone who answered at least one question (respondents who answered none are excluded).

Jobs / FAOL Sample-Size Reconciliation

The survey-flow boxes count anyone with any response in the broader Jobs / FAOL block. FAOL analyses use only the FAOL triplet, so their sample sizes can be slightly smaller.

Count definition	Fixed-years	Fixed-probabilities
Whole Jobs / FAOL block (survey-flow box)	49	44
FAOL only, any triplet answer	49	44
FAOL only, complete triplet	49	43

Here, "fixed-years" means respondents gave probabilities for fixed time horizons (hj_b_full_*), while "fixed-probabilities" means respondents gave years for fixed probabilities (hj_a_full_*).

Tasks: ta_* columns are the fixed-probability framing, tb_* the fixed-years framing (verified empirically: ta_* rows have fixedprobabilities=1, tb_* rows have fixedprobabilities=0). The Qualtrics survey-flow definition (.qsf) is not in this repo, so the exact display order may differ from the diagram.

Appendix: Supplementary Figures

B.1 fixed-prob/fixed-year CDFs

Full CDF comparison between the fixed-prob and fixed-year conditions for HLMI and FAOL. The fixed-prob framing consistently produces earlier predictions (the red curve is shifted left relative to blue).

Corresponds to Figure 18 in Grace et al. (2024). Each curve is the mixture (mean) CDF for respondents assigned to that framing condition.

B.2 Bootstrap Confidence Bands

95% bootstrap confidence intervals on the aggregate CDF, obtained by resampling the fitted individual CDFs with replacement (500 resamples).

Bootstrap CIs reflect sampling variability — if we surveyed a different random sample of AI researchers, how much would the aggregate change? This is distinct from the spread of individual predictions (which is much wider).

Appendix: Statistical Tests

F.1 Yuen's Trimmed-Mean Bootstrap Test: Demographics

Tests whether experienced researchers (≥6 years in field) give significantly different HLMI predictions than junior researchers. Uses a 10% trimmed mean with 5,000 bootstrap resamples.

N (experienced): 41, N (junior): 32. Individual 50% years are computed from per-respondent gamma fits. The trimmed mean reduces sensitivity to extreme predictions.

F.2 Framing Effect: Statistical Significance

Tests whether year-framing and probability-framing respondents give significantly different predictions, using the same Yuen's bootstrap method.

A negative difference (yr_50 − pr_50) means the year-framing produces earlier predictions. Medians shown for reference; the statistical test uses trimmed means.

Appendix: Aggregation Method Comparison

Different choices in how individual survey responses are fitted and combined into aggregate forecasts can shift the headline numbers. This appendix compares three approaches on the key milestones, motivated by Adamczewski's reanalysis of ESPAI data.

Methods compared

Mix-Mean (MSE) — the Grace et al. 2023 method. Fit a gamma CDF to each respondent's 3 data points using mean squared error, then take the pointwise mean of all fitted CDFs.

Mix-Median (MSE) — same fitting, but take the pointwise median instead of mean. More robust to outlier predictions (e.g. respondents predicting 100M years). Adamczewski argues the median better represents the "typical expert."

Mix-Mean (LogLoss) — fit using log-loss (cross-entropy) instead of MSE, then take the mean. Log-loss naturally weighs errors near p=0 and p=1 more heavily than errors near p=0.5, matching the intuition that a 5% error at p=0.95 represents a much larger shift in belief than the same error at p=0.50.

Results

Key takeaways

Aggregation method matters more than loss function. Switching from mean to median aggregation pushes HLMI from 2061 to 2066 and FAOL from 2139 to 2147. Switching the loss function (MSE→log-loss) barely changes the aggregate — consistent with Adamczewski's finding that fitting choice has "hardly any impact."

The effect is largest for far-future milestones (FAOL, AI Researcher) where a few very pessimistic respondents pull the mean CDF rightward.

See also: compare_methods.py for the full 5-method comparison across all 39 tasks, including linear interpolation and gamma-individual methods. Method comparison motivated by Adamczewski (bayes.net/espai).

Data: 2016 Expert Survey on Progress in AI (ESPAI).
Previous survey comparison values from "Thousands of AI Authors on the Future of AI" (2024 preprint).