2023 ESPAI Survey Analysis

1. Overview & Survey Methodology

Randomization Blocks

Data Cleaning Notes

The following cleaning steps were applied (see cleaning_log.md for full details):

This dataset was pre-cleaned upstream. Column renaming was applied; no further numeric cleaning was needed.

3.1 When Will 39 AI Milestones Be Feasible?

Corresponds to Figure 1 in Grace et al. (2024). Dots show the 50% probability year from the mixture CDF; horizontal lines show the 10%–90% range. The x-axis is capped at 2100; milestones whose 50% or 90% year falls beyond the cap are labeled with an arrow showing the true year. Includes 39 tasks, 4 occupations, HLMI, and FAOL.

Combined year-framing and probability-framing responses using gamma mixture CDF aggregation (matching the 2023 report methodology). A gamma CDF is fitted to each respondent's data, then all CDFs are averaged pointwise to produce a mixture CDF from which percentiles are read. Sorted by 50% probability year (earliest first). The HLMI row uses the dedicated direct-HLMI blocks (hb_a_*/hb_b_*); it does not pool the final-occupation prediction block.

Year values are years from survey date (2023), with calendar year in parentheses. "10% prob year" = year at which the mixture CDF reaches 10%. "90% prob year" = 90%. Dashes indicate insufficient data or an aggregate CDF that does not reach the target percentile within the 1e8-year "never/infinity" sentinel bound.

3.2 HLMI & Full Automation of Labor Timing

High-Level Machine Intelligence (HLMI)

HLMI is defined as machines that can accomplish every task better and cheaper than human workers.

The comparison column shows the published value for this same 2023 survey (Grace et al. 2024); any difference reflects cleaning and aggregation choices in this re-analysis, not a change in expert opinion.

N (gamma fits): 1757 (0 failed fits)

This HLMI estimate uses only the dedicated direct-HLMI question blocks (hb_a_* and hb_b_*). It does not pool the separate final-occupation prediction block (hj_*_final_pred).

Full Automation of Labor (FAOL)

All occupations fully automatable — machines carry out every task better and more cheaply than humans.

N (gamma fits): 792 (0 failed fits)

Gap Between HLMI and FAOL

The consistent gap between HLMI and FAOL predictions reflects researchers' view that achieving human-level AI capability differs from fully automating all occupations.

Methodology Note

Gamma mixture CDF method: Following the methodology of Grace et al. (2024), a gamma CDF is fitted to each respondent's three data points (whether year-framing or probability-framing). All individual CDFs are then averaged pointwise to produce a mixture CDF, from which percentiles are read off. This approach handles extrapolation naturally (pessimistic respondents contribute heavy-tailed CDFs rather than being dropped) and treats both question framings symmetrically. Respondents were randomly assigned to either a "year framing" (provide years for 10%/50%/90% probability) or a "probability framing" (provide probabilities at fixed time horizons: 10, 20, and 40 years for HLMI; 10, 20, and 50 years for FAOL). See compare_methods.py for a comparison of this method with linear interpolation.

Corresponds to Figure 3 in Grace et al. (2024). Thin lines show individual respondent gamma CDFs (random subset of 200); thick line is the mixture (average) CDF. The dashed line marks 50% probability.

3.2.4 High-Level Machine Intelligence Timing by Experience

Insufficient demographic data: the years-in-field question (hh_howlong) was redacted or absent in this year's anonymized dataset, so the experience split cannot be computed.

3.2.5 Do Participants Agree on HLMI Timing?

"How much do you think your views on when HLMI will be achieved differ from those of the typical AI researcher?" (N=670)

3.3 Framing Effects

Respondents were randomly assigned to one of two question framings for the same underlying question. The "year framing" asked how many years until a given probability, while the "probability framing" asked for the probability at a fixed time horizon.

HLMI 50% Probability Year by Framing

A consistent framing effect has been observed across survey waves: the year-framing tends to produce earlier (shorter-timeline) predictions than the probability-framing. This is a documented cognitive bias in probability elicitation.

3.4 Perceived Rates of Progress

Respondents were asked whether AI progress was faster in the first or second half of their career. (N=333)

Corresponds to Figure 4 in Grace et al. (2024).

The comparison column shows the published value for this same 2023 survey (Grace et al. 2024); any difference reflects cleaning and aggregation choices in this re-analysis, not a change in expert opinion.

How Far Along Is AI Progress? (Outside-view slider)

Respondents were shown a slider with three anchors: A = where progress was when they started working in their AI area; B = where it is now; and C = where it would need to be for AI software to have roughly human-level abilities at the tasks they study. They were asked "What fraction of the distance between where progress was when you started working in the area (A) and where it would need to be to attain human-level abilities in the area (C) have we come so far (B)?" on a 0–100 scale. This complements the year-based timing questions by anchoring an estimate to each respondent's own career window — so it is robust to disagreement about absolute calendar dates.

IQR (2023): 18–60.

3.5 What Causes AI Progress?

Respondents estimated how much less AI progress there would have been with half as much of each input. Higher values = more important to progress. Values are percentage less progress (0-100%).

Corresponds to Figure 5 in Grace et al. (2024). Red dots are means; box shows IQR with median line.

3.6 Will There Be an Intelligence Explosion?

Probability Estimates (numeric, 0-100%)

The comparison column shows the published value for this same 2023 survey (Grace et al. 2024); any difference reflects cleaning and aggregation choices in this re-analysis, not a change in expert opinion.

Is the Feedback Loop Argument Broadly Correct? (N=299)

"If AI does nearly all R&D, improvements in AI will accelerate progress including further AI progress. This could cause progress to become >10x faster within 5 years."

Corresponds to Figure 6 in Grace et al. (2024).

3.7 AI Capabilities in 2043

"In 2043, how likely do you think the following will be for at least some state-of-the-art AI systems?" Sorted by percentage rating Likely or Very Likely.

Corresponds to Figure 7 in Grace et al. (2024).

3.8 Will AI Explain Its Decisions? (2028)

"For typical state-of-the-art AI systems in 2028, will users be able to know the true reasons for decisions?" (N=912)

Corresponds to Figure 8 in Grace et al. (2024).

4.1 How Concerning Are Future AI Scenarios?

"How would you rate the level of concern these scenarios deserve over the next thirty years?" Sorted by percentage rating Substantial or Extreme concern.

Corresponds to Figure 9 in Grace et al. (2024).

4.2 How Good or Bad Will HLMI Be?

Respondents assigned probabilities to five outcome categories (summing to 100%). N=2704

Corresponds to Figure 11 in Grace et al. (2024), "Thousands of AI Authors on the Future of AI."

Mean and Median Probabilities

Individual Respondent Views

Each vertical slice below is one respondent's probability allocation, sorted from most optimistic (left) to most pessimistic (right). The chart reveals the full diversity of expert opinion.

Corresponds to Figure 10 in Grace et al. (2024).

The same respondents sorted by how much probability they assign to "extremely bad (e.g. human extinction)" outcomes. The growing black band on the right shows those assigning the highest probability to extremely bad outcomes.

No direct equivalent in Grace et al. (2024); novel visualization of the same Section 4.2 data.

No direct equivalent in Grace et al. (2024); novel visualization of the same Section 4.2 data.

Individual Response Profiles

Each small bar below represents one respondent's five-category probability distribution, sampled evenly from the most optimistic to the most pessimistic.

No direct equivalent in Grace et al. (2024); novel visualization of the same Section 4.2 data.

Extreme Outcome Analysis

The comparison column shows the published value for this same 2023 survey (Grace et al. 2024); any difference reflects cleaning and aggregation choices in this re-analysis, not a change in expert opinion.

4.3 How Likely Is AI to Cause Extinction or Severe Disempowerment?

Each respondent was randomly assigned ONE of three questions about the probability that future AI advances cause human extinction or similarly permanent and severe disempowerment. The chart below also includes "extremely bad" HLMI outcome probabilities from Section 4.2 for comparison.

Corresponds to Figure 13 in Grace et al. (2024).

Note: The "Extremely bad HLMI outcome" question was asked to all respondents (N is larger), while each extinction/disempowerment question was asked to a random subset of respondents.

Corresponds to Figure 12 in Grace et al. (2024).

The "Pooled" row combines the raw responses from all variants above (1321 + 661 + 655 = 2637 responses) and takes the median of the combined data. Because each respondent was randomly assigned exactly one variant, no respondent is counted twice. And because the other variants each add a constraint (a specific cause, a time limit) to the basic question, every response is a lower bound on that respondent's unconstrained probability — so the pooled median is a conservative basis for "the median researcher put at least 5%..." claims.

The comparison column shows the published value for this same 2023 survey (Grace et al. 2024); any difference reflects cleaning and aggregation choices in this re-analysis, not a change in expert opinion.

4.4 Are Future AI-Risk Concerns Due to Misunderstandings of AI Research?

"To what extent do you think people's concerns about future risks from AI are due to misunderstandings of AI research?" (N=671)

No direct figure equivalent in Grace et al. (2024); this data is discussed in Section 4.4 of that paper.

4.5 Rates of 5-Year Global AI Progress Prompting Most Optimism for Humanity

Question wording: "What rate of global AI progress over the next five years would make you feel most optimistic for humanity's future? Assume any change in speed affects all projects equally." (N=646)

Corresponds to Table 3 in Grace et al. (2024); no figure equivalent in that paper.

The comparison column shows the published value for this same 2023 survey (Grace et al. 2024); any difference reflects cleaning and aggregation choices in this re-analysis, not a change in expert opinion.

4.6 How Much Should AI Safety Research Be Prioritized?

"How much should society prioritize AI safety research, relative to how much it is currently prioritized?" (N=668)

Corresponds to Figure 14 in Grace et al. (2024).

4.7 The Alignment Problem

Corresponds to Figure 15 in Grace et al. (2024).

Do you think this argument points at an important problem? (N=1322)

How valuable is it to work on this problem today, compared to other problems in AI? (N=1322)

How hard do you think this problem is compared to other problems in AI? (N=1318)

5. Results by Value-Outlook Cluster

5.0 Methodology: Identifying Outlook Groups

One of the survey's core questions asks respondents to distribute 100 probability points across five possible outcomes of high-level machine intelligence: extremely good, on balance good, more or less neutral, on balance bad, and extremely bad. These five numbers form a compact signature of each respondent's overall outlook on advanced AI.

We apply a Gaussian Mixture Model (GMM) with K=4 components to these five-dimensional signatures (standardized to zero mean and unit variance). GMM is a soft-clustering method that models the data as a mixture of multivariate normal distributions; each respondent is assigned to the component with the highest posterior probability. We use 20 random initializations to avoid local optima.

The four resulting clusters are then named by inspecting each cluster's mean probability profile:

• Strong Optimists — assign high probability to extremely good outcomes and very little to bad outcomes.
• Mild Optimists — lean positive overall (combined good > 55%) but with more hedging and moderate expectations.
• Concerned — assign elevated probability to bad or extremely bad outcomes (combined pessimism > 30%).
• Polarized/Bimodal — the most distinctive group: they assign substantial probability to both extremely good and extremely bad outcomes, reflecting a worldview where advanced AI is seen as a high-stakes gamble rather than a clearly positive or negative development.

This four-way split captures meaningful variation in worldview that cuts across traditional demographic variables like experience level. 2,704 of 3,270 cleaned analysis rows had complete value-outlook data and were assigned to a cluster. The remaining subsections re-examine key survey results through this lens.

The bimodal outlook is not merely an artifact of the mixture model. Among the 2,704 respondents who answered both endpoints of the value question, 790 (29%) assigned at least 10% probability to both an extremely good and an extremely bad outcome, and 109 (4%) assigned at least 25% to both. This model-free count confirms that a substantial minority genuinely hedges across both extremes rather than settling on a single direction.

5.0.1 Cluster Profiles

5.1 HLMI Timeline Predictions

How soon do different outlook groups expect human-level machine intelligence?

5.2 Extinction/Disempowerment Estimates

Probability that future AI causes human extinction or permanent severe disempowerment, broken down by outlook cluster.

5.3 Concerning Scenarios

Mean concern level (0 = no concern, 3 = extreme concern) for each of 11 AI risk scenarios. The table below highlights which scenarios show the largest divergence across clusters.

5.4 Expected AI Capabilities in 2043

Mean rated likelihood (0 = very unlikely, 4 = very likely) that AI systems will exhibit each capability by 2043.

5.5 Intelligence Explosion

Median probability estimates for dramatic AI capability speedup and superhuman AI emergence, by outlook cluster.

5.6 Rates of 5-Year Global AI Progress Prompting Most Optimism for Humanity

What rate of global AI progress over the next five years would make each cluster feel most optimistic for humanity's future?

5.7 Safety and Alignment-Problem Views

Mean scores on whether the alignment argument points at an important problem, how much society should prioritize AI safety research, and how valuable it is to work on the alignment problem today compared with other AI problems (all 0–4 scales).

5.8 Counterfactual Progress Reduction if Factors Were Halved

Median estimated progress reduction if each factor were halved, by cluster.

Cluster assignments are based on GMM (K=4, 20 random initializations) applied to the five HLMI value-outcome probabilities (vb_1_1–vb_1_5). Respondents missing all five values are excluded (2,704 of 3,270 cleaned analysis rows assigned). Sample sizes vary across subsections due to block randomization.

Appendix: Cluster Analysis

A data-driven exploration of the natural groupings, the safety divide, and hardware vs. software beliefs among 3,270 ESPAI 2023 cleaned analysis rows.

Executive Summary

1. How Many Camps? The Natural Clusters

1.1 Clustering on Value Outlook (n=1,538)

The value-of-HLMI question asked respondents to assign probabilities (summing to 100%) across five outcomes: extremely good, on balance good, neutral, on balance bad, and extremely bad. It is the highest-coverage feature in this analysis. We cluster on these 5 dimensions using Gaussian Mixture Models.

Figure 1: Silhouette scores for K=2 through K=6 clusters on value outlook. K=2 has the highest silhouette, but K=3 and K=4 offer more interpretable structure.

Silhouette scores are modest (0.10-0.12), indicating the clusters are not sharply separated -- this is a continuous landscape of opinion, not discrete tribes. Still, the structure is meaningful.

1.2 Four-Camp Solution

Figure 2: Four natural camps in value outlook. Left: PCA projection. Center: mean probability profiles. Right: cluster sizes.

The most striking finding here is the Polarized/Bimodal group. These respondents assign high probability to both extremely good and extremely bad outcomes -- their P(extremely good) is comparable to the Strong Optimists, yet they simultaneously assign substantial probability to catastrophic outcomes. This is not a group of doomers or technophobes. They are researchers who believe AI will be hugely impactful, but who are genuinely torn on whether that impact will be positive or negative. The key axis for this group is magnitude of impact, not direction -- they have rejected the possibility that AI will be a modest or neutral development.

1.3 Three-Camp Solution (Simpler View)

Collapsing to 3 clusters (which has a higher silhouette score of 0.18 vs 0.07) merges the finer distinctions into a simpler optimist/moderate/pessimist framing. This loses the polarized group but provides a cleaner summary:

Figure 3: Three-camp simplification. The polarized group is absorbed into the moderate/pessimist clusters.

1.4 Intuitive View: How Soon vs. How Good

The PCA axes above lack intuitive meaning. Below, we plot each respondent on two directly interpretable dimensions: their predicted HLMI arrival year (x-axis) and their net optimism score (y-axis), defined as P(good + extremely good) minus P(bad + extremely bad). This reveals where the four camps sit in the space of "how soon" vs. "how beneficial."

Figure 3b: Respondents (n=1663) binned by predicted HLMI year and net optimism. Each bubble's size shows how many researchers from that cluster fall in the bin (labeled when ≥5). The vertical dashed line marks the median predicted year; the horizontal line separates net optimists from net pessimists.

1.5 Combined Clustering: Values + Concerns + Safety + Extinction

For the 323 respondents who answered all of: value outlook, concern scenarios, alignment-problem importance, and extinction/disempowerment probability, we ran a richer clustering incorporating 8 features.

Figure 4: Combined clustering incorporating worldview, concerns, safety/alignment-problem views, and extinction/disempowerment probability. Mean concern averages 11 scenario concern ratings on a 0-3 scale (0=no concern, 3=extreme concern); alignment-problem importance is a 0-4 ordinal score (0=not a real problem, 4=among the most important problems in the field); P(extinction/disempowerment) is a percentage.

HLMI year summaries include the median, interquartile range, mean, and item-level n because several groups share the same median year while their wider distributions differ.

2. Higher vs Lower Safety/Severe-Risk Concern

2.1 Defining the Groups

We built a safety composite score from (normalized 0-1 and averaged):

• Alignment-problem importance rating (0-4 ordinal scale: not a real problem to among the most important problems in the field)
• Value of working on the alignment problem today compared with other AI problems (0-4 ordinal scale)
• P(extinction/disempowerment from AI) (0-100%)

Requiring at least 2 of 3 components, we obtained scores for 1322 respondents and split at the median (0.500) into High (n=595) and Low (n=727) safety concern groups.

2.2 The Full Comparison

Figure 5: Comprehensive comparison of higher vs lower safety/severe-risk-concern respondents across six dimensions. Alignment-problem importance uses the 0-4 ordinal scale above; mean concern averages 11 scenario concern ratings on a 0-3 scale; the optimism-maximizing AI progress-rate answer is coded 0=much slower, 2=current speed, 4=much faster.

2.3 Statistical Tests

Selected Mann-Whitney U tests for scalar summaries from Figure 5. The value-outlook panel is summarized by P(extremely good) and P(extremely bad), the concern panel by mean concern, and the AI-capabilities panel is tested item-by-item in the next table. Effect size r is rank-biserial correlation (|r| > 0.3 = medium, |r| > 0.5 = large). Scale notes: mean concern 0-3, optimism-maximizing AI progress-rate answer 0-4, probabilities in percentage points. *** p<0.001, ** p<0.01, * p<0.05

2.4 AI Capabilities by 2043: What Do Safety People Expect?

Safety-concerned people don't just worry more -- they have systematically different expectations for what AI will be able to do by 2043:

Scale: 0=Very unlikely, 1=Unlikely, 2=Even chance, 3=Likely, 4=Very likely. High - Low is the high-safety mean minus the low-safety mean, so positive values mean high-safety respondents rated the capability as more likely. Hypothesized direction is marked for risk-relevant capabilities where we expected High > Low; other rows are exploratory.

2.5 Intelligence Explosion Beliefs

Figure 6: Probability estimates for intelligence explosion scenarios, by safety concern level.

3. Hardware vs. Software Progress Beliefs

3.1 Counterfactual Progress Reduction if Factors Were Halved

Respondents rated how much AI progress would decrease if each of 5 factors were cut in half (0-100% scale). Higher values mean the factor is more important.

Figure 7: Counterfactual progress-factor analysis. Top-left: estimated progress reduction distributions. Top-right: hardware vs algorithm progress-reduction estimates. Bottom-left: HW-SW index distribution. Bottom-right: factor correlations with other variables.

3.2 Hardware vs Software Index

We computed (Hardware - Algorithms) / (Hardware + Algorithms) for each respondent who rated both. Positive = hardware-leaning, negative = software-leaning.

3.3 Do Progress Beliefs Predict Other Views?

Significant cause-factor correlations (p < 0.10)

Hardware-Software Index Correlations

3.4 Progress Beliefs by Outlook and Safety Groups

Do optimists, pessimists, and safety-concerned researchers differ in what they think drives AI progress? Below we break down cause factor importance by outlook cluster (left) and safety group (right).

Figure 7b: Mean importance ratings for each progress factor, split by outlook group (left) and safety group (right). Scale: 0-100% estimated decrease in progress if factor halved.

4. How Outlook Predicts Everything Else

The three value-outlook clusters (from Section 1) predict views across many other dimensions:

Values are medians with sample sizes in parentheses, except HLMI Year, which shows median [IQR], mean, and item-level n because several groups share the same median year. Scale notes: alignment-problem importance 0-4, mean concern 0-3, optimism-maximizing AI progress-rate answer 0=much slower to 4=much faster, probabilities in percentage points.

Figure 8: How the four value-outlook camps compare across timelines, extinction/disempowerment probability, safety/alignment-problem views, concern levels, optimism-maximizing AI progress-rate answers, and P(extremely bad). Mean concern averages 11 scenario concern ratings on a 0-3 scale; alignment-problem importance and optimism-maximizing AI progress-rate answers use 0-4 ordinal scales.

5. What Dimensions Structure AI Researcher Beliefs?

Principal Component Analysis on respondents with values + concerns + safety + extinction/disempowerment data (n=301) reveals the latent axes of disagreement.

Figure 9: PCA factor loadings. Green bars = positive loading, red = negative. Each panel shows one principal component.

6. The Full Correlation Structure

Figure 10: Spearman rank correlations between key variables. Stars indicate significance. Sample sizes shown in each cell. Alignment-problem importance uses a 0-4 ordinal scale from not a real problem to among the field's most important problems; value of working on the alignment problem today uses a 0-4 ordinal scale from much less valuable to much more valuable than other AI problems; mean concern uses a 0-3 scale; HLMI Year is a calendar-year estimate; probabilities are 0-100 percentages.

Key Pairwise Correlations

7. Methods & Caveats

7.1 Approach

• Strategy: Multiple focused analyses on overlapping subsets (rather than one clustering requiring all features, which drops to n~180). This maximizes statistical power while giving complementary views.
• Clustering: Gaussian Mixture Models (GMM) with full covariance, 10-20 random initializations, model selection by BIC with parsimony preference
• Statistical tests: Non-parametric throughout (Mann-Whitney U for two groups, Kruskal-Wallis for three+, Spearman rank correlations)
• Visualization: PCA for dimensionality reduction (visualization only -- clustering done in full feature space)

7.2 Caveats

7.3 Technical Details

• Random seed: 42
• All tests two-sided; no multiple comparison correction (exploratory analysis)
• HLMI year capped at 2150 for visualization; uncapped for statistics where noted

Appendix: Survey Flow

Survey-flow / randomization diagram modeled on Figure 16 of Grace et al., Thousands of AI Authors on the Future of AI. Each box is a question block. Percentages and n's are realized fill rates: the realized number of respondents who answered each block, divided by the total N. The total N counts everyone who answered at least one question (respondents who answered none are excluded).

Jobs / FAOL Sample-Size Reconciliation

The survey-flow boxes count anyone with any response in the broader Jobs / FAOL block. FAOL analyses use only the FAOL triplet, so their sample sizes can be slightly smaller.

Count definition	Fixed-years	Fixed-probabilities
Whole Jobs / FAOL block (survey-flow box)	384	426
FAOL only, any triplet answer	380	418
FAOL only, complete triplet	379	413

Here, "fixed-years" means respondents gave probabilities for fixed time horizons (hj_b_full_*), while "fixed-probabilities" means respondents gave years for fixed probabilities (hj_a_full_*).

Tasks: ta_* columns are the fixed-probability framing, tb_* the fixed-years framing (verified empirically: ta_* rows have fixedprobabilities=1, tb_* rows have fixedprobabilities=0). The Qualtrics survey-flow definition (.qsf) is not in this repo, so the exact display order may differ from the diagram.

Appendix: Supplementary Figures

B.1 fixed-prob/fixed-year CDFs

Full CDF comparison between the fixed-prob and fixed-year conditions for HLMI and FAOL. The fixed-prob framing consistently produces earlier predictions (the red curve is shifted left relative to blue).

Corresponds to Figure 18 in Grace et al. (2024). Each curve is the mixture (mean) CDF for respondents assigned to that framing condition.

B.2 Bootstrap Confidence Bands

95% bootstrap confidence intervals on the aggregate CDF, obtained by resampling the fitted individual CDFs with replacement (500 resamples).

Bootstrap CIs reflect sampling variability — if we surveyed a different random sample of AI researchers, how much would the aggregate change? This is distinct from the spread of individual predictions (which is much wider).

Appendix: Statistical Tests

F.1 Yuen's Trimmed-Mean Bootstrap Test: Demographics

Tests whether experienced researchers (split at the median years in field) give significantly different HLMI predictions than junior researchers. Uses a 10% trimmed mean with 5,000 bootstrap resamples.

Insufficient data for demographic comparison (years-in-field was redacted in the anonymized dataset).

N (experienced): 0, N (junior): 0. Individual 50% years are computed from per-respondent gamma fits. The trimmed mean reduces sensitivity to extreme predictions.

F.2 Framing Effect: Statistical Significance

Tests whether year-framing and probability-framing respondents give significantly different predictions, using the same Yuen's bootstrap method.

A negative difference (yr_50 − pr_50) means the year-framing produces earlier predictions. Medians shown for reference; the statistical test uses trimmed means.

Appendix: Aggregation Method Comparison

Different choices in how individual survey responses are fitted and combined into aggregate forecasts can shift the headline numbers. This appendix compares three approaches on the key milestones, motivated by Adamczewski's reanalysis of ESPAI data.

Methods compared

Mix-Mean (MSE) — the Grace et al. 2023 method. Fit a gamma CDF to each respondent's 3 data points using mean squared error, then take the pointwise mean of all fitted CDFs.

Mix-Median (MSE) — same fitting, but take the pointwise median instead of mean. More robust to outlier predictions (e.g. respondents predicting 100M years). Adamczewski argues the median better represents the "typical expert."

Mix-Mean (LogLoss) — fit using log-loss (cross-entropy) instead of MSE, then take the mean. Log-loss naturally weighs errors near p=0 and p=1 more heavily than errors near p=0.5, matching the intuition that a 5% error at p=0.95 represents a much larger shift in belief than the same error at p=0.50.

Results

Key takeaways

Aggregation method matters more than loss function. Switching from mean to median aggregation pushes HLMI from 2047 to 2049 and FAOL from 2112 to 2121. Switching the loss function (MSE→log-loss) barely changes the aggregate — consistent with Adamczewski's finding that fitting choice has "hardly any impact."

The effect is largest for far-future milestones (FAOL, AI Researcher) where a few very pessimistic respondents pull the mean CDF rightward.

See also: compare_methods.py for the full 5-method comparison across all 39 tasks, including linear interpolation and gamma-individual methods. Method comparison motivated by Adamczewski (bayes.net/espai).

Data: 2023 Expert Survey on Progress in AI (ESPAI).
Previous survey comparison values from "Thousands of AI Authors on the Future of AI" (2024 preprint).