AI Governance Policy Fellowship
About CAIAC and This Fellowship
CAIAC's mission is to produce researchers who will address technical and policy questions that reduce risks from advanced artificial intelligence and steer the trajectory of AI development for the better. The Policy Fellowship helps fulfill this mission by offering an introduction on AI safety, while simultaneously preparing rising scholars to think critically about foundational questions and generate new research ideas.
This semester, there are four Fellowship sections (two Technical ML Fellowships and two Policy Fellowships). Each section is led by a facilitator who has tailored the curriculum to fit their strengths. The section I will lead will be treated as a graduate survey course and will focus on generating new research ideas in AI policy, governance, and social impact.
My purpose in focusing on generating new research ideas is to support the mission of CAIAC to produce researchers who will eventually work in AI. Before one works at a major AI lab like OpenAI, Anthropic, or DeepMind, one usually has to have participated in one or several tier one research fellowship programs like MATS, ERA:AI, CBAI, LASR, or Pivotal. These are prestigious research fellowships that have a strong track record of placing fellows into full-time roles at major AI labs. The bar of entry is significantly high to be considered for one of these research fellowship programs, as they are tailored for value-aligned people with a strong research background and record. CAIAC aims to fill this space by offering an initial launching pad that can provide rising scholars with the resources and mentorship to matriculate to a tier one research fellowship and to work at a major AI lab. Several of our alumni have successfully done this and gone on to work at Redwood Research, Constellation, MATS, CHAI, SPAR, and XLab.
What I'm Asking of You
To make this Fellowship as productive as possible, I have two main asks:
- Please come to the section having read that week's readings. This is a basic responsibility that will be asked of you as you go throughout your academic career and while working at a research lab. The sad alternative of not having read that week's readings is to sit in silence for our allotted time together as everyone tries to catch up and have a conversation that resembles the breadcrumbs that fall off the table after supper.
-
This seminar is designed to generate new research ideas. To do so, the course will push you to understand key ideas in an expansive literature, and you will be asked to submit two short papers that meaningfully extend the ideas in the readings. You may choose to write an empirical or conceptual short paper.
For an empirical paper, you should outline an original empirical study somehow informed by or related to the week's readings. Your summary should be organized in the following four sections: (1) a research question (1-2 sentences), (2) a description of methods (2-3 paragraphs), (3) your prediction(s) (1-2 sentences), and (4) why the study would be interesting and/or important (2-3 sentences).
For a conceptual paper, you should advance one or more of the week's readings in some way. Examples include: (a) connecting ideas from two or more papers to develop a more general theory; (b) critiquing ideas in one or more papers and proposing an alternative that you argue is better or more valid; (c) developing a rival theory to challenge the ideas in one or more papers. In your conceptual paper, underline your main thesis statement(s) (1-2 sentences).
Please complete these readings before the first session:
- But what is a neural network? (3Blue1Brown, 2017). Watch first 12.5 min.
- The AI Triad and what it means for national security strategy [Exec Summary Only] (Buchanan, 2020) 5 min.
- 4 charts that show why AI progress is unlikely to slow down (Henshall, 2023). 8 min.
- Can AI Scaling Continue Through 2030? [Selected sections] (Sevilla et al., 2024) 20 min.
Weekly Schedule
Week 1: What is AI Governance and why do we need it? (Sept. 29)
- International Scientific Report on the Safety of Advanced AI (Bengio et al. 2025) - Executive Summary (pp. 15-25)
- Introduction to AI Safety, Ethics and Society - Chapter 8, Governance
- AI Governance: Overview and Theoretical Lenses (Allan Dafoe, 2022)
- The AI Triad and What It Means for National Security Strategy (Ben Buchanan, 2020)
Week 2: Data Governance (Oct. 6)
Week 3: Compute Governance (Oct. 13)
- Computational power and AI. AI Now Institute. (Vipra, J, 2025)
- Computing Power and the Governance of Artificial Intelligence (Girish Sastry et al. 2024)
- The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research (Carnegie Endowment)
- Increased compute efficiency and the diffusion of AI capabilities (Pilz, K. F., Heim, L., & Brown, N. 2025)
Week 4: Algorithms, Values and Alignment (Oct. 20)
- Artificial Intelligence, Values and Alignment (Deepmind)
- The Alignment Problem from a Deep Learning Perspective
- AI 2027 (Kokotajlo et al., 2025)
Week 5: Evaluating AI – Audits, Frameworks and Corporate Governance (Oct. 27)
- Open Problems in Technical AI Governance
- Auditing large language models: A three-layered approach (Mökander et al., 2023)
- Evaluating language-model agents on realistic autonomous tasks (Kinniment et al., 2023)
- AI is Testing the Limits of Corporate Governance (Tallarita, 2023)
- Black-Box Access is Insufficient for Rigorous AI Audits (Casper et al., 2024)
Week 6: AI National Strategies and Perspectives (Nov. 10)
Required Readings:
- State of the AI Regulatory Landscape (Convergence Analysis, 2024)
- Winning the Race: America's AI Action Plan (The White House, 2025)
- High-level summary of the AI Act (EU AIA, 2024)
- AI Watch: Global regulatory tracker - China (Whitecase, 2024)
Further Reading: National AI strategies available from Brazil, Colombia, African Union, Estonia, France, India, Japan, Kazakhstan, Kenya, Mexico, Nigeria, Republic of Korea, Rwanda, Saudi Arabia, Singapore, Spain, Philippines, UAE, and UK.
Week 7: International AI Cooperation (Nov. 17)
- Envisioning a Global Regime Complex to Govern Artificial Intelligence (Klein & Patrick, 2024)
- Governing AI for Humanity Final Report, executive summary (UN AI Advisory Body, 2024)
- Hiroshima Process International Guiding Principles for Organizations Developing Advanced AI System (Hiroshima Summit, 2023)
Contact
For questions about the fellowship, please reach out to eab2291 -at- columbia.edu