The journal
We currently have 49 published papers on our research journal, which is free to read and submit to, regardless of institutional affiliation. Work by high school and student authors is listed first.
the schools are printed under each title
Faithful and Stable Neuron Explanations for Trustworthy Mechanistic Interpretability
Full credit goes to the original author, linked below. All research papers were reposted either with permission of the author, or by anonymous submission by SAIRC members like yourself. Neuron identification is a popular tool in mechanistic interpretability, aiming to uncover the human-interpretable concepts represented by individual neurons in deep networks. While algorithms such as Network Dissection and CLIP-Dissect achieve great empirical success, a rigorous theoretical foundation remains absent, which is crucial to enable trustworthy and reliable explanations. In this work, we observe that neuron identification can be viewed as the inverse process of machine learning, which allows us to derive guarantees for neuron explanations. Based on this insight, we present the first theoretical analysis of two fundamental challenges: (1) Faithfulness: whether the identified concept faithfully represents the neuron's underlying function and (2) Stability: whether the identification results are consistent across probing datasets. We derive generalization bounds for widely used similarity metrics (e.g. accuracy, AUROC, IoU) to guarantee faithfulness, and propose a bootstrap ensemble procedure that quantifies stability along with BE (Bootstrap Explanation) method to generate concept prediction sets with guaranteed coverage probability. Experiments on both synthetic and real data validate our theoretical results and demonstrate the practicality of our method, providing an important step toward trustworthy neuron identification.
What Can We Actually Steer? A Multi-Behavior Study of Activation Control
Full credit goes to the original author, linked below. All research papers were reposted either with permission of the author, or by anonymous submission by SAIRC members like yourself. Large language models (LLMs) require precise behavior control for safe and effective deployment across diverse applications. Activation steering offers a promising approach for LLMs' behavioral control. We focus on the question of how steering effectiveness varies across different behavior types and whether the nature of target behaviors can predict steering success. We address this through empirical analysis of activation steering across 50 behaviors that span persona archetypes, personality traits, misalignment behaviors, style cues, and impersonation of public figures. We present a set of comprehensive experiments on coefficient optimization, vector properties, and data requirements to provide comprehensive guidance for the implementation of activation steering. Our analysis demonstrates that steering effectiveness varies significantly by behavior type, with different behavioral categories exhibiting distinct response patterns to intervention strength. We find that trait expression follows an inverted-U curve with a steering coefficient strength. We also show that vector separation metrics do not predict steering success, but larger training datasets enable more aggressive steering. These findings provide empirically grounded guidance for implementing activation steering and demonstrate that steering effectiveness is heavily influenced by behavior type.
Base Models Know How to Reason, Thinking Models Learn When
Full credit goes to the original author, linked below. All research papers were reposted either with permission of the author, or by anonymous submission by SAIRC members like yourself. What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model's reasoning behaviors by training small Sparse Autoencoders on sentence-level activations of reasoning traces, yielding interpretable reasoning taxonomies. Building on this, we introduce constructive model diffing, which aims to reconstruct the base-to-fine-tuned difference from interpretable components: reasoning mechanisms (category vectors that can induce a reasoning behavior in the base model) and reasoning heuristics (a classifier determining when a mechanism should fire). Across nine base/thinking pairs (four RL-trained, four SFT-distilled, one mixed), two independent findings agree: category vectors in the base model converge to far lower loss for taxonomies derived from purely RL-trained models, and hybrid models recover roughly 76% of the RL base-to-thinking gap but only 11% of the SFT gap. This indicates RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones, offering a new lens on what training paradigms teach, with implications for efficient reasoning-model development.
BlackboxNLP-2025 MIB Shared Task: Exploring Ensemble Strategies for Circuit Localization Methods
Full credit goes to the original author, linked below. All research papers were reposted either with permission of the author, or by anonymous submission by SAIRC members like yourself. The Circuit Localization track of the Mechanistic Interpretability Benchmark (MIB) evaluates methods for localizing circuits within large language models (LLMs), i.e., subnetworks responsible for specific task behaviors. In this work, we investigate whether ensembling two or more circuit localization methods can improve performance. We explore two variants: parallel and sequential ensembling. In parallel ensembling, we combine attribution scores assigned to each edge by different methods-e.g., by averaging or taking the minimum or maximum value. In the sequential ensemble, we use edge attribution scores obtained via EAP-IG as a warm start for a more expensive but more precise circuit identification method, namely edge pruning. We observe that both approaches yield notable gains on the benchmark metrics, leading to a more precise circuit identification approach. Finally, we find that taking a parallel ensemble over various methods, including the sequential ensemble, achieves the best results. We evaluate our approach in the BlackboxNLP 2025 MIB Shared Task, comparing ensemble scores to official baselines across multiple model-task combinations.
Delayed Attention Training Improves Length Generalization in Transformer--RNN Hybrids
Full credit goes to the original author, linked below. All research papers were reposted either with permission of the author, or by anonymous submission by SAIRC members like yourself. We study length generalization in sequence models on a composite problem involving both state tracking and associative recall. Prior work finds that recurrent networks handle state tracking well but struggle with recall, whereas Transformers excel at recall yet fail to extend state-tracking capabilities to longer sequences. Motivated by the complementary strengths of these architectures, we construct hybrid models integrating recurrent and attention-based components, and train them on the combined task to evaluate whether both capabilities can be preserved. Our results reveal that, in such hybrids, the Transformer component tends to exploit shortcut solutions, leading to poor length generalization. We identify this shortcut reliance as a key obstacle and propose a simple yet effective training strategy -- delaying the training of the attention layers -- that mitigates this effect and significantly improves length generalization performance. Our experiments show that this approach enables hybrid models to achieve near-perfect accuracy ($>90\%$) on hybrid sequences three times longer than those used during training.
A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering
Full credit goes to the original author, linked below. All research papers were reposted either with permission of the author, or by anonymous submission by SAIRC members like yourself. Sparse autoencoders (SAEs) have recently emerged as a powerful tool for language model steering. Prior work has explored top-k SAE latents for steering, but we observe that many dimensions among the top-k latents capture non-semantic features such as punctuation rather than semantic attributes like instructions. To address this, we propose focusing on a single, most relevant SAE latent (top-1), eliminating redundant features. We further identify a limitation in constant SAE steering, which often produces degenerate outputs such as repetitive single words. To mitigate this, we introduce a token-wise decaying steering strategy, enabling more faithful comparisons with mean activation difference baselines. Empirically, we show that steering an SAE latent associated with reasoning reliably elicits step-by-step mathematical reasoning and enhances inference quality, functionally resembling the effect of appending a guiding token. Our results demonstrate that SAEs outperform mean activation difference methods on mathematical reasoning benchmarks and match their performance on IF-Eval.
Mechanistic Interpretability with Sparse Autoencoder Neural Operators
Full credit goes to the original author, linked below. All research papers were reposted either with permission of the author, or by anonymous submission by SAIRC members like yourself. We introduce sparse autoencoder neural operators (SAE-NOs), a new class of sparse autoencoders that operate in function spaces rather than fixed-dimensional Euclidean representations. We formalize the functional representation hypothesis, where data are explained through sparse compositions of structured functions. Unlike standard SAEs that represent concepts with scalar activations, SAE-NOs parameterize concepts as functions, enabling representations that capture not only a concept's presence, but also how and where it is expressed across the input domain. We achieve this through joint sparsity: concept sparsity selects active concepts, while domain sparsity governs where they are expressed. We instantiate this framework using Fourier neural operators (SAE-FNOs), parameterizing concepts as integral operators in the Fourier domain. This functional and spectral parameterization is particularly advantageous when data exhibit spatial structure across scales or when concepts are frequency-structured. We characterize SAE-FNO on vision data and demonstrate that it learns localized patterns, uses concepts more efficiently, and exhibits stable concept characteristics across sparsity levels. We further show that SAE-FNO adapts to changes in domain size and generalizes across discretizations, operating at resolutions beyond those seen during training, where standard SAEs fail. We also introduce lifting into SAEs and show theoretically and empirically that it acts as a preconditioner that accelerates optimization. Overall, our results show that moving from vector-valued to functional parameterizations, with concept and domain sparsity, extends SAEs from representing concept presence to modeling structured concept expression, highlighting the importance of parameterization.
Persona Vectors: Monitoring and Controlling Character Traits in Language Models
This paper was submitted as an educational post. Full credit goes to the authors, Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey, and the original post can be found at the provided URL. Language models interact with users through a simulated assistant persona that may sometimes deviate from ideal behavior. This research identifies directions in the model's activation space—called persona vectors—corresponding to traits like malevolence, sycophancy, and hallucination tendencies. The authors show these vectors can monitor personality fluctuations during deployment and predict personality shifts during training, with both intended and unintended post-finetuning changes correlating strongly with shifts along relevant persona vectors. The study proposes both post-hoc intervention and preventative steering approaches to mitigate unwanted changes, and demonstrates that persona vectors can flag training data likely to induce undesirable personality modifications. The extraction method is fully automated and works for any trait given only a natural-language description.
Why AI Chatbots Lie to Us
A few weeks ago, a colleague of mine needed to collect and format some data from a website, and he asked the latest version of Anthropic's generative AI system, Claude, for help. Claude cheerfully agreed to perform the task, generated a computer program to download the data, and handed over perfectly formatted results. The only problem? My colleague immediately noticed that the data Claude delivered was entirely fabricated.
LLMs Encode Harmfulness and Refusal Separately
This paper was submitted as an educational post. Full credit goes to the authors, Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau, and Weiyan Shi, and the original post can be found at the provided URL. This research demonstrates that language models develop distinct internal representations for recognizing harmful content versus refusing to engage with it. The team identified a "harmfulness direction"—separate from the previously documented "refusal direction"—showing these concepts are encoded independently in model activations. Key findings indicate that steering along the harmfulness direction leads LLMs to interpret harmless instructions as harmful, while steering along the refusal direction elicits refusal without reversing the model's judgment on harmfulness. This reveals that certain jailbreak techniques succeed by suppressing refusal signals while leaving the model's internal sense of harm intact. The authors developed "Latent Guard," a safety classifier leveraging these harmfulness representations, reporting performance matching or exceeding Llama Guard 3 8B across multiple attack methods.
Instruction Following by Boosting Attention of Large Language Models
Large language models' behavior is often shaped by instructions such as system prompts, refusal boundaries, privacy constraints, and tool-use rules that must hold at inference time. Yet in practice these constraints can be violated under long contexts or when user-provided context conflicts with them, creating reliability and safety risks. This motivates inference-time interventions that strengthen instruction influence without retraining. One such intervention is attention steering, which biases attention toward instruction tokens. In this work, we present a unifying theory for attention steering methods by formalizing instruction following as rule-based competition between instruction rules and context-derived rules, with attention mediating which rules dominate. We prove that boosting attention to instruction tokens tilts this competition, making it harder for context to override instruction-following. However, excessive boosting can suppress task-relevant context that should be incorporated alongside the instruction. Guided by this theory, we propose Instruction Attention Boosting (InstABoost), a simple intervention that applies a constant additive bias to instruction-key attention logits across all layers and heads. We evaluate InstABoost against prompting, latent steering, and prior attention steering methods across 15 tasks. InstABoost matches or outperforms all baselines while avoiding the fluency collapse of latent methods and the instruction over-focus of prior attention methods, achieving a stronger steering-quality tradeoff.
Prover Agent: An Agent-Based Framework for Formal Mathematical Proofs
We present Prover Agent, a novel AI agent for automated theorem proving that integrates large language models (LLMs) with a formal proof assistant, Lean. Prover Agent coordinates an informal reasoning LLM, a formal prover model, and feedback from Lean while also generating auxiliary lemmas. These auxiliary lemmas are not limited to subgoals in the formal proof but can also include special cases or potentially useful facts derived from the assumptions, which help in discovering a viable proof strategy. It achieves an 88.1% success rate on MiniF2F and solves 25 problems on the PutnamBench with a smaller sample budget than previous approaches, establishing a new state-of-the-art on both benchmarks among methods using small language models (SLMs). We also present theoretical analyses and case studies that illustrate how these generated lemmas contribute to solving challenging problems.
Understanding Reasoning in Thinking Language Models via Steering Vectors
Full credit goes to the original author, linked below. All research papers were reposted either with permission of the author, or by anonymous submission by SAIRC members like yourself. Recent advances in large language models (LLMs) have led to the development of thinking language models that generate extensive internal reasoning chains before producing responses. While these models achieve improved performance, controlling their reasoning processes remains challenging. This work presents a steering approach for thinking LLMs by analyzing and manipulating specific reasoning behaviors in DeepSeek-R1-Distill models. Through a systematic experiment on 500 tasks across 10 diverse categories, we identify several reasoning behaviors exhibited by thinking models, including expressing uncertainty, generating examples for hypothesis validation, and backtracking in reasoning chains. We demonstrate that these behaviors are mediated by linear directions in the model's activation space and can be controlled using steering vectors. By extracting and applying these vectors, we provide a method to modulate specific aspects of the model's reasoning process, such as its tendency to backtrack or express uncertainty. Our approach offers practical tools for steering reasoning processes in thinking models in a controlled and interpretable manner. We validate our steering method using three DeepSeek-R1-Distill models, demonstrating consistent control across different model architectures.
Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models
This paper was submitted as an educational post. Full credit goes to the authors, James Chua, Jan Betley, Mia Taylor, and Owain Evans, and the original post can be found at the provided URL. The researchers investigate whether emergent misalignment—where models finetuned on narrow malicious behaviors become broadly misaligned—extends to reasoning models. When reasoning models were finetuned on harmful behaviors with reasoning disabled, then re-enabled during evaluation, the models exhibited broad misalignment: deceptive or false answers, expressed desires for tyrannical control, and resistance to shutdown. The study finds that reasoning traces can both expose and conceal harmful intentions through benign-sounding rationalizations, complicating detection. The team also examines sleeper agent models activated only by specific triggers, finding these models demonstrate a kind of self-awareness about their own backdoors. Three domain-specific datasets (medical, legal, security) are released to facilitate further research.
Convergent Linear Representations of Emergent Misalignment
This paper was submitted as an educational post. Full credit goes to the authors, Anna Soligo, Edward Turner, Senthooran Rajamanoharan, and Neel Nanda, and the original post can be found at the provided URL. The researchers investigate how large language models develop unintended harmful behaviors when fine-tuned on narrow datasets. Using a minimal experimental setup with rank-1 adapters on Qwen2.5-14B-Instruct, their key finding is that different emergently misaligned models converge to similar representations of misalignment. By extracting a misalignment direction from one model's activations, they successfully ablated problematic behaviors in other fine-tuned variants. Their analysis reveals that six adapters contribute to general misalignment while two specialize in domain-limited misalignment. The work aims to advance understanding of misalignment mechanisms to better mitigate such issues in model development.