preprint · arXiv (Cornell University) · 2025
The role of positional encodings in the ARC benchmark
See where this sits in the topic map →Summary AI-generated
- TL;DR
- The Abstraction and Reasoning Corpus tests AI systems on abstract reasoning tasks with minimal training data—a challenge for machine learning models despite being intuitive for humans.
- Problem
- Limitations in how transformer models handle positional encoding hinder their abstract reasoning performance.
- Method
- Using CodeT5+ as a case study, this work examines the role of positional encoding across transformer architectures of varying sizes and configurations.
- Results
- The study demonstrates that positional encoding limitations directly impact reasoning performance on the ARC benchmark.
- Contributions
- Not specified in the abstract.
- Limitations
- Not specified in the abstract.
- Takeaways
- While 2D positional encoding and Rotary Position Embedding offer competitive results, 2D encoding excels in data-constrained scenarios, highlighting its effectiveness for ARC tasks.
- Applications
- Not specified in the abstract.
- Topics
- Abstract reasoning, transformer architectures, positional encodings, machine learning
- For industry
- Not specified in the abstract.
- Why it matters
- Not specified in the abstract.
Abstract
The Abstraction and Reasoning Corpus challenges AI systems to perform abstract reasoning with minimal training data, a task intuitive for humans but demanding for machine learning models. Using CodeT5+ as a case study, we demonstrate how limitations in positional encoding hinder reasoning and impact performance. This work further examines the role of positional encoding across transformer architectures, highlighting its critical influence on models of varying sizes and configurations. Comparing several strategies, we find that while 2D positional encoding and Rotary Position Embedding offer competitive performance, 2D encoding excels in data-constrained scenarios, emphasizing its effectiveness for ARC tasks