Understanding the Problem

GitHub Copilot, like other AI coding agents, faces challenges with token count and context preservation. Shorter tool responses can sometimes require additional calls or work, making tasks slower and more expensive.

The Solution

To address these issues, GitHub Copilot implemented four key changes: preserving useful context, removing repetitive output, shortening instructions, and delivering completed background work. These changes were evaluated using agentic coding benchmarks and validated through controlled online experiments.

Compressing Repetitive Output

The team developed a selective output compressor that preserves source-like and arbitrary output, reorganizes search results, and selectively compresses repetitive noise. This approach ensures that the model does not need to repeat work and maintains the necessary context for task completion.

Removing Formatting

The view tool, which agents use to read file contents, no longer prefixes lines with numbers, aligning with current file-editing tools that do not rely on line numbers for change targeting.

Evaluating the Changes

The changes were tested across various open-source repositories and benchmarks. The results showed that these optimizations led to a reduction in tokens per tool call and improved task success rates without sacrificing task quality.

Future Work

GitHub Copilot plans to continue refining these optimizations and exploring additional ways to improve efficiency and cost-effectiveness in AI coding.