AI FRONTIER
Signals worth understanding before they become obvious.
Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation
The paper reports that Arena-style rankings, which aggregate pairwise comparisons of anonymous models via the Bradley-Terry model, are vulnerable to strategic voting, strategic prompting, and strategic nomination.
- The paper demonstrates strategic voting, strategic prompting, and strategic nomination on public leaderboard data.
- The paper reports each attack can boost a model by up to 15-20 places in a ranking, and combined they can boost a model by up to 45 places.
- The paper states these issues hold for any preference-based evaluation using the Bradley-Terry model that allows open prompting, voting, or model submission.
Teams building or relying on Arena-style leaderboards need anti-manipulation safeguards across voting, prompting, and model submission, or rankings can be strategically inflated by providers.
Robust Task-Aware State Estimation from Pre-trained Perception
The paper proposes a θ-confidence Minimal Belief Set (MBS_θ), built from probabilities a perception model already produces, such as a VLM's output-layer token distributions, as the minimal subset of states whose cumulative probability meets a user-specified belief-mass threshold, along with the PRaVaSS algorithm.
- The paper is by Guy Azran and Sarah Keren and appears at NeurIPS 2026.
- MBS_θ is defined as a minimal subset of states whose cumulative probability meets a user-specified belief-mass threshold.
- PRaVaSS constructs an MBS_θ with correctness guarantees and polynomial complexity in the unconstrained case.
- Using a VLM as the backbone of the state estimator, the authors validate theoretical properties, scalability with problem size, and improved downstream decision-making in a task-planning pipeline.
For agent builders, retaining the perception model's probabilities instead of determinizing to a single best guess can improve decision robustness without maintaining a full symbolic state distribution.
Together Compiler: Continual Compilation for AI Inference
Together AI introduces Together Compiler (TKCC), a continual agent loop for AI inference optimization that pairs a Tiered Persistent Engine with a cost model for selecting parallelism configurations to ease the throughput-fidelity tradeoff.
- The paper reports TKCC outperforms highly optimized hand-written Megakernels on the Llama 1B workhorse workload by up to 1.7x.
- The paper reports up to 2.6x higher TPS than the best open source engines on the InferenceX benchmark across hardware platforms for frontier models Kimi K3, GLM 5.3, MiniMax M3.1 Flash, and DeepSeek v4.1 Flash.
- The paper reports state-of-the-art batch size 1 performance of 1111 TPS on M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash.
- The paper states TKCC serves trillions of tokens across Together AI's production fleet.
For inference serving teams, TKCC frames compilation as a continuous online loop rather than offline kernel tuning, though its speedups are paper-reported rather than independently verified.
Cleaning an Agent's History Hurts When the World Still Holds the Error
The paper freezes an agent mid-task with its exact world state on AgentCE-Bench and AppWorld, edits either the history or the world, and lets a receiver model continue, finding that a history edit's effect depends on the world it describes.
- AgentCE-Bench: after the agent commits a wrong value, holding the world fixed and adding a truthful note beside the preserved error raises recovery from 51% under a matched neutral note to 66%.
- AgentCE-Bench: adding the agent's own completed action instead lowers recovery below the unedited history, from 50% to 44%.
- AppWorld: after a database write, a success receipt lowers whole-task success from 45% to 28% while the effect is absent and raises it from 38% to 49% once present.
- The paper reports the two effects differ by 27 points pooled over two receivers, positive in six of eight receivers at 97.5% and seven at 95%.
Builders of agent memory compaction or handoff systems must keep history edits consistent with the world's actual state, since a history made to look clean while the world still holds the error can leave an agent worse off than no edit at a...
SynthCorp: A Platform for Creating Benchmarks for Data Agents
The paper presents SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time, and uses it to generate initial benchmarks of seven data estates and 256 tasks.
- Each SynthCorp data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs.
- The initial benchmarks comprise seven data estates (up to 1K operational tables) and 256 tasks across five domains.
- The paper reports that the data estates pass deterministic pipeline checks such as foreign-key violations, undeclared writes, recurrence violations, and database execution failures with a near-zero error rate.
- The paper reports evaluating three models across four data-context settings, showing the benchmarks separate data contexts and model tiers.
For teams building enterprise data agents, SynthCorp offers a customizable benchmark-generation path that verifies gold programs without manual annotation.
Yao Shunyu and Yang Zhilin co-author first joint paper: no AI independently solved any of 30 world-class math problems
A first joint paper by Yao Shunyu and Yang Zhilin reports that none of 30 world-class math problems was solved independently by an AI.
- The paper is the first joint publication by Yao Shunyu and Yang Zhilin.
- The paper covers 30 world-class math problems.
- The paper reports that none of the 30 problems was solved independently by an AI.
For AI builders, the reported result points to a clear gap in current models on math problems that must be solved independently.