What is Format Sensitivity Index?

A new article at arXiv 2607.09665v1The amazing sensitivity of large language models (LLM) illustrates the differences of small formatting in appearance. Prompt 'wrapper's can change the leadership tables even compared with stunning impact of model points. To measure this critical situation Format Sensitivity Index (FSI) and Separation Sensitivity Index (PSI) metrics were introduced. These new indices are the potential to reshape the future of LLM performance evaluations.

What is the Prompt Wrapper Sensitivity?

The prompt wrappers used while evaluating large language models; i.e. separators, JSON instructions or step-by-step templates, directly affect the figure that model understands and produces response. These differences that appear insignificantly can significantly change the accuracy points of the model. Even they can create great effects until a model will fully reverse its location in the leadership table. This precision poses a serious problem for developers and researchers who want to specifically compare and model comparisons.

Format Sensitivity Index (FSI) and Separation Sensitivity Index (PSI)

There are two complement metrics developed as a solution to this problem. Format Sensitivity Index (FSI) gauges accuracy range from choice of prompt wrapper. In the study conducted, FSI’s models differ from 30 floors. This means some of them show very high precision while they are almost never affected without formatting. The Distinctive Sensitivity Index (PSI) specifies the correct range of model respondents by the response extractor. PSI signals compatibility failures as the main reason why you’re big in FSI; if the output of the model is not accurately decomposed, the result is considered incorrect.

Token Controlled Protocol and Comprehensive Working

This comprehensive research was conducted on 4 different instruction models ranging from 7 questions-cevap task, 5 wrapper family and 7B to 72B parameter. In total, 140.000 OpenRouter production is analyzed. The study was run under a private token controlled protocol that fills prompts into a fixed character budget and saves the actual prompt token numbers. Tokens are the basic units used to process the text of large language models, directly effective on performance and cost. Platforms like OpenRouter allow you to analyze such large-scale LLM interactions.

Relationship with Accuracy of Separation and Changes in Model Performance

Analyses have proved to be a strong indicator for accuracy of separation even after being controlled by task, model and wrapper. Low decomposability is often resulted in zero-to-close accuracy. In the study, the same model has been observed in extreme examples, which can reach the accuracy of almost random accuracy under a strict JSON wrapper, under a separator-based-structured wrapper. This situation suggests not only the compatibility of the output, but also the ability of the model to perform logic. Precision differences on model are remarkable; for example, when the Qwen-2.5-72B model is almost never affected by the mean 0.024 FSI, the Phi-4 model was sensitive with average 0.763 FSI.

What is the Community

Supporters

Researchers and developers think that metrics like FSI and PSI will add transparency and reliability to the benchmarking processes. These indices allow more fair and realistic assessments by revealing hidden variable in model performance. Adopting the fact that the results of restriction vary according to formatting, more robust and repeatable methodology Development k for the first step is seen.

News

Some experts indicate that such metrics can create an additional load on model developers and already complex LLM evaluation processes can be more difficult. For each model and task, finding the optimal wrapper can be time-consuming and cost-effective, they wish that this situation can create access barrier for small teams. They also pay attention to the challenges of creating a standard wrapper set.

Frequently Asked Questions

What is Format Sensitivity Index (FSI) exactly?

Format Sensitivity Index (FSI) is a metric that measures how sensitive a large language model (LLM) accuracy scores, the prompt formatting (wrapper) changes used. High FSI shows that the model is easily affected without formatting.

What does the separation sensitivity index (PSI) work?

Distinctive Sensitivity Index (PSI) gauges the variation of model responses accurately by a benchmarking tool. PSI often reveals compatibility failures, which are the main cause of large changes in FSI.

Why does these indices affect LLM comparisons?

These indices show that existing leadership tables are sensitive to small changes in prompt formatting because they increase the reliability of LLM benchmarks. Wrapper reveals that accuracy reporting is statistically brittle without causing and compatibility issues.

How can the model developers benefit from these findings?

Model developers can use these findings to make their models more robust and make it more resistant to different prompt formats. In addition, practical recommendations such as multi-wrapper reporting for auditability of results and publishing templates publicly, help improve model distribution processes.

Measuring the performance of large language models accurately is a fundamental obligation for the progress of the artificial intelligence area. New metrics such as Format Sensitivity Index and Discreteability Responsiveness Index, adding a scientific layer to this complex evaluation process, giving the model benchmarks to the future. Better understanding and management of formatting precision will ensure that more reliable and comparable LLM results are achieved.