Suppose your spreadsheet assistant retrieves the cell containing the answer and still gets the answer wrong. Before replacing the retriever, inspect what reached the model: did it receive the value alone, or the headers and units that explain it?
Retrieval-augmented generation (RAG) gives a model retrieved evidence to answer a question. The spreadsheet pipeline discussed here is described in section 2.3 of Smoleń’s preprint. For a spreadsheet, finding the relevant value is only one requirement. The evidence must also preserve what that value refers to, and the model must use it correctly.
The useful distinction is between evidence found, meaning preserved, and answer correct. Treating those as separate checks gives you a more specific investigation than “RAG failed.”
The cell is present. Its meaning is missing.
Consider a constructed example, not an experiment from a paper. You ask: “How many returns did West record in Q2 FY2025, actual rather than budget?”
The retrieved chunk contains 12.
That might be the right cell. It does not tell the model whether the number is returns or revenue, West or East, actual or budget. It does not even establish the unit. A context-preserving record would look like this:
| Field | Illustrative evidence |
|---|---|
| Value | 12 |
| Row path | West → Returns |
| Column path | FY2025 → Q2 → Actual |
| Units | Thousands of returns |
| Table identity | Illustrative regional returns report |
Here, “12 thousand returns” answers the question; the bare number does not. A retrieval check that notices the correct cell can miss the context loss around it. Conversely, delivering the complete record does not guarantee that the model will select the right period, preserve the units, or perform a calculation correctly.
Inspect the exact chunk delivered to the generator. A header visible in the workbook is not evidence that the model received it.
What the spreadsheet study actually compares
Zofia Smoleń’s preprint, Q&A on Any Spreadsheet Requires Interpreting Its Grid Structure, examines spreadsheet structure in a RAG pipeline. Its Figure 1 illustrates how different chunk representations can contain the same answer cell while preserving different amounts of context. The grid-aware Row representation retains the sheet title and complete header paths. That is the paper’s illustration, separate from the constructed returns example above. Figure 1 and section 2.4
The evaluation covers 480 questions targeting 80 answer sheets within a corpus of 382 sheets. Generation uses one configuration: gemini-2.5-flash, temperature zero, thinking disabled, with the top five retrieved chunks. These are author-reported preprint results, not findings independently replicated here or verified as peer-reviewed. Sections 2.1 and 2.3; Appendix F.1
Two comparisons matter, and they answer different questions.
First, the baseline comparison reports improvements in retrieval as well as answer ratings. Table 2 gives these results for the STC baseline and the GAT configuration:
| Reported metric | STC | GAT |
|---|---|---|
| Recall@1 | 0.356 | 0.465 |
| Recall@5 | 0.515 | 0.640 |
| Human answer rating, 1–5 | 3.43 | 3.88 |
The answer ratings are scores on a five-point scale, not accuracy percentages. This comparison does not support “retrieval stayed the same.” Retrieval improved against the baseline. Section 2.3, Table 2
Second, the paper compares sixteen deployed checkpoints using the same Row pipeline. In that comparison, answer ratings are associated with role-label quality, measured by macro-F1, but the author reports no detectable retrieval-recall trend with that quality measure. This is a different comparison from Table 2, not a contradiction of it. Nor does “no detectable trend” establish that retrieval never matters. Appendix E also documents unequal fallback rates across checkpoints, so the association should not be treated as clean causal isolation of role quality. Section 2.4, Figure 4; Appendix E
There is a rating-method boundary, too. The main answer scores are human-rated. The auxiliary experiment that removes cell-role information uses an LLM-judge panel. Its panel-score differences should not be presented as human-rating changes or compared directly with the absolute human scores above. Section 2.5, Table 4; Appendices F.2 and G
The practical reading is narrower than “structure solves RAG”: evaluate whether retrieved evidence remains interpretable, rather than assuming a retrieval metric captures answer quality.
A diagnostic before a retriever replacement
The following is a proposed operator diagnostic, not a validated production fix or a protocol tested by the paper.
Start with one incorrect answer and the evidence actually delivered to the model. Then separate three cases:
- The necessary evidence is absent. Investigate ingestion, chunking, filters, and retrieval. Check whether the source information entered the searchable corpus and whether the retrieval path could return it.
- The value is present, but its meaning is absent or ambiguous. Inspect row and column header paths, units, period, and table identity. The question is whether extraction and representation retained enough context to distinguish this value from neighboring values.
- The evidence and its context are present, but the answer is wrong. Investigate interpretation, calculation, and generation. Do not declare retrieval or structure the sole cause simply because that was your first hypothesis.
For a local test, hold the question and generation settings fixed. Compare the isolated evidence with a context-preserving version. Save both delivered chunks and both answers, including any transformations or truncation between retrieval and generation.
If the answer improves, you have a useful local result to investigate, not proof that the same repair transfers to every workbook. If it does not, the preserved context still lets you ask a sharper question about how the model used the evidence. No such local experiment was run for this article.
Flat tables are the counterexample
Added structural machinery needs a reason to exist. On simple flat tables, the paper reports little advantage from learned roles: Table 5 lists answer scores of 3.82 for STC, 3.81 for GAT, 3.91 for DualModalityGNN, and 3.92 for the gold-role oracle. That supports a limited-benefit reading for those configurations, not a claim that every method ties or that graph models are necessary for every spreadsheet. Section 2.5, Table 5
The evaluation also does not cover literally every spreadsheet. Appendix D describes excluding tabs deemed uninterpretable, and the conclusion says the benchmark will be released separately as a dataset publication. Its release and independent reproducibility have not been verified here. Combined with the single generator configuration, these are reasons to test the diagnostic on your own documents rather than transfer the scores directly. Conclusion and Appendix D
For a flat table with explicit, unambiguous labels, a simpler representation may be sufficient. For nested headers and repeated labels, context preservation deserves closer inspection. That is an engineering judgment to evaluate against observed failures, not an architecture recommendation proved by this one preprint.
Keep an evidence record, not just a search score
For the next wrong spreadsheet answer, record:
- The exact question and expected answer, including units.
- The workbook, sheet, and source cell or range.
- The complete row path and column path.
- Units, reporting period, and table identity.
- The exact chunk delivered to the generator.
- The generated answer and relevant generation settings.
- Whether evidence was missing, context was missing, or complete evidence was used incorrectly; mark unresolved cases as unresolved.
This record is a proposed debugging aid, not a new benchmark. It makes the failure inspectable without pretending that all wrong answers share one cause.
Take one incorrect answer from your own workflow and inspect that delivered chunk before changing the retriever. The next useful change depends on whether the system failed to find the evidence, lost its meaning, or answered incorrectly despite having both.