Reducing Inter-Reader Variability in RECIST 1.1 Assessments: A Core Lab Methodology
Dr. Arjun Kalyanpur
Founder & Chief Radiologist, Image Core Lab
Executive Summary
RECIST 1.1 provides a standardized framework for assessing tumor response, but application of standardized criteria can still result in inter-reader variability. Independent double reads in oncology trials disagree routinely, as a structural feature of subjective visual assessment rather than an occasional lapse. For sponsors and CROs, that variability feeds directly into progression-free survival curves, regulatory submissions, and go/no-go decisions.
A 2019 peer-reviewed study involving investigators from Image Core Labs and Teleradiology Solutions, including our Chief Radiologist and Founder, Dr. Arjun Kalyanpur, quantified interobserver variability in a completed multicentric breast cancer trial..[1] That research has directly shaped how adjudication is structured here today.
This paper outlines the methodology used to manage that variability: how double reads are structured, when adjudication is triggered, and what a sponsor should look for when evaluating whether a vendor’s process actually controls discordance or simply documents it after the fact.
The Problem: The Impact of Inter-Reader Variability on Trial Endpoints
Response Evaluation Criteria in Solid Tumors (RECIST 1.1) exists to reduce subjectivity in oncology imaging endpoints, and in practice it does. It does not eliminate it. Two radiologists reading the same scan under the same criteria will disagree at a rate high enough to matter at the trial level.
The clearest external evidence comes from a 2021 analysis of six lung cancer trials using blinded independent central review (BICR), spanning 1,833 patients and over 10,600 time points read by 17 radiologists. The trials showed significantly different discrepancy rates, averaging 34.3% at the time-point level and 59.2% at the patient level. When the analysis focused specifically on progressive-disease calls, discrepancy rates settled at a more consistent 32.9%, while individual reader endorsement rates ranged from 27.7% to 77.8%, a spread wide enough that the reader’s identity, not just the image, was shaping the outcome.[2]
The same study found that 74.2% of adjudications stemmed from medically justifiable differences in interpretation rather than reader error: ambiguous lesion margins, borderline measurability calls, and legitimate differences in how new lesions are characterized. Variability is structural to the task itself, not a training gap.[2]
ICL’s own research team, led by Founder and Chief Radiologist Dr. Arjun Kalyanpur, has studied this problem directly. In a 2019 peer-reviewed study, they retrospectively re-read 61 cases (82 data-pairs) from a completed multicentric breast cancer trial. Overall response agreement between two conventionally trained radiologists was only moderate (Cohen’s κ = 0.510; 95% CI, 0.344–0.676), with overall-response disagreement/variation of 25.6%.. The divergence traced mainly to differences in RECIST 1.1 interpretation and image reading. The study’s conclusion was direct: structured checklists, applied before adjudication, may meaningfully reduce this gap.[1]
Left unmanaged, this creates three concrete risks:
Regulatory risk
Timeline risk
Interpretive risk
The real question for a sponsor isn’t whether variability exists in a vendor’s reads. It always will. The question is whether the read process is structured to catch and resolve it before it reaches the dataset.
Core Lab Methodology: Structured Second Read & Adjudication
At Image Core Lab, every study is assessed under a structured Blinded Independent Central Review (BICR) framework that governs the entire read process.
This methodology traces directly to a 2019 study by ICL’s research team, led by Dr. Kalyanpur, documenting real interobserver variability in a completed multicentric trial and proposing a structured checklist to close the interpretation gap before adjudication was needed.[1] What follows is that recommendation, matured into standing practice across every protocol ICL reads.
Independent double read
Every time point is read independently by two board-certified radiologists, each blinded to the other's assessment, the local site's evaluation, and, where the protocol allows, treatment arm and clinical outcome data. Blinding removes the anchoring effect that occurs when a second reader sees a first reader's call before forming their own.
Concordance Check
Once both reads are locked, target lesion measurements, non-target assessments, and new-lesion calls are compared against pre-specified discordance thresholds. Reader variability commonly traces back to a defined set of factors: radiologist experience, image quality, and the accessibility of contextual clinical information, as well as lesion-selection differences and how new lesions are confirmed.[3] Structuring the concordance check around these known failure points, rather than a generic diff of numbers, is what separates a rigorous second read from a procedural one.
Adjudication
When discordance crosses the pre-specified threshold, a third, independent reader adjudicates. The adjudicator does not simply select between the two existing reads; they re-review the full imaging history for that patient and issue an independent, binding assessment. At ICL, this adjudication tier sits under the direct methodological oversight of the lab's Chief Radiologist, ensuring adjudication decisions are calibrated against a consistent standard across the entire protocol, not only within a single reader pair.
This kind of structured escalation reflects how often adjudication is actually needed in practice: in one Phase II trial, 36.7% of patients required adjudication, with differences in reporting new lesions accounting for over half of those cases (53.7%).[4]
Reader calibration
Discordance rates are only meaningful if they're tracked. Readers are monitored for individual discordance and endorsement rates over the life of the protocol, with corrective retraining triggered when a reader's pattern drifts from the group. This is the same structured-checklist principle the research team established, applied continuously rather than as a one-time fix.[1]
Data in Practice: A Representative Case
Consider a representative, composite scenario reflecting the discrepancy patterns commonly reported in BICR literature, rather than a single real trial record.
A patient’s Week 12 follow-up scan shows a new hepatic lesion at the borderline of measurability. Reader 1 characterizes it as a new lesion and calls progressive disease. Reader 2, reading independently, judges the same finding to be an artifact of slice thickness and does not call progression. This is exactly the category of disagreement the literature identifies as a leading cause of discordance: new-lesion detection and characterization, not measurement error alone.[3]
Under an unstructured process, this discrepancy could sit unresolved until database lock, or resolve arbitrarily based on whichever read was entered first. Under a structured adjudication process, the discordance is flagged immediately at the concordance check. A third reader re-reviews the full longitudinal imaging series, cross-references the lesion against baseline and interval scans, and issues a binding call. In this scenario, that call confirms progression based on interval growth visible on a follow-up scan that neither original reader had flagged as decisive.
The mechanism matters more than the specific outcome: the date of progressive disease attached to this patient’s record is the adjudicated date, not either individual reader’s date. Because the date of progressive disease directly affects survival-curve construction, a systematic process for resolving exactly this kind of disagreement is what protects the trial’s primary endpoint from reader-level noise.
A Framework for Evaluating Read Consistency
For sponsors and CROs evaluating, or auditing, an imaging vendor’s approach to variability, five questions separate a rigorous process from a documented one:
Reader qualification and calibration
Blinding integrity
Adjudication trigger threshold
Adjudicator independence and seniority
Transparency and audit trail
Conclusion
Inter-reader variability in RECIST 1.1 assessments is a structural feature of subjective visual interpretation, and it has to be actively managed accordingly. The trials that manage it well aren’t the ones with the least disagreement; they’re the ones with a process rigorous enough to catch disagreement early and resolve it consistently. For sponsors and CROs, the imaging endpoint is only as reliable as the read process behind it.
References
1. Karmakar A, Kumtakar A, Sehgal H, Kumar S, Kalyanpur A. Interobserver Variation in Response Evaluation Criteria in Solid Tumors 1.1. Academic Radiology. 2019;26(4):489–501. doi:10.1016/j.acra.2018.05.017
2. Beaumont H, Iannessi A, Wang Y, Voyton CM, Cillario J, Liu Y. Blinded Independent Central Review (BICR) in New Therapeutic Lung Cancer Trials. Cancers. 2021;13(18):4533. doi:10.3390/cancers13184533
3. Iannessi A, Beaumont H, Ojango C, Bertrand AS, Liu Y. RECIST 1.1 Assessments Variability: A Systematic Pictorial Review of Blinded Double Reads. Insights into Imaging. 2024. doi:10.1186/s13244-024-01774-w
4. Beaumont H, Evans TL, Klifa C, Guermazi A, Hong SR, Chadjaa M, Monostori Z. Discrepancies of Assessments in a RECIST 1.1 Phase II Clinical Trial — Association Between Adjudication Rate and Variability in Images and Tumor Selection. Cancer Imaging. 2018. doi:10.1186/s40644-018-0186-0
Dr. Arjun Kalyanpur
Founder & Chief Radiologist, Image Core Lab
A board-certified radiologist, Dr. Kalyanpur founded Image Core Lab and oversees its imaging methodology, reader calibration, and adjudication standards across sponsor and CRO engagements. His peer-reviewed research includes Karmakar et al., “Interobserver Variation in Response Evaluation Criteria in Solid Tumors 1.1” (Academic Radiology, 2019)[1] — a study quantifying real-world RECIST 1.1 reader variability that directly shaped the adjudication methodology described in this paper.