Background The adequacy of leptomeningeal collateral flow has a pivotal role in determining clinical outcome in acute ischemic stroke. The American Society of Interventional and Therapeutic Neuroradiology/Society of Interventional Radiology (ASITN/SIR) collateral score is among the most commonly used scales for measuring this flow. It is based on the extent and rate of retrograde collateral flow to the impaired territory on angiography.

Objective To evaluate inter- and intraobserver agreementin angiographic leptomeningeal collateral flow assessment.

Materials and methods Thirty pretreatment angiogram video loops (frontal and lateral view), chosen from the randomized controlled trial THRombectomie des Artères CErebrales (THRACE), were sent for grading in an electronic file. 19 readers participated, including eight who had access to a training set before the first grading. 13 readers made a double evaluation, 3 months apart.

Results Overall agreement among the 19 observers was poor (κ = 0,16 ± 6,5.10 ^-3), and not improved with prior training (κ = 0,14 ± 0,016). Grade 4 showed the poorest interobserver agreement (κ=0.18±0.002) while grades 0 and 1 were associated with the best results (κ=0.52±0.001 and κ=0.43±0.004, respectively). Interobserver agreement increased (κ = 0,27± 0,014) when a dichotomized score, ‘poor collaterals’ (score of 0, 1 or 2) versus ‘good collaterals’ (score of 3 or 4) was used. The intraobserver agreements varied between slight (κ=0.18±0.13) and substantial (κ=0.74±0.1), and were slightly improved with the dichotomized score (from κ=0.19±0.2 to κ=0.79±0.11).

Conclusion Inter- and intraobserver agreement of collateral circulation grading using the ASITN/SIR score was poor, raising concerns about comparisons among publications. A simplified dichotomized judgment may be a more reproducible assessment when images are rated by the same observer(s) in randomized trials.