Background HER2 expression level is a key factor in determining the optimal treatment course for breast cancer patients. Roughly 15% of breast cancers are HER2(+), and determination of HER2 status is routinely assessed by immunohistochemistry (IHC). Accurate assessment of the HER2 IHC score by pathologists is therefore critical, especially in light of novel therapeutic approaches demonstrating efficacy in the HER2-low setting. However, there is an opportunity to improve inter-pathologist agreement at the lower levels of HER2 scoring (0, 1+, and 2+).

Methods A machine learning model (AIM-HER2) was developed to generate accurate, slide-level HER2 scores aligned with ASCO-CAP guidelines in clinical breast cancer HER2 IHC specimens. AIM-HER2 was assessed as an AI-assist tool in a retrospective reader study, where 20 HER2-trained pathologists scored breast cancer cases (N=200) with and without AIM-HER2 assistance using a 2-cohort crossover design with a 3-week washout. A separate panel of 5 expert HER2 pathologists read all 200 cases manually to establish reference scores.

Results In a significant fraction of cases examined, less than 70% inter-pathologist agreement was observed. When used as an AI assist tool, AIM-HER2 improved inter-rater agreement overall and specifically at the 0/1+ and 1+/2+ cutoffs. Similarly, AIM-HER2 AI-assist significantly increased PPA at the 0/1+ and 1+/2+ cutoffs. When interacting with the AI-assist tool, pathologists displayed a wide range of override rates, and the quality of a pathologist’s overrides was correlated with their manual accuracy. Lastly, the impact of the reference panel on AIM-HER2 accuracy metrics was assessed, revealing that measurements of model accuracy are highly dependent on reference panel composition.

Conclusions The use of AIM-HER2 as an AI-assist tool for scoring HER2 IHC in breast cancer may improve pathologist reproducibility and accuracy, particularly at the 0/1+ and 1+/2+ cutoffs.

All authors are current or former employees of, received salary and stock options from PathAI, Inc.

This study did not receive any funding.

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Anonymized breast cancer HER2 IHC specimens and corresponding digitized WSIs were obtained from Cleveland Clinic, Dana Farber Cancer Institute, and Johns Hopkins Hospital as part of a data sharing agreement between PathAI and each institution. As part of each respective agreement, the Cleveland Clinic Institutional Review Board, the Dana Farber Cancer Center Institutional Review Board, and the Johns Hopkins Medicine Institutional Review Board approved the use of these samples for this study.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Access to HER2 prediction heatmaps is available upon reasonable request to academic investigators without relevant conflicts of interest for non-commercial use who agree not to distribute the data. Access requests can be made to publicationspathai.com.