Validation vs. Verification for Whole Slide Imaging (WSI) is it Really a Grudge Match?

grudge match.jpg

Q:  My practice would like to start using Whole Slide Imaging for Primary Diagnosis.  Does my lab have to undergo verification or validation? What’s the difference between Validation and Verification for Whole slide imaging?

A:  The CAP’s definition of validation is: “…Validation:…the process used to confirm with objective evidence that a laboratory developed or modified FDA-cleared/approved test method or instrument system delivers reliable results for the intended application.”

Their definition of verification is: “…Verification….the process by which a laboratory determines that an unmodified FDA- cleared/approved test performs according to the specifications set forth by the manufacturer when used as directed.”

So validation is the process for laboratory developed tests or FDA test systems that have been modified.  Verification is the process for FDA approved systems used according to the manufacturer.  There is only one WSI system that is FDA cleared in the USA, the Philips IntelliSite Pathology Solution.  This system includes the Philips UF Scanner, Philips Display (Viewer), Philips Image Management System, and the monitors used by Philips in the FDA application (Barco, medical grade). **Note of importance - The Philips system is only approved for formalin fixed paraffin embedded H&E slides.

If a laboratory uses another viewer, image management system or monitor with the Philips scanner, it is now an FDA modified system and the laboratory must validate the same way as they would if they were using any other WSI system. The CAP’s definition of the modification of an FDA test is included at the end of this blog. But even if a laboratory uses the FDA approved system according to the manufacturer, it must still undergo a process to ensure the accuracy of the system within each laboratory. Philips states in its FDA letter that the “PIPS is not intended for use with frozen section, cytology, or non-FFPE hematopathology specimens.” Any of these specimens would still require a laboratory to validate the system for those specimen types the same as they would if they were to use any other WSI solution.

Q: What do the CAP Accreditation Checklists Say about Verification or Validation of WSI?

A: The CAP’s Lab General Checklist August 2018 revisions state:

“GEN.52920 Whole Slide Imaging System Validation/Verification Phase I

The laboratory validates or verifies whole slide imaging systems used for clinical diagnostic purposes by performing its own studies, including approval for use by the laboratory director (or designee who meets CAP director qualifications) before the technology is used for the intended diagnostic purpose(s).

NOTE: The specific components of validation or verification studies are left to the discretion of the laboratory. The approval or clearance of whole slide imaging systems by the FDA does not obviate the need for individual laboratories to verify the performance of these systems for the specific intended diagnostic purposes.

As general guiding principles, the validation or verification process should: “Closely emulate the real-world clinical environment and involve specimen preparation types and clinical settings relevant to the intended use(s); Be carried out by a pathologist(s) adequately trained to use the system; Assess intraobserver concordance between digital and glass slides; Encompass the entire whole slide imaging system, with reevaluation if a significant change is made to a previously validated or verified system. “

Summary for Implementing WSI Imaging for primary diagnosis is: The Checklist summarizes the CAP Guidelines for Implementing Whole Slide Imaging which are that a) the lab must conduct its own studies of accuracy, b) the Laboratory Medical Director must be involved and sign off on the studies, c) the study must emulate the intended use and clinical settings, d) that pathologists be adequately trained, e) that intra-observer concordance between glass and digital be assessed, f) that the study encompasses the entire system and g) revaluation takes place if there is a significant change, like a new scanner.

The complete CAP guidelines for implementing WSI can be found below.

Q: What does it mean to modify an FDA approved/cleared test or test system?

A: This is the CAP’s definition:  “Any change to the manufacturer's supplied ingredients or modifications to the assay as set forth in the manufacturer's labeling and instructions. It may include a change to specimen type, instrumentation or procedure that could affect its performance specifications for sensitivity, specificity, accuracy, or precision or any change to the stated purpose of the test, its approved test population, or any claims related to interpretation of the results.” 

For example, if a practice decided it was not necessary to purchase medical grade monitors to perform primary diagnosis on WSI, this would constitute a modification; and thus, a validation study would need to be performed.

Q: Is it easier to verify an FDA cleared test system for primary diagnosis with WSI or to validate a Lab developed WSI test system or a modified FDA cleared test system? 

A:  The verification procedure for an FDA cleared test system will depend upon the instructions from the manufacturer and will be depend on the Lab Medical Director’s determination that the test system is accurate.  A Lab Medical Director may likely elect to follow the CAP Guidelines for WSI.  In which case, the amount of effort for either verification or validation would be the same.

Q: In the event of a malpractice lawsuit involving a diagnosis performed WSI, is a pathologist better protected using an FDA cleared system?

A: In general, the pathologist or laboratory would still be responsible for a misdiagnosis, no matter the means the diagnosis was rendered. In the event of a misdiagnosis using an FDA cleared WSI system, it is unclear to what extent, if any, a pathologist would be protected.

CAP guidelines for WSI Validation for Primary Diagnosis.png

REFERENCES

1)  Pantanowitz et al, Validating whole slide imaging for diagnostic purposes in pathology: Guideline from the College of American Pathologists Pathology and Laboratory Quality Center. ARPA, 2013 http://www.archivesofpathology.org/doi/pdf/10.5858/arpa.2013-0093-CP http://www.cap.org/apps/docs/membership/wsi_faqs.pdf 


CAP WSI Validation Guidelineshttps://www.cap.org/protocols-and-guidelines/cap-guidelines/current-cap-guidelines/validating-whole-slide-imaging-for-diagnostic-purposes-in-pathology