Validation vs. Verification for Whole Slide Imaging (WSI) is it Really a Grudge Match?

grudge match.jpg

Q:  My practice would like to start using Whole Slide Imaging for Primary Diagnosis.  Does my lab have to undergo verification or validation? What’s the difference between Validation and Verification for Whole slide imaging?

A:  The CAP’s definition of validation is: “…Validation:…the process used to confirm with objective evidence that a laboratory developed or modified FDA-cleared/approved test method or instrument system delivers reliable results for the intended application.”

Their definition of verification is: “…Verification….the process by which a laboratory determines that an unmodified FDA- cleared/approved test performs according to the specifications set forth by the manufacturer when used as directed.”

So validation is the process for laboratory developed tests or FDA test systems that have been modified.  Verification is the process for FDA approved systems used according to the manufacturer.  There is only one WSI system that is FDA cleared in the USA, the Philips IntelliSite Pathology Solution.  This system includes the Philips UF Scanner, Philips Display (Viewer), Philips Image Management System, and the monitors used by Philips in the FDA application (Barco, medical grade). **Note of importance - The Philips system is only approved for formalin fixed paraffin embedded H&E slides.

If a laboratory uses another viewer, image management system or monitor with the Philips scanner, it is now an FDA modified system and the laboratory must validate the same way as they would if they were using any other WSI system. The CAP’s definition of the modification of an FDA test is included at the end of this blog. But even if a laboratory uses the FDA approved system according to the manufacturer, it must still undergo a process to ensure the accuracy of the system within each laboratory. Philips states in its FDA letter that the “PIPS is not intended for use with frozen section, cytology, or non-FFPE hematopathology specimens.” Any of these specimens would still require a laboratory to validate the system for those specimen types the same as they would if they were to use any other WSI solution.

Q: What do the CAP Accreditation Checklists Say about Verification or Validation of WSI?

A: The CAP’s Lab General Checklist August 2018 revisions state:

“GEN.52920 Whole Slide Imaging System Validation/Verification Phase I

The laboratory validates or verifies whole slide imaging systems used for clinical diagnostic purposes by performing its own studies, including approval for use by the laboratory director (or designee who meets CAP director qualifications) before the technology is used for the intended diagnostic purpose(s).

NOTE: The specific components of validation or verification studies are left to the discretion of the laboratory. The approval or clearance of whole slide imaging systems by the FDA does not obviate the need for individual laboratories to verify the performance of these systems for the specific intended diagnostic purposes.

As general guiding principles, the validation or verification process should: “Closely emulate the real-world clinical environment and involve specimen preparation types and clinical settings relevant to the intended use(s); Be carried out by a pathologist(s) adequately trained to use the system; Assess intraobserver concordance between digital and glass slides; Encompass the entire whole slide imaging system, with reevaluation if a significant change is made to a previously validated or verified system. “

Summary for Implementing WSI Imaging for primary diagnosis is: The Checklist summarizes the CAP Guidelines for Implementing Whole Slide Imaging which are that a) the lab must conduct its own studies of accuracy, b) the Laboratory Medical Director must be involved and sign off on the studies, c) the study must emulate the intended use and clinical settings, d) that pathologists be adequately trained, e) that intra-observer concordance between glass and digital be assessed, f) that the study encompasses the entire system and g) revaluation takes place if there is a significant change, like a new scanner.

The complete CAP guidelines for implementing WSI can be found below.

Q: What does it mean to modify an FDA approved/cleared test or test system?

A: This is the CAP’s definition:  “Any change to the manufacturer's supplied ingredients or modifications to the assay as set forth in the manufacturer's labeling and instructions. It may include a change to specimen type, instrumentation or procedure that could affect its performance specifications for sensitivity, specificity, accuracy, or precision or any change to the stated purpose of the test, its approved test population, or any claims related to interpretation of the results.” 

For example, if a practice decided it was not necessary to purchase medical grade monitors to perform primary diagnosis on WSI, this would constitute a modification; and thus, a validation study would need to be performed.

Q: Is it easier to verify an FDA cleared test system for primary diagnosis with WSI or to validate a Lab developed WSI test system or a modified FDA cleared test system? 

A:  The verification procedure for an FDA cleared test system will depend upon the instructions from the manufacturer and will be depend on the Lab Medical Director’s determination that the test system is accurate.  A Lab Medical Director may likely elect to follow the CAP Guidelines for WSI.  In which case, the amount of effort for either verification or validation would be the same.

Q: In the event of a malpractice lawsuit involving a diagnosis performed WSI, is a pathologist better protected using an FDA cleared system?

A: In general, the pathologist or laboratory would still be responsible for a misdiagnosis, no matter the means the diagnosis was rendered. In the event of a misdiagnosis using an FDA cleared WSI system, it is unclear to what extent, if any, a pathologist would be protected.

CAP guidelines for WSI Validation for Primary Diagnosis.png

REFERENCES

1)  Pantanowitz et al, Validating whole slide imaging for diagnostic purposes in pathology: Guideline from the College of American Pathologists Pathology and Laboratory Quality Center. ARPA, 2013 http://www.archivesofpathology.org/doi/pdf/10.5858/arpa.2013-0093-CP http://www.cap.org/apps/docs/membership/wsi_faqs.pdf 


CAP WSI Validation Guidelineshttps://www.cap.org/protocols-and-guidelines/cap-guidelines/current-cap-guidelines/validating-whole-slide-imaging-for-diagnostic-purposes-in-pathology

Companies Developing Pathology AI

AI-slide-for-blog-merged-123-1024x341.jpg

Pathology companies that are developing artificial intelligence algorithms on whole slide images from patient biopsies are popping up everywhere and all the time.  These companies state that they want to improve healthcare by improving pathology diagnoses.  Since the number of pathologists throughout the world is insufficient and the number of pathology biopsies and resections is increasing, Artificial Intelligence is one way to automate the tedious and boring aspects of pathology to free up pathologists’ time for more difficult interpretations.  The companies claim that AI will improve the detection of disease and the standardization of tumor grading.  These companies range from well-funded startups to giant internet companies, such as Google and Baidu.  There are over 30 companies around the world that profess to be performing artificial intelligence on digital pathology.  Almost half of these companies are in the USA and Canada, Twenty-three percent are in Europe (UK, Ireland, Germany, Denmark, Sweden Norway and Finland).  Nineteen percent are in Asia (Korea, India, China, Taiwan) and ten percent are in Israel.

global-distribution-of-path-ai-companies.png

I admit that I am fascinated by this sudden development.  I intend to profile many of these companies in my blog.  Some companies will be winners.  Some will be acquired by much larger companies like Philips, Leica, and Roche who want to complete their portfolio of pathology products, as well as, participate in this potentially game changing area of diagnostics.

Pixelation is Great For Art - But Bad for Diagnosis

Notre-Dame-de-la-Garde (La Bonne Mere), Marseilles by Paul Signac 1905-06.  The Metropolitian Museum of Art, NYC, personal photo

Written by Dr. David Hoak

Paul Signac was an impressionist who along with George Seurat developed the pointillists style of painting.  Signac applied paint in colorful “tiles” or mosaics to create his beautiful and evocative paintings.  Notre Dame de la Garde was the church that sailors would see as they sailed into Marseilles.

Whole slide images are made up of individual photos or tiles that are “stitched together. ”The files are stored in “pyramid” fashion to simulate the process of a microscope going from one objective magnification to the next. Think of it like Google Maps where one can zoom in close to a building or zoom out to see an entire city or country.

Figure 2. The Pyramid Image Rendering schema of pathology whole slide images.

However, sometimes when panning across a virtual slide or zooming in on a higher magnification, the image becomes “tiled” like Signac’s painting.  While the blurred image may be aesthetically pleasing it can be very fatiguing on the pathologist to wait for the image to be sharp and clear.  It can effect satisfaction and productivity. See the example below.  The usual cause of tiling is the bandwidth connectivity between the image storage location and the pathologist’s viewer.  But other causes include the WSI scanner’s proprietary image file format, the tile storage format within the pyramid, and what method the viewer determines to retrieve the next file.  Since these features differ among vendors it is important when looking to implement a WSI solution for pathologists to demo the scanner, viewer, archival system in as real a situation as possible.

Figure 3. Example of slow tiling of a WSI.

Image attribution: 1.      VirtualMicroscopy: ultra-fast interactive microscopy of gigapixel/terapixel images over internet, Ching-Wei Wang et. al, Scientific Reports volume5, Article number: 14069 (2015)

CAP and NSH Announce a Histology and WSI Scanning QC Program

blog jan 21 image.png

Blog Post by Dr. David Hoak

In the December 2018 issue of CAP Today, the College of American Pathologists and the National Society for Histology announced the 2019 release of a new quality improvement program called Whole Slide Image Quality Improvement Program. The program evaluates the quality of H&E sections cut and stained in a lab and the WSI scanned from that slide. The H&E slides are sent to CAP and the whole slide images are uploaded to CAP. Experts will then evaluate both the H&E and the WSI together.  By having both the glass and the WSI, the reviewing experts will be able to determine if an identified quality issue is due to microtomy, staining or scanning and will suggest improvements. There are two challenges a year with four slides and WSI to prepare for each challenge. The program is available at the CAP.org in the 2019 Surveys Catalog.          

When I was a Lab Medical Director, our lab participated in the CAP/NSH HistoQIP, a similar program but for histology only. I found the program very helpful and I would expect that the new program, for whole slide imaging, will be just as helpful. I would imagine that CAP/NSH will expand the program to include comparing special stains and IHC to the WSI. I would recommend that any laboratory that is currently, or is considering performing, primary diagnosis by whole slide imaging for H&E should participate in this program.

Image attribution: https://www.cap.org/laboratory-improvement/catalogs-ordering-and-shipping

http://digital.olivesoftware.com/Olive/ODN/CAPTODAY/