MONET: New AI tool improves medical imaging through deep learning and text analysis

MONET: New AI tool improves medical imaging through deep learning and text analysis

In a recent study published in Nature Medicine, researchers developed the Medical Concept Retriever (MONET) foundation model, which combines medical images with text and evaluates images based on their idea existence, which is helpful in critical tasks in medical artificial intelligence (AI) implementation.

background

Building reliable image-based medical artificial intelligence systems requires analyzing information and neural network models at every level of development, from the training phase to the post-deployment phase.

Richly annotated medical datasets with semantically relevant ideas could demystify “black box” technologies.

Understanding clinically significant terms such as darker pigmentation, atypical pigment networks, and multiple colors is medically beneficial. However, obtaining labels is laborious and most medical information sets contain only diagnostic notes.

About the study

In the current study, researchers developed MONET, an AI model that can annotate medical images with medically relevant ideas. They designed the model to identify different human-understandable ideas in two image modalities in dermatology: dermoscopic and clinical images.

The researchers collected 105,550 dermatology image-text pairs from PubMed articles and medical textbooks and then trained MONET on 105,550 dermatology photos and natural language data from a large medical literature database.

MONET assigns ratings to the photos for each idea, indicating the extent to which the image reflects the idea.

Based on contrastive learning, MONET is an artificial intelligence approach that enables the direct application of plain text descriptions to images.

This method avoids manual labeling and enables rich image-text pair information at a significantly larger scale than is possible with supervised learning. After MONET training, researchers evaluated its effectiveness in annotations and other use cases related to AI transparency.

The researchers tested MONET's concept annotation capabilities by selecting the most conceptual photos from dermoscopic and clinical images.

They compared MONET's performance with supervised learning strategies in which ResNet-50 models were trained with conceptual ground truth labels and OpenAI's Contrastive Language-Image Pretraining (CLIP) model.

The researchers also used MONET to automate data analysis and tested its effectiveness in concept difference analysis.

They used MONET to analyze data from the International Skin Imaging Collaboration (ISIC), the largest dermoscopic image collection with over 70,000 publicly available images routinely used to train dermatology AI models.

The researchers developed a model audit with MONET (MA-MONET), which uses MONET to automatically detect semantically relevant medical concepts and model errors.

The researchers evaluated MONET-MA in real-world environments by training CNN models on data from multiple universities and evaluating their automated concept annotation.

They compared the “MONET + CBM” automated idea scoring method with the human labeling method, which applies exclusively to photos containing SkinCon labels.

The researchers also examined the impact of concept selection on MONET+CBM performance, particularly on task-relevant ideas in bottleneck layers. In addition, they evaluated the impact of incorporating the concept of red in the bottleneck on MONET+CBM performance in inter-institutional transfer scenarios.

Results

MONET is a flexible medical AI platform that can appropriately annotate ideas in dermatology images as verified by board-certified dermatologists.

The concept annotation feature enables relevant trustworthiness assessments across the medical artificial intelligence pipeline, demonstrated through model audits, data audits and interpretable model development.

MONET successfully finds suitable dermoscopic and clinical images for various dermatology keywords and outperforms the basic CLIP model in both domains. MONET outperformed CLIP on dermoscopic and clinical images and remained equivalent to supervised learning models on clinical images.

MONET's automated annotation feature helps identify distinguishing features between any two groups of images in a human-readable language during idea difference analysis.

The researchers found that MONET detects differently expressed ideas in clinical and dermoscopic data sets and can help examine large data sets.

Using MA-MONET revealed features that were associated with a high error rate, such as a cluster of photos labeled “blue-whitish veil,” “blue,” “black,” “gray,” and “flattened.”

The researchers identified the cluster with the highest failure rate based on erythema, regression structure, redness, atrophy and hyperpigmentation. Dermatologists selected ten target-related ideas for the MONET+CBM and CLIP+CBM bottleneck layers that enable flexible labeling options.

MONET+CBM outperforms all baselines in mean area under the receiver operating characteristic curve (AUROC) for predicting malignancy and melanoma in clinical images. Supervised black box models consistently performed better in cancer and melanoma prediction tests.

Diploma

The study found that image-text models can increase the transparency and trustworthiness of AI in the medical field. MONET, a medical concept annotation platform, can improve the transparency and trustworthiness of dermatology AI by enabling large-scale idea annotation.

AI model developers can improve data collection, processing, and optimization practices, resulting in more reliable medical AI models.

MONET can impact the clinical use and monitoring of medical image AI systems by enabling full audit and fairness analysis through annotation of skin tone descriptors.

Sources: