AI is making precision medicine a reality, bolstered by the digital healthcare revolution, including integrated electronic health records and the advancement of computing power. In fact, prediction is not new in ophthalmology. Various risk scores have been studied to determine the individual risk of different eye diseases, in search of a personalized medicine approach. However, it is only until the rise of deep learning algorithms that the concept of predicting accurate treatment outcomes can be realized. Our study demonstrates that a deep learning neural network was effective in predicting treatment outcome over one year from baseline OCT images (AlexNet accuracy: 0.895, sensitivity: 0.824, specificity: 0.942, AUC: 0.936) , and the precision was even higher when the clinical data were combined using a new CNN model (HDF-Net precision: 0.936, sensitivity: 0.933, specificity: 0.938, AUC: 0.989).
Prediction of nAMD visual outcome derived from OCT features has been previously investigated by other groups13.18. For example, Schmidt-Erfurth et al. introduced a model to predict visual outcomes in a randomized controlled trial, and demonstrated R2= 0.34 if only baseline data were considered14; Rohm and colleagues developed and validated a model in 456 patients and showed successful VA prediction within an 8-letter margin of error after one year of anti-VEGF therapy in the real world15. All of these studies were particularly comprehensive in including “predefined” OCT measurement data, such as central retinal thickness, subretinal fluid (SRF), and intraretinal fluid (IRF), in the model of machine learning. However, some important anatomical aspects had been omitted because they were difficult to capture in currently available automated segmentation methods. In fact, the conventional machine learning approach such as random forest requires the input data to be in the form of a feature vector instead of an OCT image itself, which makes the whole of the very time-consuming process in feature labeling and highly dependent on the feature. extraction techniques19.
Based on previous evidence, baseline BCVA has become an important prognostic factor for final visual outcomes20. However, to date, automated analysis of “raw” retinal OCT images in combination with baseline BCVA in a single CNN model has not yet been explored to predict treatment outcome in nAMD. To address these limitations, we developed the HDF-Net to predict visual outcome simply by using the baseline OCT image and three demographic covariates (age, sex, and baseline BCVA). Unlike the traditional machine learning approach, the CNN model accepts a sample as an image and performs feature extraction and classification through hidden layers. But challenges remain in how to input heterogeneous hybrid image and non-image data into a single CNN architecture. Unlike conventional CNN approaches that can only allow one type of data as input, we used a data fusion approach that allows simultaneous processing of multiple data types with heterogeneous features extracted from different sources. Thus, the power of CNN can be released for image and non-image data at the same time.
The difference between HDF-net and other published CNN data fusion approaches, such as HDF-CNN, is the method of entering the numerical data. HDF-CNN uses raster format to perform heterogeneous data fusion21. Therefore, all heterogeneous data becomes a single input instance and the features of heterogeneous data are extracted through CNN. However, there may be some loss of information in the digital data after convolutions and pooling. Moreover, the relevant features among the digital data extracted by the convolution layers may turn out to be insignificant. In contrast, the HDF network we proposed here is designed to merge the image features and digital data into a single feature vector after the feature extraction layers. HDF-Net’s classification layer will later automatically determine feature weights among various input data. In this work, we showed that the HDF-Net approach is superior to models such as ResNet50 and AlexNet in accurately predicting VA outcomes in a real population. HDF-Net has many advantages, including automatic feature extraction in unlabeled samples, finding hidden structures from sparse and hyperdimensional data, and hybridization of non-image data. Therefore, it has the potential to offer robust decision support with non-image data integration, as genetic factors are known to be involved in nAMD prognosis determination.22.23.
A strength of this study is that it is a validated model based on a real population including not only treatment-naïve individuals. Previous studies regarding AI predicting treatment outcomes have frequently used a trial dataset, such as that of the HARBOR study, because it offers standardized imaging data and a well-designed treatment protocol from a large sample. Nevertheless, studies of nAMD in the real world have shown deviations in several respects from randomized controlled trials (RCTs). An analysis of 49,485 eyes assessing anti-VEGF intensity and change in VA found that real-world nAMD patients receive fewer injections and have poorer visual outcomes compared to patients receiving fixed and frequent treatment in RCTs. Additionally, older patients with low baseline VA may be particularly prone to undertreatment.24. This suggests a potential bias in the results of studies validating AI algorithms only in trials. Although this study was designed as a retrospective, all included patients were approved for anti-VEGF injections reimbursed by National Taiwan Health Insurance (NHI) after careful cross-checking of clinical diagnosis and OCT images. of the Bureau of NHI. This further supports diagnostic accuracy and standardized treatment protocol following the reimbursement scheme.
Another major strength of this study is that it aims far beyond identifying “predefined OCT features” but generates prediction rules from “raw OCT images”. In a study comparing the performance of retina specialists and an AI algorithm, it was found that retina specialists had imperfect accuracy and low sensitivity in detecting retinal fluid as the AI achieved a higher level of precision25. This confirms our hypothesis that feeding AI with only “predefined OCT functionalities” might limit the scope of its application as it became clear that AI might be able to outperform human intelligence. Previous studies have demonstrated the favorable accuracy of the deep learning approach to detect “predefined OCT features” such as retinal fluid on OCT scans25.26but the quantification of retinal fluid cannot directly guide clinical practice because controversies remain on the tolerance to SRF after the start of treatment27,28,29. Treatment of nAMD with the ultimate goal of completely drying out the retina may also increase the risk of macular atrophy, leading to poorer long-term visual outcomes.8.30.
This study has some limitations and there are several factors to improve to optimize the results. A major limitation of most deep learning models is the “black box” problem indicating that their predictions can be difficult to interpret. In this study, we applied heatmaps to locate image regions influencing classification. While the heat map is a useful clue to highlight which part of the image guided the CNN model to its decision, it does not provide information about the reason behind it. By processing both image and non-image data in a single CNN architecture, we found that adding digital clinical data from the baseline could improve model performance. But, the optimal feature weighting between image and non-image data remains a matter of major interest. Another limitation is the relatively small sample size which may have compromised our statistical analysis. However, each participant was comprehensively assessed with horizontal and vertical OCT scans, providing a rich set of reliable data.
The therapeutic response of nAMD varies widely in the real world and has been difficult to predict in the past. In this study, we presented and validated a new deep learning-based approach using baseline OCT and clinical information, including baseline BCVA, to predict visual outcome at 12 months after anti-aging treatment. Standard VEGF for active nAMD. The combination of clinical data and heterogeneous images in HDF-Net has the potential to serve as a robust decision support tool for clinicians to deliver evidence-based personalized treatment. This breakthrough marks a new era in AI to guide treatment decisions and patient expectations. Future studies are warranted to assess both economic impact and patient perceptions regarding outcome predictions.