ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal
Regular Issue, Vol. 12 N. 1 (2023), e28715
eISSN: 2255-2863
DOI: https://doi.org/10.14201/adcaij.28715

A Novel Study for Automatic Two-Class and Three-Class COVID-19 Severity Classification of CT Images using Eight Different CNNs and Pipeline Algorithm

Hüseyin Yaşar^a, Murat Ceylan^b, Hakan Cebeci^c, Abidin Kılınçer^c, Nusret Seherc, Fikret Kanat^d and Mustafa Koplay^c

^a Ministry of Health of Republic of Turkey, Ankara, Turkey.
^b Department of Electrical and Electronics Engineering, Faculty of Engineering and Natural Sciences, Konya Technical University, Konya, Turkey.
^c Department of Radiology, Selçuk University Faculty of Medicine, Konya, Turkey.
^d Department of Chest Diseases, Selçuk University Faculty of Medicine, Konya, Turkey.

mirhendise@gmail.com, mceylan@ktun.edu.tr, hcebeci16@gmail.com, akilincer@yahoo.com, nusretseher@gmail.com, fkanat@selcuk.edu.tr, koplaymustafa@hotmail.com

ABSTRACT

KEYWORDS

1. Introduction

SARS-CoV-2 has caused a severe pandemic worldwide. This virus causes respiratory distress syndrome (Huang et al., 2020). The pneumonia caused by the virus in question has been defined as COVID-19 (Coronavirus Disease 2019) (World Health Organization, 2022a). COVID-19 has put excessive workload on the healthcare systems of countries. More than two years have passed since the onset of the pandemic, and the total number of cases worldwide now approaches 540 million (World Health Organization, 2022b). The number of deaths due to the disease is around 6.3 million (World Health Organization, 2022b). Although its treatment has been attempted, no drug developed specifically for the disease exists yet. The development of vaccinations has started in some countries as of the beginning of 2021 (World Health Organization, 2022c), but issues with production, supply, storage, and transportation slow down vaccination procedures. This indicates that it may take years to be able to control the disease (Calina et al., 2020). Another major problem is the mutated versions of the virus; the protection the vaccines provide against these mutations remains unclear. This shows that the effects of the disease will affect countries and societies for a long time.

Computed tomography (CT) imaging provides important radiological information for the diagnosis and clinical evaluation of pneumonia caused by bacteria or a virus. CT imaging is widely utilized in the identification and evaluation of COVID-19. Clinical studies have been published that reveal the radiological symptoms of the disease and the differences between it and other viral or bacterial pneumonia (Wong et al., 2020; Pontone et al., 2021; Pianura et al., 2020; Zhao et al., 2020; Lin et al., 2021). It is important to establish diagnostic support systems using artificial intelligence to alleviate the workload of the healthcare system and radiologists. In this context, many system-design studies that use automatic classification of two or multi-class radiological images such as ultrasound, CT and X-ray have been carried out. Another study goal in this regard is to determine the severity of the pneumonia that occurs due to COVID-19. This is important for determining treatment procedures and follow-up of a patient’s condition. In this section, literature studies in which the severity of COVID-19 is automatically predicted through artificial intelligence and CT images are examined.

Severe and critical COVID-19 cases were classified into pneumonia levels using three-dimensional deep learning by Li et al. (2020a). CT images of a total of 217 patients, including 82 severe and 135 critical patients, were used in the study. These images were divided into174 training images and 43 testing images, and experiments were carried out. As a result of the study, the area under the receiver operating characteristic curve (AUC) value for the training set was between 0.812 and 0.909, while the AUC value for the test set was between 0.787 and 0.861. 1,475 regions of interest (ROI) were determined from 141 CT capture sets of 130 COVID-19 patients by Carvalho et al. (2020). Later, well-aerated regions (472), ground-glass opacity (413), crazy paving and linear opacities (340), and consolidation (250) regions were determined by two radiologists over the said ROI. Pulmonary involvement was obtained by dividing the size of total labelled areas by the total lung volume. In this way, three classes were created, namely mild, moderate, and severe pulmonary involvement. The results of the classification processes, obtained using artificial neural networks, were the following: sensitivity (SNST) was 0.80, specificity (SPCF) 0.86, AUC 0.90, accuracy (ACCR) 0.82, and F-score 0.85. In the severity evaluation study conducted by Xiao et al. (2020), CT images of 408 patients collected from two different hospitals were divided into two groups: 303 patients for education and 105 patients for testing. Convolutional neural network (CNN) architectures (ResNet34, AlexNet, VGGNet, and DenseNet) were utilized in the classification processes carried out as non-severe and severe within the scope of the study. In the tests performed, it was seen that the most successful CNN architecture was ResNet34, and an ACCR level of 0.819 was obtained. In addition, 0.892 AUC was achieved in the tests.

Disease severity classification was performed using CT images of 207 normal, 194 mild, and 15 severe COVID-19 patients by Huang et al. (2021). Deep learning was used in the study and the CNN architecture that was put forward was named fast assessment network (FaNet). In this study, in which the images of 300 patients were designated for training and the images of 116 patients were designated for testing, 0.9483 classification ACCR was obtained. Automatic disease severity classification was performed by Li et al. (2020b) using 95 CT images of 32 severe COVID-19 patients and 436 CT images of 164 non-severe COVID-19 patients. A deep learning approach based on UNet and ResNet34 was used for segmentation processes in the study. SNST, SPCF, and AUC values were 0.9241, 0.9049, and 0.97, respectively. The classification of COVID-19 disease severity—severe and non-severe—was performed by Yu et al. (2020). In the study, 246 severe and 483 non-severe CT image slices collected from 202 COVID-19 patients were used. Four different CNN architectures were used to extract image properties and machine learning methods were used as classifiers. The highest ACCR result was 0.9520 (10-fold cross-validation) using DenseNet-201 and cubic SVM methods.

In the COVID-19 disease severity classification study by Meng et al. (2020), two-class automatic classification, severe and critical, was performed. The de-COVID19-Net architecture was used in the study, which was carried out using 366 CT images—256 severe, and 110 critical. As a result of the study, ACCR was calculated as 0.875 and AUC as 0.943. Using 297 CT images, Ho et al. (2021) performed a two-class classification as low risk and high risk. In the study, experiments were performed using 5-fold cross-validation. As a result of the study using 3D CNN, the highest ACCR was 0.933, while the highest AUC value was calculated as 0.900. Zhu et al. (2021) classified the severity of COVID-19 disease using a total of 408 CT images, 322 non-severe and 86 severe. As a result of the experiments performed with the 5-fold cross-validation method, the ACCR was 0.8569, while the AUC was calculated as 0.8591. An automatic COVID-19 severity classification was performed using three-class (mild, moderate, and severe) and two-class (non-severe and severe). In the study, deep learning models were used for classification. CT images were used as radiological images. A total of 483 COVID-19 CT image slices—267 mild, 156 moderate and 60 severe—were used in the study. These images and labels were used directly for three-class classification. In two-class classification, mild and moderate images were accepted as non-severe. A total of eight CNN architectures were classified—MobileNetv2, ResNet101, Xception, Inceptionv3, GoogleNet, EfficientNetb0, DenseNet201, and DarkNet53. In the study, the results of the top four CNN architectures with the best performance were combined using a pipeline algorithm. The results show that deep learning methods are an important alternative for the detection of COVID-19 severity and follow-up of patients in disease processes.

2. Methods

2.1. Used Data

The CT images used within the scope of the study were collected from the real world via Selçuk University Faculty of Medicine Hospital (Konya, Turkey). Some patients have more than one CT-scan set. These shooting sets were carried out on different days of the disease process. Three CT image slices were selected from each of the CT-acquisition sets. COVID-19 severity was labeled on the CT scans by four radiologists. If lobar involvement was between 1 and 25%, it was defined as mild, if it was between 25 and 50%, it was moderate, and if it was >50%, it was severe. Information about the gender of the patients, the day of the CT scan, and the severity of the disease are provided in Table 1; M stands for male, F for female, Mi for mild, Mo for moderate, and S for Severe.

Automatic COVID-19 severity classification was carried out using three-class (mild, moderate, and severe) and two-class (non-severe and severe). A total of 483 COVID-19 CT image slices—267 mild, 156 moderate and 60 severe—were used in the study. These images and labels were used directly for three-class classification. In two-class classification, mild and moderate images were accepted as non-severe. Examples of images utilized in the study are given in Figure 1.

The CT sets used within the scope of the study were in DICOM format. Three CT slices selected from each CT set were converted to JPG format. These CT images were then framed to include the lung region, and the portion containing the actual radiological information was obtained. After this stage, the image dimensions were arranged and recorded at eight-bit gray-level depth. A total of eight CNN architectures—MobileNetv2, ResNet101, Xception, Inceptionv3, GoogleNet, EfficientNetb0, DenseNet201, and DarkNet53—were used. There are some differences in the input image sizes of the CNN architectures used. Therefore, the CT image row and column lengths have been standardized in accordance with the CNN architecture used. Table 2 shows the dimensions to which the CT images were adjusted according to the CNN architectures. In Table 2, the first two variables in the sizes column represent the pixel values and the third variable represents the number of color spaces (gray level image since it is 1).

2.2. Convolutional Neural Network

A convolutional neural network (CNN) was utilized for the extraction and automatic classification of CT images in the study. CNN is a deep learning method widely used in the extraction and analysis of the features of images in recent years. A special CNN architecture was not designed, but some architectures, which were previously put forward and tested, were transferred. Eight CNN architectures—MobileNetv2 (Sandler et al., 2018), ResNet101 (He et al., 2016), GoogleNet (Szegedy et. al., 2015), Xception (Chollet, 2017), DenseNet201 (Huang et al., 2017), EfficientNetb0 (Tan and Le, 2019), Inceptionv3 (Szegedy et al., 2016), and DarkNet53 (Redmon, 2018)—were used in the study. The original versions of the architectures in question have thousands of outputs and perform the automatic classification of thousands of classes. Automatic classification into three-class (mild, moderate, and severe) and two-class (non-severe and severe) was carried out. Another difference is the third dimension of the entrance images. The value of the third dimension of the entrance images of these CNN architectures is three. That is, input images in RGB format were used. The CT images used within the scope of the study were gray-level. So, the value of the third dimension of the images was one. For this reason, some arrangements have been made in these CNN architectures for them to be used within the scope of the study. The first arrangement made in this context was to change the CNN-input image dimensions, as shown in Table 2. The second was to make changes in the last layers of the CNN architectures so that the output class was two and three.

The software was created on the Matlab 2020 (b) platform. The software was run using an Intel (R) Xeon (R) E-2274G 4.00 GHz CPU with 16 GB of RAM, and CPU runtimes were also measured. CNN architectures were transferred within the scope of the study and later, the necessary changes were made. However, the transfer process was limited only to the transfer of the CNN architecture. No transfer was performed for the training of starting weights. All CNN training processes were initiated with starting weights randomly assigned by the Matlab platform. 10-fold cross-validation was used in experiments in the study. At this stage, we focused on the inclusion of three image slices taken from the same CT acquisition set together in the training or test set. In other words, it is not possible to have an image slice taken from the same CT acquisition set in the training set while another image slice is in the test set.

To avoid overfitting during the training process, the initial learning rate was set to be 0.01. In addition, the training set was mixed in each iteration during the training. For this, the option for data shuffling in Matlab’s 2020 (b) training options was determined to be every-epoch. The solver for training network, maximum number of epochs and the size of mini-batch options used in the training processes were determined as stochastic gradient descent with momentum (SGDM), 30 and 16, respectively. Apart from this, no additional arrangements were made in the training options; they were left as default.

2.3. Pipeline Algorithm

The pipeline algorithm used in the study has two inputs. In the said algorithm, a new result is obtained by combining the numerical average of the inputs (50%-50%). For example, suppose that the probabilities of being mild, moderate, and severe for an image were obtained as 0.16, 0.60, and 0.24, using the first CNN architecture, and 0.70, 0.14, and 0.16, using the second CNN architecture. When combining using the pipeline algorithm, the pipeline result would be 0.43, 0.37, and 0.20. Similar to the three-class classification, it is possible to combine the two-class classification. For example, suppose that the probabilities of being non-severe and severe for an image were 0.56 and 0.44, using the first CNN architecture and 0.74 and 0.26, using the second CNN architecture. When combining using the pipeline algorithm, the pipeline result would be 0.65 and 0.35. The working structure of the pipeline algorithm was visualized in Figure 2. Yasar and Ceylan have used the pipeline algorithm mentioned above, in their studies (Yasar and Ceylan, 2021a; Yasar and Ceylan, 2021b) for diagnosing COVID-19 disease from CT and X-ray images.

2.4. Evaluation Criteria of Classification Results

Many comparison parameters were used to evaluate the results obtained in the study. The most basic of these parameters were the confusion-matrix results. The true positive (TrPs), false positive (FlPs), true negative (TrNg), and false negative (FlNg) results obtained in all classification experiments were given in the study. Sensitivity (SNST), specificity (SPCF), accuracy (ACCR), and F-1 score (F-1) were calculated using the results of the confusion matrix. The formulas for the mathematical calculation of SNTS, SPCF, ACCR, and F-1 are included in equations (1–4).

$$SNST=TrPs/(TrPs+FlNg)\mathrm{ (1)}$$

$$SPCF=TrNg/(TrNg+FlPs)\mathrm{ (2)}$$

$$ACCR=(TrPs+TrNg)/(TrPs+TrNg+FlPs+FlNg)\mathrm{ (3)}$$

$$F-1=(2\times TrPs)/(2\times TrPs+FlPs+FlNg)\mathrm{ (4)}$$

Similarly, these parameters are found in two and multi-class classification operations. However, in multi-class classification, unlike two-class classification, these parameters are calculated separately for each class label. In multi-class classification, weighted SNST, SPCF, ACCR, and F-1 values can be calculated by weighting according to the number of images used in the class labels. Additionally, overall ACCR can be found by dividing the sum of correctly classified samples by the total number of test samples. Another comparison parameter used in the study is receiver operating characteristic curve (ROC) analysis and AUC values. In the multi-class classification process, the weighted AUC values were calculated in a similar way to the other calculated parameters.

3. Results

In the study, an automatic COVID-19 severity classification was performed in three-class (mild, moderate, and severe) and two-class (non-severe and severe). A total of eight CNN architectures were classified. These CNN architectures were MobileNetv2, ResNet101, Xception, Inceptionv3, GoogleNet, EfficientNetb0, DenseNet201, and DarkNet53. Table 3 shows the experimental results obtained by using these CNN architectures for three-class classification. Table 4 contains the experimental results obtained for two-class classification. Table 5 presents the acquisition times per image in CPU time, including training and testing for two-class and three-class classification. While evaluating these herds with other studies, the software platform (Matlab 2020 (b)), training-test approach (10-fold cross-validation), and training options should be considered.

Upon examining Table 3, which includes three-class classification results—mild, moderate, and severe—it is clear that the most successful four CNN architectures are Inceptionv3, DarkNet53, ResNet101, and DenseNet201, respectively. In the study, the classification results obtained by combining the most successful CNN results for the three-class classification using the pipeline algorithm are given in Table 6. Upon examining Table 4, which includes the classification results of two-class, non-severe and severe, it is clear that the most successful four CNN architectures are MobileNetv2, Inceptionv3, Xceptionv3, and DenseNet201, respectively. In the study, the classification results obtained by combining the most successful CNN results for the two-class classification using the pipeline algorithm are given in Table 7.

4. Discussion

In the study, the experimental results obtained for two-class and three-class classification using CT image slices are shown in Tables 3 and 4, respectively. Similarly, the results following the application of the pipeline algorithm are shown in Table 6 and Table 7. In this section, the presented results are compared.

Upon examining Table 3, which includes three-class classification results—mild, moderate, and severe—it is clear that the most successful four CNN architectures are Inceptionv3, DarkNet53, ResNet101, and DenseNet201, respectively. The highest weighted SNST, ACCR, F-1, AUC, and overall ACCR values were obtained using Inceptionv3 architecture: 0.7785, 0.8299, 0.7758, 0.9112, and 0.7785, respectively. The highest weighted SPCF value, 0.8351, was obtained by using the DarkNet53 architecture. The ROC analysis curve obtained using Inceptionv3 is included in Figure 3(a). When the four most successful CNN architectures are compared in terms of classification times, including training test per image, the order is DarkNet53, ResNet101, Inceptionv3, and DenseNet201, from fastest to slowest.

Upon examining Table 4, which includes the classification results of two-class, non-severe and severe, it is clear that the most successful four CNN architectures are MobileNetv2, Inceptionv3, Xceptionv3, and DenseNet201, respectively. The highest SNST value obtained was 0.9740 using the EfficientNetb0 architecture. The highest SPCF and AUC values, obtained using Inceptionv3, were 0.8500 and 0.9788, respectively. The highest ACCR and F-1 values, 0.9482 and 0.9703, respectively, were obtained using MobileNetv2. The ROC analysis curve was obtained using the Inceptionv3 architecture, which provides the highest AUC value within the scope of the study, as shown in Figure 4(a). When the four most successful CNN architectures are compared, in terms of classification times, including training-test per image, the order is MobileNetv2, Xceptionv3, Inceptionv3, and DenseNet201, from fastest to slowest.

The classification results obtained by combining the most successful CNN results for the three-class classification using the pipeline algorithm are given in Table 6. Upon examining Table 6, which includes three-class classification results—mild, moderate, and severe—it is clear that the most successful two pipeline combinations are Inceptionv3-DarkNet53, Inceptionv3-ResNet101, respectively. The highest weighted SNST, SPCF, ACCR, F-1, and overall ACCR values were obtained using the Inceptionv3-DarkNet53 pipeline combination: 0.8095, 0.8555, 0.8563, 0.8076, and 0.8096, respectively. The highest weighted AUC value, 0.9089, was obtained by using the Inceptionv3-ResNet101 pipeline combination. The ROC analysis curve obtained using Inceptionv3-ResNet101 pipeline combination is included in Figure 3(b).

The classification results obtained by combining the most successful CNN results for the two-class classification using the pipeline algorithm are given in Table 7. Upon examining Table 7, which includes the classification results of two-class, non-severe and severe, it is clear that the most successful pipeline combination is MobileNetv2-Inceptionv3. The highest SNST, SPCF, ACCR, F-1, AUC, and overall ACCR values were obtained using MobileNetv2-Inceptionv3 pipeline combination: 0.9811, 0.8333, 0.9627, 0.9788, and 0.9851, respectively. The ROC analysis curve obtained using the MobileNetv2-Inceptionv3 pipeline combination is included in Figure 4(b).

5. Conclusion

In the study, an automatic classification of COVID-19 pneumonia severity in three and two classes, namely mild, moderate, severe, and non-severe, was carried out. A total of eight architectures were used for automatic classification, namely MobileNetv2, ResNet101, Xception, Inceptionv3, GoogleNet, EfficientNetb0, DenseNet201, and DarkNet53. CT images were used as radiological images.

A total of 483 COVID-19 CT image slices—267 mild, 156 moderate, and 60 severe—were used in the study. These images and labels were used directly for the three-class classification. In the two-class classification, mild and moderate images were accepted as non-severe. Comprehensive results were obtained with the experiments performed according to the 10-fold cross-validation approach. Also, in the study, the results of the top four CNN architectures with the best performance were combined using a pipeline algorithm.

The comparison of the results obtained for the three-class classification of COVID-19 pneumonia severity as well as the results obtained in previous studies are presented in Table 8. It is understood that higher results were obtained than in Carvalho et al.’s (2020) study, in which the same class labels were used. The comparison of the results obtained for the two-class classification of COVID-19 pneumonia severity as well as those obtained in previous studies are presented in Table 9. It is understood that higher results were obtained than in the studies of Li et al. (2020a), Xiao et al. (2020), and Li et al. (2020b), in which the same class labels were used. However, the results were lower than those obtained by Yu et al. (2020). The CNN architectures used by Yu et al. (2020) were included in our study, and their highest results were obtained by using the DenseNet-201 architecture. In our study, DenseNet-201 was also used in the classification, and higher results were obtained with other CNN architectures. Therefore, it can be said that our results are more successful than those of Yu et al. (2020), although numerically lower.

Another conclusion obtained within the scope of the study is that the study results can be increased significantly by using the pipeline algorithm. For example, when the results obtained for the three-class classification of COVID-19 pneumonia severity before and after using the pipeline algorithm are examined, it is seen that the weighted results were improved by 3% thanks to the pipeline algorithm. Similarly, when the results obtained for the two-class classification of COVID-19 pneumonia severity are examined, it is understood that improvements at the level of 1% have been achieved. In this context, it is seen that the approach of using the pipeline algorithm, which has produced successful results (Yasar and Ceylan, 2021a; Yasar and Ceylan, 2021b) in the studies of automatic diagnosis of COVID-19 disease from X-ray and CT images, has also increased the results for COVID-19 pneumonia severity classification.

Future studies should aim to use different CNN architectures, especially complex-valued ones. In addition, we plan to diversify the images used with image-feature-extraction methods. Also, the proposed method is planned to be transformed directly and in real time into a diagnosis-decision support system that classifies patients with severe COVID-19 pneumonia.

6. Acknowledgements