Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

Abstract

In the last few years, the deep learning (DL) computing paradigm has been deemed the Gold Standard in the machine learning (ML) community. Moreover, it has gradually become the most widely used computational approach in the field of ML, thus achieving outstanding results on several complex cognitive tasks, matching or even beating those provided by human performance. One of the benefits of DL is the ability to learn massive amounts of data. The DL field has grown fast in the last few years and it has been extensively used to successfully address a wide range of traditional applications. More importantly, DL has outperformed well-known ML techniques in many domains, e.g., cybersecurity, natural language processing, bioinformatics, robotics and control, and medical information processing, among many others. Despite it has been contributed several works reviewing the State-of-the-Art on DL, all of them only tackled one aspect of the DL, which leads to an overall lack of knowledge about it. Therefore, in this contribution, we propose using a more holistic approach in order to provide a more suitable starting point from which to develop a full understanding of DL. Specifically, this review attempts to provide a more comprehensive survey of the most important aspects of DL and including those enhancements recently added to the field. In particular, this paper outlines the importance of DL, presents the types of DL techniques and networks. It then presents convolutional neural networks (CNNs) which the most utilized DL network type and describes the development of CNNs architectures together with their main features, e.g., starting with the AlexNet network and closing with the High-Resolution network (HR.Net). Finally, we further present the challenges and suggested solutions to help researchers understand the existing research gaps. It is followed by a list of the major DL applications. Computational tools including FPGA, GPU, and CPU are summarized along with a description of their influence on DL. The paper ends with the evolution matrix, benchmark datasets, and summary and conclusion.

Keywords: Convolution neural network (CNN); Deep learning; Deep learning applications; Deep neural network architectures; FPGA; GPU; Image classification; Machine learning; Medical image analysis; Supervised learning; Transfer learning.

Fig. 1

Search framework

Fig. 2

Deep learning family

Fig. 3

The difference between deep learning and traditional machine learning

Fig. 4

Deep learning performance compared to human

Fig. 5

An example of RvNN tree

Fig. 6

Typical unfolded RNN diagram

Fig. 7

An example of CNN architecture for image classification

Fig. 8

The primary calculations executed at each step of convolutional layer

Fig. 9

Three types of pooling operations

Fig. 10

Fully connected layer

Fig. 11

Over-fitting and under-fitting issues

Fig. 12

MLP structure

Fig. 13

Neuron activation functions

Fig. 14

The architecture of LeNet

Fig. 15

The architecture of AlexNet

Fig. 16

The architecture of network-in-network

Fig. 17

The architecture of ZefNet

Fig. 18

The architecture of VGG

Fig. 19

The basic structure of Google Block

Fig. 20

The block diagram for ResNet

Fig. 21

The basic block diagram for Inception Residual unit

Fig. 22

The architecture of DenseNet Network (adopted from [112])

Fig. 23

The basic block diagram for the ResNext building blocks

Fig. 24

The basic block diagram for the Xception block architecture

Fig. 25

The complete CapsNet encoding and decoding processes

Fig. 26

The general architecture of HRNet

Fig. 27

The performance of DL regarding the amount of data

Fig. 28

The conceptual diagram of the TL technique

Fig. 29

Examples of DL applications

Fig. 30

Workflow of deep learning tasks