ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal
Regular Issue, Vol. 14 (2025), e32609
eISSN: 2255-2863
DOI: https://doi.org/10.14201/adcaij.32609

Neural Network Training Acceleration Based on Hybrid Data and Model Parallelism

José Javier Garcia^a, Juan Ramos^b, Juan José Guerrero^b, Andrés Bustos^c, and Rafael Mayo-García^c

^a Innovation Department, NOKIA Spain, Madrid, 28050, Spain

^b Innovation Department, NOKIA TECSS, Madrid, 28050, Spain

^c CIEMAT, Madrid, 28040, Spain

jose_javier.garcia_aranda@nokia.com, juan.ramos_diaz.ext@nokia.com, juan.guerrero_lopez.ext@nokia.com, andres.bustos@ciemat.es, rafael.mayo@ciemat.es

ABSTRACT

KEYWORDS

1. Introduction

In recent years, artificial intelligence (AI) solutions based on deep learning (DL) methodologies have become accurate and reliable in a variety of disciplines such as natural language processing, computer vision, image generation, medical diagnosis, etc. One of the disadvantages of many DL models, such as neural networks (NN), is their intensive computing resource usage both in central processing unit (CPU)/graphics processing unit (GPU) time and memory (Villalobos et al., 2022). Moreover, the size and complexity of the models have increased dramatically in the last few years, as can be seen in Figure 1.

In the recent period 2018-2023, the model size has increased by four orders of magnitude. Table 1 shows specific examples of the size of current DL models, where we can see that popular models such as GPT-3 have ∼ 5 · 10¹¹ weights to adjust.

This growth is caused by several factors: the increased availability of data, which facilitates training larger DL models with more extensive datasets (Halevy et al., 2009); improvements in hardware, which have allowed for faster training (Dean et al., 2012); and advancements in algorithms and mathematical models, such as transformers, which have driven much of this progress (Vaswani et al., 2017, Brown et al., 2020).

The computing power required for artificial intelligence training is crucial to obtaining good performance, and it is even more important when the model is large (Kaplan et al., 2020). This rapid increase in computational power consumption results also in higher energy costs, making the most accurate and sophisticated AI models not only financially more expensive due to their energy demands but also environmentally detrimental because of the consequent generation of the carbon footprint (Strubell et al., 2019).

The need to reduce the training time and resource consumption has driven the onset of several data and model parallelism strategies. A reduction in the training time and resource usage gives researchers more experimentation capabilities and, consequently, facilitates greater research progress. Additionally, an effective parallel model may run on smaller and more common computers, making access to AI more affordable. The actual data and model parallelisms are adequate solutions for distributing the computations across several machines, but they have some disadvantages.

• Data parallelism requires copies of the same model on all the machines.

• Model parallelism requires heavy communications between machines, which reduces the training velocity. Moreover, the parallelism level is limited by the number of model layers or the size of each layer, respectively.

Thus, the growth of the model size and the lack of optimal parallelization strategies evidence the purpose of the present work. Ultimately, the aim is to make the latest AI tools more efficient and accessible to the scientific community by reducing the training time and computing resources. This is to be achieved through the development of a parallel computing framework called Skynnet. The objectives of this work are:

• To propose a new parallelization technique that overcomes the limitations of current data and model parallelization by generating smaller models that run in parallel.

• To reduce training time and resource consumption, since the submodels are smaller and can run in parallel on different machines.

• To validate this technique by comparing the results (e.g., accuracy, loss) of sample models with those of models divided into submodels, and by demonstrating the speedup achieved.

• To automate this process with the framework Skynnet, which enables solving AI problems without modifying the main deep learning frameworks, in an efficient and accessible way for the community.

2. Related Work

In this section, we briefly recall three existing methods devoted to speeding up the execution of DL computations: network compression, data parallelism and model parallelism. A taxonomy of the latter two (applied to graph neural networks) was described by Besta & Hoefler (2024), who also analysed a whole set of parameters for evaluating their efficiency.

For the sake of completion, additional speed up process techniques based on hardware advancements can be found in Hong et al. (2024). Also, a description of the evaluation parameters for a machine learning accelerator in terms of learning and testing performance and hardware design can be consulted in Mohaidat & Khalil (2024).

2.1. Network Compression

Network compression policies reduce the model or the dataset complexity to have a more efficient and cheaper NN execution and have been successfully applied in different problems (Sharma & Singh, 2024; Yingna et al, 2024). We can differentiate between network pruning and quantization. Both procedures can be used separately or simultaneously.

Network compression policies are designed to reduce the complexity of either the model architecture or the training dataset, thereby facilitating more efficient and cost-effective neural network execution. Such strategies have demonstrated effectiveness across a wide range of applications (Sharma & Singh, 2024; Yingna et al., 2024). Within this context, two principal approaches can be distinguished: network pruning and quantization. These methods may be implemented independently or in conjunction, depending on the computational and performance requirements of the task at hand.

Pruning is a classical technique that reduces an already trained NN into a smaller network without any extra training (Lecun et al., 1989). Pruning a network simply consists of removing unessential connections or neurons. Intuitively, if a weight is close to zero, we may eliminate that connection in the NN, reducing its size without compromising its performance. A more detailed description and specific techniques can be found in Liang et al. (2021). Unfortunately, pruning may lead to a reduction in precision and presents implementation difficulties (He et al., 2017).

Another option for network compression is quantization (Fiesler et al., 1990), which is based on simplifying the binary representation of the weights. The standard data type for the weights is usually a 32-bit float number, and for some NN this precision is unnecessary. Converting those numbers to a lower bit representation, such as using 8 bits, effectively reduces the computing time and memory usage (Jacob et al., 2018). On the other hand, Jacob et al. (2018) also studied the degradation in accuracy caused by quantization, together with mitigation techniques. The authors also underscored that high efficiency can only be achieved with dedicated hardware.

2.2. Data Parallelism

Data parallelism is a very popular choice because of its simplicity. It is based on having replicates of the same neural network distributed over all machines. The training phase is done simultaneously with a fraction of the dataset on each machine. At every training epoch, all machines calculate the loss function gradient, take the average, and finally update the model weights with a learning algorithm. The main implementations of the data parallelism method are:

• Data parallelism with replication: the weights actualization is done locally by each machine, using the average gradient computed by all. The actualization is simultaneous.

• Data parallelism with centralized parameters: the model parameters are stored in a central server, which gathers the gradients and updates the model. The updated model is sent back to the machines for further epochs.

As we stated before, these methods are limited by the communications between computing nodes (Geron, 2019). Nevertheless, interesting developments for their enhancement were proposed by Nagajaru et al. (2024), who focused on the preservation of dataset distribution across all workers, ensuring that each worker has access to representative data. Further enhancements involved reductions in communication delays between the server and the workers, and among the workers themselves. This was achieved through the localization of parameters and the quantization of gradients to three levels. The fundamentals of generative AI can be found in Corchado et al. (2023).

2.3. Model Parallelism

In model parallelism, the network is split into chunks and distributed on several independent machines. It depends strongly on the NN architecture, and it is more cumbersome to implement because of the complexity of the communications. Let us briefly describe two approaches to model parallelism, namely, pipeline and tensor parallelism (Geng et al., 2019; Brakel et al., 2024):

• Pipeline is the most intuitive model split, where the NN layers are separated into sets of layers, and then every set is assigned to a machine (inter-operator parallelism). In this way, the machines work in a pipeline, where each machine's output is the input of the following one. This indeed demands intensive communications, which penalize the global speed. During training, the back-propagation algorithm also involves communications, although in the opposite direction.

• An alternative approach to model parallelism is the so-called tensor parallelism, where the layers are divided and distributed among several computing servers, and communications are used to combine the values of the layer outputs (intra-operator parallelism). This approach is implemented in Tensorflow (Abadi et al., 2015) and is publicly available¹

There are several situations where model parallelism is adequate, improving scalability (Narayanan et al., 2021), training time (Xu et al., 2021) and resource efficiency (Shoeybi et al., 2019). On the other hand, the actual challenges of model parallelism are related to communications and work balance (Zhuang et al., 2024; Moreno-Álvarez et al., 2020; Li et al., 2020). A hybrid model parallelism approach with subnetwork data parallelism has recently been proposed by Singh et al. (2023). This approach reduces the memory requirements by training small, structured subnetworks of the model on separate workers. As aforementioned, Corchado et al. (2023) can be consulted to deepen the knowledge on model parallelism and generative AI.

3. A Novel Model Parallelism Proposal

The Skynnet platform calls for a new model parallelization paradigm that reduces computing resources and communications, avoiding the limitations of traditional strategies of intra-operator and inter-operator parallelism. We propose to perform a decomposition of the NN model into several smaller parts called submodels. The submodels will be trained and executed simultaneously, each handling specific parts of the output from the original model. Depending on the nature of the problem, the submodels can be trained either with subsets of the input dataset or on the full set. During inference, the submodels process the entire input and combine their results to produce the final output. This technique enables more efficient resource utilization, as, contrary to classical approaches, the submodels can be distributed across different devices or nodes without incurring in communications between layers. In addition, significant training time savings are expected since the smaller the model, the shorter the training.

Hence, the Skynnet parallelism strategy satisfies the following requirements:

• Communications between submodels during the training phase must not be enabled.

• The original model size must be reduced, so that submodels can be executed in machines with fewer resources.

• The output of all submodels must be combined, ensuring the same output as the original model.

• Any modification of the standard deep learning libraries is not admissible, since that would require close collaboration with the developers.

• If possible, the submodels should train fewer epochs and use a fraction of the datasets to train.

We distinguish the following kinds of problems: classification problems, where the NN output belongs to a discrete set of categories and can be subdivided into binary and multi-class classification; and regression problems, where the NN output is a continuous function.

The Skynnet approach divides the problems according to the output and not on network topology because the nature of the output determines the way the AI model is divided.

In classification problems, the proposed strategy divides the initial problem (full set of categories) into smaller problems (reduced sets of categories). The outcome involves several independent submodels that work together for a global response. However, in a regression problem, the proposed strategy divides the initial problem (complete output domain) into smaller problems (intervals of the output domain) without cooperation among submodels. In both cases, the target is to translate an original model into a set of smaller models that may be executed in parallel with lower resource consumption in terms of model size, memory usage, training data and computing time.

To establish a notation, let us call F the NN reduction factor, defined as the quotient of the output dimensions of the original and reduced models: F = N/n> 1. This factor is common to all three kinds of problems we are dealing with. Also, we denote by M the number of subnetworks to be obtained, which is equal to the number of machines for parallel training. Note that in our approach, the input data dimension is not varied. The details of the strategy for decomposing a NN into submodels/subnetworks depend on the problem, as explained below.

3.1. Multi-Class Classification Problems

In multi-class classification problems, the number of subnetworks M is calculated using combinatorics, starting with a tentative value $\hat{M}$ given by the user. The main steps are:

1. Loop over M, decreasing from $\hat{M}$ to 1.

(a) Loop over G from N to 1. G is the number of groups in which we divide the dataset classes.

If the number of subnetworks $C_2^G=\frac{G!}{(G-2)!\cdot 2!}$ is equal to M, store M in a list L.

2. If $\hat{M}$ is in the list L, then M = $\hat{M}$.

3. Otherwise, keep the largest M in the list.

It is important to remark that the user does not choose M, they only provide an initial value, and the first component of Skynnet (called «sk_tool») finds the largest M compatible with the combinations of classes of the problem. Possibly N/G is not an integer. In such case, the groups are not equal, but their sizes differ at most in one. Since some groups of classes may include one extra class with respect to other groups, subnetworks do not classify exactly the same number of classes. Their load is approximately balanced in any case. Finally, the reduction factor is $\frac{N}{n}=\frac{G}{2}$.

Since each subnetwork solves a simpler classification problem, the subnetworks might be simpler too. Following several numerical experiments, it was concluded that it was best to initially reduce the number of neurons in the Multi-Layer Perceptron (MLP) and the number of filters in the convolutional layers by the same factor F (rounded up). Logically, a subnetwork only trains with a fraction of the dataset because it must distinguish fewer categories. We followed the following specifications:

• Dense layers can be reduced by factor F until any layer reaches two neurons. The reduction must stop within that limit to keep the proportion between layers as unaltered as possible.

• In convolutional layers, the number of filters is reduced with F as well, and it must also satisfy the two filter limit in any layer. In one of our examples (AlexNet in Sec. 5.1), the smaller layer has 96 filters, so the larger reduction factor is F = 48.

• If the model has embedding or positional encoding layers, neither the dense nor the convolutional layers must be reduced in size. In those cases, most of the weights belong to the embedding/positional encoding layers, so reducing the dense or convolutional layers' dimension will not decrease the model size much and might lead to a loss of accuracy.

In the prediction phase, we must compose the final output. First, each submodel expands its output by assigning 1’s to the classes it does not classify. Then, the final output vector is the element-wise multiplication of all the submodel outputs.

Figure 2 shows this division in groups for the 10 categories of the MNIST dataset (Deng et al., 2012). Four groups are created, each one with 2 or 3 classes. Combining such groups in sets of two, we define the six subnetworks that we train with the appropriate classes of the dataset. In this way, with the output composition described below, there are no ambiguities in the 10 category classification.

This explanation is linked to the formula for computing the number of subnetworks, which is based on the standard formula for the number of combinations of G elements taken in groups of 2 elements. In this case, our elements are groups, and we mix only 2 elements (=groups) in each combination.

3.2. Binary Classification Problems

In binary classification, there are only two output categories, so the combinatorics reduction used in multi-class is not feasible. We propose a strategy where first we create two binary submodels with half the size of the original, and then we train them using half of the data. Probably those submodels are worse in accuracy than the original, but their combined output may be almost equally reliable.

Given a sample to classify, we execute both submodels and choose the result with larger certainty, that is, the one with the highest probability. Intuitively, if the sample is similar to the training samples of one of the submodels, the submodel will be very sure of its decision, so we prioritize it over the other option.

3.3. Regression Problems

When dealing with a regression problem we can reduce the original model to any number of submodels without using combinatorics. Each submodel is trained to learn an interval of the original output domain and the composed output will be the concatenation of the output of the submodels. Figure 3 shows an example of a deconstruction of an autoencoder into four submodels, which take care of a fraction of an MNIST sample. As stated above, the final output is the concatenation of the smaller outputs.

3.4. Deconstruction Rules

We propose different considerations to reduce the dimension in the dense layers, the number of convolutional filters, embedding/latent space size, number of training epochs and input/output sizes. The rules are empirical estimations and should be taken as guides, keeping in mind that exceptions may occur. They are heuristic conclusions of dozens of training experiments of several NN architectures using a variety of public datasets: MNIST (Deng et al., 2012), CalTech101 (Li et al., 2022), IMDB (Maas et al., 2011), ImageNet (Deng et al., 2009) and Omniglot (Lake et al., 2015).

We have used common network architectures depending on the task, such as MLP, convolutional neural network (CNN) and vision transformers. The tested models have up to tens of millions of parameters, hence, the observed behaviour may not be appropriate for the largest models.

Table 2 contains the details on the deconstruction rules for the three problems aforementioned. The last three rows show factors on the reduction of the model size and computational resources. We have observed empirically that the RAM memory usage is only reduced when F is smaller than 8.

3.5. Algorithm

Having described the types of problems and the principles for generating subnets, we now present a step-by-step procedure for creating subnets that solve the same problem as the original network. In this section, we provide a high-level explanation of the subnet generation process, and the following sections expand on these concepts with detailed examples. The inputs to the process are the original model and the desired number of submodels (number of available machines). The output is a decomposed model consisting of multiple submodels that can run in parallel while producing results equivalent to those of the original network.

1 Analysis of how the problem is decomposed into subnets depending on the type of problem:

1.1 Multiclass classification problem: The optimal number of subnets is calculated as indicated in section 3.1, M = $\hat{M}$

1.2 Binary classification problem: It is always reduced to two subnets. M = 2

1.3 Regression problem: It is reduced to as many subnets as requested (use desired M)

2 Preparation of the data split for the subnets:

2.1 Multiclass classification: Based on the combinations remaining in each subnet, the data is extracted from the necessary categories
If Ci is the set of classes in subnet i, then: $\mathrm{Di}=\left\{\right(\mathrm{x},\mathrm{y})\in \mathrm{D}\mid \mathrm{y}\in \mathrm{Ci}\}$ where x and y are the pair (data, label) from dataset D

2.2 Binary classification: The dataset is split in two halves.
$\mathrm{D}=\mathrm{D}1\cup \mathrm{D}2,\mathrm{D}1\cap \mathrm{D}2=\mathrm{\varnothing }$

2.3 Regression: The input data is divided into M subgroups.
$\mathrm{D}={\mathrm{U}}_{\mathrm{i}=1}^{\mathrm{M}}{\mathrm{D}}_{\mathrm{i}},\mathrm{Di}\cap \mathrm{Dj}=\varnothing \forall \mathrm{i}/=\mathrm{j}$

3 Network reduction: According to the deconstruction rules shown in Table 2.

4 Inference/prediction composition: The output of each subnetwork $\mathrm{y}={\left\{{\mathrm{f}}_{\mathrm{i}}\right(\mathrm{x}\left)\right\}}_{\mathrm{i}=1}^{\mathrm{M}}$ is collected and combined as follows:

4.1 Multiclass classification: expand each submodel output and multiplication.

$${{\mathrm{f}}^{\mathrm{\prime }}}_{\mathrm{i}}\left(\mathrm{x}\right)=\{\begin{array}{l}{\mathrm{f}}_{\mathrm{i}}\left(\mathrm{x}\right),\text{if the class of}\mathrm{x}\text{is present in subnetwork}\mathrm{i}\\ 1,\text{if the class of}\mathrm{x}\text{is not present in subnetwork}\mathrm{i}\end{array}$$

Therefore:

$\mathrm{y}=\prod _{\mathrm{i}=1}^{\mathrm{M}}{\mathrm{f}}_{\mathrm{i}}^{\mathrm{\prime }}\left(\mathrm{x}\right)$

4.2 Binary classification: Greater certainty: $\mathrm{y}={\mathrm{argmax}}_{\mathrm{i}\in \{1,2\}}{\mathrm{f}}_{\mathrm{i}}\left(\mathrm{x}\right)$

4.3 Regression: Ordered composition of the output predictions: $\mathrm{y}=\left[{\mathrm{f}}_1\right(\mathrm{x}),{\mathrm{f}}_2(\mathrm{x}),\dots ,{\mathrm{f}}_{\mathrm{M}}(\mathrm{x}\left)\right]$

4. The Skynnet Computing Platform

The Skynnet Computing Platform is based on two components named Cloudbook and sk tool. Cloudbook is a framework developed in 2020 that parallelizes Python code executions and distributes the workload among a set of machines (Garcia-Aranda et al., 2021). Thus, Cloudbook will manage the distribution and execution of the subnetworks among a set of available machines. In other words, the Cloudbook component enables the original standalone program to run in both parallel and distributed way.

Skynnet introduces a new component on top of Cloudbook, a library named sk_tool (skynnet tool). This library analyses the Python source code, paying attention to the AI parts, and automatically divides the code in smaller executables following the indications set out in Sec. 3.4 and Table 2. It requires minor modifications of the source code by the user, for instance, the inclusion of a few statements in the code, and an execution of sk tool to properly divide the code into smaller pieces.

Figure 4 shows the main components of Skynnet, which are connected in cascade. First, sk _tool reads the original source code, processes it and decomposes the AI computations following the aforementioned rules. Then the maker divides the processed source code into deployable units (DU), that will be executed in parallel.

In a DU, the minimum unit is the function, and its function aggregation criteria is the number of invocations between them. The last element in Skynnet is the deployer, that acts as an agent coordinator and assigns agents for parallel execution. An agent is a machine with computing capability that executes a DU. Agents can communicate, allowing functions from a DU to call functions from other DU.

Figure 5 exhibits how sk_tool works. The programmer must write #SKYNNET labels in certain parts of the source code, and sk_tool transforms it into a code that builds each submodel and distributes the data flow. It should be noted that, in classification problems, every submodel works with a subset of all the data classes. In regression problems, on the other hand, the submodels deal only with an interval of the output domain.

The input source code file must have certain code blocks to be processed by sk_tool. Those blocks are defined by the labels #SKYNNET:BEGIN and #SKYNNET:END and are named Skynnet blocks. Skynnet blocks contain code related to the data input, NN creation, training and prediction. The main decomposition phases are shown in Figure 4.

First, sk_tool component decomposes the source code, then the maker creates the deployable units (DU, executables) and finally the deployer sends them to the computing servers (agents). The decomposition carried out by the sk_tool («reduction tool» in Figure 5) consists of dividing the original model into submodels and adapting the data to be consumed by these submodels, in addition to preparing the output composition function.

1. Initialization

(a) The input source file is analysed and the code outside the Skynnet blocks is recognized and stored.

(b) sk_tool reads the Skynnet blocks, identifies the problem flavour (regression, multi-class or binary classification) and decides the parallelism strategy to follow.

(c) Code inside the Skynnet blocks is stored for later processing and subnetwork creation.

(d) M is calculated from a tentative value given by the user and F.

2. Skynnet Block Processing:

(a) sk_tool introduces functions to assign parts of the input dataset to the subnetworks. As an example, in multi-class problems, every subnetwork does not have the information on the categories it must classify until the training execution.

(b) The neural network is divided into subnetworks according to the deconstruction rules in Table 2.

(c) Network creation and training are separated from inference in two independent functions.

(d) If there is code after the current block, it is written in the output file. If there is another Skynnet block, we go back to step 2.b. In any other case, continue to step 2.e.

(e) sk_tool writes the corresponding functions to call the subnetworks and the main executable.

The final Python file contains the instructions to divide the input data, train with validation set and test every subnetwork. It includes the subnetwork implementation and can be executed in parallel or sequentially.

Skynnet is publicly available in a repository² under the Apache 2.0 license.

5. Practical Examples

In this section, we test the Skynnet system with three examples: a multi-class classifier and a regressor, using the well-known Caltech101 dataset; and a binary classifier using MNIST.

The purpose of these tests is to validate the new hybrid parallelism model. Data and networks are divided into smaller sets and sub-networks that can be executed in parallel, saving resources and time. In the inference phase, they are combined to compose the global results as if they were a single network.

CalTech101 is a dataset for computer vision (Li et al., 2022), widely used for testing and comparing the performance of new deep learning models. It is composed of 9000 images with variable resolution (approximately 200x300) divided into 101 categories. Due to its large number of categories, it usually requires the use of large neural networks to obtain satisfactory results.

The purpose of Skynnet is to overcome this issue, so we will apply sk_tool and use a local network with common desktop computers for the tests. All computers are the Lenovo T480 model, with an i7-8650U@1.90 GHz CPU and 16 GB of memory.

The results shown below are always the average of 10 runs each, with a standard deviation value lower than 2 % in every case. This fact ensures the robustness of the measures.

5.1. Multi-Class Classification with Alexnet

AlexNet is a classical convolutional model for image classification (Krizhevsky et al., 2012), developed to classify the ImageNet dataset (Deng et al., 2009). The main difference between CalTech101 and ImageNet is the number of classes available: 1000 categories for ImageNet and 101 for CalTech101. In any case, AlexNet is a large AI model with more than 60 million parameters, which makes it ideal for a proof of concept for Skynnet.

AlexNet processes an entry image through many convolutional, maxpooling and dense layers in order to extract a feature vector and outputs class probabilities through a final softmax activation:

$$\mathrm{P}(\mathrm{y}=\mathrm{k}\vee \mathrm{X})=\frac{\exp \left({\mathrm{w}}_{\mathrm{k}}^{\mathrm{T}}{\mathrm{f}}_{\mathrm{CNN}}\right(\mathrm{X})+{\mathrm{b}}_{\mathrm{k}})}{\sum _{\mathrm{j}=1}^{\mathrm{K}}\exp \left({\mathrm{w}}_{\mathrm{j}}^{\mathrm{T}}{\mathrm{f}}_{\mathrm{CNN}}\right(\mathrm{X})+{\mathrm{b}}_{\mathrm{j}})}$$

This formula provides the probability that an input image X belongs to class k, where f_CNN (X) is the feature vector extracted by the convolutional layers, K the number of classes, and w and b are the weight vector and the bias.

As we observed previously in Section 3, multiclass classification is performed using a reduction factor F = N/n, which results in M subnetworks Sm (1<=m<=M) according to the combinatorial formula

$$\mathrm{M}=\frac{\left(\frac{\mathrm{N}}{\mathrm{n}/2}\right)!}{2(\frac{\mathrm{N}}{\mathrm{n}/2}-2)!}$$

Therefore, by decomposing into M subnetworks, each subnetwork Sm is trained in a subset of the original classes and calculates its probability as follows:

$${\mathrm{z}}_{\mathrm{k}}^{\left(\mathrm{m}\right)}={\mathrm{w}}_{\mathrm{k}}^{(\mathrm{m}{)}^{\mathrm{T}}}{\mathrm{f}}^{\left(\mathrm{m}\right)}\left(\mathrm{X}\right)+{\mathrm{b}}_{\mathrm{k}}^{\left(\mathrm{m}\right)}$$

$${\mathrm{P}}_{\mathrm{m}}(\mathrm{y}=\mathrm{k}\vee \mathrm{X})=\frac{\exp \left({\mathrm{z}}_{\mathrm{k}}^{\left(\mathrm{m}\right)}\right)}{\sum _{\mathrm{j}\in {\mathrm{S}}_{\mathrm{m}}}\exp \left({\mathrm{z}}_{\mathrm{j}}^{\left(\mathrm{m}\right)}\right)},\mathrm{k}\in {\mathrm{S}}_{\mathrm{m}}$$

Being ${\mathrm{f}}^{\left(\mathrm{m}\right)}\left(\mathrm{X}\right)$ the lower dimension feature vector.

Empirically, as we stated in Section 4, dense and convolutional layers are reduced in size by a factor F. Thus, the model size is reduced by a factor ∼ F², because most of the AlexNet weights correspond to connections between intermediate layers.

Unlike MLP, in networks with CNN layers, we cannot reduce the number of training epochs in proportion to F⁻¹, but still, the training time is reduced because, with the submodels, each epoch is computationally simpler in terms of data (fewer classes) and computation (submodel smaller than the original one).

Figure 6 shows the training time and the model size as functions of the number of machines/submodels. Both quantities decrease as the number of submodels increases, and, as usual, the interesting region of those curves is the elbow— when the number of machines is around 15, the speedup factor is 14, resulting in a reduction of the model size by a factor of 9. Table 3 shows the numerical values of the deconstruction results: number of categories per subnet (N), model size, reduction factor (F), number of subnetworks (M), number of training epochs, computing time per epoch, total training time, and accuracy and loss value on the test set. Notice that the accuracy of the validation test remains constant or even increases when fragmenting the NN.

Each row of Table 3 is a complete exercise of dividing the original network into M models and each “acc” value is the accuracy obtained by combining the outputs of the M models.

To compare these results with standard solutions we opted for mesh-Tensorflow model parallelism. While network-compression based accelerated solutions also speed up and reduce the model, they sacrifice accuracy. Table 4 shows the results comparison in training time. Similar accuracy and loss have been obtained in both solutions.

5.2. Nonlocal Regression with Caltech101

In this regression test, we train an AI to calculate the Discrete Cosine Transform (DCT) (Ahmed, 1991) of CalTech101 images in a 32 × 32 frequency space. The model consists of two convolutional layers, the first one with 64 16x16 convolutional filters and a second with the same number of filters but with smaller size (4x4). After every convolutional layer, we encounter a maxpolling layer that reduces the dimensionality by a factor of 4. After flattening the data, two dense layers with a ReLu and sigmoid activations are applied. Figure 7 shows the model implementation with the Keras library. After training, the quality of the model will be evaluated with the loss function value.

The deconstruction of the NN is done in such a way that we divide the frequency space in patches, whose number is equal to the number of submodels. Then, each submodel learns to calculate a reduced interval of the 2D frequency domain. This regression example is distinctive because each frequency subdomain/patch may not be equivalent in terms of its importance in the image reconstruction. In fact, the patch in frequency space closer to (0,0) contains the coefficients of the low frequencies, which usually dominate over the high frequencies. This means that the submodel in charge of that subdomain performs a bigger challenge than the rest, and therefore is a major contributor to the loss function of the composed output.

Figure 8 illustrates the workflow, where the DCT is applied to a sample image, followed by the Inverse Discrete Cosine Transform (IDCT) to compare with the earliest. The image from CalTech101 is reshaped to 64×64, then the AI is applied, and finally, we check the quality of the model with the inverse transform.

The model must learn to create the DCT of 32 x 32 lowest frequency coefficients (such as a low pass filter) and the image may be recovered using the IDCT. The same process will be achieved by a set of submodels using the decomposition rules (Table 2) and combining the outputs of all submodels.

From a computational point of view, Figure 9 shows the variation of the training time and number of parameters with the number of submodels built. Table 5 contains the numerical data of the proof of concept of Skynnet.

Two conclusions can be drawn from the data in Table 5. First, since the whole sequential execution of the submodels is faster than the original model, we can still save computing time even when we do not count on a network of computers. For example, four submodels executed sequentially take 4 × 300 = 1200s, which is smaller than 3000s.

Secondly, as in Figure 9, the computing time reduction presents a region of interest with 4-6 submodels, where the training time and model size experiment the strongest reductions. Note that the loss function does not vary much with F, as compared with the previous example of a multiclass classifier.

To compare these results with standard solutions we discard the network compression method because it penalizes accuracy and loss. The data parallelism approach is the natural approach for this problem but does not reduce the model size. Mesh-tensorflow is a comparable solution because it reduces the model size and speeds up the training. Table 6 shows the results (training time) of both mesh and Skynnet for similar loss values.

5.3. Binary Classification

The dataset for the binary classification problem will be MNIST but grouped into two categories: seven and no seven. A simple classifier with two dense layers of 32 and 16 neurons and a final softmax layer is decomposed with Skynnet into two subnetworks. According to the reduction rules, the dense layers of the submodels shall have half the neurons and, thus, the parameter decrement is 25 %.

Using the most certain output criteria from 3.2, we achieve the same accuracy with the reduced model, close to 99 %. The greatest advantage is that this system requires half of the training time.

6. Conclusions

We have developed an effective and public solution that deals with the problem of parallelism for neural networks, without the drawbacks of current data and model parallelism methods. The available tool in Python, called Skynnet, does not require the modification of any standard deep learning libraries (such as Tensorflow or Pytorch). Our solution introduces strategies to reduce the model requirements in a hybrid approach, including characteristics of model and data parallelism. Skynnet processes the raw Python source code and splits it into several submodels, based on the problem typology and keeping the same performance quality as the original network. We have proposed three sorts of problems: regression and multiclass and binary classification.

The purpose of Skynnet is to save computing and researcher time and accelerate technical AI improvements, making it possible to carry out more numerical experimentation in the same time interval. Skynnet does not require deep modifications on the source code, although the user must guide sk tool with labels to identify specific parts of the code.

Model deconstruction might also be beneficial even in the case of a single machine due to the memory reduction enabled by Skynnet. In Section 5.1, we show a case when a decomposed model executed sequentially is faster than the original model.

Skynnet also brings the AI capabilities to a wider community with smaller resources, bringing AI executions to machines with low computing power, such as those devoted to the internet of things or mobile devices.

Skynnet is publicly available under the Apache 2.0 licence. (https://github.com/SKYNNET-ORG/SKYNNET and https://github.com/SKYNNET-ORG/sk_tool)

7. Funding

This work has been founded by Red.es with the project IDI2021/C005/00144167.

References

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mane, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanho Cke, V., Vasudevan, V., Viegas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., & Zheng, X. (2015). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from tensorflow.org. https://www.tensorflow.org/

Ahmed, N. (1991). How I came up with the discrete cosine transform. Digit. Signal Process, 1, 4-5. https://www.cse.iitd.ac.in/~pkalra/siv864-2019/assignment/DCT-History.pdf

Besta, M., & Hoefler, T. (2024). Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5), 2584-2606. https://doi.org/10.1109/TPAMI.2023.3303431

Brakel, F., Odyurt, U., & Varbanescu, A. L. (2024). Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studies. ArXiv. https://arxiv.org/abs/2403.03699

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, 33, 1877-1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

Corchado, J. M., López, S., Núñez, J. M., Garcia, R., & Chamoso, P. (2023). Generative Artificial Intelligence: Fundamentals. ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal, 12(1), e31704. https://doi.org/10.14201/adcaij.31704

Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M. A., Senior, A., Tucker, P., Yang, K., Le, Q., & Ng, A. (2012). Large scale distributed deep networks. In: Pereira, F., Burges, C. J., Bottou, L., Weinberger, K. Q. (eds.) Advances in Neural Information Processing Systems, 25. https://papers.nips.cc/paper_files/paper/2012/file/6aca97005c68f1206823815f66102863-Paper.pdf

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A largescale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248-255. https://doi.org/10.1109/CVPR.2009.5206848

Deng, L. (2012). The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6), 141-142. https://doi.org/10.1109/MSP.2012.2211477

Fiesler, E., Choudry, A., & Caulfield, H. J. (1990). Weight discretization paradigm for optical neural networks. In: SPIE Proceedings, 1281, 164-173. https://doi.org/10.1117/12.20700

Garcia-Aranda, J. J., Ramos-Diaz, J., Molina-Cardin, S., Larriva-Novo, X., Bustos, A., Galindo, L. A., & Mayo-Garcia, R. (2021). Dynamically distributing tasks from an unattended parallel compiler with cloudbook. In: Nesmachnow, S., Castro, H., Tchernykh, A. (eds.) High Performance Computing, 3-17. Springer, Cham.

Geng, J., Li, D., & Wang, S. (2019). Horizontal or vertical? A hybrid approach to large-scale distributed machine learning. In: Proceedings of the 10th Workshop on Scientific Cloud Computing. ScienceCloud, 19, 1-4. https://doi.org/10.1145/3322795.3331461

Geron, A. (2019). Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O'Reilly Media, Incorporated. https://books.google.es/books?id=OCS1twEACAAJ

Halevy, A., Norvig, P., & Pereira, F. (2009). The unreasonable effectiveness of data. IEEE Intelligent Systems, 24, 8-12. https://doi.org/10.1109/MIS.2009.36

He, Y., Zhang, X., & Sun, J. (2017). Channel pruning for accelerating very deep neural networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), 1398-1406. https://doi.org/10.1109/ICCV.2017.155

Hong, H., Choi, D., Kim, N., Lee, H., Kang, B., Kang, H., & Kim, H. (2024). Survey of convolutional neural network accelerators on field-programmable gate array platforms: architectures and optimization techniques. Journal of Real-Time Image Processing, 21, 64. https://doi.org/10.1007/s11554-024-01442-8

Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2704-2713. https://doi.org/10.1109/CVPR.2018.00286

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. ArXiv. https://arxiv.org/abs/2001.08361

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C. J., Bottou, L., Weinberger, K. Q. (eds.) Advances in Neural Information Processing Systems, 25. https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf

Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015). Human-level concept learning through probabilistic program induction. Science, 350(6266), 1332-1338. https://doi.org/10.1126/science.aab3050

Lecun, Y., Denker, J., & Solla, S. (1989). Optimal brain damage. Advances in Neural Information Processing Systems, 2, 598-605.

Li, F.-F., Andreeto, M., Ranzato, M., & Perona, P. (2022). Caltech 101. CaltechDATA. https://doi.org/10.22002/D1.20086

Li, M., Zhang, J., Wan, J., Ren, Y., Zhou, L., Wu, B., Yang, R., & Wang, J. (2020). Distributed machine learning load balancing strategy in cloud computing services. Wirel. Netw, 26(8), 5517-5533. https://doi.org/10.1007/s11276-019-02042-2

Liang, T., Glossner, J., Wang, L., Shi, S., & Zhang, X. (2021). Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461, 370-403. https://doi.org/10.1016/j.neucom.2021.07.045

Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 142-150. Association for Computational Linguistics, Portland, Oregon, USA. https://www.aclweb.org/anthology/P11-1015

Mohaidat, T., & Khalil, K. (2024). A Survey on Neural Network Hardware Accelerators. IEEE Transactions on Artificial Intelligence, 5(8), 3801-3822. https://doi.org/10.1109/TAI.2024.3377147

Moreno-Alvarez, S., Haut, J. M., Paoletti, M. E., Rico-Gallego, J. A., Diaz-Martin, J. C., & Plaza, J. (2020). Training deep neural networks: a static load balancing approach. The Journal of Supercomputing, 76(12), 9739-9754. https://doi.org/10.1007/s11227-020-03200-6

Nagajaru, C., Ramesh, Y., & Krishna Mohan, C. (2024). A data parallel approach for distributed neural networks to achieve faster convergence. In Proceedings Volume 13072, Sixteenth International Conference on Machine Vision (ICMV 2023); 130721A. https://doi.org/10.1117/12.3023413

Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., & Zaharia, M. (2021). Efficient large-scale language model training on gpu clusters using megatron-lm. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC '21. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3458817.3476209

Sharma, R. K., & Singh, S. (2024). Harigeeta: Cic Mechanism with Euclidean Steiner Tree for Service Latency Prediction in Delay-Sensitive Cloud Services. ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal, 13(1), e31594. https://doi.org/10.14201/adcaij.31594

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019). Megatron-LM: Training multi-billion parameter language models using model parallelism. ArXiv. https://arxiv.org/abs/1909.08053

Singh, S., Ruwase, O., Awan, A. A., Rajbhandari, S., He, Y., & Bhatele, A. (2023). A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training. Proceedings of the 37th International Conference on Supercomputing, 203–214. https://doi.org/10.1145/3577193.3593704

Strubell, E., Ganesh, A., & Mccallum, A. (2019). Energy and policy considerations for deep learning in NLP, In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Korhonen, A., Traum, D., Màrquez, L (eds), 3645-3650. https://doi.org/10.18653/v1/P19-1355

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. U., & Polosukhin, I. (2017). Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

Villalobos, P., Sevilla, J., Besiroglu, T., Heim, L., Ho, A., & Hobbhahn, M. (2022). Machine Learning Model Sizes and the Parameter Gap. https://arxiv.org/abs/2207.02852

Yingna, Z., Mohd Daud, K., Moorthy, K., & Mohamad Nor, A. N. (2024). Filtering Approaches and Mish Activation Function Applied on Handwritten Chinese Character Recognition. ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal, 13(1), e31218. https://doi.org/10.14201/adcaij.31218

Xu, W., Zhang, Y., & Tang, X. (2021). Parallelizing dnn training on gpus: Challenges and opportunities. In: Companion Proceedings of the Web Conference 2021, 174-178. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3442442.3452055

Zhuang, Y., Zhao, H., Zheng, L., Li, Z., Xing, E. P., Ho, Q., Gonzalez, J. E., Stoica, I., & Zhang, H. (2024). On optimizing the communication of model parallelism. ArXiv. https://arxiv.org/abs/2211.05322

_______________________________

¹ https://github.com/tensorflow/mesh

² https://github.com/SKYNNET-ORG/SKYNNET and https://github.com/SKYNNET-ORG/sk_tool