International Journal of Electrical and Computer Engineering (IJECE) Vol.
No.
June 2025, pp.
ISSN: 2088-8708.
DOI: 10.
11591/ijece.
Enhancing malware detection with genetic algorithms and generative adversarial networks Abid Dhiya Eddine1,2.
Ghazli Abdelkader1,2.
Bouache Mourad1,3 Department of Computer Science.
Faculty of Exact Sciences.
Tahri Mohamed University.
Bechar.
Algeria Innovations in Informatics and Engineering Laboratory (INIE LAB).
Tahri Mohamed University.
Bechar.
Algeria High Performance Computing Center.
University of Stanford.
California.
United States Article Info
ABSTRACT
Article history:
Malware detection is a critical task in cybersecurity, necessitating the creation of robust and accurate detection models.
Our proposal employs a holistic methodology for identifying and mitigating malware using deep learning techniques.
Initially, a customized genetic algorithm is employed for feature selection, reducing dimensionality and enhancing the discriminatory power of the dataset.
Subsequently, a deep neural network is trained on the selected features, achieving high accuracy and robust performance in distinguishing between malware and benign data.
Generative adversarial networks are also utilized to evaluate model effectiveness on unseen data and ensure the model's robustness and generalization Evaluation of the proposed model demonstrates accurate malware detection with high generalization capabilities.
Furthermore, future research should focus on developing and deploying practical tools or systems that implement the proposed model for real-time malware detection in operational environments.
This research makes a significant contribution to the field of malware detection and provides excellent opportunities for practical implementation in the field of cybersecurity.
Received Jul 20, 2024 Revised Jan 8, 2025 Accepted Mar 3, 2025 Keywords:
Cybersecurity Deep learning Generative artificial Genetic algorithm Malware detection This is an open access article under the CC BY-SA license.
Corresponding Author:
Abid Dhiya Eddine Department of Mathematics and Computer Sciences.
Faculty of Exact Sciences.
Tahri Mohamed University of Bechar Istiklal Street, 08000.
Bechar.
Algeria Email: abid.
dhiyaeddine@gmail.
INTRODUCTION
Technology is advancing daily and being used more frequently.
This advancement in technology also creates a conducive atmosphere for the spread of malware, .
leading to data breaches, financial losses, and disruptions of critical infrastructure .
According to the statistics, every day, the AV-TEST Institute detects over 450,000 novel potentially unwanted applications (PUA) and malicious programs .
Figure 1 shows the total amount of malware and PUA in the last five years according to AV-TEST Institute.
Malware, often known as malicious program, is widely acknowledged as one of the most dangerous cyberthreats and risks to contemporary computer systems.
This all-encompassing term describes any code that might have negative damaging effects .
It alludes to a broad spectrum of harmful software that might manifest in various ways.
These include ransomware, adware, spyware, trojan horses, worms, and viruses.
The malware aims to compromise the security and privacy of a user's computer by disrupting its operations, infiltrating unauthorized data or systems, or employing other methods .
Traditional malware detection techniques frequently depend on methods that rely on signatures, which find it difficult to keep up with the quick changes in malware types.
More advanced methods that can efficiently identify and evaluate malware Journal homepage: http://ijece.
Int J Elec & Comp Eng
ISSN: 2088-8708
in real-time are therefore becoming more and more necessary, particularly in situations where harmful activity takes place in volatile memory .
AI approaches have become effective tools in combating malware.
By utilizing machine learning techniques and deep neural networks.
AI powered malware detection systems have the ability to examine extensive volumes of data where the aim is to uncover patterns and anomalies that indicate potentially dangerous activity .
These systems have shown promising results in detecting previously unknown malwares, where conventional approaches that rely on signatures may fall short .
AI powered solutions in cybersecurity provide a proactive approach by constantly acquiring knowledge from fresh data and adjusting to changing threats .
Figure 1.
Total amount of malwares and PUA in the last 5 years according to AV-TEST Institute Recent research has made significant strides in the domain of malware detection.
Researchers in .
developed a user-friendly website for malware detection and prediction, capable of identifying the type of malware encoded in a file.
Minerva is new approach presented by .
for the ransomware detection.
Minerva uses all of the operations that files receive during a given time interval to create behavioral profiles of the files in order to detect ransomware.
Microsoft Malwares dataset was used in .
with many machine learning algorithms for classifying the malicious and the benign software based on the analysis of executable file metadata.
MalMem-2022 it is a dataset to assess the effectiveness of memory-based obfuscated malware detection methods .
, this dataset used by .
, and also .
used it but with a customized K-Nearest Neighbors algorithm, and it used by .
for detecting malwares in big data environment.
Malwares detection also can be applied to mobile phones as it is presented in paper .
, where the main idea is using deep learning techniques and explainable artificial intelligence (XAI) for the detection and the classification of android mobile malwares using CICAndMal2017 dataset .
it is another work in the field of mobile phones which is based on stacking, presented a machine learning (ML) model for detecting malware that uses an ensemble approach for Android devices.
The authors in this research merged between the CIC-MalMem 2022 and CIC-MalDroid 2020 datasets in the examination of the effectiveness of the proposed model.
The Internet of Things also has a share in previous works devoted to detecting malwares, and this is what was embodied in the research .
These studies collectively highlight the potential of machine learning and behavior-based detection in enhancing malware detection capabilities.
While all previous research has done well in creating robust models to identify malware, the question arises about the effectiveness of these models with generative data, so we will try to answer this question in this research In this research, we will use the CIC-MalMem 2022 dataset to build a model to identify malware.
We will call genetic algorithm to choose the appropriate features, then we will train the final data using deep neural networks, and then we will test the effectiveness of the model using machine learning evaluation The next stage is to create a generative adversarial network, then generate data similar to the original Finally, we will test the effectiveness of our model with the generative data in order to discover the effectiveness of the generative data in these cases.
Enhancing malware detection with genetic algorithms and generative A (Abid Dhiya Eddin.
A ISSN: 2088-8708
PROPOSED METHODOLOGY
To enhance malware detection, we implemented a multi-stage approach starting with the preprocessing of the CIC-MalMem 2022 dataset, as illustrated in Figure 2.
A customized genetic algorithm (GA) was employed to select the most discriminative features, ensuring the model was built on the most relevant data.
Using these selected features, a deep neural network (DNN) was trained to classify samples as benign or malicious.
To improve the model's robustness and generalization, a generative adversarial network (GAN) was then utilized to generate synthetic benign and malicious samples.
The GAN, consisting of a Generator and a Discriminator, created new data points that closely resembled real-world examples, which were used to further evaluate the DNN.
The model's performance was assessed on this synthetic data to determine its effectiveness.
If the model performed well, it was deemed robust.
otherwise, further refinement was necessary.
This integrated approach of feature selection, deep learning, and synthetic data generation ensured that our malware detection model was both accurate and resilient against diverse threats.
Feature selection stage Genetic Algorithm Inialization Bad Selection, crossover, mutation CIC-MalMem Evaluation Preprocessing Evaluation Good Recording & Elitism Best features Update population Training stage Data generation stage Generator Discriminator Deep Neural Network Training Trained Model Generative Adversarial Network Evaluation Classification metrics AUC Score New data Bad Good Model Refinement Needed Bad Final Model Model Evaluation on the new data Good Robust model Figure 2.
The proposed methodology Data preprocessing As mentioned previously, we used the CIC-MalMem 2022 dataset in our work.
Our database contains 58596 rows and 57 columns, divided into benign .
and malicious .
The malicious class contains three categories which are Spyware .
Ransomware .
, and Trojan .
As a pre-processing of the data, we coded the benign class as 0 and the malicious class as 1.
As for the category column, we coded the benign as 0.
Spyware as 1.
Ransomware as 2, and Trojan as 3.
Int J Elec & Comp Eng.
Vol.
No.
June 2025: 3064-3074
Int J Elec & Comp Eng
ISSN: 2088-8708
Feature selection using genetic algorithm The selection of best features is a technique aims to list the significant features that help build better classifiers and reduce computational overload .
Genetic algorithms are optimization techniques inspired by natural selection.
They use populations of candidate solutions, favoring the fittest individuals through iterative selection, crossover, and mutation.
This mimics the process of evolution, aiding in efficiently finding optimal solutions to complex problems .
In our research we created a customized genetic algorithm for feature selection, this algorithm is shown in the algorithm 1.
Algorithm 1.
Features selection genetic algorithm Input: data, population_size, generations_number, mutation_rate.
Output: optimal_features population E initialize_population_randomly() score E evaluate.
foreach generation in generations_number do parents E select_parents_for_crossover.
opulation, population_size, scor.
next_generation E create_next_generation.
next_generation E make_mutation.
ext_generatio.
score E evaluate.
ext_generatio.
add_to_results.
core, next_generatio.
next_generation E elitism.
ext_generatio.
population E next_generation optimal_features E get_best_features_from_max_score() Through the iterative application of selection, crossover, and mutation operators, genetic algorithms explore the solution space, gradually improving the quality of solutions over successive generations.
promoting the survival and reproduction of individuals with higher fitness values, genetic algorithms efficiently navigate complex search spaces and discover high-quality solutions to optimization problems in a manner inspired by natural evolution.
Data training For the data training, we trained our final data using deep neural networks.
A deep neural network is same as an artificial neural network but with many hidden layers positioned between the input and output layers, enabling it to learn complex patterns in data.
It is characterized by its depth, which allows it to represent hierarchical features and abstract representations .
The Table 1 shows the structure of our deep neural network however Table 2 shows the used parameters for the training process.
Table 1.
The structure of the deep neural network Layer .
Dense Dropout Dense Dropout Dense Activation .
Output shape .
one, .
one, .
one, .
one, .
one, .
one, .
Table 2.
The used parameters for training deep neural network Parameter Learning rate Optimizer Loss function Epochs Value Adam Binary cross entropy In order to assess the efficacity of our model, we have used assessment criteria including accuracy, precision, recall, and F1-Score.
Accuracy is calculated by the proportion of properly identified cases, and it is computed using .
yaycaycaycycycaycayc = .
cNycE ycNycA) ycNycE ycNycA yaycE yaycA Enhancing malware detection with genetic algorithms and generative A (Abid Dhiya Eddin.
A ISSN: 2088-8708 The precision measure, which quantifies the accuracy of positive predictions, is computed using .
ycEycyceycaycnycycnycuycu = .
ycNycE yaycE Recall or sensitivity, it is defined by the proportion of true positive cases that were accurately predicted.
It is determined using .
ycIyceycaycaycoyco = .
ycNycE yaycA The F1-Score is a quantitative measure that combines recall and accuracy using their harmonic mean, achieving a balanced compromise between the two metrics .
, it calculated by .
ya1 Oe ycIycaycuycyce = 2 x ycEycyceycaycnycycnycuycu x ycIyceycaycaycoyco ycEycyceycaycnycycnycuycu ycIyceycaycaycoyco Another method for evaluating the performance of classification models is called receiver operating characteristic (ROC), and it is a graphical representation that shows how a binary classifier system may be made to function as a diagnostic across different threshold settings.
The ROC curve illustrates the trade-off between the classifier's sensitivity and specificity by plotting the true positive rate (TPR) versus the false positive rate (FPR) at different threshold levels.
area under the curve (AUC) is a scale from 0 to 1, with a higher number denoting the model's superior discriminating power .
Generative adversarial networks-based evaluation Generative adversarial networks (GAN.
are a type of artificial intelligence algorithms that comprise two neural networks, which are generator and discriminator.
These networks are trained concurrently using a competitive method.
The generator makes authentic synthetic data samples, while the discriminator acquires the ability to differentiate between genuine and counterfeit data.
By employing iterative training.
GANs have the ability to produce synthetic data of exceptional quality that nearly mirrors the distributions of actual data.
This makes GANs very effective tools for many tasks, including picture synthesis, data augmentation, and unsupervised learning .
, .
In our case, we used a generative adversarial network in order to generate data similar to the original data and test the effectiveness of our model when confronted with the generated Table 3 shows the structure of our generator neural network while Table 4 shows the structure of our discriminator neural network.
Table 3.
The generator structure Generator (Input: 100-dimensional noise vecto.
Layer .
Output Shape Dense (None, .
LeakyReLU (None, .
BatchNormalization (None, .
Dense (None, .
LeakyReLU (None, .
BatchNormalization (None, .
Dense (None, 1.
LeakyReLU (None, 1.
BatchNormalization (None, 1.
Dense (None, .
Activation (Tan.
(None, .
Table 4.
The discriminator structure Discriminator (Input: Data samples with 18 column.
Layer .
Output Shape Dense (None, .
LeakyReLU (None, .
Dropout (None, .
Dense (None, .
LeakyReLU (None, .
Dropout (None, .
Dense (None, .
Activation (Sigmoi.
(None, .
Int J Elec & Comp Eng.
Vol.
No.
June 2025: 3064-3074
Int J Elec & Comp Eng
ISSN: 2088-8708
The generator takes a 100-dimensional noise vector as input.
It consists of fully connected (Dens.
layers followed by LeakyReLU activation functions and BatchNormalization layers to stabilize training.
The final layer uses the Tanh in order to ensure that the output values are within the range [-1, .
, suitable for normalization if necessary.
The discriminator takes data samples with 18 columns as input .
eal or syntheti.
It uses a fully connected (Dens.
layers with LeakyReLU activation functions and Dropout layers to avoid as max the The final layer uses the Sigmoid activation function to output a probability .
indicating the likelihood of the input being real.
Table 5 shows the used parameters for training the generative adversarial Table 5.
The used parameters for training the generator and the discriminator Parameter Learning rate Optimizer Loss function Epochs Value Adam Binary cross entropy RESULTS AND DISCUSSION In this research a deep learning model based on genetic algorithms and generative adversarial networks was successfully created and tested.
In this section we will present and discuss the results we Which include feature selection based on genetic algorithms, data training using deep neural network and generation of new data using generative adversarial network and the evaluation of the model using the generated data.
Feature selection In the stage of feature selection, we employed a customized genetic algorithm with an adaptive mutation rate, binary chromosome representation, tournament selection, single-point crossover, and an elitism strategy to retain the best solution for the next generation.
The fitness function evaluated the classification accuracy using a random forest model, resulting in the selection of 18 features from the original 57, achieving an accuracy of 99.
This approach not only simplified the model by reducing its dimensionality but also potentially improved its interpretability and generalization.
The high accuracy indicates that the selected features contain significant discriminatory information for distinguishing between malware and benign data, demonstrating the effectiveness of the genetic algorithm in identifying the most relevant features and enhancing the performance of the intrusion detection system.
Training process After selecting the optimal set of 18 features using the customized genetic algorithm, we proceeded to train a deep neural network (DNN) on the reduced feature set.
The model was trained over multiple epochs, with the hyperparameters meticulously tuned to optimize performance.
Table 6 shows our training Table 6.
Training results Evaluation metric Precision Recall F1-Score Accuracy AUC Score Value The metrics mentioned in Table 6 indicate a highly accurate model.
The precision and recall values, both exceeding 99%, suggest that the model is highly effective in distinguishing between malware and benign samples.
The F1-Score, which balances precision and recall, further confirms the model's capability to minimize both false positives and false negatives.
However, the AUC score of 0.
9955 demonstrates that the model performs exceptionally well across different classification thresholds, indicating strong overall discrimination ability.
Figure 3 presents the confusion matrix of our model which provides further insight into the model's Out of the total instances evaluated, the DNN made only 52 misclassifications, comprising 29 false positives and 23 false negatives.
This low error rate underscores the model's robustness and reliability in Enhancing malware detection with genetic algorithms and generative A (Abid Dhiya Eddin.
A ISSN: 2088-8708 detecting malware.
In summary, the training results illustrate that the DNN, trained on the reduced feature set, achieves near perfect classification performance.
The high precision, recall.
F1-Score, accuracy, and AUC score collectively demonstrate the model's effectiveness in accurately identifying malware while maintaining a low misclassification rate.
Figure 3.
Confusion matrix of our model Model evaluation using the generated data This phase involved generating new data, including both benign and malicious samples, to further evaluate the effectiveness of our model.
We created a GAN and used it to generate 100 benign and 100 malicious samples, which were then used to test the model's performance.
Figure 4 shows the loss of the generator and discriminator with the malicious generated data.
Figure 5 shows the loss of the generator and discriminator with the benign generated data.
Figure 6 illustrate some generated malicious data while the Figure 7 illustrate some generated benign data.
When the generated samples were passed to the model, it recognized the malicious samples with 100% accuracy, while the benign samples were recognized with 96% accuracy.
This means that 4 benign samples were incorrectly classified as malicious.
Utilizing a GAN to generate new data helps assess the model's effectiveness against unseen data.
Achieving 100% accuracy with the generated malicious samples demonstrates that the GAN effectively captured the underlying distribution of malicious data.
However, the 96% accuracy with benign data suggests some room for improvement, possibly due to the inherent diversity and complexity of benign data.
This evaluation highlights the model's robustness and ability to generalize to unseen data, even when that data is synthetically generated.
Figure 4.
The loss of the malicious generated data Figure 5.
The loss of the benign generated data Int J Elec & Comp Eng.
Vol.
No.
June 2025: 3064-3074
Int J Elec & Comp Eng
ISSN: 2088-8708
Figure 6.
Some of the generated malicious data Figure 7.
Some of the generated benign data COMPARISON.
LIMITATIONS AND FUTURE WORKS
As a simple comparison with previous works, our research is characterized by its use of genetic algorithm to select the most important features in addition to relying on the smallest possible number of features .
, which allows us to show more accurate results during the training process.
Our work is also characterized by adding another evaluation stage based on the use of generative adversarial networks in order evaluate the effectiveness of the model, unlike other works that were satisfied with accuracy precision, recall and F1-Score.
This can confirm the model's efficiency in working with new and unseen data.
As for accuracy, the reason for its decrease compared to works .
, .
is due to our use of two dropout layers while training our deep neural network, in order to avoid falling into a state of overfitting.
Table 7 resume the comparison of our work with the similar works.
On the other side, our dataset does not represent the full spectrum of real-world malware for this, the model's effectiveness might be limited.
We can say also that GANs might not always capture the full diversity of data, as indicated by the 96% accuracy with benign data.
Another sensitive point is, that deep neural networks and GANs are often considered black box models, making it difficult to interpret their decision-making process.
This lack of transparency can be a limitation in understanding why certain data is classified as malware.
The implementation of explainable AI techniques will be useful for these cases.
Scaling the model to handle larger datasets or real-time detection scenarios might pose a good challenge.
Future research endeavors could advance this work by focusing on the development and implementation of practical tools, software solutions, or real-time systems that leverage the proposed model for effective and efficient malware detection in operational environments with the integration of explainable AI techniques for more transparent and interpretable decision-making processes.
Enhancing malware detection with genetic algorithms and generative A (Abid Dhiya Eddin.
A ISSN: 2088-8708 Table 7.
Comparison of our work with similar works Ref Dataset Feature Number of Training Accuracy Evaluation metrics .
CICMalMem2022
Logistic regression (LR) Malware Api Call
CICMalMem2022
MalMem2022
KNN
CNN
CICMalMem
CICMalDroid
CICMalMem2022
CICMalmem2022
CICMalMem
Stacking Oe Support vector
machine (SVM) Oe Catboost
Oe Histogram
Oe Random forest
(RF)
53% (AUC
Scor.
Precision Recall F1-Score
AUC Score
Precision Recall F1-Score
Precision Recall F1-Score
Precision Recall F1-Score
AUC Score
Precision Precision Recall F1-Score
CNN Bi-LSTM
Genetic DNN
Our research
Precision Recall F1-Score
Precision Recall F1-Score
Precision Recall F1-Score
AUC score
Syntheticdata using GANs
CONCLUSION
In this research, we utilized a genetic algorithm for feature selection, trained a deep neural network on the selected features, and employed generative adversarial networks to evaluate the effectiveness of our model against the new and unseen data and make it more generalized.
The combination of feature selection.
DNN training, and data generation techniques demonstrates a holistic and sophisticated approach to malware The high accuracy and performance metrics obtained across different stages of the study indicate the effectiveness of the proposed methodology.
These results have significant implications for cybersecurity, as they provide insights into the potential of machine learning models for accurately detecting malware and distinguishing it from benign data.
Furthermore, the application of GANs to generate synthetic data played a crucial role in our evaluation process.
The GAN-generated data mimicked real-world malware scenarios, providing a robust and diverse testing environment.
This allowed us to rigorously assess the performance of our model against previously unseen and potentially sophisticated malware variants.
In conclusion, this research represents a comprehensive and impactful contribution to the field of malware detection, showcasing the potential of machine learning techniques for addressing cybersecurity challenges.
The impressive results obtained across various stages of the study underscore the effectiveness and promise of the proposed methodology.
ACKNOWLEDGMENTS