JURTEKSI (Jurnal Teknologi dan Sistem Informas.
ISSN 2407-1811 (Prin.
Vol.
XII No 3.
June 2026, hlm.
437 Ae 444 ISSN 2550-0201 (Onlin.
DOI: http://dx.
org/ 10.
33330/jurteksi.
Available online at http://jurnal.
id/index.
php/jurteksi FPR-CONSTRAINED HYBRID DEEP LEARNING
FOR IOT ANOMALY DETECTION
Salma Nurkamila1 .
Suprih Widodo 1* Pendidikan Sistem dan Teknologi Informasi.
Univeritas Pendidikan Indonesia email: *supri@upi.
Abstract: Existing IoT anomaly detection studies have achieved high classification performance, but most focus on accuracy and F1-score without explicitly controlling the false positive rate (FPR).
In addition, many approaches rely on a single detection perspective, limiting their operational reliability.
To address this gap, this study proposes a hybrid anomaly detection framework integrating Long Short-Term Memory (LSTM).
Shannon entropy, and autoencoder reconstruction error.
Shannon entropy is incorporated as an additional feature, while LSTM and the autoencoder capture temporal and reconstruction characteristics.
The resulting hybrid representation is processed by a constraint-based threshold selection mechanism that enforces FPR Experiments on the TON-IoT and Edge-IIoTset datasets achieved average F1-scores of 9250 and 0.
9934, while maintaining average FPR values of 0.
0091 and 0.
0714, respectively.
Analysis of entropy distributions showed consistent differences between normal and anomalous traffic across both datasets, indicating that Shannon entropy provides discriminative information for anomaly detection.
These results demonstrate strong detection performance with controlled false alarms, while ablation studies confirm the significant contribution of Shannon entropy to overall model performance.
Keywords: false positive rate.
hybrid deep learning.
Internet of Things.
network anomaly detection.
Shannon entropy Abstrak: Penelitian deteksi anomali Internet of Things (IoT) telah menunjukkan performa klasifikasi yang tinggi, namun sebagian besar masih berfokus pada accuracy dan F1-score tanpa mengendalikan false positive rate (FPR) secara eksplisit.
Selain itu, banyak pendekatan hanya memanfaatkan satu perspektif deteksi sehingga reliabilitas operasionalnya masih terbatas.
Untuk mengatasi kesenjangan tersebut, penelitian ini mengusulkan kerangka deteksi anomali hybrid yang mengintegrasikan Long Short-Term Memory (LSTM).
Shannon entropy, dan autoencoder reconstruction error.
Shannon entropy digunakan sebagai fitur tambahan, sedangkan LSTM dan autoencoder menangkap karakteristik temporal dan deviasi rekonstruksi.
Representasi hybrid yang dihasilkan kemudian diproses melalui mekanisme constraint-based threshold selection dengan batas FPR Hasil pengujian pada dataset TON-IoT dan Edge-IIoTset menghasilkan F1-score rata-rata sebesar 0,9250 dan 0,9934, dengan FPR rata-rata sebesar 0,0091 dan 0,0714.
Perbedaan nilai entropy yang konsisten antara trafik normal dan anomali pada kedua dataset menunjukkan bahwa Shannon entropy menyediakan informasi diskriminatif untuk deteksi anomali.
Hasil tersebut menunjukkan performa deteksi yang kuat dengan false alarm yang terkendali, sementara studi ablasi mengonfirmasi kontribusi signifikan Shannon entropy terhadap performa model.
Kata kunci: deteksi anomali jaringan.
false positive rate.
hybrid deep learning.
Internet of Things.
Shannon entropy JURTEKSI (Jurnal Teknologi dan Sistem Informas.
ISSN 2407-1811 (Prin.
Vol.
XII No 3.
June 2026, hlm.
437 Ae 444 ISSN 2550-0201 (Onlin.
DOI: http://dx.
org/ 10.
33330/jurteksi.
Available online at http://jurnal.
id/index.
php/jurteksi INTRODUCTION in an operationally acceptable range.
Despite these studies showing high detection performance, evaluations are still focused on classification metrics such as accuracy, precision, recall, and F1-score, without explicitly imposing FPR as an operational constraint.
In addition, existing approaches generally leverage only a single detection perspective, such as temporal characteristics in LSTM-based models .
, .
, distributional characteristics in entropy-based approaches .
, .
, or reconstruction characteristics in autoencoder-based approaches .
, .
As a result, the complex nature of IoT traffic anomalies is not yet comprehensively represented .
, .
Furthermore, models with high classification performance do not necessarily achieve acceptable false alarm rates in practical IoT environments dominated by normal traffic .
, .
Given these limitations, there remains a need to develop anomaly detection frameworks that simultaneously capture temporal, distributional, and reconstruction characteristics while explicitly controlling false alarms through a structured threshold selection mechanism.
To address this gap, this study proposes a hybrid anomaly detection framework combining LSTM.
Shannon entropy, and autoencoder reconstruction LSTM captures temporal dependencies.
Shannon entropy represents local distributional changes, and the autoencoder quantifies reconstruction deviation .
, .
, .
Their integration produces a unified hybrid anomaly score that is subsequently processed by a constraintbased threshold selection mechanism as the final decision layer.
Unlike previous studies that focused on optimizing accuracy or F1score, this study applied a constraintbased threshold selection mechanism The proliferation of Internet of Things (IoT) devices across smart cities, industrial automation, healthcare, and critical infrastructures has increased network complexity and exposure to cyberattacks such as Distributed Denial of Service (DDoS).
Man-in-the-Middle (MITM), data injection attacks .
, .
, .
Conventional anomaly detection approaches remain limited in detecting zero-day attacks and adapting to traffic distribution shift .
Moreover.
IoT datasets often exhibit class imbalance and distributional variations that complicate anomaly detection .
, .
, .
Numerous studies have applied machine learning and deep learning approaches for IoT anomaly detection.
Kaya et al.
and Zamanzadeh Darban et al.
demonstrated that deep learning models are effective for capturing temporal anomaly patterns, but their evaluations primarily focused on classification performance without explicit false alarm In distribution-based detection.
Pandey and Mishra .
utilized entropybased features to identify traffic distribution irregularities, yet the approach did not integrate temporal characteristics.
Similarly, autoencoder-based methods proposed by Katbi and Ksantin .
and Salehiyan et al.
Relied on reconstruction error analysis but focused on reconstruction characteristics alone.
Consequently, generally employ a single detection perspective and rarely incorporate explicit FPR constraints, limiting their operational reliability in normal-traffic-dominated IoT environments.
As a result, it remains unclear whether temporal, distributional, and reconstruction characteristics can be effectively integrated while simultaneously maintaining false alarm rates with- JURTEKSI (Jurnal Teknologi dan Sistem Informas.
ISSN 2407-1811 (Prin.
Vol.
XII No 3.
June 2026, hlm.
437 Ae 444 ISSN 2550-0201 (Onlin.
DOI: http://dx.
org/ 10.
33330/jurteksi.
Available online at http://jurnal.
id/index.
php/jurteksi with an explicit objective of an operational FPR The FPR constraint was applied during validation to limit false alarms while maintaining a balance between detection capability and operational reliability .
This threshold was selected based on the base-rate fallacy principle proposed by Axelsson .
, which states that a small increase in FPR can result in a very large number of false alarms in environments dominated by normal traffic.
Therefore, more relaxed constraints, such as FPR
FPR
, were not considered because they could substantially reduce the practical reliability of the detection system .
, .
This approach is also aligned with the need to control the tradeoff between detection capability and false alarm, as discussed by Liu et al.
and Syrby and Ruocco .
The framework was evaluated on the TON-IoT .
and Edge-IIoTset .
datasets through 30 independent multi-seed experiments to assess model stability under random initialization.
Based on the identified research gap, this study makes three contributions.
First, it proposes a hybrid anomaly detection framework integrating temporal, distributional, and reconstruction characteristics through LSTM.
Shannon entropy, and an autoencoder.
Second, it incorporates a constraint-based threshold selection mechanism with an operational FPR limit of 0.
Third, it evaluates model stability and component contributions through multi seed experiments, ablation studies, and Wilcoxon Signed Rank Tests on the TON IoT and Edge IIoTset datasets.
METHOD
detection framework for IoT networks.
This study uses IoT network traffic data obtained from two public benchmark datasets: TON-IoT .
and Edge-IIoTset .
The TON-IoT dataset, developed by the University of New South Wales, consists of 211,043 samples with 44 features covering normal traffic and 7 attack categories, while the Edge-IIoTset dataset consists of 257,800 samples with 63 features and 14 attack categories.
Both datasets were selected because they represent different IoT traffic distribution characteristics, allowing for cross-dataset generalizability evaluation .
, .
Experiments were conducted using Python with TensorFlow.
Keras, and scikit-learn.
Independent variables include original features.
Shannon entropy, and autoencoder reconstruction error, while the dependent variable is the binary classification label.
All preprocessing was performed exclusively on the training set to avoid data leakage, including the removal of irrelevant features, handling missing values with zeros, label encoding for categorical features, and Min-Max normalization to the range .
, .
The dataset was divided using stratified sampling into a 70:20:10 split: 70% training, 20% validation, and 10% test.
Sliding windows and entropy calculations were carried out after separation to avoid temporal leakage.
Figure 1.
Proposed Hybrid IoT Anomaly Detection Framework This research is an experimental, computation-based study aimed at designing and evaluating a hybrid anomaly The feature engineering approach JURTEKSI (Jurnal Teknologi dan Sistem Informas.
ISSN 2407-1811 (Prin.
Vol.
XII No 3.
June 2026, hlm.
437 Ae 444 ISSN 2550-0201 (Onlin.
DOI: http://dx.
org/ 10.
33330/jurteksi.
Available online at http://jurnal.
id/index.
php/jurteksi in this study involves two complementary stages of additional feature extraction.
Shannon entropy is calculated to quantify the uncertainty of network traffic distribution locally within sliding windows of size 10 .
, .
, as given by equation .
( )
Oc ( ) ( ) regularization, a second LSTM layer with 32 units, and a Dense layer with sigmoid activation for binary classification.
Training uses the Binary Cross-Entropy loss function and the Adam optimizer.
Class imbalance is handled using class weights based on the class distribution in the training set, while early stopping based on validation loss is used to prevent overfitting.
The ablation studies were conducted using three configurations: w/o Entropy, w/o Autoencoder, and w/o Constraint.
Each configuration was evaluated through 30 multi-seed experiments with identical data splits to ensure evaluation consistency .
Statistical significance was assessed using the Wilcoxon Signed-Rank Test to compare the F1-score distributions across 30 multi-seed ablation experiments.
This nonparametric test was chosen because it does not assume normality in deep learning model performance .
Constraint-based threshold selection was applied to control FPR .
, .
Threshold candidates were evaluated on the validation set over the range - with an interval of 0.
Thresholds satisfying FPR retained, and the optimal threshold that maximized the F1-score was selected, as formulated in equation .
( )
The value ( ) represents the probability of a feature occurrence in a given window.
The Shannon entropy value quantifies the uncertainty in the traffic distribution and is used to distinguish between normal traffic and anomalous traffic .
Autoencoder reconstruction error is calculated using the Mean Squared Error (MSE) between the input and the autoencoderAos output.
A symmetric fully connected autoencoder was used to compute reconstruction error.
shown in equation .
( )
Oc( C) ( ) The variable denotes the input feature dimension .
The entropy and autoencoder reconstruction error features are then combined with the original features through feature augmentation.
The final input dimension of the model consists of 21 features for the TON-IoT and 13 features for Edge-IIoTset.
Sequence representation employed a sliding window of size 10, producing LSTM inputs of shape ( ), where the number of samples and the number of features .
The model architecture consists of two LSTM layers .
The first layer consists of 64 LSTM units, followed by Batch Normalization and Dropout for ( ) ( ) ( ) The optimal threshold from the validation set is consistently applied to the test set for final evaluation to avoid data leakage.
In addition, performance evaluation was conducted using four main metrics: F1-score, precision, recall, and FPR, as formulated in equations .
, .
, .
, .
, .
( )
JURTEKSI (Jurnal Teknologi dan Sistem Informas.
ISSN 2407-1811 (Prin.
Vol.
XII No 3.
June 2026, hlm.
437 Ae 444 ISSN 2550-0201 (Onlin.
DOI: http://dx.
org/ 10.
33330/jurteksi.
Available online at http://jurnal.
id/index.
php/jurteksi Table 2.
Distribution of Shannon Entropy and Reconstruction Error
EdgeParameter TON-IoT
IIoTset
Normal Entropy Anomaly Entropy Entropy Difference ( ) Normal Error
Anomaly Error
Anomaly/ Normal Ratio
( )
( )
( )
TP.
FP.
FN, and TN represent true positives, false positives, false negatives, and true negatives, respectively .
, .
Model stability against random initialization variation was evaluated across 30 experiments using identical data and different seeds.
Evaluation results are reported as mean standard deviation to provide a more statistically reliable performance estimate .
In both datasets, normal traffic exhibits higher Shannon entropy than anomalous traffic .
, .
The larger entropy difference in TON-IoT ( compared to ) indicates strong discriminative power, consistent with the findings of Pandey and Mishra .
Meanwhile, the separability of the autoencoder reconstruction error is much higher in TON-IoT with a ratio of 2y.
In contrast, the ratio in EdgeIIoTset is 1.
06y, indicating that the autoencoder contributes more actively to anomaly discrimination in TON-IoT, whereas its contribution in Edge-IIoTset is relatively limited owing to the smaller separation between normal and anomalous reconstruction errors .
These consistent entropy differences across both datasets indicate that Shannon entropy provides discriminative information for separating normal and anomalous traffic distributions.
To further evaluate model robustness, threshold stability, and the contribution of individual components, additional analyses were conducted through multi-seed experiments and ablation studies.
RESULT AND DISCUSSION
Table 1.
Single Experiment Evaluation Results on Both Datasets EdgeMetric TON-IoT IIoTset F1-score Recall Precision FPR Optimal thresh0.
old ( ) Edge-IIoTset produced a higher F1-score and recall, whereas TON-IoT produced lower FPR and higher precision.
These differences reflect the distinct distribution characteristics of the two datasets .
, .
Both datasets satisfied the FPR constraint, confirming the effectiveness of the proposed threshold selection mechanism in controlling false alarms while preserving anomaly detection capability .
JURTEKSI (Jurnal Teknologi dan Sistem Informas.
ISSN 2407-1811 (Prin.
Vol.
XII No 3.
June 2026, hlm.
437 Ae 444 ISSN 2550-0201 (Onlin.
DOI: http://dx.
org/ 10.
33330/jurteksi.
Available online at http://jurnal.
id/index.
php/jurteksi Table 3.
Threshold.
Ablation, and MultiSeed Stability Characteristics TONEdgeParameter IoT
IIoTset
Mean F1 Std
Mean Recall Mean FPR
Seeds with FPR
0/30
5/30
> 0.
Highest FPR (Seed .
(Seed .
Lowest FPR (Seed .
(Seed .
Threshold 65 Ae 20 Ae Range F1 w/o Entropy FPR w/o Entro0.
F1 w/o Autoen0.
FPR w/o Auto0.
F1 w/o Con0.
FPR w/o Con0.
ly higher FPR, whereas removing the autoencoder caused only a moderate performance decline .
Wilcoxon Signed-Rank Test results showed that removing entropy significantly reduced F1-score on TON-IoT ) and Edge-IIoTset In contrast, removing the autoencoder did not produce significant differences on either dataset.
Although removing the constraint mechanism did not significantly affect F1-score on EdgeIIoTset ( ), it eliminated the operational FPR boundary, potentially reducing practical reliability under varying traffic distributions .
, .
Multi-seed analysis also shows the trade-off between recall and FPR across datasets.
TON-IoT produces a higher threshold in the range of with a low recall of 0.
8633 and tightly controlled FPR of 0.
0091, while Edge-IIoTset achieves a recall of 0.
but with more variable FPR up to 0.
These findings indicate that constraint effectiveness is influenced by dataset distribution characteristics .
, .
The multi-seed results show that TON-IoT has better constraint stability than Edge-IIoTset.
All experiments on TON-IoT met the FPR constraint, ranging from while Edge-IIoTset exhibits greater variability, with five seeds violating the constraint and one extreme case reaching an FPR of 0.
The ablation study confirms that the entropy feature provides the most consistent contribution to model performance on both datasets.
Removing entropy caused the largest F1-score reduction and increased variability, confirming its contribution to both performance and stability.
Removing the constraint mechanism resulted in substantial-
CONCLUSION
This study successfully demonstrated that constraint-based threshold selection can maintain false alarm rates within a predefined operational boundary while preserving strong anomaly detection performance across different IoT traffic distributions.
The findings further confirm that integrating temporal, distributional, and reconstruction characteristics improves detection reliability, with Shannon entropy contributing significantly to both performance and stability.
Future work could develop adaptive thresholding mechanisms that dynamically adjust FPR limits to account for distribu- JURTEKSI (Jurnal Teknologi dan Sistem Informas.
ISSN 2407-1811 (Prin.
Vol.
XII No 3.
June 2026, hlm.
437 Ae 444 ISSN 2550-0201 (Onlin.
DOI: http://dx.
org/ 10.
33330/jurteksi.
Available online at http://jurnal.
id/index.
php/jurteksi tional shifts and evaluate model performance in more complex real-time IoT
BIBLIOGRAPHY