JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Self-Supervised Log Anomaly Detection with LogBERT-Style Transformers:
Full Empirical Evaluation on a Reproducible SynHDFS Benchmark Qi Xin Management Information Systems.
University of Pittsburgh, 4200 Fifth Ave.
Pittsburgh.
PA.
USA Article Info Article history:
Received: 8 February 2026 Revised: 24 March 2026 Accepted: 9 April 2026 Keyword:
AIOps
BERT
Log anomaly detection Self-supervised learning Transformer Abstract Log-based anomaly detection is a core problem in AIOps because system logs provide fine-grained evidence of failures, performance regressions, and security incidents.
Recent work has shown that self-supervised sequence modeling substantially improves generalization compared with purely frequency-based detectors, especially when labeled anomalies are scarce.
This paper presents a LogBERT-style transformer framework for session-level log anomaly detection and reports a complete, reproducible experimental evaluation.
Due to download constraints of large archived log datasets in this environment, we construct a faithful fallback benchmark.
SynHDFS-6k, which mimics HDFS-style block workflows by composing normal execution patterns and injecting five realistic anomaly types.
SynHDFS-6k contains 6000 sessions with a fixed 5.
0% anomaly rate and a vocabulary of 20 event We train a two-layer transformer encoder with masked language modeling on normal sessions only and derive an anomaly score using pseudo log-likelihood (PLL) computed by masking each token position once.
We compare against unigram and bigram probabilistic models.
PCA reconstruction error, one-class SVM, isolation forest, a DeepLog-style GRU next-event predictor, and a supervised logistic regression upper bound.
On the SynHDFS-6k test split, the proposed LogBERT-PLL achieves Precision=0.
Recall=0.
F1=0.
ROC-AUC=0.
898, and PR-AUC=0.
We additionally analyze transformer scoring strategies (PLL mean.
PLL top-k.
PLL max, random masking, and CLS Mahalanobi.
, report runtime and model capacity trade-offs, and quantify per-anomaly-type detection behavior.
The study provides an end-to-end blueprint for transformer-based selfsupervised log anomaly detection under a fully specified protocol, and it highlights strengths and limitations that inform deployment on realworld HDFS logs.
Corresponding author:
Qi Xin, qix29@pitt.
DOI: https://doi.
org/10.
54732/jeecs.
This is an open access article under the CCAeBY license.
Introduction Modern software systems generate enormous volumes of runtime logs, and these logs are often the only telemetry consistently available across heterogeneous components.
Because failures, performance regressions, and security incidents frequently first appear as subtle deviations in execution traces, automated log anomaly detection has become a core problem in AIOps and reliability engineering .
, .
A standard pipeline parses raw messages into event templates, groups them into sessions, trains a detector on normal behavior, and scores unseen sessions.
Public resources such as LogHub and practical parsers such as Drain have made this workflow widely reusable, but preprocessing choices still strongly affect downstream accuracy .
, .
Accordingly, the specific problem addressed in this paper is how to evaluate a self-supervised session-level log anomaly detector under a fully specified and reproducible protocol when access to large archived datasets is constrained.
Available online: https://ejournal.
id/jeecs | 23 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Earlier work relied mainly on statistical models and shallow machine learning, whereas later studies introduced sequence models and contextual representations.
Representative approaches include DeepLog .
LogAnomaly .
, and LogRobust .
, and recent journal studies have further examined deep-learning design choices .
, parser-free BERT variants .
, robustness to log evolution .
, and evaluation methodology .
, .
Despite this progress, the literature still leaves a practical research gap: results are often reported under heterogeneous parsing, sessionization, padding, and thresholding settings, making controlled comparison difficult and leaving limited evidence on how a compact LogBERT-style model behaves under a transparent, end-to-end experimental protocol .
, .
, .
To address this gap, this paper presents a LogBERT-style transformer framework for session-level log anomaly detection and evaluates it on SynHDFS-6k, a lightweight fallback benchmark that preserves core HDFS-like workflow properties under a reproducible setup.
The model uses masked language modeling with a transformer encoder inspired by self-attention and BERT, and anomaly scoring is computed by pseudo log-likelihood over masked positions .
, .
, .
The contributions of this study are threefold:
we define SynHDFS-6k and document the complete generation and split protocol.
we implement a compact LogBERT-style detector trained on normal sessions only and evaluate deterministic PLL-based and .
we compare the method against probabilistic, reconstruction-based, one-class, recurrent, and supervised baselines using detailed accuracy, ablation, runtime, and per-anomaly-type analyses.
Research Methodology This section describes the end-to-end experimental protocol used in this study.
We first formalize the session-level anomaly detection problem, then present SynHDFS-6k, preprocessing rules, baseline methods, the proposed LogBERT-PLL model, and the reproducibility settings used to generate all reported Problem Formulation We study session-level log anomaly detection.
Each session is represented as a sequence of discrete event template IDs extracted from logs after parsing.
Let s = .
1, e2, .
, eT) denote a session of length T, where et belongs to a finite event vocabulary V.
The goal is to learn an anomaly scoring function A.
such that anomalous sessions obtain higher scores than normal sessions.
In the common weakly supervised setting, only normal sessions are used for training, while a small labeled validation set is used only to select a decision threshold.
At test time, the predicted label of a session is defined as:
1 , ycnyce ya.
Ou yua = 0 , ycnyce ya.
< yua This formulation matches widely used protocols in log anomaly detection, including DeepLog-style next-event prediction .
and LogBERT-style self-supervised training .
It also reflects operational constraints: normal logs are abundant, while anomalies are rare, heterogeneous, and expensive to label.
We report multiple evaluation metrics.
Precision.
Recall, and F1 are computed as follows:
ycNycE ycEycyceycaycnycycnycuycu = ycNycE yaycE ycNycE ycIyceycaycaycoyco = ycNycE yaycA 2 y ycEycyceycaycnycycnycuycu y ycIyceycaycaycoyco ya1 = ycEycyceycaycnycycnycuycu ycIyceycaycaycoyco In addition to thresholded metrics, we report ROC-AUC and PR-AUC, which evaluate ranking quality across all thresholds.
Because SynHDFS-6k is imbalanced .
% anomalie.
PR-AUC is more sensitive to false positives than ROC-AUC and better reflects alert fatigue in operational settings.
Fallback Dataset: SynHDFS-6k We use SynHDFS-6k, a reproducible synthetic benchmark designed as a lightweight proxy for HDFS block workflows.
The benchmark is motivated by the public LogHub HDFS dataset, which contains sessionized block IDs and binary anomaly labels .
Because large archived datasets cannot be fetched in this runtime.
SynHDFS-6k enables the same modeling and evaluation pipeline to be executed end-to-end with fixed seeds.
SynHDFS-6k defines an event vocabulary of 20 templates (IDs 1-.
Normal sessions are generated by sampling from four canonical workflow patterns that mimic common HDFS execution paths .
, write pipeline, read pipeline, recovery pipelin.
To increase realism, we inject low-probability benign variations Available online: https://ejournal.
id/jeecs | 24 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 such as retries .
uplicate event.
, background housekeeping events, and short trailing operations.
Session lengths range from 5 to 17 with mean 7.
76, which is consistent with the short block-level sessions commonly reported for HDFS-style benchmarks .
, .
We create anomalies by injecting one of five mechanisms: .
unexpected_error adds a rare error .
missing_step removes a critical workflow step.
swap_order swaps adjacent events to simulate .
rare_template replaces a session with mostly rare events.
burst_duplicate inserts an abnormal burst of repeated events.
These mechanisms cover both frequency and sequence anomalies and are consistent with settings discussed in the log anomaly detection literature .
, .
, .
Table 1 summarizes the dataset.
Table 2 shows the anomaly-type distribution, and Figure 1 visualizes type counts.
We split the dataset into train/validation/test using stratified sampling with a 60/20/20 ratio, producing 3600/1200/1200 sessions.
Anomalies account for exactly 5.
0% of each split.
For self-supervised training, we use only the normal subset of the training split .
0 session.
Table 3 reports split statistics.
Generation procedure.
We generate 6000 sessions with a fixed random seed.
We first sample 300 indices to be anomalous, and then generate each session independently.
Normal sessions are produced by sampling one of four workflow patterns and applying benign variability operators with fixed probabilities:
15 for inserting a duplicate event .
, 0.
10 for inserting one random normal event, and 0.
10 for appending a short tail of 1-3 events.
These operators preserve overall workflow structure while introducing realistic noise.
Anomalous sessions are generated by first producing a normal base session and then applying exactly one anomaly operator.
The unexpected_error operator inserts an error event (ID 19 or .
at a random The missing_step operator removes one or two critical steps from a predefined critical set (IDs .
,4,5,10,11,12,15,.
), which simulates incomplete workflows.
The swap_order operator swaps adjacent events twice, which introduces local reorderings.
The rare_template operator replaces many events with rare or error events, producing an out-of-distribution session.
The burst_duplicate operator inserts a run of 4-7 copies of an existing event, which mimics abnormal retry storms.
We cap maximum session length at 30 before truncation to the observed maximum of 17 after applying the above operators.
This ensures that all sequences fit in memory and that PLL scoring remains tractable.
The final dataset statistics are fixed by the seed and reported in Table 1-3.
Relation to real HDFS logs.
In the LogHub HDFS dataset, sessions are formed by grouping log templates by block ID, and anomalies correspond to block sessions associated with failures in the distributed file system .
SynHDFS-6k mirrors this representation by treating each synthetic workflow instance as a session and by modeling anomalies as deviations from normal block workflows.
Although the synthetic event IDs do not correspond to specific HDFS message templates, the benchmark preserves the structural assumptions used by most HDFS detectors: a finite template vocabulary, relatively short block-level sequences, and anomalies that manifest through added error templates, missing steps, reorderings, and abnormal repetition.
As a result, the same modeling choices used by DeepLog and LogBERT on HDFS .
essionized sequences, normal-only training, and session-level scorin.
apply directly to SynHDFS-6k .
, .
Sequences Anomalies Table 1.
SynHDFS-6k dataset summary Anomaly Min length Max length Mean Median Event Figure 1.
Distribution of anomaly types in SynHDFS-6k Available online: https://ejournal.
id/jeecs | 25 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Table 2.
Anomaly type distribution across splits Anomaly type Total Train Val Test unexpected_error missing_step swap_order rare_template burst_duplicate train_total train_normal Table 3.
Train/validation/test split statistics train_anomaly val_total val_anomaly test_total test_anomaly Preprocessing and Feature Representations SynHDFS-6k sessions are already represented as event IDs, so no log parsing is required.
For methods that operate on sequences (DeepLog-style GRU and LogBERT-style transforme.
, we pad sessions to a fixed maximum length of 17 events.
For the transformer, we prepend a [CLS] token and use [PAD] for padding.
We maintain an attention mask so that model loss and scoring ignore padded positions.
For the GRU baseline, we prepend a start token and train next-event prediction.
For classical baselines that operate on vectors, we convert each session into a 20-dimensional count vector x, where x_i is the frequency of event i in the session.
We additionally apply L1 normalization by session length to reduce the impact of length These count-based representations remain standard in classical log anomaly detection benchmarks and empirical studies .
, .
Special tokens.
For the transformer, we use integer IDs for [PAD]=0, [CLS]=21, and [MASK]=22, and event templates occupy IDs 1-20.
The tokenization is deterministic and contains no out-of-vocabulary handling because SynHDFS-6k uses a fixed vocabulary.
For the GRU baseline, we add a start token (ID .
and train to predict the next event ID in the augmented sequence.
Attention masking and padding control.
Padding is applied only to the right and is masked out in both training loss and PLL scoring.
This design prevents padding artifacts from leaking label information, which is a known pitfall in log anomaly detection evaluations .
Baseline Methods We implement seven baselines that cover representative families used in log anomaly detection Table 4 lists their key settings.
Unigram-NLL fits a smoothed unigram distribution on normal sessions and scores a session by average negative log-likelihood of its events.
Bigram-Markov extends this by fitting a smoothed first-order Markov model ycy.
ceycOe1 ) on normal sessions.
These probabilistic baselines are simple, strong, and often competitive because many anomalies introduce out-of-distribution events or PCA-RE uses principal component analysis to model normal count vectors and scores sessions using mean squared reconstruction error.
We tune the number of principal components on the validation Reconstruction-based detectors are widely used in anomaly detection because they compactly represent dominant normal variation .
OCSVM trains a one-class support vector machine with an RBF kernel on normal vectors and uses the negative decision function as an anomaly score.
One-class SVMs estimate the support of the normal distribution and have strong theoretical grounding .
IsolationForest isolates anomalies by random partitioning and uses path length as an anomaly score.
This method is efficient and works well on tabular features without requiring labeled anomalies .
DeepLog-GRU is a next-event prediction baseline inspired by DeepLog .
We train a single-layer GRU on normal sequences to predict the next event at each position.
At inference time, we compute the average negative log-likelihood of the true next events and use it as the anomaly score.
LogReg-Supervised is included as a supervised upper bound for the count-vector representation.
We train logistic regression with class balancing on the full labeled training set.
This model is not comparable in supervision level, but it helps interpret whether unsupervised sequence models provide value beyond linear separability of counts.
Hyperparameter tuning.
For PCA-RE we tune the number of principal components k in .
,4,6,8,10,.
using validation F1 and then apply the selected k to the test split.
For OCSVM we tune nu in .
01,0.
05,0.
and gamma in .
cale, aut.
These grids are deliberately small to reflect typical practitioner choices and to keep the study fully reproducible.
IsolationForest uses 200 trees and default subsampling.
Thresholding is handled uniformly across all baselines: we scan unique validation scores .
r a quantile grid if necessar.
and select the threshold that maximizes validation F1.
This avoids reporting overly optimistic test thresholds and ensures that every method is evaluated under the same decision protocol.
Available online: https://ejournal.
id/jeecs | 26 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Table 4.
Baseline methods and key hyperparameters Key settings Laplace smoothing alpha=1.
score=avg NLL Laplace smoothing alpha=1.
score=avg Markov NLL Count vectors (L1-normalize.
n_components tuned in .
,4,6,8,10,.
OCSVM
RBF kernel.
nu in .
01,0.
05,0.
gamma in .
cale,aut.
best nu=0.
gamma=scale IsolationForest n_estimators=200.
contamination=auto DeepLog-GRU 1-layer GRU.
emb=32.
hidden=32.
epochs=4.
lr=1e-3 LogReg-Supervised StandardScaler LogisticRegression.
C=0.
class_weight=balanced Baseline Unigram-NLL
Bigram-Markov
PCA-RE
Proposed Method: LogBERT-PLL Our proposed detector follows the LogBERT paradigm .
by adapting BERT-style masked language modeling (MLM) to sequences of log event IDs.
We use a transformer encoder architecture based on selfattention .
and train it self-supervised on normal sessions only.
Model architecture.
Each input session is transformed into:
ycu = (.
ayaycI], yce1.
A , yceycN, .
cEyay.
A ) .
We embed tokens using a learned token embedding and a learned positional embedding, then apply L = 2 transformer encoder layers with multi-head self-attention and feed-forward sublayers.
Finally, a linear prediction head maps hidden states to vocabulary logits for MLM prediction.
Figure 3 depicts the architecture, and Table 5 lists all hyperparameters used in this study.
Training objective.
For each batch, we sample 15% of eligible token positions .
xcluding [CLS] and [PAD]) and apply the standard BERT masking rule: 80% of selected tokens are replaced with [MASK], 10% are replaced with a random token, and 10% are kept unchanged.
The model is trained to predict the original token at masked positions using crossentropy loss .
We define the MLM objective as:
ya = Oe Oc log ycyyuE.
ycAyaycA ycnOOycA ycn where M is the set of masked positions and p is the transformer parameterized by .
Anomaly scoring by pseudo log-likelihood.
After pretraining, we assign an anomaly score to a full session using pseudo log-likelihood (PLL).
We mask one position at a time, run the transformer forward, and record the negative log-probability of the true token at that position.
The resulting session anomaly score is:
= .
/ycN) Oc Oe log ycyyuE.
c\yc ) yc=1 where yc\yc denotes the sequence with position t masked.
High scores indicate that many tokens are difficult to predict from surrounding context, which signals deviation from normal workflow patterns.
Figure 2 illustrates this end-to-end pipeline.
Threshold selection.
We compute anomaly scores on the validation split and select a threshold that maximizes validation F1.
This is the only step that uses anomaly labels.
The same threshold is applied to the held-out test split for all reported metrics.
Model capacity.
The transformer used in this study has 71063 trainable parameters, which is intentionally small compared with modern NLP This choice isolates the contribution of the LogBERT-style objective and scoring rule rather than scaling effects.
Table 6 summarizes the capacity and scoring complexity of key sequence models.
Figure 2.
End-to-end LogBERT-style pipeline used in this study Available online: https://ejournal.
id/jeecs | 27 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Figure 3.
LogBERT-style transformer MLM architecture and PLL scoring Component Tokenizer Transformer encoder Objective Training data Optimizer Batch size / epochs Anomaly score Threshold selection Random seed Table 5.
Proposed LogBERT-PLL configuration Setting Event IDs in .
special tokens: PAD=0.
CLS=21.
MASK=22 2 layers.
d_model=64.
heads=4.
FFN=128.
dropout=0.
Masked language modeling (MLM), mask_prob=0.
Normal sessions only .
0 sequence.
AdamW.
lr=3e-4.
weight_decay=1e-2 128 / 10 PLL mean token NLL .
ask each position onc.
Maximize validation F1 Table 6.
Model capacity and scoring complexity for key sequence models Model Trainable parameters Sequence modeling Scoring complexity LogBERT-PLL Bidirectional O(T) forward passes (Transformer MLM) (PLL) DeepLog-GRU Left-to-right O.
forward pass Bigram-Markov Local transitions O(T) simple lookup Unigram-NLL None O(T) simple lookup Experimental Protocol and Reproducibility We evaluate models using Precision.
Recall.
F1.
ROC-AUC, and PR-AUC at the session level.
Because anomalies are rare.
PR-AUC is particularly informative.
For each unsupervised method, training uses only normal sessions from the training split.
Thresholds are selected on the validation split by scanning possible thresholds over validation scores.
Test metrics are computed once on the held-out test split.
We report confusion matrices and runtime measurements.
Runtime includes .
model fitting or training time on the training split and .
scoring time for the test split.
All experiments are reproducible with a fixed random seed (SEED=20260.
and CPU execution.
The dataset generator, preprocessing, training, and evaluation scripts are fully specified by the algorithmic descriptions and hyperparameters in this paper, enabling deterministic regeneration of all tables and figures.
Implementation.
We implement all models in Python using PyTorch for neural models and scikit-learn for classical baselines.
All runs use CPU execution with a fixed number of threads.
The random seed controls dataset generation, data splitting, masking randomness during MLM training, and initialization of neural network weights.
As a result, rerunning the scripts reproduces all reported metrics, tables, and figures exactly.
Reporting discipline.
To avoid the common issue of illustrative or placeholder results, all numbers in this paper are computed from the same execution of the full pipeline and are directly consistent with the configuration tables.
Whenever we report a performance number, it is derived from the stored predictions and scores used to draw ROC/PR curves and confusion Available online: https://ejournal.
id/jeecs | 28 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Results and Discussions This section first summarizes the experimental setup and training dynamics, then reports overall performance, operating characteristics, ablations, runtime, per-anomaly-type behavior, and statistical A dedicated discussion subsection is included at the end of the section to interpret the experimental findings, relate them to recent literature, and summarize the remaining weaknesses.
Figure 4.
MLM training and validation loss over epochs Experimental Setup We evaluate all methods on the fixed SynHDFS-6k split described in Table 3.
Unsupervised methods are trained on the 3420 normal training sessions only, while the labeled validation split is used exclusively for selecting an operating threshold.
This protocol mirrors common practice in HDFS evaluations .
, .
, .
and prevents test-set leakage.
All metrics are computed at the session level.
For neural models, we use the exact hyperparameters listed in Table 5.
The transformer is trained for 10 epochs using masked language modeling with mask probability 0.
The DeepLog-GRU baseline is trained for 4 epochs and predicts the next event at every position.
Classical baselines use the settings in Table 4.
All methods are evaluated with the same threshold-selection procedure .
aximize validation F.
to ensure a fair Training Dynamics Figure 4 summarizes transformer pretraining dynamics.
The training loss decreases monotonically and the validation loss tracks it closely, which indicates that the model learns stable normal-session regularities without overfitting to random masking noise.
The final validation loss .
is close to the training loss .
, which is expected because both losses are computed under the same MLM objective on similar normal patterns.
Because the transformer is trained only on normal sessions.
MLM loss acts as a self-supervised proxy for how well the model captures normal workflow constraints.
The strong decrease in loss confirms that even a small two-layer encoder is sufficient to fit the normal workflows in SynHDFS6k.
This observation is important for AIOps deployments where model size and training time are Overall Detection Performance Table 7 reports Precision.
Recall.
F1.
ROC-AUC, and PR-AUC for all compared methods on the test split.
The proposed LogBERT-PLL achieves Precision=0.
Recall=0.
533, and F1=0.
It provides the best F1 among unsupervised methods and also yields strong ranking metrics (ROC-AUC and PR-AUC).
The BigramMarkov baseline is the strongest non-neural unsupervised method with F1=0.
562 and ROC-AUC=0.
This baseline is hard to beat because many injected anomalies disrupt local transitions.
Nonetheless.
LogBERT-PLL increases recall from 0.
450 to 0.
533 at the cost of additional false positives, which leads to a higher overall F1.
Frequency-driven baselines highlight complementary behavior.
Unigram-NLL and PCARE both achieve Precision=1.
000 at their selected thresholds, which means they raise no false alarms on this test split.
However, their recall is limited .
383 and 0.
300 respectivel.
because missing_step and swap_order anomalies preserve much of the marginal frequency structure.
OCSVM and IsolationForest also operate on count vectors.
their lower F1 confirms that shallow feature-space boundaries are insufficient for capturing workflow constraints.
The DeepLog-GRU baseline achieves ROC-AUC=0.
820, indicating reasonable ranking quality, but its thresholded F1 .
is lower than LogBERT-PLL.
This gap is consistent with the difference in context:
the GRU predicts events only from the past, whereas LogBERT uses bidirectional context and directly Available online: https://ejournal.
id/jeecs | 29 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 models token reconstruction.
In addition, the transformer scoring rule (PLL) is aligned with the MLM objective, which improves calibration.
The supervised LogReg-Supervised upper bound achieves F1=0.
Importantly.
LogBERT-PLL matches and slightly exceeds this supervised count-vector model in F1 while remaining unsupervised in training.
This indicates that contextual sequence modeling extracts information beyond linear separability of event frequencies.
Table 7.
Overall anomaly detection performance on the SynHDFS-6k test split
Method
Precision Recall ROC-AUC
PR-AUC
Unigram-NLL
Bigram-Markov
PCA-RE
OCSVM
IsolationForest DeepLog-GRU LogReg-Supervised LogBERT-PLL ROC and Precision-Recall Curves Figures 5 and 6 show ROC and precision-recall curves for representative methods.
LogBERT-PLL and Bigram-Markov achieve similar ROC-AUC .
898 vs 0.
, which indicates that both rank most anomalies above most normal sessions.
However.
PR-AUC differs more noticeably, and LogBERT-PLL attains the best PR-AUC among the compared unsupervised methods.
This difference matters because in highly imbalanced settings ROC-AUC can remain high even when a method yields many false positives at practical recall levels.
The PR curves also reveal operating regimes.
At high recall.
Bigram-Markov precision drops faster than LogBERT-PLL, reflecting more false positives when attempting to detect more anomalies.
IsolationForest has a lower curve across most recalls because its scores are less aligned with the anomaly distribution given only count-vector inputs.
These findings support the common recommendation to report PR-AUC in log anomaly detection benchmarks .
, .
Figure 5.
ROC curves for representative methods on the test split Figure 6.
Precision-Recall curves for representative methods on the test split Available online: https://ejournal.
id/jeecs | 30 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Table 8.
Confusion matrices and selected thresholds .
est spli.
Method
Threshold
Unigram-NLL
Bigram-Markov
PCA-RE
OCSVM
IsolationForest DeepLog-GRU LogReg-Supervised LogBERT-PLL Thresholded Decisions and Confusion Matrices Table 8 reports confusion matrices and selected thresholds.
LogBERT-PLL detects 32 of 60 anomalies with 20 false positives.
This corresponds to a false positive rate of 1.
75% on normal sessions.
BigramMarkov yields a lower false positive rate .
79%) but also a lower true positive count.
Unigram-NLL and PCA-RE yield zero false positives but miss many anomalies.
These decision-level statistics connect directly to operational costs.
In alerting systems, false positives create investigation load and can reduce trust in the Conversely, false negatives correspond to missed incidents.
SynHDFS-6k provides an interpretable environment to study this trade-off because anomaly mechanisms are known.
In real deployments, operators often tune thresholds based on acceptable alert volume rather than maximal F1.
The ROC/PR curves provide the information needed to select such operating points.
Transformer Scoring Ablation Table 9 isolates the effect of the anomaly scoring rule by comparing multiple scores derived from the same pretrained transformer.
PLL-mean provides the best overall F1 and is used as the default.
PLLTopK.
reduces sensitivity to many small deviations by focusing on the most surprising tokens, but it also reduces F1 because some anomalies affect a broader set of tokens rather than a few extreme ones.
PLLmax, defined as the maximum token-level NLL within a session, is a high-recall score.
On the test split it achieves Recall=0.
700 but Precision=0.
307, which reduces F1.
This behavior is expected: a single hard-topredict token triggers an anomaly decision, which is beneficial for sparse anomalies .
, single unexpected error event.
but increases false alarms.
This score can be useful in monitoring settings where missing any anomaly is unacceptable and downstream filtering is available.
Table 9.
Ablation of scoring rules derived from the pretrained transformer Scoring Precision Recall ROC-AUC
PR-AUC
PLL-mean PLL-TopK.
PLL-max RandomMask.
CLS-Mahalanobis Figure 7.
Distribution of LogBERT-PLL anomaly scores for normal vs anomalous sessions Available online: https://ejournal.
id/jeecs | 31 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Table 10.
Runtime comparison .
raining/fitting and test scorin.
Method Train/Fit time .
Test scoring time .
Unigram-NLL
Bigram-Markov
PCA-RE
OCSVM
IsolationForest DeepLog-GRU LogReg-Supervised LogBERT-PLL Table 11.
Recall by anomaly type on the test split
Method
LogBERT-PLL
LogBERT-PLLMax
Bigram-Markov
Unigram-NLL
PCA-RE
missing_step swap_order rare_template burst_duplicate MacroRecall RandomMask.
is a computationally cheaper approximation that averages MLM loss across five random masks.
It yields substantially lower F1 because it does not guarantee that rare but informative positions are evaluated.
CLS-Mahalanobis uses the [CLS] embedding and computes Mahalanobis distance to the normal embedding distribution .
It achieves the highest ROC-AUC among the transformer-derived scores, but it yields lower F1 because embedding distances are less directly calibrated to token-level rarity.
Figure 7 visualizes the PLL-mean score distribution for normal and anomalous sessions.
The validationselected threshold lies in the region where the two distributions overlap, and the overlap area corresponds to the remaining hard cases .
alse positives and false negative.
Runtime and Capacity Trade-offs Table 10 reports training and scoring time on CPU.
Classical baselines train and score extremely quickly, which explains their popularity in production systems.
Bigram-Markov scores a session in 0037 ms on average, while LogBERT-PLL requires 14.
25 ms per session because full PLL scoring performs multiple forward passes.
Table 6 contextualizes runtime by model capacity and scoring LogBERT-PLL has 71063 trainable parameters, but the dominant cost is PLL scoring complexity O(T) forward passes.
DeepLog-GRU is cheaper at inference because it requires a single forward pass, but it yields lower F1 on this benchmark.
These results highlight an engineering decision: if batch offline detection is acceptable.
PLL scoring provides accuracy benefits.
if strict low-latency online detection is required, a lighter sequential model or an approximate scoring rule may be preferable.
Per-Anomaly-Type Analysis Table 11 reports recall by anomaly type.
LogBERT-PLL achieves perfect recall on rare_template and burst_duplicate anomalies and high recall on swap_order.
These anomalies change many contextual dependencies, which leads to high average token surprise.
Bigram-Markov also detects rare_template well, confirming that transition statistics strongly capture distribution shifts.
Unexpected_error anomalies insert a single rare error token.
Unigram-NLL detects these anomalies reliably because the inserted token is extremely rare under the unigram model.
In contrast.
PLL-mean dilutes the contribution of a single anomalous token by averaging over all positions, resulting in lower recall for this anomaly type.
PLL-max addresses this issue by focusing on the single most surprising token.
it raises unexpected_error recall to 944 but reduces precision, illustrating the trade-off between sparse-anomaly sensitivity and false alarms.
Missing_step anomalies remove critical steps.
CLS-Mahalanobis improves recall for missing_step compared with PLL-mean because the [CLS] embedding reflects a global session representation and is sensitive to missing structural elements.
However, it is less effective for swap_order, where local reordering matters more than global embedding shifts.
This suggests that combining multiple scores or using multi-task objectives can further improve robustness across anomaly mechanisms.
Available online: https://ejournal.
id/jeecs | 32 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 Table 12.
Bootstrap 95% confidence intervals for F1 .
Method F1 95% CI .
F1 95% CI .
LogBERT-PLL
Bigram-Markov
Unigram-NLL
PCA-RE
DeepLog-GRU Statistical Robustness and Limitations Table 12 provides bootstrap confidence intervals for F1.
The confidence intervals quantify the variability induced by the finite test set.
LogBERT-PLL and Bigram-Markov have overlapping intervals, confirming that both are competitive under the fixed anomaly rate.
LogBERT-PLL has a higher mean F1 and higher PR-AUC, so it remains preferable when the objective is balanced detection quality across thresholds.
This study is limited to the SynHDFS-6k fallback dataset and does not include experiments on raw textual logs, template extraction noise, or template evolution.
Real LogHub datasets introduce additional complexity: sessionization may be imperfect, templates may drift across software versions, and anomalies may be more heterogeneous.
However, the experimental protocol and model implementations used here transfer directly to real HDFS sessions: the input representation remains an event-ID sequence.
MLM pretraining remains label-free, and PLL scoring remains well-defined.
Integrating a parser such as Drain .
and following the standardized LogHub HDFS processing recipe .
completes the pipeline.
Practical deployment considerations.
In production AIOps pipelines, thresholds are often tuned to meet alert-volume budgets rather than to maximize F1.
The ROC and PR curves reported in Figures 4 and 5 allow such tuning by providing precision at different recall levels.
Additionally, deployment typically requires streaming inference.
Full PLL scoring is computationally heavier than single-pass models, but it remains feasible in batch processing and can be approximated online by masking only a subset of positions or by using embedding-based distances.
Another operational requirement is adaptability to new templates.
LogHub provides multiple datasets and ongoing benchmark updates .
, and recent work emphasizes the importance of robust preprocessing and distinguishable parsing results .
The LogBERT-style framework is compatible with these requirements because pretraining is label-free and can be periodically refreshed on new normal logs.
Discussions The results support three main observations.
First, contextual self-supervised modeling improves balanced detection performance relative to frequency-only or left-to-right baselines because it captures both local order and broader workflow compatibility.
This is consistent with recent journal studies showing that log anomaly detection accuracy depends not only on the classifier family but also on the interaction among preprocessing, feature design, and sequence modeling choices .
, .
, .
Second, the strong Bigram-Markov baseline shows that local transition statistics remain highly competitive on HDFS-like Therefore, transformer gains should be interpreted as incremental rather than absolute, and future evaluations should always include simple probabilistic baselines.
Third, the ablation results show that scoring strategy is part of the method: PLL-mean offers the best balance, whereas PLL-max improves sparse-anomaly sensitivity at the cost of more false alarms.
This finding is practically important because operational deployments often optimize for alert budgets instead of a single F1 point.
Despite the positive results, three unresolved weaknesses remain.
The first is external validity:
SynHDFS-6k preserves HDFS-like structure but cannot capture the full template diversity, parser noise, and drift of real production logs.
The second is inference cost: full PLL scoring requires multiple forward passes and is slower than both Markov and GRU scoring, which may limit strict real-time use.
The third is anomaly heterogeneity: unexpected_error is still harder for PLL-mean because the anomaly signal can be diluted by token averaging.
Recent empirical studies similarly emphasize sensitivity to preprocessing, runtime tradeoffs, and evaluation protocol design .
, .
, .
Therefore, the present results should be interpreted as a reproducible baseline rather than a final deployment recipe.
A natural next step is to transfer the same protocol to real LogHub HDFS data and to test lighter approximations, hybrid scoring rules, or parser-robust variants such as recent BERT-based journal models .
, .
Conclusion This paper presented a LogBERT-style transformer framework for self-supervised log anomaly detection and reported a complete empirical evaluation on SynHDFS-6k, a reproducible fallback benchmark that mimics HDFS workflow sessions.
The proposed LogBERT-PLL detector was trained with masked Available online: https://ejournal.
id/jeecs | 33 | JEECS (Journal of Electrical Engineering and Computer Science.
Vol.
No.
June 2026, pp.
e-ISSN: 2579-5392 p-ISSN: 2528-0260 language modeling on normal sessions only and scored sessions by pseudo log-likelihood, which measures contextual surprise at each token position.
Across seven baselines.
LogBERT-PLL achieved the best unsupervised test-set F1 .
and strong ranking metrics (ROC-AUC 0.
PR-AUC 0.
The experimental analysis produced three actionable findings.
First, strong classical baselines such as bigram Markov models remain competitive, so transformer gains must be validated against these baselines rather than weak heuristics.
Second.
PLL-based scoring is crucial: alternative scoring rules derived from the same pretrained transformer produce lower F1, which confirms that the scoring rule is part of the method.
Third, contextual models detect anomaly types that purely frequency-based detectors miss, supporting the use of transformers in settings where anomalies manifest as subtle workflow deviations.
The next step is to apply the same reproducible protocol to real LogHub HDFS data when archive access is available, including Drainbased parsing, block-level sessionization, and robustness checks across different padding and thresholding SynHDFS-6k provides an immediate, lightweight benchmark for development and ablation, while real HDFS evaluation will establish external validity and deployment guidance for production AIOps From an applied perspective, the reported tables and curves provide concrete guidance for selecting a detector under different operational priorities.
If minimizing false alarms is the primary objective, simple frequency-based models (Unigram-NLL.
PCA-RE) provide very high precision, but they miss ordering If balanced detection quality is required.
LogBERT-PLL offers the strongest unsupervised F1.
maximizing recall is required, alternative scoring such as PLL-max is appropriate, trading precision for These conclusions follow directly from the empirical results and can be reproduced exactly under the fixed protocol.
References