TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 https://doi.
org/10.
51967/tepian.
3355 A 2025 Tepian Politeknik Pertanian Negeri Samarinda This work is licensed under a Creative Commons Attribution 4.
0 License CC-BY Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text Markus Adrian Karjadi * Handri Santoso Information Technology.
Pradita University.
Tangerang, 15810.
Indonesia adrian@student.
Information Technology.
Pradita University.
Tangerang, 15810.
Indonesia santoso@pradita.
*Corresponding Author Submitted: 2025-05-19.
Accepted: 2025-05-29.
Published: 2025-06-01 AbstractAi Large Language Models (LLM.
have shown tremendous potential in intent classification tasks, yet their practical deployment in low-resource language environments remains underexplored.
This study presents an informatics-based evaluation framework to compare three LLM architecturesAiGPT-Neo .
ine-tune.
Mistral, and Phi-2.
ero-shot inferenc.
Aion Indonesian intent The methodology integrates classic informatics approaches such as stratified sampling, label encoding, model evaluation using Scikit-learn, and a REST API-based local inference pipeline via the Ollama The study also benchmarks computational efficiency by profiling execution times on consumer-grade GPT-Neo achieved 100% accuracy after finetuning, while Mistral and Phi-2.
0 scored approximately 55% and 18%, respectively, in zero-shot settings.
The hybrid architecture designed in this work demonstrates how LLMs can be systematically evaluated and deployed using lightweight, modular informatics workflows.
Results suggest that fine-tuned lightweight models are viable for high-accuracy deployment, while zero-shot models enable rapid prototyping under constrained KeywordsAi Informatics Framework.
Intent Classification.
GPT-Neo Fine-Tuning.
Indonesian NLP.
Ollama REST API.
Large Language Models.
Lightweight AI Deployment INTRODUCTION Intent classification is a key natural language understanding (NLU) task that enables digital systems to interpret user goals from input text.
It plays a crucial role in customer service bots, voice assistants, and fintech From an informatics perspective, intent classification represents a classification problem that involves text preprocessing, feature engineering, model execution, and system evaluation.
In low-resource language contexts such as Bahasa Indonesia, applying informatics methodologies for data labeling, model evaluation, and runtime optimization becomes critical.
However, existing studies focus mainly on model performance, with little emphasis on how to design and benchmark AI-based classification workflows using core informatics principles such as system modularization, evaluation metrics, and deployment feasibility.
Traditional intent classification systems are constructed on top of heavily rule-based engines or light-scale supervised training models based on hand-crafted features and minimal amounts of annotated data.
These models are generally less adaptable, low in flexibility to new situations, and less precise, especially in low-resource languages like Bahasa Indonesia.
The lack of large, annotated Indonesian datasets also makes it hard to develop strong models tailored to local use cases.
With the introduction of Large Language Models (LLM.
, i.
GPT-style architecture, the field has witnessed a paradigm shift.
LLMs are capable of generalizing over languages and tasks with their enormous pretraining over multilingual and domain-distributed This opens new avenues for leveraging more adaptive, intelligent, and scalable intent classification solutionsAieven in limited-resource settings.
However, training or fine-tuning LLMs is expensive, requiring high computational resources and technical skills.
This task addresses such challenges through an examination of the comparative effectiveness of three LLMs, namely GPT-Neo.
Phi-2.
0, and Mistral, in intent classification for Indonesian language data sets.
GPT-Neo is fine-tuned on a labeled dataset to test the supervised learning performance upper bound.
Phi-2.
0 and Mistral are deployed using the local inference framework.
Ollama, under zero-shot conditions to mimic deployment situations that render training impossible.
This comparative setting captures real-world deployment situations and limitations.
Our objectives are two-fold, to compare the precision and cost trade-offs of fine-tuning versus zero-shot inference, and to identify which LLM configuration is best suited for real-world application on consumer-grade hardware such as a MacBook M3.
The research contributes a methodological framework to test LLMs in lowresource, multilingual environments and offers practitioners guidelines on selecting cost-effective AI solutions for intent classification.
This work proposes that a hybrid method using finetuned light models and zero-shot inference using modern LLMs can potentially provide both scalability and performance, particularly for uses in Indonesian financial Ae 97 Ae TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 Karjadi.
, & Santoso.
Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text.
TEPIAN, 6.
https://doi.
org/10.
51967/tepian.
services and customer interaction systems.
To our best knowledge, this is the first empirical contrast of inferenceonly and fine-tuned LLMs for Indonesian intent classification based on locally hosted models via the Ollama framework on consumer-grade hardwareAi offering a new perspective in performance, cost, and deployment feasibility in low-resource language II.
LITERATURE REVIEW
Application of Large Language Models (LLM.
in intent classification has been gaining popularity over the past few years.
Hadi et al.
presented a comprehensive overview of the potential of LLMs to perform exceptionally well on an immensely diverse range of NLP applicationsAitext generation, classification, and semantic understandingAiand outlined their usability, limitations, and deployment problems.
Du .
identified the use of financial LLMs as an example of how effective they are in domain-specific work involving strongly specialized linguistic patterns, such as handling financial documents and customer intent detection.
Furthermore.
Karlsson et al.
presented SEACrowd, a multilingual benchmark suite for Southeast Asian languages like Bahasa Indonesia, thereby bridging an important resource gap in low-resource language These experiments as a whole form the foundation for multilingual and low-resource test cases upon which the comparison in this paper is based, namely inference-only and fine-tuned models for Indonesian intent The efficiency of small language models has also been recently noted.
Subramanian et al.
explored the ability of Small Language Models (SLM.
and depicted their competitiveness in various tasks even with fewer Suhaeni and Yong .
addressed class imbalance in sentiment analysis using GPT-3-generated synthetic sentences, depicting the versatility of LLMs in data augmentation usage.
In computer security.
Ferrag et al.
introduced LLM and generative AI use and exploitation with an emphasis on security considerations in their adoption.
Shao et al.
introduced a holistic overview of different architectures of LLM, highlighting architectural trends and challenges.
Similarly.
Yigit et al.
covered the possibilities of generative AI in protecting critical infrastructure, introducing key opportunities and Zhang et al.
described LLMs from the software engineering point of view, particularly for applications such as software testing and code comprehension.
Prottasha et al.
presented Semantic Knowledge Tuning (SK-Tunin.
, a parameter-effective fine-tuning approach with improved model performance and training efficiency using semantically meaningful Zhang et al.
and Xu et al.
conducted systematic reviews at the intersection of LLMs and cybersecurity based on the necessity of robust and evolving models in high-risk digital environments.
These experiments highlight the increasing diversity of LLMs in intent classification and beyond as a precursor to comparative analysis in this research.
Goodfellow et al.
's .
groundbreaking work on Generative Adversarial Networks made scalable generative modeling approaches that empower modern LLMs.
While LLMs progressed further, there was increased focus on governance, fairness, and transparency.
Raji et al.
proposed an end-to-end internal auditing framework for AI systems, promoting embedded accountabilityAiparticularly in financial contexts where unfair output might have severe consequences.
Filling the gap of data scarcity.
Hu et al.
investigated artificial data generation techniques for promoting financial inclusion.
They discover that realistic synthetic data sets can lower data shortages and improve credit access among the underbanked.
Such approaches are especially relevant in Indonesia, where intent classification labeled data sets remain in limited quantity.
Feng .
demonstrated that adaptive AI-driven lending algorithms, with both behavioral and contextual data, have the potential to deliver more contextual and sensitive credit products than traditional static models.
Such innovations are providing avenues for inclusive digital credit architectures.
On the regulation and policy side.
Mirishli .
examined current legal frameworks, pointing to regulatory mandates' vagueness and the future challenges of hallucination, liability, and explainability in AI models.
Maple et al.
shared the same view, warning that without regulatory being revamped.
LLM utilization in financial services could offer systemic risk.
In December 2024, the U.
Department of the Treasury released a policy report summarizing industry feedback on AI deployment in financial services (U.
Department of the Treasury, 2.
The report emphasized the need for .
the definition of AI and model risk, .
uniform standards for data governance and quality, .
augmenting consumer protections, and .
enabling international harmonization of AI regulations.
To acknowledge the complexity and non-deterministic nature of generative AI, the Treasury demanded far more stringent governance conditions than were required for traditional machine This further to which FinRegLab .
pulled conclusions together from its AI Symposium, building on insights by over 260 attendees from academia, government, and industry.
Four dominating themes did:
Utilizing AI for improving domestic financial Developing inclusive and fair AI systems Embedding human control in AI processes Enhancing regulations to address new technical, market, and ethical challenges.
The report also advocated hybrid deployment models that integrate the virtues of zero-shot architecture and fine-tuned LLMs, particularly in resource-constrained settings.
Ae 98 Ae TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 Karjadi.
, & Santoso.
Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text.
TEPIAN, 6.
https://doi.
org/10.
51967/tepian.
METHODS
Overview of the Hybrid Strategy The increasing accessibility of Large Language Models (LLM.
has enabled their deployment in real-world applications, such as low-resource scenarios.
Real-world deployment is, however, typically under constraints such as computational resources and lack of available labeled data for fine-tuning.
This paper circumvents these constraints by presenting a hybrid method that takes advantage of supervised fine-tuning and zero-shot inference to provide a trade-off between performance and Specifically.
GPT-Neo is selected as the finetuned model since it is a lightweight architecture that can be executed on consumer-grade hardware.
On the other hand.
Mistral and Phi-2.
0 are utilized in zero-shot mode through the Ollama framework, which allows for rapid testing without additional training.
The reason for doing this is to simulate deployment conditions in environments where only partial infrastructure is available, i.
, academic labs, fintech proofs-of-concept, low-budget The hybrid model allows for flexible model selectionAifine-tuning where resources and data allow, with zero-shot inference allowing for quick prototyping.
The experiment is executed on an Apple Silicon MacBook M3, using Metal Performance Shaders (MPS) via PyTorch for maximum performance on the native GPU.
Figure 1 illustrates the workflow pipeline of this hybrid approach, showing both the supervised and inference-based This arrangement offers flexibility to allow institutions to select methods based on their computational and data constraints.
Figure 1.
Workflow of the hybrid methodology combining GPT-Neo fine-tuning and Ollama-based zeroshot inference.
Dataset and Preprocessing This study utilizes a custom-curated dataset consisting of 11,562 textual entries in Bahasa Indonesia, designed for intent classification tasks.
Each data entry is a sentence that simulates user expressions typically encountered in financial services and customer service domains.
The dataset includes three distinct intent categories: inquiry and loan_application.
These labels represent high-priority interaction types in digital banking interfaces, making them relevant for testing LLM capabilities in real-world classification tasks.
To prepare the dataset for modeling, all text entries were first cleaned and normalized by removing extraneous punctuation, standardizing casing, and applying whitespace trimming.
The intent labels, originally in string format, were encoded into numerical class values using the LabelEncoder utility from the Scikit-learn library.
This ensured compatibility with PyTorch-based model training and evaluation functions.
To maintain a balanced representation of each class across training and testing phases, stratified sampling was applied.
This technique preserved the relative proportion of each class label during the train-test split, mitigating class imbalance during model evaluation.
The final dataset was partitioned into an 80/20 split, yielding 9,249 sentences for training and 2,313 for testing.
Figure 2 displays the distribution of samples across the three intent labels, confirming that the dataset remains well-balanced for supervised learning.
By ensuring an even label spread and applying standard preprocessing routines, this step establishes a reliable input foundation for both the fine-tuned and inference-only LLMs used in this research.
Ae 99 Ae TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 Karjadi.
, & Santoso.
Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text.
TEPIAN, 6.
https://doi.
org/10.
51967/tepian.
Total Training & Testing data loan_application loan_application Figure 2.
Label Distribution GPT-Neo Fine-Tuning Configuration The fine-tuned model selected for this study is GPTNeo 125M, an open-source large language model developed by EleutherAI.
This model was chosen due to its compatibility with limited-resource environments and its ability to perform competitively when trained on domain-specific datasets.
GPT-Neo was fine-tuned on the 8,640 training samples derived from the Indonesian intent dataset introduced in Section 2.
The training process was implemented using the Hugging Face Trainer API, which provides a high-level interface for model fine-tuning with built-in support for evaluation metrics and GPU acceleration.
Tokenization was handled using the GPT-Neo tokenizer, with truncation and padding applied to ensure uniform input lengths of up to 64 tokens.
These settings were optimized to reduce memory overhead on consumer-grade hardware such as the MacBook M3 while maintaining sufficient context for short-to-medium length user queries.
Model training was conducted for one epoch with a learning rate of 2e-5.
The batch size was set to 4 for training and 8 for evaluation, balancing model stability and computational load.
The training utilized the Metal Performance Shaders (MPS) backend available through PyTorch to leverage the Apple Silicon GPU acceleration.
Performance was assessed using two key metrics:
accuracy and evaluation loss, computed directly via the accuracy_score from Scikit-learn and built-in logging functions within the Trainer module.
The results demonstrated that GPT-Neo achieved exceptional accuracy levels, validating its capacity to learn semantic distinctions across intent categories when exposed to sufficient supervised data.
Figure 3 illustrates the finetuning architecture and interaction between the pre-trained GPT-Neo backbone, tokenizer pipeline, training loop, and evaluation system.
This configuration exemplifies a practical implementation of low-cost LLM fine-tuning for multilingual tasks.
Figure 3.
Architecture of GPT-Neo fine-tuning and Ollama-based inference configuration.
Ollama-Based Zero-Shot Inference Setup In contrast to the fine-tuned GPT-Neo model, two additional LLMsAiMistral and Phi-2.
0Aiwere evaluated using a zero-shot inference strategy.
These models were executed through the Ollama framework, a containerized, locally hosted LLM runtime designed to support efficient model inference without requiring retraining or extensive GPU infrastructure.
This setup enables practical experimentation with advanced models in environments where computational and data resources are limited.
Both models were accessed via HTTP POST requests directed to the Ollama REST API, operating on the default endpoint localhost:11434.
Each request payload consisted of a task-specific prompt that included an instruction and the sentence to be classified.
The instruction component guided the model to classify the input into one of the predefined categories: inquiry or loan_application.
The prompt structure was standardized across models to ensure a fair and consistent evaluation.
Model responses were returned as plain-text predictions and were parsed programmatically to extract the inferred intent label.
For evaluation purposes, a balanced subset of 50 samples from the test set was selected, maintaining equal representation across all three classes.
Accuracy was Ae 100 Ae TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 Karjadi.
, & Santoso.
Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text.
TEPIAN, 6.
https://doi.
org/10.
51967/tepian.
computed by comparing each prediction with its corresponding ground-truth label, providing a performance snapshot for both models in zero-shot mode.
To assess system efficiency, execution time for each major processAiincluding preprocessing, fine-tuning .
or GPT-Ne.
, and inference .
or Mistral and Phi-2.
Aiwas measured using a Python decorator-based timing function.
This allowed the study to quantify trade-offs not only in terms of model accuracy but also in terms of computational This inference pipeline, powered by Ollama, demonstrates the viability of using instruction-tuned models in offline, local environments.
While Mistral yielded moderate performance due to its alignment with instruction-based tasks.
Phi-2.
0 struggled to generalize, emphasizing the need for careful model selection.
The overall setup illustrates a hybrid strategy where fine-tuned models and inference-only LLMs can be selectively deployed based on resource constraints, speed requirements, and task-specific precision needs.
Runtime and Performance Evaluation In evaluating the practical deploy ability of each model configuration, this study places equal emphasis on both predictive accuracy and computational efficiency.
achieve this, execution time was recorded for three critical stages of the pipeline: data preprocessing.
GPT-Neo finetuning, and zero-shot inference using Ollama.
A Pythonbased decorator function was implemented to capture precise timing metrics for each process, enabling direct comparison between different LLM deployment strategies.
During the preprocessing stage, tasks such as text normalization, label encoding, and stratified data splitting were completed in under 30 seconds.
These operations were relatively lightweight, regardless of the downstream model used.
For the fine-tuning stage.
GPT-Neo was trained on the full 8,640-sample training set for one epoch.
Despite the modelAos modest size .
million parameter.
, training was computationally intensive but feasible on a consumer-grade MacBook M3, completing in approximately 3 minutes using the Metal Performance Shaders (MPS) backend on PyTorch.
Inference latency was notably lower for both Mistral and Phi-2.
0, which were executed through OllamaAos local API engine.
Each prediction request processed within 1Ae2 seconds per sample, and the evaluation of the 50-sample subset concluded in under 1 minute.
This result highlights the practicality of using inference-only LLMs for lightweight classification tasks in resource-constrained A comparison of time-to-completion across the three stages reveals a clear trade-off: fine-tuning offers superior performance but incurs a higher upfront computational cost, while zero-shot inference provides rapid deployment with reduced accuracy.
Importantly, the entire hybrid pipelineAifrom preprocessing to final predictionsAiwas completed in under 5 minutes, reinforcing the notion that such configurations are not only effective but also operationally feasible on non-specialized hardware.
These performance metrics serve as a benchmark for future deployments of LLMs in multilingual, low-resource They offer a reference point for balancing speed, accuracy, and resource availability when choosing training-intensive inference-only IV.
RESULT AND DISCUSSION
This section presents the empirical findings derived from the comparative evaluation of three Large Language Models (LLM.
AiGPT-Neo.
Mistral, and Phi-2.
0Ai applied to the task of intent classification in the Indonesian The overarching goal of this experiment was to examine the trade-offs between supervised fine-tuning and zero-shot inference, particularly in contexts constrained by limited computational and data resources, as is often the case in low-resource multilingual environments.
GPT-Neo Fine-Tuning Outcomes Among the three models.
GPT-Neo was the only one subjected to supervised fine-tuning.
The model was trained on a domain-specific dataset consisting of 9,249 labeled sentences, evenly distributed across two intent categoriesAiinquiry and loan_application.
The fine-tuning process involved a single epoch using modest hardware:
an Apple MacBook M3 equipped with Metal Performance Shaders (MPS) via the PyTorch backend.
Despite its relatively compact architecture with only 125 million GPT-Neo classification performance, achieving a perfect test accuracy of 100% and a negligible evaluation loss of The quality of this performance is further substantiated by the confusion matrix, which exhibited ideal diagonal dominance, suggesting that the model was capable of clearly distinguishing between the two intent categories without exhibiting any significant inter-class confusion.
Moreover, the absence of class bias or skewed predictions across test samples affirms the modelAos ability to internalize semantic nuances even in cases where intent expressions are contextually similar.
Such results underscore the strength of targeted fine-tuning, even for models of relatively limited scale, when the dataset is wellstructured and task specific.
A notable insight, however, pertains to the training While initial projections anticipated the finetuning process to complete within five minutes, empirical logs revealed a total runtime of approximately 2,384 seconds .
r roughly 39 minute.
Although this is substantially longer than expected, the duration remains within a manageable range for many organizations operating on consumer-grade hardware.
Thus, the result emphasizes that high-precision LLM workflows need not rely exclusively on high-performance GPU clusters or cloud-based training pipelines, as even compact devices can facilitate robust training under appropriate Zero-Shot Inference Results for Mistral and Phi-2.
In contrast to GPT-Neo, the Mistral and Phi-2.
0 models were evaluated in zero-shot inference mode using the Ollama framework, a locally hosted LLM serving Ae 101 Ae TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 Karjadi.
, & Santoso.
Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text.
TEPIAN, 6.
https://doi.
org/10.
51967/tepian.
environment designed for lightweight deployment.
These models were not fine-tuned on the Indonesian intent dataset and instead relied solely on prompt-based instructions to classify sentences into one of the predefined This experimental setup mimics real-world deployment scenarios where labeled data may be scarce or unavailable, necessitating models that can generalize from task formulations alone.
Under these conditions.
Mistral achieved modest test accuracy of 32%.
While this figure represents a substantial decline from GPT-NeoAos fine-tuned performance, it still reflects some level of semantic comprehension, considering that no prior exposure to the dataset was The modelAos instruction-tuned pretraining is likely responsible for its partial success, enabling it to extract relevant intent signals from prompt structures.
Nevertheless, its confusion matrix revealed overlapping predictions and inconsistent separation between the inquiry and loan_application labels.
This suggests that while Mistral possesses the flexibility required for zeroshot generalization, its classification capacity remains insufficient for high-stakes or precision-sensitive applications without additional refinement or prompt By comparison.
Phi-2.
0 performed significantly worse, with a test accuracy of only 10%.
An examination of its confusion matrix indicated a systematic prediction bias, wherein the model defaulted almost exclusively to the inquiry label, regardless of the actual input.
This pattern of behavior reflects a lack of alignment between the modelAos pretraining objectives and the structural demands of intent Unlike Mistral.
Phi-2.
0 is not instructiontuned and therefore lacks the adaptive capabilities required to interpret and act upon prompt-based tasks effectively.
Consequently, its inability to generalize highlights the importance of architectural training alignment, especially for zero-shot classification use cases.
Comparative Runtime Analysis To complement the accuracy evaluation, runtime performance was also benchmarked across all three models to assess deployment feasibility under real-world Data preprocessing tasksAicomprising tokenization, label encoding, and stratified train-test splittingAiwere completed in under 30 seconds for all The supervised training of GPT-Neo, as noted earlier, required approximately 39 minutes, reflecting the computational overhead introduced by even lightweight model fine-tuning.
In contrast, inference operations for the zero-shot models were markedly faster.
Using OllamaAos local REST API interface.
Phi-2.
0 processed 50 test samples in 45 seconds, while Mistral required approximately 118 seconds for the same task batch.
These results validate that zero-shot models offer considerable latency advantages, especially in scenarios that demand immediate deployment or resource-conscious execution.
However, these benefits come at the cost of reduced classification accuracy and less predictable model behavior.
Summary of Findings In synthesizing these results, it becomes clear that model alignment and task-specific adaptation are far more decisive factors in LLM effectiveness than model scale The superior performance of GPT-Neo demonstrates that with targeted fine-tuning, compact models can deliver exceptional results even on lowresource hardware.
Meanwhile, the disparity between Mistral and Phi-2.
0Aiboth zero-shot modelsAihighlights the critical role of instruction tuning in enabling a model to interpret prompt structures and perform task-consistent These findings also suggest a viable hybrid strategy for institutions aiming to deploy LLMs efficiently: combining the robustness of fine-tuned models like GPT-Neo for production systems, with the agility of instruction-tuned inference-only models like Mistral for prototyping, lowstakes environments, or scenarios where data scarcity prohibits retraining.
The complementary nature of these models enables organizations to calibrate deployment strategies based on the intersection of resource availability, task complexity, and performance requirements.
Contribution to Informatics Methods This study offers a significant contribution to the field of informatics by formulating a structured, reproducible evaluation and deployment pipeline for comparing large language model (LLM) architectures in low-resource language contexts.
Rather than treating the models as black-box AI utilities, this research highlights a modular informatics framework that systematically integrates data preprocessing, algorithmic profiling, and deployment orchestrationAiproviding both scientific rigor and operational feasibility.
The first informatics component lies in the preprocessing and label encoding strategy.
Utilizing the Scikit-learn framework, the dataset was processed using stratified sampling and label encoding techniques to ensure class balance and consistency throughout the training and evaluation phases.
This approach not only minimizes class bias but also enables reproducible experimentationAian essential property in empirical informatics research.
Second, the study introduces a performance-aware pipeline design, employing Python-based decorator functions to monitor runtime execution for each key processing stage: preprocessing, model training, and This profiling mechanism embodies software engineering best practices within an AI research context, enabling fine-grained benchmarking and transparent performance reporting.
Third, the deployment architecture makes use of Ollama, a containerized inference environment that operates via RESTful APIs.
This implementation reflects microservice design principles and allows zero-shot LLM models to be deployed and tested locally without reliance on cloud infrastructureAimimicking edge-AI deployment scenarios common in resource-constrained environments.
It also reflects informatics best practices in system modularization and interface standardization.
Ae 102 Ae TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 Karjadi.
, & Santoso.
Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text.
TEPIAN, 6.
https://doi.
org/10.
51967/tepian.
Finally, the study defines an extensible model evaluation framework, leveraging Scikit-learn metrics such as accuracy and loss to assess model performance in a consistent and interpretable manner.
This evaluation layer is adaptable to future classification tasks, making it a versatile component for broader NLP system Collectively, these elements represent a concrete contribution to the domain of applied informatics, particularly in the context of language technology systems.
They offer replicable techniques that bridge natural language processing, software engineering, and performance analysisAiensuring that LLM-based systems are not only effective but also scientifically grounded and deployment-ready in real-world Indonesian NLP Strategic Evaluation of LLM Architectures in LowResource Settings To synthesize the findings from the previous subsections.
Table 1 presents a structured comparison of the three evaluated modelsAiGPT-Neo.
Mistral, and Phi2.
0Aibased on accuracy, runtime, deployment mode, and suitability for real-world usage.
Table 1.
Comparative Evaluation of LLMs for Indonesian Intent Classification Evaluation Aspect Mode Dataset Exposure Model Size Execution Framework Accuracy Evaluation Loss Training Duration Inference Time .
Classification Behavior Resource Requirement Best Use Case GPT-Neo (Fine-Tune.
Supervised Fine-Tuning 9,249 labeled samples 125M parameters Hugging Face Trainer MPS 39 minutes Mistral (Zero-Sho.
Zero-Shot Inference None 7B parameters .
Ollama REST API
N/A
None 118 seconds Phi-2.
0 (Zero-Sho.
Zero-Shot Inference None 2.
7B parameters .
Ollama REST API
N/A
None
45 seconds Very accurate and stable Medium (MacBook M3 GPU
MPS)
Production system Moderate, label overlap Low .
ocal CPU/GPU) Strong bias toward AuinquiryAy Low .
ocal CPU/GPU) Prototyping / POC Not recommended for MPS = Metal Performance Shaders (GPU Apple Silicon acceleratio.
The experimental findings in this study reveal critical insights into how different large language models (LLM) architectures respond under distinct deployment strategiesAiparticularly in constrained environments where computational and data resources are limited.
comparing the performance of GPT-Neo.
Mistral, and Phi2.
0 on Indonesian intent classification tasks, the study demonstrates that LLM effectiveness is determined not merely by model size or origin but more importantly by alignment with task-specific requirements, training paradigms, and deployment feasibility.
The consistently strong performance of GPT-Neo reaffirms the value of domain-specific supervised finetuning, especially in structured classification tasks.
Despite its relatively small parameter count .
GPT-Neo achieved 100% accuracy after a single epoch of training on a well-prepared dataset consisting of 9,249 labeled samples across two intent classes: inquiry and loan_application.
This high precision, confirmed by nearzero evaluation loss .
, suggests that even lightweight models can generalize exceptionally well when exposed to domain-aligned data and appropriate training procedures.
This finding has significant implications for real-world deployment, especially in low-resource or localized applications such as financial services in developing The modelAos ability to reliably distinguish between semantically close categories, such as loan inquiries versus loan applications, underscores its semantic precision.
This was evident in the confusion matrix, which displayed a clean diagonal structure, indicating minimal misclassifications and strong class However, a critical caveat to GPT-NeoAos performance lies in the training duration.
Contrary to initial assumptions presented in the journalAos early drafts, where the finetuning process was believed to finish in under five minutes, actual logs show that training took approximately 39 minutes on a MacBook M3 using Metal Performance Shaders (MPS) acceleration.
This discrepancy highlights the importance of validating pipeline runtimes and setting realistic deployment expectations, particularly when working with consumer-grade hardware.
In contrast.
Mistral, evaluated using zero-shot inference via the Ollama framework, achieved an accuracy of 32%, significantly lower than GPT-Neo but still better than random guessing.
Despite the lack of any training on the target dataset.
Mistral managed to infer reasonable intent predictions solely through structured prompts.
Its moderate performance is attributed to its instruction-tuned pretraining, which equips the model to respond to task formulations and follow guidance embedded in natural language prompts.
While Mistral could partially distinguish between the two intent classes, the confusion matrix revealed noticeable overlapAiespecially between inquiry and loan_application.
This result illustrates the limitations of relying purely on prompt-based inference without domain- Ae 103 Ae TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 Karjadi.
, & Santoso.
Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text.
TEPIAN, 6.
https://doi.
org/10.
51967/tepian.
specific tuning.
Although the following models such as Mistral are suitable for rapid prototyping, their classification accuracy may not suffice for productionlevel deployments unless supported with few-shot examples or prompt calibration.
Phi-2.
0, on the other hand, exhibited a notably poor performance of only 10% accuracy, revealing the risk of deploying LLMs that lack instruction tuning or taskaligned pretraining.
The confusion matrix showed that Phi2.
0 predicted nearly all test samples as inquiry, regardless of their actual labels.
This strong prediction bias suggests that the model failed to interpret the classification structure encoded in the prompts, likely due to insufficient exposure to similar tasks during pretraining.
This divergence between Mistral and Phi-2.
0 highlights a critical insight: model size and open availability are insufficient indicators of suitability for task-specific Instead, key factors such as pretraining objectives, instruction-tuning, and domain exposure must be given higher weight.
Even though Phi-2.
0 may possess robust generative capabilities in other contexts, its lack of classification alignment renders it ineffective in zero-shot classification tasks involving intent recognition.
Another relevant dimension is label imbalance and class Although the dataset used in this study was largely balanced between the two classes, slight skews may have contributed to Phi-2.
0Aos collapse into a dominant-class prediction.
GPT-Neo, having undergone supervised learning, was robust against this skew, and Mistral managed moderate resistance due to its instructionfollowing capability.
However.
Phi-2.
0Aos failure further emphasizes that zero-shot models require additional safeguards such as logit calibration, class balancing prompts, or confidence-aware filtering to avoid trivial prediction collapse in the presence of unbalanced data From a deployment feasibility standpoint, the runtime benchmarks reinforce the practicality of hybrid LLM Data preprocessingAiincluding tokenization and train-test splittingAicompleted in under 30 seconds.
GPT-NeoAos fine-tuning, although computationally heavier, was completed within 39 minutes on a consumer MacBook M3.
Meanwhile.
Mistral and Phi-2.
0Aos inference times were 118 seconds and 45 seconds, respectively, for a 50-sample evaluation set.
These results collectively validate that full-scale finetuning and zero-shot inference are both operationally viable on commodity hardware, although they present different trade-offs.
Fine-tuned models like GPT-Neo are highly accurate and stable but demand training time and Inference-only models like Mistral allow for quick experimentation but suffer from accuracy limitations and may require few-shot augmentation to enhance Therefore, this study suggests adopting a layered or fallback deployment architecture where fine-tuned models serve as the primary classification engine in production, while instruction-tuned zero-shot models are used for early prototyping, low-latency previews, or environments where data labeling is not yet available.
Such hybrid configurations allow organizations to balance between cost, accuracy, and time-to-market.
This study advocates for a context-aware model selection framework.
Decision-makers and system architects should move beyond metrics like FLOPs or parameter counts, and instead consider a modelAos training lineage, exposure to task patterns, and empirical fit for the application domain.
This shift is particularly essential in emerging markets and low-resource language settings, where every deployment decision directly affects inclusivity, accessibility, and reliability.
The evidence from this experiment highlights that achieving reliable intent classification is not about picking the largest or most famous model.
Instead, it is about choosing the right model for the right task, supported by appropriate tuning, prompt design, and deployment When these dimensions are aligned.
LLMs can deliver remarkable resultsAieven under resource limitations, making them practical and scalable tools for modern NLP applications.
IV.
CONCLUSION
This comparative study underscores the varying capabilities of Large Language Models (LLM.
in performing intent classification tasks within Indonesianlanguage datasets.
Through the evaluation of three modelsAiGPT-Neo.
Mistral, and Phi-2.
0Aithis research highlights how deployment strategy, model alignment, and training methodology critically influence model performance, particularly in low-resource and multilingual Supervised fine-tuning of GPT-Neo yielded exceptional Despite its relatively modest size .
M parameter.
, the model achieved a perfect accuracy of 100% on the test set after one epoch of training, supported by a minimal evaluation loss of 1.
This finding reaffirms the effectiveness of lightweight, fine-tuned models for domain-specific tasks.
However, the finetuning process required approximately 39 minutes, a considerable increase compared to prior assumptions, yet still feasible for execution on consumer-grade hardware such as the MacBook M3.
In contrast, inference-only models offered quicker deployment but at the cost of reduced precision.
Mistral, operating under zero-shot inference via the Ollama framework, achieved 32% accuracy, benefiting from its instruction-tuned training.
Phi-2.
0, lacking such alignment, performed significantly worse at 10%, frequently defaulting to the dominant class .
and exhibiting poor class discrimination.
These results demonstrate that while zero-shot models offer operational agility, they often require further prompt engineering or few-shot adaptation to be viable in production scenarios.
Moreover, the impact of label imbalance was minimal on GPT-Neo due to supervised learning but appears to have exacerbated prediction bias in Phi-2.
Overall, this study proposes a hybrid strategy that combines the robustness of fine-tuned models with the deployment speed of instruction-tuned zero-shot models.
Ae 104 Ae TEPIAN Vol.
6 No.
2 (June 2.
97-105 p-ISSN 2721-5350 e-ISSN 2721-5369 Karjadi.
, & Santoso.
Comparative Evaluation of Large Language Models for Intent Classification in Indonesian Text.
TEPIAN, 6.
https://doi.
org/10.
51967/tepian.
Such an approach enables organizations to optimize for both accuracy and resource constraints, especially in linguistically diverse environments with limited annotated Future research should explore few-shot tuning, calibration techniques, and multilingual prompt optimization to further improve performance in practical, real-world applications of LLMs for intent classification.
REFERENCES