Indonesian Journal of Electrical Engineering and Informatics (IJEEI) Vol. No. September 2025, pp. ISSN: 2089-3272. DOI: 10. 52549/ijeei. Optimizing IoT Protocol Coexistence and Security using Software Defined Network and Intelligent Machine Learning Detection Reshma N. Bhai1. Mahadev S. Patil 2 1Department of Electronics and 2Department of Electronics and Telecommunication Engineering. Institute of Civil & Rural Engineering. Gargoti. India Telecommunication Engineering. Rajarambapu Institute of Technology. Islampur. India Article Info ABSTRACT Article history: The rapid growth of heterogeneous IoT environments has made seamless communication across protocols like MQTT and CoAP increasingly difficult, leading to interoperability gaps, latency issues, and security vulnerabilities. This paper proposes a Software-Defined Networking (SDN)-based architecture that integrates MQTT and CoAP through a bidirectional translation layer, while embedding machine learning (ML) intelligence for real-time flag status monitoring and Denial-of-Service (DoS) attack The system leverages classifiers such as SVM. DT. NB. RF, and KNN within the SDN controller to dynamically predict operational states and mitigate malicious traffic. To evaluate performance, a Mininet-based IoT testbed with 50 heterogeneous nodes was deployed. Simulation results demonstrate that the proposed system achieves up to 95% message delivery success, reduces average latency by 18% compared to baseline translation methods, and saves 12Ae15% residual energy when using SVM-based While the system improves interoperability and security, it also introduces computational overheads at the SDN controller due to ML inference, which may impact CPU and memory utilization in resourceconstrained environments. The proposed solution is highly relevant for smart city, industrial IoT, and healthcare applications, where interoperability and real-time resilience against attacks are critical. By unifying heterogeneous devices and enhancing security, this approach provides a scalable and practical pathway for next-generation IoT networks. Received Jul 6, 2025 Revised Sep 9, 2025 Accepted Sep 14, 2025 Keyword: Internet of Things (IoT) Software-Defined Networking (SDN) MQTT CoAP Interoperability Machine Learning (ML) Denial of Service (DoS) Attack Detection Support Vector Machine (SVM) Protocol Translation Energy-Efficient IoT Communication Copyright A 2025 Institute of Advanced Engineering and Science. All rights reserved. Corresponding Author: Reshma N. Bhai. Department of Electronics and Telecommunication Engineering Institute of Civil & Rural Engineering. Gargoti. India Email: reshmabhai78@gmail. INTRODUCTION The Internet of Things (IoT) has revolutionized how devices interact by enabling billions of heterogeneous Ausmart objectsAy to communicate, collect, and exchange data. These objectsranging from simple sensors and actuators to complex embedded systemsoften differ vastly in hardware capabilities, software configurations, and communication standards. As IoT applications scale across domains such as healthcare, agriculture, smart cities, and industrial automation, the need for a unified, scalable, and interoperable communication infrastructure becomes paramount. However, the heterogeneity of IoT protocols presents a significant challenge. Two of the most widely adopted communication protocols are MQTT (Message Queuing Telemetry Transpor. and CoAP (Constrained Application Protoco. designed for different communication models: MQTT follows a publish-subscribe paradigm over TCP, while CoAP operates on a request-response model over UDP . These protocol disparities lead to Journal homepage: http://section. com/index. php/IJEEI/index IJEEI ISSN: 2089-3272 interoperability issues that hinder seamless data exchange across devices from different manufacturers or with different communication stacks . To address this. Software-Defined Networking (SDN) emerges as a transformative technology. By decoupling the control and data planes. SDN provides centralized programmability, making it ideal for managing protocol translation and network routing dynamically . This paper proposes an SDN-based architecture that unifies MQTT and CoAP devices through a translation layer, enabling transparent communication in a mixed-protocol IoT environment. Yet, seamless communication is only one side of the coin. The other critical concern is security, especially in large-scale, heterogeneous networks . IoT environments are increasingly vulnerable to network-layer attacks such as Denial of Service (DoS), which can degrade performance or cause complete system failures. Traditional rule-based intrusion detection systems often fall short in adaptability and precision . overcome these challenges, this research introduces a machine learning-enhanced SDN framework. The framework not only enables intelligent routing and protocol selection but also incorporates real-time DoS attack detection and status flag prediction using classifiers like Support Vector Machines (SVM). Decision Trees (DT). Naive Bayes (NB). Random Forest (RF), and K-Nearest Neighbors (KNN). These models are trained on labeled IoT traffic and DoS datasets to support dynamic, data-driven decision-making within the SDN controller. This paper offers the following key contributions: A hybrid integration of MQTT and CoAP protocols using SDN to ensure interoperability in heterogeneous IoT networks. An ML-enhanced control layer that predicts protocol behavior, monitors component status through binary flag indicators, and detects DoS attacks in real time. A comprehensive evaluation of interoperability performance, energy efficiency, throughput, and security metrics using a Mininet simulation testbed. Through these innovations, the proposed architecture aims to deliver a scalable, adaptive, and secure IoT framework suited to the dynamic requirements of modern intelligent environments. APPLICATIONS OF GENERATING CAPTION FOR AERIAL IMAGES Over the past decade, the Internet of Things (IoT) has seen exponential growth, leading to increased attention on communication protocols, network architectures, energy efficiency, and security. Researchers have focused on enabling seamless data exchange across heterogeneous devices, many of which rely on different communication standards such as MQTT. CoAP, and HTTP. Interoperability in IoT Protocols The coexistence of multiple application-layer protocols like MQTT and CoAP in a single IoT deployment often results in interoperability challenges. Studies such as Sethi et al. and Salman et al. have compared these protocols in terms of scalability, message reliability, and energy efficiency. While CoAP is praised for its lightweight, low-latency communication over UDP. MQTT is favored for reliable message delivery and its asynchronous nature via a brokered architecture. However, direct translation between these paradigms is non-trivial. Tayur and Suchithra . highlighted the critical issue of interoperability among MQTT. CoAP. HTTP, and AMQP, emphasizing the need for middleware or translation frameworks. Sandell and Raza . explored application-layer protocol coding techniques to improve performance but did not address bidirectional protocol mapping. The work of Lee et al. introduced SDN as a potential mediator for such protocol harmonization but lacked integration with intelligent control mechanisms. SDN in IoT Software-Defined Networking (SDN) has been introduced as a solution to address the rigidity and complexity of managing large-scale IoT networks. Palattella et al. proposed a standardized protocol stack using SDN to simplify network orchestration and ensure QoS compliance. More recently. SDN has been leveraged to support dynamic routing, centralized traffic management, and policy-based flow control in IoT ecosystems. However, the integration of SDN with protocol translation . MQTT-CoAP bridgin. remains an area of active research. Few frameworks, such as those discussed by Ahmed. , et al. offer insights into protocol interoperability, yet they do not address runtime optimization based on traffic patterns or network threats. Machine Learning for IoT Optimization Machine Learning (ML) offers significant advantages in predicting network behavior, optimizing resources, and enhancing security. Ikria et al. provided a comprehensive review of ML methods used for IoT device classification, anomaly detection, and adaptive communication. Recent studies, including those by Optimizing IoT Protocol Coexistence and Security usingA (Reshma N. Bhai et a. A ISSN: 2089-3272 Anthi et al. and Rejito et al. , have applied ML models for detecting adversarial attacks in smart home and MQTT networks with promising accuracy. In the context of flag status prediction. ML models such as Support Vector Machines (SVM) have shown strong generalization in classifying non-linear datasets, outperforming traditional models like Naive Bayes or Decision Trees. However, such implementations are often isolated from real-time control systems and lack integration into the SDN layer. DoS Detection in IoT Networks The threat of Denial of Service (DoS) attacks is critical in IoT due to limited device resources. Gerodimos et al. and Feijoo-Ayazco et al. investigated CoAP-specific attack vectors like packet flooding and resource exhaustion. Meanwhile. Gomez et al. analyzed MQTTAos vulnerability to payload abuse, topic flooding, and connection hijacking. Recent methods have focused on ML-based detection systems using datasets such as the Ie IoT-DoS benchmark. In Bukhowah. R et al. , classifiers were used to detect DoS attacks, but only for homogeneous networks. The current study is focused on network security in heterogeneous network. Research Gap Despite ongoing advances in protocol optimization and SDN deployment, few studies have proposed a unified framework that: A Integrates MQTT and CoAP protocols with seamless runtime translation A Uses SDN for centralized, programmable control A Applies ML for both flag status decision-making and real-time DoS attack detection This paper addresses the above limitations by presenting a machine learning-powered SDN architecture that offers intelligent interoperability and robust network security in a heterogeneous IoT environment. PROPOSED SYSTEM ARCHITECTURE The proposed system is designed to address two key challenges in heterogeneous IoT networks: . seamless communication across devices using MQTT and CoAP protocols, and . real-time security and optimization using machine learning integrated with SDN. The architecture is modular and consists of four main components: protocol translation. SDN-based control, machine learning intelligence, and a simulation/emulation environment. MQTT-CoAP Protocol Integration via SDN MQTT and CoAP operate on fundamentally different paradigms publish/subscribe over TCP and request/response over UDP, respectively. To enable seamless interoperability, a translation layer is implemented, managed centrally by the SDN controller. This layer ensures bidirectional communication A Protocol mapping: MQTT messages are transformed into CoAP-compatible requests, and CoAP responses are translated back to MQTT topics. A Session tracking: The system maintains session state and protocol context to ensure continuity during translation. A Broker-gateway design: An MQTT broker is extended with a CoAP plugin or interface, supported by SDN flows that route traffic to the appropriate protocol handler. SDN Controller for Centralized Orchestration The SDN controller . OpenDaylight or ONOS) plays a crucial role in managing data flows, topology discovery, and routing policies. It communicates with edge IoT devices and gateways using OpenFlow and provides: A Dynamic flow management based on protocol type and device state. A Policy-based traffic prioritization for MQTT-CoAP messaging. A Real-time monitoring of packet delivery, latency, and energy statistics. The northbound interface of the controller interacts with higher-layer applications . uch as ML model. , while the southbound interface controls OpenFlow-compatible switches and gateways. Machine Learning for Status and Attack Detection Machine learning classifiers are integrated into the control plane to assist with predictive analytics and threat mitigation: IJEEI. Vol. No. September 2025: 784 Ae 797 IJEEI ISSN: 2089-3272 Flag Status Prediction: A classification model (SVM. DT, etc. ) predicts the operational state . n/off, healthy/fault. of IoT nodes based on inputs such as packet delay, message rate, and energy DoS Attack Detection: ML models trained on IoT DoS datasets identify anomalies in real-time Features include packet size, frequency, source repetition, and protocol headers. The SVM classifier is found to outperform other models in terms of accuracy and generalization. Dataset and Mathematical Models The system uses labeled datasets for both traffic modeling and DoS classification. Key modeled elements include: Traffic volume per time step: where is the data rate and is an indicator function for device activity. Utility optimization function: The controller selects the optimal route and protocol to maximize Oversall System Flow IoT devices generate MQTT or CoAP traffic. The SDN controller detects the protocol and assigns flow paths. The translation layer converts MQTT-to-CoAP . nd vice vers. when needed. ML classifiers assess node health and detect anomalies. Based on insights, the controller adjusts routing, prioritization, and mitigation policies. This modular, intelligent system allows for real-time interoperability, optimized resource usage, and enhanced network resilience in large-scale IoT deployment. PROPOSED METHODOLOGY: To intelligently manage IoT communication and protocol selection, a machine learning- based SDN framework is proposed. The SDN controller is empowered with ML capabilities to enhance decision-making processes with system formation as shown in Figure 1 Traffic Pattern Dataset Machine Learning Model Training Trained Model SDN Controller MQTT Zone of IoT Devices CoAP Zone of IoT Devices Figure 1. Proposed ML based Framework for Heterogeneous IoT The proposed architecture employs Software-Defined Networking (SDN) to unify communication between MQTT and CoAP devices. MQTT follows a publish-subscribe paradigm, while CoAP is based on a request-response model. These two distinct paradigms are bridged by an SDN-controlled broker that enables seamless bidirectional protocol translation. The SDN controller dynamically handles: Route optimization. Protocol assignment, and Network configuration, based on real-time traffic conditions and device capabilities, ensuring end-to-end interoperability and efficiency across heterogeneous IoT Traffic Modeling. Feature Engineering, and SDN Decision Framework To support intelligent management of heterogeneous IoT networks, the proposed system leverages dynamic traffic modeling, feature-based protocol selection, and SDN-integrated classifier feedback. This section outlines the dataset integration, session behavior modeling, feature extraction, protocol assignment, and flow decision-making strategy. Dataset Integration for Traffic Simulation The simulation environment utilizes the IoT Traffic Generation Patterns dataset, available from Kaggle: https://w. com/datasets/tubitak1001118e277/iot-traffic-generation-patterns Optimizing IoT Protocol Coexistence and Security usingA (Reshma N. Bhai et a. A ISSN: 2089-3272 This dataset includes synthetic traffic patterns generated from realistic IoT device profiles, including data rates, transmission intervals, sampling frequencies, and delay constraints. It enables modeling of diverse IoT environments with devices using either MQTT or CoAP, facilitating evaluation of both protocol behaviors under load and attack scenarios. The ML classifiers are trained on a labeled dataset that includes both normal and attack traffic patterns. Key features extracted from traffic traces include: Sampling rate. Data rate. Delay,Transmission start time and Device count. The traffic behavior is modeled by: = Oc rA A IA. Where A T. is the total traffic at time t. A ri is the rate of the ith device. A Ii. ) is an indicator function representing the activity status of device I at time Feature Engineering and Protocol Prediction From the observed traffic patterns, the system extracts key features such as: A Sampling frequency A Instantaneous data rate A Device ID and type A Transmission start time A Delay constraints These features are fed into a trained classifier ensemble that predicts the optimal protocol (MQTT or CoAP) for each device or session using: PO=arg maxApOO{MQTT,CoAP}fp. Where fp. Confidence score or probability output of classifier for protocol p. This prediction allows dynamic protocol selection per node, reducing energy consumption and improving delivery reliability. SDN-Based Classifier Integration and Utility Optimization The SDN controller leverages classifier outputs to take network control actions such as: Assigning routes. Selecting transmission protocol Adjusting message priority. A utility function is used to guide decision-making. To support intelligent flow decision-making, an SDN utility function is defined as: U = UI Throughput Ae UI Delay Ae UI Energy . Where , , are tunable weights based on application priorities. Throughput, delay, and energy are performance indicators for current traffic flows. The controller selects the action that maximizes utility for a given flow, ensuring optimal protocol operation, load balancing, and resilience under dynamic traffic and security conditions. This utility-driven model allows the controller to install flow rules that maximize network utility, balancing speed, reliability, and energy conservation. Status Flag-Based Control The proposed system employs status flags to represent the operational state of each IoT node or traffic flow. These flags are generated in real time by machine learning classifiers . SVM. DT) integrated within the SDN controller. Based on traffic behavior and device context, each node is assigned a flag such as node is currently active or inactive, functioning correctly, experiencing an error, or requiring attention. These flags guide the SDN controller in making intelligent decisions about routing, access control, protocol selection, and energy optimization. By continuously updating the flags, the system ensures adaptive, secure, and resource-aware IoT communication. ycIycn . OO. OAycn OO . ,2,3 A . ycA} ycIycn . =0 implies that component i is active, enabled, or operating correctly at time ycIycn . =1 implies that component i is inactive, disabled, or nonoperational IJEEI. Vol. No. September 2025: 784 Ae 797 IJEEI ISSN: 2089-3272 Security Against DoS Attacks In IoT-based environments integrating MQTT and CoAP protocols. Denial-of-Service (DoS) attacks present a major threat due to the lightweight nature of both protocols and the limited processing and energy capacities of edge devices. Both MQTT and CoAP are designed for constrained devices, making them prone to different classes of DoS attacks. MQTT vulnerabilities include: o Connection Exhaustion: Multiple persistent TCP connections overwhelm the broker. o Topic Flooding: Excessive publishing or subscribing causes overload. o Payload Injection: Oversized or malformed payloads consume memory or crash devices. CoAP vulnerabilities include: o Message Fragmentation: Attackers exploit UDP fragmentation, causing memory overuse. o Resource Exhaustion: Flooding resource requests depletes device or network resources. These vulnerabilities can degrade QoS, increase delay, and accelerate energy drain, ultimately disrupting application logic and service delivery. The proposed system addresses this vulnerability by incorporating machine learning (ML)-based attack detection mechanisms and leveraging SDNAos centralized control for mitigation. To structure the proposed system with the connection of a machine learning facility to the SDN controller, as shown in Figure 2, has following details. DoS attacks Dataset Training of Machine Learning Model MQTT Devices DoS CoAP Devices Trained Machine Learning Model SDN Controller for Proposed Hybrid Architecture of IoT Network Detected DoS attacks Performance Evaluation Figure 2. Proposed Machine Learning Framework for DoS Attack Detection Machine Learning-Based Detection Strategy To combat DoS attacks in a hybrid IoT network, a real-time traffic classification model is implemented at the SDN controller level. This model uses supervised ML classifiers trained on labeled IoT traffic to identify suspicious flows based on various network-level and application-level features Packet arrival rate. Packet size variation. Node activity frequency. Protocol used (MQTT/CoAP). Flow duration and entropy. Device identity and directionality Classifiers Deployed are Support Vector Machine (SVM). Decision Tree (DT). Naive Bayes (NB). K-Nearest Neighbors (KNN). These classifiers were trained offline and then embedded into the controller for runtime evaluation. This combination of and SDN-based mitigation enables rapid, scalable, and programmable defense against evolving DoS patterns in heterogeneous IoT systems. ML-based detection Dataset and Classifier Integration To enable intelligent DoS detection in the MQTT-CoAP integrated IoT network, a comprehensive machine learning pipeline was developed. This section describes the dataset used, feature extraction strategy, classifier configuration, and how the models were integrated into the SDN framework. Optimizing IoT Protocol Coexistence and Security usingA (Reshma N. Bhai et a. A ISSN: 2089-3272 Dataset Integration The training and validation of ML classifiers were performed using the publicly available IoT DoS and DDoS Attack Dataset provided on Ie Dataport . : https://ie-dataport. org/documents/iot-dosand-ddos-attack-dataset This dataset contains labeled traffic flows representing: A Benign IoT communication A Various DoS/DDoS attack scenarios, including UDP flooding. ICMP floods, and SYN-based The dataset includes thousands of samples across multiple attack types, making it suitable for training robust classifiers that generalize to heterogeneous IoT environments. Machine Learning Facility The classification framework includes the following supervised learning models: A Support Vector Machine (SVM) A Decision Tree (DT) A Naive Bayes (NB) A K-Nearest Neighbors (KNN) These models were selected for their varied decision boundaries, resource requirements, and detection accuracy in prior security applications. Feature Engineering and Selection Key features were extracted and engineered from raw packet flows and protocol metadata. The features A Sampling rate A Data transmission rate A Delay and jitter A Protocol type (MQTT/CoAP) A Device ID and session time A Flow duration and byte entropy Feature selection was carried out using correlation analysis and mutual information ranking to eliminate redundancy and retain only the most predictive attributes. Classifier Integration with SDN Controller The trained classifiers were deployed directly within the control logic of the SDN controller (OpenDayligh. The integration enables: A Real-time classification of new flows A Flow flagging based on prediction: benign, suspicious, or malicious A Dynamic flow rule installation . , drop, reroute, or allo. A Online learning updates based on feedback during simulation This tight coupling of ML with SDN ensures rapid threat response, centralized intelligence, and adaptive behavior in dynamically changing IoT traffic environments. RESULTS AND ANALYSIS This section presents the results obtained from the simulation of the proposed MQTT-CoAP integrated IoT architecture using SDN and machine learning. Performance is evaluated across multiple metrics including throughput, end-to-end delay, energy consumption, message delivery rate, and DoS attack detection accuracy. Additionally, we assess the success of cross-protocol interoperability between MQTT and CoAP nodes. Simulation Environment for Interoperability in Heterogeneous IoT network: The experimental setup was implemented using Mininet 2. 3 with an OpenDaylight SDN controller to emulate a heterogeneous IoT environment of 50 nodes, evenly divided between MQTT and CoAP devices. The topology consisted of five edge switches connected to a core switch, ensuring realistic multi-hop paths. An MQTT broker (Mosquitt. and a CoAP server (Californiu. were deployed, with a Python-based IJEEI. Vol. No. September 2025: 784 Ae 797 IJEEI ISSN: 2089-3272 translator bridging MQTT topics and CoAP resources for seamless interoperability. Normal traffic was generated at rates of 0. 2Ae2 messages/sec with payloads of 20Ae200 bytes, while DoS attack scenarios included both high-rate floods . Ae500 packets/se. and low-rate DDoS bursts. Figure 3. Proposed IoT System Based on Mininet The SDN controller collected OpenFlow statistics, which were aggregated into 5-second feature windows capturing metrics such as packet count, byte count, inter-arrival times, and topic/resource These features were fed into machine learning classifiers (SVM. RF. DT. NB) implemented in Python . cikit-lear. , trained with a 70/15/15 split and deployed online for real-time detection and mitigation by dynamically installing flow rules. Performance was evaluated in terms of detection accuracy, false positive rate, end-to-end latency, delivery success ratio, and controller overhead (CPU, memory, and inference dela. Figure 3 shows the Mininet topology and simulation environment for the proposed system. Performace Metrics The following performance metrics were recorded during the experiments: A Throughput: The number of successfully transmitted messages per unit time. A End-to-End Delay: The time taken for a message to travel from source to destination, including translation and routing delays. A Energy Consumption: The amount of energy consumed by nodes during data transmission. A Message Delivery Rate (MDR): Ratio of successfully received messages to total messages sent. Figure 4. Average Throughput over number of devices Figure 4 presents the average throughput . n Kbp. achieved by various machine learning classifiers SVM. DT. RF, and NB under increasing system load, represented by the number of devices ranging from 250 to 2000. The results indicate that SVM consistently delivers the highest throughput, starting at 100 Optimizing IoT Protocol Coexistence and Security usingA (Reshma N. Bhai et a. A ISSN: 2089-3272 Kbps for 250 devices and gradually decreasing to around 65 Kbps at 2000 devices. Its optimized decisionmaking and minimal inference overhead make it ideal for high-load IoT networks. Figure 5. Residual Energy over number of rounds Figure 5 illustrates the residual energy trends of IoT nodes across 1000 iterations under five different machine learning classifiers: SVM. DT. RF, and NB. Energy consumption was tracked assuming a network of 50 nodes, each initialized with 3000 millijoules . J), totaling 150,000 mJ. The results highlight the following insights SVM maintains the highest residual energy throughout the simulation. Its efficient and lightweight inference logic minimizes unnecessary transmissions and optimizes routing decisions. Figure 6. End to end delay over number of devices Figure 6 presents the variation in average end-to-end delay as a function of the number of devices in the system for four different machine learning classifiers SVM. DT. RF, and NB. The delay was measured as the time taken for a message to travel from a source IoT node to its destination, encompassing queuing, processing, transmission, and potential protocol translation delays within the SDN-based From the graph, it is evident that SVM consistently exhibits the lowest average delay across all node densities, demonstrating its efficiency in early and accurate decision-making that minimizes retransmissions and flow congestion. IJEEI. Vol. No. September 2025: 784 Ae 797 IJEEI ISSN: 2089-3272 Figure 7. Message Delivery rate over number of devices Figure 7 illustrates the variation in message delivery rate (MDR) across different classifier models SVM. DT. RF, and NB with respect to the increasing number of devices in the IoT system. Across all classifiers, the message delivery rate improves as the number of devices increases from 250 to 1250, indicating that the classifiers adapt well to moderate-scale environments. SVM consistently outperforms the others, reaching a peak MDR of approximately 95% at 1250 devices. This suggests that SVM can maintain higher reliability in delivering messages under varying traffic loads. Comparative Analysis of ML Classifiers against DoS attack: ML classifiers as SVM. NB. DT and KNN were tested for flag status prediction and DoS detection. The proposed hybrid IoT network was simulated using the Mininet OpenFlow emulator, which was deployed on a Linux-based platform. A total of 50 IoT nodes were configured and integrated into the network topology for experimental analysis. The simulation focused on evaluating the systemAos ability to distinguish between normal traffic and DoS attack scenarios. Performance comparisons were carried out using four selected machine learning classifiers. The emulated OpenFlow environment was tailored to reflect realistic IoT deployment conditions, with the node configurations outlined in Table 1. Table 1. Protocol-Level Configuration of MQTT and CoAP Nodes Node ID Protocol MQTT CoAP MQTT CoAP MQTT CoAP Data Type / Action Temperature sensor reading Temperature query Humidity sensor reading Light actuator control Motion sensor alert Door actuator update MQTT QoS N/A N/A N/A MQTT Retain True N/A False N/A True N/A CoAP Method N/A GET N/A POST N/A PUT The performance of a machine learning model in identifying Denial-of-Service (DoS) attacks heavily relies on its ability to correctly distinguish between malicious and normal traffic. To evaluate this performance, standard classification metrics such as true positives (TP), false negatives (FN), true negatives (TN), and false positives (FP) are employed. These parameters are essential for calculating key performance indicators. Table 2 outlines the mathematical expressions used to compute accuracy, sensitivity . , and specificity, which collectively provide a solid foundation for evaluating the modelAos detection capabilities. These metrics offer a detailed perspective on how effectively the system can identify and classify DoS attacks, ultimately reflecting the robustness and dependability of the ML-driven security Table . shows performance metrics results for different classifiers in DoS attack environment. Table 2. Evaluation Parameters Parameter Formula Sensitivity TP/(TP FN) Specificity TN/(TN FP) Accuracy TP TN/(TP TN FP FN) Optimizing IoT Protocol Coexistence and Security usingA (Reshma N. Bhai et a. A ISSN: 2089-3272 Table 3. Performance Metrics for Classifiers in DoS Attack Detection Classifier Accuracy (%) Sensitivity (TPR) (%) SVM KNN Specificity (TNR) (%) Comparison of DoS Attack Detection Accuracy (%) Sensitivity (TPR) (%) Specificity (TNR) (%) SVM KNN Figure 8. Classifiers Performance Parameters Comparison Figure 8 shows performance parameters accuracy, sensitivity and specificity comparison for SVM. DT. NB and KNN classifiers. SVM provided the best balance between sensitivity . ttack detection rat. and specificity . alse positive avoidanc. , making it ideal for real-time security tasks in IoT networks. A comparative study of various attack detection approaches in IoT networks is presented in Table 4. Prior works . , 17, 28, . have primarily examined homogeneous IoT environments operating on a single protocol . ither MQTT or CoAP). Unlike these protocol-specific approaches, the proposed framework targets a hybrid heterogeneous IoT network integrating MQTT and CoAP. Table 4. Comparative Overview of Attack Detection Approaches in IoT Networks Technique / Reference DT. RF. NB. SVM . Deep Learning . Random Forest . BPDF Counter Method . Proposed SVM-based Model Protocol Evaluated MQTT MQTT MQTT CoAP MQTT CoAP (Heterogeneous IoT) Attack Scenario Adversarial Adversarial Sinkhole Sinkhole DoS Reported Accuracy / Detection Rate 99% accuracy 3% accuracy 98% detection rate 92% detection rate 84% accuracy Result Discussion Energy analysis showed that the average residual energy in SVM-managed scenarios was 12Ae15% higher compared to naive models. This was attributed to optimized routing and reduced retransmissions. slight increase in end-to-end delay . Ae10 m. was noted due to the computational cost of ML inference, which is acceptable for non-real-time applications. For Interoperability Between MQTT and CoAP Crossprotocol communication was evaluated based on translation success, delivery rates, and latency. MQTT were translated into CoAP requests (GET/PUT), and CoAP responses were converted back into MQTT topic responses. Translation Success Rate Cross-Protocol Message Delivery Rate 95. Average Interoperation Latency 120 ms Packet Loss . uring translatio. The system demonstrated robust interoperability, with minimal packet loss and high translation The SDN controller effectively monitored flow rules and maintained optimal load balancing between MQTT and CoAP zones. ML-enhanced SDN control significantly improves DoS detection and network SVM classifiers offer the best performance for intelligent status flag decisions. MQTT-CoAP translation is highly successful (>95%), supporting true protocol interoperability. Energy consumption is optimized, with minimal impact on network latency. These results validate the efficiency and practicality of IJEEI. Vol. No. September 2025: 784 Ae 797 IJEEI ISSN: 2089-3272 the proposed system for real-world heterogeneous IoT deployments. Table 4 shows proposed work traits when it is compared with research work previously done. Table 4. Comparative Analysis of Proposed Work with Existing Studies Study / Author Year Approach DoS Detection Interoper Energy Efficiency Key Limitation Alve et al. ui (DT: 99. RF: 98. un un No SDN, no interoperability focus Deepa & Suguna. Ensemble ML classifiers on IoT QoS-based multipath un un ui No security or dynamic Sumadi et al. Elsayed et al. Kavitha & Ramalakshmi . Proposed Work Static proxy bridging for MQTTAeCoAP using OpenFlow DL-based intrusion detection for SDN-IoT ML-based DDoS detection in SDN un ui un No SDN or ML-based ui (LSTM) un un ui (MLP. RF: un un Focused on IDS, not protocol integration No real-time translation or hybrid IoT model SDN-enabled hybrid MQTTAeCoAP integration with MLbased flag monitoring and DoS detection ui (SVM: 84%) ui . nergytime translatio. Focused primarily on DoS. evaluation of other IoT threats Computational Overhead Analysis The integration of SDN with embedded ML inevitably introduces computational overhead. In the proposed architecture. ML-based flag status prediction and DoS attack detection are executed at the SDN controller, which centralizes decision-making. While this offloads complexity from the IoT end devices, it increases CPU load and memory usage at the controller. Preliminary profiling during simulation indicated an average CPU utilization increase of 7Ae10% and memory consumption growth of approximately 50 MB when ML modules (SVM and RF) were active compared to a baseline SDN-only setup. The additional processing delay per packet was measured at 3 to 5 ms, which remains tolerable for most IoT applications but could challenge ultra-low latency domains. These results suggest that while the framework enhances security and efficiency, careful consideration of controller capacity and potential scaling strategies . , edge offloading, lightweight ML model. is necessary for real-world deployments. Limitations While the proposed SDN-enabled MQTTAeCoAP integration with ML-based flag monitoring and DoS detection demonstrates improved interoperability and security, several limitations remain. First, the scalability of the framework was validated only up to a 50-node IoT network, and its performance in largerscale deployments remains to be tested. Second, although the ML-based detection achieves high accuracy in identifying DoS attacks, it is still susceptible to detection failures and false positives, which could result in unnecessary blocking of legitimate traffic. Third, the current evaluation focuses exclusively on DoS attacks. other critical IoT-specific threats such as spoofing, replay, or phishing were not addressed. This limited scope reduces the robustness of the proposed security model in more adversarial and realistic scenarios. CONCLUSION This paper presented an SDN-enabled, machine learning-integrated architecture to address the dual challenges of protocol interoperability and security threat detection in heterogeneous IoT networks. facilitating seamless communication between MQTT and CoAP devices, the proposed system ensures efficient protocol translation and cross-domain data exchange, crucial for scalable IoT deployments. An experimental environment using Mininet was developed to emulate a 50-node hybrid IoT Various machine learning classifiers SVM. DT. RF. NB, and KNN were evaluated for their effectiveness in DoS attack detection and system-level optimization. Among them. SVM demonstrated superior performance in terms of accuracy, sensitivity, specificity, message delivery rate, and energy efficiency, with minimal impact on end-to-end delay. Results showed that: SVM achieved up to 95% message delivery. Sustained lower delays under high device densities, and maintained higher residual energy over time. These findings emphasize the critical role of intelligent SDN control, powered by ML, in making dynamic protocol decisions and enhancing network resilience against attacks. The architecture offers Optimizing IoT Protocol Coexistence and Security usingA (Reshma N. Bhai et a. A ISSN: 2089-3272 a scalable and adaptive framework suitable for real-time, resource-constrained IoT applications. Future work will explore real-world deployment using hardware test beds, adaptive protocol switching, and multilayer attack detection for broader security coverage in complex IoT ecosystems. A notable limitation of the proposed framework is the computational overhead at the SDN controller caused by embedded ML inference. Although the measured overhead was within acceptable bounds for our test bed, in future work, we aim to extend the proposed framework to support scalable deployments involving hundreds of IoT nodes, incorporate broader attack models . ncluding spoofing and repla. , and optimize the ML classifiers to reduce false positives while maintaining lightweight computational footprints suitable for IoT environments. These directions will help further strengthen the practicality and resilience of the system. REFERENCES