Journal of Robotics and Control (JRC) Volume 7.
Issue 2.
March 2026 ISSN: 2715-5072 DOI: 10.
18196/jrc.
Multi-Agent Reinforcement Learning with Hierarchical Reward Structures for Complex Optimization Problems Nuha Sami Mohsin1.
Asmaa Hussien Alwan2.
Saadya Fahad Jabbar 3
1, 2, 3,
College of education Ae ibn rushed for human science.
University of Baghdad.
Baghdad.
Iraq Email: 1*nuha.
sami@ircoedu.
iq, 2 asmaa.
hussien@ircoedu.
*Corresponding Author AbstractAiComplex optimization problems have now been applied in several fields such as energy management, logistics, robotics, and traffic control, among others.
The problems often have multiple goals that lie in uncertain and volatile settings that single-agent reinforcement learning and other standard optimization methods struggle to address successfully.
MultiAgent Reinforcement Learning (MARL) can be termed as a cansolve model but is vulnerable to non-stationary and needs scalability ability and provides enough credit specifically when applied in the reward scheme necessitating a high level of The embracing of hierarchical reward framework becomes an ideal solution to divide global targets into several smaller goals that enable learning to scale effectively.
To address these issues, this study provides its primary contribution by developing a new MARL system that has rewards distributed at multiple levels.
The research contribution lies in the formulation and experimentation of a hierarchical reward MARL model that explicitly improves coordination, scalability, and convergence rate of the dynamic environment.
The tested framework has high solution quality and convergence speed capabilities as well as better scalability when applied to benchmark and real-life optimization tasks in comparison with flat reward structures and traditional optimization methods.
Quantitative findings show convergence of rewards .
and over 30 percent decrease in communication overhead among global methods as compared to baseline methods.
The experimental results prove that the framework is convergent and scalable and demonstrates effective enhancement of agent coordination and learning efficiency in line with the theory The results in the dimensioned show that the baseline approaches require already thirty percent less communication overhead coupled with twenty percent higher global reward The weighting of the performance sensitive rewards (, , ) however, should be investigated in future KeywordsAicoordination.
hierarchical rewards.
multi-agent reinforcement learning.
optimization problems.
decentralized control.
credit assignment.
non-stationarity.
INTRODUCTION
The frequency of complex optimization problems grew rapidly in different domains including logistics and robotics together with energy management and traffic control systems during recent years.
The problems present numerous objectives together with dynamic environments while demanding extensive decision-making processes so traditional optimization solutions do not function effectively.
The recognized power of RL for sequential decision-making cannot solve distributed and collaborative optimization problems because its single-agent formulations lack the required capabilities .
Scientists have become increasingly interested in Multi-Agent Reinforcement Learning (MARL) since it enables various agents to work together for both mutual and individual goal accomplishment.
MARL presents several technical difficulties which include non-stationarity and scalability issues with award structures and highdimensional state-action spaces despite its challenges in conducting credit assignment .
, .
The application of hierarchical reward structures shows promise as a solution to the complexity challenges faced in such systems since they separate global objectives into sub-goals and local objectives for efficient learning .
Agents benefit from hierarchical rewards because it offers an organized method to reach both immediate and extended objectives which results in improved teamwork and simplified learning tasks .
Dividing global objectives into hierarchical local and sub-objectives through reward structures allows systems to achieve better credit assignment and ultimately enhance performance .
Current research lacks sufficient study on hierarchical reward designs for MARL systems when dealing with complex optimization problems .
The proposed research seeks to solve this problem through the development of a MARL framework which includes custom hierarchical reward structures designed for complex optimization structures .
Ae .
This framework takes advantages of hierarchical rewards through which it enhances agent coordination alongside scalability and learning efficiency to handle MARL problems.
The proposed procedure decomposes global objectives into subgoals to create appropriate reward mechanisms which enables agents to handle effective large-scale dynamic multiobjective optimization problems.
The research paper presents analyses of theoretical convergence and scalability for the framework and shows its practical results on benchmark system and real-life scenarios .
Optimization tasks across multiple agents require dealing with multiple connecting agents together with altering environments while handling conflicting goals.
Standard optimization techniques together with basic RL methods prove inadequate for managing distributed operations common in these problems .
Currently available MARL approaches show promise yet encounter three main obstacles when dealing with non- Journal Web site: http://journal.
id/index.
php/jrc Journal Email: jrc@umy.
Journal of Robotics and Control (JRC) ISSN: 2715-5072 stationary environments and lack appropriate scalability and inefficient credit assignment during learning processes in reward structures with multiple components.
Real-life scenarios suffer from ineffective reward mechanisms which results in unsatisfactory agent coordination and slows down learning processes thus preventing agents from reaching their global targets successfully .
Ae .
The primary objectives of this study are to design a MARL framework with hierarchical reward structures that address scalability, coordination, and learning efficiency challenges in complex optimization problems, investigate the impact of such structures on agent coordination, convergence, and overall performance, and evaluate the framework's effectiveness on benchmark and real-world optimization problems compared to existing approaches.
This study makes three important scientific contributions as follows.
A novel MARL framework implements hierarchical reward structures which decomposes global objectives into divisible local and sub-goals to promote efficient and numerous learning possibilities.
The investigation analyzes theoretical properties of the proposed framework including the boundary conditions it can handle.
The research shows experimental data from multiple optimization problems using both benchmark tasks and practical applications which proves the effectiveness of hierarchical rewards above flat reward structures and basic optimization approaches.
Researchers will benefit from practical implementation methods related to MARL hierarchical reward structures which will guide additional studies in this The rest of this paper is organized as follows.
Section 2 presents related works.
Section 3 presents the Section 4 presents the results and Finally.
Section 5 concludes the paper.
Although there are advances in multi-agent reinforcement learning (MARL), the current research has failed to sufficiently tackle the three issues of scalability, credit attribution, and reinforcement in dynamic dynamics.
Other historical models focused on either simplified reward models or flat reward models, lacking a connection between the successful application of hierarchical reward decomposition to real world optimization problems, such as logistics, energy management, and traffic management.
The goal of this work is to fill this gap through a direct study of how hierarchical rewards can enhance coordination and convergence in largescale systems.
The research contribution is to develop and test hierarchical reward MARL architecture that takes into account the scalability, coordination and efficiency of learning complex optimization problems.
In contrast to the rest of the work, the suggested approach incorporates global, local, and sub-goal rewards in a single framework, which is backed by theoretical analysis and experimental confirmation both on benchmark tasks and real-life case studies.
II.
RELATED WORKS
Reinforcement Learning (RL) and Multi-Agent RL Every established machine learning approach uses Reinforcement Learning (RL) to teach agents how they should act based on environmental interaction to optimize their collective reward points .
Agents in RL undertook environment observation steps followed by selected actions until they received reward-based feedback to control their learning behavior.
MARL refers to multiple agents which interact within one environment either joining forces or competing against each other to accomplish collective or individual targets .
The single-agent reinforcement learning capabilities receive additional complexity from MARL because the environment becomes non-stationary when other agents learn and credit assignment becomes essential to determine each agent's impact and the system also requires scalability due to exponential increases in interaction complexity with additional agents.
In research terms MARL presents numerous difficulties which make it a complex yet promising field that shows special potential for distributed decision-making problems .
Reward Structures in RL The behavior of RL agents significantly depends on reward structures that guide their actions.
The main design of traditional RL systems utilizes flat reward structures which produce one unified reward signal based on agent actions .
The effectiveness of agents in complex and extended environments with complex goals becomes limited by flat reward methodologies.
Agents that use hierarchical reward structures receive intermediate rewards which help them learn about decomposed sub-goals from main tasks.
rewards have two classifications: intrinsic rewards and extrinsic rewards.
Agents obtain extrinsic rewards from environmental factors yet intrinsic rewards function from within the person to promote exploration along with curiosity .
Learning efficiency together with performance level rise when hierarchies and intrinsic systems work as reward strategies specifically for complex or sparse reward areas .
Hierarchical Approaches in RL HRL provides an abstract framework for learning that splits complex tasks into different layers of complexity to enhance agent effectiveness in task completion.
Within HRL practice people use the options framework as a method which enables high-level policies to select from actions .
that represent sub-tasks or sub-goals while spanning multiple time steps .
Through hierarchical decomposition agents achieve short-term objectives leading them toward the accomplishment of long-term goals .
Ae .
Multi-Agent HRL extends hierarchical approaches to settings that enable agents to work either collaboratively or competitively at multiple abstraction levels.
Multi-agent HRL applications exist in robotic coordination functions along with resource allocation and traffic management systems.
The process of designing proper hierarchical structures for multi-agent systems presents difficulties because agents must establish coordinated communication .
Saadya Fahad.
Multi-Agent Reinforcement Learning with Hierarchical Reward Structures for Complex Optimization Problems Journal of Robotics and Control (JRC) ISSN: 2715-5072 Complex Optimization Problems The optimization of multiple object-related challenges across dynamic multi-agent systems demands complex combinations of decision-making on a large scale .
The agents must optimize their actions to achieve global or local objectives in path planning and task allocation and energy distribution .
Linear programming and heuristic algorithms demonstrate challenges when attempting to address dynamic distributed systems because they lack sufficient capabilities for these problems.
MARL presents a promising solution but currently operating methods struggle with three fundamental issues related to reward specification and coordinator needs and scalability .
Flat reward structures enable suboptimal solutions because the complex multi-objective tasks remain difficult to capture effectively by these reward structures.
Advanced frameworks must include hierarchical reward structures because they resolve the difficulties which multi-agent systems encounter with complex optimization problems .
The provided review establishes fundamental understanding regarding critical aspects of RL together with MARL and complex optimization problems and their existing solutions.
This overview serves as preparation before presenting the developed framework where hierarchical reward structures join MARL to handle the encountered challenges .
Ae .
implementation and design purposes .
The method uses mathematical models for developing effective algorithmic solutions which allow optimization of large-scale multiobjective dynamic problems efficiently.
The paper provides detailed explanations of the methodology that combines mathematical precision with real-world operational MARL operates in Markov Games as a problem space that features NN agents playing in a common environment.
The agents receive state observations ycyc OO ycI at time yc before selecting actions ycaycycn OO yaycn from which they derive rewards ycycycn .
The joint space of actions contains all choices as ycayc = .
ca1yc , ycayc2 .
A , ycaycycA ).
The transition process follows the distribution ycE.
cyc 1 O ycyc, ycay.
The agents aim to optimize OcycNyc=0 yu yc ycyc by using the discount factor that exists between .
The main issues in MARL are the joint action space growth problem when ycA increases and global cooperation challenges between agents along with uncertain observations by individual agents.
Under the hierarchical reward structure, the global reward ycI gets broken down into various fundamental rewards including ycycycn for each agent as well as ycycycycyca for sub-goals.
The world reward consists of the following factors as .
ycN i.
METHODOLOGY
The flow chart of research methodology is shown in Fig.
ycI = Ocyu yc=0 yc ycA yciycoycuycaycayco .
uycyc yu Oc ycycycn yuycycycycyca ) ycn=1 .
Weighting factors (, 89293, 151.
were added in order to balance factors of global, local and sub-goal reward.
These weights were assigned initial baseline values of .
4, 0.
, which is set according to initial experiments.
It was followed by sensitivity analysis to determine the effect of changes in each of the weights on the coordination, speed of convergence, and final performance.
The analysis reaffirmed that intermediate values usually have a stable convergence whereas extreme values can result in the lack of stability and coordination among agents.
This draws the significance of adaptive mechanisms of tuning which may be viewed as a point to work in the future.
The weight factors yu, yu and yu establish a balance between sub-goal and local and global rewards.
The decomposition mechanism directs agents toward short-term along with longterm targets.
Fig.
The flowchart of proposed methodology Problem Formulation This study's methodology deals with Multi-Agent Reinforcement Learning (MARL) complex optimization problems by developing a hierarchical reward framework for Proposed Framework The MARL framework achieves its integration of hierarchical rewards as part of its learning process .
Ae .
The framework contains three core parts beginning with the global reward system which tracks the optimization goal and continues with local rewards systems that promote agent selfperformance and completes with sub-goal rewards systems meant to guide middle-step progress.
The global reward yciycoycuycaycayco ycyc uses system-wide performance factors to generate its value yet local rewards ycycycn award agents through their independent movements.
The objective of sub-goal rewards ycycycycyca is to provide step-by-step guidance which leads agents Saadya Fahad.
Multi-Agent Reinforcement Learning with Hierarchical Reward Structures for Complex Optimization Problems Journal of Robotics and Control (JRC) ISSN: 2715-5072 toward achieving intermediate targets consisting of sub-tasks or state objectives.
The sharing of information takes place through communication protocols which enable agents to exchange both local and global information.
A centralized system integrates information into rewards that it distributes to agents yet decentralized systems enable agents to exchange information directly with their neighboring agents.
The communication overhead reduction is achieved through message-passing algorithms including graph neural networks (GNN.
to establish ycO as the agent set and ya as the communication link set in the encoded graph representation ya.
The learning algorithm introduces hierarchical rewards into MARL procedures that utilize Q-learning and policy gradient methods.
Q-learning updates its action-value function ycE.
cyc , ycayc ) according to .
cyc , ycayc ) Ia ycE.
cyc , ycayc ) yuC .
ycoycaycu ycE.
cyc 1 ,ycayc 1 )OeycE.
cyc ,ycayc ) yuycayc 1 where yuC is the learning rate.
For policy gradient methods, the policy yuUyuE( ycaycycn O ycyc ) is optimized using the gradient as .
yuyuE ya.
uE) = yayuUyuE .
uyuE ycoycuyciyuUyuE( ycaycycn O ycyc ).
cyc , ycayc )], .
where ya.
cyc , ycayc ) is the advantage function, computed using hierarchical rewards.
Algorithm 1.
Training procedure of hierarchical reward MARL framework Input: Environment states S, actions A, reward weights (, , ) Initialize: Q-values, policy parameters , communication graph G For each episode do Reset environment and agent states For each timestep t do For each agent i do Observe local state s_i Select action a_i using policy A_ end for Execute joint action A_t in environment Receive hierarchical rewards:
r_global, r_local.
, r_sub.
Compute composite reward:
r_total = * r_global * r_local * r_sub Update Q-values or policy parameters using r_total Exchange necessary information among agents end for end for Output: Optimized policy A_ for all agents Theoretical Analysis The framework analysis uses the condition of bounded rewards together with stationary policies to determine convergence .
Ae .
The proof utilizes the contraction mapping theorem to demonstrate Q-learning achieves optimal action-value function ycEO given Bellman optimality holds for the hierarchical rewards as .
max ycEO .
cyc 1 ,ycayc 1 ) ycEO .
cyc , ycayc ) = ya.
cyc yuycayc 1 The framework demonstrates polynomial scaling behavior regarding computational complexity based on the number of agents ycA as well as the state-action space size O ycI O yO ya O.
Experimental Setup A framework evaluation occurs with benchmark problems including multi-agent pathfinding along with resource allocation and task scheduling problems .
Several agents travel through a grid world following their destinations with collision avoidance enabled in this context.
The achievement of team goals at a global level creates the primary measure of success while individual achievement defines the local measure of success.
Intervening rewards are provided for attaining the specified intermediate points .
The research team performs case studies to implement the algorithm in logistics sectors and energy management fields and traffic control centers.
Agents in logistical operations determine optimized delivery pathways while energy and traffic control systems employ them to maintain balance between supply and demand and control intersection traffic flow respectively.
The solution quality reaches its best outcome when measured by global rewards while convergence speed determines the number of episodes needed to optimize performance and the system maintains high efficiency when N increases.
Performance assessment regarding coordination examines agent communication overhead .
otal message coun.
along with the extent of agent collaboration between individual goals and global targets.
The proposed framework undergoes comparison with conventional reward structures, flat outlooks, standard optimization techniques together with existing MARL The designed framework achieves superior outcomes regarding solution quality and convergence speed and scalability performance.
The mathematical method allows researchers to solve complex MARL optimization tasks through reward structures organized at different levels.
The study combines theoretical examination and algorithm development alongside practical tests which advance MARL techniques while delivering useful approaches for actual The experimental setup is also described in detail to provide reproducibility and transparency.
Table 1 provides a summary of simulation environment, programming tools and training parameters applied during the analysis of the proposed hierarchical reward MARL It is an addition to the methodological description Saadya Fahad.
Multi-Agent Reinforcement Learning with Hierarchical Reward Structures for Complex Optimization Problems Journal of Robotics and Control (JRC) ISSN: 2715-5072 and is consistent with best practices when reporting multiagent reinforcement learning experiments.
TABLE 1.
EXPERIMENTAL SETTING AND TRAINING SETUP FOR
HIERARCHICAL REWARD MARL FRAMEWORK
Component Operating System Programming Language Libraries/Frameworks Simulation Environment Hardware Training Episodes Learning Rate () Discount Factor () Reward Weights (, , ) Batch Size Optimizer Specification Ubuntu 22.
04 LTS .
-bi.
Python 3.
instances by about 2035 per cent relative to the baseline approach, attains greater global reward values, and it substantially reduces communication overhead.
Besides, the scalability analysis shows that it can withstand more than 100 agents without the performance showing any changes, which was not the case with the old methods.
These results confirm the benefit of multi-level reward decomposition as opposed to flat or partially hierarchical reward designs.
TensorFlow 2.
PyTorch 1.
NumPy 1.
SciPy 1.
Matplotlib 3.
Grid-world multi-agent pathfinding, resource allocation, task scheduling benchmarks Intel Xeon Gold 6338 CPU .
0 GHz y 32 core.
, 128 GB RAM.
NVIDIA A100 .
GB) GPU 10,000 Ae 20,000 per experiment 001 Ae 0.
uned per benchmar.
4, 0.
3, 0.
baseline, varied in sensitivity Adam optimizer Since some traditional MARL methods are based on the flat reward signals, the suggested framework proposes a hierarchical breakdown of rewards into global, local, and subgoal rewards.
This design explicitly enhances coordination, minimizes communication overhead and improves convergence stability and as such, deals with the limitations that have been noted in earlier research.
IV.
Fig.
Performance evaluation: reward convergence.
The visual representation tracks performance through varying problem size and agent number.
The system shows scalability performance when solving moderate problems with agent numbers within specific ranges.
The framework effectively manages problems of larger proportions according to Fig.
RESULTS AND DISCUSSION
The training process displays how global and local rewards and sub-goal rewards converge with each training The use of hierarchical rewards helps create an efficient learning stability for agents during training.
Fig.
illustrates how splitting rewards affects the performance of the learning process.
Each experiment was repeated 10 times using different random seeds to check the statistical validity.
The results reported give the mean and standard deviation between the Error bars have been added to figures and Table 2 summarizes the average performance with measures of This inclusion enhances credibility of the conclusions made out of experiments.
Table 2 results demonstrate that the suggested hierarchical reward MARL scheme successfully achieves a better performance in the previously mentioned metrics in comparison to the current approaches.
It lowers convergence Fig.
Scalability Analysis The framework investigates coordination effectiveness while it operates under different communication overhead Fig.
4 shows how increased communication needs lead to diminished coordination performance levels.
The plot illustrates how increased communication reduces coordination ability.
TABLE 2.
PERFORMANCE EVALUATION OF THE PROPOSED FRAMEWORK WITH EXISTING METHODS
Method Flat reward MARL .
Hierarchical MARL (Yang et al.
, 2.
HRL with dual coordination (Xu et al.
Proposed hierarchical reward MARL
Convergence Speed
15,200 A 420
12,800 A 350
11,900 A 290
Global Reward 71 A 0.
79 A 0.
82 A 0.
Communication Overhead High Medium Medium Scalability .
Limited (O .
Moderate (O .
Good (O .
9,800 A 270 86 A 0.
Low High (Ou .
Saadya Fahad.
Multi-Agent Reinforcement Learning with Hierarchical Reward Structures for Complex Optimization Problems Journal of Robotics and Control (JRC) ISSN: 2715-5072 more pronounced when using fewer agents.
Fig.
7 reveals the complexity of problem expansion as one of its main findings.
Fig.
Coordination and communication analysis A performance analysis helps compare the effects of rewards at global, local and sub-goal levels.
Fig.
5 shows that the performance advantages come from individual reward The chart demonstrates the necessity for adjusting between different reward subsystems.
Fig.
Impact of Problem Size on Performance Measurements of the end rewards from global and local levels and sub-goals are compared.
Global rewards have domination but local and sub-goal rewards continue to retain their value.
Fig.
8 reveals important details about how reward amounts distributed to agents.
Fig.
Ablation studies: impact of reward components Performance measurements of global rewards occur under different numbers of agents.
The results indicate performance enhancement through reduced agent numbers while system performance becomes steady when they Fig.
6 demonstrates how agent numbers influence the efficiency with which agents learn.
Fig.
Distribution of rewards at the end of training.
Performance measurements are analyzed regarding their reaction to variations in alpha global reward weight values.
The performance reaches its optimal point when alpha exists at intermediate levels.
Fig.
9 demonstrates the need to properly adjust global reward weights.
Fig.
Performance Comparison for Different Numbers of Agents Performance variations due to problem size changes are reviewed among different agent population levels.
Larger problems cause decreased performance which becomes even Fig.
Sensitivity analysis: impact of alpha A performance variation analysis takes place through modifications to the local reward weight .
A higher beta Saadya Fahad.
Multi-Agent Reinforcement Learning with Hierarchical Reward Structures for Complex Optimization Problems Journal of Robotics and Control (JRC) ISSN: 2715-5072 value increases team coordination yet demonstrates negative effects on overall team success.
Local reward tuning requires an evaluation of its performance-related trade-offs according to Fig.
Fig.
Convergence over episodes Fig.
Sensitivity Analysis: Impact of Beta An evaluation of how agent performance reacts to variations in sub-goal reward weight .
takes place.
The optimal performance results occur when sub-goal reward weight .
maintains moderate values.
Fig.
11 reveals the function that sub-goal rewards play in directing agent Fig.
13 shows how communication overhead between agents advances through training phases.
The system requires additional overhead throughout the training process because agent interactions grow in proportion to the increasing Multi-agent systems demonstrate a direct relationship between communication performance and operational efficiency based on the data shown.
Fig.
Communication Overhead Over Episodes Fig.
Sensitivity analysis: impact of delta Overall behavior of agents toward optimal policy discovery across time steps appears in the Convergence Over Episodes Fig.
12 through maximum Q-value tracking.
Agent exploration drives an immediate rapid increase in the metric but eventually it stabilizes when agents achieve nearly optimal policies.
The hierarchical reward structure proves its effectiveness in maintaining stable and efficient learning in MARL frameworks based on the recorded trends.
Although the results indicate significant gains, the framework is sensitive to the choice of reward weights ( , , ).
Very high/very low values can disrupt learning and diminish coordination performance.
Moreover, communication overhead, albeit less so, also increases with the number of agents and this can impose practical limits in bandwidthconstrained settings.
The observed findings highlight the relevance of adaptive weighting procedures and designs that are communication efficient in further studies.
CONCLUSION
The study develops a strong MARL framework which implements hierarchical reward structures to resolve complicated optimization problems.
Analytical and experimental testing shows that this framework enhances agent coordination performance at the same time it improves maintenance scalability and learning rate capabilities.
Optimal designs of hierarchical rewards together with framework scalability proved challenging when dealing with dynamic complex environments.
The next stage of research should concentrate on developing the framework to work with partially observable adversaries alongside deep MARL and transfer learning methods for adaptability enhancement.
Saadya Fahad.
Multi-Agent Reinforcement Learning with Hierarchical Reward Structures for Complex Optimization Problems Journal of Robotics and Control (JRC) ISSN: 2715-5072 The future developments made to the framework will enhance its potential applications for real-world situations.
REFERENCES