Browsing by Author "Martinez, Aritz D."
Now showing 1 - 12 of 12
Results Per Page
Sort Options
Item Adaptive Multifactorial Evolutionary Optimization for Multitask Reinforcement Learning(2022-04-01) Martinez, Aritz D.; Del Ser, Javier; Osaba, Eneko; Herrera, Francisco; IA; QuantumEvolutionary computation has largely exhibited its potential to complement conventional learning algorithms in a variety of machine learning tasks, especially those related to unsupervised (clustering) and supervised learning. It has not been until lately when the computational efficiency of evolutionary solvers has been put in prospective for training reinforcement learning models. However, most studies framed so far within this context have considered environments and tasks conceived in isolation, without any exchange of knowledge among related tasks. In this manuscript we present A-MFEA-RL, an adaptive version of the well-known MFEA algorithm whose search and inheritance operators are tailored for multitask reinforcement learning environments. Specifically, our approach includes crossover and inheritance mechanisms for refining the exchange of genetic material, which rely on the multilayered structure of modern deep-learning-based reinforcement learning models. In order to assess the performance of the proposed approach, we design an extensive experimental setup comprising multiple reinforcement learning environments of varying levels of complexity, over which the performance of A-MFEA-RL is compared to that furnished by alternative nonevolutionary multitask reinforcement learning approaches. As concluded from the discussion of the obtained results, A-MFEA-RL not only achieves competitive success rates over the simultaneously addressed tasks, but also fosters the exchange of knowledge among tasks that could be intuitively expected to keep a degree of synergistic relationship.Item AT-MFCGA: An Adaptive Transfer-guided Multifactorial Cellular Genetic Algorithm for Evolutionary Multitasking(2021-09) Osaba, Eneko; Del Ser, Javier; Martinez, Aritz D.; Lobo, Jesus L.; Herrera, Francisco; Quantum; IATransfer Optimization is an incipient research area dedicated to solving multiple optimization tasks simultaneously. Among the different approaches that can address this problem effectively, Evolutionary Multitasking resorts to concepts from Evolutionary Computation to solve multiple problems within a single search process. In this paper we introduce a novel adaptive metaheuristic algorithm to deal with Evolutionary Multitasking environments coined as Adaptive Transfer-guided Multifactorial Cellular Genetic Algorithm (AT-MFCGA). AT-MFCGA relies on cellular automata to implement mechanisms in order to exchange knowledge among the optimization problems under consideration. Furthermore, our approach is able to explain by itself the synergies among tasks that were encountered and exploited during the search, which helps us to understand interactions between related optimization tasks. A comprehensive experimental setup is designed to assess and compare the performance of AT-MFCGA to that of other renowned Evolutionary Multitasking alternatives (MFEA and MFEA-II). Experiments comprise 11 multitasking scenarios composed of 20 instances of 4 combinatorial optimization problems, yielding the largest discrete multitasking environment solved to date. Results are conclusive in regard to the superior quality of solutions provided by AT-MFCGA with respect to the rest of the methods, which are complemented by a quantitative examination of the genetic transferability among tasks throughout the search process.Item Collaborative Exploration and Reinforcement Learning between Heterogeneously Skilled Agents in Environments with Sparse Rewards(Institute of Electrical and Electronics Engineers Inc., 2021-07-18) Andres, Alain; Villar-Rodriguez, Esther; Martinez, Aritz D.; Del Ser, Javier; IA; QuantumA critical goal in Reinforcement Learning is the minimization of the time needed for an agent to learn to solve a given environment. In this context, collaborative reinforcement learning refers to the improvement of this learning process through the interaction between agents, which usually yields better results than training each agent in isolation. Most studies in this area have focused on the case with homogeneous agents, namely, agents equally skilled for undertaking their task. By contrast, heterogeneity among agents could arise due to the particular capabilities on how they sense the environment and/or the actions they could perform. Those differences eventually hinder the learning process and information sharing between agents. This issue becomes even more complicated to address over hard exploration scenarios where the extrinsic rewards collected from the environment are sparse. This work sheds light on the impact of leveraging collaborative learning strategies between heterogeneously skilled agents over hard exploration scenarios. Our study gravitates on how to share and exploit knowledge between the agents so as to mutually improve their learning procedures, further considering mechanisms to cope with sparse rewards. We assess the performance of these strategies via extensive simulations over modifications of the ViZDooM environment, which allow examining their benefits and drawbacks when dealing with agents endowed with different behavioral policies. Our results uncover the inherent problems of not considering the skill heterogeneity of the agents in the knowledge sharing strategy, and unleash a manifold of research directions aimed at circumventing these noted issues.Item DMFEA-II: An adaptive multifactorial evolutionary algorithm for permutation-based discrete optimization problems(Association for Computing Machinery, Inc, 2020-07-08) Osaba, Eneko; Martinez, Aritz D.; Galvez, Akemi; Iglesias, Andres; Ser, Javier Del; Quantum; IAThe emerging research paradigm coined as multitasking optimization aims to solve multiple optimization tasks concurrently by means of a single search process. For this purpose, the exploitation of complementarities among the tasks to be solved is crucial, which is often achieved via the transfer of genetic material, thereby forging the Transfer Optimization field. In this context, Evolutionary Multitasking addresses this paradigm by resorting to concepts from Evolutionary Computation. Within this specific branch, approaches such as the Multifactorial Evolutionary Algorithm (MFEA) has lately gained a notable momentum when tackling multiple optimization tasks. This work contributes to this trend by proposing the first adaptation of the recently introduced Multifactorial Evolutionary Algorithm II (MFEA-II) to permutation-based discrete optimization environments. For modeling this adaptation, some concepts cannot be directly applied to discrete search spaces, such as parent-centric interactions. In this paper we entirely reformulate such concepts, making them suited to deal with permutation-based search spaces without loosing the inherent benefits of MFEA-II. The performance of the proposed solver has been assessed over 5 different multitasking setups, composed by 8 datasets of the well-known Traveling Salesman (TSP) and Capacitated Vehicle Routing Problems (CVRP). The obtained results and their comparison to those by the discrete version of the MFEA confirm the good performance of the developed dMFEA-II, and concur with the insights drawn in previous studies for continuous optimization.Item Evolutionary Multitask Optimization: a Methodological Overview, Challenges, and Future Research Directions: a Methodological Overview, Challenges, and Future Research Directions(2022-04-12) Osaba, Eneko; Del Ser, Javier; Martinez, Aritz D.; Hussain, Amir; Quantum; IAIn this work, we consider multitasking in the context of solving multiple optimization problems simultaneously by conducting a single search process. The principal goal when dealing with this scenario is to dynamically exploit the existing complementarities among the problems (tasks) being optimized, helping each other through the exchange of valuable knowledge. Additionally, the emerging paradigm of evolutionary multitasking tackles multitask optimization scenarios by using biologically inspired concepts drawn from swarm intelligence and evolutionary computation. The main purpose of this survey is to collect, organize, and critically examine the abundant literature published so far in evolutionary multitasking, with an emphasis on the methodological patterns followed when designing new algorithmic proposals in this area (namely, multifactorial optimization and multipopulation-based multitasking). We complement our critical analysis with an identification of challenges that remain open to date, along with promising research directions that can leverage the potential of biologically inspired algorithms for multitask optimization. Our discussions held throughout this manuscript are offered to the audience as a reference of the general trajectory followed by the community working in this field in recent times, as well as a self-contained entry point for newcomers and researchers interested to join this exciting research avenue.Item Hybridizing differential evolution and novelty search for multimodal optimization problems(Association for Computing Machinery, Inc, 2019-07-13) Martinez, Aritz D.; Fister, Iztok; Osaba, Eneko; Fister, Iztok; Oregi, Izaskun; Ser, Javier Del; IA; QuantumMultimodal optimization has shown to be a complex paradigm underneath real-world problems arising in many practical applications, with particular prevalence in physics-related domains. Among them, a plethora of cases within the computational design of aerospace structures can be modeled as a multimodal optimization problem, such as aerodynamic optimization or airfoils and wings. This work aims at presenting a new research direction towards efficiently tackling this kind of optimization problems, which pursues the discovery of the multiple (at least locally optimal) solutions of a given optimization problem. Specifically, we propose to exploit the concept behind the so-called Novelty Search mechanism and embed it into the self-adaptive Differential Evolution algorithm so as to gain an increased level of controlled diversity during the search process. We assess the performance of the proposed solver over the well-known CEC'2013 suite of multimodal test functions. The obtained outcomes of the designed experimentation supports our claim that Novelty Search is a promising approach for heuristically addressed multimodal problems.Item MO-MFCGA: Multiobjective Multifactorial Cellular Genetic Algorithm for Evolutionary Multitasking(Institute of Electrical and Electronics Engineers Inc., 2021) Osaba, Eneko; Del Ser, Javier; Martinez, Aritz D.; Lobo, Jesus L.; Nebro, Antonio J.; Yang, Xin She; Quantum; IAMultiobjetive optimization has gained a considerable momentum in the evolutionary computation scientific community. Methods coming from evolutionary computation have shown a remarkable performance for solving this kind of optimization problems thanks to their implicit parallelism and the simultaneous convergence towards the Pareto front. In any case, the resolution of multiobjective optimization problems (MOPs) from the perspective of multitasking optimization remains almost unexplored. Multitasking is an incipient research stream which explores how multiple optimization problems can be simultaneously addressed by performing a single search process. The main motivation behind this solving paradigm is to exploit the synergies between the different problems (or tasks) being optimized. Going deeper, we resort in this paper to the also recent paradigm Evolutionary Multitasking (EM). We introduce the adaptation of the recently proposed Multifactorial Cellular Genetic Algorithm (MFCGA) for solving MOPs, giving rise to the Multiobjective MFCGA (MO-MFCGA). An extensive performance analysis is conducted using the Multiobjective Multifactorial Evolutionary Algorithm as comparison baseline. The experimentation is conducted over 10 multitasking setups, using the Multiobjective Euclidean Traveling Salesman Problem as benchmarking problem. We also perform a deep analysis on the genetic transferability among the problem instances employed, using the synergies among tasks aroused along the MO-MFCGA search procedure.Item More is not Always Better: Insights from a Massive Comparison of Meta-heuristic Algorithms over Real-Parameter Optimization Problems(Institute of Electrical and Electronics Engineers Inc., 2021) Del Ser, Javier; Osaba, Eneko; Martinez, Aritz D.; Bilbao, Miren Nekane; Poyatos, Javier; Molina, Daniel; Herrera, Francisco; IA; QuantumMuch controversy has been lately risen around the design and performance of modern bio-inspired optimization methods, in particular due to the alleged lack of algorithmic novelty in their definition with respect to traditional heuristic solvers. In this work we present a first attempt at shedding empirical evidence over this debate, for which results of a benchmark with unprecedented scales in terms of problems and algorithms are reported and discussed. Specifically, informed conclusions are held in what refers to the claimed superior performance of these bio-inspired solvers and their competitiveness when compared to competition-winning alternatives. Finally, we prove that the tailored selection of a subset of problems and techniques can unfairly bias the comparisons favoring any of such algorithms, ultimately arriving at illusory conclusions about their comparative performance.Item Multifactorial Cellular Genetic Algorithm (MFCGA): Algorithmic Design, Performance Comparison and Genetic Transferability Analysis(Institute of Electrical and Electronics Engineers Inc., 2020-07) Osaba, Eneko; Martinez, Aritz D.; Lobo, Jesus L.; Ser, Javier Del; Herrera, Francisco; Quantum; IAMultitasking optimization is an incipient research area which is lately gaining a notable research momentum. Unlike traditional optimization paradigm that focuses on solving a single task at a time, multitasking addresses how multiple optimization problems can be tackled simultaneously by performing a single search process. The main objective to achieve this goal efficiently is to exploit synergies between the problems (tasks) to be optimized, helping each other via knowledge transfer (thereby being referred to as Transfer Optimization). Furthermore, the equally recent concept of Evolutionary Multitasking (EM) refers to multitasking environments adopting concepts from Evolutionary Computation as their inspiration for the simultaneous solving of the problems under consideration. As such, EM approaches such as the Multifactorial Evolutionary Algorithm (MFEA) has shown a remarkable success when dealing with multiple discrete, continuous, single-, and/or multi-objective optimization problems. In this work we propose a novel algorithmic scheme for Multifactorial Optimization scenarios - the Multifactorial Cellular Genetic Algorithm (MFCGA) - that hinges on concepts from Cellular Automata to implement mechanisms for exchanging knowledge among problems. We conduct an extensive performance analysis of the proposed MFCGA and compare it to the canonical MFEA under the same algorithmic conditions and over 15 different multitasking setups (encompassing different reference instances of the discrete Traveling Salesman Problem). A further contribution of this analysis beyond performance benchmarking is a quantitative examination of the genetic transferability among the problem instances, eliciting an empirical demonstration of the synergies emerged between the different optimization tasks along the MFCGA search process.Item A Multifactorial Cellular Genetic Algorithm for Multimodal Multitask Optimization(Institute of Electrical and Electronics Engineers Inc., 2022) Osaba, Eneko; Del Ser, Javier; Martinez, Aritz D.; Lobo, Jesus L.; Quantum; IAIn multimodal optimization problems the main goal is to find as many global optima as possible by using a single search process. This type of optimization tasks emerges in many real-world scenarios in assorted fields including medicine, physics, and aerospace, among many others. However, addressing several multimodal optimization problems simultaneously has received little attention from the multitask optimization community to date. Even though solving different multimodal problems at the same time can largely benefit from the existing synergies among the modes of different tasks, this setup has been less studied than other optimization tasks. This work finds its inspiration in the incipient concepts of Evolutionary Multitasking and Multifactorial Optimization to propose a multifactorial Cellular Genetic Algorithm for solving multimodal optimization problems. Our designed algorithm expedites the search for the global optima of different problems at a time by including several algorithmic steps aimed at adapting the search itself as per the synergies found over the exploration of the problems' landscape. An extensive experimentation has been designed using 14 different functions from the CEC'2013 competition on multimodal optimization benchmark. Besides evaluating the performance of the devised algorithm to retain the global optima of every function in the benchmark, we also conduct an analysis of the transfer of knowledge among such functions. Finally, we compare its performance to that of a winning proposal in this CEC'2013 competition so as to reflect on the suitability of the multitasking paradigm to solve multimodal optimization tasks.Item On the Transferability of Knowledge among Vehicle Routing Problems by using Cellular Evolutionary Multitasking(Institute of Electrical and Electronics Engineers Inc., 2020-09-20) Osaba, Eneko; Martinez, Aritz D.; Lobo, Jesus L.; Laña, Ibai; Ser, Javier Del; Quantum; IAMultitasking optimization is a recently introduced paradigm, focused on the simultaneous solving of multiple optimization problem instances (tasks). The goal of multitasking environments is to dynamically exploit existing complementarities and synergies among tasks, helping each other through the transfer of genetic material. More concretely, Evolutionary Multitasking (EM) regards to the resolution of multitasking scenarios using concepts inherited from Evolutionary Computation. EM approaches such as the well-known Multifactorial Evolutionary Algorithm (MFEA) are lately gaining a notable research momentum when facing with multiple optimization problems. This work is focused on the application of the recently proposed Multifactorial Cellular Genetic Algorithm (MFCGA) to the well-known Capacitated Vehicle Routing Problem (CVRP). In overall, 11 different multitasking setups have been built using 12 datasets. The contribution of this research is twofold. On the one hand, it is the first application of the MFCGA to the Vehicle Routing Problem family of problems. On the other hand, equally interesting is the second contribution, which is focused on the quantitative analysis of the positive genetic transferability among the problem instances. To do that, we provide an empirical demonstration of the synergies arisen between the different optimization tasks.Item Simultaneously Evolving Deep Reinforcement Learning Models using Multifactorial optimization(Institute of Electrical and Electronics Engineers Inc., 2020-07) Martinez, Aritz D.; Osaba, Eneko; Sery, Javier Del; Herrera, Francisco; IA; QuantumIn recent years, Multifactorial optimization (MFO) has gained a notable momentum in the research community. MFO is known for its inherent capability to efficiently address multiple optimization tasks at the same time, while transferring information among such tasks to improve their convergence speed. On the other hand, the quantum leap made by Deep Q Learning (DQL) in the Machine Learning field has allowed facing Reinforcement Learning (RL) problems of unprecedented complexity. Unfortunately, complex DQL models usually find it difficult to converge to optimal policies due to the lack of exploration or sparse rewards. In order to overcome these drawbacks, pre-trained models are widely harnessed via Transfer Learning, extrapolating knowledge acquired in a source task to the target task. Besides, meta-heuristic optimization has been shown to reduce the lack of exploration of DQL models. This work proposes a MFO framework capable of simultaneously evolving several DQL models towards solving interrelated RL tasks. Specifically, our proposed framework blends together the benefits of meta-heuristic optimization, Transfer Learning and DQL to automate the process of knowledge transfer and policy learning of distributed RL agents. A thorough experimentation is presented and discussed so as to assess the performance of the framework, its comparison to the traditional methodology for Transfer Learning in terms of convergence, speed and policy quality, and the intertask relationships found and exploited over the search process.