Collaborative training of heterogeneous reinforcement learning agents in environments with sparse rewards: what and when to share?

Simple item page
dc.contributor.authorAndres, Alain
dc.contributor.authorVillar-Rodriguez, Esther
dc.contributor.authorSer, Javier Del
dc.contributor.institutionIA
dc.contributor.institutionQuantum
dc.date.accessioned2024-07-24T12:10:44Z
dc.date.available2024-07-24T12:10:44Z
dc.date.issued2023-08
dc.descriptionPublisher Copyright: © 2022, The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature.
dc.description.abstractIn the early stages of human life, babies develop their skills by exploring different scenarios motivated by their inherent satisfaction rather than by extrinsic rewards from the environment. This behavior, referred to as intrinsic motivation, has emerged as one solution to address the exploration challenge derived from reinforcement learning environments with sparse rewards. Diverse exploration approaches have been proposed to accelerate the learning process over single- and multi-agent problems with homogeneous agents. However, scarce studies have elaborated on collaborative learning frameworks between heterogeneous agents deployed into the same environment, but interacting with different instances of the latter without any prior knowledge. Beyond the heterogeneity, each agent’s characteristics grant access only to a subset of the full state space, which may hide different exploration strategies and optimal solutions. In this work we combine ideas from intrinsic motivation and transfer learning. Specifically, we focus on sharing parameters in actor-critic model architectures and on combining information obtained through intrinsic motivation with the aim of having a more efficient exploration and faster learning. We test our strategies through experiments performed over a modified ViZDooM’s My Way Home scenario, which is more challenging than its original version and allows evaluating the heterogeneity between agents. Our results reveal different ways in which a collaborative framework with little additional computational cost can outperform an independent learning process without knowledge sharing. Additionally, we depict the need for modulating correctly the importance between the extrinsic and intrinsic rewards to avoid undesired agent behaviors.en
dc.description.sponsorshipThe authors would like to thank the Basque Government for its support through the ELKARTEK program (3KIA project, KK-2020/00049). A. Andres receives funding support from the Basque Government through its BIKAINTEK PhD support program. J. Del Ser also acknowledges support received from the same institution through the consolidated research group MATHMODE (Ref. IT1456–22).
dc.description.statusPeer reviewed
dc.format.extent28
dc.identifier.citationAndres , A , Villar-Rodriguez , E & Ser , J D 2023 , ' Collaborative training of heterogeneous reinforcement learning agents in environments with sparse rewards : what and when to share? ' , Neural Computing and Applications , vol. 35 , no. 23 , pp. 16753-16780 . https://doi.org/10.1007/s00521-022-07774-5
dc.identifier.doi10.1007/s00521-022-07774-5
dc.identifier.issn0941-0643
dc.identifier.urihttps://hdl.handle.net/11556/4109
dc.identifier.urlhttp://www.scopus.com/inward/record.url?scp=85129224127&partnerID=8YFLogxK
dc.language.isoeng
dc.relation.ispartofNeural Computing and Applications
dc.relation.projectIDEusko Jaurlaritza, IT1456–22
dc.rightsinfo:eu-repo/semantics/openAccess
dc.subject.keywordsCollaborative learning
dc.subject.keywordsHeterogeneous agents
dc.subject.keywordsIntrinsic motivation
dc.subject.keywordsReinforcement learning
dc.subject.keywordsSparse rewards
dc.subject.keywordsTransfer learning
dc.subject.keywordsSoftware
dc.subject.keywordsArtificial Intelligence
dc.titleCollaborative training of heterogeneous reinforcement learning agents in environments with sparse rewards: what and when to share?en
dc.typejournal article
Files