Emergence of linguistic conventions in multi-agent reinforcement learning

Abstract

Recently, emergence of signaling conventions, among which language is a prime example, draws a considerable interdisciplinary interest ranging from game theory, to robotics to evolutionary linguistics. Such a wide spectrum of research is based on much different assumptions and methodologies, but complexity of the problem precludes formulation of a unifying and commonly accepted explanation. We examine formation of signaling conventions in a framework of a multi-agent reinforcement learning model. When the network of interactions between agents is a complete graph or a sufficiently dense random graph, a global consensus is typically reached with the emerging language being a nearly unique object-word mapping or containing some synonyms and homonyms. On finite-dimensional lattices, the model gets trapped in disordered configurations with a local consensus only. Such a trapping can be avoided by introducing a population renewal, which in the presence of superlinear reinforcement restores an ordinary surface-tension driven coarsening and considerably enhances formation of efficient signaling.

Fig 1. An elementary step of a single-object version of the model (N_w = 3).

Using the probabilities defined in Eq (1), the speaker selects one of its words (here: W₂). Next both the speaker and the hearer increase their weights of the selected word by 1.

Fig 2. An elementary step of a multi-object version of the model (N_w = 3, N_o = 2).

With a uniform probability 1/N_o, the speaker chooses an object (the corresponding section of the inventory is encircled by the dotted line). Using the relevant weights (in solid circles), the speaker calculates the probabilities defined in Eq (2) and selects one of its words (here: W₁). Next the hearer tries to guess the object the speaker is talking about by calculating the probabilites (3) based on its weights of the communicated word (in circles). When the hearer’s guess is correct, both agents increase their corresponding weights by 1.

Fig 3. Spatial distribution of s₁, the probability that an agent will select word W₁ (Eq (1)).

Results for a single-object model on a square lattice with N = 10² ⋅ 10² = 10⁴, α = 2, N_w = 2. The dynamics traps the model in a disordered state (the configurations for, e.g., t = 10³ and t = 10⁴ differ only slightly). Since s₁ is generally close to unity or to zero, it means that almost every agent developed a strong preference toward one of the words.

Fig 4. Time dependence of m_L.

Results for a single-object model with N_w = 2 on complete graphs (N = 10⁵) and Cartesian lattices with d = 1 (N = 10⁵), d = 2 (N = 300² = 9 ⋅ 10⁴), and d = 3 (N = 50³ = 125 ⋅ 10³). In the case of the ordinary reinforcement (α = 1), none of the words is even locally preferred on a complete graph (since m_L → 0), and only a small asymmetry is seen for a square lattice. The results presented (also in the following figures) are averages over 20 independent runs. Statistical errors are typically smaller than plotting symbols and are omitted.

Fig 5. Time dependence of m_G.

Results for a complete graph and Cartesian lattices with the same simulation parameters as in Fig 4. Only for a complete graph and α = 2, a global symmetry gets broken and one word dominates in the entire population of agents.

Fig 6. Time dependence of m_G (random graphs).

Results for random graphs of an average node degree z and a complete graph (α = 2, N_w = 2, N = 10⁵). Only for sufficiently large z, the behavior on the random graphs is similar to that on the complete graph. Averaging over 20 runs includes generation of independent graphs.

Fig 7. Distribution of the dominant words that agents use to talk about the first object.

Left: simulations on a square lattice with N_o = 2, N_w = 10, N = 50 ⋅ 50 = 2.5 ⋅ 10³, α = 2. Right: the same simulations but with a population renewal (with probability p = 10⁻⁵).

Fig 8. Time dependence of the success rate.

Results for the model on the complete graph of size N = 10⁴ with N_o = 10, several values of N_w, and α = 2. For N_w = 50 and α = 1 (yellow line), we can see a much slower convergence to a consensus than for N_w = 50 and α = 2. The black line shows the success rate for the version with a population renewal (with probability p = 10⁻⁴).

Fig 9. Distribution of the total (normalized) weights associated with particular words.

Results for simulations with N_o = 10 on the complete graph of size N = 10⁴ and simulation time t = 10⁶ (α = 2). Simulations for t = 10⁵ lead to nearly identical distributions.

Fig 10. Spatial distribution of s₁, the probability that an agent will select word W₁ (Eq (1)).

Results for a single-object model on a square lattice with N = 10² ⋅ 10² = 10⁴, α = 2, N_w = 2, and with the renewal probability p = 10⁻⁴. In this case, contrary to Fig 3, clusters of agents with the same dominant word grow steadily.

Fig 11. Time dependence of 1 − m_L for several values of the renewal probability p.

Results for a single-object, square-lattice version of our model. Simulations were made for α = 2, N_w = 2, and N = 200 ⋅ 200 = 4 ⋅ 10⁴, and the results are averages of 20 independent runs. The line segment has a slope corresponding to t^−0.41.

Fig 12. Time dependence of 1 − m_L for several values of α.

Results for a single-object square-lattice version of our model for the renewal probability p = 10⁻³. Simulations were made for N_w = 2 and N = 200 ⋅ 200 = 4 ⋅ 10⁴, and the results are averages of 20 independent runs.