EasyDeepLearn
Reinforcement Learning · section 9 of 12

Multi-agent RL

7 interview questions on multi-agent rl, each answered in full. Free to read, no account needed.

Multi-agent RL — what makes it fundamentally different?

hard
  • (1) Environment now non-stationary from each agent's perspective (other agents' policies change during learning) — breaks Markov assumption.
  • (2) Emergent behaviors (cooperation, competition, mixed-motive).
  • (3) Credit assignment across agents in cooperative settings.
  • (4) Coordination + equilibrium concepts (Nash, correlated).
  • (5) Scale: joint action space grows exponentially.
  • Foundational field for game theory + RL.
#multi-agentPermalink & quiz →

Centralized Training Decentralized Execution (CTDE).

hard
  • Cooperative multi-agent paradigm: during training, use global state + all agents' info (centralized critic).
  • At execution, each agent uses only its local observations (decentralized policy).
  • Best of both worlds: centralized training resolves non-stationarity + credit; decentralized execution scales + preserves partial observability.
  • MADDPG (continuous), QMIX (cooperative Q), MAPPO (cooperative PPO) all use CTDE.
#multi-agentPermalink & quiz →

Self-play — why does it work for games?

hard
  • Agent plays against copies of itself (past or current) → generates unlimited training data, difficulty auto-scales as agent improves.
  • In zero-sum games: converges to Nash equilibrium (Fictitious Self-Play, PSRO).
  • AlphaZero: naive self-play.
  • AlphaStar: League play (Nash, Main, Exploiters).
  • OpenAI Five: pool of frozen past versions.
  • Key: variety in opponents prevents collapse to narrow strategy.
#multi-agentPermalink & quiz →

League play in AlphaStar — mechanism.

hard
  • Vinyals et al. 2019.
  • Population of agents in 3 roles: (1) Main agents — general strong players.
  • (2) Main Exploiters — target Main agents specifically.
  • (3) League Exploiters — cover blind spots across whole league.
  • All train against a mixture of past + current opponents.
  • Prevents strategy cycling (rock-paper-scissors problem).
  • Grandmaster StarCraft II.
  • Foundation of modern multi-agent training for esports.
#multi-agentPermalink & quiz →

How do agents learn to cooperate?

hard
  • (1) Shared reward → both agents want same outcome (value decomposition, MAPPO).
  • (2) Communication learning (RIAL, DIAL, CommNet — learn what/when to communicate).
  • (3) Learn joint value function (QMIX).
  • (4) Difference reward: reward each agent for its marginal contribution.
  • (5) Curriculum from easier sub-tasks to full coordination.
  • (6) Social dilemmas (Iterated PD): reciprocity via memory / punishment.
#multi-agentPermalink & quiz →

Emergent communication in MARL — canonical result.

hard
  • Agents can learn to communicate over discrete tokens via reward gradient (differentiable via Gumbel-softmax or reinforce).
  • Foerster et al.'s DIAL demonstrated on switch-riddle: agents develop protocol to signal correct answer.
  • Later work: emergent grammar and compositionality (with pressure).
  • Modern: LLM-agent negotiation and collaboration extends emergent-comm to natural language.
#multi-agentPermalink & quiz →

AlphaStar StarCraft II — main technical innovations.

hard
  • Vinyals et al. 2019.
  • (1) LSTM policy + transformer over units.
  • (2) SL from replays → RL via league play.
  • (3) Auto-regressive action distribution over unit + action + coordinates.
  • (4) Distributed at scale (thousands of parallel games).
  • (5) League: Main / Main Exploiter / League Exploiter roles prevent strategy cycling.
  • Grandmaster level; imperfect information + real-time + long horizon.
#applications#multi-agentPermalink & quiz →

Practise Reinforcement Learning

All 12 sectionsFlashcardsQuiz