Reinforcement Learning

• Agent
  • This is the learner/decision maker.
• Environment
  • Everything the agents interact with.
• State (S)
  • Current situation of the environment.
  • For example, in Tic Tac Toe it’s a \(3 \times 3\) grid (so 9 boxes) and each box can be X, O or empty. Hence the number of states are \(3^{3 \times 3}\)
• Action (A)
  • Choices available to the agent.
• Policy (π(a|s))
  • This is the agent’s way of behaving at a given time.
  • It roughly maps states with the action to be taken in those states.
  • This is called the actor as it takes the current state and decides what action to take.
• Reward (R)
  • This is a single number sent by the environment to the agent.
  • It depends on the current state and action of the agent.
  • A negative reward is called a penalty. Eg. hitting a wall, losing a game, etc. are unfavourable results and must be penalized.
• Value
  • Rewards tell us what is good immediately, but value function tells us what is good in the long run.
  • It’s the total reward an agent can expect to accumlate over the future, starting from that state.
  • This is called the critic as it scores the actions.

• These are interfaces/translators between RL agents and their environment.
• They provide common functions across different agents and environments to make it normalized and consistent.

• Some common methods provided look like:

  env.reset()
  env.step(action)
  env.action_space
  env.observation_space
  

• Some common environment adaptors are:
  • Arcade learning Environment
  • CARLA
  • OpenAI Gym
  • OpenAI Retro
  • Opensim
  • PyGame Learning Environment
  • ViZDoom

The last 3 (state, action and reward) are components of RL.

The first 3 (policy, reward and value) are components of RL.

• Choosing actions that yield the highest immediate reward based on past experience is called exploitation.
• Trying new or less-frequent actions to discover better long-term strategies is called exploration.
• The goal is to balance exploring and exploiting.

RL splits into three broad approaches:

• An episode is a single, complete cycle of interaction between an agent and its environment.
• An episode starts with the starting state, and is said to have completed once a final state is encountered.
• If any task naturally ends, it is called episodic.
• For instance, a game of chess is episodic as the game does end.
• Something like managing CPU resources is non-episodic as there’s no end.
• Episodes are also known as trial, period or stage.

• You make a table like:

  State	Action	Next State	Probability	Reward
  ..	..	..	..	..

  The notations are given as:

  • State: \(s\)
  • Action: \(a\)
  • Next State: \(s'\)
  • Probability: \(P(s' \vert s, a)\) or \(T(s, a, s')\)
  • Reward: \(R(s, a, s')\)
• The sum of the Probability column should be 1. This is because: \[\Sigma P(s' \vert s, a) = 1\]

• For example:

  State	Action	Next State	Probability
  S2	Down	S4	0.8
  S2	Down	S2	0.2

• Stationarity means that the same table applies at each and every time step. The transition probabilities and the reward don’t change over time.

• This is a slot machine, which is basically a gambling device with spinning reels each with a bunch of symbols.
• If the symbol shown by each reel aligns and forms a certain desirable combination, you win a jackpot.
• The machine is called a one-arm bandit:
  • One arm because it has one lever.
  • Bandit because they were designed to take your money quickly while rarely repaying you with the jackpot.

• Here, you sit in front of a row of one-arm bandits.
• You don’t know which slot machine is more rewarding.
• You have to strike a balance between:
  • Exploit: pull the arm that has paid off best so far
  • Explore: pull a different arm to learn more about it, even though it might pay less right now

• Discount factor (0 ≤ γ < 1) tells us how much of the future rewards should the system be aware of.
  • γ = 0 would mean the system would only care about immediate rewards and won’t care about later consequences.
  • γ = 1 would mean the system is 50% concerned about immediate rewards and 50% concerned about later consequences.
• The finite return (total reward considering discount factor) is calculated as: \[G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots\] where \(R_{t+1}\), \(R_{t+2}\), … are rewards at each time step.
• Example: γ = 0.9, \(R_{t+1} = 10\), \(R_{t+2} = 20\), \(R_{t+3} = 30\) \[G_{0} = 10 + (0.9)(20) + (0.9)^{2}(30)\]

• It is given as \(P(s' \vert s, a)\)

• The number of time steps an agent plans for or interacts with the environment before the decision-making process ends or resets.
• It’s how far into the future the agent is planning/acting before the task ends.

\[\mathbb{E}[R_{t+1}\mid s,a] = \sum_{s'} P(s'\mid s,a)R(s,a,s') \] Say we have:

\(\mathbb{E}[R_{t+1}\mid s,a] = (0.7)(10) + (0.3)(-5) + (0.6)(5) = 8.5\)

• Deterministic environments might have \(P(s' \vert s, a) = 1\).
• On the other hand, stochastic environments have multiple next states like: \[P(B \vert A, \text{Right}) = 0.8\] \[P(C \vert A, \text{Right}) = 0.2\]