돗토리

3. Week_4-3_Actor-Critic for continuing tasks

doutoury2 — Mon, 21 Jun 2021 05:12:44 +0900

3. Week_4-3_Actor-Critic for continuing tasks

3. Actor-Critic for Continuing Tasks

Estimating the Policy Gradient
Actor-Critic Algorithm

$\cdot$ Estimating the Policy Gradient

Derive a sample-based estimate for the gradient of the average reward objective

We have an objective for policy optimization.
We also have the policy gradient theorem,
which gives us a simple expression for the gradient of that objective.

In this video,
we'll complete the puzzle by showing " how to estimate this gradient "
using the experience of an agent interacting with the environment.

Getting Stochastic Samples of the Gradient

We want to derive a gradient descent algorithm for our policy.
We have our objective and it's gradient due to the Policy Gradient Theorem.

Now, we need to figure out " how to approximate this gradient ".

Infact, what we will do is get a stochastic sample of the gradient.

Recall
this expression for the gradient of the average reward.

Computing the sum over states is really impractical.
But we can do the same thing we did
when deriving our stochastic gradient descent rule for policy evaluation.

We simply make updates from states we observe while following policy $\pi$.
$S_0, A_0, R_1, S_1, A_1, \;\;\; \cdots \;\;\; , S_t, A_t, R_t+1, \;\;\; \cdots$

This gradient from state $S_t$ provides an approximation
to the gradient of the average reward $\nabla r(\pi)$.
$\nabla r(\pi) \quad = \quad \displaystyle \sum_a \nabla\pi(a|\color{brown}{S_t},\theta_t) q_{\pi}(\color{brown}{S_t},a)$

As we discussed before for stochastic gradient descent,
we can adjust the weights with this approximation ,
and still guarantee you will reach a stationary point.

This is what the stochastic gradient descent update looks like
for the policy parameters.
$\theta_{t+1} \quad \doteq \quad \theta_t + \alpha \displaystyle \sum_a \nabla \pi(a|S_t,\theta_t) q_{\pi}(S_t,a)$

We could stop here,
but let's simplify this further.

Unbiasedness of the Stochastic Samples

Let's re-examine this from a perspective based on expectations.
This will help us simplify the update and give you more insight into why the update makes sense.

Notice that
the sum over states weighted by $\mu$
can be re-written as an expectation under $\mu$.
( using sampled state $S$ instead of all every state $s$ )
$= \displaystyle \sum_s \mu(s) \sum_a \nabla \pi(a|s,\theta) q_{\pi}(s,a)$
$\Rightarrow \mathbb{E}_{\mu} \big[ \displaystyle \sum_a \nabla \pi(a|S,\theta) q_{\pi}(S,a) \big]$\

Recall that
$\mu$ is the stationary distribution for $\pi$ which reflects state visitation under $\pi$.

Q. What is stationary distribution ?
A. https://en.wikipedia.org/wiki/Stationary_distribution

In fact,
the state's we oberve while following $\pi$ are distributed according to $\mu$
$S_t \sim \mu$

By computing the gradient from a state $S_t$,
we get an unbiased estimate of this expectation.

Thinking about our stachastic gradient as an unbiased estimate
suggests one other simplification !

Notice that
inside the expectation we have a sum over all actions $\displaystyle \sum_a$.
We want to make this term even simpler, and get rid of ther sum over all actions !

If this was an expectation over actions,
we could get a stochastic sample of this two(?), and avoid summing over all actions !

Getting Stochastic Samples with One Action

Here we're going to see
how we can get an unbiased gradient estimate using only one action 80,
which is the action taken by the agent.

It would be nice if the sum of our actions was weighted by $\pi$ and so was an expectation under $\pi$.

That way we could sample it
using the agent's action selection, which is distributed according to $\pi$.

It turns out this is an easy problem to solve !

To get a weighted sum correponding to an expectation
we can just multiply and divided by $\pi(a|s,\theta)$.

Now we have an expectation over actions drawn from $\pi$ for this term (in the red-box)!
$= \mathbb{E}_{\pi} \big[ \frac{\nabla \pi(A|S,\theta)}{\pi(A|S,\theta)} q_{\pi}(S,A) \big]$

Stochastic Gradient Ascent for Policy Parameters

The new stochastic gradient ascent update now looks like this.

As an aside,
it is common to rewrite this gradient as the gradient of the natural logarithm of $\pi$ !

This is based on a formula from calculus for the gradient of a logarithm.
$\nabla \big(f(x)\big) = \displaystyle \frac{\nabla f(x)}{f(x)}$

Using this rule,
we get that the gradient of log $\pi$ equals the gradient of $\pi$ over $\pi$
$\ln \pi(a|s,\theta) = \displaystyle \frac{\nabla \pi(a|s,\theta)}{\pi(a|s,\theta)}$

So this update is equivalent to what we started with.

Why do we do this ?

One reason is that
it is actually simpler to compute the gradient of the logarithm of certain distribution !

The other less important reason is that
it let's us write this gradient more compactly.

In the end,
it is just a mathematical thrick.
so don't let it distract you from the underlying algorithm.

We now have something that looks like many of the learning rules used in this course. (???)

We adjust the parameter $\theta$ proportionally to a stochastic gradient of the objective.

We use a step size parameter $\alpha$ to control the magnitude of the step in that direction.
So $\alpha$ has the same role it always has.

We now have a nice clean update rule to learn the policy parameters !

Computing the Update

The last thing to talk about is
how to actually compute the stochastic gradient for a given state and action.

We just need two components,

the gradient of the policy
$\nabla \ln \pi(A_t|S_t,\theta_t)$
an estimate of the differential values
$q_{\pi}(S_t,A_t)$

The first is easy,
we know the policy and this parameterization,
and so can compute it's gradient.

The second,
The action value can be approximated in a variety of ways.
For example,
we could use a TD algorithm that learns differential action-values.

In an upcoming video,
we will go through one particular choice in detail,
as well as how to compute the gradient for specific policy parameterization.

Summary

we derive a policy gradient learning rule for the average reward setting

In the next video,
we will see how to use this rule.

$\cdot$ Actor-Critic Algorithm

Describe the actor-critic algorithm
for control with function approximation
for continuing tasks

Do we have to choose between directly learning the policy parameters and learning a value function ?

No !
Even within policy gradient methods, valaue-learning methods like TD still have an important role to play.

In this setup,
the parameterized policy plays the role of an actor,
while the value function plays the role of a critic,
evaluating the actions selected by the actor.

These so called actor-critic methods
were some of the earliest TD-based methods introduced in Reinforcement Learning.

Approximating the Action-Value in the Policy Update

We finished off the last video with this expression for the policy gradient learning rule.
$\theta_{t+1} \quad = \quad \theta_t + \alpha \nabla \ln \pi (A_t | S_t,\theta_t) q_{\pi}(S_t,A_t)$

But, we don't have access to $q_{\pi}$,
so we'll have to approximate it !

We can do the usual TD thing,
the one-step bootstrap return.

That is the differential reward plus the value of the next state.
$R_{t+1} - \bar{R} + \hat{v}(S_{t+1},W)$

Critic part and Actor part of the actor-critic algorithm

Critic part of the actor-critic algorithm

As usual,
the parameterized function $\hat{v}(s,W)$ is learned estimate of the value function.
In this case,
$\hat{v}(s,W)$ is the differential value function.

This is the critic part of the actor-critic algorithm.

The critic provides immediate feedback.

To train the critic,
we can use any state-value learning algorithm.
We will use the average reward version of semi-gradient TD(0).

Actor part of the actor-critic algorithm

The parameterized policy is the actor !

It uses the policy gradient updates shown here.

Subtracting the Current State's Value Estimate

policy gradient update without baseline	policy gradient update with baseline

We could use this form of the update,
but there is one last thing we can do to improve the algorithm.

We can subtract off what is called a baseline !
$\hat{v}(S_t,W)$ is the baseline in this case.

Instead of using the one-step value estimate alone,
we can subtract the value estimate for the state $S_t$ to get the update that looks like this.

Notice that
this expression is equal to the TD error $\delta$ !

The expected value of this update is the same as the previous one.
Why is this ?

Adding a Baseline

Let's take the expectation of the update conditioned on a particular state $S_t$ at time $t$.

Taking the expectation of the sum is the same as the sum of the expectations.

We can use this
to seperate out the expectation of our original term
from the expectation which involves the subtracted value function.

It turns out the expectation of the second term is $0$.
So we can add this baseline to the update without changing the expectation of the update.

You can varify this for yourself.
To start, write the expectation as a sum of our(?) actions,
and pull the $\hat{v}()$ term out of the sum.
( we leave this as an exercise )

So why do we add this baseline if the update is the same in expectation ?

Subtracting this baseline tends to reduce the variance of the update
which results in faster learning !

How the Actor and the Critic Interact

This update makes sense intuitively.

After we execute an action,
we use the TD error to decide how good the action was compared to the average for that state.

If the TD error is positive,
then it means the selected action resulted in a higher value than expected.
Taking that action more often should improve our policy.

That is exactly what this update does.

It changes the policy parameters($\theta$?)
to increase the probability of actions that were better than expected according to the critic.

Correspondingly,
if the critic is disappointed and the TD error is negative,
then the probability of the action is decreased.

The actor and the critic learn at the same time constantly interacting.

The actor is continually changing the policy to exceed the critic's expectation, and
the critic is constantly updating it's value function to evaluate the actor's changing policy.

Actor-Critic algorithm

With the policy update in place,
we're ready to go through the full algorithm for average reward actor-critic.

To start,
we specify the policy parameterization and the value function parameterization.

Input : a differentiable policy parameterization $\pi(a|s,\theta)$
Input : a differentialble state-value function parameterization $\hat{v}(s,W)$

Fot example,
we might use Tile Coding to construct the approximate value function and a Soft-max policy parameterization.

We will need to maintain an estimate of the average reward $R$ just like we did in the differential SARSA algorithm.
We initialize this to $0$

Initialize $\bar{R} \in \mathbb{R}$ to $0$

We can initialize the weights and the policy parameters however we like.

Initialize state-value weights $W \in \mathbb{R}^d$ and policy parameter $\theta \in \mathbb{R}^{d'} \quad$ (e.g. to $0$)

We initialize the step size parameters for the value estimate, the policy, and the average reward
and they could all be different.

Algorithm parameters : $\alpha^W > 0, \;\; \alpha^\theta > 0, \;\; \alpha^{\bar{R}} > 0$

We get thie initial state from the environment, and then begin acting and learning.

Initialize $S \in \mathbb{E}$
Loop forever (for each time step)

On each time step,

we choose the action according to our policy
and recieve the next state and reward from the environment.

Loop forever (for each time step)
$\quad A \sim \pi(\; \cdot \; | S,\theta)$
$\quad$ Take the action $A$, $\quad$ observe $S', \; R$

Using this information,
we compute the differential TD error
and update our running estimate of the average reward.

$\delta \quad \leftarrow \quad R - \bar{R} + \hat{v}(S',W) - \hat{v}(S,W)$
$\bar{R} \quad \leftarrow \quad \bar{R} + \alpha^{\bar{R}} \delta$

We update the value function weights using the TD update.

$W \quad \leftarrow \quad W + \alpha^W \delta \nabla \hat{v}(S,W)$

Finally,
we update the policy parameters using our policy gradient update !

$\theta \quad \leftarrow \quad \theta + \alpha^\theta \delta \nabla \ln \pi(A \; |S,\theta)$

$S \quad \leftarrow \quad S;$

That's it !

This algorithm is designed for continuing tasks.
So we can run it indefinitely and continue to improve the policy forever.

Summary

It is useful to learn a value function to estimate the gradient for the policy parameters
The actor-critic algorithm implements this idea,
with a critic that learns a value function for the actor

3. Week_4-2_Policy Gradient for Continuing Tasks

doutoury2 — Mon, 21 Jun 2021 05:11:35 +0900

3. Week_4-2_Policy Gradient for Continuing Tasks

2. Policy Gradient for Continuing Tasks

The Objective for Learning Policies
The Policy Gradient Theorem

$\cdot$ The Objective for Learning Policies

Describe the objective for policy gradient algorithms

Now that we've introduced the idea of parameterizing policies directly,
we're ready to talk about how we can learn to improve a parameterized policy.

Just like with action-value based methods,
The basic idea will be to specify an objective,
and figure out how to estimate the gradient of that objective from an agent's experience.

The Goal of Reinforcement Learning $\quad : \;$ Maximizing Reward in the Long Run

Formulating an obejective for learninng a parameterized policy
is in some sense more straightforward than it was for action-value based methods !

We've said many times that the ultimate goal of Reinforcement Learning
is to learn a policy that obtain as much reward as possible in the long run.

It turns out that when we parameterize our policy directly,
we can also use this goal directly as the learning objective.

Formalizing the Goal as an Objective

Throughout this specialization,
we've introduced a few different interpretations of
what it means to obtain as much reward as possible in the long run.

For the episodic case,

we can use the undiscounted retrun,
which is the sum over a whole episode.

For the continuing case,

we introduced the discounted return,
which places more emphasis on immediate reward in order to keep the sum finite.

Most recently, we introduced the average reward formulation.
Here, we maximize the long-term average of the reward.
The appropriate return here is the sum of the differences between the immediate reward and it's average.

Because the lont-trerm average subtracted, this sum is finite even without discounting !

Here wa have three potential problem formulation to consider.
We're going to restrict our attention the continuing setting with average reward.

Remember what out aim is here
to find a way to directly optimized parameters of a policy.

The first step is to write the average reward objective in a form that we can optimize.

The Average Reward Objective

Previously, we estimated the average reward to learn action-values !
( in an average reward variant of SARSA )

Now our aim is to learn a policy
that directly optimized average reward.

Toward this aim, let's start by writhing the average reward under a policy in a more useful form.

We can write the average reward $r(\pi)$ achieved by a particular policy $\pi$ like this.
$r(\pi) \quad = \quad \displaystyle \sum_s \mu(s) \sum_a \pi(a|s,\theta) \sum_{s',r} p(s',r|s,a)r$

Undertand the Average Reward Objective (?)

Let's break this formula down starting from the inner sum and moving out.

This inner sum gives the expected reward if we starts state in $s$ and take action $a$.
$\rightarrow \quad \mathbb{E} \big[ R_t|S_t=s,A_t=a \big]$

This is simply the sum over all reward $r$ we might receive weighted by their probability $p(s',r|s,a)$ from $s$ and $a$.

We sum over next states $s'$ to get marginal probabilities (주변확률) over the reward.

The next level of the summation gives us the expected reward under the policy $\pi$ from a particular state $s$.
$\rightarrow \quad \mathbb{E}_{\pi} \big[ R_t | S_t = s \big]$

This is over all possible actions $a$ weighted by the probability $\pi(a|s,\theta)$ under (policy?) $\pi$.

Finally, we get the overall average reward
by considering the fraction of time we spend in state $s$ under policy $\pi$.
$\rightarrow \quad \mathbb{E} \big[ R_t \big]$

The distribution $\mu$ provides these probabilities $\mu(s)$.
The expected reward across states $\mathbb{E}_{\pi}(R_t)$ is a sum over $s$ of the expected reward in a state $\mathbb{E}_{\pi} \big[ R_t | S_t = s \big]$ weighted by $\mu(s)$.

$r(\pi)$ is our average reward learning objective !

Optimizing The Average Reward Objective

Our goal of policy optimization
will be to find a policy which maximizes the average reward $r(\pi)$ !

Basic approach

to estimate the gradient of the objective $\nabla r(\pi)$ with respect to the policy parameters,
ans adjust the paramters based on the estimate !

$\rightarrow \quad$ Policy Gradient method
$\qquad$ The class of methods they use this idea are often referred to as policy gradient methods.

Indirect learning policy in GPI v.s. Direct learning policy in PG

Up until now,
we've used a very different approach.
We use the Generalized Policy Iteration framework (GPI) to learn approximate action-values.

Then we use these approximate values indirectly to infer a good policy.

Now,
we're interested in learning policies directly !

There's also a superficial difference.

In GPI
minimizing the Mean Squared Error for learning

In GP
maximizing an objective for learning

That means
we will want to move in the direction of the gradient !
( rather than the negative gradient )

The Challenge of Policy Gradient Methods

However,
There're are few challenges in computing this gradient !

The main difficulty
is that modifying our policy changes the distribution $mu$ !
$\rightarrow \quad$ $\mu(s)$ is depends on $\theta$ !

This contrast the value function approximation
where we minimized Mean Squared value Error under a particular policy.

There
the distribution $\mu$ was fixed.
It does not change as the weights and the parameterized value function change.
$\rightarrow \quad \mu(s)$ is independent of $W$

We were therefore able to estimate gradients for state drawn from $\mu$ by simply following the policy.

the Policy Gradient Theorem $\quad : \;$ Solution for learning policy

This is less straightforward,
a $\mu$ itself depends on the policy we are optimizing.

Luckily,
there's an excellent theoratical answer to this challenge,
called the Policy Gradient Theorem !

( we'll talk more about that next )

Summary

We can use the averege reward as an objective for policy optimization

In upcoming lectures,
we will disucss how to actually optimize this objective from sampled experience.

$\cdot$ The Policy Gradient Theorem

Describe the result of the policy gradient theorem
Understand athe importance of the policy gradient theorem

We just discussed an objective for policy optimization.
The next step is to figure out how the agent can optimize it based on it's own experience.

In this video,
we will describe the Policy Gradient theorem.

This is a key theoretical result.
It allows us to write the gradient of the average reward so that it is easier to estimate from experience !

Gradient Ascent

To optimize the Mean Squared value Error,
we used methods based on Stochastic Gradient Descent.

We estimate the negative of the gradient of our objective,
and adjust the weights of the value function in that direction.

Policy Gradient methods use a similar approach,
but with the average reward objective and the policy parameters $\theta$.

we want to maximize the average reward rather than minimizing it !
This means we do Gradient Ascent, and move in the direction of the positive of the gradient.

Remember that
a simple recipe for solving a problem is

to first specify an objective
then estimate the gradient of that objective,
and finally, adjust the weights in that direction.

Step 1 is done.
Now the next step is to estimate the gradient of our objective.

The Gradient of the Objective

Recall
the Average Reward Objective.

Let's compute the gradient.

We can apply the product rule of calculus to the objective to yield two terms.

This is pretty complex.
So let's look at it a bit more closely.

The first term involves the gradient of the stationary distribution over states.
$\nabla \mu(s)$

Unfortunately,
the gradient of $\mu$ is not straightforward to estimate.

The stationary distribution $\mu$ depends on a long-term interaction between the policy and the environment.

The Policy Gradient Theorem

Luckily,
the Policy Gradient Theorem gives us a simpler expression for this gradient.

Here we show the results of the theorem.
It's worth walking through this expression.

In the inner sum,
we have the gradient of the policy times the action-value function.

Let's try to understand this term a little better.

Understanding $\displaystyle \sum_a \nabla \pi(a|s,\theta) q_{\pi}(s,a)$

The gradient of the policy $\pi$ is easy to compute
as long as our policy parameterization is differentiavle !

The gradient of the policy
tells you how to adjust your parameters to increase the probability of a certain action.

Example $\quad$ for understanding the policy gradient

Consider the simple grid world shown here.

As usual, the agent cam move up, donw, left, or right.
For simplicity, let's assume the policy is controlled by just two parameters ($\theta_1$ and $\theta_2$).

The parameters are curently set to the point marked here.
The arrows indicate the agent's action probabilities in a particular state with the current parameter settings.

the policy gradient for Up action case

The gradient for the up action might look something like this on the plot.

The gradient tells us
how to change the policy parameters to make that action more likely to be selected in the given state !

By moving the paramters in the direction of the gradient,
we increase the probability for the up action.

This necessarily means decrease in the probability of some of the other actions !

the policy gradient for Left action case

Different actions have different gradients.

the gradient of the left action probability may look like this.

Moving the parameters in that direction will increase the probability of the left action,
and decrease the probability of some of the other actions.

This might all sound a bit abstract.
We wll show concrete examples of computing this gradient in the coming lectures.

the policy gradient for overall action

Now,
let's bring some reward into the situation
and think about what this whole term means.

This is a sum over the gradients of each action probability weighted by the value of the associated action.

Imagine,
we've added a rewarding state in the bottom right.

The up and left actions
move away from rewarding state, and should have negative value.
The down and right actions
move toward it the rewarding state, and should have positive value.

The precise values will depend on the current policy.
but, let's say they look something like this.

The weighted sum
gives a direction to move the parameters that decreases the pro bability of moving up or left since their value is negative,
while increasing the probability of moving down or right since their value is positive.

That direction might look something like this on the plot.

Average Reward expression with the policy gradient

The gradient $\nabla r(\pi)$ expression given by the Policy Gradient Theorem
takes this expression and sums that over each state.

This gives the direction to move the policy parameters to most rapidly increase the overall average reward !

The Policy Gradient Theorem

We now have a simple expression for the gradient !

Importantly,
this expression does not contain the gradient of the state distribution $\mu$ !
( which is challenging to estimate ! )

The proof of the policy gradient theorem
can be found in the course textbook.

As we will see in the next lecture,
this gradient is straightforward to estimate !

That means we will be able to build an incremental policy gradient algorithm using an agent's experience !

Summary

The policy gradient theorem gives an expression for the gradient of the average reward
Understand the terms in this gradient

3. Week_4-1_Learning Parameterized Policies

doutoury2 — Mon, 21 Jun 2021 05:10:53 +0900

3. Week_4-1_Learning Parameterized Policies

Week_4

INDEX

Learning Parameterized Policies
- Learning Policies Directly
- Advantages of Policy Parameterization
Policy Gradient for Contunuing Tasks
- The Objective for Learning Policies
- The Policy Gradient Theorem
Actor-Critic for Continuing Tasks
- Estimating the Policy Gradient
- Actor-Critic Algorithm
Policy Parameterizations
- Actor-Critic with Softmax Policies
- Demonstration with Actor-Critic
- Gaussian Policies for Continuous Actions
- Week 4 Summary
Course Wrap-up
- Course 4 Preview

1. Learning Parameterized Policies

Learning Policies Directly
Advantages of Policy Parameterization

$\cdot$ Learning Policies Directly

Understand how to define policies as parameterized functions
Define one class of parameterized policies based on the softmax function

So far,
all the methods we've looked at for learning good policies estimate action values.
Every control algorithm we've studied was built on the framework of generalized policy iteration.

In this module,
we'll explore a new class of methods
where the policies are parameterized directly.

An Example of a Policy without Action-values

Let's think about
" what it means to specify a policy directly ".

We'll do this in Mountain Car example.

Previously,
we used $\epsilon - greedy$ to convert approximate action values into a policy.
But we can also consider policy which maps states directly to actions without first computing action values.

For example in mountain car,
we can define such a policy.

Simply choose accelerate right, when the velocity is positive,
$\pi(\text{Right}|s) = 1$
And otherwise choose the accelerate left action.
$\pi(\text{Left}|s) = 1$

Put in other way,
this policy accelerates in whatever direction we are already moving.

In fact,
this simple energy pumping policy is close to optimal.
This policy does not make use of action values at all.

But this was just an example to stimulate your intuitions
and to show you that we don't need action-values to construct policies.

We're not going to specify policies by hand.
Rather, we will learn them ( policies ) !

Parameterizing Policies Directly

We can use the language of function approximation to both represent and learn policies directly !

We'll use the greek letter $\theta$ for the policies parameter vector.
This distinguishes it from the parameters $W$ for the approximate value function.

We use the notation $\pi(a|s,\theta)$, to denote the parameterized policy.
For a given input state $s$ and action $a$,
the parameterized policy function will output the probability of taking that action in that state.

This mapping will be controlled by the parameters $\theta$.

Constraints on the Policy Parameterization

the parameterized function has to generate a valid policy.
This means, it has to generate a valid probability distribution over actions for every state.

Specifically,
the probabilities selecting an action must be greater than or equal to $0$.
And for each state, the sum of the probabilities over all actions must be $1$.

It requires some thought to satisfy these conditions for a parameterized function.

For example,
this means we can not use a linear function directly !
( like we did with value function approximation )
There is no easy way to guarantee that a linear function will sum to $1$.

Instead,
we will need to restrict the class of functions we can use to construct policies !

The Softmax Policy Parameterization

Let's consider a simple but effective way to satisfy these conditions
called the Softmax policy.

Here is the definition of a Softmax Policy.

The function $h(s,a,\theta)$ shown here
is called the action preference.

A higher preference for a particular action in a state
means that the action is more likely to be selected.

The action preference $h()$
is a function of the state $s$ and action $a$, as well as a parameter vector $\theta$.

Computing the probability of selecting an action with the softmax $\pi(a|s,\theta)$ is simple.
We take the action preference, exponentiate it, and then divide by the sum over all the actions for the same thing.
$\pi(a|s,\theta) \quad \doteq \quad \displaystyle \frac{e^{h(s,a,\theta)}}{\displaystyle \sum_{b \in A} e^{h(s,b,\theta)}}$

The exponential function $e^x$
guarantees the probability is positive for each action !

The denominator $\sum$
normalizes the ouput of each action such that the sum over actions is $1$ !

The action preference
can be parameterized in any way we like since the Softmax will enforce the constraints of a probability distribution.

For example,
the action preferences could be a linear function of the state-action features
or something more complex like the output of a Neural Network !

Example $\quad : \;$ Softmax Policies

Here's what we get when we pass a particular set of action preferences through the softmax.

The input preferences $h$ can be arbitrarily large, or even negative !

No matter how big the preference gets, the probability will never be greater than $1$.

If one preference is much larger than all the others,
the action probability would be close to one.

If one preference is very small,
the softmax policy will still select the action with non-zero probability.

For example,
if the preference is negative and the other is positive,
the negative action will still have non-zero probability.

Finally,
actions with similar preferences will be chosen with near equal probability under a softmax policy.

Action preferences are not action values

It's important to distinguish between action preferences and action values !

Preferences
it indicate how much the agent prefers each action. but they are not summaries of future reward.

Only the relative differences between preferences are impoartant. (?)
For example,
we could add $+100$ to all the preferences and it would not matter.

An epsilon greedy policy derived from action values can behave very diferently
than a softmax policy over action preferences !

In epsilon-greedy,
the action corresponding to the highest valued action is selected with high prabability.
The probability selecting all the other actions is quite small.
Actions with nearly the same but lower action value are selected with much lower probability.
Actions with very poor action values are still selected frequently due to the epsilon exploration step.

So even if the agent learns an action has terrible consequences,
it will continue to select that action much more frequently
than it would under the softmax policy.

Summary

We can parameterize policies directly
We need to define parameterizations that produce valid probability distributions
You should understand softmax policy parameterization

$\cdot$ Advantages of Policy Parameterization

Understand some of the advantages of using parameterized policies

You now know it's possible to learn parameterized policies directly.
This means we can have more flexibility in how we solve our problems.

Now, we consider learning approximate values and learnig approximate policies.

" Is it really helpful directly learn policies ? "

The Flexibility of Stochastic Policies

There are many advantages to directly learning the parameters of a policy.

First,
the agent can make it's policy more greedy over time automomously.

Why would we want that ?
In the beginning, the agent's estimates are not that accurate.
So we would want the agent to explore a lot.

As the estimates become more accurate, the agent should become more and more greedy.

Recall the Epsilon greedy policy we used before.

The epsilon step chooses a random action to ensure continual exploration.
However, the epsilon probability puts a cap on how good the resulting policy can be.

We could, of course, switch to a greedy policy
when we think the agent has explored adequately.

But we want our agents to be autonomous.
We don't want them to rely on us to decide when exploration is done.

We can avoid this issue with parameterized policies !

The policy can start off(?) stochastic to guarantee expiration.

Then,
as learning progresses, the policy can natually converge towrads a deterministic greedy policy.

A softmax policy can adequately approximate a deterministic policy by making one action preference very large.

Stochastic policies might be better than Deterministic policies under function approximation

In the Tabular setting,
we learn there's always a deterministic optimal policy.

In function approximation,
we may not be able to represent this deterministic policy.

In the function approximation,
Instead, the optimal approximate policy might be a stochstic policy !

This suggests it might be useful to learn stochastic policies.

Stochastic policies can be useful with function approximation

We can see why this is true by considering an example.

Imagine the agent is in a corridor.
It always starts in the far left state.

In the left and right states,
the actions left and right have their usual consequences.
In the middel state, however,
the left and right actions are switched.
( Moving left will take the agent right state. Moving right takes the agent to the left state )

The reward is $-1$ in every step
the task is episodic.

Imagine the functions approximation treats all these states as the same.
All three states share the same approximate value.

Deterministic policy

If we choose to limit ourselves to deterministic policies,
we would have no choice but to pick the same action in all the states.

This would give us just two choices,
always move left or always move right.

If we always move left,
we will never leave the start state.
So the expected return for that policy is negative infinity $-\infty$.
$v_{\text{always left}}(S) = -\infty$

If we always move right,
we will reach the middle state and them mover back to the start state, and continue this forever.
So the expected return is also negative infinity.
$v_{\text{always right}}(S) = -\infty$

Stochastic policy

If we are allowed to choose actions stochastically,
we can do much better ( than Deterministic policy ? )

We may get stuck for a while,
but as long as each action has a non-zero probability, we will eventually reach the terminal state.

In fact,
the best policy under this function approximation is a little particular.
Choose the right action 59% of the time and the left action the rest of the time.

This policy achieves an expected return around $-11.6$.
Clearly better than negative infinity $-\infty$ !
$v_{\pi_{\theta}} = -11.6$

This example may seem a bit contrived, but similar situation can arise in the real world !

For example,
in our own room, a robot gets s tuck in the corner of the room
because it had a deterministic policy and a limited function approximator.

Sometimes it is easier to learn a good policy

Sometimes
the policy is more simple than the value function !

Remember the mountain car problem.

We spent a lot of time designing a value-based agent to laern an optimal policy.
However, in this problem, the energy pumping policy we saw previously is nearly optimal.

The agent simply select it's action in agreement with the current velocity.
If the velocity is negative, then the agent takes the accelerate left action.
If the velocity is positive, then the agent take the accelerate right action.

This policy allows the agent to quickly escape from the valley.
As you can see, this is quite a simple policy.

On the other hand,
The value function is quite complex !

Summary

You should now understand that parameterized stochastic policies are useful !
because ...

They can autonomously decrease exploration over time
They can avoid failures due to deterministic policies with limited function approximation
Sometimes the policy is less complicated than the value function

In the upcoming videos,
we'll see there is not a strict distinction between action value methods and policy based methods !

In fact,
we can use action values to make it easier to learn a parameterized policy !

3. Week_3-4_Week 3 Review

doutoury2 — Mon, 21 Jun 2021 05:10:16 +0900

3. Week_3-4_Week 3 Review

Week 3 Review

This week, we talked about how to do control when using fucntion approximation.
Let's go over the main ideas.

Summary

That's all for this week.

We extended our Tabular control algoorithms to function approximation,
discussed how exploration changes,
and introduce a new way to think about the control problem.

Next week,
we'll talk all about how to do Reinforcement Learning without learning the value function !

Summary

That's all for this week.

We extended our Tabular control algoorithms to function approximation,
discussed how exploration changes,
and introduce a new way to think about the control problem.

Next week,
we'll talk all about how to do Reinforcement Learning without learning the value function !

Summary

That's all for this week.

We extended our Tabular control algoorithms to function approximation,
discussed how exploration changes,
and introduce a new way to think about the control problem.

Next week,
we'll talk all about how to do Reinforcement Learning without learning the value function !

Respresenting actions

1	2

First, we showed you how to estimate action values with function approximation.

If the action space is discrete,
it's probably easiest to stack the state fearues.

If the action space is continuous or you want to generalize over actions,
The action can be passed as an input like any other state variable.

TD Control with Approximation

Let's get some context abouth the next part of the module by looking at the algorithm map.

Function appoximation puts us on the left side of the map.
the focus of the first lecture was on the control algorithms in the bottom left corner,
SARSA / Expected SARSA / Q-learning

These are all extensions of the Tabular control algorithms we covered in Course 2.

The only difference between these algorithms and their tabular counterparts are the update equations.
The ipdates are all adapted for function approximation in the same way, using the gradient to update the weights.

We also saw how episodic SARSA could be used to solve the mountain car problem.
In this case, the larger step size is $0.5$ was able to learn more quickly.

Exploration with Approximation

Next,
we talked about exploration.

Optimistic initialization
Optimistic initialization can be used with some structured featrue representations like Tile-Coding.

But in general,
it's not clear how to optimistically initialize values with non-linear function approximators like neural networks.
And it might not behve as expected.

For example,
the optimism may fade too quickly.

$\epsilon - greedy$
Epsilon-greedy can be used regardless of the function approximator.

Average Reward

Finally,
we talked about a new way to think about the continuing control problem.

Instead of maximizing the discounted return from the current state,
we can think about maximizing the average reward that a policy receives overtime.

We defined differential returns and differential values.

These enable the agent to assess the relative value of actions in the average reward setting !

Differential semi-gradient SARSA

Finally,
we introduced differential semi-gradient SARSA,
that approximates differential values to learn policies.

Differential SARSA is also in the left half of the algorithm map.

But unlike the algorithms we covered earlier in the week,
it uses the average reward framework.

Summary

That's all for this week.

We extended our Tabular control algoorithms to function approximation,
discussed how exploration changes,
and introduce a new way to think about the control problem.

Next week,
we'll talk all about how to do Reinforcement Learning without learning the value function !