Christian Rivera

Correlated Q-Learning in Multi-Agent Games

Sun, 26 Jul 2020 00:00:00 GMT

Abstract

The purpose of this report is to re-implement one of the experiments found in the research paper, “Correlated Q-Learning” (Greenwald & Hall, 2003). The experiment in question is to implement four multi-agent Q-learning algorithms and compare their learning speeds in a zero-sum game referred to as “Soccer”.

Introduction and Background

Multi-agent Q-learning is an area within Reinforcement Learning to design algorithms that can learn optimal policies in an environment containing other competitors. While Markov Decision Processes (MDP) can be used to describe single-player, repeated games, a more generalized framework, a Markov Game, is necessary to describe a multi-agent game (Littman, 1994). These types of environments require a learning agent that can accounts for the possible actions of its opponents when determining the optimal move. As such, Q-Learning in a Markov Game is described as defining a value given by a state paired with an action vector:

$Q_i(s, \vec a) = (1 - \alpha)R_i(s, \vec a) + \alpha \sum_{s'}P[s'|s,\vec a]V_i(s')$

While this Markov Game framework is a strong starting point, in that it recognizes the impact of other agents’ decisions on the Q-Learning agent, more work needs to be done in defining a proper value function $V_i(s')$ that takes into account the true goals of the other agents. A proper value function enables the Q-learning agent to correctly model the environment and other agents, have its Q-values converge, and therefore find an optimal policy. Defining a proper value function is a difficult proposition with many proposed value functions only working in specific game settings. For example, in two-player adversarial, constant-sum games, a Q-learning agent can apply a minimax algorithm to its value function in order to find an equilibrium policy. This type of Q-Learning is known as Foe-Q (or alternatively Minimax-Q) (Littman, Friend-or-Foe Q-learning in General-Sum Games, 2001). In contrast, Foe-Q will not work in games with collaborating agents and a better strategy would be to implement a maximax algorithm also known as Friend-Q (Littman, Friend-or-Foe Q-learning in General-Sum Games, 2001). Both Friend-Q and Foe-Q only work in very specific circumstances and are, in effect, giving the Q-learning agent unearned information about an environment and its actors. Greenwald, and Hall looked to solve this problem with a more generalized Q-learning approach which they referred to as Correlated Equilibrium Q-Learning (CE-Q).

$V_i(s) = \text{Nash}_i(s,Q_1,Q_2,...,Q_n)$

What is interesting about Foe-Q and its use of the minimax algorithm is that you can frame the Q-Learning space as a payoff grid and treat the environment like a game theory problem. This, in turn, enables the agent to use a Nash equilibrium within this payoff grid as the output of the value function. Even better, the Nash equilibria can be discovered by treating the payoff grid as the constraint parameters for a linear programming optimization problem. A Nash equilibrium is a vector of independent probability distributions that the Q-Learning agent can use to pick an optimal action. This works if it assumes that opponent agents are only trying to minimize the Q-learning agents reward which works only in zero sum games. In non-zero-sum games, the Nash equilibria fail in describing other agents’ actions where the other agents have multiple objectives: maximize their own scores and minimize the Q-learning agent’s score. The authors of “Correlated Q-Learning” discovered a way to implement the concept of considering multi-objective opponents by considering the joint probabilities of each state for the opponents and Q-learning agent. These joint probability constraints can be added to the original payoff grid constraints and this new correlated equilibrium can still be solved as a linear programming problem. CE-Q thus enables a far more nuanced understanding by a Q-learning agent of other agents’ preferred actions beyond just the extremes of Foe-Q and Friend-Q and opens other potential value functions. Because of this, CE-Q acts as a generalization of both Foe-Q and Friend-Q while also enabling middle ground objective functions. Greenwald, Hall, and Serrano propose 4 such functions and they experiment on one of them which they referred to as Utilitarian Correlated Equilibrium Q Learning (uCE-Q).

Describing the Game and Q-Learning Experiments

In the paper, “Correlated Q-Learning”, the authors proceed to show CE-Q performance compared to regular Q-Learning, Foe-Q, and Friend-Q in several games including one referred to as “Soccer”. This game consists of replicating a “1v1” faceoff scenario in a soccer game. “Soccer” consists of a 2x4 grid space occupied by two players, A and B. At any given point, one of the players has possession of the soccer ball and attempts to “score” by entering into the opposing player’s “goal” column (columns zero and three for A and B respectively). This provides a reward of +100 however if the player enters its own goal while in possession of the ball, then the player’s reward is -100. Both scenarios result in the end of the game, with any other state on the board providing a reward of zero. Thus, “Soccer” is a zero-sum game. Each player at any point can perform 5 actions (go north, south, east or west, or stay put). Each player submits their action at the same time and the game environment decides at random who moves first thus making the environment stochastic. If both players attempt to enter the same state and the current possessor of the “soccer ball” goes second, the possession changes to the other player.

Soccer Game Environment

To compare the learning efficacy of each Q-Learning algorithm, the authors kept track of the error rate: the absolute difference between the Q value of a given state when the opponent stays put and the previous Q value of that state. Each algorithm engaged in the game against a random action opponent until it has visited the given state 100,000 times. The collected error rate values were then charted to observe whether the learning agent’s Q-values converged or not and how quickly they converged.

$\text{ERR}^t_i = |Q_i^t(s,\vec a) - Q_i^{t-1}(s, \vec a)|$

Assumptions Taken in Replicating the Experiments

Unfortunately, the paper, “Correlated Q-Learning”, left out some information regarding how to simulate the game, some edge case scenario rules of the game, how to measure the error rate, and hyperparameters of the game. Because of this, I made the following assumptions:

The opponent was not specified, so I decided that the opponent, player A, is a uniform random action opponent
If the player who moved first decided to stay put and the second player attempted to move into the space, neither player would end up moving although, the player who attempted to move would now have possession of the ball
If a player takes possession of the ball but while in his own goal column, that is still considered an own goal and the game is over.
Every game starts with the same scenario with the agent as player B in position [0,1] with possession of the ball and player A in position [0,2].
The paper does not specify exactly which state is used to log the Q-value error function but it infers that the state used is the starting scenario state so that is the state I used.
I decided that the previous Q-value state was the value from the previous time step and not the time when that state was previously visited.
While the paper specifies some hyperparameters (γ=0.9, α →0.001 for all algorithms and ε → 0.001 for the basic Q-Learning algorithm, it does not specify the starting α or decay factor which I specified as 0.5 and 0.9999, respectively.

Experiment 1: Q-Learning Agent

The first experiment in the “Soccer” environment provides a baseline by seeing how a basic Q-Learning agent performs in a multi-agent environment. Basic Q-learning does not take into consideration the actions of the other agents such that the state space is just every combination of the 112 states and 5 actions of the agent. Since the Q-Learning agent is oblivious to its opponents, each Q-value is unstable since it does not consider how the opponent’s actions impacted the value of the state. As such the Q-values are constantly being updated and never converge because they are trying to simultaneously value 25 different outcomes as just 5 outcomes for any given state. We can see this failure to converge both in the original paper as well as in the replicated results.

Experiment 2: Friend-Q Agent

$V_i(s) = \max_{\vec a \in A(s)} Q_i(s, \vec a)$

The Friend-Q agent assumes that all other agents in the game are working in tandem with the Friend-Q agent to maximize its own rewards. This is a naive and optimistic approach where it assumes that both agent and opponent will either stay put or move east so that the player can score or benefit from the opponent’s own goal. As such, the Q value for that state converges quite quickly which showed up in both the origins and my own experiments. Since this is a zero-sum game, Friend-Q ends being nothing more than an expanded basic Q-Learning that considers the extra dimension created by the opposing player.

Experiment 3: Foe-Q Agent

$V_i(s) = \max_{\sigma_1 \in \Sigma_1(s)} \min_{a_2 \in A_2(s)} Q_i(s, \sigma_1, a_2)$

The Foe-Q algorithm has the Q-learning agent assume that the other agents will try to minimize its rewards which fits the description of this zero-sum game. We see that the Foe-Q algorithm converges around 60,000 simulations in both the original and replicated experiment which suggests that Foe-Q is a viable Q-Learning method for zero sum games. The Q-values do not converge as quickly for Foe-Q as this agent must grapple with the stochastic nature of its opponent and the randomness of who initiates a move first.

Experiment 4: Utilitarian Correlated Equilibrium Agent

$\sigma \in \text{argmax}_{\sigma ]in CE} \sum_{i \in I} \sum_{\vec a \in A} \sigma(\vec a)Q_i(s,\vec a)$

As mentioned earlier, Correlated Equilibrium Q-learning provides more nuanced objective functions that enables the agent to consider opponents with multiple objectives. Utilitarian CE-Q is one of those functions where the action the agent decides is summing up all the probability adjusted potential rewards for each action and choosing the action with the highest value. As the results from both sets of experiments show, the uCE-Q agent also converges around 60,000 simulations exactly like the Foe-Q agent and it converges to the same optimal policy as the Foe-Q agent. This is not a coincidence as in a zero-sum game, the objective function of the uCE-Q is the same as the minimax objective function of the Foe-Q agent. This demonstration shows that CE-Q is a viable alternate as well as a strong generalization of Foe-Q and Friend-Q.

Conclusion

Through these replicated Q-learning experiments, we observed multiple phenomena in Reinforcement learning. We saw how a Markov Decision Process is not suitable for learning multi-agent games as the basic Q-Learning agent could not converge to a single policy. This prompted an understanding of a more generalized Reinforcement Learning framework, Markov Games, that can handle multi-agent environments. We observed Friend-Q and Foe-Q Markov Game implementations that converge in such environments before showing how Markov Game Q Learning can be generalized even further through the framework of Correlated Equilibrium Q learning. CE-Q both solved the zero-sum “Soccer” game in the same manner as the Foe-Q algorithm while maintaining an ability to be applied to other constant-sum games and collaborative games where Foe-Q would fail. Thus, CE-Q, as a generalization of Markov Games, opens new possibilities for Reinforcement Learning agents to better learn in unknown environments.

Bibliography

Greenwald, A., & Hall, K. (2003). Correlated Q-Learning. ICML.
Littman, M. L. (1994). Markov Games as a Framework for Multi-Agent Reinforcement Learning. ICML.
Littman, M. L. (2001). Friend-or-Foe Q-learning in General-Sum Games. ICML.

Deep Q-Network in Reinforcement Learning

Thu, 02 Jul 2020 00:00:00 GMT

Abstract

The purpose of this paper is to experiment with a Deep Q-Network (DQN) reinforcement learning (RL) algorithm in order to find the optimal policy for a given environment. For off-policy RL, agents use Q-Learning in order to create a policy that, for every state in an environment, a Q-learning algorithm will return the optimal action for that state. While Q-Learning is a powerful RL method, it has significant limitations. The first limitation is that Q-learning methods are viable only in environments with small, discrete state spaces. In near-infinite and/or continuous state spaces, a Q-learning matrix would be prohibitively hard to map. For example, Q-learning would require a prohibitive amount of time and memory to explore the entire $10^{46}$ sized state space for a game of Chess. We can overcome this hurdle by applying function approximations to generalize a Q-Learning space. In this paper we will be applying artificial neural networks (ANN) as the Q-Learning function approximator otherwise known as a Deep Q-Network.

Introduction and Background

This paper explores the implementation and hyperparameter tuning of a Deep Q-Network algorithm in order to create an agent that can solve the game “Lunar Lander”. In this game, an agent is presented with a virtual spaceship with the goal of landing the ship on the ground within a landing zone with the correct orientation (feet first). Landing the ship correctly provides positive rewards while landing the ship away from the landing zone or in the wrong orientation creates negative rewards. The agent is considered to have learned the optimal policy if it has successfully landed the spaceship in the landing zone the majority of the time in a given one hundred episodes. This game includes a non-trivial state space where an RL agent receives information regarding six continuous values and two binary values. For a state space like this, function approximators are needed which and where this paper uses ANNs to solve the problem.

A function approximator replaces a Q-learning matrix with weights from a supervised learning model. The supervised learning model used in this paper is a two-layer neural network constructed using the Python package PyTorch. A neural network can be thought of as a collection of interconnected logistic regressions that enable models to identify non-linear relationships in the data. This would generally cause issues of non-convergence for an RL agent however Google DeepMind discovered a combination of two methods to solve the problem: “experience replay” and a delayed “iterative update”. With “experience replay”, an RL agent keeps a record of every $(S_t,A_t,R,S_{t+1})$ and will randomly sample batches of these records for retraining which minimizes the negative impact that action sequence correlations have on a neural network’s weights. The “iterative update” method is the function approximation equivalent of temporal difference (TD) learning in classical Q-Learning where the ANN model computes the loss function by comparing the predictions of an earlier version of the model’s weights with the estimates of its current weights.

For the experiments in this paper, I wanted to observe how changing the hyperparameters for “experience replay” and “iterative update” impacted the DQN agent’s ability to discover the optimal policy of the “Lunar Lander” game. The hyperparameters were judged by both how many episodes the model needed until it averaged a score of 200 or above over the last 100 episodes.

Experiment 1: Changing the batch size of the experience replay

Experience replay provides an RL agent several advantages: it enables the agent to reuse data it has already seen to increase efficiency and it removes undesirable weight changes caused by the correlation of consecutives steps in an episode. This correlation prevents convergence or can isolate an agent at a local optimum. For the experiment, I had the agent learn the environment with different experience replay batch sizes: [4, 8, 16, 32, 64, 128, 256, 512].

As the graphs above show, the agents with the larger batch sizes (128, 256, or 512) were able to learn the environment in roughly 1650 episodes while the smaller batch sizes needed 1900 or more episodes to acquire equivalent learning. These results support the findings from the Google Deepmind team that agents reusing states they already explored reduces the number of episodes needed and enables faster convergence. These larger batch sizes also seemed to limit overcorrection of the weights as the average rolling mean scores of the larger batch-sized agents during training were significantly less volatile than their peer agents with lower batch sizes.

Experiment 2: Changing the frequency of the experience replay

Another hyperparameter I experimented on for the DQN agent’s experience replay was how frequently the agent engaged with experience replay during training. I wanted to compare the number of actions an agent takes before using experience replay with how many episodes it takes the agent to learn the environment. For this experiment, I tested agents that activate experience replays for frequencies ranging from 1-12 actions.

This experiment also reinforces the value of experience replay as there clear relationship in the data showing that, as the frequency of experience replays increases, the number of episodes needed to train the agent decreases. Inserting experience replay between the actions of an agent does appear to minimize correlation impact of consecutive actions on an agent’s neural network weights. At the same time, increasing replay step frequency, unlike increasing batch size, does not minimize learning volatility. The faster learner had a replay step frequency of one but also had the highest learning volatility with sharp upticks in knowledge gains (episodes 800-900 above) but also bouts of flat or negative knowledge gains (episodes 1000-1250). We can further observe the lack of impact that changing the frequency replay steps has on an agent’s volatility in an environment by observing the test results for the different replay step agents. While the agent with a replay step frequency of 1 is the least volatile, it does not have appreciably better performance in the testing phase. This suggests that, while having a tight frequency of replay steps is preferable, it is less important of a hyperparameter than experience replay batch size when tuning a DQN RL agent.

Why experience replay makes sense in the context of the Lunar Lander

As the results so far have shown, with both high frequency replay steps and large batch sizes, experience replay-enhanced RL agents perform better in the Lunar Lander scenario. As mentioned previously, experience replay prevents overfitting caused by the correlation effect of consecutive steps in. In the Lunar Lander scenario, this overfitting manifests itself in poor results due to the fact that the “landing zone” is located in the middle of the screen and because the agent does not receive any information regarding acceleration. The agent is only given information regarding velocity and position which leads to over-correction caused by greedy immediate knowledge. For example, If the agent starship position was located on the far left corner of the screen, the agent would be rewarded for activating the left booster thus pushing it closer to center and landing zone. The agent would repeatedly activate the left booster to get more reward but, at a certain point, would have so much acceleration that the otherwise small left booster activation would launch the agent to the far right corner, missing the center, and receiving a large negative reward. An agent with experience replay would hold a diminished value for repeatedly activating the left booster and would then not overshoot the landing zone.

Experiment 3: Changing the learning rate for iterative update

A component of a neural network model is the optimization function it uses to update its weights during training. For the DQN agent, I went with using an Adam optimizer and tuned the learning rate hyperparameter in order to investigate its impact on the agent’s training success. A learning rate is a dampening effect on an ANN’s weight update such that the ANN does not over correct itself. For this experiment, I tried multiple learning rates ranging from $1x10^{-5}$ to $1x10^{-2}$ and found that the RL agents that failed to solve the environment within 3000 episodes for learning rates both too high and too low. For a learning rate of $1x10^{-5}$ , the agent got stuck in a local optima with a score of -100 while a learning rate of $1x10^{-2}$ was too high and caused overcorrections on weight changes denoted by its much higher volatility than the other. The best performing agent had a learning rate of $1x10^{-3}$ as it learned the fastest, had the lowest training volatility and the highest average test scores. What is interesting here is that $1x10^{-3}$ is the default learning rate parameter for the Adam optimizer which supports the “adaptive moment estimation” concept of this optimizer. The Adam optimizer can apply limited dynamic changes to its learning rate towards first and second moments in the weight changes. The value $1x10^{-3}$ is a middle ground enabling the Adam optimizer to both increase and decrease the learning rate. This optimizer is thus useful in sparse environments like the Lunar Lander scenario where the agent makes anywhere between 200-500 actions before receiving non-trivial positive or negative rewards. The results from this experiment reinforce the power of the Adam optimizer as a strong choice for ANN and DQN reinforcement learning.

Optimized Model Results

For the optimized RL model we can observe how the model’s score improves over time as the number of completed episodes increases. While the model does not always score above 200 (landing the spaceship in the landing zone), it clearly does so with increased frequency and does quite well in the testing session. In the 100 testing episodes, the RL agent achieved a negative score only twice and had a score of 200 or more 71% of the time. With even more training, the model’s performance would improve even more but, at this point, these results show the effectiveness of DQN agents.

Conclusion

In this paper, we observed how artificial neural networks can be applied as Q-learning function approximators in order for reinforcement learning agents to solve non-trivial Partially-Observable Markov Decision Processes. We experimented with tuning the hyperparameters of the experience replay and iterative update algorithms that make Deep Q-Learning possible and saw how they impacted an RL agent’s ability to learn an environment.

Temporal Difference in Reinforcement Learning

Mon, 08 Jun 2020 00:00:00 GMT

Abstract

The purpose of this paper is to revisit the article, “Learning to Predict by the Methods of Temporal Differences” (1988) by Richard Sutton and replicate some of the experiments therein. In his article, Sutton introduced and formalized the theory of temporal-difference (TD) methods, which are a subset of general prediction methods. Sutton goes on to describe the advantages of TD methods over supervised learning methods when predicting dynamical systems, systems which have states evolving over time. Sutton provided a simple dynamical system, a random walk, and produced experiments showing TD methods having improved learning with lower error rates than a comparable supervised learning algorithm.

Introduction and Background

This paper contains an overview of temporal-difference (TD) methods and to reengineer experiments on TD methods from an academic paper by Richard Sutton. TD methods provide an avenue for deriving the best policy for navigating multi-step systems and systems with delayed results. A policy, in this case, provides a recommended next best action to an agent given any state within a system. Examples of such systems include games (ex: Chess), predicting the weather, forecasting stock market volatility or other pattern-recognition scenarios.

There are other types of algorithms to “learn” about a given system. For example, Dynamic Programming (DP) methods involve simulating many episodes within a system where the value of a given state is determined by the values of all potential states one step beyond the state. In contrast, Monte Carlo (MC) methods involve deriving the value of a given state based only on the terminal states found in episodes containing that state. Where DPmethods can be thought of as 1-step lookahead learning methods and MC methods are last-step learning methods, TD methods handle both extremes while also taking into account “middle-ground” scenarios. In fact, TD methodology unifies the MC and DP methodology while under a single algorithm as well as middle-ground values for faster, more efficient learning. This algorithm is known as TD( $\lambda$ ) where $0 \leq \lambda \leq 1$ with TD(0) and TD(1) equivalent to DP and MC methods respectively.

Like any learning algorithm, TD methods “learn” by updating its weights:

$\Delta w_t = \alpha(z - P_t)\triangledown_w P_t$

“Weights” can also be considered the perceived value of a given value state, $z$ as the reward of the terminal state, $\alpha$ is a positive learning rate, $P_t$ as the vector of state values, and $\triangledown_w P_t$ is the partial derivative of $P_t$ with respect to $w$ . This effectively isolates the one-to-one relationship of weight/value to a given state as the partial derivative converts all other values to 0. The TD( $\lambda$ ) algorithm’s weight changing $\delta w_t$ formula is an evolution of that formula above because it can update its “learning” through incremental steps:

$\Delta w_t = \alpha(P_{t+1} - P_t) \sum_{n=1}^T \lambda^{T-n} \triangledown_wP_k$

Where $t$ is time step within $1 \leq t \leq T$ and $T$ is the total number of steps in a given episode of a system. While TD(0) (Monte Carlo) is a 1 step process process and TD(1) (Dynamic Programming) is a last step process, the equation above takes into account every step, n-step, from 1 to $T$ where the impact of a proceeding state on the value function of the current state is degraded by the exponential decay $\lambda^{T-n}$ so that the closer states have a stronger impact. However, rather than iterating through all of the n-steps for each state sequentially, we can actually shorten the computation by vectorizing the equation. We can do this by providing an eligibility trace, vector $e$ , with all values in the vector initially set to zero. When the algorithm iterates through an episode, $S_{t-1} \rightarrow S_t$ , then $e_{t-1} = e_{t-1} + 1$ such that only the states visited in the sequence so far are updated during the weight change.

$\Delta w_t = \alpha(P_{t+1} - P_t) \lambda e_{t+1}$

With the equation above, we can iteratively search through and learn from a dynamical system. In Sutton’s paper, he provided a random walk scenario as a simple dynamical system that we can use to observe TD(\lambda) performance. The random walk system from Sutton’s paper consists of seven linearly connected states (A, B, C, D, E, F, G) where A and G are terminal states and each episode has the agent start on state D. Each state has a 50% chance of transitioning to the state to its left and a 50% to its right with every state having a reward value of zero except for the G state which has a reward of one. A given episode in this system records all the transitions the agent makes until the agent hits one of the terminal states. The learning algorithm is thus trying to approximate what the expected value of each non-terminal state of the system which are $\frac{1}{6},\frac{1}{3},\frac{1}{2}, \frac{2}{3},$ and $\frac{5}{6}$ for states B, C, D, E, and F.

For the experiments in Sutton’s paper regarding the random walk system, we replicate the steps he used in creating training data by generating 100 unique training sets with each set containing 10 randomly generated random walk episodes for a total of 1000 episodes. For each experiment, we compare the expected values the learning algorithm generates from each training set with the true values stated above, derive the root mean squared error (RMSE) for each set, and then mean average the errors from all 100 sets as the output to compare how different values of $\lambda$ perform for TD( $\lambda$ ).

Experiment 1

The first experiment in Sutton’s paper regarding random walks was comparing how TD( $\lambda$ ) performed against a base linear supervised learning module, which TD(1) replicates, under repeated training over a small amount of training data. Sutton wanted to compare the base learning of TD( $\lambda$ ) compared to supervised learning algorithms so he even modified the TD( $\lambda$ ) learning algorithm so that the weights would not update incrementally but update at the end of a training set so to provide an apples to apples comparison with supervised learning algorithms. The compared supervised learning algorithm in this instance is represented using TD(1). The different models of TD( $\lambda$ ) are repeatedly shown the same 10 sequences of a training set, updating at their weights at the end of the training set, and do not stop until the max value within the $\Delta w_t$ vector is below a certain threshold value.

Experiment 1 Results

The results derived from trying to reengineer Sutton’s work match up quite well to the original material with any data point being at most +/- 0.1 from the values Sutton derived. I attribute these differences to possible floating point arithmetic errors, and the potential different number significant digits used in optimizing $\alpha$ . These results show how, for TD( $\lambda$ ) values less than 1 perform better than TD(1). In this case, we can see the best $\lambda$ is either 0 or 0.1. This outperformance over TD(1) is attributed to the fact that TD(1) minimizes error strictly to the training set while smaller values for $\lambda$ , which considers other states besides the end state, attempt to minimize the error for future experiences. This consideration for intermediate states enables the algorithms to take into account how far away a given state is from its terminal reward state and adjust accordingly in a way that a simple one-to-one mapping of a baseline supervised learning algorithm cannot.

Experiment 1 Replication Shortfalls and Assumptions

There were significant hurdles replicating the first random walk experiment in Sutton’s paper as Sutton does not go into detail about how he altered the TD( $\lambda$ ) algorithm to not update weights until the end of a training set. Simply summing the ten $\Delta w_t$ ’s meant that the explored $\alpha$ ranges would have to be several magnitude smaller smaller than otherwise would be explored in order for $\Delta w_t$ to converge. This meant that certain $\lambda$ values would only converge at $\alpha$ values small enough to be impacted by Python floating point errors. In order to combat this, I would divide every $\Delta w_t$ by the number of sequences (10) in order to have larger learning rates that would still converge. It also meant that the $\alpha$ used in Experiment 1 would not be the same as those used in Experiment 2. Since Sutton did not provide the $\alpha$ values used in Experiment 1 nor a threshold stopping condition $D$ for training where $|\Delta w_{t+1} - \Delta w_t| > D$ , I settled for a stopping threshold of 0.0001.

Experiment 2

While Sutton’s first experiment observed the advantage of TD( $\lambda$ ) under repeated training scenarios, the second experiment was meant to observe TD( $\lambda$ )’s learning speed with only a nominal amount of data. Rather than being repeatedly shown the same training set like in the first experiment, the different models of TD( $\lambda$ ) were shown a training set exactly once. Also, unlike in the first experiment when models updated their weights at the end of a training set, the second experiment had models update their weights at the end of every sequence. The RMSE for each training set were then collected and averaged out to see the average performance of initial learning for different values of $\lambda$ . Since a training set was only observed once, the projected expected values would not have enough time to converge. In order to minimize bias in the results, each model’s value function vector was initially set to 0.5.

Experiment 2 Results

For this experiment, Sutton first compared the combinations of different $\lambda$ and $\alpha$ values. Recreating the experiment yielded results very similar to those found in Sutton’s paper. The relative performance of the 4 different $\lambda$ values is consistent between papers with TD(1), the supervised learning algorithm, immediately performing the worst for any positive $\alpha$ and $\lambda =0$ performing well for small $\alpha$ values before exponentially increasing its RMSE around $\alpha =0.35$ . In comparison, the intermediate $\lambda$ values 0.3 and 0.8 find their global minima in positive values and both have much flatter slopes than TD(0) or TD(1) suggesting less sensitivity to hyper-parameter tuning like learning rate.

By graphing the RMSE for different values of $\lambda$ with their optimized $\alpha$ values, we can observe once again how intermediate values for $\lambda$ perform better than either of their extremes at TD(0) and TD(1). The recreated results for Experiment 2 have slightly different results than what Sutton produced in that all the values seemed to have a 0.03-0.06 increase in their RMSE values in the recreated results. At the same time though, the observations from the original paper still hold where the optimal value for TD( $\lambda$ ) is 0.3. While the first experiment did not really show a valuable improvement for TD( $\lambda$ ) compared to TD(0), we can see in these results that TD() learns faster than TD(0) because it can simultaneously update all states whereas TD(0)’s forward propagation can only update one state at a time for every step in the sequence. While TD(0) can avoid this issue through backpropagation, doing so eliminates its ability to do incremental learning.

Experiment 2 Replication Shortfalls and Assumptions

Trying to recreate Sutton’s results for Experiment 2 led to new results that seemed to have a near constant-value positive difference from the original values. Since the difference is relatively constant and the TD( $\lambda$ ) improvements are observable in both results, the differences can be considered trivial. The difference suggest some sort of gamma constant applied to the $\Delta w_t$ which wasn’t previously mentioned or slightly different state space searches for optimal values. Neither of these possible shortfalls detract from the overall findings.

Conclusion

Through these experiments of algorithm performance on a random walk system, we can see the the improved learning of a TD( $\lambda$ ) algorithm with a value $0 < \lambda < 1$ compared to Monte Carlo and Dynamic Programming methods in dynamical systems. TD( $\lambda$ ) achieves both a lower average error with a smaller amount of data (Experiment 1), which beats TD(1), and a faster learning rate (Experiment 2), which is better than both TD(0) and TD(1). I also observed that, while I could not 100% replicate Sutton’s results, the recreated results were close and maintained the same data relationships and findings as what was observed in Sutton’s work thus verifying the validity of the TD methodology.

Bayesian Approach to Portfolio Allocation

Sun, 26 Apr 2020 00:00:00 GMT

1. Introduction

One of the main services of investment management is to construct portfolios of stocks and bonds (securities) for clients that provides the best risk-adjusted returns. A significant amount of research has gone into portfolio construction including the invention of mean-variance optimization. Harry Markowitz introduced mean-variance optimization in 1952 and received a Nobel prize in Economics for this research. Mean-variance optimization, also known as Modern Portfolio Theory (MPT), has been the cornerstone of portfolio construction since, but it is not without issues.

After a portfolio manager (PM) decides which securities to invest in, the PM must decide what percentage of the portfolio to invest in each security. Mean-variance optimization (MVO) is a method to discover this optimal weighting. In order to accomplish this, MVO computes the volatility (standard deviation) of the portfolio which requires assembling a covariance matrix ( $\Sigma$ ) of the relationships of all the different security returns. Once this is complete, MVO searches through an assortment of weights in order to maximize risk-adjusted returns known as the Sharpe Ratio:

$\text{Sharpe Ratio} = \frac{R_p - R_f}{\sigma_p}, \quad R_p = \text{expected portfolio return }$

$R_f = \text{risk free rate}, \quad \sigma_p = \text{portfolio volatility}, \quad w = \text{vector of weights}$

$\sigma_p = \sqrt{\sigma_p^2}, \quad \sigma_p^2 = w^T\Sigma w$

$MVO = max(\frac{R_p - R_f}{\sigma_p})$

3 Asset Portfolio Example:

\sigma_p^2 = \begin{bmatrix} 0.3 & 0.4 & 0.3 \end{bmatrix} \bullet \left( \begin{bmatrix} 0.1 & 0.2 & 0.3 \\ 0.4 & 0.5 & 0.4 \\ 0.3 & 0.2 & 0.1 \end{bmatrix} \bullet \begin{bmatrix} 0.3 \\ 0.4 \\ 0.3 \end{bmatrix} \right)

\sigma_p^2 = \begin{bmatrix} 0.3 & 0.4 & 0.3 \end{bmatrix} \bullet \begin{bmatrix} 0.2 \\ 0.44 \\ 0.2 \end{bmatrix}

$\sigma_p^2 = 0.296$

These covariance matrices are extremely fragile; they are time-dependent, too precise, and degrade in terms as the number of unique stocks and bonds in the portfolio increase (creating larger matrices). Because of this fragility, pure MVO is not used much in practice as PMs attempt to find better methods to create low volatility portfolios that are resilient against unknown future risks. This paper attempts to recreate a Bayesian modeling approach to portfolio optimization that was previously explored by Fischer Black and Robert Litterman in their Black-Litterman models.

2. Data Source

For this experiment, I chose to optimize the weights of a portfolio of four ETFs (exchange traded funds) from the investment management firm Vanguard:

Total Bond Market (BND)
Total International Bond (BNDX)
Total International Stock (VXUS)
Total Stock Market (VTI)

I chose these four ETFs because combined they represent a vast majority of the investment trading universe as each is a proxy for their respective markets. They are also the four underlying investment products Vanguard uses when creating portfolios for clients through their financial advisor, robo-advisor, and target date fund offerings. This portfolio construction strategy is not exclusive to Vanguard but representative of the robo-advisor and retail investor portfolio construction offerings provided by the entire investment management industry. The combined assets of those 4 funds alone is $1.476 Trillion. Since this portfolio of “total” ETFs represents a significant chunk of most people’s investment and retirement account, any improvement in portfolio modeling here would be a huge boon for the average investor.

For this experiment, I am going to provide 3 years worth of data (April 1st 2016 - March 30th 2019) to optimize weights by Sharpe Ratios, and then test how those strategies play out from April 1st 2019 - March 30th 2020. This means the portfolios will be tested on their performance in the market recession caused by the Covid-19 pandemic which is a strong test on how risk resilient the portfolios are.

3. Creating the Models

One avenue of research is to replace the single covariance matrix of MVO by sampling returns and covariances through a Bayesian Markov Chain Monte Carlo (MCMC) approach. By sampling from a distribution of covariances, a Bayesian approach would potentially be more resilient to covariance outliers and present more genuine relationships among the different securities within a portfolio. I created two different models with the following parameters:

Model 1

Priors:

$\mu_i \sim N(0, \mu = 0.1), \quad \sigma_i \sim \text{HalfCauchy}(x_0=0, \gamma=5)$ $L \sim LKJ(\eta=5, \sigma_i), \quad \nu \sim Exp(0.1), \quad \Sigma = LL^T$

Posterior: $\mu \sim stt(\mu_i, \nu, \Sigma)$

Model 2

Priors:

$\mu_i \sim N(0, \mu = 0.1), \quad \sigma_i \sim \text{HalfCauchy}(x_0=0, \gamma=5)$ $L \sim LKJ(\eta=5, \sigma_i), \quad \Sigma = LL^T$

Posterior: $\mu \sim N(\mu_i, \Sigma)$

I used the a Student T distribution (ST) for predicting returns in the 1st model as ST has fatter tails than a normal distribution. Since stock returns have more extreme swings than a normal distribution accounts for, a Student T distribution may be more appropriate. As such, I will compare the Student T posterior to a Normal distribution

For the covariance matrix, I used a Lewandowski-Kurowick-Joe (LKJ) distribution which creates a prior on the correlation matrix. I then use a Cholesky Decomposition ( $\Sigma = LL^T$ ) to convert the correlation matrix into a covariance matrix. This method is handling covariance matrix priors is computationally superior than creating covariance priors outright.

4. Results

Strategy	BND	BNDX	VTI	VXUS	In Sample Sharpe	Out Sample Sharpe
Classic MVO	0.00	86.13	13.87	0.00	1.748	0.462
Normal	3.76	82.28	13.94	0.01	1.748	0.478
StudentT	0.00	77.11	22.89	0.00	1.643	0.200

As the results show, the Normal Bayesian approximation from the returns yielded an identical in-sample Sharpe Ratio as the basic mean-variance optimization even though the they have slightly different portfolio compositions. Yet the Bayesian optimized portfolio actually performed better in terms of risk-adjusted returns in the out-sample. The Student-T distribution performed worse suggesting, at least in this instance, it is not as strong of a modeling agent. This would make sense since, while individual securities definitely do not have normal distributions of returns, a representation of an entire market, may exhibit more normal distribution qualities. We can see the close relationship of the classic MVO with the Bayesian Normal approximation performance in the chart below.

The Bayesian approximation maps extremely closely to the classic MVO path but, with the extra diversification, it is slightly less volatile and has handled the Covid-19 black-swan event slightly better.

5. Conclusion

In my analysis, I’ve shown how a Bayesian approximation can model a mean variance optimization. While in this specific instance, a Bayesian model was able to slightly beat the basic MVO approach, further research and more model tuning needs to occur before a definitive answer could be made on the effectiveness on the approach.

6. Errata

The code for this analysis was created with assistance and inspiration from the following sources: