Fixed E2C Implementation Mistake

*facepalm

Just realized that I made an error in my E2C implementation. The transition dynamics in the latent space $Z$ are sampled as follows:
$$
z_t \sim N(\mu_t,\Sigma_t) \\
\hat{z}_{t+1} \sim N(A_t\mu_t+B_tu_t+o_t|C_t)\\
C_t=A_t\Sigma_t{A}_t^T
$$

Previously, I had come up with transformation of a sample $i \sim N(0,\Sigma_t)$ into the desired sample $\hat{z}_{t+1}$. While the sample is indeed drawn from the right distribution, a closer inspection shows that this is totally wrong approach from a physical interpretation:

In a locally-linear dynamical system, the next state is determined via $z_{t+1} = A_tz_t+B_tu_t+o_t$, where $A$ is the Jacobian w.r.t. autonomous (latent) state dynamics and $B$ is the Jacobian w.r.t. applied controls $u$.

I don't think there should be nothing deterministic about this transition process. $\hat{z}_{t+1} \sim N(A_t\mu_t+B_tu_t+o_t|C_t)$ is a random variable not because the transition dynamics are probabilistic, but actually because $z_{t+1}$ is a deterministic transformation of random variable $z_t$. In other words, we desire the variance of $||z_{t+1}-z_t||$ to be 0.

If we were re-sample $\hat{z}_{t+1}$ probabilistically, we are adding "extra noise into the transition" from the second sampling of $N(0,1)$. The variance of $z_{t+1}$ remains the same, but we've increased the variance over $||z_{t+1}-z_t||$, so the variance of our gradients is going to be greater. Another way to look at this is that we are "de-correlating" our sample of $z_t$ and our sample of $z_{t+1}$ when they should in fact be very-coupled. We are sampling $\hat{z}_{t+1}$ assuming that each sample of $z_t$ are exactly equal to the mean $\mu_t$.

The fact that $z_{t+1}$ should be a re-parameterized version of $z_t$ rather than re-sampled seems obvious in hindsight, but I just realized that this is the entire method by which the re-parametrization trick reduces variance. If we have a set of random variables $x_1, x_2, ... x_N$ we need to sample from, instead of sampling $N$ different noise functions, we sample a single $N(0,1)$ and re-use that noise sample to compute every random variable we need.

Fixing this bug makes the latent space images nicer. The below was trained for 2e5 iterations, with max step size = 3.

Still not perfect, but this was without any of the multistep / KL / large step-size tricks I had previously tried.

sequential E2C

Learning a sequence of $X$ simultaneously under the E2C framework leads to low loss values, but the space is no longer well-formed.

Learning a sequence of $\bar{X}$ conditioned on just $X_0$ performs even more poorly (further investigation needed):

"boxbot" worm simulation framework

I spent some time re-designing how I am going to interface the Box2D/Liquidfun physics simulations with TensorFlow, and in the process I wrote a somewhat generic framework for 2D robot simulation for Reinforcement Learning. I intend to clean up the codebase and release it as a separate project after this main capstone project is done.

I used Google Protobufs to define my robot and environment models, which can be used to express rigid-body link hierarchy coupled together by flexible muscles.

Thanks to Protobufs, I can define the robot models in Python and quickly export them to C++.

Here are a couple robots I made:

octopus arm with a fixed base:

 

  

Worm:

 

These two models are actuated via a pair of dorsal and ventral muscles between adjacent segments.

obstacle encodings

Added visualization for encodings of "invalid states" in the latent space. Invalid states are never reached in the training set, so they overlap with the distribution across "reachable" states.

Better latent space unfolding

I figured out a way to achieve a nicer-looking latent space unfolding.

The trick is to increase the maximum distance that the agent is allowed to move per step (if it is too large, it will collide and stop at an obstacle).

Here is a still image:

Intuitively, the correct topology is formed by the "stitching" of continuous topological patches in latent space. Because we are uniformly sampling trajectories and not possible positions, the reconstruction fidelity is biased towards places where the agent is seen more often.  The topology gets messed up if two adjacent points are difficult to transition between (such as a narrow passageway).

Consequently, increasing the step size "forces" the reconstructions to be meaningful not just between adjacent points, but also points up to some maximum distance away. In other words, instead of just stitching together adjacent points (corresponding to 1 unit away), points are also implicitly stitched to other points within some radius from the dataset samples.

Given this result, I hypothesize  that a multi-step reconstruction loss (i.e. minimizing the reconstruction loss over trajectory forecasts, rather than single steps) will be better. I also think that this will improve performance during planning.

If the agent has $T=10$ horizon for use in iLQR, then it seems like a good idea for it's forecasting abilities to be accurate up to around $T=10$ rather than $T=1$. Otherwise, the costs computed far away may correspond to really blurry images.

But... actual reconstructions are substantially worse than the max_dist=1 step size:

There are number of parallel tasks for me at this point:

• Multi-step (recurrent) E2C
• Actual SOC algorithm (this can proceed now that I am reasonably confident about the fidelity of the latent space reconstruction)
• Implement simulation framework for the worm
• This simulation code will generate images that are stored in a database and used to train E2C offline. 
• Online training / adaptive exploration would be awesome, but I haven't thought too much about it yet.

LiquidFun Simulations

Hacked together a very simple muscle system in LiquidFun.

Basically each "muscle segment" has two axles (with limited rotation) and axles between segments are joined via prismatic joints (pistons).

LiquidFun supports particle-based soft bodies, and I got those to work, but unfortunately I don't think it supports attaching joints (actuation) to soft bodies. That's ok - rigid bodies and the above segments should be sufficient for reproducing even the results of the biped locomotion paper.

Guided Latent Space Unfolding

Tried a slightly different environment map that resulted in a better-looking space.

The latent space seems to get pretty "knotted up" in areas where the robot doesn't explore as much. Currently the dataset's distribution is uniform over the trajectories explored by the robot (but not uniform in possible states in the world).

Does training over more samples of the robot in the blue region result in a better unfolding?

Can we do some kind of interactive importance-sampling of $x_t$ from the dataset? If the training loop is coupled to the simulation loop, can we get the current latent manifold to inform what areas to search over?

Anyway, I had this really random idea that I tried, which was to extend the E2C to not only produce 1 step prediction, but actually multi-step. The intuition is that maybe the error for 1 step is minimized, but as we march forward, the errors accumulate quickly at $t=2,3,...$, which we can squash via gradient descent. So far simply extending E2C to also produce a prediction for $x2$ given $u1,x1$ is not very helpful. The plateaued error is about twice that of single step one.

But what if I predict a longer sequence using the reconstruction as the input? I'd expect results to get very poor after a few steps due to the blurriness accumulating. Maybe we can squash that.