Title
Learning and Inference in Factor Graphs with Applications to Tactile Perception
Abstract
Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception and control. Typically, these methods rely on handcrafted models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional sensor observations. For instance, consider a robot manipulating an object in hand and receiving high-dimensional tactile observations from which it must infer latent object poses. How do we couple machine learning to extract salient information from observations and graph optimization to efficiently fuse such information?
In this thesis, we address three principal challenges: (1) How do we learn observation models from data with optimizers in the loop? We show that learning observation models can be viewed as shaping energy functions that graph optimizers, even non-differentiable ones, optimize. (2) How do we impose hard constraints in graph optimizers derived from real-world physics or geometry? We expand incremental Gauss-Newton solvers into a broader primal-dual framework to efficiently solve for constraints in an online manner. (3) Finally, we look at different learned feature representations that extract salient information from tactile image observations.
We evaluate these approaches on a real-world application of tactile perception for robot manipulation where we demonstrate reliable object tracking in hundreds of trials across planar pushing and in-hand manipulation tasks. This thesis establishes novel connections between factor graph inference, energy-based learning, and constrained optimization, opening avenues for new research problems at the intersection of these topics.
Content
0:00 Introduction
1:52 Learning for robot perception
2:26 What is the perception problem?
4:26 Modeling as a factor graph optimization
7:10 What are the key challenges?
9:16 What is the observation model?
11:42 Training on a surrogate loss
12:37 Experiments: Real-world planar pushing
18:19 Gradient update rule
18:57 LEO Algorithm Overview
19:37 LEO learning a 2D energy landscape
20:36 LEO on synthetic navigation dataset
23:20 Problem Formulation
24:12 Solver for unconstrained objective
28:54 Generalization to a Bayes Tree enables online re-linearizations
30:38 Results: Keypoint features capture contact patch
32:49 Results: Learned surface normals capture contact patch geometry
34:59 Factor graph optimizer: Integrate surface normals into a local patch map
35:41 Data collection: Simulated trials
36:02 Data collection: Real-world trials
37:36 Results: Quantitative tracker performance
38:21 Thesis Contributions: Algorithms
39:14 Thesis Contributions: Representations
40:48 Future Directions: Learning Hard Constraints
42:02 Future Directions: Learning Models in Closed Loop
42:49 Future Directions: Sensor Representations
