Four Novel Approaches to Manipulating Fabric using Model-Free and Model-Based Deep Learning in Simulation

Daniel Seita, Wilson Yan, Ryan Hoque    May 5, 2020


Humans manipulate 2D deformable structures such as fabric on a daily basis, from putting on clothes to making beds. Can robots learn to perform similar tasks? Successful approaches can advance applications such as dressing assistance for senior care, folding of laundry, fabric upholstery, bed-making, manufacturing, and other tasks. Fabric manipulation is challenging, however, because of the difficulty in modeling system states and dynamics, meaning that when a robot manipulates fabric, it is hard to predict the fabric’s resulting state or visual appearance.

In this blog post, we review four recent papers from two research labs (Pieter Abbeel’s and Ken Goldberg’s) at Berkeley AI Research (BAIR) that investigate the following hypothesis: is it possible to employ learning-based approaches to the problem of fabric manipulation?

We demonstrate promising results in support of this hypothesis by using a variety of learning-based methods with fabric simulators to train smoothing (and even folding) policies in simulation. We then perform sim-to-real transfer to deploy the policies on physical robots. Examples of the learned policies in action are shown in the GIFs above.

We show that deep model-free methods trained from exploration or from demonstrations work reasonably well for specific tasks like smoothing, but it is unclear how well they generalize to related tasks such as folding. On the other hand, we show that deep model-based methods have more potential for generalization to a variety of tasks, provided that the learned models are sufficiently accurate. In the rest of this post, we summarize the papers, emphasizing the techniques and tradeoffs in each approach.

Model-Free Methods

Model-Free Learning without Demonstrations

In this paper we present a model-free deep reinforcement learning approach for smoothing cloth. We use a DM Control environment with MuJoCo. We emphasize two key innovations that help us accelerate training: a factorized pick-and-place policy, along with learning the place policy conditioned on random pick points, and then choosing pick point by maximum value. The figure below shows a visualization.


As opposed to directly learning both the pick and place policy (a), our method learns each component of a factorized pick-and-place model independently by first training with a place policy with random pick locations, and then learning the pick policy.

Jointly training the pick and place policies may result in inefficient learning. Consider the degenerate scenario when the pick policy collapses into a suboptimal restrictive set of points. This would inhibit exploration of the place policy since rewards come only after the pick and place actions are executed. In order to solve this problem, our method proposes to first use Soft Actor Critic (SAC), a state-of-the-art model-free deep reinforcement learning algorithm, to learn a place policy conditioned on pick points sampled uniformly from valid pick points on the cloth. Then, we characterize the pick policy by selecting the point with the highest value from the approximated value estimator learned when training the place policy, thus Maximal Value under Placing (MVP). We note that our approach is not tied to SAC, and can work with any off-policy learning algorithm.


An example of real robot cloth smoothing experiments with varying starting states and cloth colors. Each row shows a different episode from a start state to the achieved cloth smoothness. We observe that the robot can reach the goal state from complex start states, and generalizes outside the training data distribution.

The figure above shows different episodes on a real robot using a pick-and-place policy learned with our method. The policy is trained in simulation, and then transferred to a real robot using domain randomization on cloth physics, color, and lighting. We can see that the learned policy is able to successfully smooth cloth starting from many different complexities of state, and for different cloth colors.

The advantages of this paper’s model-free reinforcement learning approach is that all training can be done in simulation without any demonstrations, and that training can readily be applied using off-the-shelf algorithms and is faster due to the pick-and-place structure we present which (as discussed earlier) can avoid mode collapse. The tradeoff is that it trains a policy that can only do smoothing, and must be re-trained for other tasks. In addition, the actions may take relatively short pulls and might be inefficient when it comes to more difficult cloth tasks such as folding.

Model-Free Learning with Simulated Demonstrations

We now present an alternative approach for smoothing fabrics. Like the prior paper, we use a model-free method and we create an environment for fabric manipulation using a simulator. Instead of MuJoCo, we use a custom-built simulator that represents fabric as a $25 \times 25$ grid of points. We’ve open sourced this simulator for other researchers to use.

In this project, we consider a fabric plane as a white background square of the same size as a fully smooth fabric. The performance metric is coverage, or how much of the background plane gets covered by the fabric, which encourages the robot to cover a specific location. We terminate an episode if the robot attains at least 92% coverage.

One way to smooth fabric is to pull at fabric corners. Since this policy is easy to define, we code an algorithmic supervisor in simulation and perform imitation learning using Dataset Aggregation (DAgger). As briefly covered in a prior BAIR Blog post, DAgger is an algorithm to correct for covariate shift. It continually queries a supervisor agent to get corrective actions for states. This is normally a downside for DAgger, but is not a problem in this case, as we have a simulator with full access to state information (i.e., the grid of $25\times 25$ points) and can determine the optimal pull action efficiently.

In addition to using color images, we use depth images, which provide a “height scale.” In a prior BAIR Blog post, we discussed how depth was useful for various robotics tasks. To obtain images, we use Blender, an open-source computer graphics toolkit.


An example episode of our simulated corner-pulling supervisor policy. Each column of images shows one action, represented by the overlaid white arrows. While we domain-randomize these images for training, for visualization purposes in this figure, we leave images at their "default" settings. The starting state, represented to the left, is highly wrinkled and only covers 38.4% of the fabric plane. Through a sequence of five pick-and-pull actions, the policy eventually gets 95.5% coverage.

The figure above visualizes the supervisor’s policy. The supervisor chooses the fabric corner to pull based on its distance from a known target on the background plane. Even though fabric corners are sometimes hidden by a top layer, as in the second time step, the pick-and-pull actions are eventually able to get sufficient coverage.

After training using DAgger on domain randomized data, we transfer the policy to a da Vinci Surgical Robot without any further training. The figure below represents an example episode of the da Vinci pulling and smoothing fabric.


An example seven-action episode taken by a policy trained only on simulated RGB-D images. The top row has screenshots of the video of the physical robot, with overlaid black arrows to visualize the action. The second and third rows show the color and depth images that are processed as input to be passed through the learned policy. Despite the highly wrinkled starting fabric, along with hidden fabric corners, the da Vinci is able to adjust the fabric from 40.1% to 92.2% coverage.

To summarize, we learn fabric smoothing policies using imitation learning with a supervisor that has access to true state information of the fabrics. We domain randomize the colors, brightness, and camera orientation on simulated images to transfer policies to a physical da Vinci surgical robot. The advantage of the approach is that the robot can efficiently smooth fabric in relatively few actions and does not require a large workspace, as the training data consists of long pulls constrained in the workspace. In addition, implementing and debugging DAgger is relatively easy compared to model-free reinforcement learning methods as DAgger is similar to supervised learning and one can inspect the output of the teacher. The primary limitations are that we need to know how to implement the supervisor’s policy, which can be difficult for tasks beyond smoothing, and that the learned policy is a smoothing “specialist” that must be re-trained for other tasks.

Model-Based Methods

Planning Over Image States

While the previous two approaches give us solid performance on the smoothing task on real robotic systems, the learned policies are “smoothing specialists” and must be re-trained from scratch for a new task, such as fabric folding. In this paper, we consider the more general problem of goal-conditioned fabric manipulation: given a single goal image observation of a desired fabric state, we want a policy that can perform a sequence of pick-and-place actions to get from an arbitrary initial configuration to that state.

To do so, we decouple the problem into first learning a model of fabric dynamics directly from image observations and then re-using that dynamics model for different fabric manipulation tasks. For the former, we apply the Visual Foresight framework proposed by our BAIR colleagues, a model-based reinforcement learning technique that trains a video prediction model to predict a sequence of images from the image observation of the current state as well as an action sequence. With such a model, we can predict the results of taking various action sequences, and can then use planning techniques such as the cross-entropy method and model-predictive control to plan actions that minimize some cost function. We use Euclidean distance to the goal image for the cost function in our experiments.

We generate roughly 100,000 images from an entirely random policy, executed entirely in simulation. Using the same fabric simulator as in (Seita et al., 2019), we use Stochastic Variational Video Prediction (SV2P) as the video prediction model. We leverage both RGB and depth modalities, which we find in our experiments to outperform either modality alone, and thus call the algorithm VisuoSpatial Foresight (VSF).

While prior work on Visual Foresight includes some fabric manipulation results, the tasks considered are typically short horizon and have a wide range of goal states, such as covering a spoon with a pant leg. In contrast, we focus on longer horizon tasks that require a sequence of precise pick points. See the image below for typical test-time predictions from the visual dynamics model. The data is domain-randomized in color, camera angle, brightness, and noise, to facilitate transfer to a da Vinci Surgical Robot.


We show the ground truth images as a result of fabric manipulation, each paired with predictions from the trained video prediction model. Given only a starting image (not shown), along with the next four actions, the video prediction model must predict the next four images, shown above.

The predictions are accurate enough for us to plan toward a variety of goal images. Indeed, our resulting policy rivals the performance of the smoothing specialists, despite only seeing random images at training time.



We execute a sequence of pick-and-place actions to manipulate fabric toward some goal image. The top row has three different goal images: smooth, folded, and doubly folded, which has three layers of fabric stacked in the center in a particular order. In the bottom row, we show simulated rollouts (shown here as time-lapses of image observations) of our VSF policy manipulating fabric toward each of the goal images. The bottom side of the fabric is a darker shade (slightly darker in the second and much darker in the third column), and the light patches within the dark are due to self-collisions in the simulator that are difficult to model.

The main advantage of this approach is that we can train a single neural network policy to be used for a variety of tasks, each of which are set by providing a goal image of the target fabric configuration. For example, we can do folding tasks, for which it may be challenging to hand-code an algorithmic supervisor, unlike the case of smoothing. The main downsides are that training a video prediction model is difficult due to the high dimensional nature of images, and that we typically require more actions than the imitation learning agent to complete smoothing tasks as the data consists of shorter-magnitude actions.

Planning Over Latent States

In this paper, we similarly consider a model-based method, but instead of training a video prediction model to plan in pixel space, we instead plan in a learned lower-dimensional latent space since learning a video prediction model can be challenging, as the learned model must capture every detail of the environment. In addition, it is also difficult to learn proper pixel dynamics in the cases when we use frame-by-frame domain randomization to transfer to the real world.


A visual depiction of the contrastive learning framework. Given positive current-next state pairs and randomly sampled negative observations, we learn an encoder and forward model such that the estimated next states lie closer than the negative observations in the latent space.

We jointly learn an encoder and a latent forward model using contrastive estimation methods. The encoder maps raw images into a lower dimensional latent space. The latent forward model will take this latent variable, along with the action, and produce an estimate of the next state.

We train our models by minimizing a variant of the InfoNCE contrastive loss, which encourages learning latents that maximize mutual information between encoded latents and their respective future observations. In practice, this training method will bring current and subsequent latent encodings closer (in $L_2$ distance), while making other sampled non-next latent encodings to be further apart. As a result, we are able to use the learned encoder and learned forward model to effectively predict the future, similar to the image-based approach presented in the prior paper (Hoque et al., 2020), except we are not predicting images but latent variables, which are potentially easier to work with.

In our cloth experiments, we apply random actions to collect 400,000 samples in a DM Control simulator with added domain randomization on cloth physics, lighting, and cloth color. We use the learned encoder and forward model to perform model predictive control (MPC) with one-step prediction to plan towards a desired goal state image. The figure below shows examples of smoothing out different colored cloths on a real PR2 robot. Note that the same blue cloth is used as the goal image regardless of the actual cloth being manipulated. This indicates that the learned latents have learned to ignore unnecessary properties of cloth such as color when performing manipulation tasks.


Several episodes of both manipulating rope and cloth using our method, with different start and goal states. Note that the same blue cloth is used as the goal state irrespective of the color of the cloth being manipulated

Similar to (Hoque et al., 2020), this method is able to solve multi-task goal states, as shown in the example episodes above run on a real robot. Using contrastive methods to learn a latent space, we also achieve better sample complexity in model learning compared to direct video prediction models, because the latter require more samples to predict high dimensional images. In addition, planning directly in latent spaces is easier compared to planning with a visual model. In our paper, we show that using a simple one-step model-predictive control to plan in latent spaces works substantially better than one-step planning with a learned visual forward model, perhaps because the latents learn to ignore irrelevant aspects of the images. Although planning allows for cloth spreading and rope orientation manipulation, our models fail to perform long horizon manipulation since the models are trained on offline random actions.

Discussion

To recap, we presented four related papers which present different approaches for robot manipulation of fabrics. Two use model-free approaches (one with reinforcement learning and one with imitation learning) and two use model-based reinforcement learning approaches (with either images or latent variables). Based on what we’ve covered in this blog post, let’s consider possibilities for future work.

One option is to combine these methods, as done in recent or concurrent work. For example, (Matas et al., 2018) used model-free reinforcement learning with imitation learning (through demonstrations) for cloth manipulation tasks. It is also possible to add other tools from the robotics and computer vision literature, such as state estimation strategies to enable better planning. Another potential tool might be dense object descriptors which indicate correspondence among pixels in two different images. For example, we have shown the utility of descriptors for a variety of rope and fabric manipulation tasks.

Techniques such as imitation learning, reinforcement learning, self-supervision, visual foresight, depth sensing, dense object descriptors, and particularly the use of simulators, have been useful tools. We believe they will continue to play an increasing role in robot manipulation of fabrics, and could be used for more complex tasks such as wrapping items or fitting fabric to 3D objects.

Reflecting back on our work, another direction to explore could be using these methods to train six degree-of-freedom grasping. We restricted our setting to planar pick-and-place policies, and noticed that the robots often had difficulty with top-down grasps when fabric corners were not clearly exposed. In these cases, more flexible grasps may be better for smoothing or folding. Finally, another direction for future work is to address the mismatches we observed between simulated and physical policy performance. This may be due to imperfections in the fabric simulators, and it might be possible to use data from the physical robot to fine-tune the parameters of the fabric simulators to improve performance.

We thank Ajay Kumar Tanwani, Lerrel Pinto, Ken Goldberg, and Pieter Abbeel for providing extensive feedback on this blog post.

This research was performed in affiliation with the Berkeley AI Research (BAIR) Lab, Berkeley Deep Drive (BDD), and the CITRIS “People and Robots” (CPAR) Initiative. The authors were supported in part by Honda, and by equipment grants from Intuitive Surgical and Willow Garage.

Subscribe to our RSS feed.

Comments