Open Compound Domain Adaptation

Zhongqi Miao and Ziwei Liu    Jun 14, 2020

The World is Continuously Varying

Imagine we want to train a self-driving car in New York so that we can take it all the way to Seattle without tediously driving it for over 48 hours. We hope our car can handle all kinds of environments on the trip and send us safely to the destination. We know that road conditions and views can be very different. It is intuitive to simply collect road data of this trip, let the car learn from every possible condition, and hope it becomes the perfect self-driving car for our New York to Seattle trip. It needs to understand the traffic and skyscrapers in big cities like New York and Chicago, more unpredictable weather in Seattle, mountains and forests in Montana, and all kinds of country views, farmlands, animals, etc. However, how much data is enough? How many cities should we collect data from? How many weather conditions should we consider? We never know, and these questions never stop.


Figure 1: Domains boundaries are rarely clear. Therefore, it is hard to set up definite domain descriptions for all possible domains.

As for the weather, the number of different conditions can be infinite, while the words that can be used to describe weather conditions (such as sunny and rainy) are always limited. Some conditions have subtle differences that even humans can not tell. For example, overcast v.s. cloudy, the transition state of rainy to non-rainy, light snow v.s. light rain, etc. We can always try to collect data for every possible weather condition that we can imagine and use traditional domain adaptation methods to help our car to adapt to these conditions. But our imagination is limited when it comes to weather, and the main reason is that there are no clear boundaries between weather conditions. In that case, none of the traditional domain adaptation approaches, where clear boundaries are assumed, can help us in real-world weather adaption. Therefore, we start rethinking machine learning and domain adaptation systems, and try to introduce a continuous learning protocol under domain adaptation scenario.

Open Compound Domain Adaptation (OCDA)

The goal of domain adaptation is to adapt the model learned on the training data to the test data of a different distribution. Such a distributional gap is often formulated as a shift between discrete concepts of well defined data domains, e.g., images collected in sunny weather versus those in rainy weather. Though domain generalization and latent domain adaptation have attempted to tackle complex target domains, most existing works usually assume that there is a known clear distinction between domains. Such a known and clear distinction between domains is hard to define in practice, e.g., test images could be collected in mixed, continually varying, and sometimes never seen weather conditions. With numerous factors jointly contributing to data variance, it becomes implausible to separate data into discrete domains.

We propose to study Open Compound Domain Adaptation (OCDA), a continuous and more realistic setting for domain adaptation (Figure 2). The task is to learn a model from labeled source domain data and adapt it to unlabeled compound target domain data which could differ from the source domain on various factors. Our target domain can be regarded as a combination of multiple traditionally homogeneous domains where each is distinctive on one or two major factors, and yet none of the domain labels are given. For example, the five well-known datasets on digits recognition (SVHN, MNIST, MNIST-M, USPS, and SynNum) mainly differ from each other by the backgrounds and text fonts. It is not necessarily the best practice, and not feasible under some scenarios, to consider them as distinct domains. Instead, our compound target domain pools them together. Furthermore, at the inference stage, OCDA tests the model not only in the compound target domain but also in open domains that have previously unseen during training.


Figure 2: Open Compound Domain Adaptation problem. Unlike existing domain adaptation which assumes clear distinctions between discrete domains, our compound target domain is a combination of multiple traditionally homogeneous domains without any domain labels. We also allow novel domains to show up at the inference time.

While OCDA has not been defined in the literature, there are two closely related tasks which are often studied in isolation: single-target domain adaptation, and multi-target domain adaptation. Figure 3 summarizes their differences. The newly proposed Open Compound Domain Adaptation (OCDA) serves as a more comprehensive and more realistic touchstone for evaluating domain adaptation and transfer learning systems.


Figure 3: The differences between single-target domain adaptation, multi-target domain adaptation and our open compound domain adaptation (OCDA).

The Importance of Curriculum & Memory

In our OCDA setting, the target domain no longer has a predominantly uni-modal distribution, posing challenges to existing domain adaptation methods. We propose a novel approach based on two technical insights into OCDA: 1) a curriculum domain adaptation strategy to bootstrap generalization across domain distinction in a data-driven self-organizing fashion and 2) a memory module to increase the model’s agility towards novel domains, as illustrated in Figure 4.

Firstly, unlike existing curriculum adaptation methods that rely on some holistic measure of instance difficulty, we schedule the learning of unlabeled instances in the compound target domain according to their individual gaps to the labeled source domain, so that we solve an incrementally harder domain adaptation problem till we cover the entire target domain.

Our second technical insight is to prepare our model for open domains during inference with a memory module that effectively augments the representations of an input for classification. Intuitively, if the input is close enough to the source domain, the feature extracted from itself can most likely already result in accurate classification. Otherwise, the input-activated memory features can step in and play a more important role. Consequently, this memory-augmented network is more agile at handling open domains than its vanilla counterpart.


Figure 4: Overview of our approach. 1) We separate characteristics specific to domains from those discriminative between classes. The teased out domain feature is used to construct a curriculum for domain-robust learning. 2) We enhance our network with a memory module that facilitates knowledge transfer from the source domain to target domain instances, so that the network can dynamically balance the input information and the memory-transferred knowledge for more agility towards previously unseen domains.

Robustness to Compound and Open Domains

We control the complexity of the compound and open target domain by varying the number of traditional target domains / datasets in it. Here we gradually increase constituting domains from a single target domain to two, and eventually three. As demonstrated in Figure 5, we observe that our approach is more resilient to the various numbers of compound and open domains. 1) The learned curriculum enables gradual knowledge transfer that is capable of coping with complex structures in the compound target domain. 2) Furthermore, the domain indicator module in our framework helps dynamically calibrate the embedding, thus enhancing the robustness to open domains.


Figure 5: The performance change w.r.t. the number of compound and open domains.

Visualization of Learning Dynamics

Here is a visualization of disentangling domain characteristics. We separate characteristics specific to domains from those discriminative between classes. It is achieved by a class-confusion algorithm in an unsupervised manner. Figure 6 (a) and (b) visualize the examples embedded by the class encoder and domain encoder, respectively. The class encoder places instances in the same class in a cluster, while the domain encoder places instances according to their common appearances, regardless of their classes.


Figure 6: t-SNE Visualization of our (a) class-discriminative features, and (b) domain features. Our framework disentangles the mixed-domain data into class-discriminative factors and domain-focused factors.

Acknowledgements: We thank all co-authors of the paper “Open Compound Domain Adaptation” for their contributions and discussions in preparing this blog. The views and opinions expressed in this blog are solely of the authors of this paper.

This blog post is based on the following paper which will be presented at IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2020) as an oral presentation:

Subscribe to our RSS feed.

Comments