Open-World Semi-Supervised Learning

ORCA simultaneously recognizes classes previously seen in the labeled dataset and discovers novel, never-before-seen classes by grouping similar examples in the unlabeled dataset.

Supervised and semi-supervised learning methods have been traditionally designed for the closed-world setting which is based on the assumption that unlabeled test data contains only classes previously encountered in the labeled training data. In contrast to the commonly assumed closed world, the real world is inherently dynamic and open — new classes can emerge in the test data that have never been encountered during training.

Here, we introduce open-world semi-supervised learning. In this setting, the unlabeled dataset may contain classes that have never been seen in the labeled dataset, and the model needs to be able to: (i) recognize when a sample from the unlabeled data belongs to one of the seen classes present in the labeled dataset, and (ii) automatically discover novel/unseen classes without any previous knowledge by effectively grouping similar examples from the unlabeled data and assigning them to a novel class/cluster. To address the challenges of open-world SSL, we propose ORCA, an approach that learns to simultaneously classify and cluster the data. ORCA effectively assigns examples from the unlabeled data to either previously seen classes, or forms a novel class/cluster by grouping similar examples in an end-to-end deep learning framework.

Publication

Open-World Semi-Supervised Learning.
Kaidi Cao*, Maria Brbić*, Jure Leskovec.
arXiv, 2021.

@inproceedings{cao21, title={Open-World Semi-Supervised Learning}, author={Cao, Kaidi and Brbi\'c, Maria and Leskovec, Jure}, booktitle={arXiv}, year={2021}, }

Overview of ORCA


ORCA is a unique method in its ability to recognize previously seen and discover novel classes. Using both labeled and unlabeled data, ORCA learns a joint embedding function and a linear classifier consisting of classification heads for seen and additional classification heads for an expected number of novel classes. Classification heads for seen classes are used to assign the unlabeled examples to classes from the labeled set, while activating additional classification heads allows ORCA to form a novel class for examples that belong to novel classes never-before-seen in the labeled set.

To solve open-world SSL task, ORCA combines supervised objective computed on the labeled data and pairwise objective that is used to gradually generate pseudo-labels for the unlabeled set. However, naively combining supervised and pairwise objectives leads to the bias towards seen classes which reduces the ability to adapt to novel classes. To mitigate the bias, the key idea in ORCA lies in introducing uncertainty based adaptive margin in the supervised objective that gradually decreases plasticity and increases discriminability of the model during training.

Results

We apply ORCA to benchmark image classification datasets CIFAR-10, CIFAR-100 and ImageNet. Remarkably, ORCA consistently outperforms both semi-supervised and novel class discovery methods despite being the only method that can simultaneously solve both tasks. In particular, on seen classes ORCA achieves 1%, 4% and 7% improvements in accuracy over semi-supervised methods on CIFAR-10, CIFAR-100 and ImageNet datasets, respectively. On novel classes, ORCA achieves 12%, 51% and 151% improvements in accuracy over novel class discovery methods.

Figure below compares ORCA's performance to semi-supervised methods (left) and novel class discovery methods (right) when varying the ratio of seen and novel classes in the unlabeled set on the CIFAR-100 dataset. Baseline denotes open-world SSL baseline proposed in our work. Results show that ORCA consistently achieves highest accuracy across different proportion thresholds. Please refer to our paper for detailed explanations and more results.



Datasets

We use standard benchmark image classification datasets:
CIFAR-10
CIFAR-100
ImageNet

Contributors

The following people contributed to ORCA:
Kaidi Cao*
Maria Brbić*
Jure Leskovec