ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time
We introduce Zero-shot Concept Recognition and Acquisition (ZeroC), a neuro-symbolic architecture that can recognize and acquire novel concepts in a zero-shot way. ZeroC represents concepts as graphs of constituent concept models (as nodes) and their relations (as edges). For the first time, it allows acquiring new concepts, communicating its graph structure, and applying it to classification and detection tasks (even across domains) at inference time.
Method
Humans have the remarkable ability to recognize and acquire novel visual concepts in a zero-shot manner. Given a high-level, symbolic description of a novel concept in terms of previously learned visual concepts and their relations, humans can recognize novel concepts without seeing any examples. Moreover, they can acquire new concepts by parsing and communicating symbolic structures using learned visual concepts and relations. Endowing these capabilities in machines is pivotal in improving their generalization capability at inference time.In this work, we introduce Zero-shot Concept Recognition and Acquisition (ZeroC), a neuro-symbolic architecture that can recognize and acquire novel concepts in a zero-shot way. ZeroC represents concepts as graphs of constituent concept models (as nodes) and their relations (as edges). To allow inference time composition, we employ energy-based models (EBMs) to model concepts and relations. We design ZeroC architecture so that it allows a one-to-one mapping between a symbolic graph structure of a concept and its corresponding EBM, which for the first time, allows acquiring new concepts, communicating its graph structure, and applying it to classification and detection tasks (even across domains) at inference time. We introduce algorithms for learning and inference with ZeroC.
The following figure shows how ZeroC recognizes novel hierarchical concepts at inference time. (a) During training, it learns the models for constituent concepts, the concept "line" in this case, and relations, which are "parallel" and "perpendicular". (b) During inference, it takes the concept graph of "F" and use it to derive the model for "F" from the models of its constituents. Note that no training is performed on the hierarchical concept "F". (c) Based on the composed model of "F", it is able to discover, without any more training, the mask for the hierarchical concept "F" in the presence of distractors, even though it has only learned constituent concepts of "line" and relations of "parallel" and "perpendicular".
The following figure shows how ZeroC acquires and communicates novel hierarchical concepts, even across domains, at inference time (with actual example results). ZeroC$_1$ and ZeroC$_2$ are independently trained in their respective domain (2D and 3D images). At inference, ZeroC$_1$ sees the 2D demonstration of three images showing three unseen concepts $c_1$, $c_2$, $c_3$. It first parses each image into respective concept graphs. We see that except for $c_2$ that has an edit distance $d_\text{edit}$ of 1, the others have perfect parsing. ZeroC$_1$ then sends to ZeroC$_2$ the parsed concept graphs, which ZeroC$_2$ uses to perform classification that selects which 3D image corresponds to each concept.
We evaluate ZeroC on a challenging grid-world dataset which is designed to probe zero-shot concept recognition and acquisition, and demonstrate its capability. For example, for zero-shot concept recognition of hierarchical concepts, ZeroC outperforms state-of-the-art zero-shot learning method of CADA-VAE.
For more information and results, please see the paper.
Code
A reference implementation of ZeroC in PyTorch will be available on GitHub.Datasets
The datasets used by ZeroC are included and can be generated in the code repository.Contributors
The following people contributed to ZeroC:Tailin Wu
Megan Tjandrasuwita
Zhengxuan Wu
Xuelin Yang
Kevin Liu
Rok Sosič
Jure Leskovec