System overview: Input to the learning phase is a small set of RGBD images with properly annotated labels and 3D structures (highlighted cuboids), based on which the algorithm learns the probability models. In the annotating phase, the system (a) builds the initial 3D structure of an input RGBD image, and predicts object labels using the learned models. (b-d) The user supervises the system by selecting among suggestions (e.g., re-order from ‘pillow’ to ‘nightstand’) while the system automatically refines the 3D structure to resolve ambiguity due to occlusion (e.g., the nightstand is refined to stand against the floor and wall) and re-predicts object labels (e.g., object on top of the ‘nightstand’ is more likely to be a ‘lamp’ than a ‘pillow’). The process iterates until the user approving all the annotated data. The annotated image is shown on the rightmost side and is used to augment the training data.

We are grateful to the anonymous reviewers for their comments and suggestions; all the participants of the user study for their time; and Gerardo Figueroa for the video narration. The project was supported in part by the Ministry of Science and Technology of Taiwan (102-2221-E-007-055-MY3 and 103-2221-E-007-065-MY3), the Marie Curie Career Integration Grant 303541, the ERC Starting Grant SmartGeometry (StG-2013- 335373), and gifts from Adobe Research.