VISART IV
Where Computer Vision Meets Art

The Art of Vision

The main challenge of image understanding and, in particular, the analysis of art is to decompose an image into its informative attributes. The style in which a scene is presented needs to be separated from its content. The content, for instance a person, then has to be decomposed into appearance, pose, viewpoint, etc. Recently, there has been a lot of interest in models that not only learn these characteristics to then recognize a scene or detect objects therein. These generative models can also synthesize novel scenes after altering individual attributes. However, the main challenge is still the complex, highly non-linear interaction of all these characteristics. Moreover, although large datasets are easily available, labelled training data is costly and scarce.

We will discuss novel approaches for disentangling visual information into its informative constituents. To avoid the need for tedious annotations during training, the talk will cover self-supervised similarity learning and very recent improvements to it. In the context of art analysis, we will examine the benefit of these approaches for human posture recognition and synthesis, for the analysis of artistic style, and for visual retrieval in large databases of the arts.

Deep Interdisciplinary Learning. Computer Vision and Art History

Computer Vision and Art History are divided through a constructed gap between humanities and sciences with different research cultures, methods, infrastructures. The bridging of this gap is more than just a modern strategy in university politics. In my opinion it seems mandatory, because computer vision and art history are based on image description and understanding. Furthermore art is a particular field of imagery because of its complexity on the one hand and its distinctiveness on the other hand. Art historians have experience with the different ways of artistic perception, which represent several challenges for computer vision. And art historians need image processing to deal with the enhanced data basis which is accessible now. It is a chance for interdisciplinary research to have different visual cultures and methods. In my keynote I focus on the longstanding collaboration of these two fields and I will exemplify it with results. The second aim of the talk is a critical revision of the imagery of computer vision from an art historical perspective.

Deep representation of image aesthetics for visual search and content-aware in-painting.

We present a deep representation for visual aesthetics trained from a corpus of millions of artworks (BAM!), and show how this may be leveraged to distangle image content (structure) from the style in which it is depicted (aethetics). We discuss the resulting feature embedding can be leveraged for both style-aware visual search and content aware image in-painting. For the former, we propose a novel measure of visual similarity for sketch based image image retrieval that incorporates both structural and aesthetic (style) constraints. Our algorithm accepts a query as sketched shape, and a set of one or more contextual images specifying the desired visual aesthetic. For the latter, we propose a non-parametric in-painting algorithm that enforces both structural and aesthetic (style) consistency within the resulting content-aware image completion. By explicitly disentangling image structure and style during patch search and selection to ensure a visually consistent look and feel within the target image. We leverage the same model to perform adaptive stylization of patches to conform the aesthetics of selected patches to the target image, so harmonizing the integration of selected patches into the final composition. The talk comprises work presented previously at ICCV 2017 and CVPR 2018, performed in collaboration with Adobe Research.