One of the cornerstones of ERC Advanced Grant FilmColors is the development of computer assisted methods and tools for the analysis of film colors in the emerging field of digital humanities.
For the computer-assisted qualitative analysis of film colors of a large corpus of approximately 400 films, the research team has elaborated a workflow that consists of a video annotation system and various databases, currently implemented in FileMaker.
In this workflow the team segmented and annotated the films with the software ELAN, developed by the Max Planck Institute for Psycholinguistics in Nijmegen, and then imported the corresponding data into an analysis database in which a highly detailed protocol of around 600 items was used to structure the analysis process to yield standardized results over the whole range of the corpus from 1895 to 1995. The concepts of the analysis database are defined in an illustrated glossary.
Beyond the analysis of hues and color contrasts of figures and backgrounds, the protocol required the analysis of locations and periods, narrative and semantic concepts, characters’ affective and emotional states, image composition, lighting, patterns and textures, and surfaces and properties of characters, objects and environment. The concept of faktura is employed to analyze the material properties of the films as well, while the last topic addresses movements of camera, characters and illumination. An additional keyword database contains motifs and themes that describe characteristic elements of the story. Filmographic data, including film title, year and country of production, directors, cinematographers, production and costume designers, and not least technical information, such as color process applied, camera, lenses, etc., is organized in a corpus database and connected to the analysis database. In the final evaluation, currently underway, the collected data give insights into the diachronic evolution of aesthetic patterns and their shifts in connection with technical foundations, historical, cultural and institutional contexts of the films’ production, styles of filmmakers and collaborators or production companies.
The analysis team consists of the PhD students Olivia Kristina Stutz (early films from 1895 to 1930, applied colors and early mimetic colors), Michelle Beutler (mid-century from 1930 to 1955, Technicolor, early chromogenic stocks and various other additive and subtractive two to three color systems) and Joëlle Kost (more recent chromogenic stocks), postdoc researcher Dr. Bregt Lameris with an emphasis on affective dimensions of film colors, principal investigator Prof. Barbara Flueckiger, and the student assistants Manuel Joller (working on Michelle Beutler’s period), Ursina Früh (for Olivia Kristina Stutz) and Valentina Romero (for Joëlle Kost).
Since March 2017 the team has been collaborating with the Visualization and MultiMedia Lab of Prof. Renato Pajarola, also at the University of Zurich, Dr. Enrique Paredes, Dr. Rafael Ballester, Noyan Evirgen and Gaudenz Halter. It is the aim of this collaboration to adapt the methods and insights gathered during the execution of the aforementioned analyses into (semi-)automatic tools based on computer vision and deep learning.
At the core of this development is the new video annotation tool VIAN by Gaudenz Halter. VIAN is a visual annotation and analysis environment that tackles some of the issues identified in the application of ELAN. While ELAN is a highly sophisticated software for video annotation, it was not developed with aesthetic analyses of full feature films as a target of its application. By contrast, VIAN allows, in addition to temporal segmentation and annotation, a wide variety of interventions with the audio-visual data such as adding visual annotation layers, a sophisticated tool for the creation and management of screenshots, and most importantly a range of (semi-)automatic tools for the analysis and visualization of film colors.
Some of these tools were developed outside of VIAN by Noyan Evirgen with support from VMML, Dr. Paredes and Dr. Ballester, namely a deep learning tool for figure–ground separation and extraction of the corresponding color schemes. The tool combines an object recognition software with a contour extraction tool to automatically cut out characters from their background. Color schemes are created with adaptive clustering to produce finely grained representations that match the visual impression of the spatial distribution of hues (see conference paper presented at the Digital Humanities conference in Montreal, 2017).
VIAN will contain the full range of analytic concepts elaborated in the project, assisted by the illustrated glossary for users to train their understanding and perception of film color aesthetics and narrative functions. These concepts are intended to be organized in a modular way, so that each individual user can select groups of concepts and levels of complexity according to their personal requirements and interests. Furthermore, a large arsenal of visualization and analysis tools will provide a variety of results that connect the analytical concepts to the visualizations. All these results will be connected to the Timeline of Historical Film Colors.
These developments are scheduled for 2018. Starting in 2019, a beta version will be made available to selected groups of users to test the tool and to give their feedback.
In her talk FilmColors – Bridging the Gap Between Technology and Aesthetics at The Third International Conference, Colour in Film, principal investigator Barbara Flueckiger will provide insights into the current state of the project.
Starting in September 2019, our automatic film analysis tools will be available to selected groups for beta-testing.
Blog post by Dr. Christian Gosvig Olesen: In September 2017 the research team at the University of Zurich organized a workshop with external experts to receive input and feedback. Pictured are Dr. Eric Hoyt, professor of Media Studies at the University of Wisconsin, and Dr. Everardo Reyes-García, Université Paris 8.
Flueckiger, Barbara (2011): Die Vermessung ästhetischer Erscheinungen. In: Zeitschrift für Medienwissenschaft, 5, pp. 44–60 (in German). Download PDF
Flueckiger, Barbara; Evirgen, Noyan; Paredes, Enrique G.; Ballester-Ripoll, Rafael; Pajarola, Renato (2017): Deep Learning Tools for Foreground-Aware Analysis of Film Colors. Conference paper, Computer Vision in Digital Humanities, Digital Humanities Conference, Montreal.
Flueckiger, Barbara; Halter, Gaudenz (2018): Building a Crowd-sourcing Platform for the Analysis of Film Colors. In: The Moving Image, 18.2. Download on JSTOR
Flückiger, Barbara (2019): Avancierte Methoden der computer-gestützten ästhetischen Filmanalyse. In: Digital Humanities. Multimedial & multimodal. Konferenzband 6. Tagung des Verbands Digital Humanities im deutschsprachigen Raum. Frankfurt am Main, pp. 13–21 (in German). Download
Halter, Gaudenz; Ballester-Ripoll, Rafael; Flueckiger, Barbara; Pajarola, Renato (2019): VIAN. A Visual Annotation Tool for Film Analysis. In: Computer Graphics Forum, 38,1, (= https://onlinelibrary.wiley.com/doi/full/10.1111/cgf.13676).
Stutz, Olivia: (2016) Algorithmische Farbfilmästhetik. Historische sowie experimentell-digitale Notations- und Visualisierungssysteme des Farbfilms im Zeichen der Digital Humanities 2.0 und 3.0. Zürich (in German). Download PDF, 6388 KB