What if you could peek into any building, office, home and follow stories? What if you could collaborate building these stories? Attempting to answer these questions through the creation of virtual worlds may help understand transmedia experiences. These worlds constitute interfaces that allow users to interact with audiovisual content in multiple media and platforms, serving as a conduit for the exploration of transmedia universes. Departing from lessons learned from the use of animation in the design of Graphic User Interfaces (GUIs), the research being conducted explores the aesthetics and techniques of animation and motion graphics in the design of transmedia user interfaces.
The recent introduction of faster processors on multi-functional smartphones, have fostered the development of graphic user interfaces that rely on animation in order to facilitate user interaction with digital devices. Being that these devices are access terminals to audiovisual media, there are excellent opportunities to use them as interfaces for a transmedia experience on multiple platforms: laptops, game consoles, tablets, smartphones and public terminals. In all instances there are opportunities to use animation in the creation of mediated spaces that function as portals to a transmedia universe. Animation in these applications can be understood in a broader sense – as generators of artificial realities, a synthetic cinema that is naturally moldable and programmable, where the user can interact in different manners and points of entry.
Henry Jenkins presents the following definition of transmedia:
Transmedia storytelling represents a process where integral elements of a fiction get dispersed systematically across multiple delivery channels for the purpose of creating a unified and coordinated entertainment experience. Ideally, each medium makes it own unique contribution to the unfolding of the story. (2007, n.p.)
This constitutes the most widely used definition of transmedia, even though Jenkins himself agrees his definition has become outdated. More recently (2011), he suggests an update, emphasizing that there is an important difference between transmedia and cross-media: in the latter, a single contribution to the narrative is presented in multiple channels. Given the absence of physical matter, transmedia is perceived as it embodies itself in social space, has a voice in social media and is remediated as data is accessed on a variety of network interfaces. The diverse elements that compose a transmedia narrative have specific grammatical rules and audiovisual formats; these can be pre-established as is seen in film, television, literature and comics, or can be in the process of consolidation, as in games, virtual reality, viral snippets, fandom, MODS, instant messages and other emerging media. Therefore, new mediated codes are a result of diverse visual representations tied by a singular narrative.
A transmedia narrative assembles visual elements giving it “personality” and defining its identity. This can be accomplished in several fashions: in the case of literary fiction, visual representation of the narrative emerges from the reader’s imagination (Mendelsund 2014, p. 19); in audiovisual media, representation is the result of the production designer’s interpretation of the director’s vision as seen by the director of photography. Independent of crediting authorship, visual communication has an important role in creating unique narratives that can maintain the identity of a transmedia story as it is replicated on multiple platforms. Designers are usually expected to create the graphic identity for specific distribution channels of a film, game and TV series, art directors are assigned to a specific platform or medium. Rarely are these professionals hired to conceive the visual aspects of a transmedia properties as a whole.
With the objective of understanding the current state of transmedia art direction, I surveyed a range of “blockbuster” transmedia properties as listed below (Table 1).
Table 1
Images from different media of each property were collected and organized giving a panorama of the visual flow across different distribution platforms. In most cases the graphic identity and hyperlinks are the only consistent connection between different distribution platforms. However these platforms are no longer material specific: a book can be printed or an e-book, the same applies to comics, games and so forth. Given that analog media tend to have digital counterparts and digital media can be embedded in analog objects, transmediation is short of digital interfaces that connect different media.
Harry Potter is a well known example of a transmedia property: originally introduced as literary work, its graphic identity was defined by the cover art and the logo that have remained consistent in all books of the series as well as the promotional materials for the theatrical release. In the feature Harry Potter and the Sorcerer’s Stone (2001), Stuart Craig the art director for the film, expresses his visual interpretation of Hogwarts in the design and construction of the sets – translating the spatial visualization of literary constructs to cinematic space. This interpretation culminates in grand scale in Harry Potter and Deathly Hallows Part II (2011). The visual interpretation of the series “imaginary” as expressed by the art director, is replicated in transmedia extensions of the creative property, such as the theme park and games while fans offer reinterpreted versions in ARGs (Alternate Reality Games) and RPGs (Role Playing Games).
Interfaces of Transmedia
In practical and technological terms the following question emerges: Where are the interfaces of transmedia? This questioning has the objective of defining the scope of work of a designer who has an interest in developing interfaces for transmedia. With this in mind, I surveyed services, platforms, media, hardware and apps where transmedia transits, listing platforms and visually documenting these resources for consideration to be used in a practical transmedia interface project. A summary of the surveyed transmedia distribution platforms is presented on the following table (Table 2):
Table 2
The above list helps envision how broad the range of platforms is, where transmedia narratives can be mediated. From the point of view of a designer and animator, one issue that recurs is the fact of so much re-work needed to be done in order to adapt video, key artwork and text from platform to the other. Given the differences between mediums it is impossible to rely on a single interface all media, the survey will allow the analysis of which interface elements are specific to each media, and which user interaction solutions can transit within diverse media.
Animation and Human Computer Interaction
Animation has an important role in graphic user interfaces facilitating human interaction with digital devices. They help users navigate and perform tasks by establishing metaphors that represent analog methods. GUIs became popular after the launch of the Macintosh computer in 1987. The Mac OS at the time already used limited animation in its desktop interface; early Macs presented a trash can that enlarges as we place documents into it, and then empties as we delete its contents. This kind of interface is still in use today, with one great difference: videos and full blown animation are integral part of the operating system and web applications. However, it is perceived that a large number of these animations are merely effects that help ease the transition between screens.
The Macromedia Flash Application was introduced in the late 1990s as a multimedia authoring and animation software. Even being dependent on a “player” module that had to be installed in order to visualize content, the digital format became the de facto standard for distribution and display of animation on the web. One of the main reasons for the software’s success was the fact that Flash used vector based graphics and therefore was “light,” requiring fairly small network bandwidth and required minimal processing from the client machine – resources that were quite limited at the time. By the early 2000s, the use of animation on the web has become quite common; in most cases the animation was merely decorative and had no function in the usability of the interface.
Animation can be very useful and enriching when it is well applied to an interface, but it can be equally dispersive, wasting the user’s time and generating inefficient data processing. One of the reasons for the growing demand for animation on the web at the beginning of the millennium was the establishment of the web as an interactive media that was otherwise was unattractive compared to cable TV and film, due to being quite static. In the 1990s, TV channels used typography in motion as a strategy to attract a larger audience (Betancourt 2013, p. 249). The application of motion graphics on the web is still incipient and in the majority of the cases limited to the execution of pop-ups and banners. An updated version of cable TV promos is a candidate to possible form of use of animation on digital networks, for example streaming video services such as You Tube and Netflix could push animated program promotion at the end of a feature film.
One of the greatest disadvantages of distributing animation in the Flash format was the dependence of plugins, not to mention the issues related to compatibility of platforms. As a response to these limitations, alternatives have been introduced, such as the adoption of a draft specification for CSS key-frame based animation and JavaScript transitions by the W3C (World Wide Web Consortium) in 2013 (W3.org, 2017). These, on the other hand, present disadvantages related to the quality of the images imposed by GIF compression. The last 15 years have been marked by the growing popularity of videos on the web and the introduction of intelligent digital devices such as PDAs, eBooks and smartphones connected to the internet. In 2009 video data already represented 80% of internet bandwidth (Anderson 2010, p. 120). Considering that videos include animation, it is worthy to note the importance of the internet as an animation distribution platform adequate for short form formats, consequently allowing exhibition opportunities for animated shorts that would never make their way to television or movie theaters. In this context, it is suggestive that animation on the internet is a natural candidate as a transmedia interface in a similar manner that MTV’s animated promos in the 1980s were instrumental to connect viewers to cable television.
In 2002, the release of the film Minority Report (2002) brought to the general public a vision of the future of human computer interfaces. In the film, Tom Cruise’s character interacts with a spatial interface manipulating images superimposed on time on transparent surfaces. This model of interaction has its origins in the research conducted by Muriel Cooper at MIT’s Media Lab Visible Language Workshop in the 1980s (Cooper 1989). Part of the result of her research was implemented on Silicon Graphics Indigo computers, and part remained as science fiction until recently.
In the last decade researchers dedicated to human computer interaction have liberated themselves from the WIMP (window, icon, menu, pointing device) paradigm, finding alternatives in NUIs (Natural User Interfaces) and TUIs (Tangible User Interfaces) (Chung 2010, pp. 10-12), these interfaces based on gestures have become ubiquitous with the commercial introduction of touch screens and motion sensors. All these advances depend on the actions and gestures of the user; in order to allow for feedback of the user’s input, animation has become an essential component of these interfaces. TUI interface research developed by Hiroshi Ishii of the Tangible Media Group has been very influential in the interfaces designed for touch screen smartphones such as the Apple iPhone and Samsung Galaxy, Professor Ishi’s current research in three-dimensional projected surfaces constitutes a form of interactive holography (Ishii et al. 2016), indicating the possibilities of the creation of non-immersive virtual realities that are appropriate for the development of transmedia interfaces. This research is leading the path to the realization of interactive spaces that approach the Holodeck in the Star Trek TV series.
The first generation of iPhones was released by Apple in 2007. The device presented a single physical button; besides this exception, all interaction occurred using gestures on a touch sensitive screen and movements recognized by the accelerometer built into the device. This type of dynamic interaction generated the development of animated codes that correspond to users gestures. The small screen also depended on animation in order to make the connection between screens and applications that run simultaneously. Multifunctional devices that are capable of simulating a range of tools, have become dependent on animation in order to explicit metaphors that represent the functionality of these tools. For example, we can mention the iris that opens and closes when you take a picture with the iPhone camera app. It is important to note that Apple does not support Flash software in its devices; consequently the de facto standard of digital animation distribution became questioned. This policy has triggered the development of open source vector based animation tools such as CSS (Cascading Style Sheets) and HTML 5 (Hinman 2012).
While researching contemporary use of animation in computer interfaces, I have encountered a wide number of transition repositories widely available on the web, ready to be incorporated in websites and apps. These are “ready to use” coded animation that can be applied to the graphic elements of user interface, the majority of animations are transitional devices that ease the change of menus on the screen. Attention should be given to two UX (user experience) designers that have contributed to the theme and have gone beyond using animation as mere transitional devices: Pasquale D’Silva (2013) and Rachel Hinman (2012), coincidently, both present guidelines for the application of animation in interfaces that are based on Disney’s Twelve Basic Principles of Animation. By transposing these principles to human-computer interfaces we encounter an important theoretical reference that can be applied by programmers and UX designers for practical use in the field.
In the last years, the popularization of digital mobile devices such as smartphones and tablets represent an important aspect of digital inclusion for a great portion of the world population. It is noted that these devices have been increasingly used as mediated receptors as well as interaction platforms, consequently, these small devices have become more and more important as access points to a wide range of media and their transmedia extensions. My current research intends to subsidize their use as entry portals to transmedia universes.
The project
Lucia Santaella (2015), a distinguished Brazilian semiotician, made a very enlightening presentation in 2015 at the Archiinterface conference at FIESP in São Paulo. In this talk, Santaella introduces the concept of an “exosphere:” a digital layer that extends and at times mirrors our environment. The exosphere is not virtual reality, it is instead an intelligent surface that is integral to the physical world and can be graphically visualized and explored using animated interfaces. If we analyze some examples of transmedia properties we can verify the possibility of mapping diverse elements that construct a narrative: the storyline in a film or animation, posts in blogs or social networks, additional information and stats available on wikis, are examples of elements that can be located in the network and accessed via hyperlinks, manifesting themselves digitally when downloaded, consumed directly in streaming and online games. Other nodes in the networks indicate access points to components that are present or not in other platforms; movie, theater, cable TV, live events, concerts and augmented reality.
The objective of the research discussed in this article is to subsidize a practical project I am currently developing. In this project, I intend to design a human-computer user interface exploring the architecture of transmedia narratives, represented in two or three dimensions, alterable in time and space as the narrative develops and the viewer/user explores the story space. Assuming that animation is the most appropriate art form for this purpose, one of the objectives of the design in course is to apply current HCI (Human Computer Interaction) research that uses animation. For example, by mapping imaginary spaces, maps can guide viewers in the branching of non-Cartesian worlds in Game of Thrones or Star Trek; alternatively, our own surroundings can be mapped into an exosphere applied to apps such as Pokémon Go, Waze and Google maps.
The proposed project will use footage I shot in 1992 for an unfinished documentary produced for a course at the New School for Social Research in New York, and consisting of interviews with Brazilians who immigrated to the U.S.A. Currently I am contacting the people whom I interviewed, in order to discover how their situations and views have changed 25 years later. Given that the subject matter encompasses geographical references related to personal stories changing over time, I believe it is an optimal candidate for the design of a user interface that uses animation to fill narrative gaps and that can be translated into a system where transmedia properties are hosted and distributed. The system will be based on “feedable” media modules consisting of text, videos, maps, pictures and sounds uploaded by authors and users. The interface design is expected to evolve into a replicable system that encompasses the production and distribution of non-fiction media connected to a visual database. The construction of a multi-user interfaces will be proposed as a navigational method in transmedia narratives, creating a virtual transmedia space that is connected to the real. Media can converge or transit within itself: books, films, TV, comic books, live events, site-specific interventions, maps, happenings. Both public and closed circuit media can be conductors of the same story. The purpose is to create a flux between different media, using animation as an audiovisual conducting stream, assuming the connectivity of platforms in a manner that animation facilitates the flow between pages represented in the graphic interfaces of multiple applications.
Future directions
Recently there has been a growing interest of the entertainment industry and the news media in exploring virtual reality as a distribution platform. With the introduction of Google Cardboard, a low-end accessory that converts a smartphone into a VR (Virtual Reality) viewer and mid-range hardware such as the Samsung – Oculus VR partnership there has been a growing reception base for this kind of media. According to an article in Bloomberg Businessweek (Urstadt & Frier 2016, p.38), Facebook’s CEO Mark Zuckerberg, expressed his intention of distributing two million VR headsets to the social network’s users. If this scenarios consolidates, it will become even more important that designers work on spatial mapping of narratives, relating them to geo-localization in non-fiction stories and the construction of transmedia fictional universes, allowing users to transit from node to another independently of the capillarity of hyperlinks. In this scenario there is great potential for animation to become an essential component audiovisual language intrinsic to the narratives of the future.
Dr. João Paulo Amaral Schlittler is a Professor in the Department of Film, Radio and Television(School of Communication and Arts) at the Universidade de São Paulo in São Paulo, Brazil.
References
Anderson, C. & Wolff, M. (2010) “The Web Is Dead. Long Live the Internet” Wired Magazine, September, pp. 118-127.
Betancourt, M. (2013) The History of Motion Graphics: From Avant-Garde to Industry in the United States, Wildside, U.S.A.
Chung, J.L. (2010) “In search of a Natural Gesture” XRDS: Crossroads, The ACM Magazine for Students, The Future of Interaction, v. 16 i. 4.
Cooper, M. (1989) “Computers and Design,” Design Quarterly, v. 142.
D’Silva, P. (2013) “Transitional Interfaces”, Medium.com, Retrieved from https://medium.com/@pasql/transitional-interfaces-926eb80d64e3#.sntu0ytde [Viewed: 30th August 2016].
Hinman, R. (2012) “A New Mobile UX Design Material,” Smashing Magazine, Retrieved from: http://www.smashingmagazine.com/2012/10/30/motion-and- animation-a-new-mobile-ux-design-material [Viewed: 30th August 2016].
Ishii H., Nakagaki, K., Vink, L., Counts, J., Windham, D., Leithinger D., & Follmer S. (2016) “Materiable: Rendering Dynamic Material Properties in Response to Direct Physical Touch with Shape Changing Interfaces,” Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), 2764-2772.
Jenkins, H. (2007) “Transmedia Storytelling 101,” Confessions of an Aca-Fan, The official weblog of Henry Jenkins, Retrieved from: http://henryjenkins.org/2007/03/transmedia_storytelling_101.html [Viewed: 10th January 2017].
—. (2011) “Transmedia 202: Further Reflections” Confessions of an Aca-Fan The official weblog of Henry Jenkins, Retrieved from: http://henryjenkins.org/2011/08/defining_transmedia_further_re.html [Viewed: 10th January 2017].
Harry Potter and the Sorcerer’s Stone (2001) [35mm Film], Written by J.K. Rowling, Directed by Chris Columbus, USA: Warner.
Harry Potter and the Deathly Hallows: Part 2 (2011) [35mm Film], Written by J.K. Rowling, Directed by David Yates, USA: Warner.
Santaella, L. (2015) “Tecnologia e Imaginário Urbano,” Arquiinterface Conference, Presented 27th October, FIESP-SESI, São Paulo, Brazil.
Minority Report (2012) [35mm Film], Directed by Steven Spielberg. USA: Fox.
Urstadt, B. & Frier, S. (2016) “Welcome to Zucker World” Bloomberg Business Week, August 1-7 2016, pp. 38-42.
W3.org. (2017) CSS Animations [online], Retrieved from: https://www.w3.org/TR/css3-animations/ [Viewed 22 May 2017].
© João Paulo Amaral Schlittler
Edited by Amy Ratelle