IRT & LMU Cooperation: Augmented Reality in the Broadcast World

The hype about Augmented Reality (AR) is slightly going down even though the advantages remain. Especially for broadcasters the advantages of AR can be used in a variety of domains, like material planning, broadcast production, graphic development and marketing. However, AR is still a new territory for editorial and production staff. We, at the IRT, evaluate since a few years now the benefit of AR in the broadcast domain. So far, we focused on studio production with the HoloLens Spectator View and iOS Spectator View.

To develop a broader range of approaches how to use AR reasonable in TV productions the IRT cooperated with the Ludwigs-Maximilians-Universität Munich (LMU). In the 2019 summer semester 15 master students from informatics, media informatics and human computer interaction joint the project.

The scope of the project was to develop an AR application from scratch to prototype with a reasonable benefit in broadcast workflows. To design the ideas as realistic and useful as possible the concepts where pitched to representatives of different broadcast institutions and domains. In discussion with broadcasters the students could collect feedback and adapt their ideas. All three groups fund a supervisor to support them with feedback, knowledge and broadcast material. During the summer they developed their prototypes and demonstrated the approaches on the LMU OpenLabDay 2019.

To start brainstorming, we introduced the students to the basic broadcast workflows and latest AR technologies. We also demonstrated already realized AR applications, like the the WDR Kriegskinder project which was published by the Westdeutscher Rundfunk in February 2019.
With this input in mind the students started developing their own ideas. By going over the ideas again and again and constantly presenting them to the plenum we combined the single approaches and formed three groups, each with joint visions.

As next step the students focused on the technical aspects of their ideas. They researched related work and tried to figure out which technic to use for the prototype implementation.
To develop the prototypes the students coded one week together at the LMU. At the end of the week they demonstrated three AR projects.

Project 1: Interactive 360° AR

360° content can be produced in a very high quality but there are actually no VR Headsets on the market that can show the content in full quality. Another option to consume the content is a TV screens but it destroys the immersion of the video. To address this problem the Interactive 360° AR team proposed a dual approach. To display the 360° content in its full quality, they use a state-of-the-art TV display. With the help of the Microsoft HoloLens the additional parts of the 360° movie are displayed around the Screen. The TV shows a part of the movie while the rest is displayed as a sphere around the viewer. By selecting a scene out of the sphere, the viewer can actively decide what he wants to see on the TV screen. The team used Unity to synchronize the content on the HoloLens and the screen and Vuforia for the screen-recognition.  By clicking on the augmented sphere around the viewer the scene rotates and shows the selected part on the TV. They partnered with the ZDF to get feedback and 360° material for testing. The prototype proves the concept and demonstrates how high quality 360° content can be consumed without the loss of quality or immersion.

Watch the demo video – (graphics by Evelyn Müller) :

Project 2: PlaceholdAR

Efficiency and cost effectiveness are essential parts of broadcast productions. The PlaceholdAR team focuses with their approach on efficiency during the set design- and building-process. They partnered with Avirtico and the Bayerischer Rundfunk to collect feedback about today’s workflows and to discuss where to set focus. Both partners saw a potential benefit especially for non-permanent sets.
The PlaceholdAR application includes two modes. In the Set Designing mode a room can be furnished with virtual production equipment, like spots, cameras, microphones or deco elements. A Set Designer therefore uses a tablet to place the virtual objects in the real room. In the Set Construction mode, the saved virtual objects can be displayed to a technician. With the help of the Microsoft HoloLens the technician can see the virtual objects, that are placed in the room, while having both hands free to set up the physical counterpart. In their prototype the team used Unity for the scene setup and a Microsoft Azure solution to synchronize and store the changes between tablet and HoloLens. The prototype demonstrates how virtual objects are placed, manipulated and stored in a physical environment with a tablet. The objects are played back on the same physical position on a HoloLens. The PlaceholdAR team demonstrated successfully an approach to save time and enhance collaboration during a broadcast production process.

Project 3: Storytelling AR

Every TV show has their fans and loyal viewers and they are always keen to know more. Fans like to get a look behind the scenes and access to additional material. Behind-the-Scenes material, like interviews with actors, are used by many TV shows for marketing reasons. The Storytelling AR team came up with a context-based application to display additional material. The app shows viewers augmented content as if they were on a location where the show was recorded. To get material the Storytelling AR team joined a production of ndF for the ZDF series “Der Bergdoktor”.  This show has a lot of fans, who visit the film location in the mountains quite frequently. On their main location in Söll, they recorded for one day the set in 360° to recreate it virtually. The team used Unity for the scene building, Vuforia to recognize markers and Blender to build the 3D models. When a marker, like a part of the set-location, is detected the scene is augmented with virtual objects, like requisites or production equipment. The application enhances fans of “Der Bergdoktor”, when they visit the set in Söll, to see AR content through their smartphones or tablets. The prototype recognizes photographs of the original location and uses them as markers. The virtual objects are placed around the marker and additionally audio material and short video sequences can be played back.
The team demonstrated impressively how AR could be used to promote a TV show and provide additional content to a fanbase.

Watch the demo video (graphics by Franziska Haaf):

These final projects demonstrated the student’s vision in a very high level. The students where highly motivated and had a lot of fun during the project, which is reflected in the quality of the prototypes. They gained a lot of experience with AR. Thanks to their supervisors they collected many insights of the broadcast world, like filming on a real set in the mountains or working with a film crew.
For the broadcasters these three projects are good examples of how AR can be used in different broadcast areas, to support storytelling, to create immersive experiences and to enhance production workflows.
Therefore we start a new round of the project in the summer semester 2020! This time we focus on storytelling.

So, if you have a great story and want to have it told in AR, let us know and get in contact with our expert!

Back to blog