Five specific assumptions need to be addressed in order to demonstrate the feasibility of the novel approach for the creation, production, broadcast and display of omnidirectional video proposed by ImmersiaTV project:

• Create a new cinematographic language where the specificities of immersive displays are taken into account, and which conciliates immersive paradigms with traditional storytelling techniques,
• Extend the production pipeline to create omnidirectional content for a multi-platform environment,
• Re-design the distribution chain to address the specific technical challenges that omnidirectional content imposes in terms of capture, compression, distribution, reception, and rendering,
• Maximize the quality of the end-user experience, across devices, and within the technical limitations of existing production structures, distribution facilities and reception devices to create an optimal immersive experience,
• Maximize the impact of the ImmersiaTV solutions within the ecosystem of content creators, broadcasters, and consumers.

This project will create new forms of digital storytelling and broadcast production that, by putting omnidirectional video at the center of the creation, production and distribution of broadcast content, delivers an all-encompassing experience that integrates the specificities of immersive displays, and the feeling of “being there”, within the contemporary living room.  The ImmersiaTV project will propose a new form of broadcast omnidirectional video that offers end-users a coherent audiovisual experience across head mounted displays, second screens and the traditional TV set, instead of having their attention divided across them. This new experience will seamlessly integrate with and further augment traditional TV and second screen consumer habits. In other terms: the audience will still be able to watch TV sitting on their couch, or tweet comments about it. However, by putting omnidirectional content at the center of the creation, production and distribution processes, the audience will also be able to use immersive displays to feel like being inside the audiovisual stream.

ImmersiaTV will pilot an innovative end-to-end system covering the entire audiovisual value chain to enable a novel form of creative audiovisual storytelling based on omnidirectional video. The project will encompass immersive production tools, support for omnidirectional cameras, including ultra-high definition and high dynamic range images, adaptive content coding and distribution mechanisms, and immersive (HMD) & second screen visualisation.  ImmersiaTV will demonstrate via a set of live and pre-produced pilots its deploy ability in a real production and distribution platform.

The central goal of the work plan structure is to maximize the interaction among work packages and make all the efforts converge towards, first, the large-scale pilots and, second towards innovation transfer and exploitation. We have therefore chosen to organize the overall project in 2 axes: content creation, and broadcast infrastructure, and use the pilots as the meeting point for all the different efforts.

The Work packages involved in this project are the following:

• WP1 Management, is responsible for the overall coordination of the project - I2cat (Spain),
• WP2 Requirements, format and creation of Immersive experiences. is responsible for the content format ideation, translating this into concrete production needs and content creation - VRT (Belgium),
• WP3 Immersive Broadcast Platform will focus on implementing the software tools needed to create and experience this novel format of content, as well as ensure distribution and delivery provide the best possible user experience – PSNC (Poland),
• WP4 Demonstration Pilots will be the testbed of the three previous work packages, most notably in 2 large-scale pilots, followed by an evaluation of the end-user experience  - I2cat (Spain),
• WP5 Innovation, Dissemination and Exploitation will be in constant interaction with industrial stakeholders, gathering feedback and disseminating the work done in the consortium through the larger community, including particularly standardisation committees and content creators - iMinds (Belgium).

PSNC is the coordinator of the Work Package 3 - Immersive Broadcast Platform. The result of work in this work package is a full omnidirectional image chain between capturing, by processing, encoding, content distribution up to users’ display. The main objectives of this work package are:

• to design a reliable and robust system architecture of the hardware and software platform and facilitate a smooth integration of all the project technical components,
• to design, set up and deploy an omnidirectional camera system capable of capturing live high resolution high frame rate video,
• to design and implement real-time process to effectively encode multiple images from cameras into full omnidirectional video,
• to design and implement the required functionalities to adapt the existing production tools to omnidirectional inputs and across-device visualization and interaction,
• to design and implement the communication servers required to distribute omnidirectional content (incl.  live stream) to remote users through existing and next generation access networks efficiently,
• to design and implement the clients and libraries required to display omnidirectional video-based productions across devices (TV, second screen and HMD) maintaining coherence,  synchronization, and responsivity in LAN environments,
• to integrate and test the different components in an end-to-end pilot and validate it in lab conditions.

The overall architecture of the ImmersiaTV system is depicted on Pic. 1

 
Pic. 1 – ImmersiaTV system architecture

Work in this WP3 is divided into eight Tasks and five of them - the key tasks are described
below:
T3.2 Capture – This task will focus on the development of a distributed video capture and processing architecture ground up designed for omnidirectional video in a TV broadcasting context. It addresses the issues of high equipment cost, too low perceived image resolution and frame rate, too low video processing performance and/or quality, lack of versatility in deployment of current systems. In addition to camera heads, the architecture will consist of edge capture, replay and per-camera processing units, and a central video processing unit.

T3.3 Production Tools – This task addresses the field of immersive content production tools inside the ImmersiaTV framework. Both live events and off-line documentary production scenarios are envisioned. The production tools enable creative content professionals to deliver media experiences in an unprecedented, disruptive way by carefully combining immersive content and traditional storytelling techniques. Manual or semi-automatic production of the content is facilitated by providing means for preparing omnidirectional shootings, automatic pre-selection and intuitive presentation of captured content for live and non-live scenarios. It also includes the development of an advanced immersive story editor (documentary) and/or director’s tool (live event) that enable to mix immersive and non-immersive story elements into an appealing end user content experience.

T3.4 Encoding&Decoding –  The encoding and decoding of the content will be performed in three different iterations, from off-line coding to real-time coding & decoding to region of interest (ROI) & low latency coding & decoding.
Iteration 1: Off-line coding will make use of off-self video compression encoders (e.g. HEVC/H.264). The captured content will go through various processing in order to create one or more video (depending on the number of cameras used for capture) in mono or multiview (depending on the type of camera(s)) where a pre-processing has been performed in order to represent them as conventional frame based video.

Iteration 2: In this approach, the codec developed in the previous iteration will be redesigned in order to allow for a real-time encoding in addition to real time decoding, by taking into account the network conditions (bit rate).
Iteration 3: This approach will extend the previous codec in order to take into account both the content to be coded and the view point of the user or the device. A region of interest (ROI) estimation will analyse the view point of the user, the device and the content and will produce a priority map for what part of the content should be coded when and streamed when.

T3.5 Delivery and Reception – In this task delivery and reception from origin server until the end user device screen will be addressed, discussed, designed and implemented. This task covers all network transmissions since the produced stream until any user device. It will encompass the selection of the appropriate base technologies according to the current and future networks in all the phases of the distribution, providing reliable cost-effective solutions that should be applicable in nowadays advanced and emerging networks. There are two main areas of work identified in this task, (i) transmission from the content providers servers until the audience housing, and (ii) the transmissions from the media centre to the end devices, mostly over WiFi (table, smartphone, HMD) or Ethernet (PC, smartTV, Set-Top-Box).

T3.6 Interaction and Display – This task will consist, mostly, on the delivery of the end-user application. Each iteration will focus its main efforts on a specific task. The first iteration will implement interactive display mechanisms adapted to immersive displays and second screens (head movements, tablet moved around, finger gestures). It will also deliver end-user receptors for each device that can synchronize between them on the basis of a multimedia server orchestrating the different video streams. The second iteration will introduce a distributed synchronization mechanism, potentially overriding the need for a central media server in good delivery conditions, where fast and stable network communication is available and little buffering is needed. The third iteration will refine the solutions developed in the first 2 iterations and implement additional functionality needed, such as a better memory management allowing the exploratory mode, which requires interaction with a bigger amount of video streams.

This project will use omnidirectional video enriched with novel techniques of audiovisual production to deliver a novel form of Broadcast content that matches the demands of immersive displays, and can be shared with tablet and traditional TV consumers. In particular ImmersiaTV project will provide  functionality as follows:

• Omnidirectional video capture,
• Immersive content post-production,
• Encoding /Decoding of Omnidirectional video,
• Content Delivery and Reception,
• Visualization and interaction.

Using a head mounted display it is possible to render several video streams, not necessarily omnidirectional, simultaneously, smartly inserted within its very large field of view. These inserts would be experienced as audiovisual portals, which would appear, grow, cover the whole field of view or disappear, depending both on the storyteller’s choices and end-user behavior. Using this technique, the solidly proven techniques used to build narratives within an audiovisual production –close shot to show the reaction of the main characters, slow motion to repeat a crucial moment, etc.- can still be used in the context of immersive displays, where cuts between omnidirectional shots would provoke discomfort.
From the end-user perspective, in ImmersiaTV we will allow head mounted display users to access simultaneously several video streams with an immersive display. In addition, while head mounted display users will be immersed in an omnidirectional video and have free head movements to see the whole field of view delivered by the video, traditional TV audience  will still have a consistent experience, made of portions of the same omnidirectional video, selected by the production crew. Tablet and cell phone users will be able to move around the omnidirectional video, as well as watch details by zooming in.

ImmersiaTV will take advantage of the convergence of broadband, broadcast and immersive displays in the contemporary living room to develop a novel audiovisual language that addresses the specific challenges and opportunities of immersive displays. It will also seamlessly integrate with existing consumer habits on traditional broadcast TV and second screens, including social media. The solution developed will be readily available for broadcasters to deliver and further augment their content formats. Immersia TV will also implement a set of ready-to-use tools (including elaborated exploitation plans for each of these) for creating new immersive audiovisual experiences aimed at the complete value chain. This will enable the creation of new services at different stages of the value chain, including format production, omnidirectional video capture, production tools, signal distribution and home display and interaction.