Processamento, Codificação e Qualidade de Múltiplas vistas e Campos de Luz

This research project aims to establish new coding models and quality evaluation models for multiview image systems, including Light Field technology. The main rationale for this proposal comes from the observation that current compression technology and quality evaluation models do not consider the typical applications of these modalities. Usually, only visualization is considered, but even that case has limited quality evaluation models. During the development of JPEG Pleno Light Field Coding, light field quality was subjectively evaluated, displaying the distorted and reference views side-by-side as a video sequence. Each view was displayed for 0.1 seconds. Obviously, that did not allow a careful comparison of the static light fields. Further, such an evaluation did not take into account the possible application of the light field content. For instance, if the light field is displayed on a stereo head mounted display, only two views are represented at each instance, and as it is known from stereo imaging perception knowledge, it is very likely that subjects will perceive the quality based on the higher quality displayed view. Moreover, if the light field is used for view generation, multi-focus image generation, or super-resolution, the effects of compression have not been evaluated. This project intends to: - study how compression methodologies influence the performance of the different applications, notably visualization, multi-focus images, view generation, or super-resolution applications. - study if it is possible to compress the original information that results from the image sensors instead of the final image generated, like super-resolution images, multi-focus images, or fused images, as is typical on multi-view applications like those existing in mobile cameras. - develop new quality models that allow the development of light fields, structured or unstructured, or multiview image systems coding models, considering the typical applications of these imaging modalities. - develop compression models that consider the different light fields and multi-view imaging applications. These compression models will be designed considering the new quality models and tested at the application level. Currently, the existing compression models consider image visualization applications. Both the JPEG and MPEG standardization committees had developed standards for light field or multiview image coding, targeting visual applications. However, it is important that these compression models also enable different applications, like image refocusing, view generation, or super-resolution applications. While the quality models will be designed considering the imaging modalities applications, the compression models will be based on deep learning based models. This selection is based on recent observations of high compression efficiency, like in the case of JPEG AI, which reveals a 30% increase in bit rate/distortion performance over the VVC intra classic codec. The project will be based on the balance of the duality between quality evaluation and compression efficiency, considering the usual applications developed for these imaging modalities, and will use reference implementations. If it is found suitable, new models can be developed for the considered applications.