A Framework for Mediated Reality Based on Freeform Manipulation of a Real Object

Abstract

With recent advances in computer vision technology, it is possible to accurately track objects in an image and estimate camera pose in real time, which has facilitated various studies and applications related to mixed reality. On the other hand, studies on mediated reality, which can provide more immersive experiences and richer interactions by not only simply mixing the real world with the virtual world but also manipulating the real world, are also gradually increasing. In these studies, there are methods for changing material properties of real objects, and deforming real objects based on physical simulation according to user-defined physical properties as if virtual forces are applied. In regards to methods for manipulating real objects as if the original shape had changed, studies have been scarce. In this dissertation, a real-time mediated reality framework to provide freeform deformation of the shape of a real object is proposed, and the feasibility and the usefulness of the framework is shown by implementing mediated reality applications for manipulating rigid and non-rigid objects. The techniques such as object tracking, model alignment, and model deformation, required for mediated reality to freely manipulate the shape of the object, have matured enough to be individually applied to practical real-time applications. However, it is not easy to effectively integrate these techniques to implement real-time mediated reality. The proposed framework defines and abstracts the minimum essential modules (registration, manipulation, and rendering) and the input/output data for each. The modules can be flexibly configured and switched with real-time techniques depending on the scenario of mediated reality or the characteristics of the target object, and the modules are designed as threads considering concurrency so that the entire framework can operate in real time. We propose a method that provides mediated reality in which a rigid body object can be manipulated freely by applying 3D model-based tracking method and mesh deformation method to the proposed framework. In the rendering module, the texture of the object model deformed according to the user input is updated from the object region of the camera frame. Then, the region of the frame revealed by the deformation of the object, which was occluded by the object, is filled with inpainting. Finally, the deformed model is rendered on the frame image. Through the proposed method, it is shown that the user can manipulate a real object with a complex shape freely and three-dimensionally. Next, we propose a method for freeform manipulation of non-rigid objects. Among non-rigid objects, we focused on the human face, the most widely used and important subject in media contents. Facial feature detection method and fitting method for 3D morphable model (3DMM) are used for the registration module, and mesh deformation method is applied to the manipulation module. Texture update for the object model and inpainting are processed like those in manipulation of rigid body objects. For real-time processing, additional acceleration schemes are made over to control concurrency of the modules in the framework. The 3D--2D correspondence searching method for boundary facial landmarks in face model fitting, which requires a large amount of computation, is optimized and further parallelized. Texture updates, merging of fitted and deformed models, and rendering the merged model are handled efficiently with a single programmable shader. Through experiments, it was confirmed that a human face can be freely and three-dimensionally manipulated through the proposed face manipulation method, unlike existing 2D face manipulation applications. In addition, through the application in which a virtual object manipulates the real face, it showed the interactions such that the real world not only receives actions from users but also from the virtual world.