Human-Object Interaction and Projection on Dynamic Objects

Abstract

Real-time interaction through hand gesture, natural object recognition and projection based spatial augmented reality system is an interesting technology for natural user interface (NUI). In the recent evolution of hardware technology, many consumer products using NUIs came on the market. But, those devices have limited functionality in its accuracy, usability and user interface integration. Thus, we introduce a practically usable system for human-object interaction on tabletop environment. For seamless interaction experience between human gestures and objects, a tabletop human-object interaction framework for an RGB-D camera is purposed. This framework constructs synthetic depth information for accurate and efficient hand gesture recognition and object recognition. In initialization stage, the background depth information is trained using input depth data which in this case is the empty tabletop scene. In each frame, the foreground depth map can be obtained using background subtraction algorithm based on the trained background. Then, a synthetic static depth map is composed using objects depth information, which was recognized before. As the synthetic static depth map only includes recognized object’s depth information, new objects and human body parts can be resolved by subtracting from the current foreground depth map and synthetic static depth map. The subtracted data will be treated as the synthetic dynamic depth map. In synthetic static depth map, object recognition and tracking algorithms are applied to update the state. And in synthetic dynamic depth map, we apply hand gesture recognition and object recognition algorithms. As our framework constructs a specific categorized data, an ellipse layered object recognition algorithm is purposed, which features rotation and scale invariance in real-time. And, hand gesture recognition algorithm is used for natural human gestures. The natural human gestures will be used for human-object interaction without using external devices on tabletop. Besides in interaction over real-world objects, visual feedbacks are important for the user’s perception. A projector camera system is often used to achieve these circumstances. However, the latency problem in this system is hardly focused. In tabletop interaction situations, objects often move dynamically for several reasons, which directly visualize the latency to the user. A modified appliance of Kalman filter is proposed to predict the projection position which will reduce the visual latency. Experimental results are shown to justify its necessity for projection based systems on dynamic objects. In this paper, we aimed at a system for seamless human-object interaction tabletop environment. The tabletop human-object interaction framework enabled a simultaneous gesture and object recognition system, which results a seamless interaction to the user. And the prediction method on dynamic objects showed visually improved result, which enhance the augmentation. | 최근 정밀하고 고성능의 하드웨어가 많이 등장함에 따라 사물 상호작용 및 동작 인식을 기반으로 하는 상용 제품들이 등장하고 있다. 이러한 최신 기술들을 적용한 제품이 출시되고 있지만, 실제 시장에서는 아직 혁신적인 변화를 경험하지 않고 있다. 각각의 기술들은 상당히 정확하고 사용 유용성이 검증되고 있지만, 실제 소프트웨어와의 융합과 이종 기술 간 결합의 부재로 인해 실용성 측면에서는 좋은 결과를 내놓지 못하고 있다. 우리는 이러한 기술 중에서 탁상 환경에서의 일반적인 물체와 상호작용할 수 있는 통합 시스템을 구성하였다. 본 시스템에서는 탁상 위에 깊이 영상 카메라를 통해 이용자의 제스처 동작 인식과 사람-사물 사이의 상호작용을 동시에 수행할 수 있는 프레임워크를 제안한다. 본 프레임워크는 사용자의 제스처 입력 활동과 물체 인식 알고리즘을 동시에 수행하기 위하여 카메라로부터 입력 받는 깊이 영상 데이터를 그대로 사용하는 것이 아니라 다른 정보들을 활용하여 각각의 알고리즘을 적용하기 적합한 영상을 합성하여 사용하도록 설계하였다. 초기화 단계에서는 배경 깊이 영상을 학습하고 이후 프레임마다 배경 차분을 통해 전경 깊이 영상을 추출하게 된다. 그 다음 이전에 인식했던 물체의 깊이 정보를 활용하여 정적 합성 깊이 영상을 생성하고, 전경 깊이 영상에서 정적 합성 깊이 영상의 차를 구함으로써 동적 합성 깊이 영상을 획득하게 된다. 각각의 합성 깊이 영상의 특성을 살려 물체 인식, 제스처 인식 알고리즘을 개별적으로 적용하여 효율적인 사람-사물 상호작용 시스템을 구성하게 된다. 또한, 해당 시스템에 알맞은 타원 분할 히스토그램 물체 인식 알고리즘과 물체 추적 알고리즘을 소개하고 이를 검증하였다. 탁상 환경에서 사람-사물 상호작용을 수행할 때, 사용자에게 피드백 정보를 제공하기 위해서 프로젝터-카메라 시스템이 주로 사용하게 된다. 하지만 실험 과정에서 프로젝터로 움직이는 물체에 대해서는 전처리 과정에 의한 지연시간으로 인해 실제 물체의 위치와 프로젝터로 증강한 위치가 어긋나는 점을 확인하였다. 우린 이 지연시간을 측정하여 지연시간이 궁극적으로 사용자가 느끼는 시각적 효과를 충분히 방해할 수 있음을 입증한다. 그리고 Kalman 필터를 응용한 물체 위치 예측 알고리즘을 통해 물체의 위치를 예측하여 투사하는 방법으로 이 문제점을 어느 정도 해소할 수 있음을 보여준다. 본 논문은 탁상 환경에서 사람-사물 상호작용 시스템을 만들기 위한 프레임워크와 각각 요소 알고리즘을 소개하고 상호작용 과정에서 발생할 수 있는 움직이는 물체에 대한 시각적 피드백 문제를 완화하였다.; Real-time interaction through hand gesture, natural object recognition and projection based spatial augmented reality system is an interesting technology for natural user interface (NUI). In the recent evolution of hardware technology, many consumer products using NUIs came on the market. But, those devices have limited functionality in its accuracy, usability and user interface integration. Thus, we introduce a practically usable system for human-object interaction on tabletop environment. For seamless interaction experience between human gestures and objects, a tabletop human-object interaction framework for an RGB-D camera is purposed. This framework constructs synthetic depth information for accurate and efficient hand gesture recognition and object recognition. In initialization stage, the background depth information is trained using input depth data which in this case is the empty tabletop scene. In each frame, the foreground depth map can be obtained using background subtraction algorithm based on the trained background. Then, a synthetic static depth map is composed using objects depth information, which was recognized before. As the synthetic static depth map only includes recognized object’s depth information, new objects and human body parts can be resolved by subtracting from the current foreground depth map and synthetic static depth map. The subtracted data will be treated as the synthetic dynamic depth map. In synthetic static depth map, object recognition and tracking algorithms are applied to update the state. And in synthetic dynamic depth map, we apply hand gesture recognition and object recognition algorithms. As our framework constructs a specific categorized data, an ellipse layered object recognition algorithm is purposed, which features rotation and scale invariance in real-time. And, hand gesture recognition algorithm is used for natural human gestures. The natural human gestures will be used for human-object interaction without using external devices on tabletop. Besides in interaction over real-world objects, visual feedbacks are important for the user’s perception. A projector camera system is often used to achieve these circumstances. However, the latency problem in this system is hardly focused. In tabletop interaction situations, objects often move dynamically for several reasons, which directly visualize the latency to the user. A modified appliance of Kalman filter is proposed to predict the projection position which will reduce the visual latency. Experimental results are shown to justify its necessity for projection based systems on dynamic objects. In this paper, we aimed at a system for seamless human-object interaction tabletop environment. The tabletop human-object interaction framework enabled a simultaneous gesture and object recognition system, which results a seamless interaction to the user. And the prediction method on dynamic objects showed visually improved result, which enhance the augmentation.