Design of Edge-guided Interpolator Using Spectral Correlation and Its Application to Demosaicking

Abstract

Color images require multiple pixel values, such as red, green, and blue, for each pixel position. However, most digital cameras use color filter array (CFA) over the sensor so as to reduce the cost; the resulting images have only one color component level at each pixel position. Therefore, the captured mosaic-like image has to be converted to a full-color image based on the known values of adjacent pixels. This process is called demosaicking or CFA interpolation. Demosaicking algorithms depend considerably on the CFA pattern layout used. The most commonly used CFA pattern is Bayer filter mosaic CFA, in which green components are sampled twice as densely as the red or blue components. Both spatial and spectral correlations play important roles in CFA interpolation. A large portion of demosaicking algorithms has made good use of spatial correlation by using the information from neighboring pixels and considering local features. Spectral correlation has been used in the constant hue model, which is based on the assumption that an object of constant color will have a constant difference (CD) even though lighting variations may change the measured values, i.e. values (hue) are quite constant within a small region. The simplest way to interpolate CFA data is to treat each color plane separately, as if interpolating a gray-scale image. Such methods exploit spatial correlation only. Although most existing interpolation algorithms could be used for this purpose, such as bilinear or bicubic interpolations, they would produce false color artifacts in detailed regions and/or zippering artifacts near sharp edges. In order to improve the demosaicked image quality, many recently proposed algorithms have included interpolation on the CD planes. However, the way to use spectral correlation by interpolating red and blue components on the CD planes is too limited utilization. Correlations between color planes are very high, and they are important and useful information in order to obtain a high quality full color image from a given CFA image. The spectral correlation is worthwhile to be fully used for demosaicking. That is, it should be utilized when generating predictor and classifying edge direction for green plane estimation as well as red and blue planes interpolation. In this dissertation, three demosaicking algorithms are presented. The first algorithm, called demosaicking using geometric duality and dilated directional differentiation (GD4), uses two cost terms: the interpolation error of a low resolution (LR) image based on geometric duality and the dilated directional differentiation on CD planes. Since a given high resolution (HR) image and its LR image obtained by sampling have similar edge properties, GD4 first computes the interpolation errors for the candidate directions in the LR image and exploits them as a cost term for the direction. In addition, the interpolation direction can be determined accurately even in the vicinity of object boundaries by dilating the directional differentiations of the CD values. Some pixels, which are in the neighborhood of an edge but classified into a flat region by simple edge detection method like Sobel filter, are reclassified by dilation. To generate directional predictors, a modified Taylor approximation is utilized. The modified Taylor approximation is reinforced by considering gradient values for each direction. By combining this edge classifier and the weighted sum of the predictors obtained by the modified Taylor approximation, missing pixels are estimated. GD4 includes a post-processing step, in which a weighted sum strategy on CD plane is used, where the gradient values computed for generating the directional predictors are re-used as the weighting factors. An error-compensated demosaicking algorithm (ECDA) is secondly proposed in the dissertation. ECDA has three contributions: a predictor generation considering errors, an edge classifier using the constant CD model, and a post-processing scheme using a weighted sum strategy. The new predictors are generated based on the interpolation error theorem, and the predictor generation method can be used as a sound alternative to other predictor generation schemes, such as bilinear or Laplacian interpolations. The proposed edge classifier first determines whether the target pixel is located in a strong edge region or not. If so, the pixel is estimated by one of two directional predictors. Otherwise, the weighted sum strategy is used. After populating three color planes, a post-processing is applied to enhance the demosaicked image quality and reduce demosaicking artifacts. Finally, this dissertation describes Taylor series and adaptive fusion strategy (TSAFS). TSAFS consists of three steps. In the first step, the two directional predictors are generated. Specifically, the directional predictors are obtained by computing the temporal directional predictors using Taylor series-based interpolator, generating two CD planes with the temporal directional predictors, and applying a low-pass filter. In the second step, the two predictors are fused using two weighting factors, namely, the gradient weighting factor and the directional credibility weighting factor. After estimating the green plane, it is updated in the third step, and this updated green plane is used to populate the red and blue planes. The proposed algorithms have been tested on fifty images, including twenty-four Kodak images and twenty-six Laurent Condat (LC) images. The simulation has demonstrated that the proposed algorithms provide better demosaicked images. They have achieved higher color peak signal-to-noise ratio (CPSNR) and lower S-CIE L*a*b* scores, and demosaicking artifact have been significantly reduced compared to nine existing algorithms. Among the three algorithms, TSAFS has showed the best results. |컬러 영상은 각각의 화소 위치에서 빨간색, 초록색, 파란색 등 여러 색상 성분의 값들이 필요하다. 하지만 대부분의 디지털 카메라는 비용을 줄이기 위해 색상 필터 배열(Color Filter Array, CFA)이 덮여있는 구조를 사용한다. 그 결과 우리가 얻는 영상은 각각의 화소 위치에서 하나의 색상 성분 값만을 가지고 있는 모자이크 형태를 갖는다. 따라서, 이렇게 얻은 영상은 주변 위치에서 이용가능한 동일 색상성분의 값들을 이용하여 완전한 컬러 영상으로 변환되어야 한다. 이러한 프로세스는 디모자이킹 혹은 CFA 보간이라고 한다. 디모자이킹 알고리듬은 사용된 CFA 패턴에 따라 크게 달라진다. 가장 보편적으로 사용되는 CFA 패턴은 베이어 CFA인데, 이는 빨간색 혹은 파란색 성분보다 두 배 많은 초록색 성분을 갖는다. 공간적 상관관계(spatial correlation)와 스펙트럼의 상관관계(spectral correlation)는 디모자이킹에서 중요한 역할을 한다. 대부분의 디모자이킹 알고리즘은 주변에 있는 화소 정보를 이용하고 지역적 특성을 고려하는 방식으로 공간적 상관관계를 잘 이용해왔다. 스펙트럼의 상관관계는 주로 불변 색조 모델(constant hue model)에서 사용되어 왔는데, 불변 색조 모델은 빛의 변화량이 측정된 값들을 변화시키더라도 동일 색상의 물체는 일정한 색상비(color ratio, CR) 혹은 색상차(color difference, CD)를 갖는다는 가정을 기초로 한다. CFA 영상을 보간하여 완전한 컬러 영상으로 만드는 가장 간단한 방법은 각각의 색평면(color plane)을 마치 별개의 흑백 영상처럼 따로따로 다루는 것이다. 이런 방법은 공간적 상관관계만을 이용한다. 기존의 흑백 영상을 보간하기 위해 사용된 많은 방법들이 사용될 수 있지만, 이런 방식은 세세한 영역에서는 거짓 색상 인위물(false color artifacts)을 에지 영역에서는 지퍼 인위물(zippering artifacts)을 발생시켜 화질 열화가 발생한다. 디모자이킹된 영상의 화질을 향상시키기 위해 최근의 많은 알고리즘은 색상차 평면 에서 보간하는 방식을 사용해왔다. 하지만 단순히 색상차 평면에서 빨강 평면과 파랑 평면을 보간하는 것은 너무 제한적으로 스펙트럼의 상관관계를 이용하는 것이다. 색상 평면간의 상관 관계는 상당히 크고, 주어진 CFA 영상에서 고품질의 완전한 컬러 영상을 얻기 위한 매우 중요하고 유용한 정보이다. 즉, 스펙트럼의 상관관계는 빨강 평면과 파랑 평면을 보간하는데만 사용할 것이 아니라 방향적 예측값을 생성하고 에지 방향을 분류할 때도 사용함으로써 충분히 활용될 가치가 있다. 본 논문에서 세 개의 디모자이킹 알고리즘이 제시된다. 첫 번째로, 기하학적 쌍대성과 팽창된 방향 차분값을 사용하는 디모자이킹 (demosaicking using geometric duality and dilated directional differentiation, GD4)알고리즘은 기하학적 쌍대성에 기반한 저해상도 영상의 보간 오차와 색상차 평면에서 팽창된 방향 차분값을 두 개의 비용 항목으로 이용한다. 고해상도 영상과 이를 샘플링(sampling)하여 얻은 저해상도 영상은 비슷한 에지 특성을 가지고 있기 때문에 GD4는 우선 저해상도 영상에서 후보 에지 방향에 대한 보간 오차를 계산하고 이를 그 방향에 대한 첫 번째 비용항목으로 사용한다. 또한, 색상차이 값들의 방향 차분값들을 팽창시킴으로써 객체의 경계 주변에서도 보간 방향을 더욱 정확하게 얻을 수 있다. 에지의 주변에 있지만 소벨(Sobel) 필터와 같은 단순한 에지 탐색 방법에 의해 평평한 영역으로 분류된 화소들은 팽창에 의해 재분류된다. 지향성 예측값(directional predictors)을 생성하기 위해 테일러 근사로 얻은 예측값들의 가중합 방식을 사용했다. 이 방식은 각 방향에 대한 기울기값을 고려하여 테일러 근사의 성능을 강화시킨다. 지금까지 설명된 에지 분류기와 지향성 예측값을을 결합하여 화소 값들을 추정한다. GD4는 후처리 과정을 가지고 있는데, 색상차 평면에서 가중합 방법을 적용하는 방법을 사용하였다. 오차보상 디모자이킹 알고리즘 (error-compensated demosaicking algorithm, ECDA)는 본 논문에서 제안하는 두 번째 알고리즘이다. ECDA는 오차를 고려한 예측값 생성 방법, 불변 색상차 모델을 이용한 에지 분류기, 작은 영역에서 가중합 방식을 사용한 후처리 방법을 사용한다. 제안된 방식에서 지향성 예측값들은 보간 오차 정리에 기초하여 생성된다. 이 예측값 생성 방법은 쌍일차(bilinear) 보간 또는 라플라시안(Laplacian) 보간과 같은 기존 방법들의 훌륭한 대안이 될 수 있다. 제안된 에지 분류기는 우선 현재 픽셀이 강한 에지 영역에 놓여있는지 아닌지를 판단한다. 강한 에지 영역에 있다면 두 개의 지향성 예측값 중 하나의 값으로 보간되고, 그렇지 않으면 두 개의 지향성 예측값을 가중합하여 보간한다. 세 개의 평면을 모두 보간한 후, 전체적인 화질을 향상시키고 디모자이킹 인위물(demosaicking artifacts)를 줄이기 위해 후처리 과정이 적용된다. 마지막 알고리즘은 테일러 급수와 적응적 결합 전략(Taylor series and adaptive fusion strategy)이다. TSAFS는 세 개의 과정으로 구성되어 있는데 첫 번째 과정에서 두 개의 지향성 예측값들이 생성된다. 지향성 예측값은 테일러 급수기반 보간기를 사용하여 계산하고, 이 초기 지향성 예측값들을 이용해 두 개의 색상차 평면을 생성한 후, 저주파 필터링을 거침으로써 생성된다. 두 번째 과정에서 두 개의 지향성 예측값들이 두 개의 가중치 요소를 이용하여 결합된다. 두 개의 가중치 요소로 기울기와 방향적 신뢰도를 사용한다. 위의 과정을 이용해 녹색 평면을 추정하고, 세 번째 과정에서 이를 갱신한다. 갱신된 녹색평면을 이용하여 빨강색평면과 파란색평면을 채운다. 코닥(Kodak) 영상 24개와 로랑 콩다(Laurent Condat, LC) 영상 26개에 대해 제안된 세 개의 알고리즘을 실험했다. 실험결과는 제안된 알고리즘들이 더 나은 화질의 컬러영상을 제공하는 것을 보여주었다. 9개의 기존 알고리즘과 비교해서 객관적 화질 평가에서 가장 좋은 결과를 보여줬고 결과 영상의 비교는 디모자이킹 인위물이 상당히 줄었다는 것을 보여주었다.; the resulting images have only one color component level at each pixel position. Therefore, the captured mosaic-like image has to be converted to a full-color image based on the known values of adjacent pixels. This process is called demosaicking or CFA interpolation. Demosaicking algorithms depend considerably on the CFA pattern layout used. The most commonly used CFA pattern is Bayer filter mosaic CFA, in which green components are sampled twice as densely as the red or blue components. Both spatial and spectral correlations play important roles in CFA interpolation. A large portion of demosaicking algorithms has made good use of spatial correlation by using the information from neighboring pixels and considering local features. Spectral correlation has been used in the constant hue model, which is based on the assumption that an object of constant color will have a constant difference (CD) even though lighting variations may change the measured values, i.e. values (hue) are quite constant within a small region. The simplest way to interpolate CFA data is to treat each color plane separately, as if interpolating a gray-scale image. Such methods exploit spatial correlation only. Although most existing interpolation algorithms could be used for this purpose, such as bilinear or bicubic interpolations, they would produce false color artifacts in detailed regions and/or zippering artifacts near sharp edges. In order to improve the demosaicked image quality, many recently proposed algorithms have included interpolation on the CD planes. However, the way to use spectral correlation by interpolating red and blue components on the CD planes is too limited utilization. Correlations between color planes are very high, and they are important and useful information in order to obtain a high quality full color image from a given CFA image. The spectral correlation is worthwhile to be fully used for demosaicking. That is, it should be utilized when generating predictor and classifying edge direction for green plane estimation as well as red and blue planes interpolation. In this dissertation, three demosaicking algorithms are presented. The first algorithm, called demosaicking using geometric duality and dilated directional differentiation (GD4), uses two cost terms: the interpolation error of a low resolution (LR) image based on geometric duality and the dilated directional differentiation on CD planes. Since a given high resolution (HR) image and its LR image obtained by sampling have similar edge properties, GD4 first computes the interpolation errors for the candidate directions in the LR image and exploits them as a cost term for the direction. In addition, the interpolation direction can be determined accurately even in the vicinity of object boundaries by dilating the directional differentiations of the CD values. Some pixels, which are in the neighborhood of an edge but classified into a flat region by simple edge detection method like Sobel filter, are reclassified by dilation. To generate directional predictors, a modified Taylor approximation is utilized. The modified Taylor approximation is reinforced by considering gradient values for each direction. By combining this edge classifier and the weighted sum of the predictors obtained by the modified Taylor approximation, missing pixels are estimated. GD4 includes a post-processing step, in which a weighted sum strategy on CD plane is used, where the gradient values computed for generating the directional predictors are re-used as the weighting factors. An error-compensated demosaicking algorithm (ECDA) is secondly proposed in the dissertation. ECDA has three contributions: a predictor generation considering errors, an edge classifier using the constant CD model, and a post-processing scheme using a weighted sum strategy. The new predictors are generated based on the interpolation error theorem, and the predictor generation method can be used as a sound alternative to other predictor generation schemes, such as bilinear or Laplacian interpolations. The proposed edge classifier first determines whether the target pixel is located in a strong edge region or not. If so, the pixel is estimated by one of two directional predictors. Otherwise, the weighted sum strategy is used. After populating three color planes, a post-processing is applied to enhance the demosaicked image quality and reduce demosaicking artifacts. Finally, this dissertation describes Taylor series and adaptive fusion strategy (TSAFS). TSAFS consists of three steps. In the first step, the two directional predictors are generated. Specifically, the directional predictors are obtained by computing the temporal directional predictors using Taylor series-based interpolator, generating two CD planes with the temporal directional predictors, and applying a low-pass filter. In the second step, the two predictors are fused using two weighting factors, namely, the gradient weighting factor and the directional credibility weighting factor. After estimating the green plane, it is updated in the third step, and this updated green plane is used to populate the red and blue planes. The proposed algorithms have been tested on fifty images, including twenty-four Kodak images and twenty-six Laurent Condat (LC) images. The simulation has demonstrated that the proposed algorithms provide better demosaicked images. They have achieved higher color peak signal-to-noise ratio (CPSNR) and lower S-CIE L*a*b* scores, and demosaicking artifact have been significantly reduced compared to nine existing algorithms. Among the three algorithms, TSAFS has showed the best results.