In this paper, we propose a Motion Estimation (ME) based on Frame Difference (FD) for frame interpolation in video sequence with luminance variation. Proposed algorithm uses limited blocks whose motion is predicted by FD for ME. The Block average of current and previous frame for the blocks which has no motion variation is used as interpolated block. In experiments, the proposed algorithm shows better performance than conventional algorithms.