Highlight detection is one of the most important clues for semantic analysis of soccer video. A highlight scene always has a corresponding replay scene including a shot changed from zoom-out to zoom-in. Therefore, by analyzing the structure of soccer videos, we propose an efficient method for automatically detecting highlight scenes in soccer videos.