Extraction of User-Defined Data Blocks Using the Regularity of Dynamic Web Pages

Abstract

This paper proposes an enhanced method of Web information extraction by exploiting general phenomena that Web pages in a site tend to have common structures and dynamic Web pages contain multiple data blocks with repeating structural patterns. By considering this kind of regularity in dynamic Web pages, we develop a data block extraction system which basically adopts a supervised learning mechanism with training and extraction phases. In the training phase, the user selects and specifies a data block and the extraction rules for the block are generated. During this phase, the block is defined with the HTML DOM-tree path to the block and the tag sequence of the block. In the extraction phase, the rules are applied to the target pages to extract those blocks that have similar structure as the user-defined block. A series of experiments are performed to evaluate the user-defined data block extraction method for a number of well-known Web sites with dynamic Web pages, and the result of evaluation is satisfactory with high precision and recall measures.