Focused crawling for both topical relevance and qualify of medical information

Thanh Tin Tang*, David Hawking, Nick Craswell, Kathy Griffiths

*Corresponding author for this work

    Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

    43 Citations (Scopus)

    Abstract

    Subject-specific search facilities on health sites are usually built using manual inclusion and exclusion rules. These can be expensive to maintain and often provide incomplete coverage of Web resources. On the other hand, health information obtained through whole-of-Web search may not be scientifically based and can be potentially harmful. To address problems of cost, coverage and quality, we built a focused crawler for the mental health topic of depression, which was able to selectively fetch higher quality relevant information. We found that the relevance of unfetched pages can be predicted based on link anchor context, but the quality cannot. We therefore estimated quality of the entire linking page, using a learned IR-style query of weighted single words and word pairs, and used this to predict the quality of its links. The overall crawler priority was determined by the product of link relevance and source quality. We evaluated our crawler against baseline crawls using both relevance judgments and objective site quality scores obtained using an evidence-based rating scale. Both a relevance focused crawler and the quality focused crawler retrieved twice as many relevant pages as a breadth-first control. The quality focused crawler was quite effective in reducing the amount of low quality material fetched while crawling more high quality content, relative to the relevance focused crawler. Analysis suggests that quality of content might be improved by post-filtering a very big breadth-first crawl, at the cost of substantially increased network traffic.

    Original languageEnglish
    Title of host publicationCIKM'05 - Proceedings of the 14th ACM International Conference on Information and Knowledge Management
    Pages147-154
    Number of pages8
    Publication statusPublished - 2005
    EventCIKM'05 - Proceedings of the 14th ACM International Conference on Information and Knowledge Management - Bremen, Germany
    Duration: 31 Oct 20055 Nov 2005

    Publication series

    NameInternational Conference on Information and Knowledge Management, Proceedings

    Conference

    ConferenceCIKM'05 - Proceedings of the 14th ACM International Conference on Information and Knowledge Management
    Country/TerritoryGermany
    CityBremen
    Period31/10/055/11/05

    Fingerprint

    Dive into the research topics of 'Focused crawling for both topical relevance and qualify of medical information'. Together they form a unique fingerprint.

    Cite this