Data cleansing techniques for large enterprise datasets

K. Hima Prasad*, Tanveer A. Faruquie, Sachindra Joshi, Snigdha Chaturvedi, L. Venkata Subramaniam, Mukesh Mohania

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

19 Citations (Scopus)

Abstract

Data quality improvement is an important aspect of enterprise data management. Data characteristics can change with customers, with domain and geography making data quality improvement a challenging task. Data quality improvement is often an iterative process which mainly involves writing a set of data quality rules for standardization and elimination of duplicates that are present within the data. Existing data cleansing tools require a fair amount of customization whenever moving from one customer to another and from one domain to another. in this paper, we present a data quality improvement tool which helps the data quality practitioner by showing the characteristics of the entities present in the data. The tool identifies the variants and synonyms of a given entity present in the data which is an important task for writing data quality rules for standardizing the data. We present a ripple down rule framework for maintaining data quality rules which helps in reducing the services effort for adding new rules. We also present a typical workflow of the data quality improvement process and show the usefulness of the tool at each step. We also present some experimental results and discussions on the usefulness of the tools for reducing services effort in a data quality improvement.

Original languageEnglish
Title of host publicationProceedings - 2011 Annual SRII Global Conference, SRII 2011
Pages135-144
Number of pages10
DOIs
Publication statusPublished - 2011
Externally publishedYes
Event2011 Annual SRII Global Conference, SRII 2011 - San Jose, CA, United States
Duration: 30 Mar 20112 Apr 2011

Publication series

NameProceedings - 2011 Annual SRII Global Conference, SRII 2011

Conference

Conference2011 Annual SRII Global Conference, SRII 2011
Country/TerritoryUnited States
CitySan Jose, CA
Period30/03/112/04/11

Fingerprint

Dive into the research topics of 'Data cleansing techniques for large enterprise datasets'. Together they form a unique fingerprint.

Cite this