TY - GEN
T1 - Data cleansing techniques for large enterprise datasets
AU - Prasad, K. Hima
AU - Faruquie, Tanveer A.
AU - Joshi, Sachindra
AU - Chaturvedi, Snigdha
AU - Subramaniam, L. Venkata
AU - Mohania, Mukesh
PY - 2011
Y1 - 2011
N2 - Data quality improvement is an important aspect of enterprise data management. Data characteristics can change with customers, with domain and geography making data quality improvement a challenging task. Data quality improvement is often an iterative process which mainly involves writing a set of data quality rules for standardization and elimination of duplicates that are present within the data. Existing data cleansing tools require a fair amount of customization whenever moving from one customer to another and from one domain to another. in this paper, we present a data quality improvement tool which helps the data quality practitioner by showing the characteristics of the entities present in the data. The tool identifies the variants and synonyms of a given entity present in the data which is an important task for writing data quality rules for standardizing the data. We present a ripple down rule framework for maintaining data quality rules which helps in reducing the services effort for adding new rules. We also present a typical workflow of the data quality improvement process and show the usefulness of the tool at each step. We also present some experimental results and discussions on the usefulness of the tools for reducing services effort in a data quality improvement.
AB - Data quality improvement is an important aspect of enterprise data management. Data characteristics can change with customers, with domain and geography making data quality improvement a challenging task. Data quality improvement is often an iterative process which mainly involves writing a set of data quality rules for standardization and elimination of duplicates that are present within the data. Existing data cleansing tools require a fair amount of customization whenever moving from one customer to another and from one domain to another. in this paper, we present a data quality improvement tool which helps the data quality practitioner by showing the characteristics of the entities present in the data. The tool identifies the variants and synonyms of a given entity present in the data which is an important task for writing data quality rules for standardizing the data. We present a ripple down rule framework for maintaining data quality rules which helps in reducing the services effort for adding new rules. We also present a typical workflow of the data quality improvement process and show the usefulness of the tool at each step. We also present some experimental results and discussions on the usefulness of the tools for reducing services effort in a data quality improvement.
UR - http://www.scopus.com/inward/record.url?scp=80051933614&partnerID=8YFLogxK
U2 - 10.1109/SRII.2011.26
DO - 10.1109/SRII.2011.26
M3 - Conference contribution
SN - 9780769543710
T3 - Proceedings - 2011 Annual SRII Global Conference, SRII 2011
SP - 135
EP - 144
BT - Proceedings - 2011 Annual SRII Global Conference, SRII 2011
T2 - 2011 Annual SRII Global Conference, SRII 2011
Y2 - 30 March 2011 through 2 April 2011
ER -