Data Deduplication techniques and analysis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Citations (Scopus)

Abstract

Data warehouses are the repositories of data collected from several data sources, which form the backbone of most of the decision support applications. As the data sources are independent, they may adopt independent and potentially inconsistent conventions. In data warehousing applications during ETL (Extraction, Transformation and Loading) or even in OLTP (On Line Transaction Processing) applications we are often encountered with duplicate records in table. Moreover, data entry mistakes at any of these sources introduce more errors. Since high quality data is essential for gaining the confidence of users of decision support applications, ensuring high data quality is critical to the success of data warehouse implementations. Therefore, significant amount of time and money are spent on the process of detecting and correcting errors and inconsistencies. The process of cleaning dirty data is often referred to as data cleaning. To make the table data consistent and accurate we need to get rid of these duplicate records from the table. In this paper we discuss different strategies of Deduplication along with their pros and cons and some of methods used to prevent duplication in database. In addition, we have made performance evaluation with Microsoft SQL-Server 2008 on Food Mart and AdventureDB Warehouses.

Original languageEnglish
Title of host publicationProceedings - 3rd International Conference on Emerging Trends in Engineering and Technology, ICETET 2010
Pages664-668
Number of pages5
DOIs
Publication statusPublished - 2010
Event3rd International Conference on Emerging Trends in Engineering and Technology, ICETET 2010 - Goa, India
Duration: 19-11-201021-11-2010

Conference

Conference3rd International Conference on Emerging Trends in Engineering and Technology, ICETET 2010
CountryIndia
CityGoa
Period19-11-1021-11-10

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Data Deduplication techniques and analysis'. Together they form a unique fingerprint.

Cite this