Duplicate Table Discovery with Xash
| dc.contributor.author | Koch, Maximilian | |
| dc.contributor.author | Esmailoghli, Mahdi | |
| dc.contributor.author | Auer, Sören | |
| dc.contributor.author | Abedjan, Ziawasch | |
| dc.contributor.editor | König-Ries, Birgitta | |
| dc.contributor.editor | Scherzinger, Stefanie | |
| dc.contributor.editor | Lehner, Wolfgang | |
| dc.contributor.editor | Vossen, Gottfried | |
| dc.date.accessioned | 2023-02-23T13:59:48Z | |
| dc.date.available | 2023-02-23T13:59:48Z | |
| dc.date.issued | 2023 | |
| dc.description.abstract | Data lakes are typically lightly curated and as such prone to data quality problems and inconsistencies. In particular, duplicate tables are common in most repositories. The goal of duplicate table detection is to identify those tables that display the same data.Comparing tables is generally quite expensive as the order of rows and columns might differ for otherwise identical tables. In this paper, we explore the application of Xash, a hash function previously proposed for the discovery of multi-column join candidates, for the use case of duplicate table detection. With Xash, it is possible to generate a so-called super key, which serves like a bloom filter and instantly identifies the existence of particular cell values. We show that using Xash it is possible to speed up the duplicate table detection process significantly. In comparison to other hash functions, such as SimHash and other competitors, Xash results in fewer false positive candidates. | en |
| dc.identifier.doi | 10.18420/BTW2023-18 | |
| dc.identifier.isbn | 978-3-88579-725-8 | |
| dc.identifier.uri | https://dl.gi.de/handle/20.500.12116/40322 | |
| dc.language.iso | en | |
| dc.publisher | Gesellschaft für Informatik e.V. | |
| dc.relation.ispartof | BTW 2023 | |
| dc.relation.ispartofseries | Lecture Notes in Informatics (LNI) - Proceedings, Volume P-331 | |
| dc.subject | data discovery | |
| dc.subject | data lakes | |
| dc.subject | duplicate table detection | |
| dc.title | Duplicate Table Discovery with Xash | en |
| dc.type | Text/Conference Paper | |
| gi.citation.endPage | 390 | |
| gi.citation.publisherPlace | Bonn | |
| gi.citation.startPage | 367 | |
| gi.conference.date | 06.-10. März 2023 | |
| gi.conference.location | Dresden, Germany |
Dateien
Originalbündel
1 - 1 von 1
Vorschaubild nicht verfügbar
- Name:
- B4-1.pdf
- Größe:
- 590.96 KB
- Format:
- Adobe Portable Document Format