Covering essential concepts, techniques, and tools, this book is a compass for every data professional seeking to create value and transform their business.
Stéphane Heckel, Data Sommelier
1998, Ignition
My journey into the data integration world started in 1998 when the company I served as a database consultant was acquired by an American software vendor specializing in this field. Back then, the idea of a graphical ETL solution seemed far-fetched; drawing lines with a mouse between sources and target components to craft data movement interfaces for analytical applications appeared unconventional. We were accustomed to developing code in C++, ensuring the robustness and performance of applications. Data warehouses were fed through batch-mode SQL processes, with orchestration and monitoring managed in shell scripts.
The 3Vs and more !
Little did we anticipate that this low-code, no-code ETL solution would evolve into a standard embraced by global companies, marking the onset of the data integration revolution. The pace was swift1. Growing data volumes, expanding sources to profile, operational constraints, and tightening deadlines propelled changes in data tools, architectures and practices. Real-time data integration, data storage, data quality, metadata and master data management, enhanced collaboration between business and technical teams through governance programs, and the development of cloud-based applications became imperative challenges for data teams striving for operational excellence.
Ready for the AI Era !
The past 25 years flashed by, and the revolution persists, keeping my passion for data ablaze. The rise of artificial intelligence, exemplified by the success of ChatGPT, necessitates vast data processing for model building. This, in turn, compels a deeper reliance on data engineering techniques. Authored by seasoned data professionals with extensive project deployments, this book offers a comprehensive overview of data integration. My sincere gratitude to them, Pierre-Yves, Emeric, Raphaël and Mehdi for crafting this invaluable resource! Covering essential concepts, techniques, and tools, this book is a compass for every data professional seeking to create value and transform their business. May your reading journey be as enjoyable as mine!
- The 3Vs of Big Data: Volume, Velocity, Variety ↩︎