The Azure Data Lakehouse Toolkit Building and Scaling Data Lakehouses on Azure with Delta Lake, Apache Spark, Databricks, Synapse Analytics, and Snowflake

CHAPTER 4 Synapse Analytics Synapse Analytics is a data and analytics platform as a service that unifies data integration, data warehousing, big data analytics, reporting, CI/CD, and much more within the Modern Azure Data platform. It supports a variety of tools such as workspaces

Trang 2

The Azure DataLakehouse Toolkit

Building and Scaling Data

Lakehouses on Azure with DeltaLake, Apache Spark, Databricks,Synapse Analytics, and SnowflakeRon L’Esteve

The Azure Data Lakehouse Toolkit: Building and Scaling DataLakehouses on Azure

with Delta Lake, Apache Spark, Databricks, Synapse Analytics, andSnowflake

Ron L’EsteveChicago, IL, USA

ISBN-13 (pbk): 978-1-4842-8232-8

This work is subject to copyright All rights are reserved by the Publisher,whether the whole or part of the material is concerned, specifically therights of translation, reprinting, reuse of illustrations, recitation,

broadcasting, reproduction on microfilms or in any other physical way, andtransmission or information storage and retrieval, electronic adaptation,

Trang 3

computer software, or by similar or dissimilar methodology now known orhereafter developed.

Trademarked names, logos, and images may appear in this book Ratherthan use a trademark symbol with every occurrence of a trademarked name,logo, or image we use the names, logos, and images only in an editorialfashion and to the benefit of the trademark owner, with no intention ofinfringement of the trademark.

The use in this publication of trade names, trademarks, service marks, andsimilar terms, even if they are not identified as such, is not to be taken as anexpression of opinion as to whether or not they are subject to proprietaryrights.

While the advice and information in this book are believed to be true andaccurate at the date of publication, neither the authors nor the editors northe publisher can accept any legal responsibility for any errors or omissionsthat may be made The publisher makes no warranty, express or implied,with respect to the material contained herein.

Managing Director, Apress Media LLC: Welmoed SpahrAcquisitions Editor: Jonathan Gennick

Development Editor: Laura BerendsonCoordinating Editor: Jill BalzanoCover designed by eStudioCalamar

Cover image designed by Freepik (www.freepik.com)

Distributed to the book trade worldwide by Springer Science+BusinessMedia New York, 1 New York Plaza, Suite 4600, New York, NY 10004-1562, USA Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@

springer-sbm.com, or visit www.springeronline.com Apress Media, LLC isa California LLC and the sole member (owner) is Springer Science +

Trang 4

Business Media Finance Inc (SSBM Finance Inc) SSBM Finance Inc is a

Delaware corporation.

For information on translations, please e-mailbooktranslations@springernature.com; for reprint,paperback, or audio rights, please e-mail

Apress titles may be purchased in bulk for academic, corporate, or

promotional use eBook versions and licenses are also available for mosttitles For more information, reference our Print and eBook Bulk Sales webpage at http://www.apress.com/bulk-sales.

Any source code or other supplementary material referenced by the authorin this book is available to readers on GitHub via the book’s product page,located at www.apress.com/ For more detailed information, please visithttp://www.apress.com/source- code.

Printed on acid-free paper

For Cayden and Christina

Table of Contents

About the Author

��xv About the Technical Reviewer��xvii

��xix Introduction

Trang 5

��xxi Part I: GettingStarted

�� 1Chapter 1: The Data Lakehouse Paradigm

�� 3

�� 4

Ingestion and Processing

�� 6

Data Factory

��

Trang 6

�� 6

�� 10

Functions and Logic Apps

�� 11

Synapse Analytics Serverless Pools

�� 12Stream Analytics

Messaging Hubs

��15

Trang 7

Storing and Serving

Delta Lake

�� 17

Synapse Analytics Dedicated SQL Pools

�� 19

Relational Database

�� 21

Non-relational Databases

�� 23

�� 26

Trang 8

�� 28

Analysis Services

�� 28Power BI

�� 30

Power Apps

�� 33

Table of ConTenTsAdvanced Analytics

�� 33

Trang 9

Cognitive Services

�� 34Machine Learning

�� 35Continuous Integration, Deployment, and Governance

�� 36

�� 39

Summary�� 41

Trang 10

Part II: Data Platforms

�� 43

Chapter 2: Snowflake

�� 45

�� 46

�� 48

�� 49

Azure Key Vault

��

Trang 11

�� 49

Azure Private Link

�� 50Applications

�� 50

Replication and Failover

�� 50

Data Integration with Azure

�� 51

Data Lake Storage Gen2

�� 51

Real-Time Data Loading with ADLS gen2

��

Trang 12

�� 53

Data Factory

�� 54

�� 55

Data Transformation

�� 63

Column-Level Security

�� 63

Trang 13

Row-Level Security

�� 65

Access History

�� 66

Object Tagging

�� 66

�� 67

Direct Share

�� 69

Data Marketplace

��

Trang 14

�� 70Data Exchange

�� 70

Table of ConTenTs

Continuous Integration and Deployment

�� 71Jenkins

�� 71

Azure DevOps

�� 72

��

Trang 15

�� 73

Power BI

�� 73

Delta Lake, Machine Learning, and Constraints

�� 78

Delta Lake

�� 78

Machine Learning

�� 79Constraints

�� 80

Summary��

Trang 16

�� 80

Chapter 3: Databricks

�� 83

�� 84

Data Science and Engineering

�� 84

Machine Learning

�� 91SQL

�� 93

Trang 17

�� 96

�� 100

Mount Data Lake Storage Gen2 Account

�� 101

Delta Lake

�� 114

�� 115

Real-Time Analytics

��

Trang 18

�� 117Advanced Analytics

Security and Governance

�� 121

�� 125

Integration with Synapse Analytics

�� 126

Dynamic Data Encryption

�� 127

Data Profile

��

Trang 19

�� 129

Query Profile

�� 129

�� 133

Table of ConTenTs

Delta Live Tables Merge

�� 135

Summary��

Trang 20

�� 138

Chapter 4: Synapse Analytics

�� 141

�� 142

�� 144

�� 149

�� 150

Trang 21

�� 154

�� 156

�� 158

Real-Time Analytics

�� 160Structured Streaming

�� 160

Trang 22

Synapse Link

�� 160

Advanced Analytics

�� 163

�� 166

Additional Features

Delta Tables

��

Trang 23

�� 170

Machine Learning

�� 172

SQL Server Integration Services Integration Runtime (SSIS IR)

�� 172

Map Data Tool

�� 173

Data Sharing

�� 175

SQL Incremental

�� 175Constraints

��

Trang 24

�� 176

Summary�� 178

Part III: Apache Spark ELT

�� 183

Chapter 5: Pipelines and Jobs

�� 185

Data Factory

�� 191

viii

Trang 25

Table of ConTenTsMapping Data Flows

�� 191

HDInsight Spark Activity

�� 196

Scheduling and

Monitoring�� 200

Synapse Analytics Workspace

�� 202

Summary�� 206

Chapter 6: Notebook Code

�� 209

Trang 26

�� 210

�� 211

XML�� 217

�� 221

ZIP�� 225

��

Trang 27

�� 227

�� 228

Optimizing Performance

�� 229

Summary�� 232

Part IV: Delta Lake

Chapter 7: Schema

Evolution�� 235Schema Evolution Using Parquet Format

�� 236

Trang 28

Schema Evolution Using Delta Format

�� 240

�� 241

Summary�� 243

Chapter 8: Change Data Feed

�� 245

Create Database and Tables

�� 245

Trang 29

Insert Data into Tables

�� 248

Change Data Capture

�� 249

Streaming Changes

Summary�� 255

Table of ConTenTsChapter 9: Clones

�� 257

Clones��

Trang 30

�� 257

Deep Clones

�� 263

Summary�� 267

Chapter 10: Live Tables

�� 269

Advantages of Delta Live Tables

�� 270

Create a Notebook

��270

Trang 31

Create and Run a Pipeline

�� 274

Schedule a

Pipeline�� 278

Explore Event Logs

Summary�� 283

Chapter 11: Sharing

�� 285

��

Trang 32

�� 286

Share Data

�� 287

Access Data

�� 288

Sharing Data with Snowflake

�� 291

Summary�� 292

Part V: Optimizing Performance

�� 295

Chapter 12: Dynamic Partition Pruning

��

Trang 33

�� 297

�� 299

DPP Commands

Create Cluster

�� 300

Create Notebook and Mount Data Lake

�� 300

Trang 34

Create Fact Table

�� 301Verify Fact Table Partitions

�� 304

Create Dimension Table

�� 305

Table of ConTenTs

Join Results Without DPP Filter

�� 306

Join Results with DPP Filter

�� 308

Summary��

Trang 35

�� 309

Chapter 13: Z-Ordering and Data Skipping

�� 311

Prepare Data in Delta Lake

�� 312

Verify Data in Delta Lake

�� 314

Create Hive Table

�� 317

Run Optimize and Z-Order

Commands�� 318

Verify Data Skipping

��

Trang 36

�� 320Summary�� 325

Chapter 14: Adaptive Query Execution

�� 327

How It Works

�� 327

�� 328

Comparing AQE Performance on Query with Joins

�� 329

Create Datasets

��

Trang 37

Disable AQE

�� 332

Enable AQE

�� 334

Summary�� 338

Chapter 15: Bloom Filter Index

�� 339

How a Bloom Filter Index Works

�� 339

Create a Cluster

��

Trang 38

�� 340

Create a Notebook and Insert Data

�� 341

Enable Bloom Filter Index

�� 343

Tables�� 344

Create a Bloom Filter Index

�� 346

Optimize Table with Z-Order

�� 348

Verify Performance Improvements

��

Trang 39

�� 349

Summary�� 352

Table of ConTenTs

Chapter 16: Hypers pace

�� 354

Create Parquet Files

�� 358Run a Query Without an Index

��

Trang 40

�� 360

Import Hyperspace

Read the Parquet Files to a Data Frame

�� 362

Create a Hyperspace Index

�� 362

Rerun the Query with Hyperspace Index

�� 364

Other Hyperspace Management APIs

�� 365

Summary�� 366