Azure Data Factory Blog Hub

Building with Azure Data Factory

Azure Data Factory handles orchestration and data movement on many Azure data platforms. It is quick to start with. The harder decisions come later, once a factory runs more than a handful of pipelines, and those decisions are the focus of this section.

If you are new to ADF, how to create a data pipeline in ADF goes step by step from the storage account and linked services to monitoring the first run. Once pipelines run every day, incremental loading with watermarks, CDC and tumbling windows is usually the next thing to sort out.

Reaching sources inside a private network can take longer to set up than the pipelines themselves. The self-hosted integration runtime guide covers installation, high availability and firewall rules, and the Snowflake Azure PrivateLink runbook sets out the Azure and Snowflake steps for a private connection, including validation and rollback.

For design choices, ADF vs Synapse Pipelines and when to use Mapping Data Flows compare the options directly. Azure cost management for data workloads shows where the money goes and how budgets, alerts and tags help keep it under control.