Operators from Providers

Concepts covered: DAG definition

The apache-airflow package installs the scheduler and the dag processor, plus the API server behind the UI and the command line. It also installs the Task SDK: you import DAG from airflow.sdk, and dag, task and TaskGroup come from the same place. Operators live elsewhere, in provider packages named apache-airflow-providers-<name>, each released on its own schedule and installed with pip. Even BashOperator and PythonOperator live in a provider, apache-airflow-providers-standard. Core depends on that one, so it is always present, and its import paths show the pattern every other provider follows.

from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator
from airflow.providers.standard.operators.python import PythonOperator
from airflow.providers.http.operators.http import HttpOperator
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

Each path's second segment names the package, so standard comes from apache-airflow-providers-standard. The same reading gives apache-airflow-providers-http for http and apache-airflow-providers-common-sql for common.sql. To reach a system the folder cannot reach today, you install its provider, and airflow providers list prints every installed provider with its version. The provider for a particular warehouse or object store carries the operators for that system plus the plumbing they use to connect, and most SQL providers build on common-sql, so installing one pulls the other in.

The extras on the core package are a shortcut for the same install: pip install "apache-airflow[http]" brings in core and the http provider together, and the extra's name matches the provider's. Either route ends with the same package in the environment, so pick one route and use it everywhere.

What installing changes

Installing a provider adds its import paths to one Python environment, but Airflow runs as several processes, and 2 of them import the DAG file. The dag processor imports it to build the DAG, and each worker imports it when a task starts. A provider installed in one environment and absent from the other produces 2 different failures, and both look like something else until you know what each process imports.

On the workers, missing on the processor

• The import raises on the processor
• The file lands in list-import-errors
• The DAG never appears
• Looks like a broken file

On the processor, missing on the workers

• The DAG appears and its graph looks right
• The task fails at start with ModuleNotFoundError
• Every attempt fails the same way
• Looks like a bug in the task

The cure is one image or one requirements file for every Airflow process, with the providers pinned. Pinning matters because providers release on their own cadence, and each release declares the oldest core version it supports. Airflow publishes a constraints file for each core release that lists tested provider versions, and installing with that file keeps the set consistent. Upgrading a provider on its own is allowed, and worth doing when the operator you need shipped last month, as long as the same version lands on every process.

• Add the provider to the one requirements file every process installs from
• Rebuild or reinstall the processor and every worker from that file
• Run airflow providers list on each and compare the versions
• Only then change the DAG file to import the new operator

When to use a provider operator

A @task function runs any Python you write with the Task SDK alone, and for a transformation that is your own code it is the shortest route. A provider operator is the better choice when the work is a conversation with an outside system: it wraps that system's client with authentication and error handling already written, and it writes a task log that reads well, so a 20-line @task that hand-rolls an HTTP call usually loses to a 4-argument HttpOperator. Reading an unfamiliar operator starts with its constructor arguments, which are the knobs you set, and its execute method, which is what the worker runs. For BashOperator, execute runs the command and raises if the exit code is non-zero, so the task fails when the script fails.

About this lesson

This section is part of the DAG Anatomy: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.