Connections and Hooks

Concepts covered: Connections and hooks

The commanding task in the first section called warehouse.submit(...), which means something built that client from a host and port plus a username and password. Writing those into the DAG file puts a production password into a Python file that is committed to git, copied to every machine that parses DAGs, and printed in any traceback that includes the line. A connection moves those facts out of the file and into a named record that the task fetches by name at run time, so the file says warehouse and nothing more.

A connection record has a fixed set of fields. conn_id is the name tasks use and conn_type is the kind of system. The 5 fields host, port, login, password and schema hold what their names say, and extra is a JSON blob for anything they do not cover, such as a timeout or an SSL flag. The record lives in the metadata database, with the password encrypted with the deployment's key before it is stored, so a database backup holds no plain-text credential.

The UI lists records like these under Admin and Connections, with passwords shown as ***, and a record can be typed in there field by field. The command line does the same job: airflow connections add payments_api --conn-type http --conn-host api.payments.internal --conn-port 8443 --conn-password tok_live_8f3a. And an environment variable named AIRFLOW_CONN_PAYMENTS_API on the worker, holding the record as JSON, is read as the connection payments_api, which suits deployments where secrets arrive through the environment, though a record set that way does not show in the UI's list.

{
  "conn_type": "http",
  "host": "api.payments.internal",
  "port": 8443,
  "password": "tok_live_8f3a",
  "extra": {"Content-Type": "application/json"}
}

A hook opens the client for you

A hook is a Python class, shipped with the package for a kind of system, that takes a conn_id, fetches the record, builds a client from its fields, and exposes methods that use it. Every hook does those 3 things, which is what makes one worth more than a hand-built client: the task names the connection and calls a method, the record supplies where to connect and how to log in, and the hook handles the requests. The HttpHook below reaches the payments service; a hook for a warehouse looks the same from the task's side, with run replaced by a method that executes a statement.

from airflow.sdk import task
from airflow.providers.http.hooks.http import HttpHook

@task
def send_payouts(paid_hosts: int):
    hook = HttpHook(method="POST", http_conn_id="payments_api")
    response = hook.run(
        endpoint="/v1/payout-batches",
        json={"paid_hosts": paid_hosts, "day": "2026-03-01"},
    )
    return response.json()["batch_id"]

When no hook exists for a system, the task fetches the record itself with BaseHook.get_connection and builds the client by hand. The password still never appears in the file, and Airflow still masks it in the task log: any value held in a connection's password field is replaced by *** wherever it would be printed, so a traceback that dumps the client's settings shows password='***'.

from airflow.sdk import BaseHook

conn = BaseHook.get_connection("warehouse")
client = WarehouseClient(
    host=conn.host,
    port=conn.port,
    user=conn.login,
    password=conn.password,
    timeout=conn.extra_dejson["timeout_seconds"],
)

What a named record buys you

• One DAG file runs in staging and in production, because each deployment holds its own warehouse record with its own host
• Rotating a password edits one record; no DAG file changes and nothing is redeployed
• A task that cannot reach the host fails at the hook call, and its log names the conn_id it tried
• A log that prints the client's settings shows *** where the password would be

Do

• Name a connection for what the task does with it: payments_api, warehouse
• Call the hook inside the task function, where it runs once per try
• Keep structured settings in extra as JSON, with a key per setting

Avoid

• Paste a password or a token into a DAG file
• Fetch a connection at the top of the file, outside any task
• Reuse one record for 2 systems because the login happens to match

About this lesson

This section is part of the Tasks and Data: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.