Apache Airflow

Add a final task that pings your monitor, placed downstream of your real work so it only runs when everything before it succeeded.

A ping task at the end of the DAG

By default an Airflow task runs only if its upstream tasks succeeded, so a normal task at the end of the chain already gives you “ping only on success”.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

PING_URL = "https://api.didmyjobrun.com/ping/<your-key>"  # better: an Airflow Variable

with DAG(
    dag_id="nightly_pipeline",
    schedule="0 3 * * *",
    start_date=datetime(2024, 1, 1),
    catchup=False,
) as dag:

    work = BashOperator(
        task_id="work",
        bash_command="/opt/run-pipeline.sh",
    )

    ping = BashOperator(
        task_id="ping_dmjr",
        bash_command=f'curl -fsS "{PING_URL}"',
    )

    work >> ping   # ping runs only after work succeeds

If work fails, ping is skipped (its default trigger rule is all_success) and you get alerted.

Notes

• Copy your monitor’s URL from the dashboard. Storing it as an Airflow Variable or Connection keeps it out of your DAG source.
• For an HTTP-native option, the HTTP provider’s HttpOperator (called SimpleHttpOperator in older provider versions) works too — just keep it downstream of your work tasks with the default all_success trigger rule.
• Set the monitor’s period to your DAG’s schedule interval and add grace for scheduler and task latency.