Skip to main content

Daily operations

Running PDQ day to day is a short, repeatable routine: confirm that every scheduled extraction, data lake movement, warehouse load and quality check has done its job — and act on the ones that did not.

The routine is deliberately proactive. Nothing here waits for a user to report missing data. The target is to have every delivery and processing step verified before the daily deadline, typically 08:00, Monday–Saturday.

Start here​

The daily passThe routine itself — what to check in INGEST, DLS and DWA, in what order, and what each metric has to say before you sign off.
DataOps ConsolePage-by-page reference for the console: what every view shows, what each number means, and which corrective actions it offers.
TroubleshootingWhat to do when a check fails — infrastructure, ingestion, processing and warehouse loading issues, with the resolution for each.

The daily checks​

Every operational day, in this order — following the data from the source outwards:

  1. System Health — are the services themselves running, and what did they log?
  2. INGEST Tasks — did every scheduled extraction run, and did every configured file type arrive?
  3. DLS Trace — did every delivered file move through Landing → Raw Archive → Trusted → Published?
  4. DWA Loading Tasks — did every load step complete, with zero failures?
  5. DWA Additional Tasks — did the post-processors chained onto the loads run?
  6. QPI Monitor — did any quality check report a failing result?

Steps 2–6 each have a checklist view that compares configured against executed. That comparison is the only thing that catches work which never started — a failure count alone will always read zero for a task that was never scheduled.

Before you assume a failure​

Most "empty" pages in the console are answering correctly for a window that is too narrow. Check the global date range first; it defaults to the last 24 hours.

Most DWA tasks are idempotent — they can be restarted from the console without risk of duplication. Restarting a failed loading task is the normal first response, not a last resort.