Skip to main content

The daily pass

The concept of DataOps in Day-to-Day Operations within the PDQ by Simplitics platform is fundamentally realized through Proactive Data Auditing and centralized, self-service monitoring, primarily executed via the DataOps Console. DataOps ensures that the fully automated, metadata-driven pipelines execute correctly and reliably, thereby guaranteeing continuous data delivery and quality.

The daily operational workflow is structured around several conceptual pillars, ensuring efficiency and accuracy.

Core DataOps Operational Pillars​

1. Centralized Visibility and Self-Service Management​

The DataOps Console serves as the primary control center, designed to simplify daily tasks and provide immediate insights into platform performance.

  • Runtime Slicing and Dicing: The console provides the ability to inspect runtime operators and statistics, offering detailed information about ongoing processes.
  • Failure Resolution (Idempotency): Tasks, particularly those managed by DWA, are designed to be restarted safely via the console, as they are idempotent (can be rerun without risk of duplication or corruption). Restarting failed DWA Loading Tasks is a simple fix available to operators.

2. Validating End-to-End Data Integrity​

Daily operations focus on verifying that data acquisition and subsequent processing phases maintain traceability and consistency throughout the system's architectural layers.

  • Source Acquisition Audits (INGEST): Operators use the INGEST Tasks views to confirm that the required data extraction agents have run as scheduled (Expected = Completed, Failed = 0) and that all expected file types have been delivered. The INGEST Task Checklist confirms the second half of that: a green banner reading "All configured files have at least one delivery" means every configured file type arrived at least once in the selected window. A task that never started shows up here and nowhere else — a "no failures" metric will not reveal it.

  • Flow Consistency (DLS Trace): The DLS Trace view verifies data movement through the standard processing layers:

    Landing → Raw Archive → Trusted → Published → Profile

    Successful flow is validated when the counts for Landing, Raw Archive, Trusted and Profiled are identical. This confirms that the DLS Worker has performed archival, standardisation of format, and profiling. The Stage count may sit slightly lower — a difference of up to 10–20 is normal and not by itself a fault.

3. Orchestration and Model Integrity​

DataOps ensures that the sophisticated integration and automation tools are consistently maintaining the structure of the data warehouse model.

  • DWA Loading Tasks Audit: Operators use the DWA Loading Tasks view to confirm that all incoming files have been correctly processed and loaded into the integrated data model. The audit ensures that all expected loading tasks were completed and that none failed (Failed = 0).
  • Metadata Synchronization: Operational verification confirms that the DWA component's metadata-driven process has successfully generated the required SQL and orchestration at runtime, based on source-to-target mappings.

4. Data Warehouse Checks (DWA Loading Tasks)​

DWA monitoring verifies that the received and processed files have been correctly loaded into the integrated data model.

Verification Procedure​

The key check here is that:

  • Number of Expected loading tasks and Completed loading tasks must be identical
  • Failed count must be 0

Steps to verify:

  1. Access DWA Loading Tasks view
  2. Compare expected vs completed task counts
  3. Investigate any failed tasks immediately
  4. Confirm data availability in target systems

DataOps Console​

The DataOps console is the central point for monitoring and simple management of runtime operators and statistics. It groups its pages by the stage of the platform they cover — Ingestion, Data Lake, Data Warehouse, Data Quality, Reference and System — each with its own monitoring views, checklists and admin actions.

Page-by-page reference

This page describes the routine. For what each console page shows, what every metric means and which corrective actions it offers, see DataOps Console.

Daily monitoring checklist​

Follow this sequence every operational day. It is the same order used throughout these docs — services first, then the data, from the source outwards.

  1. ☑ System Health

    • Are the services up, and did anything fail or warn overnight?
    • Any row with status FAIL has to be restarted from the page that owns the work

  2. ☑ INGEST Tasks

    • Confirm that every expected run completed (Expected = Completed, Failed = 0)
    • Look for completed runs that returned zero rows — the classic silent failure
    • Confirm the green banner on the INGEST Task Checklist: "All configured files have at least one delivery"

  3. ☑ DLS Trace

    • Verify count consistency across the zones (see the rule above)
    • Check for files that stalled at a stage, using the Last seen on filter

  4. ☑ DWA Loading Tasks

    • Expected and Completed must match, and Failed must be 0
    • Restart failed tasks — they are idempotent

  5. ☑ DWA Additional Tasks

    • Confirm that the chained post-processors ran

  6. ☑ QPI Monitor

    • Did any quality check report a failing result?
    • Keep Fail (the data is wrong) apart from Error (the check itself is broken)

Document anything you could not resolve, and follow up on it before the next pass.