Skip to main content

Infrastructure & security

The deployment of the Simplitics PDQ platform is centered on a modular, container-based architecture designed for flexibility across various environments, coupled with strict dependency management and robust security configuration.

1. Core Architectural Deployment Concept​

The Simplitics architecture is fundamentally designed around Docker containers. This approach allows the platform, which is the core of Simplitics' unified data management system, to be run on any container-based environment (any cloud provider or on-premise installation).

The deployment goal is efficiency and decentralization where needed:

  • INGEST agents are preferably deployed close to the data source (on-premise or separate cloud) to minimize data transmission.
  • DWA is ideally located in the same region as the database.
  • The core metadata engine (AME) and data processing (DLS) components are preferably run as close to each other as possible, in the same region as the repository database.

2. Infrastructure Setup and System Dependencies​

Deployment on a Virtual Machine (VM), such as an EC2 Ubuntu 22.04 instance, requires specific preparatory steps.

VM and Docker Configuration​

  1. Disk Setup: Create and mount an extra disk (e.g., /datadrive) to the VM.
  2. Docker Installation: Install and configure the Docker engine, CLI, containerd.io, and necessary plugins.
  3. Data Root and Logging: Edit or create the /etc/docker/daemon.json file to define the data root ("data-root": "/datadrive/docker") and configure rotating logs ("max-size": "10m", "max-file": "3").

Service Startup Order (Systemd Dependency Chain)​

Docker Compose projects are managed by systemd service files, creating a strict, linear dependency chain to ensure services only start if the preceding service is successful.

The required startup order is:

  1. simplitics-ui.service (Base Service, starts only after Docker).
  2. simplitics-ame.service (Active Metadata Engine).
  3. simplitics-dls.service (Data Lake Service).
  4. simplitics-dwa.service (Data Warehouse Automation).
  5. simplitics-ingest.service (Data Importer).

The entire application stack can be started by executing the last service in the chain.

3. Security and Access Deployment (Caddyserver)​

The Simplitics upgrade (25.6.1) introduced Caddyserver to handle API access and new login technology, requiring specific networking and security steps:

  • DNS and Firewall: The PDQ machine must use a Fully Qualified Domain Name (FQDN). Ports 80 and 443 must be accessible on the server, but should not be public-facing.
  • Networking: A new Docker network (e.g., xx_caddynet) must be created.
  • AME Isolation (Critical Step): All mapped ports in the ame-compose.yml file must be deleted. AME services should be isolated to the internal Docker network and not accessible from outside.
  • Authentication (Entra/Azure AD): Enabling Entra authentication involves:
    • App Registration: Create an App registration (e.g., sp-pdq-caddy-demo) with a specific Redirect URI: Web - https://{dns_name}:{port}/auth/oauth2/azure/authorization-code-callback.
    • Group Claims: Define three Entra groups (pdq_web_admins, pdq_web_developers, pdq_web_viewers) to manage user permissions. Update the Token configuration to retrieve assigned user groups.
    • Permissions: Grant admin consent for User.Read permissions.

4. Maintenance and Upgrades​

Deployment includes management practices for upgrades and recovery:

  • Upgrading Components: To upgrade (e.g., to release v25.6.1), stop all Docker services first. Update the component Compose files (.yml) to point to the new release tag (e.g., Xx:release_v25.6.1), and perform a manual pull to fetch the latest image.
  • Encrypting Secrets: After a service update, connection secrets must be re-encrypted. Use the API to GET the existing unencrypted data per source system, and then POST it back into the repository.
  • Troubleshooting Resolution:
    • A common resolution for infrastructure-related problems (like a VM crash) is to stop and start the virtual machine from the cloud portal.
    • If the VM is accessible, navigate to /datadrive/configs and run docker compose down followed by docker compose up -d for all components (INGEST, TOOLS, DWA, DLS, UI, AME) to resolve exited containers or bad states.