Self-managed Deployment
=======================

This guide covers how customers deploy Scaleout Edge on their own infrastructure — either on a server they already manage, or on a cloud provider (AWS or GCP) provisioned via Terraform.

In both cases the starting point is a **release bundle**: a self-contained archive of deployment scripts and a pre-built Docker Compose configuration, distributed through the Scaleout Harbor registry.

.. contents:: On this page
   :local:
   :depth: 1

----

Prerequisites
-------------

A Harbor robot account (username and password) is required for both deployment paths.
Your Scaleout representative will provide these credentials.

**Compose deployment (on-premise)**

- Linux or macOS with Bash 4+ (on Windows use WSL2)
- `Docker Engine 24+ <https://docs.docker.com/engine/install/>`_ with the Compose plugin
- Python 3.8+

**Cloud deployment (Terraform)**

- `Terraform 1.0+ <https://developer.hashicorp.com/terraform/install>`_

----

Step 1: Install ORAS
--------------------

Release bundles are stored as OCI artifacts on Harbor. You need the `ORAS CLI <https://oras.land>`_ to pull them.

.. only:: html

   .. tabs::

      .. code-tab:: bash
         :caption: macOS

         brew install oras

      .. code-tab:: bash
         :caption: Linux (amd64)

         VERSION=$(curl -fsSL https://api.github.com/repos/oras-project/oras/releases/latest \
             | python3 -c "import sys,json; print(json.load(sys.stdin)['tag_name'].lstrip('v'))")
         curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz"
         tar -zxf "oras_${VERSION}_linux_amd64.tar.gz" oras
         sudo mv oras /usr/local/bin/
         rm "oras_${VERSION}_linux_amd64.tar.gz"

      .. code-tab:: powershell
         :caption: Windows

         winget install -e --id ORASProject.ORAS

.. only:: latex or rinoh

   *macOS:*

   .. code-block:: bash

      brew install oras

   *Linux (amd64):*

   .. code-block:: bash

      VERSION=$(curl -fsSL https://api.github.com/repos/oras-project/oras/releases/latest \
          | python3 -c "import sys,json; print(json.load(sys.stdin)['tag_name'].lstrip('v'))")
      curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz"
      tar -zxf "oras_${VERSION}_linux_amd64.tar.gz" oras
      sudo mv oras /usr/local/bin/
      rm "oras_${VERSION}_linux_amd64.tar.gz"

   *Windows:*

   .. code-block:: powershell

      winget install -e --id ORASProject.ORAS

Verify:

.. code-block:: bash

   oras version

----

Step 2: Authenticate to Harbor
-------------------------------

Log in with your Harbor robot account. Credentials are stored in ``~/.docker/config.json`` and reused for subsequent commands.

.. code-block:: bash

   oras login artifacts.scaleoutsystems.com \
     --username 'robot_<your-robot-account>' \
     --password '<your-token>'

----

Step 3: Pull the release bundle
--------------------------------

Pull the release bundle for your version. Replace ``vX.Y.Z`` with the version provided by your Scaleout representative.

.. code-block:: bash

   oras pull artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z

This writes a ``scaleout-vX.Y.Z.tar.gz`` tarball into a ``dist/`` directory. Change into it and extract:

.. code-block:: bash

   cd dist
   tar -xzf scaleout-vX.Y.Z.tar.gz
   cd scaleout-vX.Y.Z/

To check the installed version and list all available versions on the registry, use the bundled ``update-bundle.sh`` helper (this requires an existing deployment — see :ref:`Upgrading <upgrading-option-a>` once installed):

.. code-block:: bash

   bash deploy/update-bundle.sh \
     -u 'robot_<your-robot-account>' \
     -p '<your-token>' \
     --check

----

Deployment Option A: Compose (on-premise or own server)
--------------------------------------------------------

Use this path when you have a Linux server or VM that you manage directly.

**Authenticate to the container registry**

The compose deployment pulls pre-built images from Harbor. Log in with Docker using the same credentials:

.. code-block:: bash

   docker login artifacts.scaleoutsystems.com \
     --username 'robot_<your-robot-account>' \
     --password '<your-token>'

**Generate runtime configuration**

.. code-block:: bash

   ./deploy/setup.sh <hostname> [OPTIONS]

Replace ``<hostname>`` with the domain or IP address where Scaleout will be reachable (e.g. ``scaleout.mycompany.com`` or ``192.168.1.100``).

Common options:

.. list-table::
   :header-rows: 1
   :widths: 40 60

   * - Option
     - Description
   * - ``--enable-https``
     - Use ``https://`` URLs (TLS terminated externally, e.g. by a load balancer or reverse proxy)
   * - ``--enable-tls``
     - Terminate TLS in nginx using certs from ``runtime/certs/`` (implies ``--enable-https``)
   * - ``--self-signed``
     - Generate a self-signed cert for local HTTPS testing (implies ``--enable-tls``)
   * - ``--disable-signup``
     - Prevent new users from self-registering
   * - ``--google-client-id / --google-client-secret``
     - Enable Google OAuth
   * - ``--microsoft-client-id / --microsoft-client-secret``
     - Enable Microsoft OAuth

See :ref:`DNS and TLS certificates <dns-and-tls>` for when to pick each HTTPS option. Run ``./deploy/setup.sh --help`` for the full option list.

**Start services**

.. code-block:: bash

   ./deploy/compose.sh up

The platform is accessible at ``http://<hostname>`` once all containers are healthy.

See :ref:`DNS and TLS certificates <dns-and-tls>` for how to point a domain at the deployment and serve traffic over HTTPS with your own certificates.

.. _upgrading-option-a:

**Upgrading**

In-place upgrades are handled by ``deploy/update-bundle.sh``. The script stops services, backs up the current bundle, pulls the new release from Harbor, restores ``runtime/`` (secrets and generated config), and starts services again.

Run from inside the existing bundle directory. First check the installed version against what is available on the registry:

.. code-block:: bash

   sudo bash deploy/update-bundle.sh \
     -u 'robot_<your-robot-account>' \
     -p '<your-token>' \
     --check

To upgrade to the latest available version, drop ``--check``:

.. code-block:: bash

   sudo bash deploy/update-bundle.sh \
     -u 'robot_<your-robot-account>' \
     -p '<your-token>'

To target a specific version, pass ``-b``:

.. code-block:: bash

   sudo bash deploy/update-bundle.sh \
     -u 'robot_<your-robot-account>' \
     -p '<your-token>' \
     -b artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z

A timestamped ``scaleout-backup-<version>-<date>.tar.gz`` is written to the parent directory before any files are touched, so the previous bundle can be restored if needed.

.. note::

   Upgrading from v1.0.4 or earlier can fail at the "Restoring runtime configuration" step with ``Permission denied`` writing ``runtime/deploy.env``. This is a known ownership-handling bug in the update script shipped with v1.0.4 and earlier, fixed in v1.0.5 — but since the script that runs your upgrade is the one already installed, you can hit it once on the way from an affected version. If you do, fix ownership of ``runtime/`` and finish the upgrade manually as the account that owns your deployment (replace ``$(whoami)`` if upgrading via ``sudo`` from a different account):

   .. code-block:: bash

      sudo chown -R "$(whoami):$(whoami)" runtime
      bash runtime/setup-invocation.sh
      ./deploy/compose.sh up

   The bundle extraction step has already replaced ``deploy/update-bundle.sh`` with the fixed version, so this only comes up once — the next upgrade runs the fixed script from the start.

**Managing users**

Two helper scripts under ``deploy/`` wrap the Kratos admin API for common operator tasks. Run them on the deployment host. The Compose deployment publishes the Kratos admin API on ``127.0.0.1:4434`` only.

*Promote a user to system admin*

.. code-block:: bash

   ./deploy/promote-user-docker.sh <email>

*Reset a password without email*

Self-service password recovery normally emails a reset link to the user. If the deployment has no SMTP/courier configured (common in evaluation or air-gapped setups), that email is generated internally but never delivered. Generate a recovery link directly instead, and hand it to the user out of band:

.. code-block:: bash

   ./deploy/reset-password-link-docker.sh <email>

This prints a one-time recovery link (default validity ``1h``, override with a third argument) that the user opens to set a new password through the normal flow. The script exits with an error if no identity matches the given email.

Both scripts accept an optional second argument to target a non-default Kratos admin URL, e.g. ``./deploy/promote-user-docker.sh <email> http://127.0.0.1:14434`` through an SSH tunnel.

----

Deployment Option B: Cloud (AWS or GCP via Terraform)
------------------------------------------------------

Use this path to provision a cloud VM automatically. The Terraform scripts are included in the release bundle under ``infrastructure/``.

The VM bootstraps itself: Terraform creates the instance and passes a startup script that pulls the release bundle from Harbor and runs the deployment — no manual SSH required.

**Cloud provider prerequisites**

Before running Terraform, the cloud provider CLI must be installed and authenticated on your workstation. Terraform uses the credentials configured by the CLI to call the provider's APIs.

.. only:: html

   .. tabs::

      .. code-tab:: bash
         :caption: GCP

         # Install: https://cloud.google.com/sdk/docs/install
         gcloud auth application-default login
         gcloud config set project <your-gcp-project-id>

      .. code-tab:: bash
         :caption: AWS

         # Install: https://aws.amazon.com/cli/
         aws configure

.. only:: latex or rinoh

   *GCP:*

   .. code-block:: bash

      # Install: https://cloud.google.com/sdk/docs/install
      gcloud auth application-default login
      gcloud config set project <your-gcp-project-id>

   *AWS:*

   .. code-block:: bash

      # Install: https://aws.amazon.com/cli/
      aws configure

The account used by Terraform needs permission to create and manage a VM, its disk, network resources, and (optionally) snapshot schedules. Minimum required permissions:

*GCP* — the simplest path is to grant your account the ``Compute Admin`` and ``Service Account User`` roles on the target project. For a least-privilege setup, grant the equivalent ``compute.instances.*``, ``compute.disks.*``, ``compute.networks.*``, ``compute.firewalls.*``, ``compute.addresses.*``, ``compute.resourcePolicies.*`` (only if ``enable_disk_snapshots = true``), and ``iam.serviceAccounts.actAs`` permissions. The project must also have billing enabled and the Compute Engine API enabled.

*AWS* — the account needs permissions on EC2 (instances, key pairs, security groups), VPC (VPCs, subnets, route tables, internet gateways — only if ``create_vpc = true``), and IAM + DLM (only if ``enable_disk_snapshots = true``). The AWS-managed policies ``AmazonEC2FullAccess`` and ``AmazonVPCFullAccess`` cover the common case; add ``IAMFullAccess`` and ``AWSDataLifecycleManagerFullAccess`` if snapshots are enabled.

**Configure your deployment**

Copy the example variables file for your cloud provider and fill in your values:

.. only:: html

   .. tabs::

      .. code-tab:: bash
         :caption: GCP

         cd infrastructure/gcp/
         cp terraform.tfvars.example terraform.tfvars

      .. code-tab:: bash
         :caption: AWS

         cd infrastructure/aws/
         cp terraform.tfvars.example terraform.tfvars

.. only:: latex or rinoh

   *GCP:*

   .. code-block:: bash

      cd infrastructure/gcp/
      cp terraform.tfvars.example terraform.tfvars

   *AWS:*

   .. code-block:: bash

      cd infrastructure/aws/
      cp terraform.tfvars.example terraform.tfvars

Open ``terraform.tfvars`` and set at minimum:

.. only:: html

   .. tabs::

      .. code-tab:: hcl
         :caption: GCP

         project_id      = "your-gcp-project-id"
         region          = "us-central1"
         zone            = "us-central1-a"
         domain          = "scaleout.mycompany.com"
         harbor_username = "robot_<your-robot-account>"
         harbor_password = "<your-token>"

      .. code-tab:: hcl
         :caption: AWS

         region          = "eu-north-1"
         domain          = "scaleout.mycompany.com"
         harbor_username = "robot_<your-robot-account>"
         harbor_password = "<your-token>"

.. only:: latex or rinoh

   *GCP:*

   .. code-block:: hcl

      project_id      = "your-gcp-project-id"
      region          = "us-central1"
      zone            = "us-central1-a"
      domain          = "scaleout.mycompany.com"
      harbor_username = "robot_<your-robot-account>"
      harbor_password = "<your-token>"

   *AWS:*

   .. code-block:: hcl

      region          = "eu-north-1"
      domain          = "scaleout.mycompany.com"
      harbor_username = "robot_<your-robot-account>"
      harbor_password = "<your-token>"

``release_bundle_ref`` is pre-filled with the correct version. Replace the Harbor credential placeholders with the values provided by your Scaleout representative.

**Terraform variables reference**

The tables below cover the variables most customers will need to touch. See ``terraform.tfvars.example`` for the full list of optional settings (OAuth providers, HTTPS, SSH access, system admin bootstrap, snapshot scheduling).

*Common variables (both providers)*

.. list-table::
   :header-rows: 1
   :widths: 30 50 20

   * - Variable
     - Description
     - Default
   * - ``domain``
     - Hostname where Scaleout will be reachable
     - (required)
   * - ``deployment_name``
     - Unique name used as a suffix on cloud resources
     - ``default``
   * - ``release_bundle_ref``
     - Full OCI reference to the release bundle on Harbor
     - pre-filled
   * - ``harbor_username``
     - Harbor robot account username
     - (required)
   * - ``harbor_password``
     - Harbor robot account password / token
     - (required)
   * - ``enable_https``
     - Use ``https://`` URLs and secure cookies (requires valid certs)
     - ``false``
   * - ``self_signed``
     - Generate a self-signed cert for HTTPS (testing only)
     - ``false``
   * - ``letsencrypt_email`` / ``letsencrypt_staging``
     - Auto-issue and renew a real cert via Let's Encrypt (requires DNS already pointed at the deployment; port 80 must be reachable unless DNS-01 is selected on the host, see :ref:`DNS and TLS certificates <dns-and-tls>`)
     - ``""`` / ``false``
   * - ``enable_tailscale`` / ``tailscale_auth_key`` / ``tailscale_hostname``
     - Join a Tailscale network and auto-renew a Tailscale-issued cert; VM has no public IP
     - ``false`` / ``""`` / ``""``
   * - ``disable_signup``
     - Prevent new users from self-registering
     - ``false``
   * - ``google_client_id`` / ``google_client_secret``
     - Enable Google OAuth2 login
     - ``""``
   * - ``microsoft_client_id`` / ``microsoft_client_secret``
     - Enable Microsoft OAuth2 login
     - ``""``
   * - ``system_admin_email`` / ``system_admin_name`` / ``system_admin_password``
     - Bootstrap an initial system admin user
     - ``""``
   * - ``ssh_public_key``
     - SSH public key authorised on the VM
     - ``""``
   * - ``allowed_ips`` / ``ssh_allowed_ips``
     - CIDR ranges permitted to reach the app / SSH
     - ``["0.0.0.0/0"]``
   * - ``enable_disk_snapshots``
     - Take daily snapshots of the boot disk
     - ``false``
   * - ``snapshot_retention_days``
     - Number of daily snapshots to retain
     - ``7``

*GCP-specific*

.. list-table::
   :header-rows: 1
   :widths: 30 50 20

   * - Variable
     - Description
     - Default
   * - ``project_id``
     - GCP project ID
     - (required)
   * - ``region``
     - GCP region
     - ``us-central1``
   * - ``zone``
     - GCP zone
     - ``us-central1-a``
   * - ``machine_type``
     - VM machine type
     - ``e2-standard-4``
   * - ``disk_size_gb``
     - Boot disk size in GB
     - ``50``
   * - ``recovery_snapshot``
     - Snapshot name to restore the VM from
     - ``""``

*AWS-specific*

.. list-table::
   :header-rows: 1
   :widths: 30 50 20

   * - Variable
     - Description
     - Default
   * - ``region``
     - AWS region
     - ``us-east-1``
   * - ``instance_type``
     - EC2 instance type
     - ``t3.xlarge``
   * - ``disk_size_gb``
     - Root volume size in GB
     - ``50``
   * - ``create_vpc``
     - Create a new VPC for the deployment
     - ``true``
   * - ``vpc_cidr``
     - CIDR for the new VPC
     - ``10.0.0.0/16``

**Apply**

Initialise the working directory, review what Terraform is about to create, then apply:

.. code-block:: bash

   terraform init
   terraform plan
   terraform apply

Running ``terraform plan`` first lets you confirm the resources, regions, and instance sizes before any cloud charges accrue. Terraform then provisions the VM, waits for the startup script to complete, and prints the application URL when done.

See :ref:`DNS and TLS certificates <dns-and-tls>` for how to point your domain at the VM and (optionally) replace the default certificates.

**Monitoring startup and troubleshooting**

The startup script runs after the VM is created and typically takes 10–15 minutes on first boot (image pull and initial container creation). To follow progress, SSH into the VM and tail the startup logs:

.. only:: html

   .. tabs::

      .. code-tab:: bash
         :caption: GCP

         gcloud compute ssh scaleout-vm-<deployment-name> \
             --zone=<zone> --project=<project-id>

         sudo journalctl -u google-startup-scripts.service -f

      .. code-tab:: bash
         :caption: AWS

         ssh ubuntu@$(terraform output -raw public_ip)

         sudo tail -f /var/log/cloud-init-output.log

.. only:: latex or rinoh

   *GCP:*

   .. code-block:: bash

      gcloud compute ssh scaleout-vm-<deployment-name> \
          --zone=<zone> --project=<project-id>

      sudo journalctl -u google-startup-scripts.service -f

   *AWS:*

   .. code-block:: bash

      ssh ubuntu@$(terraform output -raw public_ip)

      sudo tail -f /var/log/cloud-init-output.log

The script writes ``/tmp/scaleout-startup-complete`` when finished. Check the running containers from the deployment directory:

.. code-block:: bash

   cd /opt/scaleout
   ./deploy/compose.sh ps
   ./deploy/compose.sh logs -f

Common checks if something looks wrong:

.. code-block:: bash

   # Docker is installed and running
   docker --version
   docker compose version
   sudo systemctl status docker

   # The release bundle was unpacked
   ls -la /opt/scaleout

   # DNS resolves to the VM
   nslookup <your-domain>

DNS propagation can take 5–60 minutes depending on TTL.

**Private network deployment (AWS, no public IP)**

If your environment does not permit a public IP on the VM — for example because you reach AWS through a Transit Gateway, Site-to-Site VPN, or Direct Connect — set the following in ``terraform.tfvars`` to skip the Elastic IP and launch the VM in a private subnet you already operate:

.. code-block:: hcl

   assign_public_ip = false
   create_vpc       = false
   vpc_id           = "vpc-xxxxxxxx"
   subnet_id        = "subnet-xxxxxxxx"   # must be a private subnet
   allowed_ips      = ["10.0.0.0/8"]       # internal CIDRs that may reach the application
   ssh_allowed_ips  = ["10.0.0.0/8"]       # internal CIDRs that may SSH into the VM

Prerequisites you (the customer) must provide:

- A VPC and a private subnet to host the VM, in the AWS region you chose.
- Outbound internet from that subnet (typically a NAT Gateway or a forward HTTP proxy), so the VM can pull container images from ``artifacts.scaleoutsystems.com``, GitHub, Ubuntu/Docker apt repositories, and any configured OAuth identity provider endpoints during provisioning and at runtime.
- An internal DNS A record for ``domain`` pointing at the VM's private IP. Terraform prints the private IP and the required DNS record in its output after apply.
- A path for administrative access — VPN, bastion host, or AWS Systems Manager Session Manager — since SSH is reachable only from inside your network.

**Private network deployment (AWS, no public IP)**

If your environment does not permit a public IP on the VM — for example because you reach AWS through a Transit Gateway, Site-to-Site VPN, or Direct Connect — set the following in ``terraform.tfvars`` to skip the Elastic IP and launch the VM in a private subnet you already operate:

.. code-block:: hcl

   assign_public_ip = false
   create_vpc       = false
   vpc_id           = "vpc-xxxxxxxx"
   subnet_id        = "subnet-xxxxxxxx"   # must be a private subnet
   allowed_ips      = ["10.0.0.0/8"]       # internal CIDRs that may reach the application
   ssh_allowed_ips  = ["10.0.0.0/8"]       # internal CIDRs that may SSH into the VM

Prerequisites you (the customer) must provide:

- A VPC and a private subnet to host the VM, in the AWS region you chose.
- Outbound internet from that subnet (typically a NAT Gateway or a forward HTTP proxy), so the VM can pull container images from ``harbor.scaleoutsystems.com``, GitHub, Ubuntu/Docker apt repositories, and any configured OAuth identity provider endpoints during provisioning and at runtime.
- An internal DNS A record for ``domain`` pointing at the VM's private IP. Terraform prints the private IP and the required DNS record in its output after apply.
- A path for administrative access — VPN, bastion host, or AWS Systems Manager Session Manager — since SSH is reachable only from inside your network.

**Upgrading**

Upgrading the release bundle is **not** done through Terraform. Bumping ``release_bundle_ref`` and re-running ``terraform apply`` only takes effect on a fresh VM. To upgrade an existing deployment, SSH into the VM and run ``update-bundle.sh`` exactly as in Option A — the release bundle is installed at ``/opt/scaleout``:

.. only:: html

   .. tabs::

      .. code-tab:: bash
         :caption: GCP

         gcloud compute ssh scaleout-vm-<deployment-name> \
             --zone=<zone> --project=<project-id>

         cd /opt/scaleout
         sudo bash deploy/update-bundle.sh \
             -u 'robot_<your-robot-account>' \
             -p '<your-token>' \
             --check

      .. code-tab:: bash
         :caption: AWS

         ssh ubuntu@$(terraform output -raw public_ip)

         cd /opt/scaleout
         sudo bash deploy/update-bundle.sh \
             -u 'robot_<your-robot-account>' \
             -p '<your-token>' \
             --check

.. only:: latex or rinoh

   *GCP:*

   .. code-block:: bash

      gcloud compute ssh scaleout-vm-<deployment-name> \
          --zone=<zone> --project=<project-id>

      cd /opt/scaleout
      sudo bash deploy/update-bundle.sh \
          -u 'robot_<your-robot-account>' \
          -p '<your-token>' \
          --check

   *AWS:*

   .. code-block:: bash

      ssh ubuntu@$(terraform output -raw public_ip)

      cd /opt/scaleout
      sudo bash deploy/update-bundle.sh \
          -u 'robot_<your-robot-account>' \
          -p '<your-token>' \
          --check

Drop ``--check`` to perform the upgrade, or pass ``-b artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z`` to target a specific version. The script preserves ``runtime/`` (secrets and generated config) and writes a timestamped backup to the parent of ``/opt/scaleout`` before unpacking the new bundle. See :ref:`Upgrading <upgrading-option-a>` for the full command reference.

After upgrading, update ``release_bundle_ref`` in ``terraform.tfvars`` to match the version now running on the VM so future re-creations stay in sync.

----

.. _dns-and-tls:

DNS and TLS certificates
------------------------

Both deployment options serve traffic over plain HTTP by default. To use a real domain and HTTPS:

**1. Point DNS at the deployment**

Create an ``A`` record for your domain that points at the deployment's public IP. For Option B, Terraform prints the address:

.. code-block:: bash

   terraform output public_ip

For Option A, use the public IP of the host running the compose stack. DNS propagation usually completes within 5–10 minutes for low TTLs.

**2. Enable HTTPS**

You have four options for TLS termination. Let's Encrypt and Tailscale renew automatically forever once set up; self-signed and BYO-cert are unchanged manual options for offline/internal use.

*Self-signed certificate (testing only):* ``deploy/setup.sh`` can generate a self-signed cert and configure nginx to terminate TLS with it. Re-run ``setup.sh`` with ``--self-signed`` (Option A) or set ``self_signed = true`` in ``terraform.tfvars`` (Option B). Browsers will warn about the certificate — useful for internal testing, not production.

.. code-block:: bash

   ./deploy/setup.sh <hostname> --self-signed
   ./deploy/compose.sh up

*Bring your own certificate (production):* drop a valid certificate and private key into the deployment's ``runtime/certs/`` directory as ``server.crt`` and ``server.key``, then re-run setup with ``--enable-tls`` (which expects certs to already be present) and restart the stack:

.. code-block:: bash

   # Place fullchain certificate and private key in runtime/certs/
   cp /path/to/fullchain.pem runtime/certs/server.crt
   cp /path/to/privkey.pem   runtime/certs/server.key

   ./deploy/setup.sh <hostname> --enable-tls
   ./deploy/compose.sh up

For Option B, the same files live at ``/opt/scaleout/runtime/certs/`` on the VM. Copy your certs into that directory over SSH and re-run ``runtime/setup-invocation.sh`` to regenerate the nginx config, then ``./deploy/compose.sh up``.

*Let's Encrypt (production, real public domain):* ``setup.sh`` generates a placeholder self-signed cert as before, and a ``certbot`` container issues a real one on startup and keeps renewing it forever — no further action is ever required. Let's Encrypt checks domain control with one of two methods. HTTP-01 is the default. Use DNS-01 when the deployment is not reachable on port 80 from the internet.

With either method, ``--letsencrypt-email`` must be combined with ``--enable-tls`` explicitly. Add ``--letsencrypt-staging`` to test against Let's Encrypt's staging environment first.

**Method 1: HTTP-01 (default)**

Let's Encrypt fetches a token from the deployment over port 80.

- Port 80 must be reachable from the internet.
- DNS must already point at the deployment (step 1 above).

.. code-block:: bash

   ./deploy/setup.sh <hostname> --enable-tls --letsencrypt-email you@company.com
   ./deploy/compose.sh up

For Option B, set ``letsencrypt_email`` (and optionally ``letsencrypt_staging = true``) in ``terraform.tfvars`` instead, then ``terraform apply``.

**Method 2: DNS-01**

Let's Encrypt checks a ``TXT`` record named ``_acme-challenge.<hostname>`` in your DNS zone. The ``certbot`` container writes that record through the DNS provider's API. Port 80 does not need to be open.

- The DNS zone must be hosted at one of the supported providers below.
- You create a credential for that provider once, and add it to ``runtime/secrets.env``.
- The hostname appears in public Certificate Transparency logs when the certificate is issued, even though the deployment is not reachable from outside.

Every method issues a certificate for one hostname. Wildcard certificates are not supported.

Setup for Method 2, in order:

1. Run setup with the provider option:

   .. code-block:: bash

      ./deploy/setup.sh <hostname> --enable-tls --letsencrypt-email you@company.com --letsencrypt-dns-provider <provider>

   On the first run, setup creates ``runtime/secrets.env`` and then stops, because the provider's credential is not in it yet. The message names the setting to add.

2. Add the provider's credential to ``runtime/secrets.env``. The table below lists the setting for each provider, and the steps follow it. For example, for Cloudflare:

   .. code-block:: bash

      ACME_DNS_CLOUDFLARE_API_TOKEN=<token>

3. Run the same setup command again. Setup checks the credential, and passes only the ``ACME_DNS_*`` lines of ``runtime/secrets.env`` to the ``certbot`` container.
4. Start the stack:

   .. code-block:: bash

      ./deploy/compose.sh up

.. list-table::
   :header-rows: 1
   :widths: 30 25 45

   * - Provider
     - ``--letsencrypt-dns-provider``
     - Credential in ``runtime/secrets.env``
   * - Hurricane Electric
     - ``he``
     - ``ACME_DNS_HE_DDNS_KEY``: dynamic DNS key of the challenge record
   * - Cloudflare
     - ``cloudflare``
     - ``ACME_DNS_CLOUDFLARE_API_TOKEN``: API token with Zone:Read and DNS:Edit

*Hurricane Electric* (``he``):

1. Log in at dns.he.net. Create a ``TXT`` record named ``_acme-challenge.<hostname>``. Give it any placeholder value. Dynamic DNS can only update a record that already exists.
2. Set the record's TTL to the lowest value HE offers. A longer TTL lets resolvers keep the previous challenge value.
3. Enable dynamic DNS on the record and generate its key. Use that key as ``ACME_DNS_HE_DDNS_KEY``.

*Cloudflare* (``cloudflare``):

1. In the Cloudflare dashboard, create an API token with the Zone:Read and DNS:Edit permissions for the zone that holds the hostname. Use that token as ``ACME_DNS_CLOUDFLARE_API_TOKEN``.

If something is wrong, setup stops with a message that names what to fix:

- ``--letsencrypt-dns-provider`` is given without ``--letsencrypt-email``.
- The provider name is not in the table. The message lists the names that are available.
- The provider's credential is missing or empty in ``runtime/secrets.env``.

Errors from the provider during issuance, such as a wrong key or token, show in ``./deploy/compose.sh logs certbot``. The container retries with a growing delay, up to once an hour.

To go back to HTTP-01, run setup again without ``--letsencrypt-dns-provider``. Switching method in either direction issues a new certificate once. The existing certificate keeps serving until the new one is issued.

**Adding a DNS provider:** each provider is one file in ``deploy/certbot/dns/``, named after the provider. The file handles three calls: ``check`` (the provider's credentials are set), ``auth`` (create the TXT record) and ``cleanup`` (remove it). It reads its own ``ACME_DNS_<PROVIDER>_*`` settings and uses only the Python standard library, which the ``certbot/certbot`` image includes. Add a row to the table and a short setup section here.

Let's Encrypt's ``--letsencrypt-email`` is the contact address for account and expiry notices, so use a mailbox your operators read.

For Option B, set ``letsencrypt_email`` in ``terraform.tfvars`` as for HTTP-01. Terraform does not pass the provider option. On the VM, over SSH:

1. Add ``--letsencrypt-dns-provider <provider>`` to ``runtime/setup-invocation.sh``.
2. Add the credential to ``runtime/secrets.env``.
3. Run ``runtime/setup-invocation.sh`` again, then ``./deploy/compose.sh up``.

*Tailscale (production, `*.ts.net` hostname):* for deployments reachable only over Tailscale, a ``tailscale-cert-renew`` container renews the Tailscale-issued cert automatically. Run ``tailscale up`` on the host first:

.. code-block:: bash

   ./deploy/setup.sh <tailscale-hostname> --enable-tls --tailscale-hostname <tailscale-hostname>
   ./deploy/compose.sh up

For Option B, set ``enable_tailscale = true`` plus ``tailscale_auth_key``/``tailscale_hostname`` in ``terraform.tfvars`` — Terraform joins the VM to your tailnet and the ``tailscale-cert-renew`` container takes care of renewal from there.

Let's Encrypt and Tailscale are mutually exclusive with each other and with ``--self-signed``/``self_signed`` — Let's Encrypt cannot validate a ``*.ts.net`` hostname, because that zone is not under your control.

**Upgrading an already-running deployment in place:** for Option A, edit ``runtime/setup-invocation.sh`` to add ``--letsencrypt-email ...`` (or ``--tailscale-hostname ...``), re-run it, then ``./deploy/compose.sh up`` again. The existing cert keeps serving traffic until the renewal container's first successful issuance replaces it — no manual cert deletion or extended downtime. For Option B, note that ``metadata_startup_script``/``user_data`` changes are ignored by Terraform on an already-applied VM (the "reboot recovery" path skips re-provisioning), so adding ``letsencrypt_email``/``tailscale_hostname`` to ``terraform.tfvars`` won't take effect via ``terraform apply`` alone — SSH in and follow the same ``runtime/setup-invocation.sh`` steps as Option A instead.

----

.. _health-checks:

Health checks for external load balancers and gateways
-------------------------------------------------------

If you place an external load balancer or API gateway in front of the deployment, configure it to probe one of the following unauthenticated HTTP endpoints. Both return ``200 OK`` without authentication and are reachable through the built-in nginx reverse proxy.

.. list-table::
   :header-rows: 1
   :widths: 30 25 45

   * - Endpoint
     - Service
     - Notes
   * - ``/api/v1/health``
     - REST API
     - Reflects the health of the API server. Reachable through the nginx ``/api/v1/`` proxy. Returns ``200 OK`` with body ``OK``.
   * - ``/health``
     - Frontend
     - Returns ``200 OK`` directly. Use this instead of ``/``, which returns a ``308`` redirect (via the login/session flow) and must not be used as a health probe.

For example, with the deployment reachable at ``https://scaleout.mycompany.com``:

.. code-block:: bash

   curl -fsS https://scaleout.mycompany.com/api/v1/health   # -> OK
   curl -fsS https://scaleout.mycompany.com/health          # -> OK

.. note::

   nginx also answers ``/healthz`` itself (without contacting any upstream). That endpoint only confirms that nginx is up — it does **not** reflect the health of the application services. Use ``/api/v1/health`` and ``/health`` for application-level checks from external load balancers.

**gRPC services**

The Controller, Combiner, and Hooks services expose gRPC rather than HTTP, and implement the standard `gRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_. HTTP probes will not work against them. Use `grpc_health_probe <https://github.com/grpc-ecosystem/grpc-health-probe>`_ instead, for example:

.. code-block:: bash

   grpc_health_probe -addr=<host>:<grpc-port>

This is the same mechanism the bundled Docker Compose stack uses for its container health checks.

.. _admin-api-exclusion:

Kratos and Hydra admin APIs must never be forwarded
------------------------------------------------------

The Kratos and Hydra admin APIs (ports ``4434`` and ``4445``) have no authentication of their own — anything that can reach them can enumerate identities, and create, modify, or delete OAuth2 clients. They exist only for internal orchestration (bootstrap jobs, the frontend's server-side actions) which all reach them directly over the Docker network, never through nginx.

The bundled Docker Compose stack publishes both admin ports on the loopback address ``127.0.0.1`` only, so only tools on the host itself can reach them. Postgres (``5432``) and MinIO (``9000``, ``9001``) are published the same way. Docker's published ports bypass host firewalls such as ufw, so do not change these bindings to listen on all interfaces. To reach one of these ports from another machine, use an SSH tunnel.

The bundled nginx configuration (``deploy/nginx.template/nginx.conf.template``) blocks ``/kratos/admin/`` and ``/hydra/admin/`` with an explicit ``403`` at the edge. **If you front a deployment with your own reverse proxy or load balancer instead of the bundled nginx, you must not forward these two paths to the backend.** Only the public paths — ``/kratos/`` and ``/hydra/`` — are meant to be reachable from outside.

This mirrors the Helm chart deployment (``charts/scaleout``), whose Ingress already excludes both admin ports by design and routes only the public Kratos and Hydra ports — see :doc:`chart-security`. Keep both deployment types consistent: any future change to admin-path exposure should preserve this exclusion on both.

.. _waf-file-upload:

WAF and reverse proxy requirements for file uploads
----------------------------------------------------

Scaleout uploads seed models and compute packages from the browser directly to the API server in chunks via ``POST /api/v1/file-upload/``. If a Web Application Firewall (WAF) or other inspection proxy sits between the browser and the deployment, it must be configured to allow these requests through.

**Why uploads fail behind a WAF**

The default chunk size is 900 KB. WAF rules such as AWS ``AWSManagedRulesCommonRuleSet`` / ``SizeRestrictions_BODY`` enforce a body inspection limit (commonly 8 KB) and block requests whose bodies exceed it. Because the WAF rejects the chunk before it reaches nginx, nothing appears in the application logs — only the upload initiation and abort are visible.

The symptom is an "Upload blocked by a proxy or firewall" error in the browser after every file upload attempt, while small files (under the WAF threshold) succeed.

**Fix: add a WAF allow rule for the upload path**

Add a rule with a lower priority number than your managed rule groups that unconditionally allows ``POST`` requests to ``/api/v1/file-upload/``:

.. code-block:: json

   {
       "Name": "AllowFileUploadChunks",
       "Priority": 1,
       "Statement": {
           "AndStatement": {
               "Statements": [
                   {
                       "ByteMatchStatement": {
                           "FieldToMatch": { "UriPath": {} },
                           "SearchString": "/api/v1/file-upload/",
                           "PositionalConstraint": "STARTS_WITH",
                           "TextTransformations": [
                               { "Priority": 0, "Type": "NONE" }
                           ]
                       }
                   },
                   {
                       "ByteMatchStatement": {
                           "FieldToMatch": { "Method": {} },
                           "SearchString": "POST",
                           "PositionalConstraint": "EXACTLY",
                           "TextTransformations": [
                               { "Priority": 0, "Type": "NONE" }
                           ]
                       }
                   }
               ]
           }
       },
       "Action": { "Allow": {} },
       "VisibilityConfig": {
           "SampledRequestsEnabled": true,
           "CloudWatchMetricsEnabled": true,
           "MetricName": "AllowFileUploadChunks"
       }
   }

Set ``Priority`` to a value lower than the priority of your managed rule groups so this rule evaluates first.

**Temporary workaround: reduce chunk size**

If you cannot modify the WAF rules immediately, you can reduce the chunk size so each request stays under the WAF inspection limit. Set the following environment variables on the ``api-server`` container and restart it:

.. code-block:: yaml

   SCALEOUT_UPLOAD_MAX_CHUNK_SIZE: "6144"   # 6 KB — safely under the 8 KB AWS default
   SCALEOUT_UPLOAD_MIN_CHUNK_SIZE: "2048"   # 2 KB

The frontend reads the chunk size from the server's init response and adapts automatically — no frontend restart is needed. Note that very small chunks increase the number of HTTP roundtrips and make large file uploads significantly slower. Revert these variables once the WAF rule is in place.

To confirm the variables are active inside the container:

.. code-block:: bash

   ./deploy/compose.sh exec api-server env | grep SCALEOUT_UPLOAD

----

.. _client-grpc-tls:

Connecting clients over gRPC
----------------------------

Edge clients reach the web UI and REST API on ports 80/443. They connect to a **combiner** on the hostname and port given in its public address, described below. By default the client uses the TLS mode the combiner reports on assignment, and TLS if the combiner doesn't report one. The :ref:`SCALEOUT_GRPC_SECURE <client-envs>` environment variable overrides this, except that the combiner can still upgrade it to TLS.

Whether the combiner endpoint actually speaks TLS depends on how the deployment terminates it:

.. list-table::
   :header-rows: 1
   :widths: 40 25 35

   * - Deployment
     - Combiner gRPC
     - Client setting
   * - ``--enable-tls`` / ``--self-signed``
     - TLS (terminated by nginx on the combiner ports)
     - None needed (the combiner reports TLS)
   * - Default (plain HTTP)
     - Plaintext (h2c)
     - None needed (the combiner reports plaintext)
   * - ``--enable-https`` (TLS terminated externally)
     - Plaintext from nginx, unless your external proxy is also configured to terminate TLS on the combiner ports
     - ``false`` unless the external proxy fronts the combiner ports with TLS.

With ``--enable-tls`` or ``--self-signed``, nginx terminates TLS on the combiner ports using the same certificate as the web UI and forwards plaintext gRPC to the combiner internally, so no combiner-side certificate is needed.

With ``--enable-tls`` or ``--self-signed`` and a single combiner, edge clients connect to the combiner on port 443, on the same hostname as the web UI. nginx routes their gRPC connections to the combiner, and a new deployment seeds the combiner's public address as ``<hostname>:443``. Setup prints where edge clients connect.

Without TLS, or with several combiners, each combiner is published on its own port (``12080``, ``12082``, …) instead.

**Your own load balancer in front.** The public address is the hostname and port of the endpoint that accepts the edge clients' gRPC connections. If a load balancer or proxy sits in front of the deployment, set each combiner's public address to the load balancer's hostname and port, in the combiner's edit dialog in the UI.

**Upgrading a deployment that uses the combiner's own port.** An upgrade keeps port 12080 published when the existing deployment publishes it, and records ``--enable-combiner-ports`` in the saved setup (``runtime/setup-invocation.sh``). Edge clients and load balancers that use port 12080 keep working. To move them to port 443, point the load balancer at port 443 and set the combiner's public address to its hostname and port 443. Health checks work on port 443 as well: the gRPC health check, or a plain HTTPS check of ``/healthz``. Then remove ``--enable-combiner-ports`` from the saved setup.

**Self-signed certificates.** A secure-by-default client validates the combiner certificate against the system trust store, which rejects self-signed certs. Point the client at the certificate so it can verify the connection:

.. code-block:: bash

   export SCALEOUT_GRPC_ROOT_CERT_PATH=/path/to/server.crt
   scaleout client start --api-url https://<hostname> --token <TOKEN>

This is the ``server.crt`` from the deployment's ``runtime/certs/`` directory. With a publicly-trusted (CA-issued) certificate, leave ``SCALEOUT_GRPC_ROOT_CERT_PATH`` unset — the system trust store handles verification.

See :ref:`Client Environment Variables <client-envs>` for the full list of client settings.