Self-managed Deployment ======================= This guide covers how customers deploy Scaleout Edge on their own infrastructure — either on a server they already manage, or on a cloud provider (AWS or GCP) provisioned via Terraform. In both cases the starting point is a **release bundle**: a self-contained archive of deployment scripts and a pre-built Docker Compose configuration, distributed through the Scaleout Harbor registry. .. contents:: On this page :local: :depth: 1 ---- Prerequisites ------------- A Harbor robot account (username and password) is required for both deployment paths. Your Scaleout representative will provide these credentials. **Compose deployment (on-premise)** - Linux or macOS with Bash 4+ (on Windows use WSL2) - `Docker Engine 25+ `_ with the Compose plugin - Python 3.8+ **Cloud deployment (Terraform)** - `Terraform 1.0+ `_ ---- Step 1: Install ORAS -------------------- Release bundles are stored as OCI artifacts on Harbor. You need the `ORAS CLI `_ to pull them. .. only:: html .. tabs:: .. code-tab:: bash :caption: macOS brew install oras .. code-tab:: bash :caption: Linux (amd64) VERSION=$(curl -fsSL https://api.github.com/repos/oras-project/oras/releases/latest \ | python3 -c "import sys,json; print(json.load(sys.stdin)['tag_name'].lstrip('v'))") curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz" tar -zxf "oras_${VERSION}_linux_amd64.tar.gz" oras sudo mv oras /usr/local/bin/ rm "oras_${VERSION}_linux_amd64.tar.gz" .. code-tab:: powershell :caption: Windows winget install -e --id ORASProject.ORAS .. only:: latex or rinoh *macOS:* .. code-block:: bash brew install oras *Linux (amd64):* .. code-block:: bash VERSION=$(curl -fsSL https://api.github.com/repos/oras-project/oras/releases/latest \ | python3 -c "import sys,json; print(json.load(sys.stdin)['tag_name'].lstrip('v'))") curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz" tar -zxf "oras_${VERSION}_linux_amd64.tar.gz" oras sudo mv oras /usr/local/bin/ rm "oras_${VERSION}_linux_amd64.tar.gz" *Windows:* .. code-block:: powershell winget install -e --id ORASProject.ORAS Verify: .. code-block:: bash oras version ---- Step 2: Authenticate to Harbor ------------------------------- Log in with your Harbor robot account. Credentials are stored in ``~/.docker/config.json`` and reused for subsequent commands. .. code-block:: bash oras login artifacts.scaleoutsystems.com \ --username 'robot_' \ --password '' ---- Step 3: Pull the release bundle -------------------------------- Pull the release bundle for your version. Replace ``vX.Y.Z`` with the version provided by your Scaleout representative. .. code-block:: bash oras pull artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z This writes a ``scaleout-vX.Y.Z.tar.gz`` tarball into a ``dist/`` directory. Change into it and extract: .. code-block:: bash cd dist tar -xzf scaleout-vX.Y.Z.tar.gz cd scaleout-vX.Y.Z/ To check the installed version and list all available versions on the registry, use the bundled ``update-bundle.sh`` helper (this requires an existing deployment — see :ref:`Upgrading ` once installed): .. code-block:: bash bash deploy/update-bundle.sh \ -u 'robot_' \ -p '' \ --check ---- Deployment Option A: Compose (on-premise or own server) -------------------------------------------------------- Use this path when you have a Linux server or VM that you manage directly. **Check Docker** Run these two commands. Both must print a version: .. code-block:: bash docker --version docker compose version **Authenticate to the container registry** The compose deployment pulls pre-built images from Harbor. Log in with Docker using the same credentials: .. code-block:: bash docker login artifacts.scaleoutsystems.com \ --username 'robot_' \ --password '' **Generate runtime configuration** .. code-block:: bash ./deploy/setup.sh [OPTIONS] Replace ```` with the domain or IP address where Scaleout will be reachable (e.g. ``scaleout.mycompany.com`` or ``192.168.1.100``). Common options: .. list-table:: :header-rows: 1 :widths: 40 60 * - Option - Description * - ``--enable-https`` - Use ``https://`` URLs (TLS terminated externally, e.g. by a load balancer or reverse proxy) * - ``--enable-tls`` - Terminate TLS in nginx using certs from ``runtime/certs/`` (implies ``--enable-https``) * - ``--self-signed`` - Generate a self-signed cert for local HTTPS testing (implies ``--enable-tls``) * - ``--letsencrypt-email EMAIL`` - Issue and renew a Let's Encrypt cert (requires ``--enable-tls`` and a public domain). Port 80 must be reachable unless ``--letsencrypt-dns-provider`` is given * - ``--letsencrypt-dns-provider NAME`` - Prove domain control through the DNS provider ``NAME`` (``he`` or ``cloudflare``) instead of port 80. The provider's credential goes in ``runtime/secrets.env`` (requires ``--letsencrypt-email``) * - ``--letsencrypt-staging`` - Use Let's Encrypt's staging environment, for testing ``--letsencrypt-email`` * - ``--tailscale-hostname NAME`` - Issue and renew a Tailscale cert for this host's MagicDNS name (requires ``--enable-tls`` and ``tailscale up`` already run on the host) * - ``--disable-signup`` - Prevent new users from self-registering * - ``--require-approval`` - Require a system admin to approve new users before they can sign in * - ``--google-client-id / --google-client-secret`` - Enable Google OAuth * - ``--google-hd DOMAIN`` - Only for accounts in the given Google Workspace domain * - ``--microsoft-client-id / --microsoft-client-secret`` - Enable Microsoft OAuth See :ref:`HTTPS certificates ` below for when to pick each HTTPS option and the full setup of each. Run ``./deploy/setup.sh --help`` for the full option list. **Start services** .. code-block:: bash ./deploy/compose.sh up The platform is accessible at ``http://`` once all containers are healthy. .. _dns-and-tls: **HTTPS certificates** The deployment serves traffic over plain HTTP by default. To use a real domain and HTTPS: **1. Point DNS at the deployment** Create an ``A`` record for your domain that points at the public IP of the host running the compose stack. DNS propagation usually completes within 5–10 minutes for low TTLs. **2. Enable HTTPS** You have four options for TLS termination. Let's Encrypt and Tailscale renew automatically forever once set up; self-signed and BYO-cert are unchanged manual options for offline/internal use. *Self-signed certificate (testing only):* ``deploy/setup.sh`` can generate a self-signed cert and configure nginx to terminate TLS with it. Re-run ``setup.sh`` with ``--self-signed``. Browsers will warn about the certificate — useful for internal testing, not production. .. code-block:: bash ./deploy/setup.sh --self-signed ./deploy/compose.sh up *Bring your own certificate (production):* drop a valid certificate and private key into the deployment's ``runtime/certs/`` directory as ``server.crt`` and ``server.key``, then re-run setup with ``--enable-tls`` (which expects certs to already be present) and restart the stack: .. code-block:: bash # Place fullchain certificate and private key in runtime/certs/ cp /path/to/fullchain.pem runtime/certs/server.crt cp /path/to/privkey.pem runtime/certs/server.key ./deploy/setup.sh --enable-tls ./deploy/compose.sh up *Let's Encrypt (production, real public domain):* ``setup.sh`` generates a placeholder self-signed cert as before, and a ``certbot`` container issues a real one on startup and keeps renewing it forever — no further action is required. Let's Encrypt checks domain control with one of these two methods: - HTTP-01 (default). - DNS-01, for a deployment that is not reachable on port 80 from the internet. With either method, ``--letsencrypt-email`` must be combined with ``--enable-tls`` explicitly. Add ``--letsencrypt-staging`` to test against Let's Encrypt's staging environment first. **Method 1: HTTP-01 (default)** Let's Encrypt fetches a token from the deployment over port 80. - Port 80 must be reachable from the internet. - DNS must already point at the deployment (step 1 above). .. code-block:: bash ./deploy/setup.sh --enable-tls --letsencrypt-email you@company.com ./deploy/compose.sh up **Method 2: DNS-01** Let's Encrypt checks a ``TXT`` record named ``_acme-challenge.`` in your DNS zone. The ``certbot`` container writes that record through the DNS provider's API. Port 80 does not need to be open. - The DNS zone must be hosted at one of the supported providers below. - You create a credential for that provider once, and add it to ``runtime/secrets.env``. - The hostname appears in public Certificate Transparency logs when the certificate is issued, even though the deployment is not reachable from outside. Every method issues a certificate for one hostname. Wildcard certificates are not supported. Setup for Method 2, in order: 1. Run setup with the provider option: .. code-block:: bash ./deploy/setup.sh --enable-tls --letsencrypt-email you@company.com --letsencrypt-dns-provider On the first run, setup adds the provider's setting to ``runtime/secrets.env`` with an empty value and a comment, and then stops. The message names the setting to fill in. 2. Fill in the provider's credential in ``runtime/secrets.env``. The table below lists the setting for each provider, and the steps follow it. For example, for Cloudflare: .. code-block:: bash ACME_DNS_CLOUDFLARE_API_TOKEN= 3. Run the same setup command again. Setup checks the credential, and passes only the ``ACME_DNS_*`` lines of ``runtime/secrets.env`` to the ``certbot`` container. 4. Start the stack: .. code-block:: bash ./deploy/compose.sh up .. list-table:: :header-rows: 1 :widths: 30 25 45 * - Provider - ``--letsencrypt-dns-provider`` - Credential in ``runtime/secrets.env`` * - Hurricane Electric - ``he`` - ``ACME_DNS_HE_DDNS_KEY``: dynamic DNS key of the challenge record * - Cloudflare - ``cloudflare`` - ``ACME_DNS_CLOUDFLARE_API_TOKEN``: API token with Zone:Read and DNS:Edit *Hurricane Electric* (``he``): 1. Log in at dns.he.net. Create a ``TXT`` record named ``_acme-challenge.``. Give it any placeholder value. Dynamic DNS can only update a record that already exists. 2. Enable dynamic DNS on the record and generate its key. Use that key as ``ACME_DNS_HE_DDNS_KEY``. *Cloudflare* (``cloudflare``): 1. In the Cloudflare dashboard, create an API token with the Zone:Read and DNS:Edit permissions for the zone that holds the hostname. Use that token as ``ACME_DNS_CLOUDFLARE_API_TOKEN``. If something is wrong, setup stops with a message that names what to fix: - ``--letsencrypt-dns-provider`` is given without ``--letsencrypt-email``. - The provider name is not in the table. The message lists the names that are available. - The provider's credential is missing or empty in ``runtime/secrets.env``. Errors from the provider during issuance, such as a wrong key or token, show in ``./deploy/compose.sh logs certbot``. The container retries with a growing delay, up to once an hour. To go back to HTTP-01, run setup again without ``--letsencrypt-dns-provider``. Switching method in either direction issues a new certificate once. The existing certificate keeps serving until the new one is issued. Let's Encrypt's ``--letsencrypt-email`` is the contact address for account and expiry notices. *Tailscale (production, `*.ts.net` hostname):* for deployments reachable only over Tailscale, a ``tailscale-cert-renew`` container renews the Tailscale-issued cert automatically. Run ``tailscale up`` on the host first: .. code-block:: bash ./deploy/setup.sh --enable-tls --tailscale-hostname ./deploy/compose.sh up Let's Encrypt and Tailscale are mutually exclusive with each other and with ``--self-signed`` — Let's Encrypt cannot validate a ``*.ts.net`` hostname, because that zone is not under your control. **Upgrading an already-running deployment in place:** edit ``runtime/setup-invocation.sh`` to add ``--letsencrypt-email ...`` (or ``--tailscale-hostname ...``), re-run it, then ``./deploy/compose.sh up`` again. The existing cert keeps serving traffic until the renewal container's first successful issuance replaces it — no manual cert deletion or extended downtime. .. _upgrading-option-a: **Upgrading** In-place upgrades are handled by ``deploy/update-bundle.sh``. The script stops services, backs up the current bundle, pulls the new release from Harbor, restores ``runtime/`` (secrets and generated config), and starts services again. Run from inside the existing bundle directory. First check the installed version against what is available on the registry: .. code-block:: bash sudo bash deploy/update-bundle.sh \ -u 'robot_' \ -p '' \ --check To upgrade to the latest available version, drop ``--check``: .. code-block:: bash sudo bash deploy/update-bundle.sh \ -u 'robot_' \ -p '' To target a specific version, pass ``-b``: .. code-block:: bash sudo bash deploy/update-bundle.sh \ -u 'robot_' \ -p '' \ -b artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z A timestamped ``scaleout-backup--.tar.gz`` is written to the parent directory before any files are touched, so the previous bundle can be restored if needed. .. note:: Upgrading from v1.0.4 or earlier can fail at the "Restoring runtime configuration" step with ``Permission denied`` writing ``runtime/deploy.env``. This is a known ownership-handling bug in the update script shipped with v1.0.4 and earlier, fixed in v1.0.5 — but since the script that runs your upgrade is the one already installed, you can hit it once on the way from an affected version. If you do, fix ownership of ``runtime/`` and finish the upgrade manually as the account that owns your deployment (replace ``$(whoami)`` if upgrading via ``sudo`` from a different account): .. code-block:: bash sudo chown -R "$(whoami):$(whoami)" runtime bash runtime/setup-invocation.sh ./deploy/compose.sh up The bundle extraction step has already replaced ``deploy/update-bundle.sh`` with the fixed version, so this only comes up once — the next upgrade runs the fixed script from the start. **Managing users** Two helper scripts under ``deploy/`` wrap the Kratos admin API for common operator tasks. Run them on the deployment host. The Compose deployment publishes the Kratos admin API on ``127.0.0.1:4434`` only. *Promote a user to system admin* .. code-block:: bash ./deploy/promote-user-docker.sh *Reset a password without email* Self-service password recovery normally emails a reset link to the user. If the deployment has no SMTP/courier configured (common in evaluation or air-gapped setups), that email is generated internally but never delivered. Generate a recovery link directly instead, and hand it to the user out of band: .. code-block:: bash ./deploy/reset-password-link-docker.sh This prints a one-time recovery link (default validity ``1h``, override with a third argument) that the user opens to set a new password through the normal flow. The script exits with an error if no identity matches the given email. Both scripts accept an optional second argument to target a non-default Kratos admin URL, e.g. ``./deploy/promote-user-docker.sh http://127.0.0.1:14434`` through an SSH tunnel. ---- Deployment Option B: Cloud (AWS or GCP via Terraform) ------------------------------------------------------ Use this path to provision a cloud VM automatically. The Terraform scripts are included in the release bundle under ``infrastructure/``. Terraform creates the VM and installs its prerequisites (Docker). It never sees your credentials: you then run ``install-scaleout.sh`` on the VM, which pulls the release bundle from Harbor and starts Scaleout. Harbor credentials, OAuth client secrets, the Tailscale auth key and the admin password are given to that script and are not stored in Terraform, its state, or the VM's instance metadata. **Cloud provider prerequisites** Before running Terraform, the cloud provider CLI must be installed and authenticated on your workstation. Terraform uses the credentials configured by the CLI to call the provider's APIs. .. only:: html .. tabs:: .. code-tab:: bash :caption: GCP # Install: https://cloud.google.com/sdk/docs/install gcloud auth application-default login gcloud config set project .. code-tab:: bash :caption: AWS # Install: https://aws.amazon.com/cli/ aws configure .. only:: latex or rinoh *GCP:* .. code-block:: bash # Install: https://cloud.google.com/sdk/docs/install gcloud auth application-default login gcloud config set project *AWS:* .. code-block:: bash # Install: https://aws.amazon.com/cli/ aws configure The account used by Terraform needs permission to create and manage a VM, its disk, network resources, and (optionally) snapshot schedules. Minimum required permissions: *GCP* — the simplest path is to grant your account the ``Compute Admin`` and ``Service Account User`` roles on the target project. For a least-privilege setup, grant the equivalent ``compute.instances.*``, ``compute.disks.*``, ``compute.networks.*``, ``compute.firewalls.*``, ``compute.addresses.*``, ``compute.resourcePolicies.*`` (only if ``enable_disk_snapshots = true``), and ``iam.serviceAccounts.actAs`` permissions. The project must also have billing enabled and the Compute Engine API enabled. *AWS* — the account needs permissions on EC2 (instances, key pairs, security groups), VPC (VPCs, subnets, route tables, internet gateways — only if ``create_vpc = true``), and IAM + DLM (only if ``enable_disk_snapshots = true``). The AWS-managed policies ``AmazonEC2FullAccess`` and ``AmazonVPCFullAccess`` cover the common case; add ``IAMFullAccess`` and ``AWSDataLifecycleManagerFullAccess`` if snapshots are enabled. **Configure your deployment** Copy the example variables file for your cloud provider and fill in your values: .. only:: html .. tabs:: .. code-tab:: bash :caption: GCP cd infrastructure/gcp/ cp terraform.tfvars.example terraform.tfvars .. code-tab:: bash :caption: AWS cd infrastructure/aws/ cp terraform.tfvars.example terraform.tfvars .. only:: latex or rinoh *GCP:* .. code-block:: bash cd infrastructure/gcp/ cp terraform.tfvars.example terraform.tfvars *AWS:* .. code-block:: bash cd infrastructure/aws/ cp terraform.tfvars.example terraform.tfvars Open ``terraform.tfvars`` and set at minimum: .. only:: html .. tabs:: .. code-tab:: hcl :caption: GCP project_id = "your-gcp-project-id" region = "us-central1" zone = "us-central1-a" .. code-tab:: hcl :caption: AWS region = "eu-north-1" .. only:: latex or rinoh *GCP:* .. code-block:: hcl project_id = "your-gcp-project-id" region = "us-central1" zone = "us-central1-a" *AWS:* .. code-block:: hcl region = "eu-north-1" The release version and the Harbor credentials are not set in Terraform; you provide them to ``install-scaleout.sh`` after the VM is created (see *Install Scaleout on the VM* below). **Terraform variables reference** The tables below cover the variables most customers will need to touch. See ``terraform.tfvars.example`` for the full list of optional settings (SSH access, Tailscale, snapshot scheduling). *Common variables (both providers)* .. list-table:: :header-rows: 1 :widths: 30 50 20 * - Variable - Description - Default * - ``deployment_name`` - Unique name used as a suffix on cloud resources - ``default`` * - ``private_access_only`` - Close the public HTTP/HTTPS/12080 ports, e.g. when the VM is reached over Tailscale (the VM joins your tailnet when you run ``install-scaleout.sh``) - ``false`` * - ``ssh_public_key`` - SSH public key authorised on the VM - ``""`` * - ``allowed_ips`` / ``ssh_allowed_ips`` - CIDR ranges permitted to reach the app / SSH - ``["0.0.0.0/0"]`` * - ``enable_disk_snapshots`` - Take daily snapshots of the boot disk - ``false`` * - ``snapshot_retention_days`` - Number of daily snapshots to retain - ``7`` *GCP-specific* .. list-table:: :header-rows: 1 :widths: 30 50 20 * - Variable - Description - Default * - ``project_id`` - GCP project ID - (required) * - ``region`` - GCP region - ``us-central1`` * - ``zone`` - GCP zone - ``us-central1-a`` * - ``machine_type`` - VM machine type - ``e2-standard-4`` * - ``disk_size_gb`` - Boot disk size in GB - ``50`` * - ``recovery_snapshot`` - Snapshot name to restore the VM from - ``""`` *AWS-specific* .. list-table:: :header-rows: 1 :widths: 30 50 20 * - Variable - Description - Default * - ``region`` - AWS region - ``us-east-1`` * - ``instance_type`` - EC2 instance type - ``t3.xlarge`` * - ``disk_size_gb`` - Root volume size in GB - ``50`` * - ``create_vpc`` - Create a new VPC for the deployment - ``true`` * - ``vpc_cidr`` - CIDR for the new VPC - ``10.0.0.0/16`` **Apply** Initialise the working directory, review what Terraform is about to create, then apply: .. code-block:: bash terraform init terraform plan terraform apply Running ``terraform plan`` first lets you confirm the resources, regions, and instance sizes before any cloud charges accrue. Terraform then provisions the VM and prints how to connect to it. The VM installs Docker on first boot, which takes a few minutes. See :ref:`HTTPS certificates ` below for how to point your domain at the VM and (optionally) replace the default certificates. **Install Scaleout on the VM** On GCP, run ``infrastructure/install-remote.sh`` from the deployment directory. It waits for the first boot to finish, copies ``install-scaleout.sh`` to the VM over an IAP tunnel and starts it there in the background, then follows its log. If your connection drops or you press Ctrl-C, the install keeps running; run the command again to follow it. Settings come from an ``install.env`` file in the deployment directory (``KEY=value`` lines, no secrets), and can be overridden with environment variables or ``--set KEY=VALUE``. Credentials are never put in ``install.env``. They are prompted for, read from a ``secrets.env`` file of ``KEY=value`` lines (in the deployment directory, or given with ``--secrets-file``; keep it out of version control and readable by you only), or read with ``--secret-file KEY=FILE``. They are sent to the VM over the SSH connection, so they stay out of shell history, command lines and Terraform. .. code-block:: bash cd infrastructure/gcp cat > install.env <<'EOF' DOMAIN=scaleout.mycompany.com RELEASE_BUNDLE_REF=artifacts.scaleoutsystems.com/scaleout/scaleout:v HARBOR_USERNAME=robot_ EOF ../install-remote.sh On AWS and Azure, or to run the installer yourself, wait for the first boot (``/var/lib/scaleout-bootstrapped`` exists), copy ``infrastructure/install-scaleout.sh`` to the VM and run it as root: .. code-block:: bash # From your workstation (GCP shown; use scp/ssh with the VM's public IP on AWS) gcloud compute scp ../install-scaleout.sh scaleout-vm-:~ \ --zone= --project= --tunnel-through-iap gcloud compute ssh scaleout-vm- \ --zone= --project= --tunnel-through-iap # On the VM sudo DOMAIN=scaleout.mycompany.com \ RELEASE_BUNDLE_REF=artifacts.scaleoutsystems.com/scaleout/scaleout:v \ HARBOR_USERNAME='robot_' \ ./install-scaleout.sh The script prompts for the Harbor token. It takes ``DOMAIN`` as the hostname users will reach (with Tailscale it is derived from ``TAILSCALE_HOSTNAME``). The settings below are the keys of ``install.env``, and also the environment variables of ``install-scaleout.sh``. .. list-table:: :header-rows: 1 :widths: 35 65 * - Setting - Description * - ``DOMAIN`` / ``RELEASE_BUNDLE_REF`` / ``HARBOR_USERNAME`` - Required. ``HARBOR_PASSWORD`` is prompted for unless set. * - ``SELF_SIGNED`` / ``ENABLE_HTTPS`` - ``true`` to serve HTTPS with a self-signed certificate (testing only) / with certificates you provide * - ``LETSENCRYPT_EMAIL`` / ``LETSENCRYPT_STAGING`` - Auto-issue and renew a Let's Encrypt certificate (DNS must already point at the VM and port 80 must be reachable); ``LETSENCRYPT_STAGING=true`` to test first * - ``ENABLE_TAILSCALE`` / ``TAILSCALE_HOSTNAME`` / ``TAILSCALE_AUTH_KEY`` - Join a Tailscale network and auto-renew a Tailscale-issued certificate. The auth key is prompted for unless set. Set ``private_access_only = true`` in Terraform to close the public ports. * - ``DISABLE_SIGNUP`` - ``true`` to prevent new users from self-registering * - ``GOOGLE_CLIENT_ID`` / ``GOOGLE_CLIENT_SECRET`` / ``GOOGLE_HD`` - Enable Google OAuth2 login, optionally restricted to a Workspace domain * - ``MICROSOFT_CLIENT_ID`` / ``MICROSOFT_CLIENT_SECRET`` - Enable Microsoft OAuth2 login * - ``SYSTEM_ADMIN_EMAIL`` / ``SYSTEM_ADMIN_NAME`` / ``SYSTEM_ADMIN_PASSWORD`` - Bootstrap an initial system admin user. The password is prompted for unless set. Only one of ``SELF_SIGNED``, ``LETSENCRYPT_EMAIL`` and ``ENABLE_TAILSCALE`` may be used. Installation typically takes 10–15 minutes (image pull and initial container creation). If the install fails before any service has started (for example, because the Harbor credentials were wrong), run ``install-remote.sh`` again. After the services have started, the install is not repeated, because new secrets would not match the existing data; use ``deploy/update-bundle.sh`` to update. If ``private_access_only`` is set in Terraform but Tailscale is not enabled, ``install-remote.sh`` stops and asks before installing, because nothing could then reach Scaleout from the internet. With Tailscale, the certificate can take a few minutes to arrive, and browsers report a certificate error until it does. **Monitoring startup and troubleshooting** The VM installs Docker on first boot, which takes a few minutes. To follow progress, SSH into the VM and tail the startup logs: .. only:: html .. tabs:: .. code-tab:: bash :caption: GCP gcloud compute ssh scaleout-vm- \ --zone= --project= sudo journalctl -u google-startup-scripts.service -f .. code-tab:: bash :caption: AWS ssh ubuntu@$(terraform output -raw public_ip) sudo tail -f /var/log/cloud-init-output.log .. only:: latex or rinoh *GCP:* .. code-block:: bash gcloud compute ssh scaleout-vm- \ --zone= --project= sudo journalctl -u google-startup-scripts.service -f *AWS:* .. code-block:: bash ssh ubuntu@$(terraform output -raw public_ip) sudo tail -f /var/log/cloud-init-output.log The bootstrap creates ``/var/lib/scaleout-bootstrapped`` when finished, after which ``install-scaleout.sh`` can be run. ``install-remote.sh`` shows the install's output as it runs; on the VM the same output is in ``/var/log/scaleout-install.log``. When the install script finishes, check the running containers from the deployment directory: .. code-block:: bash cd /opt/scaleout ./deploy/compose.sh ps ./deploy/compose.sh logs -f Common checks if something looks wrong: .. code-block:: bash # Docker is installed and running docker --version docker compose version sudo systemctl status docker # The release bundle was unpacked ls -la /opt/scaleout # DNS resolves to the VM nslookup DNS propagation can take 5–60 minutes depending on TTL. **Private network deployment (AWS, no public IP)** If your environment does not permit a public IP on the VM — for example because you reach AWS through a Transit Gateway, Site-to-Site VPN, or Direct Connect — set the following in ``terraform.tfvars`` to skip the Elastic IP and launch the VM in a private subnet you already operate: .. code-block:: hcl assign_public_ip = false create_vpc = false vpc_id = "vpc-xxxxxxxx" subnet_id = "subnet-xxxxxxxx" # must be a private subnet allowed_ips = ["10.0.0.0/8"] # internal CIDRs that may reach the application ssh_allowed_ips = ["10.0.0.0/8"] # internal CIDRs that may SSH into the VM Prerequisites you (the customer) must provide: - A VPC and a private subnet to host the VM, in the AWS region you chose. - Outbound internet from that subnet (typically a NAT Gateway or a forward HTTP proxy), so the VM can pull container images from ``artifacts.scaleoutsystems.com``, GitHub, Ubuntu/Docker apt repositories, and any configured OAuth identity provider endpoints during provisioning and at runtime. - An internal DNS A record for your domain pointing at the VM's private IP. Terraform prints the private IP and the required DNS record in its output after apply. - A path for administrative access — VPN, bastion host, or AWS Systems Manager Session Manager — since SSH is reachable only from inside your network. .. _https-option-b: **HTTPS certificates** The VM serves traffic over plain HTTP by default. To use a real domain and HTTPS, create an ``A`` record for your domain that points at the VM's public IP. Terraform prints the address: .. code-block:: bash terraform output public_ip Then choose a certificate method. Each install setting is passed to the setup option of the same name on the VM: put it in ``install.env`` or pass it with ``--set KEY=VALUE``. See :ref:`HTTPS certificates ` in Option A for how each method works. - *Self-signed (testing only):* ``SELF_SIGNED=true``. - *Let's Encrypt over port 80:* ``LETSENCRYPT_EMAIL`` (and optionally ``LETSENCRYPT_STAGING=true``). - *Tailscale:* ``private_access_only = true`` in ``terraform.tfvars``, and ``ENABLE_TAILSCALE=true`` plus ``TAILSCALE_HOSTNAME`` in ``install.env`` (the auth key is prompted for). The install script joins the VM to your tailnet and the ``tailscale-cert-renew`` container takes care of renewal from there. The other two methods are set up on the VM over SSH, from ``/opt/scaleout``: - *Bring your own certificate:* copy your certs into ``runtime/certs/`` as ``server.crt`` and ``server.key``, re-run ``runtime/setup-invocation.sh`` to regenerate the nginx config, then ``./deploy/compose.sh up``. - *Let's Encrypt through a DNS provider (DNS-01):* set ``LETSENCRYPT_EMAIL`` as for port 80. ``install-scaleout.sh`` does not pass the provider option. On the VM: 1. Add ``--letsencrypt-dns-provider `` to ``runtime/setup-invocation.sh`` and run it. It adds the empty setting to ``runtime/secrets.env`` and stops. 2. Fill in the credential in ``runtime/secrets.env``. 3. Run ``runtime/setup-invocation.sh`` again, then ``./deploy/compose.sh up``. ``install-scaleout.sh`` only performs the first installation. To change the certificate method on a VM that is already installed, SSH in and follow the same ``runtime/setup-invocation.sh`` steps as in Option A. **Upgrading** Upgrading the release bundle is **not** done through Terraform. To upgrade an existing deployment, SSH into the VM and run ``update-bundle.sh`` exactly as in Option A — the release bundle is installed at ``/opt/scaleout``: .. only:: html .. tabs:: .. code-tab:: bash :caption: GCP gcloud compute ssh scaleout-vm- \ --zone= --project= cd /opt/scaleout sudo bash deploy/update-bundle.sh \ -u 'robot_' \ -p '' \ --check .. code-tab:: bash :caption: AWS ssh ubuntu@$(terraform output -raw public_ip) cd /opt/scaleout sudo bash deploy/update-bundle.sh \ -u 'robot_' \ -p '' \ --check .. only:: latex or rinoh *GCP:* .. code-block:: bash gcloud compute ssh scaleout-vm- \ --zone= --project= cd /opt/scaleout sudo bash deploy/update-bundle.sh \ -u 'robot_' \ -p '' \ --check *AWS:* .. code-block:: bash ssh ubuntu@$(terraform output -raw public_ip) cd /opt/scaleout sudo bash deploy/update-bundle.sh \ -u 'robot_' \ -p '' \ --check Drop ``--check`` to perform the upgrade, or pass ``-b artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z`` to target a specific version. The script preserves ``runtime/`` (secrets and generated config) and writes a timestamped backup to the parent of ``/opt/scaleout`` before unpacking the new bundle. See :ref:`Upgrading ` for the full command reference. ---- .. _health-checks: Health checks for external load balancers and gateways ------------------------------------------------------- If you place an external load balancer or API gateway in front of the deployment, configure it to probe one of the following unauthenticated HTTP endpoints. Both return ``200 OK`` without authentication and are reachable through the built-in nginx reverse proxy. .. list-table:: :header-rows: 1 :widths: 30 25 45 * - Endpoint - Service - Notes * - ``/api/v1/health`` - REST API - Reflects the health of the API server. Reachable through the nginx ``/api/v1/`` proxy. Returns ``200 OK`` with body ``OK``. * - ``/health`` - Frontend - Returns ``200 OK`` directly. Use this instead of ``/``, which returns a ``308`` redirect (via the login/session flow) and must not be used as a health probe. For example, with the deployment reachable at ``https://scaleout.mycompany.com``: .. code-block:: bash curl -fsS https://scaleout.mycompany.com/api/v1/health # -> OK curl -fsS https://scaleout.mycompany.com/health # -> OK .. note:: nginx also answers ``/healthz`` itself (without contacting any upstream). That endpoint only confirms that nginx is up — it does **not** reflect the health of the application services. Use ``/api/v1/health`` and ``/health`` for application-level checks from external load balancers. **gRPC services** The Controller, Combiner, and Hooks services expose gRPC rather than HTTP, and implement the standard `gRPC Health Checking Protocol `_. HTTP probes will not work against them. Use `grpc_health_probe `_ instead, for example: .. code-block:: bash grpc_health_probe -addr=: This is the same mechanism the bundled Docker Compose stack uses for its container health checks. .. _admin-api-exclusion: Kratos and Hydra admin APIs must never be forwarded ------------------------------------------------------ The Kratos and Hydra admin APIs (ports ``4434`` and ``4445``) have no authentication of their own — anything that can reach them can enumerate identities, and create, modify, or delete OAuth2 clients. They exist only for internal orchestration (bootstrap jobs, the frontend's server-side actions) which all reach them directly over the Docker network, never through nginx. The bundled Docker Compose stack publishes both admin ports on the loopback address ``127.0.0.1`` only, so only tools on the host itself can reach them. Postgres (``5432``) and MinIO (``9000``, ``9001``) are published the same way. Docker's published ports bypass host firewalls such as ufw, so do not change these bindings to listen on all interfaces. To reach one of these ports from another machine, use an SSH tunnel. Through the web address of a deployment, only the public paths ``/kratos/`` and ``/hydra/`` lead to Kratos and Hydra. Requests for ``/kratos/admin`` and ``/hydra/admin`` do not reach an admin API, and the answer does not reveal any internal host name. The Docker Compose stack and the Helm chart (see :doc:`chart-security`) behave the same way. A load balancer or reverse proxy placed in front of a deployment needs no extra rules for this, because the deployment itself rejects these paths. Only if you replace the bundled nginx or the chart's ingress with your own routing must you make sure that ``/kratos/admin`` and ``/hydra/admin`` are not forwarded to the admin ports. .. _waf-file-upload: WAF and reverse proxy requirements for file uploads ---------------------------------------------------- Scaleout uploads seed models and compute packages from the browser directly to the API server in chunks via ``POST /api/v1/file-upload/``. If a Web Application Firewall (WAF) or other inspection proxy sits between the browser and the deployment, it must be configured to allow these requests through. **Why uploads fail behind a WAF** The default chunk size is 900 KB. WAF rules such as AWS ``AWSManagedRulesCommonRuleSet`` / ``SizeRestrictions_BODY`` enforce a body inspection limit (commonly 8 KB) and block requests whose bodies exceed it. Because the WAF rejects the chunk before it reaches nginx, nothing appears in the application logs — only the upload initiation and abort are visible. The symptom is an "Upload blocked by a proxy or firewall" error in the browser after every file upload attempt, while small files (under the WAF threshold) succeed. **Fix: add a WAF allow rule for the upload path** Add a rule with a lower priority number than your managed rule groups that unconditionally allows ``POST`` requests to ``/api/v1/file-upload/``: .. code-block:: json { "Name": "AllowFileUploadChunks", "Priority": 1, "Statement": { "AndStatement": { "Statements": [ { "ByteMatchStatement": { "FieldToMatch": { "UriPath": {} }, "SearchString": "/api/v1/file-upload/", "PositionalConstraint": "STARTS_WITH", "TextTransformations": [ { "Priority": 0, "Type": "NONE" } ] } }, { "ByteMatchStatement": { "FieldToMatch": { "Method": {} }, "SearchString": "POST", "PositionalConstraint": "EXACTLY", "TextTransformations": [ { "Priority": 0, "Type": "NONE" } ] } } ] } }, "Action": { "Allow": {} }, "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "AllowFileUploadChunks" } } Set ``Priority`` to a value lower than the priority of your managed rule groups so this rule evaluates first. **Temporary workaround: reduce chunk size** If you cannot modify the WAF rules immediately, you can reduce the chunk size so each request stays under the WAF inspection limit. Set the following environment variables on the ``api-server`` container and restart it: .. code-block:: yaml SCALEOUT_UPLOAD_MAX_CHUNK_SIZE: "6144" # 6 KB — safely under the 8 KB AWS default SCALEOUT_UPLOAD_MIN_CHUNK_SIZE: "2048" # 2 KB The frontend reads the chunk size from the server's init response and adapts automatically — no frontend restart is needed. Note that very small chunks increase the number of HTTP roundtrips and make large file uploads significantly slower. Revert these variables once the WAF rule is in place. To confirm the variables are active inside the container: .. code-block:: bash ./deploy/compose.sh exec api-server env | grep SCALEOUT_UPLOAD ---- .. _client-grpc-tls: Connecting clients over gRPC ---------------------------- Edge clients reach the web UI and REST API on ports 80/443. They connect to a **combiner** on the hostname and port given in its public address, described below. By default the client uses the TLS mode the combiner reports on assignment, and TLS if the combiner doesn't report one. The :ref:`SCALEOUT_GRPC_SECURE ` environment variable overrides this, except that the combiner can still upgrade it to TLS. Whether the combiner endpoint actually speaks TLS depends on how the deployment terminates it: .. list-table:: :header-rows: 1 :widths: 40 25 35 * - Deployment - Combiner gRPC - Client setting * - ``--enable-tls`` / ``--self-signed`` - TLS (terminated by nginx on the combiner ports) - None needed (the combiner reports TLS) * - Default (plain HTTP) - Plaintext (h2c) - None needed (the combiner reports plaintext) * - ``--enable-https`` (TLS terminated externally) - Plaintext from nginx, unless your external proxy is also configured to terminate TLS on the combiner ports - ``false`` unless the external proxy fronts the combiner ports with TLS. With ``--enable-tls`` or ``--self-signed``, nginx terminates TLS on the combiner ports using the same certificate as the web UI and forwards plaintext gRPC to the combiner internally, so no combiner-side certificate is needed. With ``--enable-tls`` or ``--self-signed`` and a single combiner, edge clients connect to the combiner on port 443, on the same hostname as the web UI. nginx routes their gRPC connections to the combiner, and a new deployment seeds the combiner's public address as ``:443``. Setup prints where edge clients connect. Without TLS, or with several combiners, each combiner is published on its own port (``12080``, ``12082``, …) instead. **Your own load balancer in front.** The public address is the hostname and port of the endpoint that accepts the edge clients' gRPC connections. If a load balancer or proxy sits in front of the deployment, set each combiner's public address to the load balancer's hostname and port, in the combiner's edit dialog in the UI. **Upgrading a deployment that uses the combiner's own port.** An upgrade keeps port 12080 published when the existing deployment publishes it, and records ``--enable-combiner-ports`` in the saved setup (``runtime/setup-invocation.sh``). Edge clients and load balancers that use port 12080 keep working. To move them to port 443, point the load balancer at port 443 and set the combiner's public address to its hostname and port 443. Health checks work on port 443 as well: the gRPC health check, or a plain HTTPS check of ``/healthz``. Then remove ``--enable-combiner-ports`` from the saved setup. **Self-signed certificates.** A secure-by-default client validates the combiner certificate against the system trust store, which rejects self-signed certs. Point the client at the certificate so it can verify the connection: .. code-block:: bash export SCALEOUT_GRPC_ROOT_CERT_PATH=/path/to/server.crt scaleout client start --api-url https:// --token This is the ``server.crt`` from the deployment's ``runtime/certs/`` directory. With a publicly-trusted (CA-issued) certificate, leave ``SCALEOUT_GRPC_ROOT_CERT_PATH`` unset — the system trust store handles verification. See :ref:`Client Environment Variables ` for the full list of client settings.