Self-managed Deployment

This guide covers how customers deploy Scaleout Edge on their own infrastructure — either on a server they already manage, or on a cloud provider (AWS or GCP) provisioned via Terraform.

In both cases the starting point is a release bundle: a self-contained archive of deployment scripts and a pre-built Docker Compose configuration, distributed through the Scaleout Harbor registry.

Prerequisites 

A Harbor robot account (username and password) is required for both deployment paths. Your Scaleout representative will provide these credentials.

Compose deployment (on-premise)

Linux or macOS with Bash 4+ (on Windows use WSL2)
Docker Engine 24+ with the Compose plugin
Python 3.8+

Cloud deployment (Terraform)

Terraform 1.0+

Step 1: Install ORAS 

Release bundles are stored as OCI artifacts on Harbor. You need the ORAS CLI to pull them.

macOS
Linux (amd64)
Windows

brew install oras

VERSION=$(curl -fsSL https://api.github.com/repos/oras-project/oras/releases/latest \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['tag_name'].lstrip('v'))")
curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz"
tar -zxf "oras_${VERSION}_linux_amd64.tar.gz" oras
sudo mv oras /usr/local/bin/
rm "oras_${VERSION}_linux_amd64.tar.gz"

winget install -e --id ORASProject.ORAS

Verify:

oras version

Step 2: Authenticate to Harbor 

Log in with your Harbor robot account. Credentials are stored in ~/.docker/config.json and reused for subsequent commands.

oras login artifacts.scaleoutsystems.com \
  --username 'robot_<your-robot-account>' \
  --password '<your-token>'

Step 3: Pull the release bundle 

Pull the release bundle for your version. Replace vX.Y.Z with the version provided by your Scaleout representative.

oras pull artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z

This writes a scaleout-vX.Y.Z.tar.gz tarball into a dist/ directory. Change into it and extract:

cd dist
tar -xzf scaleout-vX.Y.Z.tar.gz
cd scaleout-vX.Y.Z/

To check the installed version and list all available versions on the registry, use the bundled update-bundle.sh helper (this requires an existing deployment — see Upgrading once installed):

bash deploy/update-bundle.sh \
  -u 'robot_<your-robot-account>' \
  -p '<your-token>' \
  --check

Deployment Option A: Compose (on-premise or own server)

Use this path when you have a Linux server or VM that you manage directly.

Authenticate to the container registry

The compose deployment pulls pre-built images from Harbor. Log in with Docker using the same credentials:

docker login artifacts.scaleoutsystems.com \
  --username 'robot_<your-robot-account>' \
  --password '<your-token>'

Generate runtime configuration

./deploy/setup.sh <hostname> [OPTIONS]

Replace <hostname> with the domain or IP address where Scaleout will be reachable (e.g. scaleout.mycompany.com or 192.168.1.100).

Common options:

Option	Description
`--enable-https`	Use `https://` URLs (TLS terminated externally, e.g. by a load balancer or reverse proxy)
`--enable-tls`	Terminate TLS in nginx using certs from `runtime/certs/` (implies `--enable-https`)
`--self-signed`	Generate a self-signed cert for local HTTPS testing (implies `--enable-tls`)
`--disable-signup`	Prevent new users from self-registering
`--google-client-id / --google-client-secret`	Enable Google OAuth
`--microsoft-client-id / --microsoft-client-secret`	Enable Microsoft OAuth

See DNS and TLS certificates for when to pick each HTTPS option. Run ./deploy/setup.sh --help for the full option list.

Start services

./deploy/compose.sh up

The platform is accessible at http://<hostname> once all containers are healthy.

See DNS and TLS certificates for how to point a domain at the deployment and serve traffic over HTTPS with your own certificates.

Upgrading

In-place upgrades are handled by deploy/update-bundle.sh. The script stops services, backs up the current bundle, pulls the new release from Harbor, restores runtime/ (secrets and generated config), and starts services again.

Run from inside the existing bundle directory. First check the installed version against what is available on the registry:

sudo bash deploy/update-bundle.sh \
  -u 'robot_<your-robot-account>' \
  -p '<your-token>' \
  --check

To upgrade to the latest available version, drop --check:

sudo bash deploy/update-bundle.sh \
  -u 'robot_<your-robot-account>' \
  -p '<your-token>'

To target a specific version, pass -b:

sudo bash deploy/update-bundle.sh \
  -u 'robot_<your-robot-account>' \
  -p '<your-token>' \
  -b artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z

A timestamped scaleout-backup-<version>-<date>.tar.gz is written to the parent directory before any files are touched, so the previous bundle can be restored if needed.

Note

Upgrading from v1.0.4 or earlier can fail at the “Restoring runtime configuration” step with Permission denied writing runtime/deploy.env. This is a known ownership-handling bug in the update script shipped with v1.0.4 and earlier, fixed in v1.0.5 — but since the script that runs your upgrade is the one already installed, you can hit it once on the way from an affected version. If you do, fix ownership of runtime/ and finish the upgrade manually as the account that owns your deployment (replace $(whoami) if upgrading via sudo from a different account):

sudo chown -R "$(whoami):$(whoami)" runtime
bash runtime/setup-invocation.sh
./deploy/compose.sh up

The bundle extraction step has already replaced deploy/update-bundle.sh with the fixed version, so this only comes up once — the next upgrade runs the fixed script from the start.

Managing users

Two helper scripts under deploy/ wrap the Kratos admin API for common operator tasks. Run them on the deployment host. The Compose deployment publishes the Kratos admin API on 127.0.0.1:4434 only.

Promote a user to system admin

./deploy/promote-user-docker.sh <email>

Reset a password without email

Self-service password recovery normally emails a reset link to the user. If the deployment has no SMTP/courier configured (common in evaluation or air-gapped setups), that email is generated internally but never delivered. Generate a recovery link directly instead, and hand it to the user out of band:

./deploy/reset-password-link-docker.sh <email>

This prints a one-time recovery link (default validity 1h, override with a third argument) that the user opens to set a new password through the normal flow. The script exits with an error if no identity matches the given email.

Both scripts accept an optional second argument to target a non-default Kratos admin URL, e.g. ./deploy/promote-user-docker.sh <email> http://127.0.0.1:14434 through an SSH tunnel.

Deployment Option B: Cloud (AWS or GCP via Terraform)

Use this path to provision a cloud VM automatically. The Terraform scripts are included in the release bundle under infrastructure/.

The VM bootstraps itself: Terraform creates the instance and passes a startup script that pulls the release bundle from Harbor and runs the deployment — no manual SSH required.

Cloud provider prerequisites

Before running Terraform, the cloud provider CLI must be installed and authenticated on your workstation. Terraform uses the credentials configured by the CLI to call the provider’s APIs.

GCP
AWS

# Install: https://cloud.google.com/sdk/docs/install
gcloud auth application-default login
gcloud config set project <your-gcp-project-id>

# Install: https://aws.amazon.com/cli/
aws configure

The account used by Terraform needs permission to create and manage a VM, its disk, network resources, and (optionally) snapshot schedules. Minimum required permissions:

GCP — the simplest path is to grant your account the Compute Admin and Service Account User roles on the target project. For a least-privilege setup, grant the equivalent compute.instances.*, compute.disks.*, compute.networks.*, compute.firewalls.*, compute.addresses.*, compute.resourcePolicies.* (only if enable_disk_snapshots = true), and iam.serviceAccounts.actAs permissions. The project must also have billing enabled and the Compute Engine API enabled.

AWS — the account needs permissions on EC2 (instances, key pairs, security groups), VPC (VPCs, subnets, route tables, internet gateways — only if create_vpc = true), and IAM + DLM (only if enable_disk_snapshots = true). The AWS-managed policies AmazonEC2FullAccess and AmazonVPCFullAccess cover the common case; add IAMFullAccess and AWSDataLifecycleManagerFullAccess if snapshots are enabled.

Configure your deployment

Copy the example variables file for your cloud provider and fill in your values:

GCP
AWS

cd infrastructure/gcp/
cp terraform.tfvars.example terraform.tfvars

cd infrastructure/aws/
cp terraform.tfvars.example terraform.tfvars

Open terraform.tfvars and set at minimum:

GCP
AWS

project_id      = "your-gcp-project-id"
region          = "us-central1"
zone            = "us-central1-a"
domain          = "scaleout.mycompany.com"
harbor_username = "robot_<your-robot-account>"
harbor_password = "<your-token>"

region          = "eu-north-1"
domain          = "scaleout.mycompany.com"
harbor_username = "robot_<your-robot-account>"
harbor_password = "<your-token>"

release_bundle_ref is pre-filled with the correct version. Replace the Harbor credential placeholders with the values provided by your Scaleout representative.

Terraform variables reference

The tables below cover the variables most customers will need to touch. See terraform.tfvars.example for the full list of optional settings (OAuth providers, HTTPS, SSH access, system admin bootstrap, snapshot scheduling).

Common variables (both providers)

Variable	Description	Default
`domain`	Hostname where Scaleout will be reachable	(required)
`deployment_name`	Unique name used as a suffix on cloud resources	`default`
`release_bundle_ref`	Full OCI reference to the release bundle on Harbor	pre-filled
`harbor_username`	Harbor robot account username	(required)
`harbor_password`	Harbor robot account password / token	(required)
`enable_https`	Use `https://` URLs and secure cookies (requires valid certs)	`false`
`self_signed`	Generate a self-signed cert for HTTPS (testing only)	`false`
`letsencrypt_email` / `letsencrypt_staging`	Auto-issue and renew a real cert via Let’s Encrypt (requires DNS already pointed at the deployment; port 80 must be reachable unless DNS-01 is selected on the host, see DNS and TLS certificates)	`""` / `false`
`enable_tailscale` / `tailscale_auth_key` / `tailscale_hostname`	Join a Tailscale network and auto-renew a Tailscale-issued cert; VM has no public IP	`false` / `""` / `""`
`disable_signup`	Prevent new users from self-registering	`false`
`google_client_id` / `google_client_secret`	Enable Google OAuth2 login	`""`
`microsoft_client_id` / `microsoft_client_secret`	Enable Microsoft OAuth2 login	`""`
`system_admin_email` / `system_admin_name` / `system_admin_password`	Bootstrap an initial system admin user	`""`
`ssh_public_key`	SSH public key authorised on the VM	`""`
`allowed_ips` / `ssh_allowed_ips`	CIDR ranges permitted to reach the app / SSH	`["0.0.0.0/0"]`
`enable_disk_snapshots`	Take daily snapshots of the boot disk	`false`
`snapshot_retention_days`	Number of daily snapshots to retain	`7`

GCP-specific

Variable	Description	Default
`project_id`	GCP project ID	(required)
`region`	GCP region	`us-central1`
`zone`	GCP zone	`us-central1-a`
`machine_type`	VM machine type	`e2-standard-4`
`disk_size_gb`	Boot disk size in GB	`50`
`recovery_snapshot`	Snapshot name to restore the VM from	`""`

AWS-specific

Variable	Description	Default
`region`	AWS region	`us-east-1`
`instance_type`	EC2 instance type	`t3.xlarge`
`disk_size_gb`	Root volume size in GB	`50`
`create_vpc`	Create a new VPC for the deployment	`true`
`vpc_cidr`	CIDR for the new VPC	`10.0.0.0/16`

Apply

Initialise the working directory, review what Terraform is about to create, then apply:

terraform init
terraform plan
terraform apply

Running terraform plan first lets you confirm the resources, regions, and instance sizes before any cloud charges accrue. Terraform then provisions the VM, waits for the startup script to complete, and prints the application URL when done.

See DNS and TLS certificates for how to point your domain at the VM and (optionally) replace the default certificates.

Monitoring startup and troubleshooting

The startup script runs after the VM is created and typically takes 10–15 minutes on first boot (image pull and initial container creation). To follow progress, SSH into the VM and tail the startup logs:

GCP
AWS

gcloud compute ssh scaleout-vm-<deployment-name> \
    --zone=<zone> --project=<project-id>

sudo journalctl -u google-startup-scripts.service -f

ssh ubuntu@$(terraform output -raw public_ip)

sudo tail -f /var/log/cloud-init-output.log

The script writes /tmp/scaleout-startup-complete when finished. Check the running containers from the deployment directory:

cd /opt/scaleout
./deploy/compose.sh ps
./deploy/compose.sh logs -f

Common checks if something looks wrong:

# Docker is installed and running
docker --version
docker compose version
sudo systemctl status docker

# The release bundle was unpacked
ls -la /opt/scaleout

# DNS resolves to the VM
nslookup <your-domain>

DNS propagation can take 5–60 minutes depending on TTL.

Private network deployment (AWS, no public IP)

If your environment does not permit a public IP on the VM — for example because you reach AWS through a Transit Gateway, Site-to-Site VPN, or Direct Connect — set the following in terraform.tfvars to skip the Elastic IP and launch the VM in a private subnet you already operate:

assign_public_ip = false
create_vpc       = false
vpc_id           = "vpc-xxxxxxxx"
subnet_id        = "subnet-xxxxxxxx"   # must be a private subnet
allowed_ips      = ["10.0.0.0/8"]       # internal CIDRs that may reach the application
ssh_allowed_ips  = ["10.0.0.0/8"]       # internal CIDRs that may SSH into the VM

Prerequisites you (the customer) must provide:

A VPC and a private subnet to host the VM, in the AWS region you chose.
Outbound internet from that subnet (typically a NAT Gateway or a forward HTTP proxy), so the VM can pull container images from artifacts.scaleoutsystems.com, GitHub, Ubuntu/Docker apt repositories, and any configured OAuth identity provider endpoints during provisioning and at runtime.
An internal DNS A record for domain pointing at the VM’s private IP. Terraform prints the private IP and the required DNS record in its output after apply.
A path for administrative access — VPN, bastion host, or AWS Systems Manager Session Manager — since SSH is reachable only from inside your network.

Private network deployment (AWS, no public IP)

If your environment does not permit a public IP on the VM — for example because you reach AWS through a Transit Gateway, Site-to-Site VPN, or Direct Connect — set the following in terraform.tfvars to skip the Elastic IP and launch the VM in a private subnet you already operate:

assign_public_ip = false
create_vpc       = false
vpc_id           = "vpc-xxxxxxxx"
subnet_id        = "subnet-xxxxxxxx"   # must be a private subnet
allowed_ips      = ["10.0.0.0/8"]       # internal CIDRs that may reach the application
ssh_allowed_ips  = ["10.0.0.0/8"]       # internal CIDRs that may SSH into the VM

Prerequisites you (the customer) must provide:

A VPC and a private subnet to host the VM, in the AWS region you chose.
Outbound internet from that subnet (typically a NAT Gateway or a forward HTTP proxy), so the VM can pull container images from harbor.scaleoutsystems.com, GitHub, Ubuntu/Docker apt repositories, and any configured OAuth identity provider endpoints during provisioning and at runtime.
An internal DNS A record for domain pointing at the VM’s private IP. Terraform prints the private IP and the required DNS record in its output after apply.
A path for administrative access — VPN, bastion host, or AWS Systems Manager Session Manager — since SSH is reachable only from inside your network.

Upgrading

Upgrading the release bundle is not done through Terraform. Bumping release_bundle_ref and re-running terraform apply only takes effect on a fresh VM. To upgrade an existing deployment, SSH into the VM and run update-bundle.sh exactly as in Option A — the release bundle is installed at /opt/scaleout:

GCP
AWS

gcloud compute ssh scaleout-vm-<deployment-name> \
    --zone=<zone> --project=<project-id>

cd /opt/scaleout
sudo bash deploy/update-bundle.sh \
    -u 'robot_<your-robot-account>' \
    -p '<your-token>' \
    --check

ssh ubuntu@$(terraform output -raw public_ip)

cd /opt/scaleout
sudo bash deploy/update-bundle.sh \
    -u 'robot_<your-robot-account>' \
    -p '<your-token>' \
    --check

Drop --check to perform the upgrade, or pass -b artifacts.scaleoutsystems.com/scaleout/scaleout:vX.Y.Z to target a specific version. The script preserves runtime/ (secrets and generated config) and writes a timestamped backup to the parent of /opt/scaleout before unpacking the new bundle. See Upgrading for the full command reference.

After upgrading, update release_bundle_ref in terraform.tfvars to match the version now running on the VM so future re-creations stay in sync.

DNS and TLS certificates 

Both deployment options serve traffic over plain HTTP by default. To use a real domain and HTTPS:

1. Point DNS at the deployment

Create an A record for your domain that points at the deployment’s public IP. For Option B, Terraform prints the address:

terraform output public_ip

For Option A, use the public IP of the host running the compose stack. DNS propagation usually completes within 5–10 minutes for low TTLs.

2. Enable HTTPS

You have four options for TLS termination. Let’s Encrypt and Tailscale renew automatically forever once set up; self-signed and BYO-cert are unchanged manual options for offline/internal use.

Self-signed certificate (testing only): deploy/setup.sh can generate a self-signed cert and configure nginx to terminate TLS with it. Re-run setup.sh with --self-signed (Option A) or set self_signed = true in terraform.tfvars (Option B). Browsers will warn about the certificate — useful for internal testing, not production.

./deploy/setup.sh <hostname> --self-signed
./deploy/compose.sh up

Bring your own certificate (production): drop a valid certificate and private key into the deployment’s runtime/certs/ directory as server.crt and server.key, then re-run setup with --enable-tls (which expects certs to already be present) and restart the stack:

# Place fullchain certificate and private key in runtime/certs/
cp /path/to/fullchain.pem runtime/certs/server.crt
cp /path/to/privkey.pem   runtime/certs/server.key

./deploy/setup.sh <hostname> --enable-tls
./deploy/compose.sh up

For Option B, the same files live at /opt/scaleout/runtime/certs/ on the VM. Copy your certs into that directory over SSH and re-run runtime/setup-invocation.sh to regenerate the nginx config, then ./deploy/compose.sh up.

Let’s Encrypt (production, real public domain): setup.sh generates a placeholder self-signed cert as before, and a certbot container issues a real one on startup and keeps renewing it forever — no further action is ever required. Let’s Encrypt checks domain control with one of two methods. HTTP-01 is the default. Use DNS-01 when the deployment is not reachable on port 80 from the internet.

With either method, --letsencrypt-email must be combined with --enable-tls explicitly. Add --letsencrypt-staging to test against Let’s Encrypt’s staging environment first.

Method 1: HTTP-01 (default)

Let’s Encrypt fetches a token from the deployment over port 80.

Port 80 must be reachable from the internet.
DNS must already point at the deployment (step 1 above).

./deploy/setup.sh <hostname> --enable-tls --letsencrypt-email you@company.com
./deploy/compose.sh up

For Option B, set letsencrypt_email (and optionally letsencrypt_staging = true) in terraform.tfvars instead, then terraform apply.

Method 2: DNS-01

Let’s Encrypt checks a TXT record named _acme-challenge.<hostname> in your DNS zone. The certbot container writes that record through the DNS provider’s API. Port 80 does not need to be open.

The DNS zone must be hosted at one of the supported providers below.
You create a credential for that provider once, and add it to runtime/secrets.env.
The hostname appears in public Certificate Transparency logs when the certificate is issued, even though the deployment is not reachable from outside.

Every method issues a certificate for one hostname. Wildcard certificates are not supported.

Setup for Method 2, in order:

Run setup with the provider option:
```
./deploy/setup.sh <hostname> --enable-tls --letsencrypt-email you@company.com --letsencrypt-dns-provider <provider>
```
On the first run, setup creates runtime/secrets.env and then stops, because the provider’s credential is not in it yet. The message names the setting to add.
Add the provider’s credential to runtime/secrets.env. The table below lists the setting for each provider, and the steps follow it. For example, for Cloudflare:
```
ACME_DNS_CLOUDFLARE_API_TOKEN=<token>
```
Run the same setup command again. Setup checks the credential, and passes only the ACME_DNS_* lines of runtime/secrets.env to the certbot container.
Start the stack:
```
./deploy/compose.sh up
```

Provider	`--letsencrypt-dns-provider`	Credential in `runtime/secrets.env`
Hurricane Electric	`he`	`ACME_DNS_HE_DDNS_KEY`: dynamic DNS key of the challenge record
Cloudflare	`cloudflare`	`ACME_DNS_CLOUDFLARE_API_TOKEN`: API token with Zone:Read and DNS:Edit

Hurricane Electric (he):

Log in at dns.he.net. Create a TXT record named _acme-challenge.<hostname>. Give it any placeholder value. Dynamic DNS can only update a record that already exists.
Set the record’s TTL to the lowest value HE offers. A longer TTL lets resolvers keep the previous challenge value.
Enable dynamic DNS on the record and generate its key. Use that key as ACME_DNS_HE_DDNS_KEY.

Cloudflare (cloudflare):

In the Cloudflare dashboard, create an API token with the Zone:Read and DNS:Edit permissions for the zone that holds the hostname. Use that token as ACME_DNS_CLOUDFLARE_API_TOKEN.

If something is wrong, setup stops with a message that names what to fix:

--letsencrypt-dns-provider is given without --letsencrypt-email.
The provider name is not in the table. The message lists the names that are available.
The provider’s credential is missing or empty in runtime/secrets.env.

Errors from the provider during issuance, such as a wrong key or token, show in ./deploy/compose.sh logs certbot. The container retries with a growing delay, up to once an hour.

To go back to HTTP-01, run setup again without --letsencrypt-dns-provider. Switching method in either direction issues a new certificate once. The existing certificate keeps serving until the new one is issued.

Adding a DNS provider: each provider is one file in deploy/certbot/dns/, named after the provider. The file handles three calls: check (the provider’s credentials are set), auth (create the TXT record) and cleanup (remove it). It reads its own ACME_DNS_<PROVIDER>_* settings and uses only the Python standard library, which the certbot/certbot image includes. Add a row to the table and a short setup section here.

Let’s Encrypt’s --letsencrypt-email is the contact address for account and expiry notices, so use a mailbox your operators read.

For Option B, set letsencrypt_email in terraform.tfvars as for HTTP-01. Terraform does not pass the provider option. On the VM, over SSH:

Add --letsencrypt-dns-provider <provider> to runtime/setup-invocation.sh.
Add the credential to runtime/secrets.env.
Run runtime/setup-invocation.sh again, then ./deploy/compose.sh up.

Tailscale (production, `.ts.net` hostname):* for deployments reachable only over Tailscale, a tailscale-cert-renew container renews the Tailscale-issued cert automatically. Run tailscale up on the host first:

./deploy/setup.sh <tailscale-hostname> --enable-tls --tailscale-hostname <tailscale-hostname>
./deploy/compose.sh up

For Option B, set enable_tailscale = true plus tailscale_auth_key/tailscale_hostname in terraform.tfvars — Terraform joins the VM to your tailnet and the tailscale-cert-renew container takes care of renewal from there.

Let’s Encrypt and Tailscale are mutually exclusive with each other and with --self-signed/self_signed — Let’s Encrypt cannot validate a *.ts.net hostname, because that zone is not under your control.

Upgrading an already-running deployment in place: for Option A, edit runtime/setup-invocation.sh to add --letsencrypt-email ... (or --tailscale-hostname ...), re-run it, then ./deploy/compose.sh up again. The existing cert keeps serving traffic until the renewal container’s first successful issuance replaces it — no manual cert deletion or extended downtime. For Option B, note that metadata_startup_script/user_data changes are ignored by Terraform on an already-applied VM (the “reboot recovery” path skips re-provisioning), so adding letsencrypt_email/tailscale_hostname to terraform.tfvars won’t take effect via terraform apply alone — SSH in and follow the same runtime/setup-invocation.sh steps as Option A instead.

Health checks for external load balancers and gateways 

If you place an external load balancer or API gateway in front of the deployment, configure it to probe one of the following unauthenticated HTTP endpoints. Both return 200 OK without authentication and are reachable through the built-in nginx reverse proxy.

Endpoint	Service	Notes
`/api/v1/health`	REST API	Reflects the health of the API server. Reachable through the nginx `/api/v1/` proxy. Returns `200 OK` with body `OK`.
`/health`	Frontend	Returns `200 OK` directly. Use this instead of `/`, which returns a `308` redirect (via the login/session flow) and must not be used as a health probe.

For example, with the deployment reachable at https://scaleout.mycompany.com:

curl -fsS https://scaleout.mycompany.com/api/v1/health   # -> OK
curl -fsS https://scaleout.mycompany.com/health          # -> OK

Note

nginx also answers /healthz itself (without contacting any upstream). That endpoint only confirms that nginx is up — it does not reflect the health of the application services. Use /api/v1/health and /health for application-level checks from external load balancers.

gRPC services

The Controller, Combiner, and Hooks services expose gRPC rather than HTTP, and implement the standard gRPC Health Checking Protocol. HTTP probes will not work against them. Use grpc_health_probe instead, for example:

grpc_health_probe -addr=<host>:<grpc-port>

This is the same mechanism the bundled Docker Compose stack uses for its container health checks.

Kratos and Hydra admin APIs must never be forwarded 

The Kratos and Hydra admin APIs (ports 4434 and 4445) have no authentication of their own — anything that can reach them can enumerate identities, and create, modify, or delete OAuth2 clients. They exist only for internal orchestration (bootstrap jobs, the frontend’s server-side actions) which all reach them directly over the Docker network, never through nginx.

The bundled Docker Compose stack publishes both admin ports on the loopback address 127.0.0.1 only, so only tools on the host itself can reach them. Postgres (5432) and MinIO (9000, 9001) are published the same way. Docker’s published ports bypass host firewalls such as ufw, so do not change these bindings to listen on all interfaces. To reach one of these ports from another machine, use an SSH tunnel.

The bundled nginx configuration (deploy/nginx.template/nginx.conf.template) blocks /kratos/admin/ and /hydra/admin/ with an explicit 403 at the edge. If you front a deployment with your own reverse proxy or load balancer instead of the bundled nginx, you must not forward these two paths to the backend. Only the public paths — /kratos/ and /hydra/ — are meant to be reachable from outside.

This mirrors the Helm chart deployment (charts/scaleout), whose Ingress already excludes both admin ports by design and routes only the public Kratos and Hydra ports — see Security & Compliance. Keep both deployment types consistent: any future change to admin-path exposure should preserve this exclusion on both.

WAF and reverse proxy requirements for file uploads 

Scaleout uploads seed models and compute packages from the browser directly to the API server in chunks via POST /api/v1/file-upload/. If a Web Application Firewall (WAF) or other inspection proxy sits between the browser and the deployment, it must be configured to allow these requests through.

Why uploads fail behind a WAF

The default chunk size is 900 KB. WAF rules such as AWS AWSManagedRulesCommonRuleSet / SizeRestrictions_BODY enforce a body inspection limit (commonly 8 KB) and block requests whose bodies exceed it. Because the WAF rejects the chunk before it reaches nginx, nothing appears in the application logs — only the upload initiation and abort are visible.

The symptom is an “Upload blocked by a proxy or firewall” error in the browser after every file upload attempt, while small files (under the WAF threshold) succeed.

Fix: add a WAF allow rule for the upload path

Add a rule with a lower priority number than your managed rule groups that unconditionally allows POST requests to /api/v1/file-upload/:

{
    "Name": "AllowFileUploadChunks",
    "Priority": 1,
    "Statement": {
        "AndStatement": {
            "Statements": [
                {
                    "ByteMatchStatement": {
                        "FieldToMatch": { "UriPath": {} },
                        "SearchString": "/api/v1/file-upload/",
                        "PositionalConstraint": "STARTS_WITH",
                        "TextTransformations": [
                            { "Priority": 0, "Type": "NONE" }
                        ]
                    }
                },
                {
                    "ByteMatchStatement": {
                        "FieldToMatch": { "Method": {} },
                        "SearchString": "POST",
                        "PositionalConstraint": "EXACTLY",
                        "TextTransformations": [
                            { "Priority": 0, "Type": "NONE" }
                        ]
                    }
                }
            ]
        }
    },
    "Action": { "Allow": {} },
    "VisibilityConfig": {
        "SampledRequestsEnabled": true,
        "CloudWatchMetricsEnabled": true,
        "MetricName": "AllowFileUploadChunks"
    }
}

Set Priority to a value lower than the priority of your managed rule groups so this rule evaluates first.

Temporary workaround: reduce chunk size

If you cannot modify the WAF rules immediately, you can reduce the chunk size so each request stays under the WAF inspection limit. Set the following environment variables on the api-server container and restart it:

SCALEOUT_UPLOAD_MAX_CHUNK_SIZE: "6144"   # 6 KB — safely under the 8 KB AWS default
SCALEOUT_UPLOAD_MIN_CHUNK_SIZE: "2048"   # 2 KB

The frontend reads the chunk size from the server’s init response and adapts automatically — no frontend restart is needed. Note that very small chunks increase the number of HTTP roundtrips and make large file uploads significantly slower. Revert these variables once the WAF rule is in place.

To confirm the variables are active inside the container:

./deploy/compose.sh exec api-server env | grep SCALEOUT_UPLOAD

Connecting clients over gRPC 

Edge clients reach the web UI and REST API on ports 80/443. They connect to a combiner on the hostname and port given in its public address, described below. By default the client uses the TLS mode the combiner reports on assignment, and TLS if the combiner doesn’t report one. The SCALEOUT_GRPC_SECURE environment variable overrides this, except that the combiner can still upgrade it to TLS.

Whether the combiner endpoint actually speaks TLS depends on how the deployment terminates it:

Deployment	Combiner gRPC	Client setting
`--enable-tls` / `--self-signed`	TLS (terminated by nginx on the combiner ports)	None needed (the combiner reports TLS)
Default (plain HTTP)	Plaintext (h2c)	None needed (the combiner reports plaintext)
`--enable-https` (TLS terminated externally)	Plaintext from nginx, unless your external proxy is also configured to terminate TLS on the combiner ports	`false` unless the external proxy fronts the combiner ports with TLS.

With --enable-tls or --self-signed, nginx terminates TLS on the combiner ports using the same certificate as the web UI and forwards plaintext gRPC to the combiner internally, so no combiner-side certificate is needed.

With --enable-tls or --self-signed and a single combiner, edge clients connect to the combiner on port 443, on the same hostname as the web UI. nginx routes their gRPC connections to the combiner, and a new deployment seeds the combiner’s public address as <hostname>:443. Setup prints where edge clients connect.

Without TLS, or with several combiners, each combiner is published on its own port (12080, 12082, …) instead.

Your own load balancer in front. The public address is the hostname and port of the endpoint that accepts the edge clients’ gRPC connections. If a load balancer or proxy sits in front of the deployment, set each combiner’s public address to the load balancer’s hostname and port, in the combiner’s edit dialog in the UI.

Upgrading a deployment that uses the combiner’s own port. An upgrade keeps port 12080 published when the existing deployment publishes it, and records --enable-combiner-ports in the saved setup (runtime/setup-invocation.sh). Edge clients and load balancers that use port 12080 keep working. To move them to port 443, point the load balancer at port 443 and set the combiner’s public address to its hostname and port 443. Health checks work on port 443 as well: the gRPC health check, or a plain HTTPS check of /healthz. Then remove --enable-combiner-ports from the saved setup.

Self-signed certificates. A secure-by-default client validates the combiner certificate against the system trust store, which rejects self-signed certs. Point the client at the certificate so it can verify the connection:

export SCALEOUT_GRPC_ROOT_CERT_PATH=/path/to/server.crt
scaleout client start --api-url https://<hostname> --token <TOKEN>

This is the server.crt from the deployment’s runtime/certs/ directory. With a publicly-trusted (CA-issued) certificate, leave SCALEOUT_GRPC_ROOT_CERT_PATH unset — the system trust store handles verification.

See Client Environment Variables for the full list of client settings.