AWS SSM secret backend

Keep Preloop secret values in AWS Systems Manager Parameter Store as KMS-encrypted SecureString parameters, so the control database holds only references.

Preloop’s default credential backends keep secret values on the host: the private file store, or the operating-system keychain. A hosted, multi-tenant engine must not be able to produce a secret value from a copy of its database, its state directory, or a backup of either. The ssm backend puts every value in AWS Systems Manager Parameter Store as a SecureString, encrypted by KMS, and leaves the database with references only.

The backend is compiled only when the aws-ssm feature is enabled, so the shipped CLI carries no AWS SDK:

cargo build -p preloop-cli --features local-engine,aws-ssm

​

Configuration

Select it like any other backend, through the environment or the config file:

export PRELOOP_CREDENTIAL_STORE=ssm
export AWS_REGION=eu-west-1
export PRELOOP_SSM_PREFIX=/preloop/prod
export PRELOOP_SSM_KMS_KEY=alias/preloop-secrets
preloop serve

# config.toml
credential_store = "ssm"

Setting	Default	Purpose
PRELOOP_CREDENTIAL_STORE (or credential_store)	file	ssm selects this backend. Selecting it in a build without the aws-ssm feature is a startup error, never a fallback to a local file.
PRELOOP_SSM_PREFIX	/preloop	Hierarchy holding every value. Must start with /, must not end with /, and its first level must not be aws or ssm.
PRELOOP_SSM_KMS_KEY	Account default alias/aws/ssm	KMS key, alias, or ARN for new values (KeyId on PutParameter). Reads always request decryption, so values written under another key still resolve while the role may decrypt it.
PRELOOP_SSM_ENDPOINT	Unset (AWS)	Override the endpoint for LocalStack or a test double. Region and credentials still come from the standard AWS chain; a local endpoint with no region in the chain signs as us-east-1.
AWS_REGION / AWS_DEFAULT_REGION, profile, AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE, IMDS, ECS/EKS	—	Region and credentials come from the standard AWS chain, including profiles, SSO, instance profiles, and workload identity.

Each credential reference becomes one parameter named <prefix>/<reference>. References are validated, not rewritten: a reference that is not a legal SSM name segment (a namespaced namespace:reference, for example) is rejected before any request, so the engine can never read or write a different parameter than the one it asked for.

​

IAM policy

Least privilege for one prefix in one account, with values encrypted by a dedicated customer-managed key:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWritePreloopParameters",
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:PutParameter",
        "ssm:DeleteParameter"
      ],
      "Resource": "arn:aws:ssm:eu-west-1:111122223333:parameter/preloop/prod/*"
    },
    {
      "Sid": "UsePreloopKey",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
    }
  ]
}

• ssm:GetParameter covers both reads and the availability probe, which reads a never-written leaf, .availability-probe, under the prefix; a ParameterNotFound answer is what “reachable” means. ssm:GetParameters and ssm:GetParametersByPath are not used by this backend.
• ssm:PutParameter with Overwrite=true creates and rotates values; ssm:DeleteParameter removes them.
• The key needs kms:Decrypt for reads and kms:GenerateDataKey for writes. With the account-default alias/aws/ssm key, kms:Decrypt and kms:GenerateDataKey on * are usually granted by the AWS-managed policy; a dedicated key keeps the blast radius per deployment.
• No ssm:DescribeParameters, no ssm:AddTagsToResource, and no wildcard resource: the prefix scopes every call.

A shared account can host several deployments under separate prefixes (/preloop/prod, /preloop/staging); grant each role only its own parameter<prefix>/*.

​

Limits and throughput

Limit	Value	What it means here
Value size	4,096 characters (4 KiB for ASCII) per Standard parameter	set rejects a longer value before the request with a message naming the limit. Advanced parameters allow 8,192 but bill per parameter per month, so this backend does not use them.
Parameter name	1,011 characters including hierarchy	A reference is at most 255 characters, so a sane prefix always fits; an over-long name is rejected locally.
Free tier	10,000 Standard parameters per account and Region	One parameter per stored credential. Rotating a value versions it; it does not add a parameter.
Default throughput	40 transactions per second per account and Region, shared	Enough for a control plane that resolves a secret per job; a burst that exceeds it is answered with ThrottlingException (see below).
Parameter versions	100 per parameter	Older versions are dropped automatically; values are small, so this is a rotation history, not a leak.
Encrypt/decrypt	KMS request quota	One kms:Decrypt and one kms:GenerateDataKey per read and write of a SecureString.

​

Retries and failures

ThrottlingException and TooManyUpdates are retried with bounded exponential backoff: four attempts total, starting at 100 ms and doubling to a 1.6 s ceiling. The SDK’s own retry layer is disabled for this backend so there is exactly one policy and a bounded number of requests per call.

Every other failure is an error, never a missing secret:

• An access denial surfaces as AccessDeniedException with the parameter name, so a broken policy fails a job loudly instead of looking like a deleted credential.
• A missing parameter is the only condition reported as “no value”; delete treats ParameterNotFound as success, because the requested end state already holds.
• Network and DNS failures surface with their cause chain.

Values never appear in logs or error messages. Parameter names are identifiers and do appear; they are derived from credential references such as github-pat or engine-token-<digest>, not from secret material.

​

Operational notes

• The engine keeps the file store out of the write path when ssm is selected: a failed write fails the command. It never falls back to a local file, which would put a tenant’s password on the engine host.
• Startup and preloop doctor exercise the same backend (the sealing key and the engine’s own GitHub credentials are credential-store references), so a missing region, a denied role, or an unreachable endpoint fails there, naming the probe, before the first job runs.
• Rotating the KMS key is an SSM/KMS operation; the engine reads whatever the parameter’s current key is, so no engine restart is needed after a re-encryption.
• Deleting the hierarchy (aws ssm delete-parameters --names ...) makes every stored credential absent; jobs that need one fail closed, and a job that already resolved its values keeps them for its remaining lifetime.