Provider registrations
dl providers stores provider configuration in DreamLake and associates it with your host enrollment. These commands require a server with the provider registration API from workspace PR #321. Added in CLI 0.14.0; the corresponding Python API is in 0.11.0.
Required launch boundary
Every workload or machine launch must execute through an explicitly user-configured launch server, using that server's appropriate IAM role (and scoped provider access where needed). The control-plane/service IAM role is for service infrastructure: do not use or broaden it for launches. Ambient service credentials, an operator Vault IAM user, or direct hosted-backend provider execution are not substitutes for the configured server.
Verify the selected server, its user configuration, execution identity and the deployed routing before any launch or provisioning action. If that route is missing or unverified, stop new launches and report the gap; do not select a default provider or alternate credentials. Preserve existing resources and unrelated service operation. Read-only inspection may continue.
This is the required architecture, not a claim that the current launch APIs implement it. A successful direct-backend EC2/Kubernetes smoke run is only implementation evidence. It does not satisfy this boundary. The command syntax below remains reference material; launch examples require the verified route. The launch-server workflow below targets CLI 0.71.0 or later; its backend companion must be deployed before use. Registration or binding alone does not establish live readiness.
Bind a launch server
Requires CLI 0.71.0 and a backend supporting the launch-server route.
Enroll the user-configured server with dl hosts enroll first and verify
its owned Host enrollment. The AWS role must be attached to that server. EC2
instance profiles supply short-lived credentials; do not copy the operator's
Vault AWS credentials onto the server.
Read the current provider revision first. Binding preserves its ID, launch layers, and existing workload ownership, increments the revision, and removes the legacy credential reference. For Kubernetes, this mode targets EKS with IAM authentication and requires access through the server to the private or restricted cluster endpoint. Future operations verify the exact enrollment, current signing-key lifecycle, freshness, and server IAM role. A missing or invalid binding must not fall back to hosted service credentials.
Repeat identical binding input with the same request ID after a lost response.
For an uncertain workload submission, inspect its operation receipt before any
new request; do not blindly launch again. A pending remote execution can be
read through its saved receipt. A lost remote-dispatch acknowledgement may
remain outcome-unknown and requires reconciliation. Errors include recoveryOperationId
when a durable remote receipt exists; inspect it with
dl providers operations show team/OPERATION_ID --json.
Permission errors expose only validated AWS diagnostic code, operation, region
and request ID when available, so an agent can identify the failing permission
without logging raw cloud errors.
Fix permission limitations
IAM policies can be updated after role creation; server replacement is normally
unnecessary. On the launch server, run aws sts get-caller-identity and capture
the denied action/resource/region/request ID without credentials. Use the
identity authorized to administer the dedicated launch role to update its
Terraform policy and apply a reviewed plan, then verify the original operation
after propagation. The role's own permissions need not include editing itself.
An account credential explicitly authorized for bootstrap may create the
launch server and manage its role; subsequent launches execute on that server.
Inspect the original request's receipt before retrying a write. A recorded permission rejection remains the result for that request ID. After confirming that the write was rejected and no resource was created, use a new request ID for the corrected attempt. A pending or unknown result requires reconciliation of the existing request; changing its ID could duplicate a launch.
Separate missing IAM permissions from role trust, iam:PassRole, boundaries,
organization/session/resource policies, Kubernetes RBAC and networking. An
added Allow does not override an explicit Deny. See AWS's
policy editing guide
and AccessDenied reference.
Do not repair a denial by switching to the control-plane service role.
Register a provider
Save this declaration as provider.json. Replace the cluster, partitions and Vault entry with your values. Omit credentialRef if no reference is needed; never put credential contents in the declaration.
Keep the returned resource.id, resource.revision, operationId and your request ID. Every write requires an explicit request ID. Repeating the same request returns its original receipt. Reusing the ID with a different payload returns a conflict.
Associate your Slurm enrollment
Save an association as association.json, using your enrollment ID and the provider revision. Paths must be literal absolute paths. submissionRoot is the shared directory as seen by the submission host; computeRoot is that directory as seen by the compute node. Executable paths must be valid on compute nodes.
Replace PROVIDER_ID below with the registration's resource.id.
The enrollment must belong to you in the same namespace. The partition and optional account must be permitted by the declaration. Status reports not_checked until readiness checks exist; registration does not test SSH, paths or executables, provision machines, or submit jobs. Kubernetes declarations are accepted, but Kubernetes runner associations are not yet supported.
Declare an EC2 account and region
An EC2 declaration names an account and region scope; it does not list, adopt, start, stop or terminate instances.
region is required; accountId and credentialRef are optional. Raw AWS keys
are rejected before anything is sent. Register it with the same
providers register command.
Provider responses include capabilities: the resource unit (instance for EC2,
allocation for Slurm) and each operation with its target, effect and state.
EC2 stop keeps volumes while terminate destroys the instance; Slurm cancel
targets one allocation. state is available only when DreamLake implements
the operation. It does not mean the provider is reachable or that you may act.
Use the returned states for the deployed environment: declaration support alone
does not enable lifecycle operations. EC2 instance commands require the matching
backend; Slurm execution requires an owned runner association. Legacy records
report capabilities: null.
Declare a GCE project and zone
A Compute Engine declaration names a project and zone scope; it does not list, launch, start, stop or delete VMs.
project and zone are required; credentialRef is optional. Raw service
account keys are rejected before anything is sent. GCE capabilities use the
instance unit; stop keeps persistent disks while delete destroys the VM
and its auto-delete disks. None of them is available yet.
Layer launch configuration
Launch configuration is layered with JSON Layer
files: the provider's layer, then any number of named layers in the order
you list them (--layer repeated, at most 16), then this request's layer.
Preflight resolves and validates; it never launches.
provider.jsonl:
gpu.jsonl:
More commands:
- Layers are not pinned to a provider revision.
layers listshows when a provider change makes a layer invalid. - Layer files are checked locally first; a syntax error names its line.
- EC2 refuses
userData, Slurm refusesscript, and GCE refuses boot-script metadata: enrollment is injected at launch, and secrets are not stored. - Kubernetes providers support container launch layers with a matching backend release; they create Pods on an existing cluster.
Launch EC2 instances
CLI 0.71.0 with the matching backend accepts an enrolled launch server in an aws-ec2
declaration. Its instance role supplies the AWS credentials. For an existing
provider, use providers launch-server to preserve its ID and managed instances.
Verify the launch server boundary before launching.
Legacy Vault credential declarations remain readable for inventory; they do not
authorize new launches or lifecycle writes.
- Instances are billed from launch until terminated.
- A launch is refused before reaching AWS when preflight is invalid.
- Repeating a request ID returns the same instances; it never launches twice.
list,start,stopandterminateact only on instances this provider launched (tagdreamlake:provider).- Without enrollment options, launched instances are not enrolled. To install Nymph automatically, pass a host name; an unused service account is optional:
Automatic enrollment requires --count 1 and a server supporting machine
bootstrap. Each Nymph gets its own account, created automatically when --service-account
is omitted; --lakeshore-id optionally selects a
namespace control plane. With the matching protected-delivery release, the
backend pins installer versions and delivers an instance-bound KMS envelope at
runtime; user data contains no credential. A launch
receipt is not online/readiness evidence. See Service accounts
for grants, retry behavior and revocation.
Launch a Docker image on Kubernetes
Requires CLI 0.66.0 or later and the Kubernetes launch backend. A k8s provider
launches one Pod in an existing Kubernetes namespace; provisioning the worker
machines is a separate InfraStack operation. See the
Kubernetes setup walkthrough.
The following EKS declaration requires CLI 0.71.0 and the matching backend launch-server route. Register the exact owned server enrollment and expected IAM role. The server obtains its instance-role credentials through IMDSv2, verifies them with STS, and uses short-lived EKS authentication. Legacy static kubeconfig references are retained only for inspection/migration, not new launch authority.
Save the declaration to provider.json and the following launch layer to
container.jsonl (each JSON Layer document occupies one line). Use a registry image your cluster can pull, preferably pinned
by digest. The image's entrypoint is preserved unless command is supplied.
Pods use restartPolicy: Never; receipt acceptance does not mean the image
pulled or the workload succeeded. Check its phase and logs. Reusing the same
request ID recovers the same launch. A committed receipt can outlive its Pod;
replay does not recreate a deleted Pod. For pending or
kubernetes_launch_outcome_unknown, inspect the operation and cluster before
choosing a new request ID: retries only reconcile the original Pod and never
submit another create. A new request ID intentionally starts a separate run.
start/stop, Nymph enrollment and --host-name are EC2 options, not Pod
operations. Terminating a Pod does not stop cluster charges: scale worker pools
or destroy the task's InfraStack when finished.
Recover a missing response
After a timeout or lost connection, look up the receipt using the original request ID:
The client does not retry writes automatically. If no receipt is found, retry the original command with exactly the same input and request ID. A receipt is a historical snapshot; use status for current state. Receipts are visible to their creating user.
Retire configuration
Use the current revision of the resource being retired. To retire only an association, include its ID and revision:
Retirement changes metadata. It does not stop jobs, remove hosts or delete Vault entries. Retire associations separately when you want their records marked retired. Declarations and associations are immutable in this version; editing declarations and upgrading legacy provider records are not yet supported.
Provider checks and placed runs (unreleased)
Requires the server API in workspace PR #389 and a tracked Slurm worker. These client additions are not released yet. Replace the provider and association IDs below with your owned association.
The explicit check submits a small Slurm job (1 CPU, 512 MiB, no GPU, 120-second
CP timeout) and returns a run receipt. Inspect it with dl runs status
or cancel it with dl runs cancel, using team/<run-id>. Status reads do
not submit checks. Wait for association readiness before submitting the workload;
a pending or failed check does not authorize placement.
All placement and resource flags precede --uv-run; provider placement does not
support --uvx. Omitted resources default to 1 CPU, 512 MiB and no GPU. Readiness
lasts 15 minutes and verifies submission/environment access, not reserved capacity
or GPU isolation. Starting a new check supersedes previous evidence.
After a lost check response, repeat the identical check with the same request ID.
Checks use run receipts, not providers operations. Run retries also require the
same inputs and request ID; changing resources conflicts. No write is automatically
retried. This draft still needs paired real-server and live-worker acceptance.
The protected CLI bootstrap requires Nymph 0.1.10, CLI 0.50.0 and a matching
DreamLake backend, with configured CLI checksums, machine audience, dedicated
KMS key and instance profile. The provider's account/region and launch layer's
profile ARN must match that configuration. These are release prerequisites,
not a claim that the current hosted deployment has enabled them. See
machine authentication for readiness, login separation and
recovery. Keep interactive human login in a separate Unix account from the
background dreamlake-agent account.
Web terminal after an enrolled launch
Starting in CLI 0.53.0, an instance launch with --host-name enables web terminal by default. The CLI
waits up to 300 seconds for that exact launch's enrollment and terminal access.
--wait-seconds changes the bound; --no-web-terminal disables browser SSH.
No terminal access is granted for a launch without automatic host enrollment.
The checked default enables Nymph SSH and waits for the exact signed enrollment to appear in owner-scoped terminal discovery. Opening a browser terminal creates a bounded owner-authorized session; setup needs no manual worker grant. The command never copies your personal login to the instance. A terminal timeout preserves the launched-instance receipt, reports pending, and exits nonzero. Retry with the same request ID to reuse the instance and finish setup; do not choose a new ID to recover it.
Managed agent software
For managed EC2 launches (--host-name), a matching backend accepts an optional
agentSetup object in the existing provider, named or request launch layers.
Choose versions in your project; they are not platform defaults. Preflight
resolves overrides and reports source locations before any billed launch.
agent.layer.json:
The provider must already supply the user's image, machine/network/storage and
protected bootstrap instance profile. nodeVersion is required when the object
is present. Omit either harness version to leave it uninstalled. Versions must
be exact pins; mutable tags, unknown fields and shell fragments are rejected.
startCodexDaemon: true requires Codex and creates its user startup service.
With the entire object omitted, the shared bootstrap installs no Node/npm or
agent harness and starts no agent daemon; machine enrollment and signed
readiness still run. Raw VM launches reject agentSetup without managed
enrollment rather than silently skipping it.
Only selected harnesses and their bundled DreamLake skills are checked. Vendor login remains separate from machine identity. Never embed vendor credentials in launch layers or user data. A matching backend release is required; an older server rejects the field even with a newer CLI. Existing machines and persisted launch retries keep their original installer. A changed software choice needs a new request ID.
See machine security and rollback and the self-contained project. Verify exact enrollment, completed workload, selected skills and reboot behavior before calling your preset ready.