Mount a queue server
For the unified CLI/web connection model, see Lakeshore connections (CLI 0.107.0+). The queues mount flow below remains a compatibility entry point; existing mounts and Vault credentials are not migrated.
Host, worker and queue access are separate layers
Fabric — Add Host establishes machine identity, connectivity and management, including SSH, browser terminals and remote commands. Go directly to host enrollment. It requires an authorized execution connection in the host's DreamLake owner namespace, with no queue creation or queue mount prerequisite.
Runtime — worker process associates a process with a selected queue and consumes jobs. Configure the worker and queue association using the deployed Lakeshore tooling, start consumption, then verify a small submitted job completes. See the Lakeshore documentation for your deployment's worker and submission API/SDK. Host enrollment does not perform these Runtime steps.
Queue mount — client access configuration lets a client access queue collections. The commands below do not register a Host, register or start a worker, or submit a job. Reuse an existing authorized connection, or create a personal queue mount when that inspection registration is needed.
These are not disjoint machine categories. The same Host can support general development and also run one or more queue-consuming worker processes. A queue mount's inability to Add Host does not mean Add Host is globally unavailable.
Configure optional queue inspection access
The queue commands are present in CLI 0.81.0. Availability also depends on the
DreamLake server. Check dl --version and dl queues --help;
a successful listing does not establish enrollment or job execution capability.
-
Check your DreamLake login. These commands use the same environment and account as your other DreamLake commands. The first version supports personal DreamLake namespaces.
-
Mount your Lakeshore server. Use its HTTPS URL and choose a short prefix. The command prints the selected DreamLake server and namespace, then asks for the namespace on Lakeshore and an access token in a hidden prompt.
Paste a Lakeshore access token, created by your Lakeshore administrator. The server admin token is for creating access tokens; do not paste it here. DreamLake verifies the access token and stores it in your DreamLake Vault. The CLI does not read an admin token or reuse a saved Lakeshore CLI login.
Your DreamLake namespace and Lakeshore namespace are separate. For example,
--namespace geyang --lakeshore-namespace defaultstores the mount in DreamLake'sgeyangand accesses Lakeshore'sdefault. Neither is inferred from the other. -
Look inside the server.
labidentifies your mount. Queue names appear aslab/training, for example; the prefix does not rename the remote queue.
inspect performs a live connection check. The other commands read current
collections through DreamLake; the Lakeshore access token stays on the
DreamLake server. DreamLake controls who can use the mount, while the
Lakeshore token determines what that server permits. A prefix is a name, not
an authorization boundary.
For scripts, provide both namespaces explicitly and read a token from stdin:
The file must contain an access token, not the server admin token.
To list registrations, replace an expired or revoked access token, or remove a mount:
token prompts for a replacement token; it also accepts --token-stdin.
unmount removes the DreamLake registration. It leaves the server running,
retains its Vault entry, and does not revoke the remote token. Replacing a token
also retains the old Vault entry. Revoke unused tokens on Lakeshore separately.
All commands support --namespace <DreamLake namespace> and --json.
mounts supports --page and --page-size; jobs supports --limit and
--offset.
This first slice registers a server and reads its collections. Job submission and changes to queues or workers are separate work. For direct Python queue usage, see the dreamlake-lakeshore Python SDK.
Ownership, execution, and namespace names
A connection has several independent identifiers:
| Identifier | Meaning |
|---|---|
| DreamLake namespace | Account or organization that owns the registration |
| Connection name/prefix | Label for that registration; changing it does not rename remote resources |
| Lakeshore URL | Control plane deployment |
| Lakeshore namespace | Server-side scope selected by the access token |
| DreamLake project | Separate project authorization; not inferred from the prefix or remote namespace |
Vault-backed queues mount currently accepts only the signed-in user's personal
DreamLake namespace. On the production API checked on 2026-10-18, an org mount
was rejected with HTTP 400. An org Vault existing does not establish support for
org queue mounts. Do not silently create personal copies when the user requests
one org-owned connection, or pass an admin credential as its access token.
The Lakeshores page has a separate connection-registration flow used for execution connections. An authorized org owner can register one under the org through that flow; ordinary members receive the server-defined read permissions. This uses the connection credential store, not the personal queue-mount Vault adapter. Registering it does not grant every member execution or host enrollment rights. Verify the returned permissions and each intended operation separately.
An execution connection and matching owner namespace are necessary but not
sufficient. Its credential must explicitly authorize minting host-enrollment
grants on the deployed control plane. A normal namespace access token, an
authMode: token response, or a legacy connection label does not prove this
authority. Verify the deployed enrollment API and its credential capabilities
before claiming readiness; local source containing a scoped capability does not
prove that production supports it. Do not distribute the global admin token to
users as a workaround. If enrollment fails at grant minting, retain the existing
host identity and request ID, diagnose the authorization failure, and use the
supported retry flow after it is resolved. Do not rename or recreate the host
to bypass it.
A queue mount provides collection inspection. It is not an execution connection:
hosts enroll --lakeshore-id rejects a queue mount with
queue_mount_execution_unsupported. Obtain an explicitly authorized execution
connection before enrollment; a successful queue probe does not prove SSH,
worker enrollment, or workload readiness.
The execution connection's DreamLake owner namespace must match the host name's
namespace. For example, an execution connection owned by acme can be selected
for acme/workstations/alice-mac, subject to enrollment permissions. It cannot be
selected for alice/workstations/alice-mac: that host requires an execution
connection owned by alice. The enrollment lookup filters by owner namespace
before matching --lakeshore-id, so a cross-namespace ID can return
lakeshore_not_found even when the user can view that shared connection. A shared
Lakeshore URL, remote namespace, or token does not remove this ownership boundary.
Do not change the host's intended owner just to bypass it. See
host enrollment.
Before changing a remote namespace name, inspect the deployed API and record its immutable ID. Renaming a DreamLake connection label does not change that name. Only use a supported in-place rename that preserves the ID, resources and token bindings; verify both the new name and compatibility for active clients. Do not delete/recreate a namespace to simulate rename. A local implementation or merged commit is not proof the deployed API supports the operation. If the API rejects the name field, leave the running namespace intact and report the release dependency. Keep connection setup pending until the intended name is verified.
When a setup attempt fails ambiguously, list connections before retrying and retain the original credential receipt securely. Never mint another token just to retry registration. Tokens created for abandoned attempts should be revoked explicitly after checking that no connection uses them.
Stop a worker from claiming jobs
Requires CLI 0.92.0 or later. On a Lakeshore control plane or DreamLake deployment without the worker job-claim switch, the command reports that the switch is unavailable, and nothing changes.
A management-only host, such as a launch server, can stay reachable without
consuming any queue. Take the worker ID from dl queues workers lab:
Without a flag, the command shows the current value. --disable stops the
worker from claiming new queued jobs and tracked runs from every queue,
including default, its memberships and pool siblings. Heartbeat, control
commands, setup and remote exec keep working. The value is stored by the
control plane, so it survives daemon restarts and reconnects. Nymph needs no
configuration change. The command always reads the stored value back and prints
that, not the requested one.
Only future claims change. A job the worker already claimed keeps running; cancel
it separately if needed. A long poll already in progress stops claiming at its
next attempt. Changing the switch needs lakeshore:use on the mount.
Pin a worker to dedicated queues
Requires CLI 0.94.0 or later, and a Lakeshore control plane and DreamLake server with the worker queue pin. Otherwise the command reports that the control plane predates it, and nothing changes.
A pinned worker claims queued jobs only from its pinned queues. Its poll
queue (default for a Nymph without DREAMLAKE_QUEUE), its memberships and
pool siblings are ignored, and nothing is created automatically. Create the
queue first:
Without --queue or --clear, the command shows the stored pin. Changing it
needs lakeshore:use on the mount and ownership of the worker's host:
--host names that host, and it must be bound to WORKER_ID on this mount.
The control plane rejects default, wildcards, unknown queues and inactive
queues. While a pinned queue is archived or drained, the worker claims nothing;
it never falls back to another queue. The pin survives restarts and reconnects.
Tracked runs, heartbeat, control and exec are unaffected, and a disabled
worker-claims switch still stops all claims.
Pin a worker while its claims are disabled, read the pin back, then enable
claims. To undo, reverse the order: disable claims and confirm claimJobs: false before clearing the pin or moving to a control plane without the pin.
Otherwise the worker would claim from its poll queue again.
Maintaining the setup guide
Correct discovered setup mistakes in this source guide and its task action, then regenerate and check the DreamLake skill references. Keep account-specific URLs, token values and operational receipts in the team's private Note, not in public skills. Update that Note in place with its current revision, preserve linked setup-example directives, and read back the saved content. Track source, CLI release, docs publication, skill synchronization and live verification separately; never convert an unverified step into a success claim.
Read Slurm scheduler jobs
queues slurm requires CLI 0.57.0 or later and a DreamLake deployment with
the Slurm inventory API. Availability depends on both deployments. Existing
queues jobs <prefix> lists Lakeshore invocations; jobs list lists externally
reported tracking records. Use the Slurm commands below for scheduler jobs.
Discover configured clusters in your authorized DreamLake namespace:
Pass an exact cluster ID from that listing:
The default table shows scheduler job ID, user, job name, partition and observed state for every job in the authorized snapshot. No user filter is added. The snapshot's scope, effective Unix user, workload visibility and completeness remain visible; configured clusters and scheduler-visible jobs do not imply access to every cluster or every user's jobs.
--details shows user ID, reason, account, QoS, priority, scheduler CPU and node
counts, memory per CPU and per node in MiB, requested/allocated/per-node TRES,
submission/start/end timestamps, time limit in minutes, queue IDs, verified
allocated node IDs, canonical job and attempt IDs, revision, array identifiers
and restart count. TRES strings retain the scheduler's resource and GPU labels;
the CLI does not infer a GPU total or convert per-unit memory into total memory.
Scheduler node counts are distinct from verified allocation mappings. Start and
end times can be forecasts; they do not prove execution started or finished.
Missing, unset or invalid fields display Unknown, including details absent on
older servers. --json preserves the entire server snapshot and all allowed
fields, including raw timestamps, freshness, cache provenance and errors.
Human output includes observation, last-success and read times. Reading does not renew the observation. The server marks observations stale after 120 seconds or on cache/refresh errors, and unavailable after 24 hours. Stale output remains historical; it never establishes the current scheduler state. An unavailable or incomplete workload returns a nonzero exit code and an unknown job count rather than an empty list. These commands inspect inventory only; they do not submit, cancel or modify scheduler jobs.
Starting with CLI 0.70.1, known inventory failures include a safe error code and recovery direction in
stderr. For slurm_worker_identity_denied, an administrator should compare the
configured worker binding, enrollment identity and heartbeat freshness. An online
host alone does not establish that the inventory reader accepts that identity.
HTTP 503 leaves scheduler state unknown; it does not establish an empty or failed
cluster. Arbitrary upstream error bodies and collector output are not printed.