Skip to content

feat(k8s): support configuring workspace PVC storageClassName - #2463

Open
loveRhythm1990 wants to merge 1 commit into
NVIDIA:mainfrom
loveRhythm1990:lr90/k8s-workspace-storage-class
Open

feat(k8s): support configuring workspace PVC storageClassName#2463
loveRhythm1990 wants to merge 1 commit into
NVIDIA:mainfrom
loveRhythm1990:lr90/k8s-workspace-storage-class

Conversation

@loveRhythm1990

Copy link
Copy Markdown
Contributor

Summary

The Kubernetes compute driver's default workspace PVC never set storageClassName, so on clusters with no default StorageClass the PVC stayed Pending and sandbox creation failed. This adds a driver-level workspace_storage_class option so operators can pin the workspace PVC to a specific StorageClass.

Related Issue

Closes #2442

Changes

  • Add workspace_storage_class: String to KubernetesComputeConfig (empty default preserves current behavior — omit storageClassName, use the cluster default StorageClass).
  • Thread it through SandboxPodParams into default_workspace_volume_claim_templates(), setting storageClassName on the generated PVC only when non-empty.
  • Read OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS in the standalone driver main.rs, mirroring the existing storage-size env var.
  • Expose server.workspaceStorageClass in the Helm chart (values.yaml + gateway-config.yaml), regenerate the chart README.md via helm-docs.
  • Document the field in docs/reference/gateway-config.mdx and docs/reference/sandbox-compute-drivers.mdx.
  • Add a sandbox-PVC-pending troubleshooting row to the debug-openshell-cluster skill.

No proto/gRPC change — this is a driver-level default at the same granularity as the existing workspace_default_storage_size.

Testing

  • mise run pre-commit passes (one pre-existing, unrelated failure in openshell-supervisor-network::test_forward_public_ip_allowed_without_allowed_ips — an environment DNS-interception flake that also fails on clean main; untouched crate)
  • Unit tests added/updated — new tests cover storageClassName set when provided and omitted when empty (driver.rs), plus config default/serde-override (config.rs)
  • E2E tests added/updated (if applicable) — not run here; change is covered by unit tests and verified via helm template rendering both with and without the value set

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable) — user-facing docs + Helm README + debug skill updated

@copy-pr-bot

copy-pr-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

The Kubernetes driver's default workspace PVC never set storageClassName,
so on clusters with no default StorageClass the PVC stayed Pending and
sandbox creation failed.

Add a workspace_storage_class option to KubernetesComputeConfig, wired
through SandboxPodParams into the generated volumeClaimTemplates. When
non-empty it sets storageClassName; empty preserves the current behavior
of relying on the cluster default StorageClass.

Expose it via the OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS env var on both
the standalone driver and the embedded gateway runtime defaults, and via
the server.workspaceStorageClass Helm value.

Closes NVIDIA#2442

Signed-off-by: lr90 <qiuweimin@matrixorigin.cn>
@loveRhythm1990
loveRhythm1990 force-pushed the lr90/k8s-workspace-storage-class branch from 17a15d8 to 9a705b3 Compare July 24, 2026 11:45
@giladk1221

giladk1221 commented Jul 26, 2026

Copy link
Copy Markdown

we really need this fix and would greatly appreciate your help in pushing it forward.

@drew

drew commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 9a705b3

@drew drew left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

Validation: This PR is project-valid because it implements the maintainer-triaged Kubernetes workspace PVC storageClassName request from #2442 with a bounded driver, gateway config, Helm, and docs change.
Head SHA: 9a705b3f4b639620e08862d4bc4d60f14a80a8d0

Review findings:

  • No blocking findings remain. The independent code-only reviewer reported no actionable findings for this patch.

Docs: Fern reference docs and the Kubernetes driver documentation were updated for the new workspace_storage_class / server.workspaceStorageClass setting.

Tests: Unit coverage was added for config defaults/serde and PVC generation with set and empty storage classes. Because this changes Kubernetes driver behavior and Helm-rendered gateway config, test:e2e and test:e2e-kubernetes are required.

Next state: gator:watch-pipeline

@drew drew added gator:watch-pipeline Gator is monitoring PR CI/CD status test:e2e Requires end-to-end coverage test:e2e-kubernetes Requires Kubernetes end-to-end coverage labels Jul 28, 2026
@drew

drew commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 9a705b3

@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 9a705b3. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@github-actions

Copy link
Copy Markdown

Label test:e2e-kubernetes applied for 9a705b3. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA E2E after building the required gateway and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

@drew drew added gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed test:e2e-kubernetes Requires Kubernetes end-to-end coverage and removed gator:watch-pipeline Gator is monitoring PR CI/CD status gator:blocked Gator is blocked by process or repository gates test:e2e-kubernetes Requires Kubernetes end-to-end coverage gator:approval-needed Gator completed review; maintainer approval needed labels Jul 28, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e-kubernetes applied for 9a705b3. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA E2E after building the required gateway and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

@drew drew removed the gator:watch-pipeline Gator is monitoring PR CI/CD status label Jul 28, 2026
@drew drew added gator:approval-needed Gator completed review; maintainer approval needed gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:approval-needed Gator completed review; maintainer approval needed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Jul 28, 2026
@loveRhythm1990

loveRhythm1990 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for running the Kubernetes e2e. I've seen the kubernetes-ha-e2e failure and I'm looking
into it.

A few notes on where things stand:

  • The failures are all sandbox pods never reaching Ready — either stuck Pending
    (DependenciesNotReady: Pod exists with phase: Pending) or a running pod whose supervisor never
    relays back (supervisor session not connected / ssh 255). They surface only as CLI timeouts, since
    the HA e2e doesn't currently capture pod/PVC/controller diagnostics on failure.
  • The high-availability overlay runs the gateway with replicaCount: 2 behind a single Service, so
    I suspect a multi-replica session-routing issue (the supervisor relay landing on a different
    replica than the one holding the session) rather than anything storage-related.

I'm planning to reproduce this locally with the HA values (ci/values-high-availability.yaml + the
external Postgres fixture) to confirm the root cause. I'll report back with findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:approval-needed Gator completed review; maintainer approval needed test:e2e Requires end-to-end coverage test:e2e-kubernetes Requires Kubernetes end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(k8s): support configuring storageClassName for the default workspace PVC

3 participants