feat: Allow Fileownership change through FSGroup and VOLUME_MOUNT_GROUP #1841

mytreya-rh · 2025-06-09T17:49:45Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
(As of now pls consider it as a draft PR to discuss the solution further.)
Allows the secrets to be mounted with FSGroup as specified in the POD spec.
Thus, A pod with a non-root user should be able to read a secret, and that secret need not be world-readable.

Which issue(s) this PR fixes :
Fixes #858

Is this a chart or deployment yaml update?
There is a yaml update for secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml (generated through make manifests).
It is added in the manifest_staging/deploy
But if this PR merges after: #1622, the change in SecretProviderClassPodStatusStatus won't be required anymore and we can revert the changes related to reconciler.

Special notes for your reviewer:

Problem:

As of now, all secrets mounted by ss-csi driver are with root:root ownership, and this is unwieldy because it would require much wider permissions on the secret, or elevated privileges on the workload
The standard way to set file ownership of volume mounts is through FSGroup.
However, that does not work with SS-CSI Driver even if FSGroupPolicy is set to "File" because:
- https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_mounter.go#L454
  - Here, ‘readonly’ volumes are skipped from being chown’d by csi_mounter even if the CSIDriver CRD is created with FSGroupPolicy of “File”
  - This part executes in the kubelet while it tries to provision the volume for the POD.
- https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/main/pkg/secrets-store/nodeserver.go#L205
  - It is not possible to create a readwrite volume for ss-csi driver, and rightly so. Because, the secrets are only supposed to be read.

Solution:

outline
Do the ownership change from within the driver by advertising the VOLUME_MOUNT_GROUP capability.

Notes:

The changes also include secret rotation based on SecretProviderClassPodStatusStatus, but will be reverted if feat: Use RequiresRepublish for secret rotation #1622 merges earlier
In addition, pulled up some of common repetitive code from the unit and e2e tests to make them a bit more terse

tests added in e2e-provider:

(leaving in the test status and runtime just for reference)

ok 16 Non-root POD with no FSGroup - create in 871ms
ok 17 Non-root POD with no FSGroup - Should fail to read non world readable secret in 186ms
ok 18 Non-root POD with no FSGroup - unmount succeeds in 10143ms
ok 19 Non-root POD with FSGroup - create in 1439ms
ok 20 Non-root POD with FSGroup - should read non world readable secret in 202ms
ok 21 Non-root POD with FSGroup - rotated secret should also be readable in 37119ms
ok 22 Non-root POD with FSGroup - unmount succeeds in 10177ms

unit tests:

nodeserver_test
- TestNodePublishVolume_Errors/Invalid_FSGroup
- TestNodePublishVolume/volume_mount_with_valid_FSGroup
reconciler_test
- TestReconcileError/failed_to_parse_FSGroup
- TestReconcileNoError/reconcile_with_FSGroup

TODOs:

squashed commits
includes documentation
adds unit tests

linux-foundation-easycla · 2025-06-09T17:49:49Z

The committers listed above are authorized under a signed CLA.

✅ login: mytreya-rh / name: Mytreya Kasturi (50f04b9, cbac857, e8e905e, 598c51c)

k8s-ci-robot · 2025-06-09T17:49:54Z

Welcome @mytreya-rh!

It looks like this is your first PR to kubernetes-sigs/secrets-store-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/secrets-store-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-06-09T17:49:55Z

Hi @mytreya-rh. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dobsonj · 2025-06-27T13:04:21Z

/ok-to-test

mytreya-rh · 2025-06-30T05:56:05Z

/retest

aramase

The windows job failures are related to this PR.

E0630 16:56:06.282028   10108 atomic_writer.go:419] "unable to change file with owner" err="chown c:\\var\\lib\\kubelet\\pods\\ff425598-c3fa-480d-a6af-814831673629\\volumes\\kubernetes.io~csi\\secrets-store-inline\\mount\\..2025_06_30_16_56_06.1168464543\\secretalias: not supported by windows" logContext="secrets-store-csi-driver" fullPath="c:\\var\\lib\\kubelet\\pods\\ff425598-c3fa-480d-a6af-814831673629\\volumes\\kubernetes.io~csi\\secrets-store-inline\\mount\\..2025_06_30_16_56_06.1168464543\\secretalias" owner=-1

ref: https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_secrets-store-csi-driver/1841/pull-secrets-store-csi-driver-e2e-windows/1939725228552753152/artifacts/2025-06-30T170106/secrets-store.log

mytreya-rh · 2025-06-30T19:49:06Z

The windows job failures are related to this PR.
E0630 16:56:06.282028   10108 atomic_writer.go:419] "unable to change file with owner" err="chown c:\\var\\lib\\kubelet\\pods\\ff425598-c3fa-480d-a6af-814831673629\\volumes\\kubernetes.io~csi\\secrets-store-inline\\mount\\..2025_06_30_16_56_06.1168464543\\secretalias: not supported by windows" logContext="secrets-store-csi-driver" fullPath="c:\\var\\lib\\kubelet\\pods\\ff425598-c3fa-480d-a6af-814831673629\\volumes\\kubernetes.io~csi\\secrets-store-inline\\mount\\..2025_06_30_16_56_06.1168464543\\secretalias" owner=-1
ref: https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_secrets-store-csi-driver/1841/pull-secrets-store-csi-driver-e2e-windows/1939725228552753152/artifacts/2025-06-30T170106/secrets-store.log

Thanks @aramase !
Pushed a commit to skip Chown on Windows. Guess this is inline with FSGroup behavior as well on Windows nodes.

mytreya-rh · 2025-07-01T01:28:11Z

/retest

dobsonj

I have one comment on test/bats/e2e-provider.bats, but otherwise LGTM. It's a useful fix, implementation looks correct, good test coverage, and passing CI tests. Netlify is warning about a line unrelated from your changes.

dobsonj · 2025-07-11T17:59:29Z

test/bats/e2e-provider.bats

+  kubectl wait -n rotation --for=condition=Ready --timeout=60s pod ${curl_pod_name}
+  local pod_ip=$(kubectl get pod -n kube-system -l app=csi-secrets-store-e2e-provider -o jsonpath="{.items[0].status.podIP}")
+  run kubectl exec ${curl_pod_name} -n rotation -- curl http://${pod_ip}:8080/rotation?rotated=true
+  sleep 35 # 30 is poll interval, 5 second grace should be enough


I worry that 35 seconds may not be enough to prevent flakes. In @test "Test auto rotation of mount contents and K8s secrets" (line 472) it used to sleep 60 seconds, but now it only sleeps 35 seconds? Is it possible for a reconcile loop to be delayed for some reason that would cause this to take longer than 35?

I would probably not reduce this below 60, we had one similar case in vault.bats waiting on secret rotation where we had to increase it to 120 to improve the pass rate.

Thanks @dobsonj
Reverted the sleep back to 60s

…conds

mytreya-rh · 2025-07-21T04:51:05Z

/retest

dobsonj · 2025-07-21T19:50:34Z

/lgtm

/sig storage
/triage accepted
/priority important-soon

/assign @aramase
for approval and to decide which PR should merge first between #1841 and #1622

aramase

First pass.

I haven't looked at the test changes yet.

aramase · 2025-08-07T03:11:21Z

...secrets-store-csi-driver/crds/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml

Revert changes to charts directory. Chart changes should only be made in the manifest_staging directory.

aramase · 2025-08-07T03:12:01Z

deploy/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml

Revert changes to deploy directory. Chart changes should only be made in the manifest_staging directory.

aramase · 2025-08-07T04:29:40Z

pkg/constants/constants.go

@@ -0,0 +1,5 @@
+package constants


Need to add boilerplate to the top of the file.

aramase · 2025-08-07T04:32:12Z

pkg/rotation/reconciler.go

 	}
-	newObjectVersions, errorReason, err := secretsstore.MountContent(ctx, providerClient, string(paramsJSON), string(secretsJSON), spcps.Status.TargetPath, string(permissionJSON), oldObjectVersions)
+	gid := constants.NoGID
+	if spcps.Status.FSGroup != "" {


Suggested change

if spcps.Status.FSGroup != "" {

if len(spcps.Status.FSGroup) > 0 {

this is the code style we follow in Kubernetes

aramase · 2025-08-07T04:33:13Z

pkg/rotation/reconciler.go

+		gid, err = strconv.ParseInt(spcps.Status.FSGroup, 10, 64)
+		if err != nil {
+			errorReason = internalerrors.FailedToParseFSGroup
+			errStr := fmt.Sprintf("failed to rotate objects for pod %s/%s, err: %v, invalid FSGroup:%s", spcps.Namespace, spcps.Status.PodName, err, spcps.Status.FSGroup)


Suggested change

errStr := fmt.Sprintf("failed to rotate objects for pod %s/%s, err: %v, invalid FSGroup:%s", spcps.Namespace, spcps.Status.PodName, err, spcps.Status.FSGroup)

errStr := fmt.Sprintf("failed to rotate objects for pod %s/%s, invalid FSGroup:%s, err: %w", spcps.Namespace, spcps.Status.PodName, spcps.Status.FSGroup, err)

Done, thanks for the catch. This makes more sense to read as a log.

however, %w (wrap) format seems to be specific only to fmt.Errorf. Thus, restructured the code to use %v in the Sprintf and %w in the Errorf

aramase · 2025-08-07T04:49:42Z

pkg/secrets-store/nodeserver.go

+		gid, err = strconv.ParseInt(fsGroupStr, 10, 64)
+		klog.V(5).Info("converted gid: %v\n", gid)
+		if err != nil {
+			klog.ErrorS(err, "failed to mount secrets store object content", "pod", klog.ObjectRef{Namespace: podNamespace, Name: podName}, "invalid FSGroup: ", fsGroupStr)


the key in structured log should be a single value

Suggested change

klog.ErrorS(err, "failed to mount secrets store object content", "pod", klog.ObjectRef{Namespace: podNamespace, Name: podName}, "invalid FSGroup: ", fsGroupStr)

klog.ErrorS(err, "failed to mount secrets store object content", "pod", klog.ObjectRef{Namespace: podNamespace, Name: podName}, "fSGroup: ", fsGroupStr)

aramase · 2025-08-07T04:50:55Z

pkg/secrets-store/nodeserver.go

+			return nil, status.Error(codes.InvalidArgument, "Error parsing FSGroup")
+		}
+	} else {
+		klog.V(5).Info("mount group not set")


We need the pod context here as well, otherwise it’s impossible to tell which pod the log is for

aramase · 2025-08-07T04:54:52Z

pkg/secrets-store/utils.go

+	if gid != constants.NoGID {
+		fsGroup = strconv.FormatInt(gid, 10)
+	}
+	klog.V(5).Infof("gid string: %v for pod: %v/%v", fsGroup, namespace, podname)


Need to use structured logging klog.InfoS

aramase · 2025-08-07T04:55:15Z

pkg/secrets-store/utils.go

 		},
 	})
-
+	klog.V(5).InfoS("creating", "spcPodStatus", spcPodStatus)


Not required

aramase · 2025-08-07T04:56:44Z

pkg/util/fileutil/atomic_writer.go

 		}

-		if fileProjection.FsUser == nil {
+		if fileProjection.FsGroup == nil || runtimeutil.IsRuntimeWindows() {


I need to read through the kubernetes code to understand how this works for windows. Do you know if it's supported or they skip for windows?

Yes, as per documentation FSGroup is not supported for Windows.

And as per code as well, the Kubernetes FSGroup implementation of writable volumes for non linux is a no-op:
csi_mounter.go --> volume_unsupported.go

k8s-ci-robot · 2025-08-07T04:58:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mytreya-rh
Once this PR has been reviewed and has the lgtm label, please ask for approval from aramase. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

1. Undo changes to the charts and deploy directories 2. Add boilerplate to pkg/constants/constants.go 3. Changed logging to entire VolumeContext to just the FSGroup 4. Updated the way empty string is checked 5. changed Infof to InfoS 6. Removed un-necessary/redundant logs

k8s-ci-robot · 2025-08-13T12:49:43Z

New changes are detected. LGTM label has been removed.

mytreya-rh · 2025-08-19T10:07:06Z

/retest

k8s-ci-robot · 2025-08-19T10:10:09Z

@mytreya-rh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-secrets-store-csi-driver-image-scan	`e8e905e`	link	false	`/test pull-secrets-store-csi-driver-image-scan`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Jun 9, 2025

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 9, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 9, 2025

k8s-ci-robot requested review from aramase and ritazh June 9, 2025 17:49

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 9, 2025

enj moved this to Subprojects - Needs Triage in SIG Auth Jun 10, 2025

enj added this to SIG Auth Jun 10, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 27, 2025

mytreya-rh force-pushed the allow_file_ownership branch from 898943d to b797f9d Compare June 29, 2025 16:08

enj moved this from Subprojects - Needs Triage to In Review in SIG Auth Jun 30, 2025

aramase requested changes Jun 30, 2025

View reviewed changes

github-project-automation bot moved this from In Review to Changes Requested in SIG Auth Jun 30, 2025

squashed commit for Allow Fileownership (skip Chown on Windows)

cbac857

mytreya-rh force-pushed the allow_file_ownership branch from b797f9d to cbac857 Compare June 30, 2025 19:43

dobsonj reviewed Jul 11, 2025

View reviewed changes

Reverted the wait after setting secret rotation from 35 back to 60 se…

50f04b9

…conds

k8s-ci-robot assigned aramase Jul 21, 2025

k8s-ci-robot assigned dobsonj Jul 21, 2025

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 21, 2025

aramase requested changes Aug 7, 2025

View reviewed changes

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 13, 2025

Remove usage of %w formatter from non ErrorF call

e8e905e

mytreya-rh force-pushed the allow_file_ownership branch from 6ed5189 to e8e905e Compare August 13, 2025 16:48

	if spcps.Status.FSGroup != "" {
	if len(spcps.Status.FSGroup) > 0 {

	errStr := fmt.Sprintf("failed to rotate objects for pod %s/%s, err: %v, invalid FSGroup:%s", spcps.Namespace, spcps.Status.PodName, err, spcps.Status.FSGroup)
	errStr := fmt.Sprintf("failed to rotate objects for pod %s/%s, invalid FSGroup:%s, err: %w", spcps.Namespace, spcps.Status.PodName, spcps.Status.FSGroup, err)

	klog.ErrorS(err, "failed to mount secrets store object content", "pod", klog.ObjectRef{Namespace: podNamespace, Name: podName}, "invalid FSGroup: ", fsGroupStr)
	klog.ErrorS(err, "failed to mount secrets store object content", "pod", klog.ObjectRef{Namespace: podNamespace, Name: podName}, "fSGroup: ", fsGroupStr)

feat: Allow Fileownership change through FSGroup and VOLUME_MOUNT_GROUP #1841

Are you sure you want to change the base?

feat: Allow Fileownership change through FSGroup and VOLUME_MOUNT_GROUP #1841

Uh oh!

Conversation

mytreya-rh commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem:

Solution:

Notes:

tests added in e2e-provider:

unit tests:

Uh oh!

linux-foundation-easycla bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jun 9, 2025

Uh oh!

k8s-ci-robot commented Jun 9, 2025

Uh oh!

dobsonj commented Jun 27, 2025

Uh oh!

mytreya-rh commented Jun 30, 2025

Uh oh!

aramase left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mytreya-rh commented Jun 30, 2025

Uh oh!

mytreya-rh commented Jul 1, 2025

Uh oh!

dobsonj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mytreya-rh commented Jul 21, 2025

Uh oh!

dobsonj commented Jul 21, 2025

Uh oh!

aramase left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mytreya-rh commented Jun 9, 2025 •

edited

Loading

linux-foundation-easycla bot commented Jun 9, 2025 •

edited

Loading

aramase left a comment •

edited

Loading

mytreya-rh Aug 13, 2025 •

edited

Loading