[KFP] Connect KFP to use MLRun CE SeaweedFS Solution#261
[KFP] Connect KFP to use MLRun CE SeaweedFS Solution#261shay79il wants to merge 1 commit intomlrun:developmentfrom
Conversation
shay79il
force-pushed
the
CEML-622-kfp
branch
3 times, most recently
from
0f3d1c2 to
541cd3d
Compare
There was a problem hiding this comment.
Pull request overview
This PR configures Kubeflow Pipelines (KFP) to use the SeaweedFS S3-compatible storage gateway for artifact storage, with KFP 2.x compatibility updates and security hardening for pipeline components.
Changes:
- Configured KFP launcher with SeaweedFS artifact storage using the
minio://URI scheme (KFP's standard S3-compatible protocol) - Added KFP 2.x environment variables for artifact retention and proxy configuration with optional flags
- Hardened metadata-envoy deployment with security context restrictions
- Externalized LOG_LEVEL configuration to ConfigMap for better configurability
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| ct.yaml | New configuration file to disable version increment checks for chart-testing tool |
| charts/mlrun-ce/Chart.yaml | Bumped chart version from 0.11.0-rc9 to 0.11.0-rc10 |
| Makefile | Updated helm-lint target to use ct.yaml configuration |
| charts/mlrun-ce/templates/pipelines/configmaps/pipeline-install-config.yaml | Added LOG_LEVEL configuration with INFO as default |
| charts/mlrun-ce/templates/pipelines/configmaps/kfp-launcher.yaml | Configured defaultPipelineRoot with minio:// scheme pointing to SeaweedFS S3 gateway, includes comprehensive documentation |
| charts/mlrun-ce/templates/pipelines/deployments/ml-pipeline.yaml | Moved LOG_LEVEL to ConfigMap reference, added optional KFP 2.x environment variables (ARTIFACT_RETENTION_DAYS, ARTIFACTS_PROXY_ENABLED), and improved code organization with section comments |
| charts/mlrun-ce/templates/pipelines/deployments/metadata-envoy-deployment.yaml | Added container security context with privilege escalation prevention and capability restrictions |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| name: pipeline-install-config | ||
| key: bucketName | ||
|
|
||
| # KFP 2.15.x - Artifact retention and proxy configuration |
There was a problem hiding this comment.
The comment references "KFP 2.15.x" but the current version in use is 2.14.3 (as seen in values.yaml). The comment should either reference "KFP 2.14.x+" or be updated to reflect the actual version being used. This could cause confusion about which KFP version these features are actually available in.
| # KFP 2.15.x - Artifact retention and proxy configuration | |
| # KFP 2.14.x+ - Artifact retention and proxy configuration |
There was a problem hiding this comment.
If we want to use ARTIFACT_RETENTION_DAYS we should upgrade the KFP server version to 2.15
| allowPrivilegeEscalation: false | ||
| runAsNonRoot: true | ||
| runAsUser: 1000 | ||
| runAsGroup: 0 |
There was a problem hiding this comment.
The security context sets runAsGroup: 0 (root group), which weakens the security posture. While the container runs as a non-root user (runAsUser: 1000), running with group ID 0 can still grant elevated permissions. Consider using a non-root group ID (e.g., runAsGroup: 1000) unless the envoy proxy specifically requires root group membership for its operation.
| runAsGroup: 0 | |
| runAsGroup: 1000 |
| cacheImage: {{ .Values.pipelines.images.cacheImage.repository }}:{{ .Values.pipelines.images.cacheImage.tag }} | ||
| cacheNodeRestrictions: "false" | ||
| cronScheduleTimezone: UTC | ||
| LOG_LEVEL: INFO |
There was a problem hiding this comment.
maybe worth to add an option to the user to change this log level
| key: bucketName | ||
|
|
||
| # KFP 2.15.x - Artifact retention and proxy configuration | ||
| - name: ARTIFACT_RETENTION_DAYS |
There was a problem hiding this comment.
What are the values for ARTIFACT_RETENTION_DAYS. and ARTIFACTS_PROXY_ENABLED ?
There was a problem hiding this comment.
they are optional not mandatory
If the client does not provide them all good
There was a problem hiding this comment.
if may maybe worth adding an option for the user to set values for those ENV in the config map by using the values.yaml file.
not mandatory, choose if you want to add it
shay79il
force-pushed
the
CEML-622-kfp
branch
2 times, most recently
from
9031be9 to
d35bb5d
Compare
Configure Kubeflow Pipelines to use SeaweedFS S3 gateway for artifact storage, with KFP 2.x compatibility updates [CEML-622](https://iguazio.atlassian.net/browse/CEML-622)
shay79il
force-pushed
the
CEML-622-kfp
branch
from
d35bb5d to
f6c862e
Compare
Configure Kubeflow Pipelines to use SeaweedFS S3 gateway for artifact storage,
with KFP 2.x compatibility updates and security hardening.
CEML-622