Add the cos2cos sample for CodeEngine by subhasree91 · Pull Request #196 · IBM/CodeEngine

subhasree91 · 2025-06-11T19:29:19Z

This application processes objects (files) in IBM Cloud Object Storage (COS) between a primary and secondary bucket. The sample can be tried out using CodeEngine.

Copied from https://github.ibm.com/ibmcloud-codeengine-internship/code-engine-cos2cos.

reggeenr

Hi, thanks for putting so much work into this great sample. All in all it looks very very good. Apart from the comments that I already published, I would like to ask you whether you could some focus on making it really easy for users to get started with this example. Think about which property values can be inferred (if not set). Make assumptions. Be opinionated. Keep it easy for users to get it going... While more complex configurations should be applicable, we shouldn't confront all users with it right-away.

Keep up the great work and thanks for this contribution!!!

reggeenr · 2025-06-30T07:36:34Z

code-engine-cos2cos/Dockerfile

@@ -0,0 +1,25 @@
+FROM golang:1.23-alpine as builder


Hi, can you please adjust the Dockerfile and make sure that a) it does not pull any artifact from dockerhub (if no hostname is specified docker.io is used) and by the multi-staged build uses a distroless image in the end?

You can take the following one as an example
https://github.com/IBM/CodeEngine/blob/main/trusted-profiles/go/Dockerfile

reggeenr · 2025-06-30T07:37:00Z

code-engine-cos2cos/README.md

+### Setup & Configuration
+1. **Clone the Repository**:
+   ```bash
+   git clone https://github.ibm.com/ibmcloud-codeengine-internship/code-engine-cos2cos


This URL must be adjusted

Changed as follows

bash git clone https://github.com/IBM/CodeEngine.git cd code-engine-cos2cos

Now, that I read this, I noticed that we should call the folder cos2cos. Please wait with that renaming until we processed the entire review, otherwise this will be a mess.

However, in this PR the readme should already be adjusted

git clone https://github.com/IBM/CodeEngine.git cd cos2cos

We agree with your point. For now we have updated the readme with above changes.

code-engine-cos2cos/README.md

reggeenr · 2025-06-30T07:40:20Z

code-engine-cos2cos/README.md

+#### 2. Build Using Source Code (Local Source)
+
+To build the image from local source code using IBM Cloud Code Engine:
+
+```bash
+ibmcloud ce build create --name ${BUILD_NAME} --build-type local --image ${REGISTRY}/${NAMESPACE}/${IMAGE_NAME}
+ibmcloud ce buildrun submit --build ${BUILD_NAME} --name ${BUILD_NAME}-build-run
+```
+
+#### 3. Build Using Git-based Source
+
+To build the image using a Git repository:
+
+1. Create a deploy key or user-access key in your GitHub repository.
+2. Add the private key by creating an SSH secret in IBM Cloud Code Engine.
+3. Create a build using the Git repository:
+
+```bash
+ibmcloud ce build create \
+    --name ${BUILD_NAME} \
+    --image ${REGISTRY}/${NAMESPACE}/${IMAGE_NAME} \
+    --source ${GIT_SSH_URL} \
+    --context-dir / \
+    --strategy dockerfile \
+    --git-repo-secret ${GIT_SSH_SECRET}
+```
+
+4. Submit the build:
+
+```bash
+ibmcloud ce buildrun submit --build ${BUILD_NAME}
+```


I suggest to make use of our integrated build and building the container image along with creating the job template:

ibmcloud ce job create --name ... \ --src "." \ --wait

Suggested change implemented in config.sh file from line number 185.

code-engine-cos2cos/bucketOperations/bucketOps.go

code-engine-cos2cos/metadata/local-timestamp.go

code-engine-cos2cos/process/process.go

reggeenr · 2025-06-30T07:54:46Z

code-engine-cos2cos/process/process_test.go

+	// ginkgo.Context("should handle errors", func() {
+
+	// })


If the code isn't needed, please remove it

code-engine-cos2cos/run.sh

reggeenr

Thanks for the fast turnarounds. Really appreciate the effort.

I will take a look at mechanics of this asset more closely while trying it out.

reggeenr · 2025-07-01T13:54:27Z

code-engine-cos2cos/Dockerfile

+# Stage 2: Final stage (minimal image)
+FROM gcr.io/distroless/static-debian12
+
+# RUN apk --no-cache add ca-certificates


If not needed, would you mind to remove it?

reggeenr · 2025-07-01T13:59:49Z

code-engine-cos2cos/main.go

+	os.Setenv("IBM_COS_CRTokenFilePath_PRIMARY", "/var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token")
+	os.Setenv("IBM_COS_CRTokenFilePath_SECONDARY", "/var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token")


The Trusted Profile token path will always be the same, I don't see a reason to specify it twice and make it configurable.

Now we are using a single trusted profile and we have also changed the CRTokenFilePath to a static variable within the code and accordingly made the changes in main.go.

reggeenr · 2025-07-01T14:01:51Z

code-engine-cos2cos/config.sh

+  --from-literal IBM_COS_CRTokenFilePath_PRIMARY=${IBM_COS_CRTokenFilePath_PRIMARY} \
+  --from-literal IBM_COS_CRTokenFilePath_SECONDARY=${IBM_COS_CRTokenFilePath_SECONDARY} \


I don't see the need to have the token path configurable, please just use a static variable

Changed as requested to a single variable.

reggeenr · 2025-07-01T14:03:31Z

code-engine-cos2cos/config.sh

+  --from-literal IBM_COS_TRUSTED_PROFILE_ID_PRIMARY=${COS_TRUSTED_PROFILE_ID_PRIMARY} \
+  --from-literal IBM_COS_TRUSTED_PROFILE_ID_SECONDARY=${COS_TRUSTED_PROFILE_ID_SECONDARY}


For complexity reasons, please use a single trusted profile that grants permissions to the primary and secondary COS bucket. In the blog post that I wrote about trusted profiles, I just split it up into two trusted profiles to make it clear that is possible in general

Changed as requested to a single variable.
Note:- There has been a slight modification in main.go specifically while creating the COS client. Instead of fetching CRTokenFIlePath for primary and secondary from env file, a static value is passed which is mentioned in main.go

reggeenr · 2025-07-01T14:14:57Z

code-engine-cos2cos/config.sh

+  --env-from-secret ${AUTH_SECRET} \
+  --argument true 2>/dev/null \
+  --wait
+  # --registry-secret ${CONTAINER_REGISTRY_SECRET} \


shall be removed as not longer needed

reggeenr · 2025-07-01T14:15:02Z

code-engine-cos2cos/config.sh

+  # echo "Job '${JOB_NAME}' already exists. Exiting"
+  # exit 1


shall be removed as not longer needed

Instead of this line, we have added an if statement asking the user if they want to override the existing job or create a new one.

reggeenr · 2025-07-01T14:15:06Z

code-engine-cos2cos/config.sh

+  --env-from-secret ${BASE_SECRET} \
+  --env-from-secret ${AUTH_SECRET} \
+  --argument true 2>/dev/null
+  # --registry-secret ${CONTAINER_REGISTRY_SECRET} \


shall be removed as not longer needed

reggeenr · 2025-07-01T14:16:13Z

code-engine-cos2cos/env_sample

+
+# Primary Bucket Trusted Profile Credentials
+IBM_COS_TRUSTED_PROFILE_ID_PRIMARY= <>
+IBM_COS_CRTokenFilePath_PRIMARY=/var/run/secrets/tokens/service-account-token


I suggest to remove that one

We have removed the env_sample file since it is no longer in use.

reggeenr · 2025-07-01T14:18:18Z

code-engine-cos2cos/config.sh

+echo "Step 8.4: Compute Resource Token"
+curl \
+  --request PATCH "https://api.${PROJECT_REGION}.codeengine.cloud.ibm.com/v2/projects/$(ibmcloud ce project current --output json | jq -r .guid)/jobs/${JOB_NAME}" \
+  --header 'Accept: application/json' \
+  --header "Authorization: $(ibmcloud iam oauth-tokens --output json | jq -r '.iam_token')" \
+  --header 'Content-Type: application/merge-patch+json' \
+  --header 'If-Match: *' \
+  --data-raw "{
+    \"run_compute_resource_token_enabled\": true
+  }" 2>/dev/null


If I am not mistaken, this code can be replaced with the corresponding option in the job create/update operation, as we added support for trusted profiles for jobs

Thank you for pointing this out. We replaced it with option in job create/update --trusted-profile-enabled.

Signed-off-by: Shailza Thakur <Shailza.Thakur@ibm.com> Signed-off-by: Hamza Bharmal <Hamza@ibm.com> Signed-off-by: Subhasree Das <Subhasree.Das@ibm.com> Signed-off-by: Subhasree-Das <Subhasree.Das@ibm.com> Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

…perties Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

…uth-secret Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Signed-off-by: Hamza Bharmal <h.bharmal@ibm.com>

subhasree91 force-pushed the cos2cos-internship branch 3 times, most recently from 35704c2 to 1ec8829 Compare June 11, 2025 19:51

subhasree91 requested a review from reggeenr June 11, 2025 19:57

subhasree91 assigned subhasree91, H-Bharmal and ShailzaThakur7 Jun 11, 2025

reggeenr requested changes Jun 30, 2025

View reviewed changes

reggeenr requested changes Jul 1, 2025

View reviewed changes

subha-91 and others added 18 commits September 5, 2025 13:31

Update readme

31c8d58

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Update data.sh with fewer variables

933b00f

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Remove container registry secret creation

02dbc76

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Delete build.sh

ad798a5

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Remove dead-code

4543220

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Update readme

b6e795f

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Remove option for service-credentials. Set 3 variables to default pro…

1238621

…perties Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Update Dockerfile to use distroless image

413c873

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Remove comments

0dff284

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Remove comments

3c98de5

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

update script to use single trusted profile, Update base-secret and a…

cc31150

…uth-secret Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Remove extra variables to default values

0eb88fe

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Remove dead-code

cb93ed3

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Delete sample env

ecef454

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Update timestamp.go to use default filename

04058a8

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Update to use default values, Support single-trusted-profile

4d86018

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

Update readme

e88ca7b

Signed-off-by: Hamza Bharmal <hamza@mac.in.ibm.com>

H-Bharmal force-pushed the cos2cos-internship branch from 30b9e00 to e88ca7b Compare September 5, 2025 08:07

Update pkg versions

6da59ad

Signed-off-by: Hamza Bharmal <h.bharmal@ibm.com>

		os.Setenv("IBM_COS_CRTokenFilePath_PRIMARY", "/var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token")
		os.Setenv("IBM_COS_CRTokenFilePath_SECONDARY", "/var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token")

		--from-literal IBM_COS_CRTokenFilePath_PRIMARY=${IBM_COS_CRTokenFilePath_PRIMARY} \
		--from-literal IBM_COS_CRTokenFilePath_SECONDARY=${IBM_COS_CRTokenFilePath_SECONDARY} \

		--from-literal IBM_COS_TRUSTED_PROFILE_ID_PRIMARY=${COS_TRUSTED_PROFILE_ID_PRIMARY} \
		--from-literal IBM_COS_TRUSTED_PROFILE_ID_SECONDARY=${COS_TRUSTED_PROFILE_ID_SECONDARY}

Conversation

subhasree91 commented Jun 11, 2025

Uh oh!

reggeenr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reggeenr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reggeenr Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

reggeenr Jul 1, 2025 •

edited

Loading