Examples separate projects by Jolanrensen · Pull Request #1677 · Kotlin/dataframe

Jolanrensen · 2026-02-06T13:40:22Z

Fixes #1614
Step 3 of #1661

Introduces example structure like:
- /examples/projects/1,2,3: release versions of examples
- /examples/projects/dev/1,2,3: dev versions of examples
  Dev versions of the examples are built using the sources of the project, while release examples use the latest release versions.
  Since any example is a completely separate project, you can attach them to the IDE separately only if you need it :).
  And most importantly, each example could be extracted to a ZIP and runnable by a user without any additional setup (TODO test this)
Currently I only moved the android, gradle, and maven example projects into the new folder.
Running :promoteDevExamples copies all /dev examples upwards, making them the new "release" versions. :syncExampleFolders is run afterwards.
Running :syncAllExampleFolders (run automatically every :assemble) overwrites all relevant versions and properties in the examples from our main libs.versions.toml, gradle-wrapper.properties, .editorconfig, etc. This also works for any maven examples (see build-logic/src/.../buildExampleProjects for the specific implementation).
GH actions was updated to run this command automatically, keeping our examples up-to-date :)
Running the Test task :runBuildAllExampleFolders (run automatically every :test in debug mode, so on TC) builds all example projects using TestBuildingExampleProjects. Each example project folder gets its own auto-generated Junit test so task dependencies can be set properly and TeamCity can run them in parallel.
- Thanks to the :syncAllExampleFolders task, all dev Gradle Example projects get includeBuild("../../../..") in settings.gradle.kts, which substitutes any DataFrame dependeny with our actual sources. This also makes working inside them from the IDE as easy as if it were a subproject :).
- The Android examples are ignored if the Gradle property android.sdk.dir is unset. I modified TeamCity to provide -Pandroid.sdk.dir=%env.ANDROID_SDK_HOME% and it seems to work :) This gradle property is passed down to any example containing "android" (in any form) in its name. (This logic can change if needed)
- The Maven dev project requires a published version of DataFrame. We cannot substitute a Gradle build into Maven in the same way (AFAIK). So for that we use :publishLocal to publish the artifacts to /build/maven before running the tests.

…uild that can build the projects in /example/projects

…jects(/dev) to automatically be synced and built

…oved kotlin-dataframe-plugin-maven-example to examples/projects and examples/projects/dev

…h to root

… enum names

…ilt when specifying android.sdk.dir

…en a security risk

…example test

…htly to allow more customization which tests should run from gradle

…can run them in parallel. reenabling buildExampleFolders in debug mode.

…ze for android examples back to 2G

Jolanrensen · 2026-03-02T16:51:40Z

I might squash some commits later, as there's been a lot of trial-and-error between me and TeamCity, but all seems to build fine now :)
EDIT: Seems like this is too late... Oops. Should've done that before I merged the master into here...

Carefully read the documentation I wrote. It should explain most of the steps in detail.

koperagen · 2026-03-04T11:52:33Z

examples/projects/android-example/app/build.gradle.kts

    // Apache Arrow is not supported well on Android.
+    //
+    // Either use:
+    implementation(libs.dataframe.core)


I still believe this notation makes lives of users harder. Impossible to copy paste dependency or even remember it

Seems that it's needed for easier sync

I think it's okay, since we will provide them an all-in-one project with libs.versions.toml. This is also the recommended structure by Gradle, so I guess we would just help them this way :)
And indeed, it makes sync way easier!

koperagen · 2026-03-04T12:17:28Z

Great! Could i ask you to clone fresh clean repo on this branch and import into IDE? I'll do the same
git clone -b examples-separate-projects --single-branch https://github.com/Kotlin/dataframe.git
Let's verify build is ok

minor: promoteExamples already generates some changes:

koperagen · 2026-03-04T12:21:20Z

Some invisible changes in mvnw after running runBuildAllExampleFolders:

I assume Android examples are disabled because i don't have local.properties android sdk?

koperagen · 2026-03-04T12:27:22Z

build-logic/README.md

+        - `sync<NameOfFolder(Dev)>`: Syncs and overwrites build settings and versions.
+          Copies and overwrites versions from the main `libs.versions.toml`, `gradle.properties`, etc.
+        - `syncAllExampleFolders`: Syncs all example projects. Automatically called on `assemble`.
+        - `promoteExamples`: Promotes the `/examples/projects/dev` example projects to `/examples/project`.


promoteDevExamples?

sure, sounds like a better name :)

koperagen · 2026-03-04T12:28:48Z

build-logic/src/main/kotlin/dfbuild/buildExampleProjects/gradleTasks.kt

+            settingsGradleKts.writeText(newSettingsGradleKtsContent)
+
+            // overwrite .editorconfig
+            folder.resolve(".editorconfig").writeText(sourceEditorConfig.readText())


Should we bundle editorconfig with our rules to examples projects?

We don't necessarily have to, but we did tweak the settings to work well with DataFrame. Plus, if we bump KtLint, the version will be synced to the examples as well. If we then need to disable one of the new rules, we will need to do so in each example as well... So yeah

Doesn't it work for us even without these files, re-using root editorconfig for entire dataframe project?
People like freedom to choose codestyle in their projects, i don't see why we should enforce anything beyond what IntelliJ already does

zaleslaw

Please answer my comments, also plan the task for the documentation update (and put it in the line of our site updates which should be merged closer to the release)
https://kotlin.github.io/dataframe/guides-and-examples.html#examples

Check references, paths, its description and possible notes for a user

zaleslaw · 2026-03-04T12:27:49Z

gradle/libs.versions.toml


 gradlePlugin-gradle-foojayToolchains = { module = "org.gradle.toolchains:foojay-resolver", version.ref = "gradlePlugin-gradle-foojayToolchains" }
 typesafe-conventions = { module = "dev.panuszewski:typesafe-conventions-gradle-plugin", version.ref = "typesafe-conventions" }
+maven-invoker = { module = "org.apache.maven.shared:maven-invoker", version.ref = "maven-invoker" }


What does this invoker?

well... maven XD

It's added as dependency to the sourceSet running the build tests. Maven invoker can programmatically run a maven build, so we can even test our non-gradle examples :)

zaleslaw · 2026-03-04T12:29:25Z

examples/README.md

+The example projects in [examples/projects](projects) always target the latest stable version of the library.
+Projects compatible with the latest dev/master version are located in the [examples/projects/dev](projects/dev) folder.
+
+All examples are tested by the main project in debug mode when `gradlew test` is run


Also could be a useful information for our site https://kotlin.github.io/dataframe/guides-and-examples.html#examples

I'm not sure. We could, but then it would be mostly boasting about how well we are looking after our examples. I think most users actually already expect we test our examples (I mean, we expected it ourselves too). What is more interesting for users is that they are kept up-to-date :) meaning they can always download a working, up-to-date example from the latest green-built master :)

zaleslaw · 2026-03-04T12:30:21Z

examples/projects/kotlin-dataframe-plugin-maven-example/pom.xml

    <repositories>
+        <repository>
+            <id>mavenLocal</id>
+            <url>file://${user.home}/.m2/repository</url>


Will it work on windows? hmmm

could you try?

zaleslaw · 2026-03-04T12:31:34Z

examples/projects/kotlin-dataframe-plugin-gradle-example/.editorconfig

@@ -0,0 +1,41 @@
+root = true


do we need this file in each project?

yes, definitely. Each project should be individually downloadable as Zip. If you don't supply root=true, KtLint will scan the entire system upwards to find all .editorconfig files.

zaleslaw · 2026-03-04T12:32:58Z

examples/projects/dev/android-example/gradle.properties

 # Specifies the JVM arguments used for the daemon process.
 # The setting is particularly useful for tweaking memory settings.
-org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8
+org.gradle.jvmargs=-Xmx2g -XX:MaxMetaspaceSize=512m -Dfile.encoding=UTF-8


Does it require so much Metasspace?

Not entirely sure, but what I am sure is that we need to limit this somehow, as to not exceed TeamCity's memory. This makes it so the test won't use more than 2.5GB

zaleslaw · 2026-03-04T12:34:56Z

.../src/testBuildingExamples/kotlin/dfbuild/buildExampleProjects/TestBuildingExampleProjects.kt

+ * `dfbuild.buildExampleProjects` convention plugin.
+ */
+@Suppress("unused")
+abstract class TestBuildingExampleProjects {


I have a feeling that we need to add this to our knowledge library: for example you should demo it for particular version and we will record it

zaleslaw · 2026-03-04T12:36:14Z

build-logic/src/main/kotlin/dfbuild.buildExampleProjects.gradle.kts

+}
+
+tasks.named<JavaCompile>("compile${testBuildingExamples.name.capitalized()}Java") {
+    sourceCompatibility = JavaVersion.VERSION_21.toString()


Should we somehow parametrize it to make easier migration on the next Java?

Might be a good idea indeed :) this is the jvm version of :build-logic itself, but we can access our version catalog from there as well, so we might just be able to put it there :)

zaleslaw · 2026-03-04T12:37:59Z

build-logic/src/main/kotlin/dfbuild/buildExampleProjects/generateTestCase.kt

+typealias Code = String
+
+@Language("kt")
+fun generateTestCase(testClassName: String, folder: File, isDev: Boolean, tags: List<String>): Code =


I have a feeling that it should be a description of the processes, its goal, why do we need this generation

Jolanrensen · 2026-03-04T15:53:52Z

I assume Android examples are disabled because i don't have local.properties android sdk?

@koperagen Exactly :) see gradle.properties or /build-logic/README.md for how to set this up. I thought this was the least invasive way of skipping the tests on platforms that don't (or cannot) have an Android SDK.

created dfbuild.buildExampleProjects convention plugin for the main b…

08caf32

…uild that can build the projects in /example/projects

Jolanrensen mentioned this pull request Feb 6, 2026

Streamline our infrastructure with Gradle Convention plugins #1661

Open

9 tasks

Jolanrensen added 2 commits February 6, 2026 14:51

moved examples/kotlin-dataframe-plugin-gradle-example to examples/pro…

7c54827

…jects(/dev) to automatically be synced and built

added syncExampleFolders task to generated-sources GH actions

a20f5a7

Jolanrensen force-pushed the examples-separate-projects branch from 2ec6075 to a20f5a7 Compare February 6, 2026 13:52

Jolanrensen added 8 commits February 9, 2026 14:40

dfbuild.buildExampleProjects can now build/sync maven projects too. M…

004a6d6

…oved kotlin-dataframe-plugin-maven-example to examples/projects and examples/projects/dev

updated release checklist

1ea2779

made includeBuild() in examples/projects/dev search for the right pat…

e8252de

…h to root

tiny fix for keywords-generator, as Android/dex cannot compile "?" in…

9d4d451

… enum names

bumping gradle to 9.3.1

f8cc4d3

moved android examples to /examples/projects and /dev. They can be bu…

6767d27

…ilt when specifying android.sdk.dir

moved examples execution from gradle to junit tests

b270419

apiDump and fixed compilation regarding junit

34a5cee

Jolanrensen force-pushed the examples-separate-projects branch 2 times, most recently from 253d408 to bc241e6 Compare February 16, 2026 19:08

adding maven wrapper to examples and simplifying pom.xml

96714c6

Jolanrensen force-pushed the examples-separate-projects branch from bc241e6 to 96714c6 Compare February 16, 2026 19:20

Jolanrensen and others added 2 commits February 17, 2026 01:05

Update TestBuildingExampleProjects to disable maven home argument

286c8c4

limiting the properties set to system environment. This could have be…

8c1ec23

…en a security risk

Jolanrensen force-pushed the examples-separate-projects branch from 461e286 to 31256ad Compare February 17, 2026 14:26

using publishLocal task to publish to /build/maven for our maven dev …

bd9bd74

…example test

Jolanrensen force-pushed the examples-separate-projects branch from 31256ad to bd9bd74 Compare February 17, 2026 14:28

Jolanrensen added 8 commits February 17, 2026 15:56

reordered dfbuild.buildExampleProjects

72d0096

added more tags to TestBuildingExampleProjects and refactored it slig…

eb3e893

…htly to allow more customization which tests should run from gradle

added gradle jvmargs to gradle example project

37aba74

disabling idea-examples to avoid ksp crashing

f4c73c8

decreasing jvmargs from 8G to 6G

b83d0a6

made android a separate test target

6f17f25

limiting max heap size across the project for tests

3e5af24

disabling buildExampleFolders in debug mode

1576f0d

Jolanrensen added 3 commits February 26, 2026 14:11

Generate a separate junit test class for each example project, so TC …

8937351

…can run them in parallel. reenabling buildExampleFolders in debug mode.

improving generateTestCase to satisfy linter

c66c820

refactor of dfbuild.buildExampleProjects tasks

296d0f2

Jolanrensen force-pushed the examples-separate-projects branch from e9db608 to 296d0f2 Compare February 26, 2026 18:50

refactor of dfbuild.buildExampleProjects tasks and increasing heap si…

d97c078

…ze for android examples back to 2G

Jolanrensen force-pushed the examples-separate-projects branch 3 times, most recently from c6d4648 to 6bfabfd Compare February 26, 2026 20:25

more heap size for android examples

b8b83db

Jolanrensen force-pushed the examples-separate-projects branch from 6bfabfd to b8b83db Compare February 27, 2026 10:50

Jolanrensen added 3 commits February 27, 2026 13:42

re-enable old example projects

281dea8

korro still runs OOM on GH actions, increasing Xmx to 4G

a0fbf08

korro still runs OOM on GH actions, increasing Xmx to 5G

42fe842

Jolanrensen force-pushed the examples-separate-projects branch from 0405eb2 to 42fe842 Compare February 27, 2026 14:50

Merge branch 'master' into examples-separate-projects

eb6877d

Jolanrensen marked this pull request as ready for review February 27, 2026 19:06

updating docs

3ddedc1

Jolanrensen requested review from koperagen and zaleslaw March 2, 2026 16:49

koperagen reviewed Mar 4, 2026

View reviewed changes

zaleslaw requested changes Mar 4, 2026

View reviewed changes

Jolanrensen added 2 commits March 4, 2026 16:58

renamed promoteExample to promoteDevExamples

eba32be

migrated gradle JDK version to version catalog

c297844

Conversation

Jolanrensen commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jolanrensen commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

koperagen commented Mar 4, 2026

Uh oh!

koperagen commented Mar 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

koperagen Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zaleslaw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jolanrensen commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jolanrensen commented Feb 6, 2026 •

edited

Loading

Jolanrensen commented Mar 2, 2026 •

edited

Loading

koperagen Mar 4, 2026 •

edited

Loading

Jolanrensen commented Mar 4, 2026 •

edited

Loading