Skip to content

Conversation

@JoerivanEngelen
Copy link
Contributor

@JoerivanEngelen JoerivanEngelen commented Feb 9, 2026

Fixes #1768

Description

Fixes broken pieces of the code when updating to pandas 3, which has the following major changes which also affect iMOD Python somewhat:

  • Dedicated string data type by default: string columns are now inferred as the new str dtype instead of object, providing better performance and type safety
  • Consistent copy/view behaviour with Copy-on-Write (CoW) (a.k.a. getting rid of the SettingWithCopyWarning): more predictable and consistent behavior for all operations, with improved performance through avoiding unnecessary copies
  • New default resolution for datetime-like data: no longer defaulting to nanoseconds, but generally microseconds (or the resolution of the input), when constructing datetime or timedelta data (avoiding out-of-bounds errors for dates with a year before 1678 or after 2262)

Because of these changes, some parts of the code had to be modified slightly to get the tests to work. I think with the latter change we could simplify the code base a lot, as all logic to deal with datetimes going beyond the year 2262 and prior to 1678 wouldn't be necessary anymore. This was quite a headache in the past, and I think the choice for microseconds by default will make our lives significantly easier. I created an issue for this.

In detail this PR alters the following:

  • Pin pandas version to 3.* in pixi.toml
  • Update setup and asserts in quite some tests to work with pandas 3.0. Most of these tests will not work with pandas 2.0 anymore, as we now assert for pandas 3.0 behavior
  • Update examples to work with pandas 3.0
  • Fix statements where a view was altered (e.g. altering .values attribute), this is now forbidden by pandas
  • Update to new string handling, sometimes I had to internally enforce object dtype to get the old behavior. In other cases we could check for strings by checking dtype=="object" anymore. Luckily there is pd.api.types.is_string_dtype, which both works for pandas 2 as well as pandas 3.
  • Add unittest for to_pandas_datetime_series to check whether pandas behavior is similar in this regard.

Checklist

  • Links to correct issue
  • Update changelog, if changes affect users
  • PR title starts with Issue #nr, e.g. Issue #737
  • Unit tests were added
  • If feature added: Added/extended example
  • If feature added: Added feature to API documentation
  • If pixi.lock was changed: Ran pixi run generate-sbom and committed changes

@JoerivanEngelen JoerivanEngelen marked this pull request as ready for review February 10, 2026 15:53
@sonarqubecloud
Copy link

@JoerivanEngelen JoerivanEngelen added this pull request to the merge queue Feb 11, 2026
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to no response for status checks Feb 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support pandas 3.0

2 participants