modkit pileup does not use all cores on Apple Silicon and prefers E-cores

[SuhasSrinivasan]

Hi Arthur,

Thanks for developing modkit!

modkit v0.5.0
macOS v26.3.1 (a)
Mac Studio: M2 Ultra, 24 core (8E+16P) CPU, 76 core GPU, 192 GB Unified Memory, 8TB storage

I have been observing core loading for about 5 minutes when running modkit pileup and only for very short periods (tens of seconds) all cores are used, all other times mostly E-cores are used.
--threads 24

Please see screenshot.

This likely depends on how Rust handles multi-threading and macOS scheduling, but was wondering if there are any ways to improve this?

For example, when using PyTorch, it automatically detects P-cores and only runs threads on them because E-cores cause bottlenecks and add negligible compute capability.

============================================================
System Information
============================================================
PyTorch version: 2.10.0
Device: cpu
Processor: arm
Platform: macOS-26.3.1-arm64-arm-64bit
CPU threads (torch): 16
CPU cores (logical): 24

[ArtRand]

Hello @SuhasSrinivasan,

How does it look for v0.6.1 (since that's the active development refactor)?

[SuhasSrinivasan]

Hi @ArtRand ,

Thanks, have to test it, but was there any refactor to improve multithreading generally or specifically pileup?

For v0.5.0, it took 7 hours to pileup a 18GB modBAM into a 40GB bedMethyl.
Is this speed expected?

[SuhasSrinivasan]

Noted that v0.6.0 had efficiency improvements for pileup.

[ArtRand]

Hello @SuhasSrinivasan,

Modkit pileup before 0.6.0 was more optimized for "scale out" in terms of cores, but really got bloated with features without enough attention to performance. I spent a good amount of time reflecting on the usefulness of those features and completely re-wrote pileup in v0.6.z. At this point, any further performance optimization will happen on the v0.6.z codebase.

That being said, one notable problem with pileup v0.5.z was that the wall time varied a lot depending on the depth and motifs used (A mods, CpG, A and C mods, etc), so it's hard for me to say whether your timing is good or not (probably not).

I imagine that v0.6.0 is quite a bit faster, also you don't need to allocate as many cores as before since the program is mostly write limited (but there is some tuning you can do with --interval-size and --threads if you have a well defined use case).

[SuhasSrinivasan]

Hi @ArtRand,

Thank you

I did see the note about scaling

As of version 0.6.0 the efficiency of pileup has been greatly improved. As a result, fewer threads are often required to get high throughput. For example, increasing threads beyond 8 a 40X human genome BAM will often not yield much benefit.

I am trying running v0.6.1 on Ubuntu 24.04 LTS with homogeneous x86 cores, AMD 7975WX, 300GB/s memory bandwidth and NVMe storage.
With 8 threads for both --threads and --sampling-threads, and --sampling-frac 0.001, there was no progress after 1 hour.
This was very strange, so I interrupted the run and taking a look again at options before restarting.

Once this seems good, I can test on the Mac Studio as I have to compile v0.6.1.

[SuhasSrinivasan]

Something is not right, not sure what though.
After 50 minutes, there is no progress with sampling.
In v0.5.0 sampling the default 10042 reads would be super fast.

> parsed user-input threshold 0.5 for mod-code m
> parsed user-input threshold 0.5 for mod-code 17596
> parsed user-input threshold 0.5 for mod-code 17802
> parsed user-input threshold 0.5 for mod-code a
> discarded 223943 contigs with zero aligned reads
> parsed 4 base modification(s). Base modifications other than 'A:17596,A:a,C:m,T:17802' will be counted as 'N_other'.
> adding single-base motif: 'A 0'
> adding single-base motif: 'C 0'
> adding single-base motif: 'T 0'
> sampling 0.05% of reads

I will be reverting back to v0.5.0 for now as I have a lot of new samples to process.

[ArtRand]

@SuhasSrinivasan Could you show me the command you're running which is hanging?

[SuhasSrinivasan]

Hi @ArtRand
Thanks for the late night troubleshooting help!

Here is the command for v0.6.1.
THREADS=8

            "${MODKIT_PATH}" pileup \
            --modified-bases C:m A:a A:17596 T:17802 \
                --threads "${THREADS}" \
                --sampling-threads "${THREADS}" \
                --reference "${REFERENCE_PATH}" \
                --sampling-frac 0.0005 \
                --seed 1234567 \
                --mod-threshold "a:${MOD_THRESHOLD}" \
                --mod-threshold "m:${MOD_THRESHOLD}" \
                --mod-threshold "17802:${MOD_THRESHOLD}" \
                --mod-threshold "17596:${MOD_THRESHOLD}" \
                "${bam_sorted}" "${pile_file}"

And for v0.5.0 (this is working as previously)
THREADS=16

            "${MODKIT_PATH}" pileup \
                --threads "${THREADS}" \
                --mod-threshold "a:${MOD_THRESHOLD}" \
                --mod-threshold "m:${MOD_THRESHOLD}" \
                --mod-threshold "17802:${MOD_THRESHOLD}" \
                --mod-threshold "17596:${MOD_THRESHOLD}" \
                "${bam_sorted}" "${pile_file}"

Machine is the same for both

Ubuntu 24.04 LTS with homogeneous x86 cores, AMD 7975WX, 300GB/s memory bandwidth and NVMe storage.

[ArtRand]

@SuhasSrinivasan I have a feeling this is the problem:

--sampling-threads "${THREADS}" \
# etc
--sampling-frac 0.0005 \

remove these from the v0.6.1 run?

[SuhasSrinivasan]

Hi @ArtRand ,

Thanks, why do you suspect the --sampling-threads is stalling the sampling?

And --sampling-frac is needed and why I included it (because previously this was 10042 reads), but does not match the command usage doc and Extended subcommand help.

error: the following required arguments were not provided:
  --sampling-frac <SAMPLING_FRAC>

Usage: modkit pileup --reference <REFERENCE_FASTA> --sampling-frac <SAMPLING_FRAC> --modified-bases <MODIFIED_BASES>... --threads <THREADS> --seed <SEED> --mod-threshold <MOD_THRESHOLDS> <IN_BAM> <OUT_BED>

[ArtRand]

Hello @SuhasSrinivasan,

The threshold estimation routine is identical in v0.5.0 and v0.6.1, so why do you need --sampling-frac for the v0.6.1 command?

You can run pileup with --sampling-threads but without --sampling-frac.

[SuhasSrinivasan]

Hi @ArtRand ,

The error message is from modkit pileup not from my script. 😅
I was surprised to see that --sampling-frac is needed.

I can replicate this consistently. I am using the same script for v0.5.0 and v0.6.1.
I just change the accessible code blocks and this error is shown by modkit pileup when --sampling-frac is not used.

error: the following required arguments were not provided:
  --sampling-frac <SAMPLING_FRAC>

Usage: modkit pileup --reference <REFERENCE_FASTA> --sampling-frac <SAMPLING_FRAC> --modified-bases <MODIFIED_BASES>... --threads <THREADS> --seed <SEED> --mod-threshold <MOD_THRESHOLDS> <IN_BAM> <OUT_BED>

[SuhasSrinivasan]

After an hour, confirming that the below did not work for v0.6.1, still stalled at the sampling stage.

THREADS=8

            "${MODKIT_PATH}" pileup \
            --modified-bases C:m A:a A:17596 T:17802 \
                --threads "${THREADS}" \
                --reference "${REFERENCE_PATH}" \
                --sampling-frac 0.0005 \
                --seed 1234567 \
                --mod-threshold "a:${MOD_THRESHOLD}" \
                --mod-threshold "m:${MOD_THRESHOLD}" \
                --mod-threshold "17802:${MOD_THRESHOLD}" \
                --mod-threshold "17596:${MOD_THRESHOLD}" \
                "${bam_sorted}" "${pile_file}"

> parsed user-input threshold 0.5 for mod-code 17596
> parsed user-input threshold 0.5 for mod-code 17802
> parsed user-input threshold 0.5 for mod-code a
> parsed user-input threshold 0.5 for mod-code m
> discarded 223943 contigs with zero aligned reads
> parsed 4 base modification(s). Base modifications other than 'A:17596,A:a,C:m,T:17802' will be counted as 'N_other'.
> adding single-base motif: 'A 0'
> adding single-base motif: 'C 0'
> adding single-base motif: 'T 0'
> sampling 0.05% of reads
^C