Support multi-node tasks

[giovannipizzi]

(Note: maybe I'm completely misunderstanding how multiple allocs are supposed to be used - so please tell me if this can be solved in a different way!)

I have two usecases and I didn't understand if they are already supported (or they work already by in some way I don't know)

1. on some of our supercomputers at the Swiss supercomputing centre CSCS, we have two partitions, one with CPU only and one with GPUs. Therefore, I'd like to start two allocs, one for each (e.g. -C mc vs. -C gpu), give to each of them a name, and when I submit a job to hq, specify to which of the two they should run. Is this possible already? (The docs state that the name is only used for debug reasons)

2. If I understand correctly, when I start the alloc, I specify all parameters, including how many nodes each worker should have. Is there a way to start automatically workers with different number of nodes? E.g. if I have a machine with nodes with 128 cores, I might want to have jobs with e.g. 16, 32, 64 or 128 cores run in workers that use 1 node; but I might also have e.g. jobs that I want to run on 2 or 4 nodes, and for these I'd like to also have some workers run on 2 or 4 nodes. At the same time, I don't want the jobs with <= 128 cores to run on these workers but only on the 1-node workers. Can this be already achieved by creating 3 allocs with 1, 2, 4 nodes respectively? I tried to do this but I saw some strange result that I couldn't debug properly or test extensively yet (I think I had only the 1-node alloc first, I submitted a 2-node job to hq and I think that hq started submitting workers, when they entered the SLURM queue it realized they weren't OK for the job and it started submitting more; I then created also a 2-node alloc but it didn't enter the queue for quite some time because of the priorities of the machine, and in the meantime I kept having 1-node workers being submitted, starting, and dying after the 5-minutes idle timeout, and new ones getting launched again).
   For this usecase, I would either have something like the point 1 above (I can decide to send a job only to one specific alloc and avoid that the ones that are not compatible keep starting, timing out and restart), or some automatic way for hq to understand how many nodes a job requires, and send to the correct alloc

[Kobzol]

1. This is definitely a valid use case and we intend to support it. Currently, this can be achieved in a semi-manual way.

   You can assign resources to specific workers and jobs, e.g. you can say that worker A has the GPU resource and that job 1 needs 1 GPU in order to execute. Currently, numeric resources are implemented, while a boolean one (does have/doesn't have GPU) might be more natural here. We will introduce a shortcut for such boolean resources to the CLI.

   However, currently you can only specify these resources only when starting a worker manually. We also intend to let the user choose the resource when using automatic allocation.

2. I'm not sure whether you're talking about multi-node PBS/Slurm allocations or multi-node MPI computations here. HyperQueue currently does not support multi-node tasks, which means that a single task (currently a single executed binary) will always run on a single worker on a single node. It could in theory start a multi-node MPI computation, but this will not work as expected, even if the allocation had multiple nodes, the workers on other nodes will not know about this computation and it would most probably result in over-subscription.

   On the other hand, you can tell the auto allocator to create PBS/Slurm allocations with multiple nodes, but all it will do is to start a single independent worker on each allocated node. It's basically just a tuning parameter that can be used e.g. if you know that it will benefit queue wait times or otherwise overcome some PBS/Slurm limits. Otherwise HQ will just queue N single-node allocations in order to start the allocations as soon as possible.

[spirali]

@giovannipizzi In general, multi-node allocations tasks are on our roadmap and we definitely want to include them in HQ, but it does not have a top priority for us right now.

[spirali]

@giovannipizzi I am sorry, I used a wrong term in my previous text. We are supporting multinode (PBS/SLURM) allocations. What we are not supporting right now is multi-node tasks.

[giovannipizzi]

Ah, I see! I didn't manage to test multi-node allocations yet (I had queued a few but then I stopped them because it didn't enter the queue as quick as I had wanted ;-) ). So, I didn't realise that a 2-node alloc starts 2 1-node workers in it.

Do I therefore understand correctly that, at the moment, the best is to have a filter before submitting: if it's a 1-node job (or smaller than one) one submits to hq, if it's larger one submits directly to the scheduler (but then clearly will not benefit from running multiple shorter jobs within the same SLURM job?).

For now I think this is enough but I'd be happy to see also this usecase covered in the future, if it's of interest to you and others - so we can just submit everything to hq.

[spirali]

We want to support "everything in hq", but so far, you are right: upto 1-node tasks to HQ and multi-node tasks directly into SLURM/PBS

[manuel-g-castro]

Hello, HyperQueue devs!

I am also making some tests with multinode MPI applications, and I can vouch that HyperQueue works!

But I found a couple of issues which I think are worth reporting. I think almost all of them are related to the documentation.

1. I could not make multi-node tasks work with jobfiles.

HyperQueue does not set the HQ_NODE_LIST variable that is needed to know which nodes will be used by the task. Not sure if this is a bug or if it is by design. I think is rather the latter so it would be helpful to have it documented.

2. The documentation's MPI commands did not work for me.

mpirun --node-list=$HQ_NODE_FILE ./a-program failed complaining by the lack of --node-list flag. I have checked all the MPI implementation's that I am aware of (intel, mpich, and openmpi), and could not find this flag in neither. The closest is --hostfile, which I think is part of the standard therefore it is in every implementation.

srun --nodefile=$HQ_NODE_FILE --nodes=$HQ_NUM_NODES mpirun ... failed because HQ_NUM_NODES was not set. This issue is minor, since it can be quickly amended by $(cat $HQ_NODE_FILE | wc --lines) or any other equivalent bash shenanigans.

But, even changing to hostfile or setting the number of nodes nothing worked. Either MPI complained about the lack of slots or it hanged in the initialization.

What works for me was to rely completely on srun. So the command that I run is srun -v --overlap --nodefile=$HQ_NODE_FILE --nodes=$(cat $HQ_NODE_FILE | wc --lines) -n 320 ./mpi_application.

So maybe the documentation for Slurm platforms should be updated with this command.

Thanks! And I am happy to make a PR with documentation improvement, if you are okay with it.

[spirali]

Thanks for the report on the multi node.

1. Thanks for reporting! It is a bug. Will be fixed after merging Fix multinode tasks defined in job file #1070.

2. It seems that --node-list is a documentation error. Thanks for reporting.

If I remember correctly when browsing some MPI implementation, some MPI implementations completely ignore command line arguments if it sees some environmental variables, so it probably detects that it runs in SLURM and ignores --hostfile argument.

Regarding srun solution, I am wondering when it may happen that $HQ_NODE_FILE is set but HQ_NUM_NODES is not set. Looking into the code, it should not happen.

We would be happy for PR that updates multi-node documentation, as we do not have currently many users with multi-node pipelines, so any feedback from someone who tries it in the wild is welcome!