Task label feature

[ottointhesky]

First version of adding a task label feature that is queryable in the hubDB as discussed in #914.

What's missing?

• to check if task label is supported by all task submitting functions
• currently only dictDB supports the new column
• documentation of the feature (in docu and code)
• adding unittest

[minrk]

Nice! For applying the task_label metadata, look at how and where after is handled in the client, and copy that for a new label field (we can use label instead of task_label, since everything is a task already).

[ottointhesky]

Thanks for the feedback.

• As suggested I have renamed the parameter from task_label to label (in all places including the dictDB)
• Handling label like after works nicely for the map[_async] function (via the flags functionality) but for apply[_async] I still need to check the kwargs for the parameter (see location 1 and location 2 ). How can I improve that?

[minrk]

**kwargs are taken for apply, since view.apply(f, label=x) already means call f(label=x) remotely. That means label must be passed via the temp_flags context manager:

with view.temp_flags(label="x"):
    view.apply(f, a=b)

If we want to come up with another nicer way to avoid temp_flags (I agree it's a bit tedious), that would be appropriate, but it must not conflict with passing kwargs to f, and should probably be separate from this PR. It could be something like apply(f, args, __label, __after, etc.) where the __ prefix means to pass it to IPP instead.

[ottointhesky]

Thanks again for your feedback and pointing out the set flags concept. It's actually well documented what map and apply does with its parameter in the code…

In German we would say, the one who can read has a clear advantage ;-)

Now knowing the flags context manager concept, I’m happy to use this to set the label. So, the following is just a suggestion: The set_flags function could return self which would allow a single statement for setting the label and submitting a task, like

bview.set_flags(label=f"mylabel").map_async(wait, [2])

This concept is used in some C++ libraries and would allow defining list comprehension (as in the example script). It should not break any existing code. If you don’t like this idea/concept I will remove the two return self statements and we can find a different solution in a separate pull request, as you suggested.

By the way, do you like the idea of making the label also available in the AsyncResult objects?

[ottointhesky]

By the way, do you like the idea of making the label also available in the AsyncResult objects?

I managed to add/transfer the label into the metadata dict of the AsyncResult object. However, the information is only available after the task has been completed. We could set the label when creating the AsyncResult object, but it’s unclear for me where the most appropriate place to do so would be. But I guess this is only a small issue.

[ottointhesky]

I have added support for sqlitedb and mongodb as well. There are no functional changes to ipyparallel/controller/mongodb.py but in PyMongo 4 the API has changed. See the PyMongo 4 Migration Guide for details. Since PyMongo 4 was introduced in November 2021, I guess this functionality is not uses a lot... ;-)
I have implemented the changes in a way that it will still work with older pymongo versions. But I guess since PyMongo 4 is already that old, this is probably not needed. Should the PyMongo 4 API fix handled in a separate pull request?

[minrk]

Sorry for the long gap, it's been a busy week! Thanks for sticking with it.

The return self pattern is one that I like, I'll have to think about how it can best be used. But we can explore that separately.

Don't worry about splitting stuff that's already done for mongo. I think you're right that we can probably safely require recent versions. It may be time to drop the mongo stuff, I'm not sure it's used these days.

For setting the label on AsyncResult immediately, I think it makes sense, but if it's non-obvious how to do it in a clean way, we can open an Issue for that enhancement on its own.