commit is forbidden in queue jobs

[sbidoul]

It is forbidden to call env.cr.commit() in delayed job functions.

It has always been bad practice, but now a commit releases the job lock and can cause it to be started again (while still running) by the dead jobs requeuer.

#880 makes such commits an error to make that explicit.

I open this issue to collect feedback on this, as there might be (semi-)legitimate uses of commit.

[amh-mw]

Use case:

Import hundreds of millions of rows from external systems with ordering (chain, group).

Initial Problems:

• Memory exhausted creating job execution graph.
• The queue_job_batch module doesn't interact with (chain, group).

Initial implementation:

• Custom batching module that imports multiple rows per job.
• Use of self.env.cr.commit() introduced to save position in batch and partial progress after exception thrown or prior to thread timeout (SIGKILL).
• For lightweight records without risk of UniqueViolation or SerializationFailure, batches are orders of magnitude faster than one record per job.

Better implementation?

• Improve queue_job_batch module with some sort of permanently retrying job that completes when the entire batch is successful or failed?

To be absolutely clear, I am not arguing for self.env.cr.commit(); this antiquated design originated circa 12.0. I should have some cycles this quarter to work on queue_job_batch or whatever alternative approach seems best.