OAK-12057 Wrong index may be selected when using LIMIT OPTION#2724
OAK-12057 Wrong index may be selected when using LIMIT OPTION#2724thomasmueller wants to merge 2 commits intotrunkfrom
Conversation
|
| // if the query contains "order by" and the index can sort on that, | ||
| // then we don't need to read all entries from the index | ||
| entryCount = Math.min(maxEntryCount, entryCount); | ||
| } |
There was a problem hiding this comment.
@bhabegger , I think we should not just clear this condition. Take following example:
Lets say we have a query with sort, but only one index support the sorted filter other don't. So the cost estimation now will be based on index size but in fact index which support sort may be better.
The addition of limit brings all indexes at same level because limit is used to get MaxEntry count, may be what we need is to bring in index's entry count to picture when finalising best best index.
long entryCount = p.getEstimatedEntryCount();
There was a problem hiding this comment.
The issue with limit is that the same index will not be chose with or without the limit which in certain cases is surprising.
This said, I completely understand that taking a sorting capable index over one which doesn't support it. So here the challenge is that we have cases which favor one and cases which favor the other. But maybe, couldn't we always favor indices which support sorting systematically rather than only if their is a limit ?
There was a problem hiding this comment.
IMO yes, sorting is something that need complete result set before we can return results. If we already have index solving sorting, it saves all the jcr calls we will have to make to all data and then sorting it in memory.
There was a problem hiding this comment.
But maybe, couldn't we always favor indices which support sorting systematically rather than only if their is a limit ?
Most relational databases account for this by adding a "cost to sort". In our case, the indexes return the same cost (which is fine), and report whether they can sort or not. So the query engines task is to add a "cost to sort" (on top of the cost returned by the indexes) for indexes that can not sort. This almost exactly matches the description "favor indices which support sorting systematically" but not 100%. What it means that an index that can not sort can still be cheaper in total than index that can sort. It just depends on the cost of sorting in the query engine.
(In Jackrabbit Oak, the Lucene and Elastic indexes often report wildly inaccurate numbers (orders of magnitude wrong), because we do not have accurate statistics currently. So such improvements will likely not have a big impact currently.)
There was a problem hiding this comment.
Maybe we should split the discussion:
- General discussion on sorting and limit
- What we do in this particular PR ?
For this PR I see the following options:
- We keep it as is to favor the same index selection with or without limit (original ticket)
- We skip it to favor an speed in case of limit (and accept that a different index maybe selected with or without limit)
- We change it to add some sorting cost
If we go for the last, what should we add ? What would be the formula ?
There was a problem hiding this comment.
What we do in this particular PR ?
I think we agree we don't want to add more to the PR... Yes it's not perfect, but perfection is the enemy of the good...
On the other hand, if we don't have an agreement on whether this PR might be risky or not, I think we could introduce a feature toggle. @tihom88 WDYT?
There was a problem hiding this comment.
I think we can go ahead with this approach. The limit checks were also added quite recently.
But we should add cost for sorting logic in a separate jira.
4 participants
No description provided.