Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IP field via MultrangeQuery fix #16200 #16391

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

IP field via MultrangeQuery fix #16200 #16391

wants to merge 10 commits into from
+339 −25

Conversation

mkhludnev
Copy link
Contributor

@mkhludnev mkhludnev commented Oct 19, 2024 •
edited
Loading

Description

Combines a many concrete IPs and CIDR masks into set when querying IP field

Related Issues

Resolves #16200

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

mkhludnev and others added 5 commits October 18, 2024 01:04
@mkhludnev
use terms in set for concrete IPs keep disjunctions over ranges.
da89a31 
Signed-off-by: mikhail-khludnev <[email protected]>
@mkhludnev
forbidden api
33b8494 
Signed-off-by: mikhail-khludnev <[email protected]>
@mkhludnev
spotless
3c15413 
Signed-off-by: mikhail-khludnev <[email protected]>
mv Tests
853a344 
Signed-off-by: mikhail-khludnev <[email protected]>
@mkhludnev
Quick hack via MultiRangeQuery from Lucene sandbox
c9e2bd1 
Signed-off-by: mikhail-khludnev <[email protected]>
@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Query Capabilities labels Oct 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for c9e2bd1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for c9e2bd1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mkhludnev and others added 2 commits October 19, 2024 21:51
@mkhludnev
forbidden
a6c3148 
Signed-off-by: mikhail-khludnev <[email protected]>
javadoc
b6c3410 
Signed-off-by: mikhail-khludnev <[email protected]>
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for b6c3410: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Oct 20, 2024
Open
@mkhludnev
Copy link
Contributor Author

it's an alt of #16202

mkhludnev
mkhludnev commented Oct 20, 2024
View reviewed changes
server/src/main/java/org/opensearch/index/mapper/IpFieldMapper.java Outdated
InetAddress address;
List<InetAddress> concreteIPs = new ArrayList<>();
List<Query> ranges = new ArrayList<>();
IpMultiRangeQueryBuilder multiRange = new IpMultiRangeQueryBuilder(name());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MultiRangeQuery handles points index, but there's no a special case for DV only field.

@mkhludnev mkhludnev mentioned this pull request Oct 20, 2024
Open
relax test
6a11b54 
Signed-off-by: mikhail-khludnev <[email protected]>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 8, 2024

❌ Gradle check result for 6a11b54: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 26ff736: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mkhludnev referenced this pull request Nov 12, 2024
@harshavamsi
Updating Ip fields to use doc_values to search (#11508)
cc9ee9d 
* Updating Ip fields to use doc_values to search

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Fix IP tests

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Fix skip to allow yaml test to pass on main

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Update tests to use existing test file

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Changing skip version to match bwc

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Using exact match instead of range

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Spotless

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Fix IP field tests

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Fix spotless + precommit failure

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Get point out of query and into value

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Fix term tests

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Add skip test logic to only doc_values test

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

---------

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>
@msfroh msfroh mentioned this pull request Nov 12, 2024
Merged
8 tasks
fix dv only range search
01db875 
add filter

Signed-off-by: mikhail-khludnev <[email protected]>
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 01db875: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@mkhludnev mkhludnev mentioned this pull request Nov 16, 2024
Closed
3 tasks
@mkhludnev
Copy link
Contributor Author

TODO

hard rebase over #16628

@mkhludnev
Copy link
Contributor Author

Here we come to the question. Think about terms_query for IP/masks (this what I want to do here).

num of masks index-only docVals-only index&dvs
<maxClauses MultiRangeQ BQ { ssDvRangeQ, ..} MultiRangeQ && BQ{ssDvRangeQ,..}
>maxClauses MultiRangeQ exception (until ssDvMultiRange) MultiRangeQ (no chance for ssDvRangeQ)

The first question is to confirm this plan. So, WDYT?

Then, notice <maxClauses condition, the exact number is a question. IpFieldMapper parses terms_query only. But it has no information about number of enclosing BQs, even think about top level bool. Its' clauses are counted under overall maxClause limit. It seems like if we go with numclause condition logic we can't reasonably set the limit.
I don't want to introduce settings since the problem should gone after introducing ssDvMultiRange. The compromise is to set to something like <IndexSearcher.getMaxClausesCount()/2.
Wdyt?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz will be requested when the pull request is marked ready for review anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross will be requested when the pull request is marked ready for review andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 will be requested when the pull request is marked ready for review ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar will be requested when the pull request is marked ready for review Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE will be requested when the pull request is marked ready for review CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock will be requested when the pull request is marked ready for review dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis will be requested when the pull request is marked ready for review dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna will be requested when the pull request is marked ready for review gbbafna is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 will be requested when the pull request is marked ready for review jed326 is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal will be requested when the pull request is marked ready for review kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 will be requested when the pull request is marked ready for review mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh will be requested when the pull request is marked ready for review msfroh is a code owner

@nknize nknize Awaiting requested review from nknize nknize will be requested when the pull request is marked ready for review nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 will be requested when the pull request is marked ready for review owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta will be requested when the pull request is marked ready for review reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 will be requested when the pull request is marked ready for review Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale will be requested when the pull request is marked ready for review sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli will be requested when the pull request is marked ready for review saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja will be requested when the pull request is marked ready for review shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami will be requested when the pull request is marked ready for review sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah will be requested when the pull request is marked ready for review VachaShah is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement Enhancement or improvement to existing feature or request Search:Query Capabilities
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request] relax max Clauses Count limitation of termS query over IP field
1 participant
@mkhludnev