Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): Add support for Tantivy based time series index #1852

Merged
merged 18 commits into from
Sep 20, 2024

Commits on Jul 17, 2024

  1. misc(build): Add support for Rust JNI libraries (#1815)

    Add build integration to build Rust librarie that can be used for native methods
    in a module:
    * Opt in plugin that enables Rust integration
    * compile, test, lint, package steps
    * Support for multi-arch build allowing the final jar to support more than one platform
    
    No libraries are using this as of this commit, follow ups will add Rust code usage
    rfairfax authored Jul 17, 2024
    Configuration menu
    Copy the full SHA
    55f521d View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. refactor(core): Move common index logic to abstract base class (#1816)

    In preparation for new index types separate common logic from Lucene specific
    logic.  No functional changes are made as part of this commit, just moving
    code around and refactoring to accept multiple index implementations.
    
    In most cases this amounts to moving code out of PartKeyLuceneIndex and into PartKeyIndexRaw
    without further changes.  In a few cases things that were instaniating Lucene specific
    types now use callbacks or other approaches to allow instantiating index specific
    types.
    rfairfax authored Jul 23, 2024
    Configuration menu
    Copy the full SHA
    9779a31 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. feat(core): Add open/close support for Tantivy Index (#1817)

    Commit 1 of adding Tantivy index support.  This PR is broken down into a small subset
    of the overall Tantivy logic to allow for easier partial review.  The index code is not
    usable end to end until all parts are committed.
    
    This adds the basic Rust project skeleton and supports opening, applying schema, and closing the index.
    Many methods are unimplemented and will be added in follow up PRs.  End to end testing is not available
    in this PR as the index test suite requires ingestion and query support for verification.
    rfairfax authored Aug 1, 2024
    Configuration menu
    Copy the full SHA
    6148ee4 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Configuration menu
    Copy the full SHA
    6e45ca3 View commit details
    Browse the repository at this point in the history
  2. Merge latest develop #1828

    Merge latest develop
    rfairfax authored Aug 2, 2024
    Configuration menu
    Copy the full SHA
    50b2484 View commit details
    Browse the repository at this point in the history
  3. feat(core): Add ingestion support for Tantivy

    This implements enough of the interface to add documents to the Tantivy
    index.  This is part 2 of the series and is not usable end to end until
    additional PRs complete.
    
    Document additional and removal is covered in this PR.  Documents are
    created by calling a single native method that has many of the built-in fields,
    such as start and end time, passed as method parameters.  Dyanmic fields
    are passed to the Rust code via a simple binary encoding to reduce
    JNI overhead.
    rfairfax committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    b886661 View commit details
    Browse the repository at this point in the history
  4. Fix clippy config file

    rfairfax committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    bf797b5 View commit details
    Browse the repository at this point in the history
  5. Add _type_ support

    rfairfax committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    572ac53 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Address feedback

    rfairfax committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    2b90c68 View commit details
    Browse the repository at this point in the history
  2. Address feedback

    rfairfax committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4cc2606 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. feat(core): Add ingestion support for Tantivy #1827

    This implements enough of the interface to add documents to the Tantivy index.  This is part 2 of the series and is not usable end to end until additional PRs complete.
    
    Document additional and removal is covered in this PR.  Documents are created by calling a single native method that has many of the built-in fields, such as start and end time, passed as method parameters.  Dyanmic fields are passed to the Rust code via a simple binary encoding to reduce JNI overhead.
    
    **Pull Request checklist**
    
    - [X] The commit(s) message(s) follows the contribution [guidelines](CONTRIBUTING.md) ?
    - [ ] Tests for the changes have been added (for bug fixes / features) ?
    - [ ] Docs have been added / updated (for bug fixes / features) ?
    
    **New behavior :**
    
    Part 2 of Tantivy index support. The functionality is still incomplete and not ready for usage yet.
    rfairfax authored Aug 14, 2024
    Configuration menu
    Copy the full SHA
    4ef88b0 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. feat(core): Add Rust side of Tantivy query support (#1835)

    This contains the Rust code needed to support all query patterns for FiloDB.  This is part 3
    of the series and is not usuable end to end until additional PRs complete.
    
    This PR splits the logic into a common tantivy library, where collectors and query extensions
    are implemented, and the filodb library, where specific JVM bridge methods and business logic
    exists.
    
    The majority of this PR are custom collectors and caching to better match the performance of
    Lucene in the FiloDB use cases.  The glue logic to connect these collectors to JVM methods
    are for the most part very straight forward - parse a query, run it, serialize results.
    rfairfax authored Aug 26, 2024
    Configuration menu
    Copy the full SHA
    8dfe6f2 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. feat(core): Add Scala side of Tantivy query support (#1843)

    This contains the Scala side of query support.  This completes the basic end to
    end flow, so it also includes regression and performance tests.
    
    There will be some small cleanup PRs after this for things like additional metrics,
    but with this PR the Tantivy index can be successfully activated via config.
    rfairfax authored Sep 6, 2024
    Configuration menu
    Copy the full SHA
    38de69d View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. feat(core): Add tantivy metrics (#1845)

    Add two new metrics for commit processing time and cache hit rates.  These will
    be used for monitoring as we roll out and turn the feature on for testing.
    rfairfax authored Sep 10, 2024
    Configuration menu
    Copy the full SHA
    a5e1e50 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    54bf773 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ffe8818 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Merge pull request #1847 from rfairfax/rfairfax/feat-index-rust-merge…

    …-9-10
    
    feat(core): Merge latest develop
    rfairfax authored Sep 11, 2024
    Configuration menu
    Copy the full SHA
    faaa9c0 View commit details
    Browse the repository at this point in the history
  2. feat(core): Optimize Rust code by default (#1848)

    Previously the code only optimized if a flag was passed during sbt invocation
    or if we're building multi-arch.  This has created confusing results
    where perf drifts because debug mode was used.
    
    After doing testing and development it was found that release is a suitable
    default - it does incremental build, includes symbols for debugging,
    and while not as fast as debug builds is still fast enough for an inner loop.
    
    You can still build with debug by setting rust.optimize=false for any
    cases where it is needed during local development.
    rfairfax authored Sep 11, 2024
    Configuration menu
    Copy the full SHA
    c21e4ff View commit details
    Browse the repository at this point in the history