Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Send RetryInfo on OTel Timeouts #4294

Merged
merged 20 commits into from
Nov 13, 2024
Merged

Send RetryInfo on OTel Timeouts #4294

merged 20 commits into from
Nov 13, 2024

Commits on Sep 18, 2024

  1. WIP Send RetryInfo on OTel Timeouts 

    DataPrepper is sending `RESOURCE_EXHAUSTED` gRPC responses
whenever a buffer is full or a circuit breaker is active. These statuses do
not contain a retry info. In the OpenTelemetry protocol, this implies a
non-retryable error, that will lead to message drops, e.g. in the OTel
collector. To apply proper back pressure in these scenarios a retry info is
added to the status.

Signed-off-by: Karsten Schnitter <[email protected]>
    @KarstenSchnitter
    KarstenSchnitter committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    7dd334b View commit details
    Browse the repository at this point in the history

  2. WIP Add RequestInfo Delay Calculator 

    Implementation of exponential backoff. Idea is to start with a minimum
delay on the first time-out or circuit breaker activation. If the next such
event happens within twice the last delay after the previous event, double
the delay until a maximum delay is reached. Use the maximum delay from
then on, until a sufficiently long period (maximum delay) without an event
happens. Then the delay is reset to minimum.

TODO: Make minimum and maximum delay configurable.
Signed-off-by: Karsten Schnitter <[email protected]>
    @KarstenSchnitter
    KarstenSchnitter committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    85a7a18 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Make backoff calculation configurable 

    (cherry picked from commit 0d45f77)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    d6119b6 View commit details
    Browse the repository at this point in the history

  2. Rename RetryInfo -> RetryInfoConfig 

    (cherry picked from commit f8ac48e)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    52d2488 View commit details
    Browse the repository at this point in the history

  3. Create test for back-off calculation 

    (cherry picked from commit ff675dc)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    8c667a7 View commit details
    Browse the repository at this point in the history

  4. Return retry info in metrics source 

    (cherry picked from commit 43ba7ee)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    a42ccce View commit details
    Browse the repository at this point in the history

  5. Return retry info in logs source 

    (cherry picked from commit 1f90615)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    603f4fb View commit details
    Browse the repository at this point in the history

  6. Get rid of a few warnings 

    (cherry picked from commit 2977c1f)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    7c6bb9a View commit details
    Browse the repository at this point in the history

  7. Use Duration for delays in RetryInfoConfig 

    (cherry picked from commit 473db0e)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    c2bc33f View commit details
    Browse the repository at this point in the history

  8. Use Duration for delays in RetryInfoConfig for OTelLogsSource 

    (cherry picked from commit 6ef9b7e)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    da5b668 View commit details
    Browse the repository at this point in the history

  9. Use Duration for delays in RetryInfoConfig for OTelMetricsSource 

    (cherry picked from commit 091e9a6)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9c6d808 View commit details
    Browse the repository at this point in the history

  10. Add documentation for retry information 

    (cherry picked from commit a588b09)
Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    ccbfc9b View commit details
    Browse the repository at this point in the history

  11. Add review findings 

    Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9f3a39d View commit details
    Browse the repository at this point in the history

  12. Merge pull request #2 from sternadsoftware/4119-tomas 

    Add RetryInfo Configuration
    @KarstenSchnitter
    KarstenSchnitter authored Oct 4, 2024
    Configuration menu
    Copy the full SHA
    f4b0c60 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. Add java time module to tests 

    Signed-off-by: Tomas Longo <[email protected]>
    Tomas Longo committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    b3fe06d View commit details
    Browse the repository at this point in the history

  2. Merge pull request #3 from sternadsoftware/4119 

    Add java time module to tests
    @KarstenSchnitter
    KarstenSchnitter authored Oct 8, 2024
    Configuration menu
    Copy the full SHA
    1232498 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. 4119 Initialize with a buffer to prevent pre-mature backoff calculation 

    Signed-off-by: Tomas Longo <[email protected]>
    @KarstenSchnitter
    Tomas Longo authored and KarstenSchnitter committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    b29246a View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Apply suggestions from code review 

    Co-authored-by: David Venable <[email protected]>
Signed-off-by: Karsten Schnitter <[email protected]>
    @KarstenSchnitter @dlvenable
    KarstenSchnitter and dlvenable authored Nov 1, 2024
    Configuration menu
    Copy the full SHA
    3feda83 View commit details
    Browse the repository at this point in the history

  2. Apply suggestions from code review 

    Co-authored-by: David Venable <[email protected]>
Signed-off-by: Karsten Schnitter <[email protected]>
    @KarstenSchnitter @dlvenable
    KarstenSchnitter and dlvenable authored Nov 1, 2024
    Configuration menu
    Copy the full SHA
    9e49340 View commit details
    Browse the repository at this point in the history

  3. Rename Test-Cases to Adhere to Convention 

    Signed-off-by: Karsten Schnitter <[email protected]>
    @KarstenSchnitter
    KarstenSchnitter committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    36deb0c View commit details
    Browse the repository at this point in the history