Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AWS Lambda as Processor and Sink #4699

Closed
srikanthjg opened this issue Jul 1, 2024 · 0 comments · Fixed by #4700 · May be fixed by #5166
Closed

AWS Lambda as Processor and Sink #4699

srikanthjg opened this issue Jul 1, 2024 · 0 comments · Fixed by #4700 · May be fixed by #5166
Assignees
@srikanthjg
Labels
plugin - processor A plugin to manipulate data in the data prepper pipeline.
Milestone
v2.11

Comments

@srikanthjg
Copy link
Contributor

srikanthjg commented Jul 1, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe.

Pipeline users want to send events to AWS Lambda.

Describe the solution you'd like
Create a processor in Data Prepper which uses lambda as a remote processor . It should support

  • Retries
  • Buffering capabilities

Without Batching:

lambda-pipeline:
...
  processor:
    - lambda:
        aws:
            region: us-east-1
            sts_role_arn: <arn>
            sts_overrides:
        function_name: "uploadToS3Lambda"
        mode: synchronous
        max_retries: 3
        dlq:
            s3:
                bucket: test-bucket
                key_path_prefix: dlq/

With Batching:

lambda-pipeline:
...
  processor:
    - lambda:
        aws:
            region: us-east-1
            sts_role_arn: <arn>
            sts_overrides:
        function_name: "uploadToS3Lambda"
        mode: synchronous
        max_retries: 3
        batch:
            batch_key: "user_key"
            threshold:
                event_count: 3
                maximum_size: 6mb
                event_collect_timeout: 15s
...

Additional context
Add any other context or screenshots about the feature request here.
@srikanthjg srikanthjg changed the title Lambda as Synchronous Processor AWS Lambda as Synchronous Processor Jul 1, 2024
@srikanthjg srikanthjg mentioned this issue Jul 1, 2024
Merged
4 tasks
@dlvenable dlvenable added plugin - processor A plugin to manipulate data in the data prepper pipeline. and removed untriaged labels Jul 2, 2024
@dlvenable dlvenable added this to the v2.9 milestone Jul 2, 2024
@srikanthjg srikanthjg mentioned this issue Jul 25, 2024
Merged
4 tasks
@dlvenable dlvenable modified the milestones: v2.9, v2.10 Aug 23, 2024
@github-project-automation github-project-automation bot moved this from Unplanned to Done in Data Prepper Tracking Board Sep 24, 2024
@dlvenable dlvenable modified the milestones: v2.10, v2.11 Oct 7, 2024
@srikanthjg srikanthjg mentioned this issue Nov 2, 2024
Open
4 tasks
@srikanthjg srikanthjg changed the title AWS Lambda as Synchronous Processor AWS Lambda as Processor and Sink Nov 4, 2024
@srikanthjg srikanthjg mentioned this issue Nov 4, 2024
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@srikanthjg srikanthjg

Labels
plugin - processor A plugin to manipulate data in the data prepper pipeline.
Projects
Data Prepper Project Roadmap
Status: 2.11 (Dec 2024)
Data Prepper Tracking Board
Status: Done
Milestone
v2.11
2 participants
@dlvenable @srikanthjg