Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consume S3 persisted objects as an input stream #512

Open
ParvelAWS opened this issue Feb 16, 2024 · 2 comments
Open

Consume S3 persisted objects as an input stream #512

ParvelAWS opened this issue Feb 16, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@ParvelAWS
Copy link

Is your feature request related to a problem?

It would be nice if the replayer can have a built-in submodule that can read and stream S3 objects (in similar protobuf format) like the messages in Kafka topic and feed it into replay

What solution would you like?

A sub module, like S3InputStream similar to a Kafka stream reader, to feed protobuf messages into main replayer executor.

What alternatives have you considered?

An external Python or other shell scripts to feed S3 into stdin of the replayer pipe

Do you have any additional context?

Add any other context or screenshots about the feature request here.

@ParvelAWS ParvelAWS added enhancement New feature or request untriaged labels Feb 16, 2024
@ParvelAWS
Copy link
Author

Understood initially it may compromise the capability to pause/interrupt/resume replay offset as in Kafka topic stream. May have some workarounds like a journal tag file in S3 bucket.

@ParvelAWS ParvelAWS changed the title [FEATURE] Consume S3 persisted object as an input stream [FEATURE] Consume S3 persisted objects as an input stream Feb 16, 2024
@sumobrian
Copy link
Collaborator

This is specifically a request for capture and replay to support s3 and an input source as an alternative to Kafka.

@sumobrian sumobrian changed the title [FEATURE] Consume S3 persisted objects as an input stream Consume S3 persisted objects as an input stream Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Not Committed
Development

No branches or pull requests

2 participants