Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] jq dialect doesn't work for zoe topics produce #51

Open
whatsupbros opened this issue May 18, 2022 · 0 comments
Open

[Bug] jq dialect doesn't work for zoe topics produce #51

whatsupbros opened this issue May 18, 2022 · 0 comments

Comments

@whatsupbros
Copy link

whatsupbros commented May 18, 2022

When I run such a command:

zoe --cluster local topics produce --help

I am getting this feedback:

Usage: zoe topics produce [OPTIONS]

  produce messages into topics

Options:
  --dry-run              Do not actually produce records
  -t, --topic VALUE      Topic to write to
  --subject TEXT         Avro subject name to use
  -k, --key-path TEXT    Jmespath (or jq) expression to extract the key
  -v, --value-path TEXT  Jmespath (or jq) expression to extract the value
  --ts-path TEXT         Jmespath (or jq) expression to extract the timestamp
  --streaming            Read data line by line continuously
  --timeout INT          Timeout in millis
  --from-stdin           Consume data from stdin
  --from-file PATH       Consume data from a json file
  -h, --help             Show this message and exit

Examples:

  Produce a record into the 'input' topic from stdin (requires input to be a json array):
  > echo '[{"id": "1", "msg": "hello"}]' | zoe -c local topics produce -t input --from-stdin

  Use streaming mode to accept records one by one (does not require input to be a json array):
  > echo '{"id": "1", "msg": "hello"}' | zoe -c local topics produce -t input --from-stdin --streaming

  Use the id field of the input messages to determine the Kafka record key:
  > echo '[{"id": "1", "msg": "hello"}]' | zoe -c local topics produce -t input --from-stdin --key-path 'id'

  Write the data from a json file (requires the content in the file to be a json array):
  > zoe -c local topics produce -t input --from-file data.json

  Pipe data from another topic:
  > zoe -c remote topics consume input --continuously | zoe -c local topics produce -t output --from-stdin --streaming

Which suggests that I can use either jmespath or jq for -k and -v options.

However, when I try to use jq syntax, I am getting the following error:

zoe --cluster remote topics consume my-topic --max-records 1 --expose-metadata \
| zoe --cluster local topics produce --from-stdin --topic my-topic --subject my-topic-value \
--key-path ".__metadata__.key" \
--value-path ". | del(.__metadata__)" \
--streaming

Error:

2022-05-18 14:01:46 INFO zoe: producing '1' records to topic 'my-topic'
failure: runner 'local' failed
cause:
  error: Invalid jmespath expression '.__metadata__.key'

The option --dialect (by analogy with zoe topics consume) also doesn't work:

Error: no such option: "--dialect".

The reason I am trying to use jq and not jmespath is because I need to pipe data between topics in different clusters. And my topic has a key, which I also need to pipe.

I haven't found another way to do this easily other than to use --expose-metadata option for the consumer, and then remove the __metadata__ attribute on the producer side for the value, which is not possible with jmespath (at least I don't know the way), but can be easily done with jq.


zoe version print output:

{"projectVersion":"0.28.0","buildTimestamp":"2021-11-04T23:39:18.809435","createdBy":"Gradle 7.2","buildJdk":"14.0.2 (Azul Systems, Inc. 14.0.2+12)","buildOS":"Linux amd64 5.11.0-1020-azure"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant