Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Date Processor can't map @timestamp correctly #5119

Open
powerpower2005 opened this issue Oct 28, 2024 · 2 comments
Open

[BUG] Date Processor can't map @timestamp correctly #5119

powerpower2005 opened this issue Oct 28, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@powerpower2005
Copy link

powerpower2005 commented Oct 28, 2024

Describe the bug

I'm trying to use opensearch ingestion and data prepper for logging and visualized with opensearch

Log has a field time with format like "yy/MM/dd HH:mm:ss.SSS"
EX) 24/10/28 13:35:45.721

I want to make this to timstamp, so I use date processor like below

- date:
      match:
        - key: time
          patterns: ["yy/MM/dd HH:mm:ss.SSS"]
      destination: "@timestamp"

expect result
time and @timestamp are same value.

result
but @timestamp has been disappeared.

As I know timezone is optional and rather default value is system time zone. So I search the log within 3 days. It doesn't work
But this log doesn't have a field @timestamp anymore. and I can't see any histgram on dashboard.
So I try it with another method

I set the timezone source and destination.

- date:
      match:
        - key: time
          patterns: ["yy/MM/dd HH:mm:ss.SSS"]
      destination: "@timestamp"
      source_timezone: "Asia/Seoul"
      destination_timezone: "Asia/Seoul"

Finally, It works.

But I wonder why it happen.

I always appreciate your work.

Related component
Plugins

To Reproduce

use any date type field to @timestamp without setting timezone.
In that case, you'll never get any logs on dashboard.

Expected behavior

Expect
time : 24/10/28 21:23:25.083
-> @timestamp : 2024/10/28 21:23:25.083

Additional Details
Plugins
Data Prepper

Environment

Opensearch 2.11
fluent-bit 3.11 ( Log Source throught HTTP)

@powerpower2005
Copy link
Author

I think there is some problem with putting timestamp on a record.

DateProcessor.java line104

I think Timezone is optionally value. Even though zonedDateTime is null, we still need to put timestamp on a record.

I'm not good at java. if some logical error.

please forgive me generously.

@powerpower2005
Copy link
Author

If you think it is a bug and you didn't start yet, Can I fix it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

No branches or pull requests

2 participants