Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reformat log statements so that they aren't susceptible to formatting errors and worse when {} are in the message #1084

Merged
merged 6 commits into from
Nov 5, 2024

Conversation

gregschohn
Copy link
Collaborator

@gregschohn gregschohn commented Oct 19, 2024

Description

This converts and canonicalizes all of our existing log usage for all java projects. The rules followed are

  • setCause() should be the first thing after setting the log level.
  • setMessage() should never do any formatting & therefore never needs to create a lambda.
  • addArgument() should only create lambdas if a method reference won't work (e.g. () -> foo.get() should be replaced with foo:get).
  • addArguments and log() should be on separate lines if they can't fit onto one line.

See https://opensearch.atlassian.net/browse/MIGRATIONS-2161 for follow-up work to make these changes more durable.


  • Category Bug fix
  • Why these changes are required? For security and reliability.
  • What is the old behavior before changes and new behavior after changes? In logging, if {} was within the contents of some strings being logged as messages, those would have thrown Exceptions. Now "{}" will print out instead. A class of security bugs should also be closed down with this change.

Issues Resolved

Jira issue
Github Issue

Testing

gradle testing

Check List

  • New functionality includes testing
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link

codecov bot commented Oct 19, 2024

Codecov Report

Attention: Patch coverage is 80.16032% with 99 lines in your changes missing coverage. Please review.

Project coverage is 80.59%. Comparing base (e78d172) to head (fd43edc).
Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
...migrations/replay/kafka/TrackingKafkaConsumer.java 70.68% 17 Missing ⚠️
...y/CapturedTrafficToHttpTransactionAccumulator.java 86.20% 8 Missing ⚠️
.../opensearch/migrations/replay/TrafficReplayer.java 0.00% 6 Missing ⚠️
...rations/bulkload/common/LuceneDocumentsReader.java 37.50% 5 Missing ⚠️
...ad/workcoordination/OpenSearchWorkCoordinator.java 82.75% 5 Missing ⚠️
...ns/replay/traffic/expiration/BehavioralPolicy.java 0.00% 5 Missing ⚠️
...s/replay/traffic/source/BlockingTrafficSource.java 77.27% 5 Missing ⚠️
...replay/datahandlers/NettyPacketToHttpConsumer.java 91.30% 4 Missing ⚠️
.../datahandlers/http/NettyJsonContentAuthSigner.java 0.00% 4 Missing ⚠️
.../migrations/bulkload/common/DocumentReindexer.java 66.66% 3 Missing ⚠️
... and 19 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1084      +/-   ##
============================================
+ Coverage     80.40%   80.59%   +0.18%     
+ Complexity     2944     2881      -63     
============================================
  Files           393      393              
  Lines         14536    14554      +18     
  Branches       1001     1000       -1     
============================================
+ Hits          11688    11730      +42     
+ Misses         2242     2220      -22     
+ Partials        606      604       -2     
Flag Coverage Δ
gradle-test 78.70% <80.16%> (+0.23%) ⬆️
python-test 90.33% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

… errors and worse when {} could end up directly in the 'message' string.

I tried to codify on a couple of standards as well.
1) setMessage(Supplier...) should not be used.  Placeholders and arguments should always be used instead.
2) All arguments being added should use a supplier if the argument requires any evaluation of a function.  To confirm, the regex `.addArgument\([^\(\)]+\([^\)]*` shouldn't found present.

On stylistic levels
3) The sequence of fluent/builder operations should be setCause, setMessage, addArguments...
4) If more than one argument is added, each argument should be on a separate line, followed by a log() also on its own line.

Signed-off-by: Greg Schohn <[email protected]>
@gregschohn gregschohn marked this pull request as ready for review October 23, 2024 04:01
@gregschohn gregschohn changed the title WIP to reformat log statements so that they aren't susceptible to formatting errors and worse when {} are in the message. reformat log statements so that they aren't susceptible to formatting errors and worse when {} are in the message. Oct 24, 2024
@gregschohn gregschohn changed the title reformat log statements so that they aren't susceptible to formatting errors and worse when {} are in the message. Reformat log statements so that they aren't susceptible to formatting errors and worse when {} are in the message Oct 24, 2024
+ "ms before trying again."
)
log.atInfo().setMessage("{}")
.addArgument(() -> "After " + finalShardSetupAttemptNumber + "another process holds the lock"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's change this to use constant string with argument insertion like the rest of the log lines

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Member

@AndreKurait AndreKurait left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clean up the pr description and i'm good with this.

Updated logger usage in PredicateLoader.java and JsonTransformerTest.java to bring them up to speed with all the other logging calls

Signed-off-by: Greg Schohn <[email protected]>
…to setMessage(FORMAT).addArgument...log().

Those left behind had either one argument or was clearer to leave as it was.

Signed-off-by: Greg Schohn <[email protected]>
Signed-off-by: Greg Schohn <[email protected]>

# Conflicts:
#	RFS/src/main/java/org/opensearch/migrations/bulkload/common/OpenSearchClient.java
@gregschohn gregschohn merged commit 48da9ce into opensearch-project:main Nov 5, 2024
14 of 15 checks passed
@gregschohn gregschohn deleted the LogFormatFixes branch November 5, 2024 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants