Rework benchmarks to use different jsons (#14)

* Rework benchmarks to use different jsons * Added benchmark results * Polish documentation Co-authored-by: Breus <[email protected]> * Fix link, maybe * Fix link, maybe * Fix link, certainly --------- Co-authored-by: Breus <[email protected]>
Breus · Nov 19, 2023 · f97eb00 · f97eb00
1 parent 4d34271
commit f97eb00
Show file tree

Hide file tree

Showing 15 changed files with 123 additions and 363 deletions.
diff --git a/README.adoc b/README.adoc
@@ -6,7 +6,8 @@ image:https://img.shields.io/maven-central/v/dev.blaauwendraad/json-masker[Maven
 JSON masker which can be used to mask strings and numbers from JSON messages, corresponding to a (set of) target key(s).
 Alternatively, it can be used to mask all strings and numbers from a JSON message except the ones corresponding to the target keys.
 
-The implementation is focused on maximum (time) performance using Java and requires no additional third-party runtime dependencies.
+The library provides a convenient API and its implementation is focused on maximum (time) performance using Java.
+No additional third-party runtime dependencies are required for this library.
 
 == Features
 
@@ -144,8 +145,28 @@ String output = JsonMasker.getMasker(JsonMaskingConfig.custom(
 
 == Performance considerations
 
-This library uses a dedicated multi-target algorithm by looking for a JSON key and checking whether the target key set contains this key in constant time.
+This library uses a dedicated multi-target algorithm by looking for a JSON key and checking whether the target key set
+contains this key in constant time.
 
 The time complexity of this algorithm scales only linear in the message input length.
 Additionally, the target key set size has negligible impact on the performance.
 
+
+== Benchmarks
+
+[source]
+----
+Benchmark                           (jsonSize)  (numberOfKeys)   Mode        Score  Units
+BaselineBenchmark.noop                     4kb             100  thrpt  1567094.010  ops/s
+BaselineBenchmark.jackson                  4kb             100  thrpt    75569.597  ops/s
+JsonMaskerBenchmark.maskJsonBytes          4kb             100  thrpt   405496.513  ops/s
+JsonMaskerBenchmark.maskJsonString         4kb             100  thrpt   346559.740  ops/s
+
+BaselineBenchmark.noop                     2mb             100  thrpt     2263.683  ops/s
+BaselineBenchmark.jackson                  2mb             100  thrpt        7.923  ops/s
+JsonMaskerBenchmark.maskJsonBytes          2mb             100  thrpt     2493.899  ops/s
+JsonMaskerBenchmark.maskJsonString         2mb             100  thrpt     1312.198  ops/s
+----
+
+For full benchmark results see link:src/jmh/benchmark-history[benchmark-history]
+
diff --git a/src/jmh/benchmark-history/2022-06-09.md b/src/jmh/benchmark-history/2022-06-09.md
@@ -0,0 +1,11 @@
+CPU: MacBook Pro M1
+
+```text
+Benchmark                                                              Mode  Cnt       Score   Error  Units
+JsonMaskMultipleTargetKeysBenchmark.keyContainsMaskMultiKeysLargeJson  avgt         2649,220          ns/op
+JsonMaskMultipleTargetKeysBenchmark.keyContainsMaskMultiKeysSmallJson  avgt          246,187          ns/op
+JsonMaskMultipleTargetKeysBenchmark.loopMaskMultipleKeysLargeJson      avgt       260032,673          ns/op
+JsonMaskMultipleTargetKeysBenchmark.loopMaskMultipleKeysSmallJson      avgt        15819,945          ns/op
+JsonMaskMultipleTargetKeysBenchmark.parseAndMaskMultiKeysLargeJson     avgt       142586,845          ns/op
+JsonMaskMultipleTargetKeysBenchmark.parseAndMaskMultiKeysSmallJson     avgt         3156,930          ns/op
+```
diff --git a/src/jmh/benchmark-history/2022-07-02.md b/src/jmh/benchmark-history/2022-07-02.md
@@ -0,0 +1,11 @@
+CPU: MacBook Pro M1
+
+```text
+Benchmark                                                              Mode  Cnt       Score   Error  Units
+JsonMaskMultipleTargetKeysBenchmark.keyContainsMaskMultiKeysLargeJson  avgt         2741,496          ns/op
+JsonMaskMultipleTargetKeysBenchmark.keyContainsMaskMultiKeysSmallJson  avgt          153,333          ns/op
+JsonMaskMultipleTargetKeysBenchmark.loopMaskMultipleKeysLargeJson      avgt       259726,763          ns/op
+JsonMaskMultipleTargetKeysBenchmark.loopMaskMultipleKeysSmallJson      avgt        14396,989          ns/op
+JsonMaskMultipleTargetKeysBenchmark.parseAndMaskMultiKeysLargeJson     avgt       146207,944          ns/op
+JsonMaskMultipleTargetKeysBenchmark.parseAndMaskMultiKeysSmallJson     avgt         3102,921          ns/op
+```
diff --git a/src/jmh/benchmark-history/2022-07-12.md b/src/jmh/benchmark-history/2022-07-12.md
@@ -0,0 +1,11 @@
+CPU: MacBook Pro M1
+
+```text
+Benchmark                                                              Mode  Cnt       Score   Error  Units
+JsonMaskMultipleTargetKeysBenchmark.keyContainsMaskMultiKeysLargeJson  avgt         2512,185          ns/op
+JsonMaskMultipleTargetKeysBenchmark.keyContainsMaskMultiKeysSmallJson  avgt          165,912          ns/op
+JsonMaskMultipleTargetKeysBenchmark.loopMaskMultipleKeysLargeJson      avgt       262763,770          ns/op
+JsonMaskMultipleTargetKeysBenchmark.loopMaskMultipleKeysSmallJson      avgt        14650,831          ns/op
+JsonMaskMultipleTargetKeysBenchmark.parseAndMaskMultiKeysLargeJson     avgt       142375,488          ns/op
+JsonMaskMultipleTargetKeysBenchmark.parseAndMaskMultiKeysSmallJson     avgt         3125,639          ns/op
+```
diff --git a/src/jmh/benchmark-history/2022-09-11.md b/src/jmh/benchmark-history/2022-09-11.md
@@ -0,0 +1,11 @@
+CPU: MacBook Pro M1
+
+```text
+Benchmark                                                              Mode  Cnt       Score   Error  Units
+JsonMaskMultipleTargetKeysBenchmark.keyContainsMaskMultiKeysLargeJson  avgt         2560,033          ns/op
+JsonMaskMultipleTargetKeysBenchmark.keyContainsMaskMultiKeysSmallJson  avgt          156,414          ns/op
+JsonMaskMultipleTargetKeysBenchmark.loopMaskMultipleKeysLargeJson      avgt       258476,113          ns/op
+JsonMaskMultipleTargetKeysBenchmark.loopMaskMultipleKeysSmallJson      avgt        14488,738          ns/op
+JsonMaskMultipleTargetKeysBenchmark.parseAndMaskMultiKeysLargeJson     avgt       141392,930          ns/op
+JsonMaskMultipleTargetKeysBenchmark.parseAndMaskMultiKeysSmallJson     avgt         3079,517          ns/op
+```
diff --git a/src/jmh/benchmark-history/2023-11-17.md b/src/jmh/benchmark-history/2023-11-17.md
@@ -0,0 +1,53 @@
+CPU: AMD Ryzen 5 7600X
+
+```text
+Benchmark                           (jsonSize)  (numberOfKeys)  (obfuscationLength)   Mode  Cnt         Score   Error  Units
+BaselineBenchmark.jackson                 200b               1                  N/A  thrpt        1262056.726          ops/s
+BaselineBenchmark.jackson                 200b             100                  N/A  thrpt        1065026.221          ops/s
+BaselineBenchmark.jackson                  4kb               1                  N/A  thrpt          92561.560          ops/s
+BaselineBenchmark.jackson                  4kb             100                  N/A  thrpt          75569.597          ops/s
+BaselineBenchmark.jackson                128kb               1                  N/A  thrpt           1879.056          ops/s
+BaselineBenchmark.jackson                128kb             100                  N/A  thrpt           1911.682          ops/s
+BaselineBenchmark.jackson                  2mb               1                  N/A  thrpt             18.959          ops/s
+BaselineBenchmark.jackson                  2mb             100                  N/A  thrpt              7.923          ops/s
+BaselineBenchmark.noop                    200b               1                  N/A  thrpt       26769763.013          ops/s
+BaselineBenchmark.noop                    200b             100                  N/A  thrpt       27416983.668          ops/s
+BaselineBenchmark.noop                     4kb               1                  N/A  thrpt        1566917.285          ops/s
+BaselineBenchmark.noop                     4kb             100                  N/A  thrpt        1567094.010          ops/s
+BaselineBenchmark.noop                   128kb               1                  N/A  thrpt          36703.455          ops/s
+BaselineBenchmark.noop                   128kb             100                  N/A  thrpt          36643.225          ops/s
+BaselineBenchmark.noop                     2mb               1                  N/A  thrpt           2263.673          ops/s
+BaselineBenchmark.noop                     2mb             100                  N/A  thrpt           2263.683          ops/s
+JsonMaskerBenchmark.maskJsonBytes         200b               1                   -1  thrpt        3153816.154          ops/s
+JsonMaskerBenchmark.maskJsonBytes         200b               1                    8  thrpt        3182263.676          ops/s
+JsonMaskerBenchmark.maskJsonBytes         200b             100                   -1  thrpt        2821470.475          ops/s
+JsonMaskerBenchmark.maskJsonBytes         200b             100                    8  thrpt        2187225.371          ops/s
+JsonMaskerBenchmark.maskJsonBytes          4kb               1                   -1  thrpt         451014.030          ops/s
+JsonMaskerBenchmark.maskJsonBytes          4kb               1                    8  thrpt         445217.877          ops/s
+JsonMaskerBenchmark.maskJsonBytes          4kb             100                   -1  thrpt         405496.513          ops/s
+JsonMaskerBenchmark.maskJsonBytes          4kb             100                    8  thrpt         281729.243          ops/s
+JsonMaskerBenchmark.maskJsonBytes        128kb               1                   -1  thrpt           8979.774          ops/s
+JsonMaskerBenchmark.maskJsonBytes        128kb               1                    8  thrpt           9111.752          ops/s
+JsonMaskerBenchmark.maskJsonBytes        128kb             100                   -1  thrpt           8294.319          ops/s
+JsonMaskerBenchmark.maskJsonBytes        128kb             100                    8  thrpt           5517.200          ops/s
+JsonMaskerBenchmark.maskJsonBytes          2mb               1                   -1  thrpt           2643.338          ops/s
+JsonMaskerBenchmark.maskJsonBytes          2mb               1                    8  thrpt           2646.343          ops/s
+JsonMaskerBenchmark.maskJsonBytes          2mb             100                   -1  thrpt           2493.899          ops/s
+JsonMaskerBenchmark.maskJsonBytes          2mb             100                    8  thrpt            731.205          ops/s
+JsonMaskerBenchmark.maskJsonString        200b               1                   -1  thrpt        2945155.807          ops/s
+JsonMaskerBenchmark.maskJsonString        200b               1                    8  thrpt        2930495.471          ops/s
+JsonMaskerBenchmark.maskJsonString        200b             100                   -1  thrpt        2646630.427          ops/s
+JsonMaskerBenchmark.maskJsonString        200b             100                    8  thrpt        2091125.858          ops/s
+JsonMaskerBenchmark.maskJsonString         4kb               1                   -1  thrpt         381344.021          ops/s
+JsonMaskerBenchmark.maskJsonString         4kb               1                    8  thrpt         391209.641          ops/s
+JsonMaskerBenchmark.maskJsonString         4kb             100                   -1  thrpt         346559.740          ops/s
+JsonMaskerBenchmark.maskJsonString         4kb             100                    8  thrpt         235969.211          ops/s
+JsonMaskerBenchmark.maskJsonString       128kb               1                   -1  thrpt           7982.235          ops/s
+JsonMaskerBenchmark.maskJsonString       128kb               1                    8  thrpt           7969.799          ops/s
+JsonMaskerBenchmark.maskJsonString       128kb             100                   -1  thrpt           7284.012          ops/s
+JsonMaskerBenchmark.maskJsonString       128kb             100                    8  thrpt           5076.018          ops/s
+JsonMaskerBenchmark.maskJsonString         2mb               1                   -1  thrpt           1324.640          ops/s
+JsonMaskerBenchmark.maskJsonString         2mb               1                    8  thrpt           1314.710          ops/s
+JsonMaskerBenchmark.maskJsonString         2mb             100                   -1  thrpt           1312.198          ops/s
+JsonMaskerBenchmark.maskJsonString         2mb             100                    8  thrpt            311.324          ops/s
+```
diff --git a/src/jmh/benchmark-history/README.md b/src/jmh/benchmark-history/README.md
@@ -0,0 +1,3 @@
+### Benchmark history
+
+To prevent performance regressions, we keep track of changes in JMH benchmarks (10 s warmup, 10s average time) after each commit that could impact the performance of the masker.
diff --git a/src/jmh/java/dev/blaauwendraad/masker/json/BaselineBenchmark.java b/src/jmh/java/dev/blaauwendraad/masker/json/BaselineBenchmark.java
diff --git a/src/jmh/java/dev/blaauwendraad/masker/json/JsonMaskSingleTargetKeyBenchmark.java b/src/jmh/java/dev/blaauwendraad/masker/json/JsonMaskSingleTargetKeyBenchmark.java