HIVE-28600: Iceberg: Check that table/partition requires compaction b… #5529

difin · 2024-11-05T03:03:32Z

…efore compacting

What changes were proposed in this pull request?

Added checking if Iceberg table/partition requires compaction before starting a manual compaction request.

Why are the changes needed?

Performance improvement to skip tables/partitions which do not require compaction.

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

No

How was this patch tested?

Updated existing q-tests, added new unit test.

…efore compacting

zratkai · 2024-11-14T12:47:32Z

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java

@@ -458,6 +458,52 @@ public static List<DeleteFile> getDeleteFiles(Table table, String partitionPath)
        t -> ((PositionDeletesScanTask) t).file()));
  }

+  public static float getFileSizeRatio(Table table, String partitionPath, long fileSizeInBytesThreshold) {


The name of the method does not show what ratio it, as it does nothing with the file actual size, rather it check the number of files.
Like e.g. getUncompactedRatio would be better.

it does nothing with the file actual size, rather it check the number of files.

The method uses the actual file sizes, it checks for each file if its actual size is above or below the threshold and returns (number of files whose size is below threshold) / (total number of files).
I think getUncompactedRatio is fine, I will rename like that.

zratkai · 2024-11-14T12:53:18Z

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java

+        CloseableIterable.filter(fileScanTasks, t -> {
+          DataFile file = t.asFileScanTask().file();
+          return (!table.spec().isPartitioned() ||
+              partitionPath == null && file.specId() != table.spec().specId() ||


Please use better separation between operands to make it readable/understandable and reliable, since now the precedence can cause issues.

Like: ((a==b && c!=2) && (a!=b && c=2) && (...) )

Previously I used braces in very similar case, and @deniskuzZ asked to remove them: #5389 (comment)

The precedence won't cause issues because in Java the && operand has higher precedence over ||, so it will be evaluated deterministically.

zratkai · 2024-11-14T13:00:18Z

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java

+    CloseableIterable<ScanTask> filteredDeletesScanTasks =
+        CloseableIterable.filter(deletesScanTasks, t -> {
+          DeleteFile file = ((PositionDeletesScanTask) t).file();
+          return !table.spec().isPartitioned() ||


Please use better separation between operands to make it readable/understandable and reliable, since now the precedence can cause issues.

Like: ((a==b && c!=2) && (a!=b && c=2) && (...) )

Similar to previous comment, previously I used braces in very similar case, and @deniskuzZ asked to remove them: #5389 (comment)

The precedence won't cause issues because in Java the && operand has higher precedence over ||, so it will be evaluated deterministically.

Highly disagree. Please check this:

boolean x = false; boolean y = true; boolean z = true; if(x && y ||z){ System.out.println("true"); }else{ System.out.println("false"); } if(x && (y ||z)){ System.out.println("true"); }else{ System.out.println("false"); }

The result for the first is true, for the second it is false.

There seems to be no contradiction, your test confirms that && has higher precedence than || as it gets evaluated first.

First case:

x && y || z false && true || true false || true true

Second case:

x && (y || z) false && (true || true) false && (true) false

zratkai · 2024-11-14T13:29:19Z

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java

@@ -458,6 +458,52 @@ public static List<DeleteFile> getDeleteFiles(Table table, String partitionPath)
        t -> ((PositionDeletesScanTask) t).file()));
  }

+  public static float getFileSizeRatio(Table table, String partitionPath, long fileSizeInBytesThreshold) {
+    long uncompactedFilesCount = getDataFileCount(table, partitionPath, fileSizeInBytesThreshold, true);
+    long compactedFilesCount = getDataFileCount(table, partitionPath, fileSizeInBytesThreshold, false);


Why the fileSizeInBytesThreshold is the comparator if a file is uncompacted or compacted?
Isn't there a better approach to make this check?

A file is decided to be compacted or uncompacted based on comparing its actual size with the threshold defined in Conf depending on compaction type:

HIVE_ICEBERG_MAJOR_COMPACTION_FILE_SIZE_THRESHOLD("hive.iceberg.major.compaction.file.size.threshold", "96mb", new SizeValidator(), "Iceberg data file size in megabytes below which a file needs to be compacted."), HIVE_ICEBERG_MINOR_COMPACTION_FILE_SIZE_THRESHOLD("hive.iceberg.minor.compaction.file.size.threshold", "16mb", new SizeValidator(), "Iceberg data file size in megabytes below which a file needs to be compacted."),

fileSizeInBytesThreshold gets the value from these configs depending on compaction type.

zratkai · 2024-11-14T13:34:18Z

...erg-handler/src/test/queries/positive/iceberg_major_compaction_partition_evolution_ordered.q

 ('fn11','ln11', 3, 20, 101),
 ('fn12','ln12', 3, 20, 101);

-select * from ice_orc where company_id = 100;
-select * from ice_orc where dept_id = 2;


Why did you delete those selects?

Because with the other changes done to this q file in this PR the order of the result records from these selects is not fixed and changes randomly in different executions of the test.

how about adding -- SORT_QUERY_RESULTS ?

zratkai · 2024-11-14T13:38:59Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java

+      if (table.getStorageHandler() != null && 
+          !table.getStorageHandler().canCompact(context.getConf(), table, compactionRequest.getPartitionname(), 
+              compactionRequest.getType())) {
+        LOG.info("Table=%s%s doesn't meet requirements for compaction", table.getTableName(), 


Please add more description what requirement didn't met.
e.g.: What was expected, what is the actual?

I removed logging from this place and instead added it to HiveIcebergStorageHandler#canCompact.
This way it will log the exact reason why a table/partition doesn't meet compaction requirements.

zratkai

Please check my comments!

sonarcloud · 2024-11-15T01:02:09Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions bot requested a review from miklosgergely November 5, 2024 03:03

asf-ci-hive added tests pending tests failed and removed tests pending labels Nov 5, 2024

difin force-pushed the can_compact branch from ccc34bb to bbefd69 Compare November 5, 2024 22:28

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Nov 5, 2024

difin force-pushed the can_compact branch from bbefd69 to f8bd1d0 Compare November 6, 2024 04:18

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Nov 6, 2024

difin force-pushed the can_compact branch from f8bd1d0 to 485cb6f Compare November 6, 2024 15:35

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending labels Nov 6, 2024

difin force-pushed the can_compact branch from 485cb6f to 85d18c4 Compare November 7, 2024 19:21

asf-ci-hive added tests pending and removed tests unstable labels Nov 7, 2024

difin marked this pull request as ready for review November 7, 2024 19:22

asf-ci-hive added tests unstable and removed tests pending labels Nov 7, 2024

HIVE-28600: Iceberg: Check that table/partition requires compaction b…

7b3a4be

…efore compacting

difin force-pushed the can_compact branch from 85d18c4 to 7b3a4be Compare November 8, 2024 16:24

asf-ci-hive added tests pending and removed tests unstable tests pending labels Nov 8, 2024

asf-ci-hive added the tests passed label Nov 8, 2024

zratkai reviewed Nov 14, 2024

View reviewed changes

zratkai suggested changes Nov 14, 2024

View reviewed changes

review comment Nov 14.

8f4e3db

asf-ci-hive added tests pending and removed tests passed labels Nov 14, 2024

asf-ci-hive added tests passed and removed tests pending labels Nov 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIVE-28600: Iceberg: Check that table/partition requires compaction b… #5529

HIVE-28600: Iceberg: Check that table/partition requires compaction b… #5529

difin commented Nov 5, 2024 •

edited

Loading

zratkai Nov 14, 2024

difin Nov 14, 2024 •

edited

Loading

zratkai Nov 14, 2024

difin Nov 14, 2024 •

edited

Loading

zratkai Nov 14, 2024

difin Nov 14, 2024 •

edited

Loading

zratkai Nov 14, 2024

difin Nov 14, 2024 •

edited

Loading

zratkai Nov 14, 2024

difin Nov 14, 2024 •

edited

Loading

zratkai Nov 14, 2024

difin Nov 14, 2024 •

edited

Loading

kasakrisz Nov 15, 2024

zratkai Nov 14, 2024

difin Nov 14, 2024

zratkai left a comment

sonarcloud bot commented Nov 15, 2024

HIVE-28600: Iceberg: Check that table/partition requires compaction b… #5529

Are you sure you want to change the base?

HIVE-28600: Iceberg: Check that table/partition requires compaction b… #5529

Conversation

difin commented Nov 5, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

Is the change a dependency upgrade?

How was this patch tested?

Choose a reason for hiding this comment

difin Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

difin Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

difin Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

difin Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

difin Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

difin Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zratkai left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Nov 15, 2024

Quality Gate passed

difin commented Nov 5, 2024 •

edited

Loading

difin Nov 14, 2024 •

edited

Loading

difin Nov 14, 2024 •

edited

Loading

difin Nov 14, 2024 •

edited

Loading

difin Nov 14, 2024 •

edited

Loading

difin Nov 14, 2024 •

edited

Loading

difin Nov 14, 2024 •

edited

Loading