Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BALANCER] Improve balancer-related code execution speed #1544

Merged

Conversation

garyparrot
Copy link
Collaborator

@garyparrot garyparrot commented Mar 6, 2023

Context: #1476 (comment), #1541

目前 Balancer 在大量 Partition 下 (約 10000 Partition) 會出現程式碼執行的效能議題,這個 PR 對 Balancer 相關的程式碼做一些實作上的修正,以增進執行速度。

主要的變更包含:

  • 重寫 ShuffleTweaker,現在重複利用一個 ClusterInfoBuilder 來建立最後的結果。
  • 使 ClusterInfo.Optimized#replicas() 系列函數使用 Lazy 的 grouping 結果,避免用 default 的實作(透過 replicaStream 來重新建議一樣的 list)
  • 移動 topic filter 邏輯到 AllocationTweaker

改進前

Total Run time: 30045 ms
Total ClusterCost Evaluation: 963
Average ClusterCost Processing: 9ms
Average MoveCost Processing: 0 ms
Initial Cost: 0.20265291955203096
Final Cost: 0.15089094792725546

30 秒進行 963 次嘗試

改進後

[重寫 AllocationTweaker]
Total Run time: 30023 ms
Total ClusterCost Evaluation: 1738
Average ClusterCost Processing: 9ms
Average MoveCost Processing: 0 ms
Initial Cost: 0.20283280146387214

[ClsuterInfo#replicas 使用快取]
Total Run time: 30013 ms
Total ClusterCost Evaluation: 2280
Average ClusterCost Processing: 6ms
Average MoveCost Processing: 0 ms
Initial Cost: 0.20259621231642164
Final Cost: 0.13722788084587437

[修改 TopicFilter]
Total Run time: 30021 ms
Total ClusterCost Evaluation: 3006
Average ClusterCost Processing: 6ms
Average MoveCost Processing: 0 ms
Initial Cost: 0.20289229678372744
Final Cost: 0.06640045294631854

30 秒進行 3006 次嘗試

@garyparrot garyparrot self-assigned this Mar 6, 2023
Copy link
Contributor

@chia7712 chia7712 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

整體看起來不錯,幾個小建議請看一下

Copy link
Contributor

@chia7712 chia7712 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@garyparrot 感謝更新,我覺得就工程面來說 balancer 有越來越簡潔,幾個建議請看一下

Copy link
Contributor

@chia7712 chia7712 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 做得很好

@garyparrot garyparrot merged commit 98e6ec0 into opensource4you:main Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants