Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPU-aware pinned-inner-buffer #802

Closed
kaigai opened this issue Jul 4, 2024 · 2 comments
Closed

GPU-aware pinned-inner-buffer #802

kaigai opened this issue Jul 4, 2024 · 2 comments
Labels
feature developer considers it is the feature to be supported on the future version
Milestone

Comments

@kaigai
Copy link
Contributor

kaigai commented Jul 4, 2024

Pinned-inner-bufferがGPUメモリのサイズを越える場合、Hash-Joinは猛烈なランダムアクセスなので、
強烈なスラッシングが発生する。
通常、GPUメモリを越えるpinned-inner-bufferを使用すべきではない。

しかしマルチGPUとパーティションを併用する場合、パーティションキーがJOINの結合キーに含まれると、
そのヒントを利用してデータ配置の局所性を高める事ができる(はず)。

  • Inner-Bufferを作成する際に、パーティションキーを使ってどのパーティションに(物理GPUに)割り振られるのかを判断する kern_expression が必要。(CASE ... WHEN... で作れるか)
    • たぶん table_oid が使いやすいかも。
  • Inner-Bufferの使い方...kds_extraを使うのがよいか。要検討。

関連issue ... #801

@kaigai kaigai added the feature developer considers it is the feature to be supported on the future version label Jul 4, 2024
@kaigai kaigai added this to the v5.3 milestone Jul 4, 2024
@kaigai
Copy link
Contributor Author

kaigai commented Jul 10, 2024

こんな感じの実行計画になるようにすればよい。

[BEFORE]
GpuJoin (JOIN-Qual: h.huge_id = l.large_id)
  +- outer: huge_table h  (2TB)
  +- inner: large_table l (300GB)

[AFTER]
Append
 +- GpuJoin (JOIN-Qual: h.huge_id = l.large_id)
 |   +- outer: huge_table h (2TB) WHERE hash(huge_id) % 3 = 0
 |   +- inner: large_table l (300GB) WHERE hash(large_id) % 3 = 0
 |
 +- GpuJoin (JOIN-Qual: h.huge_id = l.large_id)
 |   +- outer: huge_table h (2TB) WHERE hash(huge_id) % 3 = 1
 |   +- inner: large_table l (300GB) WHERE hash(large_id) % 3 = 1
 |
 +- GpuJoin (JOIN-Qual: h.huge_id = l.large_id)
     +- outer: huge_table h (2TB) WHERE hash(huge_id) % 3 = 2
     +- inner: large_table l (300GB) WHERE hash(large_id) % 3 = 2

image

@kaigai
Copy link
Contributor Author

kaigai commented Oct 16, 2024

2932a693bde27b698967bfebb311773a25ad353a で一旦終結。

@kaigai kaigai closed this as completed Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature developer considers it is the feature to be supported on the future version
Projects
None yet
Development

No branches or pull requests

1 participant