Integrating ucx into ceph #8883
Replies: 6 comments 4 replies
-
|
The first version was old and could not be compiled pass. The second version, the ceph test program can work properly, but the multi-thread operation is abnormal, and it use polling mode which consumes high power. Therefore, the interrupt mode needs to be implemented. |
Beta Was this translation helpful? Give feedback.
-
|
@yosefe Do you want to do this together? |
Beta Was this translation helpful? Give feedback.
-
|
Based on https://github.com/Mellanox/ceph/tree/vasily-ucx/src/msg/async/ucx,i fixed the compilation errors，ceph ucx can work. The performance is 33% worse than that of tcp. The ceph cluster can communicate normally for several minutes and then exits with errors. |
Beta Was this translation helpful? Give feedback.
-
|
@alex-mikheev Can we work together to perfect ceph ucx? |
Beta Was this translation helpful? Give feedback.
-
|
@fengchunsong - during the work on adding ucx transport to ceph (many moons ago) we came to conclusion that ceph arch (above transport) is not built for high-speed transports (there are many SW layers above transport that are heavy weight). we discussed with ceph architects and got ack on this observation. did ceph arch change to that matter? |
Beta Was this translation helpful? Give feedback.
-
|
The ceph architecture remains unchanged, but back-end storage is upgraded from HDD to SSD, which is expected to perform better performance at the faster network transport layer. My test found that ceph, ucx completeness is about 90%, improving the handling of link exceptions. Ucx merged into the ceph mainline no problem. The new ceph architecture is based on the seastar, we can also implement ucx on the seastar. |
Beta Was this translation helpful? Give feedback.
-
|
Currently, there's no RDMA solution in seastar, right? |
Beta Was this translation helpful? Give feedback.
-
right |
Beta Was this translation helpful? Give feedback.
-
|
Ceph ucx can work properly using rdma + ud and tcp. However, when rdma + rc is used, the server will be disconnected soon after being connected. |
Beta Was this translation helpful? Give feedback.
-
|
Ucx experts, can you help analyze the logs? How to locate the problem next? [1677137866.036607] [client7:1415319:2] ib_iface.c:755 UCX DEBUG iface 0x556e540aa000: ah_attr dlid=49152 sl=0 port=1 src_path_bits=0 dgid=::ffff:192.168.37.197 sgid_index=5 traffic_class=106 |
Beta Was this translation helpful? Give feedback.
-
The RDMA of Ceph does not support RDMA_READ, XRC, and DC. If RDMA is used, large blocks of memory are reserved. There have been two previous attempts to integrate ucx into ceph, one at https://github.com/Mellanox/ceph/tree/vasily-ucx/src/msg/async/ucx in 2018 and the most recent at ceph/ceph#41826 in 2021. None of them were integrated into the ceph main line.
Integrating UCX into CEPH achieves a win-win situation. UCX expands from HPC scenarios to distributed storage scenarios. CEPH network modules can also leverage the advantages of RDMA and release all-flash performance.
I am the maintainer responsible for the ceph dpdk module. I hope to work with the ucx and ceph community to promote the implementation of ucx on ceph.
Beta Was this translation helpful? Give feedback.
All reactions