BF16 Batched GEMM Functions Available?

[moderato]

Hi everyone, I'm benchmarking BF16 GEMM functions of OpenBLAS, and so far I only see a BF16 normal GEMM function which is cblas_sbgemm. I was wondering if there are any BF16 batched GEMM functions available?

[martin-frbg]

not implemented right now, though xianyi committed something labeled as a WIP for the four "conventional" data types a few years back (see #3039) that may only need testing and trivial adaption for sbgemm

[moderato]

Also cblas_xcopy for BF16 if that's not a huge burden...

[martin-frbg]

Unfortunately I need a bit more time - had not fully realized that xianyi's previous work on GEMM_BATCH was basically abandoned on an outdated branch.

[moderato]

Unfortunately I need a bit more time - had not fully realized that xianyi's previous work on GEMM_BATCH was basically abandoned on an outdated branch.

No worries just take your time and keep me posted.

[moderato]

@martin-frbg I saw cblas_sbgemm_batch in place. Thanks for upstreaming the changes! Looks like the output array is in float. Is it possible to provide another interface with bf16 output?

[moderato]

Also cblas_xcopy for BF16 if that's not a huge burden...

I notice cblas_sbcopy is missing... Would you mind adding this function?

[martin-frbg]

that would be bgemm then, I guess (based on what was discussed some time ago in #2767). first time someone expressed interest in that, I think

[moderato]

Thanks for the pointer. This is for some deep learning applications, so I was hoping there is such a kernel that can skip any extra data type conversion.

I searched "bgemm" online and didn't see much result. Seems like this is something that hasn't been broadly implemented in any BLAS libraries? And is it difficult to add it (as it might need naming approval from netlib based on the discussion, if I understand correctly)?

[moderato]

Hey Martin, just wanna quickly checkout if there is any plan to add "bgemm" support recently?

[martin-frbg]

I haven't forgotten, and still hope it should be easily derived from sbgemm, but I got completely derailed by the breakage from fixing NaN handling in scal ( which led to the milestone for 0.3.28 slipping)

[martin-frbg]

btw according to Intel documentation, there appears to be no dedicated instruction for doing the multiplication entirely in bf16, just the "multiply and store as packed float32" (vdpbf16ps) that the existing sbgemm kernels use - so bgemm would appear to be less efficient due to the need for an additional recoding (vcvtne2ps2bf16) of the float result to bf16
("Code Sample: Intel Deep Learning Boost New Deep Learning Instructions bfloat16 - Intrinsic Functions", document ID 659650)

[moderato]

@martin-frbg I understand that. I think there's still a need of bgemm as some kernels/ops (e.g. another bgemm, softmax, etc) that are ready to use have all bf16 inputs, so it also takes extra efforts/execution time to convert from fp32 to bf16 before calling those kernels/ops that follow a bgemm. I suppose the fp32->bf16 conversion in the intrinsic level right after the fp32 result being ready should be ok in terms of cost. What do you think?