enhancement: ability to process multiple FABs in a single GPU kernel

[BenWibking]

It would be potentially useful to be able to process several FABs at once in a single GPU kernel to reduce the overhead of kernel launch latency.

I see there is now the ability to launch a ParallelFor kernel to process all local FABs in a given MultiFab. It would be ideal to have something that packs together a tunable number of grids and then passes them to, e.g., a hydro kernel in an application. This approach has successfully been used in, e.g., Parthenon.

[WeiqunZhang]

There is something that might be what you want. Please take a look at ParallelFor in Src/Base/AMReX_TagParallelFor.H. An example of using it can be found in makeFineMask_doit in Src/Base/AMReX_MultiFabUtil.cpp.

[BenWibking]

Ah, yes, that looks exactly like what I'd want, except the host-device transfer of the Vector of Array4s is not needed. Is a version that avoids the host-device transfer something you could add, or should I instead create a ParallelFor customized for my code that avoids that?

[WeiqunZhang]

I am not sure how the host to device copy of the vector of Array4s can be avoided here. For the MultiFab version of ParallelFor, we copy it once and reuse it. So in your case, you might be able to cache it like the MultiFab version.

[BenWibking]

Ideally, I'd want an MFIter-like object that returns a Vector of FABs, then I could use this object to allocate work FABs, then call the TagParallelFor-like functions (possibly with, e.g., fabVector.arrays() or fabVector.const_arrays()).

If it's unlikely to be useful for other codes, I can add a custom version of MFIter to my code. If there's a way to do so that would make it less likely for the code to break when upgrading to newer AMReX versions, let me know.

[BenWibking]

Oh, I see, it's just copying the container, not the actual array data. I misunderstood.