[API Proposal]: New attribute for interop-specific struct concerns #100896
Comments
jkoritzinsky
added
api-suggestion
|
Tagging subscribers to this area: @dotnet/interop-contrib |
dotnet-policy-service
bot
added
the
untriaged
I would expect some Roslyn integration to be a baseline requirement. The compiler already special cases If we are going to have this work as pay for play where we only go look at the extended attributes if a particular At that point, I think it's worth just minimally integrating it like we did for extended Theoretically we could just have something like
I'm not a huge fan of this name and don't think it's necessarily obvious what it means. Could we define it as |
I don't think that is a requirement. We made the transition to
This is something we should consider. I like following the CallConv approach we've developed.
It follows the same pattern as
These are not appropriate in this case since the values here will be reflected in both the managed type layout itself and not just at the interop boundary. We did consider
I would prefer to avoid C# concepts in this definition, especially as it relates to types - |
|
There is an unused value in
There are number of other issues with generics support in Swift interop. Unless we have a solution for how to make generics work well with Swift interop end-to-end, solving the struct layout issue is not that interesting.
I think it is a questionable property. As far as I know, there is no equivalent in C/C++. Also, would it make more sense to allow specifying alignment, to better match how one can control layout in C/C++?
Is this all that is required to express layout of all possible Swift structs? Is there a prior art for expressing the Swift layout struct rules in clang/C++? (I am not sure whether I like inventing names like this for language-specific layout algorithms.)
Why would we want to allow FieldOffset with the new thing? FieldOffset is problematic concept (except when used with offset 0) - I believe that you made this point earlier.
The CallConv... types exist to allow encoding the calling convention in function pointers. It is not possible to use attributes for function pointers. I think attribute works just fine for encoding type layout. |
Nit: I agree that LibraryImport does not have deep integration with Roslyn. However, we have made changes in Roslyn to make it work well (e.g. relaxed rules for
I would expect this to be useful for controlling struct layout in general, not limited to structs imported from other languages. My name choice would be |
I fully agree, that is why |
|
Maybe |
I think this name is pretty good. I think avoiding
I think it would be great if we could specify an alignment. It would be very useful.
What happens if I do this: SwiftOptionalLike<int>[] array = new SwiftOptionalLike<int>[5];
Span<SwiftOptionalLike<int>> span = array;
span[1].Value = 5; //is it misaligned? (and do we care or not?) or is sizeof is wrong and cannot be used anymore for this? or do we just disallow this entirely somehow?
I'd tend to agree that allowing FieldOffset it probably not a good idea, unless we have a good reason to combine explicit layout with the additional features this new system may provide. Here's 2 useful layouts I'd like to see that we could potentially make with this feature:
It would be good if we also kept sequential & auto with this feature so they could benefit from new options like alignment. |
I will note that the runtime doesn't support alignments larger than 8 bytes (for on-GC-heap objects). |
Indeed, that part, if approved, would presumably just be delayed until the runtime could support it (since I assume there's no fundamental reason why it couldn't work).
Also, this could presumably(?) break some of use who have struct AlignHelper<T>
{
T value;
byte field;
}
static unsafe int AlignOf<T>() => sizeof(AlignHelper<T>) - sizeof(T);
Console.WriteLine(AlignOf<SwiftOptionalLike<int>>()); //would this produce what we expect for alignment?
//it relates to the question I asked about spans earlierIf it would break the above, it would be great to get proper |
I thought about this when Jeremy first suggested the idea - "Would we ever want this for reference types?". I don't think I have the imgination to come up with a scenario where we would want to play layout games with reference types. Not pushing back on the suggestion, but is there an obvious case where it would be compelling? I personally find the
I have no push back with that name. The |
|
FWIW, the Swift struct layout algorithm is described here: https://github.com/apple/swift/blob/4b440a1d80a0900b6121b6e4a15fff2a96263bc5/docs/ABI/TypeLayout.rst#fragile-struct-and-tuple-layout I'm not sure that |
My understanding of the proposal (based on the wording and the shape of the new attribute) was that this was basically meant to supplement the existing My assumption had then been that this meant that something like a union in Swift would be defined something like: I had also assumed we were going to want this to be "pay for play" so that the VM doesn't have to look for this attribute on every struct, and the only sensible way to do that is to use one of the free bits in the existing I do think that If we were to design this from the ground up, then what I would expect to see is probably:
I would then expect us to take a look at what metadata is actually expressible by languages (both officially in the language spec and unofficially via documented compiler switches/features) we want to interoperate with and make a determination on what to expose based on that. I would then not want to reuse the existing
I would want us to be more explicit about the differences between I think we would then need to consider the best way to represent concepts like compatibility with a particular language. My biggest concern with something like |
I've spent some time looking into more specifics of Swift layout (primarily around enums and the discriminator). Swift uses a concept of "spare bits" based on the type and platform to reuse bits in types (thankfully only in the non-generic case). We could have the Swift projection handle this at the projection layer (with explicit layout with specifically placed Alternatively, we could add runtime support for this concept and add more intrinsic Swift struct types to represent the different categories of types (in particular, pointers to Swift objects and ObjC objects bridged into Swift) along with a mechanism to say how many spare bits to reserve (padding the struct length with additional bytes if necessary). I do like @tannergooding's proposal of specifying a specific target language and letting the runtime handle it, but I'm concerned that the cost of implementing more complicated concepts like the spare-bits concept would be too expensive (especially if we won't hit the limitations in the .NET 9 targets for Swift interop). Based on the reactions to the |
|
Here's my first pass at an API based on Tanner's ideas (and handling spare bits in the VM/layout). The general idea is to encode the information in the attribute by using different "*LayoutKind" enums for different target languages. I've also included the extra requirements for Swift. I've decided to not include a type for the ObjC bridged object as we're trying to avoid handling that case in the Swift interop the .NET 9 timeframe. namespace System.Runtime.InteropServices
{
public enum LayoutKind
{
Custom
}
[AttributeUsage(AttributeTargets.Struct)]
public sealed class CustomLayoutAttribute : Attribute
{
public CustomLayoutAttribute(CustomLayoutKind kind) {}
public CustomLayoutAttribute(CLayoutKind kind) {}
public CustomLayoutAttribute(Swift.SwiftLayoutKind kind, int requiredSpareBits = 0) {}
}
public enum CustomLayoutKind
{
Sequential,
Transparent
}
public enum CLayoutKind
{
Struct,
Union
}
}
namespace System.Runtime.InteropServices.Swift
{
public enum SwiftLayoutKind
{
Struct,
Enum
}
public struct SpareBits
{
public static SpareBits GetSpareBits<T>(T value) where T: unmanaged;
public static T SetSpareBits<T>(T value, SpareBits spareBits) where T: unmanaged;
// Returns the value of the spare bits as an unsigned 64-bit integer.
public ulong AsUInt64();
// Sets the value of the spare bits
public void Set(ulong bits);
}
// Represents a pointer to a Swift object (for the purposes of calculating spare bits)
public readonly unsafe struct SwiftObject
{
public SwiftObject(void* value)
{
Value = value;
}
public void* Value { get; }
}
} |
|
I do not understand the SpareBits. I assumed that the Swift-specific layout computation would take care of the spare bit allocation transparently. Could you please shed some more light on it? Can the I have mixed feeling about transparent. As I have said before, it just pushes work that can be done by interop binding generators into the runtime. |
What sort of a union is this? Is it an order dependent one or an order independent one? See this (which I linked before, but nobody seemed to look at) where I go over at an overview level what they're both useful for, and how they could be specifically represented in metadata. i.e., what do we expect the answer to the following question to be? [CustomLayout(CustomLayoutKind.Union)]
struct Union1<T1, T2>
{
T1 field1;
T2 field2;
}Is the layout of |
I don't think this is the right approach, I think there should be 1 constructor that takes a |
It is C-like union. It would work exactly same as |
The SpareBits type would provide a way to get the value stored in the spare bits of a value type to enable a Swift projection to know which element of the enum is active. It looks like the Swift compiler puts an entry into an enum type's value witness table to find the tag, so the projection can use that and not need to read them manually. I'll remove the type. https://godbolt.org/z/KMnaoTePd
I was mainly using We could use one joint enum and only read the members necessary. I was trying to use separate constructors to make it not possible to specify information that's not applicable to the target layout, but I'm not tied to the idea. Here's an updated proposal: namespace System.Runtime.InteropServices
{
public enum LayoutKind
{
Custom
}
[AttributeUsage(AttributeTargets.Struct)]
public sealed class CustomLayoutAttribute : Attribute
{
public CustomLayoutAttribute(CustomLayoutKind kind) {}
// Only valid for CustomLayoutKind.SwiftEnum
public int RequiredDiscriminatorBits { get; set; }
}
public enum CustomLayoutKind
{
Sequential, // C-style struct
Union, // C-style union
SwiftStruct, // Swift struct
SwiftEnum // Swift enumeration
}
}
namespace System.Runtime.InteropServices.Swift
{
// Represents a pointer to a Swift object (for the purposes of calculating spare bits)
public readonly unsafe struct SwiftObject
{
// Would be implemented to mask out the spare bits
// or assign while preserving spare bits
public void* Value { get; set; }
}
// Represents a bool value in Swift (for the purposes of calculating spare bits)
public readonly unsafe struct SwiftBool
{
// Would be implemented to mask out the spare bits
// or assign while preserving spare bits
public bool Value { get; }
}
} |
A sufficiently smart tool can try to get it all right, but its very error prone and prone to breaking if a new platform comes online. The general issue is that it it starts getting into concepts that are ABI specific. That is, whether or not A simple example is that people worked around the well known Windows member call difference for x64 for the longest time where a C signature that looked like A built-in layout like |
Could you please share a few examples of Swift structs and enums and what their C# equivalents would be to using these constructs? |
It is only error prone and non-portable if people are cutting corners. It is not error prone if the types are matched between managed and unmanaged signatures exactly. If the unmanaged signature has The design principle that we have established for runtime interop going forward has been to only introduce low-level features that are impossible or very hard to do in higher level bindings. Wrapping primitive types with structs is boiler plate code that is very straightforward to do in higher level bindings. One can come up with number of similar features that require quite a bit of boiler place code in interop bindings today, but that can be implemented by the runtime instead. For example, we can allow byref types in signatures and pin them implicitly in the JIT. It would save a good amount of boiler plate-code in interop bindings too. If we start introducing these types of features, where should we stop? Are we going to end up with a complicated built-in interop v2?
Right, we have introduced |
Here's some examples: @frozen
public struct InnerStruct
{
let F0: Int16;
let F1: Int8;
}
@frozen
public struct OuterStruct
{
let F0: UInt64;
let F1: Int64:
let F2 : InnerStruct;
let F3: Int8;
}
@frozen
public enum MyOptional<T>
{
case Empty;
case Some(Value: T);
}
public class MyClass
{
}
@frozen
public enum ParsedResult
{
case ParsedObject(MyClass);
case ParsedBool(Bool);
}[StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftStruct)]
public struct InnerStruct
{
public short F0; // offset 0
public sbyte F1; // offset 2
}
[StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftStruct)]
public struct OuterStruct
{
public ulong F0; // offset 0
public long F1; // offset 8
public InnerStruct F2; // offset 16
public sbyte F3; // offset 19
}
[StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftEnum, RequiredDescriminatorBits = 1)]
public struct MyOptional<T> where T : unmanaged
{
[StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftStruct)]
private struct Some_Payload
{
public T Value;
}
private Some_Payload Some;
// An API surface to describe the different cases and provide a C# API around accessing them, out of the scope of this API proposal
}
[StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftEnum, RequiredDescriminatorBits = 1)]
public struct ParsedResult
{
[StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftStruct)]
private struct ParsedObject_Payload
{
public SwiftPointer payload_0;
}
[StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftStruct)]
private struct ParsedBool_Payload
{
public SwiftBool payload_0;
}
private ParsedObject_Payload ParsedObject;
private ParsedBool_Payload SwiftBool;
// An API surface to describe the different cases and provide a C# API around accessing them, out of the scope of this API proposal
} |
No, the masking can be implemented in SwiftBool itself.
Swift's Bool type always takes up at least 1 byte. So the size of MyStruct is 2 bytes. There's no extra restrictions on its usage. Swift just allows the enumeration layout algorithm to reuse the bits that the Bool type definitely doesn't use. To do this, Swift allows types in the standard library to use LLVM's custom-bit-sized integer types. These types are always allocated as the next legal integer size (ie the |
|
I'm still confused what we expect [StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftStruct)]
public struct OuterStruct
{
public ulong F0; // offset 0
public long F1; // offset 8
public InnerStruct F2; // offset 16
public sbyte F3; // offset 19
}Based on the above, it seems like sizeof InnerStruct should be 3, but if we had [StructLayout(LayoutKind.Custom)]
[CustomLayout(CustomLayoutKind.SwiftStruct)]
public struct OuterStruct2
{
public InnerStruct F0; // offset 0
public ushort F1; // offset 4
}It seems like the size should be 4. How do we get these different values? I'd think if we're copying an arbitrary Maybe something like //Unsafe (& RuntimeHelpers overloads for RTH?)
static int SizeOf<T>(int nextAlignment);
static int AlignOf<T>();
//copy size
Unsafe.SizeOf<InnerStruct>(1); //3
//offset to next in span
Unsafe.SizeOf<InnerStruct>(Unsafe.AlignOf<InnerStruct>()); //4
//offset to next byte field
Unsafe.SizeOf<InnerStruct>(Unsafe.AlignOf<byte>()); //3
//offset to next short field
Unsafe.SizeOf<InnerStruct>(Unsafe.AlignOf<short>()); //4? Either way we set |
Ok, I have incorrectly assumed that the bit packing works for structs too. It sounds like that it works for Swift enums only and only when there is a single bool value. Just curious - is there a reason why this specific case is optimized? Are enums like this very common in Swift APIs?
So we are going to always return the lowest bit |
|
I strongly disagree with having language-specific custom layouts as a runtime concept. I think this a great UX for a source generator, but the underlying mechanism should be language agnostic (to whatever extent possible - if a language, like Swift, requires an ABI different from the normal platform ABI, that has to bleed into the runtime). Having language-specific custom layouts as a runtime concept will lead to:
Also I'm not sure that language-specific runtime mechanisms are a great idea because languages evolve. Generally they're not breaking their ABI on every major version, but it's conceivable that some Rust edition or new version of Swift breaks compat with a previous one and now we're stuck with an enum value that is ambiguous or unusable. |
|
@jkoritzinsky, After thinking about this more, I'm actually not quite sure I 100% understand the reason behind In general, ABIs are determined roughly in terms of the underlying system ABI, which is itself largely oriented around C. Accordingly, if a type or method cannot be defined using "standard" C, then it is not possible for C to call and therefore not really possible for arbitrary interop -- Where "standard" C is a little bit looser term than "spec-compliant" C and really just means, definable by GCC/Clang/MSVC Based on some of the above, it sounds like we're trying to circumvent the core interop APIs for Swift and trying to interact with it directly, rather than going through the C compatible ABI defined for the language and so its kind-of like if some C API tried to call into a .NET generic function directly. Yes you can do it, but its technically depending on implementation details that are subject to change and is largely undefined behavior. If C wants to call a .NET generic API, then it needs to use the appropriate hooks to resolve an ABI stable wrapper instead. It'd be great if this general scenario could be clarified and why we need such a feature but C does not. |
This works for enums with Bool members, Swift class object members, or Objective-C bridged object members.
There used to be more cases in Swift that were optimized in this way. The
Enums with Bools aren't particularly common, but ones with class types (represented by the
Yep
We plan to do most work a source generator/projection space, but the ABI platform differences that must be accounted for in the calling convention must be representable to the runtime, as the runtime handles the register allocation, lowering, etc. Basic type layout is included here. We tried to use the existing .NET features to describe Swift layouts, but we've realized that we can't do so with existing features. We tried to use We also can't represent the "spare bits" concept in a platform-agnostic way, as even macOS x64 and arm64 differ in which bits they consider "spare". I'd love to make these cases a source-generator-supported concept, but sadly they fall below the line of things the runtime is better at (architecture-specific differences) and things the runtime needs to handle (type layout for blittability and calling convention
We will always have users using the incorrect representation of our layout APIs that are "close enough". The majority of usage of explicit layout is done incorrectly (it's very rare for people outside of dotnet/runtime or the C# discord to represent structs containing unions as such instead of putting the explicit layout on the containing struct). I don't think we can stop users from doing this, and by not adding the features necessary to represent this, we make cases like the "represent a Swift type" case even more convoluted and difficult to understand, (which is why there was pushback on the
If we're concerned, we can name the Swift members
As you know, we already have mechanisms to represent types that don't exist in C for interop scenarios (explicit-layout w/ non-zero offsets).
For Swift interop, we are trying to introduce support to directly call into Swift APIs. That is the explicit goal of the project. In .NET 8 and earlier, Swift interop requires a significant amount of codegen in both C# and Swift to provide a C-compatible API surface on each side. Our initiative is to expand what .NET supports to enable directly calling Swift APIs with the Swift calling convention with Swift types. An explicit goal of this work is to make it possible to call Swift APIs without having to map a Swift API to a C-compatible API. We're not circumventing the core interop APIs, we're explicitly expanding what we support to include the Swift ABI on Apple platforms.
Swift's layout rules are explicitly not C-compatible. For example, the lack of trailing padding is not expressible in C, only in LLVM IR. Swift has explicitly stabilized portions of their ABI in Swift 5. We're only proposing including support in .NET for these stable portions of the ABI that are required to accurately call exposed Swift functions in the APIs we're looking at projecting into .NET. |
👍, if it's explicitly defined as stable then I think that alleviates most of the concerns I had. |
|
@lambdageek and I spoke offline and he's okay with the updated proposal given some of the mentioned concerns. I'll update the top and mark this as ready for review (and blocking). |
jkoritzinsky
added
blocking
jkoritzinsky
removed
the
untriaged
|
I think we should have good understanding of the Swift generics and Swift UI solution end-to-end before we start implementing the Swift-specific field layout in the runtime. I see it as nice-to-have at this point. I am not 100% convinced that it will be required at the end. |
|
We'll need to either support I agree that we should have a good understanding of the layout algorithms and which portions we need for our .NET 9 goals before implementing it. |
I think this would be sufficient for our .NET 9 Swift interop goals. |
|
(I am fine with doing prep-work towards this proposal, like running it through API review and removing the superfluous validation of LayoutKind in Roslyn.) |
namespace System.Runtime.InteropServices
{
public enum LayoutKind
{
Extended = 1,
}
[AttributeUsage(AttributeTargets.Struct)]
public sealed class ExtendedLayoutAttribute : Attribute
{
public ExtendedLayoutAttribute(ExtendedLayoutKind kind) {}
// Only valid for ExtendedLayoutKind.SwiftEnum
public int RequiredDiscriminatorBits { get; set; }
}
public enum ExtendedLayoutKind
{
CStruct, // C-style struct
CUnion, // C-style union
SwiftStruct, // Swift struct
SwiftEnum, // Swift enumeration
}
}
namespace System.Runtime.InteropServices.Swift
{
// Represents a pointer to a Swift object (for the purposes of calculating spare bits)
public readonly unsafe struct SwiftObject
{
// Would be implemented to mask out the spare bits
public void* Value { get; }
}
// Represents a bool value in Swift (for the purposes of calculating spare bits)
public readonly unsafe struct SwiftBool
{
// Would be implemented to mask out the spare bits
// or assign while preserving spare bits
public bool Value { get; }
}
} |
bartonjs
added
api-approved
This adds support to allow SwiftSelf<T> with a frozen struct as T. Swift allows enregistration of 'self' in these cases, but the 'self' must still be passed in the dedicated context register when the frozen struct is not enregistered, which makes this support necessary to handle as part of the calling convention. A few notes: - If `T` is not a value class we `BADCODE` - If the signature includes `SwiftSelf<T>`, then it must be the first argument of the signature, which matches how 'self' gets enregistered on the Swift side, otherwise we `BADCODE` - There is not support for reverse pinvokes for `SwiftSelf<T>`. That's because the context passed to function pointers is always a pointer, so I do not see any use case for this in reverse pinvokes. - Some care must be taken since `SwiftSelf<T>` is a generic struct whose layout generally is not going to match `T` without tail padding (until we get something like dotnet#100896).
This adds support to allow SwiftSelf<T> with a frozen struct as T. Swift allows enregistration of 'self' in these cases, but the 'self' must still be passed in the dedicated context register when the frozen struct is not enregistered, which makes this support necessary to handle as part of the calling convention. A few notes: - If `T` is not a value class we `BADCODE` - If the signature includes `SwiftSelf<T>`, then it must be the first argument of the signature, which matches how 'self' gets enregistered on the Swift side, otherwise we `BADCODE` - There is not support for reverse pinvokes for `SwiftSelf<T>`. That's because the context passed to function pointers is always a pointer, so I do not see any use case for this in reverse pinvokes. - Some care must be taken since `SwiftSelf<T>` is a generic struct whose layout generally is not going to match `T` without tail padding (until we get something like #100896).
Background and motivation
.NET's struct layout system is quite extensive, but there are still some cases it does not cover that are relevant in interop scenarios at the language/runtime boundary.
In particular, we've seen requests for features like
repr(transparent)With Swift Interop, we have a need to represent generic structs with correct Swift layouts, but we don't have a way to represent the Swift layout accurately in the general case. In the non-general case, we can represent it with an explicit
StructLayout.Sizeelement, but we can't do that for generics.For all of these cases, we'd like to extend the
StructLayoutAttributeto handle these cases. However, we can't extend that attribute as it's a pseudo-attribute that maps directly to metadata. Instead, we propose adding a new attribute, only usable on struct types, that's intended to encompass all future layout features, as well as the existing layout features provided byStructLayoutAttribute.As part of the implementation, we plan to introduce a simple source-generator to generate the corresponding
StructLayoutAttributeon the type that has our new attribute applied. To ensure that we don't accidentally make types non-blittable, we'll lower the attribute such thatCharSet = CharSet.Unicodein all cases.In the future, we may extend this attribute to also be the "trigger" attribute for an interop source generator to generate a marshaller for a given struct type.
To limit the scope of this feature, we plan to start with only the feature required by Swift interop:
Like the
LayoutKind.Sequentialsupport for structs in the runtime, we won't support these new layout requirements for any structs that (recursively) contain reference type fields.API Proposal
API Usage
Alternative Designs
We could provide a set of layout primitives instead of a set of well-known layouts (the original proposal). However, we'd then have to handle all possible combinations of these primitives or block them to only the valid combinations to match our equivalent support in the current proposal.
We could go further than the original proposal and have a mechanism for specifying OS/Arch-specific layout and ABI parameter passing rules in attributes. This design would allow the runtime to be entirely out of the business of layout and ABI handling other than reading the attributes. This design has a few problems through: We'd need to consider how to provide/validate the provided options. The code-gen backends would need to respect this information (and reading from custom attributes is expensive). Generics would also be a problem in this design space.
Original API Proposal
API Proposal
API Usage
Alternative Designs
We could provide a dedicated attribute for the Swift scenario.
We could have Roslyn support lowering the StructLayoutAttribute-corresponding members to metadata instead of introducing a source generator.
We could skip including the StructLayoutAttribute APIs and have the attributes represent separate concepts.
We could add new members to StructLayoutAttribute and require all compilers to recognize when new members are specified and not remove the attribute (and still lower the original members to metadata). This option is very expensive.
Risks
No response
The text was updated successfully, but these errors were encountered: