Consider adding support for ArraySegment<byte>

[paulomorgado]

I recently came across the need to use pooled byte[]s that usually are larger than the data.

To overcome this, in inserting new entities and was able to use ArraySegment<byte> by extending SqlServerByteArrayTypeMapping:

internal class SqlServerArraySegmentOfByteTypeMapping : SqlServerByteArrayTypeMapping
{
    private const int MaxSize = 8000;
    private readonly SqlDbType? _sqlDbType;
    public SqlServerArraySegmentOfByteTypeMapping(
        string? storeType = null,
        int? size = null,
        bool fixedLength = false,
        ValueComparer? comparer = null,
        SqlDbType? sqlDbType = null,
        StoreTypePostfix? storeTypePostfix = null)
        : base(
            new RelationalTypeMappingParameters(
                new CoreTypeMappingParameters(typeof(ArraySegment<byte>), null, comparer, jsonValueReaderWriter: JsonArraySegmentOfByteReaderWriter.Instance),
                storeType ?? (fixedLength ? "binary" : "varbinary"),
                storeTypePostfix ?? StoreTypePostfix.Size,
                System.Data.DbType.Binary,
                size: size,
                fixedLength: fixedLength),
            sqlDbType)
    {
        _sqlDbType = sqlDbType;
    }

    /// <inheritdoc/>
    protected override void ConfigureParameter(DbParameter parameter)
    {
        var sqlParameter = (SqlParameter)parameter;

        if (_sqlDbType.HasValue) // To avoid crashing wrapping providers
        {
            sqlParameter.SqlDbType = _sqlDbType.Value;
        }

        var maxSpecificSize = CalculateSize(Size);

        if (parameter.Value == null
            || parameter.Value == DBNull.Value)
        {
            parameter.Size = maxSpecificSize;
        }
        else
        {
            var value = (ArraySegment<byte>)parameter.Value;
            var length = value.Count;

            if (length <= maxSpecificSize)
            {
                // Fixed-sized parameters get exact length to avoid padding/truncation.
                parameter.Size = IsFixedLength ? length : maxSpecificSize;
            }
            else if (length is <= MaxSize)
            {
                parameter.Size = IsFixedLength ? length : MaxSize;
            }
            else
            {
                sqlParameter.Value = value.Array;
                sqlParameter.Size = value.Count;
                sqlParameter.Offset = value.Offset;
            }
        }
    }

    private static int CalculateSize(int? size)
        => size is > 0 and < MaxSize ? size.Value : MaxSize;
}
internal class JsonArraySegmentOfByteReaderWriter : JsonValueReaderWriter<ArraySegment<byte>>
{
    public static JsonArraySegmentOfByteReaderWriter Instance { get; } = new();
    private JsonArraySegmentOfByteReaderWriter()
    {
    }
    public override ArraySegment<byte> FromJsonTyped(ref Utf8JsonReaderManager manager, object? existingObject = null)
        => manager.CurrentReader.GetBytesFromBase64();
    public override void ToJsonTyped(Utf8JsonWriter writer, ArraySegment<byte> value)
        => writer.WriteBase64StringValue(value);
}

And a custom SqlServerTypeMappingSource:

internal class CustomSqlServerTypeMappingSource : SqlServerTypeMappingSource
{
    private static readonly SqlServerArraySegmentOfByteTypeMapping _variableLengthMaxBinary
        = new("varbinary(max)", storeTypePostfix: StoreTypePostfix.None);

    public CustomSqlServerTypeMappingSource(
        TypeMappingSourceDependencies dependencies,
        RelationalTypeMappingSourceDependencies relationalDependencies)
        : base(dependencies, relationalDependencies)
    {
    }

    protected override RelationalTypeMapping? FindMapping(in RelationalTypeMappingInfo mappingInfo)
    {
        if (mappingInfo.ClrType == typeof(ArraySegment<byte>))
        {
            return _variableLengthMaxBinary;
        }

        return base.FindMapping(mappingInfo);
    }
}

Since I only read the ArraySegment<byte> column using raw sql, that's all I did.

[roji]

@paulomorgado yeah, I think this could make sense - though why do this with ArraySegment<byte> as opposed to via the more modern Memory<byte>?

Note that Npgsql supports both ArraySegment<byte> and Memory<byte>, but at the ADO.NET layer rather than via an EF value converter.

/cc @ajcvickers

[paulomorgado]

@roji, I thought of that, but ArraySegment<byte> has explicit Offset while that might not be possible to obtain from a Memory<byte>.

The user can try getting an ArraySegment<byte> from a Memory<byte> using TryGetMemoryManager<T,TManager>(ReadOnlyMemory<T>, TManager, Int32, Int32).

But, because that's not always possible, it shouldn't be up to EFCore.

Having both, would be a nice to have.

[roji]

because that's not always possible

As far as I know, the only cases where that's not possible are cases where the Memory doesn't wrap an array, but rather native/stack memory; in these cases it's also not possible to write the value with SqlClient (since an array is needed).

Are you aware of a different case?

[paulomorgado]

No. And in those corner cases, it's valid that the user needs to provide a memory instance that wraps an array.

But you'll still end up with an ArraySegment<byte>, because you can't get the offset out of a Memory<byte>/ReadOnlyMemory<byte> without getting it into an ArraySegment<byte>.

[roji]

That's an internal implementation concern of the value converter - the important question is what user-facing property types we support. Anyway, I'll discuss with the team.

[roji]

Design discussion result: we think this isn't something that belongs in EF as a value converter, but rather at the ADO.NET layer, where it can also be used by non-EF consumers; Npgsql supports writing both ArraySegment<byte> and ReadOnlyMemory<byte>, for instance.

[paulomorgado]

@roji, if the provider is Npgsql, can ArraySegment<byte> and ReadOnlyMemory<byte> be used as property types on entities?

[roji]

In principle that should be possible, but I gave it a try and I see some issues: ArraySegment<byte> is mapped to smallint[] instead of bytea, and ReadOnlyMemory<byte> isn't supported (everything should already work at the ADO.NET layer, so these are EF mapping issues).

If this is something you think you need, please open an issue in https://github.com/npgsql/efcore.pg.