"Index was out of range" in ShaperProcessingExpressionVisitor.JsonEntityMaterializerRewriter

[roji]

With the repro below, the following exception is thrown:

Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonEntityMaterializerRewriter.<VisitSwitch>g__GenerateJsonPropertyReadLoop|11_0(ParameterExpression managerVariable, ParameterExpression tokenTypeVariable, List`1 finalBlockVariables, List`1 valueBufferTryReadValueMethodsToProcess) in /home/roji/projects/efcore/src/EFCore.Relational/Query/RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.cs:line 1826
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonEntityMaterializerRewriter.VisitSwitch(SwitchExpression switchExpression) in /home/roji/projects/efcore/src/EFCore.Relational/Query/RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.cs:line 1659
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at System.Dynamic.Utils.ExpressionVisitorUtils.VisitBlockExpressions(ExpressionVisitor visitor, BlockExpression block)
   at System.Linq.Expressions.ExpressionVisitor.VisitBlock(BlockExpression node)
...

Repro

await using var ctx = new BlogContext();
await ctx.Database.EnsureDeletedAsync();
await ctx.Database.EnsureCreatedAsync();

var test = await ctx.Workflows.ToListAsync();

public class BlogContext : DbContext
{
    public DbSet<Workflow> Workflows { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder
            .UseSqlServer("Server=localhost;Database=test;User=SA;Password=Abcd5678;Connect Timeout=60;ConnectRetryCount=0;Encrypt=false")
            .LogTo(Console.WriteLine, LogLevel.Information)
            .EnableSensitiveDataLogging();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Workflow>().OwnsMany(
            w => w.Activities, builder =>
            {
                builder.ToJson();
                builder.OwnsMany(a => a.Targets);
                builder.OwnsOne(a => a.Headers);
                builder.OwnsMany(a => a.FunctionMappings, navigationBuilder => navigationBuilder.OwnsMany(fm => fm.Conditionals) );
                builder.OwnsMany(a => a.Mappings);
            });
    }
}

public class Workflow
{
    public int Id { get; set; }
    public required EMethod Method { get; set; }
    public required string RequestPath { get; set; }
    public List<Activity> Activities { get; set; } = new();
}

public class Activity
{
    public required int Id { get; set; }
    public required EActivityType ActivityType { get; set; }
    public required float xPosition { get; set; }
    public required float yPosition { get; set; }
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
    public int? Source { get; set; }
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public Target[]? Targets { get; set; }


    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public Dictionary<string, string> Headers { get; set; }

    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
    public bool? Cached { get; set; }

    public Mapping[]? Mappings { get; set; }

    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
    public FunctionMapping[]? FunctionMappings { get; set; }
}

public class Target
{
    public int Id { get; set; }
    public string SourceHandleId { get; set; }
    public string TargetHandleId { get; set; }
}

public enum EActivityType
{
    InputNode,
    PostIt,
    HttpRequest,
    Mapping,
    JavaScript,
    If,
    HttpCode,
    Sql,
    Collect,
}

public class Mapping
{
    public string Variable { get; set; }
    public string Value { get; set; }
    public string DataType { get; set; }
}

public class FunctionConditional
{
    public EOption Option { get; set; }
    public EFunction Function { get; set; }
    public string Comparer { get; set; }
}

public class FunctionMapping
{
    public string JsonPath { get; set; }
    public string Variable { get; set; }
    public EGroupFunction Function { get; set; }
    public FunctionConditional[]? Conditionals { get; set; }
}

public enum EMethod
{
    GET,
    POST,
    PUT,
    DELETE
}

public enum EFunction
{
    NULL,
    EMPTY,
    EQUAL,
    NOTEQUAL,
    CONTAINS,
    NOTCONTAINS,
    STARTSWITH,
    ENDSWITH,
    GREATERTHAN,
    LESSTHAN,
    GREATERTHANEQUAL,
    LESSTHANEQUAL
}

public enum EGroupFunction
{
    SUM,
    MAX,
    MIN,
    CONCAT,
    MEDIAN,
    COUNT
}

public enum EOption
{
    IF,
    UNLESS
}

Originally opened by @maxwidergy for PostgreSQL in https://github.com/npgsql/efcore.pg/issues/2988

[maumar]

problem here is that Headers is Dictionary<string, string> so can't be an owned type mapped to json. We should be better about error/validation. Also improve robustness of the JsonEntityMaterializerRewriter - currently it throws index out of range if owned type doesn't have any property to populate from JSON

[smbecker]

I ran into this same issue today. If you do not require filtering by the JSON property, then you can use this as a work-around.

[totpero]

I get this error not only to Dictionary mapping.

[roji]

@totpero in that case we need to see a code sample.

[KeterSCP]

I got the same error without any dictionaries while trying to create repro for another issue...

Code

using Microsoft.EntityFrameworkCore;

await using var context = new TestContext();

var thisFails = context.MainModels.ToList();

public sealed class TestContext : DbContext
{
    private const string ConnectionString = "";

    public DbSet<MainModel> MainModels { get; set; }

    public TestContext()
    {
        Database.EnsureCreated();
    }

    protected override void OnConfiguring(DbContextOptionsBuilder options) => options.UseNpgsql(ConnectionString);

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<MainModel>()
            .OwnsOne(x => x.JsonModel, builder =>
            {
                builder.ToJson();
                builder.OwnsMany(d => d.InnerModels);
            });
    }
}

public class MainModel
{
    public int Id { get; set; }
    public JsonModel JsonModel { get; set; }
}

public class JsonModel
{
    public List<JsonInnerModel> InnerModels { get; set; }
}

public class JsonInnerModel
{
    public int Id { get; set; }
}

Stack trace

Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonEntityMaterializerRewriter.<VisitSwitch>g__GenerateJsonPropertyReadLoop|11_0(ParameterExpression managerVariable, ParameterExpression tokenTypeVariable, List`1 finalBlockVariables, List`1 valueBufferTryReadValueMethodsToProcess)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonEntityMaterializerRewriter.VisitSwitch(SwitchExpression switchExpression)
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at System.Dynamic.Utils.ExpressionVisitorUtils.VisitBlockExpressions(ExpressionVisitor visitor, BlockExpression block)
   at System.Linq.Expressions.ExpressionVisitor.VisitBlock(BlockExpression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitConditional(ConditionalExpression node)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonEntityMaterializerRewriter.VisitConditional(ConditionalExpression conditionalExpression)
   at System.Linq.Expressions.ExpressionVisitor.VisitConditional(ConditionalExpression node)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonEntityMaterializerRewriter.VisitConditional(ConditionalExpression conditionalExpression)
   at System.Dynamic.Utils.ExpressionVisitorUtils.VisitBlockExpressions(ExpressionVisitor visitor, BlockExpression block)
   at System.Linq.Expressions.ExpressionVisitor.VisitBlock(BlockExpression node)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonEntityMaterializerRewriter.Rewrite(BlockExpression jsonEntityShaperMaterializer)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.CreateJsonShapers(IEntityType entityType, Boolean nullable, ParameterExpression jsonReaderDataParameter, ParameterExpression keyValuesParameter, Expression parentEntityExpression, INavigation navigation)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.CreateJsonShapers(IEntityType entityType, Boolean nullable, ParameterExpression jsonReaderDataParameter, ParameterExpression keyValuesParameter, Expression parentEntityExpression, INavigation navigation)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.ProcessShaper(Expression shaperExpression, RelationalCommandCache& relationalCommandCache, IReadOnlyList`1& readerColumns, LambdaExpression& relatedDataLoaders, Int32& collectionId)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.VisitShapedQuery(ShapedQueryExpression shapedQueryExpression)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.System.Collections.Generic.IEnumerable<TEntity>.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

UPD: if I change my query to

var thisFails = context.MainModels
    .Where(mainModel => mainModel.JsonModel.InnerModels.Any(inner => inner.Id == 1))
    .ToList();

I'm getting yet another strange exception:

Stack trace 2

Unhandled exception. System.Collections.Generic.KeyNotFoundException: The given key 'Property: JsonInnerModel.Id (int) Required PK AfterSave:Throw ValueGenerated.OnAdd' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.EntityFrameworkCore.Query.StructuralTypeProjectionExpression.BindProperty(IProperty property)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.BindProperty(StructuralTypeReferenceExpression typeReference, IProperty property)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.TryBindMember(Expression source, MemberIdentity member, Expression& expression, IPropertyBase& property)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.TryBindMember(Expression source, MemberIdentity member, Expression& expression)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitMember(MemberExpression memberExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitBinary(BinaryExpression binaryExpression)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Query.Internal.NpgsqlSqlTranslatingExpressionVisitor.VisitBinary(BinaryExpression binaryExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.TranslateInternal(Expression expression, Boolean applyDefaultTypeMapping)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.Translate(Expression expression, Boolean applyDefaultTypeMapping)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateExpression(Expression expression, Boolean applyDefaultTypeMapping)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateLambdaExpression(ShapedQueryExpression shapedQueryExpression, LambdaExpression lambdaExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateWhere(ShapedQueryExpression source, LambdaExpression predicate)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateAny(ShapedQueryExpression source, LambdaExpression predicate)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Query.Internal.NpgsqlQueryableMethodTranslatingExpressionVisitor.TranslateAny(ShapedQueryExpression source, LambdaExpression predicate)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.TranslateSubquery(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Query.Internal.NpgsqlSqlTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.TranslateInternal(Expression expression, Boolean applyDefaultTypeMapping)
   at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.Translate(Expression expression, Boolean applyDefaultTypeMapping)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateExpression(Expression expression, Boolean applyDefaultTypeMapping)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateLambdaExpression(ShapedQueryExpression shapedQueryExpression, LambdaExpression lambdaExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateWhere(ShapedQueryExpression source, LambdaExpression predicate)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

[totpero]

Hi @roji , something like @KeterSCP scenario was my case to.

[attilah]

I was just troubleshooting the index out of range error that was weird. In my setup - as the project is under development - the class that was configured as Owned and using ToJson had no properties, and the column had no content. Exactly what @maumar described here

[maumar]

@attilah that issue has now been fixed - fix will be available in preview2 or is currently available in the daily build. See https://github.com/dotnet/efcore/issues/32939

[georg-jung]

currently it throws index out of range if owned type doesn't have any property to populate from JSON

Thanks for the insight @maumar! ~~I ran into this issue when I tried to create a model with polymorphy - that intentionally has no property to populate in the base type:~~

[Table("email_report")]
public class EmailReport
{
    public int EmailReportId { get; set; }

    [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
    public Instant Created { get; set; }

    public List<EmailReportParagraphBase> Paragraphs { get; set; } = [];
}

// This should probably be abstract. EF Core requires either an instantiable type or derived classes as part of the model though (didn't test the latter but the exception thrown says so).
[JsonDerivedType(typeof(MarkdownParagraph), typeDiscriminator: "markdown")]
[JsonDerivedType(typeof(TableFromSqlParagraph), typeDiscriminator: "tableFromSql")]
public class EmailReportParagraphBase
{
}

public sealed class MarkdownParagraph : EmailReportParagraphBase
{
    public required string Markdown { get; set; }
}

public sealed class TableFromSqlParagraph : EmailReportParagraphBase
{
    public required string Query { get; set; }
}

public class AppDbContext : DbContext
{
    public AppDbContext(DbContextOptions<AppDbContext> options)
        : base(options)
    {
    }

    public DbSet<EmailReport> EmailReports { get; set; }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<EmailReport>(er =>
        {
            er.Property(x => x.Created).HasDefaultValueSql("now()");
        });
        modelBuilder.Entity<EmailReport>().OwnsMany(e => e.Paragraphs, p => p.ToJson());
    }
}

~~Is this something that is unsupported for other reasons too or would this work when this is fixed?~~

Mostly duplicate of https://github.com/dotnet/efcore/issues/27779

[dks07]

I am getting the same error with version 8.0.4. I am using Jsonb but not using dictionaries. The point that I have noticed is that my Jsonb property does not have any sub-property. So as a workaround, I have to add some property to it or else Ignore that Jsonb property.

[PatrikFomin]

Seems to be the same issue in version 8.0.6.

[maumar]

I've re-tested the code the samples provided and things work fine on my end. @KeterSCP 's issue with

var thisFails = context.MainModels .Where(mainModel => mainModel.JsonModel.InnerModels.Any(inner => inner.Id == 1)) .ToList();

is caused by the fact that JsonInnerModel uses property named Id, Renaming the property solves the problem (related issues: https://github.com/dotnet/efcore/issues/29380 and https://github.com/dotnet/efcore/issues/28594)

[maumar]

@dks07 @PatrikFomin the issue throws a generic exception so it's possible this is a different bug. Can you please file new issue and provide the repro code?

[maumar]

fixed by https://github.com/dotnet/efcore/pull/32966