Projection cause "Index out of range" on Database first EF core (net 6)

[invzz]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

I Have wrapped an old DB2 database with EF and created a DbContext which does not match the table schema on the database. I had to "split" tables in different entities using the "Owned" properties

so namely the table Viaggi is splitted in infoPax, infoMerci, and so on. for each of the owned type I created a Owned navigation property using EF

for example here entity is of Type Viaggio

 _ = entity.OwnsOne(via => via.InfoPax, entity =>
                {
                    _ = entity.Property(e => e.Capacita).HasColumnName("VIPAT").HasColumnType("smallint(2)").HasDefaultValueSql("0");
                    _ = entity.Property(e => e.Venduto).HasColumnName("VIPAV").HasColumnType("smallint(2)").HasDefaultValueSql("0");
                    _ = entity.Property(e => e.Warning).HasColumnName("VIPAW").HasColumnType("smallint(2)").HasDefaultValueSql("0");
                    _ = entity.Property(e => e.BloccoVendite).IsRequired().HasColumnName("VIPBV").HasColumnType("char(1)").HasDefaultValueSql("' '");
                    _ = entity.Property(e => e.Abilitazione).IsRequired().HasColumnName("VIPFV").HasColumnType("char(1)").HasDefaultValueSql("' '");
                    _ = entity.Property(e => e.TotaleMetriLieari).HasColumnName("VIPML").HasColumnType("decimal(5, 1)").HasDefaultValueSql("0");
                    _ = entity.Property(e => e.ProgressivoBooking).HasColumnName("VIPPB").HasColumnType("integer(4)").HasDefaultValueSql("0");
                }

I Have the usual repository accessing the data and returning an IQueryable<Viaggio>

If I don't use Projections everything works great.

The problem starts when I add [UseProjection] on my query: It all works well until I try to access those owned (in theory non nullable) owned entities which really are in the same table of Viaggi.

That causes

"Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')",

I had a look in the produced Linq query :

 Compiling query expression: 
      'DbSet<Viaggio>()
          .Where(x => x.IdViaggio == __id_0)
          .Select(_s1 => new Viaggio{ 
              IdSocieta = _s1.IdSocieta, 
              InfoPax = _s1.InfoPax != null ? new InfoPax{ BloccoVendite = _s1.InfoPax.BloccoVendite }
               : null,  
                  )
                  .ToList() 
          }
          )'

and it looks fine but :

• it appears that the check _s1.InfoPax != null ? is causing the index out of range problem. If I remove the null check and the ternary operator it works and produces the right SQL

Steps to reproduce

1. Using Db First approach create a table on a server and map it onto different entities using ef core table splitting https://docs.microsoft.com/en-us/ef/core/modeling/table-splitting
2. try to access the nested owned entities using a query in graphql

Relevant log output

"errors": [
    {
      "message": "Unexpected Execution Error",
      "locations": [
        {
          "line": 2,
          "column": 2
        }
      ],
      "path": [
        "viaggiProjection"
      ],
      "extensions": {
        "message": "Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')",
        "stackTrace": "   at Microsoft.EntityFrameworkCore.Query.SqlExpressions.SelectExpression.GetProjection(ProjectionBindingExpression projectionBindingExpression)\n   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.VisitExtension(Expression extensionExpression)\n   at System.Linq.Expressions.Expression.Accept(ExpressionVisitor visitor)\n   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)\n   at System.Linq.Expressions.ExpressionVisitor.VisitMemberAssignment(MemberAssignment node)\n   at System.Linq.Expressions.ExpressionVisitor.VisitMemberBinding(MemberBinding node)\n   at System.Linq.Expressions.ExpressionVisitor.Visit[T](ReadOnlyCollection`1 nodes, Func`2 elementVisitor)\n   at System.Linq.Expressions.ExpressionVisitor.VisitMemberInit(MemberInitExpression node)\n   at System.Linq.Expressions.MemberInitExpression.Accept(ExpressionVisitor visitor)\n   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.ProcessShaper(Expression shaperExpression, RelationalCommandCache& relationalCommandCache, LambdaExpression& relatedDataLoaders, Int32& collectionId)\n   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.VisitShapedQuery(ShapedQueryExpression shapedQueryExpression)\n   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.VisitExtension(Expression extensionExpression)\n   at System.Linq.Expressions.Expression.Accept(ExpressionVisitor visitor)\n   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)\n   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)\n   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)\n   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)\n   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetAsyncEnumerator(CancellationToken cancellationToken)\n   at System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1.GetAsyncEnumerator()\n   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)\n   at HotChocolate.Data.ToListMiddleware`1.InvokeAsync(IMiddlewareContext context)\n   at HotChocolate.Types.EntityFrameworkObjectFieldDescriptorExtensions.<>c__DisplayClass2_1`1.<<UseDbContext>b__4>d.MoveNext()\n--- End of stack trace from previous location ---\n   at HotChocolate.Types.EntityFrameworkObjectFieldDescriptorExtensions.<>c__DisplayClass2_1`1.<<UseDbContext>b__4>d.MoveNext()\n--- End of stack trace from previous location ---\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)"
      }
    }
  ]
}

Additional Context?

Im using IBM entity framework core to connect to an AS400 system which has a DB2 instance

Product

Hot Chocolate

Version

12.12.0

[PascalSenn]

@invzz thanks for reporting. We are working on a better ef core integration. At the moment we do not know if EF Core (and what Features of EF Core) is used.

[invzz]

@PascalSenn thanks for your reply, I'm looking forward to it. My workaround is to work with flattened tables without tablesplitting and use automapper later to get the right entity subdivision.

[PascalSenn]

ah you use Automapper? because then this is probably the issue

[invzz]

I'm not using Automapper in graphQL, although I have it registered in services and I use it for other endpoints in a controller which expose some other pieces of functionality (basic crud db interrogations)

I also tried to exclude Automapper ( deleting the .AddAutoMapper(AppDomain.CurrentDomain.GetAssemblies()) line) from the loaded services and excluding it from my controller, but I had the same results

builder.Services

    .Configure<As400Options>(builder.Configuration.GetSection(As400Options.OptionsKey))
    .AddPooledDbContextFactory<AS400Context>(ob =>
    {
        ob.UseDb2(_connectionString, x => x.SetServerInfo(IBMDBServerType.AS400));
        ob.LogTo(Console.WriteLine);
    })
    .AddDbContext<AS400Context>( ob => {
        ob.UseDb2(_connectionString, x => x.SetServerInfo(IBMDBServerType.AS400));
        ob.LogTo(Console.WriteLine);
    })
    .AddTransient<IReadOnlyRepository<ViaggioGviat>, ReadOnlyRepository<AS400Context, ViaggioGviat>>()
    .AddAutoMapper(AppDomain.CurrentDomain.GetAssemblies())
    .ConfigureApi()
    .AddHttpClient()
    .AddGraphQLServer()
    .AddProjections()
    .AddSorting()
    .AddFiltering()
    .RegisterDbContext<AS400Context>()
    .AddQueryType<PaxQuery>();

Another "weird" and related thing is happening :

I changed my context and started using flattened tables which models mirrors the tables in the db schema:

using ConfigurationOptions.Options;
using EF400_Relational.Models.Entities;
using IBM.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Options;

namespace EF400_Relational
{
    public class AS400Context : DbContext
    {   public AS400Context(DbContextOptions<AS400Context> options) : base(options) { }
        public virtual DbSet<TestataGbigt> Testate { get; set; } = null!;
        public virtual DbSet<ViaggioGviat> Viaggi { get; set; } = null!;
        public virtual DbSet<CartaImbarcoGcimb> CarteImbarco { get; set; } = null!;
        public virtual DbSet<NaveTanav> Navi { get; set; } = null!;
        public virtual DbSet<PortoAnpor> Porti { get; set; } = null!;
        public virtual DbSet<MultitrattaGmvia> Multitratte { get; set; } = null!;
        public virtual DbSet<CartaImbarcoOld> CarteImbarcoGassv { get; set; } = null!;

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            ViaggioGviat.BuildModel(modelBuilder, e =>e.ToTable("GVIAT00F", "GNV2008"));
            TestataGbigt.BuildModel(modelBuilder, e =>e.ToTable("GBIGT00F", "GNV2008"));
            CartaImbarcoGcimb.BuildModel(modelBuilder, e =>e.ToTable("GCIMB00F", "GNV2008"));
            NaveTanav.BuildModel(modelBuilder, e =>e.ToTable("TANAV00F", "GNV2008"));
            PortoAnpor.BuildModel(modelBuilder, e =>e.ToTable("ANPOR00F", "GNV2008"));
            MultitrattaGmvia.BuildModel(modelBuilder, e =>e.ToTable("GMVIA00F", "GNVUTL"));
            CartaImbarcoOld.BuildModel(modelBuilder, e =>e.ToTable("GASSV00F", "GNVPROD"));
            CategoriaPaxAncat.BuildModel(modelBuilder, e =>e.ToTable("ANCAT00F", "GNV2008"));
           // _ = modelBuilder.Entity<NaveTanav>(e => {  });
            _ = modelBuilder.Entity<ViaggioGviat>(e =>
            {
                // relazione con testata
                _ = e.HasMany(gviat => gviat.Testate).WithOne(x => x.Viaggio).HasForeignKey(t => new { t.IdSocieta, t.IdViaggio });
                // testate di ritorno
                _ = e.HasMany(gviat => gviat.TestateRitorno).WithOne(x => x.ViaggioRitorno).HasForeignKey(t => new { t.IdSocieta, t.IdViaggioRitorno }).IsRequired();
                // relazione con Nave
                _ = e.HasOne(x => x.Nave).WithMany().HasForeignKey(x => new { x.IdSocieta, x.IdNave }).IsRequired(true);
                // Porto Arrivo
                _ = e.HasOne(x => x.PortoArrivo).WithMany().HasForeignKey(x => x.IdPortoArrivo).IsRequired(true);
                // PortoPartenza
                _ = e.HasOne(x => x.PortoPartenza).WithMany().HasForeignKey(x => x.IdPortoPartenza).IsRequired(true);
                // PortoPrimoScalo
                _ = e.HasOne(x => x.PortoPrimoScalo).WithMany().HasForeignKey(x => x.IdPortoPrimoScalo).IsRequired(false);
            });

            _ = modelBuilder.Entity<MultitrattaGmvia>(e =>
            {
                //viaggio
                _ = e.HasOne(p => p.ViaggioMultitratta).WithOne(p => p.Multitratta).HasForeignKey<MultitrattaGmvia>(p => new { p.IdViaggioMultitratta }).HasPrincipalKey<ViaggioGviat>(x => x.IdViaggio).IsRequired(false);
                //viaggo
                _ = e.HasOne(p => p.ViaggioSingolaTratta).WithOne(p => p.SingolaTratta).HasForeignKey<MultitrattaGmvia>(p => new { p.IdViaggioSingolaTratta }).HasPrincipalKey<ViaggioGviat>(x => x.IdViaggio).IsRequired(false);
            });

            _ = modelBuilder.Entity<TestataGbigt>(e =>
            {
                // relazione con logicalcartaimbarco
                _ = e.HasMany(big => big.CarteImbarcoGcimb).WithOne().HasForeignKey(c => new { c.Societa, c.PrefissoBiglietto, c.IdBiglietto, });
                // relazione con Nave
                _ = e.HasOne(x => x.Nave).WithOne(x => x.Testata).HasForeignKey<TestataGbigt>(x => new { x.IdSocieta, x.IdNave });
                // Porto Arrivo
                _ = e.HasOne(x => x.PortoArrivo).WithMany(p => p.TestateArrivo).HasForeignKey(x => x.IdPortoArrivo).IsRequired(false);
                // PortoPartenza
                _ = e.HasOne(x => x.PortoPartenza).WithMany(p => p.TestatePartenza).HasForeignKey(x => x.IdPortoPartenza).IsRequired(false);
            });

            _ = modelBuilder.Entity<CartaImbarcoGcimb>(e =>
            {
                _ = e.HasOne(x => x.OldCartaImbarcoGassv).WithOne(x => x.NewCartaImbarco).HasForeignKey<CartaImbarcoGcimb>(b => new { b.ViaggioId, b.Societa, b.IdBiglietto, b.ProgressivoCartaImbarco }).IsRequired(false);
                _ = e.HasOne(x => x.CategoriaPax).WithMany().IsRequired(false).HasForeignKey(x => x.IdCategoria);
            });
        }
    }
}

Where the buildModel method is a static method which for every property adds the configuration

entity.Property(x => x.Myproperty).HasColumnName("ColumnName").HasColumnType("MyType")

In this way I though to avoid using those owned types and table splitting approach.

My query class is a simple one:

namespace EF400RWebApi.QueryTypes
{
   public class PaxQuery
   {
       [UseDbContext(typeof(AS400Context))]
       [UseProjection]
       [UseFiltering]
       public IQueryable<ViaggioGviat> GetViaggiProjection([ScopedService] AS400Context db, int id) => db.Viaggi.Where(v => v.IdViaggio == id).Include(x => x.Nave).Include(x => x.Testate);
[UseDbContext(typeof(AS400Context))]
       public IQueryable<ViaggioGviat> GetViaggi([ScopedService] AS400Context db, int id) => db.Viaggi.Where(v => v.IdViaggio == id).Include(x => x.Nave);
   }
}

• when I query (ViaggiProjection) any property inside Nave which have Many-to-1 relation with shadowed navigation property on One side I get that index out of range and the .Select(_s1 => new ViaggioGviat{ Nave = _s1.Nave != null ? new NaveTanav{ IdNave = _s1.Nave.IdNave } : null } ) expression (which causes the error)

query projected {
  viaggiProjection(id: 19170) {
    nave {
      idNave
    }
  }
}

results in error

{
  "errors": [
    {
      "message": "Unexpected Execution Error",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "path": [
        "viaggiProjection"
      ],
      "extensions": {
        "message": "Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')",
        "stackTrace": "   at Microsoft.EntityFrameworkCore.Query.SqlExpressions.SelectExpression.GetProjection(ProjectionBindingExpression projectionBindingExpression)\n   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.VisitExtension(Expression extensionExpression)\n   at System.Linq.Expressions.Expression.Accept(ExpressionVisitor visitor)\n   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)\n   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)\n   at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)\n   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)\n   at System.Linq.Expressions.ExpressionVisitor.VisitConditional(ConditionalExpression node)\n   at System.Linq.Expressions.ConditionalExpression.Accept(ExpressionVisitor visitor)\n   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)\n   at System.Linq.Expressions.ExpressionVisitor.VisitMemberAssignment(MemberAssignment node)\n   at System.Linq.Expressions.ExpressionVisitor.VisitMemberBinding(MemberBinding node)\n   at System.Linq.Expressions.ExpressionVisitor.Visit[T](ReadOnlyCollection`1 nodes, Func`2 elementVisitor)\n   at System.Linq.Expressions.ExpressionVisitor.VisitMemberInit(MemberInitExpression node)\n   at System.Linq.Expressions.MemberInitExpression.Accept(ExpressionVisitor visitor)\n   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.ProcessShaper(Expression shaperExpression, RelationalCommandCache& relationalCommandCache, LambdaExpression& relatedDataLoaders, Int32& collectionId)\n   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.VisitShapedQuery(ShapedQueryExpression shapedQueryExpression)\n   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.VisitExtension(Expression extensionExpression)\n   at System.Linq.Expressions.Expression.Accept(ExpressionVisitor visitor)\n   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)\n   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)\n   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)\n   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)\n   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetAsyncEnumerator(CancellationToken cancellationToken)\n   at System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1.GetAsyncEnumerator()\n   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)\n   at HotChocolate.Data.ToListMiddleware`1.InvokeAsync(IMiddlewareContext context)\n   at HotChocolate.Types.EntityFrameworkObjectFieldDescriptorExtensions.<>c__DisplayClass2_1`1.<<UseDbContext>b__4>d.MoveNext()\n--- End of stack trace from previous location ---\n   at HotChocolate.Types.EntityFrameworkObjectFieldDescriptorExtensions.<>c__DisplayClass2_1`1.<<UseDbContext>b__4>d.MoveNext()\n--- End of stack trace from previous location ---\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)"
      }
    }
  ]
}

caused by the generated linq expression using projections

DbSet<ViaggioGviat>()
          .Where(v => v.IdViaggio == __id_0)
          .Include(x => x.Nave)
          .Include(x => x.Testate)
          .Select(_s1 => new ViaggioGviat{ Nave = _s1.Nave != null ? new NaveTanav{ IdNave = _s1.Nave.IdNave }
               : null }
          )

while the

query projected {
  viaggiProjection(id: 19170) {
    testate {
      idBiglietto
    }
  }
}

works and reply

 "data": {
    "viaggiProjection": [
      {
        "testate": [
          {
            "idBiglietto": 9482819
          },
          {
            "idBiglietto": 8764575
          },
         ]
      }
] }

everything works if I don't use projections as in GetFullViaggi.

In My use case projections are necessary because the entities have 200+ fields, queries with few fields are much faster (lower in volume: for 1 full viaggio query i get 6 MB of data, if I use projections i get few kb)

for clarity I here the navigation properties inside viaggioGviat

class ViaggioGviat {
         // omitted for brevity.... 
         // reference to Nave
        public string? IdNave { get; set; }
        public virtual NaveTanav Nave { get; set; } = null!;

        // reference to Porto
        public string? IdPortoPartenza { get; set; }
        public virtual PortoAnpor PortoPartenza { get; set; } = null!;

        // reference to porto
        public string? IdPortoArrivo { get; set; }
        public virtual PortoAnpor PortoArrivo { get; set; } = null!;

        // reference to porto
        public string? IdPortoPrimoScalo { get; set; }
        public virtual PortoAnpor PortoPrimoScalo { get; set; } = null!;

        //reference  to testata
        public virtual ICollection<TestataGbigt>? Testate { get; set; }
        public virtual ICollection<TestataGbigt>? TestateRitorno { get; set; }

        // references Multitratta
        public virtual MultitrattaGmvia Multitratta { get; set; } = null!;
        public virtual MultitrattaGmvia SingolaTratta { get; set; } = null!;

 // omitted for brevity.... 
}

I'm now trying to use the github repo instead of the nuget packages so I can further investigate through brakepoints what's happening, Hotchocolate project is huge and I'm not currently been able to set it up as I get some reference errors on builds

[invzz]

@PascalSenn

looks like my problem is here

 public static Expression NotNull(Expression expression)
    {
        return Expression.NotEqual(expression, _null);
    }

    public static Expression NotNullAndAlso(Expression property, Expression condition)
    {
        return Expression.Condition(
            NotNull(property),
            condition,
            Expression.Default(property.Type));
    }

Looks like theNotNullAndAlso is adding the condition which causes my problems:

• by removing the condition expression and using return Expression.Default(property.Type)); I kinda solved all my problems and works like a charm. I don't know if this might help. Thanks

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.