TTL does not fully remove document from DB on VNext Emulator

[mwgustin]

Describe the bug When documents are removed via TTL setting, they are not fully removed from the database resulting in inconsistent behavior. A "ghost" reference still exists which causes things like Get All queries to return them

To Reproduce Steps to reproduce the behavior:

Create a container with TTL Enabled but No Default specified (-1).

        var dbResp = cosmosClient?.CreateDatabaseIfNotExistsAsync("CosmosIntegrationTests").GetAwaiter().GetResult();

        var props = new ContainerProperties("TestTTLContainer", "/PartitionKey")
        {
            DefaultTimeToLive = -1
        };
        dbResp?.Database.CreateContainerIfNotExistsAsync(props).GetAwaiter().GetResult();

Upsert a doc with a TTL property set to 1 second and wait the 1 second then refresh the container. There will be a ghost reference to the document that still remains (it shows in the document list, but when you click on it, it shows no data).

When in this ghost state, querying by ID will return a Not Found result as expected, but doing a more open ended query will return the ghost document. Also, if specifying an etag match, the upsert will fail unless it can match the "ghost" document's etag. You can also query in the explorer and it WILL return result (ie where c.id = "test-id") but it will again not show any data when clicked.

Full code demo:

 public class TTLTests
{
    public class TTLTestDoc
    {
        public required string id { get; set; }
        public required string PartitionKey { get; set; }
        public int ttl { get; set; }
    }
    [Fact]
    public async Task Test()
    {
        //setup client
        var ct = TestContext.Current.CancellationToken;
        var client = new CosmosClientBuilder("AccountEndpoint=http://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==")
            .WithConnectionModeGateway()
            .Build();
        var dbResp = await client.CreateDatabaseIfNotExistsAsync("TTLTestDb", cancellationToken: ct);
        
        //create container with TTL enabled but with no default
        var containerResp = await dbResp.Database.CreateContainerIfNotExistsAsync(new ContainerProperties("TTLTestContainer", "/PartitionKey")
        {
            DefaultTimeToLive = -1
        }, cancellationToken: ct);

        var container = containerResp.Container;

        TTLTestDoc x = new TTLTestDoc { id = "test", PartitionKey = "test", ttl = 1 };

        //upsert doc to be TTL'd in 1 second
        await container.UpsertItemAsync(x, new PartitionKey(x.PartitionKey), cancellationToken: ct);

        //wait 2.5 seconds for TTL to expire
        await Task.Delay(2500, ct);


        //throws 404 not found as expected (commented out)
        // var readResp = await container.ReadItemAsync<dynamic>(x.id, new Microsoft.Azure.Cosmos.PartitionKey(x.PartitionKey), cancellationToken: ct);

        // Should be empty, but returns the item that should have been TTL'd
        var queryResp = await container.GetItemQueryIterator<TTLTestDoc>(
            new QueryDefinition("SELECT * FROM c WHERE c.id = @id")
            .WithParameter("@id", x.id),
            requestOptions: new QueryRequestOptions
            {
                PartitionKey = new PartitionKey(x.PartitionKey)
            })
            .ReadNextAsync(ct);
        Assert.Empty(queryResp.Resource); //fails - Collection was not empty
    }
}

Expected behavior When a document exceeds its specified TTL it should be fully removed from the database.

Desktop (please complete the following information):

• OS: Mac OS 15.7.1
• Browser: Chrome (and via code)
• Version: vnext-preview (hash fb2e1d720a42)
• SDK: 3.46.1
• SDK: C#/dotnet

Additional context The issue was mentioned in a previous issue ( #215 ) but it looks like the issue was improperly diagnosed in that issue and closed.

[mwgustin]

A couple images showing the "ghost" doc reference and the TTL settings.