Upsert merge and replace return 404 incorrectly when the entity does not exist

[joelverhagen]

Which service(blob, file, queue, table) does this issue concern?

table

Which version of the Azurite was used?

3.18.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

VS Code extension

What's the Node.js version?

v16.13.0

What problem was encountered?

These sort of upsert operations returned 404 when the entity does not yet exist. This is wrong for upsert.

This problem did not occur in 3.17.1.

await table.UpsertEntityAsync(entityA, TableUpdateMode.Merge);
await table.UpsertEntityAsync(entityB, TableUpdateMode.Replace);

Steps to reproduce the issue?

I made this helper app to test a bunch of different variants of delete, update, and upsert.

using System.Text.RegularExpressions;
using Azure;
using Azure.Core.Pipeline;
using Azure.Data.Tables;

var client = new TableServiceClient("UseDevelopmentStorage=true", new TableClientOptions
{
    Transport = new HttpClientTransport(new HttpClient(new LoggingHandler { InnerHandler = new HttpClientHandler() }))
});
var table = client.GetTableClient($"test{DateTimeOffset.UtcNow.Ticks}");
table.CreateIfNotExists();

var tests = new Func<int, Task>[]
{
    async i =>
    {
        Console.WriteLine("[New entity, Upsert Merge]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.UpsertEntityAsync(entity, TableUpdateMode.Merge);
    },
    async i =>
    {
        Console.WriteLine("[New entity, Update Merge *]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.UpdateEntityAsync(entity, ETag.All, TableUpdateMode.Merge);
    },
    async i =>
    {
        Console.WriteLine("[New entity, Upsert Replace]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.UpsertEntityAsync(entity, TableUpdateMode.Replace);
    },
    async i =>
    {
        Console.WriteLine("[New entity, Update Replace *]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.UpdateEntityAsync(entity, ETag.All, TableUpdateMode.Replace);
    },
    async i =>
    {
        Console.WriteLine("[New entity, Delete *]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.DeleteEntityAsync(entity.PartitionKey, entity.RowKey, ETag.All);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Upsert Merge]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.UpsertEntityAsync(entity, TableUpdateMode.Merge);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Update Merge *]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.UpdateEntityAsync(entity, ETag.All, TableUpdateMode.Merge);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Update Merge, matching etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.UpdateEntityAsync(entity, response.Headers.ETag.Value, TableUpdateMode.Merge);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Update Merge, wrong etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.UpdateEntityAsync(entity, new ETag(response.Headers.ETag.Value.ToString().Replace("20", "30")), TableUpdateMode.Merge);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Upsert Replace]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.UpsertEntityAsync(entity, TableUpdateMode.Replace);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Update Replace *]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.UpdateEntityAsync(entity, ETag.All, TableUpdateMode.Replace);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Update Replace, matching etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.UpdateEntityAsync(entity, response.Headers.ETag.Value, TableUpdateMode.Replace);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Update Merge, wrong etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.UpdateEntityAsync(entity, new ETag(response.Headers.ETag.Value.ToString().Replace("20", "30")), TableUpdateMode.Replace);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Delete *]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.DeleteEntityAsync(entity.PartitionKey, entity.RowKey, ETag.All);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Delete, matching etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.DeleteEntityAsync(entity.PartitionKey, entity.RowKey, response.Headers.ETag.Value);
    },
    async i =>
    {
        Console.WriteLine("[Existing entity, Delete, wrong etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.DeleteEntityAsync(entity.PartitionKey, entity.RowKey, new ETag(response.Headers.ETag.Value.ToString().Replace("20", "30")));
    },

    async i =>
    {
        Console.WriteLine("[Batch, New entity, Upsert Merge]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpsertMerge, entity) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, New entity, Update Merge *]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpdateMerge, entity, ETag.All) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, New entity, Upsert Replace]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpsertReplace, entity) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, New entity, Update Replace *]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpdateReplace, entity, ETag.All) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, New entity, Delete *]");
        var entity = new TableEntity("pk", $"rk{i}");
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.Delete, entity, ETag.All) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Upsert Merge]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpsertMerge, entity) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Update Merge *]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpdateMerge, entity, ETag.All) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Update Merge, matching etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpdateMerge, entity, response.Headers.ETag.Value) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Update Merge, wrong etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpdateMerge, entity, new ETag(response.Headers.ETag.Value.ToString().Replace("20", "30"))) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Upsert Replace]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpsertReplace, entity) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Update Replace *]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpdateReplace, entity, ETag.All) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Update Replace, matching etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpdateReplace, entity, response.Headers.ETag.Value) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Update Merge, wrong etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.UpdateReplace, entity, new ETag(response.Headers.ETag.Value.ToString().Replace("20", "30"))) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Delete *]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.Delete, entity, ETag.All) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Delete, matching etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.Delete, entity, response.Headers.ETag.Value) });
    },
    async i =>
    {
        Console.WriteLine("[Batch, Existing entity, Delete, wrong etag]");
        var entity = new TableEntity("pk", $"rk{i}");
        LoggingHandler.Enable = false;
        var response = await table.AddEntityAsync(entity);
        LoggingHandler.Enable = true;
        await table.SubmitTransactionAsync(new[] { new TableTransactionAction(TableTransactionActionType.Delete, entity, new ETag(response.Headers.ETag.Value.ToString().Replace("20", "30"))) });
    },
};

for (int i = 0; i < tests.Length; i++)
{
    try
    {
        await tests[i](i);
    }
    catch (RequestFailedException ex)
    {
        Console.WriteLine("  ERROR: " + ex.ErrorCode);
    }
}

class LoggingHandler : DelegatingHandler
{
    public static bool Enable = true;

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var logged = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
        {
            "If-Match",
        };

        if (Enable)
        {
            Console.Write($"  {request.Method}");
            foreach (var header in request.Headers.Concat(request.Content?.Headers ?? Enumerable.Empty<KeyValuePair<string, IEnumerable<string>>>()))
            {
                if (logged.Contains(header.Key))
                {
                    foreach (var value in header.Value)
                    {
                        Console.Write($" {header.Key}: {Regex.Replace(value, "datetime'.+?'", "datetime'...'")}");
                    }
                }
            }
            Console.Write(" => ");
        }

        var response = await base.SendAsync(request, cancellationToken);

        if (Enable)
        {
            Console.Write($"{(int)response.StatusCode}");
            foreach (var header in response.Headers.Concat(response.Content?.Headers ?? Enumerable.Empty<KeyValuePair<string, IEnumerable<string>>>()))
            {
                if (logged.Contains(header.Key))
                {
                    foreach (var value in header.Value)
                    {
                        Console.Write($" {header.Key}: {value}");
                    }
                }
            }
            Console.WriteLine();
        }

        return response;
    }
}

This diff shows the output difference between the legacy storage emulator and v3.18.0

--- legacy-emulator.txt	2022-06-23 17:16:14.611512200 -0700
+++ azurite-3.18.0.txt	2022-06-23 17:15:47.277457800 -0700
@@ -1,75 +1,73 @@
 [New entity, Upsert Merge]
-  PATCH => 204
+  PATCH => 404
+  ERROR: ResourceNotFound
 [New entity, Update Merge *]
   PATCH If-Match: * => 404
   ERROR: ResourceNotFound
 [New entity, Upsert Replace]
   PUT => 204
 [New entity, Update Replace *]
-  PUT If-Match: * => 404
-  ERROR: ResourceNotFound
+  PUT If-Match: * => 204
 [New entity, Delete *]
   DELETE If-Match: * => 404
 [Existing entity, Upsert Merge]
   PATCH => 204
 [Existing entity, Update Merge *]
   PATCH If-Match: * => 204
 [Existing entity, Update Merge, matching etag]
   PATCH If-Match: W/"datetime'...'" => 204
 [Existing entity, Update Merge, wrong etag]
   PATCH If-Match: W/"datetime'...'" => 412
   ERROR: UpdateConditionNotSatisfied
 [Existing entity, Upsert Replace]
   PUT => 204
 [Existing entity, Update Replace *]
   PUT If-Match: * => 204
 [Existing entity, Update Replace, matching etag]
   PUT If-Match: W/"datetime'...'" => 204
 [Existing entity, Update Merge, wrong etag]
   PUT If-Match: W/"datetime'...'" => 412
   ERROR: UpdateConditionNotSatisfied
 [Existing entity, Delete *]
   DELETE If-Match: * => 204
 [Existing entity, Delete, matching etag]
   DELETE If-Match: W/"datetime'...'" => 204
 [Existing entity, Delete, wrong etag]
   DELETE If-Match: W/"datetime'...'" => 412
   ERROR: UpdateConditionNotSatisfied
 [Batch, New entity, Upsert Merge]
   POST => 202
 [Batch, New entity, Update Merge *]
   POST => 202
-  ERROR: ResourceNotFound
 [Batch, New entity, Upsert Replace]
   POST => 202
 [Batch, New entity, Update Replace *]
   POST => 202
-  ERROR: ResourceNotFound
 [Batch, New entity, Delete *]
   POST => 202
   ERROR: ResourceNotFound
 [Batch, Existing entity, Upsert Merge]
   POST => 202
 [Batch, Existing entity, Update Merge *]
   POST => 202
 [Batch, Existing entity, Update Merge, matching etag]
   POST => 202
 [Batch, Existing entity, Update Merge, wrong etag]
   POST => 202
   ERROR: UpdateConditionNotSatisfied
 [Batch, Existing entity, Upsert Replace]
   POST => 202
 [Batch, Existing entity, Update Replace *]
   POST => 202
 [Batch, Existing entity, Update Replace, matching etag]
   POST => 202
 [Batch, Existing entity, Update Merge, wrong etag]
   POST => 202
   ERROR: UpdateConditionNotSatisfied
 [Batch, Existing entity, Delete *]
   POST => 202
 [Batch, Existing entity, Delete, matching etag]
   POST => 202
 [Batch, Existing entity, Delete, wrong etag]
   POST => 202
   ERROR: UpdateConditionNotSatisfied

Have you found a mitigation/solution?

[tabrath]

This also hit us today when running in CI. Guess we'll have to pin it to the last version until this is fixed.

[despian]

I also came across this issue after updating Azurite.

[jvanegmond]

@edwin-huber We have couple CI builds failing that depend on this. Would love to get an update.

[edwin-huber]

Changes have been merged via #1566, and will be made available with the next release.

[roly445]

Is there a release date for this, or is there a nightly channel that we can get a release from?

[aressler38]

Is there a release date for this, or is there a nightly channel that we can get a release from?

Looks like they release every two months or so. There's an alpha package you might try. I think npm i -g azurite@alpha should install it.

[joelverhagen]

This appears to be fixed in 3.19.0. Thanks @edwin-huber!