Inserting an whole number as a double results in a Int32 field of an entity

[jezzsantos]

Which service(blob, file, queue, table) does this issue concern?

table

Which version of the Azurite was used?

3.16.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm

What's the Node.js version?

16.3.1

What problem was encountered?

When using EntityProperty.GeneratePropertyForDouble((double?)1.0D) the data type of the property in the table is set to Int32, and not to Double, incorrectly.

When using EntityProperty.GeneratePropertyForDouble((double?)1.1D) the data type of the property in the table is set to Double, correctly.

Steps to reproduce the issue?

If possible, please provide the debug log using the -d parameter, replacing <pathtodebuglog> with an appropriate path for your OS, or review the instructions for docker containers:

-d "<pathtodebuglog>"

Please be sure to remove any PII or sensitive information before sharing!
The debug log will log raw request headers and bodies, so that we can replay these against Azurite using REST and create tests to validate resolution.

Have you found a mitigation/solution?

No

[jezzsantos]

This was working just fine in Azure Storage Emulator prior to moving to Azurite.

[XiaoningLiu]

Hi @jezzsantos , can you help with Azurite debug.log?

[fuzzlebuck]

Also experiencing this issue.

Version 3.16.0

Debug log attached.

My test was as follows:

1. Writes entity with field called TestDouble, value of 42.1
2. Writes entity with field called TestDouble, value of 42
3. Reads 2nd entity, which is returned as 0.

Log shows it's coming back as 42 which is interesting.

Storage explorer shows first entity as Type = Double and 2nd entity as Type = Int32.

Same test works fine against azure storage account.

debug.log

[edwin-huber]

Hi, Thanks for sharing the debug log. I tested this morning against the current main branch of Azurite on Windows 11 with nodejs version 14.15.4. I do not see this behavior in storage explorer. In your debug log I see that the request is for minimal metadata, using the Cosmos Table SDK: RequestHeaders:{"host":"127.0.0.1:10002","accept-charset":"UTF-8","maxdataserviceversion":"3.0;NetFx","accept":"application/json; odata=minimalmetadata","dataserviceversion":"3.0;","x-ms-client-request-id":"2cbbab00-06db-455d-8e9e-bd85866bfce8","user-agent":"Azure-Cosmos-Table/1.0.8 (.NET CLR 6.0.3; Win32NT 10.0.22000.0)"

which means that the data-type for the double value is not returned, and we return without the '.0' suffix. {\"odata.metadata\":\"http://127.0.0.1:10002/devstoreaccount1/$metadata#TestTool/@Element\",\"odata.etag\":\"W/\\\"datetime'2022-03-18T07%3A47%3A59.9140000Z'\\\"\",\"PartitionKey\":\"GenericReadWriteTest\",\"RowKey\":\"a18e5cb5-d0f6-4e47-a1a8-7fb6ff5e4609\",\"TestDouble\":42,\"[email protected]\":\"Edm.DateTime\",\"Timestamp\":\"2022-03-18T07:47:59.9140000Z\"}

[edwin-huber]

I shall look to make Azurite return the point in the double value, as the SDKs are using this to determine entity value type. That should resolve this issue.

[jag43]

I think I'm getting this on Azurite 3.19.0. It seems that azurite stores the row without type metadata

"TotalHolidays": 26,

Then when I change the type in storage explorer it adds this to the json

"[email protected]": "Edm.Double",

And when the table is queried it returns 0, is this the same issue?

[bengarrison]

I'm getting the same issue in c# using Microsoft.Azure.Costmos.Table library, running azurite in docker. Double with .0 is being saved as int32 and being returned as 0 instead of the correct double value.

Tried setting tableClient.DefaultRequestOptions.PayloadFormat = TablePayloadFormat.JsonFullMetadata; but makes no difference.

[JohnDoeKyrgyz]

Any update on this issue? We encountered it today on version 3.26.0.

[EmmaZhu]

Hi @JohnDoeKyrgyz ,

Which SDK are you using when encountering the issue? Could you help to share a repro?

I tried to repro the issue with .Net SDK with code like:

using Azure.Data.Tables;

var serviceClient = new TableServiceClient(
    new Uri("http://127.0.0.1:10002/devstoreaccount1"),
    new TableSharedKeyCredential("devstoreaccount1", "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="));

var tableClient = serviceClient.GetTableClient("anewtable123456");
tableClient.CreateIfNotExists();

var entity = new TableEntity("partitionKey1", "rowKey1");
entity.Add("key1", 54.000);
tableClient.AddEntity(entity);
foreach(var tableEntity in tableClient.Query<TableEntity>(filter: $"PartitionKey eq 'partitionKey1'"))
{
    Console.WriteLine(tableEntity.PartitionKey);
}

Got following response for the query entity request: {"odata.metadata":"http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element","value":[{"odata.etag":"W/\"datetime'2023-10-16T06%3A20%3A57.7124880Z'\"","PartitionKey":"partitionKey1","RowKey":"rowKey1","key1":54.0,"Timestamp":"2023-10-16T06:20:57.7124880Z"}]}

From the response, the issue should have been fixed.

[JohnDoeKyrgyz]

Hi @EmmaZhu,

Thanks for taking a look at this. I was using this, https://www.nuget.org/packages/FSharp.Azure.Storage/, which uses the old https://www.nuget.org/packages/Microsoft.Azure.Cosmos.Table/ package.

Good to know that the issue appears to be fixed.

[rarayudu]

Hi @EmmaZhu ,

I tried with REST apis without any SDK and I am still seeing this issue in Azurite. Here are the sample requests and responses between Azurite, Storage Emulator and Azure Storage. Running latest Azurite and latest node.js (20.9.0)

PUT http://127.0.0.1:10002/devstoreaccount1/billing(PartitionKey='f-0-20231114T044500Z',RowKey='foo')?timeout=15&sv=2018-03-28&sig=AOhz8CPbJxqpZTprwDtVQ0saPZW9iR9ENY1%2BUiTTLdg%3D&se=2025-11-14T04%3A20%3A21Z&srt=sco&ss=bqt&sp=racupwdl

Content-Type: application/json

{"Bucket":0,"UsageType":"StandardConnector","Quantity":20.0,"MeterId":"standardMeterId"}

GET http://127.0.0.1:10002/devstoreaccount1/billing(PartitionKey='f-0-20231114T044500Z',RowKey='foo')?timeout=15&sv=2018-03-28&sig=AOhz8CPbJxqpZTprwDtVQ0saPZW9iR9ENY1%2BUiTTLdg%3D&se=2025-11-14T04%3A20%3A21Z&srt=sco&ss=bqt&sp=racupwdl

Accept:application/json; odata=minimalmetadata

Azurite response: { "odata.metadata": "http://127.0.0.1:10002/devstoreaccount1/$metadata#billing/@Element", "odata.etag": "W/"datetime'2023-11-14T04%3A30%3A00.2294729Z'"", "PartitionKey": "f-0-20231114T044500Z", "RowKey": "foo", "Bucket": 0, "UsageType": "StandardConnector", "Quantity": 20, "MeterId": "standardMeterId", "Timestamp": "2023-11-14T04:30:00.2294729Z" }

Storage Emulator response: { "odata.metadata": "http://127.0.0.1:10002/devstoreaccount1/$metadata#billing/@Element", "odata.etag": "W/"datetime'2023-11-14T04%3A33%3A55.14Z'"", "PartitionKey": "f-0-20231114T044500Z", "RowKey": "foo", "Timestamp": "2023-11-14T04:33:55.14Z", "Bucket": 0, "UsageType": "StandardConnector", "Quantity": 20.0, "MeterId": "standardMeterId" }

Azure Storage response: { "odata.metadata": "https://rarayudutestedi.table.core.windows.net/$metadata#billing/@Element", "odata.etag": "W/"datetime'2023-11-14T04%3A43%3A59.7258572Z'"", "PartitionKey": "f-0-20231114T044500Z", "RowKey": "foo", "Timestamp": "2023-11-14T04:43:59.7258572Z", "Bucket": 0, "MeterId": "standardMeterId", "Quantity": 20.0, "UsageType": "StandardConnector" }

[EmmaZhu]

Thanks @rarayudu for the repro.

The issue is because this line in our code: https://github.com/Azure/Azurite/blob/main/src/table/generated/utils/serializer.ts#L150

Azurite is based on JS language, the Json parser in JS language can only recognize 20.0 as a number, it cannot tell whether it's an integer or a double. Rewriting a Json parser maybe able to fix this issue, but requires large efforts.

There's a way to work around the issue from client side to explicit specify value's type like:

{
"PartitionKey":"partitionKey1",
"RowKey":"rowKey1",
"key1":54.0,
"[email protected]":"Edm.Double"
}

[rarayudu]

Thanks for taking a look @EmmaZhu. If we need to change this on our side, it would require significant changes across different layers of integration on our side and we would like to avoid those changes.

What would be the efforts to write custom parser for double values?

[EmmaZhu]

Hi @rarayudu , to implement a Json parser we'd need to take care of encoding and special characters and guarantee our customized Json parser has totally the same behavior with the official one beside the double parsing. There's large rate to introduce regressions. We'd tend not implement this by ourselves.

Azurite is an open-source project, you can always customize one by yourself. If any help needed on how to build one, we'll always be available to be of your assistant.