REST API /tm/output returned SDR dimensions order - should be consistent

[indy-3rdman]

I am currently working on a small sample project to demonstrate how the REST API can be consumed in C#. However, there are two things I have come across that I would appreciate clarification on.

SDR dimensions

Looking at the server log below, the dimensions for predictiveCells seems to be different from the dimensions of activeCells and predictedActiveCells (cell order vs row order). Is this on purpose?

================================
GET HTTP/1.1 /network/0098/region/tm/output/activeCells
  Host: 192.168.10.88:8050
  REMOTE_ADDR: 127.0.0.1
--------------------------------
200 HTTP/1.1
  Content-Length: 29
  Content-Type: text/plain

{type: "SDR(2,8)",data: [9]}

================================
GET HTTP/1.1 /network/0098/region/tm/output/predictedActiveCells
  Host: 192.168.10.88:8050
  REMOTE_ADDR: 127.0.0.1
--------------------------------
200 HTTP/1.1
  Connection: close
  Content-Length: 29
  Content-Type: text/plain

{type: "SDR(2,8)",data: [9]}

================================
GET HTTP/1.1 /network/0098/region/tm/output/predictiveCells
  Host: 192.168.10.88:8050
  REMOTE_ADDR: 127.0.0.1
--------------------------------
200 HTTP/1.1
  Content-Length: 29
  Content-Type: text/plain

{type: "SDR(8,2)",data: [7]}

DELETE

NetworkAPI_REST.md briefly mentions DELETE under "REST Operators". Is this actually implemented?

[breznak]

am currently working on a small sample project to demonstrate how the REST API can be consumed in C#.

cool! That's something @dkeeney would like :champagne:

different from the dimensions of activeCells and predictedActiveCells (cell order vs row order). Is this on purpose?

this seems like a mistake in the API to me. Please commit a fix if you can, or we'll wait for David.

[dkeeney]

different from the dimensions of activeCells and predictedActiveCells (cell order vs row order). Is this on purpose?

Yes this is intensional.
The activeCells and predictedActiveCells outputs are always cells (rather than columns). The bottomUpOutput, and predictiveCells outputs are columns if orColumnOutputs argument is true, otherwise they are cells.

@breznak I think this is what was decided on. Is this correct.

NetworkAPI_REST.md briefly mentions DELETE under "REST Operators". Is this actually implemented?

I only partly finished this....good catch. I will finish this.

[breznak]

The bottomUpOutput, and predictiveCells outputs are columns if orColumnOutputs argument is true, otherwise they are cells.

yes, that's correct. but imho the "problem" is about why the two return sdr size (8,2) while

GET HTTP/1.1 /network/0098/region/tm/output/predictiveCells

returns sdr (2,8)

Couldn't the dimensions order be the same?

[dkeeney]

Oh, that seems to be a bug. I would guess it would be something in tm_->cellsToColumns(predictive);

[dkeeney]

I will take the REST DELETE command as a separate PR.

[dkeeney]

#830 takes care of the DELETE operator part of this Issue.

[indy-3rdman]

Thank you very much @dkeeney for taking care of this! I've just uploaded the first C# examples to https://github.com/indy-3rdman/htm.core-csharp-rest-examples

[dkeeney]

Your C# examples are really cool. Very good. 🥇 This is much simpler than the C# interface I was going to try to build using direct calls from C# directly into the C++ htm.core library.

[dkeeney]

With this latest round of api changes, all results are returned in a JSON field 'result' and the type and dimension info was removed. For an SDR the results are returned as sparse indexes and all other types of arrays are dense. Maybe that is not what we want.

For Region input and output, the data is taken from an Array object. The Array object has type and size info that it could also provide. In addition, we can get the dimensions. So what make since for returning this info? Or, do we need anything other than the data.

Suggestion:

   {"result": {"type": <type name>, "dim": [x, y], "data":  [ <the array elements in dense format> ] }}

The type names are found in enum NTA_BasicType of src/htm/types/Types.hpp In the 'data' field we could use '[ ]'s around each row for a multi-dimensional object such that a JSON parse should result in the arrays being constructed correctly. In such case do we need the dim's?

Do we want this layout for returned parameter values as well?

[breznak]

returning this info? Or, do we need anything other than the data.

I;m not saying the above is a bad idea, just for the issue mentioned, can't we just fix the order of dims returned by the region (and be done with it?): https://github.com/htm-community/htm.core/issues/828#issuecomment-638868759

[dkeeney]

The dim info was removed. It now only returns the data.

[breznak]

For an SDR the results are returned as sparse indexes and all other types of arrays are dense. Maybe that is not what we want.

optional, can all returned data be sparse, or we don't want that for some reason?

The Array object has type and size info that it could also provide. In addition, we can get the dimensions. So what make since for returning this info? Or, do we need anything other than the data

oh, ok, thanks for explanation. Now I understand it better. Then your idea with adding a "dims" field to the returned data looks good to me :+1:

In the 'data' field we could use '[ ]'s around each row for a multi-dimensional object such that a JSON parse should result in the arrays being constructed correctly.

do we need that? We could pass just a flattened array as data. And user could use SDR.reshape() to get back exactly the original multi-dim SDR.

[dkeeney]

can all returned data be sparse

and

We could pass just a flattened array as data. And user could use SDR.reshape() to get back exactly the original multi-dim SDR.

These may not be SDR's. Remember that the inputs and outputs are from the Array object which could be an SDR or it could be just an array of any type. For example, input to an encoder where 'sparse' does not make since.

[dkeeney]

@indy-3rdman What JSON structure is easiest to decode directly into a C# object of some sort?

[dkeeney]

@Lupino This will affect you as well because it would mean changing the response JSON syntax yet again. Do you have any input to this?

[dkeeney]

Another alternative:

   {"result": <data, scalar or array>,  "type": <one of the type codes>, "dim": [x,y,z]}

This would not affect @Lupino's client unless he needed the type or size for some reason.

I am thinking that all array's should be dense format to be consistent, even for SDR's however for binary arrays like SDR it is not efficient. What is the consensus on that?

[dkeeney]

Since there was no response to my question, I implemented the response as follows:

 {"result": <data, scalar or array>,  "type": <one of the type codes>, "dim": [x,y,z]}