dynamodb-toolbox icon indicating copy to clipboard operation
dynamodb-toolbox copied to clipboard

Custom parse method for attributes

Open bboure opened this issue 3 years ago • 3 comments

Hi and thanks for this awesome not-ORM 🙂

I was wondering if we could introduce a custom parse method for Entities' attributes. Basically, I need something that is the opposite of transform when an item is read from DynamoDB.

It could be useful if you want to store a value differently in DynamoDB than in your application.

A couple of examples:

  • store large texts as gzipped binaries Use transform to gzip the value, and parse to decompress it.

  • add/remove padding zeroes Sometimes, you want to store numbers as zero-padded strings. For example, if you want to index them in a string SK. Use transform to pad the value and parse to remove it and transform the value back into a number.

  • base64 encode/decode.

I'd be happy to open a PR and I am open to suggestions/ideas.

Thank you!

bboure avatar Mar 14 '21 10:03 bboure

Hi @bboure. Good thoughts.

The second use case is (kind of) supported using defaults. You need to store the number separately, but default can have a function to pad the string.

Storage is generally cheaper than compute, so this isn't an anti-pattern (data duplication). In fact, it is generally recommended to seperate your data and your keys and avoid parsing.

Your first and last use cases aren't supported out of the box but definitely can be done "manually" in your data access code.

I can see the benefits of a transform and parse method for each attribute (though we must remember that storage is cheaper than compute in serverless so you wouldn't want to abuse it).

I do wonder whether this should be a responsibility of this library though. Feature bloat makes it hard to maintain and it would be easy to implement in your data access layer.

darbio avatar Mar 14 '21 21:03 darbio

Thank you for your feedback @darbio

Storage is generally cheaper than compute, so this isn't an anti-pattern (data duplication). In fact, it is generally recommended to separate your data and your keys and avoid parsing.

You're absolutely right! In fact, this is exactly what I do. I was just trying to find examples to illustrate what I was suggesting :) My real use-case is the first one (gzip/ungzip content). That is because I have potentially very large items that will need to be compressed for cost optimization (less RCU, WCU and storage).

I do wonder whether this should be a responsibility of this library though.

Yes, I was wondering that too, but since there is a transform method (this is currently where I gzip the content), I was also wondering why there isn't an "untransform" one.

bboure avatar Mar 15 '21 06:03 bboure

Hi @bboure, I believe this was suggested before, but I like the overall idea.

jeremydaly avatar Apr 16 '21 14:04 jeremydaly

Hey @bboure, this was released in v0.5 😎

You can now use the format function to achieve what you wanted.

naorpeled avatar Nov 14 '22 08:11 naorpeled