Bug or Feature proposition on :include and :exclude not exclusive

[nonnenmacher]

The default lambda StopOnExcluded in deserialiser seem to be in the assertion that :include and :exclude are exclusive :options entries.

So if one, build an API that for example take a fields="include_property, !exclude_property" that translate to an options = {:include => [:include_property], :exclude =>[:include_property]}

Then actual code, will not exclude the exclude_property, when it seems legit to have the two options active at the same time (at least it make sense for my use case).

If the code is transformed with a comprehension or res' depending of theoptions` to test, as :

 return input unless options[:options][:exclude] || options[:options][:include]

 #res = props.include?(options[:binding].name.to_sym) # false with include: Stop. false with exclude: go!

 return input if options[:options][:include] && options[:options][:include].include?(options[:binding].name.to_sym)
 return input if options[:options][:exclude] && ! options[:options][:exclude].include?(options[:binding].name.to_sym)

Then everything work as expected.

What do you think of this ?

[apotonick]

I have trouble understanding why one would need both naturally exclusive options at the same time? Can you make a more elaborated example, please? :sunglasses:

[nonnenmacher]

culprit for me is the line problem for me is on line

props = (options[:options][:exclude] || options[:options][:include])

suppose you have something like

options = {
   options: {
      exclude: [:payments],
      include: [:shipments]
   },

   binding: {
       name: 'payments'
   }
}

then props == [:payments] and so the first test that read if options[:options][:include] && res is true because, it is true that options[:options][:include] exists and it is also true that this field payments is in the exclude list props

consequence is that the lambda exit with the

return input if options[:options][:include] && res

So for me the props = (options[:options][:exclude] || options[:options][:include]) work only if the options are exclusive (only one of the :include or :exclude could exists and have a truthy value).

I fall into that specific case, where my :excluded fields are still represented, while with my modification, all is fine.

Did that make sense ?

[nonnenmacher]

My point is that I have added an orthogonal feature, along with what some people do, with a query parameter (like &fields="prop1,!excluded,prop2"), that generate both :include and :exclude arrays.

So for the usage I have, I could have both populated at the same time, and it is not inherently two exclusives options, unless it apply for the same property (where it is fine, that one win over the other)

[apotonick]

Still struggling.. how do you decide what wins over what when you provide both values?

[nonnenmacher]

I guess the real point is not on exclusive or who win, the point is that on the sample I gave, about this 'orders' use case, the actual code does not react as expected (i.e the :payments property is not excluded).

If a property is listed in both :include and :exclude, then I guess, the order of the two test, decide who win and is not of great importance.

Truth is that this modification, make Representable works for me, and if that does not break any existing tests, then I could perhaps write one that fails with this same orders sample, and let you decide, if this is a proper fix/enhancement.

[apotonick]

I understand that, but the problem is speed. My code is optimised and doesn't do any unnecessary lookups because I simply say "either in- or exclude". You might think this doesn't matter, but it totally does - I've spent weeks on making Representable 3.0 faster and all those little hash lookups consume shitloads of time!!! Imagine, the additional lookups are made for every single property on every object when rendering/parsing.

I really would love to understand what you actually want to achieve, can you please explain that requirement a bit more? I'm really struggling to see the use. Thanks!!!

[nonnenmacher]

I understand the speed aspect (as often in the debugger, I am considering how deep the stack trace goes, considering how much work is done, just for the output presentation of filtered Hash to JSON).

My use cases, is I want to have in Grape a standard set of Helpers, one would deals with common usage parameters like : fields inclusion/exclusion, sorting, search refinements...)

One could be a fields=<properties list of fields to include or exclude> where I could express such :include or :exclude list with a simple syntax like ?fields="shipments,-payments" to take my example again.

It will be the same bug? if I just did ?include="shipments"&exclude="payments" just syntactic sugar

Speed apart, it seems to me, that the logic is flawed, unless it is expressively stated in the documentation, that :exclude is having the priority and only if it is empty then the :include will be considered. BUT if this is the case, then your two 'if order need to be inverted (i.e check the second if statement first).

[apotonick]

See, and suddenly understand what you want! :smiley:

[apotonick]

So, ?fields="shipments,-payments" means: show shipments, only?

[nonnenmacher]

it generate two listes passed to Representable options hash :
options = {:include => [:shipments], :exclude => [:payments], :user_options => {....}}

So in a sense, yes it reads *show shipments, but do not include payments`

It is used, to have summary vs detailed version of same kind of objects, for drill-down purposes, in a table for example, avoiding to have several 'summary vs detailed' Representer for the same kind of Objects and associations that are fetched from Representable inner to_hash call for those properties that are in fact associations.

[apotonick]

Ok, but that is a different semantic than "my" :include/:exclude. Mine is on a library level and is very simple to understand since mutually exclusive. Yours is your application requirement specific to local needs.

Now I'm confused again - is that only targeting nested objects, or why do you have to say "exclude A, include B". In my understanding, it's "include only A, skip everything else" or "exclude B, include everything else". Please clarify, my friend, and sorry for being so pedantic - German Precision™. Haha.

[nonnenmacher]

My uses case rose, from some of my API, where list of Things get very slow on complex objects that need to follow associations when presented on a intermediate aggregate type then detailed suite of views (e.g an Angular 3 or more steps drill-down process).

I found some usage patterns, where I started with some &detailed=1 and choose proper Representer, with more or less the same properties, but with different levels of details, for some that are too similar I tended to use my own version of your :include, but when I tried to solve my problems with the proper separations of :options vs :user_options from Representable > 2.4.0, I started replacing those Application to Representer adaptations (e.g using logic in :getter or :if) by using :include / :exclude as it allow me to have less code in Representer (let call it the logic less view analogy ;-)).

At end of the day, 80% of the API endpoint to Representer, do not use logic, but on the remaining 20%, if you consider sorting the proliferation of Representers versus using the power of :include :exclude lists (both at the same time), you end up, writing code on the client side (i.e Angular side), much quicker if you know that you could fine tune, the chain of drill-down from list view, to aggregate view to detailed view, with the same API endpoints, some caching at intermediate levels, and then choosing the proper details level using a well known/debugged Representer and all its models associations.

Sure I could stay with more Representers, but this raise the DRY level, for a burden that could be taken care by the Representer as its fall (IMHO) into his responsibilities, the API endpoint (or Controller if that still make sense) taking care of setting the proper :options or :user_options

Point is perhaps, that from a Representable's user point of view, there is no clear partition of options vs user_options as either he tend (like I did) to use, some protected options that are not swallowed magicaly from the pipeline (like :include, :wrap...), going into his own lambdas (resulting in logic inside :if or :getter where those protected or user options are in a well known state from API down to this specific code) OR start using logic/pipeline level :options you offer and then will try YOUR :include or :exclude like I did.

At some point, they will end up using both at the same time.

I think there is some kind of ambiguity here, and I'm quite sure, that sometimes yourself get in some sort of mental loop when you write some options[:options] ;-) And the consequence is that you are talking about 'My vs Your' when the truth should be on a usage level.

At least it should be outlined very clearly in the documentation, that those options usage are mutually exclusive.

I think leaving the exclusive usage lead to more power and the cost doesn't seem such high from my humble point of view (one more hash access time than actual code).

[apotonick]

Thanks for your explanation! I agree that options are better than a million representers, that's why I introduced that directive.

I still don't get how "include A, exclude B" is interpreted on the representer side, though. Can you give me an example from your API where you show me the include/exclude params and what properties will be used, please? :beers:

[nonnenmacher]

Hi Nick,

sorry to bother again (I had a heart MI problem and come back to where I left in January), I would like to know if you had make any conclusion on that issue.

I have a web application that use the proposed modification (and that use it to control precisely what to return/compute in a drill-down manner)

I would like to know, if this proposed modification add too much hash lookups and had a speed impact, or if it clear enough that the options are mutually ‘exclusive’ and you mind is definitely set on this, so I could find other means (i.e logic on the representer with my own API parameters for this filtering needs).

Regards

Vincent

On Fri, Jan 22, 2016 at 1:21 PM, Nick Sutterer [email protected] wrote:

Thanks for your explanation! I agree that options are better than a million representers, that's why I introduced that directive.

I still don't get how "include A, exclude B" is interpreted on the representer side, though. Can you give me an example from your API where you show me the include/exclude params and what properties will be used, please? [image: :beers:]

— Reply to this email directly or view it on GitHub https://github.com/apotonick/representable/issues/182#issuecomment-173903373 .