Compatibility with Frictionless lib pattern

[rufuspollock]

This is looking awesome 👏 👏 @roll

A few thoughts on compatibility with our agreed RFC https://github.com/frictionlessdata/project/blob/master/rfcs/0004-frictionless-data-lib-pattern.md (which we can tweak). I haven't reviewed thoroughly so this incomplete and for discussion ...

• File:
  • All metadata should go into descriptor attribute rather than direct on object except for computed attributes e.g. hash, size (I'm still not sure about computed attributes). so you have File.descriptor wdyt?
  • stream and buffer rather than read_... functions
• Table: i like the Table object distinct from File though it means that rows only makes sense here ...

also a question

• File vs Resource as naming for core object

[roll]

Hi @rufuspollock, thanks!

Regarding Frictionless Lib Pattern. Haven't we agreed that all the libraries/drivers/SDKs should follow it if possible e.g. tableschema-py/datapackage-rb/etc and high-level frameworks should be as pythonic/javascriptic/etc as possible?

I think that high-level frameworks should embrace the best from a targeted language. For example, in Python it's possible to subclass dicts which led to solving various metadata problems like this - https://github.com/frictionlessdata/datapackage-py/issues/213 and dropping those annoying package/resource.commit() calls, etc. In JavaScript or R it will probably be a different way to work with metadata to make it optimal/native for those platforms. Any unifications here will degrade the overall user-experience I believe.

On the other hand, for drivers/SDKs which if for low-level integrators unification should be a great idea. And for them, we have Frictionless Lib Pattern. At least it's how I understood it.

File vs Resource as naming for core object

I would follow the Specs here i.e. using resource WDYT? I did a lot to make the framework as much complaint with the Specs as possible e.g. using the Dialect concept etc.

Regarding the current frictionless.File class. I dropped a lot of useless abstractions we had with layered tabulator/tableschema/datapackage/goodtablbes but still had to preserve some extra classes just to make the whole thing work (you know "We can solve any problem by introducing an extra level of indirection."). TBH I would be happy if e.g. we can merge File and Resources classes at some point. It just needs to be thought through really closely. Another question, that it's kinda a lower level and I would rather focus e.g. on visual application work which should be more important for the project than low-lever refactorings.

PS. I found the doc from our meeting

[roll]

Closing for now as per the meeting discussion

[rufuspollock]

@roll could you summarize a bit how this was closed in the meeting discussion - i thought this was still open 😉

I had not had a chance to respond yet to your comment so i should do that now:

Regarding Frictionless Lib Pattern. Haven't we agreed that all the libraries/drivers/SDKs should follow it if possible e.g. tableschema-py/datapackage-rb/etc and high-level frameworks should be as pythonic/javascriptic/etc as possible?

Yes ... and frankly looking at frictionless-py it includes the low level driver library so i suspect people will just use that unless you plan to factor out the low level classes ... so that means the core classes here should follow the frictionless lib pattern if we can.

I think that high-level frameworks should embrace the best from a targeted language. For example, in Python it's possible to subclass dicts which led to solving various metadata problems like this - frictionlessdata/datapackage-py#213 and dropping those annoying package/resource.commit() calls, etc. In JavaScript or R it will probably be a different way to work with metadata to make it optimal/native for those platforms. Any unifications here will degrade the overall user-experience I believe.

I think it is fine to adapt in minor ways like this.

On the other hand, for drivers/SDKs which if for low-level integrators unification should be a great idea. And for them, we have Frictionless Lib Pattern. At least it's how I understood it.

See my first point - i don't see a "low level" driver library for python atm separate from frictionless lib given that it includes the core classes and methods (or are you planning to factor that out)?

The key point

Could we have the core API here follow the frictionless lib pattern ie. as doc'd in here https://github.com/frictionlessdata/frictionless-py/blob/master/docs/target/api-reference/README.md

It seems to me we are already very close and a few tweaks would get us there. Happy to chat about this to get consensus.

[roll]

I got it wrong at the meeting. Let's keep it open then :+1:

[rufuspollock]

@roll first, to confirm we are agreed that low level API in this lib (and similarly in frictionless-js lib) would follow RFC 4 (frictionless lib pattern) with appropriate Pythonic or JS adjustments?

Second, based on your experience here are there changes we should make in RFC 4 e.g.

• File rather than Resource? (I agree in preferring File)
• Dataset or Package (i prefer Dataset and plan to rename at the specs level)
• descriptor attribute on File and Dataset to get metadata ... (even if we make metadata attributes available at root level) (Why? So that one can easily retrieve the datapackage.json or dataresource content)

[roll]

@rufuspollock Let's discuss it on a call