Issues with paths when calling `steps.resource_add`

[tresoldi]

I am working on automating the releases of my team's project, and have been unable to deal with paths, base_paths, and trusted paths. I am using version 4.40.8 and I am not sure whether it something on my side or a known issue that will be corrected in 5.0. Looking at previous GitHub issues did not help, unfortunately.

In a minimal example, I have a YAML descriptor for countries.csv, which resides in the same directory:

path: countries.csv
name: countries
profile: tabular-data-resource
scheme: file
(...)

In code, we have the following (trimming to a minimal example):

package = Package(resources=[])

for entry in config:
  package = transform(
    package,
    steps=[
      steps.resource_add(Resource(entry["source"]))
    ])

The task must be perform in code and we need to add resources one by one, because it is part of a pipeline reading different kinds of transformations from a set of instructions.

The code is run both by the user and by tests, from different directories. entry["source"] holds only the filename without path (e.g., countries.yaml). If I run the code as above, I get a "no such file or directory" for the YAML file, as expected. Setting basepath (no underscore) within Resource seems to have no effect, and neither does setting base_path (underscore) within .resource_add.

Giving the full path to Resource (e.g., "/home/tresoldi/data/"+entry["source"] works, but the frictionless framework then fails to find countries.csv. Setting the full path in the YAML file (which would be a bad solution), also fails because the path is unsafe, and raises the same error when I set a trusted=True flag in Resource or a trusted: true one in the YAML file.

What is the correct way to deal with this issue? I also tried setting a base path to the Package, with no luck, but it seems I am missing how to specify a base or working path for the package as a whole. Thank you very much!

[tresoldi]

And as frequently happens, I seem to have found the solution after posting my issue... If anybody stumbles upon a similar problem, it is necessary both set the basepath in the Package and to give the full path in the Resource:

package = Package(resources=[], basepath="/home/username/data/")

for entry in config:
  package = transform(
    package,
    steps=[
      steps.resource_add(Resource("/home/username/data/"+entry["source"]))
    ])

The reason being that the package's basepath is not used for the path argument in Resource, which makes sense.

I'm keeping the issue open in case the maintainers want to add/correct something.

[roll]

Thanks @tresoldi!

We need to highlight it in the docs