joplin
joplin copied to clipboard
Clipper: Make resource creation from files faster by executing them in parallel
Changing the implementation of createResourcesFromPaths
of the notes service to create resources in "parallel".
Description
The change affects the process right that happens after the media files links found in the HTML body that is being exported to note have been downloaded to a temporary folder. Now, createResourceFromPaths
is responsible for resizing the images (if necessary) and creating Resource records in the database from the files that were downloaded.
While the current implementation uses asynchronous calls to create resources from created files, we aren't taking all the advantage possible of the host machine if we don't run the code in parallel. This implementation changes the implementation to use Promise.all
to fix that.
To do that, I also changed the data structure exported by the function:
Previously we had an array of objects that had the type:
{
[url: string]: {
path: string // where the downloaded file was stored
originalUrl: string // the original url where the image was downloaded from, same value as the url from the object key
resource: ResourceEntity
},
[url: string]: {
// ...
}
}
To something simpler:
[
{
path: string // where the downloaded file was stored
originalUrl: string // the original url where the image was downloaded from
resource: ResourceEntity
}
]
One possible issue with this approach is that we might have duplicated in the new data structure, but since the mediaUrls used as the argument are unique, this won't happen anyway.
This new data structure required that I changed the removeTempFiles
and replaceUrlsByResources
too.
Testing
I tried to add two basic tests to the function, one that fails to create the image because the file does not exist and the other that we process and create the image as a Resource.