Automatically follow links and download content?

[kyrsquir]

I have a lot of links like /api/content/[content_id].[content_type] (images and pdfs in my case) which fetch binary files from the API. Currently this content is served dynamically along with generated static app. I want to automatically download it and save to, say, /static folder during the generation process.

Do I have to write a function for that which manually replicates the file structure in order to do that and then run it at some of the build hooks (and then proxying /api/content URLs into /static when serving), or is there at least a theoretical way to retrieve all these links during generation and make Node follow them, download and store the files mimicking the original url while rewriting the links in the process?

If what I want is theoretically possible, but cannot be implemented right now with some configuration, I might consider digging into this and possibly making a PR at some point in case the idea makes sense.

[pimlie]

You could add a build:before or generate:before hook to download these files. Then in your component you could use a dynamic import statement, depending on further config you have, Webpack will then add all these files as static assets to your dist. See here for configuration options through magic comments.

[kyrsquir]

You could add a build:before or generate:before hook to download these files.

Downloading the files in advance won't really work as there is neither a comprehensive list of files to download nor a good way to build such a list with exactly the needed files before the site is generated. But perhaps there is a way for generator to access the already built routes and retrieve a list of urls from it?

[pimlie]

Can you add some example code that shows how and when you determine which content to download?

[kyrsquir]

I have two dynamic urls, each used many times with different parameters:

data: ->
  src: "/api/content/cover/#{if @issue.feature then @issue.feature else @issue.year}/#{@issue.order_within_year}"

data: ->
  if isArticle
    url = "/api/content/pdf/#{@year}/#{@orderWithinYear}/#{@orderWithinIssue}/#{@orderWithinSection}"
  else
    url = "/api/content/pdf/#{@year}/#{@orderWithinYear}"
  url: url

In case that matters, they are being set to <img> and <a> tags, respectively.