goclone icon indicating copy to clipboard operation
goclone copied to clipboard

Can we control the cloning depth? (e.g., clone only the homepage)

Open AbdallahZerfaoui opened this issue 1 year ago • 1 comments

Question:
Hi! I’m exploring GoClone for cloning specific pages (e.g., cloning just the homepage of https://apple.com without recursively downloading the entire site). Does GoClone support limiting the crawling depth?

For example:

  • A --depth flag to restrict recursion (e.g., --depth=1 for only the homepage).
  • Excluding internal links beyond the root path.

If this isn’t possible today, could you suggest a workaround or confirm if it’s on the roadmap?

Use Case:
I need to clone a website for testing/styling without pulling gigabytes of unrelated content.

Thanks for building this tool!

AbdallahZerfaoui avatar Jan 28 '25 21:01 AbdallahZerfaoui

Do you have an example of logs where the recursion can be seen?

I tested and dont see recursive

goclone https://www.apple.com/
Extracting -->  https://www.apple.com/
Css found --> /api-www/global-elements/global-header/v1/assets/globalheader.css
Extracting -->  https://www.apple.com/api-www/global-elements/global-header/v1/assets/globalheader.css
Css found --> /ac/globalfooter/8/en_US/styles/ac-globalfooter.built.css
Extracting -->  https://www.apple.com/ac/globalfooter/8/en_US/styles/ac-globalfooter.built.css
Css found --> /ac/localnav/9/styles/ac-localnav.built.css
Extracting -->  https://www.apple.com/ac/localnav/9/styles/ac-localnav.built.css
Css found --> /wss/fonts?families=SF+Pro,v3|SF+Pro+Icons,v3
Extracting -->  https://www.apple.com/wss/fonts?families=SF+Pro,v3|SF+Pro+Icons,v3
Css found --> /v/home/cd/built/styles/main.built.css
Extracting -->  https://www.apple.com/v/home/cd/built/styles/main.built.css
Css found --> /autopush/us/fam-gallery/fam-gallery.built.css
Extracting -->  https://www.apple.com/autopush/us/fam-gallery/fam-gallery.built.css
Css found --> /ac/ac-films/7.0.0/styles/modal.css
Extracting -->  https://www.apple.com/ac/ac-films/7.0.0/styles/modal.css
Js found --> /metrics/ac-target/latest/ac-target.js
Extracting -->  https://www.apple.com/metrics/ac-target/latest/ac-target.js
Js found --> /v/home/cd/built/scripts/head.built.js
Extracting -->  https://www.apple.com/v/home/cd/built/scripts/head.built.js
Js found --> /api-www/global-elements/global-header/v1/assets/globalheader.umd.js
Extracting -->  https://www.apple.com/api-www/global-elements/global-header/v1/assets/globalheader.umd.js
Js found --> /metrics/ac-analytics/2.24.0/scripts/ac-analytics.js
Extracting -->  https://www.apple.com/metrics/ac-analytics/2.24.0/scripts/ac-analytics.js
Js found --> /autopush/us/fam-gallery/fam-gallery.built.js
Extracting -->  https://www.apple.com/autopush/us/fam-gallery/fam-gallery.built.js
Js found --> /ac/globalfooter/8/en_US/scripts/ac-globalfooter.built.js
Extracting -->  https://www.apple.com/ac/globalfooter/8/en_US/scripts/ac-globalfooter.built.js
Js found --> /ac/localeswitcher/4/en_US/scripts/localeswitcher.built.js
Extracting -->  https://www.apple.com/ac/localeswitcher/4/en_US/scripts/localeswitcher.built.js
Js found --> /v/home/cd/built/scripts/main.built.js
Extracting -->  https://www.apple.com/v/home/cd/built/scripts/main.built.js
Js found --> /ac/ac-films/7.0.0/scripts/autofilms.built.js
Extracting -->  https://www.apple.com/ac/ac-films/7.0.0/scripts/autofilms.built.js
Js found --> /metrics/data-relay/1.1.4/scripts/data-relay.js
Extracting -->  https://www.apple.com/metrics/data-relay/1.1.4/scripts/data-relay.js
Js found --> /metrics/data-relay/1.1.4/scripts/auto-relay.js
Extracting -->  https://www.apple.com/metrics/data-relay/1.1.4/scripts/auto-relay.js

mesaglio avatar Mar 28 '25 21:03 mesaglio