gpt-crawler
gpt-crawler copied to clipboard
Help, why can I only climb to the first page of gitbook
I tried to crawl this page and could only crawl one page
https://layerzero.gitbook.io/docs/
The **
pattern covers all subfolders and files from the specified point.
config should be like this:
export const defaultConfig: Config = {
url: "https://layerzero.gitbook.io/docs",
match: "https://layerzero.gitbook.io/docs/**",
maxPagesToCrawl: 10,
outputFileName: "output.json",
};
The
**
pattern covers all subfolders and files from the specified point. config should be like this:export const defaultConfig: Config = { url: "https://layerzero.gitbook.io/docs", match: "https://layerzero.gitbook.io/docs/**", maxPagesToCrawl: 10, outputFileName: "output.json", };
So I tried this also:
export const defaultConfig: Config = {
url: "https://overkillgaming.com",
match: "https://overkillgaming.com/**",
maxPagesToCrawl: 10,
outputFileName: "output.json",
};
Problem is that it crawls the first page and stops. (Wordpress site)
Any resolution for this?
The
**
pattern covers all subfolders and files from the specified point. config should be like this:export const defaultConfig: Config = { url: "https://layerzero.gitbook.io/docs", match: "https://layerzero.gitbook.io/docs/**", maxPagesToCrawl: 10, outputFileName: "output.json", };
So I tried this also:
export const defaultConfig: Config = { url: "https://overkillgaming.com", match: "https://overkillgaming.com/**", maxPagesToCrawl: 10, outputFileName: "output.json", };
Problem is that it crawls the first page and stops. (Wordpress site)
Any resolution for this?
I ran it with your config and got the following result. Are you sure you followed all the steps correctly? output-1.json
The
**
pattern covers all subfolders and files from the specified point. config should be like this:export const defaultConfig: Config = { url: "https://layerzero.gitbook.io/docs", match: "https://layerzero.gitbook.io/docs/**", maxPagesToCrawl: 10, outputFileName: "output.json", };
So I tried this also:
export const defaultConfig: Config = { url: "https://overkillgaming.com", match: "https://overkillgaming.com/**", maxPagesToCrawl: 10, outputFileName: "output.json", };
Problem is that it crawls the first page and stops. (Wordpress site) Any resolution for this?
I ran it with your config and got the following result. Are you sure you followed all the steps correctly? output-1.json
100%, too bad it's not at least 100kb in size though haha. Too small for uploading to chat gpt/open ai