prashantghimire
I can't make this work with FC 25 (sofifa)
This is a great tool but i can't make it work. Any chance of you guys uploading the latest databse?
What error are you getting, also please use node and npm mentioned in the readme file.
Sorry, i'm getting a few different errors trying on Ubuntu and Windows. Obviously i'm not a pro on this topic. Can anyone upload an updated database please? You can close this topic since I won't be having enough time to look for a solucion these days unfortunately
I was able to fix all the errors and now it's running perfectly. Thanks for your work, the only thing i'm seeing different from FC24 are the names. Again, thanks for your work! You can close this ticket
I was able to fix all the errors and now it's running perfectly. Thanks for your work, the only thing i'm seeing different from FC24 are the names. Again, thanks for your work! You can close this ticket
hello my friend can you upload your data or your code? Im new in programming and i need this data.
I was able to fix all the errors and now it's running perfectly. Thanks for your work, the only thing i'm seeing different from FC24 are the names. Again, thanks for your work! You can close this ticket
hello my friend can you upload your data or your code? Im new in programming and i need this data.
Hi, yeah sure. I have the last data available, I can share it, do you want to give me your email?
Yeah of course. [email protected]
I was able to fix all the errors and now it's running perfectly. Thanks for your work, the only thing i'm seeing different from FC24 are the names. Again, thanks for your work! You can close this ticket
hello my friend can you upload your data or your code? Im new in programming and i need this data.
Hi, yeah sure. I have the last data available, I can share it, do you want to give me your email?
Of course, [email protected]
Hi @FJFuture and @kougeo25 , unfortunately I'm not able to run the code, despite downloading everything, I'm running in several issues, but don't have the knowledge to fix them. Maybe, you could help me out with a copy of the dataset too? Best would be for the first version (07.08.), but of course I'll take any available.
My e-mail is: [email protected]
thanks a lot in advance to either for helping me! :)
Hi @FJFuture and @kougeo25 , unfortunately I'm not able to run the code, despite downloading everything, I'm running in several issues, but don't have the knowledge to fix them. Maybe, you could help me out with a copy of the dataset too? Best would be for the first version (07.08.), but of course I'll take any available.
My e-mail is: [email protected]
thanks a lot in advance to either for helping me! :)
I can make a copy, but also I can try help you to make it work so you can use it whenever you want. My discord is: brivalens, you can add me there if you want and we can keep talking about this.
I want to let a huge big up for Prashant Ghimire who created this awesome tool.
Hi friend @FJFuture, I want to do some analysis and prediction using this dataset, but I don't know much about web scraping. Could you please send me the FC25 dataset?
This is my email:[email protected]
Much thanks
Hi friend @FJFuture, I want to do some analysis and prediction using this dataset, but I don't know much about web scraping. Could you please send me the FC25 dataset?
This is my email:[email protected]
Much thanks
Ok, I can do that. What database do you need? The latest one?
The latest one is OK.
By the way, the data is used for my undergraduate graduation project. So if you would like to tell me your name, I'll mention your help in my paper acknowledgements. …
Sent! Sorry for the delay
First, here's a Dockerfile for getting it to run with older node/npm versions
FROM node:18.12.1
RUN npm install -g [email protected]
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["sh", "-c", "if [ \"$RUN_FULL\" = \"true\" ]; then npm run download-urls && npm run full; else npm run test; fi"]
and then to build
docker build -t sofifa-web-scraper .
and to run the test
docker run --rm sofifa-web-scraper
and to run the full thing:
docker run --rm -e RUN_FULL=true sofifa-web-scraper
Second, if you want to get the "newest" data only, you don't need the trailing integer on each URL. By default, without a trailing integer, the server returns the most recent.
You can use in-place editing of the URL files before running, with sed (Mac OS version)
sed -i '' -E 's|(https://sofifa\.com/player/[^/]+/[^/]+)/[0-9]+/|\1/|g' files/player-urls-test.csv
sed -i '' -E 's|(https://sofifa\.com/player/[^/]+/[^/]+)/[0-9]+/|\1/|g' files/player-urls-full.csv
You need to rebuild the container if you change any files
docker build -t sofifa-web-scraper .
This all works, until this error is thrown:
/app/services/scraper.js:28
throw new Error(`Error reading page=${url}, statusCode=${response.statusCode}`);
^
ReferenceError: response is not defined
at getPageContent (/app/services/scraper.js:28:62)
at async getPlayerDetailsCsvRow (/app/services/parser.js:22:18)
at async download (/app/main.js:23:19)
at async start (/app/main.js:36:9)
Node.js v18.12.1
Hi there, Indeed there is a slight mistake in the code that doesn't handle errors well since the response is defined inside the while loop above and accessed outside of it when error occurs. This can be easily fixed by defining a lastResponse variable outside of the loop and updating it whenever a response is received. This way, the error handling will have access to the last response status code.
const getPageContent = async (url) => {
let attempts = 5;
let lastResponse = null;
while (attempts > 0) {
const response = await humanoid.get(url);
// rest of the code
}
throw new Error(`Error reading page=${url}, statusCode=${lastResponse.statusCode}`);
};
This should return the error, try it and let us know what you get :)
@philshem @SolideSpoke @FJFuture @qianbai688 Please check the latest Playwright implementation. This has been more effective against the the errors folks were facing.