sherlock-project
Fix codementor detection
This PR re-enables Codementor site detection by updating [data.json] and removing it from [removed-sites.md]. Codementor now returns 200 for claimed usernames and 404 for unclaimed ones. However, Sherlock currently does not detect the profile even when it exists (ex: "blue"). Please review the detection logic for this site.
Automatic validation of changes
| Target | F+ Check | F- Check |
|---|---|---|
| Codementor | :heavy_check_mark: Pass | :x: Fail |
Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hello @Kavindya-Kariyawasam,
Was Codementor the only site changed in data.json? There seems to be quite a large diff. Could you verify it once please ?
Automatic validation of changes
| Target | F+ Check | F- Check |
|---|---|---|
| Codementor | :x: Fail | :heavy_check_mark: Pass |
Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).
Automatic validation of changes
| Target | F+ Check | F- Check |
|---|---|---|
| Codementor | :x: Fail | :heavy_check_mark: Pass |
Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).
Hi @akh7177 , thanks a lot for pointing that out! It looks like my Prettier extension auto-formatted some parts of the files unintentionally, I am sorry about that. I've cleaned up the changes now and pushed an updated commit. Appreciate your help!
For when the F+ is resolved, there's no need for a dedicated test --- all sites are tested by the validate_targets test module (actually, the same one that's feeding the GitHub Actions comment)
poetry run pytest -q --tb no -rA -m validate_targets --chunked-sites "Codementor"
The -q --tb no -rA are optional of course, just improving output readability.
Skipping --chunked-sites "Codementor" will trigger the test against all sites (-n 20 recommended then, to use additional workers).
Automatic validation of changes
| Target | F+ Check | F- Check |
|---|---|---|
| Codementor | :heavy_check_mark: Pass | :x: Fail |
Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).
Hi @ppfeister , I've removed the dedicated Codementor test script as requested and committed the change. I ran the project’s automated validation tests. The results show that the false positive issue is resolved, but a false negative remains - Sherlock does not detect existing Codementor accounts even though the site returns 200 for claimed and 404 for unclaimed usernames. Please let me know if you'd like me to make any further changes. Thank you for your guidance!
Hi @ppfeister , I've removed the dedicated Codementor test script as requested and committed the change. I ran the project’s automated validation tests. The results show that the false positive issue is resolved, but a false negative remains - Sherlock does not detect existing Codementor accounts even though the site returns 200 for claimed and 404 for unclaimed usernames. Please let me know if you'd like me to make any further changes. Thank you for your guidance!
Hey @Kavindya-Kariyawasam , adding a GET request method seems to solve the issue. Kindly add the below line and lemme know if it fixes it
"request_method": "GET"
Automatic validation of changes
| Target | F+ Check | F- Check |
|---|---|---|
| Codementor | :x: Fail | :heavy_check_mark: Pass |
Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).
@Kavindya-Kariyawasam Hm, that's wierd. Can you see check if its working locally? Or is it returning F+/F-?
Automatic validation of changes
| Target | F+ Check | F- Check |
|---|---|---|
| Codementor | :x: Fail | :heavy_check_mark: Pass |
Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).
@akh7177 I reran the Codementor chunk locally with poetry run pytest -q --tb no -rA -m validate_targets --chunked-sites "Codementor" and both the false-positive and false-negative checks pass (2 passed, 25 deselected in 3.25s). I'll check again.
@akh7177 @ppfeister
poetry run pytest -q --tb no -rA -m validate_targets -n 20 --chunked-sites "Codementor"
This reports "2 passed in 4.17s". I also confirmed Codementor still returns 404 on GET for random usernames and 200 for "@blue". The previous CI failure looks like a transient false positive from Codementor; could you kick off another workflow run? Let me know if anything else pops up.