monocle
monocle copied to clipboard
change-metrics
Error while running RequestRateLimit with Github provider
I followed to the letter your k8s deployment https://github.com/change-metrics/monocle/blob/master/k8s/README.md
I'm using this versions
quay.io/change-metrics/monocle:1.11.1
I used the github provider
kind: ConfigMap
metadata:
name: monocle-config
labels:
app.kubernetes.io/part-of: monocle
data:
config.yaml: |+
workspaces:
- name: marketing
crawlers:
- name: marketing-crawler
update_since: "2024-07-22"
provider:
github_organization : xxxxxxxx
github_repositories:
- xxxxxxxxxxxxx
github_token: GITHUB_TOKEN
I validated my github token and added to the secret
kubectl create secret generic monocle-secrets --from-file=.secrets -n monocle
but when the crawler starts, I always see this error in the log
2024-07-24 01:20:24 WARNING Lentille.GraphQL:246: Could not fetch the current rate limit {"index":"cbi-marketing","crawler":"marketing-crawler","stream":"TaskDatas"}
2024-07-24 01:20:24 INFO Macroscope.Worker:204: Posting documents {"index":"marketing","crawler":"marketing-crawler","stream":"TaskDatas","count":1}
2024-07-24 01:20:24 WARNING Macroscope.Worker:167: Stream produced a fatal error {"index":"cbi-marketing","crawler":"marketing-crawler","stream":"TaskDatas","err":["2024-07-24T01:20:24.050374029Z",{"contents":{"fetch_error":"parse failure: Error in $: Failed reading: not a valid json value at '<html><body><h1>400Badrequest<h1>'","request":{"body":"{\"operationName\":\"GetRateLimit\",\"query\":\"\\n query GetRateLimit {\\n rateLimit {\\n used\\n remaining\\n resetAt\\n }\\n }\\n \",\"variables\":null}","resp":"<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"}},"tag":"RateLimitInfoError"}]}
2024-07-24 01:20:24 INFO Macroscope.Worker:183: Looking for oldest entity {"index":"cbi-marketing","crawler":"cbi-marketing-crawler","stream":"Changes","offset":0}
2024-07-24 01:20:24 INFO Macroscope.Worker:199: Processing {"index":"marketing","crawler":"marketing-crawler","stream":"Changes","entity":{"contents":"xxxxxxxxx","tag":"Project"},"age":"2024-07-22T00:00:00Z"}
Thanks
Perhaps the API returns 400 Bad request on auth error. Could you check the content of /etc/monocle-secrets/.secrets in the pod to verify the GITHUB_TOKEN is correct?
I verified, and the token is there, and is valid since I can use it to connect to GitHub graphql api using thunder client. but the crawler process returns
Stream produced a fatal error {"index":"cbi-marketing","crawler":"marketing-crawler","stream":"TaskDatas","err":["2024-07-24T01:20:24.050374029Z",{"contents":{"fetch_error":"parse failure: Error in $: Failed reading: not a valid json value at '<html><body><h1>400Badrequest<h1>'","request":{"body":"{\"operationName\":\"GetRateLimit\",\"query\":\"\\n query GetRateLimit {\\n rateLimit {\\n used\\n remaining\\n resetAt\\n }\\n }\\n \",\"variables\":null}","resp":"<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"}},"tag":"RateLimitInfoError"}]}