gpt4free
gpt4free copied to clipboard
xtekky
phind.com | how to set cf_clearance and user_agent
did u guys set the correct user_agent from your browser ?
Yeah, actually i set my agent from whatismybrowser.com
@zjrwtx we need to set user_agent only in the file streamlit_app.py ?
and is there a need to set any value in cf_clearance in the same file?
did u guys set the correct user_agent from your browser ?
Can you please tell me where exactly i can set the user_agent from my browser?
and where if im using phind should i insert this code? `# if using CompletionModel.load set these ora.user_id = '...' ora.session_token = '...'
model = ora.CompletionModel.load(chatbot_id, 'gpt-4')
I too get the same error:
Following are my settings in streamlit_app.py:
phind.user_agent = '<set using https://whatmyuseragent.com>'
phind.cf_clearance = ''
Main things are: make sure your using the CORRECT useragent, 2 make sure your getting the clearance token FROM THE API SITE aka if its from phind go to the phind API and get the clearance and lastly IF ITS ON A REMOTE SERVER like a vps the clearance WILL NOT WORK because the IP has to match, make sure to use a proxy server hosted on that VPS then get the clerance with that proxy
Bare with me please, i got my useragent and put it in phind.user_agent '" and im on phind.com, how am i supposed to get my clearence token to fill in the phind.cf_clearence = i suppose ?
Main things are: make sure your using the CORRECT useragent, 2 make sure your getting the clearance token FROM THE API SITE aka if its from phind go to the phind API and get the clearance and lastly IF ITS ON A REMOTE SERVER like a vps the clearance WILL NOT WORK because the IP has to match, make sure to use a proxy server hosted on that VPS then get the clerance with that proxy
so, i got my useragent from whatismybrowser, and got cf_clearance from phind.com (cookies). Still same error. Am I wrong somewhere ?
Bare with me please, i got my useragent and put it in phind.user_agent '" and im on phind.com, how am i supposed to get my clearence token to fill in the phind.cf_clearence = i suppose ?
You get it using the chrome dev tools and going to cookies
Thank you for showing me this, I got my cf_clearance and i got the user id from whatmyuseragent.com and filled them in the right places in the streamlit_app.py running it again i get this new error : An error occured, please make sure you are using a cf_clearance token and correct useragent | 'rawBingResults'
@maxolie00 , I got it working jut after couple of refreshes Thanks @localuser-isback for your help :)
@maxolie00 , I got it working jut after couple of refreshes Thanks @localuser-isback for your help :)
Same, now working after refreshes too, thanks all
Hi, phind in their panick are sometimes disabling and re-enabling cloudflare, which may require you to not set it and to set it, please refer to the comment of @localuser-isback
"make sure your using the CORRECT useragent, 2 make sure your getting the clearance token FROM THE API SITE aka if its from phind go to the phind API and get the clearance and lastly IF ITS ON A REMOTE SERVER like a vps the clearance WILL NOT WORK because the IP has to match, make sure to use a proxy server hosted on that VPS then get the clerance with that proxy"
you can also just use chatgpt and ask: "how do I get the user agent of my browser" and "how do I see the cookies a site is using"
Bare with me please, i got my useragent and put it in phind.user_agent '" and im on phind.com, how am i supposed to get my clearence token to fill in the phind.cf_clearence = i suppose ?
You get it using the chrome dev tools and going to cookies
Thanks, I managed to get this running. My question is, do i need to get the cf_clearance token everytime i start the streamlit app?
Why is there only “https://www.phind.com/” in my cookies ，and I don't see the cf_clearance
can i use the chai.openai.com cf_clearance? I can see two cf_clearance from the cookies
EDIT 2: https://github.com/xtekky/gpt4free/issues/153 - The use of phind.com is now forbidden
EDIT 1: This is just a band-aid solution to make it work. Also, please keep in mind that the cf_clearance expires and you need to refresh the page to get a new one.
Here:
Step 1: Go to phind.com and get your cf_clearance via inspect element -> application -> storage -> cookies (see the images of others above) -- They expire.
Step 2: Get your user-agent here
Step 3: Edit your phind folder's __init__.py (basically copy this and follow through. I've removed some of the lines, you may compare them later with the original code.
- Manually paste your
cf_clearanceon lines:79,139, and203 - Manually paste your
user-agenton lines:88,146, and212
from urllib.parse import quote
from time import time
from datetime import datetime
from queue import Queue, Empty
from threading import Thread
from re import findall
from curl_cffi.requests import post
cf_clearance = '' # No need to put anything here (gets wiped)
user_agent = '' # No need to put anything here (gets wiped)
class PhindResponse:
class Completion:
class Choices:
def __init__(self, choice: dict) -> None:
self.text = choice['text']
self.content = self.text.encode()
self.index = choice['index']
self.logprobs = choice['logprobs']
self.finish_reason = choice['finish_reason']
def __repr__(self) -> str:
return f'''<__main__.APIResponse.Completion.Choices(\n text = {self.text.encode()},\n index = {self.index},\n logprobs = {self.logprobs},\n finish_reason = {self.finish_reason})object at 0x1337>'''
def __init__(self, choices: dict) -> None:
self.choices = [self.Choices(choice) for choice in choices]
class Usage:
def __init__(self, usage_dict: dict) -> None:
self.prompt_tokens = usage_dict['prompt_tokens']
self.completion_tokens = usage_dict['completion_tokens']
self.total_tokens = usage_dict['total_tokens']
def __repr__(self):
return f'''<__main__.APIResponse.Usage(\n prompt_tokens = {self.prompt_tokens},\n completion_tokens = {self.completion_tokens},\n total_tokens = {self.total_tokens})object at 0x1337>'''
def __init__(self, response_dict: dict) -> None:
self.response_dict = response_dict
self.id = response_dict['id']
self.object = response_dict['object']
self.created = response_dict['created']
self.model = response_dict['model']
self.completion = self.Completion(response_dict['choices'])
self.usage = self.Usage(response_dict['usage'])
def json(self) -> dict:
return self.response_dict
class Search:
def create(prompt: str, actualSearch: bool = True, language: str = 'en') -> dict: # None = no search
if not actualSearch:
return {
'_type': 'SearchResponse',
'queryContext': {
'originalQuery': prompt
},
'webPages': {
'webSearchUrl': f'https://www.bing.com/search?q={quote(prompt)}',
'totalEstimatedMatches': 0,
'value': []
},
'rankingResponse': {
'mainline': {
'items': []
}
}
}
headers = {
'authority': 'www.phind.com',
'accept': '*/*',
'accept-language': 'en,fr-FR;q=0.9,fr;q=0.8,es-ES;q=0.7,es;q=0.6,en-US;q=0.5,am;q=0.4,de;q=0.3',
'cookie': f'cf_clearance=PASTEHERE',
'origin': 'https://www.phind.com',
'referer': 'https://www.phind.com/search?q=hi&c=&source=searchbox&init=true',
'sec-ch-ua': '"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"macOS"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': '' # PASTE HERE
}
return post('https://www.phind.com/api/bing/search', headers = headers, json = {
'q': prompt,
'userRankList': {},
'browserLanguage': language}).json()['rawBingResults']
class Completion:
def create(
model = 'gpt-4',
prompt: str = '',
results: dict = None,
creative: bool = False,
detailed: bool = False,
codeContext: str = '',
language: str = 'en') -> PhindResponse:
if results is None:
results = Search.create(prompt, actualSearch = True)
if len(codeContext) > 2999:
raise ValueError('codeContext must be less than 3000 characters')
models = {
'gpt-4' : 'expert',
'gpt-3.5-turbo' : 'intermediate',
'gpt-3.5': 'intermediate',
}
json_data = {
'question' : prompt,
'bingResults' : results, #response.json()['rawBingResults'],
'codeContext' : codeContext,
'options': {
'skill' : models[model],
'date' : datetime.now().strftime("%d/%m/%Y"),
'language': language,
'detailed': detailed,
'creative': creative
}
}
headers = {
'authority': 'www.phind.com',
'accept': '*/*',
'accept-language': 'en,fr-FR;q=0.9,fr;q=0.8,es-ES;q=0.7,es;q=0.6,en-US;q=0.5,am;q=0.4,de;q=0.3',
'content-type': 'application/json',
'cookie': f'cf_clearance=PASTEHERE',
'origin': 'https://www.phind.com',
'referer': 'https://www.phind.com/search?q=hi&c=&source=searchbox&init=true',
'sec-ch-ua': '"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"macOS"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': '' # PASTE HERE
}
completion = ''
response = post('https://www.phind.com/api/infer/answer', headers = headers, json = json_data, timeout=99999, impersonate='chrome110')
for line in response.text.split('\r\n\r\n'):
completion += (line.replace('data: ', ''))
return PhindResponse({
'id' : f'cmpl-1337-{int(time())}',
'object' : 'text_completion',
'created': int(time()),
'model' : models[model],
'choices': [{
'text' : completion,
'index' : 0,
'logprobs' : None,
'finish_reason' : 'stop'
}],
'usage': {
'prompt_tokens' : len(prompt),
'completion_tokens' : len(completion),
'total_tokens' : len(prompt) + len(completion)
}
})
class StreamingCompletion:
message_queue = Queue()
stream_completed = False
def request(model, prompt, results, creative, detailed, codeContext, language) -> None:
models = {
'gpt-4' : 'expert',
'gpt-3.5-turbo' : 'intermediate',
'gpt-3.5': 'intermediate',
}
json_data = {
'question' : prompt,
'bingResults' : results,
'codeContext' : codeContext,
'options': {
'skill' : models[model],
'date' : datetime.now().strftime("%d/%m/%Y"),
'language': language,
'detailed': detailed,
'creative': creative
}
}
headers = {
'authority': 'www.phind.com',
'accept': '*/*',
'accept-language': 'en,fr-FR;q=0.9,fr;q=0.8,es-ES;q=0.7,es;q=0.6,en-US;q=0.5,am;q=0.4,de;q=0.3',
'content-type': 'application/json',
'cookie': f'cf_clearance=PASTEHERE',
'origin': 'https://www.phind.com',
'referer': 'https://www.phind.com/search?q=hi&c=&source=searchbox&init=true',
'sec-ch-ua': '"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"macOS"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': '' # PASTE HERE
}
response = post('https://www.phind.com/api/infer/answer',
headers = headers, json = json_data, timeout=99999, impersonate='chrome110', content_callback=StreamingCompletion.handle_stream_response)
StreamingCompletion.stream_completed = True
@staticmethod
def create(
model : str = 'gpt-4',
prompt : str = '',
results : dict = None,
creative : bool = False,
detailed : bool = False,
codeContext : str = '',
language : str = 'en'):
if results is None:
results = Search.create(prompt, actualSearch = True)
if len(codeContext) > 2999:
raise ValueError('codeContext must be less than 3000 characters')
Thread(target = StreamingCompletion.request, args = [
model, prompt, results, creative, detailed, codeContext, language]).start()
while StreamingCompletion.stream_completed != True or not StreamingCompletion.message_queue.empty():
try:
chunk = StreamingCompletion.message_queue.get(timeout=0)
if chunk == b'data: \r\ndata: \r\ndata: \r\n\r\n':
chunk = b'data: \n\n\r\n\r\n'
chunk = chunk.decode()
chunk = chunk.replace('data: \r\n\r\ndata: ', 'data: \n')
chunk = chunk.replace('\r\ndata: \r\ndata: \r\n\r\n', '\n\n\r\n\r\n')
chunk = chunk.replace('data: ', '').replace('\r\n\r\n', '')
yield PhindResponse({
'id' : f'cmpl-1337-{int(time())}',
'object' : 'text_completion',
'created': int(time()),
'model' : model,
'choices': [{
'text' : chunk,
'index' : 0,
'logprobs' : None,
'finish_reason' : 'stop'
}],
'usage': {
'prompt_tokens' : len(prompt),
'completion_tokens' : len(chunk),
'total_tokens' : len(prompt) + len(chunk)
}
})
except Empty:
pass
@staticmethod
def handle_stream_response(response):
StreamingCompletion.message_queue.put(response)
P.S. Improvements here is probably setting the keys via streamlit's implementation of session_state kinda looks like the keys get wiped every call for some reason
@arianyambao is it working normally for you as of this moment? im still getting the error
i examined the response, it returns an html page, looks like its not passing the cloudflare captcha
Im also getting the same error, "An error occured, please make sure you are using a cf_clearance token and correct useragent | Expecting value: line 1 column 1 (char 0)"
I've made sure my useragent and cf_clearance are both filled in correctly as well
My solution was to set phind.cf_clearance and phind.user_agent on lines 4 and 5 in streamlit_app.py
@jarvisar Thank you, and you're right. But after trying several times, the similar errors still occur and you must gain cf_clearance and copy it again. It doesn't seem very convenient.
@jarvisar Thank you, and you're right. But after trying several times, the similar errors still occur and you must gain cf_clearance and copy it. It doesn't seem very convenient.
Yes, the clearance, similar to cookies expires and gets renewed...