Google API limit ?

[mrtngrsbch]

Hi,

I'm surprised that Google won't let me translate more than 6 files... is this normal? Does it happen to anyone else? It seems reasonable to me that Google limit the API, someone can guide me, please?

Of course, it is not a problem of Scan Tailor, but I can't find where to study the subject. The best thing I can do is to make use of my VPN (Thanks NordVPN !)

This is the command I am using trans :es file://IMG_20200911_215210.txt -o IMG_20200911_215210_ES.txt

Best, Martin

Translate Shell 0.9.6.12

platform Darwin terminal type xterm-256color bi-di emulator [N/A] gawk (GNU Awk) 5.1.0 fribidi (GNU FriBidi) 1.0.10 audio player [NOT INSTALLED] terminal pager less web browser open user locale en_US.UTF-8 (English) home language en source language auto target language en translation engine google proxy [NONE] user-agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 ip version [DEFAULT] theme default init file [NONE]

[mogando668]

google is actually far more generous than what their public API limit states. i'll give u a workaround hint - 6 versus 4.

[ben-geee]

Hi guys, is there any possibility to optionally give an Google API Key in translation shell? Like in any configuration file or via command line option?

Or is there any possibility to add parameters to the GET request which is created by translation shell?

Regards Ben

[PerseusArkouda]

[ERROR] Google did not return results because rate limiting is in effect
[ERROR] Rate limiting

It started happening today and I barely used it at all. So I guess it's dev's API which is rate limited?

[mogando668]

Dev API limit has always been there. They have a more lax limit when not using OAuth tokens.

I’ve made custom bash script and curl query that essentially mimics what the translate shell does but only for google, getting the full JSON API dump back, and benchmarked it to a hard limit of roughly 4500 bytes per 6.5-7.0 seconds (Unicode counted in bytes not chars).

Say UTF8 codepoint of U + 941A. URL encoded becomes %E9 %90 %9A

This would be counted as 3 bytes in that limit, not 1 or 9.

Remember that each new line \ n is one byte too.

And finally, remember that u, in ipv4 and ipv6, aren’t the same person ^^

PerseusArkouda [email protected] 於 2021年1月22日 13:55 寫道：

 [ERROR] Google did not return results because rate limiting is in effect [ERROR] Rate limiting It started happening today and I barely used it at all. So I guess it's dev's API which is rate limited?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[xavierbaez]

@mogando668 So can you share the script?

[xavierbaez]

Hi guys, is there any possibility to optionally give an Google API Key in translation shell? Like in any configuration file or via command line option?

Or is there any possibility to add parameters to the GET request which is created by translation shell?

Regards Ben

Were you able to pass an API key?

[mogando668]

lemme collate that code for u

but I don’t have any oauth2 api key to piggyback upon. The script doesn’t need any of that - and hardly any strange custom http headers. No vpn no obscuring via proxy or smart DNS no back door no side door. Just straight up curl to their public API, the Same URL accessed by translate shell

the main “trick”, which isn’t really one , lies upon the fact that my home ISP gives me a IPv4 and IPv6 address simultaneously, but Google’s servers track their querying frequency and volume as if those were 2 completely independent systems, thus effectively doubling the rate without any special tricks.

I’ll detail them in my email later. you’re free to do whatever u want with that code. it’s not under any sort of license, nor would I ever care about that. I simply believe in sharing.

Regards, Jason K

xavierbaez @.***> 於 2021年5月30日 02:29 寫道：

 Hi guys, is there any possibility to optionally give an Google API Key in translation shell? Like in any configuration file or via command line option?

Or is there any possibility to add parameters to the GET request which is created by translation shell?

Regards Ben

Were you able to pass an API key?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[Geremia]

@ben-geee

is there any possibility to optionally give an Google API Key in translation shell? Like in any configuration file or via command line option?

I'm wondering this, too.

[mogando668]

so sorry for the ultra long delay. my code codes pieces are still a mess. hope you can make any sense out of it. reply here if it's confusing. tsS squeeze spaces. tsL trims end of lines. uniqPB is an alias using mawk to do only get uniq non-empty lines without sorting.

The way it's written now, it looks for the most recent flat text file in the current folder named . smallerM3T_*.txt. Because i use this one liner utility via gnu-parallel to control the jobs.

The small T files are 3 columns, using equal sign "=" ( \ 075 | x 3D ) are a delimiter.

Column one is fixed width alpha-numeric indices - made, column 2 isn't need here , but just a 1-letter language code for E / C / J / K / O(thers). Column 3 will be the text that needs to be translated.

What's already residing inside the need*.txt are literally rows and rows and alpha-numeric indices, using pipe as delimiter. then each time parallel runs this shell function it'll grep those indices along column 1 , take what's indeded to be translated, and safe URL-quote-plus style encodes it before sending it over.

I try to limit them to 4500-pre-url-encoded BYTES per query. Right now the thing is overly batch-optimized but u can see a lot of overhead if you just wanna translate a few lines. I needed to translate 12.5 million lines, 3 langs each translated, for over 37,000,000, so batch was the only way viable at all.

U can see how the code doesn't have anywhere for API key cuz simply none was needed.

`nice cat need*txt | parallel -N 2 -j 1 --timeout 1800 --bar --delay 7.25 --quote zsh -c 'echo >&2; for arg; do echo "${arg}"; done | googleTransBatchLang | paste - ' _ ;`

googleTransBatchLang ()
{
    NL=$'\n';
    currFolder="$( ls -dt "${DT}"/MV80* | gsed -n 1p)";

    inputStuffAll="$( gpaste - | tsS |  gtr ' ' '\n' | tsL |  uniqPB | nonEmpty | gsed -e 's/^[=\|]*//g' | mawk -v VALPRX="${bestKnownProxy2}"  'BEGIN { OFS=FS="="; } (NF >= 3) { print $0, (NR%2) ? "-4" : "-6", NR, "" }' | gtr -s '=' | uniqPB  )";

    lookupFile="$( ls -t "${currFolder}/smallerM3T_"*txt | gsed -n 1p)";

    googleTransURLpfx='https://translate.googleapis.com/translate_a/single?client=gtx&ie=UTF-8&oe=UTF-8&dt=bd&dt=ex&dt=ld&dt=md&dt=rw&dt=rm&dt=ss&dt=t&dt=at&dt=gt&dt=qc';

    for inputStuff in $( echo "${inputStuffAll}" | nonEmpty | gg -F '=' | LC_ALL=C gsort -t= -k 5,5n ); do

         srcLang="$(echo "${inputStuff}" | cut -d= -f 1 )";
            lang="$(echo "${inputStuff}" | cut -d= -f 2 )";
            rows="$(echo "${inputStuff}" | cut -d= -f 3 )";
        ip46Flag="$(echo "${inputStuff}" | cut -d= -f 4 )";

        prxyFlag="";

       ( nice curl -w '\n\n%{url_effective}\n\n' -m 120 -f ${ip46Flag} -g ${prxyFlag} -L \
            \
            -H 'Host: translate.googleapis.com'              \
            -H 'Origin:  https://translate.google.com'       \
            -H 'Referer: https://translate.google.com/'      \
            -H 'Accept-Language: en-us'                      \
            -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15' -o -  \
            \
            "${googleTransURLpfx}&sl=${srcLang}&hl=en&q=$(nice ggrep -E "^(${rows})=" "${lookupFile}" | cut -d= -f 3 | urlencode )&tl=${lang}" | \
            \
            gpaste - | mawk -F= -v VALBEFORE="${NL} at $(date) checked ${ip46Flag} for ${srcLang}-to-${lang} using ip v${ip46Flag}...${NL}${rows}${NL}$" 'BEGIN { OFS=FS="\n"; RS="^$" } END { print (VALBEFORE) ORS ( ((NR<1) || (!($0))) ? "failed trans." : $0 ) ORS; }' ) &

    done | gcat -  ;
    date >&2;
    return;
}

And all dumped into a shared text output file ike

 at Wed Dec  9 06:28:29 EST 2020 checked -4 for en-to-ko using ip v-4...
H01062665|H02391563|H03184477|H00330271|H02165461|H05055153|H05900255|H05212152|H01924094|H05149047|H04229790|H04070049|H00336313|H00868112|H02350180|H01927905|H00959981|H01837927|H06143506|H06144387|H03786604|H00315250|H01770059|H00858803|H04713652|H00219739|H04279722|H01945104|H05053141|H04194442|H01624771|H00868491|H02127297|H05655967|H04408450|H02265092|H02268903|H01759537|H03960872|H01979541|H04386948|H03497529|H01754996|H02114834|H02678737|H03883640|H05122095|H01504223|H03289089|H01945389|H00859962|H06052899|H00301545|H05505578|H01466788|H03285121|H01609092|H05148003|H03218659|H01398322|H04240384|H04979349|H00232123|H03375207|H00170558|H03319579|H00990531|H01652046|H05230828|H04722021|H02308849|H02103412|H01079021|H00719604|H00982666|H04583237|H03550635|H04131225|H01864401|H05785373|H00318657|H01410311|H05837982|H04125532|H04528855|H05920225|H02436113|H03753663|H01775300|H03441644|H03396864|H06488276|H01965950|H01171903|H01945845|H01908329|H01616341|H05586045|H00275966|H05415533|H02102303|H04383102|H01198725|H03260539|H05430653|H01504703|H00367550|H01737861|H03297722|H01925024|H05205116|H02148289|H05928322|H01492433|H06443548|H00780560|H01158909|H04267003|H03388270|H01217325|H01401090|H05484364|H01522217|H01343472|H00196776|H03820337|H06027776|H06294930|H03966915|H03793984|H04538250|H00177766|H02038727|H00319255|H00343042|H01436584|H02344791|H05928517|H05697124|H00301295|H01977851|H01598330|H00285415|H03677380|H01611189|H02938179|H04687166|H01305810
$

[[[", Book I BWV 846-69, No.1 In C, BWV846 : Fugue\n",", Book I BWV 846-69, No.1 In C, BWV846 : Fugue\n",null,null,3,null,null,[[]],[[["e61ab6c84d6353b3a3e8cea91e18aabe","en_ko_2020q2.md"]]]],["-2 막 3 장 : 호이 호! ","- Act II Scene 3 : Hoiho!",null,null,3,null,null,[[]],[[["e61ab6c84d6353b3a3e8cea91e18aabe","en_ko_2020q2.md"]]]],["호이 호호호!\n","Hoihohoho!\n",null,null,3,null,null,[[]],[[["e61ab6c84d6353b3a3e8cea91e18aabe","en_ko_2020q2.md"]]]],["03-Die Eherne Stadt-Teil 152\n","03 - Die Eherne Stadt - Teil 152\n",null,null,3,null,null,[[]],[[["e61ab6c84d6353b3a3e8cea91e18aabe","en_ko_2020q2.md"]]]],["126Bpm\n","126 Bpm\n",null,null,3,null,null,[[]],[[["e61ab6c84d6353b3a3e8cea91e18aabe","en_ko_2020q2.md"]]]],["167.02\n","167.02\n",null,null,3,null,null,[[]],[[["e61ab6c84d6353b3a3e8cea91e18aabe","en_ko_2020q2.md"]]]],["242.70\n","242.70\n",null,null,3,null,null,[[]],[[["e61ab6c84d6353b3a3e8cea91e18aabe","en_ko_2020q2.md"]]]],["295.10\n","295.10\n",null,null,3,null,null,[[]],[[["e61ab6c84d6353b3a3e8cea91e18aabe","en_ko_2020q2.md"]]]],

enjoy !

[ccfontes]

Is it possible we pass our own API key? It's kind of important being able to do this. The reality of the day:

• corp vpns
• home vpns