web pages: better response on 404's or bad SSL

[unhammer]

e.g. "https://www.avvir.no" gives Feb 01 15:34:44 gtweb.uit.no python3[14767]: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645) in the log, but simply "translation not available" in html-tools.

• [ ] better JSON response from apy
• [ ] have html-tools give that response instead of "translation not available"

[share-with-me]

All the probable and common errors should be handled in the JSON response from APy right? For instance, error codes, 500, 502, SSL: ... , 400s etc?? I could handle it on the apertium-html-tools interface!

[unhammer]

Yeah, https://github.com/goavki/apertium-html-tools/commit/16b37220b9dd8f8fb33e2ce5c372c199b039ac26 partly does this. APy uses a function send_error, e.g. self.send_error(503, explanation="{} on fetching url: {}".format(response.code, response.error))

[share-with-me]

Cool! Will start on this :D

[share-with-me]

Um, I am trying to send an error from apy to apertium-html-tools using self.send_error(). I am unable to get it working. I am just adding the line self.send_error(503, explanation="{} on fetching url: {}".format(response.code, response.error)) and then checking the JSONresponse on frontend. But I am unable to see any. @unhammer , could you help in guiding me what I am doing wrong? I am doing this in translatePage handler and then checking the JSON response on translateWebpage() function. Thanks :D

[unhammer]

https://github.com/goavki/apertium-html-tools/commit/16b37220b9dd8f8fb33e2ce5c372c199b039ac26 is one change that does this kind of thing.

Can you show your diff?

[share-with-me]

goavki/apertium-html-tools@16b3722 is one change that does this kind of thing.

This is the code snippet: ( I am just trying to propogate an error in the apy to html-tools which will be caught in the above commit on the frontend as you have said ). I am unable to send a JSON response with this error on the frontend. The above {} values are just dummy tho!

try:
    response = yield http_client.fetch(request)
except:
    self.send_error(503, explanation="{} on fetching url: {}".format('2000', 'BAD SSL :/'))
    return

[unhammer]

could you give output from git diff instead? Without context, it's hard to know what's going wrong.

[share-with-me]

@unhammer , I have highlighted the part of the code in my file on this link

I have uploaded the git diff between the master and my branch on this gist

[unhammer]

didn't know this was pushed ;-) I just tried 28fcbbbee76f89381d4e742592cadc79a463fef6 and ./servlet.py /usr/share/apertium/ (with apertium-sme-nob installed) and then did curl -Ss "http://localhost:2737/translatePage?langpair=sme%7Cnob&markUnknown=no&url=https://avvir.no" and got back {"explanation": "2000 on fetching url: ni chal raha bhai", "message": "Service Unavailable", "status": "error", "code": 503} which looks good. The python log says [I 170628 09:23:29 servlet:732] Not working! Bad SSL!!! [E 170628 09:23:29 web:1971] 503 GET /translatePage?langpair=sme%7Cnob&markUnknown=no&url=https://avvir.no (127.0.0.1) 118.81ms

[unhammer]

But you should probably merge in the stuff from the giellatekno branch in APy first, since this and related issues have already had some work there.

What we don't want from giellatekno:

• the basic_auth / credentials stuff (that's just for a grammar checker that they don't want to be publically testable quite yet)
• the url-xsls / walkGTCorpus stuff (I might remove that from the giellatekno branch too, since it's not been very helpful)

I think we want pretty much all the other APy changes though.

[share-with-me]

Damn! haha it does work when I do curl -Ss "http://localhost:2737/translatePage?langpair=sme%7Cnob&markUnknown=no&url=https://avvir.no" . Um, what I was asking is, is there a way to send this error with the message as a JSON response to frontend ( will be handled either in the success or failure of the JSON call?

Currently I am making a function in JSON error callback, which is as follows:

function handleTranslateWebpageErrorResponse(jqXHR, textStatus, errorThrown) {
    console.log('hfewf'); // this gets printed
    console.log(jqXHR); // I want this object to have the error message in case of Bad SSL, `ni chal raha // bhai to be precise for this example`
    console.log(textStatus); // prints error
    console.log(errorThrown); // prints undefined
    //translationNotAvailableWebpage(jqXHR.responseJSON);
}

// is there a way to pass in this message as one of the attributes for the JSON response object!?

I am trying to log the JSON response but am unable to find the error message ni chal raha bhai precisely in the console. If I get this working, we could probably print that instead of Translation Not Available in case of Bad SSL (for eg: for https://avvir.no). Following is the screenshot of the response!!

[unhammer]

https://github.com/goavki/apertium-html-tools/commit/16b37220b9dd8f8fb33e2ce5c372c199b039ac26 is how it's done in the giellatekno branch of html-tools

[share-with-me]

goavki/apertium-html-tools@16b3722 is how it's done in the giellatekno branch of html-tools

Thanks @unhammer :D for this.

Um, in the diff goavki/apertium-html-tools@16b3722, you have changed jsonp to ajax. Um, in my code, when I change the function call to ajax, I get the jqXHR.responseJSON correctly while putting jsonp gives an object which has no response value. Um, is there a reason you changed it? I wonder why wrapping around from ajax to jsonp gives errors! Also, am unable to find out how responseJSON object is getting created with the required values?!! This may help me to determine how I could fetch the jqXHR with responseJSON using jsonp!! I make use of jsonp instead of ajax call.

With ajax

With jsonp

[unhammer]

$.jsonp is defined in an extra dependencyjquery.jsonp-2.4.0.min.js, not in jquery.

I don't actually know why the original code used that extra dependency, since jquery's $.ajax has built-in support for jsonp (see https://api.jquery.com/jQuery.ajax/). I have a feeling we could get rid of jquery.jsonp-2.4.0.min.js altogether and just use the stuff built in to jquery – or what do you think @sushain97 ?

[sushain97]

I have a feeling we could get rid of jquery.jsonp-2.4.0.min.js altogether and just use the stuff built in to jquery – or what do you think @sushain97 ?

I think it might have been needing one of these:

https://github.com/jaubourg/jquery-jsonp#features

I can't remember exactly though... I feel like it may have been error recovery.

[share-with-me]

Hm, I have been trying using $.jsonp() instead of $.ajax(). I receive jqXHR.responseJSON with ajax but not with jsonp. I am unable to find the cause of this. Following is my code: It works with ajax but when I replace it with jsonp, it doesn't. ( I have checked across the callApy function and my object only has the parameters which are used in that in a $.jsonp() call ). With jsonp, the output of console.log(jqXHR.responseJSON); is undefined :/ .

        $.ajax({
            url: config.APY_URL + '/translatePage',
            beforeSend: ajaxSend,
            complete: function () {
                ajaxComplete();
                synchronizeTextareaHeights();
                textTranslateRequest = undefined;
                $('iframe#translatedWebpage').animate({'opacity': 1}, 'fast');
            },
            data: {
                'langpair': curSrcLang + '|' + curDstLang,
                'markUnknown': 'no', // TODO: checkbox; also perhaps only remove the #-marks, not *
                'url': $('input#webpage').val()
            },
            success: function (data) {
                if(data.responseStatus === HTTP_OK_CODE) {
                    var iframe = $('<iframe id="translatedWebpage" class="translatedWebpage" frameborder="0"></iframe>')[0];
                    $('#translatedWebpage').replaceWith(iframe);
                    iframe.contentWindow.document.open();
                    var html = cleanPage(data.responseData.translatedText);
                    iframe.contentWindow.document.write(html);
                    iframe.contentWindow.document.close();
                    var contents = $(iframe).contents();
                    contents.find('head')
                        .append($('<base>').attr('href', $('input#webpage').val()));
                    $(iframe).load(function(){
                        contents.find('a')
                            .map(function(_i, a){
                                var href = a.href;
                                $(a).on('click', function() { window.parent.translateLink(href); });
                                a.href = "#";
                            });
                    });
                }
                else {
                    translationNotAvailableWebpage(data);
                }
            },
            error: function(jqXHR, textStatus, errorThrown){
                console.log(jqXHR.responseJSON);
                // translationNotAvailableWebpage(jqXHR.responseJSON);
            }
        });

[unhammer]

@share-with-me why not just use ajax then?

[share-with-me]

Um, @unhammer , won't using ajax affect cross domain requests? As in, if browsers provide CORS support, it'd be perfect. But in case it does not, won't jsonp help?

[unhammer]

It's the server that has to send the CORS header. And the server is APy (and if APy doesn't send the CORS header, that's a bug that we should fix).

But try it across different domains and verify! (E.g. send use apertium.org's apy or https://gtweb.uit.no/apy instead of localhost in your config.)

And as I said, ajax has dataype jsonp.

[sushain97]

It's the server that has to send the CORS header. And the server is APy (and if APy doesn't send the CORS header, that's a bug that we should fix).

Well, that's not entirely sufficient. Specifically, IE8:

https://stackoverflow.com/questions/3362474/jquery-ajax-fails-in-ie-on-cross-domain-calls

Do we want to maintain compatibility with IE8? If we want to drop it, then we can remove the dependency with little consequence.

[unhammer]

But does the dependency actually add anything? What does it do differently from calling jQuery.ajax with datatype jsonp? (That SO-question has datatype json, which does require CORS.)

[sushain97]

But does the dependency actually add anything? What does it do differently from calling ajax with datatype jsonp?

Found it: https://stackoverflow.com/a/19207855/1266600

[unhammer]

The reason I used jQuery.ajax over jsonp in the first place was that jQuery's built-in method gave me error handling, just like @share-with-me 's example at https://github.com/goavki/apertium-apy/issues/54#issuecomment-312355107 shows. Do we have a reproducible scenario where jsonp.js adds something?

[sushain97]

Do we have a reproducible scenario where jsonp.js adds something?

Yes. As mentioned in the link and in jQuery docs, $.ajax does not give you HTTP error handling:

and APy is often borked for one reason or another. Having the loading indicator spinning forever is not great.

[unhammer]

Aha, I've been testing same-domain (didn't actually need to cross domains for jorgal).

OK, so we need jsonp if we want to support IE8 where js is on a different domain from APy. If they're on the same domain, ajax's jsonp gives error handling (while if the browser is modern, we can use json).

Do we do cross-domain requests for apertium.org? Does turkic?

@share-with-me , does ajax with dataType: "jsonp" (not json) give you the right error message if you test within the same domain?

[sushain97]

Do we do cross-domain requests for apertium.org? Does turkic?

Nope, not at the moment. We used to but no longer do. Regardless, it would be nice to maintain support if we can easily.

does dataType: "jsonp" give you the right error message if you test within the same domain?

It did for me.

[unhammer]

I wasn't able to get the nice error message to show with jsonp, but yeah if it's possible it might be worth keeping it in (for as long as IE8 has a non-negligible amount of users; the csv at http://gs.statcounter.com/browser-version-market-share#yearly-2016-2016-bar says 0.62 %; IE6: 0.04 %; IE7: 0.03 %).

On the other hand, I wouldn't prioritise it if we don't actually use cross-domain requests, so it'd have to be for the even rarer combination of using html-tools on some non-apertium domain site with an 8 year old browser :-)

[share-with-me]

It did for me.

Um, @sushain97 , did it work for you? For me the loader keeps on rotating infinitely!

I wasn't able to get the nice error message to show with jsonp.

Me neither :/

Following is the ajax call am making!

        $.ajax({
            url: config.APY_URL + '/translatePage',
            dataType: 'jsonp',
            beforeSend: ajaxSend,
            complete: function () {
                ajaxComplete();
                synchronizeTextareaHeights();
                textTranslateRequest = undefined;
                $('iframe#translatedWebpage').animate({'opacity': 1}, 'fast');
            },
            data: {
                'langpair': curSrcLang + '|' + curDstLang,
                'markUnknown': 'no', // TODO: checkbox; also perhaps only remove the #-marks, not *
                'url': $('input#webpage').val()
            },
            success: function (data) {
                if(data.responseStatus === HTTP_OK_CODE) {
                    var iframe = $('<iframe id="translatedWebpage" class="translatedWebpage" frameborder="0"></iframe>')[0];
                    $('#translatedWebpage').replaceWith(iframe);
                    iframe.contentWindow.document.open();
                    var html = cleanPage(data.responseData.translatedText);
                    iframe.contentWindow.document.write(html);
                    iframe.contentWindow.document.close();
                    var contents = $(iframe).contents();
                    contents.find('head')
                        .append($('<base>').attr('href', $('input#webpage').val()));
                    $(iframe).load(function(){
                        contents.find('a')
                            .map(function(_i, a){
                                var href = a.href;
                                $(a).on('click', function() { window.parent.translateLink(href); });
                                a.href = "#";
                            });
                    });
                }
                else {
                    console.log('fewfew');
                    translationNotAvailableWebpage(data);
                }
            },
            error: function(jqXHR, textStatus, errorThrown){
                console.log(jqXHR.responseJSON);
                // translationNotAvailableWebpage(jqXHR.responseJSON);
            }
        });

[sushain97]

Um, @sushain97 , did it work for you? For me the loader keeps on rotating infinitely!

~~@share-with-me what domain is the request being executed from? Cross domain won't work with $.ajax.~~

see @unhammer 's better response below

[share-with-me]

@unhammer this is what my config.conf file has:

HTML_URL = https://www.apertium.org
APY_URL = http://localhost:3025

Should I replace HTML_URL with http://localhost:8080 where my frontend is running?

Edit: I tried above. Still the same :/