Cannot receive pictures / gifs / sounds from Facebook in Pidgin sometimes

[ae-bogdan]

This is the debug log.

Windows 10, Pidgin 2.14.1 (but this bug happens in 2.12 as well). Been happening for at least two months.

Any clue?

[ae-bogdan]

Weird thing is that it sometimes works. This is the debug log when it works.

This also applies to gifs, files sent and audio.

[ae-bogdan]

This is reproducing on Ubuntu 18.04.

Fail log:

Could there be something wrong with my Facebook account?

[ae-bogdan]

Okay so I noticed that when the request is successful there are actually two requests made towards Facebook:

When it fails only the first one is made.

[ae-bogdan]

I set up the connection via a proxy and it seems to fix the problem. Does this mean that there is a problem with my ip or ISP?

[knittl]

I found a comment which might be related: https://github.com/majn/telegram-purple/issues/55#issuecomment-107694223 – if received images are not visible at all in the chat window (but show up in the logs folder).

The culprit seems to be the "Conversation Colors" plugin (maybe only the "Ignore incoming format" option?).

[ohhai]

Have it for all (?) images sent to me in FB these days. Pictures are absent in journal as well.

purple-facebook-0.9.6-5.fc32.x86_64

[ae-bogdan]

I found a comment which might be related: majn/telegram-purple#55 (comment) – if received images are not visible at all in the chat window (but show up in the logs folder).

The culprit seems to be the "Conversation Colors" plugin (maybe only the "Ignore incoming format" option?).

Nope, The plugin is disabled.

[gradius-v]

Same here, I get maybe every 20th picture. Pidgin 2.14.1 on Windows 10.

[Malvineous]

I'm seeing this as well. Pictures I send to others through the Facebook Messenger app appear in Pidgin, but photos others send to me show nothing in Pidgin, it's like the event was never sent. But sometimes they do arrive. It seems like it could be related to the app version the sender is using or how they send the photo (attaching a photo they just took with their camera app, vs taking the photo through the Messenger app). I tried sending one from the camera app and the Facebook app and Pidgin shows them both so really not sure.

[Malvineous]

So this finally annoyed me enough to start looking into it. I captured logs of a working and failed image.

I could find no differences between the requests, except that the failed one calls /method/messaging.getAttachment and gets a response of:

{
  "error_code": 509,
  "error_msg": "Invalid attachment id (509)",
  "request_args": [
    {
      "key": "method",
      "value": "messaging.getAttachment"
...

Weirdly, if I issue this exact same request with curl, it gives me a valid response with a download link to the JPEG which shows fine when I load it in the browser.

I suspected it could be because Pidgin tries to download the attachment so fast that the remote server hasn't finished storing it yet. As a test, I had someone send me the failed image a second time and sure enough, it worked the second time. Given the speed at which you can post a video to someone after the first initial slow upload, I'm pretty sure they use hashes to avoid uploading images that have already been uploaded once already. So it would make sense that sending the pic the second time just sent another link to the same original image, which by now was available on all the servers and that's why it came through the second time.

This would also explain why the photos work sometimes, if they manage to end up on the servers in time just before purple-facebook requests them.

So my theory is that when purple-facebook gets a 509 error response from Facebook, it should wait for a few seconds and then try again, maybe 3-4 times until it gives up.

@dequis Does this delayed-retry solution sound workable? I'm afraid I lack the experience with the project to contribute a solution but I'm hoping it sounds like something that's not too hard!

[dequis]

Yeah, that's one possible solution. I'm afraid it's very unlikely I'll work on that, though.

Sister issue: https://github.com/bitlbee/bitlbee-facebook/issues/73

[knittl]

How much work would it be to display the link of the media file (in addition to the image)? I think this is a (simple?) improvement, that would already help a lot. It sounds simpler than implementing delayed retry.

Right now, end users have no idea whatsoever that an image has been sent to them if they are using Pidgin/purple-facebook.

[ohhai]

+1 for the question regarding workaround with link display. Absence of error notification is the worst. People often send images with no additional text. As for me, I lose significant part of incoming pictures.

There is even specific check for error 509 in fb_api_json_chk():

	/* 509 is used for "invalid attachment id" */
	if (code == 509) {
		errc = FB_API_ERROR_NONFATAL;
		success = FALSE;
	}

Trying to fix it for myself (thanks @Malvineous and @knittl for the info). Dumbest retry in case of error seems not possible because of asynchronous messaging sequence. Request information shall be stored elsewhere (?) to retry after possible error.

[dequis]

bitlbee-facebook can only display URLs, it doesn't embed images, and it has the same problem. If you check the issue i linked, we don't get the URL at all before calling getAttachment, just a filename.

Absence of error notification is the worst

Previously it'd disconnect the whole account. Hard to say which failure mode is worse.

[Malvineous]

FYI the API sequence goes like this:

1. Facebook: You were sent an attachment with ID 12345
2. User: What's the URL for attachment ID 12345?
3. Facebook: Attachment 12345 is at http://example.com/pic.jpg?auth=abcd
4. User: Download and display image

The problem is in step 3, instead of responding with the URL, Facebook sometimes responds with "Attachment ID 12345 doesn't exist." So at that point all you have is the attachment ID, no URL.

If a retry is not possible, I wonder whether a fake message could be returned to the user, like "Unable to retrieve attachment" or something, so that at least you know you're missing something and can go look at an alternate client.

[ohhai]

Technically retry should be possible. Currently playing with it, will report results here unless better solution appears. On another hand, I have big doubts that any of implementations I can imagine might be accepted and merged. However, should firstly verify the general idea anyway.

[Malvineous]

Sounds good. I had a quick look at the code and I see what you mean. I suppose you'd have to store all the attachment requests in some kind of array and remove them when the response comes back, unless the response is an error in which case you submit the request again and increment some limit counter.

In the meantime I have added sleep(2); to fb_api_attach() just to confirm whether this is the problem or not. Presumably Pidgin will now freeze for two seconds any time someone sends me a photo, but it should be enough time for the server to sort itself out and confirm one way or another whether this really is the issue.

[knittl]

@Malvineous wrote:

[…] If a retry is not possible, I wonder whether a fake message could be returned to the user, like "Unable to retrieve attachment" or something, so that at least you know you're missing something and can go look at an alternate client.

IMHO this would already be a viable workaround (showing the error message). At least then end users are informed that something was sent/not received. At the moment, I only find out days later by chance when I use a different client – I always wonder where all those new images come from and why I didn't see/receive them. Displaying an error/warning/message would at least instruct users to go check with a different client (e.g. the official browser one)

[ae-bogdan]

@Malvineous wrote:

I suspected it could be because Pidgin tries to download the attachment so fast that the remote server hasn't finished storing it yet.

Since I got it to work via a proxy, does this mean that if we would slow down pidgin's download speed, it would work?

[Malvineous]

Not so much the download speed as the request and response are only a couple of hundred bytes each, but rather you have to make Pidgin wait a little longer before attempting to ask Facebook for the photo's download URL. It does this by making a HTTPS POST request to https://api.facebook.com/method/messaging.getAttachment, so you want this request delayed by a couple of seconds so that when it actually hits the Facebook server, they have had time to make the photo available.

If you can configure your proxy to delay requests for that URL for a second or two (noting it's both HTTPS and POST) then I believe that would fix the problem. (EDIT: I suppose if you could slow that one URL down to a speed of 50-100 bytes/sec then it would take over one second to finish sending the request so that would probably work too. But you'd only want to slow down that one URL otherwise everything else would slow to a crawl too.)

For the record my sleep() test seems to have worked, I haven't received too many photos since setting it up but I haven't lost any of them either. So it looks like my theory is probably right, and purple-facebook just needs to be a bit slower in retrieving photos. I'll continue to test but I am now wondering whether this dodgy sleep workaround could be a simpler solution than handling retries. I guess it depends on whether @ohhai has any success with implementing retries, as that would be a far more preferable and robust solution.

[ae-bogdan]

@Malvineous thanks for the great idea. I used clumsy in order to delay the ip for api.facebook.com by 100ms. Seems to have fixed the problem.

[ohhai]

Reporting intermediate results of what I have for now:

1. Easiest and still helpful change for me is error print of error 509 to Linux shell. It allows (scripted) monitoring of this specific errors and do some kind of notification, so I don't miss such a cases at least.

2. Managed to print error text to chat window, but only for active (opened) conversations for now. Gives much better indication than previous method but still no fix.

3. Currently have some kind of retries implemented. It allows to get pictures in chat (not sure regarding correctness level), but sometimes crashes with memory allocation errors after this sequence: < my sleeping thread > -> < fb_api_cb_publish_ms() called with parameters saved by me; for FbApi parameter saving only a pointer (not copying content) > -> < fb_api_json_chk() > -> < fb_api_error_emit() > -> < g_signal_emit_by_name() > -> < generic Pidgin flow > ... Tried to just disable fb_api_error_emit() calls for sleeping thread (while still allowed for main thread). After this it stopped both crashing and getting pictures in chat. Possibly some pointers have free() inside main thread, or this thread has some signal processing routines which are absent for my sleeping thread, or just a mistake in my code, etc.

Will dig more, but not sure if successful. Can publish the changes if might make sense, however don't see the proper repository for now: current one doesn't store all the files (like libpurple/protocols/facebook/api.c); and Mercurial repo for Pidgin has these files only for 3.x branch if I'm looking at the proper place. Using Pidgin v2.14.1 (Fedora repositories).

[gradius-v]

Can publish the changes

Would you be willing to publish a Windows dll with the changes ? I'd rather have a few crashes than an unusable program.

[ohhai]

Would you be willing to publish a Windows dll with the changes ?

Sorry, I have Linux environment only. Anyway it would be better to distribute source code changes because of security point at least. I think I'll take another look at that crashes and in the meantime adding error print about error 509 seems reasonable, as for me. With no negative effects it makes possible detection of failed picture transfers, so we don't miss that moments at least.

[gradius-v]

The maintainer doesn't seem interested in this project anymore so I guess this will never be fixed ?

[ohhai]

Someone's patch might get a merge, as far as I understand.

[allanlaal]

@ohhai could you commit these changes to your own fork, so we can compile from there? :)

I'm this close to dumping using FB Messenger on desktop (the "updated" web client is unusable since december)

[Malvineous]

If you're desperate and willing to compile it yourself, my dodgy fix has been working well. No crashes or anything, Pidgin just freezes for two seconds every time someone sends a picture which is only mildly annoying. I edited pidgin/libpurple/protocols/facebook/api.c and in the function fb_api_attach() I added this sleep() call near the top:

static void
fb_api_attach(FbApi *api, FbId aid, const gchar *msgid, FbApiMessage *msg)
{
        FbHttpParams *prms;
        PurpleHttpConnection *http;
sleep(2);

It's obviously not a proper fix but it reduces my frustration levels until someone who understands the codebase better than I is able to fix it properly.

I also have only received one cut-off image since doing this whereas receiving incomplete images used to happen fairly regularly, so I guess this problem has actually been going on for some time now. I always wondered why sometimes I would receive only partial images and I guess this explains it - Pidgin is downloading the image before Facebook has finished uploading it. Seems like poor design on the FB side if they tell you where to download the image before they've actually uploaded it yet...

[ohhai]

Supporting the previously mentioned change for self-fix. I have items #1 and #2 working (from here: https://github.com/dequis/purple-facebook/issues/489#issuecomment-758232570), but not the main fix for now. Have one idea why crashes may happen, message delivery might use signals (like man 7 signal in Linux), so absence of custom handlers on my new spawned threads may lead to crash. Need to test, but even if succeed it is unlikely to be merged. Also I rebuild source package from my distro (Fedora), not the upstream sources, so patching the latter will be more complicated.

[gradius-v]

I've thrown in the towel and use the shitty website now. As I have no idea how to compile this stuff for Windows and most of the links and instructions on the pidgin website don't work, I've given up. So thanks for all the work and goodbye.