whatsapp-web.js
whatsapp-web.js copied to clipboard
pedroslopez
fix: Add support for sending large files (up to 2GB)
PR Details
Use an additional evaluate function to setup a browser variable, Break the file down into 50MB chunks and append to the browser with a loop of puppeteer evaluates
During sendMessage event, inside the browser, override the attachment's data with the variable previously created
work is based out of: https://github.com/pedroslopez/whatsapp-web.js/compare/main...matricce:whatsapp-web.js:fix/sendBigFiles
Updated patch is from @shirser121: https://github.com/pedroslopez/whatsapp-web.js/pull/2368/files/c452fe86bb0c5ff3c16863ae77e9e76ac8b66010
Description
Changes sendMessage in Client.js it does not affect messages without media, or messages whose media is below 50 Megabytes
Create client objects (chunks) of 50MB parts of the original file contents. Create metadata chunk to save the amount of data chunks
Afterwards, initiate sendMessage on the client, using a dummy MessageMedia object, which is overwritten inside the browser with the chunks sent previously
Related Issues
Fixes #2117 Fixes #1771 Fixes #1951
Motivation and Context
Currently the lib only allows <80MB files, which was fine, however, whatsapp web added support for 2GB files (2097152000 bytes)
How Has This Been Tested
npm install git+https://github.com/opssemnik/whatsapp-web.js#patch-1 Sending / Replying messages without attachments has been tested Sending / replying messages with media (sticker, non sticker, voice and documents) < 50MB Sending / replying messages with media (sticker, non sticker, voice and documents) > 50MB < 1GB
Types of changes
- [X] Bug fix (non-breaking change which fixes an issue)
Checklist
- [x] My code follows the code style of this project.
- [X] I have updated the documentation accordingly (index.d.ts).
In JS we don't use var anymore (only for very specific cases). Instead of var, try use const for unmodified variables or let for variables that will have modifications.
In JS we don't use var anymore (only for very specific cases). Instead of var, try use const for unmodified variables or let for variables that will have modifications.
I've commited the changes, thanks!
@opssemnik After the latest wwebjs update, I can't send files over 50MB again
does it work?
I have been using since i created it, i use it as an youtube video downloader Feel free to test by installing this branch npm install git+https://github.com/opssemnik/whatsapp-web.js#patch-1
@opssemnik After the latest wwebjs update, I can't send files over 50MB again
Could you share your WWEb version? the code is mostly puppeteer based than lib/wweb, so it shouldnt break unless major changes in sendMessage
for me its working in my latest branch with wweb of 2.2332.15
Can you share the version (client.getWWebVersion()), error log and file size?
@opssemnik now im using WWeb version 2.2333.11
Thanks, i will test later with this version
@opssemnik now im using WWeb version 2.2333.11
I tested now, and it works but it is much much much more slower sending the message, the weird thing is that its the actual WWeb sendMessage (window.sendMessage) that is hanging, this PR's code itself still works (it still successfully calls window.sendMessage, afterwards code is unchanged)
I will troubleshoot more to identify what are the changes waweb has done to make it work correctly on newer versions, i believe its a deeper change/effect
How do you guys do to target specific WWeb versions ?
Inside your client initiatilization add these properties: webVersionCache: { type: 'remote', remotePath: 'https://raw.githubusercontent.com/wppconnect-team/wa-version/main/html/version.html', }
replace version with the version you want https://github.com/wppconnect-team/wa-version/tree/main/html
Converted to draft Found the issue, but i need to dig deeper, window.Store.MediaUpload.uploadMedia requires some additional fields on larger files, that the lib is not passing, i was able to make it work hardcoded, but i need to debug on calling the function to add them to the media object
The blob gets rejcted by the server without those
@Sansekai fix pushed to my branch: npm install git+https://github.com/opssemnik/whatsapp-web.js#patch-1
Do note it requires the latest whatsapp web version: 2.2333.11 (you can delete the .wwebjs-cache folder, or target this version using)
Older versions are not uploading documents larger than 100mb, i assume its because they dont send the additional nodes that the latest sends (which i added to my patch)
Even them in vanilla versions (outside of waweb.js) cannot send files larger than 100mb, so i did not add a version check to the patch (since its not an issue with the lib)
In any case, i fixed support for the latest version, it is now back to life :)
I've requested re-review from @tuyuribr and @shirser121 as they were previous reviewers
Do note, i have re tested all other cases of media messages, < 100 mb files, audio files, video files (in non document) and stickers
uploadedOrigin seem to always be "2", even when uploaded in android, so i've made it a local function variable, rather than an exported constant
I will look into into as soon as i can.
I found something that can help.
mR.findModule('getUploadLimit')[0].getUploadLimit can give the max size for media type, and we should use it to check if its possible to upload the media as video for example, and return a error if not...
I will look into into as soon as i can. I found something that can help.
mR.findModule('getUploadLimit')[0].getUploadLimitcan give the max size for media type, and we should use it to check if its possible to upload the media as video for example, and return a error if not...
Thanks!, i will look into it, this probably would make the patch "safer", since when it stopped working last time, it was due to the promise being rejected(ERR_UPLOAD_STAGE_TOO_LARGE creating an AbortException), however that reject was not caught by puppeteer
await window.Store.MediaUpload.uploadMedia got stuck forever... , and further media messages were never sent
I will look into that module to get the current upload limit, it seems this limit is based on the current version of wweb you are running, with a check in place, we could make the return clearer to the developers rather than making code stuck
I also noticed there are a bunch of properties fullfilled by the server, i will see if i can get the module that returns those, should be useful for other parts of the code
getServerProps() which has max_file_size (and others)
@shirser121 whats your feeling about this function? my fear right now is people using it mistakenly
for instance, if i do MessageMedia.fromFilePath on a video file, it will auto detect mimeType = video/mp4
If i run getUploadLimits(video/mp4) it will fail, because it looks for messageType, not mimeType
additionally, if i force the message as a document, the right limit would be getUploadLimits("document"), not getUploadLimits("video")
Due to this, i made the function take a string (not a message object), however, i don't know if its the clearest way i guess i can make javadocs clearer , but i don't know if i should include checks inside the code
or i could make the function take a Message object, and check that it is not an instance of MessageMedia, and then check for forceDocument
Please fix ESLint
Done!
Great job!
Thanks, and thank you for all the help!
I don't know how to properly make staggered PRs (a PR that depends on another), but i added a small commit to fix audio waveform on downloaded voice files, this is unrelated to the main PR per say but its a very small change enabled by the new filesize field and getUploadLimits (added by this PR)
as long as the media file is audio, opus codec, and its size is less than getUploadLimits (audio), waweb will send it as a voice memo, even though mediaData.type is not ptt
goes from this:
to this:
This is useful so developers dont need to add media checks (and different message send/reply codes for each type of media) I.e. if you downloaded the voice memo from a received message
tagging @alechkos as he is the creator of the waveform feature
@opssemnik
tagging @alechkos as he is the creator of the waveform feature
it's not me, i just brought it from another lib
npm install git+https://github.com/opssemnik/whatsapp-web.js#patch-1 @opssemnik
i used this one but its not working for me
not showing any error
@opssemnik
tagging @alechkos as he is the creator of the waveform feature
it's not me, i just brought it from another lib
I brought waveforms to Baileys too from wppconnect, it is a very clean implementation, what is there to solve? I don't get the issue.
Error: Cannot create a string longer than 0x1fffffe8 characters
We can't read files bigger than 512 MB, so this is pretty useless.
We should change the MessageMedia options, to allow the creation of a pipeline stream instead of creating a base64 string, so we can read the file in chunks.
@PurpShell @alechkos can you help here, please?
Error: Cannot create a string longer than 0x1fffffe8 charactersWe can't read files bigger than 512 MB, so this is pretty useless. We should change the MessageMedia options, to allow the creation of a pipeline stream instead of creating a base64 string, so we can read the file in chunks.@PurpShell @alechkos can you help here, please?
Currently the library breaks at +-50mb because of puppeteer's evaluate function, we hit node's string limit at 10x the current limit
Locally, for files larger than that, i changed MessageMedia's data to be an array (and the underlying code to support that), however, since node's string limitation has changed in newer versions, i didnt include it in here
I tried to send media about 380MB and its not work. Sometimes crash and sometimes just not sent. Its seem like there is Missing parameter. Also, i found out that after 3-4 tries i got out of memory error