Silence XML plaintext backup conversion?

[JoeGretch]

Thank you for creating such a useful app.

Would you be willing to write a script that will convert the Silence XML plaintext backup file into a JSON file to be imported?

Here is a sample from the Silence backup:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!-- File Created By Silence -->
<smses count="12972">
 <sms protocol="0" address="+15557639101" date="1606237355859" type="1" subject="null" body="I can&apos;t fri but I can Thur I don&apos;t need assistance picking it up and can come by tonight and pay if that works" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 <sms protocol="0" address="+15557639101" date="1606237237747" type="2" subject="null" body="Hi Tracy. The boiler is all disconnected and ready to be lifted from it&apos;s pad. Can you come in Friday some time to load it up? I have to leave on Saturday and will be away for a week, and will also be tied up all day on Thanksgiving.&#10;&#10;Please let me know if that will work out for you.&#10;&#10;Thanks!" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 <sms protocol="0" address="+15557639101" date="1606077873112" type="1" subject="null" body="Yes all good thx I think I can hook every thing up now that I&apos;ve looked at it hope all is well. With y&apos;all also" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 
                                     . . . . . . .
 
  <sms protocol="0" address="5556637843" date="1689601387150" type="1" subject="null" body="We are on our way to get gas and head out" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
</smses>

Thank you again.

[tmo1]

I'm not promising to do it, but I'll consider it. The format doesn't look too complicated. Are there MMS messages, or only SMS?

[JoeGretch]

I do not think Silence exports MMS successfully. There certainly are no image or other files associated with these messages.

Thank you. I appreciate your just looking at it.

[tmo1]

A conversion script is now available here (documented here). It works on the sample XML you provided, but please test and report back here.

[JoeGretch]

Thank you!

Unfortunately, emojis, seem to break the Element Tree parser. There weren't any emojis in the sample I sent you; I apologize. This is the result I got:

xml.etree.ElementTree.ParseError: reference to invalid character number: line 12, column 93

and this is line 12 from my file:

<sms protocol="0" address="+15557639101" date="1601816103259" type="2" subject="null" body="&#55357;&#56397;" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />

I might just manually strip out the lines with emojis and see if I can make the file work that way, but I wanted to let you know how the initial test went.

Thank you again.

[tmo1]

I looked into this, and it turns out that the problem is that Silence is emitting invalid XML - this was first reported seven years ago! You can try the entityfixer code posted by Calvin-L in that thread.

[s0rau]

@tmo1 Can you implement entityfixer in the https://github.com/tmo1/sms-ie/blob/master/tools/silence-convert.py ?

[s0rau]

./silence-convert.py SilencePlaintextBackup.xml
Traceback (most recent call last):
  File "/tmp/l/./silence-convert.py", line 46, in <module>
    tree = ET.parse(input_file)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: reference to invalid character number: line 75, column 108

[s0rau]

Nice feature, SMS are imported with emoji :+1:

Do you think MMS can be save and restore ? SMS groups (2 or 3 people are not restored) -> they're not in SilencePlaintextBackup.xml

[tmo1]

Nice feature, SMS are imported with emoji 👍

I'm glad it's working!

Do you think MMS can be save and restore ? SMS groups (2 or 3 people are not restored) -> they're not in SilencePlaintextBackup.xml

We can obviously only restore data that's present in the XML file. If there's anything there that isn't being converted / restored, please post some examples and I'll take a look.

[s0rau]

We can obviously only restore data that's present in the XML file. If there's anything there that isn't being converted / restored, please post some examples and I'll take a look.

MMS are stored in /data/data/org.smssecure.smssecure/app_parts/part1001758286944762215.mms

These files are encrypted :

file part1001235047216594581.mms
part1001235047216594581.mms: data

[tmo1]

Where do those files come from? What's the encryption scheme, and where are the keys? If you provide sample files and keys, I can take a look (if you're willing expose potentially private data and metadata), but there's no way I can do anything without that.

[s0rau]

Where do those files come from? What's the encryption scheme, and where are the keys? If you provide sample files and keys, I can take a look (if you're willing expose potentially private data and metadata), but there's no way I can do anything without that.

I'd like to decrypt a file first , in order to see what file it's and then post it.

Here's the MMS database structure : https://git.silence.dev/Silence/Silence-Android/-/blob/master/src/org/smssecure/smssecure/database/MmsDatabase.java

The keys to decrpyt are located in : org.smssecure.smssecure/shared_prefs/SecureSMS-Preferences.xml

[tmo1]

Thanks. I'm not planning to comb through the code to figure out the database structure and encryption scheme, but if you make progress on it, please post your results here.

[dngray]

I created a tool to do this in python to txt format https://github.com/dngray/silence-backup-parse/blob/main/silence_backup_parse.py there were some specifics regarding emojis.

as for MMS those are not exported in the XML file.

[s0rau]

@dngray Nice, emoji and SMS are well exported by using @tmo1 source code What is missing :

• SMS with more than 2 people (which is not present in the XML)
• MMS see my previous messages

[dngray]

SMS with more than 2 people (which is not present in the XML)

That's probably because those are actually MMS messages and not SMS.

[s0rau]

Yes, I don't have found what kind of algo is used to encrypt MMS files : /data/data/org.smssecure.smssecure/app_parts/part1001758286944762215.mms Do you have any ideas about these files ? How to decrypt them ? Keys are locate in : org.smssecure.smssecure/shared_prefs/SecureSMS-Preferences.xml

[tmo1]

That's probably because those are actually MMS messages and not SMS.

Correct. SMS messages cannot have more than one recipient - messages sent to more than one recipient are automatically sent as MMS messages, even when they're only simple text messages with no "media" involved.

[Pommede]

Hi sure i'm a stupid but i tried python silence-convert.py SilencePlaintextBackup.xml and i've got: Traceback (most recent call last): File "/home/user/silence-convert.py", line 46, in tree = ET.parse(input_file) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse tree.parse(source, parser) File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse self._root = parser.parse_whole(source) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ xml.etree.ElementTree.ParseError: reference to invalid character number: line 4, column 116

then i did like recommended:

python silence-xml-fixer.py SilencePlaintextBackup.xml SilencePlaintextBackupfixed.xml and i've got nothing happened..after 15min...i quit..

and python3 silence_backup_parse.py SilencePlaintextBackup.xml

and i've got a SilencePlaintextBackup1.zip (as i wrote in the python file) but empty file....

somebody can help me ??? :-( thks !

[s0rau]

@Pommede https://github.com/tmo1/sms-ie/blob/master/tools/Tools.md#silence-convertpy

[Pommede]

that's what i did, (first command) or something wrong on my command?

[s0rau]

xml.etree.ElementTree.ParseError: reference to invalid character number: line nnn, column mmm

If this error is encountered, first use the XML fixer tool to produce valid XML:

silence-xml-fixer.py < silence-xxx.xml > silence-xxx-fixed.xml

then run the converter on the fixed XML:

silence-convert.py <silence-xxx-fixed.xml>

[Pommede]

thank for your response but i did all that (it's my text.. :-()

[s0rau]

You've missed < and >

[Pommede]

yeah the first command worked python silence-xml-fixer.py <SilencePlaintextBackup.xml> SilencePlaintextBackupfixed.xml but the second no <> if i put those i've got fish: Expected a string, but found end of the input

but python silence-convert.py SilencePlaintextBackupfixed.xml i've got a zip with something....i'll try to another sms app thks

[Pommede]

it's workin !! :-) many thks. no mms but it's a silence issue ?

[s0rau]

MMS are not located in the XML still working about it

[Pommede]

oki, very good job, i was afraid to be stuck with silence rrrr

[s0rau]

I'm looking at org.smssecure.smssecure/shared_prefs/Secure/SMS-Preferences.xml it mentions curve25519

Rapid search can lead to : https://github.com/topics/curve25519 The rage project may can decrypt file ??

-d, --decrypt Decrypt the input. https://github.com/str4d/rage#passphrase-protected-identity-files https://github.com/str4d/rage

[tmo1]

Silence / SMSSecure uses the Signal encryption protocol, which uses / can use curve25519 in its operation. I think that this may be only applicable to the encryption used for messages in transit, though, and not to the encryption used for the on-disk message databases.

There are various tools for decoding the encypted backups produced by Signal, including signalbackup-tools and signal_for_android_decryption, but they are not designed to be used with the raw database files retrieved from the Android filesystem.