firefox-history-merger
firefox-history-merger copied to clipboard
crazy-max
Do you think Firefox History Merger might be making duplicates of entries? Is there any way to deduplicate my places.sqlite files?
After merging places.sqlite files and viewing entries with browsinghistoryview, I noticed duplicate entries (from 2 to several), with the exact same URL, date, and time (down to the very second).
I don't seem to have this issue. I have repeating URL entries with increased 'Visit count' but all have different dates.
well the merged database I have, has only been merge once, so maybe that's why. I'll keep an eye out for it. will have to do a triple merge soon. (old files and I just ran into this tool)
@YousufSSyed yea I can confirm I merged an old places.sqlite (places0) with a new one: places1. then I merged my updated places1(places2) that I used for the last couple of the days with the merged places0 and I received double entries in BrowsingHistoryView. I think it's best if you delete certain the entries that are new/old in firefox and then merge with firefox-history-merger.
if you want to have two concurrent histories you have to use firefox sync anyway and get rid off manually merging histories.
you have to check the unique ID in BrowsingHistoryViewer. These entries get their own ID. It's a column. usually the last column on the right
if you want to have two concurrent histories you have to use firefox sync anyway and get rid off manually merging histories.
What do you mean by this? I keep my long-term history files away from the profile since Firefox does limit how many history entires there are (you can change it about:config but still, I don't want Firefox touching them).
@YousufSSyed well I don't know what to tell you, your worry is unwarranted when it comes to firefox sync.
I run one profile where I sync history with my mobile. I used the profile of a clean profile (no addons) in 2018 for my mobile. As with all my profiles I have places.history.expiration.max_pages to 5000000 or something huge and in that profile, that syncs with firefox sync, I have entries from 2013. I can show you via teamviewer if you don't believe me. And I don't think firefox sync deleted anything.
are you syncing with your mobile or what. ? then create a new profile and just import via https://addons.mozilla.org/en-US/firefox/addon/urls-list/reviews/ what you most need. or like I said: delete what you don't need / the older entries in firefox and merge with firefox-history-merger.
I dont think crazymaxx will be able to do anything bc doing a crossreference tool with duplication check, goes beyond this project's scope. Plus he seems to be busy. He still hasn't responded to my thread either.
I had relied on Firefox to keep all my history entries and back in 2018-19, kept wondering why old ones kept disappearing; I lost many from that time period. found out about places.history.expiration.max_pages limiting the number of history entries. I wish I had known sooner, and would've at least backed up my places.sqlite files.
@KamelittaOida I just thought about this, and I would NOT trust Firefox with years of browsing history like that, even with the adjusted places.history.expiration.max_pages preference. Personally there have been too many times when Firefox just forgot about or deleted data from my browser, especially history & extension data.
Just the other week while using Firefox, I noticed something weird happened to one of my profiles where it stopped logging history. I restarted the browser and notice my history and all bookmarks suddenly disappeared (luckily, it reappeared a few days after that).
You should backup your places.sqlite files outside of the profile directories and use the merger to combine them.
once again: unwarranted fear. you have to make backups anyway. It's not hard to create a batch file to backup upon boot and backup upon shutdown, or if you don't shut down, every 12 or 24 hours, recycle every month for firefox usage. Backups you should make anyway, regardless of firefox sync.
I cannot argue against/for a case, since I still don't understand your need for firefox merger since you haven't laid it out.
you're more likely ruining your places.sqlite with firefox merger than with firefox sync. I've shown you a places.sqlite ranging back to 2013 on a profile that syncs. I don't know what else I can show. The reason i'm advocating for firefox sync is that I feel you're complicating your life with firefox merger.
Just the other week while using Firefox, I noticed something weird happened to one of my profiles where it stopped logging history. I restarted the browser and notice my history and all bookmarks suddenly disappeared
I use firefox daily. Can't recall this happening. They release nightlies and betas. This would've been caught. If you use some history addon, then I don't think you can blame firefox for it.
only downside to firefox sync is that it doesn't sync favicons properly = not all websites. I maybe should report it but I don't care enough. Also this profile is on ESR.
@YousufSSyed I just whipped up a small script that removes all duplicate history entries. Check it out here:
I am in the process of completing a full program for combining Firefox DBs together (with UI and everything!) but this is not finished yet. Feel free to peruse the code if you want to. It's all in Python.
As far as longevity of places.sqlite entries, I have entries going back to 2014 and they're all still there!
Setting the places.history.expiration.max_pages setting in about:config does the job.
