OBIS and HSBC move to PDF-only Statements

[awsles]

Until the end of 2020, there was a nice screen which OBIS would parse but that screen has been replaced and now the only way to view previous statements is to download the PDF. I noticed you've done a lot of re-writing of OBIS. Is there a better place to view statements so that OBIS can parse them? Prhaps amend the README.md to provide more detail as to where exactly to go (maybe I'm missing it).

"Midata" is now available for download (see: https://www.hsbc.co.uk/current-accounts/midata-faqs/) but for some reason, they censor half of the payment type details, so you get a CSV full of asterisks... And so Online Banking continues to be S***.

As always, appreciate your creating OBIS! 😊

[awsles]

Folks don't like the asterisks in midata... I wonder why?! https://twitter.com/HSBC_UK/status/918790518905110528

[shuckster]

Hey @lesterw1 ,

OBIS now works on any page. :)

The best way to use it now as a Chrome Extension.

Instructions are in the README, but I will paste here in case I update that file in the future:

To build it yourself:

• Download the project from Github
• Navigate to: chrome://extensions/
• Enable Developer mode [ o ]
• Click the [ Load unpacked ] button
• Choose the downloaded dist/extension/ folder

You'll see the OBIS interface after logging-in to your HSBC account.

OBIS will find all accounts first, then statements, then entries for all statements. It might take some time, but there is a progress-bar now (it's not perfect -- the last 1% might take a few seconds.)

OBIS will also create a midata CSV without asterisks. To me it's crazy that your own data is censored after downloading it. I mean, if you can see it clearly before downloading, or in the PDF, what is the point of adding asterisks to midata only?

Hope this is helpful. Let me know if it works for you!

[Banjanki]

Hey @shuckster

Just adding to this as I may have an issue using the chrome extension method. The extension pops up and gatherers the statement however the View statements & Download all buttons are greyed out.

Is this an issue or am I doing something wrong?

Thanks and really appreciate your work!

[shuckster]

Hi @Banjanki ,

If downloading many years of transactions, it might be that the progress-bar "freezes" at the last 1% for a long time while it creates the Zip file. Perhaps try downloading just 1 year to see if the buttons become active in this case?

I'm thinking about including the Zip generation in the progress-bar to give the impression that something is still happening when many years of statements are being created. It takes long enough that it can seem like OBIS has broken.

Still, the buttons should still become "active" eventually. Perhaps open the Chrome Developer Tools + Console to see if you get any errors from OBIS before you hit the "Fetch" button?

[Banjanki]

Thanks for getting back to me @shuckster It's strange, I don't see any errors on the console. Only when HSBC times out. i waited about 5 minutes while keeping HSBC from timing out. Could be an issue on my end. I'll try and reinstall the extension. I was going to try the other manual method but I'm not clued up enough to understand the readme for that method when trying to host locally.

Thanks for your help!

[shuckster]

Sorry to hear you're still having problems @Banjanki .

It's true that the second method for using OBIS is a little trickier, but I don't think trying it would solve your problem. It loads the exact same code as the Extension version does, although in a slightly different way of course.

Let me know if reinstalling the Extension works. If not, let me know your Chrome version and which Operating System you're using, and I'll see if I can replicate the problem.

[QuicksilverBR]

Hi @shuckster,

I've been trying to get OBIS to work with the nightmare that is HSBC's online interface (stopped using accounts long ago, just want data for record-keeping), and despite repeated attempts have never got past the installation of the extension - OBIS never shows up on the online banking webpage, regardless of the method used to 'run' it - I can get it to appear at localhost:4000, but that's it. I noted that the statement download page (same PDF version as mentioned here) now has a web address of https://www.hsbc.co.uk/online/customerstatements/, rather than anything matching the default https://*.online-banking.hsbc.co.uk/*. I've tried adding https://*.hsbc.co.uk/online/* to the .json and .js files to no avail, but not being a Chrome extension developer, I may well be looking in the wrong place entirely.

It's by no means urgent, but I'd appreciate any help in getting my data out before anymore of the 6-year history is lost.

[shuckster]

Thanks for the report @QuicksilverBR .

It seems HSBC have once again updated their APIs, deprecating the old one that OBIS was using to download transactions, which is why merely updating the URLs will not make it work. 404s now exist where the old APIs used to be.

After an initial investigation I can see that it's not an insurmountable problem to update OBIS to work with the new APIs. But it will take a little time to figure out how to update everything accordingly.

[shuckster]

Hi @QuicksilverBR

I've added a new plugin that supports the new HSBC UK API. Please try it out and let me know if the numbers come out right for you. It's working for me as far as I can tell.

There is one regression: You can no longer choose how many years of history you want. For my own accounts, OBIS will download about 6 years worth of data, and it doesn't take very long. (There is a short pause before the View/Download buttons become available, just so you're aware.)

It may be possible to download data from further back, but I've not investigated this much yet. However, the state I've got OBIS to now seems good enough to be useful, so thought I'd share.

Hope it works!

[QuicksilverBR]

Thanks @shuckster; just tried it out and OBIS seemed to grab all the statements HSBC presents to me, but in a variety of actually useful formats!

I don't have time to fully check the results tonight, but a brief skim shows that all the data should be there. I'll get back you (hopefully!) tomorrow to confirm.

[QuicksilverBR]

I don't have time to fully check the results tonight, but a brief skim shows that all the data should be there. I'll get back you (hopefully!) tomorrow to confirm.

Alright; I've had a quick look, and most of the history is indeed there. Cross-referencing the data (I've been using CSVs imported into KMyMoney), there are some transactions missing, but these are mainly gross interest listings (the savings account in question has been open a long time - it's changed account types a few times, so not really a surprise.)

It's certainly more than enough to get started with - rather than entering six years of statements by hand, I only have to make a few corrections here and there. Thanks for the update; hopefully HSBC leave the layout alone this time!

[shuckster]

Thank you for getting back to me. It's encouraging to hear that most transactions made it, although I'm curious to find out why some are missing! Seems like OBIS should process everything HSBC gives. 🤔 I'll give it another look hopefully at the weekend again. And yes, it would be great if this website update could be the last one for a while. 🤞

[shuckster]

I'll close this issue for now as I've not encountered the "missing-transactions" problem myself in these months since the report. Happy to reopen/investigate if it reoccurs.