invoice2data
invoice2data copied to clipboard
invoice-x
Added area support for fields
Fields specifying a parser now also accept an "area" argument. This allows one to specify the page(s) and coordinates of the area to be extracted and then applies the specified parser to the results as normal :) Currently this only works for pdftotext but could be expanded to other converters fairly easily. Incorporated changes into all existing functions where possible to allow for ease of future maintainability. Tutorial was updated to show new option and its syntax.
Credit for the idea and base code goes to @kavinsharma in issue #305
@kavinsharma, I hope this does justice to your original idea :) Based on some of the comments on #305 I switched it into the parsers to allow people to extract areas and still be able to parse them using the other functionalities. @m3nu, let me know what you think :)
This should probably be a parser, if it only finds a single value in a specific area.
Difference between parser and plugin: https://github.com/invoice-x/invoice2data/blob/master/src/invoice2data/extract/plugins/interface.py#L4
The code in this PR is not a parser currently, since it's not one file in the parsers package.
We added parsers and plugins to avoid cluttering the rest of the code too much.
Hope you guys can work together on this. Personally, I'd build on #305, since it's more neatly separated. I'll add some comments there.
@m3nu My idea with having it as an argument rather than a plugin or separate parser is that it could be used in conjunction with any of the other parsers. E.g. if you need to pull out lines from a specific region, you can specify that area and the lines parser. If you need a single value, specify the area and the regex parser. This way any future parsers inherit the functionality automatically as the area extraction is checked before the parser is called :) It is also completely optional, so any existing templates without the area specified work as normal and if you want to search the entire document, you can just leave it out. Thoughts?
ok sure @m3nu , we will look at your comments and fix it up. @RossK1 I hope you will be fine in contributing to my PR #305 :)
ok sure @m3nu , we will look at your comments and fix it up. @RossK1 I hope you will be fine in contributing to my PR #305 :)
For sure! If you need me to test or check something in it, let me know 😄 Even if #305 is implemented as a plugin, I think the functionality in #333 is still important to have as it allows the parsers to (optionally) be directed to act on a very specific area of the invoice. I have found this to be required when dealing with invoices where multiple pieces of information are presented beside each other on the page, but move in vertical relation to each other, causing pdftotext to return slightly mis-aligned lines for each pdf. Being able to clip out the part you want and still run regex with sums or lines on it is invaluable.
@m3nu, what do you think? Would this be a useful feature? You'll see from the extra code that if the "area" parameter is passed in a field, it simply re-runs pdftotext for the region specified and uses the extracted text for that specific parser. Other fields and other plugins are unaffected and yet this functionality can benefit any current and future parsers meaning contributors can focus on the parser instead of how to extract an area of the page.
@m3nu, what do you think? Would this be a useful feature? You'll see from the extra code that if the "area" parameter is passed in a field, it simply re-runs pdftotext for the region specified and uses the extracted text for that specific parser. Other fields and other plugins are unaffected and yet this functionality can benefit any current and future parsers meaning contributors can focus on the parser instead of how to extract an area of the page.
is it available for usage?if is how can we use ?
If we make it a separate parser, it loses the ability to leverage the other parsers. Being an optional input to any field, it simply narrows down the content being sent to any other parser. This way the other parsers do not need their own area support, it is handled up-front.
True, that's an argument for adding it on top and I tend to agree. Let me consider it again tomorrow.
Yeah, as much as I would like to avoid cluttering up the code, this seems like a more elegant way of doing it without having every parser have to call some other area parser first.
I would have preferred this as standalone parser, but see the tight linkage with pdftotext. So the solution you chose makes sense.
Please just add a few test cases using the existing (or new) sample files to make sure the feature keeps working in the future.
Then we can merge and release this feature.
Any chance to have this feature available soon? I'm using invoice2data package and a feature like this would be very helpful. Cheers!
@sergiuturus not yet. Been trying to find time to add the test cases m3nu requires.
And please close any competing/superseded area plugin PRs. We had a few of them.
- [ ] Rebase and resolve conflicts
- [ ] Add tests and test files (or reuse existing files)
- [ ] Close competing or superseded PRs
@m3nu For the tests, would it be sufficient to simply reuse the Amazon one? Update the template file (src/invoice2data/extract/templates/com/com.amazon.aws.yml) to include an area and then update the expected result file (tests/compare/AmazonWebServices.json) to reflect the area to be extracted? The test_content_json in tests/test_cli.py should pick up and compare the changes, right? Would this one test be enough or would you like to see similar extractions for multiple files? Thanks :)
Sure. No problem to reuse an existing template if you don't want to share your own documents.
The test should run the new code. If you reuse a document, please add a new test and comparison output, so the old test is still there.
- [ ] Rebase and resolve conflicts
- [ ] Add tests and test files (or reuse existing files)
- [ ] Close competing or superseded PRs
@RossK1 This PR has been sitting here for a while. Would be great if it could be included in the next release. Could you attend to the outstanding tasks?
@alexis-via Great to see you back here! @m3nu Since there are approving reviews, can we still include this in the 0.4.0 release?
As the author of this PR didn't answer messages for several months, I rebased and added a test in this new PR: https://github.com/invoice-x/invoice2data/pull/438
