postcss-grid-kiss
postcss-grid-kiss copied to clipboard
sylvainpolletvillard
Text recognition
We need to go deeper. I propose to introduce a text recognition nodejs plugin. Imagine how would be cool to put a layout drawn on a napkin.
It goes well beyond the scope of this small project but it would be super fun :smile:
I think this kind of OCR tools already exist in Node, maybe you can try to chain them ? Time to experiment :+1:
This plugin is very incredible. I look forward to the support of all properties
It is a deliberate choice to not support some of the grid properties such as gaps or implicit zones. See the associated comments in the section. I do not plan to support more properties for now.
Note that you can always set these properties manually next to grid-kiss declaration if you feel the need to.
@adubovskoy I've done a few experiments with Tesseract.js today:
As you can see, there is still work to do with OCR :smile: I think the Tesseract configuration needs some tweaking to identify zones, corners, column alignment etc.. It goes beyond my skills and I need some help to progress on this feature.
If you want to test it by yourself, check out the ocr branch here : https://github.com/sylvainpolletvillard/grid-kiss-playground/tree/ocr ; and look at the OCR button in the playground header.
Would restricting letters to capitals/uppercase or specific keywords prevent OCR read errors?
Creating a second pass to adjust row widths and column addons (e.g. - between rows, etc.) might help adjust any errors/mistakes in the OCR image.
this is something really interesting, wish my PC was up to the challenge.
Yes, I tried a specific alphabet but the results have been disappointing so far. I think it would require a high-def camera, image pre-processing and a specific tesseract config to handle this kind of layout.
Since you have an automated process and your code can figure out areas, would using an automated naming convention work? At least for the current version. These automated names can be variablized like in sass (or the next version of css) and listed at the top, giving the user the ability to manually change the names should they want to.
@kartikadur are we still talking about OCR ? could you give an example ?
I'd like to borrow the image you have attached above to work on something that I think might be promising.
Instead of using the OCR reader/ program, I was thinking of using simple edge detection to figure where the section boundaries are located. this should then allow for automatic detection of areas and thus by extension the automated naming.
Sorry if this sounds a little fuzzy, but its something that I am exploring right now. I should hopefully have an example for you in a day or two.
As a preliminary example, at least for the first step, I used the code from this codepen to create the example I have posted below. It still needs work, but hopefully it should give you a basic idea.
Feel free to try everything you want, and have fun 😉 This image is pretty bad quality, it is hand drawn and the picture is from a cheap webcam with low lighting and indirect angle. Do not hesitate to make your own in high definition
This is by far the most interesting/exciting css-related issue I've stumbled upon. GitHub needs more creativity like this. 😍
lol yes but this is still an unresolved issue, I wish I had the knowledge to make it happen. This would make a really, really cool demo.
@sylvainpolletvillard I'm learning AI/ML this year and plan on playing around with all kinds of stuff like this, so eta 1 year before I can PR (unless someone beats me to it?). 👴
Hi @corysimmons , hope you're doing well ! This can indeed be a fun exercise for machine learning, but I don't have huge expectations :smile: . I think I was on the right track with Tesseract.js tho
Don't think I forget about this issue. I talked with several experts in image recognition and tried different solutions. I'm currently quite pleased with the results I got with OpenCV:
Nice work @sylvainpolletvillard ! This continues to be my favorite issue on Github.
- What are your thoughts on lining up that grid for use? Can you combine your result with a "snap-to unit" (i.e. a unit to round to, defaults to something like
30px) and some masonry math to line it up? - Can you recognize/capture the text and insert it into the appropriate cells?
- I'd imagine you'd need to make some pretty huge cells to account for big words and grid nesting, so maybe grids should live in their own sub-directory like
css/grids/homepage.css
This is obviously some very early stage work. To get a minimum viable product, I would like to :
- automatically rotate the picture to make horizontal lines truly horizontal
- improve the calculation of the coordinates by adding a margin tolerance to compensate the lack of precision of our ridiculous human hands
- crop the zone content and use optical character recognition to extract text (this is going to be hard because of handwriting)
- auto expand the grid to fit the extracted content while respecting the original proportions as close as possible
Do you have this work pushed anywhere? I'd like to take a look at it and tinker around with estimations based off of how we know CSS Grid works.
I'm looking for external contributions on this topic, but first I would like to keep the door open for other tools and approaches. I already changed my mind 2 times, using Tesseract then Tensorflow then OpenCV with Python. Before pushing any code, I want to be sure that I picked the right tool stack, so that if you decide to contribute, your contribution will not be lost if we need to change the tool later. Just so you know, my current work is based on this tutorial : https://www.pyimagesearch.com/2016/02/08/opencv-shape-detection/
What would be great if you want to help is that you try your own approach so that we can compare our results. Ideally, I would like this tool to use JavaScript or WebAssembly so that we can use it on mobile or desktop and use the camera or import pictures. Then I could call postcss-grid-kiss in realtime and get immediate feedback on a website layout. Hopefully we will get to that point one day :+1:
Progress !
Today, I converted my python code to JavaScript and I managed to run OpenCV in the browser thanks to WebAssembly. I also implemented auto-rotation, precision correction and finalized the ASCII formatter.
Next step is optical character recognition (OCR) to extract the zone selector, but there is a major issue. The two OCR solutions in the browser I tried are Tesseract.js and Google Cloud Vision. Tesseract.js runs locally in the browser, while Cloud Vision is a remote API that can be used for free for 1000 queries before Google asks for some money. Cloud Vision did a very impressive job to recognize my crappy handwriting, but unfortunately I can't say the same about Tesseract. The only characters that Tesseract can recognize accurately are high definition printed characters.
So to sum up, if we want OCR, we'll need to limit to 1000 queries or pay for it. The other option would be to let the user complete the grid code by specifying each zone selector. It's less magic but it's the guarantee to avoid OCR errors. Also, maybe it's not very natural to write on paper classes or id selectors ? What do you think ?
I think OpenCV.js is the best we can expect at this point, so I published the code here : https://github.com/sylvainpolletvillard/grid-kiss-vision ; in case you want to play with it @corysimmons
Cloud Vision: Can devs procure their own API keys and get a 1000 limit? 1000 sketches-to-layouts per person might not be too bad. 🤔
I thought about that, but I don't know if a 1000 queries limit is sustainable. There is a lot of variable adjustement before getting the expected layout, it can take between 5 and 50 tries. So we can't really limit to 1 call per picture.
@11111000000 I apologize for the joke above. I'm re-reading this issue and realize how helpful your suggestion might be (basically free OCR).