libpes
libpes copied to clipboard
treveradams
Contribute to implementation
I am the developer primarily in charge of formats on the Embroidermodder project. We have a simple implementation of writing PES files. Recently I worked to understand the entire PES format primarily version 1 and 7. I have an application written in C that parses nearly all of the data properly. I plan to make it open source, currently embroidermodder is not prepared to handle all the design information so I wanted to understand your plans for implementation. Does it make sense to you for me to contribute to this project or just spin up a new repository?
Thanks, JoshVarga
Currently, I wish to document as much as possible. My longer term goals were then to create a libpes, much like libpng or libjpeg, in that it allowed easy access (read-only at first then read-write) to all data in the file. I was hoping that eventually there could be similar libraries for the other common embroidery formats. I was hoping, if I was involved in such, that there could be a similar API for these libraries (as much as is possible).
At first, I was hoping to extend Inkscape to be a gui for these libraries. Short term this may work, but I am not sure it will work as well as I had originally hoped long term.
What do you think?
I think if I contributed what I have right now it might help with the documentation. I believe there is quite a bit that I discovered that is not currently in your documentation. Some of the more difficult portions would be more easily decoded if this program did the majority of the decoding for you. Following the libpng and libjpeg API, sounds like a good idea. We could evaluate what I have now and see how can modify it to more closely follow the API of libpng and libjpeg while still making sure that it makes sense for embroidery formats.
Currently in embroidermodder we have a portion of code named libEmbroidery that provides a common api for many different formats including pes (read and write) but it doesn't support many of the "designer" file portions of pes. Only simple designer code with the additional "machine" portion (PEC).
-------- Original message -------- From: Trever Adams [email protected] Date:02/08/2014 2:17 PM (GMT-06:00) To: treveradams/libpes [email protected] Cc: Josh Varga [email protected] Subject: Re: [libpes] Contribute to implementation (#1)
Currently, I wish to document as much as possible. My longer term goals were then to create a libpes, much like libpng or libjpeg, in that it allowed easy access (read-only at first then read-write) to all data in the file. I was hoping that eventually there could be similar libraries for the other common embroidery formats. I was hoping, if I was involved in such, that there could be a similar API for these libraries (as much as is possible).
At first, I was hoping to extend Inkscape to be a gui for these libraries. Short term this may work, but I am not sure it will work as well as I had originally hoped long term.
What do you think?
— Reply to this email directly or view it on GitHub.
I am sorry. I wasn't clear enough. I meant in concept like libpng/libjpeg in that they were fairly easy to use and could be used by anyone (LGPL or similar).
Is the license on libEmbroidery such that we could start an API from there (not the code necessarily, just the API)? We could then just extend it for the designer portions of those formats that have it.
Thank you for your contributions and interest!
@treveradams Yes, libembroidery can be used by anyone for making plugins, other apps or other uses. It is zlib licensed so there shouldn't be any concerns about license conflicts. If you are wanting to use it with Inkscape, python bindings would need to be completed first. libembroidery is written in C89 but is a long term goal of ours to provide bindings for many other languages eventually and I have been looking at SWIG to do this with since it covers many languages. If you look at the Embroidermodder repo, you will see that there are experimental Delphie/FreePascal bindings contributed by another contributor(just not using SWIG since it doesn't support Delphie/FreePascal bindings yet).
We can generate shared and static libraries. If you are using GCC/G++ you would use -lembroidery for linking. The biggest thing you should probably note right now is that since we have not locked down the API yet, upgrading to a newer version in the future may break your code somewhat.
Hello @JoshVarga, @treveradams and @redteam316. I’ve written a PES library that does decoding, encoding and transcoding of the PES format and a prototype version of an SVG based embroidery format. Eventually I’d like to make the library support all PES versions and all features, but I’m lacking some documentation and sample files to test with. Are any of you actively involved using PES these days?
On 01/06/2017 04:36 AM, frno7 wrote:
Hello @JoshVarga https://github.com/JoshVarga, @treveradams https://github.com/treveradams and @redteam316 https://github.com/redteam316. I’ve written a PES library https://github.com/frno7/libpes that does decoding, encoding and transcoding of the PES format https://github.com/frno7/libpes/blob/master/PES-format.md and a prototype version of an SVG based embroidery format. Eventually I’d like to make the library support all PES versions and all features, but I’m lacking some documentation and sample files to test with. Are any of you actively involved using PES these days?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/treveradams/libpes/issues/1#issuecomment-270887329, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-I_eerCfRJVwBKyDYdAROf98AgKpBdks5rPidWgaJpZM4BgWJO.
I have had input from an individual that adds many/much of what is missing. Unfortunately, I haven't had a chance to review it, reply and/or commit. I will be trying to do these next week. I would love to work with you going forward.
Thank you,
Trever
All,
A while back I wrote code that supported almost every version and had almost every data type at least 90% understood. I will have to dig up that code. I was planning on integrating that into the "libembroidery" portion of embroidermodder.
Josh
On Jan 6, 2017 7:56 AM, "Trever Adams" [email protected] wrote:
On 01/06/2017 04:36 AM, frno7 wrote:
Hello @JoshVarga https://github.com/JoshVarga, @treveradams https://github.com/treveradams and @redteam316 https://github.com/redteam316. I’ve written a PES library https://github.com/frno7/libpes that does decoding, encoding and transcoding of the PES format https://github.com/frno7/libpes/blob/master/PES-format.md and a prototype version of an SVG based embroidery format. Eventually I’d like to make the library support all PES versions and all features, but I’m lacking some documentation and sample files to test with. Are any of you actively involved using PES these days?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/treveradams/libpes/issues/1#issuecomment-270887329, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-I_ eerCfRJVwBKyDYdAROf98AgKpBdks5rPidWgaJpZM4BgWJO>.
I have had input from an individual that adds many/much of what is missing. Unfortunately, I haven't had a chance to review it, reply and/or commit. I will be trying to do these next week. I would love to work with you going forward.
Thank you,
Trever
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/treveradams/libpes/issues/1#issuecomment-270907665, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYcnIhNDr4YvoWyf61YMoHim3kGAPcgks5rPkglgaJpZM4BgWJO .
I just updated the documentation with the information I have received. It makes a huge amount of sense. I have not yet worked all of it into the existing format of the document, but what isn't is still included at the end.
Fantastic! @treveradams, thanks for your quick update! I’ll look through the details of it. At a glance, I believe I have a more complete description of e.g. “PES Block v6” in the “version 6 header section”, especially the “color list subsection”. Do you have files containing “CEmbCirc”, “CEmbRect”, “CEmbLine”, etc.? None of files I have contain these.
@JoshVarga, I have a few hundered files of PES versions 1 and 6, and a handful of versions 4 and 5. The files were supplied by Brothers Industries Ltd. and Bernina International AG so I assume they are properly encoded. Unfortunately the files cannot be distributed as they have restrictive licences. It would be great to have a set of known-to-be-correct files as a test suite. Would you happen to have PES files that could be used for format testing?
By the way, what would be your opinion of using a very small subset of SVG as an open and vendor neutral format for embroidery designs?
On 01/06/2017 11:23 AM, frno7 wrote:
Fantastic! @treveradams https://github.com/treveradams, thanks for your quick update! I’ll look through the details of it. At a glance, I believe I have a more complete description of e.g. “PES Block v6” in the “version 6 header section https://github.com/frno7/libpes/blob/master/PES-format.md#version-6-header-section”, especially the “color list subsection https://github.com/frno7/libpes/blob/master/PES-format.md#color-list-subsection”. Do you have files containing “CEmbCirc”, “CEmbRect”, “CEmbLine”, etc.? None of files I have contain these.
I believe I may have CEmbRect and CEmbLine. They are in PES files that I have under license and cannot share.
I would love having and all information you have and are willing ot share.
@JoshVarga https://github.com/JoshVarga, I have a few hundered files of PES versions 1 and 6, and a handful of versions 4 and 5. The files were supplied by Brothers Industries Ltd. and Bernina International AG so I assume they are properly encoded. Unfortunately the files cannot be distributed as they have restrictive licences. It would be great to have a set of known-to-be-correct files as a test suite. Would you happen to have PES files that could be used for format testing?
By the way, what would be your opinion of using a very small subset of SVG as an open and vendor neutral format for embroidery designs?
When I first started this project I had a few goals in mind. My main ones were to create a standard API that could be used to write several of the different embroidery formats. Some only have stitch, some have stitches and colors, some stitches colors and shapes. So the API would need to be able to define these.
The API would need at least three data structures. Colors (name, RGB representation, maybe vendor/vender part#) Stitches(start and stop of stitch, array likely) Shapes (not sure).
The functions/methods would simply be: Read Write
For formats that didn't support various parts, a standard UTF-8 text file could be created listing things. Some formats may only support part of the color information, for example, all of it could be written to the text file.
This way a program could load these data structures and call a write function, the read would be the reverse.
I was hoping to make a small change to Inkscape, or similar SVG editor, that would allow SVG to be used a "lossless" format.
If I remember correctly there were a few attributes which would have to be added:
- A way of defining stitch direction for the primitive instance (poly, line, arc, etc.). This may seem unnecessary, but proper stitch direction can make all the difference in how an embroidered piece looks.
- A way of defining stitch length, again for the given primitive instance. (In documentation or for libpes, or what not, good minimums and maximums would be good to know as well as a good standard.)
- Stitch density for primitive instance. Again, this can make all the difference. In some designs it may even be desirable to have some parts more dense than others. (Again good defaults documented, and recommended minimum/maximum would be nice.)
- Use gradients for the colors. Why gradients? This has more to do with thumbnails. Some embroidery thread has multiple colors. A gradient could show a rough idea of this in the SVG editor as well as in a preview image. (This last one would not require changes to Inkscape or what not, but would just require that designers follow some standard in how they created/used such gradients. Not following it may not change any functionality, just preview results.)
I never got to this as I got stuck on figuring out the PES format and my time was demanded by another project.
The final piece would be the stitch <-> SVG converter. Stitch to SVG might be relatively straight forward. The SVG to stitch has to be "smart" enough to avoid potential damage to the machine due to knotting (stitches being too close, etc.). It would need to read the 4 items added to SVG and use them to adjust settings.
A preview might also do something like Gimp's clothify, adjusting each parameter automatically, to each part of the image based on stitch direction, length and density. It could also use the gradients for multi color thread. We would not (and probably don't want to anyway) attempt to simulate each stitch, just a rough representation such as clothify does.
I would love to have feed back on this API idea and the SVG idea.
Thank you. Trever
Caveat: I have never used any of the embroidery programs, so pardon my ignorance. I never go that far.
Further ideas on SVG subset. I think the path stroke and fill colors are important. Again, the use of a gradient if the thread is multi color. Stroke style. We should support whatever primitives that SVG supports that would commonly be used: circles, squares, ellipses, lines, arcs, letters, etc.
Was I missing attributes needed beyond stitch density, stitch length, stitch density?
@frno7 would you mind if I merged information from your document into mine and maybe even copy the format? (Get it into md out of odt.)
@frno7 , It seems in version 6, you are skipping a field before "Length of following string". Another individual has that as being the # of stitchgroups. I saw this same field but didn't know what it was. Are you sure you have it complete?
As I am looking at this maybe I have been a bit naive. I saw a demo of one of the software pieces once upon a time, it showed simply drawing objects, setting stitch direction, density and length and it did the work.
@treveradams, many thanks for your comments! Your ideas on a high-level “generic” embroidery API sound good. In addition, I believe format specific low-level APIs (specifically for PES etc.) are useful as well, with the generic API implemented in terms of the low-level APIs. My libpes implementation is mostly low-level except, perhaps, for the SVG transcoder prototype. One design choice made is to avoid direct file operations in the library and instead rely on a set of callbacks, as this is much more flexible.
Some form of Inkscape integration, as you suggest, would be excellent! Similarly to the API, I think the SVG embroidery format could be split into two parts: one fairly low-level part mostly involving stitching instructions (PEC type instructions), and another high-level part concerning rendering of shapes, typefaces, etc. Details like SVG namespaces etc. need to be thought out as well.
My libpes repo is GPL-3.0 so feel free to clone and modify the documentation etc. I’m happy to accept patches as well.
Regarding the PES version 6 header, I‘ve been unable to make sense of the fields (the 4 bytes u16 + u8 + u8 with typical values of 1, 0x30 and 0x31 or 0x32) just before the description string. Is some field missing? Not sure how any of them could be the number of stitch groups? [ The hoop width and height are given in millimeters after the string so why would those be encoded in reduced form in the u16 before the string, as suggested in your ODT? ] One way to try out the specification is to run the pes-info tool that comes with the library. Does it work properly with all your PES files?
@frno7, I have been looking around today. I think what I am looking for (if I could make it work) is somewhere between https://github.com/lexelby/inkscape-embroidery and https://github.com/sergstetsuk/inkscape-embroidery/tree/shapesplit. I cannot make it work because it complains about self intersection or holes.
I am not entirely liking the algorithm results for stitches with it (the samples shown). I like the idea of all the stitch length, density, direction (maybe stitch fill type such as those in Wilcom Hatch) being attached as SVG/XML attributes to the object. Your SVG format seems to convert each stitch to a path, which is NOT bad and allows for things to be manually edited. Would it be possible to mark each such SVG object you create with some attribute so that modified versions of inkscape-embroidery wouldn't try to restitch it, but would dump it directly? Or am I missing how your pes to svg conversion works? I do work with Python, but I am having a difficult time figuring out python shapely and why it throws the self intersection error. I also am also not sure what to do about the lack of deal with holes properly. (https://en.wikipedia.org/wiki/File:Flag_of_the_United_States.svg has self intersection, http://www.freepik.com/free-vector/coloured-eagle-design_952146.htm#term=eagle&page=1&position=13 will need eps to svg conversion by Inkscape, it causes the holes problem).
As for the hoop, that was provided by someone else. It does seem to match what I am seeing. It also fills in a gap, that at least in my copy of v6 and v1 headers, you don't even seem to have a place holder for.
I will try to better compare my documentation, yours and the other contributor's. Unfortunately, I have pressing time requirements for at least the next 2 weeks related to work and home obligations. So, my work here will be sporadic until those are met.
@treveradams, yes, the SVG embroidery format I’ve been prototyping is a simple almost direct stitch-for-stitch transcoding for the PES format. I think it could be improved using attributes, as you mention, especially the technical parts such as handling of jumps and trims.
I imagine at least two kinds of workflow. Firstly, one can to transcode PES into SVG in order to adjust a design (in a very technical and precise way using for example Inkscape, manually or otherwise), and then transcode the modified SVG back into PES again (preferably losslessly). Secondly, one can author original designs in SVG and have them transcoded into PES. In this case it helps to work with shapes in SVG and then translate those shapes into SVG paths before transcoding into PES. I’ve made a few proof-of-concept designs using Inkscape and my own tools in this way (SVG shapes → SVG paths → PES, so SVG has dual purpose here).
I work mainly with C/C++ and haven’t tried the Python tools you mention. I believe a well rounded set of extensions to Inkscape could certainly improve usability of the workflows.
I haven't had time to document the format, however I just published some code that should be able to answer questions about many missing details. Take a look at the code here: https://github.com/JoshVarga/libpes
It should help to get an understanding of the difficulty level of making lossless conversion between PES and SVG.
If anyone has time they may be able to extract much of the code into the documentation.
It covers: Version 1 - 8 Affine Transforms Embedded patterns Circles D Text Object groups differences in the header of the files regular letters Lines Motif patterns M Text N Text Rectangles Satin stitches Special shapes Stamps S Text Tile lines
@frno7 yes, I think you and are have the same work flow in mind. SVG -> PES being the major part. @JoshVarga, PES -> SVG may lose information. While I am interested in that, it isn't my main concern. A complete documentation of the PES format and a way to use SVG, say through Inkscape with extensions/modifications, is what I am interested in. My work flow as I am working to get going uses SVG for many things and it would be nice to be able to use it for the emroidery part as well for consistency.
Thanks @JoshVarga! I’ve updated the PES format document with many of the findings in the headers especially.
@treveradams, my Brother Innovis 955 machine seems to only make use of the stitches in the PEC, ignoring most parts of the PES. If this holds true for embroidery machines in general then the PES encoding format can be quite minimal in any SVG path → PES transcoding workflow.
The problem of translating SVG shapes (polygons, curves, text, etc.) into suitable SVG paths could then be treated separately, independently of PES and possibly any other embroidery format. Translating shapes into paths is a difficult problem but at least things simplify if one can ignore PES.
@JoshVarga, do you have any comments on these thoughts?
@frno7, Mine also uses the color table (names, rgb values). As for translating shapes into stitches, that is why I am looking at the python projects I mentioned. Inkscape can convert just about any SVG shapes into paths, once there, the python extension (barring holes and self intersection problems) solves the rest for me (then manual editing to fixes mistakes, etc.)
@frno7, would you mine chiming in on my suggestions for the SVG related stuff in https://github.com/stesie/inkscape-embroidery/pull/4?
PEC is the machine format, I don't know of a machine that uses the PES portion. PES is for designing only as far as I know. Embroidermodder is capable of writing version 1 PES files that are accepted by every machine that I have tested on. It's possible to just use that code. In the libEmbroidery portion of Embroidermodder most of this is supported, SVG is partially supported but is definitely lacking.
On Mon, Jan 16, 2017, 1:32 PM frno7 [email protected] wrote:
@treveradams https://github.com/treveradams, my Brother Innovis 955 machine seems to only make use of the stitches in the PEC, ignoring most parts of the PES. If this holds true for embroidery machines in general then the PES encoding format can be quite minimal in any SVG path → PES transcoding workflow.
The problem of translating SVG shapes (polygons, curves, text, etc.) into suitable SVG paths could then be treated separately, independently of PES and possibly any other embroidery format. Translating shapes into paths is a difficult problem but at least things simplify if one can ignore PES.
@JoshVarga https://github.com/JoshVarga, do you have any comments on these thoughts?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/treveradams/libpes/issues/1#issuecomment-272945134, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYcnN5xnyF5XuSl4I73_IoC12ELesVxks5rS8XKgaJpZM4BgWJO .
@treveradams, about the min/max stitch lengths etc. represented in SVG? It sounds interesting but unfortunately I don’t have much to say at the moment since I haven’t worked with those kinds of designs. So far I’ve simply programmed my own algorithms for stitch patterns, etc.
By the explanation of @JoshVarga (thanks!) I favour as a basis having a vendor neutral “machine format” representation in SVG (equivalent to PEC), essentially made up of SVG paths and perhaps a few necessary attributes but not much else.
As for “digitizing” general designs (for example SVG shapes) into machine format (assuming this is what you’re trying to do here?), there are obviously many degrees of freedom, and it’s a quite interesting problem. It seems likely that this too could be defined within SVG in some sense.
All the SVG shapes can be converted into SVG paths, and in theory PES would support those regardless. And much of these are generated within Brother's PE Design program. So in theory some parts will be necessarily lossy. For example, there's no stopping and starting points in SVG because there's nothing to be done there. There's no angle of the stitch, or stitch densities or any qualifiers on how the stitches are laid down. But, both formats certainly share shapes, not just the line segments EMBSewSeg but the other shapes would certainly be able to be directly transcoded. Even without explicitly specifying what the sew-commands are. You'd end up with a bunch of shapes as such, that can be loaded in Brother's PE Design and then that program could make the stitches then save it out, in the default way it would go about filling those shapes.
You'd only be feeding in the basic shapes and hoping Design PE fills them in somehow coherent. But, note this is similar to how a lot of modern Embroidery Software goes with them being addons to Corel Draw or Illustrator. They simply jump to the program after all the vector shapes are in, and tweak those settings that aren't relevant to vector shapes.
@frno7 As for the format neutral general design format, the degrees of freedom and the interesting problem it reveals. I've actually dedicated a very considerable amount of time to address it. There are expressly a wide range of different freedoms here. And we need to capture them all. What actually is needed is a format that can properly express the simple as well as the complex and do so just as easily. So you can express not just the shape that needs to be rendered but every way which it can be filled with stitches. It is not sufficient to simply list these ways, and start implementing what will end up being a massive program with every single way to articulate stitches, and every feature of every program has ever had. It needs to be done in a much more coherent fashion from what amounts to first principles.
A programming language for example, is useful because you can express anything in very useful and expressive syntax and explain large amounts what you're doing in a coherent fashion, while ultimately compiling down to bitcode/machine-code that a simple machine use. We have the same problem here with embroidery. Where you need to be able to express something complex like take this the letters TAT and express them as stitches. Each layer could well have added degrees of freedom, and power.
- "embroidered word TAT at location m"
- The letter T ending at the bottom, A ending at the bottom right, T starting from the bottom and ending where-ever.
- the glyph data of the various shapes with the starting and ending points without knowing they are letters.
- The boundaries of the glyph data, the satin stitch information, the underpathing information, the jump stitch information, the data for the next letter etc. *The physical x,y change location of the stitches, cut/jump etc.
But, what these actually make is a nested hierarchy, where you can make the lower level items from the expression of the higher level items. If you have the stitch information making the physical X,Y changes result in BAT rather than TAT is basically a non-starter. Each step you go up in the hierarchy it becomes more and more easy to make this change. There are also really crazy expressive vector line things like guilloches that could be expressed in embroidery but nobody does, as they don't at their core work directly on shapes. And so you need a format which can properly express all higher level and lower level things to do, and express the interrelationship of all the elements. Without, also limiting the format, to just those methods.
I've solved the intermediate issue. And can express a really useful format that is much more expressive than stitches while not being as limiting as, fill this particular shape. It basically amounts to a rendering pipeline of particular line types. Where we can express not just simple stitches, cuts, color-change, and jumps, but we can also express things like fill stitches and satin stitches without regard to the boundaries of the shapes that we're using to limit those geometric ideas. After all, a fill stitch relates directly with the geometric idea of monotony, equally there are satin stitches which relate to the idea of sweepability, and radials with have direct correlation with star-shaped polygons and all the maths therein. But, all of these can ignore the actual property of the shape they are filling and simply render in a ray-traced fashion (so you can fill a non-montone polygon as if it were montone and just miss a section. So you can take complex things like the algorithm for a tatsumi fill and break it down into intermediate steps. Thereby describing all the different quirks with all the different tatsumi fill algorithms that differ greatly between various programs. Or things like start and stop points within a shape and the complex patterns that describes or the underlay and underpathing as their own thing. We can break down the complex (fill this shape, with these start and end points) into the simple (draw this sequence of lines, but a few of these lines ray-trace to the edge of the shape they might be filling).
The ultimate goal of all of this is to figure out a language that is as expressive as stitches themselves, while also being simpler to express some superset of predefined fill patterns. And we can do this by understanding everything as a path the sewing machine generally takes and perturbations of these paths. Thus resulting in the simplest and most expressive language for embroidery. It's also, consequently nothing like how most modern embroidery is done, but is entirely able to express all of those forms of embroidery and a huge amount of forms that are basically missed. At the highest level you can express that you'd like to fill this shape starting at point p and ending at point q, but we need to break that down into the specific very expressive steps taking into account underlay, underpathing, the density of the thread, randomness at the edges, and all the huge number of potential options and turn that into an intermediate form then into the final stitch form. Since the intermediate form could well be final stitch form, it would rightly be, by definition, properly expressive. And properly with pathing and the core underlying geometry we can make considerably more expressive forms that still following paths, but because these forms are still fully free they can properly express the work of all higher level commands and all embroidery programs.
Mostly this is required because when you're faced with something new that doesn't fit into the classic colorforms style of modern embroidery, you're screwed. Say you want to add a spirograph. to your embroidery program. How does one possibly articulate that? What about if one program makes the needlepoints within a fill fit a particular pattern? What about if the fill they use includes a particular pattern itself? Or if you want a flat fill but having it curve up a little bit at the very end thus filling a non-monotone shape in an interesting way? How about if you're doing a series of different fills in the same direction, and plotting out all the underpathing, and end up needing make thousands of objects? What if somebody develops a better more interesting way of doing that same thing that basically amounts to finger painting monotone shapes in vorono diagrams shapes? There's way more things in embroidery than any modern embroidery program can express (except as a very naive running stitch), and a significant amount of these shortfalls is because of the assumption that embroidery is about shapes and how to fill them, rather than how can we quickly put down a lot of stitches in this interesting way. Modern embroidery is just one way of doing these things, and the trick is to develop something that can easily capture modern embroidery without limiting it. Much as modern programming languages allow you to express very complex ideas easily without losing the power of what machine code can do.
It's a very interesting problem and requires a lot of thought and work. But, it's well beyond the SVG spec. Even if we account for foreignObjects and switch statements and extending the format a lot in some reasonable and consistent ways. While expressing vector objects as svg paths would be fine and such, you cannot express the basically endless ways of filling that shape in any coherent fashion.
@tatarize, thanks for your thoughtful comments! SVG accommodates foreign namespaces and private data which I believe would be appropriate to handle many if not all of the embroidery stops, angles, densities, qualifiers, etc. that you mention.
As regards to an SVG based embroidery design format, I agree there are many degrees of freedom. For this reason I’ve decided to concentrate on the embroidery machine format (PEC in this case), which in analogy with your programming language example would be equivalent to the assembly/byte code. I’m optimistic we can capture all aspects of PEC within SVG in a reasonable manner. This simplifies tooling in as much that designs can be transcoded into SVG instead of having to deal with PEC directly.
One way to resolve the design complexity is to accept that there are several design formats for different purposes, some perhaps based on SVG, much like no single programming language is able to handle all desirable abstractions.
Even if we ignore the command level commands of PEC we could easily view what following those commands would result in as a completed embroidery very largely just say the command sequences are just line plots. Though this is admittedly lossy with regard to stops. It is valid to stop the machine in places where the actual resulting embroidery would not have any effect generally for things like applique. These also record the machine's work product and not how it did the jumps. It's rare that it's ever important but the embroidery machines really will jump around in a circle a bunch of times if you tell them to. Actually PEC blocks also contain the graphics data which cannot be represented in SVG though one assumes it to not contain much of anything.
There's a lot to PES that is specific to the Brother embroidery software used to produce it. Things like background colors, and hoops, and the handles for the objects. I tried breaking down EmbRect a bit, and determined that the block stores all the rectangles end to end, but also stores not just the 4 things you'd need but a bunch stuff like the position of the actual handles used to move the rectangle within that data. So you can within Brother define the rectangle in one place and the bit that moves the rectangle as being in another place. These are part of what PES actually is.
There are also things like how shapes are filled that aren't recorded anywhere save in the Software used to fill the area. Even if you are going to fill a "U" shape for example with a density of 4 lines per mm, and do a monotone fill from top to bottom going purely horizontal. There's still a huge number of ways to fill that. You can lay down an underlay first, to right side or left side first. Attach the bottom as part of the right or part of the left likely with some underpathing. To make this worse PES doesn't record that as a shape. It literally calls it a letter object (EmbLetter).
Assuming we could go through all the work to make a PES to SVG converter we could certainly do a lot of that. We'd need to realize most of the shapes perhaps the basic ones and skip over text, and get that working. And maybe there's some interesting ways to convert SVG equally into some of the core PES objects assuming they are mapped out well enough to write them. The only thing this would do is allow Brother PE to edit this stuff and then also use things like inkscape then convert it to PES. And then have brother generate the core stitches. It's entirely possible. In fact, I can currently do that with just the line objects themselves. Read the lines, write the svg lines, and back again. That's really easy. It wouldn't be super-hard to add to that some ability to do rectangles.
I program Touch Embroidery for the Android and one of its core formats is SVG. I know the standard well enough to know its power, and its limitations. But, the biggest issue isn't with SVG, it's in the heart of the computational embroidery: it's not vectors.
The lines, sure those work. And shapes and filling the shapes is a great way to lay down a lot of stitches very quickly, but the fact that the shapes are being filled with stitches means we're not really dealing with vector shapes. We're dealing with fills constrained by vector shapes. Yeah, if you want to convert shapes in PES to shapes in SVG and back again, that can be done. It might even be very nicely useful. You could equally do as a number of high end embroidery programs do and take a bunch of vector shapes and apply a series of algorithms to those vectors and get an embroidery design that you laid down a bunch of underpaths, fill stitches, underlay, and used some pretty solid metrics for how to do that. -- My objection to this is if you wanted to how would you define a spirograph or a satin fill with a curved element in the center.
It's certainly not needed for this task. Honestly one could hack out circles, rectangles and likely punches fairly quickly and perhaps get them working both ways in the PES<->SVG arena, in a week or so. At least provisionally working even if you don't plot out all the fills (after all the SVG doesn't have them, and your SVG file would be the only thing that cares). You might even just do punches for the same reason in SVGs you can get away with only doing paths. You can represent all the shapes with just that one element. The differences are that they are edited in different ways, you can change a circle in different ways than 4 bezier or 2 arcs that basically perfectly represent a circle.
Really I've explored SVG as embroidery rather extensively and I keep coming back to the core problem: embroidery isn't vectors. And when you address the proper problem it requires asking mathematically what is embroidery, like, as a thing. And secondly, what can you do to make that actually usable so that a puncher can put down all the stitches they need to put down in the shortest most efficient ways, which basically brings dozens of different solutions to that to mind, a goodly number of them that cannot be properly captured as vectors.
@tatarize, my personal workflow is to generate whatever layouts I want programmatically. So if a spirograph is desired, I’d make a program to output machine format SVG, and when the look is finalised transcode that to PEC using libpes.
So essentially I sidestep the embroidery shapes format complexities. Sometimes I block out a base design using Inkscape SVG, and then programmatically fill in the missing parts to obtain machine format SVG for conversion to PEC.
As for “EmbLetter”, I suppose FreeType would be a good start for rendering? Admittedly, it’s more difficult to layout stitches than pixels.
Yeah, but then if you wanted to adjust the design any, you have it in a bunch of line segments and can no longer modify it after saving. Part of the problem with the embroidery as vector ideas is that all this other stuff about how the shapes are filled must be saved. They simply are many times more important than the shape itself is. Basically unless you have all the parts you used to generate the stitches, you can't edit the stitches. Because really what needs to happen is that the stitches rendering elements need to change and the stitches need to be rerendered.
If it came down to getting the letters right, SVG does have text. It might be possible to simply have a text object there and match the letters as such. So you can get the shape more or less. But, in a lot of embroidery they have internal fonts and they are embroidery fonts which differ from actual fonts in the same way that embroidery differs from vectors, namely they have a bunch of information on how to fill the particular font with stitches.
The shapes might entirely be doable in a short period of time. The stitches are doable now. It's also worth noting that within PES it's often correct to use the PEC commands for stitches simply because if the PES has a higher level object like a circle or text or rectangle it will produce resulting stitches but those won't be found in the EmbOne/EmbSewSeg block. That only governs sew segments. Like any proper embroidery format PES stores the workers and the rendered stitches even though they are equal after you use PE Design to render them.
But, it seems like these things could be incrementally added. Figure out how to read and write EmbRect to Svg rect, and whatever scheme PES is using to combine two rectangles, it doesn't use a 1:1 ratio of blocks to shapes, it puts all the rectangles in the same place, so a rectangle, then circle, then rectangle will produce a EmbRect block with both rectangles, then a EmbCirc block with the circle, and somehow not screw up the ordering. It isn't too tall of an order to convert a number of simple shapes into the right class of shapes, or convert them all into a punch object (which would fill with stitches but be technically wrong in a lot of ways). Then if this is correct enough that Brother PE Design can load the file (it crashes a lot when you give it weird stuff) then that one high priced program could convert those shapes into stitches and save out a different PES object with the PEC blocks too.
The very lowest level parts of that conversion would certainly work. PES only has so many shapes. And while the info on how to fill them with stitches would be lost going to SVG and from SVG we'd have to throw on default settings it would be entirely plausible to do this and moreover wouldn't take too long. If I knew the precise ins and outs of what it's doing I'd toss it into my rather giant breakdown of PES.
https://jsfiddle.net/Tatarize/4hyx4e06/
Which reads a number of the blocks that it can, that are understood and simply hex dumps the bits it doesn't know yet. But, if things like EmbRect and EmbCirc were a bit better figured out it wouldn't be too hard to simply write that to the svg like I do for all the stitches. We'd simply need to ignore the parts of SVG that don't convert like things like gradients or higher level definitions etc. In theory going from PES to SVG would be easier, (reading is always easier than writing an unknown format), and that would be doable to even make the text, rectangles, punch -> path, etc. All with notable lossy nature losing the information of how those shapes are filled with stitches. But, that project would be entirely doable. Though I'm not sure how many people have fairly intricate PES files that they actually made entire vector pieces out of and would want a fairly worthless SVG representation of vectors used in the project.
The conversion, even if we used the SVG markup which I cannot guarantee the SVG editing software will maintain or that they will make any sense after the modifications is made, will generally be lossy. We will lose things in the conversion, even if they are commented what they were originally.
In the conversion from proper SVG to PES things are definitely lossy, things like gradients, paths with arcs, stroke widths etc, are all going to be lost. Assuming we breakdown any curves with small line segments, and arcs with small line segments, and maybe just use punch for everything and convert SVGs transforms either directly into actual physical shapes of those positions. An svg rect can be rotated by 45° and so can the EmbRect and all that would need to be accounted for, including guessing where PE Design would put the handles for the object. And to get stitches it would need to be loaded in the proprietary and expensive program "Brother PE design" and saved again to generate stitches.
There's a need for a format that can accurately work as a coherent open source object type that can capture all the intricacies of any embroidery with the least amount of information loss, but certainly SVG is not that format. Because embroidery is not vectors. And I seriously believe they are doing massive harm to embroidery as a whole, because clipart colorforms like drawing is not maximally expressive even if it does have the advantage of being quick and allowing re-editing.