Need file version strategy

[32Beat]

It appears there is no version key encoded in an inkpad file. If so, I would like to suggest the following:

It would be useful to transition to encoding textbased values. Textbased values are platform agnostic and therefore have several benefits: they are not affected by byte-order, byte-size, or format. For example CGPoint may be 32bit floats or 64bit floats, not even sure which ieee format. Encoding them via NSStringFromCGPoint etc., solves a lot of headaches.

It probably would be sensible to implement this transition together with file-versioning.

[sprang]

At one point I wanted to make SVG the native file format. I think it's probably not too much work to make that happen now. Scott Vachalek was working in this direction a long time ago, but it was never thoroughly tested. There are still issues like https://github.com/sprang/Inkpad/issues/17

[ghost]

We had great success with this in Inkscape (using svg) since the standard is complete and solves thinking too much about too many things.

[32Beat]

I would personally argue against using SVG as the native format, since you might want to store metadata and user specific data that has no interchangeable value. (e.g. workspace data like placement of tools or something...)

In addition, it might be possible to subclass a keyed archiver to write SVG, so the initial problem is merely storing the data in the coder.

I have build an example in my latest WDBezierSegment rewrite branch. DO NOT try to merge it, as there have been too many changes made, but it serves as an example of what seems to work perfectly with existing files. I can open a new branch for just this specific change, if we agree that it is useful.

The pertinent parts: First we add some keys for the different parts of the WDBezierNode object Note that the coding keys end with "Key", as opposed to the actual keystring. It is possible to write textbased plist files which then read like:

WDBezierNodeAnchorPoint = { 0.0, 0.0 }

Note also that the current version is set to 1. We are indicating the version of the WDBezierNode contents, not of the entire file. In addition, due to keyed archiving, we do not expect to change the internal data very often.

static NSString *WDBezierNodeVersionKey = @"WDBezierNodeVersion";

static NSInteger WDBezierNodeVersion = 1;
static NSString *WDBezierNodeAnchorPointKey = @"WDBezierNodeAnchorPoint";
static NSString *WDBezierNodeOutPointKey = @"WDBezierNodeOutPoint";
static NSString *WDBezierNodeInPointKey = @"WDBezierNodeInPoint";

static NSInteger WDBezierNodeVersion0 = 0;
static NSString *WDBezierNodePointArrayKey = @"WDPointArrayKey";

Next we encode the WDBezierNode by only storing the necessary data. That is: if our node does not have handles, they can be skipped.

- (void)encodeWithCoder:(NSCoder *)coder
{
    // Save format version
    [coder encodeInteger:WDBezierNodeVersion forKey:WDBezierNodeVersionKey];

    // Save Anchorpoint
    NSString *A = NSStringFromCGPoint(anchorPoint_);
    [coder encodeObject:A forKey:WDBezierNodeAnchorPointKey];

    // Save outPoint if necessary
    if ([self hasOutPoint])
    {
        NSString *B = NSStringFromCGPoint(outPoint_);
        [coder encodeObject:B forKey:WDBezierNodeOutPointKey];
    }

    // Save inPoint if necessary
    if ([self hasInPoint])
    {
        NSString *C = NSStringFromCGPoint(inPoint_);
        [coder encodeObject:C forKey:WDBezierNodeInPointKey];
    }
}

The useful part about keyed archiving is that decoding the version if it is not available results in 0. More over, we could still attempt to read the keys we do think we can understand, even if the archived data version doesn't match. (Although that isn't implemented here):

- (id)initWithCoder:(NSCoder *)coder
{
    self = [super init];
    if (!self) return nil;


    NSInteger version =
    [coder decodeIntegerForKey:WDBezierNodeVersionKey];

    if (version == WDBezierNodeVersion)
        [self readFromCoder:coder];
    else
    if (version == WDBezierNodeVersion0)
        [self readFromCoder0:coder];

    // Test for valid node
    if ([self isValid])
    { return self; }


#ifdef WD_DEBUG
NSLog(@"%@",[self description]);
#endif

    // Test for recoverability
    if ([self recoverContents])
    { return self; }

    //TODO: corrupt file notification stategy
    return nil;
}

To recover from corrupt nodes, we could add a "state" to the WDBezierNode and try to recover from partly corrupt data, and report this in the state. The application could then check all the node->state and report back to the user accordingly.

- (BOOL) recoverContents
{
    long bitMask =
    CGPointIsValid(self->anchorPoint_)+
    2*CGPointIsValid(self->outPoint_)+
    4*CGPointIsValid(self->inPoint_);

    // Test for valid anchorpoint, recover handles if necessary
    if (bitMask & 0x01)
    {
        if ((bitMask & 0x02) == 0)
        { self->outPoint_ = self->anchorPoint_; }
        if ((bitMask & 0x04) == 0)
        { self->inPoint_ = self->anchorPoint_; }
    }
    else
    // Test for 2 valid handles, must recover anchor
    if (bitMask == 6)
    {
        self->anchorPoint_ =
        WDInterpolatePoints(self->inPoint_, self->outPoint_, 0.5);
    }
    else
    // Must recover anchor & out
    if (bitMask == 4)
    {
        self->anchorPoint_ = self->inPoint_;
        self->outPoint_ = self->inPoint_;
    }
    else
    // Must recover anchor & in
    if (bitMask == 2)
    {
        self->anchorPoint_ = self->outPoint_;
        self->inPoint_ = self->outPoint_;
    }
    else
    {
        self->anchorPoint_ =
        self->outPoint_ =
        self->inPoint_ = CGPointZero;
    }

    // Invert bitMask: bits then indicate recovered values
    self->state_ = bitMask^0x07;

    // Report recoverability
    return bitMask != 7;
}

The reading otherwise is trivial

- (BOOL) readFromCoder:(NSCoder *)coder
{
    // Read anchorPoint first, assign to all
    NSString *P = [coder decodeObjectForKey:WDBezierNodeAnchorPointKey];
    if (P != nil)
    {
        anchorPoint_ =
        outPoint_ =
        inPoint_ = CGPointFromString(P);
    }

    // Assign outPoint if available
    P = [coder decodeObjectForKey:WDBezierNodeOutPointKey];
    if (P != nil) { outPoint_ = CGPointFromString(P); }

    // Assign inPoint if available
    P = [coder decodeObjectForKey:WDBezierNodeInPointKey];
    if (P != nil) { inPoint_ = CGPointFromString(P); }

    return YES;
}

The previous decoding

- (BOOL) readFromCoder0:(NSCoder *)coder
{
    const uint8_t *bytes =
    [coder decodeBytesForKey:WDBezierNodePointArrayKey returnedLength:NULL];

    CFSwappedFloat32 *swapped = (CFSwappedFloat32 *) bytes;

    inPoint_.x = CFConvertFloat32SwappedToHost(swapped[0]);
    inPoint_.y = CFConvertFloat32SwappedToHost(swapped[1]);
    anchorPoint_.x = CFConvertFloat32SwappedToHost(swapped[2]);
    anchorPoint_.y = CFConvertFloat32SwappedToHost(swapped[3]);
    outPoint_.x = CFConvertFloat32SwappedToHost(swapped[4]);
    outPoint_.y = CFConvertFloat32SwappedToHost(swapped[5]);

    return YES;
}

[ghost]

For workspace and metadata make a namespace like inkpad and then add in what you like. Inkscape does this very well inkscape svg should be studied.

[32Beat]

The problem is not that it can't be stored in SVG, the problem is that it has no interchange value there. It doesn't need to be read by everyone, and in fact, it could pose a security risk. And again: the NSCoder can probably already pose as an interface between our coding strategy and writing SVG or some other desired format. So, eventually you could make it a user preference how the data is written, but internally we always get a coder object.

[eonist]

Using SVG has another benefit. It's version controllable. If you want to collaborate over git. Although the best format is probably an SQLite .db file. Similar to how Sketch stores .sketch formats. Why use SQLite? Its speedy. But not version controllable. I guess one could toggle the two formats in prefs.

[JanX2]

You can try and do what I do in one of my apps that handles CSV files: use extended file attributes for workspace and metadata. The problem is, that they can get lost in transfer. This is probably more of a problem when exchanging files via email and such. Maybe they can survive Dropbox and other such hosters.

[eonist]

@janX2 Didnt know you could store any meta of any significant length in the fileformat it self. Nice to know! I'm planing to make an opensource vector fileformat. My current implementation is just xml. I plan to make it compatible with svg similar to inkscape. But im also planining on parsing it into another fileformat while in use. Sqlite of some sort so that the app can read and write really fast. And while in use in the app I will update the svg file, on a background thread. Why do I want to keep it svg? Because I want the fileformat to be version controll friendly.

[JanX2]

@eonist I should have thought of that, of course. Having edited Inkscape SVG in a text editor before. ;)

[eonist]

@JanX2 im not 100% sure sqlite is needed in the mix. But my research into the matter suggests that this is how .sketch files and .ai files do it. Performance tests will assert if the idea is good or not. As i mentioned, the format will be opensource so this or other projects may use it. For future reference the format will be on my github account.

[alistairmcmillan]

@eonist I'm assuming you know this already but just in case.

Gus Mueller of FMDB fame uses an sqlite file format for his Acorn image editor.

http://shapeof.com/archives/2013/4/we_need_a_standard_layered_image_format.html

https://github.com/ccgus/lift

[eonist]

@alistairmcmillan Thx. Didn't not know that. Added the Links to my future blog post about this format. I read the article in the link and the reason Adobe doesn't use an open format is because they want to keep people in. Not let them escape. :) Sketch is no better in this regard, as its almost impossible to migrate to .ai file or .svg file if you happen to have a .sketch file and not the app it self.

The problem with a pure .sqlite based format is that it isn't version controllable. I think using git with design assets is the future. Figma design will use a format that is version controllable and they will sync the file-format to other designers etc.

So keeping the SQLite bit just for internal usage to benefit performance and then siphon off data to a xml/svg like format might just work.

• I might just have the experience to build such a format because:

1. I built a SQLite CRUD application in applescript (shell so its portable to swift with little effort) https://github.com/eonist/SqliteEdit
2. I have built an automated Git application (already ported to swift) https://github.com/eonist/GitSync
3. I have a SVG engine that parses and writes SVG: https://github.com/eonist/swift-utils/tree/master/geom/svg
4. I have an implementation of an XML format that supports layers, graphic states, etc.

This is the an example of the format I use:

<?xml version="1.0"?>
<document x="264" y="42" width="800" height="600">
  <canvas width="800" height="600">
    <selectpath name="rect">
      <commands>2 2 2 2</commands>
      <pathdata>100 100 200 100 200 200 100 200</pathdata>
      <linestyle color="#000000" alpha="1" pixelHinting="true" lineScaleMode="normal" capStyle="none" jointStyle="miter" miterLimit="1.414"/>
    </selectpath>
    <layer name="layer" isLocked="false"/>
    <selectpath name="b">
      <commands>1 2 2 2 2</commands>
      <pathdata>497.5 544 503 477.5 607.5 477 627.5 541 719.5 540</pathdata>
      <linestyle color="#000000" alpha="1" lineScaleMode="normal" capStyle="none" jointStyle="miter" miterLimit="1.414"/>
    </selectpath>
  </canvas>
</document>

The format is by no means done. It will have similarities to SVG. And I could even make it compatible with SVG by doing what InkScape are doing. adding meta data around each node so that its still parseable with old SVG parsers.

The reason I like SVG is that its viewable in Finder preview and works in any browsers. However I think i had some issues to get around the layer vs group problem. You can store shapes in groups and use that as layers when they come in to your application. But then you cant have groups. And you want both sort of. I can recall that I didn't like how InkScape handled this so I just made my own .xml format to get things working. By starting a fresh file format you can add stuff you like faster so I will continue to do that to keep a nice speed. and then when I feel its in good shape I will see if i can merge the .xml format with .svg by utilising meta data in the SVG nodes.