packr
packr copied to clipboard
gobuffalo
packr2: allow mixed virtual + file system boxes
packr is setup such that a box is purely virtual or purely file system, and virtual is always preferred over file system.
In our environment, we need the opposite: our asset compilation pipeline is fairly complex, and run only by the engineers who work on the service. Those engineers need the file system to dominate, since it represents their in-progress work. Afterwards they regenerate and commit updated packr boxes, but not any of the intermediate or compiled on-disk assets. This allows everyone else (who isn't working on that service) to just run the service without doing any asset compilation themselves.
I have a fairly clean patch (I think) to support this that I would like to submit, if there's a chance it might be accepted?
If I am not mistaken, packr2 offers this exact scenario.
Also, I believe that packr1 is not developed anymore. Is is possible for you to migrate to packr2 ?
So I played around with this in packr2, and there's no obvious way to make this work out of the box. My initial stab was to try this:
type resolveFunc func(string, string) (file.File, error)
func (f resolveFunc) Resolve(box string, name string) (file.File, error) {
return f(box, name)
}
func chainResolvers(resolvers ...resolver.Resolver) resolver.Resolver {
return resolveFunc(func(box string, name string) (file.File, error) {
var ret file.File
var err error
for _, r := range resolvers {
ret, err = r.Resolve(box, name)
if err == nil {
return ret, nil
}
}
return ret, err
})
}
func LocalDevBox(box *packr.Box) {
r, ok := box.DefaultResolver.(*resolver.HexGzip)
if !ok {
panic(fmt.Sprintf("not a hex gzip box: %T", box.DefaultResolver))
}
box.DefaultResolver = chainResolvers(&resolver.Disk{Root: box.ResolutionDir}, r)
}
The problem centers around the special treatment of box.resolvers resolversMap. The construction makes it supremely difficult to do what I'm trying to do -- I don't really grok the reason having different resolvers at each file path.
I think an easier-to-work-with construction would be:
- generated boxes have a DefaultResolver setup with static data
- get rid of
box.resolvers - make Resolver implement FileMap
If we did this, it would be easy to chain / layer resolvers, and the implementation of box.Walk would be much simpler.
I'm not against a PR on this. My biggest question is is it a breaking change? if so, what does that look like?
Good question-- is the ability to set a different resolver for per path an important design choice, or an implementation detail for constructing pointer boxes?
I've been looking through packr2 today, and there seems to be a lot more there than packr1, but I haven't yet been able to distill out the design philosophy for packr2. Do you have something that explains packr2 vs. packr1, in terms of what packr2 is trying to accomplish?
It was, and is, import to be able to resolve a "file" a different way than others. for example, a CMS is using it so they can have some files local, and others in a DB/cache.
Anyway, I can give you more details, but you get the idea.
I think we're definitely talking a breaking change, but one that could be done as a v3, as I quite like the idea.
My thoughts for would this pattern would look like is something like this:
type ResolverBox interface {
packd.Box
Resolve(box string, file string) (file.File, error)
}
func WithResolver(b ResolverBox, boxName string, file string, res resolver.Resolver) ResolverBox {
rb := resolverBox{
ResolverBox: b,
boxName: boxName,
file: file,
res: res,
}
return rb
}
type resolverBox struct {
ResolverBox
boxName string
file string
res resolver.Resolver
}
func (b resolverBox) Resolve(box string, file string) (file.File, error) {
if box == b.boxName && file == b.file {
return b.res.Resolve(box, file)
}
return b.ResolverBox.Resolve(box, file)
}
that's a quick thought on the subject.
any interest in working up a proposal/PR on this?
Ignore my previous bad code idea. A box could have a single resolver and you can just wrap it like middleware On my phone or I’d type it out. But basically what you’re were going for.
Been stewing on this and I think I may have a backwards compatible idea that retains the ability to do file-specific resolution. Seems like the core of Box could be something like:
type Box struct {
Path string
Name string
resolvers []resolver.Resolver // resolvers in priority order
}
resolvers would be an ordered list of resolvers to try. Resolver would have to do slightly more-- more of what's in Box would probably be delegated. For example, implementing Walk would probably entail all Resolvers implementing FileMappable, and Box would just merge the results together such that earlier resolver paths would take precedence over later ones.
The upside is that Box gets really clean: at the moment it does a bunch of dirty things like look at disk directly, when in reality it should just have a Disk resolver to delegate these parts to.
AddString / AddBytes could be done by checking if the first resolver in the chain is an in-memory resolver; if so, add to that; if not prepend a new in-memory resolver to the chain first.
What do you think of the general idea?
(A wrap-and-delegate chain could also an effective strategy, although it might be harder to correctly implement resolvers.)
@dragonsinth If I am not mistaken, you could achieve this a bit easier by using spf13/afero‘s CopyOnWriteFS, defining a boxes HttpFS as the read-only part and your OS‘s FS as the overlay part. Did not test that, but I guess it should work, it puts the layer of abstraction where it belongs to (from an actual FS implementation to a dedicated abstraction layer) and it is pretty much done.
