Issue with clearing cache

[jwoertink]

I have a private shard that my app uses. Right now I just point it to master branch. When I do these steps, I seem to run in to a cache issue.

• Make update to shard
• Push master to github
• run shards update from project
• boot project and get error because it never actually updated the shard
• rm -rf lib/my_shard
• shards install
• boot project and now it's all good.

$ shards --version
Shards 0.10.0 (2020-04-06)
$ crystal -v
Crystal 0.34.0 (2020-04-07)

LLVM: 10.0.0
Default target: x86_64-apple-macosx

This is how I have my shard defined.

dependencies:
  my_shard:
    git: https://jwoertink:[email protected]/group/my_shard.cr
    branch: master

[jwoertink]

@waj I will try with the latest version to make sure and then report back. If it works fine, I'll close this out. Thanks!

[jwoertink]

Ok, just ran in to this again, but maybe a slightly different issue. I installed my shards, then I created an override and ran my shards update and things were good. Next I wanted to go back, so I removed shard.lock lib/ ~/.cache/crystal, and shard.override.yml. Then I shards install and I was back to original. Finally, I wanted to get back to the override one more time. I delete all those files, add my shard override, and run shards install and it still had the original version even though the lock showed the override. The lib still contained the wrong one. To fix this, I went in to the lib and removed just that one shard, and ran shards update and it was all good..

Seems like some combo of me running install vs update. Even though the lock file is being updated properly, the lib directory isn't.

[jwoertink]

Looks like I'm still running in to this issue. Using Shards 0.15.0. The shards install command says I'm installing the correct version. The shard.lock shows # Overridden and has the correct commit, but the lib/ directory contains the wrong one. Deleting just that folder from the lib/ and running shards update seems to sort of fix the issue... though, I'm running into an issue of trying to override a dependency of a dependency of a dependency...

A Lucky app requires the Lucky shard. I want to override the crystal-db shard which is nested like this:

Lucky > Avram > PG > DB

So in order to do this, I have to first add a local Avram to my app's shard.override.yml. Then I update the local Avram to point to a specific pg.

Running shards install installs the wrong Avram. Then I delete the lib/avram directory, and run shards update and it gives me the local one, but the wrong pg. I then delete lib/pg and update again to get the correct one, and so on..

It's either some weird race condition, or being cached somehow in a non ~/.cache/ spot?? I have no idea 😕

[straight-shoota]

It would help a lot if you could give us instructions on how to reproduce this issue from scratch.

[jwoertink]

Sure thing.

1. Create some shard, and push to github
2. Create an "app" that depends on that shard using git resolver and some branch
3. Run shards install
4. Make update to that shard, and push to github
5. Run shards update
6. Check that the change you just made exists in lib/that_shard

This doesn't quite follow the issues I'm running in to today, but that's essentially what I said in my original post at the top. My issue today might be related to something else (still digging around).

[straight-shoota]

I already tried to reproduce the issue on your original description, but I can't get to the failure state. That's why I'm asking for specific instructions, ideally a couple of shell commands which reproduces the issue in isolation. Bonus points if it doesn't depend on Github as an external service, but it's fine if that's in the loop.

[jwoertink]

Ok. I'll see if I can break it down further when I have a little more time to spend on it.

[matthewmcgarvey]

I think I'm seeing this issue when a shard I'm installing has a dependency on the overridden shard and has a postinstall step that calls shards build. It doesn't see that the dependency is satisfied through an override and install the version it wants, overwriting the override install.

To be specific, if I add an override of luckyframework/avram that points at the master branch and my project also has luckyframework/lucky then avram is installed as expected but then it install lucky and runs the postinstall step and I'm fairly sure that is re-installing the avram version it is expecting though I'm not 100% sure.

[jgaskins]

@matthewmcgarvey This precisely matches what I'm seeing at the moment. I'm using interro, which has a dependency on db and runs shards build to build the migration tool. So I'm trying to use the version of db from this PR in my override but db v11.0 is being installed from the canonical repo.

[jgaskins]

I found a workaround here. After running shards install, I remove the dependency I've overridden and rerun shards install:

shards install
rm -rf lib/db
shards install

It's a little messy, but it works. I just used it in my Dockerfile for one of my apps that needed an unreleased feature in crystal-lang/crystal-db.