nocopy option doesnt work without moving files (defeating part of the purpose)

[PCSmith]

Version information: 0.4.10

Type: Bug / Implementation Flaw

Severity: high

Description:

The "filestore" capability should be considered baseline for this project IMO. The expectation cant be to duplicate all bytes you want to place on IPFS by keeping a copy in the datastore... Nor can it be to expect users to throw their existing directory organisational structure out the window to manually copy IPFS candidate files to a central location.

Need the ability to --nocopy any file in any location and have that file not move or be copied anywhere. Also, if this --raw-leaves thing is needed for this it should be done by default. Not sure if it is though.

Thanks for your work on this world changing project.

[whyrusleeping]

@PCSmith ipfs can only add files within its directory context to the filestore as a security measure. Think of it like a git repository. If i remember correctly, symlinks work fine, so you can symlink your /datamount into your homedir and use the filestore from there.

[kevina]

Note that the .ipfs default location is the home directory. This exposes the entire home directory that is likely to contain a lot of sensitive information. Thus, at least with the current setup, I do not see this is a very convincing argument. Apologies if I am missing something obvious.

[PCSmith]

A few thoughts: I'm not linux right now, as with the majority of your likely intended audience for this project, I'm Windows. The "home" directory isnt where most data is kept. I keep nothing in there in fact, and dont care to as thats on my SSD.

My use case is making a several hundred videos from my youtube channel accessible via IPFS. The video files are dispersed through a file system meant to organize them with their other assets (clips, adobe premiere files, audio, etc). Copying them would be a ridiculous waste of space. Moving them all away from their support structure and accompanying assets would be more than inconvenient.

I do not argue the security concerns and I'm glad you guys are keeping an eye on it. Though exceptions accepted through positive actions should be possible for usability. And please dont put Windows last on your list of considerations. I'd argue it should be near the front for adoptions sake, not because I dont use and love linux.

If symlinking my video tree into the ipfs path will work for this I will def give it a shot. Thanks guys!

[PCSmith]

Before I spend the time deleting my store and recreating everything let me make sure I understand how this will work on Windows.

The IPFS executable existes in a folder on drive X. Lets say X:\IPFS\IPFS.exe which has been added to my paths. The data store through the IPFS_PATH environment variable exists in X:\IPFS\Datastore. If I symlink my youtube video tree in on X:\IPFS\YouTubes then filestore can work with those files without copying or moving them?

[whyrusleeping]

@PCSmith Hey, thanks for the feedback. Getting feedback and people pushing for better windows support definitely helps us prioritize things.

The symlink as you describe it should work. Let me know if you run into any issues, I have a windows VM around now and should be able to help debug.

[PCSmith]

Is raw-leaves required?

[djdv]

Random anecdote, I've added over a terabyte of data across many different files and folders via nocopy, utilising symlinks in the IPFS_PATH without issues on Windows (outside of issues relating to filestore commands that aren't implemented yet). I made a shell extension for Windows that makes a symlink of the target's parent folder and this is the primary method I use for now.

---

2021 edit: Key formats seem to have changed and I still get questions about this project. The last published version is at /ipfs/zDMZof1m1fX98cTLyC2VLe9iDQQhWgDLu5foshBSsxSWHQNuiyYV and the IPNS key is now /ipns/k51qzi5uqu5di8iluwqo958r5wf6vw7imzfww3zg1gi7br27ze7h3k93ddisr8 (it's the same keyfile after I imported it with ipfs key import from the old /ipns/QmaUgENG66kp6cyYUoiKREJWRaaQZmFt7EfFEnoMN1UvJZ key)

I haven't maintained this but if you replace the bundled ipfs.exe it probably still work. If you find it useful and want me to update it, reach out to me and I'll try to fix up the code so that the binary works and can release the code with it. Or consider IPFS Desktop.

[whyrusleeping]

@PCSmith yes, raw leaves is required. In the near future we will be defaulting that option to true for normal adds as well.

[PCSmith]

crap -- I added my entire library without raw-leaves. Do I need to clean and redo? What happens without it?

[whyrusleeping]

If youre using the --nocopy option, it turns --raw-leaves on automatically for you.

[PCSmith]

oh! perfect. thanks. You can close this out I suppose. But I think my criticism still stands if your target audience is your typical computer user. Symlinks and environment variables are probably not going to work for them. If thats not the audience then disregard. I'm just having trouble figuring how this is going to mainstream without laymen being able to easily seed / pin things / publish things.

The example is Dtube -- most of these people are uploading videos without realizing that unless their vids pay that dude enough to keep their files pinned on his hosting platform that their videos are not going to live long unless they're running their own node and pinning everything themselves. Because the browser version of ipfs cant exactly pin most videos (50 mb limit right?), and even if it could it would only be running while that page was open. I guess we're OT at this point. Just rambling.

Is there someone available for the project to be interviewed on my channel? I'd love to promote the project.

[PCSmith]

btw djdv -- it is so freaking awesome that you were able to deploy that site with video and download to IPFS. Loving the possibilities here.

[whyrusleeping]

But I think my criticism still stands if your target audience is your typical computer user

Well, the typical computer user won't be using the command line either. When we have a nicer user interface for ipfs, this sort of thing could be more easily automated and hidden away from the user.

and even if it could it would only be running while that page was open

Not true actually, using an ipfs service worker, you could have a js-ipfs node running in the background being used by any website that needs it.

Is there someone available for the project to be interviewed on my channel?

I would be interested, but i'm going to be traveling for a few weeks so it might be difficult.

[PCSmith]

When we have a nicer user interface for ipfs, this sort of thing could be more easily automated and hidden away from the user.

Is anyone working on that or is that an area I might contribute?

and even if it could it would only be running while that page was open

oh yeah! I forgot about those.

I would be interested, but i'm going to be traveling for a few weeks so it might be difficult.

Awesome. Let me know how or who I can get in touch with my producer to set it up?

[whyrusleeping]

Is anyone working on that or is that an area I might contribute?

there are a lot of different projects, but nothing that nice yet. I think the biggest issue is nobody really knows what is needed or wanted. Who are the users? What are the use cases? how do we best support that, etc.

Let me know how or who I can get in touch with my producer to set it up?

Grab my email from a commit (spam avoidance)

[kevina]

@PCSmith can I have a link to your channel? Your GitHub profile doesn't say much about you.

[PCSmith]

I emailed you a while back why...

Channel is here: https://steemit.com/@disenthrall https://dtube.video/#!/c/disenthrall https://www.youtube.com/DisenthrallMe https://www.facebook.com/Disenthrall/

[whyrusleeping]

Hey, I got your email. Just been moving around a lot without stable internet since then

On Sun, Oct 8, 2017, 7:17 AM Patrick [email protected] wrote:

I emailed you a while back why...

Channel is here: https://steemit.com/@disenthrall https://dtube.video/#!/c/disenthrall https://www.youtube.com/DisenthrallMe https://www.facebook.com/Disenthrall/

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ipfs/go-ipfs/issues/4224#issuecomment-334981952, or mute the thread https://github.com/notifications/unsubscribe-auth/ABL4HJUYLskhjX_W9h4dQ4rEU7xZtbMWks5sqEzhgaJpZM4PUJyC .

[PCSmith]

no worries. I look forward to the talk. :)

2 side questions:

1. How do I update a nocopy file in ipfs when I move it?

2. Is there a way to search IPFS to see if a file has been nocopy pinned?

[kevina]

1. How do I update a nocopy file in ipfs when I move it?

You can't right now please see https://github.com/ipfs/go-ipfs/issues/4260

2. Is there a way to search IPFS to see if a file has been nocopy pinned

The best you can do now is ipfs filestore ls to get the contents of the filestore this will only list the leafs not the pinned roots.

[Voker57]

@whyrusleeping could you please explain the security importance of root restriction, from which attack vector does it protect?

[renich]

I am typing randomly here but, I think this is solvable in a not-so-convoluted way by:

# create systemd service unit
cat << 'EOF' > /etc/systemd/system/[email protected]
[Unit]
Description=InterPlanetary File System
After=network.target

[Service]
ExecStart=/usr/local/bin/ipfs daemon --enable-gc --migrate
ExecStop=/usr/local/bin/ipfs shutdown
Group=%i
Restart=always
Type=simple
User=%i

[Install]
WantedBy=multi-user.target
EOF

# create a user for this purpose
useradd --create-home --home-dir=/var/lib/ipfs/ --system --shell=/bin/bash ipfs


# login as that user
su - ipfs


# init
ipfs init


# create relevant dirs
## mounts
mkdir -m 2770 mounts
mkdir -m 2770 mounts/{foo,bar}

## ipfs and ipns
mkdir -m 2770 ipfs ipns


# configure ipfs
## enable filestore
ipfs config --bool Experimental.FilestoreEnabled true

## set ipfs and ipfs mount points
ipfs config Mounts.IPFS $( pwd -P )/ipfs
ipfs config Mounts.IPNS $( pwd -P )/ipns


# exit ipfs user
exit


# go back to the user
su - ipfs


# check peers
ipfs swarm peers

# mount whatever directories you want
## bind existing directories. You could, also, add the entry at /etc/fstab:
## /home/renich/foo /var/lib/ipfs/mounts/foo none bind
## note: remember that the directory has to be readable by the ipfs user now. It's entirely up to you how you do this. I can think of: 
## * common group between users and ipset
## * ACLs
## * bindfs UID and GID mapping
## * add the ipfs user to the user's group (not recommended but pretty much how it currently works)
mount -o bind /home/renich/foo ~ipfs/mounts/foo

## mount a drive
## you could, also, add it to /etc/fstab
## /dev/sdXi /var/lib/ipfs/mounts/bar btrfs defaults
mount /dev/sdXi ~ipfs/mounts/bar


# exit ipfs user
exit


# start the daemon
systemctl start [email protected]


# back to ipfs user
su - ipfs


# add stuff
ipfs add --progress --recursive --nocopy $HOME/mounts/foo
ipfs add --progress --recursive --nocopy $HOME/mounts/bar

I mean, it's not as easy as curl some-script | bash but it works more less.

[ghost]

Hey @Renich please dont use IPFS for anything that could be regarded as copyright infringement. I've edited your comment slighty. For more info see https://github.com/ipfs/community/blob/master/code-of-conduct.md

Thanks for the script though, very nice.

[renich]

@lgierth sure thing. Just joking a bit. ;) Won't happen again.

[renich]

@lgierth btw, you missed a few. Updated again.

[PCSmith]

You are attempting to police content available on IPFS now? interesting... noted.

-Patrick Intellectual property is not a valid form of property.

[whyrusleeping]

Nope, not policing content on ipfs. Just the community forums that we spend our time maintaining for the sake of the community.

[iain17]

I'm wanting to do something similar. I have a use case where every peer in my IPFS network shares a certain directory with files up to 10gb or more. Where this directory is located differs per user. I'm a little confused as to what --nocopy actually does now.... How can I add these files to IPFS without duplicating each file to the ~/.ipfs/ home directory?

The app will mostly be used on Windows so I doubt the symlink solutions discussed in this issue are available right?

Perhaps a possible solution would be to add a option to add hashes of the files to the network, telling the network that this peer has the files. Then when a peer asks for a hash which IPFS can't find in its own filestore, we have a callback to return a io.Reader of different locations where it could... Giving the end user some flexibility.

[Voker57]

@iain17 if users store data in their $HOME, they can use --nocopy, if not, they can add a symlink to their $HOME.

I'm still mystified on how that restriction improves security, however.

[iain17]

@Voker57 ah thanks for clearing that up. Same here. Do you know by any chance where in the code base this check if its inside of the home directory is done?