git-sync
git-sync copied to clipboard
kubernetes
Add git-lfs to image
I'm requesting git-lfs be added to the image. This issue came up back in 2019 #174. The issue was closed without adding git-lfs to the image. I tested v3.6.2 by adding git-lfs to the image and I'm syncing LFS files from BitBucket without any additional configuration.
This is my dockerfile.
FROM k8s.gcr.io/git-sync/git-sync:v3.6.2
USER root
RUN apt-get update
RUN apt-get -y install git-lfs
USER 65533:65533
This is my container spec within my deployment in k8s.
spec:
template:
spec:
containers:
- name: git-sync-lfs
image: example-docker-hub/git-sync-lfs:v3.6.2
args:
- -ssh
- [email protected]:example-private-repo/lfs.git
- -root=/data
- -dest=lfs
- -branch=master
- -depth=1
- -wait=60
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: lfs-data
mountPath: /data
- name: git-secret
mountPath: /etc/git-secret
securityContext:
runAsUser: 65533
I would want an e2e test for this - how do I force this to be used so I know it works?
Do existing tests not cover this? Can I not just add git-lfs package in a Dockerfile using apt-get then run tests on new image?
My background is sysadmin. I'm new to devops. I know some bash, docker, and k8s. You want me to write a test to prove that it works? Not sure how to do that.
I forked the repo. I'm looking at Dockerfile.in and test_e2e.sh. I'm not sure where to add git-lfs package in Dockerfile.in. Do I pass it an image:tag as an argument to run tests?
If you can show me how to craft an e2e setup which specifically proves LFS works, that would be great. You can see in that file that each testcase is a little snip of shell code which sets up a scenario. What I don't quiote grok is how to set up a repo to use LFS and how to prove that it works.
On Thu, Dec 22, 2022 at 4:56 PM James Ringer @.***> wrote:
Do existing tests not cover this? My background is sysadmin. I'm new to devops. I know some bash. You want me to write a test to prove that it works? Not sure how to do that. I went ahead and forked the repo. I'm looking at test_e2e.sh. How can I help?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
Does this help? GitHub - Configuring Git LFS
GitHub provides some storage and bandwidth for LFS for free. GitHub - About Storage and Bandwidth
We can do a call and work through it together if you'd like. I set it up on BitBucket, but it's probably similar on GitHub. The BitBucket repo I have setup is private, but I could setup a public repo if that helps.
Afaik the only thing that will be different after adding the git-lfs package will be that if a repo contains lfs files they will be synced rather than the text files that point to them. This makes me think existing tests should be fine, but I'm far from and expert.
To do an e2e I need some way to replicate it locally or at least without any auth to an external service. Possible?
On Thu, Dec 22, 2022, 5:23 PM James Ringer @.***> wrote:
Does this help? GitHub - Configuring Git LFS https://docs.github.com/en/repositories/working-with-files/managing-large-files/configuring-git-large-file-storage
We can do a call and work through it together if that helps. I set it up on BitBucket, but it's probably similar on GitHub.
— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1363494506, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVG5E3GCH32QE7KMKMDWOT5I7ANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>
If you setup a public GitHub repo with LFS I don't think you need authentication.
Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo?
The existing e2e tests use a file:// repo or an SSH repo (still backed by
files).
It looks like we can build an LFS test-server binary ( https://github.com/git-lfs/lfs-test-server) to run tests against, but there's not a standard apt pkg for it, so we'll have to pull and build it as part of the test. Look in _test_tools. That's not impossible, it just makes this a little more involved. At least it is written in Go, so we can drop it into tools/go.mod. :)
If you have bandwidth, it's appreciated, otherwise I'll get to this eventually, I hope. :)
On Fri, Dec 23, 2022 at 7:10 AM James Ringer @.***> wrote:
Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo?
— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1364029692, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVFGCCNKCP5ONWDXENDWOW6E7ANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>
In my fork I have setup lfs-test-server using the sshd folder as an example.
_test_tools
├── lfs-test-server
│ ├── Dockerfile
│ ├── README.md
│ └── run.sh
├── ncsvr
│ ├── Dockerfile
│ ├── README.md
│ └── ncsvr.sh
└── sshd
├── Dockerfile
├── README.md
├── sshd.sh
└── sshd_config
I am using env vars LFS_USERNAME and LFS_PASSWORD to configure lfs-test-server management username and password. Looking at sshd example I'm assuming we don't want these committed to the repository. Not sure how I'm going to do that. If you don't set these the management interface is not accessible. Also, it looks like you have to login and create a user to use lfs-test-server. Not how I'm going to work around that either. Any ideas?
Ugh, what a cumbersome piece of software to automate. How about something like this Dockerfile:
FROM alpine AS base
RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out \
alpine-baselayout \
apk-tools \
curl \
&& true
###############
FROM scratch
ENTRYPOINT []
WORKDIR /
COPY --from=base /out/ /
COPY lfs-test-server /
RUN LFS_ADMINUSER=admin \
LFS_ADMINPASS=admin \
timeout 2 \
/lfs-test-server & \
sleep 0.5 \
&& curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e
ENTRYPOINT ["/lfs-test-server"]
I did a quick test and that SEEMS to work, as long as lfs-test-server is built with CGO_ENABLED=0 (static binary).
Now you can run the image in docker, get the IP (see ssh examples) and us username "e2e" password "e2e".
I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed.
That's because it's not easy to make "trivial" SSH keys, whereas this case is super easy :)
On Mon, Dec 26, 2022 at 4:34 PM James Ringer @.***> wrote:
I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed.
— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1365522636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVFVFQ4LEKCP6F7JCZLWPI2TDANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>
That makes sense. I just wanted to check before moving forward. Btw adding the user with curl that way is just what I was looking for. I'll keep working on this. Thanks.
Haven't circled back to this yet, but I will. Please don't let the issue go stale and close. Thanks!
Hi. I am hoping to have a v4.0.0 release candidate some time in the next month or so - would love to have this in. No pressure, we can always add it later.
I'm working on test_e2e.sh. I added a section to init lfs-test-server.
# Init Git LFS Test Server
# Clone lfs-test-server to _test_tools
git clone https://github.com/git-lfs/lfs-test-server ./_test_tools/lfs-test-server
# Overwrite lfs-test-server Dockerfile
tee ./_test_tools/lfs-test-server/Dockerfile <<EOF
FROM golang:1.18
WORKDIR /go/src/github.com/git-lfs/lfs-test-server
COPY . .
RUN go build
EXPOSE 8080
ENV LFS_ADMINUSER=admin
ENV LFS_ADMINPASS=admin
RUN /go/src/github.com/git-lfs/lfs-test-server/lfs-test-server & \
sleep 0.5 \
&& curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e
ENTRYPOINT ["/go/src/github.com/git-lfs/lfs-test-server/lfs-test-server"]
EOF
# Build the lfs-test-server image
docker build ./_test_tools/lfs-test-server -t example.com/lfs-test-server:latest
# Run the lfs-test-server container
docker run --rm -d -p 8080:8080/tcp example.com/lfs-test-server:latest
The shell snippet works. Idk if that's what you had in mind. I'm also looking at modifying the init_repo function.
function init_repo() {
rm -rf "$REPO"
mkdir -p "$REPO"
git -C "$REPO" init -q -b "$MAIN_BRANCH"
touch "$REPO"/file
git -C "$REPO" add file
git -C "$REPO" commit -aqm "init file"
cd "$REPO"
git lfs install
git lfs track "*.jpg"
cd ..
echo "[lfs]" > "$REPO"/.lfsconfig
echo " url = \"http://localhost:8080/\"" >> "$REPO"/.lfsconfig
git -C "$REPO" add .lfsconfig .gitattributes
git -C "$REPO" commit -aqm "lfs config"
touch "$REPO"/lfs.jpg
git -C "$REPO" add lfs.jpg
git -C "$REPO" commit -aqm "add lfs.jpg"
rm -rf "$REPO2"
cp -r "$REPO" "$REPO2"
}
To confirm the lfs tracked file is sent to lfs-test-server we need to git push, but for that we need a remote.
I don't think you need to do this in the common init_repo, but in the specific LFS test case?
On Sun, Mar 12, 2023, 3:28 AM James Ringer @.***> wrote:
I'm working on test_e2e.sh. I added a section to init lfs-test-server.
Init Git LFS Test Server
Clone lfs-test-server to _test_tools
git clone https://github.com/git-lfs/lfs-test-server ./_test_tools/lfs-test-server
Overwrite lfs-test-server Dockerfile
tee ./_test_tools/lfs-test-server/Dockerfile <<EOF FROM golang:1.18 WORKDIR /go/src/github.com/git-lfs/lfs-test-server COPY . . RUN go build EXPOSE 8080 ENV LFS_ADMINUSER=admin ENV LFS_ADMINPASS=admin RUN /go/src/github.com/git-lfs/lfs-test-server/lfs-test-server &
sleep 0.5
&& curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e ENTRYPOINT ["/go/src/github.com/git-lfs/lfs-test-server/lfs-test-server"] EOFBuild the lfs-test-server image
docker build ./_test_tools/lfs-test-server -t example.com/lfs-test-server:latest
Run the lfs-test-server container
docker run --rm -d -p 8080:8080/tcp example.com/lfs-test-server:latest
The shell snippet works. Idk if that's what you had in mind. I'm also looking at modifying the init_repo function.
function init_repo() { rm -rf "$REPO" mkdir -p "$REPO" git -C "$REPO" init -q -b "$MAIN_BRANCH" touch "$REPO"/file git -C "$REPO" add file git -C "$REPO" commit -aqm "init file"
cd "$REPO" git lfs install git lfs track "*.jpg" cd .. echo "[lfs]" > "$REPO"/.lfsconfig echo " url = \"http://localhost:8080/\"" >> "$REPO"/.lfsconfig git -C "$REPO" add .lfsconfig .gitattributes git -C "$REPO" commit -aqm "lfs config" touch "$REPO"/lfs.jpg git -C "$REPO" add lfs.jpg git -C "$REPO" commit -aqm "add lfs.jpg" rm -rf "$REPO2" cp -r "$REPO" "$REPO2"}
To confirm the lfs tracked file is sent to lfs-test-server we need to git push, but for that we need a remote.
— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1465152804, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVAHFJNHWUSATBMSGLDW3WQMDANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>
I've been looking at the repo trying to understand how to do this, but I don't quite follow how everything works. Would it be better if I put the script at /test-lfs.sh?
Do you have a branch pushed to GitHub? I can clone it later this week and see what I see.
On Mon, Mar 13, 2023, 12:00 PM James Ringer @.***> wrote:
I've been looking at the repo trying to understand how to do this, but I don't quite follow how everything works. Would it be better if I put the script at /test-lfs.sh?
— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1466774042, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVDC4EU4YWEQ22QIPHDW35VF3ANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>
Tbh, there's not much to see at this point. After working on it a bit last weekend I have a better idea of which direction to go. I'm going to work on it again this weekend a bit more, so we'll see how it goes.
I worked with it a bit more and made a little progress. I got a lfs-test-server container along with a git-server-docker container setup and was able to push a commit to git-server-docker containing a jpg that was pushed to lfs-test-server.
I'll throw a bit more time at it next weekend. I feel it's starting to come together, but there is still a ways to go. Ultimately, I want to automate the setup of an environment where we have a repo in git-server-docker and a large file in lfs-test-server such that when we clone the repo with git-sync it pulls the large file with the repo.
Are we on the same page with that goal?
I think that's right - enough to prove that LFS was activated and works.
On Sun, Mar 19, 2023 at 9:40 PM James Ringer @.***> wrote:
I worked with it a bit more and made a little progress. I got a lfs-test-server container along with a git-server-docker container setup and was able to push a commit to git-server-docker containing a jpg that was pushed to lfs-test-server.
I'll throw a bit more time at it next weekend. I feel it's starting to come together, but there is still a ways to go. Ultimately, I want to automate the setup of an environment where we have a repo in git-server-docker and a large file in lfs-test-server such that when we clone the repo with git-sync it pulls the large file with the repo.
Are we on the same page with that goal?
— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1475611199, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVEEDNZECMWQG2CD7UTW47NTLANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>
I have not had time the past couple of weeks. Tomorrow I'll have some time to pick this up again. Idk how far I'll get, but I'll keep at it until it's done.
I talked with my manager and received approval to start dedicating some of my work time to OSS. This is at the top of my list!
@thockin I have not forgotten about this, but I'm on a modified schedule until the end of June with no time for oss. I'll circle back to this in July. Sorry I'm slow and inexperienced.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue as fresh with
/remove-lifecycle stale - Close this issue with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
