attic
attic copied to clipboard
jborg
helping out with merge reviews and issue triage
hi
there seems to be some people around here that could be useful to help out with bug triage, documentation and issue review. having submitted #155 and others and waiting for it for a few weeks, I do feel there is a worrisome bottleneck there, especially if we end up in a situation where there is an urgent security fix to push through (for example).
wouldn't it be a good idea to start opening up access to the repository or at least issue queue? github supports organisations that could reflect such a workflow. unfortunately, access control isn't very granular in those setups so may this repo could keep on being the canonical repository until trust is build up? the atticbackup namespace is free right now. :)
as a data point: there are currently 18 pending pull requests and 101 opened issues. a lot of those should be just merged and closed as they are fairly trivial documentation fixes or answered support requests. in the last month, 19 new issues were created, and only 2 of those closed, by the authors of the issues. 6 new pull requests were created and none were merged in.
i may have some time to contribute to the triage effort.
I'ld also help.
I contacted github team, trying to get "attic" as org name - github has some policy about existing, but unused accounts that might enable getting this.
I really like attic and I'd love to see further improvements! Right now the progress is kinda slow, so maybe it's the right way to go.
+1.
If you scroll to the bottom there's a guide on how to transfer a project. (if @jborg would like to do so) http://git-scm.com/book/en/v2/GitHub-Maintaining-a-Project
just to be clear here, i'm not talking about transfering ownership: i believe that @jborg should stay as a maintainer. i am just looking for ways to help.
i think it's great that people are joining up here, i hope it actually means that people will contribute. note that it's already possible for people to file issues, comments and pull requests here :)
I made a "merge" branch in my repo https://github.com/thomaswaldmann/attic where I merged all the stuff that looked safe (I reviewed it first, did unit and practical tests).
https://github.com/attic \o/
github support was so nice to free this for us and I quickly grabbed it for our org before someone not from us takes it again. Sent out some first invitations, @jborg was first.
@jborg: if you have some time to help transfering to the org, you could enable a lot of collaboration. :)
https://help.github.com/articles/transferring-a-repository/
i put a link to this issue in the new team to clarify its purpose and it's ... well, unofficial nature until @jborg chimes in!
@ThomasWaldmann which criteria did you use to add people to the team?
which criterion should be used?
I just grabbed some people who made pull requests (which I reviewed and found safe to merge). The list is not expected to be the final list, I just wanted to avoid creating another bus factor problem there.
I hope we can (with the help of @jborg) transfer this repo with all issues and pull requests to there.
Hi guys, sorry for chiming in so late.
I'm glad for all the attention and interest people are showing for Attic lately, and I know you've been keeping busy (30+ Github notification mails most days)
As you probably know, and as I explained on the mailing list I currently have pretty much 0 amount of time right now. Besides the kids, we're also in the process of buying and moving to a new house.
Anyway, I see that people are concerned (and rightfully so) about the lack of progress lately and wondering if there is any way to help.
First of all, I'm not ready to hand over the project to somebody else just yet. I still want to work on Attic, I just need to wait until my life slows down a bit. It's hard to make predictions about these things. But I suspect it will take weeks or months before this project will pick up any significant speed :(
I want to be in control of what goes into the Attic code base (I.e, I want to merge PRs). Additionally, when it comes to the critical sections of the code base (pretty much anything that touches encryption, storage format and backwards compatibility) I prefer to write all code myself to ensure I understand every part of it.
On the other hand, help with answering and investigating bug reports on the mailing list and Github is always appreciated. Smaller PRs fixing bugs are also welcomed. But please try to refrain from spending too much time creating large PRs that touch the critical sections I mentioned above without first discussing things with me. Otherwise chances are that these will not be merged. And I don't want to waste anybody's time or hurt anybody's feelings.
Hi @jborg, nice to see you here!
I can follow that you are a uncomfortable with people who you do not really know merging stuff (I would somehow feel same I guess).
In case it helps getting to know me better and trust in my changes:
Python: I do Python since about 14years now, primary maintainer of MoinMoin Wiki + nsupdate.info. Plus a lot of contributions elsewhere / other smaller projects. I am also frequently reviewing other people's code for the projects I participate in. I use to attend EuroPython.
Crypto: I am interested in crypto since quite a while and recently successfully completed Dan Boneh's (Stanford) "Crypto I" course on coursera.org (which I can recommend and which covers everything used in attic). I use to attend CCC conferences. That doesn't make me a cryptographer of course, but it gives me working knowledge to do stuff and avoid stupid mistakes. Review is of course always welcome.
Storage/Backwards compat: I carefully made the changes in a way so that they are backwards compatible. Hardcoded stuff before is now just a parameter fed to generalized code. Parameters are same for the old type bytes.
So, as your availability seems to be rather low now and in near future, maybe at least consider if REVIEWING merges and giving feedback is maybe not a better (less time consuming for you) option than waiting until you can do it all yourself.
About rather writing it on your own: I think you could also try to understand my code (which shouldn't be hard as I like to write pretty clear code, IMHO) [or other people's code], instead of writing it yourself. If there are any questions or feedback, I am happy to answer them / improve the code. The nice thing about Python is that there are a lot of people using it AND love to write clean code.
Problem is that there can be dependencies between PRs, so as long as one such PR is not merged, it basically blocks work on stuff depending on that change.
Cheers, Thomas
BTW, you said "handing over the project". In case that was unclear: We do not want to take the project from you (that's why you got an invitation to the attic gh org), we just thought it would unblock the project if the group of maintainers would be bigger than 1. so more people could effectively care for it. AKA "increase bus factor".
I can follow that you are a uncomfortable with people who you do not really know merging stuff (I would somehow feel same I guess).
No, that's not really it. My thinking is a lot more selfish than that. The reason I started coding on Attic several years ago was not because I thought it would be the most time efficient method to solve my own backup needs. And not because I felt the need to provide the best and most feature complete backup solution to the world.
I started working on Attic because I like writing code on my spare time and Attic includes a nice mix of technologies and problems I like think about and work on. And even today Attic is my "feel good" pet project that I want to be able to code on whenever I happen to get some spare time. And when I have as little spare time as I currently have, I really do not want to spend it all on reviewing, merging, debugging and rewriting other peoples code. I rather listen on peoples feedback and feature requests and write the code myself. As soon as I get more time for this I'll be able to review other peoples code. But right now only looking at other peoples code sort of takes all the fun out of this project for me.
One concrete example is PR #207, this PR implements something I've already done but not yet published (See http://librelist.com/browser//attic/2015/1/27/status-update-i-m-still-alive/). Right now it would be best if people just reported issues for bugs and feature requests, but held off on creating pull requests until I get more time reviewing code. Unfortunately I don't think it's possible to disable pull requests on Github.
About PR #207 and your ML post: I read your ML post, but IIRC it was after I had started working on that.
i can vouch for what @ThomasWaldmann is saying, assuming he is the real @ThomasWaldmann.
for my part, i also happen to be a MoinMoin contributor (although minor). i have written bup-cron and monkeysign. i am a good release manager, tester, sysadmin and old programmer.
if that matters at all of course. :)
i understand your concerns, @jborg, but people are really enthousiastic about the project and it'd be great to allow more people to contribute...
thanks for the feedback at least. :)
@jborg: Now that about 2 months have passed and you likely had some time to look at other people's code, to read some mailing list posts, to think about the project's future, etc. ... - do you still think the same way as when writing above comments of March 10, 2015?
Because I think some people here (including me) think that attic is a great piece of software and should be more than just your personal pet project (thinking of project "bus factor" also). They would like to have a more open, more collaborative project (which would likely be good for speed of progress also). They would also like to have you as part of such a bigger project rather than starting a new one (with initially maybe the same bus_factor == 1 problem).
But of course nobody wants to take the fun away from you or try to force you into something that you don't like. So in case you still feel and think the same, a fork of the project is maybe the best for everybody. I am prepared to do so (if necessary), at https://github.com/borgbackup .
Maybe sleep a night over it and then please reply.
Note: I would transfer the attic organisation on github to you in case the fork happens, I am not interested in it if you aren't interested in doing a more collaborative attic development there and rather want to continue it as a personal project.
@jborg Come on, surely you are not that selfish since you shared Attic in the first place. We would all be better off if a fork is avoided and more (knowledgeable enough) people get to help you with maintaining the code. If you prefer coding to merging, why not let @ThomasWaldmann maintain contributions jointly with you?
What do you think about an "unstable" fork that merges all changes jborg does back in? Jborg himself could backport all changes of the "unstable" fork to attic if he wishes to. As long as the unstable fork merges all fixes by jborg, both applications will stay compatible.
People of this new repository could be the ones that do all the "community management" like answering feedback and PR requests, therefore removing all community related work jborg may not have time to attend to.
The end game for me has never been to keep this project as a one man shop where every single line of code is written by me. Just to make things clear I do accept contributions from other people, Attic already contains both bug fixes and new features from a number of contributors (including you @ThomasWaldmann). But probably not not as fast and to the same extent as some people like...
Here's an attempt at answering the obvious question: "Why don't you let people help you?"
Before answering that question I feel the need to explain a bit about my vision and philosophy for Attic:
- Attic should be compatible with previous versions as long as possible (preferably forever). This means that great care needs to be used when the format is changed since the codebase most likely will need to support the new and old format forever. This means that you can't change the format very often, and when you do you need to get it right on your first try. Design mistakes can be costly to live with.
- Attic should stay lightweight and user friendly. I think Attic currently compares fairly well feature wise against other similar backup solutions while at the same time being user friendly. So in order to keep these properties it's important to make sure new features are of good quality and of general interest. I don't want to keep adding a bunch of obscure features that only a minority needs or just because a pull request exists.
until I find somebody I trust that shares my vision for Attic I will keep my role as sole maintainer. Hopefully I'll get more time to work on Attic soon. This will both speed up the project itself. But probably more important allow me to spend more time reviewing PRs and hopefully finding somebody I would feel comfortable delegating responsibility to.
Note: I would transfer the attic organisation on github to you in case the fork happens, I am not interested in it if you aren't interested in doing a more collaborative attic development there and rather want to continue it as a personal project.
Using the org as a bargaining chip is not the way to gain trust...
Hey guys, seeing this from the sidelines and strictly as a user, I think this may be a train wreck in slow motion. Let me tell you about that movie as I see in my head:
The guy with the power and the vision wants to keep it pristine, but doesn't have the time to move it forward and he distrusts anybody by default, so anybody who wants to help needs to suffer first to get some degree of trust later. That makes sense to some degree, but still to be a leader means to have followers, and deterring followers is not a great leadership move for starters.
The guy with the time and the will to make things happen, he may not even know what that grand vision is like (nobody told him and nobody heard his ideas either, as far as I see), he is probably tired of dedicating time to tasks that don't get thanked, and probably doesn't want to submit to menial tasks under a leader who does not want to be followed and who may not even be available in the future as far as we know.
The users appreciate the vision, because that's what made Attic great in the first place: its careful and well-thought design. Some unit testing and a fast merging pace is no substitute for it, it can only rot the code over time if it's not lead with a guiding vision. At the same time, the users need some pretty basic features that are long overdue (no renaming, seriously?), some proposed enhancements stay frustratingly quiet, lots of issues need tending, documentation is needed, etc. Much from this stuff is not going to threaten the grand vision and should be doable under some light supervision, but even that is not available. The whole project is stuck under repeated half-promises of "hopefully" getting some time in the future.
As far as I see it, there's a communication problem on @jborg 's side. The leader should share his vision and get followers to share it, listening for any good ideas they may bring. Followers need to get trained on this vision, because assuming someone will pop up already having the leader's ideas on his head is not realistic. The leader should delegate, and then verify. To find people to trust, the leader should be trustable, he must commit to something even if it's a small commitment ("hopefully" is not a promise to be trusted, especially when it's been said before to no effect). To receive, the leader must ask. (If you want the attic org to get passed to you, ask for it directly). The leader must be just (implying the other is holding the attic org for bargaining even after he said he's willing to pass it to you is not fair, even if you hold that suspicion, when you haven't asked for it).
This communication problem, which results in Attic currently having an owner but no leadership to speak of, probably originates in @jborg 's lack of time. It may also be that he's not keen on delegating (I know I find it hard). I also know having a kid can suck most of your time, not to mention three (two of them twins, to make matters worse). As they grow and for the next few years, you know, they tend to take more time, not less. As far as I see it, @jborg, either you focus right now on leadership and on communicating your vision and on delegating as much as possible really soon, or you'll end up overwhelmed and Attic will stall and it may end up being forked. And if it is forked successfully, and it may, you probably won't find it very appealing to return to it in a couple of years when you have some free time again.
Now, maybe this is just fiction in my head. Maybe.
@jborg maybe reread what I wrote. I told I'll give the org to you in the outcome that is less preferable (and more work) for me. Not sure how you interpret that as a "bargaining chip".
@pguth haha. looks like, pity.
ok, I'm out of here (at least development wise I'll continue at https://github.com/borgbackup ).
@jborg feel free to pull anything from there as you like, I'll also try to pull your stuff, although this will be harder in future. wake me up if you change your mind.
bye and thanks for the fish ehrm attic! :)
Not sure how you interpret that as a "bargaining chip".
Yeah, found that weird too.
No other choice than to fork then. I will be switching - the number of open tickets and pull requestswith no response, lack of feedback from @jborg (in respect to stuff I wrote about last year) and the non collaborative attitude is a shame. Also the accusation of you using the attic org as a bargaining chip - which was obviously not what you were saying.
shame it had to be like this, but there was no choice really in the end.
well, this is certainly not what I had in mind when i opened up this discussion.
i think this is an unfortunate result, although I am not sure what other outcome would happen at this point. people are of course free to fork the project, but I am not sure I see exactly what are the technical pain points that warrant such a fork.
if stability and reliability is such a concern for people, forking the project hardly seems like a good way forward: it introduces uncertainty and variability. it seems we are already talking about changing the storage format in "borg" (https://github.com/borgbackup/borg/issues/1) - how would that bring more stability??
anyways, that's a really disappointing outcome. i hope this can all merge back at some point...
another way to put this: we were considering Attic at work to replace our current backup software. it was either this or bup, with a strong incentive to use attic because it supported removing old backups.
now that there's this fork, it's unclear what the future of Attic will be and it makes it basically impractical to use attic for the time being, as we'll probably have to wait a year or two to figure out which fork will end up being more stable, maintained and (basically) popular.
which will probably just make us use bup because it will support removing old backups in the old release.
a shame, really.
For all passive readers I would suggest checking out borgbackup/borg#1 for a deeper discussion about the forks features.
@ThomasWaldmann the name borgbackup strikes me as odd. Is that meant to be a play on @jborg's name? If so that seems a little provoking. Is it just a nod at the heritage of the software or something more insidious?
@skarekrow I asked him that same question. He denied it being made in mockery, he noted @jborg 's last name is not Borg but Borgström and referred to the Star Trek character. I think it's an obvious pun and in a way an homage to the original author, even if his name was an involuntary source of inspiration for the fork's name. There's also the idea of Borg assimilation being somewhat akin to deduplication, maybe.
Still, there seem to be some sour grapes between @jborg and @ThomasWaldmann. Forks don't usually happen in perfectly amicable terms, that's for sure. I think it's understandable, there were some underhanded accusations from @jborg and there was frustration on both parts due to @jborg 's lack of available time. Still I hope they could apologize as needed and come to work as a team, but that's pretty much up to @jborg now I guess.
Same. If it is a pun relating to Star Trek and paying homage, it's very clever. If not, that's not a great way to start a fork off. Regardless I think this should maybe have stayed private between @ThomasWaldmann and @jborg as having all the users piling on certainly won't help things be amicable. It's akin to being ganged up on. Then the conversation becomes defensive. Which is good for no party. I understand why @ThomasWaldmann wants to keep this conversation going as I'm sure he's frustrated also, but I don't think we should really be weighing in as it doesn't directly concern us. We just want a stable program with good support. Whether a fork will provide that or @jborg getting more time does remains to be seen. But I don't think issues like these or the mailing list post really helps accomplish that.
My 2cents of course.
I think that is somewhat dramatic. It's worth noting that one of the problems is users who are also devs, wanting some changes to attic that haven't happened. I'm not placing blame or pointing fingers, and I know how real life works, we are not all young with lots of free time, however, attic momentum from users, and 3rd party devs was moving faster than attic itself. The best thing would be for @jborg to put some trust in other skilled programmers, but development is almost completely stalled (minus a few updates recently).
I'm pretty sure the name has nothing to do with the username, but even if it does, it's not something to get upset over. Let's not get silly about that,
Maybe it will end up like ffmpeg/libav, which despite the fork being somewhat unpleasant to many, triggered a much better development model for ffmpeg, and in the end ffmpeg (imho) remained the better product. I'm not taking sides here either.
Last year I started working on the retropie project. The main dev had less free time, and I had some. After some patchsets, and a feature branch, I am now able to improve it as needed, as I have commit access to the main project - however I still do pull requests for stuff that changes anything significantly - and the original dev still runs the show. It's his baby.
No-one is being attacked here, but I have felt that attic is not moving along as it should. I also do think @jborg could delegate and trust @ThomasWaldmann a bit more - I've looked through the patches. @ThomasWaldmann is not an amateur - he is a very skilled programmer.
I think this boils down to ego. Which is a shame :/ I hope I am proved wrong.
@skarekrow it does directly concern us precisely because, as you say, we just want a stable program with good support. Attic is a stable program, with not much support (in the only form free software is supported, which is through development, community participation and feedback). Borg may be starting to have some support in that sense, but it's far from stable at this point. As attic's user community, we can only voice our opinion and try to nudge the developers into some more reasonable collaboration between them. If we don't do that, then we might as well go back to bup as I'm sure many are considering at this point. But we know attic's design is better, and it's a shame all this work is being left to rot.
@aride I agree with that, but @jborg has nicely stated his intentions a few times now. Clearly if more work wants to be done it needs to be in a fork of Attic. I cannot, as I imagine other users cannot, change his mind. He has other time issues that are more pressing then Attic. It would be a shame if all the work rots, but that hasn't been shown yet. He has been popping in to let everybody know his thoughts. Until he completely abandons the project, I see no reason for all the worrying.
@joolswills I don't think what I've said was dramatic or silly ;) But I would think that this is not a matter of ego, but rather an issue of trust. @jborg certainly isn't required to trust @ThomasWaldmann, and this is his code. If he wants complete understanding of what's going in and trusting the program does what he thinks it does, then who are we to say he needs to open it up and have us try to force cooperation? That happens naturally. In this case it's clear that @ThomasWaldmann will have to continue with his fork and encourage contributions there. There's going to be a clear division of how the two operate. One will remain protective and the other much more open.
Like I said, I think time will show us which approach ends up being better for the user. But haranguing @jborg to change his mind certainly doesn't help or speed things up.
Most of the forks in github happen silently. I don't quite get the fuss about forking attic. As suggested above, I would keep those two applications as much in sync as possible. This would provide better compatibility, something I guess both applications would benefit from.
@jborg raised two very valid points:
- "Attic should be compatible with previous versions as long as possible (preferably forever)": 100% ack. This saves money and time client side.
- "Attic should stay lightweight and user friendly.": True, but I wouldn't deem attic to be very user friendly at the moment. TW had some good PRs to make it more userfriendly. Additionally there were some efforts to document attic that have not (?) been pulled.
@ThomasWaldmann : Looking at borgbackup/borg#1 I see an urge for revolution. In my opinion only sensitive evolution is needed here.
@anarcat : I don't see why you shouldn't use attic. At the moment, I would definitely go for attic as it has already proven itself in an enterprise environment. This trust is something Borg-Backup would need to establish. However, borg seems more promising in the future as it has community momentum. Time will tell if this benefits attic / borg or not.
My 2c
Aside from my personal use, I work professionally in the backup industry with a range of commercial backup software products. We use attic on our own infrastructure as the storage algorithm is leaps and bounds ahead of the commercial competition. Our repositories date back over a year and contain several hundred GB.
I will continue to watch development, but intend to remain using the original version because
- it's now packaged in debian jessie, inheriting certain security and maintainability guarantees
- jborg has a long proven track record as project maintainer
- demonstrable commitment to backward compatibility
- our internal investment in testing, deployment and monitoring procedures
- i am risk-averse
Today node.js did announce it would merge back with its major fork io.js. I'm sure that one day in the future, this situation will similarly culminate in an amicable merge, and no work will go to waste.
My best wishes to all parties involved, especially jborg for whom i hope this mess is not too great a distraction,
@anarcat : I don't see why you shouldn't use attic. At the moment, I would definitely go for attic as it has already proven itself in an enterprise environment. This trust is something Borg-Backup would need to establish. However, borg seems more promising in the future as it has community momentum. Time will tell if this benefits attic / borg or not.
For the record, when i said "i can't use attic" i also meant "i can't use borg". In fact, i can use borg less now because it explicitely states things are changing and breaking. I am also curious to see where attic was used in enterprise, i didn't have the feeling it was the case.
I will continue to watch development, but intend to remain using the original version because
- it's now packaged in debian jessie, inheriting certain security and maintainability guarantees
So for the record, something landing in Jessie doesn't magically means it's maintained for bugs and security issues. Sure, one more person has eyeballs on the project, but Debian maintainers are also very busy people and can't carry the weight of the project on their own. So to give a very practical example: attic is actually out of date in Debian, already. I also don't see bugfixes being backported into jessie either.
- jborg has a long proven track record as project maintainer
- demonstrable commitment to backward compatibility
- our internal investment in testing, deployment and monitoring procedures
- i am risk-averse
I agree with all of this. I also just noticed that jborg merged a few PRs and made two releases this year already.
My point was more that the fork introduces instability in the community, which makes me consider other software again.
@anarcat On May 14 you said:
For the record, when i said "i can't use attic" i also meant "i can't use borg". i can use borg less now because it explicitely states things are changing and breaking.
I'm interested to know if you still feel that way, or if you are know using either attic or borg. And if you are, what change prompted your decision. Thanks!!
i am still using attic, out of procrastination, mostly. i would need to convert my existing archive to the new borg format to switch, which would involve writing a converter.
so basically, the status quo is simplest for me right now. but i'm considering switching to borg, just because there seems to be more momentum there.
Thanks for your input. @ThomasWaldmann gave a talk on Borg at CCC and it gave me some confidence in his abilities to extend borg in useful directions. So for now I'm watching both projects and hoping they both continue for long time to come.
Don't you worry, linkmaster 3000 is here! https://www.youtube.com/watch?v=Nb5nXEKSN-k
i'm currently using both and that's what i would recommend when asked... ;-) (the dedup works wonders)
How are you able to combine backup sets between the two forks? Is that even possible?
oh, no no, i'm not doing that. just using attic and borg separately (so yes, it takes up twice the space but at least both are deduped). i don't believe such combining is possible.
i've migrated to borg completely since then. at that time i was using both for extra safety as borg just started its life. as of today, i consider it (borg) being good enough to trust my data to it so i've dumped attic (because its developement seems to have stalled).
unless it wasn't made clear here already, borg supports an "upgrade" command that converts an attic repo into a borg repo safely, inline and without duplication. so there's a clear upgrade path as well.
as far as i am concerned, borg is the new attic. we are actively merging any changes that happen in the attic source tree (which is basically two commits since the fork) and borg people (especially @ThomasWaldmann) are even responding to attic issues here. we are also tracking attic issues that may affect borg in https://github.com/borgbackup/borg/issues/5.
so i see very little reason to still run attic.
i would close this bug report as it obviously won't be fixed, but i feel it may shed some light on the history of the project and could serve as a warning for people that would look into attic in the future as well. i hope that is alright.
