exiv2
exiv2 copied to clipboard
Exiv2
Rehost exiv2.org on pixls.us server
We've had a generous offer from Mica and/or Pat to host exiv2.org on their server. Currently it's hosted on AWS and I pay about $20/month (depends on bandwidth). Let's move the domain by the end of 2021. We should wait until after v0.27.5 ships on 2021-09-30. Let's aim for November 2021.
I am the current registered domain owner of Exiv2.org and I will totally cooperate. I'm confident Nehal (@nehaljwani) will contribute to getting this done. I think we should investigate transferring the registration for exiv2.org to Mica somebody involved with pixls.us.
The future of exiv2.org should be discussed by Team Exiv2. I think it should remain because it has unique documentation and copies of every release of Exiv2 going back to 2004. Additionally, closing exiv2.org broadcasts an incorrect and very negative message about the status of Exiv2.
The effort involved to make a release on exiv2.org can be easily reduced. Today, I build on Ubuntu, macOS/x66 MinGW, Cygwin64 and msvc2019. This can be replaced by the nightly build when it is augmented with MinGW.
If Nehal, or another Team Member, is happy to take on "Release Engineering", I will cooperate with the effort to modify the current release procedures which are here: svn://dev.exiv2.org/svn/team/website
My opinion is that we should stop building binaries for MinGW and Cygwin after 0.27.5. We have an Action which automatically builds binaries for Linux, MacOS, and Windows. Those are the major platforms, so I think that is sufficient. I did try to add support for MinGW and Cygwin, but couldn't get either to work.
Well, thank you for trying to get MinGW to build. I agree 0.27.5 is the final release for which binaries will be built on the mac-mini build server.
I hope we find a volunteer to take over responsibility for release engineering. I will cooperate to make the appropriate (small) changes to the web-site and the release process .
FWIW, I was behind the last two releases of the MinGW package for the MSYS2 repo - no problems building over there so far, but that's not really CI (although I use MinGW to build and debug locally both the main and 0.27-maintenance branches from time to time). Generally if both the Linux and Windows/MSVC actions are in good shape, MinGW should be as well.
Those are the major platforms, so I think that is sufficient.
Actually quite a few apps use the MSYS2/MinGW route to provide Windows support, so it might still be worthwhile to make sure it has a decent level of testing. But I agree, there is no need to release the MinGW binaries, they should come from the repo.
Side note: not sure about Cygwin, but the MinGW actions should be relatively straightforward using https://github.com/msys2/setup-msys2 - this is e.g. what both darktable and rawspeed use for their only Windows CI.
I added both Cygwin and MinGW to Appveyor and they are used on the 0.27-maintenance branch. I don't know if they are deployed for branch 'main'.
I believe the MinGW build is popular with Qt users and we should continue to support it. Can we discontinue supporting Cygwin/64? I think so.
@alexvanderberkel @postscript-dev @kmilos @kevinbackhouse @nehaljwani @hassec @piponazo @1div0 @dhoulder
Alison says the bill from AWS this month for exiv2.org was $35 (usually about $20). My guess is that server used more band-width to deliver v0.27.5 downloads. Com'on guys, it's time to deal with relocating exiv2.org
I think it would be useful to hold a team meeting on Zoom to discuss 2022. Exiv2 v1.00 does not appear likely. Perhaps we could begin a new series of dots v0.9, v0.91, .... Yet another v0.27 security "dot" v0.27.6 may be required.
I am willing to support/mentor a new release engineer. I'm happy to attend a team meeting and will only speak when asked a question.
Alison here!! Be in no doubt the Credit Card paying AWS will be cancelled by the end of January so please deal with this ASAP Thank you.
Dear Exiv2 Team,
I've set up a demo at https://njw.exiv2.org/ . It uses Github Pages to host the content. We can update the artifact urls for *.zip and *.tar.gz files to link to the 'assets' uploaded on each official release on Github. We can do the same for https://pre-release.exiv2.org/ as well. Please check if this is acceptable and then we can retire the AWS instance and the only portion remaining will be the DNS domain ownership.
Please note that this does not include Redmine (https://dev.exiv2.org/). It is theoretically possible to generate static content for old URLs and host it as well, but I'll need a list of URLs for which this needs to be done.
As for Release Engineering - happy to take it up. I've access to Linux, MacOS, Windows and can also cross-compile for non x86_64 architectures if needed.
Cheers! Nehal
Looks like a miracle to me, Nehal.
@mica offered to host. Maybe he can run Redmine on his server and we give him a back up of the database. The URL of the redmine server doesn't need to have anything much to do with exiv2. exiv2-redmine.pixls.us (or something like that).
Very happy to hear that you're willing to do the releases. Let's discuss the scripts involved when the team decides to make a release.
@nehaljwani I've sent Mica a private message on discuss.pixls.us and asked him to join the discussion here.
I've pinged mica again: https://discuss.pixls.us/t/hosting-exiv2-org/28039 (I suspect you can't read my message).
Alex asked me to make you on "Owner" of Team Exiv2 on GitHub. I've done that. Not at all familiar with GitHub Teams. I'm not sure we use them. Anyway, you should be set up. And I've boosted your GitHub permissions from "Write" to "Admin" (the top access level). "With great power, comes great responsibility".
Hi, sorry for the delay. I spoke to @nehaljwani on matrix, and there were two things to host: the website and redmine. I had not previously heard about hosting redmine, and since it isn't actively used, I've asked @nehaljwani to try and make a static site out of it.
As for the website, we can host that on the pixls.us infrastructure if you'd like.
@paperdigits Very pleased to hear your voice, Mica.
For sure @nehaljwani is has much greater knowledge of servers that I do. So, I totally defer to your's and Nehal's judgement about what should be done. I know nothing about setting up Redmine, however I'm sure it's fairly straightforward. When it's configured, it should be "read-only" to ensure that folks can't post new issues on the site.
There are team resources stored in svn as svn://svn://dev.exiv2.org/svn/team/ For example the team directory includes the code for the web-site, the Exiv2 logo graphics, and my book. That's there because it has always lived there and never been moved. If Nehal wants to relocate those resources to GitHub, that's fine with me. The only restriction is that we must avoid "bloating" the product download from GitHub.com/exiv2/exiv2.git. The GitHub Wiki is at: https://github.com/Exiv2/exiv2.wiki.git Maybe that's a suitable place. @kevinbackhouse is an engineer at GitHub and can probably advise.
The only other feature on my "wish list" is email addresses such as "[email protected]" for use by team members.
At a later time, let's discuss the registered owner of the domain. I'd like that to be transferred somewhere.
My suggestion is to create a few new repositories in the Exiv2 organization. For example, the svn repository can probably be converted to git and pushed to a new repo in the Exiv2 org. Similarly, we could create a repo there for the website (using @nehaljwani's GitHub Pages work).
Thanks Kev @kevinbackhouse.
Nehal @nehaljwani can you take the lead of this please and say what's to be done.
I'm happy to move stuff (team, book, web-site) from svn to git. It's on svn because it was born there and has never been moved. Time to change.
Thanks, @nehaljwani You are a splendid fellow. Very pleased that you are dealing with this. Thank You.
@clanmills: Something occurred to me, the new manpage uses https://clanmills.com/Stonehenge.jpg as a test image. At the moment this version of the manpage is only on 'main' and few people use it, but once it is on https://exiv2.org/ this will change. While the file's location may bring more traffic to your website, all downloads would be at your cost. If you want to, you could add a copy of Stonehenge.jpg to https://exiv2.org/? If you decide to do this, then let me know the new URL and I will update the manpage references.
Great idea, Peter. I'll move it. I don't run up bandwidth charges on clanmills.com.
Alison is dusting her rolling pin and looking forward to hitting everybody in 2022 about the web-site.
I love Alison. She doesn't have a rolling pin. We've been married for 47+ years. Photo at our son Jamie's wedding in 2019.
@clanmills
Great idea, Peter. I'll move it. I don't run up bandwidth charges on clanmills.com.
Perhaps it would help to add a reference to https://clanmills.com/ in the file's metadata. This would show where the image came from and point anyone interested to your site. Maybe adding other licensing tags would also be useful for you.
On a related note, when I was looking for a test file for the manpage, I was surprised to discovered that the Exiv2 logos on https://www.exiv2.org don't contain much/any metadata.
The big logo (which is the same as exiv2.png in git) has:
$ exiv2 --verbose --print a exiv2-logo-big.png
File 1/1: exiv2-logo-big.png
exiv2-logo-big.png: No Exif data found in the file
exiv2-logo-big.png: No IPTC data found in the file
exiv2-logo-big.png: No XMP data found in the file
and the small logo has:
$ exiv2 --verbose --print a exiv2-logo-small.png
File 1/1: exiv2-logo-small.png
Xmp.exif.PixelYDimension XmpText 4 1370
Xmp.exif.PixelXDimension XmpText 4 2401
Xmp.tiff.ImageWidth XmpText 4 2401
Xmp.tiff.ImageHeight XmpText 4 1370
Xmp.tiff.Orientation XmpText 1 top, left
exiv2-logo-small.png: No Exif data found in the file
exiv2-logo-small.png: No IPTC data found in the file
This is something that I think needs to be addressed by someone before the next release.
I've updated the web site: https://www.exiv2.org/Stonehenge.jpg I submitted a change to svn: svn://dev.exiv2.org/svn/team/website r5321 I donate 100% of the ownership of the image Stonehenge.jpg to Team Exiv2. I don't remember why I started using it to discuss features of the exiv2 command-line tool with users.
That version of Stonehenge.jpg is almost the original. The only edit was the Caption "Classic View" (in Picasa).
$ exiv2 -pa -g Caption -g description https://exiv2.org/Stonehenge.jpg
Iptc.Application2.Caption String 12 Classic View
Xmp.dc.description LangAlt 1 lang="x-default" Classic View
$
I used an on-line service to design the logo and purchased the package of files. I've never thought about the metadata in those files.
I can't imagine anybody being interested in clanmills.com. Friends/family follow our trips. If I miss putting up photos for a day on vacation, I'll get email from somebody asking if we're OK! It's currently being totally re-engineered in JavaScript and the 80,000+ photos have been moved to Google Photos. The site has decreased from 10GB to 50mb. Instead of 2 million lines of script generated HTML, there are 300 lines of JavaScript. It's been a huge task to deal with this. I started in August and it'll be finished next month. Delighted by the result. It's clean, consistent, faster, smaller and provides an up-to-date user experience.
I know very little about the use of metadata, so I don't know what metadata should be inserted into the logo files. The whole package is stored at: svn://dev.exiv2.org/svn/team/logo_files. I think I created Logo.graffle from the SVG for use in OmniGraffle (Eh? Why don't I use Adobe Illustrator for drawing?). You can modify them on SVN.
I think @nehaljwani is going to move the team resources (including the website) to GitHub, so it's better to wait for that transition.
@nehaljwani and @paperdigits. I've lost the plot with this project. Nothing seems to have happened. It's easier to forget about jobs that are finished, than to have lots on the TODO list. Let's get the web-site moved in January (2022).
540 rmills@rmillsm1:~/gnu/exiv2/team/logo_files $ find . -type f | grep -v svn
svg/SVGlogo.svg svg/SVGlogoAI.svg svg/Albertus_Medium.ttf
svg/readme.txt EPSLogo.eps svg.zip
twitterheader.jpg 400dpiLogo.png 400dpiLogo.jpg
65377083.pdf facebookcover.jpg Logo.graffle
SmallLogo.png SmallLogo.jpg icons/favicon-16x16.png
icons/favicon.ico icons/apple-icon.png icons/apple-icon-144x144.png
icons/android-icon-192x192.png icons/apple-icon-precomposed.png icons/apple-icon-114x114.png
icons/ms-icon-310x310.png icons/ms-icon-144x144.png icons/apple-icon-57x57.png
icons/apple-icon-152x152.png icons/ms-icon-150x150.png icons/android-icon-72x72.png
icons/android-icon-96x96.png icons/android-icon-36x36.png icons/apple-icon-180x180.png
icons/favicon-96x96.png icons/manifest.json icons/android-icon-48x48.png
icons/apple-icon-76x76.png icons/apple-icon-60x60.png icons/browserconfig.xml
icons/android-icon-144x144.png icons/apple-icon-72x72.png icons/apple-icon-120x120.png
icons/favicon-32x32.png icons/ms-icon-70x70.png SmallLogoBW.png
SmallLogoBW.jpg PSLogo.ps linkedinstandard.png
twitterprofile.jpg 400dpiLogoCropped.jpg 400dpiLogoCropped.png
linkedinsquare.png facebookprofile.jpg Exiv2-64x64.png
541 rmills@rmillsm1:~/gnu/exiv2/team/logo_files $
Apologies for the delay, a few updates:
- After spending some time, I was able to generate a static copy of the Redmine website (read-only). I've pushed the content at https://github.com/Exiv2/dev.exiv2.org. One can take a look at the commit history to find out how it was done. I've hosted it at https://dev2.exiv2.org/ using Github pages. @clanmills could you please test it a bit to see if there are any broken links that need to be fixed? 🙂 Once I receive a green signal, I'll proceed with moving it to
dev.exiv2.org. ❗Note: The current size of the static content is ~1.6GB and Github hasn't complained (I think there is a soft quota of 1GB). If Github does complain in the future, I'll seek help from @paperdigits for moving the static content to the pixls.us server. - I've converted the team/website folder into a git repository (with the history intact), available at https://github.com/Exiv2/exiv2.org. It has a makefile, so my plan is to add a Github action by following https://github.com/marketplace/actions/github-pages-action to publish the content. If this approach creates hurdles, I'll simply generate the static content and push it to a different branch on the repo and serve the content from there.
- The 'team' folder has more bonus content. @clanmills Could you please point out the remaining folders that need to be a repo of their own (I can combine multiple into one too)?
$ du -sh team/*
87M team/XMPsdk
292M team/book
88K team/contrib
64M team/drawings
115M team/libraries
364K team/license
2.6M team/logo_files
3.0M team/meetings
384M team/releases
46M team/rmills
1.9M team/tuan
11M team/website
@nehaljwani Thanks. I'll go over this carefully today and give you an update. Here are my initial comments. Please don't feel daunted by the length of my comments. You're not far away.
- The website should be about 10m:
505 rmills@rmillsm1:~/gnu/exiv2/team/website $ cd /tmp
506 rmills@rmillsm1:/tmp $ svn checkout svn://dev.exiv2.org/svn/team/website
A website/Makefile
A website/master
A website/master/Stonehenge.jpg
A website/master/news.xml
...
A website/exiv2.png
A website/license.txt
A website/README
Checked out revision 5366.
508 rmills@rmillsm1:/tmp $ du -sh website/
22M website/
507 rmills@rmillsm1:/tmp $ du -sh website/*
16K website/Checklist.txt
8.0K website/Makefile
4.0K website/README
56K website/bin
120K website/doxygen
16K website/exiv2.png
3.1M website/html
20K website/license.txt
6.7M website/master
400K website/var
509 rmills@rmillsm1:/tmp $
The equivalent in GitHub is 3.2G. So we're storing lots of stuff on https://github.com/Exiv2/dev.exiv2.org.git that doesn't need to be there.
552 rmills@rmillsm1:~/gnu/github/exiv2 $ git clone https://github.com/Exiv2/dev.exiv2.org.git
Cloning into 'dev.exiv2.org'...
remote: Enumerating objects: 14880, done.
...
Updating files: 100% (14199/14199), done.
553 rmills@rmillsm1:~/gnu/github/exiv2 $ du -sh dev.exiv2.org/
3.2G dev.exiv2.org/
554 rmills@rmillsm1:~/gnu/github/exiv2 $
- The releases directory on exiv2.org are tarballs.
I think we can store them on GitHub using the releases feature on GitHub. In future this page https://www.exiv2.org/archive.html can be replaced by this: https://github.com/Exiv2/exiv2/releases
Every release of Exiv2 starting with v0.26 is already on GitHub, so the following should be added:
675 rmills@rmillsmm-local:~/gnu/exiv2/team/website/html $ ls -1 releases | grep -v 26 | grep -v 27 | sort --numeric
exiv2-0.10-win.zip
exiv2-0.10.tar.gz
exiv2-0.11-win.zip
exiv2-0.11.tar.gz
exiv2-0.12-win.zip
exiv2-0.12.tar.gz
...
exiv2-0.25-win.zip
exiv2-0.25.tar.gz
exiv2-0.3-win.zip
exiv2-0.3.tar.gz
...
exiv2-0.9-win.zip
exiv2-0.9.1-win.zip
exiv2-0.9.1.tar.gz
exiv2-0.9.tar.gz
676 rmills@rmillsmm-local:~/gnu/exiv2/team/website/html $ 555 rmills@rmillsm1:~/gnu/github/exiv2
- Here's what's in team.
87M team/XMPsdk Remove.
292M team/book Keep. That's the raw "code" for my book (mostly test images).
88K team/contrib Keep.
64M team/drawings Remove. Graphics for GSoC 2013 and the LGM/Rio/2017 presentation.
115M team/libraries Remove.
364K team/license Keep.
2.6M team/logo_files Keep. Discussed above.
3.0M team/meetings Keep.
384M team/releases Remove. Discussed above.
46M team/rmills Keep. My "stuff". I should find another home for this stuff.
1.9M team/tuan Remove. Tuan's "stuff".
11M team/website Keep. Discussed above.
You can remove XMPsdk and libraries. I was working with them in 2017 when investigating how to remove the xmpsdk/ directory from the code-base. XMPsdk has copies of different editions of the Adobe's code. libraries has copies of zlib and expat to be used when building the Adobe code (and openssl, libssh and curl). See libraries/ReadMe.txt
- Don't forget about testfiles.
The following repos has test images. svn://svn.exiv2.org/svn/testdata
We don't need to do anything quickly about this. Let's discuss it once we've dealt with the web site.
- Validating the website html.
website/Checklist.txt recommends
Validate uploaded webpages with W3C HTML checker # https://validator.w3.org
I usually skip this step (out of laziness/exhaustion). We should use this to "polish" the HTML to be as good as possible.
Hello Robin, thank you for checking. I just wanted to clarify that https://github.com/Exiv2/dev.exiv2.org.git is the static content from Redmine, not the exiv2.org website, and while calculating the disk usage, you should skip the .git folder, otherwise you end up counting twice. The largest portion in the checkout comes from the various attachments in Redmine from issues/posts/wiki. As of now, GitHub hasn't complained so I am a bit optimistic that we can leave it as is.
Also, I can also put the entire team folder as a repository on GitHub, instead of separating each or a combination of them (i.e. defer that decision for later)
@nehaljwani Wow. We're looking in different directions. Not a problem. Damn glad to realise we're not on the same page yet.
I cloned dev.exiv2.org and didn't inspect it. I see now that it's Redmine magic. You're right. We can probably "leave that as it is". I'll have to investigate what you have there. It's likely that you can transfer that to pixels.us and not store that on GitHub. We need to pay attention to backing up the Redmine database. I know Andreas used to back it up to a server in his office. I know very little about Redmine.
I know the .git/.svn folder doubles the storage. However that's no problem. It's the release directory that should be removed and those tar-balls put on GitHub. Can you focus on "releases", please?
Can we have the following repos:
| Name | Copied from | Comment |
|---|---|---|
| team.exiv2.org.git | svn://dev.exiv2.org/svn/team | Remove XMPsdk, libraries, tuan and drawings |
| testdata.exiv2.org.git | svn://svn.exiv2.org/svn/testdata | To be used by the test harness (see below) |
I believe @vog is going to work on #2042 in 2022. He has EPS test images which can be stored in testdata.exiv2.org.git and downloaded when needed by the test code. I'd like to minimise the size of the source tar-ball (it's currently about 30mb). It includes lots of small test files in test/data. Bigger files (for video and EPS tests) can be downloaded when required. At this time, we only have to establish the repos. The test files and scripts will be handled in #2042.
I added Stonehenge.jpg to the website yesterday for @postscript-dev. Maybe Stonehenge.jpg should also be stored in testdata.exiv2.org.git and copied into website/html directory when the website/html gets built by website/Makefile.
These are now available:
https://github.com/Exiv2/team.exiv2.org https://github.com/Exiv2/testdata.exiv2.org
Robin, could you please share your steps to build the website from the team repo? I keep running into a problem about a missing Exif.xml file
I've pushed the tags for versions older than v0.26, will upload the releases later today.
Happy New Year. Best Wishes for 2022.
Sounds as though you are making great progress. Have you been working during the night? Are you in India or the States at the moment?
The file Exif.xml is generated in
$ cd <exiv2dir>
$ mkdir build ; cd build ; cmake .. -DEXIV2_ENABLE_BMFF=1 ; make ; sudo make install
$ cd ../doc/template
$ make
I assume you have the file Checklist.txt which documents the build process. I have the following environment strings set up.
835 rmills@rmillsmm-local:~/gnu/exiv2/team/website $ set | grep EXIV2
EXIV2HOME=/Users/rmills/gnu/github/exiv2/0.27-maintenance # You need this
EXIV2WEB=/Users/rmills/gnu/exiv2/team/website # You need this
EXIV2BUILDER=/Users/rmills/gnu/exiv2/team/contrib/buildserver # you probably don’t need this
EXIV2BUILDS=/Users/Shared/Jenkins/Home/userContent/builds/all # you probably don’t need this
836 rmills@rmillsmm-local:~/gnu/exiv2/team/website $ type web
web is aliased to `cd ${EXIV2WEB}'
837 rmills@rmillsmm-local:~/gnu/exiv2/team/website $ type home
home is aliased to `cd ${EXIV2HOME}'
Some of the scripts in $EXIV2WEB need $EXIV2HOME. When you run $ make update, it expects the tags (Exif.xml) to be generated.
Let me know if you want to screen share. The build is fairly robust. But you know what it’s like with home made scripts. What works for the person who wrote the scripts usually doesn’t work for anybody else!
I’ve had a security alert: GitGuardian has detected the following Google API Key exposed within your GitHub account. There’s something in my ~/bin/.profile that shouldn’t be public. My ~/bin is stored in team/rmills/bin and this thing is public. It has something to do with IMaEA (my book Image Metadata and Exiv2 Architecture). I don’t know what this Google API Key is and I never use it. So, I’ll kill the key after breakfast.
The security alerts are valid and have been serviced. They are not a threat to Exiv2 in any way. I had exposed my Google Photos API key. I've modified the file involved and cancelled the key on google. I had accidentally revealed the ftp password to a web site I managed about 15 years ago. I've removed the script with that password.