Improve console output under Windows for UTF-8 strings

[tester0077]

I'm always frustrated when Exiv2 and its associated utilities do not display the sample output as expected when the sample files contain UTF-8 strings in the metadata when the samples are executed under Windows. It seems a common enough issue under Windows and while there seems to have been some discussion, I have not seen the underlying issue addressed and resolved as part of the Exiv2 code. The attached sample/test file Metalith.jpg was supplied as part of and referred to in an earlier discussion about UTF-8 strings, though mainly in the slightly different context of UserComment strings. However, it also contains a sample of a UTF-8 string in the XMP metadata which is helpful in this context. And, while this comment uses xmpprint as an example, the same fix can be applied to the other Exiv2 utilities.

The solution which works for me, is to include some conditional code in xmpprint.cpp At the top of the code add the snippet

#if defined(_MSC_VER)
#include "Windows.h"
#endif

and at the point where output is to be displayed add the following code snippet before the loop

 #if defined(_MSC_VER)
     	// Change the default DOS window to use a code page which can handle UTF-8 strings.
        SetConsoleOutputCP( 65001 );
        std::cout << std::endl; // extra space at top of window
 #endif
     // existing code
     for (const auto& md : xmpData) {
         std::cout << std::setfill(' ') << std::left << std::setw(44) << md.key() << " " << std::setw(9)
                   << std::setfill(' ') << std::left << md.typeName() << " " << std::dec << std::setw(3)
                   << std::setfill(' ') << std::right << md.count() << "  " << std::dec << md.value() << std::endl;
     }
 

The same effect can be obtained by executing the following command at the DOS prompt CHCP 65001 This would be useful for testing compiled Exiv2 utilities outside the IDE.

Also note, that once the "CHCP 65001" command has been executed in the DOS window, it is only effective as long as the particular instance is running. The command MUST be executed every time a new DOS prompt window is opened. The CHCP 65001 command does NOT seem to work for a Powershell prompt (no real surprise) and I have not tried to find an alternative for PS.

Within the MSVC IDE (MSVC 2019 Community edition, in my case), once the code page is switched to 65001, it seems to remain active until the current instance of the IDE terminates. The 2 attached screenshot show the difference. The first one includes the extra code, the other does not.

[clanmills]

Arnold

If I'm going to get my book finished in 2020, I can't take on any more challenges/puzzles. I spent all of this week dealing with RAF files #1402. I've had substantial new issues reported about XMP #1415 and Iptc #1416 and #1395 and the on-going legal/ISOBMFF issue: #1229

You'll probably remember in the "The Temple/Lepers" in Jesus Christ Superstar: https://www.allmusicals.com/lyrics/jesuschristsuperstar/thetemple.htm https://www.youtube.com/watch?v=c5MgI7zpHKs

.... Will you touch, will you mend me Christ? Won't you touch, will you heal me Christ? Will you kiss, you can cure me Christ? Won't you kiss, won't you pay me Christ?

JESUS

There's too many of you...Don't push me. There's too little of me...Don't crowd me. Heal yourselves!

I have a constructive suggestion. Please signup and open the subject of a maintainer for Exiv2 on this forum: https://discuss.pixls.us

[clanmills]

@tester0077 I've had another thought about this. Does this have anything at all to do with Exiv2? Isn't this some to do with the terminal emulator? Microsoft have a new terminal application which runs cmd.exe, bash (on WSL), or PowerShell - and presumably other shells. This is a replacement for the venerable (and hideous) console. Perhaps the solution is to run exiv2 in the new terminal with the appropriate terminal emulator in use. There are other terminal emulators for Windows.

As you know, I know very little about code pages, characters sets, terminal emulators and everything outside of the world of 7-bit ascii. First and foremost, Exiv2 is a cross-platform library and the sample applications are provided for testing and documenting the use of the library. I don't think the Microsoft command-line tools or any of the Unix command tools for Windows (MinGW/msys2, Cygwin/64 and others) do what you are asking of Exiv2.

[clanmills]

https://superuser.com/questions/269818/change-default-code-page-of-windows-console-to-utf-8

[tester0077]

@clanmills I very much appreciate your position and eagerness to complete your book and retire. I am sure you will finish the book, but whether you will really be able to bring yourself to retire to the point where you will be completely divorced from Exiv2, I find hard to believe ;-) OTOH, the issues I have raised regarding Exiv2 stem from a very similar eagerness to complete some of my projects dealing with the classification and sorting of my family pictures by embedding sufficient useful metadata - all before my second retirement.

As for the current issue, I will be quite happy if the issue stays unresolved in the current dot releases. For myself, I know how to circumvent or fix the issue, much like the free() function in the bowels of Exiv2 we have corresponded about before. Having a 'solution' at least in the project history, will be of some help to others who might end up with the same problem and perhaps these problems will be resolved as part of 2.8. I must confess to be very puzzled to find myself as the only one struggling with issues of this sort. Surely I can't be the lone Windows user with text strings encoded in anything but plain ASCII. Then again, I would expect very few console apps to be in development. Unfortunately, either some eggheads at M$ are still bothered that the world did not accept their multi-byte string solutions, or else the multi-byte string code is so pervasive and deeply buried in the OS code, that they cannot afford to upset the Win OS apple cart by changing it all to UTF-8. Even if they did, it would still leave them with plenty of backwards compatibility issues.

The last link you shared, was where I had found my solution. Just now I even tried to 'fix' the problem by fiddling with the registry to use the UTF-8 code page, only to very quickly find that a number of application threw up their guts and forced me to promptly revert back to the old value.

When I started my journey into using metadata in images, Jpg had barely outgrown sidecar files. For quite some time all was well with my using exiv2lib in my work, but soon enough, some of those devilish Umlauts appeared and caused me no end of trouble - almost literally, since I am still fighting with the likes of them. Working in the Win environment is one of the major reasons for my troubles. Though, even recognizing this, I can not see myself switching. For part of my time before retirement, I was working in a Solaris environment. At various times I had considered and dabbled with Linux, but what drew me back to Windows, was the debugging and development IDE. At the time, I knew nothing about metadata and UTF-8, still I would not trade MSVC IDE just to be able to handle UTF-8 strings more easily :-)

[clanmills]

I've always really liked Visual Studio for C++ development. I've used just about everything at different times including Eclipse, CLion, Xcode, CodeBlocks, QtCreator and other IDEs for embedded, python, perl, javascript. Visual Studio is really good. I used it once for an embedded system with an external makefile to cross-build (over ssh) for the device. The controller was debugged with printf statements and lights blinking on the board!

If you find that adding the following code to every main(), does the business for you, then I'm happy that you are happy.

#if defined(_MSC_VER)
     	// Change the default DOS window to use a code page which can handle UTF-8 strings.
        SetConsoleOutputCP( 65001 );
        std::cout << std::endl; // extra space at top of window
 #endif

I will be 100% done with Exiv2 after LGM in Rennes which I hope will happen in 2021. If Rennes is cancelled, I'll be 100% after the book's finished in 3 weeks time.

I've given the community fair warning of my intention. If Exiv2 falls into disrepair, so be it. At least I've gone to the trouble to document how Exiv2 works and how metadata is stored and processed. My knowledge has been acquired by perseverance and determination. I have to point out that my program tvisitor.cpp in the book is MUCH simpler than Exiv2 (which is hard to understand). I hope somebody will read/understand tvisitor.cpp and write a brand new metadata library.

I would appreciate you opening a topic on the forum about finding a Maintainer for Exiv2. I'm not bluffing. I'm retiring.

[tester0077]

@clanmills I for one will seriously miss your help, though I can understand your wish to retire after al this time. I have been on this forum long enough to have seen some of the 'conversations'. If I have been a 'pain' at times, (& I know I have been,) I offer my sincere apologies. In many instances, the problems I had were much more related to building the library, or my understanding of metadata and the standards. I remember when I first found exiv2lib and had no clue as to how to use it, I asked some questions and was surprised at the quick and helpful response from you. At the time, I was trying to port an application which seemed useful to me, but the interface to exiv2 was a complete mystery to me and looking back, at least a part of that was because the original code was not using the interface in any resembling what I now believe to be the 'proper' way. Instead of querying the library to find out which data was available, the code checked for all possibilities known to the original coder. At about that time, I also first discovered the Adobe XMP SDK and had several exchanges with the developers, also about handling of UTF-8 strings. As for looking for a maintainer, I see no problem in starting a new topic, if you so wish. However, it had been my understanding that a & Luis had taken on that job.

[clanmills]

To avoid misunderstanding, the forum I am discussing is: https://discuss.pixls.us

Saying something on the Exiv2 Facebook page would also be helpful. https://www.facebook.com/exiv2/

Luis is very helpful and friendly and does code reviews. However, he's busy and I doubt if he'll take on maintenance. All other contributors have busy lives and unlikely to take it on. I've written the book to encourage a new maintainer to get involved. I will of course support and mentor. Because of C-19, folks are working-at-home. So with not having to commute, and with the book, somebody would volunteer. Hasn't happened.

I've always found working with you to be pleasant.

[tester0077]

Hi Robin, just signed up for pixls.us and left a call for a maintainer for Exiv2. I see there is some sort of video meeting planned for this Saturday, perhaps you might want to look in on that. As I have an abiding aversion for FB, I have not posted anything there, though I have scrolled through some of the content.

[clanmills]

Thanks. I saw the meeting. They will be discussing WhiteBalance and stuff that I don't understand.

My real hopes are pinned on discussing this at LGM in Rennes in May. There will be about 150 at the conference and between us we'll think of somebody, or declare Exiv2 dead. Google or somebody, might come to the rescue and take it on. We'll see. For sure, I'm not immortal and can't be expected to work for eternity. I'm in good health and want to pursue other goals. I've given more than anybody else I know to the community.

If Exiv2 dies, there isn't a viable alternative. My book explains how to build a new library.

Thanks for doing this. It'll be interesting to see the response.

[tester0077]

If these users/consumers of the library don't respond, .....

[clanmills]

Thank You, Arnold (@tester0077) for your sensible words on the forum about the consequences of doing nothing. Much appreciated. Thank You.

I'm pleased that the book explains how to parse metadata. This is a very important low-level technology. It's far from trivial and easily as challenging as the world-class products on which I worked in Silicon Valley such as PostScript and Acrobat.

[tester0077]

You are welcome & my pleasure; it is the little bit I can do.

I, for one, very much appreciate all the work you have done and, IMO, you deserve better than to have your request for a successor simply ignored by the folks who most depend on it.

The only thing I can think of in their favour, is that I expect few of them actually pay much attention to what transpires on the Exiv2 forum until they need help, and at that time, they most likely pay little if any attention to any other topic, such as succession.

Aside from that, it is my firm conviction that "Rennes" will be virtual at best and well past your intended retirement date. In any case, by that time, a successor ought to be involved and be more or less well on his/her way to being able to handle the task.

FWIW, as part of my trying to understand metadata better and the underlying standards, I have, in a limited way, written some ad hoc code snippets to parse some aspects of PNG & JPG data and, as a result, I can better appreciate the effort and dedication that went and is going into development & maintenance/stability of the Exiv2 code base.

[clanmills]

I'm going to assign this the milestone: v1.00. I am delighted that there are more contributors working on Exiv2 and somebody may accept this challenge.

[norbertwg]

@tester0077 : I just had a look on this issue. I checked 0.28.0 and there it seems to be fixed, but I did not see where it is changed. Can you confirm, that the issue is solved? If yes: do you know how it was solved?

We have a similar problem with IPTC, which I would try to solve: When Iptc.Envelope.CharacterSet is filled with "ESC % G", texts are encoded as UTF8. Then also display is fine. But if Iptc.Envelope.CharacterSet is empty, the text is not displayed correct.