German script translation: charset UTF-8 deficient on Windows 10

On Windows 10 I find that translations that are generated with the charset “UTF-8” show bad deficiencies:

• In the button text as well as in printed strings umlauts are substituted by some fancy characters.
• The tooltip doesn’t show at all.

I find that my Windows installation has “ISO-8859-1” as the default. But when I generate mo-files with this set or with “ISO-8859-15”, the translation doesn’t show, the English version is presented instead. I can’t see the reason for it, because I get no output at all in the terminal, when I invoke darktable with the “-d lua” option.

Surprisingly, when I translate with charset “charset=CHARSET” or some other not defined string as “charset=” or even with “charset==CHARSET” or other, the translation is performed the way it should be. All umlauts and the sharp s are there in label, tooltip and print.

By the way, translations filed in the “lua\locale\de_DE\LC_MESSAGES” directory are not found. They need to reside in “lua\locale\de\LC_MESSAGES”.

dt 3.0.1

[wpferguson]

@preklov Any recommendations on how to fix this?

I'm afraid, no. With an undefined charset like "CHARSET" the translation is performed perfectly (although with error message) on Windows 10 and macOS Catalina, but on Linux (openSUSE Tumbleweed) dt doesn't accept this mo file at all.

I know too little about Windows to find a way to overcome the need of .../locale/de/... A link from .../locale/de_DE/... didn't work for me. Maybe a new line .../lua-scripts/locale/de/LC_MESSAGES has to be established with a copy of all the .../de_DE/... files.

[wpferguson]

I looked at how darktable organizes it's locale directory. The locale subdirectories are all 2 letter except for pt_BR and pt_PT. Perhaps we should align our subdirectories to mirror darktables. @supertobi, @preklov Thoughts?

[supertobi]

If that fixes the problem, I'm fine.

I was surprised to see that after the update from dt 3.0.1 to 3.0.2 on macOS (here Catalina), lua script translations are not found any more in de_DE, but are expected to reside in de, just like on Windows. Is it the same on Linux by now?

But the other problem prevails. Which charset does a .po file need to translate correctly on Windows?

[wpferguson]

@preklov I've attached a script translation file that I generated. Could you put it in either the de_DE/LC_MESSAGES directory or the de/LC_MESSAGES directory and then change the gettext.bindtextdomain and gettext.dgettext lines to uses scripts instead of gimp. Test it with de_DE and de and let me know what works.

Thanks
scripts_mo.zip

[wpferguson]

@preklov If you would like to test the changes I made and see if they work for you, you can do the following

open a terminal
cd to your lua scripts folder
git checkout -b translation_test master
git pull https://github.com/wpferguson/lua-scripts.git autostyle_i18n

run darktable and see if translations are correct.

I tried scripts.mo on macOS Catalina (dt 3.2.1), where it works just like with the original translation files. But on Windows 10 (dt 3.0.2) the German special characters still are not displayed. Btw, it helped that you inserted the translation of OpenInExplorer in addition to the gimp one, as the gimp translation file doesn't contain any special character.

[ChristianBirzer]

EDIT: Please ignore this post for a while. I just tried to reproduce the problem in order to debug a bit deeper but now everything works as expected. Just using the current master build of dt now. I'm confused.... I will continue watching this issue...
EDIT 2: Even with the official build of dt 3.8.1 the Umlauts now work. It looks like it's a Windows issue here.

Hi,
I just came across this problem when I was trying to translate my Helicon Focus script to German (at least I guess this is the same problem).
My translation file gets loaded but the German umlauts are displayed as question marks.
My locale dir is "de", not "de-DE".

Because I did not know if my translation output was correct, I used the darktable.mo file that came with dt and is known as good as all "built in" strings get translated with correct Umlauts. I just copied the darktable.mo into the locale dir of my script and renamed it to HeliconFocus.po.

I then let all strings translate to "mid-tones" which should get translated to "Mitteltöne" with Umlaut.
When using the dgettext function with the domain of my Script (which then uses my 1:1 copy of the darktable.mo) I get corrupt Umlauts.
When I use the gettext without a domain (which uses the dt mo file, I get the correct Umlauts.
So it seems, the problem is somewhere in the difference between those two functions?

With a call to
return gettext.gettext( 'mid-tones' ):

With a call to
return gettext.dgettext( 'HeliconFocus', 'mid-tones' )

The po file that is uses is the exact same copy in both cases.

BTW: I know, It's a stupid example but I just had this script in my hands and the easiest way was to set all source strings to the same in the _ function ;-)

[ChristianBirzer]

EDIT 2: Even with the official build of dt 3.8.1 the Umlauts now work. It looks like it's a Windows issue here.

After doing some more tests, I'm a little bit smarter now...
As I wrote in my last comment, the Umlauts from my own script translation did not work even with the official build, installed on Win10.
I noticed that they did work after I launched my Visual Studio Code with the darktable project. (On launching, the cmake configure is executed once.) I did not start dt from VSCode but closed VSCode and started the installed official dt instead. And the Umlauts did work now there. I compared the system variable before and after and noticed that after launching VSCode the LANG=DE was set which wasn't before.

So we can conclude that this environment variable is necessary for the translation of the scripts to work properly. The "built in" translation work either way.

IIRC the LANG variable is very common on linux systems and set anyway there but on windows it's rather uncommon. So maybe there could be some check if it is set somewhere or even derive the variable from the language settings in the preferences and set it when launching the script manager?