Installation fails to start Arango service on Windows 10

[aviadmini]

trafficstars

My Environment

• ArangoDB Version: 3.4.3
• Storage Engine: MMFiles
• Deployment Mode: Single Server
• Operating System: Windows 10 build 1809
• Total RAM in your machine: 16Gb
• Disks in use: 1
• Used Package: Official installer

Steps to reproduce

1. Run installer with default settings

Problem: ArangoDB service fails to start and points to check Windows Event log Event log contains the following

{config} specified language 'en_US' does not match previously used language ''
c:\c17-windows.build\oskar\work\arangodb\arangod\restserver\languagecheckfeature.cpp
arangodb::LanguageCheckFeature::start
146

[Simran-B]

Related:

• https://github.com/arangodb/arangodb/pull/7490 Persist and check default language selection

[Simran-B]

@aviadmini May I ask whether this was a first time installation? Did you start the server before, or upgraded from a previous version?

What is the language of your operating system? Is it something other than English (US)? I wonder if this would also occur under a Windows in en_US language - it might, because comparing an empty string to en-US will result in a mismatch. My theory is that the server is started without a specified language, some parts get initialized, but then the language appears to default to something and eventually there is a check which fails because of the now mismatching language settings...

[aviadmini]

Yes it was installed first time on this computer. ArangoDB itself works just fine when launched as console app, but service fails to start.

My OS locale is en_US

Tell me if there is anything I can do to help

[Simran-B]

@aviadmini Do you see the error each time you attempt to start the service, e.g. via the task manager > services tab, or via the services.msc panel, or in an (elevated) command prompt with sc start arangodb? And if you start it in user mode by running the arangod binary you never see the error? Or did the error occur just once, when the installer tried to start the service for the first time, and now it can also be started as service without issues?

[aviadmini]

I cannot start the service from any of the options above - each attempt results in the error above (I look up the error in Event log) I can start the "Arango Server" as an application from programs menu (it shows console and works correctly)

[Simran-B]

Thanks! It could be related to the installer, how it sets up the service, but I am not sure. This will need further investigation on our end.

[aringot]

Hi, same problem here.

My Environment

• ArangoDB Version: 3.4.3
• Storage Engine: RocksDB
• Deployment Mode: Single Server
• Operating System: Windows 10 build 17763
• Total RAM in your machine: 8Gb
• Disks in use: 1
• Used Package: Official installer

I was unable to start the server so I installed 3.3.22 for now (that seems to work fine). Please keep us posted.

[ile]

My Environment

• ArangoDB Version: 3.7.6
• Storage Engine: RocksDB
• Deployment Mode: Single Server
• Deployment Strategy: Manual Start
• Operating System: Ubuntu 20.04
• Total RAM in your machine: 8Gb.
• Disks in use: SSD
• Used Package: Ubuntu .deb

I have the same problem in a new Ubuntu installation (from ssdnodes.com). Started with arangodb3.service.

2021-01-25T18:37:25Z [54451] FATAL [7ef60] {config} specified language 'en_US' does not match previously used language ''

Tried dpkg-reconfigure locales but didn't help.

[ile]

A correction: reboot did help. So I suppose dpkg-reconfigure locales required a reboot to work.

Problem fixed.

[dothebart]

I don't think this is related. but yes - you mustn't change the locale of an existing installation; in that case you need to dump, start a fresh DB and restore.

[ile]

This was still a strange error. Not clear why it happened and how to fix.

No dbs were created though so I suppose just changing/reconfiguring locale helped.

[dothebart]

Please (if) create a new report, and if you feel that other issues are similar reference them by link.

[fsoft72]

I had the same problem on Ubuntu 20.04

I fixed this with the following changes:

1. edit "/etc/systemd/system/multi-user.target.wants/arangodb3.service"
2. change line: Environment=GLIBCXX_FORCE_NEW=1 with Environment=GLIBCXX_FORCE_NEW=1 LANG=C.UTF-8
3. exit
4. systemctl daemon-reload
5. restart the service

Maybe it is my installation, my default locale is LANG=C.UTF-8 I am running a vanilla Ubuntu Server minimal

[Simran-B]

@fsoft72 The server should fall back to the system locale if default-language isn't set explicitly. It looks like there is none for services however, as you had to set it explicitly? I wonder what the content of your LANGUAGE file (in the database directory) is and whether setting default-language eliminates the need for setting LANG in the service environment.

I tried to reproduce the issue with a Debian base installation (debootstrap) in Alpine in Docker, but it worked just fine. It's unclear how it ends up with an empty language and what the persisted LANGUAGE is at that point under Linux.

[fsoft72]

@Simran-B This is the content of LANGUAGE file

/var/lib/arangodb3# cat LANGUAGE 
{"default":""}

As you can see, it is empty.

Could it be a bug in the installation script? I am using the "Installation via Package Manager" as described here: https://www.arangodb.com/download-major/ubuntu/

[Simran-B]

Thanks! It's still unclear how the server gets into this situation. I suspect the system locale to be empty in rare cases, and because arangod's default-language is also not set, it falls back to the empty system locale and persists that to the LANGUAGE file. There might be another fallback to en_US somewhere (possibly related to ICU), which then causes the mismatch.

What's your exact Ubuntu version and can you remember how you installed it? It would be very valuable to have an environment to reproduce the problem in.

[fsoft72]

@Simran-B

Ubuntu version:

root@n01:~# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

This is the install process (from bash history)

  428  curl -OL https://download.arangodb.com/arangodb37/DEBIAN/Release.key
  429  sudo apt-key add - < Release.key
  430  echo 'deb https://download.arangodb.com/arangodb37/DEBIAN/ /' | sudo tee /etc/apt/sources.list.d/arangodb.list
  431  sudo apt-get install apt-transport-https
  432  sudo apt-get update
  433  sudo apt-get install arangodb3=3.7.6-1

[Simran-B]

Sorry, I meant how the operating system was installed. If I docker run --rm -it ubuntu:18.04 then I get this:

> cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

> locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

> locale -a
C
C.UTF-8
POSIX

After installing ArangoDB as described, /var/lib/arangodb3/LANGUAGE is {"default":"en_US"} and I can start it fine with service arangodb3 start. It looks like the behavior is different with Docker or the operating system.

[fsoft72]

@Simran-B sorry for late answer. Here is my output:

root@n01:~#  cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

root@n01:~# locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC=C.UTF-8
LC_TIME=C.UTF-8
LC_COLLATE="C.UTF-8"
LC_MONETARY=C.UTF-8
LC_MESSAGES="C.UTF-8"
LC_PAPER=C.UTF-8
LC_NAME=C.UTF-8
LC_ADDRESS=C.UTF-8
LC_TELEPHONE=C.UTF-8
LC_MEASUREMENT=C.UTF-8
LC_IDENTIFICATION=C.UTF-8
LC_ALL=

root@n01:~# locale -a
C
C.UTF-8
POSIX
en_US.utf8

The system is a standard Ubuntu server minimal installed automatically by my hosting provider on my virtual machine

[robsiera]

I'm having the same issue as the title. Win10. But I was upgrading from v3.6.4 to v3.7.6. First I had issue that the installer failed. I then tried to run arangod.exe but it complained on not being able to write to the data folder. I was able to fix that by refreshing the NTSF permissions on the Data folder.
So arangod is now running from the commandline, but not as a service (as the original post of this issue explains). I'm getting this error in the windows eventlog:

2021-02-06T20:41:51Z [12900] FATAL [7ef60] {config} specified language 'en' does not match previously used language ''
D:\hw24-windows\oskar\work\ArangoDB\arangod\RestServer\LanguageCheckFeature.cpp
arangodb::LanguageCheckFeature::start
148

I tried to rerun the installer but that ended with the same error as above.

Please advise.

--- edit --- The issue does not seem to be related to the installer. I'm saying this because I un installed the new v3.7.6, restored the old data and tried to upgrade to v3.6.5. I ran into the same issue. The only difference now: I cannot even start arangod from the commandline. I get this error:

FATAL [7ef60] {config} specified language 'root' does not match previously used language ''

while the eventlog give a little bit more detail

[7ef60] {config} specified language 'root' does not match previously used language '' C:\vm02-windows\oskar\work\ArangoDB\arangod\RestServer\LanguageCheckFeature.cpp arangodb::LanguageCheckFeature::start 147

--- edit --- More observations:

• clean install of v3.6.4: no problem. But after restoring the database, which was also of v3.6.4, the above error occurs again, even at the commandline. I can not use Arango.
• clean install of v3.7.6: installation fails. Service cannot be installed due to above error.

ticket #12550 is related to the same issue.

--- SOLUTION --- A clean install of v3.6.4 (downloaded today 9th feb 2021) creates a file C:\ProgramData\ArangoDB\LANGUAGE which has this content

{"default":"root"}

while the content in my data backup was

{"default":""}

Changing this empty string to "root" solved the error. Subsequent upgrade to v3.6.11 when smooth less. I'm not going to upgrade to v3.7.4 for now. I'm going to enjoy my new stable environment first ;-)

[mahadi-hossain]

Me also not being able to run 3.7 and 3.8 versions. But 3.6 was OK I am using Ubuntu server 18.04 on Oracle VM VirtualBox.

My log-- https://github.com/mahadi-hossain/arangodbLog/blob/main/file.txt

[sorenhoyer]

I was having problems installing v 3.8 on Windows 10 after having just uninstalled v 3.6. The installer would fail and I could not start the service no matter what I tried. After simply opening C:\ProgramData\ArangoDB\LANGUAGE and adding "root" as @robsiera suggested in the above comment worked like a charm. The install succeeded without any errors and service now runs fine.

[dothebart]

please note that root is not a valid configuration value for this. Its intended to be i.e. en_US . The reason to refuse to start is, that if you index non-ascii values, ICU is used to encode them. Depending on the language this encoding may be different. Hence if you then access the index with a different locale, look ups to readily existing values may simply return nothing:

FOR x IN col FILTER x.indexedValue == 'Motörhead' RETURN x

so to mitigate this, you should drop & recreate such indices.

[Fennec-hub]

Same problem and none of the above worked for me.

Ubuntu 22.04 TLS, (ArangoDB 3.9.2 [linux] 64bit, using jemalloc, build tags/v3.9.2-0-g8bf70c5f5b6, VPack 0.1.35, RocksDB 6.27.0, ICU 64.2, V8 7.9.317, OpenSSL 1.1.1o 3 May 2022)

arangod.log FATAL [7ef60] {config} Specified language 'en_US' with type 'default' does not match previously used language '' with type 'default'

locale LANG=en_US.UTF-8 LANGUAGE= LC_CTYPE="en_US" LC_NUMERIC=en_AU.UTF-8 LC_TIME=en_AU.UTF-8 LC_COLLATE="en_US.UTF-8" LC_MONETARY=en_AU.UTF-8 LC_MESSAGES="en_US.UTF-8" LC_PAPER=en_AU.UTF-8 LC_NAME=en_AU.UTF-8 LC_ADDRESS=en_AU.UTF-8 LC_TELEPHONE=en_AU.UTF-8 LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=en_AU.UTF-8 LC_ALL=

[dothebart]

you have installed arangodb on a system with not properly set up locale. try running

sudo locale-gen "en_US.UTF-8"

and retry from scratch.

[Fennec-hub]

Believe me I tried that with en_US, en_US.UTF_8 and dpkg-reconfigure locales, even rebooted after each reconfiguration/uninstall then install again.

I gave up on the deb package, for now I'm using the docker image and it works fine.

Thanks for the quick response.