[Sign In/Sign Out - v14.0.0] Problem with Ruby on Sign In and Sign out

[gregmzs]

When users sign in or sign out, Gitlab goes into an error 500.

In production.log :

On Sign In

Started POST "/users/auth/ldapmain/callback" for 172.18.0.3 at 2021-06-24 10:16:14 +0200
  
ArgumentError (marshal data too short):
  
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

On Sign out

Started POST "/users/sign_out" for 172.18.0.3 at 2021-06-24 10:17:18 +0200
  
ArgumentError (marshal data too short):
  
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

It looks like a problem with Ruby

[ghost]

I have same problems. I use oAuth login.

Why can I log in once in a while ...

[sachilles]

Hi,

I can reproduce the problem, but I don't have a solution yet.

ArgumentError (marshal data too short):

lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

I will analyze the problem in more detail and hopefully be able to fix it.

[sachilles]

After several login attempts, I was able to log in. However, the problem seems to appear very frequently and differently. I use 2FA, so after the login page, the second factor query should actually appear.

Sometimes this happens, and sometimes I get a 500's message right away. On the second page (when the second factor is requested), I am very (rarely) successful in logging in and usually get a 500'er response as well.

This is very strange. Gitlab's self-tests (see https://docs.gitlab.com/ee/administration/raketasks/maintenance.html#check-gitlab-configuration) show no abnormalities.

[tDo]

Was now also mentioned in official gitlab issue tracker: https://gitlab.com/gitlab-org/gitlab/-/issues/334681

(Since someone linked this ticket in there, I guess a "link back" to the gitlab issue tracker might be helpful as well)

[omueller]

+1, same issue here (sometimes just after login, sometimes only after 2FA validation), also after 14.0.5 upgrade.

[omueller]

According to https://gitlab.com/gitlab-org/gitlab/-/issues/334681#note_624402042,

Disabling Rack Attack fixes the issue (but is obviously only a temporary workaround).

The next question (until the issue is fixed upstream) is how to disable this in docker-gitlab, I'll have a try later today if possible.

[sachilles]

Based on your suggestion, I'm using one additional entry in the file docker-compose.yml as an environmental variable (see README.md in the root of this repository):

    environment:
      - ...
      - RACK_ATTACK_ENABLED=false

So far, after this additional configuration (and after restarting the gitlab microservices) any sign in/sign out works without any issues. However, this is just a temporary workaround.

[omueller]

Thanks @sachilles, it works perfectly this way (until a definitive fix is found). Merci!

[EugRomanchenko]

Thank you for suggesting a workaround. But for some reason it stopped working for us today. There were problems in authorization again with an error from the first message.

[omueller]

@EugRomanchenko maybe you could post the detailled logs/docker-compose.yml here and also to https://gitlab.com/gitlab-org/gitlab/-/issues/334681 ? It seems the Gitlab staff is working on it, but they still need a way to reproduce it.

[EugRomanchenko]

@omueller logs are the same

Started POST "/users/sign_in" for 10.60.12.152 at 2021-07-12 07:21:47 +0000

ArgumentError (marshal data too short):

lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Started GET "/favicon.ico" for 10.60.12.152 at 2021-07-12 07:21:47 +0000

I apologize. While browsing the contents of docker-compose.yml, I noticed that the RACK_ATTACK_ENABLED variable is being overridden twice. Removed duplicate entry.

[omueller]

Not sure anything will happen here unfortunately... On gitlab, the issue was tagged as: "Backlog - Issues that we want to do but are not planned right now. Open for contribution from the community.", with 26'002 other issues of the same category.

14.0.6 will most probably also do nothing here (https://about.gitlab.com/releases/2021/07/20/gitlab-14-0-6-released/). The temporary workaround will need to remain a bit longer...

[dimm0]

Isn't paid version affected?

[taladar]

I noticed that updating config/initializers/rack_attack.rb to the version from the repository helps in our source based installation, perhaps it might be useful here too (see gitlab bug).

[sachilles]

Thanks for the tip!

@sameersbn and @solidnerd do you think that template file can be replaced or reduced without major problems?

[kkimurak]

+1 for @taladar , Removing installation of rack_attack.rb works for me (tested on self build image based on sameersbn/gitlab:14.0.0 and sameersbn/gitlab:14.1.1-1, still working for a few days).

[gazben]

According to a comment updating the config/initializers/rack_attack.rb file fixes the issue.

Source: https://gitlab.com/gitlab-org/gitlab/-/issues/334681#note_641824184

Can the issue be fixed by overriding the file during the docker build?

[kkimurak]

I found same reported in #2202 . BTW, I'm running 14.2.3 without overwriting rack_attack.rb for a month. It works with no problems, at least for me. Test with 14.2.4 also succeed just now. I will test 14.3.0 soon.

[kkimurak]

It should have been fixed in #2426.