request.raw_post has encoded content

[ArturT]

I noticed when I updated rack-utf8_sanitizer form 1.2.4 to 1.3.1 then request.raw_post in controller has encoded content. I had to use URI::decode(request.raw_post) to get the same content I used to have.

[whitequark]

@sj26 this seems to be your change at fault. Care to take a look?

[ArturT]

I checked changes between https://github.com/whitequark/rack-utf8_sanitizer/compare/v1.2.4...v1.3.0

I noticed problem happened because of sanitize_io method https://github.com/whitequark/rack-utf8_sanitizer/blob/master/lib/rack/utf8_sanitizer.rb#L113

My case is, someone is sending POST xml to my rails action and I used to get that xml in controller action from request.raw_post. Since I updated rack-utf8_sanitizer version I need to do URI::decode(request.raw_post).

[whitequark]

Yeah, I consider that a bug. Unfortunately I've no time to work on it in near future, maybe @bf4 can take a look.

[Fivell]

any update guys?

[whitequark]

No update since my previous comment. I haven't done any Rails in years.

[bf4]

@Fivell would be helpful to have a failing test showing the regression.

[edudepetris]

I would say it's not a bug.

From my understand,

It's ok has encoded content on request.raw_post if your content-type is application/x-www-form-urlencoded. That's what sanitize_io does.

My case is, someone is sending POST xml to my rails action and I used to get that xml in controller action from request.raw_post. Since I updated rack-utf8_sanitizer version I need to do URI::decode(request.raw_post).

I'm guessing here, but I think this request has a bad content-type, actually it must to have application/x-www-form-urlencoded instead of 'application/xml'

In these tests I tried to replicate what I've mentioned above https://github.com/edudepetris/rack-utf8_sanitizer/commit/93762bffe3c7d59d3681d3860aaf6f5f604d83cd

[ndbroadbent]

I just installed rack-utf8_sanitizer in my application, and this issue is causing all of my API controller actions to fail with a Error occurred while parsing request parameters when someone uses curl -d to send a test API request (as form post data.)

I am doing something similar to the original poster, where I force the content-type into application/json instead of application/x-www-form-urlencoded, because this makes it much easier to handle any kind of POST request:

module Rack
  class ApiContentType
    def initialize(app)
      @app = app
      @methods = %w[POST PATCH]
      @path = /^\/api\/v\d+/
    end

    def call(env)
      req = Rack::Request.new(env)

      if @methods.include?(req.request_method) && @path.match(req.path.to_s)
        if req.content_type.blank? ||
            req.content_type == 'application/x-www-form-urlencoded'
          env['CONTENT_TYPE'] = 'application/json'
        end
      end

      @app.call(env)
    end
  end
end

I would definitely consider this to be a bug because it is not related to sanitizing UTF-8 characters, and it is unexpectedly changing the default behavior in Rails (which was working fine before.)

UPDATE: I fixed my issue by moving the ApiContentType middleware to the very beginning (before Rack::UTF8Sanitizer.) This prevents UTF8Sanitizer from sanitizing any form data.