self-hosted icon indicating copy to clipboard operation
self-hosted copied to clipboard

Unable to upload source maps

Open moroine opened this issue 3 years ago • 10 comments

Version

22.2.0

Steps to Reproduce

  1. I have the following .env file
SENTRY_PROJECT=agg
SENTRY_URL=https://sentry.nugit.co
SENTRY_ORG=nugit
SENTRY_DSN="https://[email protected]/6"
SENTRY_AUTH_TOKEN=yyyyy
SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
NODE_OPTIONS="--use-openssl-ca"

☝️ SENTRY_AUTH_TOKEN is generated using internal integration

  1. Try to upload sourcemaps
$ ./node_modules/.bin/sentry-cli releases files 17fdb74bdd064a534efaf612bed673418c867a8d upload-sourcemaps ./dist --log-level=debug
> Found 16 release files
> Analyzing 16 sources
> Analyzing completed in 0.057s
> Rewriting sources
> Rewriting completed in 0.03s
> Adding source map references
> Bundling files for upload...
> Bundling completed in 0.046s
> Optimizing completed in 0.003s
error: API request failed
  caused by: sentry reported an error: [Errno 13] Permission denied: '/app' (http status: 400)
  1. But I have all the permissions
$ ./node_modules/.bin/sentry-cli info
Sentry Server: https://sentry.nugit.jco
Default Organization: nugit
Default Project: agg

Authentication Info:
  Method: Auth Token
  User:
  Scopes:
    - project:read
    - project:write
    - project:admin
    - team:read
    - team:write
    - team:admin
    - project:releases
    - org:read
    - org:write
    - org:admin

Expected Result

Successfully uploading sourcemaps

Actual Result

When I check logs using docker-compose logs -f here is what I have:


web_1                                       | 14:09:16 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.chunk.ChunkUploadEndpoint' response=200 user_id='8' is_app='True' token_type='ApiToken' organization_id='1' auth_id='2' path='/api/0/organizations/nugit/chunk-upload/' caller_ip='10.xx.xx.xx' user_agent='sentry-cli/1.71.0')
nginx_1                                     | 10.37.209.37 - - [20/Dec/2021:14:09:16 +0000] "GET /api/0/organizations/nugit/chunk-upload/ HTTP/1.0" 200 317 "-" "sentry-cli/1.71.0" "10.xx.xx.xx"
nginx_1                                     | 2021/12/20 14:09:16 [warn] 22#22: *272927 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000011199, client: 10.xx.xx.xx, server: , request: "POST /api/0/organizations/nugit/chunk-upload/ HTTP/1.0", host: "sentry.nugit.co"
web_1                                       | 14:09:16 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.chunk.ChunkUploadEndpoint' response=400 user_id='8' is_app='True' token_type='ApiToken' organization_id='1' auth_id='2' path='/api/0/organizations/nugit/chunk-upload/' caller_ip='10.xx.xx.xx' user_agent='sentry-cli/1.71.0')
web_1                                       | 14:09:16 [WARNING] django.request: Bad Request: /api/0/organizations/nugit/chunk-upload/ (status_code=400 request=<WSGIRequest: POST '/api/0/organizations/nugit/chunk-upload/'>)
nginx_1                                     | 10.37.209.37 - - [20/Dec/2021:14:09:16 +0000] "POST /api/0/organizations/nugit/chunk-upload/ HTTP/1.0" 400 48 "-" "sentry-cli/1.71.0" "10.xx.xx.xx"

moroine avatar Feb 16 '22 17:02 moroine

The original issue was posted in https://github.com/getsentry/sentry-cli/issues/1088

I tried the workaround in https://github.com/getsentry/self-hosted/issues/302 and https://github.com/getsentry/self-hosted/issues/679#issuecomment-696862913 but nothing change.

Also, try to upgrade from 21.10.0 to 22.2.0 but the issue persist

moroine avatar Feb 16 '22 17:02 moroine

Is a warning like this mentioned in any of the previous threads?

web_1                                       | 14:09:16 [WARNING] django.request: Bad Request: /api/0/organizations/nugit/chunk-upload/ (status_code=400 request=<WSGIRequest: POST '/api/0/organizations/nugit/chunk-upload/'>)

chadwhitacre avatar Feb 16 '22 17:02 chadwhitacre

@chadwhitacre I don't think so

moroine avatar Feb 16 '22 17:02 moroine

Okay, so that looks like the place to start debugging. What are you sending to that endpoint that you're getting a 400 from it? What checks does that endpoint have in general that can result in a 400?

chadwhitacre avatar Feb 16 '22 18:02 chadwhitacre

Here is the output with cli:

$ ./node_modules/.bin/sentry-cli releases files ""17fdb74bdd064a534efaf612bed673418c867a8d"" upload-sourcemaps --no-rewrite ./dist --log-level=debug
> Found 16 release files
> Analyzing 16 sources
> Analyzing completed in 0.058s
> Adding source map references
> Bundling files for upload...
> Bundling completed in 0.046s
> Optimizing completed in 0.003s
error: API request failed
  caused by: sentry reported an error: [Errno 13] Permission denied: '/app' (http status: 400)

moroine avatar Feb 16 '22 18:02 moroine

I have a nginx proxy with the following config:

upstream sentry {
	server  xx.xxx.xxx.xxx:8889;
}

server {
	listen  443 ssl;
	server_name  sentry.example.com;

	root  /var/www/dummy_webroot;

	access_log  /var/log/nginx/sentry.access.log  jlog;
	error_log   /var/log/nginx/sentry.error.log;

	ssl_certificate      /etc/nginx/ssl/star_example.com.crt;
	ssl_certificate_key  /etc/nginx/ssl/star_example.com.key;

	client_max_body_size 50m;
	add_header   Strict-Transport-Security  "max-age=31536000"  always;

	location / {
		proxy_http_version  1.1;
		proxy_set_header  Host  $http_host;
		proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

		proxy_pass  http://sentry;
	}
}

moroine avatar Feb 16 '22 20:02 moroine

"Permission denied: '/app'" seems like a pretty strong clue. What's going on there? Any hypotheses for this error?

chadwhitacre avatar Feb 18 '22 23:02 chadwhitacre

The only app folder I found is:

root@a51d371851aa:/# find / -type d -name "app"
/usr/local/lib/python3.8/site-packages/celery/app

And it belongs to root user.

Meanwhile in another instance of Sentry (which is working) I have the following folders, but all belong to root

/usr/local/lib/python3.8/site-packages/celery/app
/usr/local/lib/python3.8/site-packages/sentry/static/sentry/dist/static/app
/usr/local/lib/python3.8/site-packages/sentry/static/sentry/dist/static/app/components/events/contexts/app
/usr/local/lib/python3.8/site-packages/sentry/static/sentry/dist/static/app/views/app

moroine avatar Feb 22 '22 03:02 moroine

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

github-actions[bot] avatar Mar 16 '22 00:03 github-actions[bot]

@chadwhitacre any idea how to fix/debug this?

moroine avatar Mar 16 '22 07:03 moroine

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

getsantry[bot] avatar Aug 11 '23 07:08 getsantry[bot]

Are you still experiencing this issue @moroine ?

hubertdeng123 avatar Aug 11 '23 16:08 hubertdeng123

I disabled it long time ago but never face it again in other projects

moroine avatar Aug 13 '23 21:08 moroine

feel free to reopen if necessary, closing for now

hubertdeng123 avatar Aug 15 '23 17:08 hubertdeng123