More helpful logs when service "Create Failed"

[kdhillon]

Tell us about your request When creating a new service, create fails without any errors in deployment logs. The last visible logs are

02-14-2022 10:18:10 PM [Build] Successfully tagged application-image:latest
02-14-2022 10:18:10 PM [Build] Successfully built 2a53f3e899cb

Which suggests that build succeeded. Seems like there should always be an error logged in deployment log anytime a Service Create fails.

Describe alternatives you've considered Tried looking through forums and past posts, all of them suggest checking logs, but event logs and/or deployment logs show no errors explaining why it failed.

Additional context ARN: arn:aws:apprunner:us-west-2:520917683465:service/arpeggi-web-beta2/600285cea79a48fbb137786c78d31b52 Another ARN: arn:aws:apprunner:us-west-2:520917683465:service/arpeggi-web-beta/ea870ffb4d0a44979811a77ac1bf96f5

Current Service Quota: 10. Existing Services: 3.

Repro steps (reproducible 5/5 tries):

• "Create Service"
• "Source code repository"
• Connect github repo + branch (we have another Service correctly set up and running from the exact same github branch)
• Deployment settings: Manual or Automatic (I've tried both)
• Configure settings to be exact same as existing Service
• Default settings (AWS managed key, etc)
• Wait for "Create service in progress..." to complete, always get "Create Failed"
• Event log is always:

02-14-2022 10:08:30 PM [AppRunner] Starting source code build.
02-14-2022 10:08:29 PM [AppRunner] Successfully pulled source code.
02-14-2022 10:07:41 PM [AppRunner] Service status is set to OPERATION_IN_PROGRESS.
02-14-2022 10:07:41 PM [AppRunner] Service creation started.

Deployment Log for "Create Service" contains no errors: https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Faws$252Fapprunner$252Farpeggi-web-beta2$252F600285cea79a48fbb137786c78d31b52$252Fservice/log-events/deployment$252F1ecf04a9c05c4bf1a7410d61a652aff6

[tsheaff]

Yeah this is crazy. I've run into the same issue. I have absolutely no way to debug why it's failing!

[tsheaff]

I'm sure it's an easily solvable issue for why the creation is failing, but with no way to inspect what went wrong, all I can do is poke around in the dark for hours 😠

All I see is something like this. None of the other tabs or CloudWatch has any extra details:

[tsheaff]

cc @akshayram-wolverine

[tsheaff]

In your official guide it says this, which is completely unhelpful because there are no logs

[McLeroy]

I couldn't use App Runner because of this :(

[abba-logico-srl]

Same problem here :(

[michael-newsrx]

This morning I can't resume or create any services. :-(

Nothing logged in either case as to why.

[vladnicula]

same problem here...

[laurenprete]

I faced this issue as well. Failed service creation and deployment both produce empty/useless logs.

I ended up giving up and deploying my app with ECS on Fargate instead. This got Cloudwatch to log an actual error, so I fixed the problem, then redeployed to app runner and it worked. It would have been helpful if App Runner had logged the underlying error in the first place.

[snnles]

Hi everyone,

We are working on providing more actionable logs for create service workflow. You will see incremental improvements in the 'Deployment Logs' and 'Event Logs' over the time.

Meanwhile, If you have any idea to share around how we can make logging and troubleshooting in App Runner better and what information you would like to see in 'Deployment Logs' and 'Event Logs', please share your ideas/feedback here.

Thanks!

[simllll]

Experience something simliar right now, "[AppRunner] Failed to deploy your application source code." but no reason why, is this also part of the improvements around logs?

[hariohmprasath]

Hi @simllll Thanks for reaching out, we are currently working on different options to improve the log experience. Meanwhile can you let us know the service ARN so we can provide you with some details around the service creation failure.

[simllll]

@hariohmprasath thanks for your quick response, e.g. arn:aws:apprunner:eu-west-1:770785616967:service/app-runner-git-deploy-service/aa6ff0e1cbc9414b84b9b104e0373d83 or arn:aws:apprunner:eu-west-1:770785616967:service/website-app-test/c017277194e04c08b05977db5dd7b11a

I guess both have the same issue though. Thanks a lot!

[hariohmprasath]

Hi @simllll, I had a quick look at these services looks like both of them failed at build stage when running the build command. These build logs are available under Deployment logs section in the console. If you are fine you can share us a sample project without any proprietary code so we can try recreating this locally and help you fix this.

[simllll]

The deployment logs just show: not really helpful.

Is there any possibility to get the build output?

[simllll]

Interesting, tried again just now, and now I got some output :-) I added to the npm install command the --ignore-scripts tag, not sure if this is related or some update just got rolled out. anyways, this makes things a lot easier (y)

[simllll]

It really seems to be related to the "build" command, I changed it agian, and now I'm back with the "no output" issue.

build-command: npm install --ignore-scripts && patch-package && npm run build

I guess patch-package is failing, but somehow this leads to the behaviour that there is no output logged at all. I will try if it improves if I run a script instead.

[hariohmprasath]

It really seems to be related to the "build" command, I changed it agian, and now I'm back with the "no output" issue.

build-command: npm install --ignore-scripts && patch-package && npm run build

I guess patch-package is failing, but somehow this leads to the behaviour that there is no output logged at all. I will try if it improves if I run a script instead.

Hi @simllll, Yes based on our internal analysis its related to the build command, will you be able to share a sample project that you are using to run these tests? We would love to look at it and understand this better and improve the experience on our end. Thanks

[hariohmprasath]

It really seems to be related to the "build" command, I changed it agian, and now I'm back with the "no output" issue. build-command: npm install --ignore-scripts && patch-package && npm run build I guess patch-package is failing, but somehow this leads to the behaviour that there is no output logged at all. I will try if it improves if I run a script instead.

Hi @simllll , Thanks for your patience. We are able to recreate this issue locally, looks like any errors related to build commands doesn't get transmitted to customer environment. We will work on a fix and update this thread in the upcoming weeks. Meanwhile you can workaround this problem by updating the build command to the pattern mentioned below and you will start seeing any command execution errors in the Deployment logs.

(buildCommand) 2>&1

I was able to give it a try with an invalid command and was able to identify the error using the Deployment logs (refer below):

Hopefully this helps, let us know if you have any questions.

[simllll]

Thanks @hariohmprasath I succeeded with building now, at least according to the log (arn:aws:apprunner:eu-west-1:770785616967:service/website-app-deploy-test-envs/9b8c8ad1fa4a419f9960bac46f58b75a), last few log lines:

	2022-11-30T09:28:02.824+01:00	[Build] [0mRemoving intermediate container 65911fcd9654
	2022-11-30T09:28:02.824+01:00	[Build] ---> 46860de98696
	2022-11-30T09:28:02.824+01:00	[Build] Step 5/5 : EXPOSE 18000
	2022-11-30T09:28:02.824+01:00	[Build] ---> Running in a425b9b34785
	2022-11-30T09:28:02.824+01:00	[Build] Removing intermediate container a425b9b34785
	2022-11-30T09:28:02.824+01:00	[Build] ---> a7eb9cc657cf
	2022-11-30T09:28:02.824+01:00	[Build] Successfully built a7eb9cc657cf
	2022-11-30T09:28:02.824+01:00	[Build] Successfully tagged application-image:latest

looks actually good, but "Create service" is in state failed?

Am I missing something?

I found earlier in the log:

2022-11-30T09:27:38.261+01:00	[AppRunner] Failed to deploy your application source code.

and it started at

2022-11-30T09:07:11.816+01:00	[AppRunner] Starting to build your application source code.

is it maybe related to some timeout?

[hariohmprasath]

Hi @simllll, I have moved this discussion to repost (https://repost.aws/questions/QUHRZ_kd9hR--kIUfAfrDdCg/app-runner-service-creation-failed) and also updated my observation to this service ARN

[asmfadholi]

Hi @hariohmprasath , I'm facing the same issue when deploying to app runner, and the error just said: [AppRunner] Failed to deploy your application source code.

there is no detail provided. I just added @sentry/nextjs library in my nextjs project. Do you have insight about it?

Thank you.

[hariohmprasath]

Hi @asmfadholi, I have moved this discussion to https://repost.aws/questions/QUYEjXJekeQ3O5Xz_YU5W1qw/app-runner-failed-to-deploy. Can you provide the serviceARN, part of your response? We can look into this and share some insights. Thanks

[asmfadholi]

@hariohmprasath I added arn to comment in aws discussion. Thank you.

[hariohmprasath]

Hi, @asmfadholi,

Looks like you have run into a service bug because of a conflict in the NodeJS versions. We will prioritize this and work on a fix soon and keep you posted on the progress. As a temporary workaround you can install nodeJS 14 part of the pre-build phase and use that as a default runtime for building your code. This can be done by creating a configuration based service (using apprunner.yaml) instead of API based one.

Here is a sample apprunner.yaml file that you can use with the suggested fix:

version: 1.0
runtime: nodejs14
build:
  commands:
    pre-build:
      - n 14.18.3
    build:      
      - <<build_command>>
run: 
  command: <<start_command>>
  network:
    port: <<port>>
    env: APP_PORT

Give it a try and let us know if you are still blocked on this. Thanks

[asmfadholi]

I created new app runner service, and config as you suggested above. It works well now. Thanks for your help. @hariohmprasath

[RichardGrant-93]

Hi everyone,

We are working on providing more actionable logs for create service workflow. You will see incremental improvements in the 'Deployment Logs' and 'Event Logs' over the time.

Meanwhile, If you have any idea to share around how we can make logging and troubleshooting in App Runner better and what information you would like to see in 'Deployment Logs' and 'Event Logs', please share your ideas/feedback here.

Thanks!

Well this didn't age well did it.

[januszm]

Not to mention that logs are flipped upside down, newest at the top, which completely breaks most of the multiline messages

[hariohmprasath]

Not to mention that logs are flipped upside down, newest at the top, which completely breaks most of the multiline messages

Hi @januszm A follow up question: Did the service creation fail in your case? If so can you please create a post in repost.aws with the 'serviceARN', screenshot of your logs and share us the post URL? We will be happy to look into this and help you out. Thanks.

[januszm]

oh yeah it did fail. I'm actually still unable to launch a Rails application on AppRunner using "Source code repository".