ssm start-session AWS-StartPortForwardingSession blocking behavior

[ghost]

trafficstars

Confirm by changing [ ] to [x] below:

• [x ] I've gone though the User Guide and the API reference
• [x ] I've searched for previous similar issues and didn't find any solution

Issue is about usage on:

• [ ] Service API : I want to do X using Y service, what should I do?
• [ ] CLI : passing arguments or cli configurations.
• [X] Other/Not sure.

Platform/OS/Hardware/Device Fedora 31 EC2 instance

Describe the question This concern the behavior of the SSM Session Manager command used to forward port: aws ssm start-session --target i-XXXXX --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["XXXX"], "localPortNumber":["XXXX"]}'

The command works as intended and I succesfully used it to forward ports, but, I noticed the following behaviour:

• The command is blocking.
• If I exit the command with for instance CTRL+C, the port forwarding is still enabled.
• The blocking command exit itself only if the session is closed externally (For instance from the console or with a terminate-session call from another shell)

If port forwarding is still enabled when the command is exited, why the command is blocking? I assume there is a background service that manages the port forwarding.

Is it possible to run the command in a non-blocking way and retrieve the session ID as output ?

Is it possible to forward the remote instance port to a local Unix socket ?

All of this is not detailed in the documentation.

Logs/output Get full traceback and error logs by adding --debug to the command.

[qoomon]

I'm facing the same problem.

[qoomon]

When I hit Control+ C I only get the message Terminate signal received, exiting. however nothing is happening after that. 'session-manager-plugin and aws ssm start-session --document-name 'AWS-StartPortForwardingSession' ... process are still running.

[qoomon]

My current workaround looks like this

aws ssm start-session --document-name 'AWS-StartPortForwardingSession' \
  --target "${INSTANCE_ID}" \
  --parameters "{\"portNumber\": [\"${INSTANCE_PORT}\"], \"localPortNumber\": [\"${LOCAL_PORT}\"]}" &
# WORKAROUND - stop session-manager-plugin on Ctrl + C. See https://github.com/aws/aws-cli/issues/5149
AWS_SSM_PID=$!
sleep 2
SESSION_MANAGER_PLUGIN_PID=$(ps o pid,ppid,command | grep "^[0-9]* $AWS_SSM_PID session-manager-plugin" | cut -d ' ' -f1)
trap 'exit_code=$?; kill $SESSION_MANAGER_PLUGIN_PID; exit $exit_code' INT TERM EXIT
wait $AWS_SSM_PID
# WORKAROUND END

[kdaily]

Hi @xlz-jgoutin, thanks for the report. Looks like some improvements need to be made to the documentation, and to change the behavior to make it exit properly.

[quentin9696]

Hi, Thanks for this issue, I got the same with AWS cli aws-cli/1.18.173 Python/3.8.6 Linux/5.9.6-arch1-1 botocore/1.19.13

What about this issue ?

Quentin.

[dylanlive]

@quentin9696 I resolved this issue by reinstalling sessionmanagerplugin https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-macos

[SuchismitaGoswami]

This will be an important addition. Any update when this feature would be available?

[seanamos]

Stumbled onto this myself using SSM with scripts. Sending a TERM control signal to the AWS CLI process leaves a session-manager-plugin process running.

To clean it up, you need to recurse the child processes aws start-session creates and send a TERM signal to the session-manager-plugin process directly. Quite unexpected.

[sbarakat]

Just ran into this issue as well.

The main point from the original issue raised I feel is this:

Is it possible to run the command in a non-blocking way and retrieve the session ID as output ?

I'm currently working around this using this snippet of commands:

# Start session
aws ssm start-session --target $instance_id \
    --document-name AWS-StartPortForwardingSession \
    --parameters '{"portNumber":["XXXX"], "localPortNumber":["XXXX"]}' &

# Immediately get the session-id, sessions ordered by StartDate descending 
my_identity=$(aws sts get-caller-identity --query 'Arn' --output text)
session_id=$(aws ssm describe-sessions --state "Active" \
    --filters "key=Owner,value=$my_identity" "key=Target,value=$instance_id" "key=Status,value=Connected" \
    --query 'Sessions[].{SessionId:SessionId,StartDate:StartDate} | reverse(sort_by(@, &StartDate)) | [0].SessionId' --output text)

# Do stuff...

# Terminate session
aws ssm terminate-session --session-id $session_id

If aws ssm start-session returned the session-id directly I'd have the assurance its the same session that's terminated later. It is visible on the session-manager-plugin child pid, but extracting it involves a lot of text fiddling.

[gbooth27]

Any updates on this? I am running into the same issue trying to extract the session ID returned

[dalibor-petras]

+1 I need this functionality as well in my CI workflow.

This is my workaround that could work in your scenario as well. 1, Start sessions with nohup or with '&' at the end. 2, At the end of the CI pipeline run:

killall -u $(whoami) -e session-manager-plugin -s SIGINT

to terminate any active aws cli ssm plugin sessions started by current user.

[dan-carlson]

Just found this, not sure if you are still having the issue, but we are now getting the session-id on stdout. Here is my solution (in context of github actions yml)

  jobs:
    aws-session:
      runs-on: ubuntu-latest
      outputs:
        SSM_SESSION_ID: ${{steps.start-ssm-session.outputs.SSM_SESSION_ID}}
      steps:
      - name: Start AWS SSM Session
          id: start-ssm-session
          run: |
            aws ssm start-session --target i-xxxxxxxxxxxx --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["5432"], "localPortNumber":["5436"]}' > ssm_output.txt 2>&1 &
            sleep 10 # Give it a moment to ensure the command has output the session Id
            echo "Contents of ssm_output.txt:"
            cat ssm_output.txt
            echo "Attempting to extract Session Id..."
            SESSION_ID=$(grep -oP 'SessionId: \K[a-zA-Z0-9-]+' ssm_output.txt | head -1)
            if [ -z "$SESSION_ID" ]; then
              echo "Session Id not found in the output."
              exit 1
            fi
            echo "Extracted Session ID: $SESSION_ID"
            echo "SSM_SESSION_ID=$SESSION_ID" >> $GITHUB_OUTPUT

Then in a later job

    aws-session-terminate:
     runs-on: ubuntu-latest
     needs: [aws-session]
     if: always()
     env:
       SSM_SESSION_ID: ${{needs.aws-session.outputs.SSM_SESSION_ID}}
     steps:
     - name: Terminate SSM Session
       run: |
         aws ssm terminate-session --session-id $SSM_SESSION_ID