Multiline strings are not handled when slack orb parses them from an environment variable.

[williamfalconeruk]

Orb version: 4.4.0

What happened:

• we created a commit which has been squashed and merged in github, with multiple commits to your default branch.
• this adds a multiline string to the $COMMIT_MESSAGE environment variable.
• use a slack dynamic template or in custom option the following json:

{
    "blocks": [],
    "attachments":
    [
        {
            "color": "#508c18",
            "blocks":
            [
                {
                    "type": "header",
                    "text":
                    {
                        "type": "plain_text",
                        "text": "Production Release Successful",
                        "emoji": false
                    }
                },
                {
                    "type": "section",
                    "text":
                    {
                        "type": "mrkdwn",
                        "text": "public facing url: <https://${PUBLIC_FACING_DOMAIN}/home>"
                    }
                },
                {
                    "type": "section",
                    "text":
                    {
                        "type": "mrkdwn",
                        "text": "admin url: <https://${ADMIN_DOMAIN}>"
                    }
                },
                {
                    "type": "context",
                    "elements":
                    [
                        {
                            "type": "plain_text",
                            "text": "by user: ${CIRCLE_USERNAME} - branch: ${CIRCLE_BRANCH} - Commit Message: ${COMMIT_MESSAGE}"
                        }
                    ]
                }
            ]
        }
    ]
}

this generated the following:

{
    "blocks": [],
    "attachments":
    [
        {
            "color": "#508c18",
            "blocks":
            [
                {
                    "type": "header",
                    "text":
                    {
                        "type": "plain_text",
                        "text": "Production Release Successful",
                        "emoji": false
                    }
                },
                {
                    "type": "section",
                    "text":
                    {
                        "type": "mrkdwn",
                        "text": "public facing url: <https://www.redacted.service.gov.uk/home>"
                    }
                },
                {
                    "type": "section",
                    "text":
                    {
                        "type": "mrkdwn",
                        "text": "admin url: <https://admin.redacted.gov.uk>"
                    }
                },
                {
                    "type": "context",
                    "elements":
                    [
                        {
                            "type": "plain_text",
                            "text": "by user: some user - branch: master - Commit Message: multiline squash and merge commit example
* first commit

* second commit

* 4th commit

* final commit

Co-authored-by: william Falconer <redacted>"
                        }
                    ]
                }
            ]
        }
    ]
}

this is not handled by the orb and returns the following message:

parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 1, column 1867

Expected behaviour:

the orb should sanitise multiline strings passed in environment variables and replace with \n instead.

Additional Information:

N/A

[williamfalconeruk]

A simple workaround is to use variable substitution e.g. ${COMMIT_MESSAGE//$'\n'/\\n}, which gives the desired result.

[williamfalconeruk]

A simple workaround is to use variable substitution e.g. ${COMMIT_MESSAGE//$'\n'/\\n}, which gives the desired result.

For further info: the above doesn't work for the custom option

[gmemstr]

@williamfalconeruk Given the project in question is open source, would you mind linking to a job where you saw this output? I'm very curious to know what command from the orb is producing this. I don't see anything necessarily wrong with the JSON payload itself.

[williamfalconeruk]

@williamfalconeruk Given the project in question is open source, would you mind linking to a job where you saw this output? I'm very curious to know what command from the orb is producing this. I don't see anything necessarily wrong with the JSON payload itself.

i would give this link, but sadly the job details are behind authentication and controlled by our org, and probably are too old now.

In the end I resorted to writing a python script, using Jinja2 templates and the slack notification libraries to get a robust solution.

There appears to be a number of related tickets to this:

• #265
• #274
• #276
• #279 Sorry if i'm not able to help on this further.

[gmemstr]

All good! Appreciate the issue being raised :) We'll dig in a bit and see what can be done or replicated.

[williamfalconeruk]

ah didn't realise you were a circleci dev rep - i'll take a look again....

[williamfalconeruk]

No sorry - long gone. i'd add

• #264 to this family of issues. I've experienced most of these issues when using the orb. hope this helps.

[katebygrace]

Hi! When is this getting fixed? It's a big pain point for my team.

Posting Status Checking For JQ + CURL: Debian parse error: Invalid numeric literal at line 1, column 2 parse error: Invalid numeric literal at line 1, column 2

[comp615]

This is extremely frustrating. Feels like doing gymnastics to get variables in or make a custom command :(

[Antman261]

Yep, spent a day trying to get commit messages into our slack notifications, only for it to fall over with well written multiline commit messages 😭

[ianjamieson]

We try to get the latest commit message with:

- run: echo 'export GIT_COMMIT_MESSAGE="$(git log --format=%B -n 1 $CIRCLE_SHA1)"' >> $BASH_ENV

This seems to be causing the error for us.

We generally use squash merges as well.

[galvarado]

I am using circleci/[email protected] and I am obtaining some container logs using kubectl logs command and want to send them to slack but I am hitting this. I was trying to "clean" the output with sed but doesn't work.

      - run:
          name: Check Failed Tests
          when: on_fail
          command: |
            FAILING_TESTS=$(kubectl logs job.batch/integration-tests | grep -A 30 -P "failing" | sed s/\"//g  | sed '/^$/d'))
            echo "${FAILING_TESTS}"
            echo export TESTS_FAIL_MESSAGE=\" Tests has failed!! "${FAILING_TESTS}" \" >> $BASH_ENV
      - notify-error

Then I try to use the variable as:

  notify-error:
    description: "Custom slack notification"
    parameters:
      message:
        type: string
        default: ""
    steps:
      - slack/notify:
          event: fail
          custom: |
            {
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": ":red_circle: A $CIRCLE_JOB job has failed"
                  }
                },
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Project:* $CIRCLE_PROJECT_REPONAME"
                  },
                  "accessory": {
                    "type": "button",
                    "text": {
                      "type": "plain_text",
                      "text": "View Job"
                    },
                    "value": "view_job",
                    "url": "${CIRCLE_BUILD_URL}",
                    "action_id": "button-action"
                  }
                }
                 ,
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "${TESTS_FAIL_MESSAGE}",
                  }
                }
              ]
            }

The result is:

parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 61, column 3

Any help would be appreciate it.

[aspin]

Here to add another :+1:. I even tried dumping a escaped string to $BASH_ENV but it just gets unescaped on its way out.

[alexcheuk]

A weird behavior that I have noticed is that if the multiline commit message was written on the GitHub web interfaces like the one below:

and I use git show --pretty=format:%B --no-patch to fetch the commit message, the new line is written with \r\n instead of just \n. This will cause the error mentioned parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at.

The solution for us was:

echo "export COMMIT_MESSAGE=\"$(git show --pretty=format:%B --no-patch | tr -d '\r')\"" >> $BASH_ENV

[attrivivekFF]

Is there any solution to this problem. We are facing this issue but unable to resolve it since last 2 days?

[attrivivekFF]

This was happening because of control characters in the variable. Below command helped me fix it: sed -e 's/[^[:print:]\r\t]//g'

@alexcheuk I think it would be helpful for other to add this into the orb code itself.