Spectacles Cannot update Looker Git branch

[sdblanchard]

trafficstars

When running spectacles versions 2+ on a Looker PR I am getting the following error at the SQL validation step:

  spectacles sql --project app-sales-config --branch ${GITHUB_REF##*/} --remote-reset --incremental --explores sales_analytics/opportunity_arr_stamp_history sales_analytics/arr sales_analytics/commercial_metrics_master_ledger sales_analytics/commercial_metrics_editable_ledger_audit -v
  shell: /usr/bin/bash -e {0}
  env:
    LOOKER_PROJECT: salesloft
    LOOKER_BASE_URL: https://salesloft.looker.com/
    LOOKER_PORT: 19999
    LOOKER_API_VERSION: 3.1
    LOOKER_CLIENT_ID: VMPkXmNFFmNHdCw5hXbg
    LOOKER_CLIENT_SECRET: ***
    pythonLocation: /opt/hostedtoolcache/Python/3.7.1[2](https://github.com/SalesLoft/looker-sales-analytics/runs/6097631259?check_suite_focus=true#step:6:2)/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/[3](https://github.com/SalesLoft/looker-sales-analytics/runs/6097631259?check_suite_focus=true#step:6:3).7.12/x6[4](https://github.com/SalesLoft/looker-sales-analytics/runs/6097631259?check_suite_focus=true#step:6:4)/lib
Authenticating Looker API credentials
Checking Looker instance release version
Connected to Looker version 22.4.29 using Looker API 3.1
Setting up branch manager in project 'app-sales-config'
Getting the workspace in use by this session
Getting active branch for project 'app-sales-config'
The active branch is 'master'
Getting manifest details
Project 'app-sales-config' imports the following projects: ['app-sales-base', 'app-sales-adapter', 'salesloft_hub']

Setting Git state for project 'app-sales-config' @ ISE-3270-Add-SFDC-Views-to-Commercial-Metrics-Explores
--------------------------------------------------------------------------------
Updating session to use the dev workspace
Getting the workspace in use by this session
Getting active branch for project 'app-sales-config'
The active branch is 'tmp_spectacles_e3ed7b06[5](https://github.com/SalesLoft/looker-sales-analytics/runs/6097631259?check_suite_focus=true#step:6:5)3'
Branching 'tmp_spectacles_1338[7](https://github.com/SalesLoft/looker-sales-analytics/runs/6097631259?check_suite_focus=true#step:6:7)552ea' off 'ISE-3270-Add-SFDC-Views-to-Commercial-Metrics-Explores'. Afterwards, restoring to branch 'master' in project 'app-sales-config'
Creating branch 'tmp_spectacles_133[8](https://github.com/SalesLoft/looker-sales-analytics/runs/6097631259?check_suite_focus=true#step:6:8)7552ea' on project 'app-sales-config'
Hard resetting branch 'tmp_spectacles_13387552ea' on project 'app-sales-config' to ref 'ISE-3270-Add-SFDC-Views-to-Commercial-Metrics-Explores'
Completed SQL validation in 3 seconds.


Couldn't update Git branch. Unable to update branch 'tmp_spectacles_13387552ea' in project 'app-sales-config' using ref 'ISE-3270-Add-SFDC-Views-to-Commercial-Metrics-Explores'. Please try again.

Run in verbose mode (-v) or check your log file to see the full response from the Looker API. For support, please create an issue at https://github.com/spectacles-ci/spectacles/issues

Spectacles received a 422 response code from the Looker API with the following details: {
  "message": "Unable to resolve ref 'ISE-3270-Add-SFDC-Views-to-Commercial-Metrics-Explores'",
  "documentation_url": "http://docs.looker.com/"
}

Error: Process completed with exit code [10](https://github.com/SalesLoft/looker-sales-analytics/runs/6097631259?check_suite_focus=true#step:6:10)1.

[DylanBaker]

Hey @sdblanchard. Thanks for creating the issue.

I can't tell if this has to do with pasting into GitHub or something else, but I noticed some weird text in the logs, i.e.:

The active branch is 'tmp_spectacles_e3ed7b06[5](https://github.com/SalesLoft/looker-sales-analytics/runs/6097631259?check_suite_focus=true#step:6:5)3'

I'm not sure what the second half of that is doing in there. As a first step, could you just confirm that that isn't in the logs themselves? Thanks.

As a second sanity check, can you (a) try re-running this to see if it occurs again, and (b) if it does, confirm you can check out the branch ISE-3270-Add-SFDC-Views-to-Commercial-Metrics-Explores in Looker?

[msandt3]

@DylanBaker - I'm able to reproduce this error intermittently - I think part of the frustration is the behavior is unpredictable so it becomes a battle of making minor changes to the PR structure multiple times in hopes that it will pass.

Approach:

I spun off a branch on a smaller project we have running spectacles as a CI step in Github actions. I'm able to check out the branch I'm working against in Looker so we're all clear there. I made a few LookML changes that wouldn't break anything (removing timeframes from a time dimension_group is what I chose).

I can attach the full logs for multiple runs if you'd like to see them. One thing I did notice (and where I think its tripping up) is this:

Cleaning up Git state in 'salesloft_hub'
2022-05-25T19:34:43.9364650Z .. Restoring project 'salesloft_hub' to branch 'master' and deleting temporary branch
2022-05-25T19:34:43.9365130Z      'tmp_spectacles_0dea0b3e63'
2022-05-25T19:34:43.9365997Z .. Setting project 'salesloft_hub' branch to 'tmp_spectacles_2f60feda5c'
2022-05-25T19:34:44.5872419Z Completed SQL validation in 22 seconds.
2022-05-25T19:34:44.5872656Z 
2022-05-25T19:34:44.5873588Z 
2022-05-25T19:34:44.5874699Z Couldn't checkout Git branch. Unable to checkout Git branch 'tmp_spectacles_2f60feda5c'. If you have uncommitted changes on the current branch, please commit or revert them, then try again.
2022-05-25T19:34:44.5875341Z 
2022-05-25T19:34:44.5875913Z [2mRun in verbose mode (-v) or check your log file to see the full response from the Looker API. For support, please create an issue at https://github.com/spectacles-ci/spectacles/issues[0m
2022-05-25T19:34:44.5876366Z 
2022-05-25T19:34:44.5879373Z .. Spectacles received a 422 response code from the Looker API with the following
2022-05-25T19:34:44.5879891Z      details: {
2022-05-25T19:34:44.5880897Z   "message": "Missing branch 'tmp_spectacles_2f60feda5c'",
2022-05-25T19:34:44.5881336Z      "documentation_url": "http://docs.looker.com/"
2022-05-25T19:34:44.5881634Z }
2022-05-25T19:34:44.9954275Z ##[error]Process completed with exit code 101.

I can't find any reference to the tmp_spectacles_2f60feda5c branch in Looker. Since we're interacting with git branching structure via the looker API - I wonder if there's something preventing Looker's visibility of the temp branch we created to store the changes.

Additionally - locally I've run git branch r | grep tmp_spectacles_2f60feda5c with no luck.

This may be GitHub actions related and how the repository is initialized in the tests - e.g. the tmp branch that gets created doesn't actually exist on the remote that github and looker are reading from and is only persisted for the duration of the test in isolation.

2022-05-25T19:25:16.3521372Z [command]/usr/bin/git init /home/runner/work/salesloft_hub/salesloft_hub
2022-05-25T19:25:16.3599523Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-05-25T19:25:16.3600404Z hint: is subject to change. To configure the initial branch name to use in all
2022-05-25T19:25:16.3600925Z hint: of your new repositories, which will suppress this warning, call:
2022-05-25T19:25:16.3601316Z hint: 
2022-05-25T19:25:16.3601839Z hint: 	git config --global init.defaultBranch <name>
2022-05-25T19:25:16.3602230Z hint: 
2022-05-25T19:25:16.3603067Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-05-25T19:25:16.3603660Z hint: 'development'. The just-created branch can be renamed via this command:
2022-05-25T19:25:16.3604080Z hint: 
2022-05-25T19:25:16.3604397Z hint: 	git branch -m <name>
2022-05-25T19:25:16.3616096Z Initialized empty Git repository in /home/runner/work/salesloft_hub/salesloft_hub/.git/
2022-05-25T19:25:16.3629074Z [command]/usr/bin/git remote add origin https://github.com/SalesLoft/salesloft_hub
2022-05-25T19:25:16.3723480Z ##[endgroup]
2022-05-25T19:25:16.3724499Z ##[group]Disabling automatic garbage collection
2022-05-25T19:25:16.3732477Z [command]/usr/bin/git config --local gc.auto 0
2022-05-25T19:25:16.3770151Z ##[endgroup]
2022-05-25T19:25:16.3770936Z ##[group]Setting up auth
2022-05-25T19:25:16.3783917Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-25T19:25:16.3824535Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-25T19:25:16.4214924Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-25T19:25:16.4245865Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-25T19:25:16.4510482Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-25T19:25:16.4552999Z ##[endgroup]
2022-05-25T19:25:16.4554339Z ##[group]Fetching the repository
2022-05-25T19:25:16.4563962Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +a4dae7055a871e85358fa909ef084a4781fd18a7:refs/remotes/origin/tmp_spectacles_fbd0dff6f3

I'm near the limit of my git knowledge but I hope that helps get the conversation frame up in the right way.

[msandt3]

I did run this locally with as similar of a configuration as I could (minus the github actions setup & checkout boilerplate) and didn't encounter the same issue fwiw

[DylanBaker]

@msandt3, is it possible you've got multiple jobs happening simultaneously with the same credential?

[msandt3]

@DylanBaker its certainly possible Our architecture looks like the following

• We split up our LookML into several projects / repos for re-usability
• Each project/repo runs spectacles for every commit, the spectacles run is isolated to only views and tests contained within that project/repo. E.G. spectacles runs of project A do not invoke tests for project B
• All of the spectacles traffic is run through a single user (just a machine user for us, no one actually has access to this user)

TL;DR - there is a possibility of multiple concurrent runs happening. However, for the example I put together there were no concurrent runs on the project, but could have been in the global environment.

[DylanBaker]

Thanks. Makes sense.

When Spectacles finishes a run, we try to reset to the branch we found you on. Based on the logs you sent, my best hypothesis is that that branch no longer exists by the time you get to the end of the run. The fact it's trying to reset to a branch that was created by Spectacles would lend some credence to that.

There's a change in an upcoming PR that would have us default bank to a random dev branch if we fail to reset on the branch you were on. This should resolve the issue if my hypothesis is correct.

You can either way for that to be merged in and deployed (probably next week) or pull from that specific branch now if you want.

[daniel-anya]

Hi 👋 @DylanBaker @joshtemple I'm currently running into this issue. I see that the proposed change from here later got reverted here. I'm wondering if y'all have some more insight on why that fix didn't end up being a permanent one?

[joshtemple]

Hey @daniel-anya, that commit is over a year old and I'm afraid I'm struggling to remember why it was reverted. I seem to remember that we reverted it because it wasn't actually fixing the problem we needed to fix at the time, but I'm drawing a blank.

@DylanBaker does this jog your memory at all?