Ability to redeploy only changed files

[decibel]

Use case 1: While developing patches, changes are made to deploy, revert or verify files, possibly including "old" ones. The rebase command should be able to determine what change to use for the --onto option so that all new changes get deployed.

Use case 2: Same as above, but developer needs to do additional stuff besides a simple rebase. There should be a way to get the file that needs to be given to the --onto target so this can be captured as a variable (ie: oldest = $(sqitch log --oldest), though maybe log isn't the right command for this).

Discussion on mailing list: https://groups.google.com/forum/#!topic/sqitch-users/8BnceXktxV0

[eric-brechemier]

I have a similar use case for which I am near to a working solution, based on git. I compare two versions (two git tags) and revert to the last common change before deploying an updated plan.

Our complete use case is based on two plans:

• the plan written in stone (tables, etc.), which is deployed first and only in a forward way (no revert ever)
• the plan written in sand (functions, views, indexes, etc.), which can be reverted in full and redeployed, but it is not very efficient and rarely necessary to revert it completely, thus a differential update is preferred

Given the version currently deployed and the target version

echo "Server: Current Version: $currentVersion" 
echo "Server: Target Version: $targetVersion"

We first look for the last line unchanged in the plan:

echo "Server: look for last common line in plans" \ 
     "of $currentVersion and $targetVersion" 
lastCommonLineInPlan="$( 
  git \ 
    --git-dir="/home/git/$repository.git" \ 
    diff \ 
      -U1 \ 
      --relative="$sandPlanFolder/sqitch.plan" \ 
      "$currentVersion".."$targetVersion" \ 
  | tail -n +6 \
  | head -n 1 
)"
case "$lastCommonLineInPlan" in 
  ('') 
    echo 'No difference found in sqitch.plan.' 
  ;; 
  (' '[!\ ]*) 
    lastCommonChange="$( echo "$lastCommonLineInPlan" | cut -d' ' -f2 )"
    echo "Revert to last common change: $lastCommonChange"
    sudo -u postgres sqitch revert -y "$lastCommonChange"
  ;;
  (*)
    echo 'The whole plan is modified. Revert all changes.'
    sudo -u postgres sqitch revert -y
    exit 0
esac

Then we look for deployment scripts updated or deleted and revert these changes as well (on second thought, only updated changes need to be checked here, deleted files should be taken care of by the previous step):

echo \
  "Server: Revert changes with deploy scripts" \
  "updated or deleted between $currentVersion and $targetVersion"
git \
  --git-dir="/home/git/$repository.git" \
  diff \
    --diff-filter=MDR \
    --name-only \
    --relative="$sandPlanFolder/deploy" \
    "$currentVersion".."$targetVersion" \
| while read -r changedFile
do
  change="${changedFile%\.sql}"
  sudo -u postgres \
    sqitch show --exists 'change' "$change" &&
  echo "Server: revert change $change" &&
  sudo -u postgres \
    sqitch revert --to "$change^1" -y
done

The above code is still under development and not considered free of bugs. Caveat emptor.

[theory]

Couldn't you could use the checkout command for that, @eric-brechemier?

[eric-brechemier]

@theory thanks for the suggestion! I was not aware of this command.

It is very close to what I need, but I do not have a checked out repository on the production server, only a bare repository from which I export each version into a separate folder. This setup allows to deploy changes in an atomic way by changing the target of a symbolic link to the directory for the current version.

Is there an access to the underlying diff between two versions somehow, without the git checkout parts?

[eric-brechemier]

I was not aware of this command

More precisely, I had no idea from its short description Revert, checkout another VCS branch, and re-deploy changes that it was doing a differential update and not a full revert and deploy.

[theory]

There isn't, but there's no reason why it couldn't be abstracted out of the source.

[eric-brechemier]

It seems that this code only lists changes in the plan, and does not detect changes in deployment files, correct?

[theory]

Right.

[eric-brechemier]

Also, it does not detect a mere reordering of the changes, if I guess correctly.

[theory]

Right. See #200 for some background on addressing that particular challenge.

[eric-brechemier]

(this is not just a hypothetical use case: we had to move the definition of indexes to the start of a plan to solve an issue recently)

[eric-brechemier]

We are now using the following script to identify the minimum revert point for a sqitch plan before deploying a modified plan, both stored at different tags (or commits) in git:

#!/bin/sh
# Revert sqitch plan to prepare differential update
#
# The plan in current version is rolled back up to the last change
# not modified in the target version, for differential update.
# Both the plan and the deployment scripts are considered, using git
# to find the history of changes between the two versions.
#
# Parameters:
#   $1 - relative path to the sqitch plan for current version,
#        relative to the root of the working directory
#   $2 - git commit or tag for the current version
#   $3 - git commit or tag for the version to deploy
#   $4 - path to the root of the working directory for current version
#   $5 - optional, command to run git; defaults to just 'git'.
#        A more useful example, which configures the path to the git repository
#        and runs the command as a different user called 'gituser' would be:
#        'sudo -u gituser git --git-dir=/path/to/git-repository.git'.
#   $6 - optional, command to run sqitch; defaults to just 'sqitch'.
#        Likewise, this parameter can be used to run the command as a
#        different user or with extra configuration parameters, e.g.:
#        'sudo -u postgres sqitch' to run sqitch as 'postgres' user.
#
sqitchPlan="$1"
fromVersion="$2"
toVersion="$3"
workingDir="$4"
gitCommand="${5:-git}"
sqitchCommand="${6:-sqitch}"

die()
{
  die_code="$?"
  die_message="$1"
  echo "$die_message" 1>&2
  exit "$die_code"
}

test -n "$sqitchPlan" \
  -a -n "$fromVersion" \
  -a -n "$toVersion" \
  -a -n "$workingDir" \
  -a -n "$gitCommand" \
  -a -n "$sqitchCommand" \
|| die "Usage: $0 sqitchPlan fromVersion toVersion workingDir [gitCommand] [sqitchCommand]"

test -d "$workingDir" \
  || die "Working directory not found: $workingDir"

test -f "$workingDir/$sqitchPlan" \
  || die "Sqitch plan not found: $workingDir/$sqitchPlan"

$gitCommand tag >/dev/null \
  || die "Could not run git command: $gitCommand"

$gitCommand show --oneline --name-only "$fromVersion" >/dev/null \
  || die "Version not found: $fromVersion"

$gitCommand show --oneline --name-only "$toVersion" >/dev/null \
  || die "Version not found: $toVersion"

sqitchProject="$(
  grep -m 1 '^%project=' "$workingDir/$sqitchPlan" \
  | cut -d'=' -f2
)"
test -n "$sqitchProject" || die "Not a sqitch plan: $workingDir/$sqitchPlan"

echo "Revert project '$sqitchProject' for migration $fromVersion->$toVersion"

sqitchProjectDirectory="$( dirname "$workingDir/$sqitchPlan" )"
sqitchPlanFile="$( basename "$workingDir/$sqitchPlan" )"

cd "$sqitchProjectDirectory" \
  || die "Failed to change to directory '$sqitchProjectDirectory'"

$sqitchCommand tag >/dev/null \
  || die "Could not run sqitch command: $sqitchCommand"

# list of modified deployment scripts, separated and surrounded by ';'
modifiedDeployScripts=";$(
  $gitCommand diff \
    --diff-filter=M \
    --name-only \
    --relative="$( dirname "$sqitchPlan" )/deploy" \
    "$fromVersion".."$toVersion" \
  | tr "\n" ';'
)"

# Check whether the deployment script for given change has been modified
# between the current version and the target version
#
# Parameter:
#   $1 - name of the sqitch change
#
# Exit Code:
#   0 (true) when a difference was found
#   1 (false) when no difference was found
hasDeployScriptChanged()
{
  case "$modifiedDeployScripts" in
    *";$1.sql;"*) return 0;;
    *) return 1 ;;
  esac
}

currentChange=''
lastCommonChange=''
$gitCommand show "$toVersion:$sqitchPlan" |
while read -r oldLine <&4 && read -r newLine
do
  case "$oldLine" in
    ''|'%'*) # no change on this line
      currentChange=''
    ;;
    *) # change found
      currentChange="${oldLine%% *}"
    ;;
  esac

  # check first whether plan lines differ
  if test "$oldLine" != "$newLine" ||
    {
      # else, only when outside of header,
      test -n "$currentChange" &&
      # check whether deployment scripts differ
      hasDeployScriptChanged "$currentChange"
    }
  then
    if test -z "$lastCommonChange"
    then
      echo 'No common change. Revert the whole plan.'
      $sqitchCommand revert -y
    else
      echo "Revert to last common change: $lastCommonChange"
      $sqitchCommand revert -y "$lastCommonChange"
    fi
    # stop at first change
    exit 1
  fi
  lastCommonChange="$currentChange"
done 4<sqitch.plan

if test $? -eq 0 # no error and no change found
then
  echo 'No change found.'
fi