data-api-builder icon indicating copy to clipboard operation
data-api-builder copied to clipboard

Published 20 hours ago • verified publisher icon Azure

Add tests to check staleness of checked-in reference config files

Open severussundar opened this issue 1 year ago • 4 comments

There are a few reference config files checked-in in the Service.Tests directory. Whenever there is a change in the way CLI generates a config file, these reference config file needs to be updated to reflect the latest changes.

At the moment, there is no validation to ensure that these reference files are as per the latest format. So, these config files tends to go out of date.

Hence, add tests to ensure that these reference config files stay up to date.

severussundar avatar Oct 02 '23 16:10 severussundar

Re-assigning to 1.1 in favor of finishing Nested Inserts work critical to .12

seantleonard avatar Mar 12 '24 19:03 seantleonard

reassigning to 1.2 since non-critical for 1.1

seantleonard avatar Apr 23 '24 21:04 seantleonard

Any simpler mechanism to keep the test configs updated besides a test? Perhaps pre-commit hook that runs the config automation script and updates the test configs. (this would also highlight how slow the CLI is with our config scripts... takes minutes to run).

seantleonard avatar May 16 '24 23:05 seantleonard

Currently, the checked-in reference config files and config file generated through CLI can go out of sync. The most common cause of this is that the reference config files are not updated when the command file is updated. The goal of this validation to catch when the generated config file (which is used in integration tests) and the reference config files go out of sync.

Generating a second config file in the integration testing pipeline is not a viable approach as it adds ~2-3 minutes to the pipeline.

Alternative approach I was experimenting:

  1. Reading the commands from the text file (such as mssql-commands.txt), splitting them as an array of strings and executing them using Program.Execute() (The same approach is being done in EndToEndTests - link ) seems viable. Executing commands (and thereby generating the config file) this way, does not add significant delays to the test.
  2. One challenge to note: When the generated json file (created using the approach mentioned in point 1) is read programatically, there exists lots of escape characters. So, direct string comparison with the checked in reference config file is not possible. Some logic to remove the escape characters needs to written

severussundar avatar Jun 03 '24 11:06 severussundar