would it make sense to allow `js {}` variables in `config {}`

[andres-lowrie]

So for example say I have a file structure like this

definitions/
   system1/
      sharedLogicStuff.js
      table_a.sqlx
   system2/
      ...

and let's say that sharedLogicStuff.js looks like this

module.exports = { 
  colsToAppearInManyPlaces: {
      col_1: 'some description that I want to define once',
  }
}

what I'm hinting is being able to do something like this in the .sqlx files

js {
  const {colsToAppearInManyPlaces} = require('definitions/system1/sharedLogicStuff')
}

config {
  columns: colsToAppearInManyPlaces,
  // or for the use case I have right now 
  columns: Object.assign({}, colsToAppearInManyPlaces, {colX: 'just for this output'}),
}

.... 

currently that doesn't work. I get back colsToAppearInManyPlace is not defined which I'm assuming is by design

I'm wondering if being able to reference js {} stuff in the config {} makes sense from a big picture dataform design idea in order to add it as a feature.

In the use case I laid out, the goal is to keep the shared stuff close to the sqlx files to make it easier to find/edit the shared logic when working on a table. Imagine the .js file having a bunch of other variables and snippets that's used in the actual "sqlx body" as well

Currently what I have to do to in order to share declarations of variables is split out the declarations across includes/ and definitions depending on where I want to use them in the definitions (sqlx files)

includes/
   system1/
       sharedLogicStuffThatsUsableInConfig.js
   ...
definitions/
   system1/
      sharedLogicStuffThatsUsableInSqlx.js
      table_a.sqlx
   system2/
      ...

it's a bit cumbersome and a bit complex (cognitive load wise) to have to split out variables/logic/etc into 2 locations in order to be able to reference them in different places within the .sqlx file; would be nice if both the "config {}" block and the "sqlx block" had access to the js {} as well as the includes

[qchuchu]

Thanks @andres-lowrie for your issue, I'm also very interested by this problem.

I've started to look at the compiler.ts file in order to understand where js variables were resolved.

I think I would make a lot of sense to resolve constant in the js block before the config one, but I can't really understand how hard it could be.

We also had the discussion here : https://www.googlecloudcommunity.com/gc/Data-Analytics/Ho-to-reference-a-variable-from-js-script-within-sqlx-file/m-p/730521#M5442

My usage :

• Most of the times columns definitions that are being reused are in the same folder. And here, it seems that we can't use the Principle of Proximity.
• I want to sync column definitions with rowConditions.

@Ekrekr Maybe you can point out the steps in the compiler where we could look at ? :)

[BenBirt]

For scoping reasons, js {} blocks are inserted inside Contextables upon SQLX -> JS transpilation. It may be possible to work around that with some thought, but it's not immediately obvious that there is a way to make it work.

That said, there is nothing stopping you doing something like:

config {
  columns: require('definitions/system1/sharedLogicStuff').colsToAppearInManyPlaces,
  // or for the use case I have right now 
  columns: Object.assign({}, require('definitions/system1/sharedLogicStuff').colsToAppearInManyPlaces, {colX: 'just for this output'}),
}

[andres-lowrie]

ahh.. good to know. That's much better than splitting across, sweet