Node package

[Goldziher]

Search before asking

• [X] I searched the issues and found no similar issues.

Description

I would like to request publishing this package into the NPM registry with a JS binding - for installation.

Use case

JS/TS is a widely used language and ecosystem. There is though a need of sqlfluff there, and it could be highly useful. For example, I am using sqlfluff via pre-commit in a TS based project relying on supabase.

I am though coming from python, so using python tooling is natural to me. Most JS/TS developers are not used to pre-commit, and do not use pyproject.toml or other config files.

Note, I am willing to help as I can.

Dialect

all

Are you willing to work on and submit a PR to address the issue?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[alanmcruickshank]

I'm open to doing this, but I have no idea how to do it. In particular how an NPM package can effectively specify a python dependency. If you're able to suggest a viable route forward for this - I'm very interested however.

[Goldziher]

lemme look into this. im sure we can do this, I suspect we need to have wheels and then install them in a wrapped binary

[Goldziher]

I created the basics for this: https://github.com/Goldziher/sqlfluff-js

I created the repository for you guys. If you are pleased, I want to transfer ownership of the repo to you and you take it over.

Look at the readme in the repository.

[Goldziher]

@alanmcruickshank

[zineanteoh]

+1 having sqlfluff as an npm package would be extremely valuable. struggling to find alternative tools as mature as sqlfluff -- but it currently is limited to the python ecosystem.

[alanmcruickshank]

Thanks @Goldziher! I've taken a look, and it looks like a really good start. Ideally I'd love to bring it into the sqlfluff org so that it's easier for people to find, but I'd love to get confidence that it does what it needs to first.

First, I think it's worth clarifying something from @zineanteoh perhaps: when you say you want sqlfluff as an npm package, is that to use it on the command line, or as something you can interact with as an API? The repository above is an npm wrapper around the python cli, to expose it as a node cli - but it still relies on being able to install python packages and interact with a CLI, rather than a native javascript api. Does this solve your use case?

My main question for @Goldziher is whether you'd be able to add a small CI suite to your repo? I'm not familiar with current best practices on test frameworks for javascript, but something that tests the main ways you'd want people to interact with the package. In particular you noted that many JS/TS developers are not familiar with configuring python dependencies, and so testing that this package would work using the methods that you do think they would use to specify dependencies.

Lastly one thing I'm not sure about is how we manage versions:

• On one side, I'm tempted to bind the version numbers of the sqlfluff core package and of sqlfluff-js, i.e. that version 3.1.1 of sqlfluff-js installs version 3.1.1 of sqlfluff. This is probably simpler for users, but does involve a new deploy of the js package every time there's a new sqlfluff release.
• The other option would be to allow users of sqlfluff-js to specify the version they want in configuring the package. That allows a little more seperation, but it forces an extra configuration step.

Do either of you have views on which you think might work best?

[Goldziher]

I made a small video of how this works now with the repo I created

[Goldziher]

https://github.com/user-attachments/assets/914b0130-834d-4f5e-8399-19a47acde6a8

[Goldziher]

Is there an e2e test suite for the sqlfluff cli? If you have a test suite that runs the cli via a shell, we can reuse it. Otherwise I don't think this is necessary - the library just proxies all the commands to sqlfluff, its really a pretty thin wrapper.

As for the python and releasing - the library doesn't actually rely on any specific version of sqlfluff. It simply checks if the sqlfluff command is installed, and if it isn't it installs it with

if (await tryExec("sqlfluff --version")) {
		console.log("sqlfluff is already installed.");
		return null;
	}
	console.log("Installing sqlfluff...");

	let command = "pip install sqlfluff";

	if (platform() === "win32") {
		// Check if py launcher is available on Windows
		try {
			await execAsync("py --version");
			command = "py -m pip install sqlfluff";
		} catch {
			command = "pip install sqlfluff";
		}
	}

	return await execAsync(command);

This can be further refined as you like.

You can very easily run update operations as well or add a self-update command to sqlfluff, and then it will be available there.

For releasing the package - the steps are pretty simple and I can automate them for you in CI. You will need to:

1. create a user in npm
2. create a token in npm
3. add the token to github

Then to publish

1. update the version in the package.json file and push to main
2. draft a new release with a new version tag in the releases view
3. publish the release

You can also do this with the cli:

1. npm login
2. npm publish

Let me know if you want to take ownership.

[Goldziher]

Also- you should consider supporting sqlfluff config via a json file. it will be best for the js community

[Goldziher]

final thing - you can add me as a co-maintainer of this particular repo, and I will help maintaining it. I will need to have membership of the organization, at least temporarily to transfer the repo apparently:

[DanteOz]

@Goldziher You could also use uv in your package.json scripts to run sqlfluff.

{
  "scripts": {
    "sql:fmt": "uvx sqlfluff@latest format",
    "sql:lint": "uvx sqlfluff@latest check",
    "sql:fix": "uvx sqlfluff@latest fix"
  }
}

https://docs.astral.sh/uv/guides/tools/#running-tools

[Goldziher]

@Goldziher You could also use uv in your package.json scripts to run sqlfluff.

{
  "scripts": {
    "sql:fmt": "uvx sqlfluff@latest format",
    "sql:lint": "uvx sqlfluff@latest check",
    "sql:fix": "uvx sqlfluff@latest fix"
  }
}

https://docs.astral.sh/uv/guides/tools/#running-tools

UV needs to be installed on the system for this

[alanmcruickshank]

@Goldziher - I haven't forgotten about this PR, I'd still love some input from someone planning to use this package on whether this fulfils their needs. Assuming it does, then I'm very happy to bring it into the project and add you as a maintainer 👍 .

My concern with the current approach is that:

1. It only exposes a CLI interface.
2. Under the hood it relies on being able to install python packages in the environment.

If someone is able to install python packages, and has access to the CLI - then I would assume they'd use the existing SQLFluff CLI directly. Which means I'm not sure what use this Node package adds (beyond managing the interaction with pip, which removes the ability to manage version information etc).

However, I'm not an active JS developer, so I'm not sure if I'm missing the point here. I'd love to get input from a wider range of JS developers who might use the package to resolve that. Do you have any you might be able to get some input from?

[Goldziher]

@Goldziher - I haven't forgotten about this PR, I'd still love some input from someone planning to use this package on whether this fulfils their needs. Assuming it does, then I'm very happy to bring it into the project and add you as a maintainer 👍 .

My concern with the current approach is that:

1. It only exposes a CLI interface.
2. Under the hood it relies on being able to install python packages in the environment.

If someone is able to install python packages, and has access to the CLI - then I would assume they'd use the existing SQLFluff CLI directly. Which means I'm not sure what use this Node package adds (beyond managing the interaction with pip, which removes the ability to manage version information etc).

However, I'm not an active JS developer, so I'm not sure if I'm missing the point here. I'd love to get input from a wider range of JS developers who might use the package to resolve that. Do you have any you might be able to get some input from?

I used this myself now in npm scripts.

Anyhow the only added value it adds is that it doesn't require developers to know how to install python packages, since it handles that part for them.

It's not unusual for packages to depend on external os level packages, e.g. a native of driver etc.

At least here python is widely available and easy to install on practically all environments.

Anyhow, the repo is available. I can publish the package if it's preferable. Or I can transfer it.

Anyone can install the repo directly using npm or pnpm.

Also, it's worthwhile interesting the supabase folks in this package.

[jasonaowen]

I'd still love some input from someone planning to use this package on whether this fulfils their needs.

I'd be interested in using SQLFluff on a Node project, but I share this concern:

2. Under the hood it relies on being able to install python packages in the environment.

I think it's important that the npm package does not change anything outside the project. My goal is for teammates who are less familiar with Python to be able to just run npm install && npm run lint:sql or whatever, which would ensure that the Python package and all its dependencies are installed and run SQLFluff on the appropriate directory with the appropriate config file.

I suspect we need to have wheels and then install them in a wrapped binary

Including all the dependencies in the npm package sounds great, however that would work; that'd avoid the need for a postinstall script.

I think it's safe to require some version of Python 3 installed on the system as a prerequisite - managing Python installations sounds well outside of the scope of this.

Based on the blog post Bundling Python files into a stand-alone executable, another option might be to just package all the source and dependencies into the npm package, and run it with python3 sqlfluff/ or so. I'm not super clear on the best way to vendor the dependencies so that Python can find them automatically, but it seems like it ought to be possible.

Responding to other questions in this thread:

1. It only exposes a CLI interface.

That's fine with me; all I want is to be able to run SQLFluff as an application, not use it as a library.

Lastly one thing I'm not sure about is how we manage versions

Either of the proposed alternatives sound good to me, with a preference for binding the npm package version to the SQLFluff version, so that @dependabot or @renovatebot can keep track of updates.

Long term, I think what would be most useful is for the npm package to be generated as part of a SQLFluff release, rather than as a separate build in a downstream repo (even if it's a first-party repo). But starting as a separate repo seems like a good start, and if it works out then it can be merged into the primary repo later.

consider supporting sqlfluff config via a json file

This would be helpful, but I'd suggest that should be a separate issue, and again not block this on that.