psf
Black support for Databricks notebook formatting - Ignore lines starting with ! or %
I have a .py file which runs as notebook
# Databricks notebook source
!pip install -U dspy python-dotenv
!pip install torchmetrics
dbutils.library.restartPython()
# COMMAND ----------
from tqdm.notebook import tqdm
tqdm.pandas()
import logging
# COMMAND ----------
!pip install matplotlib seaborn
The above is python file which runs as notebook where each cell is defined by # COMMAND in databricks. When applying black formatting I want to skip the pip install related lines. I tried using noqa but still above is causing warnings. How do I skip checking lines which have ! or % as starting character?
My pre-commit file config is
repos:
- repo: https://github.com/psf/black
rev: 23.3.0
hooks:
- id: black
name: black
description: "Black: The uncompromising Python code formatter"
entry: black
language: python
minimum_pre_commit_version: 2.9.2
require_serial: true
types_or: [ python, pyi ]
- id: black-jupyter
name: black-jupyter
description:
"Black: The uncompromising Python code formatter (with Jupyter Notebook support)"
entry: black
language: python
minimum_pre_commit_version: 2.9.2
require_serial: true
types_or: [ python, pyi, jupyter ]
additional_dependencies: [ ".[jupyter]" ]
On running
pre-commit run --all-files
Warnings which I get is
Cannot parse: 7:0: !pip install -U dspy python-dotenv
Use --skip-string-normalization and # noqa Ensure you add # noqa to lines you want to skip and pass the --skip-string-normalization option to Black to prevent reformatting lines unnecessarily. For proper handling of non-Python lines like ! and %, use tools like black_nbconvert or preprocess the notebook file to filter such lines.
I feel like this would be a somewhat unnecessary change - see the similar issue #3557 and https://github.com/psf/black/issues/3668
(This is also much fiddlier to support than those ones, since this is not valid Python syntax but those are)
Write a script to preprocess the .py file, comment out ! and % lines, run Black, and then uncomment the lines. Example:
Preprocess
with open("notebook.py", "r") as f: lines = f.readlines()
with open("notebook.py", "w") as f: for line in lines: if line.startswith(("!", "%")): f.write(f"# {line}") else: f.write(line)
Run Black
...
Postprocess
with open("notebook.py", "r") as f: lines = f.readlines()
with open("notebook.py", "w") as f: for line in lines: if line.startswith("# !") or line.startswith("# %"): f.write(line[2:]) else: f.write(line)
We're probably not going to support this. It's difficult to support things that do not parse as Python. Building a little hacky wrapper script around Black might be your best option.