The question mark in comment results in extra spaces on the second pass.

[nick863]

Copied from the thread Repro steps:

1. Create the minimal jupyter notebook, which will contain one cell with the function ands a comment, containing question mark;

def example_fn(param):
    """
    param: str    what to print?
    """
    print(param)

2. Close the file and run

black-nb  Repro.ipynb

The error will be shown: error: cannot format /path/to/nb/Repro.ipynb: INTERNAL ERROR: Black produced code that is not equivalent to the source on pass 1. Please report a bug on https://github.com/psf/black/issues. This diff might be helpful: ${TEMP}/blk_y4fjze4u.log

The diff will show that at the second pass extra whitespaces were added:

--- src
+++ dst
@@ -23,11 +23,11 @@
       body=
         Expr(
           value=
             Constant(
               value=
-                '###MAGIC###    param: str    what to print?',  # str
+                '###MAGIC###        param: str    what to print?',  # str
             )  # /Constant
         )  # /Expr
         Expr(
           value=
             Call(

If the question mark will be removed, the notebook will pass

Example notebook:

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5c38d2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def example_fn(param):\n",
    "    \"\"\"\n",
    "    param: str    what to print?\n",
    "    \"\"\"\n",
    "    print(param)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}

• contains_magic makes no consideration for whether or not it's looking at a string. This explains why the issue doesn't happen if you remove the question mark. I'm not familiar with jupyter, so I'm not sure if this is intended behavior or not.

• As a consequence of 1, black-nb will add a magic comment block to the beginning of the line after 0 spaces, which is not appropriate for a docstring. Black will correctly try to indent it, adding 4 spaces. When the magic is removed, those 4 spaces remain, which accounts for the diff you are seeing.

[liamcoatman]

I think this is a bit tricky to fix. Our contains_magic function works line by line, but in order to know whether the current line is inside a multi-line comment we would need to do something fancier with the AST, which is what black does.

As HassanAbouelela notes in the black issue, I would recommend using black to format notebooks since it now supports it natively.