hera
hera copied to clipboard
Published
20 hours ago •
argoproj-labs
argoproj-labs
f-strings break the workflow generator
Hi, I'm trying to use an f-string in my script and am running into issues. I think the presence of curly brackets confuses the workflow compiler somehow. I'm using hera 5.1.3.
For example, I can run the basic example script just fine. But if I make a small change to include an f-string:
from hera.workflows import Steps, Workflow, script
@script()
def echo(message: str):
print(f"message: {message}")
with Workflow(
generate_name="single-script-",
entrypoint="steps",
) as w:
with Steps(name="steps"):
echo(arguments={"message": "A"})
w.create()
I get an error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[5], line 18
15 with Steps(name="steps"):
16 echo(arguments={"message": "A"})
---> 18 wf = {"Workflow": w.to_dict()}
20 # submit the workflow to Argo
21 argo.create_workflow(wf)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/workflow.py:319](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/workflow.py:319), in Workflow.to_dict(self)
317 def to_dict(self) -> Any:
318 """Builds the Workflow as an Argo schema Workflow object and returns it as a dictionary."""
--> 319 return self.build().dict(exclude_none=True, by_alias=True)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/workflow.py:212](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/workflow.py:212), in Workflow.build(self)
209 template = template._dispatch_hooks()
211 if isinstance(template, Templatable):
--> 212 templates.append(template._build_template())
213 elif isinstance(template, get_args(TTemplate)):
214 templates.append(template)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:172](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:172), in Script._build_template(self)
144 def _build_template(self) -> _ModelTemplate:
145 assert isinstance(self.constructor, ScriptConstructor)
146 return self.constructor.transform_template_post_build(
147 self,
148 _ModelTemplate(
149 active_deadline_seconds=self.active_deadline_seconds,
150 affinity=self.affinity,
151 archive_location=self.archive_location,
152 automount_service_account_token=self.automount_service_account_token,
153 daemon=self.daemon,
154 executor=self.executor,
155 fail_fast=self.fail_fast,
156 host_aliases=self.host_aliases,
157 init_containers=self.init_containers,
158 inputs=self._build_inputs(),
159 memoize=self.memoize,
160 metadata=self._build_metadata(),
161 metrics=self.metrics,
162 name=self.name,
163 node_selector=self.node_selector,
164 outputs=self._build_outputs(),
165 parallelism=self.parallelism,
166 plugin=self.plugin,
167 pod_spec_patch=self.pod_spec_patch,
168 priority=self.priority,
169 priority_class_name=self.priority_class_name,
170 retry_strategy=self.retry_strategy,
171 scheduler_name=self.scheduler_name,
--> 172 script=self._build_script(),
173 security_context=self.pod_security_context,
174 service_account_name=self.service_account_name,
175 sidecars=self.sidecars,
176 synchronization=self.synchronization,
177 timeout=self.timeout,
178 tolerations=self.tolerations,
179 volumes=self._build_volumes(),
180 ),
181 )
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:201](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:201), in Script._build_script(self)
183 def _build_script(self) -> _ModelScriptTemplate:
184 assert isinstance(self.constructor, ScriptConstructor)
185 return self.constructor.transform_script_template_post_build(
186 self,
187 _ModelScriptTemplate(
188 args=self.args,
189 command=self.command,
190 env=self._build_env(),
191 env_from=self._build_env_from(),
192 image=self.image,
193 image_pull_policy=self._build_image_pull_policy(),
194 lifecycle=self.lifecycle,
195 liveness_probe=self.liveness_probe,
196 name=self.container_name,
197 ports=self.ports,
198 readiness_probe=self.readiness_probe,
199 resources=self._build_resources(),
200 security_context=self.security_context,
--> 201 source=self.constructor.generate_source(self),
202 startup_probe=self.startup_probe,
203 stdin=self.stdin,
204 stdin_once=self.stdin_once,
205 termination_message_path=self.termination_message_path,
206 termination_message_policy=self.termination_message_policy,
207 tty=self.tty,
208 volume_devices=self.volume_devices,
209 volume_mounts=self._build_volume_mounts(),
210 working_dir=self.working_dir,
211 ),
212 )
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:336](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:336), in InlineScriptConstructor.generate_source(self, instance)
330 script += "\n"
332 # We use ast parse/unparse to get the source code of the function
333 # in order to have consistent looking functions and getting rid of any comments
334 # parsing issues.
335 # See https://github.com/argoproj-labs/hera/issues/572
--> 336 content = roundtrip(inspect.getsource(instance.source)).splitlines()
337 for i, line in enumerate(content):
338 if line.startswith("def") or line.startswith("async def"):
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:949](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:949), in roundtrip(source)
947 if hasattr(ast, "unparse"):
948 return ast.unparse(tree)
--> 949 return unparse(tree)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:942](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:942), in unparse(ast_obj)
940 def unparse(ast_obj):
941 unparser = _Unparser()
--> 942 return unparser(ast_obj)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:70](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:70), in _Unparser.__call__(self, node)
67 """Outputs a source code string that, if converted back to an ast
68 (using ast.parse) will generate an AST equivalent to *node*"""
69 self._source = []
---> 70 self.traverse(node)
71 return "".join(self._source)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183), in _Unparser.traverse(self, node)
181 self.traverse(item)
182 else:
--> 183 super().visit(node)
File [~/anaconda3/envs/quilt/lib/python3.8/ast.py:371](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/ast.py:371), in NodeVisitor.visit(self, node)
369 method = 'visit_' + node.__class__.__name__
370 visitor = getattr(self, method, self.generic_visit)
--> 371 return visitor(node)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:194](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:194), in _Unparser.visit_Module(self, node)
192 def visit_Module(self, node):
193 self._type_ignores = {ignore.lineno: f"ignore{ignore.tag}" for ignore in node.type_ignores}
--> 194 self._write_docstring_and_traverse_body(node)
195 self._type_ignores.clear()
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:190](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:190), in _Unparser._write_docstring_and_traverse_body(self, node)
188 self.traverse(node.body[1:])
189 else:
--> 190 self.traverse(node.body)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:181](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:181), in _Unparser.traverse(self, node)
179 if isinstance(node, list):
180 for item in node:
--> 181 self.traverse(item)
182 else:
183 super().visit(node)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183), in _Unparser.traverse(self, node)
181 self.traverse(item)
182 else:
--> 183 super().visit(node)
File [~/anaconda3/envs/quilt/lib/python3.8/ast.py:371](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/ast.py:371), in NodeVisitor.visit(self, node)
369 method = 'visit_' + node.__class__.__name__
370 visitor = getattr(self, method, self.generic_visit)
--> 371 return visitor(node)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:374](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:374), in _Unparser.visit_FunctionDef(self, node)
373 def visit_FunctionDef(self, node):
--> 374 self._function_helper(node, "def")
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:392](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:392), in _Unparser._function_helper(self, node, fill_suffix)
390 self.traverse(node.returns)
391 with self.block(extra=self.get_type_comment(node)):
--> 392 self._write_docstring_and_traverse_body(node)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:190](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:190), in _Unparser._write_docstring_and_traverse_body(self, node)
188 self.traverse(node.body[1:])
189 else:
--> 190 self.traverse(node.body)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:181](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:181), in _Unparser.traverse(self, node)
179 if isinstance(node, list):
180 for item in node:
--> 181 self.traverse(item)
182 else:
183 super().visit(node)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183), in _Unparser.traverse(self, node)
181 self.traverse(item)
182 else:
--> 183 super().visit(node)
File [~/anaconda3/envs/quilt/lib/python3.8/ast.py:371](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/ast.py:371), in NodeVisitor.visit(self, node)
369 method = 'visit_' + node.__class__.__name__
370 visitor = getattr(self, method, self.generic_visit)
--> 371 return visitor(node)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:233](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:233), in _Unparser.visit_Assign(self, node)
231 self.traverse(target)
232 self.write(" = ")
--> 233 self.traverse(node.value)
234 if type_comment := self.get_type_comment(node):
235 self.write(type_comment)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183), in _Unparser.traverse(self, node)
181 self.traverse(item)
182 else:
--> 183 super().visit(node)
File [~/anaconda3/envs/quilt/lib/python3.8/ast.py:371](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/ast.py:371), in NodeVisitor.visit(self, node)
369 method = 'visit_' + node.__class__.__name__
370 visitor = getattr(self, method, self.generic_visit)
--> 371 return visitor(node)
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:511](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:511), in _Unparser.visit_JoinedStr(self, node)
509 for value in node.values:
510 meth = getattr(self, "_fstring_" + type(value).__name__)
--> 511 meth(value, self.buffer_writer)
512 buffer.append((self.buffer, isinstance(value, Constant)))
513 new_buffer = []
File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:546](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:546), in _Unparser._fstring_FormattedValue(self, node, write)
544 unparser.set_precedence(_Precedence.TEST.next(), node.value)
545 expr = unparser.visit(node.value)
--> 546 if expr.startswith("{"):
547 write(" ") # Separate pair of opening brackets as "{ {"
548 if "\\" in expr:
AttributeError: 'NoneType' object has no attribute 'startswith'
Hey @sergiynesterenko90! Thanks for reporting this! We are aware of this, and it's actually clearly manifesting in the 2 PRs we currently have up #613 and #606. This is caused by the AST unparse functionality we added for backwards compatibility with Py3.8. I think #613 has a potential fix but we also have to fix Hera's CI for this because different Py versions AST modules result in different scripts being tested, which breaks CI. Going to try to fix this issue as part of those PRs and will post back here
I think this should be highlighted in the docs as the error is ambiguous and took me quite some time to figure out. Perhaps adding a better warning message?
This only happens with Python 3.8 which is deprecated and will be end of life soon. Our preference would be to drop support for 3.8 soon and ask users to upgrade to the next minor version of python instead.
