crawlee
crawlee copied to clipboard
Improve type safety
- There are multiple opportunities throughout Crawlee to improve type safety and optionally introduce runtime validation:
- Dataset items (
CrawlingContext.pushData) - Key-value store content
Request.userData- Request routing labels
- ? possibly somewhere else
- Dataset items (
- One option is allowing the user to declare some special types (
DatasetItem,RequestLabel, ...) in a.d.tsfile and using it for type checking- TanStack router utilizes a similar approach
- A safer and more versatile approach would be to accept standard-schema models (either via
BasicCrawlerorConfiguration) and infer the types from those- this would also add runtime validation (in contrast to the
.d.tsapproach which is more of a gentleman agreement)
- this would also add runtime validation (in contrast to the
Related: #1485