brimdata
Skip input rows
A community user asked in a recent Slack thread:
I keep getting CSV files where I have to skip rows before the header - things like pandas and knime let you skip rows when you're trying to ingest tabular data - not sure if that's something that makes sense for the shaping wizard, but it would save me a step
Details
At the time this issue is being opened, Zed is at commit 6ff0586.
As noted by the user above, Pandas has its skiprows option in its CSV reader and KNIME has Limit Rows options in its CSV reader which includes "Skip first lines".
The community user described their requirement in the context of the shaping as part of Zui's Preview & Load. However, I could imagine this functionality being just as useful for constructing zq pipelines at the shell. Since Zui invokes zq for shaping purposes, if we add it at the Zed layer first Zui can ultimately take advantage as well.
@nwt pointed out that adding the many advanced bells & whistles to deal with all the CSV variants will likely only be possible when we implement our own CSV parser, since the "stock" one we rely on right now doesn't support all the knobs that we see in implementations like Pandas.