How to read tsv?

[Hoeze]

I'm trying to run:

glow.transform(
    "pipe",
    input_df.limit(10),
    cmd=json.dumps(shlex.split(cmd)),
    inputFormatter='vcf',
    inVcfHeader='infer',
    outputFormatter='csv',
    out_delimiter="\t"
)

but I still only get a single column. How can I read tsv output?

[karenfeng]

Hi @Hoeze! I just ran a quick test locally, and the CSV piper should still work for TSVs. However, the CSV datasource exposes many options, including whether there is a header (out_header) or there are comments (out_comment). Can you tell me more about what the command you're running?

[Hoeze]

@karenfeng The issue that I had was that the option should be called outDelimiter instead of out_delimiter.

However, I'm having another issue now. For some reason, the options outNullValue and outEmptyValue do not work:

import json
import shlex

vep_transformed_df = glow.transform(
    "pipe",
    input_df.limit(10).distinct(),
#     cmd=json.dumps(shlex.split("cat | grep -v '^##'")),
    cmd=json.dumps(shlex.split(vep_cmd)),
    inputFormatter='vcf',
    inVcfHeader='infer',
    outputFormatter='csv',
#     outQuote="##",
    outHeader=True,
    outDelimiter="\t",
    outNullValue="-",
    outEmptyValue="-",
)

# vep_transformed_df.toPandas()["cDNA_position"].iloc[0]
'-'

Is there again some difference in naming?

[henrydavidge]

Closing since we now support only the text piper and to/from csv functions in Spark