Compatibility Issue: java.lang.NoSuchMethodError with glow.py on PySpark 3.5.1

[nickzren]

When using glow.py in a Conda environment with the following setup:

• Python version: 3.10
• PySpark version: 3.5.1
• Glow.py version: 2.0.0 (installed via pip install glow.py)

Attempting to load a VCF file results in the following error:

java.lang.NoSuchMethodError: 'java.lang.String org.apache.spark.sql.execution.datasources.PartitionedFile.filePath()'

Steps to Reproduce:

1. Create a Conda environment with Python 3.10.
2. Install PySpark 3.5.1 and glow.py via pip.
3. Attempt to load a VCF file using Glow.

Workaround: Downgrading to PySpark 3.3.2 resolves the issue, and the VCF file loads as expected.