Sonal

zingg0.4.0-spark3.5.0 match phase works on emr -7.0.0 cluster, but not on emr serveless 7.0.0.

3

 

{project.parent.basedir} etc are getting created. please check pom.xml and fix

spark block and sparkfeaturefactory both have this method. move to one common place

protected Dataset getBlocks(Dataset blocked) throws Exception{ return DSUtil.joinWithItself(blocked, ColName.HASH_COL, true).cache(); } this method is not called form the code and will lead to issues with the linker and resolver?

this makes readthedoc build fail ` sonal@sonal-mac docs % rm -rf _build/html; make html Running Sphinx v7.2.6 path is /Users/sonal/zingg/python/zingg making output directory... done loading pickled environment... done building [mo]:...

rethink args validation

1

some of the methods are tied to a phase, some are probably invoked directly while jackson sets the property? Need to see whats really needed here and if the code...

multiple registration of sim functions

1

2023-11-24 17:49:58,143 [main] WARN org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry - The function affinegapsimilarityfunction replaced a previously registered function. 2023-11-24 17:49:58,143 [main] WARN org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry - The function jarowinklerfunction replaced a previously registered function. 2023-11-24 17:49:58,143...

Check https://gist.github.com/UsAndRufus/7b46bb349f5016b5995f8feb786597b3 and see places to improve the python api experience.

Unit Testing Framework

1

To support multiple backends like Spark, Snowflake etc as well as to test the base algorithms and concrete implementations, we need to build a unit testing framework. Broadly, we need...