Error during matcher.fit for pyspark==3.4.0

[mkopec87]

I started to get errors during matcher.fit method call after updating to pyspark==3.4.0:

  Caused by: java.lang.IllegalArgumentException: requirement failed: Index 0 follows 0 and is not strictly increasing

Still digging to find the cause (may be specific to other dependencies I have), but do you plan to add github actions here in github to test e.g. against different python versions or updated dependencies?

[mkopec87]

Some more details:

../.venv/lib/python3.10/site-packages/spark_matcher/matching_base/matching_base.py:227: in fit
    pairs_table = self._create_train_pairs_table(sdf_1, sdf_2)
../.venv/lib/python3.10/site-packages/spark_matcher/matching_base/matching_base.py:128: in _create_train_pairs_table
    hashed_pairs_table = self.table_checkpointer.checkpoint_table(h_sampler.create_pairs_table(sdf_1, sdf_2),
../.venv/lib/python3.10/site-packages/spark_matcher/sampler/training_sampler.py:181: in create_pairs_table
    sdf_1_vectorized, sdf_2_vectorized = self._vectorize(self.col_names, sdf_1, sdf_2)
../.venv/lib/python3.10/site-packages/spark_matcher/sampler/training_sampler.py:127: in _vectorize
    sdf_merged = self.table_checkpointer(
../.venv/lib/python3.10/site-packages/spark_matcher/table_checkpointer.py:25: in __call__
    return self.checkpoint_table(sdf, checkpoint_name)
../.venv/lib/python3.10/site-packages/spark_matcher/table_checkpointer.py:89: in checkpoint_table
    sdf.write.parquet(file_name, mode='overwrite')