databricks-sql-python icon indicating copy to clipboard operation
databricks-sql-python copied to clipboard

Enhance Arrow to Pandas conversion with type overrides and additional kwargs

Open madhav-db opened this issue 9 months ago • 7 comments

  • Introduced _arrow_pandas_type_override and _arrow_to_pandas_kwargs in Connection class for customizable dtype mapping and DataFrame construction parameters.
  • Updated ResultSet to utilize these new options during conversion from Arrow tables to Pandas DataFrames.
  • Added unit tests to validate the new functionality, including scenarios for type overrides and additional kwargs handling.

What type of PR is this?

  • [ ] Refactor
  • [x] Feature
  • [ ] Bug Fix
  • [ ] Other

Description

How is this tested?

  • [x] Unit tests
  • [ ] E2E Tests
  • [ ] Manually
  • [ ] N/A

Related Tickets & Documents

#578

madhav-db avatar May 30 '25 06:05 madhav-db

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions[bot] avatar May 30 '25 06:05 github-actions[bot]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions[bot] avatar May 30 '25 06:05 github-actions[bot]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions[bot] avatar May 30 '25 07:05 github-actions[bot]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions[bot] avatar Jun 16 '25 06:06 github-actions[bot]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions[bot] avatar Jun 16 '25 06:06 github-actions[bot]

@madhav-db Very interested in seeing this pass. 🤞

j-bennet avatar Jun 27 '25 15:06 j-bennet

Plz merge the master once, as the last commit is pretty old

jprakash-db avatar Sep 11 '25 08:09 jprakash-db