Improve token counting for messages with package

[pamelafox]

Purpose

As I needed our token counting and message building functionality for another sample, I moved it to a separate package. That also gave me the opportunity to increase test coverage and receive community contributions for improving it, so it is more accurate than what's currently in this repo (notably, it can now count tools, which it was ignoring before).

You can see the interface for the other package here: https://pypi.org/project/openai-messages-token-helper/

This PR deletes the code/tests from this repo and replaces them with equivalent functionality in that package.

Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error? If you're not sure, try it out on an old environment.

[ ] Yes
[X] No

Does this require changes to learn.microsoft.com docs?

This repository is referenced by this tutorial which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial, check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

[ ] Yes
[X] No

Type of change

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[X] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Code quality checklist

See CONTRIBUTING.md for more details.

• [X] The current tests all pass (python -m pytest).
• [ ] I added tests that prove my fix is effective or that my feature works: There is already a test, test_chat_with_long_history, that effectively uses the functionality.
• [X] I ran python -m pytest --cov to verify 100% coverage of added lines
• [ ] I ran python -m mypy to check for type errors
• [X] I either used the pre-commit hooks or ran ruff and black manually on my code.

[pamelafox]

I've merged from main, this is ready for review.