[package import issue]: Autogen does not install markdownify package required by WebSurfer.

[gauravdhiman]

Describe the issue

Autogen does not install markdownify package, a required dependency for by WebSurfer. I use poetry for env namagement and installing the packages. Used poetry add pyautogen command, it installed the autogen in my venv managed by poetry, but when I run from autogen.agentchat.contrib.web_surfer import WebSurferAgent in python script (with poetry env active), it throws below stacktrace.

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[7], [line 9](vscode-notebook-cell:?execution_count=7&line=9)
      [7](vscode-notebook-cell:?execution_count=7&line=7) from typing import List, Dict
      [8](vscode-notebook-cell:?execution_count=7&line=8) from autogen.agentchat.assistant_agent import AssistantAgent  # noqa E402
----> [9](vscode-notebook-cell:?execution_count=7&line=9) from autogen.agentchat.contrib.web_surfer import WebSurferAgent  # noqa: E402
     [10](vscode-notebook-cell:?execution_count=7&line=10) from pathlib import Path

File ~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/agentchat/contrib/web_surfer.py:12
      [9](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/agentchat/contrib/web_surfer.py:9) from typing_extensions import Annotated
     [11](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/agentchat/contrib/web_surfer.py:11) from ... import Agent, AssistantAgent, ConversableAgent, GroupChat, GroupChatManager, OpenAIWrapper, UserProxyAgent
---> [12](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/agentchat/contrib/web_surfer.py:12) from ...browser_utils import SimpleTextBrowser
     [13](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/agentchat/contrib/web_surfer.py:13) from ...code_utils import content_str
     [14](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/agentchat/contrib/web_surfer.py:14) from ...oai.openai_utils import filter_config

File ~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/browser_utils.py:10
      [7](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/browser_utils.py:7) from typing import Any, Dict, List, Optional, Tuple, Union
      [8](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/browser_utils.py:8) from urllib.parse import urljoin, urlparse
---> [10](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/browser_utils.py:10) import markdownify
     [11](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/browser_utils.py:11) import requests
     [12](https://file+.vscode-resource.vscode-cdn.net/Users/gauravdhiman/projects/python/ai/notebooks/notebooks/~/projects/python/ai/notebooks/.venv/lib/python3.12/site-packages/autogen/browser_utils.py:12) from bs4 import BeautifulSoup

ModuleNotFoundError: No module named 'markdownify'

Steps to reproduce

1). cd 2). poetry shell (to activate the poetry env) 3). poetry add pyautogen (install the latest version of autogen in env managed by poetry) 4). add line from autogen.agentchat.contrib.web_surfer import WebSurferAgent to a python file and execute that file

Screenshots and logs

Additional Information

Autogen version: 0.2.32