Onboarding Neomi Sule

[sonniki]

• Source: intern.onboarding_checklist.reference.md

Org

• [x] HiringMeister: File an issue with this checklist

  • The title is "Onboarding {{Name}}"
  • Copy-and-paste the whole checklist starting from here
  • The issue should be assigned to the intern

• [x] Intern: Update this GitHub issue if you face any problems. If applicable, do a PR proposing improvements to the checklist (or any other docs), since this will allow us to improve the process as we move forward

• [x] Intern: Join the Telegram channel - https://t.me/+DXZXsWoEHR1mNWIx

• [x] HiringMeister: Establish contact by email or Telegram with the intern with a few words about the next steps

• [x] Intern: Post your laptop's OS (Windows, Linux, Mac) in the comments of this issue

• [x] Intern: Confirm access to the public GH repos

  • [x] helpers
  • [x] tutorials

• [x] IT: @Shayawnn Add the intern to the mailing group [email protected] so that they can send morning TODO emails

  • The intern's personal e-mail address can be found in the corresponding Asana task in the Hiring project

IT setup

• [ ] Intern: Set up the development environment following instructions in intern.set_up_development_on_laptop.how_to_guide.md

Must-read

• [ ] Intern: Carefully study all the documents in the must-read list
  • [ ] General rules of collaboration
  • [ ] Coding style guide
  • [ ] How to write unit tests
  • [ ] How to run unit tests
  • [ ] Creating a Jupyter Notebook
  • [ ] What to do before opening a PR
  • [ ] Code review process
  • [ ] Git workflows and best practices
  • [ ] GitHub organization
  • [ ] Tips for writing documentation
  • They will help you get up to speed with our practices and development style
  • Read them carefully one by one
  • Ask questions
  • Memorize / internalize all the information
  • Take notes
  • Mark the reading as done
  • Open a GH issue/PR to propose improvements to the documentation

Final checks

• [ ] Intern: Exercise all the important parts of the systems
  • [ ] Create a GitHub issue
  • [ ] Check out and pull the latest version of the repo code
  • [ ] Create a branch
  • [ ] Run regressions (i run_fast_tests)
  • [ ] Run Linter (i lint --files="...")
  • [ ] Start a Docker container (i docker_bash)
  • [ ] Start a Jupyter server (i docker_jupyter)
  • [ ] Do a PR

Mentor: @gpsaggese

[neomisule]

Hi, I have windows OS on my laptop.

[neomisule]

Successfully Set Up Ubuntu on VMWare Workstation Pro After Resolving Multiple Setup Issues.

• Switched from VirtualBox to VMWare Workstation Pro after encountering persistent issues with VirtualBox, including the "system program problem detected" error.

• Spent a day troubleshooting the "unsupported hypervisor" problem while setting up Ubuntu.

• Spent a couple more days resolving internet connectivity issues, had to try several different fixes before finally getting it to work.

• Sharing all the progress at once now since I was advised to provide regular updates!

[gpsaggese]

@sonniki and @samarth9008 is it worth to add @neomisule to document this in a markdown? Other students can leverage her work, instead of rediscovering the wheel.

I would have something like all.running_Causify_on_Windows.how_to_guide.md

[sonniki]

@sonniki and @samarth9008 is it worth to add @neomisule to document this in a markdown? Other students can leverage her work, instead of rediscovering the wheel.

I would have something like all.running_Causify_on_Windows.how_to_guide.md

Good idea. @neomisule do you think you can reproduce what you've done for a doc? This could be the issue and PR for the "Final checks" part of the checklist.

[neomisule]

@sonniki and @samarth9008 is it worth to add @neomisule to document this in a markdown? Other students can leverage her work, instead of rediscovering the wheel. I would have something like all.running_Causify_on_Windows.how_to_guide.md

Good idea. @neomisule do you think you can reproduce what you've done for a doc? This could be the issue and PR for the "Final checks" part of the checklist.

Yes definitely! I'd be happy to create a document on that.

[neomisule]

Hi, I initially tried setting up the Thin environment, but ran into a bunch of issues, so I switched to tmux. Now I’m getting an error, and I think it might be because I’m running it on a virtual machine (causifyns).

This is the command I entered: dev_scripts_helpers/thin_client/tmux.py --index 1

This is the error I encountered:

##> /home/neomisule/causifygit/dev_scripts_helpers/thin_client/tmux.py
10:13:59 - INFO  hdbg.py init_logger:1013                               Saving log to file '{'/home/neomisule/causifygit/dev_scripts_helpers/thin_client/thin_client_utils.py.log'}'
10:13:59 - INFO  hdbg.py init_logger:1018                               > cmd='dev_scripts_helpers/thin_client/tmux.py --index 1'
10:13:59 - INFO  thin_client_utils.py create_tmux_session:270           tmux_name=helpers1
10:13:59 - INFO  thin_client_utils.py create_tmux_session:299           The tmux session doesn't exist, creating it
10:13:59 - INFO  thin_client_utils.py create_tmux_session:308           Inferred server setup
Traceback (most recent call last):
  File "/home/neomisule/causifygit/dev_scripts_helpers/thin_client/tmux.py", line 41, in <module>
    tcu.create_tmux_session(
  File "/home/neomisule/causifygit/dev_scripts_helpers/thin_client/thin_client_utils.py", line 310, in create_tmux_session
    hdbg.dassert_in(server_name, ["dev1", "dev2", "dev3"])
  File "/home/neomisule/causifygit/helpers/hdbg.py", line 295, in dassert_in
    _dfatal(txt, msg, *args, only_warning=only_warning)
  File "/home/neomisule/causifygit/helpers/hdbg.py", line 142, in _dfatal
    dfatal(dfatal_txt)
  File "/home/neomisule/causifygit/helpers/hdbg.py", line 71, in dfatal
    raise assertion_type(ret)
AssertionError: 
################################################################################
* Failed assertion *
'Causifyns' in '['dev1', 'dev2', 'dev3']'
################################################################################

[gpsaggese]

1. You should call the dir helpers1 instead of causifygit (I think there should be a note on this in the doc). If things are not called consistently, the system doesn't know where are things.
2. I'll check, but I believe the problem is because so far we only had mac and AWS servers, and now we started supporting Linux outside our AWS VPC, so the logic should be extended to handle that.

• You can spend 5 mins trying to debug it, otherwise one of us can take a look

[sonniki]

You should call the dir helpers1 instead of causifygit (I think there should be a note on this in the doc). If things are not called consistently, the system doesn't know where are things.

Correct, this is indeed given in the docs, in the cloning instructions. Moreover, @neomisule, as it says there, the local repo folder should be under ~/src, i.e. the path to it would end in ~/src/{REPO_NAME}{IDX}. Please fix this -- otherwise you'll be getting various errors in the future.

I'll check, but I believe the problem is because so far we only had mac and AWS servers, and now we started supporting Linux outside our AWS VPC, so the logic should be extended to handle that.

Indeed. @neomisule please open a new issue (following our conventions of how to file issues), and try to propose a fix there (since you don't have the full dev setup yet, one of us will actually do the implementation). Hint: you should find helpful functions in helpers/hserver.py

[neomisule]

1. You should call the dir helpers1 instead of causifygit (I think there should be a note on this in the doc). If things are not called consistently, the system doesn't know where are things.
2. I'll check, but I believe the problem is because so far we only had mac and AWS servers, and now we started supporting Linux outside our AWS VPC, so the logic should be extended to handle that.

• You can spend 5 mins trying to debug it, otherwise one of us can take a look

I initially tried using helpers1 and placed the local repository folder under src. When that didn't work, I experimented with different configurations, including using causifygit and other variations. After several attempts, I have come up with potential solution and as @sonniki suggested, I'll create an issue and add the proposed fix there.

[neomisule]

Correct, this is indeed given in the docs, in the cloning instructions. Moreover, @neomisule, as it says there, the local repo folder should be under ~/src, i.e. the path to it would end in ~/src/{REPO_NAME}{IDX}. Please fix this -- otherwise you'll be getting various errors in the future.

I got rid of the other variations that I was trying to work with and I have fixed it according to the documentation.

Indeed. @neomisule please open a new issue (following our conventions of how to file issues), and try to propose a fix there (since you don't have the full dev setup yet, one of us will actually do the implementation). Hint: you should find helpful functions in helpers/hserver.py

I believe the is_external_linux() function can help address this issue. I’ll open a new issue and outline the details of the proposed fix there.

[neomisule]

I was able to set up tmux and successfully run docker pull hello-world. However, when I try to run i docker_pull, I encounter the error s3fs is not defined. I’ve tried installing s3fs using pip install s3fs and also importing it, but both show that the requirements are already satisfied.

Here’s the additional context for the error:

  File "/usr/lib/python3/dist-packages/invoke/program.py", line 695, in load_collection
    module, parent = loader.load(coll_name)
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/invoke/loader.py", line 76, in load
    module = imp.load_module(name, fd, path, desc)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/zombie_imp/imp_3_11.py", line 238, in load_module
    return load_source(name, filename, file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/zombie_imp/imp_3_11.py", line 175, in load_source
    module = _load(spec)
             ^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 966, in _load
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/neomisule/src/helpers1/tasks.py", line 10, in <module>
    from helpers.lib_tasks import (  # This is not an invoke target.
  File "/home/neomisule/src/helpers1/helpers/lib_tasks.py", line 15, in <module>
    from helpers.lib_tasks_docker import *  # isort:skip  # noqa: F401,F403 # pylint: disable=unused-import,unused-wildcard-import,wildcard-import
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/neomisule/src/helpers1/helpers/lib_tasks_docker.py", line 26, in <module>
    import helpers.hs3 as hs3
  File "/home/neomisule/src/helpers1/helpers/hs3.py", line 57, in <module>
    AwsProfile = Optional[Union[str, s3fs.core.S3FileSystem]]
                                     ^^^^
NameError: name 's3fs' is not defined

[samarth9008]

also importing it

Where did we try importing it?

I think the reason for this error is due to this LOCs

https://github.com/causify-ai/helpers/blob/master/helpers/hs3.py#L21-L25

Unable to import s3fs. Lets try to debug little more.

[neomisule]

Where did we try importing it?

I tried pip install s3fs and then I went to python shell using python3 and imported it there.

I think the reason for this error is due to this LOCs https://github.com/causify-ai/helpers/blob/master/helpers/hs3.py#L21-L25

Yeah I was checking this as well, trying to understand what's going on.

[gpsaggese]

I might not be in the loop with what's going on, but here are my 2c.

s3fs is installed by build.py, when the thin environment is installed with virtual env. After that you either use tmux.py or you need to do setenv.sh to configure a shell and you can only run invoke commands from that shell.

To check if you are running from the right shell you can do

> echo $VIRTUAL_ENV
/Users/saggese/src/venv/client_venv.helpers

The prompt should look like

Does it make sense? If not, I would re-read the instructions and restart from scratch.

[neomisule]

To check if you are running from the right shell you can do

I checked, it is the correct shell.

The prompt should look like

Does it make sense? If not, I would re-read the instructions and restart from scratch.

I ran this command and got the following output:

boto3 >= 1.20.17
# Restricted because of CmTask8340
# https://github.com/cryptokaizen/cmamp/issues/8340
# See https://github.com/psf/requests/issues/6707 for more details.
requests <= 2.31.0
# Keep in sync with `.github/gh_requirements.txt`, see CmTask6488.
docker < 7
docker-compose >= 1.29.0
invoke >= 1.5.0
poetry
pytest >= 6.0.0
s3fs  # For tools like `publish_notebook.py`.
tqdm
# On Mac and locally on Linux there is an issue related to this package
# (see HelpersTask377), so we might want to pin it down.
# pyyaml == 5.3.1

I've already tried restarting the process a couple times with different changes and potential solutions, now I'll try to check other aspects based on this output.

[neomisule]

I've already tried restarting the process a couple times with different changes and potential solutions, now I'll try to check other aspects based on this output.

I've checked and confirmed that all installations are correct, but when trying to use docker-compose and pyyaml, I'm encountering the following error:

          self.add_defaults()
        File "/tmp/pip-build-env-0k6ycb8u/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 581, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-0k6ycb8u/overlay/lib/python3.12/site-packages/setuptools/command/sdist.py", line 109, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-0k6ycb8u/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 245, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-0k6ycb8u/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 330, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-0k6ycb8u/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 131, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Additionally, the first link https://github.com/cryptokaizen/cmamp/issues/8340 is returning a page not found error.

[samarth9008]

Can you also post the sequence of commands you're running?

[neomisule]

Can you also post the sequence of commands you're running?

Sure, I ran (client_venv.helpers) neomisule@Causifyns:~/src/helpers1$ pip install docker-compose>=1.29.0 and (client_venv.helpers) neomisule@Causifyns:~/src/helpers1$ pip install pyyaml==5.3.1

[samarth9008]

I meant the sequence when setting up the env. Not sure if the above installations are needed bcz these are done when running build.py

[neomisule]

I meant the sequence when setting up the env. Not sure if the above installations are needed bcz these are done when running build.py

I tried to check the versions, but the packages weren't found, so I tried to install the correct ones.

I have set up tmux, as thin environment was encountering errors. So I haven't explicitly run build.py, assuming the installations and necessities would be covered while setting up tmux, as thin environment is activated automatically in a tmux session. These are the instructions:

neomisule@Causifyns:~$ cd src
neomisule@Causifyns:~/src$ cd helpers1
neomisule@Causifyns:~/src/helpers1$  dev_scripts_helpers/thin_client/tmux.py --create_global_link
##> /home/neomisule/src/helpers1/dev_scripts_helpers/thin_client/tmux.py
18:12:37 - INFO  hdbg.py init_logger:1013                               Saving log to file '{'/home/neomisule/src/helpers1/dev_scripts_helpers/thin_client/thin_client_utils.py.log'}'
18:12:37 - INFO  hdbg.py init_logger:1018                               > cmd='dev_scripts_helpers/thin_client/tmux.py --create_global_link'
18:12:37 - INFO  thin_client_utils.py create_tmux_session:267           Creating the global link
################################################################################
ln -sf /home/neomisule/src/helpers1/dev_scripts_helpers/thin_client/tmux.py ~/go_helpers.py
################################################################################
18:12:37 - INFO  thin_client_utils.py create_tmux_session:271           Link created: exiting
neomisule@Causifyns:~/src/helpers1$  dev_scripts_helpers/thin_client/tmux.py --index 1

[neomisule]

It worked!! So, as it kept showing the packages weren't found, which they should have been, I installed them all manually. Then, when I ran i docker_pull, it indicated that the yaml package was missing. Since I couldn't download the specified version from the requirements.txt file due to the error, I installed the latest version (6.0.2). After making a few minor fixes and addressing disk space issues, I was able to successfully run i docker_pull and i docker_pull_helpers.

[samarth9008]

As mentioned in the docs, the correct solution is to run build.py file once and only when doing the setup for the first time before running tmux.py. When you run tmux.py, you don't need to run setenv.sh file as tmux will handle that automatically but that doesn't mean env should not be "build" using build.py.

[neomisule]

As mentioned in the docs, the correct solution is to run build.py file once and only when doing the setup for the first time before running tmux.py. When you run tmux.py, you don't need to run setenv.sh file as tmux will handle that automatically but that doesn't mean env should not be "build" using build.py.

I think the documentation is a little confusing. The thin environment section says it can be skipped if you're using tmux, and the tmux part mentions that the thin environment runs automatically, but it doesn’t clearly say that build.py needs to be run at least once.

When you and @gpsaggese mentioned build.py, I gave it a shot, but it warned that it would delete the old virtual environment, so I didn’t want to risk it. That said, I was still able to finish the setup and start the Jupyter server.

Let me know if you'd like me to do anything differently or make any changes!

[sonniki]

@neomisule feel free to open a PR with proposed changes to the docs based on your experience

[samarth9008]

I think the documentation is a little confusing. The thin environment section says it can be skipped if you're using tmux, and the tmux part mentions that the thin environment runs automatically, but it doesn’t clearly say that build.py needs to be run at least once.

Yes you are right. Lets modify it according to the right thing