Lean init not working behind firewall

[nick-voisin]

Hi,

I'm currently testing out the Lean CLI for my firm. Unfortunately I'm stuck at the very beginning:

I'm sure it has something to do with our IT team blocking every possible port we can think of. Is there a workaround on your end, otherwise I'm going to have to wait weeks before an internal ticket gets approved.

I went in the init.py file -> _download_repository function -> line 37 and added verify = False to the get method. I get a warning and nothing :

Thanks for your help,

Nicolas Voisin

[jmerle]

Hi Nicolas,

I went in the init.py file -> _download_repository function -> line 37 and added verify = False to the get method. I get a warning and nothing :

Do you mean the download ran successfully after adding verify = False, or do you mean the command exited immediately?

[nick-voisin]

Hi Jasper,

The terminal just freezes with the urllib3 warning. I don't have access to the windows terminal (i.e can't type any command), and nothing appears in the ./LeanCLI folder. I end up just closing the tab because Ctrl+C or Ctrl+Z doesn't work either.

[jmerle]

I just released a new version which supports the Windows Certificate Store as a source of CA certificates to verify SSL vertificates on Windows. This should make it work in corporate environments like yours where your company stores its own CA certificate in the certificate stores of all its machines.

Can you please update using pip install --upgrade lean (you can use the production version, you shouldn't need the verify = False thing) and try running lean init again? Keep in mind that it may take a few minutes for the command to complete as it needs to download and extract 200MB+, but the errors and warnings should be gone.

[nick-voisin]

Hi Jasper,

I don't get the error messages anymore, I'm just stuck at downloading :

Waited 30 minutes at this point and given our bandwidth I'm sure it would have been done by now. Unfortunately I'm going to have to go through our IT and Security Team. Thanks for your efforts and help though ! Will keep you posted.

Nicolas

[nick-voisin]

Small update :

IT team says no ports are blocked, since I can download the file via the browser (MS Edge). I've tried the following code to test the request. The file length is hardcoded because the headers response does not contain the file size (via content-length).

import requests
import time

url = "https://github.com/QuantConnect/Lean/archive/master.zip"
start_time = time.time()

local_filename = url.split('/')[-1]
chunk_size = 8192

with requests.get(url, stream=True) as r:
    r.raise_for_status()
    total_size = 211932341

    with open(local_filename, 'wb') as f:
        count = 1
        for chunk in r.iter_content(chunk_size=chunk_size):
            duration = time.time() - start_time
            progress = count * chunk_size
            percent = progress / total_size
            print(f"...{percent:.0%} complete, with {progress} bytes downloaded, {duration:.1f} seconds passed")
            f.write(chunk)

            count += 1

print("Download finished!")

I end up being stuck with :

87% complete, with 184164352 bytes downloaded, 103.0 seconds passed

And that's it. Nothing after that.

[jmerle]

Weird, that script finishes successfully in ~20 seconds for me.

Can you please try the following script? I am wondering whether the streaming is the issue here.

import requests
import time
from pathlib import Path

url = "https://github.com/QuantConnect/Lean/archive/master.zip"
start_time = time.time()

response = requests.get(url)
response.raise_for_status()

local_file = Path.cwd() / url.split("/")[-1]
with local_file.open("wb") as file:
    file.write(response.content)

bytes_length = len(local_file.read_bytes())
duration = time.time() - start_time

print(f"Download finished, downloaded {bytes_length} bytes in {duration:.1f} seconds")

As an alternative, since you can download the file manually using Edge, you can follow these steps to manually perform the actions performed by lean init so you can continue using the CLI:

1. Download https://github.com/QuantConnect/Lean/archive/master.zip using your browser.
2. Extract the Data directory in master.zip to the data directory in the directory you are running lean init in.
3. Save the content of this gist to lean.json in the directory you are running lean init in.
4. Run lean config set default-language csharp or lean config set default-language python to set the default language of the lean create-project <name> command to either C# or Python.

[nick-voisin]

Hi Jasper,

I tried your script and I get the same issue - it just hangs. I ve tried my script at home and it works perfectly fine so I'm confident the issue is within our coporate environment but unless I pinpoint the exact issue the IT support will not budge.

I will try your alternative solution.