algo
algo copied to clipboard
Published
20 hours ago •
trailofbits
trailofbits
Error with BlobServiceClient from azure.storage.blob during deployment to Azure (on Windows with WSL2 running Ubuntu 22 LTS)
Describe the bug
Deployment to Azure fails with an error (on Windows with WSL2 running Ubuntu 22 LTS). The message is as follows:
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'BlobServiceClient' from 'azure.storage.blob' (/root/algo/.env/lib/python3.10/site-packages/azure/storage/blob/__init__.py)
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on DESKTOP-STQVHKJ's Python /root/algo/.env/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}
I have already got the auth token from Azure using azure-cli's az login (although algo didn't guide me through that either, and I think it should have, but that would be the topic of another issue).
To Reproduce
Steps to reproduce the behavior:
- On WSL2 command line, clone algo.
- pip-install
virtualenvand more. - "Install Algo's remaining dependencies" using the snippet provided in readme.md.
- Install
azure-cli. Receive auth token from Azure withaz login. - Run
./algo. - Choose "4. Microsoft Azure" when prompted.
- Choose all the defaults until the region selection.
- Choose "29. (Europe) Germany West Central" as the region.
- Right at
TASK [cloud-azure : Create AlgoVPN Server], the provided in the code block above in the description is generated.
Expected behavior
The installation should continue.
Additional context
- Windows
- WSL2 with Ubuntu 22.04.2 LTS
Full log
This is the whole output of ./algo. There is another one from ./algo -vvv beneath it.
(.env) root@DESKTOP-STQVHKJ:~/algo# ./algo
PLAY [localhost] *********************************************************************************************
TASK [Gathering Facts] ***************************************************************************************
ok: [localhost]
TASK [Playbook dir stat] *************************************************************************************
ok: [localhost]
TASK [Ensure Ansible is not being run in a world writable directory] *****************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature will be removed from
ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
[WARNING]: The value '' is not a valid IP address or network, passing this value to ipaddr filter might
result in breaking change in future.
TASK [Ensure the requirements installed] *********************************************************************
ok: [localhost]
TASK [Set required ansible version as a fact] ****************************************************************
ok: [localhost] => (item=ansible==9.1.0)
TASK [Just get the list from default pip] ********************************************************************
ok: [localhost]
TASK [Verify Python meets Algo VPN requirements] *************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [Verify Ansible meets Algo VPN requirements] ************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
[WARNING]: Found variable using reserved name: no_log
PLAY [Ask user for the input] ********************************************************************************
TASK [Gathering Facts] ***************************************************************************************
ok: [localhost]
[Cloud prompt]
What provider would you like to use?
1. DigitalOcean
2. Amazon Lightsail
3. Amazon EC2
4. Microsoft Azure
5. Google Compute Engine
6. Hetzner Cloud
7. Vultr
8. Scaleway
9. OpenStack (DreamCompute optimised)
10. CloudStack (Exoscale optimised)
11. Linode
12. Install to existing Ubuntu latest LTS server (for more advanced users)
Enter the number of your desired provider
:
TASK [Cloud prompt] ******************************************************************************************
ok: [localhost]
TASK [Set facts based on the input] **************************************************************************
ok: [localhost]
[VPN server name prompt]
Name the vpn server
[algo]
:
TASK [VPN server name prompt] ********************************************************************************
ok: [localhost]
[Cellular On Demand prompt]
Do you want macOS/iOS clients to enable "Connect On Demand" when connected to cellular networks?
[y/N]
:
TASK [Cellular On Demand prompt] *****************************************************************************
ok: [localhost]
[Wi-Fi On Demand prompt]
Do you want macOS/iOS clients to enable "Connect On Demand" when connected to Wi-Fi?
[y/N]
:
TASK [Wi-Fi On Demand prompt] ********************************************************************************
ok: [localhost]
[Retain the PKI prompt]
Do you want to retain the keys (PKI)? (required to add users in the future, but less secure)
[y/N]
:
TASK [Retain the PKI prompt] *********************************************************************************
ok: [localhost]
[DNS adblocking prompt]
Do you want to enable DNS ad blocking on this VPN server?
[y/N]
:
TASK [DNS adblocking prompt] *********************************************************************************
ok: [localhost]
[SSH tunneling prompt]
Do you want each user to have their own account for SSH tunneling?
[y/N]
:
TASK [SSH tunneling prompt] **********************************************************************************
ok: [localhost]
TASK [Set facts based on the input] **************************************************************************
ok: [localhost]
PLAY [Provision the server] **********************************************************************************
TASK [Gathering Facts] ***************************************************************************************
ok: [localhost]
--> Please include the following block of text when reporting issues:
Algo running on: Ubuntu 22.04.2 LTS (Virtualized: wsl)
Created from git fork. Last commit: 74051d0 Update README.md dependencies (#14634)
Python 3.10.6
Runtime variables:
algo_provider "azure"
algo_ondemand_cellular "False"
algo_ondemand_wifi "False"
algo_ondemand_wifi_exclude "X251bGw="
algo_dns_adblocking "False"
algo_ssh_tunneling "False"
wireguard_enabled "True"
dns_encryption "True"
TASK [Display the invocation environment] ********************************************************************
changed: [localhost]
TASK [Install the requirements] ******************************************************************************
ok: [localhost]
TASK [Generate the SSH private key] **************************************************************************
ok: [localhost]
TASK [Generate the SSH public key] ***************************************************************************
ok: [localhost]
TASK [Copy the private SSH key to /tmp] **********************************************************************
ok: [localhost]
TASK [Include a provisioning role] ***************************************************************************
TASK [cloud-azure : Install requirements] ********************************************************************
ok: [localhost]
TASK [cloud-azure : set_fact] ********************************************************************************
ok: [localhost]
TASK [cloud-azure : Set the default region] ******************************************************************
ok: [localhost]
[cloud-azure : pause]
What region should the server be located in?
1. Asia
2. Asia Pacific
3. Australia
4. (Asia Pacific) Australia Central
5. (Asia Pacific) Australia Central 2
6. (Asia Pacific) Australia East
7. (Asia Pacific) Australia Southeast
8. Brazil
9. (South America) Brazil South
10. (South America) Brazil Southeast
11. Canada
12. (Canada) Canada Central
13. (Canada) Canada East
14. (Asia Pacific) Central India
15. (US) Central US
16. (US) Central US EUAP
17. (US) Central US (Stage)
18. (Asia Pacific) East Asia
19. (Asia Pacific) East Asia (Stage)
20. (US) East US
21. (US) East US 2
22. (US) East US 2 EUAP
23. (US) East US 2 (Stage)
24. (US) East US (Stage)
25. Europe
26. (Europe) France Central
27. (Europe) France South
28. (Europe) Germany North
29. (Europe) Germany West Central
30. Global
31. India
32. Japan
33. (Asia Pacific) Japan East
34. (Asia Pacific) Japan West
35. (Asia Pacific) Jio India Central
36. (Asia Pacific) Jio India West
37. (Asia Pacific) Korea Central
38. (Asia Pacific) Korea South
39. (US) North Central US
40. (US) North Central US (Stage)
41. (Europe) North Europe
42. (Europe) Norway East
43. (Europe) Norway West
44. (Europe) Qatar Central
45. (Africa) South Africa North
46. (Africa) South Africa West
47. (US) South Central US
48. (US) South Central US (Stage)
49. (Asia Pacific) Southeast Asia
50. (Asia Pacific) Southeast Asia (Stage)
51. (Asia Pacific) South India
52. (Europe) Sweden Central
53. (Europe) Sweden South
54. (Europe) Switzerland North
55. (Europe) Switzerland West
56. (Middle East) UAE Central
57. (Middle East) UAE North
58. United Kingdom
59. (Europe) UK South
60. (Europe) UK West
61. United States
62. (US) West Central US
63. (Europe) West Europe
64. (Asia Pacific) West India
65. (US) West US
66. (US) West US 2
67. (US) West US 2 (Stage)
68. (US) West US 3
69. (US) West US (Stage)
Enter the number of your desired region
[20]
:
TASK [cloud-azure : pause] ***********************************************************************************
ok: [localhost]
TASK [cloud-azure : set_fact] ********************************************************************************
ok: [localhost]
TASK [cloud-azure : Create AlgoVPN Server] *******************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'BlobServiceClient' from 'azure.storage.blob' (/root/algo/.env/lib/python3.10/site-packages/azure/storage/blob/__init__.py)
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on DESKTOP-STQVHKJ's Python /root/algo/.env/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}
TASK [include_tasks] *****************************************************************************************
included: /root/algo/playbooks/rescue.yml for localhost
TASK [debug] *************************************************************************************************
ok: [localhost] => {
"fail_hint": [
"Sorry, but something went wrong!",
"Please check the troubleshooting guide.",
"https://trailofbits.github.io/algo/troubleshooting.html"
]
}
TASK [Fail the installation] *********************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed as requested from task"}
PLAY RECAP ***************************************************************************************************
localhost : ok=31 changed=1 unreachable=0 failed=1 skipped=1 rescued=1 ignored=0
The output of ./algo -vvv, but only the part where the error is raised (the rest is too long for me to find and eliminate sensitive bits):
TASK [cloud-azure : Create AlgoVPN Server] *******************************************************************task path: /root/algo/roles/cloud-azure/tasks/main.yml:14
The full traceback is:
Traceback (most recent call last):
File "/tmp/ansible_azure_rm_deployment_payload_lv4gewn4/ansible_azure_rm_deployment_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 242, in <module>
from azure.storage.blob import BlobServiceClient
ImportError: cannot import name 'BlobServiceClient' from 'azure.storage.blob' (/root/algo/.env/lib/python3.10/site-packages/azure/storage/blob/__init__.py)
fatal: [localhost]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"ad_user": null,
"adfs_authority_url": null,
"api_profile": "latest",
"append_tags": true,
"auth_source": "auto",
"cert_validation_mode": null,
"client_id": "",
"cloud_environment": "AzureCloud",
"deployment_mode": "incremental",
"deployment_name": "algo",
"location": "germanywestcentral",
"log_mode": null,
"log_path": null,
"name": "algo",
"parameters": {
"SshPort": {
"value": 4160
},
"UserData": {
"value": "..."
},
"WireGuardPort": {
"value": 51820
},
"imageReferenceOffer": {
"value": "0001-com-ubuntu-minimal-jammy-daily"
},
"imageReferencePublisher": {
"value": "Canonical"
},
"imageReferenceSku": {
"value": "minimal-22_04-daily-lts"
},
"imageReferenceVersion": {
"value": "latest"
},
"osDiskType": {
"value": "Standard_LRS"
},
"sshKeyData": {
"value": "..."
},
"vmSize": {
"value": "Standard_B1S"
}
},
"parameters_link": null,
"password": null,
"profile": null,
"resource_group": "algo",
"resource_group_name": "algo",
"secret": "",
"state": "present",
"subscription_id": "",
"tags": null,
"template": {
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json",
"contentVersion": "1.0.0.0",
"outputs": {
"publicIPAddresses": {
"type": "string",
"value": "[reference(resourceId('Microsoft.Network/publicIPAddresses',resourceGroup().name),providers('Microsoft.Network', 'publicIPAddresses').apiVersions[0]).ipAddress]"
}
},
"parameters": {
"SshPort": {
"type": "int"
},
"UserData": {
"type": "string"
},
"WireGuardPort": {
"type": "int"
},
"imageReferenceOffer": {
"type": "string"
},
"imageReferencePublisher": {
"type": "string"
},
"imageReferenceSku": {
"type": "string"
},
"imageReferenceVersion": {
"type": "string"
},
"osDiskType": {
"type": "string"
},
"sshKeyData": {
"type": "string"
},
"vmSize": {
"type": "string"
}
},
"resources": [
{
"apiVersion": "2015-06-15",
"location": "[resourceGroup().location]",
"name": "[resourceGroup().name]",
"properties": {
"securityRules": [
{
"name": "AllowSSH",
"properties": {
"access": "Allow",
"description": "Allow SSH",
"destinationAddressPrefix": "*",
"destinationPortRange": "[parameters('SshPort')]",
"direction": "Inbound",
"priority": 100,
"protocol": "Tcp",
"sourceAddressPrefix": "*",
"sourcePortRange": "*"
}
},
{
"name": "AllowIPSEC500",
"properties": {
"access": "Allow",
"description": "Allow UDP to port 500",
"destinationAddressPrefix": "*",
"destinationPortRange": "500",
"direction": "Inbound",
"priority": 110,
"protocol": "Udp",
"sourceAddressPrefix": "*",
"sourcePortRange": "*"
}
},
{
"name": "AllowIPSEC4500",
"properties": {
"access": "Allow",
"description": "Allow UDP to port 4500",
"destinationAddressPrefix": "*",
"destinationPortRange": "4500",
"direction": "Inbound",
"priority": 120,
"protocol": "Udp",
"sourceAddressPrefix": "*",
"sourcePortRange": "*"
}
},
{
"name": "AllowWireGuard",
"properties": {
"access": "Allow",
"description": "Locks inbound down to ssh default port 22.",
"destinationAddressPrefix": "*",
"destinationPortRange": "[parameters('WireGuardPort')]",
"direction": "Inbound",
"priority": 130,
"protocol": "Udp",
"sourceAddressPrefix": "*",
"sourcePortRange": "*"
}
}
]
},
"type": "Microsoft.Network/networkSecurityGroups"
},
{
"apiVersion": "2015-06-15",
"location": "[resourceGroup().location]",
"name": "[resourceGroup().name]",
"properties": {
"publicIPAllocationMethod": "Static"
},
"type": "Microsoft.Network/publicIPAddresses"
},
{
"apiVersion": "2015-06-15",
"location": "[resourceGroup().location]",
"name": "[resourceGroup().name]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"10.10.0.0/16"
]
},
"subnets": [
{
"name": "[resourceGroup().name]",
"properties": {
"addressPrefix": "10.10.0.0/24"
}
}
]
},
"type": "Microsoft.Network/virtualNetworks"
},
{
"apiVersion": "2015-06-15",
"dependsOn": [
"[concat('Microsoft.Network/networkSecurityGroups/', resourceGroup().name)]",
"[concat('Microsoft.Network/publicIPAddresses/', resourceGroup().name)]",
"[concat('Microsoft.Network/virtualNetworks/', resourceGroup().name)]"
],
"location": "[resourceGroup().location]",
"name": "[resourceGroup().name]",
"properties": {
"ipConfigurations": [
{
"name": "ipconfig1",
"properties": {
"privateIPAllocationMethod": "Dynamic",
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses', resourceGroup().name)]"
},
"subnet": {
"id": "[variables('subnet1Ref')]"
}
}
}
],
"networkSecurityGroup": {
"id": "[resourceId('Microsoft.Network/networkSecurityGroups', resourceGroup().name)]"
}
},
"type": "Microsoft.Network/networkInterfaces"
},
{
"apiVersion": "2016-04-30-preview",
"dependsOn": [
"[concat('Microsoft.Network/networkInterfaces/', resourceGroup().name)]"
],
"location": "[resourceGroup().location]",
"name": "[resourceGroup().name]",
"properties": {
"hardwareProfile": {
"vmSize": "[parameters('vmSize')]"
},
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces', resourceGroup().name)]"
}
]
},
"osProfile": {
"adminUsername": "algo",
"computerName": "[resourceGroup().name]",
"customData": "[parameters('UserData')]",
"linuxConfiguration": {
"disablePasswordAuthentication": true,
"ssh": {
"publicKeys": [
{
"keyData": "[parameters('sshKeyData')]",
"path": "/home/algo/.ssh/authorized_keys"
}
]
}
}
},
"storageProfile": {
"imageReference": {
"offer": "[parameters('imageReferenceOffer')]",
"publisher": "[parameters('imageReferencePublisher')]",
"sku": "[parameters('imageReferenceSku')]",
"version": "[parameters('imageReferenceVersion')]"
},
"osDisk": {
"createOption": "FromImage",
"managedDisk": {
"storageAccountType": "[parameters('osDiskType')]"
}
}
}
},
"type": "Microsoft.Compute/virtualMachines"
}
],
"variables": {
"subnet1Ref": "[concat(variables('vnetID'),'/subnets/', resourceGroup().name)]",
"vnetID": "[resourceId('Microsoft.Network/virtualNetworks', resourceGroup().name)]"
}
},
"template_link": null,
"tenant": "",
"thumbprint": null,
"wait_for_deployment_completion": true,
"wait_for_deployment_polling_period": 10,
"x509_certificate_path": null
}
},
"msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on DESKTOP-STQVHKJ's Python /root/algo/.env/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"
}
Read vars_file 'config.cfg'
I've encountered with the same issue, and got it working with the change from this PR. https://github.com/trailofbits/algo/pull/14680
I still have this issue today. Though I'm running the command through ansible-playbook.
One additional comment, the README says to run this command on Ubuntu:
sudo apt install -y --no-install-recommends python3-virtualenv file lookup
When I execute this on Ubuntu 22.04.3 LTS, I receive this error:
sudo apt install -y --no-install-recommends python3-virtualenv file lookup
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package lookup
So I didn't install lookup as part of setting up the dependencies.
Full log of algo install afterwards:
~/algo$ ansible-playbook main.yml -e \
"provider=azure \
server_name=[REMOVED] \
ondemand_cellular=false \
ondemand_wifi=false \
dns_adblocking=true \
ssh_tunneling=false \
store_pki=false \
azure_secret=[REMOVED] \
azure_tenant=[REMOVED] \
azure_client_id=[REMOVED] \
azure_subscription_id=[REMOVED] \
region=eastus2"
PLAY [localhost] *******************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************
ok: [localhost]
TASK [Playbook dir stat] ***********************************************************************************************
ok: [localhost]
TASK [Ensure Ansible is not being run in a world writable directory] ***************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature will be removed from ansible.netcommon
in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
[WARNING]: The value '' is not a valid IP address or network, passing this value to ipaddr filter might result in
breaking change in future.
TASK [Ensure the requirements installed] *******************************************************************************
ok: [localhost]
TASK [Set required ansible version as a fact] **************************************************************************
ok: [localhost] => (item=ansible==9.1.0)
TASK [Just get the list from default pip] ******************************************************************************
ok: [localhost]
TASK [Verify Python meets Algo VPN requirements] ***********************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [Verify Ansible meets Algo VPN requirements] **********************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
[WARNING]: Found variable using reserved name: no_log
PLAY [Ask user for the input] ******************************************************************************************
TASK [Gathering Facts] *************************************************************************************************
ok: [localhost]
TASK [Set facts based on the input] ************************************************************************************
ok: [localhost]
TASK [Set facts based on the input] ************************************************************************************
ok: [localhost]
PLAY [Provision the server] ********************************************************************************************
TASK [Gathering Facts] *************************************************************************************************
ok: [localhost]
--> Please include the following block of text when reporting issues:
Algo running on: Ubuntu 22.04.3 LTS (Virtualized: microsoft)
Created from git clone. Last commit: 74051d0 Update README.md dependencies (#14634)
Python 3.10.12
Runtime variables:
algo_provider "azure"
algo_ondemand_cellular "False"
algo_ondemand_wifi "False"
algo_ondemand_wifi_exclude "X251bGw="
algo_dns_adblocking "True"
algo_ssh_tunneling "False"
wireguard_enabled "True"
dns_encryption "True"
TASK [Display the invocation environment] ******************************************************************************
changed: [localhost]
TASK [Install the requirements] ****************************************************************************************
changed: [localhost]
TASK [Generate the SSH private key] ************************************************************************************
changed: [localhost]
TASK [Generate the SSH public key] *************************************************************************************
changed: [localhost]
TASK [Copy the private SSH key to /tmp] ********************************************************************************
changed: [localhost]
TASK [Include a provisioning role] *************************************************************************************
TASK [cloud-azure : Install requirements] ******************************************************************************
changed: [localhost]
TASK [cloud-azure : set_fact] ******************************************************************************************
ok: [localhost]
TASK [cloud-azure : set_fact] ******************************************************************************************
ok: [localhost]
TASK [cloud-azure : Create AlgoVPN Server] *****************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'BlobServiceClient' from 'azure.storage.blob' (/home/[REMOVED]/algo/.env/lib/python3.10/site-packages/azure/storage/blob/__init__.py)
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on [REMOVED]'s Python /home/[REMOVED]/algo/.env/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}
TASK [include_tasks] ***************************************************************************************************
included: /home/[REMOVED]/algo/playbooks/rescue.yml for localhost
TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
"fail_hint": [
"Sorry, but something went wrong!",
"Please check the troubleshooting guide.",
"https://trailofbits.github.io/algo/troubleshooting.html"
]
}
TASK [Fail the installation] *******************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed as requested from task"}
PLAY RECAP *************************************************************************************************************
localhost : ok=22 changed=6 unreachable=0 failed=1 skipped=10 rescued=1 ignored=0
