iam-floyd icon indicating copy to clipboard operation
iam-floyd copied to clipboard

Published 20 hours ago • verified publisher icon udondan

autocomplete broken for python (pyright bug?)

Open rirze opened this issue 2 years ago • 6 comments

I'm using the package

  • [X] iam-floyd
  • [ ] cdk-iam-floyd

I'm using the package in language

  • [ ] TypeScript/JavaScript (npm)
  • [X] Python (pip)

Describe the problem When using the python package, I'm unable to get a full list of autocomplete functions for any service statement. Both of my setups depend on pyright, a language server that provides autocomplete recommendations as well as doing other functions.

Machine Details Currently on macOS Monterey (v12.1) using Python3.10

Emacs

My primary editor is Emacs and and I use a combination of lsp and pyright to obtain autocomplete. The following is a full list of autocomplete recommendations given to me within my emacs configuration. It's quite the list but at the second location, it doesn't have relevant ones like "on_bucket", "on_objects", etc..

Specifically for this situation:

statement.S3.on_bucket(bucket_name='some_bucket') 
#           ^loc1                               ^loc2
all autocomplete options reported by emacs + pyright at `loc1` 
all_actions  (Method)
all_list_actions  (Method)
all_matching_actions  (Method)
all_permission_management_actions  (Method)
all_read_actions  (Method)
all_tagging_actions  (Method)
all_write_actions  (Method)
allow  (Method)
compact  (Method)
deny  (Method)
effect  (Property)
for_  (Method)
for_account  (Method)
for_assumed_role_session  (Method)
for_canonical_user  (Method)
for_federated  (Method)
for_federated_amazon  (Method)
for_federated_cognito  (Method)
for_federated_facebook  (Method)
for_federated_google  (Method)
for_public  (Method)
for_role  (Method)
for_saml  (Method)
for_service  (Method)
for_user  (Method)
has_actions  (Method)
has_conditions  (Method)
has_principals  (Method)
has_resources  (Method)
if_  (Method)
if_access_point_network_origin  (Method)
if_auth_type  (Method)
if_aws_called_via  (Method)
if_aws_called_via_first  (Method)
if_aws_called_via_last  (Method)
if_aws_current_time  (Method)
if_aws_epoch_time  (Method)
if_aws_multi_factor_auth_age  (Method)
if_aws_multi_factor_auth_present  (Method)
if_aws_principal_account  (Method)
if_aws_principal_arn  (Method)
if_aws_principal_is_aws_service  (Method)
if_aws_principal_org_id  (Method)
if_aws_principal_org_paths  (Method)
if_aws_principal_service_name  (Method)
if_aws_principal_service_names_list  (Method)
if_aws_principal_tag  (Method)
if_aws_principal_type  (Method)
if_aws_referer  (Method)
if_aws_request_tag  (Method)
if_aws_requested_region  (Method)
if_aws_resource_tag  (Method)
if_aws_secure_transport  (Method)
if_aws_source_account  (Method)
if_aws_source_arn  (Method)
if_aws_source_identity  (Method)
if_aws_source_ip  (Method)
if_aws_source_vpc  (Method)
if_aws_source_vpce  (Method)
if_aws_tag_keys  (Method)
if_aws_token_issue_time  (Method)
if_aws_user_agent  (Method)
if_aws_userid  (Method)
if_aws_username  (Method)
if_aws_via_aws_service  (Method)
if_aws_vpc_source_ip  (Method)
if_data_access_point_account  (Method)
if_data_access_point_arn  (Method)
if_delimiter  (Method)
if_existing_job_operation  (Method)
if_existing_job_priority  (Method)
if_existing_object_tag  (Method)
if_job_suspended_cause  (Method)
if_location_constraint  (Method)
if_locationconstraint  (Method)
if_max_keys  (Method)
if_object_lock_legal_hold  (Method)
if_object_lock_mode  (Method)
if_object_lock_remaining_retention_days  (Method)
if_object_lock_retain_until_date  (Method)
if_prefix  (Method)
if_request_job_operation  (Method)
if_request_job_priority  (Method)
if_request_object_tag  (Method)
if_request_object_tag_keys  (Method)
if_resource_account  (Method)
if_signature_age  (Method)
if_signatureversion  (Method)
if_tls_version  (Method)
if_version_id  (Method)
if_versionid  (Method)
if_x_amz_acl  (Method)
if_x_amz_content_sha256  (Method)
if_x_amz_copy_source  (Method)
if_x_amz_grant_full_control  (Method)
if_x_amz_grant_read  (Method)
if_x_amz_grant_read_acp  (Method)
if_x_amz_grant_write  (Method)
if_x_amz_grant_write_acp  (Method)
if_x_amz_metadata_directive  (Method)
if_x_amz_object_ownership  (Method)
if_x_amz_server_side_encryption  (Method)
if_x_amz_server_side_encryption_aws_kms_key_id  (Method)
if_x_amz_storage_class  (Method)
if_x_amz_website_redirect_location  (Method)
not_actions  (Method)
not_principals  (Method)
not_resources  (Method)
on  (Method)
on_accesspoint  (Method)
on_all_resources  (Method)
on_bucket  (Method)
on_job  (Method)
on_multiregionaccesspoint  (Method)
on_multiregionaccesspointrequestarn  (Method)
on_object  (Method)
on_objectlambdaaccesspoint  (Method)
on_storagelensconfiguration  (Method)
service_prefix  (Property)
sid  (Property)
to  (Method)
to_abort_multipart_upload  (Method)
to_bypass_governance_retention  (Method)
to_create_access_point  (Method)
to_create_access_point_for_object_lambda  (Method)
to_create_bucket  (Method)
to_create_job  (Method)
to_create_multi_region_access_point  (Method)
to_delete_access_point  (Method)
to_delete_access_point_for_object_lambda  (Method)
to_delete_access_point_policy  (Method)
to_delete_access_point_policy_for_object_lambda  (Method)
to_delete_bucket  (Method)
to_delete_bucket_policy  (Method)
to_delete_bucket_website  (Method)
to_delete_job_tagging  (Method)
to_delete_multi_region_access_point  (Method)
to_delete_object  (Method)
to_delete_object_tagging  (Method)
to_delete_object_version  (Method)
to_delete_object_version_tagging  (Method)
to_delete_storage_lens_configuration  (Method)
to_delete_storage_lens_configuration_tagging  (Method)
to_describe_job  (Method)
to_describe_multi_region_access_point_operation  (Method)
to_get_accelerate_configuration  (Method)
to_get_access_point  (Method)
to_get_access_point_configuration_for_object_lambda  (Method)
to_get_access_point_for_object_lambda  (Method)
to_get_access_point_policy  (Method)
to_get_access_point_policy_for_object_lambda  (Method)
to_get_access_point_policy_status  (Method)
to_get_access_point_policy_status_for_object_lambda  (Method)
to_get_account_public_access_block  (Method)
to_get_analytics_configuration  (Method)
to_get_bucket_acl  (Method)
to_get_bucket_cors  (Method)
to_get_bucket_location  (Method)
to_get_bucket_logging  (Method)
to_get_bucket_notification  (Method)
to_get_bucket_object_lock_configuration  (Method)
to_get_bucket_ownership_controls  (Method)
to_get_bucket_policy  (Method)
to_get_bucket_policy_status  (Method)
to_get_bucket_public_access_block  (Method)
to_get_bucket_request_payment  (Method)
to_get_bucket_tagging  (Method)
to_get_bucket_versioning  (Method)
to_get_bucket_website  (Method)
to_get_encryption_configuration  (Method)
to_get_intelligent_tiering_configuration  (Method)
to_get_inventory_configuration  (Method)
to_get_job_tagging  (Method)
to_get_lifecycle_configuration  (Method)
to_get_metrics_configuration  (Method)
to_get_multi_region_access_point  (Method)
to_get_multi_region_access_point_policy  (Method)
to_get_multi_region_access_point_policy_status  (Method)
to_get_object  (Method)
to_get_object_acl  (Method)
to_get_object_legal_hold  (Method)
to_get_object_retention  (Method)
to_get_object_tagging  (Method)
to_get_object_torrent  (Method)
to_get_object_version  (Method)
to_get_object_version_acl  (Method)
to_get_object_version_for_replication  (Method)
to_get_object_version_tagging  (Method)
to_get_object_version_torrent  (Method)
to_get_replication_configuration  (Method)
to_get_storage_lens_configuration  (Method)
to_get_storage_lens_configuration_tagging  (Method)
to_get_storage_lens_dashboard  (Method)
to_json  (Method)
to_list_access_points  (Method)
to_list_access_points_for_object_lambda  (Method)
to_list_all_my_buckets  (Method)
to_list_bucket  (Method)
to_list_bucket_multipart_uploads  (Method)
to_list_bucket_versions  (Method)
to_list_jobs  (Method)
to_list_multi_region_access_points  (Method)
to_list_multipart_upload_parts  (Method)
to_list_storage_lens_configurations  (Method)
to_object_owner_override_to_bucket_owner  (Method)
to_put_accelerate_configuration  (Method)
to_put_access_point_configuration_for_object_lambda  (Method)
to_put_access_point_policy  (Method)
to_put_access_point_policy_for_object_lambda  (Method)
to_put_account_public_access_block  (Method)
to_put_analytics_configuration  (Method)
to_put_bucket_acl  (Method)
to_put_bucket_cors  (Method)
to_put_bucket_logging  (Method)
to_put_bucket_notification  (Method)
to_put_bucket_object_lock_configuration  (Method)
to_put_bucket_ownership_controls  (Method)
to_put_bucket_policy  (Method)
to_put_bucket_public_access_block  (Method)
to_put_bucket_request_payment  (Method)
to_put_bucket_tagging  (Method)
to_put_bucket_versioning  (Method)
to_put_bucket_website  (Method)
to_put_encryption_configuration  (Method)
to_put_intelligent_tiering_configuration  (Method)
to_put_inventory_configuration  (Method)
to_put_job_tagging  (Method)
to_put_lifecycle_configuration  (Method)
to_put_metrics_configuration  (Method)
to_put_multi_region_access_point_policy  (Method)
to_put_object  (Method)
to_put_object_acl  (Method)
to_put_object_legal_hold  (Method)
to_put_object_retention  (Method)
to_put_object_tagging  (Method)
to_put_object_version_acl  (Method)
to_put_object_version_tagging  (Method)
to_put_replication_configuration  (Method)
to_put_storage_lens_configuration  (Method)
to_put_storage_lens_configuration_tagging  (Method)
to_replicate_delete  (Method)
to_replicate_object  (Method)
to_replicate_tags  (Method)
to_restore_object  (Method)
to_statement_json  (Method)
to_update_job_priority  (Method)
to_update_job_status  (Method)
_access_level_list  (Property)
_actions  (Property)
_add_principal  (Method)
_cdk_applied  (Property)
_cdk_apply_principals  (Method)
_conditions  (Property)
_principals  (Property)
_resources  (Property)
_skip_auto_resource  (Property)
_use_not_principals  (Property)
__annotations__  (Variable)
__class__  (Property)
__delattr__  (Method)
__dict__  (Variable)
__dir__  (Method)
__doc__  (Variable)
__eq__  (Method)
__format__  (Method)
__getattribute__  (Method)
__hash__  (Method)
__init__  (Method)
__init_subclass__  (Method)
__module__  (Variable)
__ne__  (Method)
__new__  (Method)
__reduce__  (Method)
__reduce_ex__  (Method)
__repr__  (Method)
__setattr__  (Method)
__sizeof__  (Method)
__slots__  (Variable)
__str__  (Method)
all autocomplete options reported by emacs + pyright at `loc2` (this is a smaller list) 
all_actions  (Method)
all_list_actions  (Method)
all_matching_actions  (Method)
all_permission_management_actions  (Method)
all_read_actions  (Method)
all_tagging_actions  (Method)
all_write_actions  (Method)
allow  (Method)
compact  (Method)
deny  (Method)
effect  (Property)
has_actions  (Method)
has_conditions  (Method)
has_principals  (Method)
has_resources  (Method)
if_  (Method)
if_aws_called_via  (Method)
if_aws_called_via_first  (Method)
if_aws_called_via_last  (Method)
if_aws_current_time  (Method)
if_aws_epoch_time  (Method)
if_aws_multi_factor_auth_age  (Method)
if_aws_multi_factor_auth_present  (Method)
if_aws_principal_account  (Method)
if_aws_principal_arn  (Method)
if_aws_principal_is_aws_service  (Method)
if_aws_principal_org_id  (Method)
if_aws_principal_org_paths  (Method)
if_aws_principal_service_name  (Method)
if_aws_principal_service_names_list  (Method)
if_aws_principal_tag  (Method)
if_aws_principal_type  (Method)
if_aws_referer  (Method)
if_aws_request_tag  (Method)
if_aws_requested_region  (Method)
if_aws_resource_tag  (Method)
if_aws_secure_transport  (Method)
if_aws_source_account  (Method)
if_aws_source_arn  (Method)
if_aws_source_identity  (Method)
if_aws_source_ip  (Method)
if_aws_source_vpc  (Method)
if_aws_source_vpce  (Method)
if_aws_tag_keys  (Method)
if_aws_token_issue_time  (Method)
if_aws_user_agent  (Method)
if_aws_userid  (Method)
if_aws_username  (Method)
if_aws_via_aws_service  (Method)
if_aws_vpc_source_ip  (Method)
not_actions  (Method)
not_resources  (Method)
on  (Method)
on_all_resources  (Method)
service_prefix  (Property)
sid  (Property)
to  (Method)
to_json  (Method)
to_statement_json  (Method)
_access_level_list  (Property)
_actions  (Property)
_conditions  (Property)
_resources  (Property)
_skip_auto_resource  (Property)
__annotations__  (Variable)
__class__  (Property)
__delattr__  (Method)
__dict__  (Variable)
__dir__  (Method)
__doc__  (Variable)
__eq__  (Method)
__format__  (Method)
__getattribute__  (Method)
__hash__  (Method)
__init__  (Method)
__init_subclass__  (Method)
__module__  (Variable)
__ne__  (Method)
__new__  (Method)
__reduce__  (Method)
__reduce_ex__  (Method)
__repr__  (Method)
__setattr__  (Method)
__sizeof__  (Method)
__slots__  (Variable)
__str__  (Method)

As you could inductively guess, any further chaining only show the reduced list of autocomplete recommendations as well. I would like to see the full list.

VSCode

VSCode is performing oddly as well. I have the standard Python extension installed and this allows use of pyright+pylance for Python files. Specifically when I type:

statement.S3.on_bucket(bucket_name='some_bucket') 
#           ^loc1                               ^loc2

I'm able to get all relevant autocomplete at loc1 but I get nothing at loc2. Even though the return type is S3, for some reason the language server is unable to pick up any recommendations.

Final notes

I was wondering if you had any insight into this-- is this a local issue or one that's reproducible on other machines? If it ends being a package issue, do you think the way you're generating functions and translating them into usable Python APIs may be interfering with the autocomplete here?

Overall, I am impressed with the expressibility of this library-- I'd love to have working autocomplete recommendations in my Python setup before adopting it fully. Thanks.

rirze avatar Jan 06 '22 20:01 rirze

Hi Sourabh,

thanks for the detailed report. I think you just forgot to add parentheses behind the S3 class. Try

statement.S3().on_bucket(bucket_name='some_bucket')

instead of

statement.S3.on_bucket(bucket_name='some_bucket')

I'm not too deep into Python anymore, but I think w/o the parentheses you directly access the class methods without instantiating an object. All the methods return self... not sure what self would refer to, when it's not an object method. Apparently something that does not have the expected methods . 😸

Cheers, Daniel

udondan avatar Jan 09 '22 15:01 udondan

Thanks for the reply and catching that typo.

I've tested it again on my personal machine and I run into a very similar error. This time, I observed a pattern:

Case 1 (works)

When typing:

statement.S3().on_bucket(bucket_name='some_bucket') 
#             ^loc1                               ^loc2

At loc1, I obtain the full list of methods/attributes. At loc2, I still receive a full list of options.

Case 2 (doesn't work)

When typing:

statement.S3().all_actions()
#             ^loc1        ^loc2

At loc1, I obtain the full list of methods/attributes. At loc2, I only receive the limited list of attributes and functions. (Same as documented in the original ticket). For example, on_bucket does not appear as an autocomplete option at loc2 but all the generic service functions do show.

Notes

Could this be because .all_actions() returns the parent class to S3 (i assume the parent class of all aws service objects?), and so it doesn't show the options for the service specific class?

rirze avatar Jan 09 '22 17:01 rirze

Ok, I think I found out why this is the case-- this is probably an effect from the situation you described on your ticket on the jsii project. I'm guessing other typed language servers are able to derive the inherited type from return this;, but somehow the Python language (pyright) is not. I'll try to see if there's something I can do from the language server side.

rirze avatar Jan 09 '22 19:01 rirze

I think I found the crux of the issue. While the original TS code doesn't specify a return type for any method that returns this, jsii takes the liberty to apply the return type to be the class in the generated Python code. The language server then only shows methods/attributes from the superclass because it expects the return object to be of type(superclass).

I wrote a ticket to jsii to see if there's a way to generate the Python code with this in mind. https://github.com/aws/jsii/issues/3330

rirze avatar Jan 09 '22 23:01 rirze

Thanks so much for digging so deep. Yes, now I remember the this issue.

I think I will have to look into other ways to generate the code. jsii is great, if you don't work with this/self. I already dropped support for Java and .Net because the resulting libraries were broken, because of the same exact issue.

jsii and Python also is the reason why this lib is still in 0.x state, because I wasn't able to get some imports right when working on full IAM policy support.

Maybe generating Python code via ast is the way to go. Of course this is a huge chunk of work you shouldn't be waiting for. 🙂

udondan avatar Jan 10 '22 06:01 udondan

I've thought about a short-term solution, but I don't know how you would feel about it:

What if we ran a string-replace on the jsii-generated Python __init__.py (that contains the all the relevant classes) and delete all the explicit -> "PolicyStatement" strings? That way, the Python files would not restrict autocomplete options to the current class.

I would imagine we'd need to run this replace command for each of the classes like you listed on the jsii ticket: cdk.Iam.PolicyStatement <- PolicyStatementBase <- PolicyStatementWithCondition <- PolicyStatementWithActions <- PolicyStatementWithResources <- PolicyStatementWithEffect <- PolicyStatementWithPrincipal <- PolicyStatementWithCDKPrincipal <- PolicyStatement

This isn't the solution I would like (preferably would like jsii to make this possible in the generation step), but I know it would definitely work. It would involve untar'ing the jsii-generated Python source tarball, running the string replace commands, and then bundling it again into a .whl and .tar.gz file.

If you'd like, I could write a PR integrating a shell script that does the above steps into the Makefile.

rirze avatar Jan 10 '22 20:01 rirze

Closing this as I have dropped jsii and with that Python support completely. The project was completely stuck for a year because of jsii problems and i decided to rather build a good TypeScript project than to deal with jsii issues over and over again.

udondan avatar Feb 17 '24 08:02 udondan