Steampipe query unexpectedly reads from Stdin when parameterized queries

[RyanJarv]

Describe the bug When executing steampipe as apart of a while loop, with a static query passed on the command line steampipe will consume stdin input meant for the while loop.

Steampipe version (steampipe -v) Example: 0.13.6

To reproduce

steampipe % steampipe query --output json "SELECT table_name FROM information_schema.tables WHERE table_name LIKE 'aws%'"|jq '.[].table_name' -r | grep -v 'metric' | while read table; do steampipe query --output json "select * from $table"; done

This will run a 'select * from aws_accessanalyzer_analyzer' and then produces the following error.

Warning: executeQueries: query 2 of 2 failed: ERROR: syntax error at or near "aws_accountaws_acm_certificateaws_api_gateway_api_keyaws_api_gateway_authorizeraws_api_gateway_rest_apiaws_api_gateway_stageaws_api_gateway_usage_planaws_api_gatewayv2_apiaws_api_gatewayv2_domain_nameaws_api_gatewayv2_integrationaws_api_gatewayv2_stageaws_appautoscaling_targetaws_auditmanager_assessmentaws_auditmanager_controlaws_auditmanager_evidenceaws_auditmanager_evidence_folderaws_auditmanager_frameworkaws_availability_zoneaws_backup_planaws_backup_protected_resourceaws_backup_recovery_pointaws_backup_selectionaws_backup_vaultaws_cloudcontrol_resourceaws_cloudformation_stackaws_cloudfront_cache_policyaws_cloudfront_distributionaws_cloudfront_origin_access_identityaws_cloudfront_origin_request_policyaws_cloudtrail_trailaws_cloudtrail_trail_eventaws_cloudwatch_alarmaws_cloudwatch_log_eventaws_cloudwatch_log_groupaws_cloudwatch_log_resource_policyaws_cloudwatch_log_streamaws_codebuild_projectaws_codebuild_source_credentialaws_codecommit_repositoryaws_codepipeline_pipelineaws_config_configuration_recorderaws_config_conformance_packaws_config_ruleaws_cost_by_account_dailyaws_cost_by_account_monthlyaws_cost_by_service_dailyaws_cost_by_service_monthlyaws_cost_by_service_usage_type_dailyaws_cost_by_service_usage_type_monthlyaws_cost_forecast_dailyaws_cost_forecast_monthlyaws_cost_usageaws_dax_clusteraws_directory_service_directoryaws_dms_replication_instanceaws_dynamodb_backupaws_dynamodb_global_tableaws_dynamodb_tableaws_ebs_snapshotaws_ebs_volumeaws_ec2_amiaws_ec2_ami_sharedaws_ec2_application_load_balanceraws_ec2_autoscaling_groupaws_ec2_capacity_reservationaws_ec2_classic_load_balanceraws_ec2_gateway_load_balanceraws_ec2_instanceaws_ec2_instance_availabilityaws_ec2_instance_typeaws_ec2_key_pairaws_ec2_launch_configurationaws_ec2_load_balancer_listeneraws_ec2_managed_prefix_listaws_ec2_network_interfaceaws_ec2_network_load_balanceraws_ec2_regional_settingsaws_ec2_reserved_instanceaws_ec2_ssl_policyaws_ec2_target_groupaws_ec2_transit_gatewayaws_ec2_transit_gateway_routeaws_ec2_transit_gateway_route_tableaws_ec2_transit_gateway_vpc_attachmentaws_ecr_repositoryaws_ecrpublic_repositoryaws_ecs_clusteraws_ecs_container_instanceaws_ecs_serviceaws_ecs_taskaws_ecs_task_definitionaws_efs_access_pointaws_efs_file_systemaws_efs_mount_targetaws_eks_addonaws_eks_addon_versionaws_eks_clusteraws_eks_identity_provider_configaws_elastic_beanstalk_applicationaws_elastic_beanstalk_environmentaws_elasticache_clusteraws_elasticache_parameter_groupaws_elasticache_replication_groupaws_elasticache_subnet_groupaws_elasticsearch_domainaws_emr_clusteraws_emr_instance_groupaws_eventbridge_busaws_eventbridge_ruleaws_fsx_file_systemaws_glacier_vaultaws_glue_catalog_databaseaws_guardduty_detectoraws_guardduty_findingaws_guardduty_ipsetaws_guardduty_threat_intel_setaws_iam_access_advisoraws_iam_access_keyaws_iam_account_password_policyaws_iam_account_summaryaws_iam_actionaws_iam_credential_reportaws_iam_groupaws_iam_policyaws_iam_policy_attachmentaws_iam_policy_simulatoraws_iam_roleaws_iam_server_certificateaws_iam_useraws_iam_virtual_mfa_deviceaws_identitystore_groupaws_identitystore_useraws_inspector_assessment_targetaws_inspector_assessment_templateaws_kinesis_consumeraws_kinesis_firehose_delivery_streamaws_kinesis_streamaws_kinesis_video_streamaws_kinesisanalyticsv2_applicationaws_kms_keyaws_lambda_aliasaws_lambda_functionaws_lambda_layeraws_lambda_layer_versionaws_lambda_versionaws_macie2_classification_jobaws_media_store_containeraws_organizations_accountaws_rds_db_clusteraws_rds_db_cluster_parameter_groupaws_rds_db_cluster_snapshotaws_rds_db_event_subscriptionaws_rds_db_instanceaws_rds_db_option_groupaws_rds_db_parameter_groupaws_rds_db_snapshotaws_rds_db_subnet_groupaws_redshift_clusteraws_redshift_event_subscriptionaws_redshift_parameter_groupaws_redshift_snapshotaws_redshift_subnet_groupaws_regionaws_route53_domainaws_route53_recordaws_route53_resolver_endpointaws_route53_resolver_ruleaws_route53_zoneaws_s3_access_pointaws_s3_account_settingsaws_s3_bucketaws_sagemaker_endpoint_configurationaws_sagemaker_modelaws_sagemaker_notebook_instanceaws_sagemaker_training_jobaws_secretsmanager_secretaws_securityhub_hubaws_securityhub_productaws_securityhub_standards_subscriptionaws_serverlessapplicationrepository_applicationaws_sfn_state_machineaws_sfn_state_machine_executionaws_sfn_state_machine_execution_historyaws_sns_topicaws_sns_topic_subscriptionaws_sqs_queueaws_ssm_associationaws_ssm_documentaws_ssm_maintenance_windowaws_ssm_managed_instanceaws_ssm_managed_instance_complianceaws_ssm_parameteraws_ssm_patch_baselineaws_ssoadmin_instanceaws_ssoadmin_managed_policy_attachmentaws_ssoadmin_permission_setaws_tagging_resourceaws_vpcaws_vpc_customer_gatewayaws_vpc_dhcp_optionsaws_vpc_egress_only_internet_gatewayaws_vpc_eipaws_vpc_endpointaws_vpc_endpoint_serviceaws_vpc_flow_logaws_vpc_flow_log_eventaws_vpc_internet_gatewayaws_vpc_nat_gatewayaws_vpc_network_aclaws_vpc_peering_connectionaws_vpc_routeaws_vpc_route_tableaws_vpc_security_groupaws_vpc_security_group_ruleaws_vpc_subnetaws_vpc_vpn_connectionaws_vpc_vpn_gatewayaws_waf_rate_based_ruleaws_waf_ruleaws_wafv2_ip_setaws_wafv2_regex_pattern_setaws_wafv2_rule_groupaws_wafv2_web_aclaws_wellarchitected_workloadaws_workspaces_workspace" (SQLSTATE 42601)

Expected behavior

The expected behavior would be to close or avoid reading from stdin when the full SQL statement is already known, when it's been passed as a string or as a file path. This way it will avoid eating all the stdin intended for the read command.

Additional context If you've ever tried to automate something with ssh you've probably ran into the same issue, it's kind of just a limitation of shells in general. There's several ways to work around at the moment but the easiest is probably to close stdin by prepending 0<&- to the steampipe command.

steampipe query --output json "SELECT table_name FROM information_schema.tables WHERE table_name LIKE 'aws%'"|jq '.[].table_name' -r | grep -v 'metric' | while read table; do 0<&- steampipe query --output json "select * from $table" | tee "$table".json ; done;

fyi, looks like this was a side effect of #499

also I know this is a ridiculous use of steampipe, it's just I'm much more familiar with bash then SQL, for better or worse. My excuse for dumping everything is basically just to have a snapshot of the account and grep for random strings.

[kaidaguerre]

thanks for raising this @RyanJarv

let me have a dig and see if we can find a solution which maintains our stdin functionality but would work in a loop scenario

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.