AWS plugin for Steampipe

Overview

AWS Plugin for Steampipe

Query AWS with SQL

Use SQL to query IAM users, EC2 instances and more from your AWS account. For example:

select
  name,
  user_id,
  path,
  create_date,
  password_last_used
from
  aws_iam_user;

Learn about Steampipe.

Get started

Table documentation and examples →

Install the plugin:

steampipe plugin install aws

Get involved

Community

The Steampipe community can be found on GitHub Discussions, where you can ask questions, voice ideas, and share your projects.

Our Code of Conduct applies to all Steampipe community channels.

Contributing

Please see CONTRIBUTING.md.

Comments
  • ERROR: failed to start plugin 'aws': error reading from server: EOF (SQLSTATE HV000)

    ERROR: failed to start plugin 'aws': error reading from server: EOF (SQLSTATE HV000)

    Describe the bug I am running a custom benchmark against AWS accounts. At some point in time, Steampipe will just hang in the middle of a benchmark. The tracing logs show the following error message: query failed with plugin connectivity error ERROR: failed to start plugin 'aws': error reading from server: EOF (SQLSTATE HV000) - retrying... This is then followed by ERROR: failed to start plugin 'aws': connection error: desc = "transport: error while dialing: dial unix /tmp/plugin157052978: connect: connection refused" (SQLSTATE HV000) - retrying...

    Steampipe version (steampipe -v) v0.14.6

    Plugin version (steampipe plugin list) latest -> 0.61.0

    To reproduce This is the difficult part: the error is not deterministic. Out of seven execution runs, this problems will appear usually once, for a random AWS account.

    Steampipe is being executed in a loop over multiple accounts inside a container environment. The structure is basically like this:

    steampipe plugin install aws
    for accountid in $ACCOUNTLIST
     # generate steampipe config file of the following structure
     # connection "aws" {
     #    plugin  = "aws"
     #    profile = "$accountid"
     #    regions = ["*"]
     #  }
      steampipe \
        check benchmark.mybenchmark \
        --theme plain \
        --progress=false \
        --output json \
        > /output/$accountid.json || true
    done
    
    
    

    This is using AWS assume role with a particular role_arn for each AWS account.

    Expected behavior Steampipe does not hang.

    Additional context

    The relevant parts of the tracing log for the AWS plugin installation and the benchmark execution where this problem occurs.

    2022-05-31 18:45:53.159 UTC [TRACE] steampipe: ociDownloader.Download: downloading us-docker.pkg.dev/steampipe/plugins/turbot/aws:latest
    2022-05-31 18:45:53.159 UTC [TRACE] steampipe: ociDownloader.Pull: pulling us-docker.pkg.dev/steampipe/plugins/turbot/aws:latest
    2022-05-31 18:45:53.717 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.plugin.linux-amd64.layer.v1+gzip
    2022-05-31 18:45:53.717 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.plugin.docs.layer.v1+tar
    2022-05-31 18:45:53.717 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.plugin.spc.layer.v1+tar
    2022-05-31 18:45:53.717 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.plugin.license.layer.v1+text
                
    2022-05-31 18:45:54.678 UTC [TRACE] steampipe: GetRunStatus - loadRunningInstanceInfo returned nil
    2022-05-31 18:45:54.678 UTC [TRACE] steampipe: calling removeRunningInstanceInfo
    2022-05-31 18:45:54.679 UTC [TRACE] steampipe: ociDownloader.Download: downloading us-docker.pkg.dev/steampipe/steampipe/db:14.2.0
    2022-05-31 18:45:54.679 UTC [TRACE] steampipe: ociDownloader.Pull: pulling us-docker.pkg.dev/steampipe/steampipe/db:14.2.0
    2022-05-31 18:45:57.736 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.db.linux-amd64.layer.v1+tar
    2022-05-31 18:45:57.736 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.db.doc.layer.v1+text
    2022-05-31 18:45:57.740 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.db.license.layer.v1+text
    2022-05-31 18:45:57.837 UTC [TRACE] steampipe: GetRunStatus - loadRunningInstanceInfo returned nil
    2022-05-31 18:45:57.838 UTC [TRACE] steampipe: ociDownloader.Download: downloading us-docker.pkg.dev/steampipe/steampipe/fdw:1.1.1
    2022-05-31 18:45:57.838 UTC [TRACE] steampipe: ociDownloader.Pull: pulling us-docker.pkg.dev/steampipe/steampipe/fdw:1.1.1
    2022-05-31 18:45:58.278 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.fdw.linux-amd64.layer.v1+gzip
    2022-05-31 18:45:58.278 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.fdw.control.layer.v1+text
    2022-05-31 18:45:58.278 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.fdw.sql.layer.v1+text
    2022-05-31 18:45:58.278 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.fdw.doc.layer.v1+text
    2022-05-31 18:45:58.278 UTC [TRACE] steampipe: looking for application/vnd.turbot.steampipe.fdw.license.layer.v1+text
    2022-05-31 18:45:59.474 UTC [TRACE] steampipe: initdb start: /home/steampipe/.steampipe/db/14.2.0/postgres/bin/initdb --auth=trust --username=root --pgdata=/home/steampipe/.steampipe/db/14.2.0/data --encoding=UTF-8 --wal-segsize=1 --debug
    2022-05-31 18:46:07.229 UTC [TRACE] steampipe: Connection string:  host=localhost port=40787 user=root dbname=postgres sslmode=disable
    2022-05-31 18:46:07.244 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.244 UTC [39] LOG:  starting PostgreSQL 14.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4, 64-bit
    2022-05-31 18:46:07.245 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.245 UTC [39] LOG:  listening on IPv4 address "127.0.0.1", port 40787
    2022-05-31 18:46:07.245 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.245 UTC [39] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
    2022-05-31 18:46:07.248 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.248 UTC [39] LOG:  listening on Unix socket "/tmp/.s.PGSQL.40787"
    2022-05-31 18:46:07.250 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.250 UTC [39] LOG:  hostssl record cannot match because SSL is disabled
    2022-05-31 18:46:07.251 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.250 UTC [39] HINT:  Set ssl = on in postgresql.conf.
    2022-05-31 18:46:07.251 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.250 UTC [39] CONTEXT:  line 2 of configuration file "/home/steampipe/.steampipe/db/14.2.0/data/pg_hba.conf"
    2022-05-31 18:46:07.271 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.271 UTC [40] LOG:  database system was shut down at 2022-05-31 18:46:05 UTC
    2022-05-31 18:46:07.275 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.275 UTC [39] LOG:  database system is ready to accept connections
    2022-05-31 18:46:07.287 UTC [TRACE] steampipe: valid database name: steampipe
    2022-05-31 18:46:07.287 UTC [TRACE] steampipe: installing database with name steampipe
    2022-05-31 18:46:07.846 UTC [TRACE] steampipe: Install Foreign Server:  drop extension if exists "steampipe_postgres_fdw" cascade
    2022-05-31 18:46:07.846 UTC [TRACE] steampipe: Install Foreign Server:  create extension if not exists "steampipe_postgres_fdw"
    2022-05-31 18:46:07.872 UTC [TRACE] steampipe: SERVICE: 2022/05/31 18:46:07 [INFO]
    2022-05-31 18:46:07.872 UTC [TRACE] steampipe: SERVICE: ******************************************************
    2022-05-31 18:46:07.872 UTC [TRACE] steampipe: SERVICE: 		steampipe postgres fdw init
    2022-05-31 18:46:07.872 UTC [TRACE] steampipe: SERVICE: ******************************************************
    2022-05-31 18:46:07.872 UTC [TRACE] steampipe: SERVICE: 2022/05/31 18:46:07 [INFO] Log level trace
    2022-05-31 18:46:07.878 UTC [TRACE] steampipe: Install Foreign Server:  create server "steampipe" foreign data wrapper "steampipe_postgres_fdw"
    2022-05-31 18:46:07.879 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.879 UTC [39] LOG:  received smart shutdown request
    2022-05-31 18:46:07.881 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.880 UTC [TRACE] hub: hub: close
    2022-05-31 18:46:07.886 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.886 UTC [39] LOG:  background worker "logical replication launcher" (PID 46) exited with exit code 1
    2022-05-31 18:46:07.927 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.927 UTC [41] LOG:  shutting down
    2022-05-31 18:46:07.988 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:07.988 UTC [39] LOG:  database system is shut down
    2022-05-31 18:46:08.030 UTC [TRACE] steampipe: GetRunStatus - loadRunningInstanceInfo returned nil
    2022-05-31 18:46:08.030 UTC [TRACE] steampipe: StartDB invoker plugin
    2022-05-31 18:46:08.030 UTC [TRACE] steampipe: StartDB started plugin manager
    2022-05-31 18:46:08.030 UTC [TRACE] steampipe: loadRootPrivateKey - failed to load key from /home/steampipe/.steampipe/db/14.2.0/data/root.key: open /home/steampipe/.steampipe/db/14.2.0/data/root.key: no such file or directory
    2022-05-31 18:46:10.335 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=postgres sslmode=disable
    2022-05-31 18:46:10.379 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:10.379 UTC [54] LOG:  redirecting log output to logging collector process
    2022-05-31 18:46:10.379 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:46:10.379 UTC [54] HINT:  Future log output will appear in directory "/home/steampipe/.steampipe/logs".
    2022-05-31 18:46:10.428 UTC [TRACE] steampipe: status:  {"pid":54,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"plugin","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:46:10.428 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=postgres sslmode=require
    2022-05-31 18:46:10.529 UTC [TRACE] steampipe: status:  {"pid":54,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"plugin","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:46:10.529 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:46:10.630 UTC [TRACE] steampipe: Install Foreign Server:  drop extension if exists "steampipe_postgres_fdw" cascade
    2022-05-31 18:46:10.630 UTC [TRACE] steampipe: Install Foreign Server:  create extension if not exists "steampipe_postgres_fdw"
    2022-05-31 18:46:10.645 UTC [TRACE] steampipe: Install Foreign Server:  create server "steampipe" foreign data wrapper "steampipe_postgres_fdw"
    2022-05-31 18:46:10.685 UTC [TRACE] steampipe: plugin manager state file not found
    2022-05-31 18:46:10.685 UTC [TRACE] steampipe: plugin manager state file not found
    2022-05-31 18:46:10.743 UTC [TRACE] steampipe: start: started plugin manager, pid 70
    2022-05-31 18:46:10.846 UTC [TRACE] steampipe: created local client 0xc0003cd940
    2022-05-31 18:46:10.846 UTC [TRACE] steampipe: getSchemaHashesForDynamicSchemas
    2022-05-31 18:46:10.846 UTC [TRACE] steampipe: CreateConnectionPlugin creating 0 connections
    2022-05-31 18:46:10.846 UTC [TRACE] steampipe: plugin manager is running - returning client
    2022-05-31 18:46:10.848 UTC [TRACE] steampipe: fetched schema for 0 dynamic plugins
    2022-05-31 18:46:10.848 UTC [TRACE] steampipe: connection aws is out of date or missing
    2022-05-31 18:46:10.848 UTC [TRACE] steampipe: CreateConnectionPlugin creating 1 connections
    2022-05-31 18:46:10.849 UTC [TRACE] steampipe: plugin manager is running - returning client
    2022-05-31 18:46:10.942 UTC [TRACE] steampipe: plugin manager returned reattach config for connection 'aws' - pid 78, reattach protocol:"grpc" addr:{Network:"unix" Address:"/tmp/plugin3105144512"} pid:78
    2022-05-31 18:46:10.942 UTC [TRACE] steampipe: NewPluginClient for plugin hub.steampipe.io/plugins/turbot/[email protected]
    2022-05-31 18:46:11.235 UTC [TRACE] steampipe: created connection plugin for connection: 'aws', pluginName: 'hub.steampipe.io/plugins/turbot/[email protected]'
    2022-05-31 18:46:11.253 UTC [TRACE] steampipe: status:  {"pid":54,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"plugin","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:46:11.255 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:46:11.333 UTC [TRACE] steampipe: executeConnectionUpdateQueries: num updates 1
    2022-05-31 18:46:11.333 UTC [TRACE] steampipe: executing update query 0 of 1 for connection 'aws'
    2022-05-31 18:46:11.835 UTC [TRACE] steampipe: all update queries executed
    2022-05-31 18:46:11.835 UTC [TRACE] steampipe: executing comment query 0 of 1 for plugin 'aws'
    2022-05-31 18:46:12.173 UTC [TRACE] steampipe: executeUpdateQueries complete
    2022-05-31 18:46:12.174 UTC [TRACE] steampipe: status:  {"pid":54,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"plugin","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:46:12.174 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:46:12.241 UTC [TRACE] steampipe: setting user search path to [public aws internal]
    2022-05-31 18:46:12.245 UTC [TRACE] steampipe: user search path sql: alter user "steampipe" set search_path to "public","aws","internal";
    2022-05-31 18:46:12.245 UTC [TRACE] steampipe: status:  {"pid":54,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"plugin","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:46:12.245 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:46:12.342 UTC [TRACE] steampipe: close local client 0xc0003cd940
    2022-05-31 18:46:12.342 UTC [TRACE] steampipe: local client not NIL
    2022-05-31 18:46:12.343 UTC [TRACE] steampipe: DbClient.Close &{0 0xc0009f74a0 0 {0 0} [0xc00019d440] map[] 0 1 0xc0006b8000 false map[0xc00019d440:map[0xc00019d440:true]] map[] 5 5 0 0 <nil> 0 0 0 0 0x4907a0}
    2022-05-31 18:46:12.343 UTC [TRACE] steampipe: local client close complete
    2022-05-31 18:46:12.343 UTC [TRACE] steampipe: shutdown local service plugin
    2022-05-31 18:46:12.345 UTC [TRACE] steampipe: status:  {"pid":54,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"plugin","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:46:12.345 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:46:12.432 UTC [TRACE] steampipe: StopDB invoker plugin, force false
    2022-05-31 18:46:12.436 UTC [TRACE] steampipe: plugin manager stop
    2022-05-31 18:46:12.436 UTC [TRACE] steampipe: pluginManager.Shutdown
    2022-05-31 18:46:12.436 UTC [TRACE] steampipe: PluginManagerClient Shutdown
    2022-05-31 18:46:12.449 UTC [TRACE] steampipe: pluginManager state.kill
    
    Installed plugin: [[email protected] v0.61.0
    Documentation:    https://hub.steampipe.io/plugins/turbot/aws
    
    [...]
    
    2022-05-31 18:50:18.982 UTC [TRACE] steampipe: modfile exists in workspace folder - creating pseudo-resources and loading files recursively
    2022-05-31 18:50:18.987 UTC [TRACE] steampipe: parse complete after 1 decode passes
    2022-05-31 18:50:18.993 UTC [TRACE] steampipe: parse complete after 2 decode passes
    2022-05-31 18:50:18.993 UTC [TRACE] steampipe: ensuring check export/output templates
    2022-05-31 18:50:18.996 UTC [TRACE] steampipe: GetRunStatus - loadRunningInstanceInfo returned nil
    2022-05-31 18:50:18.996 UTC [TRACE] steampipe: StartDB invoker check
    2022-05-31 18:50:18.996 UTC [TRACE] steampipe: StartDB started plugin manager
    2022-05-31 18:50:18.997 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=postgres sslmode=disable
    2022-05-31 18:50:19.034 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:50:19.034 UTC [2722] LOG:  redirecting log output to logging collector process
    2022-05-31 18:50:19.035 UTC [TRACE] steampipe: SERVICE: 2022-05-31 18:50:19.034 UTC [2722] HINT:  Future log output will appear in directory "/home/steampipe/.steampipe/logs".
    2022-05-31 18:50:19.072 UTC [TRACE] steampipe: status:  {"pid":2722,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"check","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:50:19.072 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=postgres sslmode=require
    2022-05-31 18:50:19.144 UTC [TRACE] steampipe: status:  {"pid":2722,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"check","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:50:19.144 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:50:19.220 UTC [TRACE] steampipe: plugin manager state file not found
    2022-05-31 18:50:19.220 UTC [TRACE] steampipe: plugin manager state file not found
    2022-05-31 18:50:19.286 UTC [TRACE] steampipe: start: started plugin manager, pid 2740
    2022-05-31 18:50:19.361 UTC [TRACE] steampipe: created local client 0xc0001c7ac0
    2022-05-31 18:50:19.361 UTC [TRACE] steampipe: getSchemaHashesForDynamicSchemas
    2022-05-31 18:50:19.361 UTC [TRACE] steampipe: CreateConnectionPlugin creating 0 connections
    2022-05-31 18:50:19.361 UTC [TRACE] steampipe: plugin manager is running - returning client
    2022-05-31 18:50:19.365 UTC [TRACE] steampipe: fetched schema for 0 dynamic plugins
    2022-05-31 18:50:19.365 UTC [TRACE] steampipe: connection aws is out of date or missing
    2022-05-31 18:50:19.365 UTC [TRACE] steampipe: CreateConnectionPlugin creating 1 connections
    2022-05-31 18:50:19.366 UTC [TRACE] steampipe: plugin manager is running - returning client
    2022-05-31 18:50:19.420 UTC [TRACE] steampipe: plugin manager returned reattach config for connection 'aws' - pid 2748, reattach protocol:"grpc" addr:{Network:"unix" Address:"/tmp/plugin157052978"} pid:2748
    2022-05-31 18:50:19.420 UTC [TRACE] steampipe: NewPluginClient for plugin hub.steampipe.io/plugins/turbot/[email protected]
    2022-05-31 18:50:19.586 UTC [TRACE] steampipe: created connection plugin for connection: 'aws', pluginName: 'hub.steampipe.io/plugins/turbot/[email protected]'
    2022-05-31 18:50:19.604 UTC [TRACE] steampipe: status:  {"pid":2722,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"check","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:50:19.604 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:50:19.664 UTC [TRACE] steampipe: executeConnectionUpdateQueries: num updates 1
    2022-05-31 18:50:19.664 UTC [TRACE] steampipe: executing update query 0 of 1 for connection 'aws'
    2022-05-31 18:50:20.056 UTC [TRACE] steampipe: all update queries executed
    2022-05-31 18:50:20.056 UTC [TRACE] steampipe: executing comment query 0 of 1 for plugin 'aws'
    2022-05-31 18:50:20.298 UTC [TRACE] steampipe: executeUpdateQueries complete
    2022-05-31 18:50:20.299 UTC [TRACE] steampipe: status:  {"pid":2722,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"check","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:50:20.299 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:50:20.384 UTC [TRACE] steampipe: setting user search path to [public aws internal]
    2022-05-31 18:50:20.389 UTC [TRACE] steampipe: user search path sql: alter user "steampipe" set search_path to "public","aws","internal";
    2022-05-31 18:50:20.389 UTC [TRACE] steampipe: status:  {"pid":2722,"port":9193,"listen":["localhost","127.0.0.1"],"listen_type":"local","invoker":"check","password":"XXXX-XXXX-XXXX","user":"steampipe","database":"steampipe","struct_version":20220411}
    2022-05-31 18:50:20.389 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=steampipe sslmode=require
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: begin ExecutionTree.Execute
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: begin ResultGroup.Execute: root_result_group
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: begin ResultGroup.Execute: mybenchmark.benchmark.mybenchmark_aws
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_compute_1
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_compute_1
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_network_1
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_network_1
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_iam_1
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_iam_1
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_iam_2
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_iam_2
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_iam_3
    2022-05-31 18:50:20.453 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_iam_3
    2022-05-31 18:50:20.492 UTC [TRACE] steampipe: ensureSessionSearchPath
    2022-05-31 18:50:20.492 UTC [TRACE] steampipe: updated the required search path to "public","aws","internal"
    2022-05-31 18:50:20.492 UTC [TRACE] steampipe: session search path will be updated to  "public","aws","internal"
    2022-05-31 18:50:20.495 UTC [TRACE] steampipe: CreatePreparedStatements
    2022-05-31 18:50:20.528 UTC [TRACE] steampipe: ensureSessionSearchPath
    2022-05-31 18:50:20.540 UTC [TRACE] steampipe: updated the required search path to "public","aws","internal"
    2022-05-31 18:50:20.540 UTC [TRACE] steampipe: session search path will be updated to  "public","aws","internal"
    2022-05-31 18:50:20.547 UTC [TRACE] steampipe: ensureSessionSearchPath
    2022-05-31 18:50:20.548 UTC [TRACE] steampipe: updated the required search path to "public","aws","internal"
    2022-05-31 18:50:20.549 UTC [TRACE] steampipe: session search path will be updated to  "public","aws","internal"
    2022-05-31 18:50:20.555 UTC [TRACE] steampipe: ensureSessionSearchPath
    2022-05-31 18:50:20.555 UTC [TRACE] steampipe: updated the required search path to "public","aws","internal"
    2022-05-31 18:50:20.555 UTC [TRACE] steampipe: session search path will be updated to  "public","aws","internal"
    2022-05-31 18:50:20.555 UTC [TRACE] steampipe: CreatePreparedStatements
    2022-05-31 18:50:20.562 UTC [TRACE] steampipe: CreatePreparedStatements
    2022-05-31 18:50:20.562 UTC [TRACE] steampipe: ensureSessionSearchPath
    2022-05-31 18:50:20.562 UTC [TRACE] steampipe: updated the required search path to "public","aws","internal"
    2022-05-31 18:50:20.562 UTC [TRACE] steampipe: session search path will be updated to  "public","aws","internal"
    2022-05-31 18:50:20.657 UTC [TRACE] steampipe: ResolveQueryFromQueryProvider for mybenchmark.control.mdbi_aws_compute_1
    2022-05-31 18:50:20.657 UTC [TRACE] steampipe: control defines inline SQL
    2022-05-31 18:50:20.657 UTC [TRACE] steampipe: GetPreparedStatementExecuteSQL source: mybenchmark.control.mdbi_aws_compute_1, sql: execute mybenchmark_mdbi_aws_compute_1_c80a1, args: <empty>
    2022-05-31 18:50:20.657 UTC [TRACE] steampipe: setting search path mybenchmark.control.mdbi_aws_compute_1
    2022-05-31 18:50:20.657 UTC [TRACE] steampipe: execute start for, mybenchmark.control.mdbi_aws_compute_1
    2022-05-31 18:50:20.668 UTC [TRACE] steampipe: CreatePreparedStatements
    2022-05-31 18:50:20.793 UTC [TRACE] steampipe: ResolveQueryFromQueryProvider for mybenchmark.control.mdbi_aws_network_1
    2022-05-31 18:50:20.793 UTC [TRACE] steampipe: control defines inline SQL
    2022-05-31 18:50:20.793 UTC [TRACE] steampipe: GetPreparedStatementExecuteSQL source: mybenchmark.control.mdbi_aws_network_1, sql: execute mybenchmark_mdbi_aws_network_1_cd58f, args: <empty>
    2022-05-31 18:50:20.793 UTC [TRACE] steampipe: setting search path mybenchmark.control.mdbi_aws_network_1
    2022-05-31 18:50:20.793 UTC [TRACE] steampipe: execute start for, mybenchmark.control.mdbi_aws_network_1
    2022-05-31 18:50:20.796 UTC [TRACE] steampipe: CreatePreparedStatements
    2022-05-31 18:50:20.834 UTC [TRACE] steampipe: ResolveQueryFromQueryProvider for mybenchmark.control.mdbi_aws_iam_1
    2022-05-31 18:50:20.835 UTC [TRACE] steampipe: control defines inline SQL
    2022-05-31 18:50:20.835 UTC [TRACE] steampipe: GetPreparedStatementExecuteSQL source: mybenchmark.control.mdbi_aws_iam_1, sql: execute mybenchmark_mdbi_aws_iam_1_c0ebd, args: <empty>
    2022-05-31 18:50:20.835 UTC [TRACE] steampipe: setting search path mybenchmark.control.mdbi_aws_iam_1
    2022-05-31 18:50:20.835 UTC [TRACE] steampipe: execute start for, mybenchmark.control.mdbi_aws_iam_1
    2022-05-31 18:50:20.906 UTC [TRACE] steampipe: ResolveQueryFromQueryProvider for mybenchmark.control.mdbi_aws_iam_3
    2022-05-31 18:50:20.906 UTC [TRACE] steampipe: control defines inline SQL
    2022-05-31 18:50:20.906 UTC [TRACE] steampipe: GetPreparedStatementExecuteSQL source: mybenchmark.control.mdbi_aws_iam_3, sql: execute mybenchmark_mdbi_aws_iam_3_c7c39, args: <empty>
    2022-05-31 18:50:20.906 UTC [TRACE] steampipe: setting search path mybenchmark.control.mdbi_aws_iam_3
    2022-05-31 18:50:20.906 UTC [TRACE] steampipe: execute start for, mybenchmark.control.mdbi_aws_iam_3
    2022-05-31 18:50:21.001 UTC [TRACE] steampipe: ResolveQueryFromQueryProvider for mybenchmark.control.mdbi_aws_iam_2
    2022-05-31 18:50:21.001 UTC [TRACE] steampipe: control defines inline SQL
    2022-05-31 18:50:21.001 UTC [TRACE] steampipe: GetPreparedStatementExecuteSQL source: mybenchmark.control.mdbi_aws_iam_2, sql: execute mybenchmark_mdbi_aws_iam_2_cc24e, args: <empty>
    2022-05-31 18:50:21.001 UTC [TRACE] steampipe: setting search path mybenchmark.control.mdbi_aws_iam_2
    2022-05-31 18:50:21.001 UTC [TRACE] steampipe: execute start for, mybenchmark.control.mdbi_aws_iam_2
    2022-05-31 18:50:21.070 UTC [TRACE] steampipe: execute finish for, mybenchmark.control.mdbi_aws_iam_3
    2022-05-31 18:50:21.070 UTC [TRACE] steampipe: control mybenchmark.control.mdbi_aws_iam_3 query failed with plugin connectivity error ERROR: failed to start plugin 'aws': error reading from server: EOF (SQLSTATE HV000) - retrying...
    2022-05-31 18:50:21.070 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_iam_3
    2022-05-31 18:50:21.070 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_iam_3
    2022-05-31 18:50:21.070 UTC [TRACE] steampipe: execute finish for, mybenchmark.control.mdbi_aws_iam_1
    2022-05-31 18:50:21.070 UTC [TRACE] steampipe: control mybenchmark.control.mdbi_aws_iam_1 query failed with plugin connectivity error ERROR: failed to start plugin 'aws': error reading from server: EOF (SQLSTATE HV000) - retrying...
    2022-05-31 18:50:21.070 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_iam_1
    2022-05-31 18:50:21.070 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_iam_1
    2022-05-31 18:50:21.071 UTC [TRACE] steampipe: execute finish for, mybenchmark.control.mdbi_aws_compute_1
    2022-05-31 18:50:21.071 UTC [TRACE] steampipe: control mybenchmark.control.mdbi_aws_compute_1 query failed with plugin connectivity error ERROR: failed to start plugin 'aws': error reading from server: EOF (SQLSTATE HV000) - retrying...
    2022-05-31 18:50:21.071 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_compute_1
    2022-05-31 18:50:21.071 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_compute_1
    2022-05-31 18:50:21.071 UTC [TRACE] steampipe: execute finish for, mybenchmark.control.mdbi_aws_network_1
    2022-05-31 18:50:21.071 UTC [TRACE] steampipe: control mybenchmark.control.mdbi_aws_network_1 query failed with plugin connectivity error ERROR: failed to start plugin 'aws': error reading from server: EOF (SQLSTATE HV000) - retrying...
    2022-05-31 18:50:21.071 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_network_1
    2022-05-31 18:50:21.071 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_network_1
    2022-05-31 18:50:21.077 UTC [TRACE] steampipe: execute finish for, mybenchmark.control.mdbi_aws_iam_2
    2022-05-31 18:50:21.077 UTC [TRACE] steampipe: control mybenchmark.control.mdbi_aws_iam_2 query failed with plugin connectivity error ERROR: failed to start plugin 'aws': connection error: desc = "transport: error while dialing: dial unix /tmp/plugin157052978: connect: connection refused" (SQLSTATE HV000) - retrying...
    2022-05-31 18:50:21.077 UTC [TRACE] steampipe: begin ControlRun.Start: mybenchmark.control.mdbi_aws_iam_2
    2022-05-31 18:50:21.077 UTC [TRACE] steampipe: control start, mybenchmark.control.mdbi_aws_iam_2
    
    
    bug stale 
    opened by chr-b 14
  • Add sso auto run

    Add sso auto run

    replace #839 which needed to be closed as changing commit email caused history issues

    Integration test logs

    Logs Not sure how you want to show this one, as it's very behavioral with SSO but here is the trace from the plugin
    plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] AWSPlugin: FetchType=list
    plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] AWSPlugin: Config="Value => {[eu-west-1] 0x140068c6960 <nil> <nil> <nil>}"
    plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] AWSPlugin: ConnectionManager="Type => *connection.Manager"
    plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] AWSPlugin: ConnectionManager="Value => &{0x14000010028}"
    plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="Starting for Legacy-Prod-ReadOnly"
    plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="CommandInput was for aws sts get-caller-identity --profile Legacy"
    plugin-2021-12-17.log:2021-12-17T22:35:59.255Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="exit status 255"
    plugin-2021-12-17.log:2021-12-17T22:35:59.323Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="exit status 255"
    plugin-2021-12-17.log:2021-12-17T22:35:59.430Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="exit status 255"
    plugin-2021-12-17.log:2021-12-17T22:35:59.436Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="exit status 255"
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: runAWSCLISSOLogin="Attempting to automatically open the SSO authorization page in your default browser.
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: If the browser does not open or you wish to use a different device to authorize this request, open the following URL:
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: 
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: https://device.sso.eu-west-1.amazonaws.com/
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: 
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: Then enter the code:
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: 
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: XXXX-XXXX
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: Successully logged into Start URL: https://XXXX.awsapps.com/start
    plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: "
    plugin-2021-12-17.log-2021-12-17T22:36:13.313Z [TRACE] aws.plugin: [TRACE] WithCache no function lock key getCommonColumns
    plugin-2021-12-17.log-2021-12-17T22:36:13.313Z [TRACE] aws.plugin: [TRACE] WithCache added lock to map key getCommonColumns
    plugin-2021-12-17.log-2021-12-17T22:36:13.313Z [TRACE] aws.plugin: [TRACE] WithCache no function lock key getCallerIdentity
    plugin-2021-12-17.log-2021-12-17T22:36:13.313Z [TRACE] aws.plugin: [TRACE] WithCache added lock to map key getCallerIdentity
    plugin-2021-12-17.log-2021-12-17T22:36:14.587Z [TRACE] aws.plugin: [TRACE] rowData chan select - channel CLOSED
    plugin-2021-12-17.log-2021-12-17T22:36:14.587Z [TRACE] aws.plugin: [TRACE] wait for rows
    plugin-2021-12-17.log-2021-12-17T22:36:14.587Z [TRACE] aws.plugin: [TRACE] getOrganizationDetails
    plugin-2021-12-17.log-2021-12-17T22:36:15.098Z [TRACE] aws.plugin: [TRACE] accountARN
    plugin-2021-12-17.log-2021-12-17T22:36:15.098Z [TRACE] aws.plugin: [TRACE] getAwsAccountAkas
    plugin-2021-12-17.log-2021-12-17T22:36:15.098Z [TRACE] aws.plugin: [TRACE] accountARN
    
    dbmurphy commented 10 days ago
    Hi @rajlearner17 as a secondary comment I am not sure this is a critical change. If someone as they do today, runs aws sso login prior to the steampipe command, it will have no change in behavior. If however they did not it will just attempt to run it for them. The design here was specifically to maintain existing behavior and just improve on it.
    
    Work in the main steampipe repo would be needed to avoid duplicate web page opening for authorizing the CLI ( as it would need to be done in the aggregator vs child plugin due to the way they are called in parallel.
    
    Am I wrong in thinking this is simply extended existing behavior where steampipe would have failed with no SSO creds anyhow?
    
    @rajlearner17
     
    Contributor
    rajlearner17 commented 9 days ago
    @dbmurphy I agree with your point it's important to have aws sso login be part of the execution model itself. @cbruno10 would you like to share some feedback on this?
    

    Updated the above code to better protect against a nil awsConfig.Profile entry in some cases.

    Additionally, I added a bool var we may want to expose in the awsConfig struct which is "isSSO". I did not include it, but it could make things easier code wise if we had something like:

    if awsConfig.authType == "SSO" {
       xxxxx
    } else if  awsConfig.authType="KEY" {
        if  awsConfig.accessKey != nil && awsConfig.secretKey != nil {
           xxxxx
        }
    }
    
    opened by dbmurphy 14
  • Installing a specific version of the plugin results in a loading error

    Installing a specific version of the plugin results in a loading error

    When I install a specific version, I get the following error after running steampipe query:

    $ steampipe plugin install [email protected]
    
    Installed plugin: [email protected] v0.36.0
    Documentation:    https://hub.steampipe.io/plugins/turbot/aws
    
    $ steampipe query
    Welcome to Steampipe v0.9.0
    For more information, type .help
    > 
    Error: failed to load connection 'aws': no plugin installed matching aws
    /Users/chrism/.steampipe/config/aws.spc:1,1-11
    

    I'm able to edit the ~/.steampipe/config/aws.spc file and change the plugin to [email protected] to get things working again.

    Steampipe version (steampipe -v) v0.9.0

    Plugin version (steampipe plugin list) v0.36.0

    bug 
    opened by mildebrandt 14
  • Initial implementation of global accelerator tables

    Initial implementation of global accelerator tables

    Adds the following tables:

    • aws_globalaccelerator_accelerator
    • aws_globalaccelerator_listener
    • aws_globalaccelerator_endpoint_group

    Integration test logs

    Logs
    Add passing integration test logs here
    

    Example query results

    Results
    Add example SQL query results here (please include the input queries as well)
    
    opened by nmische 13
  • Removes error check for missing regions

    Removes error check for missing regions

    This PR reverts commit 08d619d0b4e35d8e5e4ab832ec2d01d2ccbf3730 which added a check for empty AWS regions list. Unfortunately, this broke the fallback options for defining regions elsewhere, such as the profiles and AWS_REGION env var, and made regions a required option.

    opened by joraff 11
  • support for AWS Organizations / organisational unit

    support for AWS Organizations / organisational unit

    Is your feature request related to a problem? Please describe. AWS accounts can be managed using AWS Organizations and grouped with organisational units (OU). This account structure also supports a login to a "central" AWS account and assuming roles within member accounts. It would be very helpful to configure a single login AWS account to login into and the assuming a role while connecting to all AWS accounts within the organization or just a subset (OU).

    Describe the solution you'd like Configure login credentials to the "billing" account and a role being used when connecting to member accounts.

    Describe alternatives you've considered Basically the profile can be also prepared using scripts and query member account IDs of an OU.

    Additional context N/A

    enhancement stale 
    opened by tbugfinder 11
  • Add profile from aws spc

    Add profile from aws spc

    Extends aws_account to include account_profile column which is pulled get GetConfig from the spc file. This can be useful if someone wants to join on account and display account_profile vs account_aliases if they don't have aliases defined in AWS.

    Example query results

    Results
    > select account_id,account_profile from aws_all.aws_account
    +---------------+-------------------+
    | account_id    | account_profile   |
    +---------------+-------------------+
    | 000000000000  | shared-services   |
    | 000000000001  | nonprod           |
    | 000000000002  | prod              |
    +---------------+-------------------+
    

    Updated the code so that during awsConfig setup, if awsConfig.Profile is nil , sets an empty string as the pointer source

    > select  account_profile,account_id from aws_key.aws_account
    +-----------------+--------------+
    | account_profile | account_id   |
    +-----------------+--------------+
    | default         | XXXXXXXXXXXX |
    +-----------------+--------------+
    Using:
    
    connection "aws_key" {
      plugin      = "aws"
      access_key     = "XXXXXXXXXXXXXXXXXXXX"
      secret_key     = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      regions     = ["eu-west-1"]
    }
    

    Additional mixed-mode testing by mixing SSO and Key modes in an aggregator: (SSO profile name and account_ids replaced for account safety.

    > select account_profile,account_id from aws_test_account_key.aws_account
    +------------------------+--------------+
    | account_profile        | account_id   |
    +------------------------+--------------+
    | David Personal Profile | XXXXXXXXXXXX |
    +------------------------+--------------+
    > select account_profile,account_id from aws_test_db_prod.aws_account
    +-----------------------+--------------+
    | account_profile       | account_id   |
    +-----------------------+--------------+
    | David SSO Profile | YYYYYYYYYYYY |
    +-----------------------+--------------+
    > select account_profile,account_id from aws_test_all.aws_account
    +------------------------+--------------+
    | account_profile        | account_id   |
    +------------------------+--------------+
    | David Personal Profile | XXXXXXXXXXXX |
    | David SSO Profile  | YYYYYYYYYYYY |
    +------------------------+--------------+
    > 
    
    opened by dbmurphy 11
  • Listing EC2 instances fails with status code: 400

    Listing EC2 instances fails with status code: 400

    Describe the bug When running select * from aws_ec2_instances steampipe exits with the following error after running for a bit.

    Error: :
            status code: 400, request id:
    

    This doesn't appear to happen outside a specific account, but so far haven't been able to narrow it down outside of that.

    Log during query part 1 Log during query part 2

    Steampipe version (steampipe -v) steampipe version 0.7.3

    Plugin version (steampipe plugin list) 0.31.0

    To reproduce Currently haven't been able to reproduce this outside of a specific an environment. Will update if I can narrow this down though.

    Expected behavior Steampipe doesn't fail.

    Additional context The environment is fairly large so this fails sometime into the scan (unsure how long exactly, can double check soon and update). At least one attempt succeeded, so this doesn't appear to happen every time, but occured about 3 out of the 4 times attempted.

    bug priority:high 
    opened by RyanJarv 11
  • hydrate function getVpcEipARN failed with panic runtime error

    hydrate function getVpcEipARN failed with panic runtime error

    Describe the bug We're getting a crash in certain regions when querying aws_vpc_eip and including the arn column. The error is:

    Error: rpc error: code = Internal desc = hydrate function getVpcEipARN failed with panic runtime error: invalid memory address or nil pointer dereference (SQLSTATE HV000)
    

    Steampipe version (steampipe -v) v0.16.1

    Plugin version (steampipe plugin list) [email protected] 0.76.0

    To reproduce

    steampipe query "select arn from <profile>.aws_vpc_eip"
    
    Error: rpc error: code = Internal desc = hydrate function getVpcEipARN failed with panic runtime error: invalid memory address or nil pointer dereference (SQLSTATE HV000)
    

    Expected behavior No crash, the arn and allocation_id fields should be null.

    Additional context Using the aws cli, I noticed that this region has an EIP that doesn't have an allocation ID, etc.. Other EIPs in this region, and all EIPs in other regions that don't crash have allocation IDs and some other fields.

    aws --profile <profile> --region us-east-1 ec2 describe-addresses
    {
      "Addresses": [
        {
          "InstanceId": "",
          "PublicIp": "x.x.x.x",
          "Domain": "standard",
          "PublicIpv4Pool": "amazon"
        },
       ...
      ]
    }
    
    bug 
    opened by ben417 10
  • rapidly repeated DNS query

    rapidly repeated DNS query

    Describe the bug Ran the below query for cloudwatch log streams last week. After a few moments, received an error stating my DNS server is "misbehaving." I'm not clear if that error text is from Steampipe, an AWS API response, or something else.

    As it turns out, my own local DNS server was the source of the timeout, there were 1000 DNS queries made from my laptop to the DNS server in less than 60 seconds. If I'm interpreting this correctly, my steampipe query resulted in ~1000 consecutive repeated lookups for logs.us-east-1.amazonaws.com before my DNS server (rightfully) cut me off. For a few minutes, my laptop is completely cut-off from being able to query DNS, at all, and I lose general connectivity. I'm basically banned from using DNS for hammering it a thousand consecutive times with the same query.

    Steampipe version (steampipe -v) steampipe (mac) v0.15.0 aws plugin v0.66.0

    To reproduce steampipe query 'select region,first_event_timestamp,last_event_timestamp,log_group_name,name from my_connection_name.aws_cloudwatch_log_stream' --output csv > log-streams.csv

    Expected behavior a csv file from "my_connection_name.aws_cloudwatch_log_stream" containing the fields from my select statement

    Additional context

    • I was able to get around this by adding logs.us-east-1.amazonaws.com to my hosts file so no queries go to my DNS server, but I don't feel like this behavior of rapidly repeating the same DNS query is efficient.
    • querying a smaller AWS account with less streams seems to be successful.

    Including a screenshot of my console output as well as my local eth & wifi ip addresses getting temporarily banned on my home pi-hole DNS server. image

    bug 
    opened by jhodal 10
  • aws_ecr_repository queries hitting api limits

    aws_ecr_repository queries hitting api limits

    Describe the bug queries will timeout with an api limit error (Error running query: rpc error: code = Unknown desc = ThrottlingException: Rate exceeded status code: 400, request id: ....) when querying using repository_name in ( select distinct repository_name from aws_ecr_repository limit 1) or similar (using a subselect). However, if the query doesn't use a subselect like `repository_name in ('a/b','b/c',.....) it works fine

    Steampipe version (steampipe -v) Example: v0.15.1

    Plugin version (steampipe plugin list) Example: v0.71.0

    To reproduce

      select
        deets -> 'ImageTags' as image_tags,
        deets -> 'ImageDigest' as image_digest,
        deets -> 'ImagePushedAt' as image_pushedat,
        deets -> 'LastRecordedPullTime' as image_lastpulled,
        ecr.repository_name as repository_name,
        fin -> 'Uri' as uri,
        fin -> 'Name' as cve,
        fin -> 'Severity' as severity,
        fin -> 'Description' as description,
        fin -> 'Attributes' as attributes
      from
        aws_cloudhub.aws_ecr_repository as ecr,
        jsonb_array_elements(image_details) as deets,
        jsonb_array_elements(image_scanning_findings) as details,
        jsonb(details) as detail,
        jsonb_array_elements(detail -> 'ImageScanFindings' -> 'Findings') as fin
      where
        ecr.repository_name in (select distinct repository_name from aws_ecr_repository limit 1)
        and fin ->> 'Severity' in ('CRITICAL', 'UNDEFINED')
        and deets -> 'ImageDigest' = detail -> 'ImageId' -> 'ImageDigest'
    

    Expected behavior subselect in quals should not hit api limits

    bug 
    opened by tinder-tder 9
  • Add table aws_drs_source_server . closes #1457

    Add table aws_drs_source_server . closes #1457

    Integration test logs

    Logs
    Add passing integration test logs here
    

    Example query results

    Results
    Add example SQL query results here (please include the input queries as well)
    
    opened by sdsubhajitdas 0
  • Add table aws_backup_legal_hold closes #1439

    Add table aws_backup_legal_hold closes #1439

    Integration test logs

    Logs
    N/A
    

    Example query results

    Results
    > select
      legal_hold_id,
      arn,
      creation_date,
      cancellation_date
    from
      aws_backup_legal_hold;
    +--------------------------------------+---------------------------------------------------------------------------------------+---------------------------+-------------------+
    | legal_hold_id                        | arn                                                                                   | creation_date             | cancellation_date |
    +--------------------------------------+---------------------------------------------------------------------------------------+---------------------------+-------------------+
    | 2c72c4e3-2936-4d53-8eb6-fdafce991da6 | arn:aws:backup:us-east-1:384702153875:legal-hold:2c72c4e3-2936-4d53-8eb6-fdafce991da6 | 2022-12-01T14:27:17+05:30 | <null>            |
    +--------------------------------------+---------------------------------------------------------------------------------------+---------------------------+-------------------+
    
    Time: 6.1s. Rows fetched: 1. Hydrate calls: 0.
    > select
      legal_hold_id,
      arn,
      creation_date,
      creation_date,
      retain_record_until
    from
      aws_backup_legal_hold
    where
      creation_date >= current_date - interval '1' day
    order by
      creation_date;
    +--------------------------------------+---------------------------------------------------------------------------------------+---------------------------+---------------------------+------------------
    | legal_hold_id                        | arn                                                                                   | creation_date             | creation_date             | retain_record_unt
    +--------------------------------------+---------------------------------------------------------------------------------------+---------------------------+---------------------------+------------------
    | 2c72c4e3-2936-4d53-8eb6-fdafce991da6 | arn:aws:backup:us-east-1:384702153875:legal-hold:2c72c4e3-2936-4d53-8eb6-fdafce991da6 | 2022-12-01T14:27:17+05:30 | 2022-12-01T14:27:17+05:30 | <null>           
    +--------------------------------------+---------------------------------------------------------------------------------------+---------------------------+---------------------------+------------------
    
    Time: 2.4s. Rows fetched: 1. Hydrate calls: 1.
    > select
      title,
      legal_hold_id,
      recovery_point_selection -> 'DateRange' ->> 'ToDate' as to_date,
      recovery_point_selection -> 'DateRange' ->> 'FromDate' as from_date,
      recovery_point_selection -> 'VaultNames' as vault_names,
      recovery_point_selection ->> 'ResourceIdentifiers' as resource_identifiers
    from
      aws_backup_legal_hold;
    +--------+--------------------------------------+--------------------------+----------------------+-------------+--------------------------------+
    | title  | legal_hold_id                        | to_date                  | from_date            | vault_names | resource_identifiers           |
    +--------+--------------------------------------+--------------------------+----------------------+-------------+--------------------------------+
    | test43 | 2c72c4e3-2936-4d53-8eb6-fdafce991da6 | 2022-12-01T08:57:16.295Z | 2022-11-29T18:30:00Z | []          | ["arn:aws:ec2:*:*:instance/*"] |
    +--------+--------------------------------------+--------------------------+----------------------+-------------+--------------------------------+
    
    Time: 2.5s. Rows fetched: 1. Hydrate calls: 1.
    > 
    
    opened by ParthaI 0
  • Add table aws_securitylake_subscriber closes #1440

    Add table aws_securitylake_subscriber closes #1440

    Integration test logs

    Logs
    N/A
    

    Example query results

    Results
    > select
      subscriber_name,
      subscription_id,
      created_at,
      role_arn,
      s3_bucket_arn,
      subscription_endpoint
    from
      aws_securitylake_subscriber;
    +-----------------+--------------------------------------+---------------------------+----------------------------------------------------------------------------------------+---------------+-------------------------------------------------------------------------------
    | subscriber_name | subscription_id                      | created_at                | role_arn                                                                               | s3_bucket_arn | subscription_endpoint                                                         
    +-----------------+--------------------------------------+---------------------------+----------------------------------------------------------------------------------------+---------------+-------------------------------------------------------------------------------
    | sub-001         | c0b04ca2-7c0a-460a-bbc2-23ca588308af | 2022-12-01T17:24:22+05:30 | arn:aws:iam::948502153456:role/AmazonSecurityLake-c0b04ca2-7c0a-460a-bbc2-23ca588308af | <null>        | arn:aws:sqs:us-east-1:948502153456:AmazonSecurityLake-c0b04ca2-7c0a-460a-bbc2-
    | sub02-rk        | de4c9289-7fae-4425-bea4-a94c3119963a | 2022-12-01T18:02:56+05:30 | arn:aws:iam::948502153456:role/AmazonSecurityLake-de4c9289-7fae-4425-bea4-a94c3119963a | <null>        | arn:aws:sqs:us-east-1:948502153456:AmazonSecurityLake-de4c9289-7fae-4425-bea4-
    +-----------------+--------------------------------------+---------------------------+----------------------------------------------------------------------------------------+---------------+-------------------------------------------------------------------------------
    
    Time: 1.6s. Rows fetched: 2. Hydrate calls: 0.
    > select
      s.subscriber_name,
      s.subscription_id,
      r.arn,
      r.inline_policies,
      r.attached_policy_arns,
      r.assume_role_policy
    from
      aws_securitylake_subscriber as s,
      aws_iam_role as r
    where
      s.role_arn = r.arn;
    +-----------------+--------------------------------------+----------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------
    | subscriber_name | subscription_id                      | arn                                                                                    | inline_policies                                                                                                           
    +-----------------+--------------------------------------+----------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------
    | sub-001         | c0b04ca2-7c0a-460a-bbc2-23ca588308af | arn:aws:iam::948502153456:role/AmazonSecurityLake-c0b04ca2-7c0a-460a-bbc2-23ca588308af | [{"PolicyDocument":{"Statement":[{"Action":["sqs:GetQueueUrl","sqs:ReceiveMessage","sqs:SendMessage","sqs:DeleteMessage","
    | sub02-rk        | de4c9289-7fae-4425-bea4-a94c3119963a | arn:aws:iam::948502153456:role/AmazonSecurityLake-de4c9289-7fae-4425-bea4-a94c3119963a | [{"PolicyDocument":{"Statement":[{"Action":["sqs:GetQueueUrl","sqs:ReceiveMessage","sqs:SendMessage","sqs:DeleteMessage","
    +-----------------+--------------------------------------+----------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------
    
    Time: 11.9s. Rows fetched: 191 (2 cached). Hydrate calls: 378.
    
    
    opened by ParthaI 0
  • Add column platform_family to table aws_ecs_service

    Add column platform_family to table aws_ecs_service

    Is your feature request related to a problem? Please describe.

    DescribeServices API supports a new attribute PlatformFamily in github.com/aws/aws-sdk-go-v2/service/ecs: v1.20.0

    Describe the solution you'd like A clear and concise description of what you want to happen.

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

    enhancement 
    opened by ParthaI 0
Owner
Turbot
Get cloud work done with Turbot — Creators of https://turbot.com/v5 and https://steampipe.io
Turbot
AWS SDK for the Go programming language.

AWS SDK for Go aws-sdk-go is the official AWS SDK for the Go programming language. Checkout our release notes for information about the latest bug fix

Amazon Web Services 8k Nov 29, 2022
AWS Lambda to work around index.html S3/CloudFront mess

No more index.html mess with AWS CloudFront/S3 Problem Consider you have a statically generated site — a bunch of usual resources, including html file

Artyom Pervukhin 26 Jun 18, 2022
Browse your AWS ECS Clusters in the Terminal

Browse your AWS ECS Clusters in the Terminal. The ecsview application is a terminal-based UI for browsing Amazon Elastic Container Service (ECS) clust

Jason Swartz 101 Nov 1, 2022
Generate a basic IAM policy from AWS client-side monitoring (CSM)

iamlive Generate a basic IAM policy from AWS client-side monitoring (CSM) Installation Pre-built binaries Pre-built binaries for Windows, macOS and Li

Ian Mckay 2.2k Nov 24, 2022
This example shows how to serve private contents on AWS S3 through CloudFront signed URL.

AWS CloudFront with Signed URL This is the repository of my blog post. This example shows how to serve private contents on AWS S3 through CloudFront s

Hao-Ming, Hsu 7 Oct 19, 2022
A Cloud Foundry cli plugin that offers a faster and customizable alternative for cf apps

Panzer cf cli plugin A plugin for faster interaction (less API calls) with Cloud Foundry, and choose the columns you want in your output. Instead of "

Harry Metske 0 Feb 14, 2022
Steampipe plugin to query your Scalingo apps, addons and more

Scalingo plugin for Steampipe Use SQL to query infrastructure including applications and addons from Scalingo. Get started → Documentation: Table defi

François de Metz 11 Nov 4, 2022
Steampipe plugin for the Hypothesis annotation system

Steampipe plugin for the Hypothesis annotation system

Jon Udell 1 Nov 17, 2021
Steampipe plugin to query your Baleen namespaces, custom rules and more

Baleen plugin for Steampipe Use SQL to query namespaces, rules and more from Baleen. Get started → Documentation: Table definitions & examples Quick s

François de Metz 2 Jun 16, 2022
This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances

Session Manager Plugin This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances. Sess

Amazon Web Services 178 Nov 28, 2022
null 2 Feb 7, 2022
Simple no frills AWS S3 Golang Library using REST with V4 Signing (without AWS Go SDK)

simples3 : Simple no frills AWS S3 Library using REST with V4 Signing Overview SimpleS3 is a golang library for uploading and deleting objects on S3 b

Rohan Verma 95 Nov 4, 2022
Run the same Docker images in AWS Lambda and AWS ECS

serverlessish tl;dr Run the exact same image for websites in Lambda as you do in ECS, Kubernetes, etc. Just add this to your Dockerfile, listen on por

Glass Echidna 182 Sep 18, 2022
CLI for exploring AWS EC2 Spot inventory. Inspect AWS Spot instance types, saving, price, and interruption frequency.

spotinfo The spotinfo is a command-line tool that helps you determine AWS Spot instance types with the least chance of interruption and provides the s

Alexei Ledenev 74 Sep 28, 2022
Terraform provider to help with various AWS automation tasks (mostly all that stuff we cannot accomplish with the official AWS terraform provider)

terraform-provider-awsutils Terraform provider for performing various tasks that cannot be performed with the official AWS Terraform Provider from Has

Cloud Posse 24 Oct 1, 2022
Infrastructure testing helper for AWS Resources that uses AWS SSM to remotely execute commands on EC2 machines.

Infrastructure testing helper for AWS Resources that uses AWS SSM to remotely execute commands on EC2 machines, to enable infrastructure engineering teams to write tests that validate behaviour.

Ankit Wal 18 Sep 5, 2022
CLI tool to update ~/.aws/config with all accounts and permission sets defined in AWS SSO

aws-sso-profiles Generate or update ~/.aws/config with a profile for each SSO account you have access to, by using an existing AWS SSO session. Bootst

SpareBank 1 Utvikling 3 Nov 3, 2022
Integrate AWS EKS Anywhere cluster with AWS Services

This article provides step-by-step instruction on integrating AWS EKS Anywhere with AWS Services so the applications running on customer data center can securely connect with these services.

Hari Ohm Prasath 2 Mar 6, 2022
Apis para la administracion de notifiaciones, utilizando servicios como AWS SNS y AWS SQS

notificacion_api Servicio para envío de notificaciónes por difusión en AWS SNS Especificaciones Técnicas Tecnologías Implementadas y Versiones Golang

Universidad Distrital Francisco José de Caldas 0 Jan 7, 2022