replace #839 which needed to be closed as changing commit email caused history issues

Integration test logs

plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] AWSPlugin: FetchType=list
plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] AWSPlugin: Config="Value => {[eu-west-1] 0x140068c6960 <nil> <nil> <nil>}"
plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] AWSPlugin: ConnectionManager="Type => *connection.Manager"
plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] AWSPlugin: ConnectionManager="Value => &{0x14000010028}"
plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="Starting for Legacy-Prod-ReadOnly"
plugin-2021-12-17.log:2021-12-17T22:35:59.153Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="CommandInput was for aws sts get-caller-identity --profile Legacy"
plugin-2021-12-17.log:2021-12-17T22:35:59.255Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="exit status 255"
plugin-2021-12-17.log:2021-12-17T22:35:59.323Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="exit status 255"
plugin-2021-12-17.log:2021-12-17T22:35:59.430Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="exit status 255"
plugin-2021-12-17.log:2021-12-17T22:35:59.436Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: checkAWSCallerIdent="exit status 255"
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [TRACE] aws.plugin: [TRACE] getSessionWithMaxRetries: runAWSCLISSOLogin="Attempting to automatically open the SSO authorization page in your default browser.
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: If the browser does not open or you wish to use a different device to authorize this request, open the following URL:
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: 
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: https://device.sso.eu-west-1.amazonaws.com/
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: 
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: Then enter the code:
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: 
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: XXXX-XXXX
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: Successully logged into Start URL: https://XXXX.awsapps.com/start
plugin-2021-12-17.log-2021-12-17T22:36:13.290Z [DEBUG] aws.plugin: "
plugin-2021-12-17.log-2021-12-17T22:36:13.313Z [TRACE] aws.plugin: [TRACE] WithCache no function lock key getCommonColumns
plugin-2021-12-17.log-2021-12-17T22:36:13.313Z [TRACE] aws.plugin: [TRACE] WithCache added lock to map key getCommonColumns
plugin-2021-12-17.log-2021-12-17T22:36:13.313Z [TRACE] aws.plugin: [TRACE] WithCache no function lock key getCallerIdentity
plugin-2021-12-17.log-2021-12-17T22:36:13.313Z [TRACE] aws.plugin: [TRACE] WithCache added lock to map key getCallerIdentity
plugin-2021-12-17.log-2021-12-17T22:36:14.587Z [TRACE] aws.plugin: [TRACE] rowData chan select - channel CLOSED
plugin-2021-12-17.log-2021-12-17T22:36:14.587Z [TRACE] aws.plugin: [TRACE] wait for rows
plugin-2021-12-17.log-2021-12-17T22:36:14.587Z [TRACE] aws.plugin: [TRACE] getOrganizationDetails
plugin-2021-12-17.log-2021-12-17T22:36:15.098Z [TRACE] aws.plugin: [TRACE] accountARN
plugin-2021-12-17.log-2021-12-17T22:36:15.098Z [TRACE] aws.plugin: [TRACE] getAwsAccountAkas
plugin-2021-12-17.log-2021-12-17T22:36:15.098Z [TRACE] aws.plugin: [TRACE] accountARN

dbmurphy commented 10 days ago
Hi @rajlearner17 as a secondary comment I am not sure this is a critical change. If someone as they do today, runs aws sso login prior to the steampipe command, it will have no change in behavior. If however they did not it will just attempt to run it for them. The design here was specifically to maintain existing behavior and just improve on it.

Work in the main steampipe repo would be needed to avoid duplicate web page opening for authorizing the CLI ( as it would need to be done in the aggregator vs child plugin due to the way they are called in parallel.

Am I wrong in thinking this is simply extended existing behavior where steampipe would have failed with no SSO creds anyhow?

@rajlearner17
 
Contributor
rajlearner17 commented 9 days ago
@dbmurphy I agree with your point it's important to have aws sso login be part of the execution model itself. @cbruno10 would you like to share some feedback on this?

Updated the above code to better protect against a nil awsConfig.Profile entry in some cases.

Additionally, I added a bool var we may want to expose in the awsConfig struct which is "isSSO". I did not include it, but it could make things easier code wise if we had something like:

if awsConfig.authType == "SSO" {
   xxxxx
} else if  awsConfig.authType="KEY" {
    if  awsConfig.accessKey != nil && awsConfig.secretKey != nil {
       xxxxx
    }
}

When I install a specific version, I get the following error after running steampipe query:

$ steampipe plugin install [email protected]

Installed plugin: [email protected] v0.36.0
Documentation:    https://hub.steampipe.io/plugins/turbot/aws

$ steampipe query
Welcome to Steampipe v0.9.0
For more information, type .help
> 
Error: failed to load connection 'aws': no plugin installed matching aws
/Users/chrism/.steampipe/config/aws.spc:1,1-11

I'm able to edit the ~/.steampipe/config/aws.spc file and change the plugin to [email protected] to get things working again.

Steampipe version (steampipe -v) v0.9.0

Plugin version (steampipe plugin list) v0.36.0

Extends aws_account to include account_profile column which is pulled get GetConfig from the spc file. This can be useful if someone wants to join on account and display account_profile vs account_aliases if they don't have aliases defined in AWS.

Example query results

Results
> select account_id,account_profile from aws_all.aws_account
+---------------+-------------------+
| account_id    | account_profile   |
+---------------+-------------------+
| 000000000000  | shared-services   |
| 000000000001  | nonprod           |
| 000000000002  | prod              |
+---------------+-------------------+

Updated the code so that during awsConfig setup, if awsConfig.Profile is nil , sets an empty string as the pointer source

> select  account_profile,account_id from aws_key.aws_account
+-----------------+--------------+
| account_profile | account_id   |
+-----------------+--------------+
| default         | XXXXXXXXXXXX |
+-----------------+--------------+
Using:

connection "aws_key" {
  plugin      = "aws"
  access_key     = "XXXXXXXXXXXXXXXXXXXX"
  secret_key     = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  regions     = ["eu-west-1"]
}

Additional mixed-mode testing by mixing SSO and Key modes in an aggregator: (SSO profile name and account_ids replaced for account safety.

> select account_profile,account_id from aws_test_account_key.aws_account
+------------------------+--------------+
| account_profile        | account_id   |
+------------------------+--------------+
| David Personal Profile | XXXXXXXXXXXX |
+------------------------+--------------+
> select account_profile,account_id from aws_test_db_prod.aws_account
+-----------------------+--------------+
| account_profile       | account_id   |
+-----------------------+--------------+
| David SSO Profile | YYYYYYYYYYYY |
+-----------------------+--------------+
> select account_profile,account_id from aws_test_all.aws_account
+------------------------+--------------+
| account_profile        | account_id   |
+------------------------+--------------+
| David Personal Profile | XXXXXXXXXXXX |
| David SSO Profile  | YYYYYYYYYYYY |
+------------------------+--------------+
> 

Describe the bug When running select * from aws_ec2_instances steampipe exits with the following error after running for a bit.

Error: :
        status code: 400, request id:

This doesn't appear to happen outside a specific account, but so far haven't been able to narrow it down outside of that.

Log during query part 1 Log during query part 2

Steampipe version (steampipe -v) steampipe version 0.7.3

Plugin version (steampipe plugin list) 0.31.0

To reproduce Currently haven't been able to reproduce this outside of a specific an environment. Will update if I can narrow this down though.

Expected behavior Steampipe doesn't fail.

Additional context The environment is fairly large so this fails sometime into the scan (unsure how long exactly, can double check soon and update). At least one attempt succeeded, so this doesn't appear to happen every time, but occured about 3 out of the 4 times attempted.

… closes #1077

Integration test logs

Add passing integration test logs here

Example query results

Add example SQL query results here (please include the input queries as well)

Is your feature request related to a problem? Please describe. I'm writing controls for a custom AWS security mod. I want my controls to check every resource, but when my connection is denied access to a subset of resources for a given table, the query corresponding to my control fails altogether.

Describe the solution you'd like For my use case, dropping the row for the resource to which access is denied is suitable. (see below for problems associated with this solution)

Describe alternatives you've considered Alternatively, returning a null row for the resource to which access is denied would also work. I've proposed a "status" column which would contain API responses to give insight into issues. (see below for problems associated with this solution)

Based on the AWS CIS Benchmark, we've setup an alert for AccessDenied and UnauthorizedOperation events (https://github.com/turbot/steampipe-mod-aws-compliance/blob/main/query/cloudwatch/log_metric_filter_unauthorized_api.sql#L27).

When running the CIS 1.4 compliance check (https://github.com/turbot/steampipe-mod-aws-compliance/tree/main/cis_v140) this alert always triggers for kms:GetKeyRotationStatus.

This happens due to the check https://github.com/turbot/steampipe-mod-aws-compliance/blob/main/query/kms/kms_cmk_rotation_enabled.sql and the corresponding hydrate implementation within the plugin https://github.com/turbot/steampipe-plugin-aws/blob/main/aws/table_aws_kms_key.go#L269.

This Issue could be mitigated by just calling svc.GetKeyRotationStatus() for customer managed keys which of course need's another svc.DescribeKey() roundtrip first

Integration test logs

Add passing integration test logs here

Example query results

Add example SQL query results here (please include the input queries as well)

Is your feature request related to a problem? Please describe. Localstack requires to use custom endpoint url. Lack of aws endpoint configuration introduces a huge complexity for steampipe + localstack integration testing.

Describe the solution you'd like It wil be great if custom endpoint url can be specified through plugin configuration mechanism.

Describe the bug I am receiving the following error while using role based credentials in a primary account to access a series of secondary accounts via STS:AssumeRole.

ERROR: rpc error: code = Unknown desc = SharedConfigAssumeRoleError: failed to load assume role for {ROLE}, source profile {PROFILE} has no shared credentials (SQLSTATE HV000)

Based on my initial research of the error, it looks like it is originating from the go aws sdk. Threads referencing the same error: https://github.com/99designs/aws-vault/issues/410 https://github.com/aws/aws-sdk-go/issues/2063

I was successfully able to authenticate and pull data from various secondary accounts using the aws cli utility.

**Steampipe version ** 0.13.6

Plugin version 0.54.0

Describe the bug When I run query for workspaces, steampipe returns empty table even though there are multiple Workspaces in the AWS Account.

18:16 $ steampipe query
Welcome to Steampipe v0.12.2
For more information, type .help
> .cache clear
> select
  workspace_id
from
  aws_workspaces_workspace
+--------------+
| workspace_id |
+--------------+
+--------------+

Output from AWS CLI from same account with same credentials. (Output truncated and obfuscated)

18:10 $ aws workspaces describe-workspaces --query 'Workspaces[].WorkspaceId'
[
    "ws-wr......",
    "ws-bc......",
    "ws-62......",
    "ws-d1......",
    "ws-97......"
]

Steampipe version (steampipe -v)

steampipe version 0.12.2

Plugin version (steampipe plugin list)

+---------------------------------------------+---------+-------------+
| Name                                        | Version | Connections |
+---------------------------------------------+---------+-------------+
| hub.steampipe.io/plugins/turbot/[email protected]  | 0.49.0  | aws         |
| hub.steampipe.io/plugins/turbot/[email protected] | 0.1.0   | jira        |
+---------------------------------------------+---------+-------------+

To reproduce

1. In AWS Account where Workspaces exist, run query select workspace_id from aws_workspaces_workspace

Expected behavior Output should list the Workspaces IDs for the existing Workspaces

Integration test logs

Add passing integration test logs here

Example query results

Add example SQL query results here (please include the input queries as well)

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

It will be helpful if we fetch the Certificate authority date, which can help to rotate the new rds-ca as mentioned here

Go SDK Ref

Describe the solution you'd like A clear and concise description of what you want to happen.

$ aws --profile dev-test --region us-east-2 rds describe-certificates
{
    "Certificates": [
        {
            "CertificateIdentifier": "rds-ca-2019",
            "CertificateType": "CA",
            "Thumbprint": "e9ffsfgfddfdffdgdfgfdg71f6fbd1e28d3",
            "ValidFrom": "2019-09-13T17:06:41+00:00",
            "ValidTill": "2024-08-22T17:08:50+00:00",
            "CertificateArn": "arn:aws:rds:us-east-2::cert:rds-ca-2019",
            "CustomerOverride": false
        }
    ]
}

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

Describe the bug List API is failing with AccessDeniedException, however get API is working fine.

select * from aws_auditmanager_framework Error: AccessDeniedException: Please complete AWS Audit Manager setup from home page to enable this action in this account. (SQLSTATE HV000)

select * from aws_auditmanager_framework where id = 'f7cdaffa-69c5-367a-8c58-c9a4bc5704e1' and region = 'us-east-1' +-------+------------------------------------------------------------------------------------------+--------------------------------------+----------+---------------------------+------------+-----------------+----- | name | arn | id | type | created_at | created_by | compliance_type | cont +-------+------------------------------------------------------------------------------------------+--------------------------------------+----------+---------------------------+------------+-----------------+----- | HIPAA | arn:aws:auditmanager:us-east-1::assessmentFramework/f7cdaffa-69c5-367a-8c58-c9a4bc5704e1 | f7cdaffa-69c5-367a-8c58-c9a4bc5704e1 | Standard | 2022-05-06T22:38:02+05:30 | | HIPAA | <nul | | | | | | | |

Steampipe version (steampipe -v) Example: v0.14.6

Plugin version (steampipe plugin list) Example: v0.60.0

Integration test logs

Outputs:

aws_account = "533793682495"
aws_region = "us-east-1"
database_name = "turbottest4657"
resource_aka = "arn:aws:glue:us-east-1:533793682495:security-configuration/turbottest4657"
resource_name = "turbottest4657"

Running SQL query: test-get-query.sql
[
  {
    "cloud_watch_encryption": {
      "CloudWatchEncryptionMode": "DISABLED",
      "KmsKeyArn": null
    },
    "job_bookmarks_encryption": {
      "JobBookmarksEncryptionMode": "DISABLED",
      "KmsKeyArn": null
    },
    "name": "turbottest4657",
    "s3_encryption": [
      {
        "KmsKeyArn": null,
        "S3EncryptionMode": "SSE-S3"
      }
    ]
  }
]
✔ PASSED

Running SQL query: test-list-query.sql
[
  {
    "cloud_watch_encryption": {
      "CloudWatchEncryptionMode": "DISABLED",
      "KmsKeyArn": null
    },
    "job_bookmarks_encryption": {
      "JobBookmarksEncryptionMode": "DISABLED",
      "KmsKeyArn": null
    },
    "name": "turbottest4657",
    "s3_encryption": [
      {
        "KmsKeyArn": null,
        "S3EncryptionMode": "SSE-S3"
      }
    ]
  }
]
✔ PASSED

Running SQL query: test-notfound-query.sql
null
✔ PASSED

Running SQL query: test-turbot-query.sql
[
  {
    "akas": [
      "arn:aws:glue:us-east-1:533793682495:security-configuration/turbottest4657"
    ],
    "name": "turbottest4657",
    "region": "us-east-1",
    "title": "turbottest4657"
  }
]
✔ PASSED

POSTTEST: tests/aws_glue_security_configuration

TEARDOWN: tests/aws_glue_security_configuration

SUMMARY:

1/1 passed.

Example query results

> select
  name,
  cloud_watch_encryption ->> 'CloudWatchEncryptionMode' as encyption_mode,
  cloud_watch_encryption ->> 'KmsKeyArn' as kms_key_arn
from
  aws_glue_security_configuration
where
  cloud_watch_encryption ->> 'CloudWatchEncryptionMode' != 'DISABLED';

+---------+----------------+-----------------------------------------------------------------------------+
| name    | encyption_mode | kms_key_arn                                                                 |
+---------+----------------+-----------------------------------------------------------------------------+
| test234 | SSE-KMS        | arn:aws:kms:us-east-1:533793682495:key/304c9953-df86-4373-8574-46a50ed1deab |
| testsc  | SSE-KMS        | arn:aws:kms:us-east-1:533793682495:key/304c9953-df86-4373-8574-46a50ed1deab |
+---------+----------------+-----------------------------------------------------------------------------+

> select
  name,
  job_bookmarks_encryption ->> 'JobBookmarksEncryptionMode' as encyption_mode,
  job_bookmarks_encryption ->> 'KmsKeyArn' as kms_key_arn
from
  aws_glue_security_configuration
where
  job_bookmarks_encryption ->> 'JobBookmarksEncryptionMode' != 'DISABLED';
+--------+----------------+-----------------------------------------------------------------------------+
| name   | encyption_mode | kms_key_arn                                                                 |
+--------+----------------+-----------------------------------------------------------------------------+
| testsc | CSE-KMS        | arn:aws:kms:us-east-1:533793682495:key/34c20b6e-9efe-4ebd-aa26-14ce1c0fae64 |
+--------+----------------+-----------------------------------------------------------------------------+

Describe the bug

select * from aws_guardduty_member where detector_id = '*************' and member_account_id = '********' Error: 16 list calls returned errors: BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "5a1b3bad-b954-4247-86a0-c0d533db4bea" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "24495ee2-9cf7-45ef-a423-4ead82871def" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "4136584b-ee5d-48b8-a07c-7c3280a7d8ac" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "b397d53a-c62e-4d31-a758-c9f129fa8396" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "f0837f51-40b0-4a6c-be26-8cb973c17daa" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "6e7f2c94-1825-47ca-8cea-667e5e2b9566" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "0c1ae641-fde3-4b1a-ab59-5b665dc97511" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "a04c0378-d795-4a38-a41f-79b0d20e1e74" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "25445bf8-7283-4831-8fe2-fa804e2d28a2" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "beb3fd8b-f715-4c31-895e-4e649bbc5fd8" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "95f81afc-25ad-4b20-b65f-b4f7b2f6a0f6" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "e820f74d-1ef3-4a50-91ff-d0992a45163b" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "bfd014c8-4a7b-4e93-8ae7-683ad2687e85" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "fc6e39bb-111e-4806-ba5f-d9f9f570efec" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "a60b4431-ca0c-4ea0-9a54-3384ff2da9ff" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } BadRequestException: The request is rejected because the input detectorId is not owned by the current account. { RespMetadata: { StatusCode: 400, RequestID: "9eb8ca07-6312-4067-89c8-5ed2ae638693" }, Message_: "The request is rejected because the input detectorId is not owned by the current account.", Type: "InvalidInputException" } (SQLSTATE HV000)

Steampipe version (steampipe -v) Example: v0.14.6

Plugin version (steampipe plugin list) Example: v0.60.0