Inject another failure

Let’s create a new failure and generate an insight for that. We will modify the Lambda functions’s resource-based policy by removing the permissions for API Gateway to access this function.

  1. After you close the insight from the previous section i.e. when it is on the Resolved state, trigger the HTTP traffic generation loop from the AWS Cloud9 terminal using the following command:
python sendAPIRequest.py
  1. Follow this link to open the Lambda function ScanFunctionMonitorOper.

  2. On the Permissions tab, access the Resource-based policy section.

Accessing the permissions tab for the Lambda
  1. Save a copy of the policy offline as a backup before making any changes.

  2. Note down the Sid values for the AWS:SourceArn that ends with prod/*/ and prod/*/*.

Checking the Resource-based policy for the Lambda
  1. Run the following command to remove the Sid JSON statements:
aws lambda remove-permission --function-name ScanFunctionMonitorOper \
    --statement-id <Sid-value-ending-with-prod/*/>
  1. Run the same command for the second Sid value:
aws lambda remove-permission --function-name ScanFunctionMonitorOper \
    --statement-id <Sid-value-ending-with-prod/*/*>

You should see several 5XX errors, as in the following screenshot:

Terminal output now showing 500 errors for the script output

After less than 8 minutes, you should see a new ongoing reactive insight on the DevOps Guru dashboard.

Let’s take a closer look at the insight. The following screenshot shows the anomalous metric 5XXError Average of API Gateway and its duration (this insight shows as closed because I had already restored permissions):

Terminal output now showing 500 errors for the script output

If you have configured to enable creating OpsItem in Systems Manager, you would see the link to OpsItem ID created in the insight, as shown above. This is an optional configuration, which will enable you to track the insights in the form of open tickets (OpsItems) in Systems Manager OpsCenter.

The recommendations provide guidance based upon the related events and anomalous metrics. After the insight has been generated, reviewed, and verified, restore the permissions by running the following command:

aws lambda add-permission --function-name ScanFunctionMonitorOper  \
    --statement-id APIGatewayProdPerm --action lambda:InvokeFunction \
    --principal apigateway.amazonaws.com

If needed, you can insert the condition to point to the API Gateway ARN to allow only specific API Gateways to access the Lambda function.