Disable AWS CloudFront Distributions if Budget is Exceeded

I am scared of using AWS. In AWS, you can’t easily put a hard cap on the money that you will spend, meaning if you are the target of a DDoS attack or if something goes wrong, you might be overcharged massively. For me, this means that if my CloudFront distributions are under a DDoS attack, I might get a budget notification in my email, but if I don’t act fast enough, I might be overcharged. I would like to avoid that. So in this post, I will try to explain how we can disable all of our CloudFront Distributions if the budget is exceeded or if we exceed some threshold we set for our CloudFront requests.

The first solution is not very reliable because the billing info gets updated up to 3 times in AWS^[1]. So there is a chance that a DDoS will last up to 24 hours and that could still incur thousands of dollars. Using a Lambda function to monitor CloudFront requests from CloudWatch and disable the distributions after a threshold is exceeded is more reliable. AWS also provides a GetFreeTierUsage API, but I haven’t found a way to get it to report CloudFront free tier usage, unfortunately. But I am still open to any suggestions if you have any to improve the situation^[2], including tips about moving this solution and the blog setup to CloudFormation, and not depending on hard coded free tier limits.

I considered moving to Hetzner, but it seemed like too big of a change to my eyes: moving everything, including the TLS certificate, etc., and making GitHub Actions work with deploying to Hetzner. For a cheap option, I considered using Hetzner Webhosting, but every little thing there costs extra money. Want more than 1 domain? Costs money. Want a static IPv4 address? Costs money. So I am still in the consideration phase. Is a possible DDoS attack worth the extra amount of work necessary to move to Hetzner? Anyway, on with the blog post…

First of all, log in to the AWS Console and go to Identity and Access Management (IAM). Click on Policies on the sidebar.

Then, click on Create Policy. In the Policy Editor, click on JSON and paste the following policy to allow CloudFront ListDistributions and UpdateDistribution permissions.

 1{
 2  "Version": "2012-10-17",
 3  "Statement": [
 4    {
 5      "Effect": "Allow",
 6      "Action": [
 7        "cloudfront:ListDistributions",
 8        "cloudfront:UpdateDistribution",
 9        "cloudfront:GetDistributionConfig"
10      ],
11      "Resource": "*"
12    }
13  ]
14}

Click on Next and name your policy. I chose lambda-cloudfront-disable-policy and added the description It allows the policyholder to list all CloudFront distributions, get CloudFront distribution configs, and update CloudFront distributions.

Then click on Create Policy.

Next, on the left sidebar, click on Roles.

Click on Create Role. For the Trusted entity type, select AWS Service, and then under Use Case, from the dropdown menu select Lambda.

Click on Next. For permissions, we need to select the policy we created previously, lambda-cloudfront-disable-policy, and AWSLambdaBasicExecutionRole policy to allow the Lambda functions to write logs.

Then click on Next. For Role Name, I chose BudgetCloudFrontDisableRole, and review.

Then click on Create Role. Wait while the role is created.

Once the role is created, go to AWS Lambda.

AWS Lambda is a serverless code running method provided by AWS. In our case, we will be in the Free Tier because we will run the disable function once, if we exceed our limits and the monitoring function every 5 minutes. That means 8760 invocations of the function, each with 3-10 seconds of runtime. If we calculate 8760*512*10 = 44 851 200 MB-seconds of compute time, which is less than 400 000 GB-seconds (400 000 000 MB-seconds) given in the free tier^[3].

Then click on Create a Function. Then we name our Lambda function. I chose DisableCloudFrontDistributions. Then for the Runtime, you have a couple of different options. You could use almost whatever language you prefer, but it becomes difficult to deploy if you choose a compiled language, for example, Go or Rust. That’s why it seems to me that the industry best practice is to go with Node.js. And as for Architecture, I go with arm64, no real reason, as we will probably never have to pay. Then for the Execution role, I chose Use an existing role and from the dropdown, I chose BudgetCloudFrontDisableRole.

And then click on Create function. Here, at the top of the page, let’s copy the Function ARN as we will need it later when we subscribe to our SNS.

Now we see a code editor on the page, and an index.mjs file. Paste the following code into the code editor and click on Deploy.

 1import { CloudFrontClient, GetDistributionConfigCommand, ListDistributionsCommand, UpdateDistributionCommand } from "@aws-sdk/client-cloudfront";
 2
 3const cloudFrontClient = new CloudFrontClient();
 4
 5export async function handler(event, context) {
 6  console.log(`Event: ${JSON.stringify(event)}`);
 7  console.log(`Context: ${JSON.stringify(context)}`);
 8  try {
 9    // Get a list of all CloudFront distributions
10    const distributionsResponse = await cloudFrontClient.send(new ListDistributionsCommand({}));
11
12    // Disable each distribution
13    const disablePromises = distributionsResponse.DistributionList.Items.map(async (item) => {
14      const distributionId = item.Id;
15      await disableDistribution(distributionId);
16    });
17
18    await Promise.all(disablePromises);
19
20    return {
21      statusCode: 200,
22      body: "All CloudFront distributions disabled successfully."
23    };
24  }
25  catch (error) {
26    console.error("Error:", error);
27    return {
28      statusCode: 500,
29      body: "An error occurred while disabling CloudFront distributions."
30    };
31  }
32}
33
34async function disableDistribution(distributionId) {
35  try {
36    // Get distribution configuration
37    const configResponse = await cloudFrontClient.send(new GetDistributionConfigCommand({ Id: distributionId }));
38    const config = configResponse.DistributionConfig;
39
40    // Check if distribution is already disabled
41    if (!config.Enabled) {
42      console.log(`Distribution ${distributionId} is already disabled.`);
43      return; // Skip updating
44    }
45
46    const disabledConfig = {
47      ...config,
48      Enabled: false,
49    };
50
51    // Update distribution with modified configuration
52    await cloudFrontClient.send(new UpdateDistributionCommand({
53      Id: distributionId,
54      DistributionConfig: disabledConfig,
55      IfMatch: configResponse.ETag
56    }));
57
58    console.log(`Disabled CloudFront distribution: ${distributionId}`);
59  }
60  catch (error) {
61    console.error(`Error disabling CloudFront distribution ${distributionId}:`, error);
62    throw error;
63  }
64}

After deployment, if you click Test, all of your CloudFront distributions should be disabled in a short time. However, we don’t want to manually run the Lambda function; we want it to be run automatically once our budget or our CloudFront metrics are exceeded. Let’s configure that as well.

To run the Lambda function, we will create an Amazon SNS. Open Amazon SNS page, and again make sure that the region of your SNS is us-east-1.

Then click on Create topic and choose Standard as SNS Type and name the topic. I named it BudgetAlertSNS. On the Access Policy, we need to add a new Statement so that our budget can access the SNS and publish a message there.

Click on Access Policy and copy the JSON preview under the Basic method. Then click on Advanced method and paste the copied JSON. Afterward, add the following Statement in the Statement array, replacing YOUR_SNS_ARN with the ARN from the copied JSON^[4]:

    {
      "Sid": "BudgetAction",
      "Effect": "Allow",
      "Principal": {
        "Service": "budgets.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "YOUR_SNS_ARN"
    }

And at the end, you should have a policy similar to the following:

 1{
 2  "Version": "2008-10-17",
 3  "Id": "__default_policy_ID",
 4  "Statement": [
 5    {
 6      "Sid": "__default_statement_ID",
 7      "Effect": "Allow",
 8      "Principal": {
 9        "AWS": "*"
10      },
11      "Action": [
12        "SNS:Publish",
13        "SNS:RemovePermission",
14        "SNS:SetTopicAttributes",
15        "SNS:DeleteTopic",
16        "SNS:ListSubscriptionsByTopic",
17        "SNS:GetTopicAttributes",
18        "SNS:AddPermission",
19        "SNS:Subscribe"
20      ],
21      "Resource": "YOUR_SNS_ARN",
22      "Condition": {
23        "StringEquals": {
24          "AWS:SourceOwner": "YOUR_ACCOUNT_ID"
25        }
26      }
27    },
28    {
29      "Sid": "BudgetAction",
30      "Effect": "Allow",
31      "Principal": {
32        "Service": "budgets.amazonaws.com"
33      },
34      "Action": "SNS:Publish",
35      "Resource": "YOUR_SNS_ARN"
36    }
37  ]
38}

You can leave everything else as is and press Create topic.

Let’s copy the ARN of our SNS Topic for later use and then create a subscription for this SNS. This subscription will run our Lambda function.

Scroll down to the bottom of this page and click on Create Subscription button. For the protocol, choose AWS Lambda, and for the Endpoint, paste the Function ARN you copied previously.

Click on Create Subscription.

At this page, you can also optionally create a subscription with your email. This way, you will get notified whenever there is a message in this SNS topic. Note that you will have to go to your email and confirm the subscription.

Now we can try to publish a message in this topic, and our Lambda function should automatically run and disable our CloudFront distributions.

Budget Setup

As for the next step, let’s connect this SNS to our budget and disable the distributions automatically if the budget is exceeded.

Let’s go to the AWS Billing and Cost Management page and on the sidebar, click on Budgets.

If you have no Budgets created yet, click on the Create a budget button and choose the Customize (advanced) box. Then, if you see that you do not have Cost Explorer enabled, click on the Enable Cost Explorer button. For the Budget Type, choose Cost budget - recommended.

And then click Next. On this page, enter a Budget name and select the period and enter your budget amount. Keep Budget scope to All AWS Services (Recommended).

And then click Next. On this page, you should create an alert threshold (I chose 95% of the actual budget as the threshold) and expand the Amazon SNS Alerts - Optional Info section and enter your SNS ARN. Additionally, you can enter an email to get notified or create other alerts to get notified as well.

Click on Next, and in the Actions page, no changes are necessary.

Click Next and review the budget.

And click Create budget. The budget should be created and should be ready to send notifications and disable all of your CloudFront distributions once the budget reaches 95% of the budgeted amount.

Using a Lambda Function to Monitor CloudFront Metrics

Here, let’s begin again by creating a Policy. This time, our policy should allow us to get CloudFront metrics from CloudWatch and send an SNS to the topic we created previously so that our CloudFront Distributions can be disabled. Like before, go to Identity and Access Management (IAM), click on Policies on the sidebar, and click on Create Policy. In the Policy Editor, click on JSON and paste the following policy, replacing YOUR_SNS_ARN with the SNS ARN we copied previously.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:GetMetricData",
        "cloudfront:ListDistributions"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "YOUR_SNS_ARN"
    }
  ]
}

Click on Next and name your policy. I chose lambda-monitor-metrics-send-sns and added the description It allows the policyholder to list all CloudFront distributions, get CloudFront distribution metrics and send an SNS that can disable CloudFront Distributions.

Now that we have our Policy created, let’s create a Role and attach this Policy to it, like before. Once again, on the left sidebar click on Roles. Then click on Create Role. For the Trusted entity type, select AWS Service, and then under Use Case, from the dropdown menu select Lambda. Click on Next. For permissions, we need to select the policy we created previously, lambda-monitor-metrics-send-sns, and AWSLambdaBasicExecutionRole policy to allow the Lambda functions to write logs. Then click on Next. For Role Name, I chose CloudFrontMetricsMonitor and review.

Then click on Create Role. Wait while the role is created.

Once the role is created, go to AWS Lambda. Then click on Create a Function. Then we name our Lambda function. I chose MonitorCloudFrontMetrics. Like previously, I chose Node.js for Runtime and arm64 for Architecture. Then for the Execution role, I chose Use an existing role and from the dropdown, I chose CloudFrontMetricsMonitor.

Like before, we see a code editor. Paste the following code into the code editor, replacing YOUR_SNS_ARN with the SNS Topic ARN we copied before and click on Deploy.

  1import { CloudFrontClient, ListDistributionsCommand } from "@aws-sdk/client-cloudfront";
  2import { CloudWatchClient, GetMetricDataCommand } from "@aws-sdk/client-cloudwatch";
  3import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";
  4
  5const cloudFrontClient = new CloudFrontClient();
  6const cloudWatchClient = new CloudWatchClient();
  7const snsClient = new SNSClient();
  8
  9const SNSTopicARN = 'YOUR_SNS_ARN';
 10
 11const metricsToTrack = {
 12  distribution: ["Requests", "BytesDownloaded"],
 13  function: ["FunctionInvocations"],
 14  limits: {
 15    Requests: 10_000_000,
 16    BytesDownloaded: 1_000_000_000_000,
 17    FunctionInvocations: 2_000_000,
 18  },
 19};
 20
 21export async function handler(event, context) {
 22  console.log(`Event: ${JSON.stringify(event)}`);
 23  console.log(`Context: ${JSON.stringify(context)}`);
 24  try {
 25    // Get a list of all CloudFront distributions
 26    const distributionsResponse = await cloudFrontClient.send(new ListDistributionsCommand({}));
 27
 28    // Creating a Set to store unique FunctionARN values
 29    const uniqueFunctionARNs = new Set();
 30
 31    // Extracting DefaultCacheBehavior FunctionAssociations
 32    distributionsResponse.DistributionList.Items.forEach(item => {
 33      if (item.DefaultCacheBehavior && item.DefaultCacheBehavior.FunctionAssociations && item.DefaultCacheBehavior.FunctionAssociations.Items) {
 34        item.DefaultCacheBehavior.FunctionAssociations.Items.forEach(assoc => {
 35          uniqueFunctionARNs.add(assoc.FunctionARN);
 36        });
 37      }
 38    });
 39
 40    // Extracting CacheBehaviors FunctionAssociations
 41    distributionsResponse.DistributionList.Items.forEach(item => {
 42      if (item.CacheBehaviors && item.CacheBehaviors.Items) {
 43        item.CacheBehaviors.Items.forEach(cacheBehavior => {
 44          if (cacheBehavior.FunctionAssociations && cacheBehavior.FunctionAssociations.Items) {
 45            cacheBehavior.FunctionAssociations.Items.forEach(assoc => {
 46              uniqueFunctionARNs.add(assoc.FunctionARN);
 47            });
 48          }
 49        });
 50      }
 51    });
 52
 53    // Converting Set to array
 54    const allFunctionARNs = Array.from(uniqueFunctionARNs);
 55
 56    // Get the start and end dates for the current month
 57    const today = new Date();
 58    const startDate = new Date(today.getFullYear(), today.getMonth(), 1);
 59    const endDate = new Date(today.getFullYear(), today.getMonth() + 1, 0);
 60
 61    // Create an array to hold all the MetricDataQueries
 62    const metricDataQueries = [];
 63
 64    distributionsResponse.DistributionList.Items.forEach((distribution, i) => {
 65      metricsToTrack.distribution.forEach(metric => {
 66        metricDataQueries.push({
 67          Id: `distribution_${i + 1}_metric_${metric.toLowerCase()}`,
 68          MetricStat: {
 69            Metric: {
 70              Namespace: 'AWS/CloudFront',
 71              MetricName: metric,
 72              Dimensions: [{ Name: 'DistributionId', Value: distribution.Id }, { Name: 'Region', Value: 'Global' }]
 73            },
 74            Period: 300,
 75            Stat: 'Sum'
 76          },
 77          ReturnData: true
 78        });
 79      });
 80    });
 81
 82    allFunctionARNs.forEach((functionARN, i) => {
 83      metricsToTrack.function.forEach(metric => {
 84        metricDataQueries.push({
 85          Id: `function_${i + 1}_metric_${metric.toLowerCase()}`,
 86          MetricStat: {
 87            Metric: {
 88              Namespace: 'AWS/CloudFront',
 89              MetricName: metric,
 90              Dimensions: [{ Name: "FunctionName", Value: functionARN.split(':function/')[1] }, { Name: 'Region', Value: 'Global' }]
 91            },
 92            Period: 300,
 93            Stat: 'Sum'
 94          },
 95          ReturnData: true
 96        });
 97      });
 98    });
 99
100    // Execute all promises concurrently and wait for all of them to resolve
101    const metricsResults = await fetchCloudWatchMetrics(metricDataQueries, startDate, endDate);
102
103    var exceeded = false;
104
105    Object.keys(metricsResults).forEach(metricType => {
106      const sum = metricsResults[metricType];
107      console.log(`Metric Type: ${metricType}, Sum: ${sum}, Limit: ${metricsToTrack.limits[metricType]}`);
108      if (sum >= metricsToTrack.limits[metricType]) {
109        exceeded = true;
110      }
111    });
112
113    if (exceeded) {
114      const params = {
115        Subject: 'CloudFront Metric Alert',
116        Message: `Metric limit has been exceeded. Disabling all CloudFront Distributions!`,
117        TopicArn: SNSTopicARN
118      };
119      const snsResult = await snsClient.send(new PublishCommand(params));
120      console.log("SNS notification sent successfully.");
121
122      return {
123        statusCode: 201,
124        body: "All CloudFront Distributions will be disabled. Limits has been exceeded."
125      };
126    }
127
128    return {
129      statusCode: 200,
130      body: JSON.stringify(metricsResults)
131    };
132  }
133  catch (error) {
134    console.error("Error:", error);
135    return {
136      statusCode: 500,
137      body: "An error occurred while retrieving CloudFront requests."
138    };
139  }
140}
141
142async function fetchCloudWatchMetrics(metricDataQueries, startDate, endDate) {
143  try {
144    // Get CloudWatch metrics for the specified metrics and dimensions
145    const metricDataResponse = await cloudWatchClient.send(new GetMetricDataCommand({
146      MetricDataQueries: metricDataQueries,
147      StartTime: startDate,
148      EndTime: endDate
149    }));
150
151    const aggregatedMetrics = {};
152
153    // Process the responses for each metric
154    metricDataResponse.MetricDataResults.forEach(metricData => {
155      const metricType = metricData.Label.split(' ')[1];
156      const sum = metricData.Values.reduce((accumulator, currentValue) => accumulator + currentValue, 0);
157      aggregatedMetrics[metricType] = (aggregatedMetrics[metricType] || 0) + sum;
158    });
159
160    return aggregatedMetrics;
161  }
162  catch (error) {
163    console.error("Error fetching metric data:", error);
164    return null;
165  }
166}

If you click Test, and then Invoke, depending on how many CloudFront distributions and functions you have, your Lambda might time out. To increase the timeout or give more resources, click on the Configuration tab and click Edit:

I increase the Memory to 256~512 MB and the Timeout to 5~10 seconds. As we increase the Memory, we also get more CPU resources. Test the function and increase the Timeout depending to your needs. Now we can go to the Code tab and Test our function again. As a response, you should see the total Requests, BytesDownloaded, and FunctionInvocations for all of your CloudFront Distributions:

{
  "statusCode": 200,
  "body": "{\"Requests\":3038,\"BytesDownloaded\":4319762,\"FunctionInvocations\":2875}"
}

In case one of those exceeds the limit, this Lambda will send a message to the SNS Topic and the SNS Topic will disable all of the CloudFront Distributions.

Now the only thing that is left is to schedule this Lambda function so that it can run. For that, we will use the Amazon EventBridge Scheduler. As of today, Scheduler can run 14 000 000 invocations per month for free^[5].

Let’s go to the Amazon EventBridge Scheduler home page.

Then let’s click on Create schedule.

I changed the Schedule name to lambda-metrics-monitoring and Occurrence to Recurring schedule, and by default, Cron-based schedule is set. I leave it as is and fill the Cron expression as follows (you can copy-paste it into the first field):

*/5 * ? * * *

After you click Next, you will see what to do on schedule. We want to Invoke a Lambda, so we choose AWS Lambda - Invoke option, and then choose the Lambda we want to run, that is MonitorCloudFrontMetrics.

Then, at the bottom of the page, select Skip to Review and create schedule.

If everything looks good, click on Create schedule. Wait while the schedule is created. After creation is completed, you will see the created schedule.

One last thing I did was to change the retention setting for the Log Groups automatically created by the Lambda functions. In the free tier CloudWatch, 5 GB Data is allowed. And by default, the Log Groups created by Lambda functions do not have a retention policy. Go to CloudWatch Log groups page and select all the log groups. Click on Actions and choose Edit retention setting(s).

And when the selection screen pops up, select any time you find appropriate. I went with 2 months (60 days).

And click Save.

And now, everything should be set up. If you ever exceed the budget, the CloudFront Distributions will get disabled.

Footnotes: