Use RDS Proxy with AWS Lambda and IAM authentication

AWS Lambda RDS Proxy

RDS Proxy is a database proxy that makes your applications more scalable. Because Lambda functions can scale up to hundreds of instances, RDS Proxy allows the functions to share existing connections by handling all the database traffic. Concurrent Lambda functions no longer create new connection for every invocation, and the database needs much less memory and CPU resources to manage the connections.

The main objective of this tutorial is to connect a Lambda function to a database proxy using IAM authentication. With IAM authentication, you manage your database credentials the same way you manage other credentials in IAM console, and your source code is a lot more secure.

The examples provided here use MySQL and Python. At this time, RDS Proxy works with (Aurora) MySQL 5.6 or 5.7 and (Aurora) PostgreSQL 10.10 or 11.5. Let’s do this!

Prerequisite

Make sure the RDS instance has Password authentication enabled.

RDS database authentication option

Create a user the proxy will use to query the database on behalf of the Lambda functions. I know I said we’ll be using IAM authentication to connect. Here’s how it works: the Lambda functions use IAM authentication to authenticate to the proxy. On success, the proxy connects to the database and then uses the user we’re creating to run queries. Make sure you grant the right access to this user.

Here we need at least 2 secrets: the first is to store the database credentials, it’s the only way the proxy connects to the database, and the second is to store the credentials created in Step 1. Once the Lambda is successfully authenticated, the proxy makes queries using the second secret.

Go to Secrets Manager console and click Store a new secret. Choose Credentials for RDS database, enter the username and password, choose the database instance and click Next.

Secrets manager — store RDS credentials

Name the new secret, add a description and click Next. Keep the default options and click Next. Review everything and then click Store.

Now click Store a new secret and choose Other type of secrets. Enter the key “username” and the value is the same user name created in Step 1. Enter the key “password” and the value is the password for the user created in Step 1. Click Next.

Secrets manager — store user credentials

Name the new secret, add a description, keep the default options in the following steps, then click Store.

When working with RDS Proxy, each database user for the RDS instance accessed by a proxy must have a corresponding secret in Secrets Manager.

We have created the secret the proxy will use to authenticate to the database and the secret to access the user’s account and make queries. Keep both secrets ARN, we’ll need them later.

Open the EC2 dashboard and go to Security Groups. Click Create security group. Name the group, add a description and select the same VPC as the RDS instance. Keep the default options for Inbound rules (no inbound) and Outbound rules (all traffic out) and click Create security group.

Security group for Lambda functions

Click Create security group. Name the group, add a description and select the same VPC as the RDS instance. For Inbound rules, click Add rule. You must allow the security group created in Step 3 for Lambda functions, on port 3306 used for MySQL. At this time, RDS Proxy does not support a custom port (although the underline database does, which is stored in the Secret from Step 2).

Security group for RDS Proxy

For Outbound rules, leave the default to all traffic out and click Create security group.

Open the security group associated with the RDS instance and add a rule to allow the security group created in Step 4.

You may use existing security groups to allow access. Just make sure the security group you assign to the Proxy allows traffic from Lambda functions to the Proxy, and traffic from the Proxy to RDS.

Open the IAM console and click Create policy. For Service, choose Secrets Manager. The required action is Read — GetSecretValue. Under Resources, add the ARN of the 2 secrets created earlier in Step 2.

IAM secrets manager access

Click Add additional permissions. For Service, choose KMS. The required action is Write — Decrypt. Under Resources, enter your region name (or check Any) and check Any for Key id.

KMS ARN

Under Request conditions, click Add condition. For Condition key, select kms:ViaService. For Operator, select StringEquals. For Value, enter secretsmanager.REGION.amazonaws.com (use the correct region name). Click Add.

KMS condition

Click Next: Tags, Next: Review. Name the policy and click Create policy.

Here’s a template for the policy we’ve just created. Make sure you set the right values for your account.

Here we add a role to authorize RDS Proxy to access the secrets created in Step 2 via the policy defined in Step 6. Click Create role. Select RDS in the list and then choose RDS — Add Role to Database.

IAM add role to RDS

Click Next: Permissions. Filter the Customer managed policies and select the policy created in Step 6. Click Next:Tags, Next: Review. Name the role and click Create Role.

So far we have defined and stored the database credentials the proxy will use to connect to RDS and make queries on behalf of the Lambda functions. We also created a security group to associate to the Lambda functions, and allowed traffic from this security group to the proxy, and from the proxy to RDS. Finally we created an IAM role to authorize the proxy to read the secrets. Now let’s create the proxy.

Open the RDS dashboard, go to Proxies and click Create proxy. Name the proxy, select the engine and check Require Transport Layer Security. This is mandatory to use IAM authentication.

RDS Proxy configuration

Under Target group configuration, select the database the proxy will connect to. You can leave the maximum connections to 100 percent because we’re using only 1 proxy. If you have many proxies, you can share the percentage between them.

Under Connectivity, for Secrets Manager secret(s), select the 2 secrets created in Step 2. For IAM role, select the role created in Step 7. For IAM authentication, select Required. For Subnets, select at least 2 subnets from the VPC. Expand the Additional connectivity configuration section. For VPC security group, remove the default security group and select the security group created in Step 4.

RDS Proxy connectivity

Under Advanced configuration, I suggest you check Enable enhanced logging. It will help you debug in case you run into troubles. You may uncheck it later. Click Create proxy. The process then takes a few minutes to complete.

Click on the proxy name to access the details. Under the Target group(s) section, wait for the status to become Available before continuing. It means the proxy has access to the database. Otherwise there’s something wrong with your configuration (more likely an issue with secrets manager or IAM role).

RDS Proxy — status unavailable
RDS Proxy — status available

Save the proxy endpoint for later.

The proxy is created and is able to connect to the database. Now we create a Lambda function to make a simple request.

Open the IAM dashboard and click Create role. For service choose Lambda and click Next: Permissions. Select the policy AWSLambdaVPCAccessExecutionRole and click Next: Tags, Next: Review. Name the role and click Create role.

Open the Lambda dashboard and click Create function. Name the function, select Python 3.8 as Runtime, expand Change default execution role section and select the role created in Step 9.

AWS Lambda basic information

Under Advanced settings section, for VPC select the same VPC as the RDS instance, select at least 2 subnets, and for Security groups, select the security group created in Step 3. Then click Create function.

AWS Lambda advanced settings

Access the function details, scroll down to the Environment variables section and add the following variables keys and values (they will be referenced in the code):

  • DB_HOST — enter the proxy endpoint from Step 8.
  • DB_USER — enter the user name created in Step 1.
  • DB_NAME — enter the database name to access.
AWS Lambda environment variables

Go to the Database proxies section and click Add database proxy. Select the proxy created in Step 8.

AWS Lambda database proxy

This function generates a token and connects to the proxy, then makes a simple SELECT statement. I use mysql-connector-python, and you will need to create a deployment package (instructions here). The source code is below, edit it to your needs before deploying.

Notice there is no password referenced in the code, we use a token instead. Test the function with a simple test event and check the logs. You should see the rows from you query.

Congratulations, your Lambda functions can now access RDS Proxy using IAM authentication.

Full-stack developer based in Quebec city, QC. I enjoy coding, data analysis, tennis and photography (https://mariusbio.photos).

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store