Use RDS Proxy with AWS Lambda and IAM authentication
RDS Proxy is a database proxy that makes your applications more scalable. Because Lambda functions can scale up to hundreds of instances, RDS Proxy allows the functions to share existing connections by handling all the database traffic. Concurrent Lambda functions no longer create new connection for every invocation, and the database needs much less memory and CPU resources to manage the connections.
The main objective of this tutorial is to connect a Lambda function to a database proxy using IAM authentication. With IAM authentication, you manage your database credentials the same way you manage other credentials in IAM console, and your source code is a lot more secure.
The examples provided here use MySQL and Python. At time of writing, RDS Proxy works with (Aurora) MySQL 5.6 or 5.7 and (Aurora) PostgreSQL 10.10 or 11.5. Let’s do this!
Update: RDS Proxy now supports MySQL 8.
Prerequisite
Make sure the RDS instance has Password authentication enabled.
Step 1 — Create a user for Lambda
Create a user the proxy will use to query the database on behalf of the Lambda functions. I know I said we’ll be using IAM authentication to connect. Here’s how it works: the Lambda functions use IAM authentication to authenticate to the proxy. On success, the proxy connects to the database and then uses the user we’re creating to run queries. Make sure you grant the right access to this user.
CREATE USER 'lambdauser'@'%' IDENTIFIED BY 'lambdapassword';
GRANT SELECT, INSERT, UPDATE ON dbname.* TO 'lambdauser';
FLUSH PRIVILEGES;Step 2 — Store the credentials in Secrets Manager
Here we need at least 2 secrets: the first is to store the database credentials, it’s the only way the proxy connects to the database, and the second is to store the credentials created in Step 1. Once the Lambda is successfully authenticated, the proxy makes queries using the second secret.
Go to Secrets Manager console and click Store a new secret. Choose Credentials for RDS database, enter the username and password, choose the database instance and click Next.
Name the new secret, add a description and click Next. Keep the default options and click Next. Review everything and then click Store.
Now click Store a new secret and choose Other type of secrets. Enter the key “username” and the value is the same user name created in Step 1. Enter the key “password” and the value is the password for the user created in Step 1. Click Next.
Name the new secret, add a description, keep the default options in the following steps, then click Store.
When working with RDS Proxy, each database user for the RDS instance accessed by a proxy must have a corresponding secret in Secrets Manager.
We have created the secret the proxy will use to authenticate to the database and the secret to access the user’s account and make queries. Keep both secrets ARN, we’ll need them later.
Step 3 — Create a security group for the Lambda functions
Open the EC2 dashboard and go to Security Groups. Click Create security group. Name the group, add a description and select the same VPC as the RDS instance. Keep the default options for Inbound rules (no inbound) and Outbound rules (all traffic out) and click Create security group.
Step 4 — Create a security group for RDS Proxy
Click Create security group. Name the group, add a description and select the same VPC as the RDS instance. For Inbound rules, click Add rule. You must allow the security group created in Step 3 for Lambda functions, on port 3306 used for MySQL. At this time, RDS Proxy does not support a custom port (although the underline database does, which is stored in the Secret from Step 2).
For Outbound rules, leave the default to all traffic out and click Create security group.
Step 5 — Edit the security group for RDS instance
Open the security group associated with the RDS instance and add a rule to allow the security group created in Step 4.
You may use existing security groups to allow access. Just make sure the security group you assign to the Proxy allows traffic from Lambda functions to the Proxy, and traffic from the Proxy to RDS.
Step 6 — Create a policy for RDS Proxy
Open the IAM console and click Create policy. For Service, choose Secrets Manager. The required action is Read — GetSecretValue. Under Resources, add the ARN of the 2 secrets created earlier in Step 2.
Click Add additional permissions. For Service, choose KMS. The required action is Write — Decrypt. Under Resources, enter your region name (or check Any) and check Any for Key id.
Under Request conditions, click Add condition. For Condition key, select kms:ViaService. For Operator, select StringEquals. For Value, enter secretsmanager.REGION.amazonaws.com (use the correct region name). Click Add.
Click Next: Tags, Next: Review. Name the policy and click Create policy.
Here’s a template for the policy we’ve just created. Make sure you set the right values for your account.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": [
"RDS_SECRET_ARN",
"RDS_DATABASE_USER_SECRET_ARN"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/*",
"Condition": {
"StringEquals": {
"kms:ViaService": "secretsmanager.REGION.amazonaws.com"
}
}
}
]
}Step 7 — Add a role for RDS Proxy
Here we add a role to authorize RDS Proxy to access the secrets created in Step 2 via the policy defined in Step 6. Click Create role. Select RDS in the list and then choose RDS — Add Role to Database.
Click Next: Permissions. Filter the Customer managed policies and select the policy created in Step 6. Click Next:Tags, Next: Review. Name the role and click Create Role.
So far we have defined and stored the database credentials the proxy will use to connect to RDS and make queries on behalf of the Lambda functions. We also created a security group to associate to the Lambda functions, and allowed traffic from this security group to the proxy, and from the proxy to RDS. Finally we created an IAM role to authorize the proxy to read the secrets. Now let’s create the proxy.
Step 8 — Create the proxy
Open the RDS dashboard, go to Proxies and click Create proxy. Name the proxy, select the engine and check Require Transport Layer Security. This is mandatory to use IAM authentication.
Under Target group configuration, select the database the proxy will connect to. You can leave the maximum connections to 100 percent because we’re using only 1 proxy. If you have many proxies, you can share the percentage between them.
Under Connectivity, for Secrets Manager secret(s), select the 2 secrets created in Step 2. For IAM role, select the role created in Step 7. For IAM authentication, select Required. For Subnets, select at least 2 subnets from the VPC. Expand the Additional connectivity configuration section. For VPC security group, remove the default security group and select the security group created in Step 4.
Under Advanced configuration, I suggest you check Enable enhanced logging. It will help you debug in case you run into troubles. You may uncheck it later. Click Create proxy. The process then takes a few minutes to complete.
Click on the proxy name to access the details. Under the Target group(s) section, wait for the status to become Available before continuing. It means the proxy has access to the database. Otherwise there’s something wrong with your configuration (more likely an issue with secrets manager or IAM role).
Save the proxy endpoint for later.
The proxy is created and is able to connect to the database. Now we create a Lambda function to make a simple request.
Step 9 — Add a role for Lambda
Open the IAM dashboard and click Create role. For service choose Lambda and click Next: Permissions. Select the policy AWSLambdaVPCAccessExecutionRole and click Next: Tags, Next: Review. Name the role and click Create role.
Step 10 — Create the Lambda function
Open the Lambda dashboard and click Create function. Name the function, select Python 3.8 as Runtime, expand Change default execution role section and select the role created in Step 9.
Under Advanced settings section, for VPC select the same VPC as the RDS instance, select at least 2 subnets, and for Security groups, select the security group created in Step 3. Then click Create function.
Access the function details, scroll down to the Environment variables section and add the following variables keys and values (they will be referenced in the code):
- DB_HOST — enter the proxy endpoint from Step 8.
- DB_USER — enter the user name created in Step 1.
- DB_NAME — enter the database name to access.
Go to the Database proxies section and click Add database proxy. Select the proxy created in Step 8.
Step 11 — Test the function
This function generates a token and connects to the proxy, then makes a simple SELECT statement. I use mysql-connector-python, and you will need to create a deployment package (instructions here). The source code is below, edit it to your needs before deploying.
from mysql.connector import Error as mysql_error
import boto3
import logging
import mysql.connector
import os
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging.info('Generate database token...')
rds_client = boto3.client('rds')
database_token = rds_client.generate_db_auth_token(
DBHostname=os.environ['DB_HOST'],
Port=3306,
DBUsername=os.environ['DB_USER'],
Region=os.environ['AWS_REGION']
)
logging.info('Token successfully obtained. Connecting to database...')
database_connection = mysql.connector.connect(
host=os.environ['DB_HOST'],
port=3306,
database=os.environ['DB_NAME'],
user=os.environ['DB_USER'],
password=database_token
)
logging.info('Connected!!')
def lambda_handler(event, context):
db_cursor = database_connection.cursor(dictionary=True)
query = "SELECT * FROM tablename LIMIT 10"
db_cursor.execute(query)
print(db_cursor.fetchall())Notice there is no password referenced in the code, we use a token instead. Test the function with a simple test event and check the logs. You should see the rows from you query.
Congratulations, your Lambda functions can now access RDS Proxy using IAM authentication.
References
- Limitations for RDS Proxy : https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html#rds-proxy.limitations
