Setup AWS Lambda to Use Amazon RDS Proxy
Adam C. |

In the previous post, we learned that to ultimately solve the issue of AWS Lambda overwhelming RDS instance is using Amazon RDS Proxy. In this post, we are looking into this.

AWS provides some documents, we found the below two links are helpful:

  1. Using Amazon RDS Proxy with AWS Lambda
  2. Introduction to RDS Proxy (Youtube Video)

The idea behind this is pretty simple. The flow that we had before was:  Amazon API Gateway → Lambda function → Amazon RDS, like the diagram below:

Lambda to RDS Diagram by AWS Compute Blog

And then we would like to add an RDS Proxy between Lambda function and Amazon RDS, so then we could use RDS Proxy to share the database connection pool, improve database efficiency, application scalability, and security.  The architecture looks like this:

RDS Proxy  by AWS

Get Started

Like most of AWS Services, the setup of RDS Proxy is painful. We hope you have an AWS Certified DevOps Engineer in your team.  In this article, we are NOT going to cover the details of how to configure this which involves AWS Secrets Manager, IAM Role/Policy, and attaching a proxy to AWS Lambda. Please follow the links at the beginning of this article and good luck!

We will go over some issues we had on the Lambda function side. Here is what we had before in dbConnection:

import getSecret from './secrets';

require('mysql');

const dbContext = require('knex')({
  client: 'mysql',
  connection: async () => {
    return {
      host: await getSecret(`${process.env.ENV}_DB_HOST`),
      user: await getSecret(`${process.env.ENV}_DB_USER`),
      password: await getSecret(`${process.env.ENV}_DB_PASSWORD`),
      database: await getSecret(`${process.env.ENV}_DB_NAME`),
      ssl: 'Amazon RDS'
    };
  },
  pool: {
    min: 0,
    max: 10,
    createTimeoutMillis: 30000,
    acquireTimeoutMillis: 30000,
    idleTimeoutMillis: 30000,
    reapIntervalMillis: 1000,
    createRetryIntervalMillis: 100,
  },
  debug: false
});

export default dbContext;

Updated (10/19/2020)  We removed the pool settings propagateCreateError: false (Sorry for the mistake, and it's should be true)  Per Mikael (knex/tarn maintainer) :

Please don't set propagateCreateError: false it will make knex behave badly. … it should never be touched

And we need to point the connection to RDS Proxy instead of RDS, and since we use IAM authentication, we add RDS.singer to get the token.  The code like this:

const getToken = (hostName, userName) => {
  const signer = new AWS.RDS.Signer({
    region: 'us-east-1',
    hostname: hostName,
    port: 3306,
    username: userName
  });

  const token = signer.getAuthToken({
    username: userName
  });
  return token;
};

const dbContext = require('knex')({
  client: 'mysql',
  const rdsProxyEndpoint = await getSecret(`${process.env.ENV}_Proxy_Endpoint`);
  const dbUser = await getSecret(`${process.env.ENV}_DB_USER`);
  const dbName = await getSecret(`${process.env.ENV}_DB_NAME`);

  const token = getToken(rdsProxyEndpoint, dbUser);
  const connectionConfig = {
      host: rdsProxyEndpoint,
      user: dbUser,
      database: dbName,
      ssl: { rejectUnauthorized: false },
      password: token,
      authSwitchHandler: ({ pluginName, pluginData }, cb) => {
        console.log('Setting new auth handler.');
      }
    };

    // Adding the mysql_clear_password handler
    connectionConfig.authSwitchHandler = (data, cb) => {
      if (data.pluginName === 'mysql_clear_password') {
        console.log('pluginName: ' + data.pluginName);
        const password = token + '\0';
        const buffer = Buffer.from(password);
        cb(null, password);
      }
    };
 ...
 }

And then, we got an error in RDS log, 

INFO Error: MySQL is requesting the mysql_clear_password authentication method, which is not supported.

   at Handshake.AuthSwitchRequestPacket (/var/task/node_modules/mysql/lib/protocol/sequences/Handshake.js:47:17) …..

Regarding MySQL Doc:

Hashing or encryption cannot be done for authentication schemes that require the server to receive the password as entered on the client side. In such cases, the client-side mysql_clear_password plugin is used, which enables the client to send the password to the server as cleartext. There is no corresponding server-side plugin. Rather, mysql_clear_password can be used on the client side in concert with any server-side plugin that needs a cleartext password.

Since AWS RDS Proxy is using mysql_clear_password plugin, we have to add this Auth Handler. But we use ‘mysql’ node npm, which does not yet support mysql_clear_password. The good news is that there is a better mysql node npm, ‘mysql2’ available.

But we got second error: 

"String cannot represent value: { introText: "", breadcrumbFields: [{}] }"

That's because we have some database columns are JSON type, and node mysql2 library parses the JSON string to a native object, which is different in node mysql library. Since our code base were written based on node mysql library, it would be a lot of changes if using JSON object. The good thing is that mysql2 allows typeCast to treat JSON type as String type. The modified code looks like this:

const connectionConfig = {
   host: rdsProxyEndpoint,
   user: dbUser,
   database: dbName,
   ssl: { rejectUnauthorized: false },
   password: token,
   authSwitchHandler: ({ pluginName, pluginData }, cb) => {
     console.log('Setting new auth handler.');
   },
   typeCast: (field, next) => {
     if (field.type === 'JSON') {
       return field.string();
     }
     return next();
   }
};

After that we also got some other errors in RDS:

Error1 - unable to get local issuer certificate

INFO Error: unable to get local issuer certificate

   at TLSSocket.<anonymous> (/var/task/node_modules/mysql/lib/Connection.js:317:48)

That's because, during the debugging, we set ssl back to ‘Amazon RDS’, like ssl: 'Amazon RDS'. This means we should set ssl: { rejectUnauthorized: false } when pointing the connection to RDS Proxy.

Error2 - Access denied for user … (using password: YES)

INFO Error: Access denied for user 'db_admin'@'10.0.3.240' (using password: YES)

   at Packet.asError (/var/task/node_modules/mysql2/lib/packets/packet.js:712:17)

That's because, during the debugging, we set the connection host to be RDS endpoint instead of RDS Proxy endpoint.

Error3 - Access denied for user … (using password: NO)

INFO Error: Access denied for user 'db_admin'@'%' (using password: NO)

   at Packet.asError (/var/task/node_modules/mysql2/lib/packets/packet.js:712:17)

This error drove us crazy. We had gone thru all settings in AWS RDS Proxy to make sure they are the same as what AWS doc says, but had not made it work.  Finally, we checked the RDS Proxy log (it's new log in addition to RDS log and Lambda log after using RDS Proxy) and saw this:

Proxy authentication with IAM authentication failed for user "nccih_admin" with TLS on. Reason: The proxy couldn't authenticate using IAM. The expected "Credential's region" value is "us-east-1" but was "us-eat-1". Generate a new authentication token and use it in a new client connection.

  

Oh no! There is a typo: ‘us-eat-1’ So lesson learned:  check logs first (all available logs) it would help you!

Last thing

In the local we still use local mySQL server, so there is no RDS proxy involved, and we still use min:0/max:10 in the pool setting, but in the AWS, we now have RDS proxy for the connection pooling, so we should make min/max based on the total connections available in RDS. We are trying min:5/max:30 for now, but looks like it could be set to much higher numbers.  Here is the final dbConnection.js:

import getSecret, { getToken } from './secrets';

require('mysql2');

// 8/26/2020
// LOCAL - no pool share b/w requests, so use 0 for min
// DEV/PORD - use rds proxy for pooling connecgtion, so use 5 for min

const minNum = process.env.ENV === 'local' ? 0 : 5;
const maxNum = process.env.ENV === 'local' ? 10 : 30;
const idleTimout = process.env.ENV === 'local' ? 1000 : 30000;

const dbContext = require('knex')({
  client: 'mysql2',
  connection: async () => {
    console.log('connecting to db...');
    if (process.env.ENV === 'local')
      return {
        host: '127.0.0.1',
        user: 'db_admin',
        password: 'db_pass',
        database: 'db_name',
        typeCast: (field, next) => {
          if (field.type === 'JSON') {
            return field.string();
          }
          return next();
        }
      };

    const rdsProxyEndpoint = await getSecret(`${process.env.ENV}_Proxy_Endpoint`);
    const dbUser = await getSecret(`${process.env.ENV}_DB_USER`);
    const dbName = await getSecret(`${process.env.ENV}_DB_NAME`);

    const token = getToken(rdsProxyEndpoint, dbUser);

    const connectionConfig = {
      host: rdsProxyEndpoint,
      user: dbUser,
      database: dbName,
      ssl: { rejectUnauthorized: false },
      password: token,
      authSwitchHandler: ({ pluginName, pluginData }, cb) => {
        console.log('Setting new auth handler.');
      },
      typeCast: (field, next) => {
        if (field.type === 'JSON') {
          return field.string();
        }
        return next();
      }
    };

    // Adding the mysql_clear_password handler
    connectionConfig.authSwitchHandler = (data, cb) => {
      if (data.pluginName === 'mysql_clear_password') {
        console.log('pluginName: ' + data.pluginName);
        const password = token + '\0';
        const buffer = Buffer.from(password);
        cb(null, password);
      }
    };

    console.log(connectionConfig);
    return connectionConfig;
  },
  pool: {
    min: minNum, 
    max: maxNum,
    createTimeoutMillis: 30000,
    acquireTimeoutMillis: 30000,
    idleTimeoutMillis: idleTimout,
    reapIntervalMillis: 1000,
    createRetryIntervalMillis: 100
  },
  debug: false
});

export default dbContext;

That's it. Hope this helps.

Updated (10/19/2020)   We removed the pool settings propagateCreateError: true. Per Mikael (knex/tarn maintainer) - 

the setting, propagateCreateError, should never be touched.