In the <a target="_blank" rel="noopener noreferrer" href="https://deniapps.com/blog/mysql-rds-aborted-connection-error-aws-lambda">previous post</a>, we learned that to ultimately solve the issue of AWS Lambda overwhelming RDS instance is using Amazon RDS Proxy. In this post, we are looking into this.AWS provides some documents, we found the below two links are helpful:<ol><li><a target="_blank" rel="noopener noreferrer" href="https://aws.amazon.com/blogs/compute/using-amazon-rds-proxy-with-aws-lambda/">Using Amazon RDS Proxy with AWS Lambda</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.youtube.com/watch?v=ULRnn6tIYu8">Introduction to RDS Proxy</a> (Youtube Video)</li></ol>The idea behind this is pretty simple. The flow that we had before was: &nbsp;Amazon API Gateway → Lambda function → Amazon RDS, like the diagram below:<figure class="image"><img src="https://d2908q01vomqb2.cloudfront.net/1b6453892473a467d07372d45eb05abc2031647a/2019/12/03/serverless-app-rds.png"><figcaption>Lambda to RDS Diagram by AWS Compute Blog</figcaption></figure>And then we would like to add an RDS Proxy between Lambda function and Amazon RDS, so then we could use RDS Proxy to share the database connection pool, improve database efficiency, application scalability, and security. &nbsp;The architecture looks like this:<figure class="image"><img src="https://d1.awsstatic.com/re19/Rialto/product-page-diagram_RDS%20Proxy_How-it-works@2x.a18916586f49718a16fd11579d168ab08c83d333.png"><figcaption>RDS Proxy &nbsp;by AWS</figcaption></figure><h3>Get Started</h3>Like most of AWS Services, the setup of RDS Proxy is painful. We hope you have an AWS Certified DevOps Engineer in your team. &nbsp;In this article, we are NOT going to cover the details of how to configure this which involves AWS Secrets Manager, IAM Role/Policy, and attaching a proxy to AWS Lambda. Please follow the links at the beginning of this article and good luck!We will go over some issues we had on the Lambda function side. Here is what we had before in dbConnection:<pre><code class="language-javascript">import getSecret from './secrets';

require('mysql');

const dbContext = require('knex')({
 client: 'mysql',
 connection: async () =&gt; {
 return {
 host: await getSecret(`${process.env.ENV}_DB_HOST`),
 user: await getSecret(`${process.env.ENV}_DB_USER`),
 password: await getSecret(`${process.env.ENV}_DB_PASSWORD`),
 database: await getSecret(`${process.env.ENV}_DB_NAME`),
 ssl: 'Amazon RDS'
 };
 },
 pool: {
 min: 0,
 max: 10,
 createTimeoutMillis: 30000,
 acquireTimeoutMillis: 30000,
 idleTimeoutMillis: 30000,
 reapIntervalMillis: 1000,
 createRetryIntervalMillis: 100,
 },
 debug: false
});

export default dbContext;</code></pre>Updated (10/19/2020) &nbsp;We removed the pool settings <code>propagateCreateError: false</code> (Sorry for the mistake, and it's should be <code>true</code>) &nbsp;Per Mikael (knex/tarn maintainer) :<blockquote>Please don't set <code>propagateCreateError: false</code> it will make knex behave badly. … it should never be touched</blockquote>And we need to point the connection to RDS Proxy instead of RDS, and since we use IAM authentication, we add RDS.singer to get the token. &nbsp;The code like this:<pre><code class="language-javascript">const getToken = (hostName, userName) =&gt; {
 const signer = new AWS.RDS.Signer({
 region: 'us-east-1',
 hostname: hostName,
 port: 3306,
 username: userName
 });

 const token = signer.getAuthToken({
 username: userName
 });
 return token;
};

const dbContext = require('knex')({
 client: 'mysql',
 const rdsProxyEndpoint = await getSecret(`${process.env.ENV}_Proxy_Endpoint`);
 const dbUser = await getSecret(`${process.env.ENV}_DB_USER`);
 const dbName = await getSecret(`${process.env.ENV}_DB_NAME`);

 const token = getToken(rdsProxyEndpoint, dbUser);
 const connectionConfig = {
 host: rdsProxyEndpoint,
 user: dbUser,
 database: dbName,
 ssl: { rejectUnauthorized: false },
 password: token,
 authSwitchHandler: ({ pluginName, pluginData }, cb) =&gt; {
 console.log('Setting new auth handler.');
 }
 };

 // Adding the mysql_clear_password handler
 connectionConfig.authSwitchHandler = (data, cb) =&gt; {
 if (data.pluginName === 'mysql_clear_password') {
 console.log('pluginName: ' + data.pluginName);
 const password = token + '\0';
 const buffer = Buffer.from(password);
 cb(null, password);
 }
 };
 ...
 }</code></pre>And then, we got an error in RDS log,&nbsp;<blockquote>INFO Error: MySQL is requesting the mysql_clear_password authentication method, which is not supported.&nbsp; &nbsp;at Handshake.AuthSwitchRequestPacket (/var/task/node_modules/mysql/lib/protocol/sequences/Handshake.js:47:17) …..</blockquote>Regarding MySQL Doc:<blockquote>Hashing or encryption cannot be done for authentication schemes that require the server to receive the password as entered on the client side. In such cases, the client-side <code>mysql_clear_password</code> plugin is used, which enables the client to send the password to the server as cleartext. There is no corresponding server-side plugin. Rather, <code>mysql_clear_password</code> can be used on the client side in concert with any server-side plugin that needs a cleartext password.</blockquote>Since AWS RDS Proxy is using <code>mysql_clear_password</code> plugin, we have to add this Auth Handler. But we use ‘mysql’ node npm, which does not yet support <code>mysql_clear_password</code>. The good news is that there is a better mysql node npm, ‘mysql2’ available.But we got second error:&nbsp;<blockquote>"String cannot represent value: { introText: "", breadcrumbFields: [{}] }"</blockquote>That's because we have some database columns are JSON type, and node <code>mysql2</code> library parses the JSON string to a native object, which is different in node <code>mysql</code> library. Since our code base were written based on node <code>mysql</code> library, it would be a lot of changes if using JSON object. The good thing is that <code>mysql2</code> allows <code>typeCast</code> to treat JSON type as String type. The modified code looks like this:<pre><code class="language-javascript">const connectionConfig = {
 host: rdsProxyEndpoint,
 user: dbUser,
 database: dbName,
 ssl: { rejectUnauthorized: false },
 password: token,
 authSwitchHandler: ({ pluginName, pluginData }, cb) =&gt; {
 console.log('Setting new auth handler.');
 },
 typeCast: (field, next) =&gt; {
 if (field.type === 'JSON') {
 return field.string();
 }
 return next();
 }
};</code></pre>After that we also got some other errors in RDS:<h4>Error1 - unable to get local issuer certificate</h4><blockquote>INFO Error: unable to get local issuer certificate&nbsp; &nbsp;at TLSSocket.&lt;anonymous&gt; (/var/task/node_modules/mysql/lib/Connection.js:317:48)</blockquote>That's because, during the debugging, we set ssl back to ‘Amazon RDS’, like <code>ssl: 'Amazon RDS'</code>. This means we should set <code>ssl: { rejectUnauthorized: false }</code> when pointing the connection to RDS Proxy.<h4>Error2 - Access denied for user … (using password: YES)</h4><blockquote>INFO Error: Access denied for user 'db_admin'@'10.0.3.240' (using password: YES)&nbsp; &nbsp;at Packet.asError (/var/task/node_modules/mysql2/lib/packets/packet.js:712:17)</blockquote>That's because, during the debugging, we set the connection host to be RDS endpoint instead of RDS Proxy endpoint.<h4>Error3 - Access denied for user … (using password: NO)</h4><blockquote>INFO Error: Access denied for user 'db_admin'@'%' (using password: NO)&nbsp; &nbsp;at Packet.asError (/var/task/node_modules/mysql2/lib/packets/packet.js:712:17)</blockquote>This error drove us crazy. We had gone thru all settings in AWS RDS Proxy to make sure they are the same as what AWS doc says, but had not made it work. &nbsp;Finally, we checked the RDS Proxy log (it's new log in addition to RDS log and Lambda log after using RDS Proxy) and saw this:<blockquote>Proxy authentication with IAM authentication failed for user "nccih_admin" with TLS on. Reason: The proxy couldn't authenticate using IAM. The expected "Credential's region" value is "us-east-1" but was "us-eat-1". Generate a new authentication token and use it in a new client connection.&nbsp;&nbsp;</blockquote>Oh no! There is a typo: ‘us-eat-1’ So lesson learned: &nbsp;check logs first (all available logs) it would help you!<h3>Last thing</h3>In the local we still use local mySQL server, so there is no RDS proxy involved, and we still use min:0/max:10 in the pool setting, but in the AWS, we now have RDS proxy for the connection pooling, so we should make min/max based on the total connections available in RDS. We are trying min:5/max:30 for now, but looks like it could be set to much higher numbers. &nbsp;Here is the final dbConnection.js:<pre><code class="language-javascript">import getSecret, { getToken } from './secrets';

require('mysql2');

// 8/26/2020
// LOCAL - no pool share b/w requests, so use 0 for min
// DEV/PORD - use rds proxy for pooling connecgtion, so use 5 for min

const minNum = process.env.ENV === 'local' ? 0 : 5;
const maxNum = process.env.ENV === 'local' ? 10 : 30;
const idleTimout = process.env.ENV === 'local' ? 1000 : 30000;

const dbContext = require('knex')({
 client: 'mysql2',
 connection: async () =&gt; {
 console.log('connecting to db...');
 if (process.env.ENV === 'local')
 return {
 host: '127.0.0.1',
 user: 'db_admin',
 password: 'db_pass',
 database: 'db_name',
 typeCast: (field, next) =&gt; {
 if (field.type === 'JSON') {
 return field.string();
 }
 return next();
 }
 };

 const rdsProxyEndpoint = await getSecret(`${process.env.ENV}_Proxy_Endpoint`);
 const dbUser = await getSecret(`${process.env.ENV}_DB_USER`);
 const dbName = await getSecret(`${process.env.ENV}_DB_NAME`);

 const token = getToken(rdsProxyEndpoint, dbUser);

 const connectionConfig = {
 host: rdsProxyEndpoint,
 user: dbUser,
 database: dbName,
 ssl: { rejectUnauthorized: false },
 password: token,
 authSwitchHandler: ({ pluginName, pluginData }, cb) =&gt; {
 console.log('Setting new auth handler.');
 },
 typeCast: (field, next) =&gt; {
 if (field.type === 'JSON') {
 return field.string();
 }
 return next();
 }
 };

 // Adding the mysql_clear_password handler
 connectionConfig.authSwitchHandler = (data, cb) =&gt; {
 if (data.pluginName === 'mysql_clear_password') {
 console.log('pluginName: ' + data.pluginName);
 const password = token + '\0';
 const buffer = Buffer.from(password);
 cb(null, password);
 }
 };

 console.log(connectionConfig);
 return connectionConfig;
 },
 pool: {
 min: minNum, 
 max: maxNum,
 createTimeoutMillis: 30000,
 acquireTimeoutMillis: 30000,
 idleTimeoutMillis: idleTimout,
 reapIntervalMillis: 1000,
 createRetryIntervalMillis: 100
 },
 debug: false
});

export default dbContext;</code></pre>That's it. Hope this helps.Updated (10/19/2020) &nbsp; We removed the pool settings <code>propagateCreateError: true. </code>Per Mikael (knex/tarn maintainer) -&nbsp;<blockquote>the setting, <code>propagateCreateError,</code> should never be touched.</blockquote>

In this article, we would like to share some issues we had and resolved during setting up AWS Lambda to use Amazon RDS Proxy.

Setup AWS Lambda to Use Amazon RDS Proxy

Recently, we noticed a ton of Errors in AWS CloudWatch Logs, the message like this:<blockquote>2020-08-21T17:43:32.142855Z 8812 [Note] Aborted connection 8812 to db: 'dbname' user: 'db_user' host: 'x.x.x.x' (Got an error reading communication packets)</blockquote>“Aborted connection" is a common MySQL communication error. &nbsp;The possible reasons are numerous includes:<ol><li>It takes more than connect_timeout seconds to obtain a connect packet.</li><li>The client had been sleeping more than wait_timeout or interactive_timeout seconds without issuing any requests to the server.</li><li>The max_allowed_packet variable value is too small or queries require more memory than you have allocated for mysqld.</li><li>The client program did not call mysql_close() before exiting.</li><li>The client program ended abruptly in the middle of a data transfer.</li><li>A connection packet does not contain the right information.</li><li>Any DNS related issues and the hosts are authenticated against their IP address instead of Hostnames.</li></ol>Yes, it's a long list, unfortunately.&nbsp;<h3>MySQL Tuning</h3>We went thru the list, the first thing we tried was increasing those three variables:<ol><li>max_allowed_packet = 500M</li><li>innodb_log_buffer_size = 32M</li><li>innodb_log_file_size = 2047M</li></ol>But they did not help, then we checked some timeout variables:<ol><li><code>wait_timeout</code> -this parameter refers to the number of seconds the server waits for activity on a non-interactive connection before closing it<ul><li>Allowed value: 1-31536000</li><li>The default value for MySQL 5.7: 28800</li></ul></li><li><code>connect_timeout</code> - this parameter the number of seconds that the server waits for a connect packet before responding with Bad handshake.<ul><li>Allowed value: 2- 31536000</li><li>The default value for MySQL 5.7:10</li></ul></li><li><code>interactive_timeout</code> - this parameter refers to the number of seconds the server waits for activity on an interactive connection before closing it.<ul><li>Allowed value: 1-31536000</li><li>The default value for MySQL 5.7: 28800</li></ul></li></ol>We have default values for all, so unlikely it is due to “The client had been sleeping more than wait_timeout or interactive_timeout seconds without issuing any requests to the server.”<h3>Application Troubleshooting</h3>It's enough for the MySQL tuning. We looked into the Application side. We use Lambda Serverless &nbsp;Apollo Server. The database connection is using KnexJS library. &nbsp;Our DB connection was something like this:<pre><code class="language-javascript">require('mysql');

const dbContext = require('knex')({
 client: 'mysql',
 connection: async () =&gt; {
 return {
 host: await getSecret(`${process.env.ENV}_DB_HOST`),
 user: await getSecret(`${process.env.ENV}_DB_USER`),
 password: await getSecret(`${process.env.ENV}_DB_PASSWORD`),
 database: await getSecret(`${process.env.ENV}_DB_NAME`),
 ssl: 'Amazon RDS'
 };
 },
 pool: {
 min: 2,
 max: 10, 
 createTimeoutMillis: 30000,
 acquireTimeoutMillis: 30000,
 idleTimeoutMillis: 30000,
 reapIntervalMillis: 1000,
 createRetryIntervalMillis: 100
 },
 debug: false
});

export default dbContext;</code></pre>Updated (10/19/2020) &nbsp; We removed the pool settings <code>propagateCreateError: true. </code>Per Mikael (knex/tarn maintainer) -&nbsp;<blockquote>the setting, <code>propagateCreateError,</code> should never be touched.</blockquote>There are many people having this issue on KnexJS GitHub issues channel (see the reference links below.) The key point here is that we are using serverless,&nbsp;<blockquote>Lambda functions are stateless, so there is no way to share a connection pool between functions. - rusher</blockquote>And our DB connection had pool setting as <code>min:2 / max: 10</code>. The good thing about connection pooling is that regarding wiki:<blockquote>In software engineering, a connection pool is a cache of database connections maintained so that the connections can be reused when future requests to the database are required. Connection pools are used to enhance the performance of executing commands on a database.</blockquote>The first expression for us was that since Lambda functions doe not share a connection pool, maintaining a connection pool becomes a waste here. Those connections sleep there do nothing, and they end up taking all the available connections. So as many people suggested, we changed the pool setting to be&nbsp;<pre><code class="language-javascript"> pool: {
 min: 1,
 max: 1, 
 ...</code></pre>Okay, after that, we saw some improvement. And we also did one thing:&nbsp;<pre><code class="language-javascript">const APIGatewayProxyHandler = (event, context, callback) =&gt; {
 context.callbackWaitsForEmptyEventLoop = false;</code></pre>By specifying <code>context.callbackWaitsForEmptyEventLoop = false</code>, we allow a DB connection to be maintained in a global variable in the lambda's container resulting in a faster connection.However, we noticed the performance issue after changing the connection pool to be <code>min:1/max:1</code>. Why? Did we just said, “Lambda functions doe not share a connection pool”? &nbsp;Let's be more clear, although Lambda functions do not share a connection pool, the queries in the same function do share a connection pool. For example, we have a GraphQL call, getCustomPages. When it's called, a DB connection is created, and then all queries in this call can share the connection pool, especially, those queries are written in an async way (i.e., with Promise) To demonstrate this, we can use MySQLWorkbench under Management/Client Connections. When the pool size is 1, we see only one connection created for one GraphQL call, but when the pool size is set to 2/10, we see more connections (about 10) are created for one call, and then we can see all queries are sharing those connections perfectly. We see page load time drops from ~10 seconds to a half-second by changing the connection pool from 1/1 to 2/10. That's because without sharing the pool, every query has to wait until another query finished before it can be executed. If a lambda function has thousands of queries in a loop, the big connection poll will truly boost the performance.&nbsp;So for better performance, we should still use the default pool connection. But because AWS Lambda functions do not share the pool properly, the pool connection will be open for each Lambda function call and end up using all of the available MySQL connections.&nbsp;After looking into <a target="_blank" rel="noopener noreferrer" href="https://github.com/vincit/tarn.js">tarn.js</a>, the library KnexJS used to manage the connection pool, we found out that the setting <code>idleTimeoutMillis</code> controls how many mill seconds before the free resources are destroyed. For example, if we set it to 30000, ie. 30 seconds, then the idle connections will be destroyed after 30 seconds. However, it will not destroy all sleeping connections, but keep the minimum number of connections set in the pool configuration. Since we cannot reuse the pool connection in the next Lambda function call, why not set the minimum to 0?Brilliant! The final solution we have is:<ul><li>Use the pool settings as: (Updated: 8/26/2020 - since the connections are not shared by next Lambda function call, we could set idelTiemoutMillis to be much shorter, and we changed it to 1000 from 30000.)</li></ul><pre><code class="language-javascript">pool: {
 min: 0,
 max: 10, 
 createTimeoutMillis: 30000,
 acquireTimeoutMillis: 30000,
 idleTimeoutMillis: 1000, //changed it from 30000 to 1000
 reapIntervalMillis: 1000,
 createRetryIntervalMillis: 100,
 propagateCreateError: true
 },</code></pre><ul><li>Set <code>context.callbackWaitsForEmptyEventLoop</code> to be <code>false</code> (Note: the DB connection outside of Lambda handler function)</li></ul><pre><code class="language-javascript">const APIGatewayProxyHandler = (event, context, callback) =&gt; {
 context.callbackWaitsForEmptyEventLoop = false;</code></pre><h3>The Ultimate Solution</h3>As we repeated many times above, the bottleneck of the AWS Lambda functions is that they don't share the resource. To better understand this, please check the following images:<figure class="image"><img src="https://images.deniapps.com/lambda-function-verwhelming-rds.png"><figcaption>Many concurrent connections kill Amazon RDS - Photo By <a target="_blank" rel="noopener noreferrer" href="https://blog.thundra.io/can-lambda-and-rds-play-nicely-together">thundra</a></figcaption></figure>Since each Lambda function is an individual process, it has to establish its connection to the DB instance. This design becomes very resource-intensive.Luckily, Amazon launched the preview of Amazon RDS Proxy in December 2019 and then made it generally available for both Mysql and PostgreSQL engines in<a target="_blank" rel="noopener noreferrer" href="https://aws.amazon.com/blogs/aws/amazon-rds-proxy-now-generally-available/"> Jun 2020</a>, which address this issue. We call it “the Ultimate Solution” here. :-)<blockquote>Amazon RDS Proxy allows applications to pool and share connections established with the database, improving database efficiency, application scalability, and security. RDS Proxy&nbsp;<a target="_blank" rel="noopener noreferrer" href="https://aws.amazon.com/blogs/database/improving-application-availability-with-amazon-rds-proxy/">reduces client recovery time</a>&nbsp;after failover by up to 79% for Amazon Aurora MySQL and by up to 32% for Amazon RDS for MySQL. Also, its authentication and access can be managed through integration with&nbsp;<a target="_blank" rel="noopener noreferrer" href="https://aws.amazon.com/secrets-manager">AWS Secrets Manager</a>&nbsp;and&nbsp;<a target="_blank" rel="noopener noreferrer" href="https://aws.amazon.com/iam">AWS Identity and Access Management</a>&nbsp;(IAM).</blockquote>Again, we borrow a beautiful diagram from <a target="_blank" rel="noopener noreferrer" href="https://blog.thundra.io/can-lambda-and-rds-play-nicely-together">Thundra</a>:<figure class="image"><img src="https://images.deniapps.com/lambda-function-with-rds-proxy.png"><figcaption>Many concurrent connections connect to Amazon RDS via RDS Proxy - Photo By <a target="_blank" rel="noopener noreferrer" href="https://blog.thundra.io/can-lambda-and-rds-play-nicely-together">thundra</a></figcaption></figure>Please follow the links to see how to set up Amazon RDS Proxy if you are interested. We are about the end of this long long post. Thank you for reading. &nbsp;Updated (08/27/2020): We added <a target="_blank" rel="noopener noreferrer" href="https://deniapps.com/blog/setup-aws-lambda-to-use-amazon-rds-proxy">AWS RDS Proxy</a>.References:<ul><li><a target="_blank" rel="noopener noreferrer" href="https://github.com/knex/knex/issues/1875">https://github.com/knex/knex/issues/1875</a> (How do I use Knex with AWS Lambda</li><li><a target="_blank" rel="noopener noreferrer" href="https://github.com/vincit/tarn.js">https://github.com/vincit/tarn.js</a> (Another Resource Pool)</li><li><a target="_blank" rel="noopener noreferrer" href="https://blog.thundra.io/can-lambda-and-rds-play-nicely-together">https://blog.thundra.io/can-lambda-and-rds-play-nicely-together</a> (Play Lambda and RDS Nicely Together)</li><li><a target="_blank" rel="noopener noreferrer" href="https://aws.amazon.com/blogs/aws/amazon-rds-proxy-now-generally-available/">https://aws.amazon.com/blogs/aws/amazon-rds-proxy-now-generally-available/</a> (AWS RDS Proxy)</li></ul>

Starting for AWS Cloudwatch logs, we spent a week looking into 'Aborted Connection" errors on RDS. Finally, we figured out that although Lambda functions do not share a connection pool, the queries in the same function do share a connection pool, so we should keep on using the pool, but we should set the pool with min 0, so it could be destroyed without taking space. And the ultimate solution is to use Amazon RDS.

Troubleshooting on MySQL RDS Aborted Connection Error with AWS Lambda

We have a serverless application running on AWS Lambda. Recently, we saw some warnings/errors in the AWS CodeBuild log as below, although the application is successfully built:<blockquote>Serverless: Deprecation warning: Starting with next major version, API Gateway naming will be changed from "{stage}{service}" to "{service}{stage}". Set "provider.apiGateway.shouldStartNameWithService" to "true" to adapt to the new behavior now. More Info: <a target="_blank" rel="noopener noreferrer" href="https://www.serverless.com/framework/docs/deprecations/#AWS_API_GATEWAY_NAME_STARTING_WITH_SERVICE">https://www.serverless.com/framework/docs/deprecations/#AWS_API_GATEWAY_NAME_STARTING_WITH_SERVICE</a></blockquote><blockquote>gyp ERR! configure error gyp ERR! stack Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/serverless/node_modules/snappy/.node-gyp'</blockquote><figure class="image"><img src="https://images.unsplash.com/photo-1571244470733-589930a67a8c?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=MXwxNTQwMTB8MHwxfHJhbmRvbXx8fHx8fHx8&amp;ixlib=rb-1.2.1&amp;q=80&amp;w=1080"><figcaption>Photo by <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/@diegitane?utm_source=DNX&amp;utm_medium=referral">Diego Fernandez</a> on <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/?utm_source=DNX&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>apiGateway.shouldStartNameWithService</h3>The first one is pretty straightforward. Regarding the documentation from serverless.com:<blockquote>Starting with v3.0.0, API Gateway naming will be changed from <code>${stage}-${service}</code> to <code>${service}-${stage}</code>.Adapt to this convention now by setting <code>provider.apiGateway.shouldStartNameWithService</code> to <code>true</code>.</blockquote>To fix this, we just need to add the following in serverless.yml:<pre><code class="language-javascript" id="XEAboOrmA">provider:
 ...
 apiGateway:
 shouldStartNameWithService: true</code></pre>And we also need to add a bunch of new IAM permissions to the role, otherwise, the build will fail due to the following errors:<blockquote>An error occurred: ApiGatewayRestApi - User: arn:aws:sts::514231989137:assumed-role/codebuild-NCCIH-API-service-role/AWSCodeBuild-dd964d15-ed63-4bb3-b152-48109817bbab is not authorized to perform: apigateway:PATCH on resource: arn:aws:apigateway:us-east-1::/restapis/9ohallxqbi (Service: AmazonApiGateway; Status Code: 403; Error Code: AccessDeniedException; Request ID: f81ccfd9-aa65-4aba-9016-e1c925dbae0a; Proxy: null).</blockquote>Check AWS docs for IAM policy examples for managing API Gateway APIs: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-iam-policy-examples.html<h3>gyp ERR!</h3>We are confused with this one. It seems to be failed on our local sometimes. However, everything works fine without that. Please leave your comment if you know why.<h3>What's gyp</h3><blockquote><code>node-gyp</code> is a cross-platform command-line tool written in Node.js for compiling native addon modules for Node.js. It contains a vendored copy of the <a target="_blank" rel="noopener noreferrer" href="https://github.com/nodejs/gyp-next">gyp-next</a> project that was previously used by the Chromium team, extended to support the development of Node.js native addons.</blockquote>We don't have node-gyp as a dependency in our package.json, so it must be used by one of our dependency, in our case it's <code>serverless</code>.The error we had is so obvious that the user who runs npm does not have the right permission to ‘<code>mkdir</code>’. We might use ‘root’ in the local to fix this, but how about a non-root user? &nbsp;When code is built &amp; deployed in the AWS serverless, it's not using ‘root’ user and cannot ‘sudo’ as well, so we need a way to allow a non-root user to run package scripts. The good news is that npm allows us to set a flag: ‘unsafe-perm’ to true to run within the context of the running script.So to fix this, we need to add a flag in buildspec.yml where we specify the install commands as below:<pre><code class="language-javascript" id="Sl9MayT0a">phases:
 install:
 runtime-versions:
 nodejs: 12
 commands:
 - npm install
 - npm install knex -g
 - npm install -g serverless --unsafe-perm</code></pre>&nbsp;&nbsp;&nbsp;&nbsp;

We have a serverless application running on AWS Lambda. Recently, we saw some warnings/errors in the AWS CodeBuild log as below, although the application is successfully built: provider.apiGateway.shouldStartNameWithService and gypErr. In the article, we will explain the issues and solutions.

Fix Serverless Build Errors on AWS Lambda

Our Apollo Server is running on AWS Lambda using Serverless Framework, and then it's accessed thru AWS API Gateway using Cognito User Pool as an authorizer.&nbsp;<blockquote>More information about Amazon Cognito User Pools can be found in <a target="_blank" rel="noopener noreferrer" href="https://deniapps.com/blog/how-tokens-work-in-amazon-cognito-user-pools">this article</a>.</blockquote><figure class="image"><img src="https://images.unsplash.com/photo-1600695268275-1a6468700bd5?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=MXwxNTQwMTB8MHwxfHJhbmRvbXx8fHx8fHx8&amp;ixlib=rb-1.2.1&amp;q=80&amp;w=1080"><figcaption>Photo by <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/@markusspiske?utm_source=DNX&amp;utm_medium=referral">Markus Spiske</a> on <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/?utm_source=DNX&amp;utm_medium=referral">Unsplash</a></figcaption></figure>Our frontend (CMS) is running on top of Create React App with Apollo Client.On the localhost, we use serverless offline, so there is not API Gateway, Cognito User Pool, and Lambda in between the Apollo Client and Apollo Server.We received a lot of complaints from the CMS users that they got the “Failed to Fetch” error a few times almost every day when they were working on the website. &nbsp;When that happened, the CMS cannot communicate to the API server. With console log enable, we saw the information like below:<blockquote>[Network error]: TypeError: Failed to fetch &nbsp;&nbsp;Access to fetch at ‘https://API-URI/graphql’ from origin ‘https://CMS-URL’ has been blocked by CORS policy: "No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request's mode to ‘no-cors’ to fetch the resource with CORS disabled.&nbsp;&nbsp;&nbsp;Failed to load resource: https://API-URL/graphql:1 net:ERR_FAILED&nbsp;</blockquote>The hard part to solve this issue is that:<ol><li>There wasn't useful information that could be found in the AWS Cloudwatch</li><li>We could not reproduce it on the local, or on production (with brief test)</li></ol>After spending a lot of time checking the network connection (the CMS can only access via VPN), API Gateway, Lamba, RDS proxy, MySQL database, etc., everything that is different between Localhost and Production, we still could not find the clues.&nbsp;<blockquote>We were pretty sure that the CORS is enabled on Apollo Server. &nbsp;Somehow the server's response does not send the No "Access-Control-Allow-Origin’ header, and the client does not set the request's mode to ‘no-cors’, &nbsp;so the browser throws this error.</blockquote><blockquote>More informatio about CORS error could be found in <a target="_blank" rel="noopener noreferrer" href="https://deniapps.com/blog/false-alarm-of-cors-error">this article</a></blockquote>So, how about eliminating the CORS log first? We need to see the real error log.&nbsp;<blockquote>All debugging starts from the log - Adam C.&nbsp;</blockquote>We took a second look at the API Gateway and found that we should manually add the “Access-Control-Allow-Origin” header to 4xx and 5xx response, otherwise, this header will be missing from the response, and then the real 4xx and 5xx error will be hidden, because the browser will complain CORS errors as shown above.After adding this header with the value “*”, we got the new error report from our CMS users. As expected, we saw the real error message:<blockquote>Failed to load resource: the server responded with a status of 401 ()&nbsp;&nbsp;[Network error]: ServerError: Response not successful: Reveived status code 401</blockquote>After finding the root of failure, the fixing became much more targetable. We quickly figured out that it's related to the Cognito user pool, which we use as an authorizer. The AccessToken and IDToken are set with a short expiration time of one hour, so if the user is on the website for more than one hour, &nbsp;he/she will use pass an expired token for API call, therefore, the 401 unauthorized error will be received. Also, after refreshing the page, everything is back to normal, that's because the lifetime of refreshToken is one day, and Amplify will auto-refresh the AccessToken and IDToken as long as the refreshToken is not expired.&nbsp;So the solution is clear that we just need to refresh the AccessToken/IDToken when they are expired, but we need to do this in the background without refreshing the page, otherwise, the user's unsaved work will get lost. &nbsp;We use Amplify React UI Component to handle user login. When the login is successful or the user is already logged in, the ID token is passed to Apollo authLink as below:<pre><code class="language-javascript" id="80kA0Le8f">Auth.currentAuthenticatedUser().then(async (user) =&gt; {
 const token = user.signInUserSession.idToken.jwtToken;
 const authLink = new ApolloLink((operation, forward) =&gt; {
 // Use the setContext method to set the HTTP headers.
 operation.setContext({
 headers: {
 authorization: token ? `Bearer ${token}` : ""
 }
 });

 // Call the next link in the middleware chain.
 return forward(operation);
 });
});</code></pre>The logic above is implemented in the App.js, which is the parent component of all, and it's only run once at the componentDidMount lifecycle, so without refreshing the page, the token is never changed. But as we learned above, when the token is expired, the 401 unauthorized error is received. The Apollo Client allows us to check for a certain failure condition or error code, and retry the request if rectifying the error is possible. Below is what we came up using the ErrorLink: (If you are not familiar with Apollo Link, check it out <a target="_blank" rel="noopener noreferrer" href="https://www.apollographql.com/docs/react/api/link/introduction/">here</a>)<pre><code class="language-javascript" id="hAPivEyJR">const handlErrorLink = onError(
 ({ graphQLErrors, networkError, operation, forward }) =&gt; {
 if (graphQLErrors)
 graphQLErrors.forEach(({ message, locations, path }) =&gt;
 console.log(
 `[GraphQL error]: Message: ${message}, Location: ${locations}, Path: ${path}`
 )
 );
 if (networkError) {
 if (String(networkError).match(/Received status code 401/i)) {
 return fromPromise(
 Auth.currentAuthenticatedUser()
 .then(async (user) =&gt; {
 return user.signInUserSession.idToken.jwtToken;
 })
 .catch((error) =&gt; {
 console.log("refresh token failed: ", error);
 // Handle token refresh errors e.g clear stored tokens, redirect to login, ...
 return;
 })
 )
 .filter((value) =&gt; {
 return Boolean(value);
 })
 .flatMap((token) =&gt; {
 const oldHeaders = operation.getContext().headers;
 // modify the operation context with a new token
 operation.setContext({
 headers: {
 ...oldHeaders,
 authorization: `Bearer ${token}`,
 },
 });
 return forward(operation);
 });
 }

 console.log(`[Network error]: ${networkError}`);
 }
 }
);</code></pre>Note the in our case, the 401 unauthorized is captured in networkError, and the error string contains “<code>Received status code 401”</code> which we used to filter, and <code>Auth.currentAuthenticatedUse,</code> the function provided by ‘Amplify’ is asynchronous, so we have to use <code>fromPromise,</code> the function provided by ‘apollo-link’ to token from a Promise object, and then replace the old header in operation context, finally forward the operation to retry.After this, we had the annoying ‘failed to fetch’ error fixed. :-)&nbsp;

In this article, we learned how to fix the 'failed to fetch' error from API-Gateway in our AWS Lambda Serverless Framework. The root cause is ended up to be the expired AWS Cognito Access/ID tokens. We were able to fix this by utilizing the Apollo Error Link to replace the old token with the refreshed one, and then retry the request.

Finally We Fixed "Failed to Fetch" Error

AWS provides us Amazon Cognito User Pools, which could be used as authorizer to control access to our application. Per <a target="_blank" rel="noopener noreferrer" href="https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-with-identity-providers.html">Amazon Doc</a>: &nbsp;Amazon Cognito user pools implement ID, access, and refresh tokens as defined by the OpenID Connect (OIDC) open standard:<ol><li>The <a target="_blank" rel="noopener noreferrer" href="http://openid.net/specs/openid-connect-core-1_0.html#IDToken">ID token</a> contains claims about the identity of the authenticated user such as <code>name</code>, <code>email</code>, and <code>phone_number</code>.</li><li>The <a target="_blank" rel="noopener noreferrer" href="https://tools.ietf.org/html/rfc6749#section-1.4">access token</a> contains scopes and groups and is used to grant access to authorized resources.</li><li>The <a target="_blank" rel="noopener noreferrer" href="https://tools.ietf.org/html/rfc6749#section-1.5">refresh token</a> contains the information necessary to obtain a new ID or access token.</li></ol>When we sign in the user pool using Amplify, we will obtain all those three tokens.&nbsp; &nbsp;<figure class="image"><img src="https://images.unsplash.com/photo-1475701240414-7842311479e7?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjE1NDAxMH0"><figcaption>Photo by <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/@genessapana?utm_source=DNX&amp;utm_medium=referral">Genessa Panainte</a> on <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/?utm_source=DNX&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Which Token to Call API?</h3>Normally, the ID token is for Authentication, and the access token is for Authorization. When calling the API method, we typically set the token to the request's Authorization header. So, it sounds like we should use the access token, doesn't it? Based on the doc, we could use either the ID token or the access token. However, &nbsp;as we tested, we should use the ID token. That's because:<ol><li>The ID token is used to authenticate users to our resource servers or server applications (ex: API)</li><li>The purpose of the access token is to authorize API operations in the context of the user in the user pool. For example, you can use the access token to grant your user access to add, change, or delete user attributes.</li></ol><h3>What's the Refresh Token?</h3>Usually, we set the access token/the ID token expiration to be much shorter than the refresh token expiration. And then when the access token/the ID token expires, we can use an unexpired refresh token to get a new access token/ID token without asking users to re-login. The reason behind this is that the access token/the ID token is used to API method, in case they are stolen, the short expiry time could help minimize the damage.&nbsp;<h3>Will the Refresh Token Expires?</h3>Yes, with Amazon Cognito User Pool, we can set the app's refresh token expiration to any value between 60 minutes and 10 years.&nbsp;<h3>How to Check if the Refresh Token Expired or Not?</h3>Different to the access token/the ID token, which is the JWT token where we can get the expiration date, we cannot tell if the Refresh Token Expired or not from the token. But we can tell it from the auth_time of the refresh token/the ID token. For example, we set the refresh token expiration to 1 day, then we can use the following equation to get the refresh token expiration DateTime:<pre><code class="language-javascript">const authTime = user.signInUserSession.accessToken.payload["auth_time"];
const refreshTokenExpirationTime = 
	new Date(authTime * 1000 + 24 * 60 * 60 * 1000)</code></pre><h3>Possible Issues When Refresh Token is Expiring</h3>We set the access token expiration to be 60 mins, and the refresh token expiration to be 1 day. We use the Amplify library, which auto-refreshes the token when the access token expires, we basically get the 1-day session duration. Note that when the refresh token expires, the user has to re-login to get the new access token, ID token, and refresh token. So if the user is working on something when both the ID token and the refresh token are expired, then the unsaved work will be lost.To understand how this works, let us use an example:<ul><li>The user logged in today at 9:00 am, he got an&nbsp;accessToken (valid until 10:am) and a refresh token (valid until 9 am tomorrow)</li><li>He comes back at 10:30 am, and his access token expires, but he is still logged in because his session is refreshed by the refresh token and he gets the new access token (valid until 11:30 am) and his refreshToken is still valid until 9 am tomorrow.</li><li>He comes back tomorrow at 8:30 am, he is still logged in, again thanks to the refresh token, and he gets a new access token valid until 9:30 am, then there is an issue because at 9:31 am, he has to re-login since&nbsp;his refresh token will be&nbsp;expired at that time. So his unsaved work will be lost at 9:31 am.</li></ul><h3>How to Handle When Refresh Token is Expiring</h3>If your application has an auto-save feature, like Gmail App, then how the refresh token works as we discussed above should be fine. Otherwise, we may look into some solution to manually log out the user at the right time to avoid losing unsaved work.<h4>Solution One</h4>Log out the user when the refresh token is about to expire in 60 mins, and we can check this when the page loads and when the user is active right after idle. In the other words, if the refresh token is not expired in 1 hour, then the user could get one more refresh, which could mitigate the session losing.&nbsp;<h4>Solution Two</h4>Implement the strict mode, for example, log out the user after the idle time is more than &nbsp;15 mins, which could be a popup to allow the user to extend the session (by setting a longer refresh token expiration.) Also, log out the user when we detect the refresh token expiration time is in a day at the initial page load. But this solution will sacrifice the benefit of the refresh token.We know both solutions are not great. So if you have any ideas, please leave your comments.

AWS provides us Amazon Cognito User Pools, which could be used as authorizer to control access to our application. The article will explain all three tokens implemented by Cognito User Pools, and discuss some common questions of how to use those tokens.

How Tokens Work in Amazon Cognito User Pools

My client's website is using AWS Amplify to handle the user log-in via AWS Cognito. It's quick to set up the login using <code>aws-amplify-react</code>. &nbsp;As of <code>aws-amplify-react@4.x.x</code>, the Authenticator is not styled, so we had installed the <code>aws-amplify/ui 2.0.3,</code> and imported the style from '<code>@aws-amplify/ui/dist/style.css'</code>. But it stopped working after updating to V3. It throws a ton of “Module not found” errors.&nbsp;<figure class="image"><img src="https://images.unsplash.com/photo-1613806667512-478ce88c5983?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=MnwxNTQwMTB8MHwxfHNlYXJjaHwxfHxucG18ZW58MHx8fHwxNjU2MzYxOTY2&amp;ixlib=rb-1.2.1&amp;q=80&amp;w=1080"><figcaption>Photo by <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/@jalalkelink?utm_source=DNX&amp;utm_medium=referral">Jalal Kelink</a> on <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/?utm_source=DNX&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Package path ./dist/style.css is not exported from package /Users/PROJECT/node_modules/@aws-amplify/ui (see exports field in /Users/PROJECT/node_modules/@aws-amplify/ui/package.json)</blockquote>After digging into this, we found out that we don't need to install <code>@aws-amplify/ui</code> at all, because it's a dependency of <code>aws-amplify-react</code>. Try ‘<code>npm list @aws-amplify/ui</code>’, it returns:<pre><code class="language-javascript" id="YmEZDKVdF">├─┬ aws-amplify-react@5.1.9
│ └── @aws-amplify/ui@2.0.5
└─┬ aws-amplify@4.3.26
 └── @aws-amplify/ui@2.0.5 deduped</code></pre>Since we don't use any UI elements except the login form, so there is no point to install the whole package of @aws-amplify/ui. Just import the the style like this:<pre><code class="language-javascript" id="dfUo0fpym">import "@aws-amplify/ui/dist/style.css";</code></pre>It would work without installing <code>@aws-amplify/ui.</code> If we install the V3, then there must be some conflicts between the V2 installed by <code>aws-amplify-react</code> and V3 itself.&nbsp;Note that, the ESlint will complaints no package found, like:<blockquote>'@aws-amplify/ui' should be listed in the project's dependencies. Run 'npm i -S @aws-amplify/ui' to add iteslint<a target="_blank" rel="noopener noreferrer" href="https://github.com/import-js/eslint-plugin-import/blob/v2.26.0/docs/rules/no-extraneous-dependencies.md">import/no-extraneous-dependencies</a></blockquote>We could ignore it by adding the line before the import.<pre><code class="language-plaintext" id="rNGv0NtH0">// eslint-disable-next-line import/no-extraneous-dependencies
import "@aws-amplify/ui/dist/style.css";</code></pre>That's it!

My client's website is using AWS Amplify to handle the user log-in via AWS Cognito. As of aws-amplify-react@4.x.x, the Authenticator is not styled, so we had installed the aws-amplify/ui 2.0.3, and then imported the style from '@aws-amplify/ui/dist/style.css'. But it stopped working after updating to V3. Here is the solution.

Issue When Upgrading aws-amplify/ui from 2.03 to 3.10.1

AWS provides two ways to store and manage your application configuration data.&nbsp;<ol><li>Parameter Store - it's designed to say any application variable like URLs, Token, etc.</li><li>Secrets Manager - as the name says, it is mainly for secret keys, like passwords and API Keys, because the entry is encrypted by default.</li></ol>Both access can be restricted through IAM, although Secrets Manager provides an additional layer of security. The main differences are the cost and secrets rotation. Check this <a target="_blank" rel="noopener noreferrer" href="https://medium.com/awesome-cloud/aws-difference-between-secrets-manager-and-parameter-store-systems-manager-f02686604eae">excellent article </a>to see the comparison.<figure class="image"><img src="https://images.unsplash.com/photo-1483706600674-e0c87d3fe85b?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=MnwxNTQwMTB8MHwxfHNlYXJjaHwxfHxzZWNyZXR8ZW58MHx8fHwxNjYzMjg4ODkw&amp;ixlib=rb-1.2.1&amp;q=80&amp;w=1080"><figcaption>Photo by <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/@tinaflour?utm_source=DNX&amp;utm_medium=referral">Kristina Flour</a> on <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/?utm_source=DNX&amp;utm_medium=referral">Unsplash</a></figcaption></figure>We can use <code>@aws-sdk/client-ssm</code> to retrieve the Parameters from Parameter Store, like below:<pre><code class="language-javascript" id="Jh5H4FiBj">import { SSMClient, GetParameterCommand } from "@aws-sdk/client-ssm";

const getParameter = async (parameterName) =&gt; {
 const params = {
 Name: parameterName,
 WithDecryption: true,
 };

 const command = new GetParameterCommand(params);
 const result = await ssm.send(command);
 return result.Parameter.Value;
};</code></pre>And we can use <code>@aws-sdk/client-secrets-manager</code> to retrieve the Secrets from Secret Manager, like below:<pre><code class="language-javascript" id="xb90F1EwH">import {
 	SecretsManagerClient,
	GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";

export const getSecretConnection = async (secretId) =&gt; {
 
 try {
 const command = new GetSecretValueCommand({
 SecretId: secretId,
 });
 const result = await secretsmanager.send(command);

 if (result) {
 return JSON.parse(result.SecretString);
 }
 } catch (error) {
 handleErrorLogging('GET DB CONNECTION FAILED: ', error,'');
 }
 return {};
};</code></pre>And today, I learned that we could &nbsp;use &nbsp;<code>@aws-sdk/client-ssm</code> to retrieve Secrets from the secret manager as well. That is called <a target="_blank" rel="noopener noreferrer" href="https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-ps-secretsmanager.html">Referencing AWS Secrets Manager secrets from Parameter Store parameters</a>.Just keep in mind,&nbsp;<blockquote>When you retrieve a Secrets Manager secret from Parameter Store, the name must begin with the following reserved path: /aws/reference/secretsmanager/<code>secret_ID_in_Secrets_Manager</code>.</blockquote>That's cool, we can simplify the code as below:<pre><code class="language-javascript" id="0msTa5qF-">import { SSMClient, GetParameterCommand } from "@aws-sdk/client-ssm";

const retrieve = async (key, type="secret") =&gt; {
. const parameterName = type === "secret" ? 
	`/aws/reference/secretsmanager/${key}` : key;
 const params = {
 Name: parameterName,
 WithDecryption: true,
 };

 const command = new GetParameterCommand(params);
 const result = await ssm.send(command);
 return result.Parameter.Value;
};</code></pre>Note:&nbsp;<ol><li>use <code>retrieve(key, “parameter”)</code> to retrieve the parameter</li><li>use <code>retrieve(key)</code> to retrieve the secret</li><li>you may add try-catch as I did in the previous example</li></ol>That's it!

Today, I learned that we could use @aws-sdk/client-ssm to retrieve Secrets from the secret manager as well. That is called Referencing AWS Secrets Manager secrets from Parameter Store parameters.

Use SSM to Retrieve Both Parameter Store Parameters and Secrets Manager Secrets

In the world of modern web development, keeping your tools and dependencies up to date is essential. Recently, I embarked on an upgrade journey, transitioning to npm v9 on my local machine. Little did I know that this seemingly innocuous upgrade would lead me to discover an issue with the AWS CodeArtifact command. In this blog post, I'll share my experience and the solution I found to make things right.<figure class="image image_resized"><img src="https://images.unsplash.com/photo-1613806667512-478ce88c5983?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=M3wxNTQwMTB8MHwxfHNlYXJjaHwxfHxucG18ZW58MHx8fHwxNjk1ODU3NzQwfDA&amp;ixlib=rb-4.0.3&amp;q=80&amp;w=1080"><figcaption>Photo by <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/@jalalkelink?utm_source=DNX&amp;utm_medium=referral">Jalal Kelink</a> on <a target="_blank" rel="noopener noreferrer" href="https://unsplash.com/?utm_source=DNX&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>The Problem Unveiled</h4>It all started when I upgraded my local npm to version 9.0. The upgrade went smoothly, but soon I encountered a baffling issue with the AWS CodeArtifact command. A cryptic error message stared back at me:<pre><code class="language-plaintext" id="Cz1y_kIjG">Command '['npm', 'config', 'set', '//123.codeartifact.us-east-1.amazonaws.com
/npm/my-repo-name/:always-auth', 'true']' returned non-zero exit status 1.
 ELIFECYCLE  Command failed with exit code 255.</code></pre>After some diligent investigation, I uncovered the root cause of the problem: NPM v9 had introduced a breaking change that affected the AWS CodeArtifact command.The breaking change, as it turns out, was related to npm's configuration system. In NPM v9, the <code>npm config set</code> command no longer accepts deprecated or invalid config options. This change had a direct impact on the AWS CodeArtifact command, which relied on a specific npm configuration.<h4>The NPM Configuration Conundrum</h4>To clarify, when you use AWS CodeArtifact to install packages, it requires authentication. This authentication is achieved by setting an npm configuration that includes the <code>:always-auth</code> option. Here's an example of what that configuration looks like:<pre><code class="language-plaintext" id="wwsM6G2Bl">//123.codeartifact.us-east-1.amazonaws.com/npm/my-repo-name/:always-auth=true</code></pre>This configuration is stored in your local <code>~/.npmrc</code> file, and it is crucial for AWS CodeArtifact to function correctly.With NPM v9, attempting to set this configuration using the <code>npm config set</code> command resulted in an error, as I experienced. This presented a serious obstacle to using AWS CodeArtifact with the latest NPM version.<h4>The Solution: Upgrading AWS CLI</h4>To resolve this predicament, the solution was clear: I needed to upgrade the AWS Command Line Interface (CLI) on my local machine. The version I had at the time was as follows:<pre><code class="language-plaintext" id="sXpApzfJc">$ aws --version
aws-cli/2.5.8 Python/3.9.11 Darwin/22.6.0 exe/x86_64 prompt/off
</code></pre>To upgrade AWS CLI to a version compatible with npm v9, I executed the following commands in my macOS terminal:Download the latest AWS CLI package:<pre><code class="language-plaintext" id="S8G5jMjkm">curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
</code></pre>Install the downloaded package:<pre><code class="language-plaintext" id="hJHZVfDbw">sudo installer -pkg ./AWSCLIV2.pkg -target /
</code></pre>After successfully upgrading AWS CLI to version 2.13.21, I retried the AWS CodeArtifact command, specifically <code>npm run ca:login</code>, and it worked flawlessly. The npm configuration was set as expected, and AWS CodeArtifact was back in action.&nbsp;

In the world of modern web development, keeping your tools and dependencies up to date is essential. Recently, I embarked on an upgrade journey, transitioning to npm v9 on my local machine. Little did I know that this seemingly innocuous upgrade would lead me to discover an issue with the AWS CodeArtifact command. In this blog post, I'll share my experience and the solution I found to make things right.

All Posts In Aws