It’s no secret (pun intended) that cybersecurity is big news these days. On a daily basis, we are hearing about data breaches, leaked personal information and stolen passwords. Many of these incidents are centered around mismanagement of credentials such as improper storage of encryption keys, databases with default or no password for the admin account and access keys stored in application source code in GitHub or in an unprotected object storage bucket . The problem is compounded when companies move to the public cloud where mistakes in security implementations are magnified.

Werner Vogels, CTO of Amazon Web Services, has proclaimed in recent conference keynotes that “Every developer should be a security engineer’ and that “Security is everyone’s responsibility.” Backing up words with actions, Amazon Web Services (AWS) has been busy pushing out new security services such as GuardDuty and default encryption for Amazon S3 buckets. One of the latest editions to their security portfolio is AWS Secrets Manager which was recently announced at the AWS Summit in San Francisco.

What is Secrets Management?

Secrets management has been a hot topic of late, even before AWS got into the game, thanks to the efforts of folks like those at Hashicorp. They has educating the IT community about secrets and their vision for managing secrets using their open-source tool, Vault. A secret, as defined by Hashicorp, is “anything that you want to tightly control access to, such as API keys, passwords, certificates, and more.” Secrets management involves the storage, protection, lifecycle management and auditing of secrets. This includes encrypting secrets so it can’t be viewed by anyone without access to the decryption key, rotating credentials on-demand or based on policies. and tracking for unauthorized access. And I would argue that the most important component is lifecycle management. Encrypting and storing secrets is not difficult; you could just use an encrypted S3 bucket or encrypted DynamoDB table for that task. But managing the lifecycle of a secret at scale and making sure that secrets are successfully rotated on a regular basis and are revoked when credentials are compromised is where users typically run into challenges.

Managing secrets has always been important but the need for it has increased with the adoption of cloud computing and microservices. While we may have previously only had to be concerned about secrets in one of two servers behind the corporate firewall, secrets now have to be protected and managed for fleets of  instances in the public cloud. It is, therefore, no surprise that AWS has seen fit to jump into this market.

What is AWS Secrets Manager?

According to the official user guide, AWS Secrets Manager is “an AWS service that makes it easier for you to manage secrets—such as database credentials, passwords, third-party API keys, and even arbitrary text—that you use to access AWS, on-premises, or third-party resources.” The canonical example AWS has been giving is the use case for accessing a large number of databases which have a common user credential. Instead of storing these credentials in plaintext on application servers that need access to those databases, user can store the credentials as encrypted secrets in AWS Secrets Manager. The application servers can then be modified to request the database credentials programmatically via API calls. When the credentials are periodically rotated, as they should be, Secrets Manager can do that automatically and it will be picked up by the application servers with minimal changes.

You can get more details on features and how-to’s from the user guide and Randall Hunt’s blog post. I want to focus, instead, on the architecture and some observations I have on the service.

Architectural Overview

The available resources on Secrets Manger is very light, which is to be expected from a new service. The product page and user guide essentially talks about Secrets Manager as a black box with integration to a few AWS services. Here’s what it looks like based on what has been published:

• The Secrets Manager service is responsible for storing secrets and encrypting them using keys provided by the AWS Key Management Service (KMS).
• Lambda functions provide automatic secrets rotation triggered by time-based events from a scheduler in Secrets Manager
• As stated in the user guide, CloudTrail and CloudWatch are used to monitor and log Secrets Manager with the Simple Notification Service (SNS) being leveraged for alerting.

If i were to go further and speculate, I think it is very likely that Secrets Manager is a serverless framework that integrates a collection of AWS services. It might look something like this:

• I think it is likely that secrets are being stored in S3 buckets, encrypted via KMS. It seems to be the most logical place to store a large amount of secrets that are being modeled as key value pairs. The secrets naming convention fits with S3 object naming conventions and encryption seems to be mapping to S3 default encryption using KMS.
• It looks to me like there are two categories of Lambda functions being used. One set of functions perform ad hoc management tasks such as creating and storing secrets  while a second set of functions, which can be customized and added to, is performing automated lifecycle management tasks such as secrets rotation. I assume that additional functions will be added out of the box for other tasks.
• If I am correct that all compute for Secrets Manager is leveraging Lambda, I think it is likely that AWS is using DynamoDB to store state information about tasks that are being run ad hoc or automated.
• As stated in the user guide, CloudTrail and CloudWatch are used to monitor and log Secrets Manager with SNS being leveraged for alerting. Since automated rotation is using a time trigger, I believe  that CloudWatch is also being used to provide that trigger.

It will be interesting to see how right and how wrong I end up being about the architecture as more details come out about Secrets Manager.

My Observations

I’ve had a chance to explore AWS Secrets Manager since it was announced and while it has some shortcomings that I will discuss later, I see a bright future for the service and believe it will get even better over time. Historically, AWS has released new services that target specific use cases and then expand those use cases over time by adding new features. Here are some specific observations I have.

What is Working

There is much to be commended with this first release of Secrets Manager, particularly if you are a heavy user of AWS services such as KMS and Relational Database Service (RDS).

• It is easy to get started with Secrets Manager using the console or the CLI. The console interface is clean and does a good job of walking a user through the steps to store and rotate their secrets.

The sample code generated when the secret is created and stored is helpful for updating the applications that will be accessing the secret.

In addition, the out of the box integration with AWS services like KMS and Identity Access and Management (IAM) makes getting started very easy for an existing AWS user.

• Secrets Manager is clearly designed to work best with RDS. If you use Secrets Manager with one of the supported RDS databases, creating, storing and automatically rotating  secrets are is extremely simple. The Lambda function to perform the rotation has already been created and made available.

If you are an RDS user and aren’t currently using a secrets management system, AWS Secrets Manager is pretty compelling with its out of the box integration.

• AWS is all about freeing users from having to perform “undifferentiated heavy lifting.” By offering secrets management as a fully managed service, Amazon removes the need for users to install and to maintain their own secrets management system. With one exception, which I will discuss in the next section, Secrets Manager abstracts away complexities such as encryption key management.

What Needs Improvement

Any product in its first release will have issues and AWS Secrets Manager is no exception. My main criticism of the service is the requirement to have a working knowledge of AWS Lambda for many use cases. Note that I am not critiquing the use of Lambda here but the fact that knowledge of Lambda is required.

To rotate secrets outside of what will be used with an RDS database requires you to know how to write functions in Lambda, the AWS serverless service. In fact, if you want to rotate secrets for a database that you are running on an EC2 instance and managing yourself, you have to write your own Lambda function to automate secrets rotation.

I expect that AWS will create new Lambda rotation functions to address use cases other than RDS. But until then, the need to write Lambda for non-RDS use cases adds complexity to the solution.

What is Missing

I would argue that version 1 of AWS Secrets Manager is more of a secrets store with some management than a full secrets management system. I expect that to change eventually but for now, the management in Secrets Manager seems to be limited to secrets rotation. Along those lines, here are a couple of  key features that I believe are missing:

• I was disappointed to see that Secrets Manager does not have support for dynamic secrets. A dynamic secret is generated on-demand and typically has a lease attached to it that expires the secret after a set time. An example of this would be an IAM user that is created only when an application needs to access a resource and the account is automatically deleted after 1 hour. You would be able to do the same with database credentials and API keys. Below is an example of how Hashicorp Vault implements dynamic secrets.

The benefits of dynamic secrets include mitigating against poorly secured application that may “leak” credentials in logs and artifacts. It also limits the impact of a compromised credential since credentials could be set up as a single-use one-time credential that is mapped to specific request.

• I was surprised to see that Secret Manager does not provide the ability to manually revoke secrets. Forced or manual revocation is a break glass procedure that a user should be able to enact in the event that a secret is compromised. I would like to see the ability for a user to revoke a secret which would override any rotation schedule and invalidate any credentials that is being reference in the secret.

Overall, AWS Secrets Manager is a  solid version 1 of a service that is particularly valuable for AWS users who are currently using or plan to use RDS. But Secrets Manager lacks the functionality and maturity of products such as Hashicorp Vault which also has the benefits of not being tied to a specific cloud provider.. However, I look forward to what is coming in future releases.