Serverless architectures are a hot topic right now, popularized in large part by the growing NoOps movement. Products like Heroku, Elastic Beanstalk, and Convox aim to build a layer of abstraction between software engineers and the operations environment surrounding their code. Lambda takes yet another step past automating operations by eliminating them entirely; Lambda functions are stateless, created and destroyed per request.
Because Lambdas are such a raw, context-driven solution, deployment processes are equally raw and context-driven. The only solution provided out of the box is to zip your Node.js package (or Python, or Java, though in this post we’ll be focusing on Node.js) – node_modules folder included, try not to gasp too loudly – compress it to a zip file, and manually upload it through Amazon’s UI, or use their cli tools to do an API call.
I’ll go out on a limb at this point and assume we’re thinking the same thing: that’s probably not a tenable or scalable solution. It’s time to automate.
Let’s survey the landscape of tooling available to us. Lambda is an offering from AWS (Amazon Web Services), which means we have the entirely of AWS at arm’s reach to support our Lambda deployments. We also have tools such as Terraform, a Hashicorp offering that automates AWS infrastructure setup and teardown. For our Lambda, we settled on the following tooling:
- Node.js 4.3
- Gulp.js (with gulp-clean, gulp-zip)
- Another Lambda function (we’ll discuss this later)
- The S3 Node.js module
The Basic Idea
The way the deployment system works is like this: we have two Lambda functions, one of which is the actual Lambda we want to deploy, and one of which is the Lambda function that actually deploys other Lambdas. Let that sink in for a second.
The flow looks like this: we have a task in Jenkins to deploy a Lambda. The Jenkins task calls a Gulp.js task called deploy. That Gulp.js task runs clean (delete any existing archives), build (zip everything up), and then deploys the archive to an S3 bucket specified in an environment variable.
The Lambda function that deploys other Lambda functions is listening for events on our S3 bucket. So, when a new archive is uploaded to the bucket, that Lambda function runs and calls updateFunctionCode on the other Lambda function to update it from the archive it received in S3. When the event says a file was deleted, the function looks up previous revisions of the file and restores the most recent non-deleted version, again calling updateFunctionCode to do so. There’s also a Gulp.js task called revert that will do this manually – which is called by Jenkins for rollbacks.
Here’s what it looks like:
Everything within AWS – the API Gateway, S3 Bucket, Lambda definitions, policies, roles – is provisioned by Terraform.
As you can see in the diagram, we also use NGINX to reverse proxy requests to an AWS API Gateway, which calls the Lambda function. This approach means that you can effectively build an API layer with Lambda functions and very little operational overhead.
The Web Request
If you’re using Lambda to build an API endpoint, you’re going to need a way to get that request all the way through to Lambda. That’s where Amazon’s API Gateway comes in handy.
API Gateway will do exactly what the name promises – provide an HTTP endpoint that acts as an API Gateway into a number of different services, including Lambda. There’s one caveat with Lambda, however: it can only be invoked via an HTTP POST. So, when setting up the integration between API Gateway and Lambda, regardless of the incoming HTTP verb, make sure that the gateway POSTs to the Lambda.
API Gateway will give you a kind of nonsensical-looking URL, so if you’re consuming this elsewhere in your codebase you probably want another layer on top of that. For us, this came down to either using Cloudfront or our existing NGINX setup to forward requests on from our existing API request paths to the new endpoint. We decided to set up an NGINX reverse proxy that passed the request on to API Gateway. This actually gets a little tricky as well, because of the expectations API Gateway has for secure handshakes. Our proxy ended up looking like this:
This does a few things: it uses Google’s DNS to perform lookups, enables TLS for the proxy (which API Gateway expects), and then passes any query parameters on.
So now we have an NGINX reverse proxy that sends a request to API Gateway, which invokes a Lambda function and pipes the response all the way back upstream.
If you find this kind of work interesting – or even if you don’t, but you love music and code – we’re hiring.