Automating AWS Lambda Layer Creation for Python with Makefile

AWS Lambda is an event-driven serverless computing service, which is a “done-for-you” service provided by AWS. It removes the hassle of server management from developers, allowing you to focus on what matters: the code.

The concept of Lambda is simple: You write code in your supported language of choice and upload it to a magic box, where it will (theoretically) be executed in response to defined triggers. However, veteran Lambda users will be familiar with the following error message:

deployment package of your Lambda function is too large to enable inline code

Yes, that’s right. You have deployed your code but can’t see it in your browser because it’s too large! For a service that talks about “done for you”, there’s always a catch.

The reason this error happens is because of the third party packages (python lingo) that you have included in your code. Your code may be just a couple of lines spread across 2 files but if it requires 15 different packages to tie everything together, it’s referred to as a “deployment package.” If your deployment package is too big, you’re out of luck- AWS Lambda won’t be able to show your code in the browser.

Don’t worry- there’s a workaround to this! This is where Lambda Layers comes into play. Lambda Layers are the zip archives that contain these “packages” (or libraries). The libraries enable you to add the dependency code in your Lambda without making it a part of your deployment package.

If you use Lambda Layers, all 15 of your dependencies will be included in a layer- but they’ll be hidden, enabling your Lambda to show the whole code in your browser. It’s an annoying problem, but the solution is simple.

How to automate the creation of Lambda Layers

Important Note: If you create a zip package from a non-Linux environment (even Mac), it will NOT WORK with Lambda. If you (like me) prefer developing on Mac or Windows, your best bet is to use an Ubuntu docker and build your layer there.

Your workflow could be as follows:

1. Pull an Ubuntu docker
2. Exec into it
3. Get your requirements.txt file in container
4. And install them at a specific path
5. Once everything looks good, zip up your layer
6. Get your .zip file

If you have to do it often, or for different Lambdas, the process can get tedious. Luckily, we have an automation script which can save you loads of time.

Here’s how you do it.

Create a Makefile with the following content.

Keep it in the same folder which has requirements.txt file. This will do everything we enlisted in the above steps, both sequentially & automatically.

Run the following command:

make get-lambda-layer

Which should create a file named packages.zip in your current directory.

Use this file to create your Lambda layer.

Exceptions:

1. This script is only for Python
2. This downloads 3.6.8 (as of today) and build on top of that. This should work for most Python 3+ versions
3. You need to have docker installed.

This script can be used as a good starting point for anyone who loves automation.

For Further Automation:

1. Create an image which already has everything pre-installed
2. Add different scripts for each language.
3. Go one step beyond, upload your zip to s3, and create a layer automatically.

I hope this helps you automate the annoying process of creating Lambda Layers!

If you find this blog helpful, be sure to give it a few claps or follow me on LinkedIn