In a previous post, I mentioned using Amazon Linux EC2 to create AWS Lambda compatible packages. While this works, another way to create packages that can run on AWS Lambda is to create them locally via a Docker Amazon Linux image . One downside I’ve found to this method is that sometimes these images are incompatible with some of the system files in the Lambda runtime, but at the time of writing this, I found the docker-lambda project to both create compatible lambda linux images as well as a great way to shorten lambda development cycles by emulating a lambda environment you can invoke locally.
To start, here are the instructions to build a Python 3.6 docker lambda image (of course, make sure you have Docker installed):
git clone https://github.com/lambci/docker-lambda.git # clone project from git cd docker-lambda/ # go to project directory npm install # install project node.js dependencies cd python3.6/build # go to the python Dockerfile build docker build . # build the image as per instructions in # the Dockerfile (takes time...) docker images # show docker images, note the id of the built image docker tag 32e7f5244861 lambci/python3.6:build # name and tag the built docker image using its id docker run -it lambci/python3.6:build /bin/bash # create a new container based on new image and # run it interactively (/bin/bash command is needed # because CMD ["/bin/bash"] is not included as the # last line in the Dockerfile exit # leave docker container docker ps -a # locate the newly created container from the above # above command, and note the name given to it docker start -i vibrant_heyrovsky # resume interactive session with the container # using the container name found above
So, now you have a console to a compatible Amazon Linux shell. To create lambda functions, you basically zip all the relevant files and upload to AWS lambda and after that, you can remotely invoke the required function on Lambda .
My current method will be to have two console windows – one is the above console to the docker bash, and another is a console of the host operating system (whatever OS you are running Docker on). This way, you can easily zip the lambda packages in the Docker console, and then copy them from your OS console (and from there upload them to AWS Lambda)
Setting up an AWS lambda user ∞
Now that we have a local Lambda-compatible environment, let’s create an actual AWS user that will be used to upload and run the packages that we’ll create in our local Lambda-compatible Docker container.
To run the following, make sure you first have the AWS CLI installed on your OS.
Let’s create our lambda user using the above CLI. Of course, the assumption is that you already have a credentials file in your .aws directory which enable you to do the next part. If not, you’ll need to create a user with the appropriate privileges from the AWS IAM console, get that user’s aws key id and aws secret, then locally run aws configure and follow the instructions. This will create your initial credentials file.
We’ll now create a user that we’ll use for AWS lambda. The information here is based on this excellent simple tutorial with some minor changes to suit this one.
# Create a user group 'lambda_group' $ aws iam create-group --group-name lambda_group # Create a user 'lambda_user' $ aws iam create-user --user-name lambda_user # Add our user to the group $ aws iam add-user-to-group --user-name lambda_user --group-name lambda_group # Create a password for this user $ aws iam create-login-profile --user-name lambda_user --password _your_password_here_ # Create a CLI access key for this user $ aws iam create-access-key --user-name lambda_user # Save user's Secret and Access Keys somewhere safe - we'll need them later
Now that we have a user, let’s authorise this user to run lambda functions, copy s3 files etc. To do this, we create a policy and grant that policy to the user we just created.
For that, create a file with the following json, and name it lambda_policy.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"iam:*",
"lambda:*",
"s3:*"
],
"Resource": "*"
}]
}
now grant the above policy to our lambda user:
aws iam put-user-policy --user-name lambda_user --policy-name lambda_all --policy-document file://lambda_policy.json
Now, let’s configure our AWS CLI so that we can perform actions as lambda_user
$ aws configure --profile lambda_user > AWS Access Key ID [None]: <your key from the above create-access-key command> > AWS Secret Access Key [None]: <your secret from the above create-access-key command> > Default region name [None]: us-east-1 (or whatever region you use) > Default output format [None]: json # AWS stores this information under [lambda_user] at ~/.aws/cretentials file
Finally, we need to create a role which is needed when creating a lambda function and determines what actions the lambda function is permitted to perform.
To create the role, create a file named basic_lambda_role.json with the following json text:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": { "AWS" : "*" },
"Action": "sts:AssumeRole"
}]
}
Then create the role via the CLI:
$ aws iam create-role --role-name basic_lambda_role --assume-role-policy-document file://basic_lambda_role.json
The above will return the role identifier as an Amazon Resource Name (ARN), for example: arn:aws:iam::716980512849:role/basic_lambda_role . You’ll need this ARN whenever you create a new lambda function so hold on to it.
We now have all the ingredients to create, update and invoke AWS Lambda functions. We’ll do that later, but first – let’s get back to creating the code package that is required when creating a lambda function. The code package is just a zip file which contains all your code and its dependencies that are uploaded to lambda when you create or update your lambda function. The next section will explain how to do this.
Creating a AWS Lambda code package ∞
We’ll start with creating and invoking a python package that has some dependencies, and then show how to create a package that can run arbitrary executables on AWS Lambda
Creating a local Python 3.6 package
So now, let’s make a package example that will return the current time in Seoul. To do this, we’ll install a python module named arrow, but we’ll install it in a local directory since we need to package our code with this python module. To do this, open your docker console that is running the lambda compatible environment and:
cd /var/task # move to the base lambda directory in the docker image mkdir arrowtest # Create a directory for the lambda package we're going to make cd arrowtest # move in to the directory pip install arrow -t ./ # install the arrow python library in this directory ls # take a look at what has been added
next, we’ll create our lambda function which we’ll later invoke. (you might want to install an editor of your choice on the docker console using yum, for example via yum install vim).
So, let’s create arrowtest.py :
import sys import arrow def lambdafunc(event, context): utc = arrow.utcnow() SeoulTime = utc.to('Asia/Seoul') return "The time in Seoul is: %s" % (SeoulTime.format()) #just for local testing if __name__ == "__main__": print(lambda_func(None, None))
and test that it works locally in the docker shell:
python arrowtest.py
Ok, so we have the python file with the lambda function, we have the dependencies, now all we need to do is zip the contents of the entire directory and add this zip file as a parameter to the lambda function creation.
This would work, however with larger Python libraries, you might want to remove certain files that aren’t being used by you python code and would just waste space on lambda. My rather primitive but effective method for doing this is cloning the complete directory and start removing files that seem pointless until something breaks, and then I put them back and try other things until I’m happy with the size reduction. In the cloned directory, I actually rename directories before removing them as it’s easier to run the script after renaming and rename them back if we see that the directory is needed by the script.
Let’s do it for this example:
cd .. pwd # should be /var/task cp -r arrowtest arrowtest_clone cd arrowtest_clone ls # let's see what's in here du -hd1 # note how much space each directory takes (1.2MB)
Installed python libraries can contain many directories and files of different types. There are python files, binary dynamic libraries (usually with .so extensions) and others. Knowing what these are can help decide what can be removed to make the zipped package more lean. In this example, the directory sizes are a non issue, but other python libraries can get much larger.
an example of some stuff I deleted
rm -rf *.dist-info rm -rf *.egg-info rm six.py rm -rf dateutil # we're not making use of this - it's just wasting space # test that the script is still working after all we've deleted python arrowtest.py test du -hd1 # we're down to 332K from 1.2MB and the script still works.
now, let’s package this directory in a zip file. if you don’t have zip installed on your docker container yet then
yum install zip
and now after removing unneeded files and dependencies, let’s pack our directory:
zip -r package.zip .
now that we have the package on the docker container, let’s copy it to our OS from our OS console:
docker cp vibrant_heyrovsky:/var/task/arrowtest_clone/package.zip .
(replace vibrant_heyrovsky with the name of your docker image).
So we have a zipped package that we tested on docker – let’s create a lambda function from this package and invoke it (replace arn:aws:iam::716980512849:role/basic_lambda_role with your own ARN):
aws lambda create-function --region us-east-1 --function-name lambdafunc --zip-file fileb://package.zip --role arn:aws:iam::716980512849:role/basic_lambda_role --handler arrowtest.lambdafunc --runtime python3.6 --profile lambda_user
and finally, let’s see if we can get AWS lambda to tell us the current time in Seoul:
aws lambda invoke --invocation-type RequestResponse --function-name lambdafunc --region us-east-1 --log-type Tail --profile lambda_user out.txt #invoke the function cat out.txt # check the result
the file out.txt contains the return value of the called lambda function. Next we’ll see how to update to a new package and how to pass parameters to the lambda function.
To be continued…