Linux on Windows 10 via Docker

Wanting to run Linux command line under windows usually brings you to one of two options:

  1. Install WSL (Windows Subsystem for Linux) – the main advantage here is performance optimizations made by Microsoft to take the most advantage of the hardware and OS infrastructure.
  2. Install Docker for Windows 10 and then create a Linux container which you can run and get the command line prompt from

There are other options such as running a full graphical virtual machine like VirtualBox, VMWare and the like, but our focus will be getting a quick Linux command line prompt from Windows.

My personal preference is using option 2, mainly due to it being easy to install and the ability to easily create multiple sandboxed instances and easily use them for different purposes.

The following describes how to install the latest Ubuntu version and run it as a docker container.

  1. If you haven’t already done so, install Docker for Windows
  2. open a command prompt and type:
    docker pull ubuntu
  3. next, type:
    docker images
    copy the image id of the repository named ubuntu
  4. suppose the image id of the repository you copied was f6a191f19b5f, type:
    docker run -it --name=ubuntu-container f6a191f19b5f /bin/sh
  5. that’s it, you have the root prompt on the latest Ubuntu running on your Windows system!
  6. After you exit the ubuntu container, you can always enter it again by typing from the command prompt:
    docker start -i ubuntu-container

Training “cat or dog” on Windows without GPU

I was curious to see how much slower the training of chapter 1 of fast.ai would be on my GPU-less home desktop vs. a basic cloud GPU servers. The first issue was to get the code running after installing torch, torchvision and fastai modules. Apparently, there are some code modifications to make to get things running on my local system – I’ve summarized them here:

from fastai.vision.all import *

# To fix: Can't get attribute 'is_cat' on <module '__main__' (built-in)>
# https://stackoverflow.com/questions/41385708/multiprocessing-example-giving-attributeerror
from iscat import *

path = untar_data(URLs.PETS)/'images'

# added num_workers=0 to avoid:
# AttributeError: '_FakeLoader' object has no attribute 'noops'
# https://github.com/fastai/fastai/issues/3047

dls = ImageDataLoaders.from_name_func(
    path, get_image_files(path), valid_pct=0.2, seed=42,
    label_func=is_cat, item_tfms=Resize(224), num_workers=0 )

learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(1)

And the results ?

Well, the dataset used to train the network to distinguish between cats and dogs takes a decent GPU around 20 seconds to complete for each epoch.

Giving this task to my CPU took around 40 minutes per epoch.

Now you know why a GPU is highly recommended for machine learning, at least until someone will discover a more efficient way to do it (like this?)

cannot import name ‘mobilenet_v2’ from ‘torchvision.models’

This was the error I received when trying to import fastbook to study the excellent fast.ai course.

I had Python 3.9.1 installed and used pip to install the latest versions of torch, torchvision, and fastai

To spare the reader from boring war stories, here’s the bottom line:

  • Download the latest version of torchvision for your platform from https://download.pytorch.org/whl/torch_stable.html
    (for example cpu/torchvision-0.8.2%2Bcpu-cp39-cp39-win_amd64.whl is torchvision 0.8.2 for cpu on Windows for version 3.9 of C Python)
  • run pip install torchvision-0.8.2+cpu-cp39-cp39-win_amd64.whl (assuming this is the version you downloaded)
  • run pip install fastai --upgrade

You should now be able to import fastbook

Compiling PHP 7.3 with MySQL support for AWS Lambda

This is probably easier to do for PHP 7.4, but I wanted PHP 7.3 to make sure everything is compatible with existing code. The code below assumes you are doing this in an interactive shell in a Docker container build for creating binaries for AWS Lambda. Doing this is explained here

First, install the necessary tools to build PHP

$ yum update -y
$ yum install autoconf bison gcc gcc-c++ libcurl-devel libxml2-devel -y

Next, build the SSL version required for PHP 7.3 build

$ curl -sL http://www.openssl.org/source/openssl-1.0.1k.tar.gz | tar -xvz
$ cd openssl-1.0.1k
$ ./config & make & make install

Then download and build PHP 7.3, and configure with all the support for accessing MySQL databases

$ cd ..
$ curl -sL https://github.com/php/php-src/archive/php-7.3.0.tar.gz | tar -xvz
$ cd php-src-php-7.3.0
$ ./buildconf --force
$ /configure --prefix=/home/ec2-user/php-7-bin/ --with-openssl=/usr/local/ssl --with-curl --without-libzip --with-zlib --enable-zip --with-pdo-mysql --with-mysqli=mysqlnd
$ make install

Finally, check that the MySQL modules are there:

$ /home/ec2-user/php-7-bin/bin/php -m

Backing up AWS lambda functions

There might be a built-in method to do so, but I haven’t found one. I wanted to download packages of all the lambda files I have in AWS and ended up creating a script to do it. Feel free to copy. Don’t forget to chmod u+x … 🙂

#!/bin/bash

# Get all the function names (assuming my region is us-east-1)
list=`aws lambda list-functions --region us-east-1 | jq -r .Functions[].FunctionName | perl -pe 's/\n/ /g'`

# For each lambda function, get the function's download url and download it
for val in $list; do
url=`aws lambda get-function --region us-east-1 --function-name $val --profile lambda_user | jq -r .Code.Location`
wget $url
done

# Rename all the downloaded files by removing the long url params and adding a zip extension
find . -type f -name '*versionId*' -print0 |
while IFS= read -d '' file_name; do
newname=`echo $file_name | perl -pe 's/^(.+)\?versionId=.+/\1.zip/g'`
mv "$file_name" "$newname"
done

Duplicated log lines using Python in AWS Lambda

The short version:

logname = 'processing'
formatter = logging.Formatter(fmt='%(asctime)s %(levelname)-8s %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger = logging.getLogger(logname)
logger.setLevel(logging.DEBUG)
logger.handlers = [] # === Make sure to add this line
logger.addHandler(handler)
logger.propagate = False # === and this line

For details, check out this link

Standard C++ libraries on AWS Lambda

While attempting to compile a test sample together a library I needed, I received the following

/usr/bin/ld: cannot find -lstdc++

installing the following solved the issue. I didn’t even check if both installations are necessary – if you have the curiosity to dig further then send your conclusions, but all I wanted was a solution to the problem at hand, and here it is:

yum install libstdc++-devel
yum install libstdc++-static

lupa (LuaJIT/Lua-Python bridge) on AWS Lambda

Last post discussed launching LuaJIT from Python on AWS Lambda.

Suppose you want to receive events from the Lua code in the Python code that invoked it.

If you don’t have a Python-Lua API then you are left with either using sockets, files or even getting feedback by reading the realtime output from the process stdout object (which is not that trivial, see this implementation)

A more elegant way would be if there was a Python API for accessing the Lua code (and vice versa). This is what lupa does, and it works with both Lua and LuaJIT.

I’ll explain how to install lupa so it can be used with Python on the amazon linux docker container, and thereby on AWS Lambda.

So, Connect to your amazon linux docker container and

mkdir -p /root/lambdalua
cd /root/lambdalua
# Download the lupa source code
# link for latest source: https://pypi.org/project/lupa/#files - look for tar.gz file
wget https://files.pythonhosted.org/packages/f2/b2/8295ebabb417173eaae05adc63353e4da3eb8909e1b636b17ff9f8e1cdce/lupa-1.7.tar.gz
# extract the source in /root/lambdalua
tar xzfv lupa-1.7.tar.gz 
# enter the lupa source directory
cd lupa-1.7
wget http://luajit.org/download/LuaJIT-2.0.5.zip  # download latest source of LuaJIT
unzip LuaJIT-2.0.5.zip                            # unzip it in /root/lambdalua/lupa-1.7
cd LuaJIT-2.0.5                                   # genter LuaJIT source directory
make CFLAGS=-fPIC                                 # Make LuaJIT with -fPIC compile flag
cd ..                                             # back to lupa source dir
python setup.py install                           # Create lupa Python module
cd /root/lambdalua
mkdir lupa_package             # create a directory for a lupa test aws package

# copy the Lupa module that we previously compiled to the lupa_package directory
# so that the "import lupa" will work from the python code on AWS lambda
cp -r ./lupa-1.7/build/lib.linux-x86_64-3.6/lupa lupa_package/

Now create /root/lambdalua/lupa_package/lambdalua.py so that it makes use of Lupa:

import subprocess
import sys
import os

import lupa                 # use the local lupa module
from lupa import LuaRuntime

def lambda_luajit_func(event, context):

    def readfile(filename):
        content = ''
        with open(filename, 'r') as myfile:
            return myfile.read()

    lua = LuaRuntime(unpack_returned_tuples=True)

    # define fhe Python function that will be called from LuaJIT
    def add_one(num):
        return num + 1

    # Load the Lua code
    lua_func = lua.eval(readfile('./test.lua'))

    params = { "add_one_func":add_one, "num":42 }
    # call the Lua function defined in test.lua with the above parameters
    res = lua_func(params)

    return res

if __name__ == "__main__":
    print(lambda_luajit_func(None, None))

and also create the /root/lambdalua/lupa_package/test.lua that the above python file will load:

function(params_table)
    -- get the number passed from python
    num = params_table.num
    -- get the python function to invoke
    py_func = params_table.add_one_func

    -- return the result of invoking the Python function on the number
    return "The result is:"..py_func(num)
end

To see that this actually works in our amazon linux docker, just type:

python lambdalua.py # output is: The result is:43

To make this run on AWS lambda, we pack the contents of /root/lambdalua/lupa_package in a zip file.

zip -r package.zip .

If the AWS credentials and command line is in your OS shell, then from an OS terminal copy the zip file to your OS from the docker container, e.g (you can skip this step if you copied the aws credentials to ~/.aws in the docker container):

docker cp lucid_poincare:/root/lambdalua/lupa_package/package.zip .

Create a function (if we haven’t done so already) or update the function (if we want to update the function from the previous post) and invoke it.

The details of creating the lambda user, profile, function etc from the aws command line are detailed in a previous post, but a quick overview assuming you already have a lambda user names lambda_user:

Next, assuming you already have a lambda account and user (if you don’t, see a previous post) either create a new function

aws lambda create-function --region us-east-1 --function-name lambda_luajit_func --zip-file fileb://package.zip --role arn:aws:iam::123456789012:role/basic_lambda_role --handler lambdalua.lambda_luajit_func --runtime python3.6 --profile lambda_use

or, if you already created the function lambdalua.lambda_luajit_func from the previous post, you can update it:

# the following assumes myzippackage is a bucket you own
aws s3 rm s3://myzippackage/package.zip              # remove previous package if exists
aws s3 cp package.zip s3://myzippackage/package.zip  # copy new package

# update lambda_luajit_func function with contents of new package
aws lambda update-function-code --region us-east-1 --function-name lambda_luajit_func --s3-bucket myzippackage --s3-key package.zip --profile lambda_user

finally, invoke the Lupa test on AWS lambda

aws lambda invoke --invocation-type RequestResponse --function-name lambda_luajit_func --region us-east-1 --log-type Tail  --profile lambda_user out.txt
cat out.txt # output should be: "The result is:43"