Running Python Script from PHP as www-data

The problem
Python script invoked from PHP via shell_exec and runs fine when PHP invoked from command line but fails when PHP triggered by browser access.

Reason
PHP, when triggered by a browser access is invoked by web server with the user www-data, while from the command line it is run as user ubuntu.

Attempting to see what happens when running PHP from the command line as user www-data would help to understand why running the script fails

One method to run php as www-data from the command line is to enable a shell for www-data user. This is done by modifying /etc/passwd so that user www-data has a shell (change the existing /usr/sbin/nologin or whatever to /bin/bash or something similar) and then sudo su www-data and try to run the python script again (see this reply for details).

Doing the above, quickly showed that the one of the imports in the python script fails when running it under www-data.
Comparing python3 -m site when running under user www-data vs. when running under ubuntu showed there is a difference in the module search paths.

Adding the missing path found for user ubuntu to user www-data via sys.path.insert was not scalable, nor possible (since the ubuntu user path is inaccessible to the www-data user), so the best way was to install the python modules (in my case, imagehash) in a way that will be accessible to the www-data user

The solution, found here illustrated how this is done

sudo mkdir /var/www/.local
sudo mkdir /var/www/.cache
sudo chown www-data.www-data /var/www/.local
sudo chown www-data.www-data /var/www/.cache
sudo -H -u www-data pip install imagehash

Quick, fresh Ubuntu 16.04 image in VirtualBox

Download the VirtualBox image from https://www.virtualbox.org/wiki/Downloads

Download the 16.04 Ubuntu image for VirtualBox from www.osboxes.org (the credential details for logging in are here: https://www.osboxes.org/faqs/)

Open VirtualBox, click the New icon, Select ‘Linux’ for type, ‘Ubuntu 64-bit’ for version (assuming you downloaded the VirtualBox 64bit vdi), click next a couple of times until you reach the ‘Hard Disk’ section and then click the ‘use an existing virtual hard disk file’ radio button
and browse for the vdi file you downloaded.

Log in using the credentials given (osboxes.org as password). After logging in it is recommended to install the guess additions (to be able to have things like shared folders and shared clipboard). Do this by selecting Devices->Insert Guest additions CD image… from the VirtualBox menu. This will add a CD icon to the unity bar on the left, meaning the guest additions CD has been mounted. Open a terminal window and type the following at the command prompt:
sudo /media/osboxes/VBox_GAs_5.2.8/VBoxLinuxAdditions.run
This will install the guest additions on the Ubuntu guest system. Ignore the #modprobe vboxsf failed# error at the end and restart the system via typing sudo restart at the command line
After successfully installing the guest additions you can eject the CD (right-click the CD icon and select eject)

After logging in again, open the terminal window again and do:
sudo apt-get update
sudo apt-get upgrade

The above update and upgrade will take a few minutes so after it is done, it is recommended to export your image should you want to have run the updated VirtualBox image on another machine without going through the entire process above yet again. Do this by closing the window and selecting ‘Power off the machine’. Next, Open the Oracle VirtualBox manager. Select File->Export Appliance from the menu, select the name of machine and click Next a couple of times. The resulting file/files can then be imported to a different VirtualBox app.

Selecting columns in Sublime Text turning your screen upside down?

The way to select a column in Sublime Text on Windows is by using + and then pressing the up or down arrow keys. You might get a surprise though if you try it in the form of you entire screen display flipping upside down. Some people don’t even know this is possible to do so they get extra surprised.

This is because on certain Intel graphic cards, there are the hotkeys to trigger certain functions of the graphic card, for example being the functions to flip and rotate the screen (this is handy for instance if you want to work with your screen physically rotated to be in portrait rather than landscape mode and you’d obviously want to render the desktop accordingly). The keys to select a text column in Sublime Text happens to be taken by the hotkey for flipping the screen by the graphic card.

To disable this key combination being hijacked by the graphic card, you can either customize the graphic card to use a different key combination (Ctrl+Alt+F12 > Options) or disable the hotkeys for graphic functions altogether (Ctrl+Alt+F12 > Options > Uncheck “Enable Hot Keys”)

Found the above thanks to a comment here (by L_7337)

Importing an .ics file to Google calendar

Someone sent you an invitation to an event as an .ics file, and you wish to add it to your Google calendar

The first part is to import the file. To do this, click the + icon to the right of the “Add a friends calendar” located on the left side of the page. After clicking the icon, from the available options select ‘Import’

Now, the obvious thing to do would be to just to click the button that says “Select file from your computer”, select the .ics file and click import, but if you would do that, chances are that you’ll see the dreaded:

‘Failed to import events: Could not upload your events because you do not have sufficient access on the target calendar..’

The solution for this is to manually edit the .ics file prior to importing it and replace all occurrences of “UID:” with “UID:X” (without the quotes). After doing this and saving the file, proceed with the import and all should be fine.

Learned this from here

Notepad tricks in Google Docs

Many people are not aware that since the early days of the simple notepad app that comes bundled with Windows, it had the following undocumented feature: If you enter the text .LOG as the first line of the file, then every time you open the file with notepad, it will append the current date and time to the end of the document and scroll there. This is quite handy when you want a file that keeps track of the time when you added new entries.

I wanted to have the same functionality with Google Docs (with the added benefit of not needing to write .LOG at the beginning of the file). The following script (built via multiple shameless plagiarism from various sources) enables that functionality:

function onOpen() {
  var ui = DocumentApp.getUi();
  // Or FormApp or SpreadsheetApp.
  ui.createMenu('Custom Menu')
      .addItem('Insert Date', 'insertDate')
      .addToUi();

  setCursorToEnd()
  insertDate();
  setCursorToEnd()
}

function setCursorToEnd()
{
  var doc = DocumentApp.getActiveDocument();
  var paragraph = doc.getBody().appendParagraph('');
  var position = doc.newPosition(paragraph, 0);
  doc.setCursor(position);
}

function formatDate()
{
  var date = new Date();
  var datestr = date.getFullYear() + '-'+  
    ('0' + (date.getMonth()+1)).slice(-2) + '-' + 
    ('0' + date.getDate()).slice(-2);  
  var hours = date.getHours();
  var minutes = date.getMinutes();
  var ampm = hours >= 12 ? 'pm' : 'am';
  hours = hours % 12;
  hours = hours ? hours : 12; // the hour '0' should be '12'
  minutes = minutes < 10 ? '0'+minutes : minutes;
  var strTime = hours + ':' + minutes + ' ' + ampm;
  return datestr + "  " + strTime;
}

function insertDate()
{
  var activeDoc = DocumentApp.getActiveDocument();
  var cursor = activeDoc.getCursor();
  if (cursor) {
      var date = formatDate();    
      var element = cursor.insertText(date);
  } else {
      DocumentApp.getUi().alert('Cannot find a cursor in the document.');
  }
}

To enable this for a document, In Google Docs : Tools -> Script Editor, enter and save the above script and then accept the permissions.

Status of specific brew service

Switching between various development environments, to save restart time, I wanted to start various services from OS/X only if they were not already running. Brew does not have a method to know whether a particular service is running (AFAIK), so the following example code parses the list of services returned by brew services list and then enables quickly checking the status of a given service and acting accordingly:

#!/bin/bash
brew services list | awk -F " " '{print $1 "=\"" $2 "\""}' > __tmp__.sh
chmod u+x __tmp__.sh
source ./__tmp__.sh # so that variables are maintained in this shell
rm ./__tmp__.sh

if [ "$php72" == "started" ]; then
        echo "PHP service is up"
fi

if [ "$openresty" == "started" ]; then
        echo "Openresty service is up"
fi

Playing with TLSH – Trend Micro Locality Sensitive Hash

There are cryptographic hashes, whose output are very sensitive to any change in the input given, and then then there are hashes which try to convey similarity between given inputs. One such hash is TLSH (github page here)

Here I make a quick demonstration on how simple it is to make TLSH generate a false positive, i.e give a far better similarity score to a text that has no resemblance to the original, than to a text that is almost identical to the original.

Three texts are compared – the original text t1, a second text t2, whose only difference is changing one character from a lowercase to an uppercase B, and a third text t3 which has no resemblance to either t1 or t2.

The text t3 was generated in very quickly using a genetic algorithm…

import tlsh 

# two texts with the only difference being that the second text (t2) uses a capital B for the first occurance of the word "building"
t1 = "Because highly fit schemata of low defining length and low order play such an important role in the action of genetic algorithms, we have already given them a special name: building blocks. Just as a child creates magnificent fortresses through the arrangement of simple blocks of wood, so does a genetic algorithm seek near optimal performance through the juxtaposition of short, low-order, high-performance schemata, or building blocks"
t2 = "Because highly fit schemata of low defining length and low order play such an important role in the action of genetic algorithms, we have already given them a special name: Building blocks. Just as a child creates magnificent fortresses through the arrangement of simple blocks of wood, so does a genetic algorithm seek near optimal performance through the juxtaposition of short, low-order, high-performance schemata, or building blocks"

# A third text, designed to generate a good Trend Micro Locality Sensitive Hash (TLSH) to the first string
t3 = "DYoFBaxhRRRzivXGOrBSglJZ,R.mEkbsf pbANUkMjnIeCwHaqdLWtyu, gqGlmiMpRwqzQmjayETZtxOfYP!Wnl..fsKHVQIzykSvJsUcbfXrLuFJmX.iBEAvDNZPcaqRVFZf ,PYQsMGkUnHKnlSgyXNtbqW.iLiMowTk!MJCNcp,AjqU!jhbGLbOenLnYElxuGKaSgmXdKMsZWtoyHRVTHPzKci.Q!H,aflYkKIFKyZgMENOsnzjKcdLtvVGDBiqCCxnPweTuPmbWShUodpZrJwWVWKGzEpgktsCproAbXufQe,LcxwPivaeFZOOZYHKnkVOPhIqDbxd.sbyzdpQQzWERIQJsvSogXutiINfJM.whcBYVCKEbx npXAFjmqQKkWe!MyzjTFBO,qTZc.ZjnAIptaxMYmSwGOx eXLuBhfoN!thQ"

h1 = tlsh.hash(t1.encode())
h2 = tlsh.hash(t2.encode())
h3 = tlsh.hash(t3.encode())

print("text 1\n--------\n%s\ntext 2\n--------\n%s\ntext 3\n--------\n%s\n" % (t1, t2, t3))
print("h1=%s\nh2=%s\nh3=%s\n" % (h1, h2, h3))

print("hash diff h1,h2 = %d\nhash diff h1,h3 = %d" % (tlsh.diff(h1, h2), tlsh.diff(h1, h3)))

the output of the above program is:

text 1
--------
Because highly fit schemata of low defining length and low order play such an important role in the action of genetic algorithms, we have already given them a special name: building blocks. Just as a child creates magnificent fortresses through the arrangement of simple blocks of wood, so does a genetic algorithm seek near optimal performance through the juxtaposition of short, low-order, high-performance schemata, or building blocks
text 2
--------
Because highly fit schemata of low defining length and low order play such an important role in the action of genetic algorithms, we have already given them a special name: Building blocks. Just as a child creates magnificent fortresses through the arrangement of simple blocks of wood, so does a genetic algorithm seek near optimal performance through the juxtaposition of short, low-order, high-performance schemata, or building blocks
text 3
--------
DYoFBaxhRRRzivXGOrBSglJZ,R.mEkbsf pbANUkMjnIeCwHaqdLWtyu, gqGlmiMpRwqzQmjayETZtxOfYP!Wnl..fsKHVQIzykSvJsUcbfXrLuFJmX.iBEAvDNZPcaqRVFZf ,PYQsMGkUnHKnlSgyXNtbqW.iLiMowTk!MJCNcp,AjqU!jhbGLbOenLnYElxuGKaSgmXdKMsZWtoyHRVTHPzKci.Q!H,aflYkKIFKyZgMENOsnzjKcdLtvVGDBiqCCxnPweTuPmbWShUodpZrJwWVWKGzEpgktsCproAbXufQe,LcxwPivaeFZOOZYHKnkVOPhIqDbxd.sbyzdpQQzWERIQJsvSogXutiINfJM.whcBYVCKEbx npXAFjmqQKkWe!MyzjTFBO,qTZc.ZjnAIptaxMYmSwGOx eXLuBhfoN!thQ

h1=50E0DC15DE2103E00AD2082113E8284263228498022001C1C0F8883021DFDB905BEBDE
h2=32E0D515EE2203E00AD2082113E8288263228898022002C2C0F8883022EEDB906BEBEE
h3=86E0DC15DE2203E01BD2402212E8284263228898032001C4C0F8882021DFDBA15BEBEE

hash diff h1,h2 = 84
hash diff h1,h3 = 17

PHP – switch statement vs. function dispatching

When you start having too many cases in your switch statement, it might be worth looking into using a function lookup. This is possible in all languages where functions are first class citizens…

<?php
$post = "hello";
$info = "3.1415";
//-----------------------------------------------------------
// Version 1 - using a switch statement
//-----------------------------------------------------------

switch ($post) {
    case "hey": {
        print("I've been called with $info\n");
        break;
    }

    case "there": {
        print("I've been called with $info\n");
        break;
    }

    case "hello": {
        print("I've been called with $info\n");
        break;
    }

    default: {
        print("shouldn't have gotten here\n");
    }
}

//-----------------------------------------------------------
// version 2 - same thing using a dictionary of functions
//-----------------------------------------------------------

$funcArray = Array(
    "hey" => function ($param) {
        print("I've been called with $param\n");
    },
    "there" => function ($param) {
        print("I've been called with $param\n");
    },
    "hello" => function ($param) {
        print("I've been called with $param\n");
    }
);

// The data driven invocation of the correct function:

if (isset($funcArray[$post])) {
    // Invoke the relevant function if the string is found
    $funcArray[$post]($info);
}
else {
    print("shouldn't have gotten here\n");
}
?>