Various Tips – unsorted

Open MySQL running on EC2 to remote connections

First, let it be on the record that I don’t think this is a good idea security-wise, but for those that want/have to ignore this advice, here are the steps:

  • Open TCP port 3306 using your AWS EC2 console
  • edit either /etc/mysql/my.cnf or /etc/mysql/mysql.conf.d and search for the string bind-address
  • if bind-address is 127.0.0.1 change it to 0.0.0.0 (this will enable the mysql process to accept remote connections)
  • restart the mysql service (via sudo /etc/init.d/mysql restart or whatever method you prefer)
  • login to mysql with administrator privileges (e.g as root) and create a new mysql user that can access your database remotely – the following example assumes the username is remoteuser and the database you want to user to have access to is mydatabase:

mysql> CREATE USER 'remoteuser'@'%' IDENTIFIED BY 'your_password';
mysql> GRANT ALL PRIVILEGES ON mydatabase.* TO 'remoteuser'@'%' WITH GRANT OPTION;

finally, to check that it works, from your local command line, assuming your database runs at your.dbserver.com:

mysql -u remoteuser -h your.dbserver.com -pyour_password

Some ffmpeg tricks

Convert VOB file to mp4 using ffmpeg:

ffmpeg -i outss.VOB -acodec libmp3lame -ac 2 -ar 22050 video.mp4

___

To extract only the video track to a new VOB file (VOB is the container, the codec is mpeg2):

First, find the channel where the video stream is located using:

ffmpeg -i video1.VOB

The list of existing streams in the file will be returned. Note what follows the ‘Stream #’ in the video stream. If for example the video stream you’re interested in is 0:1 then:

ffmpeg -i video1.VOB -map 0:1 -vcodec copy silent_video.VOB

___

To extract a stream between two points in time:

ffmpeg -i bigfile.VOB -ss 00:28:00 -to 00:35:30 -c copy section.VOB

the above will extract the section starting at minute 28 in the video until minute 35:30 into the video
___

To change properties, for example increase brightness by a factor of two – first try the values using ffplay, e.g

ffplay -vf lutyuv=y=val*2 myvideo.VOB

Then create a modified video with these parameters:

ffmpeg -i myvideo.VOB -vf lutyuv=y=val*2 brighter.VOB

More examples here

Good cross platform PDF utils

pdf to text, pdf to images, pdf to html, pdf info – great command line utilities and open source:
http://www.xpdfreader.com/download.html

pdf2ppm is also good but hard to find a version that works on Windows

Iterating through maps (tables/hash tables/dictionaries/associative arrays)

If you also find yourself making use of more programming languages and software technologies than you care to count, then you probably also forget from time to time the syntax for doing things. A quick reminder:

Lua

Lua doesn’t distinguish (at least not externally) between dictionaries and arrays

local hash={ 1, 'two', three=3, ['& four']='quatro' }
for i=1,4 do print( hash[i] ) end
print('n')
for k,v in pairs(hash) do print(k,v) end
1
two
nil
nil

1       1
2       two
three   3
& four  quatro

Python

hash={ 1:1, 2:'two', 'three':3, '& four':'quatro' }

for i in range(0,4):
     if i in hash:
         print(hash[i])
     else:
         print("no key")
         
print("")
      
for i in hash:
    print("%st%s" % (i,hash[i]))
no key
1
two
no key

1	1
2	two
& four	quatro
three	3

JavaScript

var hash={ 1:1, 2:'two', 'three':3, '& four':'quatro' }
for (var i=0; i<4; i++) 
    if (i in hash) console.log(hash[i]);
else
    console.log("no key");

console.log('')

for (i in hash)
    console.log( i+"t"+hash[i] )
no key
1
two
key

1	1
2	two
three	3
& four	quatro

PHP

<?php
$hash = array( 1=>1, 2=>'two', 'three'=>3, 'four'=>'quatro' );
for ($i=0; $i<4; $i++) 
    if (isset($hash[$i]))
         print("$hash[$i]n");
    else
         print("no keyn");

print("n");

foreach ( $hash as $key => $value )
    printf( "$keyt$valuen" );
?>
no key
1
two
no key

1       1
2       two
three   3
four    quatro

Perl

my %hash = ( 1=>1, 2=>'two', 'three'=>3, 'four key'=>'quatro' );
for ($i=0; $i<4; $i++) {
        if (exists $hash{$i}) {
                print("$hash{$i}n");
        }
        else {
                print("no keyn");
        }
}

print("n");

while ( ($key, $value) = each %hash ) {
        print("$keyt$hash{$key}n");
}
no key
1
two
no key

2       two
1       1
three   3
four    quatro

Ubuntu – check which processes are listening on which addresses/ports

sudo netstat -plnt

Free Mac utility for hardware monitoring

istats – CPU temperature, fan speed, battery health and more

Add user to MySQL database

CREATE USER 'newuser'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON * . * TO 'newuser'@'localhost';
FLUSH PRIVILEGES;

change root password of MySQL

mysqladmin -u root -p"<old_pass>" password <new_pass>

import csv to table

mysqlimport -v --local --fields-enclosed-by='"' --fields-escaped-by='\' --fields-terminated-by=',' --lines-terminated-by='\n' -u<username> -p<password> -h localhost <dbname> '<filename_identical_to_tablename>.csv'

Find hard disk space hogs

Need to see what’s using up all that space on your hard drive?

cd /
sudo du -hd1

and drill down from there…

Quickly searching through bash history

If you know your way around the bash command line, you know that <Ctrl>r searches the last command in your history that started with the letters that you proceed type afterwards. While that is nice, it’s not always that practical since many times you want to find the command that starts the same way but was previous to the one that <Ctrl>r found.

one way is to combine history with grep, but a better way is to add the following lines to your .bashrc (or .bash_profile on Mac):

if [[ $- =~ i ]]; then
    # only do this if we are in an interactive shell
    bind '"e[A":history-search-backward'
    bind '"e[B":history-search-forward'
fi

Next time you open a bash window and want to search for a previous command, simply type the first letters and hit the up arrow. Each additional press of the up arrow will bring a matching command further up the history.

Viewing root directories in Finder window on OS/X

For example, if you want that the /usr directory will be visible:

sudo chflags nohidden /
sudo chflags nohidden /usr

If you want to hide the /usr directory again:

sudo chflags hidden /usr

Codesigning gdb on OS/X

Taken from the GNU docs

The Darwin Kernel requires the debugger to have special permissions
before it is allowed to control other processes. These permissions
are granted by codesigning the GDB executable. Without these
permissions, the debugger will report error messages such as:

Starting program: /x/y/foo
Unable to find Mach task port for process-id 28885: (os/kern) failure (0x5).
(please check gdb is codesigned - see taskgated(8))

Codesigning requires a certificate. The following procedure explains
how to create one:

  • Start the Keychain Access application (in
    /Applications/Utilities/Keychain Access.app)
  • Select the Keychain Access -> Certificate Assistant ->
    Create a Certificate… menu
  • Then:

    • Choose a name for the new certificate (this procedure will use
      “gdb-cert” as an example)
    • Set “Identity Type” to “Self Signed Root”
    • Set “Certificate Type” to “Code Signing”
    • Activate the “Let me override defaults” option

  • Click several times on “Continue” until the “Specify a Location

    For The Certificate” screen appears, then set “Keychain” to “System”

  • Click on “Continue” until the certificate is created
  • Finally, in the view, double-click on the new certificate,

    and set “When using this certificate” to “Always Trust”

  • Exit the Keychain Access application and restart the computer

    (this is unfortunately required)

Once a certificate has been created, the debugger can be codesigned
as follow. In a Terminal, run the following command…

codesign -f -s  "gdb-cert"  <gnat_install_prefix>/bin/gdb

… where “gdb-cert” should be replaced by the actual certificate
name chosen above, and should be replaced by
the location where you installed GNAT.


Quick HTTP Server for testing stuff

python -m SimpleHTTPServer

set default editor for commands like crontab -e

sudo select-editor

Mac – refresh /etc/hosts configuration

OS X Mountain Lion or Lion

Use the following Terminal command to reset the DNS cache:

sudo killall -HUP mDNSResponder

Mac OS X v10.6

Use the following Terminal command to reset the DNS cache:

sudo dscacheutil -flushcache

Also reset browser cache if necessary.


Unix – Recursively search files

beginning with “myfile” from current path

find . -name "myfile*"

vim tricks to remember

When editing CSV files

  • If the columns are nice and straight throughout the file, you can make use of <ctrl>-v which puts you in column visual mode and lets you mark the columns that you wish to copy/paste/delete etc.

  • If the CSV is less well behaved, you can use regex with capture groups to search and replace, for example: :%s/^.+,.+,.+,(.+),.+/1/g will remove the first 3 columns and last column from a CSV file


Extracting bz2 files

taken from here:

First, make sure you have bzip2 installed

apt-get install bzip2

and to do the magic:

tar jxvf phantomjs-1.9.2-linux-i686.tar.bz2

Saving video streams to a file

To save video that is streamed to a Web page as a file, you should first locate the uri that the plugin that renders the stream uses to fetch the media. For example, if the web page uses CastUP technology to stream video then you need to look for the point where the plugin requests the mms://<some-identifier>.wmv

To do this, open the a terminal console window and enter:

tcpdump -A 'tcp port 80' | grep mms

Then, open the web browser at the page where the video is rendered. You should be able to see in the terminal console the references to the mms:// url.

Copy the url and feed it either to VLC or mencoder as specified here.
and just in case the link above will be lost:

using mencoder:

mencoder mms://ip-address/foo.wmv -ovc copy -oac copy -o foo.avi
mencoder mms://1.2.3.4/bar.wmv -ovc copy -oac copy -o bar.avi

using VLC:

vlc mms://1.2.3.4/foo/bar --sout=file/ps:output.mpg

sed, awk, tr etc

replace whitespaces between words with a newline:

tr ' ' 'n' < infile > outfile

Converting between text file formats

Suppose you have a file encoded in one text format, say UTF16-LE and you wish to convert it to another format, for example UTF-8

iconv is an amazing piece of work that can convert between many different text encoding formats and is also available for all major platforms.

In general using iconv goes like this

iconv -f from_encoding -t to_encoding input_file.txt > converted_file.txt

where from_encoding/to_encoding are the existing encoding of the file and the encoding you want to convert to respectively.
The list of encodings iconv supports can be viewed bu issuing iconv -l

If you don’t know the current encoding of the input_file.txt in the above example, on unix based systems issue a

file -i input_file.txt

This will show the existing encoding of the file you wish to convert so that you can enter the -f parameter for the conversion

In cases where file -i claims that what you have is a regular text file but gives you no additional information as to the file’s encoding, you should use enca , which is a great tool for discovering the existing text encoding of a file. In most cases all you’ll need to do is (assuming infile.txt is the file whose encoding you want to know):

enca infile.txt

Or, if the above doesn’t work, for multibyte encodings try:

enca -L none infile.txt

Check man enca for more info.


Batch Image Processing

I’ve done this before but already forgot about it. You have a directory full of images and you want to process them all in one shot – either resize, convert format, flip, join, crop, blur etc. What you do is use mogrify. It works like this:

  • Make sure you have imagemagick installed. If not then sudo apt-get install imagemagick
  • Then just run mogrify (one member of the ImageMagick suite of tools) on the set of images you want to transform. Note – the images will be overwritten with the new files – make sure you have copies of the images if you use mogrify else use the convert tool.

An example on running mogrify to resize all the jpg files in a directory to a width of 200 pixels (preserving the aspect ratio):

mogrify -resize 200 *.jpg

Create Desktop Launcher/Shortcut

This was removed from the UI in Ubuntu 12.04, but you can still do it as follows:
If you don’t have it already installed, then:

sudo apt-get install gnome-panel

Next, simply

gnome-desktop-item-edit ~/Desktop/ --create-new

(thanks)

Burn a directory to multiple CDs/DVDs

its called dirsplit and is used thus:
Suppose you have a directory with gadzillion files (mp3s, images, whatever), and you want to burn them all on CD/DVD media. It will take a few volumes but you don’t want to start dividing the files into folders of CD/DVD size. For the example say these are images that are stored in /home/myplace/CamUploads/

What you do is:

cd /home/myplace/CamUploads/
dirsplit -s 700M /home/myplace/CamUploads/

The above will create of files with extension .list (vol_1.list, vol_2.list, etc. depending on the total size). These files can be processed to create corresponding iso images for burning on the CD/DVD by issuing:

mkisofs -o vol1.iso -D -r --joliet-long -graft-points -path-list vol_1.list
mkisofs -o vol2.iso -D -r --joliet-long -graft-points -path-list vol_2.list

Use your favorite CD burner (read: K3b) to burn the iso images to your CD/DVD media.


Batch deletion of files

When you have files that have a common pattern dispersed throughout a complex directory structure and you wish to delete them, you can use the Linux powerful find command. Among its many parameters are the -exec switch which enables executing a command for every file found.

Suppose you want to delete all the files within a directory structure whose root is at /home/me and that end with _dev.db3. To achieve this simply do the following:

cd /home/me
find . -type f -name "*_dev.db3" -exec rm -f "{}" ;

{} represents the files that are found for the exec switch. It is quoted to avoid surprises when running rm on it. The command after the exec has to be terminated by a ; which itself is escaped by a backslash to avoid additional unintentional shell interpretations.

man find for more information

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.