Excerpt of my .bashrc file

Misspelling aliases are probably the most useful feature!

# aliases for terminal use
alias ls='ls --color'
alias ll='ls -lh'
alias la='ls -lha'


# protection against ninja (and stupid) "rm"
alias rm='mv $1 ~/trash/'

# aliases for misspelling
alias mroe='more'
alias nore='more'
alias mero='more'

alias enacs='emacs'
alias emacxs='emacs'

# aliases for separator

alias cut='cut -d "|"'
alias awk='awk -F "|"'

# alias for navigation
alias mor='cd /data/$USER'

# variables for navigation
UMLS2012AB='/export/home/$USER/data/umls/2012AB/META/'

 # line prefix
PS1='$USER@$HOST:\W $'

# mask to correct rights for file creation
umask 0022

# functions
function emacs() { command emacs -name "$@" "$@" & }
function firefox() { command firefox "$@" & }
extract () {
    if [ -f $1 ] ; then
        case $1 in
            *.tar.bz2)    tar xjf $1        ;;
            *.tar.gz)    tar xzf $1        ;;
            *.bz2)        bunzip2 $1        ;;
            *.rar)        rar x $1        ;;
            *.gz)        gunzip $1        ;;
            *.tar)        tar xf $1        ;;
            *.tbz2)        tar xjf $1        ;;
            *.tgz)        tar xzf $1        ;;
            *.zip)        unzip $1        ;;
            *.Z)        uncompress $1    ;;
            *)            echo "'$1' cannot be extracted via extract()" ;;
        esac
    else
        echo "'$1' is not a valid file"
    fi
}

# manage history
export HISTCONTROL=ignoredups:erasedups  # no duplicate entries
export HISTSIZE=100000                                  # big big history
export HISTFILESIZE=100000                          # big big history
shopt -s histappend                                             # append to history, don't overwrite it

# Save and reload the history after each command finishes
export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"

Error with exec() in java (and wget tips)

Of course, i (almost) never read extensively the documentations. I often regret it...

I came upon a problem on which i lost some precious minutes:
- Executing wget from java exec()
- The code runs well, slows down, stops and finally returns an exception.

After investigation, the problem comes from exec() which keep all the standard and error outputs in memory. If you use wget on a large file, or on a large number of file. It comes a time, where memory becomes a problem.

Here is a quick and dirty solution: not output at all.

‘-q’

‘--quiet’

Turn off Wget's output

‘-nv’

‘--no-verbose’

Turn off verbose without being completely quiet (use ‘-q’ for that), which means that error messages and basic information still get printed. 

See http://www.gnu.org/software/wget/manual/html_node/Logging-and-Input-File-Options.html#Logging-and-Input-File-Options for more information.

        String file = extractFileName(url);
        System.err.println("Download: " + url);
        try {
            Process proc = Runtime.getRuntime().exec("wget -q -nv " + url);
            proc.waitFor();
        } catch (IOException e) {
            System.err.println("wget -q -nv " + url+ " failed");
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

 The proper way to deal with the problem is to capture and flush the ouput and error stream.

That's all folks!

Passing arguments to parallelized threads in java

Java proposes a very useful class to feed threads: the BlockingQueue.

Example of declaration:

private BlockingQueue<String> _blockingQueue = new ArrayBlockingQueue<String>(3000000);

 BlockingQueue is an interface, ArrayBlockingQueue is initialized with a fixed capacity (here 3,000,000 strings).

The BlockingQueue is thread safe.

Each thread can get element with the method take().

There is no indicator that the queue is empty (as new element can be added while the queue is read). A way to send a termination signal is to use a "poisonous element" at the end of the queue. Better, is to send an interrupt signal  to all thread.

That's all folks!

Using xargs or parallel to parallelize processes on linux

The command line:

find . -name '.txt' | xargs -n 1 -P 10 -I % sh -c 'cat % | perl command.pl > %.out'

The arguments:
-n number of arguments passed at each line, might not be useful here
-P number of processors used - enable parallelization of the processes
-I % get the argument passed by the command.

Equivalent command line with parallel:

find . -name '.txt' | parallel 'perl command.pl {} > {.}.out'

The arguments:
{} the argument
{.} the argument without extension

That's all folks!