Where is Mozilla Firefox Cache

This one is not about programming. Mozilla Firefox is popular web browser and it comes as default browser on many Linux distros. OK it may be called Iceweasel instead of Firefox, but that is about the same. Cache used to be in $HOME/.mozilla/firefox/(profile)/Cache but now it is not there. During the time one get used to delete or copy files from cache to some safe location, for example:

find (path to Cache) -type f -size +100k -exec cp --parents {} (where to copy) \;

But now cache is gone and nothing works. Initial reaction is google for solution, we are turning into web search addicts, and when that doesn’t bring us joy we are searching some more.
So, how we find cache? For the beginning we start Firefox and terminal. In terminal we execute:

ps aux | grep firefox

That will tell us what is process ID for Firefox. Should look something like this:

(user name)   16025 19.6  5.9 1065852 226284 ?      Sl   16:10   0:13 /usr/lib/firefox/firefox

That 16025 is process ID or pid. Knowing pid we can easily find out what files that proces is keeping open:

ls -l /proc/16025/fd | grep cache

We execute that in terminal, no need to be root. On Mint it is in $HOME/.cache/mozilla/firefox/(profile).
That was so simple, how is possible that I couldn’t find it on Google ;-)

Using POSIX MQ from Java - another part

In introduction we managed to create our message queue and to close it. Only thing which is unclear is where is it? Message queue is maintained by kernel and it is created in virtual file system. If we execute in terminal:

cat /proc/filesystems

we will see entry:

nodev    mqueue

If we want to take a closer look at our JNIMQ_1 we will have to mount mqueue. It is nicely described in man pages, just look for mq_overview entry. For lazy people here are commands for mounting message queue, you must do that as root:

# mkdir /dev/mqueue
# mount -t mqueue none /dev/mqueue

After this we should exit root level. Again using man pages we find out that second line is standard form for mount and looks like this mount -t type device dir. Once mounted we can use our standard set of tools for examining file content on Linux, for example:

ls -l /dev/mqueue
cat /dev/mqueue/JNIMQ_1

Now we can go on sending and receiving messages. If we do not specify time-out, those messages and queues will be persisted until system reboots.
Since we have functional message queue, we can nicely start chat. For that reason we will send some messages. We already have queue from last time and we will not create it:

public class SendMessage{ public native int send(String name, int flags, String text); public static void main(String[] args) { int status = new SendMessage().send("/JNIMQ_1",1,"Hello World!"); if(status!=-1) System.out.println("Message dispatched."); else System.out.println("Sending message failed."); } static { System.loadLibrary("SendMessage"); } }

That flags = 1 is open in write only mode. Boring part with javah we are skipping and here is implementation:

#include "SendMessage.h" #include <stdlib.h> #include <stdio.h> #include <string.h> #include <unistd.h> #include <sys/stat.h> #include <sys/types.h> #include <mqueue.h> JNIEXPORT jint JNICALL Java_SendMessage_send (JNIEnv *env, jobject obj, jstring name, jint flags, jstring text){ mqd_t mq; const char *cstr = (*env)->GetStringUTFChars(env, name, 0); mq = mq_open(cstr, flags); if((mqd_t)-1 == mq){ (*env)->ReleaseStringUTFChars(env, name, cstr); perror("mq_open"); return -1; } const char *ctxt = (*env)->GetStringUTFChars(env, text, 0); int res = mq_send(mq, ctxt, strlen(ctxt)+1, 0); if(res==-1){ perror("mq_send"); return -1; } if((mqd_t)-1 == mq_close(mq)){ perror("mq_close"); return -1; } (*env)->ReleaseStringUTFChars(env, name, cstr); (*env)->ReleaseStringUTFChars(env, text, ctxt); return 0; }

We compile it as described in previous blog entry and send some messages. If we cat /dev/mqueue/JNIMQ_1 we will see that it is growing.
Receiving is equally simple, here is the Java code:

public class GetMessage{ public native String getMessage(String name, int flags, int priority); public static void main(String[] args) { System.out.println(new GetMessage().getMessage("/JNIMQ_1",0,0)); } static { System.loadLibrary("GetMessage"); } }

We want queue opened as read only and we want messages with priority 0. After applying javac and javah we write implementation:

#include "GetMessage.h" #include <stdlib.h> #include <stdio.h> #include <string.h> #include <unistd.h> #include <sys/stat.h> #include <sys/types.h> #include <mqueue.h> JNIEXPORT jstring JNICALL Java_GetMessage_getMessage (JNIEnv *env, jobject obj, jstring name, jint flags, jint priority){ unsigned int up = priority; char ret[1025]; const char *cstr = (*env)->GetStringUTFChars(env, name, 0); mqd_t mq; mq = mq_open(cstr, flags); if((mqd_t)-1 == mq){ (*env)->ReleaseStringUTFChars(env, name, cstr); perror("mq_open"); return (*env)->NewStringUTF(env,"dud"); } ssize_t bytes_read = mq_receive(mq, ret, 1024, &up); if(bytes_read==-1){ perror("mq_receive"); return (*env)->NewStringUTF(env,"dud"); } ret[bytes_read] = '\0'; if((mqd_t)-1 == mq_close(mq)){ perror("mq_close"); return (*env)->NewStringUTF(env,"dud"); } (*env)->ReleaseStringUTFChars(env, name, cstr); return (*env)->NewStringUTF(env,ret); }

This 1024 should be retrieved from mq attributes, but I am lazy to do that and I still remember how it was created.
In this two examples I deviated from passing mq descriptor to Java, that is what jtux does. Not sure what is more expensive, crossing managed-native border or opening file, so do not ask.

Using POSIX MQ from Java - Introduction

All this is taking place on Linux Mint but should work on any other Linux. JDK is Oracle JDK, I am doing Android programming and it requires Oracle JDK, OpenJDK should do fine but path will differ. Goal of this introduction is to create new message queue and to close it - nothing else.
As usually in JNI workflow we write Java code:

public class PosixMQ { public native long mq_open(String name, int flags); public native int mq_close(long desc); public static void main(String[] args) { PosixMQ pmq = new PosixMQ(); long desc = pmq.mq_open("/JNIMQ_1",64); if(desc!=-1L){ System.out.println("We have MQ open."); System.out.println("Close returned " + pmq.mq_close(desc)); }else{ System.out.println("Open failed."); } } static { System.loadLibrary("PosixMQ"); } }

Idea about storing pointer to data structure into long is from jtux project, otherwise it should be message queue descriptor or (mqd_t)-1 in case of failure. Queue name must start with slash and 64 is O_CREAT | O_RDONLY we want queue to be created and to be read only for us. Now we compile PosixMQ.java and we run javah on class to get header for C implementation. On my box it looks like this:

/usr/lib/jvm/java-7-oracle/bin/javah -jni PosixMQ

Interesting part of PosixMQ.h looks like this:

/* * Class: PosixMQ * Method: mq_open * Signature: (Ljava/lang/String;I)J */ JNIEXPORT jlong JNICALL Java_PosixMQ_mq_1open (JNIEnv *, jobject, jstring, jint); /* * Class: PosixMQ * Method: mq_close * Signature: (J)I */ JNIEXPORT jint JNICALL Java_PosixMQ_mq_1close (JNIEnv *, jobject, jlong);

Finally we write trivial C implementation, being careful to include all required imports:

#include "PosixMQ.h" #include <stdlib.h> #include <stdio.h> #include <string.h> #include <unistd.h> #include <sys/stat.h> #include <sys/types.h> #include <mqueue.h> JNIEXPORT jlong JNICALL Java_PosixMQ_mq_1open (JNIEnv *env, jobject obj, jstring name, jint flags){ struct mq_attr attr; attr.mq_flags = 0; attr.mq_maxmsg = 10; attr.mq_msgsize = 1024; attr.mq_curmsgs = 0; const char *str = (*env)->GetStringUTFChars(env, name, 0); mqd_t desc; desc = mq_open(str, flags, 0644, &attr); (*env)->ReleaseStringUTFChars(env, name, str); return (intptr_t)desc; } JNIEXPORT jint JNICALL Java_PosixMQ_mq_1close (JNIEnv *env, jobject obj, jlong desc){ return mq_close((mqd_t)(intptr_t)desc); }

I was too lazy to pass attributes around, it should be done at later stage. Permissions are also conveniently hardcoded, though passing octal through parameter list is not difficult. To compile this I used:

gcc -I/usr/lib/jvm/java-7-oracle/include -I/usr/lib/jvm/java-7-oracle/include/linux -o libPosixMQ.so -shared -fPIC PosixMQ.c -lrt

Switch -fPIC will not be required on 32 bit Linux and -lrt instruction to link against the real-time is required everywhere. To run it we do:

java -Djava.library.path=. PosixMQ

in directory where are binaries and we see:

We have MQ open.
Close returned 0

Not very impressive but again very simple. Maybe next time we send and receive some messages, who knows.

Java writes to named pipe

I did while ago article where Java talks to C++ using standard input/output, now this is just addition to it. As in last example all that happens on Linux and we are using named pipe or FIFO. Need to transfer result of some periodic operation to remote listener and do not want to block process with network transfer. It slows it down and also introduces new opportunities for failure. In original problem that process is written in some funny functional language where many things may go wrong. Naturally I will not bother you with networking or what is being processed. It will be just small IPC example.
So here is C++ listener:

#include <iostream> #include <sys/stat.h> #include <stdio.h> #include <fcntl.h> using namespace std; int main() { char fifo_name[] = "MYFIFO"; char buf[100]; int fifo_handle; int size; mknod(fifo_name, S_IFIFO | 0666, 0); cout << "client is ready" << endl; if ((fifo_handle = open(fifo_name, O_RDONLY)) < 0) { perror("open failed"); return -1; } do{ size = read(fifo_handle, buf, 100); if(size<0) perror("read failed"); else if(size==0) cout<<"server left us"; else{ buf[size] = '\0'; cout<<"message is "<<buf<<endl; } }while(size>0); close(fifo_handle); return 0; }

We run it the first, to create MYFIFO. BTW on Ubuntu 12.10 it will fail to compile, to remedy that add unistd.h to includes. Then comes Java class which is talking to C++ listener:

import java.io.*; public class Orator { public static void main(String[] args) throws IOException { //Runtime.getRuntime().exec("mkfifo MYFIFO"); BufferedReader in = new BufferedReader(new InputStreamReader(System.in)); BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream("MYFIFO")); String s; while ((s = in.readLine()) != null && s.length() != 0){ byte[] arr = s.getBytes(); out.write(arr,0,arr.length); out.flush(); } } }

We run listener in terminal, then Orator class in different terminal and type short messages. When we hit enter message goes through pipe and appears in terminal. If you didn’t know how to turn string into byte array and write it to buffer now you know. Only tricky point is that you need to flush output stream to go through pipe, it’s all about plumbing.

Easy light polluted night sky workflow

I got four 120 seconds ISO 800 RAW frames. Better option would be to go for 60 seconds ISO 1600 and minimize tracking errors, stacking should eliminate noise anyway. Since pictures are taken under suburban sky, they are overexposed and have ugly orange background color. When we open them in Darktable default white balance preset is camera white balance and that looks like this.

Now we want to eliminate that ugly orange background and we switch from camera white balance to spot white balance.

If images were saved as JPEG, camera white balance would be applied, colors shifted to allow better compression and we won’t be able to do much processing. Before exporting images to TIFF, to export hit Ctrl+e, we will tweak exposure as on picture:

I export them as 16 bit integer per channel TIFF, if you do not know how to manage export settings it is explained in previous blog entries.
Now to do stacking I will open terminal and execute magic formula:

align_image_stack -a tif *.tiff

gmic tif0000.tif tif0001.tif -div 256 -gimp_blend 3,1,0 -mul 256 -c 0,65536 -type ushort -output one.tiff
gmic tif0002.tif tif0003.tif -div 256 -gimp_blend 3,1,0 -mul 256 -c 0,65536 -type ushort -output two.tiff
gmic one.tiff two.tiff -div 256 -gimp_blend 3,1,0 -mul 256 -c 0,65536 -type ushort -output tutorial.tiff

If you compare it with previous blog entries about G’MIC stacking you will see that new version of G’MIC is not completely backward compatible. Some people would maybe like to use DSS instead and that is also OK. Finally we open image in GIMP bump up contrast and LAB color decompose image. We duplicate A and B component, set copy mode to overlay and then merge them down. Do not flatten layers, there should be L, A and B layers. L layer can be slightly stretched or left how it is. When we LAB compose layers back into RGB image we will have nice saturated colors. Now, some people will proceed playing with curves but I will just add background gradient. I am not really using it as intended. I switch from divide to overlay and reduce opacity to 50%. That background gradient is part of astronomy plugin for GIMP by Georg Hennig and 2.8 compatible version is available from here git://gitorious.org/gimp-plugins-ambulance/gimp-plugin-astronomy.git. Building plugin is trivial. Here is result:

 
 Complete damage control was done in Darktable in three straightforward steps and that is why I call it easy. If level of black was lower and and for example exposure was reduced only -0.5EV, we could further increase contrast and get more of that Flame nebula. Though it will be more fiddling in GIMP and may be not so simple as it sounds.

Homework: Towers of Hanoi

This puzzle was popularised in Europe by French mathematician Edouard Lucas. There are three poles, 64 disks of different sizes and number of monks trying to move those disks from first to third pole, respecting the following rules:
1) You can move only one disk at the time.
2) You can pick only top disk and place it on top of other disks on different pole.
3) You can place disk only on bigger disk.
When monks finish moving those 64 disks and puzzle is solved, end of the world will come. According to Lucas all that takes place in Vietnam. Now, this puzzle is known in India as Towers of Brahma and there could be other versions of it, for example Chinese, over last seven thousands years they invented many puzzles. One may be under impression that end of the world is near but to solve 64 disk puzzle, 2^64-1 moves are required and that is quite big number 18446744073709551615. Though end of the world may come before puzzle is solved for many other reasons.
Puzzle is quite interesting because it allows to develop simple algorithm. Now we will name poles start, aux and end. With single disk, we take it from start and place on end. With two disks, we take small to aux, big to end and finally smal from aux to end. Now with three disks we already see that we should move top two to aux, big disk to end and after that top two to end. So, moving top two to aux is like previous case, but aux and end are swapping places. Moving top two from aux to end is again the same as previous case, but start and aux are swapping places. We can even write moves, so it is easy to see what is going on:

One disk: start to end
Two disks: start to aux, start to end, aux to end
Three disks: (start to end, start to aux, end to aux), start to end, (aux to start, aux to end, start to end)

We can generalize and say that solution is moving n-1 disks to aux, moving bottom disk to end and moving n-1 disks from aux to end. So we got self algorithm for solving puzzle. Turning it into code is more than simple:

def towers(n, start, aux, end): if n>1 : towers(n-1, start, end, aux) print 'moving disk {} from {} to {}'.format( n, start, end ) towers(n-1, aux, start, end) elif n==1 : print 'moving disk {} from {} to {}'.format( n, start, end )

That was Python implementation. If you do not know Python, here is Java implementation:

public class Towers { public static void main(String[] args) { towers(5,1,2,3); } static void towers(int n, int start, int aux, int end){ if(n>1){ towers(n-1, start, end, aux); System.out.println("moving disk "+n+" from "+start+" to "+end); towers(n-1, aux, start, end); }else System.out.println("moving disk "+n+" from "+start+" to "+end); } }

Very simple. There are many other ways to solve this puzzle but recursion is simplest and most logical. Also excellent interview question, if candidate is not capable of developing simple algorithm and understanding recursion, then candidate is not really programmer.

Recursion in Python

And now something completely different. During last week I was busy polishing my Python skills or rather removing rust from them. I went through book Think Python: How to Think Like a Computer Scientist by Allen Downey. Mentioned book is available for download from http://www.thinkpython.com. My impression is that book is intended for army of different bookkeepers and alike trying to learn programming. So author is threading rather gently and slowly. It reminds me of joke from Monty Python where accountant, Palin, wants to go directly into taming lions, but career consultant, Cleese won’t let him, he must do that gradually, first investment banking. Now I will be kind and let accountant do lion taming without going first into investment banking.
In one of introductory chapters recursion is introduced and stack transition is explained. For example we can use factorial:

def fac(n): if n <= 1 : return 1 else: return n*fac(n-1)

It is easily readable and looks good. If we call it with n=6 this will happen:

6*(5*(4*(3*(2*1))))

CPU pushes on stack value for n and * operation and dispatches call to the same function but with n-1 parameter, when right brace is closed, pops frame from stack and finishes multiplication. Problem with recursion and Python is that depth of stack is rather small, only thousand frames. If you try to calculate factorial for 2^10 using this function it will cause stack overflow. So instead of doing recursive calls one can rewrite factorial and use iterations:

def fac(n): res = 1 while n>1: res *= n n-=1 return res

Still simple and shallow stack problems are avoided, but that is so investment banking. What else could be done? We can use binary splitting. Story about binary splitting comes from Xavier Gourdon and Pascal Sebah and you can find it here. Instead of multiplying accumulator with all numbers in range we do recursive preparation similar to binary search and build tree like flow to multiply partial products between themselves. We define product like this:

P(a, b) = P(a, (a+b)/2)*P((a+b)/2, b)

Then we divide each range in half until distance between upper and lower bound is small. Here is the code:

def product(a, b): d = b - a m = (a+b)/2 result = 0 if d > 1 : result = product(a,m)*product(m,b) elif d == 1 : result = b else : result = 1 return result

If we say a is 0 we will get factorial b as result. I tested it for b up to 2^16 and it went through without stack overflow as expected. While it theoretically could be used for values up to 2^1000, try not to exceed 2^16 unless your box have really good cooling and plenty of time to wait for result. Tree is not tall, but there is quite few branches. Python is partly functional language and that bring us to yet another idea. Functional languages have compilers which will optimize recursive calls into so called tail calls if function is written in certain way. For example this could be optimized:

def fac(n, a=1): if n<2 : return a else: return fac(n-1, n*a)

But Python compiler is not that much functional and it will not generate tail calls. Through the years people are bothering Guido with tail call optimisations but he just doesn’t want to implement it. When gccpy front-end is released, there will be tail call optimisation since GCC is doing it anyway. Instead of waiting for gccpy we can help ourselves with generators and so called trampolining. We should also switch from factorial to Fibonacci numbers, due to size of result. Plain Fibonacci looks like this:

def fibo(n): if n<2: return n else: return fibo(n-1)+fibo(n-2)

In order to get generators we will replace return with yield and rewrite function so it ends with single function call:

def fibo(a, b, c):
    if c==0:
        yield b
    else:
        yield fibo(a+b, a, c-1)

and one would like to call it like this fibo(1, 0, n). Now we just need trampoline to execute those tail calls and we can calculate first thousand or so Fibonacci numbers:

import types

def trampoline(func, *args, **kwargs):
    f = func(*args, **kwargs)
    while isinstance(f, types.GeneratorType):
        f=f.next()
    return f

def fibo(a, b, c):
    if c==0:
        yield b
    else:
        yield fibo(a+b, a, c-1)

for x in range(1024):
    print "F(", x, ") =", trampoline(fibo, 1, 0, x)

There is user friendly version of trampoline where decorator is used, so no need to replace return with yield and normal function call is used, but is more difficult to understand, you can download it from here.