Building GIMP 2.9.1 from source on Linux Mint 13

On Open Source Photography group on Google+ https://plus.google.com/u/0/communities/110647644928874455108 goes discussion: how nobody is providing deb for GIMP 2.9.1 and you can download installer for Mac or Windows, for example from here http://www.partha.com/. I guess most of Linux users are capable of compiling and building software on its own and that could be reason. There are distros like Gentoo Linux where building packages from source is common and in that way your binaries will be best match for your hardware. On the other side building GIMP is not so easy, for example one of dependencies is Gtk+ and that one is never easy to build. Since I am doing programming for living, I decided to help and write tutorial how to build GIMP 2.9.1 from source. Before I start, GIMP 2.9.1 is unstable development release, it is not intended for final user, it is slow and if you still want to build it do that in VM, place build environment in KVM or VBox. On positive side - yes it can handle 16 bit integer channel.
Starting point for me was Compiling Gimp 2.9.x on PCLinuxOS KDE4 2012.02 http://sparewotw.wordpress.com/2012/03/19/compiling-gimp-2-7-6-development-version-on-pclinuxos-kde4-2012-02/, thanks Andrzej. My desktop is Mate and his KDE but PCLinuxOS is based on Ubuntu like Mint, though packages are called slightly differently. If one is starting from plain new install he can start with:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential

Next libs for processing image and media formats like libtiff, libjpeg, libavcodec-extra-53, libvpx and so on. Naturally you will need corresponding dev packages also, since you want to compile against them. From development you will need Flex, Bison, Lua, Yasm; Python and Ruby you should already have. To get source code you need Git, source is at git://git.gnome.org/gimp and it is not marked as 2.9 but as 2.8, so do not get confused, check configure.ac and look for:

m4_define([gimp_app_version], [2.9])

I will not try to exactly list dependencies, you will see what is missing from error message if ./configure fails. Also going through Andrzej’s recommended list, one can do:

apt-cache search [package name without version number]

and see what is it called on Mint or Ubuntu or Debian.
Now following Andrzej’s tutorial download recommended dependencies, extract them where you want to build them, I do that in ~install. Do not deviate in package version since interfaces are changing and you will not be able to build. Now is time to start build, open terminal and elevate it to root level, cd where you will start build and set $PREFIX, $PATH and $PKG_CONFIG_PATH:

export PREFIX=`echo /usr/`
export PATH=$PREFIX/bin:$PATH
export PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig:$PREFIX/share/pkgconfig

While this is OK on clean box, rather go for /usr/local value for $PREFIX if you not doing it in VM or you do not have spare box for build. I also did

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib

First dependency is libffi-3.0.11 and as in Andrzej’s tutorial you just do, from the same terminal where those environment variables are exported:

./configure --prefix=$PREFIX
make
make install

This one should go without problems. You should see messages like this one:

Libraries have been installed in:
   /usr/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,--rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

Next is glib-2.32.2 and quite few other packages are depending on it. When you finish building it do:

ldconfig
ldconfig -n /usr/lib

The first one clears library cache and the second one renews links in /usr/lib, that is where we are installing build output. To check situation do:

ldconfig -p | grep glib
ls -l /usr/lib | grep glib

The first prints links and the second one will say to what links are pointing, for example:

lrwxrwxrwx  1 root   root           23 Jan 20 22:24 libglib-2.0.so -> libglib-2.0.so.0.3200.2

If there are links to something like libglib-2.0.so.0.3400.3 they must be removed, together with libs. Check situation for libgio-2.0, libglib-2.0, libgmodule-2.0, libgobject-2.0 and libgthread-2.0. You should not have them unless you downloaded and did build of different version of glib. Leave alone lower versions like libglib-1.2.so.0.0.10, which may be there, but higher ones you should remove and links pointing to them. Tricky to configure and build is gtk+-2.24.10, if you run into problems double check are you bilding against right dependencies, if not get right ones and

make clean
ldconfig
./configure --prefix=$PREFIX

Like that all ten downloaded dependencies are built and installed and we can clone from GIMP repository source and finish build:

git clone git://git.gnome.org/babl
git clone git://git.gnome.org/gegl
git clone git://git.gnome.org/gimp
git clone git://git.gnome.org/gimp-gap

I didn’t even try gimp-gap since Andrzej was complaining that he can’t build it and GIMP works without it. For top three in the same order, cd in their root directory and:

./autogen.sh --prefix=$PREFIX
make
make install

GEGL was complaining about my custom FFmpeg and few other libs so I temporarily renamed offending libs and installed missing libx264-120 and libx264-dev. For example this is error message when make fails:

/usr/local/lib/libavformat.a: could not read symbols: Bad value

When GIMP is built you can run it executing:

/usr/bin/gimp-2.9

I run it from terminal because it prints debug messages to terminal. Plugins for GIMP 2.8 will naturally work, so install gimp-plugin-registry. If you see messages like this one

Gtk-WARNING **: Unable to locate theme engine in module_path: "murrine"

you will understand why is good idea to build it in VM. Enjoy.

Orion’s belt and sword, processing with open source tools

Capturing photos

To do processing one needs to take few photos first. Minimal hardware is DSLR with 50mm lens, remote shutter release for DSLR and any kind of cheap telescope with equatorial mount. It is not necessary to buy 50mm lens, kit lens usually 18-55mm will do, if your DSLR have live view and you can set focus. Manually setting focus on kit lens without live view is quite difficult. But it will be nice to have one 50mm F1.8 lens which is much faster and sharper than kit lens. If you have longer focal length lens it is even better, 100mm or even up to 200mm, over it tracking becomes big problem. Why remote shutter release? Camera supports up to 30 seconds exposure and after that BULB, you should press and hold shutter button what will cause lot of shaking and ruin photo. For that reason remote shutter release is used, we can have long exposure without shaking and strain. Telescope is used so that we can piggyback camera on it and achieve long exposure without trailing stars. How to attach camera to scope, there are piggyback brackets, piggyback camera mounts, you can make them on your own using hose clamp. If your telescope got RA motor drive, do polar alignment and switch it on, shorter focal length of the lens is more tolerant towards tracking errors. If there is no RA motor drive you will have to do manual tracking. Telescope with alt-azimuth mount won’t do it must be with equatorial mount. There are specialized devices like Vixen Polarie or AstroTrac which can be used instead, but they are more expensive than small 5" Newtonians with RA motor drive. Why one shouldn’t just do few hundreds shots from tripod and later stack that in Deep Sky Stacker? Because 5 seconds would be longest acceptable exposure for 50mm lens and to get to 5 minutes you need 3600 photos. Way to go is increasing exposure time.
If you are not sure where is Orion or M42 inside Orion install Stellarium http://www.stellarium.org/ it is open source and works on all major operating systems.

Processing

So I went out under my light polluted suburban sky and managed to get few decent photos between clouds. I did the same night before and now I have 13 frames 55mm F5.6 ISO 800 where exposure is from one minute to two minutes. When I decided to crop them and stack them together align_image_stack from Hugin did poor job. Was that too narrow cropping or something else I do not know, for that reason new strategy was stack them by night and later stack final results. So, imported RAWs into darktable, applied chromatic aberration and lens correction and exported them to 16 bit TIF. Should do hot pixels removal but I forgot to do it and did it later in G’MIC, like this:

gmic IMG_0451.tiff -remove_hotpixels 3,10 -c 0,65536 -type ushort -output IMG_0451hp.tiff

Now I aligned stack:

align_image_stack -a tif  *.tiff

and averaged them two by two, saving output:

gmic tif0000.tif tif0001.tif -div 256 -gimp_compose_average 1,0 -mul 256 -c 0,65536 -type ushort -output step1.tiff

Later I combined those steps, two by two, until final result. To stack those two images I need to rotate one and to crop both of them:

gmic m4291.tiff -rotate -66 -crop 1400,2500,3500,4700 -c 0,65536 -type ushort -output m4291c.tiff
gmic m4281.tiff -crop 1450,767,3550,2967 -c 0,65536 -type ushort -output m4281c.tiff

To find out how much to rotate and crop I used GIMP. Now fine alignment of those two with align_image_stack:

align_image_stack -a tif *.tiff

and final blending for16 bit output:

gmic tif0000.tif tif0001.tif -div 256 -gimp_compose_average 1,0 -mul 256 -c 0,65536 -type ushort -output m4289.tiff

and also one 8 bit for GIMP:

gmic tif0000.tif tif0001.tif -div 256 -gimp_compose_average 1,0 -output m4289.jpg

Light pollution and clouds contributed to final result.

It is too bright, too much orange in it, but M42 is visible and Flame nebula is just barely visible. If we look at histogram, this is where we are and where we want to be:

From GIMP menu we select Color->Levels. In Adjust Color Levels we set Channel to Red and move upper slide from the bottom towards middle.

We do the same for green, looking at histogram set to RGB and picture itself. After this image is still too bright and Flame nebula is invisible. To remedy that I will rise contrast, Colors->Brightness-Contrast and I set contrast on 40. Now LAB decompose color boost, described in one of previous tutorials and some more contrast stretching via Colors->Auto->White Balance and after that Edit->Fade, we want histogram stretched but not that much, again checking image and histogram to find out how much.
Here is the final result:

Conclusion I need lens and light pollution filter ;-)

EasyCAP story

On Astrophotography group on Google + https://plus.google.com/u/0/communities/118208937662082340807 one beginner wanted advice what camera he can attach to his Powerseeker 127EQ and to pay it not more than $250. Astrophotography is known as money hole and it is difficult to imagine decent setup below $2000, and that is using entry level DSLR and not specialized CCD camera. Now I remembered that maybe year ago I was looking Deva camera by Astrolocker, GSTAR-EX http://www.myastroshop.com.au/guides/gstar/index.htm and JWT Messier http://www.jtwastronomy.com/products/messier.php.
Today cheapest Deva, it used to be $275, is gone from the market and others are significantly more expensive. So I advised using Samsung Ultra High Resolution SCB-2000 or similar CCTV camera with sense-up and EasyCAP DC 60 for video capture. What is idea, with sense-up activated up to x512, camera will internally stack up to 512 frames and capture will not take more than few seconds. About any equatorial mount with motor drive must be able to pull few seconds of decent tracking without significant errors. Naturally those cameras are not mach for DSLR camera. They are able to produce 752 × 582 on PAL and with sense-up activated noise ratio is not nominal 50dB. So how pictures actually look like? For example South African amateur astronomer Lucas Ferreira did quite few captures with Samsung SCB-4000 through 10" Newtonian Reflector and his pictures are here http://www.lucastronomy.co.za/Gallery/categories.php?cat_id=17. Looking at picture description we see that complete capture takes about 12 seconds with sense-up x512. While quality is very far from Hubble to get any kind of picture in 12 seconds with any normal camera one needs about two meters of aperture. So, I guess you agree with me CCTV camera can pull maximum out of small 5" Newtonian and even utilize his for proper astrophotography wobbly motor drive.
After giving advice I decided to put together CCTV camera and capture device and plug all that into PC. It is not horribly complicated and may be interesting. So, I ordered EasyCAP D60+ on local website and it arrived in couple of days. I plugged it in USB port and executed lsusb and it says ID 1c88:0007 Somagic, Inc. SMI Grabber (EasyCAP DC60+ clone) [SMI-2021CBE]. Before I ordered I did read up and I was aware that there are four EasyCAP clones, two good ones, one so-so and one bad. Whole introductory story about EasyCAP on Linux is available on Linux TV, here http://linuxtv.org/wiki/index.php/Easycap. That was so-so case. Following instructions from Linux TV I am on easycap-somagic-linux website and now following their instructions. It says install Wine - I hate Wine, but OK. Try to install driver from CD under Wine it reports error. After few attempts, start Windows Server 2003 inside VM, install driver and copy over SmiUsbGrabber3C.sys to extract firmware. Everything goes well and device is initialised, it says now ID is 1c88:003c and time to plug in DVD player and try it. There is no sound but I do not really need it. Here is screen capture of MPlayer:

Not very promising. Try different settings, different capture parameters and most important eject DVD. The last one brings color:

While I got video and even color, it is not usable for what I need it. Attempted capture in Windows inside VM, nothing it is black screen for duration of capture. Give it to my son to install driver to his gaming box - doesn’t want to install, Windows 7 doesn’t like driver and doesn’t want to use it. Try booting into Windows XP, the same story. Maybe it works on Mac? Sorry do not have Mac to try it. Time to put aside Somagic, no magic today, and give to guys from easycap-somagic-linux time to finish driver, it produces some results already.
Ordering more EasyCAP devices with probability of 50% to get usable one or going for more expensive one which works? Think that will go for slightly more expensive device and wait to see it working before I recommend it to anybody.
If anybody managed to use Somagic EasyCAP DC60+ on any operating system, I would like to hear about it, so please leave comment.

Download YouTube video and convert it for Android

For downloading videos from YouTube I am using YouTubeDownloader2 and you can find it here http://sourceforge.net/projects/ytd2/. Why I am using that and not using Firefox add-ons? YouTubeDownloader2 is standalone program written in Java which is not storing my browsing habits in some remote DB. How do I know that? That is OSS, downloaded code, checked what it does, compiled it. If you don’t do Java, trust others who do ;-)
Once video is on file system we can encode it. It should be easy on Linux unless you are have common repo FFmpeg which is compiled without support for H.264, mp3 and few others codecs. If you are experienced Linux user you will visit FFmpeg wiki where is detailed instruction https://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide how to configure and build FFmpeg. I am on Mint so that is valid set of instructions for me, people using other distro will look for their set of instructions. If you are not experienced user maybe mencoder which comes by default with everything enabled is better option. So, install libavcodec-extra-53 or maybe 54 and mencoder. We can ask mencoder what is supported if in terminal we execute:

~ $ mencoder -ovc help
MEncoder svn r34540 (Ubuntu), built with gcc-4.6 (C) 2000-2012 MPlayer Team

Available codecs:
copy     - frame copy, without re-encoding. Doesn't work with filters.
frameno  - special audio-only file for 3-pass encoding, see DOCS.
raw      - uncompressed video. Use fourcc option to set format explicitly.
nuv      - nuppel video
lavc     - libavcodec codecs - best quality!
xvid     - XviD encoding
x264     - H.264 encoding

and it produces video codecs list. For audio, query looks like this:

~ $ mencoder -oac help
MEncoder svn r34540 (Ubuntu), built with gcc-4.6 (C) 2000-2012 MPlayer Team

Available codecs:
   copy     - frame copy, without re-encoding (useful for AC3)
   pcm      - uncompressed PCM audio
   mp3lame  - cbr/abr/vbr MP3 using libmp3lame
   lavc     - FFmpeg audio encoder (MP2, AC3, ...)
   faac     - FAAC AAC audio encoder

There is small problem with mencoder, there is no WinFF for it and one have to do quite a bit of reading to assemble proper set of parameters. To save you from learning, here is what I do:

~ $ mencoder -of lavf -lavfopts format=mp4 -oac lavc -ovc lavc -lavcopts vbitrate=480:aglobal=1:vglobal=1:acodec=libfaac:abitrate=64:vcodec=mpeg4:keyint=25 -ofps 15 -af lavcresample=44100 -vf harddup,scale=480:320 -mc 0 "/home/yourlogin/in.flv" -o "/home/yourlogin/out.mp4"

I want H.263 video and AAC audio in MP4 container. Video should have bitrate up to 480kb/s and must be resized to 480x320 pixels, also frame rate is 15 frames per second.  For audio requested was resampling at 44.1 KHz and bit rate up to 64kb/s. It is not requirement that input video must be FLV, it should work for any container which you can find at YouTube. If your screen is different from 480x320, you may want to resize video differently. For example for 320x240 screen, vbitrate=256 and scale=320:240 should be good choice.
Not so nice as WinFF but not very complicated either.

How to stack images using G’MIC

After chat on Astophotography Google+ community https://plus.google.com/u/0/communities/118208937662082340807 I concluded that it may be enough interest for 16 bit image processing using G’MIC tutorial and here it is.
G'MIC stands for GREYC's Magic Image Converter and we know it as plugin for GIMP. It is less known that G'MIC framework can be used on it’s own through simple shell scripting or possible C++ or web interface. What is very important it supports 16 bits integers per channel and that is something what GIMP is lacking.
So, what we are going to do here is to take a look at G’MIC tutorial http://gmic.sourceforge.net/tutorial.shtml and load images into stack average them and save them as 16 bit TIFF.
Tutorial says if you are using G’MIC as GIMP plugin and set logging to verbose it will print what it does.

I prefer writing output to log file to suggested running GIMP from terminal (command line for Windows users). Standard path and name for log file is /tmp/gmic_log
Also from tutorial we see that 16 bit processing starts with division with 256 and ends with multiplication with 256. While that represents reduction to 8 bit it is important to note that G’MIC internally works with floating point numbers and there is no loss of precision when we convert it back to 16 bit integers. Once when we are done with processing it is important how we are going to specify TIFF output. G’MIC will pick format from extension but it by default writes 32 bit TIFF. So, to get 8 bit TIFF we need to specify output type uchar and to get 16 bit TIFF we need to specify output type ushort.
Typical operations which we are going to use are:

1.     -gimp_haar_smoothing 0.1,10,2,0,0 what is Smooth [wavelets] under Enhancements
2.     -gimp_compose_average 1,0 what is Blend [average]  under Layers
3.     -gimp_compose_screen 1,0 what is Blend [screen] under Layers

So complete workflow is we pick nicer RAWs export them as 16 bit PPMs using darktable as in http://grumpyoldprogrammer.blogspot.com/2012/11/more-about-stacking.html then we align them using align_image_stack from hugin-tools as in http://grumpyoldprogrammer.blogspot.com/2012/12/how-to-align-image-stack.html and finally we process them with G’MIC.
Averaging layers will produce at the end two layers, if we want we can save them as separate pictures, like this:

$ gmic tif0000.tif tif0001.tif tif0002.tif tif0003.tif -div 256 -gimp_compose_average 1,0 -mul 256 -c 0,65536 -type ushort -output[1] imageA.tiff -output[0] imageB.tiff

or we can stack those two layers in screen mode to boost light and stretch contrast, like this:

$ gmic tif0000.tif tif0001.tif tif0002.tif tif0003.tif -div 256 -gimp_compose_average 1,0 -gimp_compose_screen 1,0 -mul 256 -c 0,65536 -type ushort -output imageS.tiff

If we want to do resizing we can as in G’MIC tutorial use -resize2dx 1600,5 what is bi-cubic resizing to 1600 pixels width.
Instead of averaging we can try other options like -compose_median what is median or any other available filter or combination of filters, we try in GIMP, set logging to verbose and later run it in terminal without loss of data.

How to align image stack

Aligning images in GIMP is not very difficult if we do not have rotation to attend to, if rotation is present we rather leave that task to computer.
Aligning stack of images is not only used in astrophotography, it is common task in creation of HDR images. So, we can use align_image_stack from hugin-tools to align images. Installation procedure on Mint or Ubuntu is simple, we find it:

$ apt-cache search hugin
enblend - image blending tool
enfuse - image exposure blending tool
hugin - panorama photo stitcher - GUI tools
hugin-data - panorama photo stitcher - common data files
hugin-tools - panorama photo stitcher - commandline tools

and after that we install it:

$ sudo apt-get install hugin

It is available from repositories and there is no need to compile it from source. People using different operating system from Linux should visit Hugin website http://hugin.sourceforge.net/download/ and download installer for their operating system.
Once Hugin is installed align_image_stack should be available and we can align image stack. If you do not have own images in this tutorial http://grumpyoldprogrammer.blogspot.com/2012/11/even-more-astrophotography.html you will find download links.
We place in some empty directory JPEGs, if you have RAWs convert them, cd to that directory and execute:

$ align_image_stack -a tif *.JPG

Option -a tif means we want tif indexed output, so after a while we will see tif0000.tif, tif0001.tif and so on. Parameter *.JPG is input list, align all JPEGs. Alternatively we can specify file by file. Once alignment is done we can start GIMP and open all images as layers.

In order to check alignment we can change mode for top layer from Normal to Difference in Layers - Brushes floating window. To check further we make top layer invisible and repeat the same procedure for next layer.

Once we are happy we can stack images as described in previous tutorials or we can use GMI’C plugin. GMI’C is located under Filters and it opens as separate window. We expand Layers, select Average and set Input layers to All and Output mode to New image.

Clicking Apply or OK button will create new image. If you have new version of GIMP Median is also interesting. Average produces two layers and Median only one, we can stack them again. For final processing you may like to do some contrast stretching, maybe some curves as well.

RESTful web service for picture upload Android client

In previous tutorial http://grumpyoldprogrammer.blogspot.com/2012/11/image-upload-via-restful-web-service.html I described how to create simple server application and deploy it on Tomcat. Now we are going to take a look at client application.
Simplest is to use Apache HttpComponents Client. It is possible to convert HttpMime jar for Android and import it as library. Since not all classes are required for picture upload we can download source and add the following files to project:

AbstractContentBody.java
ByteArrayBody.java
ContentBody.java
ContentDescriptor.java
FormBodyPart.java
Header.java
HttpMultipart.java
HttpMultipartMode.java
MIME.java
MinimalField.java
MultipartEntity.java
StringBody.java

Naturally whole project like this becomes Apache licensed. Some of those Java files are in org.apache.http.entity.mime.content package and others are in org.apache.http.entity.mime.
In this way we avoid compiling two files FileBody.java and InputStreamBody.java
Now will we go refactoring package names to fit them into project is of minor relevance, I usually do.
We create new Android project and we code onCreate like this:

public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    try {
        bm = BitmapFactory.decodeFile("/sdcard/DCIM/FinalM8.JPG");
        if(bm==null)
            throw new Exception("no picture!");
        new FetchItemsTask().execute();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Please note that image path is hardcoded, so change it accordingly. Bitmap bm is loaded, defined as field, and if it is different from null we create and execute AsyncTask to upload picture. To keep things simple as possible we will implement only doInBackground, like this:

protected Void doInBackground(Void... arg0) {
    // TODO Auto-generated method stub
    try {
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        bm.compress(CompressFormat.JPEG, 50, bos);
        byte[] data = bos.toByteArray();
        HttpClient httpClient = new DefaultHttpClient();
        HttpPost postRequest =
            new HttpPost("http://localhost:8080/WebApplication3/xyz/generic/images");
        ByteArrayBody bab = new ByteArrayBody(data, "FinalM8.JPG");
        MultipartEntity reqEntity = new MultipartEntity(
                HttpMultipartMode.BROWSER_COMPATIBLE);
        reqEntity.addPart("image", bab);
        postRequest.setEntity(reqEntity);
        HttpResponse response = httpClient.execute(postRequest);
        System.out.println("Status is "+response.getStatusLine());
        BufferedReader reader = new BufferedReader(new InputStreamReader(
                response.getEntity().getContent(), "UTF-8"));
        String sResponse;
        StringBuilder s = new StringBuilder();
        while ((sResponse = reader.readLine()) != null) {
            s = s.append(sResponse);
        }
        System.out.println("Response: " + s);
    } catch (Exception e) {
        // handle exception here
        e.printStackTrace();
    }
    return null;
}

Image is compressed on 50%, loaded into byte array and upload is attempted. In order to run code just push onto emulator some image and change path.