Even More Astrophotography

Everything what was described in previous tutorial should work on Windows as well. Fiji uses Java and works everywhere as advertised and GIMP installer for Windows certainly exists. How to install plugin registry on Windows I really do not know, Google is your friend.

We can go downloading and processing FITS files from many places beside mentioned LCOGT we can use Hubble data from http://hla.stsci.edu/hlaview.html or maybe data from Misti Mountain Observatory http://www.mistisoftware.com/astronomy/index_fits.htm to name few.

All that is nice, but real fun begins when we capture own data using our own camera.

Taking pictures without tracking

Any kind of camera, any kind of lens, capable of delivering sharp pictures will do. There is another inexpensive piece of equipment which is a must - tripod. I am using old Sony A290 DSLR with SAL75300 telephoto lens.
Now we go out place tripod and put camera on it. Select manual mode, adjust ISO to 800, maybe 1600 and exposure long as possible bat not too long to avoid star trailing. Shorter focal length will allow longer exposures. How long exposure can be? That depends on many things, your position on the globe, declination of target and so on. With 75mm focal length on DSLR what corresponds to 112.5mm on 35mm SLR I am happy with 4 to 5 seconds of exposure in Johannesburg, South Africa. If 50mm lens is available I would go for 6 to 8 seconds exposure. So, select exposure, go into drive mode select three or five shots burst, aim and fire.
If you are going to use some stacking software like Deep Sky Stacker you can take RAW and JPEG picture simultaneously. Deep Sky Stacker will not work on Linux and you will have to run Windows inside virtual machine to use it.
When you have few nice snapshots of for example Milky Way you can go back to computer and stack them using GIMP. We will describe stacking in the last part of article.

Manual tracking

Soon as you stack few snapshots appetite will start growing. Going for hundred snapshots is not way forward. Way forward is to increase exposure time. If you do not have $$$$ to spend on real telescope with computerised equatorial mount, you need to look for cost-effective solution. For example to build barn door tracker or you may just have cheap telescope with equatorial mount which is only good for taking Moon snapshots. Cheap telescopes are coming with poor quality equatorial mounts which are very shaky. Now worst thing which you may attempt is spending money to stabilise cheap mount. Attach weight to its tripod, few rounds of rope around legs to tighten it and that’s it. Doesn’t look nice but does job.

Placing camera on telescope could be done using proper piggyback mount or you can made one. I am using ordinary wire ties tightened between camera and quick release head. Don’t go too sloppy you may destroy camera in that way.
Placing eyepiece with higher magnification in and locating bright star close to edge of viewing field you are ready to go. My camera goes up to 30 seconds and after that BULB, for BULB I need remote, so 30 seconds is what I am aiming for. How long exposure can be? It depends of equatorial mount setup, with quick alignment you should be able to pool one minute. Tolerance for your tracking errors is again function of focal length, 300mm is likely just waste of time with maybe one  good frame out dozen.

Stacking frames in GIMP

I uploaded four re-sized frames of area around M 8 if you want to practice before you take own snapshots. Here are links:

https://docs.google.com/open?id=0B0cIChfVrJ7WbkJtZjg3LWlGbXM
https://docs.google.com/open?id=0B0cIChfVrJ7WSWF1LUZ5UnBVRTg
https://docs.google.com/open?id=0B0cIChfVrJ7WMzdnRXdmZmJOSDQ
https://docs.google.com/open?id=0B0cIChfVrJ7WNFplbk5kb2ozYUU

We open the first frame and possibly reduce noise, if required, as in previous tutorial.
If you are using my frames, which are re-sized, there is no need for noise reduction or hot pixels removal. Hot pixels removal will remove quite few stars on re-sized image.
Since those are longer exposure captures we will have hot pixels. To eliminate hot pixels we open Filters -> G’MIC -> Enhancement -> Hot pixels filtering and apply it with default values. Now we open as layer next frame, select it in layers and in Filters we Repeat “G’MIC”. Now we set mode from Normal to Screen, zoom to 100% or more and align layers. We repeat the same for remaining frames. At the end we Merge Visible Layers from context menu for layers (right click one) accepting default expand as necessary option. If picture is too bright what would be a case with supplied pictures, we will do contrast stretching. As we add frames we increase level of signal and histogram changes like this:

So, Colors -> Auto -> White Balance and after that Edit -> Fade Levels where we set mode to Multiply. This is what final result should look like:

That would be simple manual stacking of JPEG frames with satisfactory final result. We could also align color levels on this picture but that was not goal of this tutorial.

GIMP, Fiji and astrophotography on Linux

Ever wanted to know how to process those wonderful Hubble like pictures on Linux? It is not difficult, I will show you how. We will do RGB processing and LRGB is very similar.

On the Web we will encounter plenty tutorials where author uses some proprietary tool to do contrast stretching and after that Photo Shop to do final processing. Usually none of those will be available on Linux. Replacement for Photo Shop is no brainer, it is naturally popular GIMP. Tool for contrast stretching is trickier. For nonlinear stretching I am using Fiji, what is almost the same as ImageJ. Guess there is no distribution which doesn’t offer GIMP so just follow usual install path for your distro. It is good idea to install gimp-plugin-registry as well. For Fiji you will have to download it from http://fiji.sc/wiki/index.php/Downloads and untar it. It comes with or without Java runtime, so we pick what we need depending do we already have Java or not.

Now we installed required programs and we need data. Typically data consists of three or more gray images in FITS format. FITS is abbreviation for Flexible Image Transport System. Good source of FITS data is Las Cumbres Observatory Global Telescope Network and here is their website http://lcogt.net/

They have two two meter reflectors and few smaller telescopes and observation data is freely available under Creative Commons license. If you get data from 2m telescope you will end-up with three 2008x2008 pixels images about eight megabytes each. So go there and pick some galaxy from Observations section, I will go for NGC6946, looks nice. After downloading Blue, Green and Red FITS we can start.

And we immediately see why we need contrast stretching, there is barely few stars. Now from menu we do Process -> Enhance Contrast, tick equalize histogram and hit OK button. Result looks like this:

There is much more to see but also huge amount of noise. We repeat the same story for remaining files and save them as Tiff. If we want we can go to Image -> Color -> Merge Channels and create composite to see how approximately it will look like.

It is nice but too much noise, time for GIMP.

We open all three tifs in GIMP and we do Filters -> Enhance ->Wavelet Denoise with settings like on picture. If you don't have gimp-plugin-registry installed there will be no Wavelet Denoise then just do Despeckle few times.

We do the same on remaining two pictures doing Ctrl+F repeating last filter. Next step is Image -> Mode -> RGB followed by Colors -> Colorify and we apply what is actual color on it.

Now we copy green and paste as layer over red, rename Pasted Layer to something and change layer mode to Screen, we do the same with blue one.

If alignment is OK we can merge layers.

Now we can play with curves, levels or do decomposition to enhance colors and so on, get imaginative here. Here is how it looks like without any additional processing.

If frames are properly aligned we could place them as layers into single image and do Colors -> Components -> Compose what is simpler than doing Mode and Colorify.

If we have LRGB, then we process RGB as above. L we stretch, denoise and at the end use it as value layer and RGB as color layer.

Tutorial part 3

In this part we are going to take a look at AsyncTask which is light-weight framework intended to save programmers from concurrency traps and handling of Java threads.

AsyncTask

Most important thing is AsyncTask is intended to be used on UI thread, so do not try to use it elsewhere. Signature is android.os.AsyncTask and you can use them or ignore them and replace unused ones with Void. If you are planning to pass array of strings as parameters and not use progress or result than it becomes android.os.AsyncTask.
There are four commonly used and overridden methods:

protected void onPreExecute ()
protected abstract Result doInBackground (Params... params)
protected void onProgressUpdate (Progress... values)
protected void onPostExecute (Result result)

obviously only doInBacground is compulsory to override and that is only one which is not executing on UI thread. Typical implementation looks like this:

private class FetchItemsTask extends AsyncTask {
    @Override
    protected void onPreExecute() {
        super.onPreExecute();
        progressDialog.setMessage("Downloading. Please wait...");
        progressDialog.show();
    }
    @Override
    protected Void doInBackground(Void... params) {
        do some heavy I/O work here
        return null;
    }
    @Override
    protected void onPostExecute(Void result) {
        pass to UI result of work here
        progressDialog.dismiss();
    }
}

Usually it is declared as inner class inside Activity and we have instance of ProgressDialog accessible.
It is not full size threading framework, it should be used as suggested to do lengthy I/O operations. That lengthy is few seconds not few minutes. So good candidate for downloading image from Internet or pulling data from DB. Very good idea here is to go to your Android SDK Manager and to download source code and see how implementation of AsyncTask looks like. Equally good idea is to place logging at every method and log ID of current thread, so that we can be sure where is what executed.
If we take a look at my example in git://github.com/FBulovic/grumpyoldprogrammer.git we will see that my BroadcastReceiver contains  AsyncTask. Knowing that  AsyncTask can bi instantiated only from UI thread we can conclude that  BroadcastReceiver is also executed on UI thread. That was nice intro to CursorAdapter and ContentProvider where execution happens outside of UI thread but they need to communicate with UI thread and that may course some problems here and there. Especially you want to access SQLite DB directly without using ContentProvider.
Next part of this tutorial will follow, soon.

Tutorial part 2

Since we have scheduler to retrieve data maybe we want to put data into database. On Android we have SQLite as part of environment so choice is easy.

Subclassing SQLiteOpenHelper

Things are straightforward here. There are two methods which must be implemented:

@Override
public void onCreate(SQLiteDatabase arg0) {
    arg0.execSQL(createSQL);
}
@Override
public void onUpgrade(SQLiteDatabase arg0, int arg1, int arg2) {
    arg0.execSQL("DROP TABLE IF EXISTS "+TABLE_NAME);
    onCreate(arg0);
}

variable createSQL is String containing simple SQL statement to create table:

CREATE TABLE tweets (_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, 
    user_name TEXT NOT NULL, date_name TEXT NOT NULL, 
    text_name TEXT NOT NULL)

Self-incrementing primary key _id is required by Android, to use it in data binding, must be named like this.
Another thing which is required is constructor:

public DbHelper(Context context) {
    super(context, DATABASE_NAME, null, 1);
}

Beside context we need to pass to super-class constructor DB name, optional CursorFactory and version.
Using helper looks like this:

DbHelper dbW = new DbHelper(this);
SQLiteDatabase db = dbW.getReadableDatabase();
Cursor cursor = db.query("tweets", new String[] { "_id", DbHelper.DATE,
        DbHelper.TEXT, DbHelper.USER }, null, null, null, null, null,
        null);
cursor.moveToFirst();
...
do something with cursor
...
cursor.close();

It is obviously called DbHelper, since query needs to be done readable database is enough. Query method looks odd and since we are selecting everything we just provide table name and collection of columns. Once when work with cursor is done we need to close it, SQLite is quite tough on resource managemet. Other parameters of query method are selection, order, limit, having, group by. If we want to specify selection it will be something like this:

cursor = db.query("route", new String[] { "_id",
        DbHelper.ADDRESS, DbHelper.PHONE,
        DbHelper.SURNAME }, DbHelper.STATUS + "="
        + Integer.toString(filterEquals) + " and "
        + DbHelper.ADDRESS + " like '%" + searchString + "%'",
        null, null, null, null, null);

So selection is “where”.
Instead of this pseudo OO approach raw SQL can be used:

String updateSQL = "UPDATE my_table SET complete = 4  WHERE status > 0;";
dbW.getWritableDatabase().execSQL(updateSQL);

It is certainly easier at the beginning to use normal SQL – less unknown stuff to fight with.
If we are take a look at code for this (and other) tutorial, we see that I never used directly SQL except inside provider. If you use it directly be prepared for really strict resource management and Android likes executing things on different threads what doesn't go nicely with SQLite which doesn't let you close cursors created on different tread from yours. Cure for this is to call startManagingCursor and leave to Android to close dangling cursor. Do not be discouraged with deprecation of startManagingCursor if you can't control execution it is much better than leaking resources. All that nightmare suddenly stops if you do not use SQLite DB directly but wrapped up in provider. Political approach, you are free to do what you want but we will make you use what we want you to use.
Code for this tutorial (and others) is here git://github.com/FBulovic/grumpyoldprogrammer.git
Next part of this tutorial will follow.

Android tutorial (multipart)

This tutorial will cover AsyncTask, SQLite, SQLiteOpenHelper, ContentProvider, AlarmManager, BroadcastReceiver, custom filterable CursorAdapter, ListView, TextWatcher, JSONArray and JSONObject.
We will actually use HttpClient but I was talking about it already. Goal of tutorial is to fetch periodically search results from Twitter, store them in DB and display them in list view. We will make those search results searchable.

Cron job Android style

Android is Linux but Cron daemon is missing, we must use  AlarmManager and  BroadcastReceiver. Simplest form of  BroadcastReceiver looks like this:

public class CronJobScheduler extends BroadcastReceiver {
    @Override
    public void onReceive(Context context, Intent intent) {
        System.out.println("inside onReceive");
    }
}

To use it from your Activity you will do something like this:

Context context = getBaseContext();
AlarmManager manager=(AlarmManager)context.getSystemService(Context.ALARM_SERVICE);
Intent intent = new Intent(context, CronJobScheduler.class);
PendingIntent pending = PendingIntent.getBroadcast(context, 0, intent, 0);
manager.setRepeating(AlarmManager.RTC_WAKEUP, System.currentTimeMillis(), 1000 * 20 , pending);

and at appropriate place you will also stop scheduled job, like this:

Context context = getBaseContext();
Intent intent = new Intent(context, CronJobScheduler.class);
PendingIntent pending = PendingIntent.getBroadcast(context, 0, intent, 0);
AlarmManager alarmManager = (AlarmManager) context.getSystemService(Context.ALARM_SERVICE);
alarmManager.cancel(pending);

But that is not all, in AndroidManifest.xml we must place this:



It must be within application element. If we build app and run it in emulator it will print every 20 seconds inside onReceive.
In order to do something useful (like download JSON, parse it and insert data in DB) we will use later this code:

PowerManager manager = (PowerManager) context.getSystemService(Context.POWER_SERVICE);
PowerManager.WakeLock lock = manager.newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, "HOWZIT:)");
lock.acquire();
cr = context.getContentResolver();
new FetchItemsTask().execute();
lock.release();

In order to have this executed we will need to add some permissions to  AndroidManifest.xml, like this:



We are acquiring PowerManager.PARTIAL_WAKE_LOCK and that is CPU lock, we want CPU to be awake so that scheduled work could be done. As usual it is good idea to release CPU lock when work is done ;-)
If anybody needs code for this it is available here git://github.com/FBulovic/grumpyoldprogrammer.git there is few other things, also.
Next part of this tutorial will follow, soon.

Is it safe?

Looking at tech headlines I spotted this:

Over 1,000 Android Apps Contain SSL Flaws by Tom Brewster

If they are claiming existence of some virus or Trojan which works on Android, I would just ignore it as complete nonsense. Now after initial “this is FUD” reaction I proceeded reading and encountered the following claims:

More than 1,000 legitimate Android apps contain SSL (Secure Sockets Layer) weaknesses, leaving them vulnerable to Man-in-the-Middle (MITM) attacks, researchers have claimed.

Without naming names, the students found one “generic online banking app”, which was trusting all certificates, even the MITM proxy with a self-signed certificate set up by the researchers. It had between 100,000 and 500,000 users.

Not naming names is really nice touch, maybe more unsuspecting users can fall victim to MITM criminals.
Knowing that software companies lately relying on do idiotic test recruitment process, I started to change opinion from trolling-FUD to it is actually possible. In pursuit of margins, software companies are inclined to employ younger, inexperienced and cheaper candidates, but in the same time they will force them to work faster and be productive as more experienced programmers. At the end it hurts them (couldn't care less for them), final users and whole development platform.
Naturally FUD ingredient can't be completely omitted, is situation with security implementation on proprietary mobile operating systems better? We can mess-up SSL configuration on every kind of OS using the same HttpComponents.
So, problem is not in platform, it is difficult to imagine something safer than Linux (Android uses Linux kernel), but in incompetency of would be programmers and their would be project managers. All those C# .NET Windows developers suddenly are becoming experts for Linux and Android – crazy stuff.

Apache HttpComponents and  HttpClient

Android uses 4.x version of HttpClient and that is old and reputable library. Problem is that in pursuit of deadlines people jump on Google search and finish with some half-baked solution from “Stack Overflow”. “Stack Overflow” can not replace help files and it is not its purpose. Read the manual. If you are not sure what help file says, download source code, yes that is open source you can do it, see how they are using that class in examples and tests. That is faster than reading threads on different forums.
It is possible to configure HttpClient , something like this:

private static final DefaultHttpClient client;
private static HttpParams params = new BasicHttpParams();
static {
    HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
    HttpProtocolParams.setContentCharset(params, "utf-8");
    params.setBooleanParameter("http.protocol.expect-continue", false);
    SchemeRegistry registry = new SchemeRegistry();
    registry.register(new Scheme("http", PlainSocketFactory
            .getSocketFactory(), 80));
    final SSLSocketFactory sslSocketFactory = SSLSocketFactory
            .getSocketFactory();
    sslSocketFactory
            .setHostnameVerifier(SSLSocketFactory.BROWSER_COMPATIBLE_HOSTNAME_VERIFIER);
    registry.register(new Scheme("https", sslSocketFactory, 443));
    ClientConnectionManager manager = new ThreadSafeClientConnManager(
            params, registry);
    ConnManagerParams.setMaxTotalConnections(params, 20);
    ConnManagerParams.setMaxConnectionsPerRoute(params,
            new ConnPerRoute() {
                public int getMaxForRoute(HttpRoute httproute) {
                    return 10;
                }
            });
    client = new DefaultHttpClient(manager, params);
}

It doesn't have to be used in just give me defaults style. We need thread safe connection manager since we are going to use threads to check later is singleton thread safe.
If one does search on “How to ignore SSL certificate errors in Apache HttpClient 4.0”, there is nice example how not to configure HttpClient. May be used in early development stage, but releasing things like that is asking for lawsuit.
Once client is configured, one may want to provide it in organized manner to the rest of application, singleton looks like ideal storage for it.

Lazy Initialization Holder Class Idiom

This one is quite popular in Java world since appearance of book Java Concurrency in Practice by Goetz and others. It relies on Java Language Specification and JVM loading. Here is how it aproximately looks like:

public class SafeLazy {
    private static class SingletonHolder {
        private static final DefaultHttpClient client = new DefaultHttpClient();
    }

    public static DefaultHttpClient getInstance() {
        return SingletonHolder.client;
    }
}

Since outer class doesn't have static fields it will be loaded without initializing HttpClient and SingletonHolder. JVM will perform initialization of inner class only when it needs to be accessed. Naturally one may want to configure slightly that client, like in the previous code example.
Now you can avoid problems of type “if I try successive downloads it just hangs”, “I trust every certificate from everywhere” and so on.
If you need complete sample with test it is available here https://github.com/FBulovic/isitsafe.git you know how to use Git, don't you?

Who you follow but he or she does not follow you back on Twitter

This tutorial primarily targets DIY inclined nonprogrammers or very lazy programmers. In order to do professional interaction with Twitter one would use their API which is quite well documented and development portal is available here https://dev.twitter.com/
For DIY approach we are happy to pull HTML, page source and work with it. I use Firefox and add-on Firebug by Joe Hewitt. If you do not have Firebug between your add-ons - install it.  If you are using Google Chrome Cedric created similar tutorial here http://www.cedricve.me/2012/04/10/howto-scrape-your-twitter-followers-with-perl/ though code and goal is somewhat different.
Now login into your Twitter account and expand all your followers to the bottom of the page. If we save page now there will be no required data, it is loaded via Java script how we scrolling and expanding list. To see data we invoke Firebug via context menu, right click on the page and select "Inspect Element with Firebug".

In search box (right from context menu on the picture) we type "stream-items-id" and hit enter. Selecting whole "div" we right click and copy inner HTML. Paste that in your favorite text editor and save, people using Windows should use Notepad++. Do the same for list following. I saved followers into file called fm and following into ft, if you like different naming scheme, please rename files in code accordingly.
In directory where fm and ft are saved create Perl script with following content:

open file1, "<" , "ft"; open file2, "<" , "fm"; %ht=(); while(<file1>) { chomp; if(( $_ =~ /\<div data-user-id=\"(.*?)\" data-screen-name=\"(.*?)\" class=\"account js-actionable-user js-profile-popup-actionable \">/ )){ if(( $1 && $2 )) { $ht{$2} = $1; } } } while(<file2>) { chomp; if(( $_ =~ /\<div data-user-id=\"(.*?)\" data-screen-name=\"(.*?)\" class=\"account js-actionable-user js-profile-popup-actionable \">/ )){ if(( $1 && $2 )) { if ( exists $ht{$2} ){ delete $ht{$2}; } } } } close(file1); close(file2); foreach $h (keys %ht){ print "$h - $ht{$h}\n"; }

when you run it, terminal or command line or from IDE it should produce output like this:

Queen_Europe - 407347022
FaulkesSouth - 102351359
lcogt - 80797776
glenn_hughes - 15565190
FaulkesNorth - 102350867
Rogozin - 36980694
wikileaks - 16589206
Nigel_Farage - 19017675
StoppINDECT – 169043724

those are screen names and user ID pairs.