Tutorial part 3

In this part we are going to take a look at AsyncTask which is light-weight framework intended to save programmers from concurrency traps and handling of Java threads.

AsyncTask

Most important thing is AsyncTask is intended to be used on UI thread, so do not try to use it elsewhere. Signature is android.os.AsyncTask and you can use them or ignore them and replace unused ones with Void. If you are planning to pass array of strings as parameters and not use progress or result than it becomes android.os.AsyncTask.
There are four commonly used and overridden methods:

protected void onPreExecute ()
protected abstract Result doInBackground (Params... params)
protected void onProgressUpdate (Progress... values)
protected void onPostExecute (Result result)

obviously only doInBacground is compulsory to override and that is only one which is not executing on UI thread. Typical implementation looks like this:

private class FetchItemsTask extends AsyncTask {
    @Override
    protected void onPreExecute() {
        super.onPreExecute();
        progressDialog.setMessage("Downloading. Please wait...");
        progressDialog.show();
    }
    @Override
    protected Void doInBackground(Void... params) {
        do some heavy I/O work here
        return null;
    }
    @Override
    protected void onPostExecute(Void result) {
        pass to UI result of work here
        progressDialog.dismiss();
    }
}

Usually it is declared as inner class inside Activity and we have instance of ProgressDialog accessible.
It is not full size threading framework, it should be used as suggested to do lengthy I/O operations. That lengthy is few seconds not few minutes. So good candidate for downloading image from Internet or pulling data from DB. Very good idea here is to go to your Android SDK Manager and to download source code and see how implementation of AsyncTask looks like. Equally good idea is to place logging at every method and log ID of current thread, so that we can be sure where is what executed.
If we take a look at my example in git://github.com/FBulovic/grumpyoldprogrammer.git we will see that my BroadcastReceiver contains  AsyncTask. Knowing that  AsyncTask can bi instantiated only from UI thread we can conclude that  BroadcastReceiver is also executed on UI thread. That was nice intro to CursorAdapter and ContentProvider where execution happens outside of UI thread but they need to communicate with UI thread and that may course some problems here and there. Especially you want to access SQLite DB directly without using ContentProvider.
Next part of this tutorial will follow, soon.

Tutorial part 2

Since we have scheduler to retrieve data maybe we want to put data into database. On Android we have SQLite as part of environment so choice is easy.

Subclassing SQLiteOpenHelper

Things are straightforward here. There are two methods which must be implemented:

@Override
public void onCreate(SQLiteDatabase arg0) {
    arg0.execSQL(createSQL);
}
@Override
public void onUpgrade(SQLiteDatabase arg0, int arg1, int arg2) {
    arg0.execSQL("DROP TABLE IF EXISTS "+TABLE_NAME);
    onCreate(arg0);
}

variable createSQL is String containing simple SQL statement to create table:

CREATE TABLE tweets (_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, 
    user_name TEXT NOT NULL, date_name TEXT NOT NULL, 
    text_name TEXT NOT NULL)

Self-incrementing primary key _id is required by Android, to use it in data binding, must be named like this.
Another thing which is required is constructor:

public DbHelper(Context context) {
    super(context, DATABASE_NAME, null, 1);
}

Beside context we need to pass to super-class constructor DB name, optional CursorFactory and version.
Using helper looks like this:

DbHelper dbW = new DbHelper(this);
SQLiteDatabase db = dbW.getReadableDatabase();
Cursor cursor = db.query("tweets", new String[] { "_id", DbHelper.DATE,
        DbHelper.TEXT, DbHelper.USER }, null, null, null, null, null,
        null);
cursor.moveToFirst();
...
do something with cursor
...
cursor.close();

It is obviously called DbHelper, since query needs to be done readable database is enough. Query method looks odd and since we are selecting everything we just provide table name and collection of columns. Once when work with cursor is done we need to close it, SQLite is quite tough on resource managemet. Other parameters of query method are selection, order, limit, having, group by. If we want to specify selection it will be something like this:

cursor = db.query("route", new String[] { "_id",
        DbHelper.ADDRESS, DbHelper.PHONE,
        DbHelper.SURNAME }, DbHelper.STATUS + "="
        + Integer.toString(filterEquals) + " and "
        + DbHelper.ADDRESS + " like '%" + searchString + "%'",
        null, null, null, null, null);

So selection is “where”.
Instead of this pseudo OO approach raw SQL can be used:

String updateSQL = "UPDATE my_table SET complete = 4  WHERE status > 0;";
dbW.getWritableDatabase().execSQL(updateSQL);

It is certainly easier at the beginning to use normal SQL – less unknown stuff to fight with.
If we are take a look at code for this (and other) tutorial, we see that I never used directly SQL except inside provider. If you use it directly be prepared for really strict resource management and Android likes executing things on different threads what doesn't go nicely with SQLite which doesn't let you close cursors created on different tread from yours. Cure for this is to call startManagingCursor and leave to Android to close dangling cursor. Do not be discouraged with deprecation of startManagingCursor if you can't control execution it is much better than leaking resources. All that nightmare suddenly stops if you do not use SQLite DB directly but wrapped up in provider. Political approach, you are free to do what you want but we will make you use what we want you to use.
Code for this tutorial (and others) is here git://github.com/FBulovic/grumpyoldprogrammer.git
Next part of this tutorial will follow.

Android tutorial (multipart)

This tutorial will cover AsyncTask, SQLite, SQLiteOpenHelper, ContentProvider, AlarmManager, BroadcastReceiver, custom filterable CursorAdapter, ListView, TextWatcher, JSONArray and JSONObject.
We will actually use HttpClient but I was talking about it already. Goal of tutorial is to fetch periodically search results from Twitter, store them in DB and display them in list view. We will make those search results searchable.

Cron job Android style

Android is Linux but Cron daemon is missing, we must use  AlarmManager and  BroadcastReceiver. Simplest form of  BroadcastReceiver looks like this:

public class CronJobScheduler extends BroadcastReceiver {
    @Override
    public void onReceive(Context context, Intent intent) {
        System.out.println("inside onReceive");
    }
}

To use it from your Activity you will do something like this:

Context context = getBaseContext();
AlarmManager manager=(AlarmManager)context.getSystemService(Context.ALARM_SERVICE);
Intent intent = new Intent(context, CronJobScheduler.class);
PendingIntent pending = PendingIntent.getBroadcast(context, 0, intent, 0);
manager.setRepeating(AlarmManager.RTC_WAKEUP, System.currentTimeMillis(), 1000 * 20 , pending);

and at appropriate place you will also stop scheduled job, like this:

Context context = getBaseContext();
Intent intent = new Intent(context, CronJobScheduler.class);
PendingIntent pending = PendingIntent.getBroadcast(context, 0, intent, 0);
AlarmManager alarmManager = (AlarmManager) context.getSystemService(Context.ALARM_SERVICE);
alarmManager.cancel(pending);

But that is not all, in AndroidManifest.xml we must place this:



It must be within application element. If we build app and run it in emulator it will print every 20 seconds inside onReceive.
In order to do something useful (like download JSON, parse it and insert data in DB) we will use later this code:

PowerManager manager = (PowerManager) context.getSystemService(Context.POWER_SERVICE);
PowerManager.WakeLock lock = manager.newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, "HOWZIT:)");
lock.acquire();
cr = context.getContentResolver();
new FetchItemsTask().execute();
lock.release();

In order to have this executed we will need to add some permissions to  AndroidManifest.xml, like this:



We are acquiring PowerManager.PARTIAL_WAKE_LOCK and that is CPU lock, we want CPU to be awake so that scheduled work could be done. As usual it is good idea to release CPU lock when work is done ;-)
If anybody needs code for this it is available here git://github.com/FBulovic/grumpyoldprogrammer.git there is few other things, also.
Next part of this tutorial will follow, soon.

Is it safe?

Looking at tech headlines I spotted this:

Over 1,000 Android Apps Contain SSL Flaws by Tom Brewster

If they are claiming existence of some virus or Trojan which works on Android, I would just ignore it as complete nonsense. Now after initial “this is FUD” reaction I proceeded reading and encountered the following claims:

More than 1,000 legitimate Android apps contain SSL (Secure Sockets Layer) weaknesses, leaving them vulnerable to Man-in-the-Middle (MITM) attacks, researchers have claimed.

Without naming names, the students found one “generic online banking app”, which was trusting all certificates, even the MITM proxy with a self-signed certificate set up by the researchers. It had between 100,000 and 500,000 users.

Not naming names is really nice touch, maybe more unsuspecting users can fall victim to MITM criminals.
Knowing that software companies lately relying on do idiotic test recruitment process, I started to change opinion from trolling-FUD to it is actually possible. In pursuit of margins, software companies are inclined to employ younger, inexperienced and cheaper candidates, but in the same time they will force them to work faster and be productive as more experienced programmers. At the end it hurts them (couldn't care less for them), final users and whole development platform.
Naturally FUD ingredient can't be completely omitted, is situation with security implementation on proprietary mobile operating systems better? We can mess-up SSL configuration on every kind of OS using the same HttpComponents.
So, problem is not in platform, it is difficult to imagine something safer than Linux (Android uses Linux kernel), but in incompetency of would be programmers and their would be project managers. All those C# .NET Windows developers suddenly are becoming experts for Linux and Android – crazy stuff.

Apache HttpComponents and  HttpClient

Android uses 4.x version of HttpClient and that is old and reputable library. Problem is that in pursuit of deadlines people jump on Google search and finish with some half-baked solution from “Stack Overflow”. “Stack Overflow” can not replace help files and it is not its purpose. Read the manual. If you are not sure what help file says, download source code, yes that is open source you can do it, see how they are using that class in examples and tests. That is faster than reading threads on different forums.
It is possible to configure HttpClient , something like this:

private static final DefaultHttpClient client;
private static HttpParams params = new BasicHttpParams();
static {
    HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
    HttpProtocolParams.setContentCharset(params, "utf-8");
    params.setBooleanParameter("http.protocol.expect-continue", false);
    SchemeRegistry registry = new SchemeRegistry();
    registry.register(new Scheme("http", PlainSocketFactory
            .getSocketFactory(), 80));
    final SSLSocketFactory sslSocketFactory = SSLSocketFactory
            .getSocketFactory();
    sslSocketFactory
            .setHostnameVerifier(SSLSocketFactory.BROWSER_COMPATIBLE_HOSTNAME_VERIFIER);
    registry.register(new Scheme("https", sslSocketFactory, 443));
    ClientConnectionManager manager = new ThreadSafeClientConnManager(
            params, registry);
    ConnManagerParams.setMaxTotalConnections(params, 20);
    ConnManagerParams.setMaxConnectionsPerRoute(params,
            new ConnPerRoute() {
                public int getMaxForRoute(HttpRoute httproute) {
                    return 10;
                }
            });
    client = new DefaultHttpClient(manager, params);
}

It doesn't have to be used in just give me defaults style. We need thread safe connection manager since we are going to use threads to check later is singleton thread safe.
If one does search on “How to ignore SSL certificate errors in Apache HttpClient 4.0”, there is nice example how not to configure HttpClient. May be used in early development stage, but releasing things like that is asking for lawsuit.
Once client is configured, one may want to provide it in organized manner to the rest of application, singleton looks like ideal storage for it.

Lazy Initialization Holder Class Idiom

This one is quite popular in Java world since appearance of book Java Concurrency in Practice by Goetz and others. It relies on Java Language Specification and JVM loading. Here is how it aproximately looks like:

public class SafeLazy {
    private static class SingletonHolder {
        private static final DefaultHttpClient client = new DefaultHttpClient();
    }

    public static DefaultHttpClient getInstance() {
        return SingletonHolder.client;
    }
}

Since outer class doesn't have static fields it will be loaded without initializing HttpClient and SingletonHolder. JVM will perform initialization of inner class only when it needs to be accessed. Naturally one may want to configure slightly that client, like in the previous code example.
Now you can avoid problems of type “if I try successive downloads it just hangs”, “I trust every certificate from everywhere” and so on.
If you need complete sample with test it is available here https://github.com/FBulovic/isitsafe.git you know how to use Git, don't you?

Who you follow but he or she does not follow you back on Twitter

This tutorial primarily targets DIY inclined nonprogrammers or very lazy programmers. In order to do professional interaction with Twitter one would use their API which is quite well documented and development portal is available here https://dev.twitter.com/
For DIY approach we are happy to pull HTML, page source and work with it. I use Firefox and add-on Firebug by Joe Hewitt. If you do not have Firebug between your add-ons - install it.  If you are using Google Chrome Cedric created similar tutorial here http://www.cedricve.me/2012/04/10/howto-scrape-your-twitter-followers-with-perl/ though code and goal is somewhat different.
Now login into your Twitter account and expand all your followers to the bottom of the page. If we save page now there will be no required data, it is loaded via Java script how we scrolling and expanding list. To see data we invoke Firebug via context menu, right click on the page and select "Inspect Element with Firebug".

In search box (right from context menu on the picture) we type "stream-items-id" and hit enter. Selecting whole "div" we right click and copy inner HTML. Paste that in your favorite text editor and save, people using Windows should use Notepad++. Do the same for list following. I saved followers into file called fm and following into ft, if you like different naming scheme, please rename files in code accordingly.
In directory where fm and ft are saved create Perl script with following content:

open file1, "<" , "ft"; open file2, "<" , "fm"; %ht=(); while(<file1>) { chomp; if(( $_ =~ /\<div data-user-id=\"(.*?)\" data-screen-name=\"(.*?)\" class=\"account js-actionable-user js-profile-popup-actionable \">/ )){ if(( $1 && $2 )) { $ht{$2} = $1; } } } while(<file2>) { chomp; if(( $_ =~ /\<div data-user-id=\"(.*?)\" data-screen-name=\"(.*?)\" class=\"account js-actionable-user js-profile-popup-actionable \">/ )){ if(( $1 && $2 )) { if ( exists $ht{$2} ){ delete $ht{$2}; } } } } close(file1); close(file2); foreach $h (keys %ht){ print "$h - $ht{$h}\n"; }

when you run it, terminal or command line or from IDE it should produce output like this:

Queen_Europe - 407347022
FaulkesSouth - 102351359
lcogt - 80797776
glenn_hughes - 15565190
FaulkesNorth - 102350867
Rogozin - 36980694
wikileaks - 16589206
Nigel_Farage - 19017675
StoppINDECT – 169043724

those are screen names and user ID pairs.

Quantum Superpositions and other Scary Stuff

Quantum::Superpositions, Quantum::Entanglement and Quantum::Usrn, as one may note, are related to quantum mechanics. Those are not packages to solve equations characteristic to quantum mechanics, but to bring quantum ideas and logic into Perl and programming in general. All three packages are interesting and deserve attention but only Quantum::Superpositions achieved wider recognition and become part of Perl 6. For that reason (limited space also plays a role) I would concentrate only on Quantum::Superpositions.

Welcome to Hilbert space and no, you do not have to have quantum computer at home to play with madness. With a little help from your friends (whom you manage to find on Internet) your good old computer will do just fine.

If you are running Ubuntu or Debian, use sudo; if not, just run terminal as root:

sudo perl -MCPAN -e shell

As mentioned, to invoke CPAN shell you must use root. If you never used it then do the update.

cpan[1]> install Bundle::CPAN

That may take some time.

cpan[2]> reload cpan

And finally, why we started all this - installation of Quantum::Superpositions:

cpan[3]> install Quantum::Superpositions

CPAN shell will ask for permission to download dependencies, and when all is done type in q or quit and hit enter to release the lock.

There is a philosophycal background to this package and the author, Damian Conway, gently mentioned it in documentation. Naturally, very few readers have the knowledge of quantum mechanics (that thing is worse than maths) and for the majority we will nicely disect the package through trusty old trial and error aproach. Quantum::Superpositions introduces two operators - any and all. Those operators are mapping a set of scalars to a single scalar variable, the author calls those scalar variables eigenstates. You see how that quantum mechanic may be dangerous and contageous, short exposure to it and you are already appending German words to English ones creating a new language which nobody understands. Without further ado we start with the code:

use Quantum::Superpositions;

use Data::Dumper;

$temp = all(1 .. 5);

print "$temp";

print "\n";

print Dumper($temp);

From the output we may conclude that to string method is overloaded and that $temp is a reference to anonymous array blessed to Quantum::Superpositions::Conj. If we do the same for any we see that $temp is a reference to anonymous array now blessed to Quantum::Superpositions::Disj. If you can't remember that reference to anonymous array, here it is:

$temp = [1, 2, 3]

Beside string conversion, the package overloads all other commonly used operators. Locating Superpositions.pm and inspecting the code may be highly beneficial here. Trivial:

$test++;

will increase all eigenvalues, irrespective of whether we are using any or all. Multiplication and addition becomes real fun:

$temp = any(1 .. 3)*any(4, 5);

print "$temp";

will produce:

any(8,4,10,12,15,5)

We will also note that any ignores duplicate values (or eigenvalues?) and all doesn't. Commutativity is satisfied for any*any or all*all but not for all*any. For example:

any(1 .. 3)*all(4, 5) = any()

but

all(4, 5)*any(1 .. 3) = all(any(8,4,12),any(10,15,5))

The type of the first operand determines the type of the result of multiplication. If we try addition then:

$temp = all(4, 5)+any(1 .. 3);

print "$temp";

print "\n";

print Dumper($temp);

produces a very interesting result:

6 7
$VAR1 = bless( [
                 bless( [
                          5,
                          6,
                          7
                        ], 'Quantum::Superpositions::Disj' ),
                 bless( [
                          6,
                          7,
                          8
                        ], 'Quantum::Superpositions::Disj' )
               ], 'Quantum::Superpositions::Conj' ); 

The result is an array containing common subset. From this (pretending that we did not read help file) we can easily draw the formula for finding common subset:

$temp = all(any(1 .. 5), any(4, 5, 6), any(3, 4, 5));

Small warning here: if one tries @$temp it will not produce expected 4 and 5 but:

any(4,1,3,2,5) any(6,4,5) any(4,3,5)

To pull subset into array properly we must use eigenstates method.

@subset = eigenstates(all(any(1 .. 5), any(4, 5, 6), any(3, 4, 5)));

Now the question do elements of array @b contain all the elements of array @a becomes:

@a = (1, 2, 3);

@b = (1 .. 5);

if(@a == eigenstates(all(any(@a), any(@b)))){

print "Yes @a is subset of @b\n";

}

Which is really a neat way of coding compared to usual nested loops.

If you download the source as well, it contains demo directory where we can see GCD and factoring and generally get some idea on how the package may be used. While the package brings ideas from quantum computing, it doesn't turn your old computer into a quantum one, so some algorithms may still execute in exponential time. If you find this interesting here is a small advertisment for Quantum::Entanglement, its demo directory contains implementation of Shor's algorithm for factoring numbers.

Pascal's triangle, Perl, and homework

Frequently, an evil professor gives to his students homework and sometimes it may be a request to write code which prints Pascal's triangle. Naturally, the student cranks up Firefox (that is synonym for web browser) and goes googling around for examples. A nice elegant solution comes up in the form of:

sub pascal { 
   my @row; 
   foreach (1 .. shift) { 
      push @row => 1; 
      $row [$_] += $row [$_ - 1] for reverse 1 .. @row - 2; 
      print "@row\n"; 
   } 
}

I picked up that one on http://www.perlmonks.org/?node=175586, naturally very few students will understand, right away, what is happening in this code. In order to make the code shorter the author did some Perl tricks, which made it difficult to read and understand the code. In the first transformation I will introduce one unnecessary variable and also make function call more C or Java like.

sub pascal {
   my $size = shift();
   my @row;
   foreach (1 .. $size) {
      push (@row, 1);
      $row [$_] += $row [$_ - 1] for reverse 1 .. @row - 2;
      print "@row\n";
   }
}

Already looks better. Inner loop still looks odd and we are not quite sure what it does and which $_ is used, one from foreach or one from for reverse. In order to see, let us add one debug loop:

sub pascal {
   my $size = shift();
   my @row;
   foreach (1 .. $size) {
      push (@row, 1);
      for (reverse 1 .. @row - 2){
          print $row [$_]," + ",$row [$_ - 1]," store in \$row[$_]\n";
       }
      $row [$_] += $row [$_ - 1] for reverse 1 .. @row - 2;
      print "@row\n";
   }
}

Output will look like this:

1 
1 1 
1 + 1 store in $row[1] 
1 2 1 
1 + 2 store in $row[2] 
2 + 1 store in $row[1] 
1 3 3 1 
1 + 3 store in $row[3] 
3 + 3 store in $row[2] 
3 + 1 store in $row[1] 
1 4 6 4 1 

Now it is obvious that outer loop just pushes ones and inner loop does all difficult work.
One small thing is left, printout doesn't look nice, it is somehow skew. To remedy that we add some pretty printing:

sub pascal {
    my $size = shift;
    my $count;
    my $margin;
    my @row;
    foreach (1 .. $size) {
       push (@row, 1);
       $row [$_] += $row [$_ - 1] for reverse 1 .. @row - 2;
       $margin = "   " x ($size - $count - 1);
       print "$margin";
       $count++;
       foreach (@row){
           printf "%6d",$_;
        }
       print "\n";
}

If we call it and pass 16 as parameter it will print this:

                                                 1
                                              1     1
                                           1    2     1
                                        1    3    3     1
                                     1    4    6    4     1
                                  1    5    10   10    5     1
                               1    6    15   20   15    6     1
                            1    7    21   35    35   21    7     1
                         1    8    28   56    70   56   28    8     1
                      1    9    36   84   126  126    84   36    9     1
                   1    10   45   120   210  252   210  120    45   10     1
                1    11   55   165   330  462   462   330  165    55  11     1
             1    12   66   220   495  792   924   792  495   220   66   12     1
          1    13   78   286   715 1287  1716  1716  1287  715   286   78   13     1
       1    14   91   364  1001  2002 3003  3432  3003  2002 1001   364   91   14     1
    1    15  105   455  1365  3003 5005  6435  6435  5005 3003  1365   455  105   15     1 

Not quite equilateral, one would say.