This is going to be a two part introduction to the machine learning technique/framework called Random Forests. Random forests the have gained much popularity in recent years owing to it’s ability to handle extremely large training data efficiently as well as provide $O(n)$ test time efficiency.

Before we can talk about random forests we must understand decision trees, and decision tree learning. I feel that it’s much easier to learn through example, so I will go through a simple one from the book: Machine Learning by Tom Mitchell.

The goal is to be able to predict whether or not to play tennis on a Saturday morning given the following attributes $\{weather outlook, temperature, humidity, wind\}$ . In this example we will be performing supervised learning, which is where we are given labelled training data. Going back to our example, the labelled training would look something like this:

Note:The labels are the playTennis column (yes,no).

From this data we wish to build a decision tree which, such that for an input $x=\{weather outlook, temperature, humidity, wind\}$ we can follow a path to a leaf node which contains the answer ${yes, no}$ .

In order for us to reach the correct leaf node, at each non-leaf node (called split nodes), a function $g(x)$ is evaluated on the input and based on the result we branch either right or left until we reach a leaf node which will contain the result (to play tennis or not).

The question now is, what qualities should the functions $g(x)$ have and how do we generate a good function, $g(x)$ , using the training data?

Well, Ideally we would just have one split node which would contain a test function that managed to split the training data into two sets, $S_i and S_j$ , such that all members $x_i \in S_i$ have attribute $playTennis=No$ and $x_j \in S_j$ have attribute $playTennis=Yes$ .

This situation may arise in very simple problems, or training data that doesn’t fully represent the problem. For example, look at the graph below, it shows that the data points can easily be split into two classes by the simple function $f(x) = x>5$ . However, in real life situations data is seldom separated this easily, in fact real data is rarely linearly separable and can possibly only be split by a high dimensional hyperplane represented by a very complex function. It is, in fact, the combination of the simple split node tests, that together form a highly non-linear separation of the data in to the desired classes.

An example of linearly separable data

Even though it is unlikely that just one split function will be able to partition the data perfectly, I hope it has become clear that we would like a split function that maximizes this separation. In fact, we wish to maximize the information gain at each split node. The information gain of the function at a split node is a measure of much the function decreases the uncertainty of the class label in the data. For example, given data $\{a,a,a,a,b,b,b\}$ , a split function that splits the data into $\{b, b,b\} \& \{a,a,a,a\}$ would have a higher information gain than one that splits it into $\{a,a,b,b\} \& \{a,a,b\}$ as clearly the uncertainty of class label is lower in the sets produced by the first split than the second.

More formally, information gain is defined as the change in entropy ( $H$ ) from a prior state (the training data at the node, $S_i$ ) to a new state (the subsets generated $S_{i}$ ). This is denoted by the following formula (in this case we are using the Shannon Entropy as our measure):

note: $C$ is the set of all classes, in our example this is $\{yes,no\}$

So, at each node we wish to choose the function that maximizes the information gain. Now, returning to decision tree learning, the following algorithm, called C4.5, is used to grow our tree:

1. If we should terminate, create leaf node. Else
2. For each attribute a
3. Find the normalized information gain from splitting on a
4. Let a_best be the attribute with the highest information gain
5. Create a decision node that splits on a_best
6. Recurse on the subsets obtained by splitting on a_best, and add those nodes as children of the current node

Returning to our example, for our terminating criterion we will stop if all data samples at the node have the same class, which is all yes or all no in our case. So given the training data, we start at the root of the tree and check, for each attribute, the information gain. For example, given the 3 data examples at the top, using the attribute $wind$ , we calculate the information gain as follows:

First, we calculate the entropy, $H(S)$ , of the whole set $S = \{d1,d2,d3\}$ as follows:

We now recurse on the two subsets produced ( $S_1$ & $S_2$ as above), which we can see both contain data from just one class which matches our termination criterion. So, we have now how two leaf nodes, and have built our tree.

Given our tree, given a new input $x$ , we check the $wind$ attribute and if it is $weak$ we answer $no$ else if it’s $strong$ we answer $yes$ . Obviously this was highly simplified by the fact that we only used 3 training samples, which is hardly enough to capture the problem accurately. Nevertheless, I hope this example has made the algorithm clear.

In the second part of this post I will discuss the down side of using this approach and why we use random trees. Additionally (hopefully) I will have a walk through, with code, of a real application using random forests. The application will probably be some form of image classification, such as classifying images as either images of cats or dogs, or if I have the time perhaps I can try and tackle a where’s Wally solver.

Roadside weeds, literally.

However this is confined to Old Manali, and with just a short walk to new Manali you can feel like you’re back in India. Walk a little bit further and you will hit the town of Vashisht, which is where many of the Indian tourists tend to stay. Vashisht is also the starting points to many treks into the Himalayas and therefore has many wonderful viewpoints.

Me at the Vashisht Waterfall holding a friends drum

As with any tourist town in India, you will find the snake charmers (here’s a video if you’ve never seen one). But what I particularly enjoyed was a street show by two tourists which attracted all of the locals:

Bhrigu Lake is a holy lake where it is said that Maharishi Bhrigu, one of the seven sages created by the God Brhama, meditated while he wrote the Bhrigu Samhita. It is located at an altitude of 4250m (~14,000 ft) and the trip from Manali (at 2000m) is done over 2 nights and 3 days. It was probably the most physically challenging thing I have ever done in my life.

Day one started at 9am and we ascended up to 3800m in about 4 hours, during which I realised I am not as fit as I thought I was and that altitude sickness is no myth! During the ascent we stumbled across a herd of mountain goats and a sheppard, who was kind enough to let me hold a baby goat .

Me and a baby goat

Once we reached camp we were greeted with hot chai and food. The views over the Himalayas at this altitude are amazing and because we were camping make for possible the best toilet in the world.

The best toilet in the world

The next day we made the chilling ascent up to the lake. This involved snow trekking up the edge of the mountain which meant that the guide had to use his ice axe to carve out steps for us. It gave me a real adrenaline rush as if you slipped you could very well slide half way down the mountain. Due to this, I couldn’t find an opportune moment to take a picture of this. After about 4 hours of this we finally reached the lake and I must say it was worth it as it is one of the most beautiful things I’ve every seen.

Bhrigu Lake

Waggah is about 30km from central Amritsar, so I got the hotel to hire me a driver for the afternoon. The drive was fun, especially because the driver didn’t speak a word of English, luckily my Hindi was sufficient enough to have a reasonable conversation.

When I arrived I was shocked to see how many local people had attended at 4:30 on a Friday. Apparently this is a common occurrence and the viewing gallery gets filled every day.

Once I had cleared security (no bags allowed), I had no idea where to go. A few meters ahead I saw a few American chaps go through a gated area which was being guarded by a soldier. I decided to approach the gate and the soldier (not so) kindly told me that I cannot go through so I joined the crowd of Indian people who appeared to be standing around waiting for something.

Suddenly, another gate opened and the crowd swarmed towards the entrance carrying me with them like a strong tide. During this complete lack of order I saw once man get trampled and another loose his sandals. Luckily I am slightly bigger than the average in India and was not easily run over. Once I was through the gate it wasn’t so bad, people took their seats and waited for the show to start. This experience made me appreciate one English thing in particular, the orderly queue.

The ceremony consisted of patriotic chants from both sides of the border, some dancing from the crowd and some fancy marching by the soldiers. Granted I could only partially see into Pakistan, but from what I could see there were many more people on the Indian side of the border, and plenty of noise was made.

India’s Viewing Gallery

Pakistan’s Viewing Gallery

The crowd getting involved

The next day I met another (American) tourist who told me that if I had shown my passport to the Soldier I could have gone through the mysterious gate, behind which exists an orderly queue for foreigners. So if any Indian, or Indian looking, tourists plan on visiting Waggah, I would recommend speaking no Hindi and just flash your passport for some VIP treatment.

I’ve decided to go traveling in India and am actually writing this post from a hotel in Manali overlooking the Himalayan mountains.

The view from my hotel balcony

The first stop was Amritsar, Punjab. Here I visited the Golden Temple, which is one of Sikhism’s most holy places as it houses the Adi Granth, which is the Sikh holy book. It is called Golden as the temple itself is covered in gold, which was done for protection rather than beauty both of which have been achieved.

The Golden Temple in the day

The Golden Temple at night

I visited in the day and at night as the views are very different. At both times there were many pilgrims taking dips in the holy lake, however at night the temperature is more bearable and I queued to go inside the temple (no photography allowed).

A pilgrim taking a dip in the holy lake

I’d like to say a big thank you to everyone who donated. Together we managed to raise £360 fine English Pounds for the prostate cancer charity!

Now, as promised, the moment you all paid for, the Motos (Photos). Enjoy!

List<String> names = new LinkedList<String>(){{
    add("Bob");
    add("Pam");
    add("Aly");
}};

Which is a nicer, slightly more concise way of writing

List<String> names = new LinkedList<String>();
names.add("Bob");
names.add("Pam");
names.add("Aly");

So, what exactly does double brace initialization do under the covers? Well, the first set of braces creates an anonymous class that extends the class you are trying to instantiate. The second set creates an initialization block for this anonymous class. This means that the initialization block can refer to any public/protected fields and methods in the base type. So, this is not just a bit of syntactic sugar to populate collections on initialization, in fact, this technique can be used with any object.

There are however a few things to note before you use this technique:

If your object has a complicated equals method that depends on the type, class name or private fields then you probably want to avoid using double brace initialization. This is because the new type that is created will have a different equals method and may fail on .equals() checks.

The second gotcha is slightly more subtle and involves serialization. When classes implement the Serializable interface (as do most of the Collections in the Collections API) they have to provide a serialVersionUID and if they don’t the JVM will create one for them. The serialVersionUID is used as a version control mechanism for Java serialization. This is an important topic, and one for another blog post, but it is important to note that changing it can cause serialization on the client side to fail.

Now, when you use double brace initialization without explicitly adding the same serialVersionUID as the base type (which would defeat point of the nice shorthand syntax) the JVM will generate a serialVersionUID for the new anonymous type. The generated serialVersionUID can be based on the fields as well as the class name which is different for the anonymous type. This will lead to serialization conflicts, so if you are serializing you object, you will want to avoid this shorthand syntax.

—UPDATE-14-11-2011————————————–

A third gotcha, as pointed out by a friend of mine, Viral Shah, is that when the anonymous class is created, it implicitly gets a reference to the surrounding class which can be accessed with the Person.this syntax as shown below:

public class Person{
  public String name = "Aly";
  List<String> favouriteNames = new LinkedList(){{
    add(Person.this.name);
    add("Viral");
  }};
}

Whilst this implicit reference looks harmless, it can cause two subtle but relevant issues.

The first is that as long as a reference to the favouriteNames list is live, in any object, a reference to the Person object will be kept alive and thus consume heap space.

The second issue relates to (surprise, surprise) serialization. As the anonymous class has an implicit reference to the outer class, when the anonymous object is serialized it will (try to) serialize the Person object as well. Now, using the example above, Person does not implement the Serializable interface. So trying to serialize it, directly or via serializing the favouriteNames list, will throw a java.io.NotSerializableException exception. Additionally, this can not be avoided by using the transient keyword, as the reference is implicit and therefore we cannot mark it as transient.

It itches like hell, but I can stroke it whilst reading and look more intelligent than I really am. As the saying goes, “pictures or it didn’t happen”, so here it is:

If you found that funny and/or you want to help a good cause please donate to the cause here: http://mobro.co/AlyTejani

Why just final methods? Well, as stated in the article:

A final method cannot be overridden in a subclass.

Ok, but why shouldn’t we override these initialization methods? After all I often see code like this:

public class Super{
 
    public Super(){
        //some code here
        init();
    }
 
    public void init(){
        //initialzation code here
    }
}
 
public class Sub extends Super{
 
    public Sub(){
        super();
    }
 
    @Override
    public void init(){
        //different initialization code here
    }
}

They then go on to say:

“Calling non-final methods during instance initialization can cause problems. Joshua Bloch describes this in more detail in Effective Java”

Now, I assume that like me, when people are reading this page, usually at work, they generally don’t have their copy of Effective Java with them. So, I’m going to try my best to explain this so that its available via a simple google search.

Let’s look at a simple example. We have a Duck and at design time we decide to make the assumption that when Ducks speak they all say “quack”. At some point in the future we decide to add a SqueakyDuck; which when speaking says “squeak”. The code for this is below:

Disclaimer: Now, I know this is a bad example and there are better ways to achieve this behaviour, such as the Strategy Pattern, but that is a post for another day. Nevertheless, I feel this general pattern occurs in practice quite a lot.

public class Duck{
 
    String sound = "quack";
    protected String speech;
    public Duck(){
        //some code here
        speech = initSpeech();
    }
 
    protected void initSpeech(){
        speech = "Hi, I'm a Duck and I say " + sound;
    }
 
    public void speak(){
        System.out.println(speech);
    }
 
    public String getSound(){
        return sound;
    }
}
 
public class SqueakyDuck extends Duck{
 
    String squeakSound = "squeak";
 
    public SqueakyDuck(){
        super();
        //some code here
    }
 
    @Override
    protected void initSpeech(){
        speech = "Hi, I'm a Squeaky Duck, and I say " + squeakSound;
    }

    @Override
    protected void getSound(){
        return squeakSound;
    }
}

The problem arises when we run the following code:

public static void main(String[] args){
    Duck squeaky = new SqueakyDuck();
    squeaky.speak();
    System.out.println(squeaky.getSound());
}

and we get the following output:

Hi, I’m a Squeaky Duck, and I say null
squeak

Chapter 12.5 “Creation of New Class Instances” of the Java Language Specification helps to clarify why this occurs (relevant bits highlighted below):

Just before a reference to the newly created object is returned as the result, the indicated constructor is processed to initialize the new object using the following procedure:

1. Assign the arguments for the constructor to newly created parameter variables for this constructor invocation.
2. If this constructor begins with an explicit constructor invocation of another constructor in the same class (using this), then evaluate the arguments and process that constructor invocation recursively using these same five steps. If that constructor invocation completes abruptly, then this procedure completes abruptly for the same reason; otherwise, continue with step 5.
3. This constructor does not begin with an explicit constructor invocation of another constructor in the same class (using this). If this constructor is for a class other than Object, then this constructor will begin with an explicit or implicit invocation of a superclass constructor (using super). Evaluate the arguments and process that superclass constructor invocation recursively using these same five steps. If that constructor invocation completes abruptly, then this procedure completes abruptly for the same reason. Otherwise, continue with step 4.
4. Execute the instance initializers and instance variable initializers for this class, assigning the values of instance variable initializers to the corresponding instance variables, in the left-to-right order in which they appear textually in the source code for the class. If execution of any of these initializers results in an exception, then no further initializers are processed and this procedure completes abruptly with that same exception. Otherwise, continue with step 5. (In some early implementations, the compiler incorrectly omitted the code to initialize a field if the field initializer expression was a constant expression whose value was equal to the default initialization value for its type.)
5. Execute the rest of the body of this constructor. If that execution completes abruptly, then this procedure completes abruptly for the same reason. Otherwise, this procedure completes normally.

This means that on execution of the SqueakyDuck constructor, it in turn calls super(), which in turn calls the overridden initSpeech() method, inside the SqueakyDuck class, before the SqueakyDuck instance members have been fully initialized. This means that the when initSpeech() is run the squeakySound field has the default value, which is null for all objects. However, once the SqueakyDuck object is fully initialized the squeakySound field is set to “squeak” which is why the call to getSound() returns “squeak”.