PredictGuru

Page Rank kind of ranking for test cricket teams

2010-10-23T04:26:00.000-07:00

The official ranking for test teams in ICC is quite complicated http://en.wikipedia.org/wiki/ICC_Test_Championship#Test_championship_calculations . It seems a bit confusing with all the arbitrary point calculations. (40 points, 90 points etc). And although it tries to give more points to a weaker team that beats a stronger team, the effect is not uniform. The scoring scheme can be made more elegant by a simple page rank kind of algorithm.

The idea is, if India beats Australia in a series 2-1, Indias score would be (score-of-australia)*2/3 and australias score would be (score-of-india)*1/3. At the end of a given time period, lets say india beat aus 2-0, SA 3-1 and lost to bangladesh 1-0. Indias score would be:
india-score = k(india-score + 2/2*aus_score + 3/4*sa_score +0*bangladesh_score)
similarly for bangladesh, bang-score = k(bang-score + 1*india-score + other series)

where k is a constant.

This leads to a linear set of equations which can be solved by using the eigen-value decomposition. Each eigen vector is a solution, which in this case becomes a possible team score. We can take the vector that corresponds to the maximum eigen value.

The data for all the test series played until now is available from cricinfo at http://stats.cricinfo.com/ci/content/records/335431.html. I used a slightly modified version of the above scoring scheme, The score I used was (number of wins)/(total-matches + 1) . Just to score more for a comprehensive 3-0 win (3/(3+1) = 0.75) v/s a 1-0 win (0.5) The scores for the last 3 years, that is 2007 and beyond are

"India" 0.5994906420666286
"Australia" 0.48114889162515917

"South Africa" 0.46906326930865216
"England" 0.27645475105538375,
"Sri Lanka" 0.29340243047144315
"West Indies" 0.11112461214397185
"Pakistan" 0.09815656282323282
"New Zealand" 0.05997455303303149
"Bangladesh" 0.03152476327152151,

If scored this way, Australia still seem to have a higher ranking as compared to the ICC ranking where they are number currently number 5. I also tried plotting the scores for the last 130 years of test cricket. Here are the results.

As expected, India is rising now whereas Australia is falling, WI dominated a decade after 75. Surprising thing is England started both the world wars winning against Australia and immediately after the wars, Australia was on top.

Here is the clojure code for doing all this, jblas is needed for eigen decomposition and jfreechart for plotting.

Data says no Warming of Bangalore in the last 37 years.

2010-04-26T09:22:00.000-07:00

It is a constant complaint that you hear from people these days. "Bangalore has grown warmer over the years." or this year is the warmest over the years. And the blame inevitably is on global warming. But how much of this is real? What is the magnitude of the temperature increase? Was this year warmer than the one before? . I downloaded data from 1973 to 2009 to see what happened and I have posted it here. The data can be downloaded from . Looks like the data is fairly accurate, but I have not tested it against any other sources.

The graph on the left is the temperature data for the last 37 years, averaged over 3 months (x axis is the number of days since 1973 Mar 1). You can see the yearly rise and fall of the temperature over the year. It may not entirely be clear, but the graph shows that there is no huge (> 1 degree) increase in the temperature of Bangalore. Also, as the media tend to assume, Bangalore does not seem to have huge variance in the temperature over the years.

We can make it more clear by taking a longer term average of the temperature of Bangalore.

When averaged over 5000 days, looks like the average temperature has only slightly increased from 23.57C to 23.8C. Hardly noticeable by humans.

In effect the overall increase in Bangalore temperature is very small and may not be a proof of global warming as we keep hearing.

Mobile meter for auto rickshaws

2010-03-13T21:27:00.000-08:00

Would be nice to have an app for location aware mobile phones that you can use while traveling in Autos / Taxis in Bengalooru or elsewhere. The app should track the distance and waiting times and calculate the exact amount for the trip. With mobile phones connected to the net, the per kilometer fares can be downloaded from a common server, which maintains the rates for lots of places. May be there is an app already out there.. but too lazy to check.

Flying bridges for bangalore traffic?

2008-09-10T08:45:00.000-07:00

Can bridges suspended by helium / hydrogen / hot air filled balloons provide an easy alternative to flyovers in Bangalore?

Some of the advantages:
1. No need of any space on the ground for building pillars. Only need pillars where the vehicles can climb onto the bridge.

2. Bridges can be moved based on the traffic. Move all the balloons (air ships) to a new position.

Challenges :

1. Amount of Helium needed: At 1000ltrs to lift a kg, and having a limit of 8 tonnes weight for each balloon, they need to be filled by 8 million liters of helium.. Which would make each of the balloons to be about 100 mtr by 10 mtr dia cylinders.
2. Handling monsoon winds.

Product review tracking from motiflabs

2006-11-25T07:34:00.001-08:00

A prototype of the Product Buzz tool is online.

Automatic news ranking

2006-02-20T06:47:00.000-08:00

Ranking is one of the most important problems in machine learning these days. All the search engines including google uses ranking for sorting the web pages according to the relevance to the entered query. It has been shown that using machine learning for ranking provides satisfactory results. (No idea if google uses machine learning also to get the page rank.) The idea is simple, a user manually ranks a set of documents (movies, webpages anything). Using this as reference, we train a function, which, given a new document, identifies the correct rank of the document in a set of documents. (Based on movie name, actors, director, contents if it is webpage...).
Ranking can also be used in many other areas. Here is a list

Email Ranking: Lets assume you are a person with lots of contacts. Or you are very famous and get 100s of legitimate emails everyday. Some mails require your immediate attention and some mails are not so important Machine learning can be used here to order the mails for you based on your past behaviour. So all the important mails will be at the top and the not so important mails will be at the bottom. If you do not have a spam filter all spams will naturally fall at the bottom with this approach. An implementation would be a plugin for thunderbird which puts any new email in its rightful place.
News Ranking: Consider news sites like slashdot and digg. The problem with slashdot is that a editor has to read through thousands of postings, find good stories edit it and post it. And this takes a few hours, the problem is similar in case of digg. It takes a few hours for a story to get enough diggs to push it to the front page. A story may be stale by the time it makes it to front pages. Now consider an algorithm that learns from the previous stories that have made it to the front page and automatically decides if a new story is front page material. If the algorithm is good, we get a near real time appearance on the front page. A simple implementation would be to get RSS feeds of the story from digg, rank it and if the rank is good enough post it on your site.
Blog ranking on blogspot, Photo ranking on flickr if you think of something leave a comment.

Of Flying Cars

2006-02-16T08:47:00.000-08:00

There have been so many attempts to build small cars that can fly from short runways. The latest is an attempt by MIT guys which may or may not be successful. The problem may be our limited imagination. No human alive can conjure up a design in his/her head and predict if the machine is going to fly. To get around the problem I suggest we use a variant of genetic algorithm and CFD analysis tool to obtain a design. Here is an algorithm.

Convert an airplane outer body shape into a vector. This can be done by breaking the airplane into millions of small volumes or however you want. Ideally any vector imaginable should define a structure.
Create a base set (population) of vectors from available plane designs and probably birds.
Write code to convert the vectors back into a shape that can be simulated in a CFD (Computational Fluid Dynamics) solver like CFX or OpenFoam.
Define a survivability criterion for each design. This can be based on criteria like lift provided by the design, Volume, Surface area, stability. These data can be obtained by analysis of the output of a CFD solver.
Now in a loop, until a good design is found.

Select two designs from the population. Decide a crossover point break the vectors of the 2 designs at the crossover point and rejoin to the half vector from the other design,(crossover)
Perform mutations with some probability.
Determine if both the new vectors are valid design. (Plane with holes in wings.. Whatever u can think of that is obvious it not going to work.)
Run CFD analysis on the designs and determine the survivability.
If the designs are good (survivability > cutoff) add designs to the population and repeat loop. Else discard design and repeat loop.

Now a CFD analysis on a simple car takes a few days to complete. So to perform CFD on the millions of possible designs will take ages. So we should think of ideas from machine learning to minimize repeated effort, Also large scale distributed processing like SETI can be probably thought of. Also I coding for any of the above steps will be a project in itself. May be the idea itself is not practical. But its fun to think about it.

Natural Language Processing

2006-01-15T06:13:00.000-08:00

Natural Language Processing has turned out to be a very difficult challenge. One of the reasons is that the way language has evolved. An advantage the human brain has is the availability of thousands of classifiers (read as neurons) making decisions. We will understand a sentence written in natural language probably only when the output of these agrees. Some of the neurons possibly are trying to see if a sentence makes sense in the currently understood form (Context information). However these facilities are currently unavailable to a computer.
May be we can use the procedures of statistical machine learning here. Here is the details of a system I wish to suggest.

As everyone understands, English language (all languages apart from Sanskrit possibly) has some disadvantages like, a word can mean different things in different cases (for example table in table of contents and a table as in a dining table). Multiple words can be used to represent the same meaning (synonyms). A set of words representing a concept (rear view mirror). Etc. One way to simplify the mess is to use an intermediate language. Some of the properties of this intermediate language should be:

One word, one meaning.
One meaning one word. (All synonyms are represented by one word).
All set of words that mean something are condensed into one word. A word in the intermediate language is a concept and not necessarily an English word. But it makes sense to make it as close to English as possible to reduce the effort. After all we will be using English most of the times)

Such an intermediate language can now be converted into any form of representation. One such form is an "association graph of concepts". Here every noun (or a noun with adjectives) forms a node and verbs form the edges. Example "Ram is a good cat". Translates to a node for Ram, A node for "good cat" and a node for cat. There is an "is" edge from "Ram" to "Good Cat" and another "is" relation from "good cat" to "cat". (Guess this can be added by default). There can be a base graph already built after reading millions of documents say, to get the base knowledge.
Once such a graph is built, the same graph can be used for word disambiguation as it would have a strong set of links for related concepts.
The graph can form a common base for all human languages with "intermediate to particular language" translator built on top of it. So machine translation can be made to work in this way.
Once the computer starts reading a new document, the concepts of the document will be close to a particular part of the graph. These ideas can be possibly used for text summarization and sentiment analysis.

Anyway this is still in the idea stage.. If we ever make any progress on such a system, I will post it here.

Statistical Machine Learning

2005-09-23T20:57:00.000-07:00

We spent the whole of last month discussing statistical bounds for function classes. The equations kept getting horrible, makes me wonder if we are trying to fix the wrong problem. As of now SVMs can do much better generalization than the human brain. (After all we as humans tend to have superstitions, and we can look at a superstition as a failure in generalizing). So where else can we improve.
What would be nice to have is a system that tries to explain the features logically. Maybe that is where inductive programming comes in, but I don't have much knowledge in that area.

CAPTCHA

2005-08-28T08:09:00.000-07:00

Anyone who has tried to create an account in any of the portals these days would have seen CAPTCHAs these are the wiggly words that google and yahoo use to keep out the bots. Generally CAPTCHA are anything that does that trick, not just a wiggly piece of text. http://www.captcha.net/
Would be an interesting thing to try and break these stuff.

Restart

2005-08-28T08:04:00.000-07:00

One of the ideas currently doing rounds is research blogs. http://hunch.net/
is a good one for machine learning. They are also planning to make it a place where people put up links to papers they like and which they think are worthy compared to the other millions that gets published just for the record.

Clustered File System

2005-05-19T05:50:00.000-07:00

The suceess of desktop search tools will probably signal an end to the way data is organized on the disk. After installing google desktop search i found accessing my files through desktop search was way easier than windows explorer. Explorer like interface is probably on its way out, i guess we will see file systems in the future which automatically store the data in a more logical way based on the contents, and freeing up the users from managing their data in seperate folders.
If i were to implement such a file system (in linux), i would provide a single folder where the user can copy all thier files. Whenever a new file is copied, it is indexed and all related files are probably clustered, so for example a user request for a listing of files (ls) containg a word (say simulated annealing) will be witten as "ls simulated annealing" and the fs should provide all files with the seachword in them. It can also list all files with some related words too (ofcourse with a lower priority). Would also be nice if the command "acroread simulated annealing" will open all pdf files contaning the word simulated annealing in them.

p8

2005-05-09T21:41:00.000-07:00

Some students from MIT recently generated a paper using context free grammar which lead to some controversy, however the idea of machines generating a document based on some rules is pretty old. One interesting idea is http://www.eblong.com/zarf/markov/ where a makov chain is learnt based on the text from a big collection and the text is generated probabilistically. Some more details (and some more fun) can be found here: http://www.cs.princeton.edu/courses/archive/spr05/cos126/assignments/markov.html
It would be interesting to train the Markov chains over sentences too (this requires that each sentence be given a state based on the set of words in the sentence) also we can try out filtering a sentence based on a grammar and look at the result. The first part can be implemented in I guess half an hour. It will probably take a bit more thinking to get the second and third part working. Guess I can steal the implementation of the cfg from somewhere.

p7

2005-05-06T22:39:00.000-07:00

I was asked to think about some of the applications of handwriting recognition on a simputer, here are some of the areas where i as a student would love to use it.

1. It would be nice if i can take down all my notes on a simputer and transfer it to my computer every day, i would also prefer to read my notes on a simputer (Instead of a computer or my handwriting, it is always better to let silicon handle the deciphering of my handwriting rather than me worry about deciphering it at exam time). Also cut and paste features would be valuable to have when writing notes.

2. Also once we have recognition features, it needs to be studied if we can somehow use it to compress the handwritten data. i.e compression based on strokes.

3. If the simputer is targetted at say. a shopkeeper, things he would be interested in would be very different from me i guess. He would probably be interested in a way to track his customers who have bought things from him. It frequently happens that the customers generally pay the shopkeeper monthly instead of at the time of purchase. Tracking the purchases and calculating the exact amount at the payment time would be made much easier with a simputer, wherein th e shopkeeper can just notedown the items and the cost (as if on a piece of paper) and the simputer will find the right customer from the database and debit the amount against the customer.

4. Signature verification in banks. In most of the banks, money is still withdrawn through self drawn cheques, simputers can be used there..Also simputer when connected to the internet can be used for secure transfer of money with the help of signature capture and transmission.

5. simputers can also be used for quick sharing of designs over the internet. It can be used like the back of the envolope between people on phone in different parts of the world

p6

2005-05-03T08:05:00.000-07:00

Determining right foods for the right person.
Every person has a different need for food and getting the balance right is difficult. Can pattern recognition stuff be used to determine what is right considering that persons BP, Sugar, Calories reqquired, Carbohyderates.. Now the trick would be to not just provide a chart providing the amount of each permissable food. but to take in the details about the food that a person has eaten till now and suggest a suitable recipie for the next food item. These softwares are already available but are not effective and have an inbuilt set of food items. But a better system would crawl the net to pick up new recipies and calculate the amounts of each nutrient in the food and suggest a bunch of them appropriately.
The difficulty in implementing this is the enormous need for domain knowledge .

p5

2005-05-02T20:28:00.000-07:00

15 billion dollars is up for grabs. That is the amount that banks and other financial institutions are supposed to spend in this year on systems that can track dirty money, frauds and payments to terrorist organizations. These systems are mostly built on the concepts of anomaly detection. Anamoly detection is a concept that was probably first introduced to figure out network intrusion detection to seperate out newer attacks. Traditional systems were rule based, but they can be easily defeated by any hacker. So the solution was to model what is normal (i.e normal usage) probabilistically. And whenever a newer entry in the logs that is not similar to any existing entry appears, classify it as an anomaly for human experts to review. The similarity measures and the modelling techniques for such things are already available in the pattern recognition world.
Now this idea is also applied to all banking transactions, as and when a new transaction is seen (such as suddenly 110 crore Rs transferred to the account of a person who doesnt even have a pan number) or a person getting a salary of 2lack pa suddenly getting a cheque for 30 lakhs) an anamoly can be signalled and the tax (possibly corrupt) officials can be allowed to handle it in a suitable way.
Guess would be fun to implement it and monitor it especially if u have contacts with the abpve mentioned tax officials.

p4

2005-05-02T08:32:00.000-07:00

NadiShasthra is a form of astrology where u are supposed to give the either the hand impresssion or the thumb impression (i am not sure which some even take the kundali) . The astrologer matches it with a huge database of palm leaves and comes up with a matching palm leaves and reads out, and somehow most of the things he says about ur past is correct (although most of the things he says about your future doesnt happen. lets assume it does. it is more fun that way). Ok so hoe does he manage to generate such information about you in such a small time.
Before that where did those palm leaves come from.. One of the explaination is that those were assignment submissions from disciples of a particular guru who was an expert in these fields.. Apparantly the students were asked to predict the future births / incarnations of some of the people in their time. (And looks like they did a pretty good job).
So how does pattern matching come in here:
One simple idea (courtesy Bhargav) is to get the data base from the pundit and put it online.. People submit their hand prints in jpg format, we use pattern matching techniques to get the appropriate palm leaf and spell out their future. (All for a few ofcourse).

The other more intersting idea is that if there is a real correlation between the human hand and the human destiny, it should be more logical to use a computer and mathematics to study it rather than just the human brain. All one would have to prove or disprove astri=ology is get the hand prints of a million people, cluster it based on their hand prints and see if some criterias like the area of work, ...(basically whatever an astrologer tries to predict) falls in the same clusters more often than random. A simple chi square test would be enough.

Also if the hand print does not throw up anything we can look at the thumb print, stars, planets whatever.
But i guess doing so would also lead to massive job losses in India So in the larger interest of the country i guess i will stay away from this area.

p3

2005-04-28T21:35:00.000-07:00

This post is about self driving cars. If u follow the news u would know that there is already an effort to create auto cars or cars that drive themselves. How is statistical pattern recognition applicable here?
Regression can possibly help us here. Regression is a tool which learns a function whose output is continuous (Models in previous posts were discrete). The idea is simple. Let a human drive a car which also has a on board camera and a device to capture the drivers response to various events on the camera. Over time with a good learner and a good driver, it is possible that the learning system may learn how to drive. There can be a preprocessing stage in front of the camera to seperate out obstacles and other moving bodies. Also it may be necessary to find out the speed of a moving object and try to project it in time and provide the projection data to the learner, this may be necessary as it is impossible to encounter all the auto drivers for a trainer in the training period. (This is based on the assumption that autos are quantum vehicles whose absolute position given its current position can only be determined probabilistically).
Since a rigorous proof that this system will work cannot be given it is better to prove it by building such a system. The only problem is the funding.

p2

2005-04-22T22:55:00.000-07:00

some of the other stolen ideas include

stock market prediction: People are already using machine learning techniques to predict the movement of the stock market. It is said that some of the banks make a lot of money by predicting the fluctuations in the currencies of different countries. I guess there is a lot of money that can be made by creating the write model by predicting the movemets of a financial institution.
an other idea that one of my friends is working on is to identify related tunes from different songs, or given a tune find the most appropriate song from a database.

p1

2005-04-21T04:12:00.000-07:00

Welcome to the first predict guru idea, this blog is about our experiments to get to know the ideas behind machine learning. Since the time i started learning this fascinating subject, (more scary than fascinating for equation haters like me) i felt like we can apply this technique for many more things other than Fishers IRIS data or the US postal departments ocr data. The following are the things i am planning to try out.

Cricket player score prediction: Lots of money is involved in cricket betting, imagine the amount of money u can make if u know how much Tendulkar or Sehwag is going to score in a series. I guess given a right model this can be easily predicted, I mean if not the exact number of runs, a range in which each player will score in a series of say 5 matches. Look out for ideas for the model later. This idea can be extended to other games as well.
This idea is about predicting people. Companies spend a lot of money trying to get the right person, right now interviewing and testing is more of an art which is left to the existing employees who may not be readily available. Why not use machine learning techniques here. One way to do it is to provide a questionnaire to the interviewee as well as some of the ideal employees of an organization. Based on the distance between the expected and the interviewees response, a HR person can quickly discover if a candidate can move forward to the next level.
Software code review is another area where people use subjective assesments. Again code reviewed by experts can be compared with the new code to evaluate it.

Although some of these look like jobs for expert systems, the idea here is to learn the rules statistically.