Notes on codes, projects and everything
2016 January
-
Further Hack on the Multi-Dimensional Approximate Neighbour Search
Traversing a tree structure often involves writing a recursive function. However, Python isn’t the best language for this purpose. Therefore I started flattening the tree into a key-value dictonary structure. Logically it is still a tree, but it is physically stored as a dictionary. Therefore it is now easier to write a simple loop to traverse it.
-
Approximate Neighbour Search in Multiple Dimensions
In the previous post, I re-implemented Annoy in 2D with some linear algebra maths. Then I spent some time going through some tutorial on vectors, and expanded the script to handle data in 3D and more. So instead of finding gradient, the perpendicular line in the middle of two points, I construct a plane, and find the distance between it and points to construct the tree.
-
Re-implementing Approximate Nearest Neighbour Search
Recently I switched my search code to Annoy because the input dataset is huge (7.5mil records with 20k dictionary count). It wasn’t without issues though, however I would probably talk about it next time. In order to figure out what each parameters meant, I spent some time watching through the talk given by the author @fulhack.
-
Information Retrieving with ….. a lot of libraries
Implementing a Information Retrieval system is a fun thing to do. However, doing it efficiently is not (at least to me). So my first few attempts didn’t really end well (mostly uses just Go/golang with some bash tricks here and there, with or without a database). Then I jumped back to Python, which I am more familiar with and was very surprised with all the options available. So I started with Pandas and Scikit-learn combo.
Random Posts
-
Release of my app on bitbucket
Should have done this earlier, I was just being lazy to go through all the steps to publish it properly. So here it is, the full source is published to bitbucket. Feel free to fork the project if you are interested. I have not attach a licence to it but it will most probably be BSD licence. I have also uploaded the latest 0.0.2 release to bitbucket and would update the download link posted previously soon.
-
Regression with 2-means clustering Annoy (non-scientific)
So apparently Annoy is now splitting points by using the centroids of 2 means clustering. It is claimed that it provides better results for ANN search, however, how does this impact regression? Purely out of curiosity, I plugged a new point splitting function and generated a new set of points.
(more…) -
Real Life Enterprise Class Website Development
Recently I am involved in developing some small modules for a enterprise class website using CodeIgniter (CI). There was no restriction given on which framework should I use for the development and I chose CI as I learned a bit on it (when I was considering whether to shift my personal development project). Of course there are other reasons why I chose to learn CI, for example the superior documentation and screencasts available.
-
Quick Hack: Terminal Appearing from Right
It is useful to have the terminal around whenever I code. However, while real screen estate is finite, having a terminal further limiting the amount of information that can be displayed at the same time. The problem with the terminal is that I don’t really need it all the time, so I usually find it buried under a group of windows.
-
Maintaining State with YUI Event
When one start writting Javascript in patterns like the module pattern, then sooner or later he would want to maintain the state when an event handler is called. The reason I am still using YUI to handle my event handling code is because I like how state can be maintained.
