Notes on codes, projects and everything
Information Retrieval
-
Further Hack on the Multi-Dimensional Approximate Neighbour Search
Traversing a tree structure often involves writing a recursive function. However, Python isn’t the best language for this purpose. Therefore I started flattening the tree into a key-value dictonary structure. Logically it is still a tree, but it is physically stored as a dictionary. Therefore it is now easier to write a simple loop to traverse it.
-
Approximate Neighbour Search in Multiple Dimensions
In the previous post, I re-implemented Annoy in 2D with some linear algebra maths. Then I spent some time going through some tutorial on vectors, and expanded the script to handle data in 3D and more. So instead of finding gradient, the perpendicular line in the middle of two points, I construct a plane, and find the distance between it and points to construct the tree.
-
Re-implementing Approximate Nearest Neighbour Search
Recently I switched my search code to Annoy because the input dataset is huge (7.5mil records with 20k dictionary count). It wasn’t without issues though, however I would probably talk about it next time. In order to figure out what each parameters meant, I spent some time watching through the talk given by the author @fulhack.
-
Information Retrieving with ….. a lot of libraries
Implementing a Information Retrieval system is a fun thing to do. However, doing it efficiently is not (at least to me). So my first few attempts didn’t really end well (mostly uses just Go/golang with some bash tricks here and there, with or without a database). Then I jumped back to Python, which I am more familiar with and was very surprised with all the options available. So I started with Pandas and Scikit-learn combo.
Random Posts
-
The Little Pythoner, Part 2
In the last part, I implemented a couple of primitive functions so that they can be applied in the following chapters. The second chapter of the book, is titled “Do it again, and again, and again…”. The title already hints that readers will deal with repetitions throughout the chapter.
-
Setting Up Virtuoso for Redland Under Ubuntu
Just a quick update to the previous post, the virtuoso storage engine works with redland provided the required packages are properly installed (yes, yes, yes, I know I haven’t release my PHP OO wrapper for Redland). Now that the package is installed, we need to do some configuration so that Redland can use it.
-
Sync’ MP4 Tracks through MTP
I’m not sure why Windows Media Player 11 doesn’t like mp4/aac format but not being able to transfer my tracks in mp4 through MTP to my N81 really piss me off. At first I installed Songbird 1.0.0 to try synchronizing the tracks through the MTP addon but it doesn’t even play *.mp4 tracks out of the box under Windows XP SP3 (although it claimed otherwise). Weird thing happens when I was trying to synchronize anyway by having the files renamed to *.mp4 extension and it succeeded but without any meta-tag information.
-
Information Retrieving with ….. a lot of libraries
Implementing a Information Retrieval system is a fun thing to do. However, doing it efficiently is not (at least to me). So my first few attempts didn’t really end well (mostly uses just Go/golang with some bash tricks here and there, with or without a database). Then I jumped back to Python, which I am more familiar with and was very surprised with all the options available. So I started with Pandas and Scikit-learn combo.
-
Random notes on Pandas and Scikit-learn
So I first heard about Panda probably a year ago when I was in my previous job. It looked nice, but I didn’t really get the chance to use it. So practically it is a library that makes data looks like a mix of relational database table and excel sheet. It is easy to do query with it, and provides a way to process it fast if you know how to do it properly (no, I don’t, so I cheated).
