Notes on codes, projects and everything
Information Retrieval
-
Further Hack on the Multi-Dimensional Approximate Neighbour Search
Traversing a tree structure often involves writing a recursive function. However, Python isn’t the best language for this purpose. Therefore I started flattening the tree into a key-value dictonary structure. Logically it is still a tree, but it is physically stored as a dictionary. Therefore it is now easier to write a simple loop to traverse it.
-
Approximate Neighbour Search in Multiple Dimensions
In the previous post, I re-implemented Annoy in 2D with some linear algebra maths. Then I spent some time going through some tutorial on vectors, and expanded the script to handle data in 3D and more. So instead of finding gradient, the perpendicular line in the middle of two points, I construct a plane, and find the distance between it and points to construct the tree.
-
Re-implementing Approximate Nearest Neighbour Search
Recently I switched my search code to Annoy because the input dataset is huge (7.5mil records with 20k dictionary count). It wasn’t without issues though, however I would probably talk about it next time. In order to figure out what each parameters meant, I spent some time watching through the talk given by the author @fulhack.
-
Information Retrieving with ….. a lot of libraries
Implementing a Information Retrieval system is a fun thing to do. However, doing it efficiently is not (at least to me). So my first few attempts didn’t really end well (mostly uses just Go/golang with some bash tricks here and there, with or without a database). Then I jumped back to Python, which I am more familiar with and was very surprised with all the options available. So I started with Pandas and Scikit-learn combo.
Random Posts
-
Maintaining State with jQuery Event
Maintaining state in Javascript is not too difficult once you catch the idea. However, as I am not a super brilliant programmer, it takes me some time to find a way to maintain state as YUI Event does in jQuery.
-
Development virtual machines on OS X using VirtualBox and Ubuntu
I have been following this excellent guide written by Benjamin Thomas to set up my virtual machine for development purpose. However, when I am starting to configure a Ubuntu Quantal alpha machine, parts of the guide became inapplicable. Hence, this post is written as a small revision to the previously mentioned guide.
-
Semantic Network
Recently the term “Semantic Web” becomes extremely popular that Sitepoint blogs keep posting articles on this topic (1, 2). In my college days, I learned about Semantic Network and I wonder if there is some relationship between them. I’m not sure whether I get the concept correctly but in this article I would like to revise a bit on semantic network before going to semantic web. Please correct me if I’m wrong.
-
Adding complexity to the bot: threads and processes
Being new to asyncio, after publishing the previous post on running multiple applications in one event loop, I also cross posted it to the discussion board for feedback. So apparently instead of cramming everything to the same event loop, it would be better if each application run on a separate thread. That makes sense, considering all the code that was written for that.
(more…) -
Blending ACL and FSM-based Workflow
After reading through the documentation, I find that the role based ACL and work flow can be more tightly integrated. Therefore I made all the transaction into many FSMs and my work flow component now consists of one work flow library and one work flow management model. As I am going a more normalized design (I use denormalized design in work as it deals with a lot of documents, however for a small project like mine, a denormalized design should do well).
