Notes on codes, projects and everything
ann
-
Approximate Neighbour Search in Multiple Dimensions
In the previous post, I re-implemented Annoy in 2D with some linear algebra maths. Then I spent some time going through some tutorial on vectors, and expanded the script to handle data in 3D and more. So instead of finding gradient, the perpendicular line in the middle of two points, I construct a plane, and find the distance between it and points to construct the tree.
-
Re-implementing Approximate Nearest Neighbour Search
Recently I switched my search code to Annoy because the input dataset is huge (7.5mil records with 20k dictionary count). It wasn’t without issues though, however I would probably talk about it next time. In order to figure out what each parameters meant, I spent some time watching through the talk given by the author @fulhack.
Random Posts
-
Real Life Enterprise Class Website Development
Recently I am involved in developing some small modules for a enterprise class website using CodeIgniter (CI). There was no restriction given on which framework should I use for the development and I chose CI as I learned a bit on it (when I was considering whether to shift my personal development project). Of course there are other reasons why I chose to learn CI, for example the superior documentation and screencasts available.
-
Resource Definition Framework
As the name implies, Resource Definition Framework, or RDF in short, is a language to represent information about resources in world wide web. Information that can be represented is mostly metadata like title (assuming the resource is a web-page), author, last modified date etc. Besides representing resource that is network-accessible, it can be used to represent things that cannot be accessed through the network, as long as it can be identified using a URI.
-
Adding a new settings applet for my daemon
This is basically a small incremental update to my script published here. For some reason, the previous version of the script didn’t really work, so this release should fix the problem. Besides fixing the problem where the daemon did not actually launched at start up, I have added a settings applet for this script as well.
-
Symlinking two Blogs
I am currently doing some organization to my blogs. Few weeks ago, after spending months struggling to work in Ubuntu 7.10, I learned about symbolic links. Then I thought this would be good for my project file management. Therefore I started to re-organize my project file structure to utilize symbolic links. One of the projects that uses symbolic link is the current wordpress theme.
-
Processing JSON with dask.bag
Often times, I am dealing with JSONL files, though panda’s DataFrame is great (and blaze to certain extend), however it is offering too much for the job. Most of the received data is in the form of structured text and I do all sorts of work with them. For example checking for consistency, doing replace based on values of other columns, stripping whitespace etc.
