Notes on codes, projects and everything
In the previous post, I re-implemented Annoy in 2D with some linear algebra maths. Then I spent some time going through some tutorial on vectors, and expanded the script to handle data in 3D and more. So instead of finding gradient, the perpendicular line in the middle of two points, I construct a plane, and find the distance between it and points to construct the tree.
Recently I switched my search code to Annoy because the input dataset is huge (7.5mil records with 20k dictionary count). It wasn’t without issues though, however I would probably talk about it next time. In order to figure out what each parameters meant, I spent some time watching through the talk given by the author @fulhack.
array_map function is a function that I use the most in my php scripts recently. However, there are times where I want to pass some non-array into it, therefore often times I have code like the snippet shown below:
$result = array_map(
'some_callback',
array_fill(0, count($some_array), 'some_string'),
array_fill(0, count($some_array), 'some_other_string'),
$some_array
)
It doesn’t look good IMO, as it makes the code looks complicated. Hence, after seeing how the code may vary in all different scenarios, I created some functions to clean up the array_map call as seen above. Code snippet after the jump
While working on a text classification task, I spent quite some time preparing the training set for a given document collection. The project is supposed to be a pure golang implementation, so after some quick searching I found some libraries that are either a wrapper to libsvm, or a re-implementation. So I happily started to prepare my training set in the libsvm format.
Often times, I am dealing with JSONL files, though panda’s DataFrame is great (and blaze to certain extend), however it is offering too much for the job. Most of the received data is in the form of structured text and I do all sorts of work with them. For example checking for consistency, doing replace based on values of other columns, stripping whitespace etc.
There are a lot of things I want to post to both here and my personal blogs. However I was sucked into sanctuary for the most of last month. I guess after a month of playing, it is probably time to slowly resume my personal projects.
Not sure about the others, but the obsession to my coding tools is probably more than I would admit. I have just managed to do a dirty quick hack to manage my VIM configuration settings. While I am sure there are other people doing this, I would like to show my reinvented wheels.