2016

• Protobuf and dask

  While JSON is a fine data-interchange format, however it does have some limitations. It is well-known for its simplicity, that even a non-programmer can easily compose a JSON file (but humanity will surprise you IRL). Therefore, it is found almost everywhere, from numerous web APIs, to geospatial data (GeoJSON), and even semantic web (RDF/JSON).

  (more…)

• The Little Pythoner, Part 3

  Previously, I started practising recursions by implementing a type check on lat (list of atoms), and ismember (whether an atom is a member of a given lat). Then in the third chapter, named “Cons the Magnificent”, more list manipulation methods are being introduced.

  (more…)

• Generating grids in javascript

  Recently I find some of my pet projects share a common pattern, they all are based on some kind of grids. So I find myself writing similar piece of code over and over again. While re-inventing wheels is quite fun, especially when you learn new way of getting things done with every iteration, it is actually quite tedious after a while.

  (more…)

• The Little Pythoner, Part 2

  In the last part, I implemented a couple of primitive functions so that they can be applied in the following chapters. The second chapter of the book, is titled “Do it again, and again, and again…”. The title already hints that readers will deal with repetitions throughout the chapter.

  (more…)

• The Little Pythoner, Part I

  I saw this article from alistapart, which is about Javascript’s prototypal object orientation. So the article mentioned Douglas Crawford, and I was immediately reminded about my struggle in understanding the language itself. Back then I used to also refer to his site for a lot of notes in Javascript. So I went back to have a quick read, and found this article that discusses the similarity between Javascript and Lisp.

  (more…)

• Re-implementing a rule engine in Javascript (Introducing Ruler)

  Sometimes, letting a piece of code evolving by itself without much planning does not usually end well. However I was quite pleased with a by-product of it and I am currently formalizing it. So the by-product is some sort of DSL for a rule engine that I implemented to process records. It started as some lambda functions in Python but eventually becomes something else.

  (more…)

• Oh NO! My print output is chopped!

  So my cheat with dask worked fine and dandy, until I started inspecting the output (which was to be used as an input for another script). While the script seemed to work fine, however when I started to parse each line I was hit with some funny syntax errors. After some quick inspection I found some of the lines was not printed completely.

  (more…)

• Processing JSON with dask.bag

  Often times, I am dealing with JSONL files, though panda’s DataFrame is great (and blaze to certain extend), however it is offering too much for the job. Most of the received data is in the form of structured text and I do all sorts of work with them. For example checking for consistency, doing replace based on values of other columns, stripping whitespace etc.

  (more…)

• Estimating the value of pi with a line of Python

  I came across a video on Youtube on Pi day. Coincidently it was about estimating the value of Pi produced by Matt Parker aka standupmaths. While I am not quite interested in knowing the best way to estimate Pi, I am quite interested in the algorithm he showed in the video however. Specifically, I am interested to find out how easy it is to implement in Python.

  (more…)

• Further Hack on the Multi-Dimensional Approximate Neighbour Search

  Traversing a tree structure often involves writing a recursive function. However, Python isn’t the best language for this purpose. Therefore I started flattening the tree into a key-value dictonary structure. Logically it is still a tree, but it is physically stored as a dictionary. Therefore it is now easier to write a simple loop to traverse it.

  (more…)

Random Posts

• Performing Statistical Analysis with Python

  This is the year I kept digging my old undergraduate notes on Statistics for work. First was my brief attempt wearing the Data Scientist performing ANOVA test to see if there’s correlation between pairs of variables. Then just recently I was tasked to analyze a survey result for a social audit project.

  (more…)

• Magic SQL

  Ever wanted to find the number of days between two dates without counting weekend (Saturdays and Sundays)? In PHP you typically needs to do a lot of calculation and a lot of factors needs to be considered. Therefore, in the end you will end up having a whole bunch of code that you will probably start asking yourself whether you are programming a web-calendar or something similar.

  (more…)

• Blending ACL and FSM-based Workflow

  After reading through the documentation, I find that the role based ACL and work flow can be more tightly integrated. Therefore I made all the transaction into many FSMs and my work flow component now consists of one work flow library and one work flow management model. As I am going a more normalized design (I use denormalized design in work as it deals with a lot of documents, however for a small project like mine, a denormalized design should do well).

  (more…)

• Linked Data Principle

  Semantic Web is not just about putting data on the web, but also making links to allow a person as well as a machine to explore the web of data. Links are made in the web of data connects arbitrary things together as described by RDF as opposed to links in the web of hypertext, where links connects to only web-resources. Linkage of arbitrary things then allow related things to be found while performing search.

  (more…)

• Over-doing a task: reading CSV file in #golang

  I was invited to try Go (the programming language, not that board game) a few months ago, however I didn’t complete back then. The main reason was because it felt raw, compared to other languages that I know a fair bit better (for example Ruby). There was no much syntatic sugar around, and getting some work done with it feels “dirty”.

  (more…)