Notes on codes, projects and everything

Sorting Multilingual Data

Been trying my best to stick to the well-known UNIX Philosophy – “Do one thing and do it well”, so I have been breaking down my projects into numerous pieces of small tasks and rely on existing tools whenever possible. One of the existing tool that I use a lot is the GNU sort tool. Generally sort utility is really doing fine and dandy without having to configure anything, at least not until I realize the problem that leads to this post.

Considering I am having input in this form


http://flickr.com/photos/tags/スラム http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/46857830@N03/5651576112 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/99996011@N00/5396566822 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/99996011@N00/5396566822 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/99996011@N00/5396566822 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/スラム http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/46857830@N03/5651576112 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/46857830@N03/5651576112 0

I am expecting result in this order


http://flickr.com/photos/tags/スラム http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/スラム http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/46857830@N03/5651576112 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/99996011@N00/5396566822 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/46857830@N03/5651576112 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/99996011@N00/5396566822 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/46857830@N03/5651576112 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/99996011@N00/5396566822 0

but I am getting this instead


http://flickr.com/photos/tags/スラム http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/13980928@N03/6001200971 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/21207178@N07/5441742937 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/25845846@N06/3033371575 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/30366924@N08/5772100510 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/31343451@N00/5957189406 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/36662563@N00/4815218552 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/38583880@N00/5686968462 1
http://flickr.com/photos/tags/スラム http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/43335486@N00/5794673203 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/46857830@N03/5651576112 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/46857830@N03/5651576112 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/46857830@N03/5651576112 0
http://flickr.com/photos/tags/スラム http://flickr.com/photos/99996011@N00/5396566822 0
http://flickr.com/photos/tags/ホーム http://flickr.com/photos/99996011@N00/5396566822 0
http://flickr.com/photos/tags/ブラジル http://flickr.com/photos/99996011@N00/5396566822 0

After a quick search, it turns out that GNU sort actually rely on LC_COLLATE to do the sorting. I don’t really care about the character sequence order as long as I get consistent output, therefore the easiest way to fix this problem is obviously changing either LC_COLLATE, LC_ALL or LANG environment variable to “C”. However, to ensure the utility works the same elsewhere without interfering other scripts/programs relying on those environment variables, it would be easier to do this instead


$ <whatever command that produce output to stdOut> | LC_COLLATE="C" sort
Related Posts Plugin for WordPress, Blogger...

leave your comment

name is required

email is required

have a blog?

This blog uses scripts to assist and automate comment moderation, and the author of this blog post does not hold responsibility in the content of posted comments. Please note that activities such as flaming, ungrounded accusations as well as spamming will not be entertained.

Click to change color scheme