Flatten a directory with a filter in Clojure

I always wanted cross platform and stand-alone script to flatten a directory with an optional filter. I have relied on Ant builder from Groovy but it is slow and one needs to have Groovy installed along with Ant. So, I wrote a small script in Clojure that I can use and share with friends easily. Without further ado, here is the entire script:

Here is the source file for you to use, experiment or do whatever you want to have fun with Clojure.

Ubiquitous note taking with Vim and Dropbox

Finally I found the combination of a best data sync tool called Dropbox and my favorite text editor Vim for ubiquitous note taking. To make it even seamless, I added following to my .vimrc and sharing it with you guys:
map <leader>nn :tabedit ~/Dropbox/YourName/notes/
<leader> here is a mapleader, which is a special variable in vim that you can bind to a key in your vimrc. I use following to map it to ","
let mapleader = "," 
After that you can use comma with the combination of any characters for mapping things that you end up doing a lot. In this case, I mapped it to creating a new file in a new tab within ~/Dropbox/YourName/notes/ folder with whatever name that I will type before hitting the enter key.  Above setting will allow following work flow:
1. Open vim
2. Just type ,nn and enter the file name you want
3. Hit enter and start typing
Creating a new notes could not be more easier and pleasant than this knowing that your notes will be automatically synchronized across multiple computers even on mobile phones. Enjoy!

On learning idiomatic Clojure

If you hang around in Clojure google groups and Clojure user groups, one topic that keep popping up is this: How one go about learning idiomatic way of writing Clojure code? After reading almost every source that I could get my hands on, I finally settled for this:

1. Study Clojure.core daily
2. Read "Joy of Clojure" The Joy of Clojure
3. Code, Code, and Code in Clojure as much as you can

If you are really serious about Clojure, grab a MEAP edition of "Joy of Clojure" and the book bundle. Here is how I follow above steps: I find interesting functions from Clojure.core and will implement it by adding my- in front of standard functions. Then I search for the forms used within the function in the "Joy of Clojure".  It is quite an enlightening experience to see how the creator of the language structure his code. On top of that, you have an idiomatic Clojure reference book's help within your reach. Boy, it feels like a happy kid with a cotton candy in your hand.  Let me show you by an example:

 (def
 my-assoc
 (fn my-assoc
   ([map key val] (assoc map key val))
   ([map key val & kvs]
    (let [ret (my-assoc map key val)]
      (if kvs
        (recur ret (first kvs) (second kvs) (nnext kvs))
        ret))))) 
(println (my-assoc {} :symbol "assoc")) => {:symbo assoc}

my-assoc here is a Var that holds a reference to a function that named my-assoc, which takes a map and one or more key-val pairs, and returns a map that contains the mapping.  This actually is a classic Clojure style of tail recursive function. Namely, it is an idiomatic way to use function signatures for base and recursive cases. It is also very elegant to use let binding to hold accumulative value that will be referred in multiple places in a block. The use of recur in the tail position is just what it is intended for. It recursively invokes my-assoc function with the correct number of arguments, which in this case
(my-assoc [map key val & kvs])  until kvs are nil, which is one of only two false (nil and false) in idiomatic Clojure.  (first kvs) (second kvs) (nnext kvs) is also idiomatic way of traversing a seq that will save you from unexpected nil punning. (Again, read "Joy of Clojure" for detailed explanation for this). 

My way of learning idiomatic Clojure is very opinionated and might not work for everyone. What is your way of learning idiomatic Clojure? Please share it if you can so a Clojure noob will have a good start.

Stat 101 and little more in 25 lines of Clojure

If you like numbers or statistics to be exact, Clojure is your best friend. There is this wonderful statistical graphing library called Incanter, which is an R like environment for all of your data crunching and visualization needs. And there is such an expressive power of Clojure at your fingertips that you can pretty much express your stat 101 (probability, permutation, combination, number of trial and degree of certainty) with a mere 25 (blank lines are not counted) lines of Clojure code. OK, without further ado, here is the whole thing:
Note: DC stands for degree of certainty that an event will appear, P stands for probability of the event and N stands for number of trials (events). from the auther of Probability Theory, Live!
1:  (defn N [p dc] 
2:   (/ (Math/log (- 1 dc)) 
3:     (Math/log (- 1 p)))) 
4:   
5:  (defn DC [p n] 
6:   (let [np (- 1 p) 
7:      pp (Math/pow np n)] 
8:    (- 1 pp))) 
9:   
10:  (defn k-factorial [n] 
11:   (reduce * (range 1 (inc n)))) 
12:   
13:  (defn permutation [n k] 
14:   (/ (k-factorial n) 
15:     (k-factorial (- n k)))) 
16:   
17:  (defn combination [n k] 
18:   (/ (k-factorial n) 
19:     (* (k-factorial k) 
20:      (k-factorial (- n k))))) 
21:   
22:  (defn- k-filter [el coll] 
23:   (filter #(not (= el %)) coll))   
24:   
25:  (defn permutations [n coll] 
26:    (if (= n 1) 
27:     (map #(vector %) coll) 
28:     (for [el coll nlis (permutations (- n 1) (k-filter el coll))] 
29:      (conj nlis el))))  
30:   
31:  (defn combinations [n coll] 
32:   (set (map set (permutations n coll)))) 


The combinations and permutations implementation is recursive and will result in stack over flow for large data. For non recursive implementation, you can check clojure-contrib's combinatorics library. It is fast and makes use of lazy-seq, meaning you can virtually generate combinations for arbitrary large amount of data. However, the point of this post is to show you that it is very easy to implement statistical formulas in Clojure. You might wander what you can do with the above code. Here is one interesting statistical riddle you can solve with it: Let us say you have a bag with 100 coins. There are 7 gold coins and 93 silver coins. If you are to pick up a coin without looking inside the bag, what is minimum number of coins that you had to pick up to be able to say with 50% confidence that this bag has 7 gold coins? Now, we can put the code above into action:
First we need to clarify the riddle is asking us to find the number of trials, which is N as defined above.
Second, we know that the possibility P of picking up a gold coin is 7/100.  The degree of certainty, which is denoted as DC above, is given as .5 or 1/2. So let us just plug in those numbers to the function N:

=>(println (N (/ 7 100) 0.5))

This will produce 9.551337509447352, which roughly around 10 trials.  So we can say that after picking about 10 coins, you can say with 50% percent confidence that the bag has 7 gold coins.

Again, if you are math inclined person, you will have a lot more fun with Clojure than you might have imagined.  Note that the DC and N formula is directly from following website:
http://saliu.com/Saliu2.htm. I also got help and feedback to my permutations function from nice folks hang around in Google Clojure groups, which I highly recommend you sign up immediately if you want to learn from a master or masters.  I might do another post with the non recursive version of the function in my next post when time permits. Until then, enjoy your Clojure journey.
 

The most portable Clojure REPL box - Eee PC 701

I have been trying to put my Eee PC 701 into a good use. This weekend I realized that turning it into a portable Clojure Box could not be more useful considering I can have REPL access whenever I want. As it turned out, it is quite easy, and here is how I got it done:

1. Download the Arch Linux ISO (here: http://www.archlinux.org/download/) and burn it onto a CD.
2. Use an external CD drive to boot your Eee PC with Arch Linux Image and follow the instructions to install the base system.  You will find very detailed step by step instruction from here: http://wiki.archlinux.org/index.php/Asus_Eee_PC_701. Even though the installation process is pretty straight forward, here are few things you need to pay attention:
• Make sure you select ext2 file system
• Run "pacman -Syu" to do system upgrade before installing any extra programs
• Make sure to add hal and fam to the deamon list by adding "DAEMONS=(syslog-ng network netfs crond @hal @fam)" to your /etc/rc.conf if you are planning to add a graphical desktop environment
• Now you can install Mercurial, Git, Java and Clojure. (You can install all from Arch repo except Clojure. You can easily install Clojure by: hg clone http://bitbucket.org/kasim/clojurew/, which is a project I created to make setting up Clojure less painful.)   

You can also install Xorg and a lightweight desktop called lxde very easily if you want. However, All I needed is a shell log in so I can play with Clojure REPL. The Clojurew project include latest versions of Clojure-contrib and JLine so you pretty much have all Clojure API doc and source code browsing at your fingertips with command line history. After installing Mercurial, Git, Java and Clojure with clojure-contrib, you will still have close to 3 Gig empty space left on your hard drive(It has only 4 Gig hard drive). That means you can still install Incanter, which I have been playing with a lot recently. (Note: you can just copy Incanter app jar from the binary download to Clojurew's lib directory to have Incanter available at the REPL).  It took me around an hour to set everything up and the joy of being able to carry a little Clojure box around is well worth the effort and amazing. Go ahead and try it out if you own Eee PC 701.

Easy Clojure developement with Vim

Starting with Vim or Clojure can be difficult. Starting with both at the same time is not recommended. But it can get difficult for someone who knows both. Following is a simple way how I do my development:

I edit clojure scripts in Vim. I found putting following lines in .vimrc very helpful:

set showmatch 

map <F5> <Esc>:!clj '%:p'<CR>

The first one is for highlighting matching (,{ and [. The second one is for executing the currently edited file.  Please note that !clj '%:p' just invokes shell command clj with the fully qualified path of the current buffer you are editing and it is mapped to the F5 function key.  If you are using MacVim, make sure that you put following lines in your .vimrc file:
let $PATH="path/to/clojure/home/bin:".$PATH
I also use a vim plug in called “AutoClose” (here: http://www.vim.org/scripts/script.php?script_id=1849) that will automatically close parenthesis, brackets and curly braces. You can also do yourself a favor by installing a plug in called “VimClojure” (here: http://www.vim.org/scripts/script.php?script_id=2501). I only use syntax highlighting feature and do not run interactive REPL with it(I do not recommended using nailgun server). The above is just enough to do small scale development. However, for involved development, there is dependency and class path hurdles to overcome. It is even worse for someone who is new to Java World from other dialects of LISP.  To ease the pain, I created a project called Clojurew (here: http://bitbucket.org/kasim/clojurew/src) that will do automatic class path resolution. Any jar dependency will be resolved by just coping the jar to the lib folder of Clojurew. If you want to include your own script to the class path,  you can even create a soft link to your script within the lib directory and it will be included in the class path as well. Here is how you can do this:

 ln  -s path/to/your/script linkname

After this, you can use any function in your script that is defined in other files.  Happy Vimming and Clojuring!

I hope someone find this helpful.

let and binding macros in Clojure

Clojure has a special form called "let" and "binding", which are actually implemented as macros. It can get confusing sometimes.  If you have read "Programming Clojure" book, you might be thinking that it is easy to use them correctly considering following straightforward example:

Basically, "let" creates a lexical scoped Vars, which is local to the code block or a procedure where it is initially defined. Therefore, in the example above, "let" just created a new "x" with the value of "new value". But it is not local to the function "print-x".  That is to say, "print-x" have no knowledge of the local scoped binding that is created by "let". "binding" macro instead creates a new binding for the already existed var, "x" with the "new value". Since "print-x" is evaluated within the "binding" form, the newly bound value is visible to any chained calls within the form.  All nice and easy. However, it is not that obvious to understand and use them correctly. Look at following example:

"let" example is just fine. It created a new var "y" with the value of 5 bound to it.  But what is going on with the "binding" example? If we dig little deeper to Clojure API doc, we learn that the new bindings in "binding" are made in parallel, not sequential as it is in "let". Therefore, existing var "y" gets bound to the root binding of "x" not the overshadowed value of "x", which is the case in the "let example".

OK, so far so good. Let us look at another example:

Line 20-29 is pretty straight forward. But how about line 30? Shouldn't  it call "print-y" function? Not really. It is because the anonymous function is evaluated outside of the "binding" form.

To see this in action let us try following:

The "#<user$eval__28$fn__30 user$eval__28$fn__30@bdb503>" is an indication that anonymous function did not get evaluated within the "binding" form.  Ok, now, how do you make sure it evaluates within the "binding" form? Here is what you do:

Just enclose the anonymous function within parenthesis.  What do we learn from all of the above? Here are the summary that helps one use "let" and "binding" correctly:

• Use "binding" only if you are trying to alter a behavior dynamically for special-vars that is already defined. Enclose your special vars within * as recommended by idiomatic clojure to make your intent clear.
• Use "let" for locally scoped vars and avoid shadowing already defined vars.
• Remember "let" binding is sequential and "binding" binding is done in parallel.
• Pay close attention to the scope of the "binding" form. Any expression that is not evaluated inside the form, will not see the binding. That is to say, binding in "binding" form is thread local starting from the "binding" form until it ends including any expression that is chained within the form, which is what the thread-local are all about.

I posted this so that it helps someone trying to find his or her way in the wonderful world of Clojure.  

Easy way to start with Clojure developement

For any software developer, I recommend learning functional programming, which is especially true for anyone who spent quite a bit of time on doing Object Oriented Programming.  As far as my experience from Ruby -> Groovy -> Scala -> Clojure journey concerned, I strongly recommend learning Clojure to get into functional way of thinking in practical way.  One of the hurdles that I encountered in getting started with Clojure is lack of a simple way of setting the environment. Yes, there are many ways one can get started such as Lein, Maven+Clojure Plugin, Emacs+Swank+Slime, Vim+VimClojure, Eclipse+CounterClockWise, NetBeans+Enclojure, IDEA+LaClojure, you name it. But it all makes a newbie to spend days if not hours to get his or her environment right.  After playing the hard way( Emac+Swank+Slime, Vim+VimClojure), which I do not recommend for a new comer,  I ended up creating a project called ClojureW.  It is the easiest way to start with Clojure as far as I know for all three major OSes such as Windows, OS X and Linux.  Here is all you need to get going:

1.       Download Clojurew from: http://bitbucket.org/kasim/clojurew/get/tip.zip

2.       Unzip it to a folder and change to bin directory of this folder

3.       Just run clj to get a REPL or run clj cljscript.clj to execute your script.

For beginners, I recommend just using a text editor that highlights matching parenthesis.  Believe me, trying to set up an “IDE” for Clojure is painful if not disappointing at best.  The main thing that you learn from Clojure is to think bottom-up when solving problems.  You can practice this approach with any language but Clojure will force you to think this way.  Again, setting up IDE is a hindrance to what Clojure is really about at the beginning. You can play with IDEs once you get into idiomatic Clojure way of thinking.  As stated by a book titled, “97 Things Every Programmer Should Know”, being able to quickly type up a simple “Hello World” program and run it via command line can prove extremely valuable after all.#  

Books everyone should read not just geeks or hackers

It has been a while since I did not write anything on my blog. I have been busy with work and reading among other things. But two books I read recently made me to want to blog about them. One is Paul Graham's Hackers and Painters, the other is Pomodoro Technique Illustrated by Staffan Noteberg.

On Hackers and Painters:

It is a great book especially for programmers. The breadth of common sense and not so common sense knowledge is breathtaking. It offers unique perspective on programming languages. It is pretty much biased towards Lisp, which is understandable considering his success and expertise on it. However, his advice of not learning Lisp for people over the age of 25 does not do justice to such a great language that every programmer should learn even if they do not use it. I am recent Lisp convert and I can already see its benefit to my daily work. I think earlier Lisp Dialects did not catch on because of its "strange" syntax, which made it more implementer friendly but not user friendly. But a recent dialect called Clojure is poised to make Lisp very very popular. Because the creator of Clojure is making very very nice middle road approach between user and implementer of the language. Anyway, Paul Graham's collection of essays in his book is really eye opening. There is chapter on how America's public school system works, why it is works the way it is and how adults should help kids between the age of 11 and 17. There is also a chapter about how to create wealth. His take on start ups are really thoughtful.

On Pomodoro Technique Illustrated:

Pomodoro means tomato in Italian. It is a focus training method that helps every goal oriented person by allowing them to get things done. What is unique about it is:

1. It is simple

2. It is easy

3. It is effective

Here is what you need to do:

1. Pick a to do item now

2. Set 25 minutes timer and start doing it now

3. Repeat above with new to do item if you finish or continue working after a short break.

All you need is a Timer, a sheet of paper, and your attention. In this memory overflow day and age, everyone needs to stay focused to get anything done. Countless interruption and stimulus is making people not even be able to concentrate for 25 minutes. Try it and you will be more productive.

To roman implementation in Clojure

Here is my take of to roman implementation in Clojure:
I posted this in http://rosettacode.org/wiki/Roman_Numerals#Clojure

1:  (def arabic-roman-map
2:     {1 "I", 5 "V", 
3:     10 "X", 50 "L", 
4:     100 "C", 500 "D", 
5:     1000 "M", 
6:     4 "IV", 9 "IX", 
7:     40 "XL", 90 "XC", 
8:     400 "CD", 900 "CM" })
9:  (def arabic-roman-map-sorted-keys
10:     (sort (keys arabic-roman-map)))
11:  (defn find-value-in-coll
12:   [coll k]
13:   (let [aval (find coll k)]
14:   (if (nil? aval) "" (val aval))))
15:  (defn to-roman
16:   [result n]
17:   (let
18:     [closest-key-for-n (last (filter #(> n %) arabic-roman-map-sorted-keys))
19:      roman-value-for-n (find-value-in-coll arabic-roman-map n)
20:      roman-value-for-closet-to-n (find-value-in-coll arabic-roman-map
21:                                   closest-key-for-n)]
22:      (if (or (<= n 0) (contains? arabic-roman-map n))
23:        (conj result roman-value-for-n)
24:        (recur (conj result roman-value-for-closet-to-n)
25:            (- n closest-key-for-n)))))
26:  Usage: >(to-roman [] 1999)
27:  result: ["M" "CM" "XC" "IX"]