Mu

Zen and the art of...

2010-12-12

Improved Sandbar Forms

There has been a flurry of improvements made to the Clojure web stack during the past year. Compojure has matured, there's some non-trivial code available (for example, see Brian Carper's cow-blog) and the new Sandbar library which brings a higher-level of abstraction on top of Compojure and Ring. For now, it provides a stateful session mechanism, authorization + authentication and a clever way of defining forms layout, processing and validation. Also there's much more to come, you can look at the details in the following roadmap.

For today, I'll discuss the recent changes to the forms namespace. There has been a lot of work done on that part during the past week, but the changes to the API are minimal. I'll go over each options of the defform macro, but first lets talk about the code behind. As Stuart Halloway stated in his book, the first rule of the Macro Club is: "Don't Write Macros." So the most significant alteration is the rewrite of the defform macro. Previously, it was a big piece of code (134 lines) generating a bunch of definitions of all sorts. It was quite hard to modify and had the usual constraints of using macros that way. It now has been replaced by a much more simple macro that call the new make-form function.

Lets see a sample form written for the current (0.3) version of Sandbar.

(forms/defform group-form "/group/edit"
  :fields [(forms/textfield :name)
           (forms/select :region
                         (db/all-region)
                         {:id :name :prompt {"" "Select a Region"}})
           (forms/textarea :description)]
  :load #(db/fetch-group %)
  :on-cancel "/groups"
  :on-success #(do (db/store-group %)
                   (session/flash-put! :user-message
                                       "Group has been saved.")
                   "/groups")
  :properties {:name "Group's name:"
               :description "Description:"})

(defroutes group-form-routes
  (group-form (fn [request form] (views/layout form))))

It was nice but had a severe limitation: you were forced to use a fixed set of routes. When creating a record, "/new" was appended to the given URI else the "id" key was used. Also, the fields option tended to become cluttered with the data source bindings. These are the two main point I'll address here.

Firstly, lets rewrite the fields option, it's quite similar in taking a vector of field descriptions. The difference is in the field description functions that are taking the name of the field (as a keyword) followed by a list of optional key/value pairs.

  :fields [(forms/textfield :name
                            :label "Group's name:")
           (forms/select :region
                         :prompt {"" "Select a Region"})
           (forms/textarea  :description
                            :label "Description:")]

Each field functions can take a label option, then most have an optional boolean required option which auto-generate the corresponding validator for that field and finally there's the prompt option for the select field. Any other options will be added to the field's HTML attribute.

Secondly, there are the new options for managing the form action and method attributes. Each are prefixed by create or update followed by -action or -method whether it's for an action or method. They can take parametrized routes which will get their parameters replaced by the matching route parameters from the incoming request.

  :create-action "/groups"
  :update-action "/groups/:id"
  :update-method :put

Thirdly, here's the new bindings option which take a map of field names followed by their respective binding information in a map.

  :bindings {:region {:value :id
                      :visible :name
                      :source (constantly (db/all-regions))
                      :data :id}}

The source option needs a function that fetch the relevant data, the visible option determines what field to show on the page and the value and data options represent the actual value to use in the source data map and the form data map respectively.

Finally here's the whole code for this example using the new forms features using RESTful routes.

(forms/defform group-form
  "Form handler for Group entity."
  :fields [(forms/textfield :name
                            :label "Group's name:")
           (forms/select :region
                         :prompt {"" "Select a Region"})
           (forms/textarea  :description
                            :label "Description:")]
  :load #(db/fetch-group %)
  :on-cancel "/groups"
  :on-success #(do (db/store-group %)
                   (session/flash-put! :user-message
                                       "Group has been saved.")
                   "/groups")
  :create-action "/groups"
  :update-action "/groups/:id"
  :update-method :put
  :bindings {:region {:value :id
                      :visible :name
                      :source (constantly (db/all-regions))
                      :data :id}})

(defroutes group-form-routes
  (GET  "/groups/new"      request (group-form request))
  (POST "/groups"          request (group-form request))
  (GET  "/groups/:id/edit" request (group-form request))
  (PUT  "/groups/:id"      request (group-form request)))

Furthermore, other modifications include that most defform options can take a function of the request instead of just a value and that redirection URIs in the on-success and on-cancel options can be parametrized.

The forms namespace is still a work in progress and is still being improved. Particularly there is talk concerning the way forms get rendered, which currently lack flexibility as pointed out by David Nolen in this thread. Nothing has been done yet in this regard, so it's the right time for anyone interested to chime in this discussion.

2010-07-23

Getting Back on Rails

I've been pretty busy in the past months and didn't felt inspired enough to write. Today is time to get back to writing in a human language instead of a computer one.

Coming Back Home

I've been involved in web development for more than ten years, but for a long time I could only be labeled a designer. A little more than four years ago I began developing web sites using that awesome little framework named Ruby on Rails. After toying with it for less than a year, I gave it up to go on the dangerous path of trying to really understand what was going on behind all that magic. After learning a lot while dabbling into some lesser known frameworks like Pylons, Helma and then Compojure, I now got the chance to code in a more professional setting using Ruby on Rails.

Good Old Rails Has Changed

For the better! Since four years ago, Rails has visibly matured and it is now used in a lot of very successful web sites. All this popularity pushed the framework to evolve and it has become much more capable with the help of an army of plugins. Some simple features were kicked out of the main codebase to be replaced by plugins. For example, the built-in pagination was removed from Rails 2.0 and turned into a plugin called classic_pagination. It has fallen out of favor for another plugin, will_paginate, which is really much better.

Back then there was a generator called login to build basic authentication and user management features into your application. This generator has been deprecated and now there's multiple plugins available. I've only tried Authlogic by now and I'm pleased by its simplicity and customizability so far.

And there's a lot more! More conventions, more syntactic sugar, more ready to use plugins. One notable change is that Rails is now RESTful, meaning that it uses the HTTP protocol intelligently for routing requests. For example, if a DELETE request with only the name of the controller is sent, by convention, Rails will call the destroy action. One last thing to note is that ERb template files have given up the .rhtml file extension for a more versatile .erb appended after the output file extension.

Development Environment

Well, I'm a staunch Emacs user, so I ended up using it for developing Rails applications. Nonetheless, I was curious about the degree of support offered by some IDEs, especially from the latest release of NetBeans which some people say is getting pretty good. I've tried it for a couple of hours to find out that it was really not the kind of environment I'd like to work in.

First, the wizard wasn't able to create a database in Derby because it tried using the inexistent root user. Then it was the included GlassFish server that didn't want to use another port than 8080, completely ignoring the provided configuration file (which is WEBrick specific it appears). Those are all minor glitch, but it continued like that for an hour or two and soon I felt the need for this trusty Emacs text editor. The most glaring bug that really pushed me off NetBeans was the configuration screen for syntax highlighting, it was at some point using 90% of both cores on my machine! No wonder everybody says Microsoft have the lead in the IDE world, even I must say that it's a decent development environment after manually setting Emacs like shortcuts (the included ones just don't cut it).

Emacs Configuration

There's two mode for Rails on Emacs, namely emacs-rails and Rinari. The first one is older and is quite outdated. It has been replaced by a new mode named emacs-rails-reloaded by the same author. It provides navigation between the multitude of files in a Rails project, make it easy to call rake or generators and a lot more. Rinari is a lightweight alternative that I haven't tried yet but it looks quite capable too.

For editing the ERb templates, you can use the nXhtml mode, which is already included in the Windows port of Emacs. I had some compatibility issues between that version and emacs-rails-reloaded and had to install the latest version.

Deployment

I had the opportunity of deploying an application and find out that there were great improvements made on this side.

First and foremost, is that incredible tool called Capistrano. It make deploying Rails application in a consistent and reversible manner a breeze. With very little configuration, it will copy your files in a date/time tagged directory, help you easily start/stop your application server(s) and get back online previously deployed versions easily. It's even more than just a tool for RoR projects, you can understand it as being a sort of Makefile for remotely controlling multiple servers.

Finally, back in the days, most hosting provider offered only fastCGI or mod_rails on top of Apache for running your applications. Today there's more options, but I won't go through all of them as I've still got much to learn about them. For now I've learned that Passenger (previously mod_rails or mod_rack) is the recommended way of deploying a Rails app, it can use Apache or Nginx as its web server.

2010-04-18

Cygwin Tip #1

Calling Java Under Cygwin

Cygwin is a fantastic piece of software for those who, like me, can't live without a Linux-like command line and environment. But for any person who have tried to blend two completely different cultures, we know it's not always easy to make them play well together. In this post I'll explain how to make Java's Windows version feel more like a Cygwin application.

The main problem comes from the way a file path differs between Windows and the POSIX standard. The most obvious difference is the path separator issue, in Windows world it's \ while for the rest of the planet it's /. Another one is the way the root path(s) are being represented, in Windows each drive/partition have it's own letters with its own root path. Cygwin being POSIX-based has a single root path (/) and maps the Windows letters in a directory called /cygdrive. Those problems are easily circumvented using a the cygpath command:

$ cygpath -aw '/cygdrive/c/Program Files/Java/jdk1.6.0_18/bin/java'
C:\Program Files\Java\jdk1.6.0_18\bin\java

That's Not Enough

It wouldn't be hard to call this command on the classpath argument when calling java from cygwin, but there's another hindrance. The classpath usually contains more than one path, these paths being separated by a colon (:) in POSIX systems or by a semicolon (;) under Windows. It would be also quite tiresome edit java calls manually so I came up with a little Ruby script that can do the job for us.

#!/bin/ruby
# Slightly obfuscated cygwin + windows java wrapper, automate cygpath

cpi = ARGV.index("-cp") + 1
cp = ARGV[cpi] if cpi

XBCP = "-Xbootclasspath/a:"

xbcpi = ARGV.index{|i|i=~/^#{XBCP}.*/}
xbcp = ARGV[xbcpi] if xbcpi

if cp or xbcpi
  def convert_paths(paths)
    paths = paths.gsub(':', ';').split(';')
    paths.map{|p|`cygpath -aw #{p}`.strip}.join ';'
  end
  ARGV[cpi] = convert_paths(cp) if cp
  ARGV[xbcpi] = XBCP + convert_paths(xbcp.sub(XBCP, '')) if xbcp
end

java = '/cygdrive/c/Program Files/Java/jdk1.6.0_18/bin/java'
cmd = [java].concat ARGV

def e(s); "\"#{s.strip.gsub('"','\"')}\""; end

exec(cmd.map{|a|e a}.join(' '))

This is more of a hack than a solid solution, but it's enough for my current usage and hopefully yours. Now the Windows java command can take POSIX paths without complaining under Windows, isn't that marvellous!

2010-04-15

Gödel, Escher, Bach and Smartphones

After many many years, I'm still (slowly) reading GEB. That's because I'm wasting to much time reading the (whole) Internet to bother with printed books! Yet, I like to finish what I start (eventually) and yesterday I found a really interesting quote in that book. Mind you, it has been written thirty one years ago, at a time when personal computers were a novelty that only a few lucky ones could afford. Most computer usage (Punched Cards!) was being done in Universities laboratories and these machines weren't fast enough to do most of what we take for granted today.

Dogs, Garbage and Light Bulbs...

The world have changed a lot since then and computers are incredibly more useful now a day's. While discussing the Church-Turing thesis in chapter seventeen, Mr. Hofstadter goes on the representation of knowledge about the real world and say:

Because of the complexity of the world , it is hard to imagine a little pocket calculator that can answer questions put to it when you press a few buttons bearing labels such as "dog", "garbage", "light bulb", and so forth. In fact, so far it has proven to be extremely complicated to have a full-size high-speed computer answer questions about what appear to be rather simple subdomains of the real world.

Hard to imagine back then, but today it's quite commonplace to see people asking such questions to their smartphones. Which bring me to the point of this post, is the Web can be considered a form of artificial intelligence? I'd say no, it's just a hack, yet a pretty useful one. The raw data that can be found here isn't that different than the information that could be gathered in any other format. It is the search capabilities offered by various engines that truly makes the greatness of the World Wide Web. Still, these facilities aren't quite smart enough, we still have to choose carefully the wording of our queries and click around.

The Semantic Web

This is the next step, and a big one also. A lot of have been written on the subject, a lot of code too and thus far that concept hasn't yielded (easily) usable tools. I think that most problems concerning semantical representation of data are intrinsic to the nature of information. As Oleg once said: "Without a reader, a book has no meaning." It's all about asking the right question. And here is the missing link, does natural languages are good enough? Where's the real burden located, in formulating a query or in understanding it?

2010-02-22

Redesigning ClojureQL's Frontend

I've not updated this blog in a while as I've been pretty busy for the past few weeks. I've got many projects on the table and one of them is starting to get really interesting. I'm obviously talking about the subject of this post, ClojureQL. I've mainly been working on the backend since the end of last year. In the meantime, Meikel has started a rework of the frontend to provide a cleaner API, free from the magical artifacts introduced by using macros like in the current (0.9.7) version. This rework have been triggered by a post carefully explaining the issue, courtesy of Zef.

I won't go further than a linkfest so here are the important links for those interested in the future of ClojureQL:

We're also ready to receive your suggestions on the brand new clojureql group. Please, visit the Wiki pages and tell us about what you think.

2010-02-07

Google AI Challenge in Clojure

These days, the web have been noisy about the brand new Google sponsored AI Challenge organized by the University of Waterloo Computer Science Club. I find that it's a great initiative and is a good occasion for amateur AI researchers like me to do something else than web development or database related coding. Being fond of Clojure, I've taken on the task of making a starter package for this language and making it available on a GitHub repository. It's not an official package yet, but it shouldn't be a problem as the organizers seems really open.

The code has been translated from the Java starter package. I've tried to keep the map code close to the original, so it may not be the most idiomatic Clojure code. There's four global refs containing the state of the game: width, height, walls and players. The two first are simple integers, walls is an hash map keyed with points (which are vectors) and the last one a vector of points. There's some convenient functions to access that data, wall? to tell if a wall is found at the given coordinates and two others to get each players position, you and them. To move your bike you must call make-move with one of :north, :east, :south or :west. The map code end up being nearly half the size of the Java version.

With this code we can rewrite the sample random bot in Clojure, I'll spare you any more explanation and only show the code.

(load "map")

(defn valid-moves [x y]
  [(when-not (wall? x (- y 1)) :north)
   (when-not (wall? (+ x 1) y) :east)
   (when-not (wall? x (+ y 1)) :south)
   (when-not (wall? (- x 1) y) :west)])

(defn compact [coll]
  (filter identity coll))

(defn choose-at-random [coll]
  (let [size (count coll)]
    (when (< 0 size)
      (nth coll (rand-int size)))))

(defn next-move []
  (choose-at-random
   (compact
    (apply valid-moves (you)))))

(defn game-loop []
  (loop []
    (initialize)
    (make-move (next-move))
    (recur)))

(game-loop)

Hack and be mery!

Update: I've updated the GitHub repository with some improvements and instructions on how to compile your entry. While discussing with one of the contest organizer over their forum, I realized that even though Clojure is more than fast enough for this kind of task, the Clojure jar loading time take a good chunk of the first turn time, so be careful. By the way, it's now an official package.

2010-01-31

Closure Library Tutorial: Tasks (part 2)

It's been nearly two months since I've written about JavaScript and now is the time to follow up on the previous part of this tutorial. Since then, there have been some changes to the Closure library, nothing major, mostly improvements to the documentation and bug fixes.

Sliders

Adding a Slider is unsurprisingly as easy to do as any other components. Contrary to those we already used though, it doesn't come with a nice interface with pre-made CSS and images. Closure let you decide how your sliders will look like, which is comprehensible as this control is really versatile and it would be hard to come up with a good generic design for it. A slider element is composed of two DIVs, one representing the slider itself, the other being the thumb. Both have a class name to let you control their appearance. For the slider element this name is composed of a prefix found in the Slider's CSS_CLASS_PREFIX static property followed by either of "-horizontal" or "-vertical" depending on the orientation. The thumb has only a single class name determined by the THUMB_CSS_CLASS property, note that as the thumb is contained in the slider element, it can be styled depending on orientation too.

There's some rules to obey while styling a slider:

  • a slider cannot be inlined, but you can use inline-block;
  • a slider cannot be floated;
  • the thumb position style must be set to absolute or relative.

Note that it's preferable to make both elements hide their overflowing contents to prevent scrolling bars from appearing.

Using Sliders

As sliders cannot be floated, we have a problem with the existing code. If you remember the way tasks were constructed in the previous tutorial, the action buttons were added to the summary DIV directly with their float styles set to right. This isn't the correct manner of doing things anyway, so we'll fix that first. We'll group the task controls into a single floating element.

    this.summaryControlsDiv = goog.dom.createDom('div', { 'class': 'controls' });
    this.summaryDiv = goog.dom.createDom('div', { 'class': 'summary' }, this.summary, this.summaryControlsDiv);

This break some ugly part of our code, the makeActionButton function is being passed the task DIV and renders the button being created by selecting the appropriate child node. We'll just do a quick fix and correct this issue in the next section.

    button.render(element.childNodes[0].childNodes[1]);

Another more subtle change we need to make is in makeButtons, we must change the order in which the action buttons are created. We must also change some styles and add an entry for the new CSS class.

      .task .summary .controls { float: right; }
      .task .description { background: #eee; border-top: solid 1px #333; }
      
      .taskButton { margin-top: -6px; margin-left: 7px; }
      .editorButton { margin: 0.2em 0.3em 0.5em; }

The following code create a slider, we'll stop event propagation the same way it's done for tasks' action buttons.

mu.tutorial.tasks.Task.prototype.makeSlider = function(task, element) {
    if (task.parent.id == 'taskList') {
        var slider = new goog.ui.Slider;
        slider.createDom();
        slider.render(element.childNodes[0].childNodes[1]);

        slider.addEventListener(
            goog.ui.Component.EventType.CHANGE,
            function() { /* TODO: something... */ });

        goog.events.listen(
            slider.getContentElement(),
            goog.events.EventType.CLICK,
            function(e) { e.stopPropagation(); });
    }
}

We need to call this function from makeDom just before creating the task buttons. To make the page actually show the sliders, we'll add the styles below.

      .goog-slider-horizontal, .goog-slider-thumb {
        overflow: hidden;
        height: 20px;
      }
      
      .goog-slider-horizontal {
        background-color: #ccc;
        display: inline-block;
        position: relative;
        width: 200px;
      }
 
      .goog-slider-thumb {
        background-color: #777;
        position: absolute;
        width: 20px;
      }

Cleaning Up

Earlier, we talked about how bad it was for makeActionButton to have to know where to render the button. We'll take the time to clean that up. If we look back at the code, one thing is obvious, we pass lots of arguments around. That make the code overly functional and this isn't a good approach here. Another thing is that we carefully initialized prototype members for every elements contained in a task and we end up not using them.

For this section I'll skip the code, so use your imagination. First thing to do is to remove the element argument from functions having it. For the controls, we'll use the object's properties, summaryControlsDiv for action buttons and sliders, and editorContainer for editor buttons. Finally, we need to replace the taskDiv local variable in makeDom by a property that we'll name element. We can't use it directly in our callback function as it is a closure and this doesn't point to the task object. We can work around that simply by using a temporary variable, we'll rewrite that code later anyway.

Making the Sliders Do Something

To put the sliders to good use, we'll make them control the priority of a task. We'll first make the tasks display their priorities, to be able to test things manually.

    this.priorityDiv =  goog.dom.createDom('div', { 'class': 'priority' }, this.priority.toString());
    this.summaryDiv = goog.dom.createDom('div', { 'class': 'summary' },
                                         this.summary,
                                         this.summaryControlsDiv,
                                         this.priorityDiv);

We'll also have to add styling for the new priority DIV, it must float to the right and have some padding on the right side. Now, lets add two things to the makeSlider function. Firstly, setting the slider value to the priority of the task currently being created. Secondly, replacing our TODO comment by changing the priority property as well as the content of the priority DIV.

        slider.setValue(this.priority);

        var task = this;
        slider.addEventListener(
            goog.ui.Component.EventType.CHANGE,
            function() {
                task.priority = slider.getValue();
                task.priorityDiv.innerHTML = task.priority;
            });

There's a small bug with this code, can you find it?

If you delete or mark a task as done, then undo that action, you'll see that the resurrected tasks' slider thumb is set at its lowest value. If you insert an alert to display the sliders' value when the task is recreated, you'll see that it's value is correct. Also the slider behave as if the thumb was at the correct position strangely, it can even go into infinite loop. This all points out toward the fact that when we recreate our tasks, they're being added to an invisible element. We'll reorganize our code in the next section to circumvent this issue.

Sorting Tasks

The sliders are now working (with some issues) that's good but it doesn't add much to our little application feature-wise. The obvious use of tasks priority would be to sort them. We'll need to do some major modification to our code to do this though. Currently the tasks add themselves to their containers and appear in whatever order they're being appended. We'll need to create a new object to represent task lists. I won't be showing all the changes made, just the most important ones, you can always refer to the final code if you need more details. First, we'll need to provide a new class name for our TaskList prototype.

goog.provide('mu.tutorial.tasks.TaskList');

mu.tutorial.tasks.TaskList = function(name, data, container) {
    this.name = name;
    this.container = container;
    this.tasks = [];

    for (var i = 0; i < data.length; i++) {
        this.tasks.push(new mu.tutorial.tasks.Task(data[i], this));
    }

    this.sort()
    this.render();
}

It's followed by its constructor which creates, sorts and renders the given tasks, this greatly simplify the makeTasks function. With this modification, we introduced some unwanted duplication of information, our tasks objects already have a reference to their containers. We'll just change that reference for another pointed at the list it's associated to. We'll also change the way we're using the taskLists namespace variable, it will now contains TaskList objects. Now lets see how the render function works.

mu.tutorial.tasks.TaskList.prototype.noTasks = function() {
    this.container.appendChild(
        goog.dom.createDom('h2', { 'class': 'empty' }, 'This list is empty!'));
}

mu.tutorial.tasks.TaskList.prototype.render = function() {
    goog.dom.removeChildren(this.container);

    if (this.tasks.length == 0)
        this.noTasks();
    else
        for (var i = 0; i < this.tasks.length; i++) {
            this.tasks[i].makeDom();
        }
}

It's basically the same code that was previously in makeTasks with the only difference that it clean up its container before adding new elements. We also moved noTasks into the TaskList object for convenience.

Next, some new code. We've got an array of tasks and we'll sort it using the default JavaScript sort function. Google has thought about adding helper functions for arrays including one for sorting, its advantage over the default JavaScript one is that it sorts numbers correctly. Here this sort function wouldn't be helping us as we'll write our own compare function.

mu.tutorial.tasks.defaultCompare = function(t1, t2) {
    return goog.array.defaultCompare(
        t2.priority,
        t1.priority);
};

mu.tutorial.tasks.TaskList.prototype.sort = function() {
    this.tasks.sort(mu.tutorial.tasks.defaultCompare);
}

We'll need to insert and remove tasks from our task lists. As they're always sorted, the Closure Library has two functions to help us: binaryInsert and binaryRemove. These allow for fast manipulation of sorted arrays by using a binary search algorithm.

mu.tutorial.tasks.TaskList.prototype.add = function(task) {
    task.list = this;
    
    goog.array.binaryInsert(
        this.tasks,
        task,
        mu.tutorial.tasks.defaultCompare);
}

mu.tutorial.tasks.TaskList.prototype.remove = function(task) {
    this.container.removeChild(task.element);
    
    goog.array.binaryRemove(
        this.tasks,
        task,
        mu.tutorial.tasks.defaultCompare);

    if (this.tasks.length == 0)
        this.render();
}

The add method is very simple, it just set the current task list as the given task parent and insert it into the array. For the remove method though, it's actually in charge of cleaning up the element of the task being removed. It also ensures to call noTask if there's no task left in that task list.

Next, we'll replace the clickActionButton listener code by a call to a new Task method called moveTo. This method will be in charge of removing the task from its parent list and inserting it into the specified one using the above methods.

mu.tutorial.tasks.Task.prototype.moveTo = function(target) {
    this.list.remove(this);
    mu.tutorial.tasks.taskLists[target].add(this);
}

Yet another change that merit a mention is the switchPanel function which takes a new argument to know what TaskList to render. This could be made cleaner, but we'll keep it that way for this part of the tutorial.

When a user change the priority of a task, we better have them sorted. We'll create a new Task method that will be called when someone stop using the slider. It make use of a new property of the Task object to remember the previous value so as not to reload a task list if the priority didn't changed.

mu.tutorial.tasks.Task.prototype.reload = function() {
    if (this.priority != this.previous_priority) {
        this.list.sort();
        this.list.render();
        this.previous_priority = this.priority;
    }
}

But now we have a problem. When shall we call this function? I've thought about it for some time and came up with a complex and convoluted solution. The correct solution, that is to use a timer and accumulate events, being too much bothersome for the scope of this tutorial, I've decided to settle on a small hack I've found that should do the job.

By the principles of YAGNI, I figured out that the sliders are to heavy to handle. Why would we need four different ways of modifying a task priority? So, from now on, we'll only listen for MOUSEOUT and KEYUP events.

        goog.events.listen(
            slider.getContentElement(),
            [goog.events.EventType.MOUSEOUT,
             goog.events.EventType.KEYUP],
            function() { task.reload(); });

There's still a problem with dragging the sliders, if only we could make that issue disappear.

Invisible Sliders

Sometimes the best style is no styles at all. We must face it, the sliders are ugly. Making them look awesome can take some time for a non-designer like me. While thinking about this problem, I've came up with a solution that would also simplify sliders event handling code. We could make the sliders invisible and put them on top of the priority numbers shown. That doesn't actually fix our issue, we only prevent users from thinking about dragging the slider thumb. This is just a hack and should only be done if you're really short on time. Talking of time, this part is much more longer than I expected. So I'll let you look at the result rather than the code as it's a very simple modification, mostly new DOM elements and CSS.

Less Rendering and More Stability

One last thing, our code is working like a charm and is more than fast enough for the small data set we're testing it with. But in real world situations, there's some bottlenecks that could potentially slow our application down. I'll let the testing for another part, but we'll remove one of those bottleneck.

The most processor hungry function in our code is certainly the render one. Presently, we're calling it every time the priority of a task change. But if the order of tasks don't change, we don't need to render them again. Moreover, the JavaScript sort function isn't stable, two tasks having the same priority could be reordered. So, to kill two birds with one stone a second time, we'll make our own stable sort function. Well, in fact, we'll just take the code from goog.array.stableSort and alter it to return true if the array changed.

mu.tutorial.tasks.TaskList.prototype.sort = function() {
    var arr = this.tasks
    for (var i = 0; i < arr.length; i++) {
        arr[i] = {index: i, value: arr[i]};
    }

    function stableCompareFn(obj1, obj2) {
        return mu.tutorial.tasks.defaultCompare(obj1.value, obj2.value) ||
            obj1.index - obj2.index;
    };
    
    goog.array.sort(arr, stableCompareFn);

    var changed = false;
    for (var i = 0; i < arr.length; i++) {
        if (i != arr[i].index)
            changed = true;
        arr[i] = arr[i].value;
    }
    return changed;
}

Then, it's only a matter of calling render when the sort call returns true in the reload method.

mu.tutorial.tasks.Task.prototype.reload = function() {
    if (this.priority != this.previous_priority) {
        if (this.list.sort())
            this.list.render();
        this.previous_priority = this.priority;
    }
}

Phew, that was a big post, I hope you enjoyed it! I'm still not quite sure about what will be the subject of the next part, if you have any suggestion, drop by a comment.

2010-01-24

Emacs Tip #1

Emacs is a great editor, I won't elaborate on this as there's countless blog posts already celebrating its countless features. After some years of usage, anyone using Emacs end up with a bloated .emacs file that may contains some good tips. I'll be explaining some of the tricks I'm using in a series of posts of which this one is the first.

Web Search Shortcuts

It's a known fact (in the sense that it's a common Intertubes meme) that some people use their computer exclusively through Emacs. I'm not that fond of this editor and prefer to do things like navigating the web with a real browser. Emacs as a way to send URLs to one, using the browse-url function. Feeding an unencoded URL to it could yield a bad request though. To fix that, we can use url-hexify-string from the url library. Here's a simple example of putting these functions to good use.

(require 'url)

(defun google ()
  (interactive)
  (let ((url (if (and transient-mark-mode mark-active)
               (buffer-substring-no-properties
                 (region-beginning)
                 (region-end))
               (thing-at-point 'symbol))))
    (browse-url
      (concat "http://www.google.com/search?"
        (format "q=%s" (url-hexify-string url))))))

This interactive function let us do a simple search with the text under the currently active region or with the symbol found under the cursor if there's no active region. This can be useful, you're no longer forced to copy-paste text from Emacs to your browser.

Next, we could try to send more complex queries to our preferred search engine, which in this case is Google. Standard library documentation such as Java's API specification generally have URLs that are easily queryable. With Google's allinurl keyword, we can make a function to search for a specific class symbol.

(defun search-all-in-url (query site inurl)
  (concat "http://www.google.com/"
    (format "search?q=allinurl:%s+site:%s+inurl:%s&btnI"
      (url-hexify-string query)
      (url-hexify-string site)
      (url-hexify-string inurl))))

(defun web-help (site inurl)
  (browse-url (search-all-in-url (thing-at-point 'symbol) site inurl)))

With web-help, we can now define custom interactive functions. Currently in my .emacs file, I've got two such functions, one for Java and the other for Ruby.

(defun java-help ()
  (interactive)
  (web-help "java.sun.com" "/javase/6/docs/api/"))

(defun ruby-help ()
  (interactive)
  (web-help "www.ruby-doc.org" "/core/"))

Then, it's a simple matter of adding key bindings for our new interactive functions. Personally, I like to have them bound to the lower F* keys.

(global-set-key [(f1)] 'google)
(global-set-key [(f2)] 'java-help)
(global-set-key [(f3)] 'ruby-help)

You might already use these shortcuts for other things (I think that F3 serves to start recording macros) but you certainly still have some unbound key combinations!

Update: I've corrected the second code example that I messed up while writing this post.

2010-01-19

A Recursive Walk on a String

Sometimes you don't pay attention to some details and this lead you right into the wrong direction. That's exactly what happened to me the other day. I had a specific problem I intended to solve for a while, nothing particularly difficult, but an interesting one. After hacking around for an hour, the situation was getting weird and I decided to ask for help on the Clojure Google Group. This post is an account of how that situation arose and what have been done to solve it.

Walking Recursively

The task is nearly as simple as explaining it. I needed a way to apply a function to each strings contained in a tree of nested data structures in Clojure. Not that easy to explain as you see, well you didn't really saw me rewrite the previous sentence a thousand times but you can probably picture it. My first attempt is really naive and leverages the clojure.walk library written by Stuart Sierra. It's a small library that provides functions to perform non-recursive walking of generic Clojure data structures. Lets first have a look at the walk function documentation.

user> (doc clojure.walk/walk)
-------------------------
clojure.walk/walk
([inner outer form])
  Traverses form, an arbitrary data structure.  inner and outer are
  functions.  Applies inner to each element of form, building up a
  data structure of the same type, then applies outer to the result.
  Recognizes all Clojure data structures except sorted-map-by.
  Consumes seqs as with doall. 
nil

It isn't hard to take this function and use it in a recursive one to do the job. Our function to transform strings could go in the inner argument, it's also where we'll add the recursive call. We don't need the outer one, so we will just use the identity function.

(use 'clojure.walk)

(defn recursive-string-walk [f form]
  (walk #(if (string? %) (f %) (recursive-string-walk f %))
    identity form))

It works well but there's an issue with this way of doing things, recursion will end up filling up the stack which will ultimately overflow. For the use I intend to make of this function, it isn't really a big problem, but I always like to take these kind of challenges.

A Buggy Test

The first thing that came to my mind to test the above issue was to create a nested list of arbitrary depth. We can take the iterate function to create an infinite sequence of nested function calls to list, then take and last to fetch the pit we want to test with.

user> (def pit (iterate list "bottom!"))
#'user/pit
user> (take 6 pit)
("bottom!" ("bottom!") (("bottom!")) ((("bottom!"))) (((("bottom!")))) ((((("bottom!"))))))
user> (last (take 6 pit))
((((("bottom!")))))
user> (recursive-string-walk identity (last (take 6 pit)))
((((("bottom!")))))
user> (recursive-string-walk #(str "reached " %) (last (take 6 pit)))
((((("reached bottom!")))))
user> (recursive-string-walk #(str "reached " %) (last (take 1000 pit)))
; Evaluation aborted.
user> (recursive-string-walk #(str "reached " %) (last (take 100 pit)))
((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((("reached bottom!")))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))

Work great! Now lets create an informal test function. This function should take a range and a recursive walk function then will test it using pits starting with the shallowest. This way we can find exactly at which point the overflow happen.

(defn test-walk [walker shallowest deepest]
  (doseq [depth (range shallowest deepest)]
    (println (walker #(str depth " reached " %)
               (last (take depth pit))))))

We'll use it to make sure it works and see which depth our recursive-string-walk can reach.

user> (test-walk recursive-string-walk 400 500)
...
((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((("486 reached bottom!")))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))
(...(; Evaluation aborted.

On my computer, this function cannot go deeper than 486 (I wonder how it would fare on a 486?) which isn't that much. It shouldn't be all that difficult to make a lazy version of that function. It boils down to look for sequences and wrap their recursive calls with lazy-seq.

(defn lazy-recursive-string-walk [f form]
  (walk #(cond
           (string? %) (f %)
           (seq? %)    (lazy-seq (lazy-recursive-string-walk f %))
           :default    (lazy-recursive-string-walk f %))
    identity form))

From my understanding of lazy data structures and the walk library, this version should be able to handle data structures of infinite depth given infinite resources. However the test function didn't seems to be in agreement with me as it was crashing at 828. That's about the time I asked for help and stopped thinking about this problem for a while.

Lazy Zippers

A week-end passed by and in the meantime, some people from the group tried to help me, yet nobody found what was the real problem. That would have been miraculous given the actual source of the bug. Furthermore, I was now more convinced than before than there was no bug in the lazy version. Tom Hicks provided a lazy version of walk, which was behaving in the same way. Laurent Petit also suggested that I try to write a version using zippers.

It was an intriguing suggestion, I never used zippers but heard often about them. Additionally, that day, MarkCC from Good Math, Bad Math just posted an article about them. So I learned what they are and how to use them. I must say this is a very neat functional programming technique, really worth the time to learn. It's not that hard, Mark has done a great job explaining it. I won't go into the details, but this version is quite easy to understand when you know that the next function do a depth-first traversal automatically.

(require '[clojure.zip :as z])

(defn lazy-recursive-string-walk-with-zipper [f form]
  (loop [loc (z/seq-zip form)]
    (if (z/end? loc)
      (z/root loc)
      (recur (z/next
               (if (string? (z/node loc))
                 (z/replace loc (f (z/node loc)))
                 loc))))))

To my surprise, the test function still overflowed the stack using this version. And it could simply not throw the StackOverflowError as it executes in a loop. Maybe the error wasn't coming from the lazy walk function after all.

So Where's The Bug?

I've been negligent about one thing, I didn't look closely enough at the stack traces. The one thrown by recursive-string-walk happen to be completely different than the one thrown by lazy-recursive-string-walk. Here's both of them one after the other.

No message.
  [Thrown class java.lang.StackOverflowError]

Restarts:
 0: [ABORT] Return to SLIME's top level.

Backtrace:
  0: clojure.lang.RT.boundedLength(RT.java:1128)
  1: clojure.lang.RestFn.applyTo(RestFn.java:135)
  2: clojure.core$apply__4370.invoke(core.clj:436)
  3: clojure.walk$walk__7942.invoke(walk.clj:43)
  4: recursive_string_walk$recursive_string_walk__4137.invoke(NO_SOURCE_FILE:1)
  5: recursive_string_walk$recursive_string_walk__4137$fn__4139.invoke(NO_SOURCE_FILE:1)
---
No message.
  [Thrown class java.lang.StackOverflowError]

Restarts:
 0: [ABORT] Return to SLIME's top level.

Backtrace:
  0: clojure.lang.RT.assoc(RT.java:666)
  1: clojure.core$assoc__4259.invoke(core.clj:146)
  2: clojure.lang.AFn.applyToHelper(AFn.java:179)
  3: clojure.lang.RestFn.applyTo(RestFn.java:137)
  4: clojure.lang.Ref.alter(Ref.java:174)
  5: clojure.core$alter__4907.doInvoke(core.clj:1542)

The actual source of the exception is the print function that is recursive on nested sequences. This make sense, why would someone want to print such monsters. Because of that inattention I lost time searching for an imaginary bug. Only a small modification is needed to make our test function working.

(defn get-bottom [p]
  (loop [p (first p)]
    (if (seq? p) (recur (first p)) p)))

(defn test-walk [walker shallowest deepest]
  (doseq [depth (range shallowest deepest)]
    (println (get-bottom
               (walker #(str depth " reached " %)
                 (lazy-seq (last (take depth pit))))))))

We simply search for the bottom item and print it, this will also make the test output clearer. We can now see what our original walk function can handle.

user> (test-walk lazy-recursive-string-walk 100000 100001)
...
; Evaluation aborted.
---
Java heap space
  [Thrown class java.lang.OutOfMemoryError]

Well, the heap too has its limit! We would need to augment the JVM heap memory size to be able to run the previous example. Lets reduce our ambitions and do some benchmarking instead.

user> (time (test-walk lazy-recursive-string-walk 10000 10001))
10000 reached bottom!
"Elapsed time: 25.138697 msecs"
nil
user> (time (test-walk lazy-recursive-string-walk-with-zipper 10000 10001))
10000 reached bottom!
"Elapsed time: 122.691972 msecs"
nil

As you see, the version using clojure.walk is much faster. That's perfectly understandable considering the more complex concept of zippers which have multiple uses aside walking trees. As both versions are fully lazy, the first one is the clear winner here.

What Can We Use This For

After all these trials and tribulations, you may wonder what use I may have for such a function. It's to help create functions or macros that perform string replacement on all strings in a Clojure form. This can be useful with Compojure while deploying your application to a different root path than "/" for example. The idea come from my experience with ASP.NET in which URLs can contain a character (~) to denote the web site root path.

(use 'clojure.contrib.str-utils)

(defn expand* [expansions string]
  (reduce (fn [s [regex replacement]]
            (re-gsub regex replacement s))
    string (partition 2 expansions)))

(defn expand-form [url form]
  (recursive-string-walk (partial expand* [#"^~" url]) form))

Having the same functionality in a macro would be nice too.

(defmacro with-expansions [expansions & body]
  `(do ~@(recursive-string-walk (partial expand* expansions) body)))

(defmacro expand [url & body]
  `(with-expansions [#"^~" ~(str url)] ~@body))

To put this to use I've taken a function written by James Reeves a few months ago during a discussion on the Compojure group. A function with the same name (with-context) already exists in Compojure though, so we'll rename it to with-root-path. The way it work is by binding the given context to a var and then we can use a function called url to add the context so that internal links still work once the application is deployed elsewhere. Using the expand-form function on html's arguments can reduce repetition and make the code less cluttered if there's a lot of such URLs.

(use 'compojure)

(declare *context*)

(defn with-root-path
  [ctx & route-seq]
  (let [handler (apply routes route-seq)
         pattern (re-pattern (str "^" ctx "(/.*)?"))]
    (fn [request]
      (if-let [[_ uri] (re-matches pattern (:uri request))]
        (binding [*context* ctx]
          (handler (assoc request :uri uri)))))))

(defn url [path]
  (str *context* path))

(defn html-expand [& trees]
  (apply html (expand-form *context* trees)))

(defroutes test-routes
  (with-root-path "/foo"
    (GET "/"    (html-expand [:h1 "test"] [:p "~/"]))
    (GET "/bar" (html [:h1 "test"] [:p (url "/bar")]))
    (ANY "*" (page-not-found))))

(defn start-test-server []
  (run-server {:port 8000} "/*" (servlet test-routes)))

The first route use a special version of the html function, it's only a slight improvement but I really like that way of doing things. There's plenty of other possible use cases though, but that should be enough for this post.

P.S.: I know the title isn't correct, but it sounds way better than A Recursive Walk on Strings Contained in Arbitrary Nested Data Structures.

2010-01-12

Documenting Clojure Code

Clojure is a truly amazing language and still a very young one. There is already a substantial quantity of documentation available for Clojure and some of its libraries. Yet, when talking about documentation, some is not enough. Sure, there's the REPL with all its bells and whistles, but sometimes I prefer to navigate through a well formatted API document using a browser. There's already some solutions for this, one on which I worked for the defunct Compojure website (which was based on similar code for Clojure) and another more serious attempt in clojure-contrib, gen-html-docs by Craig Andera. This library only output HTML and there arise a small (but annoying) problem: not all projects have their documentation formatted in HTML. For example, the new Compojure website runs on DokuWiki which have its own syntax and Gitorious integrated Wiki, used for ClojureQL, is using Markdown. If only every Wiki engine would understand Creole!

clj-doc

To be able to solve the above problem, I decided to start a new project called clj-doc. For now it's pretty much just a weekend coding session, but I'll continue to work on it so that one day it will be the best tool available to generate documentation for Clojure code. The basic usage is done through the gen-doc macro.

user> (use 'clj-doc)
nil
user> (gen-doc clj-doc.utils)
("<!DOCTYPE html ...Documentation for clj-doc.utils...")

This gives us a sequence of strings being the rendered output of each namespaces given using the default markup language. Namespaces could also be grouped using nested sequences, for now clj-doc support only one level of nesting. Another notable feature is that namespace groups can be specified using regular expressions. Here's an example demonstrating all these methods at once.

user> (use 'clojure.contrib.pprint)
nil
user> (pprint (gen-doc clj-doc
                       (clj-doc.markups clj-doc.generator clj-doc.utils)
                       #"clj-doc\.markups\."))
("<!DOCTYPE html ...Documentation for clj-doc..."
 "<!DOCTYPE html ...Documentation for clj-doc.markups, clj-doc.generator, clj-doc.utils..."
 "<!DOCTYPE html ...Documentation for clj-doc.markups.creole, clj-doc.markups.dokuwiki, ...")

Writing to Files

Usually you would want to write the documentation for your library to a file. I've added basic support for this with the gen-doc-to-file macro. It behaves exactly as gen-doc but writes each of the resulting documentation strings to a file.

user> (gen-doc-to-file "~/test.html" clj-doc)
nil

If you use more than one group of namespaces, this macro will write them into multiple files that will be numbered starting with zero by appending a number before the last dot if there's one, at the end otherwise. I intend to make this argument a format string where you can specify more complex filenames in a next version. So, if you got any ideas about how to design or implement such feature, I encourage you to add a comment to this post.

Supported Markups

Up until now, we've only been using the default output markup. The main point of this library being generating something else than HTML, we better have a way to use another markup language. Before talking about how to do that, lets look under the hood.

(ns clj-doc.markups.creole
  "Creole markup."
  (use clj-doc.markups))

(defmarkup
  #^{:doc "Creole markup."}
  creole
  :title        #(str "\n= "     %     " =")
  :namespace    #(str "\n== "    %    " ==")
  :section      #(str "\n=== "   %1  " ===\n" %2)
  :var-name     #(str "\n==== "  %  " ====")
  :var-arglist  #(str "\n**" % "**\\\\")
  :var-doc      #(str "\n\n" %))

This is the current implementation for the Creole Wiki markup generator. All markups are defined in the same way, they're basically only struct maps, for more details see the clj-doc.markups namespace. The library currently provides four output methods. We've already seen two of them, here's the complete list:

  • creole
  • dokuwiki
  • html-simple
  • markdown

To define your own markups, you can take one of the implementations, modify it and see how it work out, the keywords should be explicit enough. The library only look for markups into namespaces starting with "clj-doc.markups." though, but that's just a temporary limitation, eventually you'll be able to pass a markup struct map as an option.

The Option Map

The option map is an optional argument that must be specified before any namespaces. It presently only support two keywords, :markup and :separated-by. The first letting you choose the output method and the second used to change the default page separation behavior.

user> (gen-doc {:markup creole} clj-doc.markups.creole)
("\n= Documentation for clj-doc.markups.creole =\n== clj-doc.markups.creole namespace ==\n=== others ===\n\n==== creole ====\n\nCreole markup.")

The :separated-by option can only take one value, namespace, which makes the macros separate pages between namespaces instead of groups of them.

user> (pprint (gen-doc {:markup markdown :separated-by namespace} #"clj-doc\.markups\."))
("\n#Documentation for clj-doc.markups.creole..."
 "\n#Documentation for clj-doc.markups.dokuwiki..."
 "\n#Documentation for clj-doc.markups.html-simple..."
 "\n#Documentation for clj-doc.markups.markdown...")
nil

The Future

This is a very early release done the worse is better way, so it's still a pretty bare-bone tool. I'll be taking a pause this week to work on more (hopefully) profitable projects, so in the meantime I'll be pleased to ear about any suggestions to improve the way clj-doc works.

Links

2010-01-11

Git Tip #1

Changing Filename Case Under Cygwin

When using Git on Windows XP, you're confronted with a basic problem, case changes aren't accounted for while renaming files as Windows internals doesn't have a clue about the difference between lower and upper case. You can actually change the casing, but you can't have two files using the same name with different casing for example. This behavior obviously affect Git, which cannot detect filename case changes.

$ git init Initialized empty Git repository in /home/budu/tmp/.git/

$ touch foo

$ git add .

$ git commit -m 'test' [master (root-commit) 5a1fbb3] test 0 files
changed, 0 insertions(+), 0 deletions(-) create mode 100644 fo

$ mv foo FOO

$ l total 0 drwxr-xr-x+ 1 budu None 0 Jan 11 18:15 .git/ -rw-r--r-- 1
budu None 0 Jan 11 17:59 FOO

$ git status # On branch master nothing to commit (working directory
clean)

This is an unfortunate situation which can be easily resolved by using a simple trick. The first thing you need to do is to copy the file you want to modify to a different temporary name. Then you must use git rm on the original file and finally rename the temporary file to the new name with changed casing.

$ cp FOO bar

$ git rm foo rm 'foo'

$ mv bar FOO

$ git add .

$ git status # On branch master # Changes to be committed: # (use "git
reset HEAD ..." to unstage) # # renamed: foo -> FOO #

One thing to note here is that when calling git rm you have to use the original file name if its casing is already changed.

About Me

My photo
Quebec, Canada
Your humble servant.