Macros and Backquote Macros make up an important aspect of Common Lisp. Experienced lisp programmers use macros a lot. They are also primary vehicles by which various extensions to the language are considered, experimented with, and adopted. Writing macros is easy in lisp because its data and programs are very much alike, and it is easy to write programs that write programs, which are what macros are! Think of adding a new kind of loop, called WHILE loop, to lisp. Its syntax is (WHILE ). It should work as follows: 1. Evaluate . 2. If evaluates to TRUE, is evaluated and control goes to step 1. 3. Else the value of in the previous iteration is returned. (if there is no previous value, return NIL). Consider writing it as a function using defun. (defun while (condition body) (do ((temp nil body)) ((null condition) temp) )) How to call this? >(setf x 3) 3 >(while (> x 0) (progn (print x) (setf x (1- x)))) What happens now? The arguments of while are evaluated. In this case (> x 0) evaluates to T. The body, (progn (print x) (setf x (- x 1)))) is evaluated next. It prints 3 and evaluates to 3-1 = 2. Then While will be called with arguments T and 2. (while T 2) will execute the loop (do ((temp nil 2)) ((null T) temp) ) This loop will execute forever, since the condition is always T! The problem was that the arguments of while, the condition and the body, were executed before entering the loop. We have to stop them from being evaluated before the body of while is evaluated! One could do this by quoting the arguments of while. >(setf X 3) 3 >(while '(> X 0) '(progn (print x) (setf x (- x 1)))) With the above call, the body of the while executes: (do ((temp nil '(progn (print x) (setf x (- x 1))))) ((null '(> x 0)) temp) ) temp <- nil (null '(> x 0)) <- nil temp <- (progn (print x) (setf x (- x 1))) (null '(> x 0)) <- nil temp <- (progn (print x) (setf x (- x 1))) (null '(> x 0)) <- nil . . . The problem now is that the and are not being evaluated during the loop. One solution is to surround them with evals. So the function now looks like: (defun while (condition body) (do ((temp nil (eval body))) ((null (eval condition)) temp) )) This works correctly when called with quoted arguments. >(setf X 3) 3 >(while '(> X 0) '(progn (print x) (setf x (- x 1)))) 3 2 1 0 But the arguments of the function still need to be quoted when called. Macros allow you to call them without quoting their arguments, because the macro is executed on the UNEVALUATED arguments during the first phase. This phase is called the translation phase. After this is done, the result of translation is again evaluated. Macros are written using defmacro instead of defun. You can write the while macro as follows. (defmacro while (condition body) (list 'do (list (list 'temp nil body)) (list (list 'null condition) 'temp))) Suppose the above macro is called with: >(setf X 3) 3 >(while (> X 0) (progn (print x) (setf x (- x 1)))) The above will be translated as follows in the first phase. >(do ((temp nil (progn (print x) (setf x (- x 1))))) ((null (> x 0)) temp)) Now the above loop is evaluated, resulting in the behavior we saw before. 3 2 1 0 One important aspect of macros is that the translation is done at the compile time since translation phase does not involve evaluating the arguments! This means that run-time efficiency does not suffer at all by writing macros. At run time, it is as if we never wrote the macro and just used the macro body. Macros allow us to take advantage of a repeated structure in the code without paying for it in efficiency -- unlike what happens when we write functions. But the above macro is messy. Why not just write (defmacro while (condition body) '(do (( temp nil body)) ((null condition) temp))) What happens if we call the above macro with >(setf X 3) 3 >(while (> X 0) (progn (print x) (setf x (- x 1)))) After translation, it gives (do (( temp nil body)) ((null condition) temp))) During evaluation neither condition nor body have values, and hence it does not evaluate. Error: Attempt to take the value of the unbound variable `CONDITION'. [condition type: UNBOUND-VARIABLE] The problem is that defmacro is not able to go inside the quotes and modify things, i.e., condition and body. We need a mechanism to selectively evaluate forms inside a quoted expression. We cannot do it with the ordinary quote ('), but we can do it with the backquote (`). Backquote has the same effect as quote, but the forms inside can be selectively evaluated by preceding them with a comma. >(setf x 3) 3 >`(She is ,X years old) (SHE IS 3 YEARS OLD) Backquote is great for template filling! >(setf x (READ))Johnson JOHNSON >`(Good Morning! Mr. ,X) GOOD MORNING! MR. JOHNSON Comma can be used to evaluate any forms, not just variables. >(SETF X '(ORANGES APPLES)) (ORANGES APPLES) >`(I like ,(first x) but not ,(second x)) (I LIKE ORANGES BUT NOT APPLES) Finally, a comma should only be used inside a backquote. Using it without a backquote gives an error. Now we can write our WHILE macro using backquote and comma. (defmacro while (condition body) `(do ((temp nil ,body)) ((null ,condition) temp))) >(setf X 3) 3 >(WHILE (> X 0) (PROGN (PRINT X) (SETF X (1- X)))) Note that 1- is a macro itself that subtracts 1 from its argument. (1- X) is equivalent to (- X 1). After translation: (DO ((TEMP NIL (PROGN (PRINT X) (SETF X (1- X))))) ((null (> X 0)) temp)) After evaluation: 3 2 1 0 Backquote also allows comma followed by @sign (,@) to evaluate the following expression. The expression must then evaluate to a list, which is spliced into the parent list. > (setf x '(apples and oranges)) (APPLES AND ORANGES) >`(,x DO NOT MIX) ((APPLES AND ORANGES) DO NOT MIX) >`(,@x DO NOT MIX) (APPLES AND ORANGES DO NOT MIX) Defmacro also allows optional, key, and rest parameters. Suppose we want to generalize our WHILE loop so that the body might consist of any number of forms. So the while loop might look like: (WHILE ... ) This can be easily done using the &rest parameter and ,@. (defmacro while (condition &rest body) `(do ((temp nil (progn ,@body))) ((null ,condition) temp))) Now we can say: >(setf x 3) 3 >(while (> X 0) (print X) (setf X (1- X))) After translation: (do ((temp nil (progn (print x) (setf X (1- X))))) ((null (> X 0)) temp)) which evaluates as before. Although the above macro works correctly in many cases, it still has a subtle bug. Consider what happens when you call it as follows. >(setf temp 3) 3 >(while (> temp 0) (print temp) (setf temp (1- temp)) 'done) This gets translated to: (do ((temp nil (progn (print temp) (setf temp (1- temp))) 'done)) ((null (> temp 0)) temp)) Note that the macro variable temp, and the variable temp in the call have fused into one! Not surprisingly, the macro does not work as intended, because temp gets set to NIL the first time and is compared with a > sign, which gives error. This comes as a nasty surprise to the user of the WHILE loop, who in all probability is going to be a different person than the person who wrote the while macro. Even if he or she is the same person, this kind of bugs are extremely difficult to catch. We say that the macro variable "temp" occurs freely inside the body of the macro, because it is neither a parameter nor a local variable. Such free variables cause what are called "variable captures", an example of which we have just seen. This problem can be avoided by creating a new variable name which is unique and has no chance of occurring anywhere else. Lisp has a function called gensym, which creates just such variables. >(gensym) #:G12 #:G12 is a uniquely named symbol that is guaranteed to be not eq to any other symbol. Hence, there is no danger of name conflicts with the user program. Now, how do we write our WHILE macro using gensyms? First we embed our previous macro inside a LET. The LET has a local variable which holds a freshly generated gensym. The gensym should be substituted in the place of the variable temp. In other words, the macro looks as follows: (defmacro while (condition &rest body) (let ((var (gensym))) `(do ((,var nil (progn ,@body))) ((null ,condition) ,var)) )) Now, how does the following call get translated? >(setf var 3) 3 >(while (> var 0) (print var) (setf var (1- var)) 'done) Translation: (do ((#:G13 nil (progn (print var) (setf var (1- var))) 'done)) ((null (> var 0)) #:G13)) Which executes as follows: 3 2 1 DONE Note that the local variable var in the macro definition does not conflict with the variable var in the macro call, because it is declared inside the let, and is allocated space separately during the macro expansion. It is more difficult to debug macros than functions, because macros have two phases -- translation and evaluation. Only the results of the second stage are visible when a macro is called. We can see the results of the translation phase, however, by using the function macroexpand, as follows: > (macroexpand '(while (> var 0) (print var) (setf var (1- var)) 'done)) (LET ((#:G556 NIL)) (DECLARE) (BLOCK NIL (TAGBODY #:G557 (COND (# #)) (TAGBODY) (PSETQ #:G556 (PROGN # # 'DONE)) (GO #:G557)))) T This does not appear anything like what we expected, because macroexpand expands all macros to the lowest level. Notice the use of GO statement here! It is a primitive construct that is often used to implement loops as macros. To stop it expanding to the lowest level, we can use macroexpand-1. > (macroexpand-1 '(while (> var 0) (print var) (setf var (1- var)) 'done)) (DO ((#:G559 NIL (PROGN (PRINT VAR) (SETF VAR (1- VAR)) 'DONE))) ((NULL (> VAR 0)) #:G559) ) Macroexpand and macroexpand-1 are very useful to debug macros! Use them to see how your macros are getting translated before calling them. Destructuring ------------- Sometimes the parameters of a macro are themselves in lists as in all the loops such as DO, DOLIST, and DOTIMES. For example, consider writing a FOR loop whose syntax is as below: (FOR (

) ) To write this as a macro, we have to pick out the and the start and stop forms from the first list. We could start this macro as (defmacro FOR (params &rest body) ... and use ,(first params) ,(second params) etc. for , , etc. This works, but is quite untidy and unreadable. It would be nice to be able to refer to the individual contents of the parameter list by names -- so that we can use them in the macro by name. This can be done by the facility called "destructuring". Basically, the parameters of the macro are allowed to be lists, and the arguments of the macro call are parsed according to this parameter structure. Destructuring allows us to write the above macro as: (defmacro FOR ((var start stop) &rest body) ... and use ,var ,start and ,stop inside the macro body. Macros vs. Functions -------------------- What are the reasons for using a macro as opposed to a function? a) Sometimes we do not want to evaluate the arguments before evaluating the function. We can only have this ability with macros, because functions always evaluate their arguments before the body. E.g., (SETQ X 3) SETQ cannot be a function, because, if it were, it first wants to evaluate X. If X already does not have a value, we cannot give it a new value using setq, if it is defined as a function! We can implement setq as a macro using setf as follows: (defmacro my-setq (var form) `(setf ,var ,form)) b) Another reason is run time efficiency. If a function is small and inexpensive, the overhead of preparing for the function call, by allocating stack space and updating the pointers, dominates the cost of executing the function itself. But small functions are sometimes useful for readability and elegance. With macros, we can have the cake and eat it too! We can write small macros without fearing the overhead, because there is no overhead at run time! The macros are expanded at the compile time, so that the run time efficiency does not suffer. E.g., suppose we have a books data base that we talked about earlier, in which each book is represented as a list. The first item of the book is its title. So book-title is the following trivial function: (defun book-title (book) (first book)) The overhead of setting up a function book-title is not worth the cost, because it dominates the cost of executing it. But on the other hand, using first instead of book-title as our function defeats the purpose of data abstraction. By making book-title a macro, we can win both ways! (defmacro book-title (book) `(first ,book)) However nice this use of macros is, using macros for run time efficiency of small functions is becoming less and less important. The reason is that Common lisp provides a facility by which such functions can be declared `inline'. This tells the compiler directly that these functions should be compiled into the code of the calling functions. E.g., (proclaim '(inline book-title)) would have had the same effect as making it a macro. However compiler writers are free to ignore such proclamations, and hence all compilers are not guaranteed to take this seriously. But there are situations where the run time efficiency is more than a matter of reducing the over-head time relative to function execution time. Macros give more flexibility to the programmer by the ability to push things from run time to compile time. E.g., The average of a set of numbers can be easily computed with a function. (defun avg (&rest args) (/ (apply #'+ args) (length args))) But it can also be computed with the following macro: (defmacro avg (&rest args) `(/ (+ ,@args) ,(length args))) If we do it as a function, the function length is called each time it is executed. But as macro, length is called only at the time the macro is expanded -- i.e., at the compile time. We know the length of the list at the compile time, because it is the number of arguments of the macro call! (avg 50 10 20) will be expanded by the above macro into : (/ (+ 50 10 20) 3) There is no call to length at the compile time. Note that the above would not have worked if the argument to avg is a list, rather than the numbers themselves. In that case, the length of the list is not available at the compile time and must be computed at the run time. c) A third use of macros is evaluating the arguments conditionally or iteratively. Functions cannot do this, because they always evaluate their arguments only once before evaluating their body. We have seen this use of macros in implementing the WHILE and FOR loops. Conditional forms such as IF, WHEN, WHILE, and COND are similar. They are implemented using the GO form we have seen before in typical implementations. d) A fourth use of macros is in creating new environments, as in LET and LET*. The following implements LET as a lambda expression. (defmacro my-let (bindings &rest body) `((lambda ,(mapcar #'(lambda (x) ; collect all the variables (if (consp x) (first x) x)) ; of the let and make them bindings) ; parameters of lambda, ,@body) ; substitute the body, ,@(mapcar #'(lambda (x) ; apply the lambda (if (consp x) (second x) nil)) ; expression to the bindings))) ; values of init forms ; of let. This macro expands the following (my-let ((X 1) (Y 2) (Z 3)) (print (* X X)) (print (* Y Y)) (print (* Z Z))) to ((LAMBDA (X Y Z) (PRINT (* X X)) (PRINT (* Y Y)) (PRINT (* Z Z))) 1 2 3) When executed, it does the correct thing and prints: 1 4 9 9 Recursive macros ---------------- Macros can be recursive. But you should be careful to recurse only on things which are available at compile time, e.g., argument lists. Recursive macros can cause unforeseen infinite loops during compilation as the following example illustrates. Consider implementing nth as a recursive macro. Suppose you write it as: (defmacro my-nth (n list) `(if (= ,n 0) (first ,list) (my-nth (- ,n 1) (rest ,list)))) How does it expand (my-nth K X)? Expanding the first recursive call, we get (IF (= K 0) (FIRST X) (MY-NTH (- K 1) (REST X))) Since MY-NTH is itself a macro, it will be expanded further. (IF (= K 0) (FIRST X) (IF (= (- K 1) 0) (FIRST (REST X)) (MY-NTH (- (- K 1) 1) (REST (REST X))))) This will be expanded into even longer expressions by expanding MY-NTH until the compiler runs out of memory! An innocent looking macro can thus cause the compiler to an irrecoverable loop. Note that if the values of K and X are available, the interpreter would work fine, because after the first expansion, the interpreter attempts the second expansion only after it evaluates the condition part of the IF loop and it turns out to be NIL. We can fix the MY-NTH macro by translating it into an iterative loop. We can illustrate a good use of recursion in macros using the example of OR. We call the macro MY-OR. One way to translate MY-OR is by using COND. (MY-OR exp1 exp2 ... expn) will be translated to (COND (exp1) (exp2) ... (expn)) Note that by the semantics of COND, if the first cond clause whose condition evaluates to a non-null value does not have any consequents, it returns the value of its condition as its value. Hence the previous COND clause returns the value of the first expression that evaluates to a non-null value. The above translation can be done by the following simple macro that uses mapcar to surround each of the arguments in parentheses and splice it into an empty COND statement. (defmacro my-or (&rest exps) `(cond ,@(mapcar #'list exps))) >(macroexpand-1 '(my-or nil (+ 2 3) 6)) (COND (NIL) ((+ 2 3)) (6)) >(my-or nil (+ 2 3) 6) 5 Note that my-or has the lazy evaluation property of OR! But MY-OR is not recursive. How do we implement it recursively? The following shows a completely different way of implementing MY-OR, and it is recursive. (defmacro my-or (&rest args) (if (null args) nil (let ((sym (gensym))) `(let ((,sym ,(first args))) (if ,sym ,sym (my-or ,@(rest args))))))) Note the use of let (at the top level) and gensym to avoid variable capture. It is necessary to compute (first args) once and store it in the gensym variable, because otherwise (if ,sym ,sym ...) would have to be coded as (if ,(first args) ,(first args) ...) which results in evaluating the first argument twice! This is not only inefficient but could result in unintended bugs if the expressions have side effects. The above macro expands (my-or (> 4 3) (< 4 3)) into (LET ((#:G588 (> 4 3))) (IF #:G588 #:G588 (LET ((#:G589 (< 4 3))) (IF #:G589 #:G589 NIL)))) which returns T when executed. Summary ------- 1. Macros are useful devices to abstract and take advantage of repeated structure in the code. 2. Macros are expanded before evaluating the arguments, and the result of macro expansion (first evaluation of macro) is again evaluated. 3. Macro expansion takes place at compile time. So there is no cost for using macros at run time. 4. Backquote is like a quote except that everything inside it which is preceded by comma is evaluated. Backquote and comma are convenient devices used to write easily readable macros. 5. You should be careful to avoid variable capture when you write macros. Avoiding free variables and using gensym to generate new variable names are safe ways for doing this. 6. Macroexpand and macroexpand-1 show the translated macros and are useful in debugging. 7. Destructuring is useful for writing macros that take arguments which are structured into parameter lists. 8. Macros are chosen over functions for many reasons. a) When arguments should not be evaluated before the function bodies. b) Run-time efficiency. c) To implement conditionals and iterative statements. d) To create new environments. 9. Recursive macros should only recurse over unevaluated forms. They lead to infinite loops at compile time if recursion is over variables whose values are not available at compile time.