Macros

Jonathan Bachrach MIT AI Lab

Macro

• Syntactic extension to core language • Defines meaning of one construct in terms of other constructs • Their declarations are called

macro definitions

• Their uses are called

macro calls

unless Example

• Definition: (ds (unless ,test ,@body) `(if (not ,test) ,@body)) • Call: (unless done? (go)) • Expansion: (if (not done?) (go)) • Evaluation: …

Macro Motivation

• “From now on, a main goal in designing a language should be to plan for growth.” – Guy Steele, “Growing a Language”, OOPSLA-98, invited talk • Power of abstraction where neither functional nor object abstraction will suffice, affording: – Clarity – Concision – Implementation hiding • People are lazy – do the right thing

Mathematical Notation

• Continually inventing new abbreviations appropriate for expressing new concepts • Many mathematical concepts become usable only after someone invents a suitable notation to express them.

Proper Macros

• Macros control the evaluation of their arguments • Four basic types of control: – Transformation – Binding – Conditional Evaluation – Multiple Evaluation pick apart before eval vars directly in code e.g., unless e.g., do

Useful Macros

• With macros – (with-open-file (s “dump” :direction :output) (princ 99 s)) • Conditional evaluation – (nif x ‘p ‘z ‘n) • Iteration – (forever (yell)) • Anaphoric macros – (aif (big-long-calculation) (foo it))

Improper Macros

• General Rule: It is inelegant to use a macro where a function would do.

– Example: 1+ versus while • Specific Rule: don’t use macros for inlining.

• Caveat: sometimes macros are used when function calls would be awkward (e.g., closures)

Quasiquote

• WSYWIG specification of lists – Permits the writing a template of desired list with substitutions in place of the variable parts of the list • Evaluation – yields a list – turned off by default – turned on with unquote • Unquote ,

expr

• Splicing ,@

expr

- inserts value of

expr

- splices in value of

expr

• Example (let ((x 1) (y ‘(2 3))) `(,x ,@y)) ==> (1 2 3)

Pattern Matching

• An intuitive and accessible way of expressing a parser – Permits the writing out of the general shape of the expected input, but with pattern bindings in place of the variable parts of the code • Proto uses quasiquote notation for patterns – Default is to match exactly and recursively – Unquote ,

name

matches one expr bound to

name

– Splicing ,@

name

binds them to

name

matches zero or more exprs and • Example – (mif ((,x ,@y) ‘(1 2 3)) (lst x y)) ==> (1 (2 3))

for Example

(for (e ‘(1 2 3)) (println out e)) ==> (let ((c ‘(1 2 3))) (rep loop ((s (ini-state c))) (unless (fin-state? S c) (let ((e (cur-elt s c))) (println out e) (rep (nxt-state s c))))))) (ds (for (,name ,collection) ,@body) `(let ((c ,collection)) (rep loop ((s (ini-state c))) (unless (fin-state? s c) (let ((,name (cur-elt s c))) ,@body (rep (nxt-state s c)))))))

TestSuite Macros

• Want test-suite that’s – Concise to write tests – runs to completion – records intelligent failure messages (test-group whales (test-eqv (moby) 37) …) (ds (test-eqv ,input ,output) `(do-test-eqv *group-name* ,(to-str input) (fun () ,input) (fun () ,output))) (dm do-test-eqv (group-name doc input-thunk output-thunk) (unless (= (input-thunk) (output-thunk)) (sig (cat group-name ": " doc))))

Declarative Data

• Defining tables can be awkward • (deftab t (‘jb => ‘boy) (‘kk => ‘girl))

Declarative Data Examples

f(list(x, y, z)); Studies { course Math101 { title “Mathematics 101”; 2 points fall term; } … exclusions { Math101 <> MathA; Math102 <> MathB; } prerequisites { (Math101, Math102) < (Math201, Math202, Math204); (CS101,CS102) < (CS201, CS203); } … }

Compilation

• Parse into sexprs • Recursively expand macros top-down • Create IR • Optimize IR • Emit code

Scope of Macros

• Macros usually are limited to known positions • In lisp, – Particular forms • Head named lists • Symbols – In particular contexts • Avoids macro calls in parameter specs, let bindings, …

Evaluation with Macros

• Preparation for Macros – (run (prepare

expression

)) • Preparation can be split into many phases – Lexical – Syntactic e.g., read macros • Good design dictates that – run and prepare are strictly separated – Their relation is clarified – And most importantly that experiments can be repeated

Preparation

• Multiple worlds – Preparation produces a result often stored in a file – Preparation and running typically execute in distinct processes – Expression being prepared is only means of communicating between preparer and evaluator • Unique world – All computations cohabit same memory – Communication can be neither limited nor controlled

Exogenous Mode

• macroexpand is provided independently of expression to prepare, for example, by preparation directives: (dm prepare (expression directives) (let ((macroexpand (generate-macroexpand directives)) (really-prepare (macroexpand expression)))) • Macro expansion might either occur as – A cascade using piping – A synthesis of a new compiler • Ensures separation between macro expansion and evaluation • No connection between language of macro expander and the language that is being prepared

Endogenous Mode

• All information necessary for expansion must be found in the expression to prepare: (dm prepare (expression) (really-prepare (macroexpand expression))) • Macro expansion algorithm preexists – Finds macro definitions as well as calls – Use eval rather than invent a special language • Complicated because requires an evaluator inside the macro expander – Must not confuse language for writing macros with the language that is being prepared – There are actually two different evaluators or at least evaluation contexts

Proto’s Endogenous Macros

• Can side-effect macro evaluator with ct form – (ct (dm map-boot-forms (f) …)) special • ct forms can then be used during the expansion of subsequent macros – (ds (define-parents) – `(seq ,(map-boot-forms (fun (x) …)))) • ct forms can be stripped from resulting image if in compilation setting • Experiments can be more faithfully repeated because ct world is insulated from rt world

Macro Expansion

• Classic Mode – Result returned by expander can include more macro calls – Result is re-expanded until it no longer contains any macro calls • Expansion Passing Style – Expander must return an expression without macro calls – Pass expander macroexpand function as argument – Easy to express local macros

Macro-Defining Macros

• Patterns in code signal need for abstraction – Same applies to macros • Example: (ds (mvbind ,@args) `(multiple-value-bind ,@args)) (alias mvbind multiple-value-bind) (ds (mvbind ,@args) (let ((name ‘multiple-value-bind)) `(,name ,@args)) `(ds (,short ,@args) (let ((name ‘,long)) `(,name ,@args)) `(ds (,short ,@args) `(,’,long ,@args)) (ds (alias ,short ,long) `(ds (,short ,@args) `(,’,long ,@args)))

Self Generating Code Quote

Fragment f = #{ #{ Fragment f = #{ ??f }; ?f; }; }; #{ Fragment f = #{ ??f }; ?f; }; Based on Mike McMahon’s self generating quasiquote solution published in Alan Bawden’s quasiquote paper.

(let ((let '`(let ((let ',let)) ,let))) `(let ((let ',let)) ,let))

Unexpected Captures

(ds (apair ,key ,value ,alist) `(let ((f pair)) (f (f ,key ,value) ,alist))) (let ((pair lst) (f #f)) (apair ‘false f ‘())) • Captures – Definition – Call pair f

Simple Hygiene Solutions

• Renaming for call captures – Use gensym for bindings introduced by macro • Call Example: (ds (apair ,key ,value ,alist) (let ((f (gensym)) `(let ((,f pair)) (,f (,f ,key ,value) ,alist))) • Package prefixing for definition captures (doesn’t work though for local macros) • Definition Example: (ds (apair ,key ,value ,alist) `(let ((f proto::pair)) (f (f ,key ,value) ,alist)))

Semi-automatic Hygiene

• Explicitly mention the words whose meaning is to be frozen: (ds (apair ,key ,value ,alist) (let ((f (gensym))) (with-aliases ((cons pair)) `(let ((,f ,cons)) (,f (,f ,key ,value) ,alist)))

Automatic Hygiene

• Variables record definition (e.g., module) origin • Expansion variables record hygiene context which – Is unique number – Is created afresh during each macro expansion • After expansion – Names are renamed such that names connect if they have both same name and hygiene context – If free then use module origin to start lookup • Example (ds (apair ,key ,value ,alist) #[let ((f pair)) (f (f ,key ,value) ,alist)])

Hygiene Escapes

• There are important macros that aren’t hygienic: there is a need to escape hygiene (ds (loop ,@body) #[lab (#=exit) (rep again () ,@body (again))]) (let ((i 0)) (loop (if (= i 10) (exit 37)) (set i (+ i 1))))

R5RS Macros Examples

(define-syntax or (syntax-rules () ((or) #f) ((or test) test) ((or test1 test2 ...) (let ((x test1)) (if x x (or test2 ...)))))) (define-syntax let (syntax-rules () ((let ((name val) ...) body1 body2 ...) ((lambda (name ...) body1 body2 ...) val ...)) ((let tag ((name val) ...) body1 body2 ...) ((letrec ((tag (lambda (name ...) body1 body2 ...))) tag) val ...))))

R5RS Macros

• syntax-rules is not procedural • Two environments • … is cute but brittle • Pattern variables are defaults • No hygiene escapes • Local syntax • Other Scheme macro systems exist

Code Walking

• Think of macros as compiler extensions • Macros are still limited (no with-slots nor series) – Can’t access lexical context – Can’t build up form dependent information • Code walking is a framework that makes this possible – Current state of the art is AST/OO walkers – See LiSP

Conventional Syntax Macros

• Power of macros has been limited to Lisp family of languages because Lisp has – a very regular syntax based on lists – a large library of list fabs and xforms • Has been some recent work towards remedying this situation

Grammar Extensions

“Programmable Syntax Macros” by Weise and Crew and “Growing Languages with Metamorphic Syntax Macros” by Brabrand and Schwartzbach • User writes new rewrite rules • Example Syntax unless () ::= { if (!) }

Grammar Extensions Critique

• Advantages – Can ensure that expansions produce admissible code fragments • Disadvantages – Users need to know grammar (e.g., nonterminals) – Limited to rewrite rules only and thus awkward to write more complicated macros

Dylan Macros

• Simple regular conventional syntax • Converts surface syntax to SST form • Offers constraint-based pattern matching and code quotes all with automatic hygiene • Rewrite rule only but allows auxiliary rules that take arguments • Simple Example: define macro unless { unless ?test:expression ?:body } => { if (~?test) ?body } end macro;

Sophisticated Dylan Example

define macro iterate { iterate ?:name (?bindings:*) ?:body end } => { local method ?name (?@vars{ ?bindings }) ?body end; ?name(?@inits{ ?bindings }) } vars: { } => { } { ?:variable = ?:expression, ... } => { ?variable, ... } inits: { } => { } { ?:variable = ?:expression, ... } => { ?expression, ... } end macro;

Dylan Macro Critique

• Advantages – High level – Surface syntax – Hygiene support • Disadvantages – Rewrite rule only – No guarantees about producing admissible code fragments

Procedural Macro Motivation

• Analysis and rewriting no longer constrained • Simplified pattern matching and rewrite rule engine • Can package and re-use syntax expansion utilities • Pattern matching engine is extensible

Java Syntactic Extender

• Builds on Dylan macros having constraint-based pattern matching and code quotes • Adds procedural macros • Macros are classes – Uses CLASSPATH to find them • Example: public syntax unless { case #{ unless (?test:expression) ?:statement }: return #{ if (!?test) ?statement }; }

JSE Assert Example

assert(moby() == 37); public syntax assert { case #{ assert(?:expression); } : return #{ if (?expression) System.out.println("Error"); } ; case #{ assert(?:expression, ?message:expression); } : return #{ if (?expression) System.out.println(?message); } ; } public class assertSExpander implements SyntaxExpander { … public Expansion expand (Fragment fragments) throws SyntaxMatchFailure { syntaxSwitch (fragments) { case #{ assert(?:expression); } : return #{ if (?expression) System.out.println("Error"); } ; case #{ assert(?:expression, ?message:expression); } : return #{ if (?expression) System.out.println(?message); } ; } } }

JSE Parallel forEach Example

public syntax forEach { case #{ forEach (?clauses:*) ?:statement }: Fragment inits = #{ }; Fragment preds = #{ true }; Fragment nexts = #{ }; return #{ ?(loop(clauses, statement, inits, preds, nexts)) }; } private Fragment loop (Fragment clauses, Fragment statement, Fragment inits, Fragment preds, Fragment nexts) throws MatchFailure { syntaxSwitch (clauses) { case #{ }: return #{ ?inits while (?preds) { ?nexts ?statement } }; case #{ ?:type ?:name in ?c:expression, ... }: Fragment newInits = #{ ?inits Iterator i = ?c.iterator(); }; Fragment newPreds = #{ ?preds & i.hasNext() }; Fragment newNexts = #{ ?nexts ?name = (?type)i.next(); }; return #{ ?(loop(..., statement, newInits, newPreds, newNexts)) }; } }

Open Problems

• Conventional syntax macros • More flexibility in macro shapes • Provable admissibility

Reading List

• Bawden: Quasiquotation in Lisp • Queinnec: Lisp In Small Pieces • Graham: Advanced Techniques for Common Lisp • Kelsey, Clinger, and Rees: R5RS • Clinger: Hygienic Macros through Explicit Renaming.

• Shalit: The Dylan Reference Manual • Braband and Schwartzbach: Growing Languages with Metamorphic Syntax Macros