The Object Teams Blog

In the first part I demonstrated how Object Teams can be used to extend the JDT compiler for a custom language to be embedded in Java. I concluded by saying that more substantial features like Refactoring might need more rocket science which I wanted to show next.

The “bad news” is: before I started to do some strong adaptations of DOM AST etc to make Refactorings work, I just made a few experiments of how Refactorings actually behaved in my hybrid language. To my own surprise a lot of things already worked OK: I could extract a custom syntax expression into a local variable and inline the variable again and more stuff of that kind. Just look at this example:

Actually this reflects an experience I’ve made more than once: If you reuse some module and perform some adaptations in terms of provided API and extension points etc. more often than not one adaptation entails the next, adding tweaks to workarounds because you keep scratching at the surface. If, OTOH, you succeed to make your adaptation right at the core where the decisions are made, just one or two cuts and stitches may suffice to get your job done. Clean, effective and consistent. That’s what we see when cleanly inserting a custom AST node into the JDT: if our CustomIntLiteral behaves well a lot of JDT functionality can just work with this thing without knowing it’s not a genuine Java thing.

Now this means for my next example I had to look for an extra challenge. I decided to enhance the example in two ways:

• The custom syntax should be a bit more realistic, so I chose to create a syntax for money, consisting of a number and the name of a currency
• I wanted source formatting to work for the whole hybrid language

A word of warning: this post uses some bells and whistles of OT/J and applies it to the non-trivial JDT. This might be a bit overwhelming for the novice. If you prefer lower dosage first, you may want to check out our example section in the wiki. It’s still far from complete but I’m working on it.

A syntax for money

The new syntax should allow me to write this:

int getMoney() {
    return <% 13 euro %>;
}

and the stuff should internally be stored as a structured AST node. This is how class CurrencyExpression starts:

public class CurrencyExpression extends Expression {
 
    public IntLiteral value;
    public String currency;
 
    final static String[] CURRENCIES = { "euro", "dollar" };
 
    public CurrencyExpression(int sourceStart, int sourceEnd) { ...
 
    public boolean setCurrency(String string) { ...
 
    @Override
    public StringBuffer printExpression(int indent, StringBuffer output) { ...
    ....
}

For creating a CurrencyExpression from source I wrote a little CustomParser, normal boring stuff with 40% just reading individual chars and manipulating character positions, another 45% actually does some error reporting and only 3 lines are relevant: those that create a new CurrencyExpression, create an IntValue for the value part and invoke setCurrency with the currency string.

In the ScannerAdaptor from the previous post I simply replaced this

Expression replacement = new CustomIntLiteral(source, start, end, start+2, end-2);

with this:

Expression replacement = customParser.parseCurrencyExpression(source, start, end, this.getProblemReporter());

That suffices to make the above little method compile and run just as expected.

Interlude: DOM AST

Well, with this slightly more realistic syntax you’d actually see a number of exceptions in the IDE that can all be fixed by letting the DOM AST know about our addition. For those who don’t regularly program against the JDT API: the DOM AST is the public data structure by which tools outside the JDT core manipulate Java programs. Inside the JDT extending the DOM AST would mean to subclass either org.eclipse.jdt.core.dom.ASTNode or one of its subclasses. Unfortunately, all constructors in this hierarchy are package private, and even with OT/J we respect what the javadoc says: “clients are unable to declare additional subclasses“.

But we can do something similar: instead of subclassing we can use instances of a regular DOM class and attach a role instance to them. As the base I chose org.eclipse.jdt.core.dom.SimpleName which inside the JDT could mean a lot of different things, so for most parts a node of this kind is regarded as a black box, just what we need. This is the role I added to the team SyntaxAdaptor from the previous post:

1
2
3
4
5
6
7
8
9
10
11
12

protected class DomCurrencyLiteral playedBy SimpleName {
    protected String currency;
 
    void setSourceRange(int sourceStart, int length) -> void setSourceRange(int sourceStart, int length);
 
    @SuppressWarnings("decapsulation")
    public DomCurrencyLiteral(AST ast, CurrencyExpression expression) {
        base(ast);
        this.currency = expression.currency;
        setSourceRange(expression.sourceStart, expression.sourceEnd-expression.sourceStart+1);
    }
}

So this almost looks like subclassing except we use playedBy instead of extends and base() instead of super(). And yes, when creating an instance with “new DomCurrencyLiteral(ast, expr)” inside the constructor we create a SimpleName from DOM using the package private constructor. But by using role playing instead of sub-classing this has become part of the aspectBinding relationship, which makes analysis of the state of encapsulation much easier.

So, who actually creates these nodes? Inside the JDT this is the responsibility of the ASTConverter, which takes an AST from the compiler and converts it to the public variant. In order to tell the ASTConverter how to handle our currency nodes I added this role to the existing team SyntaxAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

@SuppressWarnings("decapsulation")
protected class DomConverterAdaptor playedBy ASTConverter {
 
    // whenever convert(Expression) is called ...
    org.eclipse.jdt.core.dom.Expression convertCurrencyExpression(CurrencyExpression expression)
    <- replace org.eclipse.jdt.core.dom.Expression convert(Expression expression)
        // ... and when the literal is actually a CurrencyExpression ...
        base when (expression instanceof CurrencyExpression)
        // ... perform the cast we just checked for and feed it into the callin method below.
        with { expression <- (CurrencyExpression)expression }
 
    /**
     * Convert a CustomIntLiteral from the compiler to its dom counter part.
     * This method uses inferred callouts (OTJLD §3.1(j))
     * which need to be enabled in the OT/J compiler preferences.
     */
    @SuppressWarnings({ "basecall", "inferredcallout" })
    callin org.eclipse.jdt.core.dom.Expression convertCurrencyExpression(CurrencyExpression expression){
        final DomCurrencyLiteral name = new DomCurrencyLiteral(this.ast, expression);
        if (this.resolveBindings) {
            recordNodes(name, expression);
        }
        return name;
    }
}

I deliberately used some special OT/J syntax worth explaining:

• Lines 5ff. define a callin bindings like we’ve seen before.
• Line 8 adds a guard predicate to the binding, saying that this binding should only fire when the argument expression is actually of type CurrencyExpression
• After passing the guard we know that we can safely cast to CurrencyExpression so I added a parameter mapping (line 10) which feeds a casted value into the role method.
• Inside the role method convertCurrencyExpression everything looks normal, but at a closer look this.ast and this.resolveBindings seem to be undefined in the scope of the current class. In fact these fields are defined in the base class ASTConverter and we could use explicit callout accessors like in the previous post. However, this time I chose to let the compiler infer these callouts so that the method would look exactly like existing methods in ASTConverter do (this option has to be enabled in the OT/J compiler preferences).

OK, with this little addition our CurrencyExpressions are converted to something that the JDT can handle and we’re already prepared for doing real AST manipulation including our syntax.

Source Formatting

Inside the JDT source formatting (Ctrl-Shift-F) is essentially performed by class CodeFormatterVisitor. This class is one of many subclasses of the general ASTVisitor. If one wanted to make these visitors aware of our CurrencyExpression we would have to add one visit method to ASTVisitor and each of its sub-classes! That’s certainly not viable, so with plain Java we’re pretty much out of luck.

The situation that needs adaptation can be described as follows:

• A visitor will be created and invoked in order to descend into the AST
• At the point when traversal finds a CurrencyExpression it will invoke its traverse(ASTVisitor) method.

Of course we could manually inspect the type of visitor within the traverse method, but that would defy the whole purpose of having visitors: keep all those add-on functions out from your data structures. Instead I only gave a default implementation to CurrencyExpression.traverse and used OT/J for the cleanest implementation of double dispatch (which is what the visitor pattern painstakingly emulates): we need dispatch that considers both the visitor type and the node type for finding the suitable method implementation.

In green-field development this would be still easier but even on top of an existing visitor infrastructure it get’s pretty concise.

Visitor adaptation - version 1

My first version looks like this (explanations follow below):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

public team class VisitorsAdaptor {
 
    protected team class AstFormatting playedBy CodeFormatterVisitor {
        // whenever visiting something that could contain an expression
        // activate this team to enable callins of the inner role
        callin void visiting() {
            within(this) {
                base.visiting();
            }
        }
        @SuppressWarnings("decapsulation")
        void visiting()
            <- replace
                boolean visit(Block block, BlockScope scope),
                boolean visit(FieldDeclaration fieldDeclaration, MethodScope scope),
                void formatStatements(BlockScope scope, final Statement[] statements, boolean insertNewLineAfterLastStatement);
 
        Scribe getScribe() -> get Scribe scribe;
 
        /** This role implements formating of our custom ast: */
        protected class CustomAst playedBy CurrencyExpression
        {
            void traverse() <- replace void traverse(ASTVisitor visitor, BlockScope scope);
 
            @SuppressWarnings({ "inferredcallout", "basecall" })
            callin void traverse() {
                Scribe scribe = getScribe();
                Scanner scanner = scribe.scanner;
 
                // format this AST node into a StringBuffer:
                StringBuffer replacement = new StringBuffer();
                replacement.append("<% ");
                this.value.printExpression(0, replacement);
                replacement.append(' ');
                replacement.append(this.currency);
                replacement.append(" %>");
 
                // feed the formatted string into the Scribe:
                int start = this.sourceStart();
                int end = this.sourceEnd();
                scribe.addReplaceEdit(start, end, replacement.toString());
 
                // advance the scanner:
                scanner.resetTo(end+1, scribe.scannerEndPosition - 1);
                scribe.pendingSpace = false;
            }
        }
    }
}

The key trick in this example is nesting:

• Role AstFormatting is responsible for detecting when a CodeFormatterVisitor is visiting any subtree that may contain expressions. This is done using a callin binding that lists three relevant base methods which all should be intercepted by the same role method (lines 12-16).
• Inside role AstFormatting (which is also marked as a team) an inner role CustomAst will only be triggered if a CodeFormatterVisitor calls the traverse method of a CurrencyExpression (see callin binding in line 23).
• The connection between both levels is wired in method AstFormatting.visiting: the block statement within() { } temporarily and locally activates the given team instance, here denoted by this. Only during this block the nested team AstFormatting is active - meaning that only during this block the callin binding in role CustomAst will fire.
• Within role CustomAst we can naturally access the CodeFormatterVisitor via the enclosing instance of AstFormatting. No instanceof and casting needed, because all this only happens in the context of a CodeFormatterVisitor

The body of method traverse contains only domain logic: pretty-printing the current node into a string buffer and interacting with the underlying infrastructure (Scanner, Scribe) that drives the formatting.

That’s it, with these classes in place, we can write this method:

int getMoney() {
   int myMoney = <% 
		  3
	  euro %>
	; 
		System
	.out.println("myMoney ="+myMoney);
					return myMoney;
}

then hit Ctrl-Shift-F et voilà:

private static int getMoney() {
    int myMoney = <% 3 euro %>;
    System.out.println("myMoney =" + myMoney);
    return myMoney;
}

How’s that?
The formatter smoothly operates on the full hybrid language, not just skipping over our nodes but handling them as well.

Generalizing visitor adaptations

After success wrt both challenges I’d like to clean up even more and prepare for further adaptations of other visitors. Given how many subclasses of ASTVisitor are used within the JDT we wouldn’t want to write the infrastructure for double dispatch over and over again. So let’s generalize, that is: extract a common super-class, by extracting everything re-usable out off class AstFormatting

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

public team class VisitorsAdaptor
{
    protected abstract team class AstVisiting playedBy ASTVisitor {
        // whenever visiting something that could contain an expression
        // activate this team to enable callins of the inner role
        callin void visiting() {
            within(this)
                base.visiting();
        }
        void visiting()
            <- replace
                boolean visit(Block block, BlockScope scope),
                boolean visit(FieldDeclaration fieldDeclaration, MethodScope scope);
 
        protected abstract class CustomAst playedBy CurrencyExpression {
            // variant of traversal that should be used when the enclosing team is active:
            // (implement in subclasses)
            abstract callin void traverse();
            void traverse() <- replace void traverse(ASTVisitor visitor, BlockScope scope);
        }
        // Insert more roles for binding more AST nodes...
    }
 
    protected team class AstFormatting extends AstVisiting playedBy CodeFormatterVisitor
    {
        // one more trigger that should activate the team:
        @SuppressWarnings("decapsulation")
        visiting <- replace formatStatements;
 
        Scribe getScribe() -> get Scribe scribe;
 
        /** This role implements formating of our custom ast: */
        @Override
        protected class CustomAst {
            @SuppressWarnings({ "inferredcallout", "basecall" })
            callin void traverse() {
                // method body as before
            }
        }
    }
    protected team class OtherVisitorAdaptor extends AstVisiting playedBy XYVisitor
    {
        @Override
        protected class CustomAst {
            callin void traverse() {
                // domain logic
            }
        }
        // Insert more roles for actually handling more AST nodes ...
    }
}

Now team class AstVisiting contains the part that is common for all visitors. At this level several things are still abstract: method traverse, role class CustomAst and even the whole team AstVisiting.

Team class AstFormatting extends the abstract team and defines everything specific to formatting. We have one more trigger for visiting, one callout binding to a field of class CodeFormatterVisitor and then we only refine the previously abstract role class CustomAst. At this level it is no longer abstract because we give an implementation for traverse.

I’ve also sketched another nested team showing a minimal specialization of AstVisiting for adapting some other visitor and adding another implementation for CustomAst.traverse plus potentially more roles for more node types.

Conclusion

For those who don’t work in the compiler business on a day-to-day basis this is probably pretty tough stuff, but let me summarize what we’ve just achieved:

• Embed a custom syntax into Java, showing how a custom parser can be plugged in to create custom AST from a region of the Java source.
• Adapt the conversion between two different AST structures (internal -> DOM) to also handle custom nodes.
• Adapt the code formatter so that hybrid sources can be formatted with a single command.
• Prepared the infrastructure for adapting other visitors, too. By this we have achieved that new visitor adaptations will only need to add their specific implementation with close to zero scaffolding.
• Cleanly separated each implemented concern in one module.
• Keep each module in the scale of only tens of lines of code.
• Yet implement significant steps towards a production quality IDE for our custom hybrid language.

Maybe I shouldn’t have told you, how easy these things can be - if your tools are sharp - maybe.
But professional carvers know: if your knife is sharp, it’s actually easy to handle. Only if it is blunt you are in real danger of hurting yourself - because you need to apply disproportionate force to cut your wood. So:

Spare your fingers, sharpen your knife!

PS: Here’s the archive of all sources, ready to be imported into the OTDT.

Have you ever thought of making Java a bit smarter? Perhaps, for some task you would prefer a custom syntax, and snippets using that syntax should then be embedded into Java? Sure, many never seriously think about this because of the prohibitively high effort to create the compiler for such hybrid language. And even if you are a compiler guru, knowing your toolkits so that translation wouldn’t be a problem for you, you’ll probably surrender at the mere thought of how to create a mature IDE that would allow efficiently productive work with you hybrid language.

You shouldn’t give up. Think: If you build your own IDE you’ll never be able to really compete with the JDT, right? Still anything falling back behind the quality of the JDT won’t raise your productivity but will stand in your way at the most common tasks during development, right?
What does this tell you? Give up? No. If you can’t beat us, join us. Don’t write a new IDE for any Java-based language. Join the JDT. Well, but the JDT doesn’t provide an extension point for embedding a different syntax, does it? Sure they don’t, but it’s actually not their job to do so because every embedded language will probably have slightly different requirements so designing such an extension point would be a battle you can never win.

I have developed a tiny extension to Java and integrated this into the JDT by a mere 204 lines of code including comments and a plugin.xml. As some may guess the only trick needed is to use Object Teams. By this post I will explain how Object Teams can be used for extending the JDT in this way. And I will also argue against the most common fear in this context: “Is that solution maintainable?” From my very own experience this design is not just barely manageable, but from all I’ve seen this is the best maintainable solution for this kind of task, but I’m getting ahead of myself.

In order not to distract from the interesting design issues I’ll be using the most simply language extension: I want to be able to write integer constants in natural language, and while I’m at it, I want it to work in an multilingual setting. So, this should, e.g., be a legal program:

public class EmbeddingTest {
    private static int foo() {
        return <% one %>;
    }
    public static void main(String[] args) {
        System.out.println(foo());
    }
}

I’m using <% and %> tokens to switch between Java syntax and custom syntax.

The first step can be achieved in plain Java, it’s creating a class for ASTNodes representing my custom int literals within the compiler. If you really want you may inspect class CustomIntLiteral, but it’s actually pretty boring old Java. Its main job is to lookup a given string from an array of known number words and thus translate the word into an int. It even detects the language used and remembers this for later use. The behaviour is hooked into the JDT compiler by overriding method TypeBinding resolveType(BlockScope scope) — just normal Java practice.

Drilling down into the example

Here’s an overview of the module that does all the rest:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

package embedding.jdt;
 
import org.eclipse.jdt.core.compiler.CharOperation;
import org.eclipse.jdt.core.compiler.InvalidInputException;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTNode;
import org.eclipse.jdt.internal.compiler.ast.Expression;
import org.eclipse.jdt.internal.compiler.ast.IntLiteral;
import org.eclipse.jdt.internal.compiler.parser.TerminalTokens;
 
import embedding.custom.ast.CustomIntLiteral;
 
import base org.eclipse.jdt.core.dom.ASTConverter;
import base org.eclipse.jdt.core.dom.NumberLiteral;
import base org.eclipse.jdt.internal.compiler.parser.Parser;
import base org.eclipse.jdt.internal.compiler.parser.Scanner;
 
public team class SyntaxAdaptor {
 
        /**
         * <h3>Part 1 of the adaptation:</h3>
         * Wait until '<' is seen and check if it actually is a special string enclosed in '<%' and '%>'.
         */
        protected class ScannerAdaptor playedBy Scanner { ... }
 
        /**
         * <h3>Part 2 of the adaptation:</h3>
         * If the ScannerAdaptor found a match intercept creation of the faked null expression
         * and replace it with a custom AST.
         *
         * This is a team with a nested role so that we can control activation separately.
         *
         * This team should be activated for the current thread only to ensure that
         * concurrent compilations don't interfere: By using thread activation any state of
         * this team is automatically local to that thread.
         */
        protected team class InnerCompilerAdaptor {
                /** This inner role does the real work of the InnerCompilerAdaptor. */
                protected class ParserAdaptor playedBy Parser { ... }
        }
 
        /**
         * Dom representation of CustomIntLiteral.
         * Since the constructor of NumberLiteral is package private we cannot subclass, so use a role instead.
         */
        protected class DomCustomIntLiteral playedBy NumberLiteral
                // don't adapt plain NumberLiterals, just those that already have a DomCustomIntLiteral role:
                base when (SyntaxAdaptor.this.hasRole(base, DomCustomIntLiteral.class))
        { ... }
 
        /**
         * <h3>Part 3 of the adaptation:</h3>
         * This adaptor role helps the ASTConverter to convert CustomIntLiterals, too.
         */
        @SuppressWarnings("decapsulation")
        protected class DomConverterAdaptor playedBy ASTConverter { ... }
}

Imports

Why am I showing you boring import declarations to begin with? Well, with OT/J there’s a fine distinction that is worth looking at: all imports starting with import base indicate that these classes are imported for attaching a role to them. So just from these lines you see that the given module adds roles to classes from org.eclipse.jdt.internal.compiler.parser and org.eclipse.jdt.core.dom (2 classes each). All other imports are plain Java imports and won’t let you apply any OT/J tricks.

Teams and Roles

Line 18 above tells you that the class SyntaxAdaptor is actually a team. Teams are used for grouping a set of roles - nested classes of a team. Using the playedBy keyword a role declares that it adapts the specified base class (which are the same classes we base-imported above). The purpose of these roles should be roughly clear by the doc comments.

So, role ScannerAdaptor will be responsible for switching between both syntaxes.

Role ParserAdaptor (line 40) will be responsible for creating our AST node (CustomIntLiteral). But wait, what’s that: the role is nested within an intermediate team, InnerCompilerAdaptor. This team will show you, how to define a role that is only effective in specific situations, here, the ParserAdaptor should only be effective after the ScannerAdaptor has detected a syntax switch. Details follow below.

The other two roles will do advanced stuff so I’ll discuss them later.

Role implementation (1)

Here is the full(!) code of role ScannerAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

protected class ScannerAdaptor playedBy Scanner {
 
    // access fields from Scanner ("callout bindings"):
    int getCurrentPosition()                     -> get int currentPosition;
    void setCurrentPosition(int currentPosition) -> set int currentPosition;
    char[] getSource()                           -> get char[] source;
 
    // intercept this method from Scanner ("callin binding"):
    int getNextToken() <- replace int getNextToken();
 
    callin int getNextToken() throws InvalidInputException {
        // invoke the original method:
        int token = base.getNextToken();
        if (token == TerminalTokens.TokenNameLESS) {
            char[] source = getSource();
            int pos = getCurrentPosition();
            if (source[pos++] == '%') {                                        // detecting the opening "<%" ?
                int start = pos;                                               // inner start, just behind "<%"
                try {
                    while (source[pos++] != '%' || source[pos++] != '>')       // detecting the closing "%>" ?
                        ;                                                      // empty body
                } catch (ArrayIndexOutOfBoundsException aioobe) {              // not found, proceed as normal
                        return token;
                }
                setCurrentPosition(pos);                                       // tell the scanner what we have consumed (pointing one past '>')
                int end = pos-2;                                               // position of "%>"
                char[] fragment = CharOperation.subarray(source, start, end);  // extract the custom string (excluding <% and %>)
                // prepare an inner adaptor to intercept the expected parser action
                new InnerCompilerAdaptor(fragment, start-2, end+1).activate(); // positions include <% and %>
                return TerminalTokens.TokenNamenull;                           // pretend we saw a valid expression token ('null')
            }
        }
        return token;
    }
}

Comments describing the logic are in the right column. Inline comments describe the usage of OT/J:

• Lines 3-6 define accessors for two fields from the base class Scanner.
• Line 9 defines that calls to method getNextToken() should be intercepted by our version of this method
• Line 11 marks the role method as callin which is a pre-requisite for line 13
• Line 13 invokes the original method from Scanner
• In line 29 we are in the situation that we have detected a region delimited by <% and %>. We have extracted the text fragment between delimiters, and we know the start and end positions within the source file. Only now we create an instance of InnerCompilerAdaptor and immediately activate it for the current thread (activate()).

At this point the ScannerAdaptor is done and now an InnerCompilerAdaptor is watching what comes next.

Here’s the nested team InnerCompilerAdaptor with its role ParserAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

protected team class InnerCompilerAdaptor {
 
    char[] source;
    int start, end;
    protected InnerCompilerAdaptor(char[] source, int start, int end) {
        this.source = source;
        this.start = start;
        this.end = end;
    }
 
    /** This inner role does the real work of the InnerCompilerAdaptor. */
    protected class ParserAdaptor playedBy Parser {
        // import methods from Parser ("callout bindings"):
        @SuppressWarnings("decapsulation")
        void pushOnExpressionStack(Expression expr) -> void pushOnExpressionStack(Expression expr);
 
        // intercept this method from Parser ("callin binding"):
        void consumeToken(int type) <- replace void consumeToken(int type);
 
        @SuppressWarnings("basecall")
        callin void consumeToken(int type) {
            if (type == TerminalTokens.TokenNamenull) {                 // 'null' token is the faked element pushed by the SyntaxAdaptor
                // this inner adaptor has done its job, no longer intercept
                InnerCompilerAdaptor.this.deactivate();
                // TODO analyse source to find what AST should be created
                Expression replacement = new CustomIntLiteral(source, start, end, start+2, end-2);
                this.pushOnExpressionStack(replacement);                // feed custom AST into the parser:
                return;
            }
            // shouldn't happen: only activated when scanner returns TokenNamenull
            base.consumeToken(type);
        }
    }
}

• Lines 3-4 define state of the nested team, which is used for passing the information collected by the ScannerAdaptor down the pipe
• Line 15 provides access to a protected method from Parser. By @SuppressWarnings("decapsulation") we document that this access inserts a tiny little hole into the encapsulation of Parser
• Line 18 defines a callin binding as we have seen it before.
• Line 24 already deactivates the enclosing InnerCompilerAdaptor, ensuring this is a one-shot adaptation, only.
• Line 26/27 perform the payload: feed a CustomIntLiteral node into the parser

Coming to life

Wow, if you’ve read so far, you’ve seen a lot of OT/J on just a few lines of code. Let’s wire things together, by throwing the code into an Object Teams Plug-in Project and declaring one extension:

I have defined one aspectBinding between the existing plugin org.eclipse.jdt.core and my team classes SyntaxAdaptor and InnerAdaptor (there’s a man behind the curtain pushing an ugly __OT__ prefix into the declaration, please ignore him - he’ll be gone in the next release of the tool).
Please note that for team SyntaxAdaptor I have set the activation to ALL_THREADS which means that at application launch an instance of this team will be created and activated globally. Without this flag the whole thing would actually have no effect at all.

That’s all the wiring needed, so kick up a runtime workbench, create a Java project and class, insert the code for class EmbeddingTest from the top of this post and boldly select Run As > Java Application. In the console we see a result:

1

Oops, the compiler for our little language extension already works? Did you see me writing a compiler?

Well, beginner’s luck, let’s assume. But, oops, watch this: When I mistype the return type of foo and ask the JDT for help, this is what I see:

The problem view tells me it knows that <% one %> has type int, which doesn’t match the declared return type boolean. Next I positioned the cursor on “one” (the element that’s definitely not Java) and hit Ctrl-1, and the standard JDT quickfix knows that I should change the return type of foo to int.
Did you watch me implementing a quickfix??

Summary so far

Here’re the stats:

• 204 lines of code including plugin.xml
• roles adapting two base classes from org.eclipse.jdt.core.
• callout bindings to two fields and one method
• callin bindings to two methods
• all adaptation is cleanly encapsulated in one team class. If you wish you could even deactivate this one team in a running workbench and thus disable all our adaptions with a single click.
• one plain Java class to implement the semantics of our extension

As for maintainability: The only dependencies are the items mentioned above: two classes, two fields and three methods. Only if one of these are modified under evolution, my adaptation has to be updated accordingly - and: if this happens I will definitely be told by the compiler because one of the bindings will break. If it doesn’t break there’s no need to worry.

With this implementation the compiler seamlessly works with our new syntax and even UI features that operate on the compiler AST can handle our extension, too.

What’s next?

I’m sure some think that the above is probably a forged example. You might challenge me to do something real, like refactoring. If you do so, you actually got me (mumble, mumble) - with the above implementation refactoring does not work with our custom syntax. Now that you’ve seen the start, what do you expect, how much additional rocket science does it take to add minimal refactoring support? (to be continued)

I’d like to return the warm “welcome” Wayne recently pronounced under “Heady Times”.
The Object Teams Project is thrilled to finally move into the Eclipse house. I guess we are not complete strangers, as I’ve spoken at two EclipseCons and met quite a few of you in bugzilla etc. But still a majority might not be sure what the Object Teams Project is all about, so here comes a little introduction:

Origin

Formally, the Object Teams Project is a cross-over between ambitious research in language design and an extremely pragmatic approach to writing highly modular code even under high pressure of re-use, maintenance, evolution and that sort of conditions. Our pragmatism made us refrain from designing a language from scratch but made us create OT/J as an extension of Java. This pragmatism has furthermore led us to spending >99% of our time on building actual working tools in contrast to <1% of tweaking new language features. That’s why we decided as early as 2003 to not develop a standalone compiler for our language but to base everything on the JDT (v2.1.1!) so we would soon have a full-featured, high-quality IDE. From this the Object Teams Development Tooling (OTDT) was born which had its first public appearance in 2005. This development was a tremendous joint effort (2003-2006) of the Fraunhofer FIRST institute and the Technical University Berlin.

Two ways of looking at Object Teams

(1) Conceptually: Object Teams solves the dilemma of structuring software by functions (procedural or functional approaches) or by data (object oriented approach). In Object Teams you do both simultaneously. Suppose you are writing a banking application. After some core concepts have been implemented and tested you are about to implement a new use case, say Wire-Transfer. Do you add new methods to existing classes like Customer, Account, ATM etc? Or do you consider those classes as dumb data objects and create one new class WireTransfer? The Object Teams answer: you do the best of both approaches: you create a role for each relevant class, implement the new behaviour in terms of role methods calling each other. Roles are grouped into a team (here: WireTransfer) and roles are bound to the underlying core objects (here: of type Customer, Account etc.). - Looking for a data-module, an entity? Just inspect the Account together with all its roles. - Looking for a behavioral module? Just inspect the WireTransfer team with all its contained roles. Voila.

Sounds esoteric? “Role modeling”, “collaboration based design” and similar off-springs of the same ideas have been embraced for analysis and design for quite some time (cf. OOram, Catalysis etc.). Object Teams finally gives a natural implementation to such models. Finally we can cleanly compose a system from one module per use case.

It turns out, however, that you may also look at Object Teams from an even more pragmatic point of view:

(2) Pragmatically: As one of its core capabilities a role can override a method of its bound base. Given that roles can be added to existing entities at any time (add role classes at compile-time, add role instances to base instances even at run-time), roles provide a mechanism for extending and adapting existing things without touching them (”non-invasively”). As an example consider the JDT/UI: a plug-in providing an excellent user experience for working with Java code. Guess what: just by adding a number of roles and without changing a single source line of the the JDT/UI we created the OTDT/UI: a plug-ing providing an excellent user experience for working with OT/J code .

While the mere possibility to non-invasively adapt existing plug-ins is fascinating by itself, using Object Teams for this task also guides you towards a code structure that is actually easy to read and highly suitable for future maintenance and evolution.

OT/Equinox

Here is why all this is highly relevant for everyone writing Eclipse plug-ins: we have married OT/J to Equinox (first presented at Eclipse Technology eXchange 2006 in Nantes) so that all the concepts of Object Teams can seamlessly be leveraged when developing plug-ins. The consequences of this I will leave for your imagination at this point. More to follow soon.

Show me the code

This post has become a little verbose, with no real examples. Some examples will follow in subsequent posts. In the meantime you may want to look at the

• Object Teams Quick-Start with actual code to pimp the Eclipse workbench “in less then 15 minutes”.

What’s next?

Currently our initial contribution awaits the initial OK so that we can start committing to SVN. Pretty soon after you should expect the first milestone release (incubation) from eclipse.org. “If all goes well” we will apply for graduation by the time of the Helios release, so you would see an OTDT 2.0.0 accompanying the Helios GA. After 70+ releases (incl. milestones) in our previous identity at objectteams.org I say we should be able to demonstrate the maturity required for that next big step.

Please stay tuned and join the discussion in our forum.

Thanks to everybody who made our move to Eclipse.org possible!