The Object Teams Blog

In the first part I demonstrated how Object Teams can be used to extend the JDT compiler for a custom language to be embedded in Java. I concluded by saying that more substantial features like Refactoring might need more rocket science which I wanted to show next.

The “bad news” is: before I started to do some strong adaptations of DOM AST etc to make Refactorings work, I just made a few experiments of how Refactorings actually behaved in my hybrid language. To my own surprise a lot of things already worked OK: I could extract a custom syntax expression into a local variable and inline the variable again and more stuff of that kind. Just look at this example:

Actually this reflects an experience I’ve made more than once: If you reuse some module and perform some adaptations in terms of provided API and extension points etc. more often than not one adaptation entails the next, adding tweaks to workarounds because you keep scratching at the surface. If, OTOH, you succeed to make your adaptation right at the core where the decisions are made, just one or two cuts and stitches may suffice to get your job done. Clean, effective and consistent. That’s what we see when cleanly inserting a custom AST node into the JDT: if our CustomIntLiteral behaves well a lot of JDT functionality can just work with this thing without knowing it’s not a genuine Java thing.

Now this means for my next example I had to look for an extra challenge. I decided to enhance the example in two ways:

• The custom syntax should be a bit more realistic, so I chose to create a syntax for money, consisting of a number and the name of a currency
• I wanted source formatting to work for the whole hybrid language

A word of warning: this post uses some bells and whistles of OT/J and applies it to the non-trivial JDT. This might be a bit overwhelming for the novice. If you prefer lower dosage first, you may want to check out our example section in the wiki. It’s still far from complete but I’m working on it.

A syntax for money

The new syntax should allow me to write this:

int getMoney() {
    return <% 13 euro %>;
}

and the stuff should internally be stored as a structured AST node. This is how class CurrencyExpression starts:

public class CurrencyExpression extends Expression {
 
    public IntLiteral value;
    public String currency;
 
    final static String[] CURRENCIES = { "euro", "dollar" };
 
    public CurrencyExpression(int sourceStart, int sourceEnd) { ...
 
    public boolean setCurrency(String string) { ...
 
    @Override
    public StringBuffer printExpression(int indent, StringBuffer output) { ...
    ....
}

For creating a CurrencyExpression from source I wrote a little CustomParser, normal boring stuff with 40% just reading individual chars and manipulating character positions, another 45% actually does some error reporting and only 3 lines are relevant: those that create a new CurrencyExpression, create an IntValue for the value part and invoke setCurrency with the currency string.

In the ScannerAdaptor from the previous post I simply replaced this

Expression replacement = new CustomIntLiteral(source, start, end, start+2, end-2);

with this:

Expression replacement = customParser.parseCurrencyExpression(source, start, end, this.getProblemReporter());

That suffices to make the above little method compile and run just as expected.

Interlude: DOM AST

Well, with this slightly more realistic syntax you’d actually see a number of exceptions in the IDE that can all be fixed by letting the DOM AST know about our addition. For those who don’t regularly program against the JDT API: the DOM AST is the public data structure by which tools outside the JDT core manipulate Java programs. Inside the JDT extending the DOM AST would mean to subclass either org.eclipse.jdt.core.dom.ASTNode or one of its subclasses. Unfortunately, all constructors in this hierarchy are package private, and even with OT/J we respect what the javadoc says: “clients are unable to declare additional subclasses“.

But we can do something similar: instead of subclassing we can use instances of a regular DOM class and attach a role instance to them. As the base I chose org.eclipse.jdt.core.dom.SimpleName which inside the JDT could mean a lot of different things, so for most parts a node of this kind is regarded as a black box, just what we need. This is the role I added to the team SyntaxAdaptor from the previous post:

1
2
3
4
5
6
7
8
9
10
11
12

protected class DomCurrencyLiteral playedBy SimpleName {
    protected String currency;
 
    void setSourceRange(int sourceStart, int length) -> void setSourceRange(int sourceStart, int length);
 
    @SuppressWarnings("decapsulation")
    public DomCurrencyLiteral(AST ast, CurrencyExpression expression) {
        base(ast);
        this.currency = expression.currency;
        setSourceRange(expression.sourceStart, expression.sourceEnd-expression.sourceStart+1);
    }
}

So this almost looks like subclassing except we use playedBy instead of extends and base() instead of super(). And yes, when creating an instance with “new DomCurrencyLiteral(ast, expr)” inside the constructor we create a SimpleName from DOM using the package private constructor. But by using role playing instead of sub-classing this has become part of the aspectBinding relationship, which makes analysis of the state of encapsulation much easier.

So, who actually creates these nodes? Inside the JDT this is the responsibility of the ASTConverter, which takes an AST from the compiler and converts it to the public variant. In order to tell the ASTConverter how to handle our currency nodes I added this role to the existing team SyntaxAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

@SuppressWarnings("decapsulation")
protected class DomConverterAdaptor playedBy ASTConverter {
 
    // whenever convert(Expression) is called ...
    org.eclipse.jdt.core.dom.Expression convertCurrencyExpression(CurrencyExpression expression)
    <- replace org.eclipse.jdt.core.dom.Expression convert(Expression expression)
        // ... and when the literal is actually a CurrencyExpression ...
        base when (expression instanceof CurrencyExpression)
        // ... perform the cast we just checked for and feed it into the callin method below.
        with { expression <- (CurrencyExpression)expression }
 
    /**
     * Convert a CustomIntLiteral from the compiler to its dom counter part.
     * This method uses inferred callouts (OTJLD §3.1(j))
     * which need to be enabled in the OT/J compiler preferences.
     */
    @SuppressWarnings({ "basecall", "inferredcallout" })
    callin org.eclipse.jdt.core.dom.Expression convertCurrencyExpression(CurrencyExpression expression){
        final DomCurrencyLiteral name = new DomCurrencyLiteral(this.ast, expression);
        if (this.resolveBindings) {
            recordNodes(name, expression);
        }
        return name;
    }
}

I deliberately used some special OT/J syntax worth explaining:

• Lines 5ff. define a callin bindings like we’ve seen before.
• Line 8 adds a guard predicate to the binding, saying that this binding should only fire when the argument expression is actually of type CurrencyExpression
• After passing the guard we know that we can safely cast to CurrencyExpression so I added a parameter mapping (line 10) which feeds a casted value into the role method.
• Inside the role method convertCurrencyExpression everything looks normal, but at a closer look this.ast and this.resolveBindings seem to be undefined in the scope of the current class. In fact these fields are defined in the base class ASTConverter and we could use explicit callout accessors like in the previous post. However, this time I chose to let the compiler infer these callouts so that the method would look exactly like existing methods in ASTConverter do (this option has to be enabled in the OT/J compiler preferences).

OK, with this little addition our CurrencyExpressions are converted to something that the JDT can handle and we’re already prepared for doing real AST manipulation including our syntax.

Source Formatting

Inside the JDT source formatting (Ctrl-Shift-F) is essentially performed by class CodeFormatterVisitor. This class is one of many subclasses of the general ASTVisitor. If one wanted to make these visitors aware of our CurrencyExpression we would have to add one visit method to ASTVisitor and each of its sub-classes! That’s certainly not viable, so with plain Java we’re pretty much out of luck.

The situation that needs adaptation can be described as follows:

• A visitor will be created and invoked in order to descend into the AST
• At the point when traversal finds a CurrencyExpression it will invoke its traverse(ASTVisitor) method.

Of course we could manually inspect the type of visitor within the traverse method, but that would defy the whole purpose of having visitors: keep all those add-on functions out from your data structures. Instead I only gave a default implementation to CurrencyExpression.traverse and used OT/J for the cleanest implementation of double dispatch (which is what the visitor pattern painstakingly emulates): we need dispatch that considers both the visitor type and the node type for finding the suitable method implementation.

In green-field development this would be still easier but even on top of an existing visitor infrastructure it get’s pretty concise.

Visitor adaptation - version 1

My first version looks like this (explanations follow below):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

public team class VisitorsAdaptor {
 
    protected team class AstFormatting playedBy CodeFormatterVisitor {
        // whenever visiting something that could contain an expression
        // activate this team to enable callins of the inner role
        callin void visiting() {
            within(this) {
                base.visiting();
            }
        }
        @SuppressWarnings("decapsulation")
        void visiting()
            <- replace
                boolean visit(Block block, BlockScope scope),
                boolean visit(FieldDeclaration fieldDeclaration, MethodScope scope),
                void formatStatements(BlockScope scope, final Statement[] statements, boolean insertNewLineAfterLastStatement);
 
        Scribe getScribe() -> get Scribe scribe;
 
        /** This role implements formating of our custom ast: */
        protected class CustomAst playedBy CurrencyExpression
        {
            void traverse() <- replace void traverse(ASTVisitor visitor, BlockScope scope);
 
            @SuppressWarnings({ "inferredcallout", "basecall" })
            callin void traverse() {
                Scribe scribe = getScribe();
                Scanner scanner = scribe.scanner;
 
                // format this AST node into a StringBuffer:
                StringBuffer replacement = new StringBuffer();
                replacement.append("<% ");
                this.value.printExpression(0, replacement);
                replacement.append(' ');
                replacement.append(this.currency);
                replacement.append(" %>");
 
                // feed the formatted string into the Scribe:
                int start = this.sourceStart();
                int end = this.sourceEnd();
                scribe.addReplaceEdit(start, end, replacement.toString());
 
                // advance the scanner:
                scanner.resetTo(end+1, scribe.scannerEndPosition - 1);
                scribe.pendingSpace = false;
            }
        }
    }
}

The key trick in this example is nesting:

• Role AstFormatting is responsible for detecting when a CodeFormatterVisitor is visiting any subtree that may contain expressions. This is done using a callin binding that lists three relevant base methods which all should be intercepted by the same role method (lines 12-16).
• Inside role AstFormatting (which is also marked as a team) an inner role CustomAst will only be triggered if a CodeFormatterVisitor calls the traverse method of a CurrencyExpression (see callin binding in line 23).
• The connection between both levels is wired in method AstFormatting.visiting: the block statement within() { } temporarily and locally activates the given team instance, here denoted by this. Only during this block the nested team AstFormatting is active - meaning that only during this block the callin binding in role CustomAst will fire.
• Within role CustomAst we can naturally access the CodeFormatterVisitor via the enclosing instance of AstFormatting. No instanceof and casting needed, because all this only happens in the context of a CodeFormatterVisitor

The body of method traverse contains only domain logic: pretty-printing the current node into a string buffer and interacting with the underlying infrastructure (Scanner, Scribe) that drives the formatting.

That’s it, with these classes in place, we can write this method:

int getMoney() {
   int myMoney = <% 
		  3
	  euro %>
	; 
		System
	.out.println("myMoney ="+myMoney);
					return myMoney;
}

then hit Ctrl-Shift-F et voilà:

private static int getMoney() {
    int myMoney = <% 3 euro %>;
    System.out.println("myMoney =" + myMoney);
    return myMoney;
}

How’s that?
The formatter smoothly operates on the full hybrid language, not just skipping over our nodes but handling them as well.

Generalizing visitor adaptations

After success wrt both challenges I’d like to clean up even more and prepare for further adaptations of other visitors. Given how many subclasses of ASTVisitor are used within the JDT we wouldn’t want to write the infrastructure for double dispatch over and over again. So let’s generalize, that is: extract a common super-class, by extracting everything re-usable out off class AstFormatting

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

public team class VisitorsAdaptor
{
    protected abstract team class AstVisiting playedBy ASTVisitor {
        // whenever visiting something that could contain an expression
        // activate this team to enable callins of the inner role
        callin void visiting() {
            within(this)
                base.visiting();
        }
        void visiting()
            <- replace
                boolean visit(Block block, BlockScope scope),
                boolean visit(FieldDeclaration fieldDeclaration, MethodScope scope);
 
        protected abstract class CustomAst playedBy CurrencyExpression {
            // variant of traversal that should be used when the enclosing team is active:
            // (implement in subclasses)
            abstract callin void traverse();
            void traverse() <- replace void traverse(ASTVisitor visitor, BlockScope scope);
        }
        // Insert more roles for binding more AST nodes...
    }
 
    protected team class AstFormatting extends AstVisiting playedBy CodeFormatterVisitor
    {
        // one more trigger that should activate the team:
        @SuppressWarnings("decapsulation")
        visiting <- replace formatStatements;
 
        Scribe getScribe() -> get Scribe scribe;
 
        /** This role implements formating of our custom ast: */
        @Override
        protected class CustomAst {
            @SuppressWarnings({ "inferredcallout", "basecall" })
            callin void traverse() {
                // method body as before
            }
        }
    }
    protected team class OtherVisitorAdaptor extends AstVisiting playedBy XYVisitor
    {
        @Override
        protected class CustomAst {
            callin void traverse() {
                // domain logic
            }
        }
        // Insert more roles for actually handling more AST nodes ...
    }
}

Now team class AstVisiting contains the part that is common for all visitors. At this level several things are still abstract: method traverse, role class CustomAst and even the whole team AstVisiting.

Team class AstFormatting extends the abstract team and defines everything specific to formatting. We have one more trigger for visiting, one callout binding to a field of class CodeFormatterVisitor and then we only refine the previously abstract role class CustomAst. At this level it is no longer abstract because we give an implementation for traverse.

I’ve also sketched another nested team showing a minimal specialization of AstVisiting for adapting some other visitor and adding another implementation for CustomAst.traverse plus potentially more roles for more node types.

Conclusion

For those who don’t work in the compiler business on a day-to-day basis this is probably pretty tough stuff, but let me summarize what we’ve just achieved:

• Embed a custom syntax into Java, showing how a custom parser can be plugged in to create custom AST from a region of the Java source.
• Adapt the conversion between two different AST structures (internal -> DOM) to also handle custom nodes.
• Adapt the code formatter so that hybrid sources can be formatted with a single command.
• Prepared the infrastructure for adapting other visitors, too. By this we have achieved that new visitor adaptations will only need to add their specific implementation with close to zero scaffolding.
• Cleanly separated each implemented concern in one module.
• Keep each module in the scale of only tens of lines of code.
• Yet implement significant steps towards a production quality IDE for our custom hybrid language.

Maybe I shouldn’t have told you, how easy these things can be - if your tools are sharp - maybe.
But professional carvers know: if your knife is sharp, it’s actually easy to handle. Only if it is blunt you are in real danger of hurting yourself - because you need to apply disproportionate force to cut your wood. So:

Spare your fingers, sharpen your knife!

PS: Here’s the archive of all sources, ready to be imported into the OTDT.

Have you ever thought of making Java a bit smarter? Perhaps, for some task you would prefer a custom syntax, and snippets using that syntax should then be embedded into Java? Sure, many never seriously think about this because of the prohibitively high effort to create the compiler for such hybrid language. And even if you are a compiler guru, knowing your toolkits so that translation wouldn’t be a problem for you, you’ll probably surrender at the mere thought of how to create a mature IDE that would allow efficiently productive work with you hybrid language.

You shouldn’t give up. Think: If you build your own IDE you’ll never be able to really compete with the JDT, right? Still anything falling back behind the quality of the JDT won’t raise your productivity but will stand in your way at the most common tasks during development, right?
What does this tell you? Give up? No. If you can’t beat us, join us. Don’t write a new IDE for any Java-based language. Join the JDT. Well, but the JDT doesn’t provide an extension point for embedding a different syntax, does it? Sure they don’t, but it’s actually not their job to do so because every embedded language will probably have slightly different requirements so designing such an extension point would be a battle you can never win.

I have developed a tiny extension to Java and integrated this into the JDT by a mere 204 lines of code including comments and a plugin.xml. As some may guess the only trick needed is to use Object Teams. By this post I will explain how Object Teams can be used for extending the JDT in this way. And I will also argue against the most common fear in this context: “Is that solution maintainable?” From my very own experience this design is not just barely manageable, but from all I’ve seen this is the best maintainable solution for this kind of task, but I’m getting ahead of myself.

In order not to distract from the interesting design issues I’ll be using the most simply language extension: I want to be able to write integer constants in natural language, and while I’m at it, I want it to work in an multilingual setting. So, this should, e.g., be a legal program:

public class EmbeddingTest {
    private static int foo() {
        return <% one %>;
    }
    public static void main(String[] args) {
        System.out.println(foo());
    }
}

I’m using <% and %> tokens to switch between Java syntax and custom syntax.

The first step can be achieved in plain Java, it’s creating a class for ASTNodes representing my custom int literals within the compiler. If you really want you may inspect class CustomIntLiteral, but it’s actually pretty boring old Java. Its main job is to lookup a given string from an array of known number words and thus translate the word into an int. It even detects the language used and remembers this for later use. The behaviour is hooked into the JDT compiler by overriding method TypeBinding resolveType(BlockScope scope) — just normal Java practice.

Drilling down into the example

Here’s an overview of the module that does all the rest:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

package embedding.jdt;
 
import org.eclipse.jdt.core.compiler.CharOperation;
import org.eclipse.jdt.core.compiler.InvalidInputException;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTNode;
import org.eclipse.jdt.internal.compiler.ast.Expression;
import org.eclipse.jdt.internal.compiler.ast.IntLiteral;
import org.eclipse.jdt.internal.compiler.parser.TerminalTokens;
 
import embedding.custom.ast.CustomIntLiteral;
 
import base org.eclipse.jdt.core.dom.ASTConverter;
import base org.eclipse.jdt.core.dom.NumberLiteral;
import base org.eclipse.jdt.internal.compiler.parser.Parser;
import base org.eclipse.jdt.internal.compiler.parser.Scanner;
 
public team class SyntaxAdaptor {
 
        /**
         * <h3>Part 1 of the adaptation:</h3>
         * Wait until '<' is seen and check if it actually is a special string enclosed in '<%' and '%>'.
         */
        protected class ScannerAdaptor playedBy Scanner { ... }
 
        /**
         * <h3>Part 2 of the adaptation:</h3>
         * If the ScannerAdaptor found a match intercept creation of the faked null expression
         * and replace it with a custom AST.
         *
         * This is a team with a nested role so that we can control activation separately.
         *
         * This team should be activated for the current thread only to ensure that
         * concurrent compilations don't interfere: By using thread activation any state of
         * this team is automatically local to that thread.
         */
        protected team class InnerCompilerAdaptor {
                /** This inner role does the real work of the InnerCompilerAdaptor. */
                protected class ParserAdaptor playedBy Parser { ... }
        }
 
        /**
         * Dom representation of CustomIntLiteral.
         * Since the constructor of NumberLiteral is package private we cannot subclass, so use a role instead.
         */
        protected class DomCustomIntLiteral playedBy NumberLiteral
                // don't adapt plain NumberLiterals, just those that already have a DomCustomIntLiteral role:
                base when (SyntaxAdaptor.this.hasRole(base, DomCustomIntLiteral.class))
        { ... }
 
        /**
         * <h3>Part 3 of the adaptation:</h3>
         * This adaptor role helps the ASTConverter to convert CustomIntLiterals, too.
         */
        @SuppressWarnings("decapsulation")
        protected class DomConverterAdaptor playedBy ASTConverter { ... }
}

Imports

Why am I showing you boring import declarations to begin with? Well, with OT/J there’s a fine distinction that is worth looking at: all imports starting with import base indicate that these classes are imported for attaching a role to them. So just from these lines you see that the given module adds roles to classes from org.eclipse.jdt.internal.compiler.parser and org.eclipse.jdt.core.dom (2 classes each). All other imports are plain Java imports and won’t let you apply any OT/J tricks.

Teams and Roles

Line 18 above tells you that the class SyntaxAdaptor is actually a team. Teams are used for grouping a set of roles - nested classes of a team. Using the playedBy keyword a role declares that it adapts the specified base class (which are the same classes we base-imported above). The purpose of these roles should be roughly clear by the doc comments.

So, role ScannerAdaptor will be responsible for switching between both syntaxes.

Role ParserAdaptor (line 40) will be responsible for creating our AST node (CustomIntLiteral). But wait, what’s that: the role is nested within an intermediate team, InnerCompilerAdaptor. This team will show you, how to define a role that is only effective in specific situations, here, the ParserAdaptor should only be effective after the ScannerAdaptor has detected a syntax switch. Details follow below.

The other two roles will do advanced stuff so I’ll discuss them later.

Role implementation (1)

Here is the full(!) code of role ScannerAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

protected class ScannerAdaptor playedBy Scanner {
 
    // access fields from Scanner ("callout bindings"):
    int getCurrentPosition()                     -> get int currentPosition;
    void setCurrentPosition(int currentPosition) -> set int currentPosition;
    char[] getSource()                           -> get char[] source;
 
    // intercept this method from Scanner ("callin binding"):
    int getNextToken() <- replace int getNextToken();
 
    callin int getNextToken() throws InvalidInputException {
        // invoke the original method:
        int token = base.getNextToken();
        if (token == TerminalTokens.TokenNameLESS) {
            char[] source = getSource();
            int pos = getCurrentPosition();
            if (source[pos++] == '%') {                                        // detecting the opening "<%" ?
                int start = pos;                                               // inner start, just behind "<%"
                try {
                    while (source[pos++] != '%' || source[pos++] != '>')       // detecting the closing "%>" ?
                        ;                                                      // empty body
                } catch (ArrayIndexOutOfBoundsException aioobe) {              // not found, proceed as normal
                        return token;
                }
                setCurrentPosition(pos);                                       // tell the scanner what we have consumed (pointing one past '>')
                int end = pos-2;                                               // position of "%>"
                char[] fragment = CharOperation.subarray(source, start, end);  // extract the custom string (excluding <% and %>)
                // prepare an inner adaptor to intercept the expected parser action
                new InnerCompilerAdaptor(fragment, start-2, end+1).activate(); // positions include <% and %>
                return TerminalTokens.TokenNamenull;                           // pretend we saw a valid expression token ('null')
            }
        }
        return token;
    }
}

Comments describing the logic are in the right column. Inline comments describe the usage of OT/J:

• Lines 3-6 define accessors for two fields from the base class Scanner.
• Line 9 defines that calls to method getNextToken() should be intercepted by our version of this method
• Line 11 marks the role method as callin which is a pre-requisite for line 13
• Line 13 invokes the original method from Scanner
• In line 29 we are in the situation that we have detected a region delimited by <% and %>. We have extracted the text fragment between delimiters, and we know the start and end positions within the source file. Only now we create an instance of InnerCompilerAdaptor and immediately activate it for the current thread (activate()).

At this point the ScannerAdaptor is done and now an InnerCompilerAdaptor is watching what comes next.

Here’s the nested team InnerCompilerAdaptor with its role ParserAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

protected team class InnerCompilerAdaptor {
 
    char[] source;
    int start, end;
    protected InnerCompilerAdaptor(char[] source, int start, int end) {
        this.source = source;
        this.start = start;
        this.end = end;
    }
 
    /** This inner role does the real work of the InnerCompilerAdaptor. */
    protected class ParserAdaptor playedBy Parser {
        // import methods from Parser ("callout bindings"):
        @SuppressWarnings("decapsulation")
        void pushOnExpressionStack(Expression expr) -> void pushOnExpressionStack(Expression expr);
 
        // intercept this method from Parser ("callin binding"):
        void consumeToken(int type) <- replace void consumeToken(int type);
 
        @SuppressWarnings("basecall")
        callin void consumeToken(int type) {
            if (type == TerminalTokens.TokenNamenull) {                 // 'null' token is the faked element pushed by the SyntaxAdaptor
                // this inner adaptor has done its job, no longer intercept
                InnerCompilerAdaptor.this.deactivate();
                // TODO analyse source to find what AST should be created
                Expression replacement = new CustomIntLiteral(source, start, end, start+2, end-2);
                this.pushOnExpressionStack(replacement);                // feed custom AST into the parser:
                return;
            }
            // shouldn't happen: only activated when scanner returns TokenNamenull
            base.consumeToken(type);
        }
    }
}

• Lines 3-4 define state of the nested team, which is used for passing the information collected by the ScannerAdaptor down the pipe
• Line 15 provides access to a protected method from Parser. By @SuppressWarnings("decapsulation") we document that this access inserts a tiny little hole into the encapsulation of Parser
• Line 18 defines a callin binding as we have seen it before.
• Line 24 already deactivates the enclosing InnerCompilerAdaptor, ensuring this is a one-shot adaptation, only.
• Line 26/27 perform the payload: feed a CustomIntLiteral node into the parser

Coming to life

Wow, if you’ve read so far, you’ve seen a lot of OT/J on just a few lines of code. Let’s wire things together, by throwing the code into an Object Teams Plug-in Project and declaring one extension:

I have defined one aspectBinding between the existing plugin org.eclipse.jdt.core and my team classes SyntaxAdaptor and InnerAdaptor (there’s a man behind the curtain pushing an ugly __OT__ prefix into the declaration, please ignore him - he’ll be gone in the next release of the tool).
Please note that for team SyntaxAdaptor I have set the activation to ALL_THREADS which means that at application launch an instance of this team will be created and activated globally. Without this flag the whole thing would actually have no effect at all.

That’s all the wiring needed, so kick up a runtime workbench, create a Java project and class, insert the code for class EmbeddingTest from the top of this post and boldly select Run As > Java Application. In the console we see a result:

1

Oops, the compiler for our little language extension already works? Did you see me writing a compiler?

Well, beginner’s luck, let’s assume. But, oops, watch this: When I mistype the return type of foo and ask the JDT for help, this is what I see:

The problem view tells me it knows that <% one %> has type int, which doesn’t match the declared return type boolean. Next I positioned the cursor on “one” (the element that’s definitely not Java) and hit Ctrl-1, and the standard JDT quickfix knows that I should change the return type of foo to int.
Did you watch me implementing a quickfix??

Summary so far

Here’re the stats:

• 204 lines of code including plugin.xml
• roles adapting two base classes from org.eclipse.jdt.core.
• callout bindings to two fields and one method
• callin bindings to two methods
• all adaptation is cleanly encapsulated in one team class. If you wish you could even deactivate this one team in a running workbench and thus disable all our adaptions with a single click.
• one plain Java class to implement the semantics of our extension

As for maintainability: The only dependencies are the items mentioned above: two classes, two fields and three methods. Only if one of these are modified under evolution, my adaptation has to be updated accordingly - and: if this happens I will definitely be told by the compiler because one of the bindings will break. If it doesn’t break there’s no need to worry.

With this implementation the compiler seamlessly works with our new syntax and even UI features that operate on the compiler AST can handle our extension, too.

What’s next?

I’m sure some think that the above is probably a forged example. You might challenge me to do something real, like refactoring. If you do so, you actually got me (mumble, mumble) - with the above implementation refactoring does not work with our custom syntax. Now that you’ve seen the start, what do you expect, how much additional rocket science does it take to add minimal refactoring support? (to be continued)

In response to our “Hello Eclipse” I was asked about “the distinction between OT, AOP and delegation”
and also Wayne suspected some overlap. So here’s an attempt at answering.

What OT/J is not

If the only problems you see with pure Java are of the kind as non-invasively adding
tracing/logging to a system, than you’re probably fine with AspectJ and OT/J does not
compete for a better solution in this discipline. This is because AspectJ is specialized
at defining sophisticated pointcuts: use powerful patterns and wildcards to capture
a large set of joinpoints that shall trigger your aspect.

I’m personally not enthusiastic about targeting problems of this category because
the focus is too narrow for my taste. So in this league where AspectJ performs best
OT/J is not applying as a replacement for AspectJ.

Goals for OT/J

Positively speaking, let’s call five toplevel goals for OT/J:

1. powerful modules
2. powerful ways of connecting modules
3. maximum support for re-use
4. evolvable architectures
5. intuitive metaphors

Most of these goals are so broad and common place, that we’ll soon agree that
we all scream for ice cream. For (1) classes, objects and bundles are a pretty good
starting point. Not much need to improve. For (3) & (4) the proof of the pudding is
only in the eating. You can’t directly boil them down to specific language features.
(5) is what makes a language suitable for manipulation by humans, it’s the least
technical goal in the list and thus ‘difficult’ to discuss among geeks

The issue of connecting modules (2) is, however, extremely interesting and creates
the backbone for anything we can say about an architecture. And this is where OT/J
excels in my view, based on three kinds of relations:

• OT/J takes inheritance to the extreme
• OT/J introduces a real meaningful containment relation
• OT/J introduces the role playing relation

Mentioning inheritance lets me add that we pay very close attention to not
side-step object orientation, but rather to put object orientation on steroids.

As I’m zooming in I will leave inheritance and containment aside for now
so as to focus on the role playing relation.

Role playing

The way role playing is defined in OT/J it is actually very similar to inheritance
with three decisive differences

• role playing is a dynamic thing happening among runtime instances
• role playing separates two sides of inheritance: acquisition and overriding
• control is more fine grained as individual methods (and fields) can be acquired
  and overridden selectively

This dynamism is one of the strongest points in OT/J: roles can be added to specialise
existing instances at any point during runtime and multiple roles can specialise the same
base instance simultaneously. Neither is possible if inheritance is defined between
classes rather than instances. Yet, OT/J is not careless about possible runtime effects,
so in order for a role instance to be attached to a base instance the role’s class must
statically declare a playedBy relation to the corresponding base class. The ensures that
possible runtime effects are analysable from the source code.

Ingredients to role playing

Separating acquisition from overriding yields the following pictures of possible
communications between a role and its base
- in all pictures assume a role class with this header:

public class ARole playedBy ABase { ...

Here we go:

Here class ABase implements baseMethod() which ARole would like to “inherit”.
It does so by this little callout declaration:

void roleMethod() -> void baseMethod(); // make baseMethod known under a new name

Now when a client sends a roleMethod() call to the :ARole instance, this is
automatically forwarded to the associated base instance, invoking its baseMethod().

Great, so a role may acquire individual methods from its base using callout.
No big deal so far.

Here’s the opposite direction:

This time the client talks to the :ABase instance saying “baseMethod()”.
Assume that the role has defined this callin binding:

void roleMethod() <- replace void baseMethod();

Now the original method call is intercepted and redirected to the role.
This has the same effect as overriding has in traditional inheritance.

The full glory only shines when both directions are involved in the same control flow:

This picture shows the role version of the template&hook pattern: :ARole inherits the
template method baseMethod2() which issues a self call to the hook method baseMethod1().
Even during this self call, method dispatch may be aware of the overridingM() in the role,
which intercepts the self call.

This situation is what is widely termed as delegation in the literature:
forwarding with the option to still override methods called within this control flow.

Comparison

Now that I have elaborated on the role playing relation in OT/J, how does it compare?
To AOP? To delegation?

Role playing vs. delegation

Role playing supports full delegation with overriding. In OT/J delegation is configured
selectively for individual methods whereas the declarative style of method bindings
keeps the effort at a minimum.

Furthermore, the effect of callin bindings can be controlled by several mechanisms
which I haven’t shown here (”team activation“, “guard predicates“), which means you
have the free choice between the weaker forwarding and the stronger delegation.

Delegation usually doesn’t imply overriding when directly addressing the base instance
as in the second picture. In OT/J you can freely choose, whether or not overriding
is effective in this situation.

Additionally, OT/J takes away the burden to manually manage the additional instances
involved in delegation. That’s what a team as the container for roles does for you.

Role playing vs. AOP

My explanation didn’t sound much like AOP, did it? The only connection here is in
the term “interception”. That’s the core mechanism that is used in both approaches.
Other than that I see little similarity.

In the same way as all languages providing dynamic method dispatch can solve a
similar set of design issues, also all languages providing method call interception
can solve similar issues. In OT/J we blend interception into the general concept
of dynamic dispatch as best as we can, so that it doesn’t stick out from other
concepts of object oriented programming. So, instead of featuring three new
concepts (”join point”, “pointcut”, “advice”) , OT/J only has callin bindings to declare
method call interception.

Two examples for those who like the details of what I mean by
“blend with other concepts”: “advice” in AspectJ is an oddish animal,
it is, e.g., impossible to override inherited advice.
Callin bindings refer to methods, which can be overridden like normal.
Also “aspects” are limited regarding inheritance: it is illegal to extend a
non-abstract aspect. Roles in OT/J have no such restriction.

All the rest

OK, role playing is key for re-using (and adapting) existing things.

Remember, role playing is only one of three strong ways in OT/J to specify connections
between modules. The enhanced inheritance and the strong containment relation
are both unrelated to AOP and delegation, but add even more value as they help to
create evolvable architectures.
It’s a major contribution of OT/J that these three ways of connecting things are not added
as isolated language features but in a way that creates the best synergy among them.

I even think that roles and teams are great metaphors representing the mechanisms
at hand in an intuitive way, but before you actually tried eating the pudding,
you may perhaps not feel this way - yet

I’d like to return the warm “welcome” Wayne recently pronounced under “Heady Times”.
The Object Teams Project is thrilled to finally move into the Eclipse house. I guess we are not complete strangers, as I’ve spoken at two EclipseCons and met quite a few of you in bugzilla etc. But still a majority might not be sure what the Object Teams Project is all about, so here comes a little introduction:

Origin

Formally, the Object Teams Project is a cross-over between ambitious research in language design and an extremely pragmatic approach to writing highly modular code even under high pressure of re-use, maintenance, evolution and that sort of conditions. Our pragmatism made us refrain from designing a language from scratch but made us create OT/J as an extension of Java. This pragmatism has furthermore led us to spending >99% of our time on building actual working tools in contrast to <1% of tweaking new language features. That’s why we decided as early as 2003 to not develop a standalone compiler for our language but to base everything on the JDT (v2.1.1!) so we would soon have a full-featured, high-quality IDE. From this the Object Teams Development Tooling (OTDT) was born which had its first public appearance in 2005. This development was a tremendous joint effort (2003-2006) of the Fraunhofer FIRST institute and the Technical University Berlin.

Two ways of looking at Object Teams

(1) Conceptually: Object Teams solves the dilemma of structuring software by functions (procedural or functional approaches) or by data (object oriented approach). In Object Teams you do both simultaneously. Suppose you are writing a banking application. After some core concepts have been implemented and tested you are about to implement a new use case, say Wire-Transfer. Do you add new methods to existing classes like Customer, Account, ATM etc? Or do you consider those classes as dumb data objects and create one new class WireTransfer? The Object Teams answer: you do the best of both approaches: you create a role for each relevant class, implement the new behaviour in terms of role methods calling each other. Roles are grouped into a team (here: WireTransfer) and roles are bound to the underlying core objects (here: of type Customer, Account etc.). - Looking for a data-module, an entity? Just inspect the Account together with all its roles. - Looking for a behavioral module? Just inspect the WireTransfer team with all its contained roles. Voila.

Sounds esoteric? “Role modeling”, “collaboration based design” and similar off-springs of the same ideas have been embraced for analysis and design for quite some time (cf. OOram, Catalysis etc.). Object Teams finally gives a natural implementation to such models. Finally we can cleanly compose a system from one module per use case.

It turns out, however, that you may also look at Object Teams from an even more pragmatic point of view:

(2) Pragmatically: As one of its core capabilities a role can override a method of its bound base. Given that roles can be added to existing entities at any time (add role classes at compile-time, add role instances to base instances even at run-time), roles provide a mechanism for extending and adapting existing things without touching them (”non-invasively”). As an example consider the JDT/UI: a plug-in providing an excellent user experience for working with Java code. Guess what: just by adding a number of roles and without changing a single source line of the the JDT/UI we created the OTDT/UI: a plug-ing providing an excellent user experience for working with OT/J code .

While the mere possibility to non-invasively adapt existing plug-ins is fascinating by itself, using Object Teams for this task also guides you towards a code structure that is actually easy to read and highly suitable for future maintenance and evolution.

OT/Equinox

Here is why all this is highly relevant for everyone writing Eclipse plug-ins: we have married OT/J to Equinox (first presented at Eclipse Technology eXchange 2006 in Nantes) so that all the concepts of Object Teams can seamlessly be leveraged when developing plug-ins. The consequences of this I will leave for your imagination at this point. More to follow soon.

Show me the code

This post has become a little verbose, with no real examples. Some examples will follow in subsequent posts. In the meantime you may want to look at the

• Object Teams Quick-Start with actual code to pimp the Eclipse workbench “in less then 15 minutes”.

What’s next?

Currently our initial contribution awaits the initial OK so that we can start committing to SVN. Pretty soon after you should expect the first milestone release (incubation) from eclipse.org. “If all goes well” we will apply for graduation by the time of the Helios release, so you would see an OTDT 2.0.0 accompanying the Helios GA. After 70+ releases (incl. milestones) in our previous identity at objectteams.org I say we should be able to demonstrate the maturity required for that next big step.

Please stay tuned and join the discussion in our forum.

Thanks to everybody who made our move to Eclipse.org possible!

OK, I use Eclipse for getting some work done. Eclipse is software so we know it contains bugs. Given that Eclipse is open source software, we all can only expect it to run smoothly if we diligently report back all errors we encounter (and provide steps on how to reproduce etc.). I know all that and I really want to be a good bug reporter because I really want a good experience using Eclipse because I really want to get the work done (and I may even want to sustain the impression that I’ve got everything under control and thus working with Eclipse is a delight).

A task

The other day, I was a very bad bug reporter, and only today I find some time to reason about what happened. This was my task: In preparing the initial contribution for the Object Teams Project I had to rename a bunch of things from org.objectteams to org.eclipse.objectteams. Simply, huh? Back in the days of emacs+bash I might have chosen to just use one big

find . -exec /bin/sed -i -e "s/org.objectteams/org.eclipse.objectteams/g" {} \;

and voila, if fortuna was with me, I might have been done at the return of that single command. But things were actually just a little bit more challenging, like, a few occurrences would have to remain unchanged plus while touching about every single file in our software I was going to also do some clean-up: rename some packages to foo.internal.bar, fixing some copyright headers, etc. Also I preferred to change one plug-in at a time which would mean that all references to plug-ins not yet processed should stay unchanged, too. Plus a few more little deviations from the grand search-and-replace-globally.

OK, since I like to see myself an Eclipse-wizard this was a nice challenge for its refactoring support. Plug-in by plug-in I renamed, I renamed packages with / or without subpackages, and after each step I wanted to see that compiler and PDE agree with all changes and signal that everything is still (or again) consistent, ready to be built, actually. Perhaps things started the get wrong when I estimated the effort as one or two hours. So, after a day or so, I wasn’t perfectly relaxed any more. My fault, should’ve known better about that estimate. BTW, one of the reasons it took so long was simply the size of my workspace in comparison to the power of my computer / hard-disk: every time I performed a rename with updates in non-Java files, I was nervously looking at the screen: “should I sit and wait for the preview page, or should I go to the kitchen, get a chocolate, coffee, just something?“. I did some emailing in parallel, but let’s just keep this: due to those response times I wasn’t perfectly relaxed any more.

A story of bug reporting

What I was going to tell here is a story of bug reporting, because as a safe bet doing a real-life stress test to an Eclipse component should give you a good chance to discover and report a few bugs that have not yet been reported by others. And indeed, I was successful in discovering some bugs, in various components actually.

I think one of the first things that occurred was that the svn synchronize view would sometimes fail to open the compare editor, more precisely, I had to explicitly close the compare editor before comparing the next file. At first this really **** me off, because the error dialog was popping up in some kind of infinite loop. Fun!#$ Once I’d figure out how to work around this problem it soon became a habit to just close the compare editor before clicking the next. Next, the svn plugin made a refactoring fail, because it was trying to create a directory which the previous refactoring had already created. The most creative bug having to do with subversive was a chain-reaction of first failing to undo a refactoring and than during reporting this error blocking the UI of Eclipse so I could only kill the Eclipse process, insert a new coin and start a new game.

I don’t intend to blame a particular component. For clean-up of license headers I have a little home-grown plugin that I just wanted to quickly install into the running Eclipse instance, so I went for the cool new feature to export/install into the host. Oops, my plugin depends on another plugin that only exists in the workspace but not in the host, install failed for good reasons. I removed the dependency and tried again. Installation still failed for the same reason: the ghost of this removed dependency prevented installation into the host Eclipse. Oh, I should have incremented the version or let a version qualifier do this automatically, of course. Tried again, still failed. Tried something so slightly different I cannot recall, from there on it worked. Can I reproduce the two or three different levels of failure? I didn’t even take the time to think of it. Well I would’ve been disappointed without a bug from p2 in this list .

PDE did its share by reporting overlapping text edits in plugin.xml and therefore disabling its refactoring participant. What the **** caused those overlapping text edits, and how do I re-enable the refactoring participant to give it one more chance to behave well?

The list could go on if only I could remember. Instead I was happy to finish this 1.5 hours task after 2.7 days, ready to submit our initial code contribution, wow!

Looking back, I / we missed a great opportunity: we could have identified plenty of bugs in various components of Eclipse. With only a few more days of debugging I might have been able to present reproducing steps for all those bugs. And, if triaged and fixed by the corresponding devs, this might have contributed to M6 containing fewer of those bugs that just only occur in real world, never during testing. I failed, I managed only to submit two bug reports, with very little information on how to reproduce.

Lesson learned

Susan McCourt responded to an earlier bug report of mine in a very descriptive comment:

That is one of those things I’ve been meaning to fix forever, never wrote a
bug, and so keep forgetting to fix. And it seems like if I’m actually
[doing what triggers the bug], it’s because something is wrong, and so I again postpone
writing a bug.

Sure, when we hit a bug (or a bug hits us) we are always in some context of doing something challenging. Something that requires our mind to stay in focus. Something we want to get done.
Well, work isn’t perfectly linear, so we know how to react to interrupts. Bugs are such interrupts. Sometimes I like the challenge of isolating a bug etc. Sometimes I’m sufficiently relaxed when the bug occurs so I actually take the challenge. Sometimes the bug is sufficiently good-natured so making a small note and going back to the bug after the actual work is done is a perfect strategy. Some bugs, however, smell like depending on so many factors from your current context that reproduction an hour later seems extremely unlikely.

I think I have a solution to all this: given we don’t want to be distracted from our actual work, given also that some bugs need immediate care or they will escape our attempts to identify. Given some of the worst moments are when we start to isolate a bug and during that task a second bug stops us from working on the first bug etc. The only way to relentlessly follow all those tasks is to enable branching universes in your working environment. The frequent use of the work “task” may give a hint that I should finally start using Mylyn (I have no excuse for not doing so), but I would need a Mylyn that is able to capture full contexts: the complete state of my machine plus the full state of my brain. As a start I’m dreaming of always working in a virtual machine, and whenever something strange happens, I quickly (!!) create a snapshot of the virtual machine. Then I could first isolate (and fix ) the bug that just occurred, and then go back to the exact point where I interrupted my work and act as if nothing had happened. Branching universes with the ability of back porting fixes between branches is what I need. Of course the clock needs to be reset when returning from a bug reporting / fixing branch.

Yeah, that’s why I can’t quite live up to my dreams of good participation in open source development: I haven’t figured out how to enable branching universes for my character. If anybody has a hint on how to do this or any other strategy to not get overwhelmed between work and bug reporting, I’d really appreciate.

And if I don’t smile while writing a bug report, please excuse, I might just be terribly stressed because your bug interrupted my work on isolating another bug that stopped me from doing …

”Always look at the bright side of life…”

Welcome to The Object Teams Blog.

Here you will find various examples of how OT/J can be used to create great software, plus updates on recent developments.

Why do we bother to run our own instance of wordpress? Simply because we want syntax highlighting for our code snippets, and we do want it for OT/J code, too.

So, here’s a welcome also from OT/J:

1
2
3
4
5
6
7
8

public team class HelloOTJ {
    protected class HelloAdaptor playedBy HelloWorld {
        void hello2 () {
            System.out.println("Hello from OT/J, too!");
        }
        hello2 <- after main;
    }
}