The Object Teams Blog

Less than a week ago I happily announced that Object Teams is on the Indigo Train.
Much water has gone under the bridge since then and the above statement is history.

Events where triggered by what was actually a little bug in the b3 aggregator. Only by way of this bug some people noticed that there was a ”’patch feature”’ inside the repository, i.e., a feature (”Object Teams Patch for JDT/Core”) that replaces the jdt.core plugin with a variant.

One part of me is very happy this bug occurred because finally an issue got the attention I had tried to raise at various occasions before. The lesson is:

(see this post, e.g.).

The other part of me got very worried because during that debate some harsh statements occurred that would effectively amount to excluding Object Teams from Eclipse.org. That’s a little more attention than I had intended. As in any heated debate some of the arguments sounded to me more like ideology than anything that could possibly be discussed open-mindedly.

I had mixed feelings regarding the technical scope: the outcry only banned one specific technology: patch features. I don’t see how the goal to protect a project’s bits and bytes against influence from other projects can be achieved without also discussing: access to internal, byte-code weaving and - worst of all, I believe - reflection. To be perfectly open: Object Teams uses all these techniques except for reflection. Personally, I would even argue for banning projects that do use setAccessible(true), but that’s not a realistic option because then quite likely the whole Train would dissolve into a mist.

I am actually guilty of a technical simplification during this debate: I focused too much on the idea that a user would explicitly select features to install, not accounting for the possibility that the jdt.core plugin can well be pulled in invisibly due to dependencies among plugins. So, yes, if Object Teams would still be in the repository, and if a user installed a package without the JDT and if that user did never select to install the JDT and if that user selects another feature that implicitly requires the jdt.core plugin, then that user would unexpectly install the OT variant of the jdt.core. I agree that this is not ideal. I personally would have been happy to take this risk because I know how thoroughly the OT variant is tested for compatibility. And for the remaining minuscule risk I would have been happy to promise same-day fixes. But risk assessment naturally depends on perspective and I understand that others come to different conclusions when weighing the issues.

From two days distance I can already laugh at one implication of the central argument, paraphrased as: the JDT/Core team must be protected against harmful actions from the OT team. When spelling this out in names, one of the sentences reads: “Stephan Herrmann must be protected against harmful actions by Stephan Herrmann”. I should really be careful, because I’ll never be able to escape him!

Where to go?

• As we’re banned from the Train, I willed hurriedly book a plane ticket to Indigo. Make sure we come by a Graduation office on the way.
• I do hope that bug 316702 get’s sufficient attention now. Seriously: if you are so detrimentally determined about patch features, then the UI must report it. And if it reports this, it might as well report other techniques that have similar effects!
• I appreciate any offers for helping OT/J towards a solution that avoids replacing a plugin. As of today and after more than seven years of looking at this, I see no way how this can be done, but that doesn’t mean we shouldn’t try still harder.

Unfortunately, the debate consumed all the time and energy I had planned for preparing a presentation at the EclipseCON Audition. However, Lynn finally made my day by letting me know that my submission is the lucky #42. So I’m making progress towards my all-seasons Eclipse collection

I’m flattered to see my name appear on one more Eclipse web-page:

Thanks to the JDT/Core team for accepting me as a new member!

Actually, I’ve been poking about the JDT source code since 2003 when we started implementing the Object Teams Development Tooling (OTDT) on top of the JDT. So it was only natural that I would stumble upon a bug in the JDT every once in a while (incl. one “greatbug” ). In this specific situation, reporting bugs and trying to help find solutions was more than just good Eclipse citizenship: the discussions in bugzilla also helped me to better understand the JDT sources and thus it helped me developing the OTDT. And that’s what this post is about: synergy!

Personal Goals

Actually, watching the JDT/Core bug inbox provides a constant stream of cool riddles: spooky Java examples producing unexpected compiler/runtime results. Sometimes those are real fun to crack. Which compiler is right, which one isn’t? What’s the programmer trying to do? I’d say there’s no better way to test your Java knowledge

Also, watching this stream helps me stay informed, because every patch will eventually need to be merged into the Object Teams branch of the JDT/Core.

But I also have some pet projects that I’d like to actively push forward. Currently I’m thinking about these two:

• Supporting inter-procedural null analysis
• Making the compiler closer match the OSGi semantics

Inter-procedural null analysis

Naturally, the JDT/Core is a facilitator for tons of cool functionality in the IDE, but it hardly communicates directly with the user. But we have one channel, where the compiler can actually talk to the developer and give advice that’s well worth your bucks: errors and warnings. Did you know, that the compiler can immediately tell you what’s wrong about this piece of code:

void foo(String in) {
   if (in == null) new IllegalArgumentException("Null is not allowed here");
   ... // real work done here
}

(Yep, that was bug 236385, see the New&Noteworthy on how to enable).

Many Java developers share the experience that one of the most frequent problems is also one of the most mundane: NPE. Those, who are aware of this easily produce the opposite problem: cluttering there code with meaningless null-checks (many of which have no better action than the above IllegalArgumentException which isn’t much better than throwing the NPE in the first place). The point is, we’d want to know which values can actually be null (and why!) so we write only the necessary null-checks and for those we should think really hard what would be a suitable action, right? (“This shouldn’t happen” is not an answer!).

Thus I’m happy I could get involved in some improvements of the compiler’s flow analysis. I also experienced that null-analysis combined with aggressive optimization is a delicate issue - remember SDK 3.7M2a? Sorry about that, but to my justification I might add that bug 325755 has always been dormantly present, I only woke’em up. I must admit for those 105 minutes my heart beat was a bit above average .

Anyway, the current null analysis is still fundamentally limited: it knows nothing about nullness of method arguments nor method call results. So it would be a real cool enhancement if we could feed such information into the compiler. Fortunately, the theory of how to do this (you may either think of improved type systems or of design by contract) is a well-explored subject. Actually, bug 186342 already has some patches in this direction. Since discussing in bugs with long history is a bit tedious I created this wiki page. Sadly, it’s not a technical difficulty that’s keeping us from immediately releasing that patch, but a stalled standardization process .

However, as outlined in the wiki, here comes another synergy: in case we can’t find a solution that will be sufficiently compatible with future standards, I can easily create an early-adopters release, so sorting out the details of implementation and usage can be done in parallel with the standardization process. How that? I’d simply ship the compiler enhancement as an OT/Equinox-enabled plug-in. This would allow us to ship the compiler enhancement as separate plug-in ready to be tried by early adopters.

OSGi-aware compiler

Some of you may have noticed (here, or here, or here, or here or one of the many duplicates) that the compiler’s concept of a (single, linear) classpath is not a good match for compiling OSGi code. I will soon write another wiki page with my current thinkings about that issue. Stay tuned …

Comments?

For both issues I’d appreciate any comments / feedback. I’ll be checking updates of the bugs I listed, or the wiki page or … talk to me during ESE! Let’s move something together.

See you all in Ludwigsburg!
Stephan

Ralf Ebert recently blogged about how he extended Java to support a short-hand notation for throwing exceptions, like:

   throw "this is wrong";

It’s exactly the kind of enhancement you’d expect from Project Coin, but neither do they have it, nor would you want to wait until they release a solution.

At this point I gave it a few minutes, adapted Ralf’s code, applied Olivier’s suggestion wrapped it in a little plugin et voilà:

Install

Use this p2 repository, check two features…

…install and restart, and you’re ready to use your “Medal” IDE:

So that’s basically the same as what Ralf already showed except:

It’s a module!

In contrast to Ralf’s patch of the JDT/Core my little plugin can be easily deployed and installed into any Eclipse (≥3.6.0). It just requires another small feature called “Object Teams Equinox Integration” or “OT/Equinox” for short.

So we’re all going to use our private own ”’dialects of Java?”’ Hm, firstly, once compiled this is of course plain Java, you wouldn’t be able to tell that the sources looked “funny”.

And: here’s the Boss Key: when somebody sniffs about your monitor, a single click will make Eclipse behave “normal”:

In other words, you can ”’dynamically enable/disable”’ this feature at runtime. The OT/Equinox Monitor view in the snapshot shows all known Team instances in the currently running IDE, and the little check boxes simply send activate() / deactivate() messages to the selected instance.

I coined the name Medal as our own playground for Java extensions of this kind. Feel free to suggest/contribute more!

Implementation

For a quick introduction on how to setup an OT/Equinox project in Eclipse I’d suggest our Quick Start (let me know if anything is unclear). For this particular case the key is in defining one little extension:

which the package explorer will render as:

Drilling into the Team class ThrowString you’ll see:

The Team class contains two Role classes:

• Role DontReport binds to class ProblemReporter (not shown in the Outline), intercepts calls to ProblemReporter.cannotThrowType and if the type in question is String, simply ignores the “error”
• Role Generate binds to class ThrowStatement to make sure the correct bytecodes for creating a RuntimeException are generated

Also, in the Outline you see both kinds of method bindings that are supported by Object Teams:

• getExceptionType/setExceptionType are getter/setter definitions for field ThrowStatement.exceptionType (callout-to-field in OT/J jargon)
• Things like “adjustType <- after resolve” establish method call interception (callin bindings in OT/J jargon - the “after” is symbolized by the specific icon)

The actual implementation is really simple, like (full listing of the first role):

protected class DontReport playedBy ProblemReporter {
	cannotThrowType <- replace cannotThrowType;
 
	@SuppressWarnings("basecall")
	callin void cannotThrowType(ASTNode exception, TypeBinding exceptionType) {
		if (exceptionType.id != TypeIds.T_JavaLangString)
			// do the actual reporting only if it's not a string
			base.cannotThrowType(exception, exceptionType);
	}		
}

The base-call (base.cannotThrowType) delegates back to the original method, but only if the exception type is not String. The @SuppressWarnings annotation documents that not all control flows through this method will issue a base-call, a decision that deserves a second thought as it means the base plugin (here JDT/Core) does not perform its task fully as usual.

Intercepting resolve has the purpose of replacing type String with RuntimeException so that other parts of the Compiler and the IDE see a well-typed structure.

The method that performs the actual work is generateCode. Since this method is essentially based on the original implementation, the best way to see the difference is (select either the callin method or the callin binding):

which gives you this compare editor:

This neatly shows the two code blocks I inserted, one for creating the RuntimeException instance, the other for invoking its constructor. Or, if you just want to read the full role method:

/* This method is partly copied from the base method. */
@SuppressWarnings({"basecall", "inferredcallout"})
callin void generateCode(BlockScope currentScope, CodeStream codeStream) {
	if ((this.bits & ASTNode.IsReachable) == 0)
		return;
	int pc = codeStream.position;
	// create a new RuntimeException:
	ReferenceBinding runtimeExceptionBinding = (ReferenceBinding) this.exceptionType;
	codeStream.new_(runtimeExceptionBinding);
	codeStream.dup();
	// generate the code for the original String expression:
	this.exception.generateCode(currentScope, codeStream, true);
	// call the constructor RuntimeException(String):
	MethodBinding ctor = runtimeExceptionBinding.getExactConstructor(new TypeBinding[]{this.stringType});
	codeStream.invoke(Opcodes.OPC_invokespecial, ctor, runtimeExceptionBinding);
	// throw it:
	codeStream.athrow();
	codeStream.recordPositionsFrom(pc, this.sourceStart);			
}

You may also fetch the full sources of this little plug-in (plus a feature for easy deployment) to play around with and extend.

Next?

Ralf mentioned that he’d like to play with ways for also extending the syntax. For a starter on how this can be done with Object Teams I recommend my previous posts IDE for your own language embedded in Java? (part 1) and part 2.

During the last week or so I modernized a part of the Object Teams Development Tooling (OTDT) that had been developed some 5 years ago: the type hierarchy for OT/J. I’ll mention the basic requirements for this engine in a minute. While most of the OTDT succeeds in reusing functionality from the JDT, the type hierarchy was implemented as a full replacement of the original. This is a pretty involved little machine, which took weeks and months to get right. It provides its logic to components like Refactoring and the Type Hierarchy View.

On the one hand this engine worked well for most uses, but over so many years we did not succeed to solve two remaining issues:

Give a faithful implementation for getSuperclass()

This is tricky because a role class in OT/J can have more than one superclass. Failing to implement this method we could not support the “traditional” mode of the hierarchy view that shows both the tree of subclasses of a focus type plus the path of superclasses up to Object (this upwards path relies on getSuperclass).

Support region based hierarchies

Here the type hierarchy is not only computed for supertypes and subtypes of one given focus type, but full inheritance structure is computed for a set of types (a “region”). This strategy is used by many JDT Refactorings, and thus we could not precisely adapt some of these for OT/J.

In analyzing this situation I had to weigh these issues:

• In its current state the implementation strategy was a show stopper for one mode of the type hierarchy view and for precise analysis in several refactorings.
• Adding a region based variant of our hierarchy implementation would mean to re-invent lots of stuff, both from the JDT and from our own development.
• All this seemed to suggest to discard our own implementation and start over from scratch.

Object Teams to the rescue: Let’s re-build Rome in ten days.

As mentioned in my previous post, the strength of Object Teams lies in building layers: each module sits in one layer, and integration between layers is given by declarative bindings:

Applying this to the issue at hand we now actually have three layers with quite different structures:

Java Model

The bottom layer is the Java model that implements the containment tree of Jave elements: A project contains source folders, containing packages, containing compilation units, containing types containing members. In this model each Java type is represented by an instance of IType

Java Type Hierarchy

This engine from the JDT maintains the graph of inheritance information as a second way for navigating between ITypes. Interestingly, this module pretty closely simulates what Object Teams does natively, I may come back to that in a later post.

Object Teams Type Hierarchy

As an extension of Java, OT/J naturally supports the normal inheritance using extends, but there is a second way how an inheritance link can be established: based on inheritance of the enclosing team:

team class EcoSystem {
   protected class Project { }
   protected class IDEProject extends Project { }
}
team class Eclipse extends EcoSystem {
   @Override
   protected class Project { }
   @Override
   protected class IDEProject extends Project { }
}

Here, Eclipse.Project is an implicit subclass of EcoSystem.Project simply because Eclipse is a subclass of EcoSystem and both classes have the same simple name Project. I will not go into motivation and consequences of this language design (that’ll be a separate post — which I actually promised many weeks ago).

Looking at the technical challenge we see that the implicit inheritance in OT/J adds a third layer, in which classes are connected in yet another graph.

Three Layers — Three Graphs

Looking at the IType representation of Eclipse.IDEProject we can ask three questions:

Each question is implemented in a different layer of the system. Things get a little complicated when asking a type for all its super types, which requires to collect the answers from both the JDT hierarchy layer and the OT hierarchy. Yet, the most tricky part was giving an implementation for getSuperclass().

An "Impossible" Requirement

There is a hidden assumption behind method getSuperclass() which is pervasive in large parts of the implementation, especially most refactorings:

When searching all methods that a type inherits from other types, looping over getSuperclass() until you reach Object will bring you to all the classes you need to consider, like so:

IType currentType = /* some init */;
while (currentType != null) {
   findMethods(currentType, /*some more arguments*/);
   currentType = hierarchy.getSuperclass(currentType);
}

There are lots and lots of places implemented using this pattern. But, how do you do that if a class has multiple superclasses?? I cannot change all the existing code to use recursive functions rather than this single loop!

Looking at Eclipse.IDEProject we have two direct superclasses: Eclipse.Project (normal inheritance, “extends”) and EcoSystem.IDEProject (OT/J implicit inheritance), which cannot both be answered by a single call to getSuperclass(). The programming language theory behind OT/J, however, has a simple answer: linearization. Thus, the superclasses of Eclipse.IDEProject are:

• Eclipse.IDEProject → EcoSystem.IDEProject → Eclipse.Project → EcoSystem.Project

… in this order. And this is how this shall be rendered in the hierarchy view:

The final callenge: what should this query answer:

        getSuperclass(ecoSystemIDEProject);

According to the above linearization we should answer: Eclipse.Project, but only if we are in the context of the superclass chain of Eclipse.IDEProject. Talking directly to EcoSystem.IDEProject we should get EcoSystem.Project! In other words: the function needs to be smarter than what it can derive from its arguments.

Layer Instances for each Situation

Let’s go back to the layer thing:

At the bottom you see the Java model (as rendered by the package explorer). In the top layer you see the OT/J type hierarchy (lets forget about the middle layer for now). Two essential concepts can be illustrated by this picture:

• Each layer is populated with objects and while each layer owns its objects, those objects connected with a red line between layers are almost the same, they represent the same concept.
• The top layer can be instantiated multiple times: for each focus type you create a new OT/J hierarchy instance, populated with a fresh set of objects.

It is the second bullet that resolves the “impossible” requirement: the objects within each layer instance are wired differently, implementing different traversals. Depending on the focus type, each layer may answer the getSuperclass(type) question differently, even for the same argument.

The first bullet answers how these layers are integrated into a system: Conceptually we are speaking about the same Java model elements (IType), but we superimpose different graph structure depending on our current context.

Inside the hierarchy layer, we actually do not handle IType instances directly, but we have roles that represent one given IType each. Those roles contain all the inheritance links needed for answering the various questions about inheritance relations (direct/indirect, explicit/implicit, super/sub).

A cool thing about Object Teams is, that having different sets of objects in different layers ( teams) doesn’t make the program more complex, because I can pass an object from one layer into methods of another layer and the language will quite automagically translate into the object that sits at the other end of that red line in the picture above. Although each layer has its own view, they “know” that they are basically talking about the same stuff (sounds like real life, doesn’t it?).

Summing up

OK, I haven’t shown any code of the new hierarchy implementation (yet), but here’s a sketch of before-vs.-after:

Code Size

The new implementation of the hierarchy engine has about half the size of the previous implementation (because it need not repeat anything that’s already implemented in the Java hierarchy).

Integration

The previous implementation had to be individually integrated into each client module that normally uses Java hierarchies and then should use an OT hierarchy instead. After the re-implementation, the OT hierarchy is transparently integrated such that no clients need to be adapted (accounting for even more code that could be discarded).

Linearization

Using the new implementation, getSuperclass() answers the correct, context sensitive linearization, as shown in the screenshot above, which the old implementation failed to solve.

Region based hierarchies

The old implementation was incompatible with building a hierarchy for a region. For the new implementation it doesn’t matter whether it’s built for a single focus type or for a region, so, many clients now work better without any additional efforts.

The previous implementation only scratched at the surface – literally worked around the actual issue (which is: the Java type hierarchy is not aware of OT/J implicit inheritance). The new solution solves the issue right at its core: the new team OTTypeHierarchies assists the original type hierarchy (such that its answers indeed respect OT/J’s implicit inheritance). By performing this adaptation at the issue’s core, the solution automatically radiates to all clients. So I expect that investing a few days in re-writing the implementation will pay off in no time. Especially, improving the (already strong) refactoring support for OT/J is now much, much easier.

Need I say, how much fun this re-write was?

In the first part I demonstrated how Object Teams can be used to extend the JDT compiler for a custom language to be embedded in Java. I concluded by saying that more substantial features like Refactoring might need more rocket science which I wanted to show next.

The “bad news” is: before I started to do some strong adaptations of DOM AST etc to make Refactorings work, I just made a few experiments of how Refactorings actually behaved in my hybrid language. To my own surprise a lot of things already worked OK: I could extract a custom syntax expression into a local variable and inline the variable again and more stuff of that kind. Just look at this example:

Actually this reflects an experience I’ve made more than once: If you reuse some module and perform some adaptations in terms of provided API and extension points etc. more often than not one adaptation entails the next, adding tweaks to workarounds because you keep scratching at the surface. If, OTOH, you succeed to make your adaptation right at the core where the decisions are made, just one or two cuts and stitches may suffice to get your job done. Clean, effective and consistent. That’s what we see when cleanly inserting a custom AST node into the JDT: if our CustomIntLiteral behaves well a lot of JDT functionality can just work with this thing without knowing it’s not a genuine Java thing.

Now this means for my next example I had to look for an extra challenge. I decided to enhance the example in two ways:

• The custom syntax should be a bit more realistic, so I chose to create a syntax for money, consisting of a number and the name of a currency
• I wanted source formatting to work for the whole hybrid language

A word of warning: this post uses some bells and whistles of OT/J and applies it to the non-trivial JDT. This might be a bit overwhelming for the novice. If you prefer lower dosage first, you may want to check out our example section in the wiki. It’s still far from complete but I’m working on it.

A syntax for money

The new syntax should allow me to write this:

int getMoney() {
    return <% 13 euro %>;
}

and the stuff should internally be stored as a structured AST node. This is how class CurrencyExpression starts:

public class CurrencyExpression extends Expression {
 
    public IntLiteral value;
    public String currency;
 
    final static String[] CURRENCIES = { "euro", "dollar" };
 
    public CurrencyExpression(int sourceStart, int sourceEnd) { ...
 
    public boolean setCurrency(String string) { ...
 
    @Override
    public StringBuffer printExpression(int indent, StringBuffer output) { ...
    ....
}

For creating a CurrencyExpression from source I wrote a little CustomParser, normal boring stuff with 40% just reading individual chars and manipulating character positions, another 45% actually does some error reporting and only 3 lines are relevant: those that create a new CurrencyExpression, create an IntValue for the value part and invoke setCurrency with the currency string.

In the ScannerAdaptor from the previous post I simply replaced this

Expression replacement = new CustomIntLiteral(source, start, end, start+2, end-2);

with this:

Expression replacement = customParser.parseCurrencyExpression(source, start, end, this.getProblemReporter());

That suffices to make the above little method compile and run just as expected.

Interlude: DOM AST

Well, with this slightly more realistic syntax you’d actually see a number of exceptions in the IDE that can all be fixed by letting the DOM AST know about our addition. For those who don’t regularly program against the JDT API: the DOM AST is the public data structure by which tools outside the JDT core manipulate Java programs. Inside the JDT extending the DOM AST would mean to subclass either org.eclipse.jdt.core.dom.ASTNode or one of its subclasses. Unfortunately, all constructors in this hierarchy are package private, and even with OT/J we respect what the javadoc says: “clients are unable to declare additional subclasses“.

But we can do something similar: instead of subclassing we can use instances of a regular DOM class and attach a role instance to them. As the base I chose org.eclipse.jdt.core.dom.SimpleName which inside the JDT could mean a lot of different things, so for most parts a node of this kind is regarded as a black box, just what we need. This is the role I added to the team SyntaxAdaptor from the previous post:

1
2
3
4
5
6
7
8
9
10
11
12

protected class DomCurrencyLiteral playedBy SimpleName {
    protected String currency;
 
    void setSourceRange(int sourceStart, int length) -> void setSourceRange(int sourceStart, int length);
 
    @SuppressWarnings("decapsulation")
    public DomCurrencyLiteral(AST ast, CurrencyExpression expression) {
        base(ast);
        this.currency = expression.currency;
        setSourceRange(expression.sourceStart, expression.sourceEnd-expression.sourceStart+1);
    }
}

So this almost looks like subclassing except we use playedBy instead of extends and base() instead of super(). And yes, when creating an instance with “new DomCurrencyLiteral(ast, expr)” inside the constructor we create a SimpleName from DOM using the package private constructor. But by using role playing instead of sub-classing this has become part of the aspectBinding relationship, which makes analysis of the state of encapsulation much easier.

So, who actually creates these nodes? Inside the JDT this is the responsibility of the ASTConverter, which takes an AST from the compiler and converts it to the public variant. In order to tell the ASTConverter how to handle our currency nodes I added this role to the existing team SyntaxAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

@SuppressWarnings("decapsulation")
protected class DomConverterAdaptor playedBy ASTConverter {
 
    // whenever convert(Expression) is called ...
    org.eclipse.jdt.core.dom.Expression convertCurrencyExpression(CurrencyExpression expression)
    <- replace org.eclipse.jdt.core.dom.Expression convert(Expression expression)
        // ... and when the literal is actually a CurrencyExpression ...
        base when (expression instanceof CurrencyExpression)
        // ... perform the cast we just checked for and feed it into the callin method below.
        with { expression <- (CurrencyExpression)expression }
 
    /**
     * Convert a CustomIntLiteral from the compiler to its dom counter part.
     * This method uses inferred callouts (OTJLD §3.1(j))
     * which need to be enabled in the OT/J compiler preferences.
     */
    @SuppressWarnings({ "basecall", "inferredcallout" })
    callin org.eclipse.jdt.core.dom.Expression convertCurrencyExpression(CurrencyExpression expression){
        final DomCurrencyLiteral name = new DomCurrencyLiteral(this.ast, expression);
        if (this.resolveBindings) {
            recordNodes(name, expression);
        }
        return name;
    }
}

I deliberately used some special OT/J syntax worth explaining:

• Lines 5ff. define a callin bindings like we’ve seen before.
• Line 8 adds a guard predicate to the binding, saying that this binding should only fire when the argument expression is actually of type CurrencyExpression
• After passing the guard we know that we can safely cast to CurrencyExpression so I added a parameter mapping (line 10) which feeds a casted value into the role method.
• Inside the role method convertCurrencyExpression everything looks normal, but at a closer look this.ast and this.resolveBindings seem to be undefined in the scope of the current class. In fact these fields are defined in the base class ASTConverter and we could use explicit callout accessors like in the previous post. However, this time I chose to let the compiler infer these callouts so that the method would look exactly like existing methods in ASTConverter do (this option has to be enabled in the OT/J compiler preferences).

OK, with this little addition our CurrencyExpressions are converted to something that the JDT can handle and we’re already prepared for doing real AST manipulation including our syntax.

Source Formatting

Inside the JDT source formatting (Ctrl-Shift-F) is essentially performed by class CodeFormatterVisitor. This class is one of many subclasses of the general ASTVisitor. If one wanted to make these visitors aware of our CurrencyExpression we would have to add one visit method to ASTVisitor and each of its sub-classes! That’s certainly not viable, so with plain Java we’re pretty much out of luck.

The situation that needs adaptation can be described as follows:

• A visitor will be created and invoked in order to descend into the AST
• At the point when traversal finds a CurrencyExpression it will invoke its traverse(ASTVisitor) method.

Of course we could manually inspect the type of visitor within the traverse method, but that would defy the whole purpose of having visitors: keep all those add-on functions out from your data structures. Instead I only gave a default implementation to CurrencyExpression.traverse and used OT/J for the cleanest implementation of double dispatch (which is what the visitor pattern painstakingly emulates): we need dispatch that considers both the visitor type and the node type for finding the suitable method implementation.

In green-field development this would be still easier but even on top of an existing visitor infrastructure it get’s pretty concise.

Visitor adaptation - version 1

My first version looks like this (explanations follow below):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

public team class VisitorsAdaptor {
 
    protected team class AstFormatting playedBy CodeFormatterVisitor {
        // whenever visiting something that could contain an expression
        // activate this team to enable callins of the inner role
        callin void visiting() {
            within(this) {
                base.visiting();
            }
        }
        @SuppressWarnings("decapsulation")
        void visiting()
            <- replace
                boolean visit(Block block, BlockScope scope),
                boolean visit(FieldDeclaration fieldDeclaration, MethodScope scope),
                void formatStatements(BlockScope scope, final Statement[] statements, boolean insertNewLineAfterLastStatement);
 
        Scribe getScribe() -> get Scribe scribe;
 
        /** This role implements formating of our custom ast: */
        protected class CustomAst playedBy CurrencyExpression
        {
            void traverse() <- replace void traverse(ASTVisitor visitor, BlockScope scope);
 
            @SuppressWarnings({ "inferredcallout", "basecall" })
            callin void traverse() {
                Scribe scribe = getScribe();
                Scanner scanner = scribe.scanner;
 
                // format this AST node into a StringBuffer:
                StringBuffer replacement = new StringBuffer();
                replacement.append("<% ");
                this.value.printExpression(0, replacement);
                replacement.append(' ');
                replacement.append(this.currency);
                replacement.append(" %>");
 
                // feed the formatted string into the Scribe:
                int start = this.sourceStart();
                int end = this.sourceEnd();
                scribe.addReplaceEdit(start, end, replacement.toString());
 
                // advance the scanner:
                scanner.resetTo(end+1, scribe.scannerEndPosition - 1);
                scribe.pendingSpace = false;
            }
        }
    }
}

The key trick in this example is nesting:

• Role AstFormatting is responsible for detecting when a CodeFormatterVisitor is visiting any subtree that may contain expressions. This is done using a callin binding that lists three relevant base methods which all should be intercepted by the same role method (lines 12-16).
• Inside role AstFormatting (which is also marked as a team) an inner role CustomAst will only be triggered if a CodeFormatterVisitor calls the traverse method of a CurrencyExpression (see callin binding in line 23).
• The connection between both levels is wired in method AstFormatting.visiting: the block statement within() { } temporarily and locally activates the given team instance, here denoted by this. Only during this block the nested team AstFormatting is active - meaning that only during this block the callin binding in role CustomAst will fire.
• Within role CustomAst we can naturally access the CodeFormatterVisitor via the enclosing instance of AstFormatting. No instanceof and casting needed, because all this only happens in the context of a CodeFormatterVisitor

The body of method traverse contains only domain logic: pretty-printing the current node into a string buffer and interacting with the underlying infrastructure (Scanner, Scribe) that drives the formatting.

That’s it, with these classes in place, we can write this method:

int getMoney() {
   int myMoney = <% 
		  3
	  euro %>
	; 
		System
	.out.println("myMoney ="+myMoney);
					return myMoney;
}

then hit Ctrl-Shift-F et voilà:

private static int getMoney() {
    int myMoney = <% 3 euro %>;
    System.out.println("myMoney =" + myMoney);
    return myMoney;
}

How’s that?
The formatter smoothly operates on the full hybrid language, not just skipping over our nodes but handling them as well.

Generalizing visitor adaptations

After success wrt both challenges I’d like to clean up even more and prepare for further adaptations of other visitors. Given how many subclasses of ASTVisitor are used within the JDT we wouldn’t want to write the infrastructure for double dispatch over and over again. So let’s generalize, that is: extract a common super-class, by extracting everything re-usable out off class AstFormatting

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

public team class VisitorsAdaptor
{
    protected abstract team class AstVisiting playedBy ASTVisitor {
        // whenever visiting something that could contain an expression
        // activate this team to enable callins of the inner role
        callin void visiting() {
            within(this)
                base.visiting();
        }
        void visiting()
            <- replace
                boolean visit(Block block, BlockScope scope),
                boolean visit(FieldDeclaration fieldDeclaration, MethodScope scope);
 
        protected abstract class CustomAst playedBy CurrencyExpression {
            // variant of traversal that should be used when the enclosing team is active:
            // (implement in subclasses)
            abstract callin void traverse();
            void traverse() <- replace void traverse(ASTVisitor visitor, BlockScope scope);
        }
        // Insert more roles for binding more AST nodes...
    }
 
    protected team class AstFormatting extends AstVisiting playedBy CodeFormatterVisitor
    {
        // one more trigger that should activate the team:
        @SuppressWarnings("decapsulation")
        visiting <- replace formatStatements;
 
        Scribe getScribe() -> get Scribe scribe;
 
        /** This role implements formating of our custom ast: */
        @Override
        protected class CustomAst {
            @SuppressWarnings({ "inferredcallout", "basecall" })
            callin void traverse() {
                // method body as before
            }
        }
    }
    protected team class OtherVisitorAdaptor extends AstVisiting playedBy XYVisitor
    {
        @Override
        protected class CustomAst {
            callin void traverse() {
                // domain logic
            }
        }
        // Insert more roles for actually handling more AST nodes ...
    }
}

Now team class AstVisiting contains the part that is common for all visitors. At this level several things are still abstract: method traverse, role class CustomAst and even the whole team AstVisiting.

Team class AstFormatting extends the abstract team and defines everything specific to formatting. We have one more trigger for visiting, one callout binding to a field of class CodeFormatterVisitor and then we only refine the previously abstract role class CustomAst. At this level it is no longer abstract because we give an implementation for traverse.

I’ve also sketched another nested team showing a minimal specialization of AstVisiting for adapting some other visitor and adding another implementation for CustomAst.traverse plus potentially more roles for more node types.

Conclusion

For those who don’t work in the compiler business on a day-to-day basis this is probably pretty tough stuff, but let me summarize what we’ve just achieved:

• Embed a custom syntax into Java, showing how a custom parser can be plugged in to create custom AST from a region of the Java source.
• Adapt the conversion between two different AST structures (internal -> DOM) to also handle custom nodes.
• Adapt the code formatter so that hybrid sources can be formatted with a single command.
• Prepared the infrastructure for adapting other visitors, too. By this we have achieved that new visitor adaptations will only need to add their specific implementation with close to zero scaffolding.
• Cleanly separated each implemented concern in one module.
• Keep each module in the scale of only tens of lines of code.
• Yet implement significant steps towards a production quality IDE for our custom hybrid language.

Maybe I shouldn’t have told you, how easy these things can be - if your tools are sharp - maybe.
But professional carvers know: if your knife is sharp, it’s actually easy to handle. Only if it is blunt you are in real danger of hurting yourself - because you need to apply disproportionate force to cut your wood. So:

Spare your fingers, sharpen your knife!

PS: Here’s the archive of all sources, ready to be imported into the OTDT.

Have you ever thought of making Java a bit smarter? Perhaps, for some task you would prefer a custom syntax, and snippets using that syntax should then be embedded into Java? Sure, many never seriously think about this because of the prohibitively high effort to create the compiler for such hybrid language. And even if you are a compiler guru, knowing your toolkits so that translation wouldn’t be a problem for you, you’ll probably surrender at the mere thought of how to create a mature IDE that would allow efficiently productive work with you hybrid language.

You shouldn’t give up. Think: If you build your own IDE you’ll never be able to really compete with the JDT, right? Still anything falling back behind the quality of the JDT won’t raise your productivity but will stand in your way at the most common tasks during development, right?
What does this tell you? Give up? No. If you can’t beat us, join us. Don’t write a new IDE for any Java-based language. Join the JDT. Well, but the JDT doesn’t provide an extension point for embedding a different syntax, does it? Sure they don’t, but it’s actually not their job to do so because every embedded language will probably have slightly different requirements so designing such an extension point would be a battle you can never win.

I have developed a tiny extension to Java and integrated this into the JDT by a mere 204 lines of code including comments and a plugin.xml. As some may guess the only trick needed is to use Object Teams. By this post I will explain how Object Teams can be used for extending the JDT in this way. And I will also argue against the most common fear in this context: “Is that solution maintainable?” From my very own experience this design is not just barely manageable, but from all I’ve seen this is the best maintainable solution for this kind of task, but I’m getting ahead of myself.

In order not to distract from the interesting design issues I’ll be using the most simply language extension: I want to be able to write integer constants in natural language, and while I’m at it, I want it to work in an multilingual setting. So, this should, e.g., be a legal program:

public class EmbeddingTest {
    private static int foo() {
        return <% one %>;
    }
    public static void main(String[] args) {
        System.out.println(foo());
    }
}

I’m using <% and %> tokens to switch between Java syntax and custom syntax.

The first step can be achieved in plain Java, it’s creating a class for ASTNodes representing my custom int literals within the compiler. If you really want you may inspect class CustomIntLiteral, but it’s actually pretty boring old Java. Its main job is to lookup a given string from an array of known number words and thus translate the word into an int. It even detects the language used and remembers this for later use. The behaviour is hooked into the JDT compiler by overriding method TypeBinding resolveType(BlockScope scope) — just normal Java practice.

Drilling down into the example

Here’s an overview of the module that does all the rest:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

package embedding.jdt;
 
import org.eclipse.jdt.core.compiler.CharOperation;
import org.eclipse.jdt.core.compiler.InvalidInputException;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTNode;
import org.eclipse.jdt.internal.compiler.ast.Expression;
import org.eclipse.jdt.internal.compiler.ast.IntLiteral;
import org.eclipse.jdt.internal.compiler.parser.TerminalTokens;
 
import embedding.custom.ast.CustomIntLiteral;
 
import base org.eclipse.jdt.core.dom.ASTConverter;
import base org.eclipse.jdt.core.dom.NumberLiteral;
import base org.eclipse.jdt.internal.compiler.parser.Parser;
import base org.eclipse.jdt.internal.compiler.parser.Scanner;
 
public team class SyntaxAdaptor {
 
        /**
         * <h3>Part 1 of the adaptation:</h3>
         * Wait until '<' is seen and check if it actually is a special string enclosed in '<%' and '%>'.
         */
        protected class ScannerAdaptor playedBy Scanner { ... }
 
        /**
         * <h3>Part 2 of the adaptation:</h3>
         * If the ScannerAdaptor found a match intercept creation of the faked null expression
         * and replace it with a custom AST.
         *
         * This is a team with a nested role so that we can control activation separately.
         *
         * This team should be activated for the current thread only to ensure that
         * concurrent compilations don't interfere: By using thread activation any state of
         * this team is automatically local to that thread.
         */
        protected team class InnerCompilerAdaptor {
                /** This inner role does the real work of the InnerCompilerAdaptor. */
                protected class ParserAdaptor playedBy Parser { ... }
        }
 
        /**
         * Dom representation of CustomIntLiteral.
         * Since the constructor of NumberLiteral is package private we cannot subclass, so use a role instead.
         */
        protected class DomCustomIntLiteral playedBy NumberLiteral
                // don't adapt plain NumberLiterals, just those that already have a DomCustomIntLiteral role:
                base when (SyntaxAdaptor.this.hasRole(base, DomCustomIntLiteral.class))
        { ... }
 
        /**
         * <h3>Part 3 of the adaptation:</h3>
         * This adaptor role helps the ASTConverter to convert CustomIntLiterals, too.
         */
        @SuppressWarnings("decapsulation")
        protected class DomConverterAdaptor playedBy ASTConverter { ... }
}

Imports

Why am I showing you boring import declarations to begin with? Well, with OT/J there’s a fine distinction that is worth looking at: all imports starting with import base indicate that these classes are imported for attaching a role to them. So just from these lines you see that the given module adds roles to classes from org.eclipse.jdt.internal.compiler.parser and org.eclipse.jdt.core.dom (2 classes each). All other imports are plain Java imports and won’t let you apply any OT/J tricks.

Teams and Roles

Line 18 above tells you that the class SyntaxAdaptor is actually a team. Teams are used for grouping a set of roles - nested classes of a team. Using the playedBy keyword a role declares that it adapts the specified base class (which are the same classes we base-imported above). The purpose of these roles should be roughly clear by the doc comments.

So, role ScannerAdaptor will be responsible for switching between both syntaxes.

Role ParserAdaptor (line 40) will be responsible for creating our AST node (CustomIntLiteral). But wait, what’s that: the role is nested within an intermediate team, InnerCompilerAdaptor. This team will show you, how to define a role that is only effective in specific situations, here, the ParserAdaptor should only be effective after the ScannerAdaptor has detected a syntax switch. Details follow below.

The other two roles will do advanced stuff so I’ll discuss them later.

Role implementation (1)

Here is the full(!) code of role ScannerAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

protected class ScannerAdaptor playedBy Scanner {
 
    // access fields from Scanner ("callout bindings"):
    int getCurrentPosition()                     -> get int currentPosition;
    void setCurrentPosition(int currentPosition) -> set int currentPosition;
    char[] getSource()                           -> get char[] source;
 
    // intercept this method from Scanner ("callin binding"):
    int getNextToken() <- replace int getNextToken();
 
    callin int getNextToken() throws InvalidInputException {
        // invoke the original method:
        int token = base.getNextToken();
        if (token == TerminalTokens.TokenNameLESS) {
            char[] source = getSource();
            int pos = getCurrentPosition();
            if (source[pos++] == '%') {                                        // detecting the opening "<%" ?
                int start = pos;                                               // inner start, just behind "<%"
                try {
                    while (source[pos++] != '%' || source[pos++] != '>')       // detecting the closing "%>" ?
                        ;                                                      // empty body
                } catch (ArrayIndexOutOfBoundsException aioobe) {              // not found, proceed as normal
                        return token;
                }
                setCurrentPosition(pos);                                       // tell the scanner what we have consumed (pointing one past '>')
                int end = pos-2;                                               // position of "%>"
                char[] fragment = CharOperation.subarray(source, start, end);  // extract the custom string (excluding <% and %>)
                // prepare an inner adaptor to intercept the expected parser action
                new InnerCompilerAdaptor(fragment, start-2, end+1).activate(); // positions include <% and %>
                return TerminalTokens.TokenNamenull;                           // pretend we saw a valid expression token ('null')
            }
        }
        return token;
    }
}

Comments describing the logic are in the right column. Inline comments describe the usage of OT/J:

• Lines 3-6 define accessors for two fields from the base class Scanner.
• Line 9 defines that calls to method getNextToken() should be intercepted by our version of this method
• Line 11 marks the role method as callin which is a pre-requisite for line 13
• Line 13 invokes the original method from Scanner
• In line 29 we are in the situation that we have detected a region delimited by <% and %>. We have extracted the text fragment between delimiters, and we know the start and end positions within the source file. Only now we create an instance of InnerCompilerAdaptor and immediately activate it for the current thread (activate()).

At this point the ScannerAdaptor is done and now an InnerCompilerAdaptor is watching what comes next.

Here’s the nested team InnerCompilerAdaptor with its role ParserAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

protected team class InnerCompilerAdaptor {
 
    char[] source;
    int start, end;
    protected InnerCompilerAdaptor(char[] source, int start, int end) {
        this.source = source;
        this.start = start;
        this.end = end;
    }
 
    /** This inner role does the real work of the InnerCompilerAdaptor. */
    protected class ParserAdaptor playedBy Parser {
        // import methods from Parser ("callout bindings"):
        @SuppressWarnings("decapsulation")
        void pushOnExpressionStack(Expression expr) -> void pushOnExpressionStack(Expression expr);
 
        // intercept this method from Parser ("callin binding"):
        void consumeToken(int type) <- replace void consumeToken(int type);
 
        @SuppressWarnings("basecall")
        callin void consumeToken(int type) {
            if (type == TerminalTokens.TokenNamenull) {                 // 'null' token is the faked element pushed by the SyntaxAdaptor
                // this inner adaptor has done its job, no longer intercept
                InnerCompilerAdaptor.this.deactivate();
                // TODO analyse source to find what AST should be created
                Expression replacement = new CustomIntLiteral(source, start, end, start+2, end-2);
                this.pushOnExpressionStack(replacement);                // feed custom AST into the parser:
                return;
            }
            // shouldn't happen: only activated when scanner returns TokenNamenull
            base.consumeToken(type);
        }
    }
}

• Lines 3-4 define state of the nested team, which is used for passing the information collected by the ScannerAdaptor down the pipe
• Line 15 provides access to a protected method from Parser. By @SuppressWarnings("decapsulation") we document that this access inserts a tiny little hole into the encapsulation of Parser
• Line 18 defines a callin binding as we have seen it before.
• Line 24 already deactivates the enclosing InnerCompilerAdaptor, ensuring this is a one-shot adaptation, only.
• Line 26/27 perform the payload: feed a CustomIntLiteral node into the parser

Coming to life

Wow, if you’ve read so far, you’ve seen a lot of OT/J on just a few lines of code. Let’s wire things together, by throwing the code into an Object Teams Plug-in Project and declaring one extension:

I have defined one aspectBinding between the existing plugin org.eclipse.jdt.core and my team classes SyntaxAdaptor and InnerAdaptor (there’s a man behind the curtain pushing an ugly __OT__ prefix into the declaration, please ignore him - he’ll be gone in the next release of the tool).
Please note that for team SyntaxAdaptor I have set the activation to ALL_THREADS which means that at application launch an instance of this team will be created and activated globally. Without this flag the whole thing would actually have no effect at all.

That’s all the wiring needed, so kick up a runtime workbench, create a Java project and class, insert the code for class EmbeddingTest from the top of this post and boldly select Run As > Java Application. In the console we see a result:

1

Oops, the compiler for our little language extension already works? Did you see me writing a compiler?

Well, beginner’s luck, let’s assume. But, oops, watch this: When I mistype the return type of foo and ask the JDT for help, this is what I see:

The problem view tells me it knows that <% one %> has type int, which doesn’t match the declared return type boolean. Next I positioned the cursor on “one” (the element that’s definitely not Java) and hit Ctrl-1, and the standard JDT quickfix knows that I should change the return type of foo to int.
Did you watch me implementing a quickfix??

Summary so far

Here’re the stats:

• 204 lines of code including plugin.xml
• roles adapting two base classes from org.eclipse.jdt.core.
• callout bindings to two fields and one method
• callin bindings to two methods
• all adaptation is cleanly encapsulated in one team class. If you wish you could even deactivate this one team in a running workbench and thus disable all our adaptions with a single click.
• one plain Java class to implement the semantics of our extension

As for maintainability: The only dependencies are the items mentioned above: two classes, two fields and three methods. Only if one of these are modified under evolution, my adaptation has to be updated accordingly - and: if this happens I will definitely be told by the compiler because one of the bindings will break. If it doesn’t break there’s no need to worry.

With this implementation the compiler seamlessly works with our new syntax and even UI features that operate on the compiler AST can handle our extension, too.

What’s next?

I’m sure some think that the above is probably a forged example. You might challenge me to do something real, like refactoring. If you do so, you actually got me (mumble, mumble) - with the above implementation refactoring does not work with our custom syntax. Now that you’ve seen the start, what do you expect, how much additional rocket science does it take to add minimal refactoring support? (to be continued)