The Object Teams Blog

IDE Innovation

Every now and then some folks report that the IDE they’re developing now supports this or that cool new feature. Sometimes I envy them for such progress - but more often than not I end up realizing that the OTDT already has that feature or something very similar. Is that just my personal bias (which certainly I have) or are we cheating in some way, or what?

There’s a little detail in the design of the OTDT that turns out to make such a difference as normally can only achieved by cheating: by the way how the OTDT extends and adapts the JDT it is like saying we’re starting the race not at the line saying “START” but at the other one saying “FINISH” and run on from there. While many projects define “JDT-like user experience” as their long-term goal, the OTDT basically has this since the first release. How come? The OTDT basically is the JDT, with adaptations.

There’s a fine point in the word basically. To tell the truth, every feature that the JDT supports for Java development is not automatically fully available in the OTDT for OT/J development. In fact most JDT features need adaptation to provide equal convenience for OT/J development. It’s just that all these adaptations can be brought into the system very very easily - thanks to the self-application of OT/Equinox. And now, here is actually an example of a JDT feature that lacked OT/J support - until yesterday:

Change Method Signature Refactoring

If you refactor mercilessly the “Change Method Signature” refactoring is certainly one of your friends. Add/rename/remove/reshuffle parameters of a method without (too easily) breaking existing code, cool. It knows about the connections from method invocation to method declaration and about overriding. That’s good enough for Java, but not good enough for OT/J since OT/J introduces method bindings (”callout” and “callin”) that create a wiring between methods of different objects. Obviously, if one of the methods being wired changes its signature so must the method binding.

Technically, the JDT implementation of that refactoring bailed out when it asked a parameter/argument for its parent in the AST and found neither a method declaration nor a method invocation. The JDT refactoring does not know about OT/J method bindings, so it just failed to update those.

After a little of coding this is what happens now when you apply “Change Method Signature” on a piece of OT/J code. Assume you have a plain Java class:

public class BaseClass {
	public void bm(int i, boolean b) {
 
	}
	void other() {
		bm(3, false);
	}
}

and a TeamClass whose contained RoleClass is bound to BaseClass:

1
2
3
4
5
6
7
8
9
10

public team class TeamClass {
	protected class RoleClass playedBy BaseClass {
 
		void rm(int i2, boolean b2) <- after void bm(int i, boolean b);
 
		private void rm(int i2, boolean b2) {
			System.out.println((b2?i2:-i2));
		}
	}
}

Here the right-hand side of the callin binding in line 4 refers to the normal method bm(int,boolean) defined in BaseClass.

What happens if you start messing around with the signature of bm?
Like:

I.e., we are adding a parameter str and also change the order of parameters (from i,b to b,str,i). I don’t have to tell you what this refactoring does to BaseClass, but here’s the preview of those changes affecting TeamClass:


(sorry the screenshot is a bit wide, you may have to click to really see).

The preview shows that the refactoring will do three things:

1. Update the right-hand side of the method binding.
2. Add a parameter mapping (the part starting with “with“) to ensure that the role method rm receives the arguments it needs, the way it needs them.
3. Do not update any part of the role implementation, because that’s what the parameter mapping is for: shield the role implementation from any outside changes.

Some words on these parameter mappings: each element like “b2 <- b” feeds a value from the right-hand side (representing things at the baseclass side) into the parameter at the left-hand side (representing the role). The list of parameter mappings is not ordered, which means further swapping of base side parameters requires no further action. And indeed the current implementation of the refactoring does not attempt to adjust an existing parameter mapping (which might be a quite complex task). If adjustments are required which the refactoring cannot perform automatically, it will inform the user that perhaps a parameter mapping may need manual adjustment.

The refactoring applies no AI to guess what the intended solution should look like, but it performs a number of obvious adaptations and gives note when these adaptations may not suffice and manual cleanup may be needed.

Implementation

Those who have read previous posts may (almost) know the kind of statistics that follows:

• 299 LOC implementation
• 315 LOC testcode
• 170 LOC testdata

I should really show one of these OT/J based implementation, one of these days. For this post I will just give you an Outline, literally:

Role class Processor is bound to the JDT’s ChangeSignatureProcessor, and the nested roles OccurrenceUpdate and MethodSpecUpdate are bound to two inner classes of ChangeSignatureProcessor, which shows how even class nesting at the base level can be mapped to the team & role level. Instances of the innermost roles will only ever come into being, if a Processor role has detected that it needs to work in order to handle OT/J specific code. Inside each role - apart from regular fields and methods - you see those green arrow-things, denoting method bindings between a role and its base. The highlighted binding to createOccurrenceUpdate is actually the initial entry into the logic of this module. Further down you see how the base behavior reshuffleElements is intercepted to additionally add parameter mappings if needed (and yes, I’m hiding the details of role MethodSpecUpdate, but there are no secrets inside, I just more methods and more method bindings).

Voilà, we indeed have a new refactoring for OT/J. I’ve been planning this one for a while but in the end it took me little more than a day

During the last week or so I modernized a part of the Object Teams Development Tooling (OTDT) that had been developed some 5 years ago: the type hierarchy for OT/J. I’ll mention the basic requirements for this engine in a minute. While most of the OTDT succeeds in reusing functionality from the JDT, the type hierarchy was implemented as a full replacement of the original. This is a pretty involved little machine, which took weeks and months to get right. It provides its logic to components like Refactoring and the Type Hierarchy View.

On the one hand this engine worked well for most uses, but over so many years we did not succeed to solve two remaining issues:

Give a faithful implementation for getSuperclass()

This is tricky because a role class in OT/J can have more than one superclass. Failing to implement this method we could not support the “traditional” mode of the hierarchy view that shows both the tree of subclasses of a focus type plus the path of superclasses up to Object (this upwards path relies on getSuperclass).

Support region based hierarchies

Here the type hierarchy is not only computed for supertypes and subtypes of one given focus type, but full inheritance structure is computed for a set of types (a “region”). This strategy is used by many JDT Refactorings, and thus we could not precisely adapt some of these for OT/J.

In analyzing this situation I had to weigh these issues:

• In its current state the implementation strategy was a show stopper for one mode of the type hierarchy view and for precise analysis in several refactorings.
• Adding a region based variant of our hierarchy implementation would mean to re-invent lots of stuff, both from the JDT and from our own development.
• All this seemed to suggest to discard our own implementation and start over from scratch.

Object Teams to the rescue: Let’s re-build Rome in ten days.

As mentioned in my previous post, the strength of Object Teams lies in building layers: each module sits in one layer, and integration between layers is given by declarative bindings:

Applying this to the issue at hand we now actually have three layers with quite different structures:

Java Model

The bottom layer is the Java model that implements the containment tree of Jave elements: A project contains source folders, containing packages, containing compilation units, containing types containing members. In this model each Java type is represented by an instance of IType

Java Type Hierarchy

This engine from the JDT maintains the graph of inheritance information as a second way for navigating between ITypes. Interestingly, this module pretty closely simulates what Object Teams does natively, I may come back to that in a later post.

Object Teams Type Hierarchy

As an extension of Java, OT/J naturally supports the normal inheritance using extends, but there is a second way how an inheritance link can be established: based on inheritance of the enclosing team:

team class EcoSystem {
   protected class Project { }
   protected class IDEProject extends Project { }
}
team class Eclipse extends EcoSystem {
   @Override
   protected class Project { }
   @Override
   protected class IDEProject extends Project { }
}

Here, Eclipse.Project is an implicit subclass of EcoSystem.Project simply because Eclipse is a subclass of EcoSystem and both classes have the same simple name Project. I will not go into motivation and consequences of this language design (that’ll be a separate post — which I actually promised many weeks ago).

Looking at the technical challenge we see that the implicit inheritance in OT/J adds a third layer, in which classes are connected in yet another graph.

Three Layers — Three Graphs

Looking at the IType representation of Eclipse.IDEProject we can ask three questions:

Each question is implemented in a different layer of the system. Things get a little complicated when asking a type for all its super types, which requires to collect the answers from both the JDT hierarchy layer and the OT hierarchy. Yet, the most tricky part was giving an implementation for getSuperclass().

An "Impossible" Requirement

There is a hidden assumption behind method getSuperclass() which is pervasive in large parts of the implementation, especially most refactorings:

When searching all methods that a type inherits from other types, looping over getSuperclass() until you reach Object will bring you to all the classes you need to consider, like so:

IType currentType = /* some init */;
while (currentType != null) {
   findMethods(currentType, /*some more arguments*/);
   currentType = hierarchy.getSuperclass(currentType);
}

There are lots and lots of places implemented using this pattern. But, how do you do that if a class has multiple superclasses?? I cannot change all the existing code to use recursive functions rather than this single loop!

Looking at Eclipse.IDEProject we have two direct superclasses: Eclipse.Project (normal inheritance, “extends”) and EcoSystem.IDEProject (OT/J implicit inheritance), which cannot both be answered by a single call to getSuperclass(). The programming language theory behind OT/J, however, has a simple answer: linearization. Thus, the superclasses of Eclipse.IDEProject are:

• Eclipse.IDEProject → EcoSystem.IDEProject → Eclipse.Project → EcoSystem.Project

… in this order. And this is how this shall be rendered in the hierarchy view:

The final callenge: what should this query answer:

        getSuperclass(ecoSystemIDEProject);

According to the above linearization we should answer: Eclipse.Project, but only if we are in the context of the superclass chain of Eclipse.IDEProject. Talking directly to EcoSystem.IDEProject we should get EcoSystem.Project! In other words: the function needs to be smarter than what it can derive from its arguments.

Layer Instances for each Situation

Let’s go back to the layer thing:

At the bottom you see the Java model (as rendered by the package explorer). In the top layer you see the OT/J type hierarchy (lets forget about the middle layer for now). Two essential concepts can be illustrated by this picture:

• Each layer is populated with objects and while each layer owns its objects, those objects connected with a red line between layers are almost the same, they represent the same concept.
• The top layer can be instantiated multiple times: for each focus type you create a new OT/J hierarchy instance, populated with a fresh set of objects.

It is the second bullet that resolves the “impossible” requirement: the objects within each layer instance are wired differently, implementing different traversals. Depending on the focus type, each layer may answer the getSuperclass(type) question differently, even for the same argument.

The first bullet answers how these layers are integrated into a system: Conceptually we are speaking about the same Java model elements (IType), but we superimpose different graph structure depending on our current context.

Inside the hierarchy layer, we actually do not handle IType instances directly, but we have roles that represent one given IType each. Those roles contain all the inheritance links needed for answering the various questions about inheritance relations (direct/indirect, explicit/implicit, super/sub).

A cool thing about Object Teams is, that having different sets of objects in different layers ( teams) doesn’t make the program more complex, because I can pass an object from one layer into methods of another layer and the language will quite automagically translate into the object that sits at the other end of that red line in the picture above. Although each layer has its own view, they “know” that they are basically talking about the same stuff (sounds like real life, doesn’t it?).

Summing up

OK, I haven’t shown any code of the new hierarchy implementation (yet), but here’s a sketch of before-vs.-after:

Code Size

The new implementation of the hierarchy engine has about half the size of the previous implementation (because it need not repeat anything that’s already implemented in the Java hierarchy).

Integration

The previous implementation had to be individually integrated into each client module that normally uses Java hierarchies and then should use an OT hierarchy instead. After the re-implementation, the OT hierarchy is transparently integrated such that no clients need to be adapted (accounting for even more code that could be discarded).

Linearization

Using the new implementation, getSuperclass() answers the correct, context sensitive linearization, as shown in the screenshot above, which the old implementation failed to solve.

Region based hierarchies

The old implementation was incompatible with building a hierarchy for a region. For the new implementation it doesn’t matter whether it’s built for a single focus type or for a region, so, many clients now work better without any additional efforts.

The previous implementation only scratched at the surface – literally worked around the actual issue (which is: the Java type hierarchy is not aware of OT/J implicit inheritance). The new solution solves the issue right at its core: the new team OTTypeHierarchies assists the original type hierarchy (such that its answers indeed respect OT/J’s implicit inheritance). By performing this adaptation at the issue’s core, the solution automatically radiates to all clients. So I expect that investing a few days in re-writing the implementation will pay off in no time. Especially, improving the (already strong) refactoring support for OT/J is now much, much easier.

Need I say, how much fun this re-write was?

On a day like this (2010/07/07), it should be evident what is better than one or a few stars: a strong team!

But when you look at what happens in your software at runtime, all you see is a soup of individuals (called objects) running around all over the place (remember: standard module systems do not create boundaries around runtime objects!).

In all humbleness, are you surprised to learn that it was a German invention back in 2002, that objects should team up?

This is the first release after the move of the project from objectteams.org to Eclipse.org, at what point it is time to express a big thank you to all who helped along the way, Mentors, EMO, Legal, the Tools-PMC, and - of course - the many Contributors. This is: a team-effort!

Now you might say: that’s a pretty scattered team: some people at the Eclipse Foundation in Ottawa, some students in Berlin, people from Austin, and where-not. But that’s actually the point about a team: you start from a set of individuals who initially do not necessarily have any particular relationship. Then you create a team where each individual takes one particular role. This means you further specialize these existing individuals (you don’t want a team of 11 goal keepers, would you? Even with a Manual Neuer giving a perfect forward pass, it takes a Miroslav Klose to make the goal). And then you unite all members of the team towards a common goal, giving the team a new identity, so that team acts like one.

Still, each team member brings into the team the strengths of his particular background, meaning: the individuals do not completely disappear, but some properties of the individuals shine through when they play their roles in the team.

Now, what’s that got to do with software?

Given you already have a core of an application implemented, and now it’s your task to implemented one or two more user stories on top of the existing code. You look at the existing classes and mark those that in some way or other are related to what you need to implement. One user story relates to all classes marked red, another one to those marked greenish etc (and do expect overlap):

How do you implement the user story that involves all the red classes, such that the new implementation sits in a nice new module that concentrates on only this one task/user story?

Consider the red entities as plain individuals, they don’t know about the new task they should contribute to. Also keep in mind, that not all instances of those red classes will participate in the new user story. What we need to do is: specialize a few of those individuals so they can play particular roles wrt the new task and unite those roles within a new team.

If you live in flat-land, this is tremendously difficult, but if you’re able to just add one more dimension to the picture …

… the solution is very straight forward:

Now:

• The implementation of the red user story is a strong, cohesive team
• Its members are roles specialized in their particular sub-tasks.
• Each role relates (↓) to one individual from the application core and specializes what is already given towards what the team requires.
• How exactly each role relates to its base is declared using two kinds of atomic bindings: callout and callin method bindings (see, e.g., this post).
• These roles only affect the system as long as the team context is active, and activation happens per team instance (with options for fine-tuning).
• Other teams may be formed for other purposes / user stories (see the greenish team)
• Even if you start already with this 3-D picture, with Object Teams you can always add one more dimension to your architecture, if needed.

Last Friday I received some wonderful meta-feedback. What’s that you’ll say?
It’s feedback on feedback, or, second order feedback.

First Order Feedback

Initially, I’m thinking of feedback whereby a tool tells its user what s/he’s done wrong and where to go in order to improve. As I mentioned earlier, I’m not interested in a tool that just works when it works, as that might require its users to get everything 100% correct right from the beginning. In our business I’m interested in tools that help the user to get it to work, from initial buggy attempts towards a full working solution.

So, when working on the Object Teams Development Tooling, how can we make the tool speak to the user in really helpful ways?

First we really care to give precise error messages and warnings regarding all kinds of situations that look funny, strange or plain bogus. Last time I counted the OT/J compiler featured 314 messages specific to OT/J. This is excellent for a seasoned OT/J developer but someone still trying to learn the language might be a little bit puzzled by messages like:

The clue on how to help the puzzled users lies in the suffix “OTJLD 4.3(e)”: that’s exactly the paragraph in the OT/J Language Definition, that defines what’s going on here. But what’s a reference like “§4.3(e)” good for? So the next thing we added to the OTDT was a context menu action on any OT/J related problem in the Problems view:

What do you see:

• At the top you see an editor with an underlined, buggy piece of code
• Next you see the Problems view with the corresponding error message
• Next you see a context menu on the problem with an entry “Go to Language Definition“.
• At the bottom finally you see a small extract from the language definition, exactly that paragraph that is violated by the buggy code. If that doesn’t provide sufficient context there are plenty of hyperlinks and breadcrumbs that help to find the required explanation

This is our specific feedback system and I think its already quite nice, however …

Second Order Feedback

Last Friday I presented Object Teams at the Vienna Helios Demo Camp. When I showed the “Go to Language Definition” action Peter Kofler gave some excellent feedback on our feedback system. He must have feeled the too-much-stuff syndrome you can easily see when looking at the screenshot above. So he requested that the same action be available even without the Problems view. So once back home I file bug 318071. In the most recent build you now have two more options:

Use the context menu of the left gutter:

Use the toolbar of the problem hover

Need I say that adding a button to the problem hover is not normally possible? With the action already in place the following OT/J code is all we need to integrate into the JDT/UI’s implementation of that hover:

/**
 * Add OT-Support to hovers for java problems.
 *  
 * @author stephan
 * @since 0.7.0 (Incubation at Eclipse.org)
 */
@SuppressWarnings({ "restriction", "decapsulation" })
public team class HoverAdaptor {
 
	/** Add the "Go to Language Definition" action to the hover's toolbar. */
	protected class ProblemHoverAdaptor playedBy ProblemInfo {
 
		void addAction(ToolBarManager manager, Annotation annotation) <- after void fillToolBar(ToolBarManager manager, IInformationControl infoControl)
			base when (isOTJProblem(base.annotation))
			with {  manager    <- manager,
				annotation <- base.annotation }
 
		void addAction(ToolBarManager manager, Annotation annotation) 
		{
			manager.add(ShowOTJLDAction.createAction(null/*site*/, annotation.getText()));
		}
 
		static boolean isOTJProblem(Annotation annotation) {
			if (annotation instanceof IJavaAnnotation) {
				int problemId = ((IJavaAnnotation) annotation).getId();
				return problemId > IProblem.OTJ_RELATED && problemId < IProblem.TypeRelated;
			}
			return false;
		}
	}
}

Thanks Peter, I think your RFE made a clear point for usability of the OTDT!

Remember the last time you had to cram your whole hosehold into boxes, bags and cases? You may feel excited about your new home etc. but the whole boxing business is quite a drag, ain’t it? There’s of course at least two ways of approaching this:

1. Don’t look, just shovel everything randomly into boxes
2. Look at each single piece, indulge in memories associates with it and sort it to its likes

Obviously, (1) means that on the other side you will unpack a whole mess of junk. OTOH, (2) won’t be finished before the moving truck arives. Still deep inside there lives a bit of hope, that you’ll move to your new home with only that stuff you’ll actually want, and everything ready to be neatly deployed into its new destination. Moving could free you from all the junk you don’t want any more, right? And even more, after the move, you may want to know, where everything is, right?

When moving the Object Teams project to Eclipse I was in the lucky situation that I could indeed use the occasion to sort through some of our stuff. The software engineer might be tempted to even speak of some “quality assurance” along the way, but let’s be careful with our wording for now.

On January 26 this year, the Eclipse Object Teams Project was created and we started to put up signs “Object Teams is moving to Eclipse.org”. Recently, I changed that sign to “Object Teams has moved to Eclipse.org”. So, what exactly happened between then and now?

Learning the infrastructure

At first the newly appointed eclipse committer and project lead is overwhelmed with all the shiny technology: web server, wiki, version repositories, build servers, download servers, the portal, project metadata, accounts for this and that, bugzilla components, versions and milestones and what-not. “Alles so schön bunt hier!”

I won’t indulge in talking about the paper work needed at this stage, after some four days I had most my accounts and the project was registered as “incubation - conforming”, so we were ready to go into the parallel IP process.

Initial contribution

On project day #12 I submitted my first CQ and, yes, that submission was already the result of heavy refactoring: I had renamed most (not all!!) occurrences of org.objectteams to org.eclipse.objectteams. A piece of cake? Well not exactly if the code piles up to a zip of 35 MBytes (not including .class files and nested archives), and not if your team-support plug-in goes berserk on some of the renamings and if … (see also this post).

Parallel IP process - our version

In fact our version of the parallel IP process looked like this:

On one thread I was chasing after some people and institutions to just provide the necessary signed statements of code provenance. Wrt the individuals this was painless, however, the university and the research institute involved both had their very specific strategies for delaying the project. All-in-all it took them more than one week per sentence in the final document. Or would you prefer the words-per-day count?

The much feared IP analysis turned out to be a very constructive collaboration with Sharon Corbett. I was really amazed about the obscure pieces of code (and comments) she brought to light, things that I never knew where in there. So that was helpful information, actually . Most of all I was pleased by her quick responsiveness - quite in contrast to the other thread. Thanks Sharon! I should also thank Jan Wloka who from the outset of the project took care that we’d have copyright headers and that stuff.

The effect was: at the time I got the signatures that cleared us to check our sources into svn the IP analysis was already done and complete! And not only that, during that process we’ve done some significant cleanup.

Code cleanup triggered by the move:

• Proviously, Object Teams used the JMangler framework for launching with our bytecode transformers in place. This was a great thing to have back in the olden Java 1.3 times. But in 2010 our Java 5 based alternative had matured and we didn’t even have to put JMangler into any of our moving boxes
• We used to maintain a patched version of BCEL 4.4.1 (developed at Freie Universität Berlin, as the namespace de.fub still announced). I consulted the AspectJ folks who maintain a patched version, too. But their patch only vaguely resembles the original, and they clearly stated that they saw now chances of these changes being adopted upstream. So, I went back to our sources, checked the patches, checked the current version 5.2 of bcel and found that the remaining bugs could actually be easily worked around from the outside. That’s when I learned the details of Orbit: since our initial commits to Eclipse we never had to bother about the bcel version, it just comes right flying from the Orbit. So we got that legacy version cleared up.
• My heart skipped a tiny single beat, when I learned that one of our most central data structures could not be accepted at Eclipse: I had patched class WeakHashMap from the OpenJDK to create a DoublyWeakHashMap with quite unique characteristics concerning garbage collection. We need that! Yet, the license (GPL with “classpath exception”) was not accepted. I made a quick experiment with wrapping instead of patching and guess what I learned (again): while the patched version was created in the pursuit of performance, still, after changing the strategy (to what was destined to be slower) my measurements could not show any performance penalty. So, carve that in stone: never optimize without measuring. The new version has the same performance - and no license issues!

Of course, there were more issues like needing to file a new CQ just for using files like xhtml1-strict.dtd, but those caused no grief after all. Enter the next phase.

Getting everything to build and test on eclipse servers

OK, when we opened the boxes at our new home, some of the content was actually broken on the way.

Fixing broken builds

The ugliest part was getting an ancient set of PDE/Build scripts to run on build.eclipse.org. Digging through a 30MByte build-log looking for the cause of a build failure never was fun. The point that dissatisfies me with all the build technology I’ve seen so far: You have a build that works on one machine and with one version of the software. Then, one arbitrary piece of the setup changes, let’s take a big change as the example: moving your JDK from 5 to 6. It’s OK that things may break at this point, but the kind of breakage frequently seems to have nothing to do with what you’ve changed, e.g., after moving to a different JDK the compiler can no longer resolve java.long.Object, whereas everybody knows: that’s not the difference between the two JDKs. The problem is not broken builds, the problem is how little clues the logs give you for finding the root cause of the breakage, or even: telling you how to fix it. A technology that works when it works is one thing. A technology that helps you get it to work is another (and we’re working hard to make Object Teams fall into the second category).

Modernizing the build

OK, enough complained, the move to Eclipse again gave reason to cleanup that monster build and even update to using some of the automatic built-in p2 stuff (yes, finally we use p2.gathering=true), rather than manually invoking the various p2 applications (publisher, director). When it runs, you may even get the impression you know why it does.

The final round in improving our build was adding bundle signing, yeah! Of course, that’s when all the p2-metadata generated during the build don’t help you any more because those include the checksums of your unsigned jars. So I created a tiny little shell script to automate those steps required after a successful build&test. I ended up with 7 more transformations of our metadata needed at this stage. So we’re back at directly invoking p2 publisher, p2 director, do various XSL transformations etc. Most of these could actually be done by a PDE/Build - p2 integration, but let’s not expect too much, not now.

Did I mention the almost 50000 tests that successfully run during each build? Well, that’s what we owe our users, right?

It builds - let’s ship it!

OK, let’s move ahead to the success-story-part. Less than a month after the initial commit we had our first milestone. It’s so good to be back in business After fixing all those migration-induced regressions I’m sure our code has better quality than before.

User side migration

Only one burden we had to pass on to users: due to the changed namespace, the configuration files of existing OT/J projects have to be updated. Luckily, it wasn’t too difficult to add some specific build-path problems and quickfixes, which should make the migration for users pretty smooth.

Installing

Right while I was publishing our first milestones a new cool tool came around the corner: the Market Place Client. So, now if you download, e.g., the Helios Package “Eclipse IDE for Java Developers” you’ll get the OTDT installed without having to know the download address, just select Help > Eclipse Marketplace and search for “Object Teams”, and you’re ready to hit “Install“:

Interestingly, in order for this to work I had to ask for this feature, which later down the road triggered this blocker security issue. At the end of the day this made me ponder about generalizing various things that the user might want to know when installing software. And indeed, Object Teams should play an active role in this discussion: the whole business of OT/Equinox is based on the assumption that the user agrees with what we are doing. We already do our best in treating the users as grown-ups who can make their own decisions, if we provide sufficient information, like:

This little screenshot tells you a whole story about this version of the bundle org.eclipse.jdt.core (see last column):

• The icon in the 1st column says: this bundle is signed, but that signed content is going to be woven at load time before the JVM sees it
• Columns 2 & 3 give the obvious information that this is not the version provided by the JDT team but something from the Object Teams project, which BTW. is still in its incubation phase.
• Column 4 finally gives you all the gory details: a sophisticated version number plus the list of OT/Equinox plugins that have declared to adapt the current plugin.

That’s the kind of transparency we show upon request after the software is installed. The mentioned bug 316702 is about providing similar transparency already during install.

So, what’s the plan?

Given that all legal and technical matters have been sorted out to this point, and given that the tool is in an even better shape than the final OTDT 1.4.0 from objectteams.org, what’s our plan?

• Just recently I requested a Release Review, tentatively scheduled for July 7, so with only little delay after Helios we should have an actual, stable release.
• I decided to defer the project graduation some more months to give us time to define which parts of the software are actually API.

Where can I see it in action?

Well, given the current milestone releases (and the ease of installing) and all the documentation we have in the wiki nothing should stop you from running your first Object Teams demo in do-it-yourself mode

Otherwise, if you happen to be in Vienna on June 25, just come to this DemoCamp and I’ll help you to get started with Object Teams.

So, indeed for the Object Teams project that past 6 months were used to turn this:

into this:

Last week I had the great opportunity to speak at GeeCON 2010.

Wow, that was a big screen in Multikino 51 to use for presentations!

I’ve uploaded my slides, so if you missed the presentation or want to recap some points, just grab the slides. Please post your feedback or questions to the Object Teams forum.

I very much enjoyed all those discussions at the conference and I’m very sorry I had to leave early. I was specifically happy to hear that quite a few people in the audience had actually experienced the exact problems that Object Teams addresses (erm, no, I’m not happy to see people suffering but about the chance to help ‘em with our approach ).

Another good thing about the conference was that I found some minutes to chat with one of the mentors of the Eclipse Object Teams Project, Chris Aniszczyk. He gave me some new ideas for our project plan, which I will update shortly.

Thanks for the warm welcome by the organizers and the audience!

Today I received my hard copy of the German Eclipse Magazin issue 3.10, which features a 4-pages article on Object Teams, so if you want to let a colleague know about Object Teams (and if that colleague understands German) give him/her a copy of that Magazin

The article gives an overview of …

• what deficiencies in OOP are addressed by Object Teams
• fundamentals of OT/J: teams, roles, playedBy, callout and callin
• the Object Teams Development Tooling
• OT/Equinox
• exemplary applications of OT/J

… and summarizes the gains in flexibility and maintainability.

There are a few errata which I’d like to correct here:

• The pictures of figures 1 & 2 are swapped
• Some text in table 1 is unrelated to Object Teams, in the manuscript it reads:
  

  Ende 2001	Beginn der Arbeiten an der Technischen Universität Berlin
  2003-2006	Kooperation TU Berlin & Fraunhofer FIRST, gefördert mit Mitteln des BMBF
  2005	erste öffentliche Präsentation des OTDT
  2006	erste Plug-ins mit OT/Equinox geschrieben
  März 2007	Version 1.0.0 des OTDT veröffentlicht
  2007-2010	Kontinuierliche Verbesserungen, Versionen 1.1.0 – 1.4.0
  Januar 2010	Eclipse Object Teams Projekt kreiert, Beginn des Umzuges

• The CD claims to contain the Object Teams Development Tooling, however, you’ll only find the jar of the command line compiler. But no problem, simply visit the download page and you’ll find all you need to install the OTDT into Eclipse 3.6 M6 (or earlier versions)

Edit: The magazine has provided an online version with these errata fixed.

Enjoy the read!

The basic elements of programming are methods. If you have many methods you want to group them in classes. If you have many classes you want to group them in packages. If you have many packages you want to group them in bundles. If you have many bundles you group them in features, but if you have many features …stop!, STOP!!!

Isn’t this insane? Every time we scale up to the next level we need a new programming concept? Like, someone invented the +1 operator and I can trump him by inventing a +2 operator, and you trump me by …? Haven’t we learned the 101 of programming: abstraction? I guess not many folks in Java-land are aware of a language called gbeta, where classes and methods are unified to “patterns” and no other kinds of modules are needed than patterns. It’s good news that the guy behind gbeta receives one of this year’s prestigious Dahl-Nygaard prizes: Erik Ernst. Object Teams owes much to Erik and I will speak more about his contributions in a sequel post.

Another really smart guy is Gilad Bracha, who after working on Java generics (together with Erik actually) and even JSR 294 decided to do something better, actually doubleplusgood. While he upholds the distinction between methods and classes he makes strong claims that no other modules than classes are needed, if, and here is the rub: if classes can be nested (see “Modules as Objects in Newspeak“).

Modules in Java: man or boy?

Let me briefly check this hypothesis:

The proliferation of module concepts in Java is due to the lack of truly nestable modules.

Which of the above mentioned concepts supports nesting? Features support inclusion which isn’t exactly nesting, but since features are actually defied by OSGi purists, I don’t want to burn my fingers by promoting features. Bundles cannot be nested, bummer! Some people actually think packages support nesting, with two reasons for believing so: packages are mapped to directories in a files system or in a jar and directories can be nested, plus: packages have compound names. However, semantically the dot in a package name is just a regular part of the name, it has no special semantics. Speaking of package foo.bar being a “sub-package” of foo is strongly misleading as the relation between them two is in no way different from the relation between any other pair of packages. All packages are equalst. Perhaps you recall the superpackage hero (or the strawman of that hero), which introduced the capability that a public class can actually be seen by classes from specific superpackages only. And: superpackages where designed to support nesting. Now superpackages are “superceded” by Jigsaw modules, which don’t support nesting. Great, they invented the +3 operator. That’s award winning!

Finally classes: surprisingly, yes, classes can be nested. Unfortunately, still today it shines through that nested classes are an after-thought, the real core of Java 1.0 was designed without that. E.g., serializing nested classes is strongly discouraged. How many O/R mappers support non-static inner classes? The last time I looked: zero!

Nested classes: flaws and remedies (part 1)

I see three conceptual flaws in the design of nested classes in Java:

1. scoping (2 problems actually)
2. forced representation
3. poor integration with inheritance

I discuss the first two in this post, the third, inheritance, deserves a post on its own right.
For each problem I will briefly show how it is resolved in OT/J. First off, I should mention that OT/J maintains compatibility with Java by applying any modified semantics only inside classes with the team modifier.

Scoping

Consider the following Java snippet:

public class City {
   TownHall theTownHall = new TownHall();
   class TownHall {
       class Candidate { }
       Candidate[] candidates;
       void voteForMayor(Candidate candidate) { ... }
   }
   class Citizen {
       void performCivilDuties() {
           ...
           theTownHall.voteForMayor(theTownHall.candidates[n]);
       }
   }
}

In this code we can actually make all methods, fields and nested classes private, to the end that external clients see none of these internal details of a City, whereas classes at the inside can blithely communicate with each other. Thus we have created a wonderful module City which can use all accomplishments of object-oriented programming for its internal structure - well hidden from the outside. In Java, however, this is flawed in two ways:

1. Nested classes cannot protect any members from access by sibling classes, so I (a Citizen) can actually see the wallets of all mayor Candidates (well, maybe that’s not a bug but a feature). It would be much more useful if only inside-out visibility was given, i.e., a Citizen can see all members of his/her City, but not inverse - the City looking into its glass Citizens.
2. Scoping rules in Java are purely static, i.e., permissions are given to classes not objects. As a result I could not only vote in my home city, but in any City (and every person can see the wallet of any other person etc.).

OT/J solves (1) by reinterpreting the access modifiers. Within a team class a private field Candidate.wallet, e.g., would not be visible outside its class, whereas a Citizen could still access theTownHall if this field were private.

(2) is basically solved by applying different rules to self-references (incl. outer self) than to explicitly qualified expressions, so City.this.theTownHall (OK, uses the enclosing this instance) applies different rules than newYork.theTownHall (not OK if theTownHall is private).

Well, issue (1) is a matter of tedious details, where I see no excuse for Java’s “solution”. Issue (2) has always been a differentiation between good old Eiffel, e.g., and its “successor” Java. This issue stands for a conceptual crossroad: are we interested in code nesting, or are we more interested in expressing how run-time objects are grouped? I personally don’t see much use in making the definition of a Citizen a nested part of the definition of a City, but speaking of many Citizens (instances) forming a City (instance) makes a lot of sense.

Forced Representation

By this I mean the fact that programmers are forced to store all inner classes textually embedded in their enclosing class. After we’ve already seen the discrepancy between the semantics and the representation of a package (flat structure stored in nested directories), now we see the opposite: semantic nesting is forcefully tied to representational nesting. This sounds logical until you start to write significant amounts of code as one big outer class with lots of (deeply) nested classes contained. You probably wouldn’t even think of this, because pretty soon the file becomes unmanageably huge. This is a very mundane reason why class nesting in Java simply doesn’t scale.

The solution in OT/J is pretty trivial. The following structure

// file community/City.java
package community;
public team class City {
    protected class Citizen {}
}

is strictly equivalent to these two files:

// file community/City.java
package community;
/**
 * @role Citizen
 */
public team class City {
}

// file community/City/Citizen.java
team package community.City;
protected class Citizen {}

The trick is: City is both a class and a special package. So semantically the team contains the nested Citizen but physical storage may happen in a separate file stored in a directory community/City/ that corresponds to the enclosing team class.

As a special treat the package explorer of the OTDT provides two display options for team packages: logical vs. physical. In the physical view the 2-files solution looks like this:

Just by selecting the logical view, where the team-package is not shown, the display changes to

Small files in separate directories are good for team support etc. Logical nesting is good for modularity. Why not have the cake and eat it?

These are just little tricks introduced by OT/J and the OTDT. But with these there’s no excuse any longer for not using class nesting even for large structures. And remember: this is real nesting, so you can use the same concept for 1-level, 2-level, … n-level containment. Good news for developers, bad news for technology designers, because now we simply don’t need any “superpackages”, “modules” etc. I actually wonder, how nestable classes could be unified with bundles, but that’s still a different story to be told on a different day.

Before that I will discuss how nesting and inheritance produce tremendous synergy (instead of awkwardly standing side-by-side). From a research perspective this has been solved some 9 years ago. I strongly believe that the time is ripe to bring these fruits from ivory towers to the hands of real world developers. Stay tuned for part 2.

This is a response to Chris Aniszczyk’s post “Eclipse and Academia“.

I’m glad he raises the issue of academia producing cool stuff vs. consuming Eclipse in class, which seems to be better supported.

Clearly, Object Teams is one of the projects that has crossed the line between both worlds, so let my try to summarize the experience I made along the way.

First, it’s important to distinguish 2 kinds of academic projects: There are individual (PhD) projects, which indeed have quite limited resources. However, there’s also funded projects that typically run about 3 years, involving a number of people, perhaps from different institutions. Object Teams is of the second kind.

When I submitted presentations for EclipseCon the main goal certainly was to increase visibility (both for getting feedback and setting the grounds for a business, eventually). I made the experience that an academic project is not very likely to meet the criteria of an EclipseCon program committee: EclipseCon wants presentations about topics that everybody already talks about. By definition a research project covers topics that nobody has been talking about yet. I once discussed this with Bjorn and apparently the poster session at ESE is an offspring of discussions like that. However, I think posters is not enough. I had a poster at ESE ‘07, to the effect that I could talk to about 4 persons. That doesn’t really help a lot.

I would really suggest to have a track with regular presentations where selection criteria are just shifted from already-hyped to potentially-changing-things-in-the-future.

Another thing that would help integrating academic projects that actually move to Eclipse.org would be a track of New Project presentations. When the Object Teams project was in its proposal phase I submitted four presentations to EclipseCon none of which was accepted, so on that channel I cannot introduce the project to the community. I know of another project that made the same experience.

Sorry, if this sounds a bit negative. Summarizing my experience, currently it’s much easier to get involved bottom-up like in Eclipse Demo Camps (great!), blogging, forums etc. Regarding “top-level” visibility academic projects seem to be starting with a handicap in Eclipse land.

I also like the idea of collecting Eclipse-related publications. If things go well I’d actually expect a huge number of such publications to show up, which would soon require smart categorization to ensure that the list is digestable.

Regarding the opposite direction, going to academic conferences: Sure there was a successful series of Eclipse Technology eXchange events at OOPSLA and ECOOP and I once tried to organize one at ECOOP but failed. My feeling is that by now Eclipse is so pervasive in research that just the fact of using Eclipse for your prototypes isn’t enough of a commonality any more to foster an “academic” workshop. There might be a case for tutorials to jump start young researchers: “write your first dataflow analysis (refactoring, metrics, Java extension …) in Eclipse“. I’m not sure how big the gap would be between existing articles and what a typical PhD would need.

In the first part I demonstrated how Object Teams can be used to extend the JDT compiler for a custom language to be embedded in Java. I concluded by saying that more substantial features like Refactoring might need more rocket science which I wanted to show next.

The “bad news” is: before I started to do some strong adaptations of DOM AST etc to make Refactorings work, I just made a few experiments of how Refactorings actually behaved in my hybrid language. To my own surprise a lot of things already worked OK: I could extract a custom syntax expression into a local variable and inline the variable again and more stuff of that kind. Just look at this example:

Actually this reflects an experience I’ve made more than once: If you reuse some module and perform some adaptations in terms of provided API and extension points etc. more often than not one adaptation entails the next, adding tweaks to workarounds because you keep scratching at the surface. If, OTOH, you succeed to make your adaptation right at the core where the decisions are made, just one or two cuts and stitches may suffice to get your job done. Clean, effective and consistent. That’s what we see when cleanly inserting a custom AST node into the JDT: if our CustomIntLiteral behaves well a lot of JDT functionality can just work with this thing without knowing it’s not a genuine Java thing.

Now this means for my next example I had to look for an extra challenge. I decided to enhance the example in two ways:

• The custom syntax should be a bit more realistic, so I chose to create a syntax for money, consisting of a number and the name of a currency
• I wanted source formatting to work for the whole hybrid language

A word of warning: this post uses some bells and whistles of OT/J and applies it to the non-trivial JDT. This might be a bit overwhelming for the novice. If you prefer lower dosage first, you may want to check out our example section in the wiki. It’s still far from complete but I’m working on it.

A syntax for money

The new syntax should allow me to write this:

int getMoney() {
    return <% 13 euro %>;
}

and the stuff should internally be stored as a structured AST node. This is how class CurrencyExpression starts:

public class CurrencyExpression extends Expression {
 
    public IntLiteral value;
    public String currency;
 
    final static String[] CURRENCIES = { "euro", "dollar" };
 
    public CurrencyExpression(int sourceStart, int sourceEnd) { ...
 
    public boolean setCurrency(String string) { ...
 
    @Override
    public StringBuffer printExpression(int indent, StringBuffer output) { ...
    ....
}

For creating a CurrencyExpression from source I wrote a little CustomParser, normal boring stuff with 40% just reading individual chars and manipulating character positions, another 45% actually does some error reporting and only 3 lines are relevant: those that create a new CurrencyExpression, create an IntValue for the value part and invoke setCurrency with the currency string.

In the ScannerAdaptor from the previous post I simply replaced this

Expression replacement = new CustomIntLiteral(source, start, end, start+2, end-2);

with this:

Expression replacement = customParser.parseCurrencyExpression(source, start, end, this.getProblemReporter());

That suffices to make the above little method compile and run just as expected.

Interlude: DOM AST

Well, with this slightly more realistic syntax you’d actually see a number of exceptions in the IDE that can all be fixed by letting the DOM AST know about our addition. For those who don’t regularly program against the JDT API: the DOM AST is the public data structure by which tools outside the JDT core manipulate Java programs. Inside the JDT extending the DOM AST would mean to subclass either org.eclipse.jdt.core.dom.ASTNode or one of its subclasses. Unfortunately, all constructors in this hierarchy are package private, and even with OT/J we respect what the javadoc says: “clients are unable to declare additional subclasses“.

But we can do something similar: instead of subclassing we can use instances of a regular DOM class and attach a role instance to them. As the base I chose org.eclipse.jdt.core.dom.SimpleName which inside the JDT could mean a lot of different things, so for most parts a node of this kind is regarded as a black box, just what we need. This is the role I added to the team SyntaxAdaptor from the previous post:

1
2
3
4
5
6
7
8
9
10
11
12

protected class DomCurrencyLiteral playedBy SimpleName {
    protected String currency;
 
    void setSourceRange(int sourceStart, int length) -> void setSourceRange(int sourceStart, int length);
 
    @SuppressWarnings("decapsulation")
    public DomCurrencyLiteral(AST ast, CurrencyExpression expression) {
        base(ast);
        this.currency = expression.currency;
        setSourceRange(expression.sourceStart, expression.sourceEnd-expression.sourceStart+1);
    }
}

So this almost looks like subclassing except we use playedBy instead of extends and base() instead of super(). And yes, when creating an instance with “new DomCurrencyLiteral(ast, expr)” inside the constructor we create a SimpleName from DOM using the package private constructor. But by using role playing instead of sub-classing this has become part of the aspectBinding relationship, which makes analysis of the state of encapsulation much easier.

So, who actually creates these nodes? Inside the JDT this is the responsibility of the ASTConverter, which takes an AST from the compiler and converts it to the public variant. In order to tell the ASTConverter how to handle our currency nodes I added this role to the existing team SyntaxAdaptor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

@SuppressWarnings("decapsulation")
protected class DomConverterAdaptor playedBy ASTConverter {
 
    // whenever convert(Expression) is called ...
    org.eclipse.jdt.core.dom.Expression convertCurrencyExpression(CurrencyExpression expression)
    <- replace org.eclipse.jdt.core.dom.Expression convert(Expression expression)
        // ... and when the literal is actually a CurrencyExpression ...
        base when (expression instanceof CurrencyExpression)
        // ... perform the cast we just checked for and feed it into the callin method below.
        with { expression <- (CurrencyExpression)expression }
 
    /**
     * Convert a CustomIntLiteral from the compiler to its dom counter part.
     * This method uses inferred callouts (OTJLD §3.1(j))
     * which need to be enabled in the OT/J compiler preferences.
     */
    @SuppressWarnings({ "basecall", "inferredcallout" })
    callin org.eclipse.jdt.core.dom.Expression convertCurrencyExpression(CurrencyExpression expression){
        final DomCurrencyLiteral name = new DomCurrencyLiteral(this.ast, expression);
        if (this.resolveBindings) {
            recordNodes(name, expression);
        }
        return name;
    }
}

I deliberately used some special OT/J syntax worth explaining:

• Lines 5ff. define a callin bindings like we’ve seen before.
• Line 8 adds a guard predicate to the binding, saying that this binding should only fire when the argument expression is actually of type CurrencyExpression
• After passing the guard we know that we can safely cast to CurrencyExpression so I added a parameter mapping (line 10) which feeds a casted value into the role method.
• Inside the role method convertCurrencyExpression everything looks normal, but at a closer look this.ast and this.resolveBindings seem to be undefined in the scope of the current class. In fact these fields are defined in the base class ASTConverter and we could use explicit callout accessors like in the previous post. However, this time I chose to let the compiler infer these callouts so that the method would look exactly like existing methods in ASTConverter do (this option has to be enabled in the OT/J compiler preferences).

OK, with this little addition our CurrencyExpressions are converted to something that the JDT can handle and we’re already prepared for doing real AST manipulation including our syntax.

Source Formatting

Inside the JDT source formatting (Ctrl-Shift-F) is essentially performed by class CodeFormatterVisitor. This class is one of many subclasses of the general ASTVisitor. If one wanted to make these visitors aware of our CurrencyExpression we would have to add one visit method to ASTVisitor and each of its sub-classes! That’s certainly not viable, so with plain Java we’re pretty much out of luck.

The situation that needs adaptation can be described as follows:

• A visitor will be created and invoked in order to descend into the AST
• At the point when traversal finds a CurrencyExpression it will invoke its traverse(ASTVisitor) method.

Of course we could manually inspect the type of visitor within the traverse method, but that would defy the whole purpose of having visitors: keep all those add-on functions out from your data structures. Instead I only gave a default implementation to CurrencyExpression.traverse and used OT/J for the cleanest implementation of double dispatch (which is what the visitor pattern painstakingly emulates): we need dispatch that considers both the visitor type and the node type for finding the suitable method implementation.

In green-field development this would be still easier but even on top of an existing visitor infrastructure it get’s pretty concise.

Visitor adaptation - version 1

My first version looks like this (explanations follow below):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

public team class VisitorsAdaptor {
 
    protected team class AstFormatting playedBy CodeFormatterVisitor {
        // whenever visiting something that could contain an expression
        // activate this team to enable callins of the inner role
        callin void visiting() {
            within(this) {
                base.visiting();
            }
        }
        @SuppressWarnings("decapsulation")
        void visiting()
            <- replace
                boolean visit(Block block, BlockScope scope),
                boolean visit(FieldDeclaration fieldDeclaration, MethodScope scope),
                void formatStatements(BlockScope scope, final Statement[] statements, boolean insertNewLineAfterLastStatement);
 
        Scribe getScribe() -> get Scribe scribe;
 
        /** This role implements formating of our custom ast: */
        protected class CustomAst playedBy CurrencyExpression
        {
            void traverse() <- replace void traverse(ASTVisitor visitor, BlockScope scope);
 
            @SuppressWarnings({ "inferredcallout", "basecall" })
            callin void traverse() {
                Scribe scribe = getScribe();
                Scanner scanner = scribe.scanner;
 
                // format this AST node into a StringBuffer:
                StringBuffer replacement = new StringBuffer();
                replacement.append("<% ");
                this.value.printExpression(0, replacement);
                replacement.append(' ');
                replacement.append(this.currency);
                replacement.append(" %>");
 
                // feed the formatted string into the Scribe:
                int start = this.sourceStart();
                int end = this.sourceEnd();
                scribe.addReplaceEdit(start, end, replacement.toString());
 
                // advance the scanner:
                scanner.resetTo(end+1, scribe.scannerEndPosition - 1);
                scribe.pendingSpace = false;
            }
        }
    }
}

The key trick in this example is nesting:

• Role AstFormatting is responsible for detecting when a CodeFormatterVisitor is visiting any subtree that may contain expressions. This is done using a callin binding that lists three relevant base methods which all should be intercepted by the same role method (lines 12-16).
• Inside role AstFormatting (which is also marked as a team) an inner role CustomAst will only be triggered if a CodeFormatterVisitor calls the traverse method of a CurrencyExpression (see callin binding in line 23).
• The connection between both levels is wired in method AstFormatting.visiting: the block statement within() { } temporarily and locally activates the given team instance, here denoted by this. Only during this block the nested team AstFormatting is active - meaning that only during this block the callin binding in role CustomAst will fire.
• Within role CustomAst we can naturally access the CodeFormatterVisitor via the enclosing instance of AstFormatting. No instanceof and casting needed, because all this only happens in the context of a CodeFormatterVisitor

The body of method traverse contains only domain logic: pretty-printing the current node into a string buffer and interacting with the underlying infrastructure (Scanner, Scribe) that drives the formatting.

That’s it, with these classes in place, we can write this method:

int getMoney() {
   int myMoney = <% 
		  3
	  euro %>
	; 
		System
	.out.println("myMoney ="+myMoney);
					return myMoney;
}

then hit Ctrl-Shift-F et voilà:

private static int getMoney() {
    int myMoney = <% 3 euro %>;
    System.out.println("myMoney =" + myMoney);
    return myMoney;
}

How’s that?
The formatter smoothly operates on the full hybrid language, not just skipping over our nodes but handling them as well.

Generalizing visitor adaptations

After success wrt both challenges I’d like to clean up even more and prepare for further adaptations of other visitors. Given how many subclasses of ASTVisitor are used within the JDT we wouldn’t want to write the infrastructure for double dispatch over and over again. So let’s generalize, that is: extract a common super-class, by extracting everything re-usable out off class AstFormatting

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

public team class VisitorsAdaptor
{
    protected abstract team class AstVisiting playedBy ASTVisitor {
        // whenever visiting something that could contain an expression
        // activate this team to enable callins of the inner role
        callin void visiting() {
            within(this)
                base.visiting();
        }
        void visiting()
            <- replace
                boolean visit(Block block, BlockScope scope),
                boolean visit(FieldDeclaration fieldDeclaration, MethodScope scope);
 
        protected abstract class CustomAst playedBy CurrencyExpression {
            // variant of traversal that should be used when the enclosing team is active:
            // (implement in subclasses)
            abstract callin void traverse();
            void traverse() <- replace void traverse(ASTVisitor visitor, BlockScope scope);
        }
        // Insert more roles for binding more AST nodes...
    }
 
    protected team class AstFormatting extends AstVisiting playedBy CodeFormatterVisitor
    {
        // one more trigger that should activate the team:
        @SuppressWarnings("decapsulation")
        visiting <- replace formatStatements;
 
        Scribe getScribe() -> get Scribe scribe;
 
        /** This role implements formating of our custom ast: */
        @Override
        protected class CustomAst {
            @SuppressWarnings({ "inferredcallout", "basecall" })
            callin void traverse() {
                // method body as before
            }
        }
    }
    protected team class OtherVisitorAdaptor extends AstVisiting playedBy XYVisitor
    {
        @Override
        protected class CustomAst {
            callin void traverse() {
                // domain logic
            }
        }
        // Insert more roles for actually handling more AST nodes ...
    }
}

Now team class AstVisiting contains the part that is common for all visitors. At this level several things are still abstract: method traverse, role class CustomAst and even the whole team AstVisiting.

Team class AstFormatting extends the abstract team and defines everything specific to formatting. We have one more trigger for visiting, one callout binding to a field of class CodeFormatterVisitor and then we only refine the previously abstract role class CustomAst. At this level it is no longer abstract because we give an implementation for traverse.

I’ve also sketched another nested team showing a minimal specialization of AstVisiting for adapting some other visitor and adding another implementation for CustomAst.traverse plus potentially more roles for more node types.

Conclusion

For those who don’t work in the compiler business on a day-to-day basis this is probably pretty tough stuff, but let me summarize what we’ve just achieved:

• Embed a custom syntax into Java, showing how a custom parser can be plugged in to create custom AST from a region of the Java source.
• Adapt the conversion between two different AST structures (internal -> DOM) to also handle custom nodes.
• Adapt the code formatter so that hybrid sources can be formatted with a single command.
• Prepared the infrastructure for adapting other visitors, too. By this we have achieved that new visitor adaptations will only need to add their specific implementation with close to zero scaffolding.
• Cleanly separated each implemented concern in one module.
• Keep each module in the scale of only tens of lines of code.
• Yet implement significant steps towards a production quality IDE for our custom hybrid language.

Maybe I shouldn’t have told you, how easy these things can be - if your tools are sharp - maybe.
But professional carvers know: if your knife is sharp, it’s actually easy to handle. Only if it is blunt you are in real danger of hurting yourself - because you need to apply disproportionate force to cut your wood. So:

Spare your fingers, sharpen your knife!

PS: Here’s the archive of all sources, ready to be imported into the OTDT.