The Object Teams Blog

During the Juno cycle a lot of work in the JDT has gone into more sophisticated static analysis, and some more is still in the pipe-line. I truly hope that once Juno is shipped this will help all JDT users to find more bugs immediately while still typing. However, early feedback regarding these features shows that users are starting to expect miracles from the analysis

On the one hand seeing this is flattering, but on the other hand it makes me think we should perhaps explain what exactly the analysis can see and what is beyond its vision. If you take a few minutes learning about the concepts behind the analysis you’ll not only understand its limitations, but more importantly you will learn how to write code that’s better readable - in this case for reading by the compiler. Saying: with only slightly rephrasing your programs you can help the compiler to better understand what’s going on, to the effect that the compiler can answer with much more useful error and warning messages.

Since there’s a lot of analysis in this JDT compiler I will address just one topic per blog post. This post goes to improvements in the detection of resource leaks.

Resource leaks - the basics

Right when everybody believed that Eclipse Indigo RC 4 was ready for the great release, another blocker bug was detected: a simple resource leak basically prevented Eclipse from launching on a typical Linux box if more than 1000 bundles are installed. Coincidentally, at the same time the JDT team was finishing up work on the new try-with-resources statement introduced in Java 7. So I was thinking: shouldn’t the compiler help users to migrate from notoriously brittle handling of resources to the new construct that was designed specifically to facilitate a safe style of working with resources?

What’s a resource?

So, how can the compiler know about resources? Following the try-with-resources concept, any instance of type java.lang.AutoCloseable is a resource. Simple, huh? In order to extend the analysis also to pre Java 7 code, we also consider java.io.Closeable (available since 1.5).

Resource life cycle

The expected life cycle of any resource is : allocate—use—close. Simple again.

From this we conclude the code pattern we have to look for: where does the code allocate a closeable and no call to close() is seen afterwards. Or perhaps a call is seen but not all execution paths will reach that call, etc.

Basic warnings

With Juno M3 we released a first analysis that could now tell you things like:

• Resource leak: “input” is never closed
• Resource leak: “input” is never closed at this location (if a method exit happens before reaching close())

If the problem occurs only on some execution paths the warnings are softened (saying “potential leak” etc.).

Good, but…

Signal to noise - part 1

It turned out that the analysis was causing significant noise. How come? The concepts are so clear and all code that wouldn’t exhibit the simple allocate—use—close life cycle should indeed by revised, shouldn’t it?

In fact we found several patterns, where these warnings were indeed useless.

Resource-less resources

We learned that not every subtype of Closeable really represents a resource that needs leak prevention. How many times have you invoked close() on a StringWriter, e.g.? Just have a look at its implementation and you’ll see why this isn’t worth the effort. Are there more classes in this category?

Indeed we found a total of 7 classes in java.io that purely operate on Java objects without allocating any resources from the operating system:

• StringReader
• StringWriter
• ByteArrayInputStream
• ByteArrayOutputStream
• CharArrayReader
• CharArrayWriter
• StringBufferInputStream

For none of these does it make sense to warn about missing close().

To account for these classes we simply added a white list: if a class is in the list suppress any warnings/errors. This white list consists of exactly those 7 classes listed above. Sub-classes of these classes are not considered.

Wrapper resources

Another group of classes implementing Closeable showed up, that are not strictly resources themselves. Think of BufferedInputStream! Does it need to be closed?

Well? What’s your answer? The correct answer is: it depends. A few examples:

1
2
3
4
5
6
7
8

	void wrappers(String content) throws IOException {
		Reader r1, r2, r3, r4;
		r1 = new BufferedReader(new FileReader("someFile"));
		r2 = new BufferedReader(new StringReader(content));
		r3 = new FileReader("somefile");
		r4 = new BufferedReader(r3);
		r3.close();
	}

How many leaks? With same added smartness the compiler will signal only one resource leak: on r1. All others are safe:

• r2 is a wrapper for a resource-less closeable: no OS resources are ever allocated here.
• r3 is explicitly closed
• r4 is just a wrapper around r3 and since that is properly closed, r4 does not hold onto any OS resources at the end.
• returning to r1, why is that a leak? It’s a wrapper, too, but now the underlying resource (a FileReader) is not directly closed so it’s the responsibility of the wrapper and can only be triggered by calling close() on the wrapper r1.

Summarizing: wrappers don’t directly hold an OS resource, but delegate to a next closeable. Depending on the nature and state of the nested closeable the wrapper may or may not be responsible for closing. In arbitrary chains of wrappers with a relevant resource at the bottom, closing any closeable in the chain (including the bottom) will suffice to release the single resource. If a wrapper chain is not properly closed the problem will be flagged against the outer-most wrapper, since calling close() at the wrapper will be delegated along all elements of the chain, which is the cleanest way of closing.

Also for wrappers the question arises: how does the compiler know? Again we set up a white list with all wrapper classes we found in the JRE: 20 classes in java.io, 12 in java.util.zip and 5 in other packages (the full lists are in TypeConstants.java, search for “_CLOSEABLES”).

Status and outlook

Yes, a leak can be a stop-ship problem.

Starting with Juno M3 we have basic analysis of resource leaks; starting with Juno M5 the analysis uses the two white lists mentioned above: resource-less closeables and resource wrappers. In real code this significantly reduces the number of false positives, which means: for the remaining warnings the signal-to-noise ratio is significantly better.

M5 will actually bring more improvements in this analysis, but that will be subject of a next post.

The recent release of Juno M4 brought an interesting combination: The Object Teams Development Tooling now natively supports annotation-based null analysis for Object Teams (OT/J). How about that?

The path behind us

Annotation-based null analysis has been added to Eclipse in several stages:

Using OT/J for prototyping

As discussed in this post, OT/J excelled once more in a complex development challenge: it solved the conflict between extremely tight integration and separate development without double maintenance. That part was real fun.

Applying the prototype to numerous platforms

Next I reported that only one binary deployment of the OT/J-based prototype sufficed to upgrade any of 12 different versions of the JDT to support null annotations — looks like a cool product line

Pushing the prototype into the JDT/Core

Next all of the JDT team (Core and UI) invested efforts to make the new feature an integral part of the JDT. Thanks to all for this great collaboration!

Merging the changes into the OTDT

Now, that the new stuff was mixed back into the plain-Java implementation of the JDT, it was no longer applicable to other variants, but the routine merge between JDT/Core HEAD and Object Teams automatically brought it back for us. With the OTDT 2.1 M4, annotation-based null analysis is integral part of the OTDT.

Where we are now

Regarding the JDT, others like Andrey, Deepak and Aysush have beaten me in blogging about the new coolness. It seems the feature even made it to become a top mention of the Eclipse SDK Juno M4. Thanks for spreading the word!

Ah, and thanks to FOSSLC you can now watch my ECE 2011 presentation on this topic.

Two problems of OOP, and their solutions

Now, OT/J with null annotations is indeed an interesting mix, because it solves two inherent problems of object-oriented programming, which couldn’t differ more:

1.: NullPointerException is the most widespread and most embarrassing bug that we produce day after day, again and again. Pushing support for null annotations into the JDT has one major motivation: if you use the JDT but don’t use null annotations you’ll no longer have an excuse. For no good reasons your code will retain these miserable properties:

• It will throw those embarrassing NPEs.
• It doesn’t tell the reader about fundamental design decisions: which part of the code is responsible for handling which potential problems?

Why is this problem inherent to OOP? The dangerous operator that causes the exception is this:

2.: Objectivity seems to be a central property on any approach that is based just on Objects. While so many other activities in software engineering are based on the insight that complex problems with many stakeholders involved can best be addressed using perspectives and views etc., OOP forces you to abandon all that: an object is an object is an object. Think of a very simple object: a File. Some part of the application will be interested in the content so it can decode the bytes and do s.t. meaningful with it, another part of the application (maybe an underlying framework) will mainly be interested in the path in the filesystem and how it can be protected against concurrent writing, still other parts don’t care about either but only let you send the thing over the net. By representing the “File” as an object, that object must have all properties that are relevant to any part of the application. It must be openable, lockable and sendable and whatnot. This yields bloated objects and unnecessary, sometimes daunting dependencies. Inside the object all those different use cases it is involved in can not be separated!

With roles objectivity is replaced by a disciplined form of subjectivity: each part of the application will see the object with exactly those properties it needs, mediated by a specific role. New parts can add new properties to existing objects — but not in the unsafe style of dynamic languages, but strictly typed and checked. What does it mean for practical design challenges? E.g, direct support for feature oriented designs - the direct path to painless product lines etc.

Just like the dot, objectivity seems to be hardcoded into OOP. While null annotations make the dot safe(r), the roles and teams of OT/J add a new dimension to OOP where perspectives can be used directly in the implementation. Maybe it does make sense, to have both capabilities in one language although one of them cleans up what should have been sorted out many decades ago while the other opens new doors towards the future of sustainable software designs.

The road ahead

The work on null annotations goes on. What we have in M4 is usable and I can only encourage adopters to start using it right now, but we still have an ambitious goal: eventually, the null analysis shall not only find some NPEs in your program, but eventually the absense of null related errors and warnings shall give the developer the guarantee that this piece of code will never throw NPE at runtime.

What’s missing towards that goal:

1. Fields: we don’t yet support null annotations for fields. This is next on our plan, but one particular issue will require experimentation and feedback: how do we handle the initialization phase of an object, where fields start as being null? More on that soon.
2. Libraries: we want to support null specifications for libraries that have no null annotations in their source code.
3. JSR 308: only with JSR 308 will we be able to annotate all occurrences of types, like, e.g., the element type of a collection (think of List<@NonNull String>)

Please stay tuned as the feature evolves. Feedback including bug reports is very welcome!

Ah, and one more thing in the future: I finally have the opportunity to work out a cool tutorial with a fellow JDT committer: How To Train the JDT Dragon with Ayushman. Hope to see y’all in Reston!

I’ve been involved in the release of different versions of the JDT lately, supporting different flavors of Java.

Classical release management

At the core we have the plain JDT, of which we published the 3.7.0 release in June and right now first release candidates are being prepared towards the 3.7.1 service release, which will be the first official release to support Java 7. At the same time the first milestones towards 3.8 are being built. OK, this is almost normal business — with the exception of the service release differs more than usual from its base release, due to the unhappy timing of the release of Java 7 vs. Eclipse 3.7.

First variant: Object Teams

The same release plan is mirrored by the Object Teams releases 2.0.0, 2.0.1RC1, 2.1.0M1. Merging the delta from JDT 3.7 to 3.7.1 into the OTDT was a challenge, given that this delta contained the full implementation of all that’s new in Java 7. Still with the experience of regularly merging JDT/Core changes into the OT variant, the pure merging was less than one day plus a couple more days until all 50000+ tests were green again. The nice thing about the architecture of the OTDT: after merging the JDT/Core, I was done. Since all other adaptations of the JDT are implemented using OT/Equinox adopting, e.g., all the new quick assists for Java 7 required a total of zero minutes integration time.

I took the liberty of branching 2.0.x and 2.1 only after integrating the Java 7 support, which also means that 2.1 M1 has only a small number of OT-specific improvements that did not already go into 2.0.1.

Prototyping annotation based null analysis

As I wrote before, I’m preparing a comprehensive new feature for the JDT/Core: static analysis for potential NullPointerException based on annotations in the code. The latest patch attached to the bug had almost 3000 lines. Recent discussions at ECOOP made me change my mind in a few questions, so I changed some implementation strategies. Luckily the code is well modularized due to the use of OT/Equinox.

Now came the big question: against which version of the JDT should I build the null-annotation add-on? I mean, which of the 6 versions I have been involved in during the last 2 months?

As I like a fair challenge every now and then I decided: all six, i.e., I wanted to support adding the new static analysis to all six JDT versions mentioned before.

Integration details

Anybody who has worked on a Java compiler will confirm: if you change one feature of the compiler chances are that any other feature can be broken by the change (I’m just paraphrasing: “it’s complex”). And indeed, applying the nullity plug-in to the OTDT caused some headache at first, because both variants of the compiler make specific assumptions about the order in which specific information is available during the compilation process. It turned out that two of these assumptions where simply incompatible, so I had to make some changes (here I made the null analysis more robust).

At the point where I thought I was done, I tripped over an ugly problem that’s intrinsic to Java.
The nullity plug-in adapts a method in the JDT/Core which contains the following switch statement:

	while ((token = this.scanner.getNextToken()) != TerminalTokens.TokenNameEOF) {
		IExtendedModifier modifier = null;
		switch(token) {
			case TerminalTokens.TokenNameabstract:
				modifier = createModifier(Modifier.ModifierKeyword.ABSTRACT_KEYWORD);
				break;
			case TerminalTokens.TokenNamepublic:
				modifier = createModifier(Modifier.ModifierKeyword.PUBLIC_KEYWORD);
				break;
			// more cases
		}
	}

I have a copy of this method where I only added a few lines to one of the case blocks.
Compiles fine against any version of the JDT. But Eclipse hangs when I install this plugin on top of a wrong JDT version. What’s wrong?

The problem lies in the (internal) interface TerminalTokens. The required constants TokenNameabstract etc. are of course present in all versions of this interface, however the values of these constants change every time the parser is generated anew. If constants were really abstractions that encapsulate their implementation values, all would be fine, but the Java byte code knows nothing about such an abstraction, all constant values are inlined during compilation. In other words: the meaning of a constant depends solely on the definitions which the compiler sees during compilation. Thus compiling the above switch statement hardcodes a dependency on one particular version of the interface TerminalTokens. BAD.

After recognizing the problem, I had to copy some different versions of the interface into my plug-in, implement some logic to translate between the different encodings and that problem was solved.

What’s next?

Nothing is next. At this point I could apply the nullity plug-in to all six versions of the JDT and all are behaving well.

Mix-n-match

Would you like Java with our without the version 7 enhancements (stable release or milestone)? May I add some role and team classes? How about a dash more static analysis? It turns out we have more than just one product, we have a full little product line with features to pick or opt-out:

• OTDT: http://www.eclipse.org/objectteams/download.php
• Null ananlysis: http://wiki.eclipse.org/JDT_Core/Null_Analysis

Just make your choice
Happy hacking with null annotations and try-with-resources in OT/J.

BTW: if you want to hear a bit more about the work on null annotations, you should really come to EclipseCon Europe — why not drop a comment at this submission

At our EclipseCon tutorial we mentioned a bonus excercise, for which we didn’t have the time at the tutorial.

Now it’s time to reveal the solution.

Task

“Implement the following demo-mode for the JDT:

• When creating a Java project let the user select:

❒ Project is for demo purpose only

• When creating a class in a Demo project:

insert class name as “Foo1”, “Foo2” …”

So creating classes in demo mode is much easier, and you’ll use the names “Foo1″… anyway
(See also our slides (#39)).

Granted, this is a toy example, yet it combines a few properties that I frequently find in real life and which cause significant pains without OT/J:

• The added behavior must tightly integrate with existing behavior.
• The added behavior affects code at distant locations,
  here two plug-ins are affected: org.eclipse.jdt.ui and org.eclipse.jdt.core.
• The added behavior affects execution at different points in time,
  here creation of a project plus creation of a class inside a project.
• The added behavior requires to maintain more state at existing objects,
  here a JavaProject must remember if it is a demo project.

Despite these characteristics the task can be easily described in a few English sentences. So the solution should be similarly concise and delivered as a single coherent piece.

Strategy

With a little knowledge about the JDT the solution can be outlined as this

• Add a checkbox to the New Java Project wizard
• When the wizard creates the project mark it as a demo project if the box is checked.
• Let the project also count the names of Foo.. classes it has created.
• When the new class wizard creates a class inside a demo project pre-set the generated class name and make the name field unselectable.

From this we conclude the need to define 4 roles, playedBy these existing types:

• org.eclipse.jdt.ui.wizards.NewJavaProjectWizardPageOne.NameGroup:
  the wizard page section where the project name is entered and where we want to add the checkbox.
• org.eclipse.jdt.ui.wizards.NewJavaProjectWizardPageTwo:
  the part of the wizard that triggers setup of the JavaProject.
• org.eclipse.jdt.core.IJavaProject:
  this is where we need to add more state (isDemoProject and numFooClasses).
• org.eclipse.jdt.ui.wizards.NewTypeWizardPage:
  this is where the user normally specifies the name for a new class to be created.

Note, that 3 classes in this list resided in org.eclipse.jdt.ui, but IJavaProject is from org.eclipse.jdt.core, which leads us to the next step:

Plug-in configuration

Our solution is developed as an OT/Equinox plug-in, with the following architecture level connections:

This simply says that the same team demohelper.ProjectAdaptor is entitled to bind roles to classes from both org.eclipse.jdt.ui and org.eclipse.jdt.core.
One more detail in these extensions shouldn’t go unmentioned: Don’t forget to set “activation: ALL_THREADS” for the team (otherwise you won’t see any effect …).

Now we’re ready to do the coding.

Implementing the roles

18
19
20
21
22
23
24
25
26
27
28
29
30
31

protected class DialogExtender playedBy NameGroup {
 
	protected SelectionButtonDialogField isDemoField;
 
	void createControl(Composite parent) <- after Control createControl(Composite composite)
		with { parent <- (Composite) result }
 
	private void createControl(Composite parent) {
		isDemoField= new SelectionButtonDialogField(SWT.CHECK);
		isDemoField.setLabelText("Project is for demo purpose only");
		isDemoField.setSelection(false);
		isDemoField.doFillIntoGrid(parent, 4);
	}
}

Our first role adds the checkbox. The implementation of createControl is straight-forward UI business. Lines 22,23 hook our role method into the one from the bound base class NameGroup. After the with keyword, we are piping the result from the base method into the parameter parent of the role method (with a cast). This construct is a parameter mapping.

Here’s the result:

Next we want to store the demo-flag to instances of IJavaProject, so we write this role:

52
53
54
55
56
57
58
59
60

protected class EnhancedJavaProject playedBy IJavaProject {
 
	protected boolean isDemoProject;
	private int numFooClasses = 1;
 
	protected String getTypeName() {
		return "Foo"+(numFooClasses++);
	}		
}

Great, now any IJavaProject can play the role EnhancedJavaProject which holds the two additional fields, and we can automatically serve an arbitrary number of class names Foo1 …
In the IDE you will actually see a warning, telling you that binding a role to a base interface currently imposes a few restrictions, but these don’t affect us in this example.

Next comes a typical question: how to transfer the flag from role DialogExtender to role EnhancedJavaProject?? The roles don’t know about each other nor do the bound base classes. The answer is: use a chain of references.

33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

protected class FirstPage playedBy NewJavaProjectWizardPageOne {
 
	DialogExtender getFNameGroup() -> get NameGroup fNameGroup;
 
	protected boolean isDemoProject() {
		return getFNameGroup().isDemoField.isSelected();
	}		
}
 
protected class WizardExtender playedBy NewJavaProjectWizardPageTwo {
 
	FirstPage getFFirstPage() -> get NewJavaProjectWizardPageOne fFirstPage;
 
	markDemoProject <- after initializeBuildPath;
	private void markDemoProject(EnhancedJavaProject javaProject) {
		if (getFFirstPage().isDemoProject())
			javaProject.isDemoProject = true;
	}
}

Role WizardExtender intercepts the event when the wizard initializes the IJavaProject (line 46). Method initializedBuildPath receives a parameter of type IJavaProject but the OT/J runtime transparently translates this into an instance of type EnhancedJavaProject (this - statically type safe - operation is called lifting). Another indirection is needed to access the checkbox: The base objects are linked like this:

This link structure is lifted to the role level by the callout bindings in lines 35 and 44.

We’re ready for our last role:

62
63
64
65
66
67
68
69
70
71
72
73

protected class NewTypeExtender playedBy NewTypeWizardPage {
 
	void setTypeName(String name, boolean canBeModified) -> void setTypeName(String name, boolean canBeModified);
 
	void initTypePage(EnhancedJavaProject prj) <- after void initTypePage(IJavaElement element)
		with { prj <- element.getJavaProject() }
 
	private void initTypePage(EnhancedJavaProject prj) {
		if (prj.isDemoProject)
			setTypeName(prj.getTypeName(), false);
	}
}

Here we intercept the initialization of the type page of a New Java Project wizard (lines 66,67). Another parameter mapping is used to perform two adjustments in one go: fetch the IJavaProject from the enclosing element and lift it to its EnhancedJavaProject role. This follows the rule-of-thumb that base-type operations (like navigating from IJavaElement to IJavaProject) should happen at the right hand side, so that we are ready to lift the IJavaProject to EnhancedJavaProject when the data flow enters the team.

The EnhancedJavaProject can now be asked for its stored flag (isDemoProject) and it can be asked for a generated class name (getTypeName()). The generated class name is then inserted into the dialog using the callout binding in line 64. Looks like this:

See this? No need to think of a good class name

Wrap-up

So that’s it. All these roles are collected in one team class and here is the fully expanded outline:

All this is indeed one concise and coherent module. In the tutorial I promised to do this no more than 80 LOC, and indeed the team class has 74 lines including imports and white space.

Or, if you are interested just in how this module connects to the existing implementation, you may use the “binding editor” in which you see all playedBy, callout and callin bindings:

The full sources are also available for download.

have fun

So, I’ve been working on annotations @NonNull and @Nullable so that the Eclipse Java compiler can statically detect your NullPointerExceptions already during compile time (see also bug 186342).

By now it’s clear this new feature will not be shipped as part of Eclipse 3.7, but that needn’t stop you from trying it, as I have uploaded the thing as an OT/Equinox plugin.

Behind the scenes: Object Teams

Today’s post shall focus on how I built that plugin using Object Teams, because it greatly shows three advantages of this technology:

• easier maintenance
• easier deployment
• easier development

Before I go into details, let me express a warm invitation to our EclipseCon tutorial on Thursday morning. We’ll be happy to guide your first steps in using OT/J for your most modular, most flexible and best maintainable code.

Maintenance without the pain

It was suggested that I should create a CVS branch for the null annotation support. This is a natural choice, of course. I chose differently, because I’m tired of double maintenance, I don’t want to spend my time applying patches from one branch to the other and mending merge conflicts. So I avoid it wherever possible. You don’t think this kind of compiler enhancement can be developed outside the HEAD stream of the compiler without incurring double maintenance? Yes it can. With OT/J we have the tight integration that is needed for implementing the feature while keeping the sources well separated.

The code for the annotation support even lives in a different source repository, but the runtime effect is the same as if all this already were an integral part of the JDT/Core. Maybe I should say, that for this particular task the integration using OT/J causes a somewhat noticable performance penalty. The compiler does an awful lot of work and hooking into this busy machine comes at a price. So yes, at the end of the day this should be re-integrated into the JDT/Core. But for the time being the OT/J solution well serves its purpose (and in most other situations you won’t even notice any impact on performance plus we already have further performance improvements in the OT/J runtime in our development pipeline).

Independent deployment

Had I created a branch, the only way to get this to you early adopters would have been via a patch feature. I do have some routine in deploying patch features but they have one big drawback: they create a tight dependency to the exact version of the feature which you are patching. That means, if you have the habit of always updating to the latest I-build of Eclipse I would have to provide a new patch feature for each individual I-build released at Eclipse!

Not so for OT/Equinox plug-ins: in this particular case I have a lower bound: the JDT/Core must be from a build ≥ 20110226. Other than that the same OT/J-based plug-in seemlessly integrates with any Eclipse build. You may wonder, how can I be so sure. There could be changes in the JDT/Core that could break the integration. Theoretically: yes. Actually, as a JDT/Core committer I’ll be the first to know about those changes. But most importantly: from many years’ experience of using this technology I know such breakage is very seldom and should a problem occur it can be fixed in the blink of an eye.

As a special treat the OT/J-based plug-in can even be enabled/disabled dynamically at runtime. The OT/Equinox runtime ships with the following introspection view:

Simply unchecking the second item dynamically disables all annotation based null analysis, consistently.

Enjoyable development

The Java compiler is a complex beast. And it’s not exactly small. Over 5 Mbytes of source spread over 323 classes in 13 packages. The central package of these (ast) comprising no less than 109 classes. To add insult to injury: each line of this code could easily get you puzzling for a day or two. It ain’t easy.

If you are a wizard of the patches feel free to look at the latest patch from the bug. Does that look like s.t. you’d like to work on? Not after you’ve seen how nice & clean things can be, I suppose.

First level: overview

Instead of unfolding the package explorer until it shows all relevant classes (at what time the scrollbar will probably be too small to grasp) a quick look into the outline suffices to see everything relevant:

Here we see one top-level class, actually a team class. The class encapsulates a number of role classes containing the actual implementation.

Navigation to the details

Each of those role classes is bound to an existing class of the compiler, like:

   protected class MessageSend playedBy MessageSend { ...

Ctrl-click on the right-hand class name takes you to that base class (the packages containing those base classes are indicated in the above screenshot). This way the team serves as the single point of reference from which each affected location in the base code can be reached with a single mouse click - no matter how widely scattered those locations are.

When drilling down into details a typical roles looks like this:

The 3 top items are  ”callout” method bindings providing access to fields or methods of the base object. The bottom item is a regular method implementing the new analysis for this particular AST node, and the item above it defines a  ”callin” binding which causes the new method to be executed after each execution of the corresponding base method.

Locality of new information flows

Since all these roles define almost normal classes and objects, additional state can easily be introduced as fields in role classes. In fact some relevant information flows of the implementation make use of role fields for passing analysis results directly from one role to another, i.e., the new analysis mostly happens by interaction among the roles of this team.

Selective views, e.g., on inheritance structures

As a final example consider the inheritance hierarchy of class Statement: In the original this is a rather large tree:

Way too large actually to be fully unfolded in a single screenshot. But for the implementation at hand most of these
classes are irrelevant. So at the role layer we’re happy to work with this reduced view:

A tale from the real world

I hope I could give an impression of a real world application of OT/J. I couldn’t think of a nicer structure for a feature of this complexity based on an existing code base of this size and complexity. Its actually fun to work with such powerful concepts.

Did I already say? Don’t miss our EclipseCon tutorial

See you at

As I announced earlier most of my time for the JDT currently goes into static analysis, more specifically into analyzing various issues relating to null references.

Since the compiler can’t read our minds on whether we intend to allow null for a method parameter, a method return etc. we should give it a little help: annotate types in method signatures with either @Nullable or @NonNull, thus creating a contract between a method and its callers.

With such contracts it is possible to write code that is provably free of NPEs — wouldn’t that be great?

Let null contracts become mainstream

The theory and some tools for checking null contracts have been around for a while. I think it is time that the Eclipse Java compiler will make this feature directly available within your favorite IDE.

Towards this goal work in bug 186342 has made good progress, but some issues still need discussing and experimenting before this feature can be released into the JDT.

First prototype available

The current prototype is available for download — see these steps for installing into a fresh Eclipse SDK 3.7 M5.

What the prototype supports:

• @NonNull and @Nullable annotations for method parameters, method return and local variables, seemlessly integrated with the existing flow analysis of the JDT compiler.
• Inheritance of null contracts with the option of compatible refinement in subtypes.
• Configuration of a global default (either @NonNull or @Nullable) to reduce the number or required annotations.
• Configuration of specific qualified names for these annotation types to support compatibility with (some) other tools
• A few quickfixes for adding obviously missing annotations.

If you like to play with these existing features you’re highly welcome to install the plugin and give feedback in the bug.

If you want to see the full story you may want to wait for some more items from my agenda:

• Null annotations for fields
• Nullity profiles for libraries
• Configurability of the default annotation per package and per type.
• More quickfixes

Behind the scenes

There’s a second piece of good news in this development: before this feature gets approved for committing into JDT head, I was faced with the usual choice to either evolve the feature outside CVS or create a branch. Since I didn’t like the efforts incurred by either approach – regarding maintenance as well as deployment – I chose the third option: Object Teams.

I transformed my recent patch from the bug into a regular OT/Equinox plug-in. The Object Teams based solution not only avoids the double maintance associated with branching but also makes the specific code for this feature much easier to read because it is localized in exactly one module. Also, the plug-in can be easier comsumed because it can be packaged and deployed independently of the concurrent evolution of the JDT.

By the way: those interested in these and other cool things that can be done with Object Teams: don’t miss our EclipseCon tutorial:

See you at

Less than a week ago I happily announced that Object Teams is on the Indigo Train.
Much water has gone under the bridge since then and the above statement is history.

Events where triggered by what was actually a little bug in the b3 aggregator. Only by way of this bug some people noticed that there was a ”’patch feature”’ inside the repository, i.e., a feature (”Object Teams Patch for JDT/Core”) that replaces the jdt.core plugin with a variant.

One part of me is very happy this bug occurred because finally an issue got the attention I had tried to raise at various occasions before. The lesson is:

(see this post, e.g.).

The other part of me got very worried because during that debate some harsh statements occurred that would effectively amount to excluding Object Teams from Eclipse.org. That’s a little more attention than I had intended. As in any heated debate some of the arguments sounded to me more like ideology than anything that could possibly be discussed open-mindedly.

I had mixed feelings regarding the technical scope: the outcry only banned one specific technology: patch features. I don’t see how the goal to protect a project’s bits and bytes against influence from other projects can be achieved without also discussing: access to internal, byte-code weaving and - worst of all, I believe - reflection. To be perfectly open: Object Teams uses all these techniques except for reflection. Personally, I would even argue for banning projects that do use setAccessible(true), but that’s not a realistic option because then quite likely the whole Train would dissolve into a mist.

I am actually guilty of a technical simplification during this debate: I focused too much on the idea that a user would explicitly select features to install, not accounting for the possibility that the jdt.core plugin can well be pulled in invisibly due to dependencies among plugins. So, yes, if Object Teams would still be in the repository, and if a user installed a package without the JDT and if that user did never select to install the JDT and if that user selects another feature that implicitly requires the jdt.core plugin, then that user would unexpectly install the OT variant of the jdt.core. I agree that this is not ideal. I personally would have been happy to take this risk because I know how thoroughly the OT variant is tested for compatibility. And for the remaining minuscule risk I would have been happy to promise same-day fixes. But risk assessment naturally depends on perspective and I understand that others come to different conclusions when weighing the issues.

From two days distance I can already laugh at one implication of the central argument, paraphrased as: the JDT/Core team must be protected against harmful actions from the OT team. When spelling this out in names, one of the sentences reads: “Stephan Herrmann must be protected against harmful actions by Stephan Herrmann”. I should really be careful, because I’ll never be able to escape him!

Where to go?

• As we’re banned from the Train, I willed hurriedly book a plane ticket to Indigo. Make sure we come by a Graduation office on the way.
• I do hope that bug 316702 get’s sufficient attention now. Seriously: if you are so detrimentally determined about patch features, then the UI must report it. And if it reports this, it might as well report other techniques that have similar effects!
• I appreciate any offers for helping OT/J towards a solution that avoids replacing a plugin. As of today and after more than seven years of looking at this, I see no way how this can be done, but that doesn’t mean we shouldn’t try still harder.

Unfortunately, the debate consumed all the time and energy I had planned for preparing a presentation at the EclipseCON Audition. However, Lynn finally made my day by letting me know that my submission is the lucky #42. So I’m making progress towards my all-seasons Eclipse collection

I’m flattered to see my name appear on one more Eclipse web-page:

Thanks to the JDT/Core team for accepting me as a new member!

Actually, I’ve been poking about the JDT source code since 2003 when we started implementing the Object Teams Development Tooling (OTDT) on top of the JDT. So it was only natural that I would stumble upon a bug in the JDT every once in a while (incl. one “greatbug” ). In this specific situation, reporting bugs and trying to help find solutions was more than just good Eclipse citizenship: the discussions in bugzilla also helped me to better understand the JDT sources and thus it helped me developing the OTDT. And that’s what this post is about: synergy!

Personal Goals

Actually, watching the JDT/Core bug inbox provides a constant stream of cool riddles: spooky Java examples producing unexpected compiler/runtime results. Sometimes those are real fun to crack. Which compiler is right, which one isn’t? What’s the programmer trying to do? I’d say there’s no better way to test your Java knowledge

Also, watching this stream helps me stay informed, because every patch will eventually need to be merged into the Object Teams branch of the JDT/Core.

But I also have some pet projects that I’d like to actively push forward. Currently I’m thinking about these two:

• Supporting inter-procedural null analysis
• Making the compiler closer match the OSGi semantics

Inter-procedural null analysis

Naturally, the JDT/Core is a facilitator for tons of cool functionality in the IDE, but it hardly communicates directly with the user. But we have one channel, where the compiler can actually talk to the developer and give advice that’s well worth your bucks: errors and warnings. Did you know, that the compiler can immediately tell you what’s wrong about this piece of code:

void foo(String in) {
   if (in == null) new IllegalArgumentException("Null is not allowed here");
   ... // real work done here
}

(Yep, that was bug 236385, see the New&Noteworthy on how to enable).

Many Java developers share the experience that one of the most frequent problems is also one of the most mundane: NPE. Those, who are aware of this easily produce the opposite problem: cluttering there code with meaningless null-checks (many of which have no better action than the above IllegalArgumentException which isn’t much better than throwing the NPE in the first place). The point is, we’d want to know which values can actually be null (and why!) so we write only the necessary null-checks and for those we should think really hard what would be a suitable action, right? (“This shouldn’t happen” is not an answer!).

Thus I’m happy I could get involved in some improvements of the compiler’s flow analysis. I also experienced that null-analysis combined with aggressive optimization is a delicate issue - remember SDK 3.7M2a? Sorry about that, but to my justification I might add that bug 325755 has always been dormantly present, I only woke’em up. I must admit for those 105 minutes my heart beat was a bit above average .

Anyway, the current null analysis is still fundamentally limited: it knows nothing about nullness of method arguments nor method call results. So it would be a real cool enhancement if we could feed such information into the compiler. Fortunately, the theory of how to do this (you may either think of improved type systems or of design by contract) is a well-explored subject. Actually, bug 186342 already has some patches in this direction. Since discussing in bugs with long history is a bit tedious I created this wiki page. Sadly, it’s not a technical difficulty that’s keeping us from immediately releasing that patch, but a stalled standardization process .

However, as outlined in the wiki, here comes another synergy: in case we can’t find a solution that will be sufficiently compatible with future standards, I can easily create an early-adopters release, so sorting out the details of implementation and usage can be done in parallel with the standardization process. How that? I’d simply ship the compiler enhancement as an OT/Equinox-enabled plug-in. This would allow us to ship the compiler enhancement as separate plug-in ready to be tried by early adopters.

OSGi-aware compiler

Some of you may have noticed (here, or here, or here, or here or one of the many duplicates) that the compiler’s concept of a (single, linear) classpath is not a good match for compiling OSGi code. I will soon write another wiki page with my current thinkings about that issue. Stay tuned …

Comments?

For both issues I’d appreciate any comments / feedback. I’ll be checking updates of the bugs I listed, or the wiki page or … talk to me during ESE! Let’s move something together.

See you all in Ludwigsburg!
Stephan

Ralf Ebert recently blogged about how he extended Java to support a short-hand notation for throwing exceptions, like:

   throw "this is wrong";

It’s exactly the kind of enhancement you’d expect from Project Coin, but neither do they have it, nor would you want to wait until they release a solution.

At this point I gave it a few minutes, adapted Ralf’s code, applied Olivier’s suggestion wrapped it in a little plugin et voilà:

Install

Use this p2 repository, check two features…

…install and restart, and you’re ready to use your “Medal” IDE:

So that’s basically the same as what Ralf already showed except:

It’s a module!

In contrast to Ralf’s patch of the JDT/Core my little plugin can be easily deployed and installed into any Eclipse (≥3.6.0). It just requires another small feature called “Object Teams Equinox Integration” or “OT/Equinox” for short.

So we’re all going to use our private own ”’dialects of Java?”’ Hm, firstly, once compiled this is of course plain Java, you wouldn’t be able to tell that the sources looked “funny”.

And: here’s the Boss Key: when somebody sniffs about your monitor, a single click will make Eclipse behave “normal”:

In other words, you can ”’dynamically enable/disable”’ this feature at runtime. The OT/Equinox Monitor view in the snapshot shows all known Team instances in the currently running IDE, and the little check boxes simply send activate() / deactivate() messages to the selected instance.

I coined the name Medal as our own playground for Java extensions of this kind. Feel free to suggest/contribute more!

Implementation

For a quick introduction on how to setup an OT/Equinox project in Eclipse I’d suggest our Quick Start (let me know if anything is unclear). For this particular case the key is in defining one little extension:

which the package explorer will render as:

Drilling into the Team class ThrowString you’ll see:

The Team class contains two Role classes:

• Role DontReport binds to class ProblemReporter (not shown in the Outline), intercepts calls to ProblemReporter.cannotThrowType and if the type in question is String, simply ignores the “error”
• Role Generate binds to class ThrowStatement to make sure the correct bytecodes for creating a RuntimeException are generated

Also, in the Outline you see both kinds of method bindings that are supported by Object Teams:

• getExceptionType/setExceptionType are getter/setter definitions for field ThrowStatement.exceptionType (callout-to-field in OT/J jargon)
• Things like “adjustType <- after resolve” establish method call interception (callin bindings in OT/J jargon - the “after” is symbolized by the specific icon)

The actual implementation is really simple, like (full listing of the first role):

protected class DontReport playedBy ProblemReporter {
	cannotThrowType <- replace cannotThrowType;
 
	@SuppressWarnings("basecall")
	callin void cannotThrowType(ASTNode exception, TypeBinding exceptionType) {
		if (exceptionType.id != TypeIds.T_JavaLangString)
			// do the actual reporting only if it's not a string
			base.cannotThrowType(exception, exceptionType);
	}		
}

The base-call (base.cannotThrowType) delegates back to the original method, but only if the exception type is not String. The @SuppressWarnings annotation documents that not all control flows through this method will issue a base-call, a decision that deserves a second thought as it means the base plugin (here JDT/Core) does not perform its task fully as usual.

Intercepting resolve has the purpose of replacing type String with RuntimeException so that other parts of the Compiler and the IDE see a well-typed structure.

The method that performs the actual work is generateCode. Since this method is essentially based on the original implementation, the best way to see the difference is (select either the callin method or the callin binding):

which gives you this compare editor:

This neatly shows the two code blocks I inserted, one for creating the RuntimeException instance, the other for invoking its constructor. Or, if you just want to read the full role method:

/* This method is partly copied from the base method. */
@SuppressWarnings({"basecall", "inferredcallout"})
callin void generateCode(BlockScope currentScope, CodeStream codeStream) {
	if ((this.bits & ASTNode.IsReachable) == 0)
		return;
	int pc = codeStream.position;
	// create a new RuntimeException:
	ReferenceBinding runtimeExceptionBinding = (ReferenceBinding) this.exceptionType;
	codeStream.new_(runtimeExceptionBinding);
	codeStream.dup();
	// generate the code for the original String expression:
	this.exception.generateCode(currentScope, codeStream, true);
	// call the constructor RuntimeException(String):
	MethodBinding ctor = runtimeExceptionBinding.getExactConstructor(new TypeBinding[]{this.stringType});
	codeStream.invoke(Opcodes.OPC_invokespecial, ctor, runtimeExceptionBinding);
	// throw it:
	codeStream.athrow();
	codeStream.recordPositionsFrom(pc, this.sourceStart);			
}

You may also fetch the full sources of this little plug-in (plus a feature for easy deployment) to play around with and extend.

Next?

Ralf mentioned that he’d like to play with ways for also extending the syntax. For a starter on how this can be done with Object Teams I recommend my previous posts IDE for your own language embedded in Java? (part 1) and part 2.

During the last week or so I modernized a part of the Object Teams Development Tooling (OTDT) that had been developed some 5 years ago: the type hierarchy for OT/J. I’ll mention the basic requirements for this engine in a minute. While most of the OTDT succeeds in reusing functionality from the JDT, the type hierarchy was implemented as a full replacement of the original. This is a pretty involved little machine, which took weeks and months to get right. It provides its logic to components like Refactoring and the Type Hierarchy View.

On the one hand this engine worked well for most uses, but over so many years we did not succeed to solve two remaining issues:

Give a faithful implementation for getSuperclass()

This is tricky because a role class in OT/J can have more than one superclass. Failing to implement this method we could not support the “traditional” mode of the hierarchy view that shows both the tree of subclasses of a focus type plus the path of superclasses up to Object (this upwards path relies on getSuperclass).

Support region based hierarchies

Here the type hierarchy is not only computed for supertypes and subtypes of one given focus type, but full inheritance structure is computed for a set of types (a “region”). This strategy is used by many JDT Refactorings, and thus we could not precisely adapt some of these for OT/J.

In analyzing this situation I had to weigh these issues:

• In its current state the implementation strategy was a show stopper for one mode of the type hierarchy view and for precise analysis in several refactorings.
• Adding a region based variant of our hierarchy implementation would mean to re-invent lots of stuff, both from the JDT and from our own development.
• All this seemed to suggest to discard our own implementation and start over from scratch.

Object Teams to the rescue: Let’s re-build Rome in ten days.

As mentioned in my previous post, the strength of Object Teams lies in building layers: each module sits in one layer, and integration between layers is given by declarative bindings:

Applying this to the issue at hand we now actually have three layers with quite different structures:

Java Model

The bottom layer is the Java model that implements the containment tree of Jave elements: A project contains source folders, containing packages, containing compilation units, containing types containing members. In this model each Java type is represented by an instance of IType

Java Type Hierarchy

This engine from the JDT maintains the graph of inheritance information as a second way for navigating between ITypes. Interestingly, this module pretty closely simulates what Object Teams does natively, I may come back to that in a later post.

Object Teams Type Hierarchy

As an extension of Java, OT/J naturally supports the normal inheritance using extends, but there is a second way how an inheritance link can be established: based on inheritance of the enclosing team:

team class EcoSystem {
   protected class Project { }
   protected class IDEProject extends Project { }
}
team class Eclipse extends EcoSystem {
   @Override
   protected class Project { }
   @Override
   protected class IDEProject extends Project { }
}

Here, Eclipse.Project is an implicit subclass of EcoSystem.Project simply because Eclipse is a subclass of EcoSystem and both classes have the same simple name Project. I will not go into motivation and consequences of this language design (that’ll be a separate post — which I actually promised many weeks ago).

Looking at the technical challenge we see that the implicit inheritance in OT/J adds a third layer, in which classes are connected in yet another graph.

Three Layers — Three Graphs

Looking at the IType representation of Eclipse.IDEProject we can ask three questions:

Each question is implemented in a different layer of the system. Things get a little complicated when asking a type for all its super types, which requires to collect the answers from both the JDT hierarchy layer and the OT hierarchy. Yet, the most tricky part was giving an implementation for getSuperclass().

An "Impossible" Requirement

There is a hidden assumption behind method getSuperclass() which is pervasive in large parts of the implementation, especially most refactorings:

When searching all methods that a type inherits from other types, looping over getSuperclass() until you reach Object will bring you to all the classes you need to consider, like so:

IType currentType = /* some init */;
while (currentType != null) {
   findMethods(currentType, /*some more arguments*/);
   currentType = hierarchy.getSuperclass(currentType);
}

There are lots and lots of places implemented using this pattern. But, how do you do that if a class has multiple superclasses?? I cannot change all the existing code to use recursive functions rather than this single loop!

Looking at Eclipse.IDEProject we have two direct superclasses: Eclipse.Project (normal inheritance, “extends”) and EcoSystem.IDEProject (OT/J implicit inheritance), which cannot both be answered by a single call to getSuperclass(). The programming language theory behind OT/J, however, has a simple answer: linearization. Thus, the superclasses of Eclipse.IDEProject are:

• Eclipse.IDEProject → EcoSystem.IDEProject → Eclipse.Project → EcoSystem.Project

… in this order. And this is how this shall be rendered in the hierarchy view:

The final callenge: what should this query answer:

        getSuperclass(ecoSystemIDEProject);

According to the above linearization we should answer: Eclipse.Project, but only if we are in the context of the superclass chain of Eclipse.IDEProject. Talking directly to EcoSystem.IDEProject we should get EcoSystem.Project! In other words: the function needs to be smarter than what it can derive from its arguments.

Layer Instances for each Situation

Let’s go back to the layer thing:

At the bottom you see the Java model (as rendered by the package explorer). In the top layer you see the OT/J type hierarchy (lets forget about the middle layer for now). Two essential concepts can be illustrated by this picture:

• Each layer is populated with objects and while each layer owns its objects, those objects connected with a red line between layers are almost the same, they represent the same concept.
• The top layer can be instantiated multiple times: for each focus type you create a new OT/J hierarchy instance, populated with a fresh set of objects.

It is the second bullet that resolves the “impossible” requirement: the objects within each layer instance are wired differently, implementing different traversals. Depending on the focus type, each layer may answer the getSuperclass(type) question differently, even for the same argument.

The first bullet answers how these layers are integrated into a system: Conceptually we are speaking about the same Java model elements (IType), but we superimpose different graph structure depending on our current context.

Inside the hierarchy layer, we actually do not handle IType instances directly, but we have roles that represent one given IType each. Those roles contain all the inheritance links needed for answering the various questions about inheritance relations (direct/indirect, explicit/implicit, super/sub).

A cool thing about Object Teams is, that having different sets of objects in different layers ( teams) doesn’t make the program more complex, because I can pass an object from one layer into methods of another layer and the language will quite automagically translate into the object that sits at the other end of that red line in the picture above. Although each layer has its own view, they “know” that they are basically talking about the same stuff (sounds like real life, doesn’t it?).

Summing up

OK, I haven’t shown any code of the new hierarchy implementation (yet), but here’s a sketch of before-vs.-after:

Code Size

The new implementation of the hierarchy engine has about half the size of the previous implementation (because it need not repeat anything that’s already implemented in the Java hierarchy).

Integration

The previous implementation had to be individually integrated into each client module that normally uses Java hierarchies and then should use an OT hierarchy instead. After the re-implementation, the OT hierarchy is transparently integrated such that no clients need to be adapted (accounting for even more code that could be discarded).

Linearization

Using the new implementation, getSuperclass() answers the correct, context sensitive linearization, as shown in the screenshot above, which the old implementation failed to solve.

Region based hierarchies

The old implementation was incompatible with building a hierarchy for a region. For the new implementation it doesn’t matter whether it’s built for a single focus type or for a region, so, many clients now work better without any additional efforts.

The previous implementation only scratched at the surface – literally worked around the actual issue (which is: the Java type hierarchy is not aware of OT/J implicit inheritance). The new solution solves the issue right at its core: the new team OTTypeHierarchies assists the original type hierarchy (such that its answers indeed respect OT/J’s implicit inheritance). By performing this adaptation at the issue’s core, the solution automatically radiates to all clients. So I expect that investing a few days in re-writing the implementation will pay off in no time. Especially, improving the (already strong) refactoring support for OT/J is now much, much easier.

Need I say, how much fun this re-write was?