The Object Teams Blog

During the Juno cycle a lot of work in the JDT has gone into more sophisticated static analysis, and some more is still in the pipe-line. I truly hope that once Juno is shipped this will help all JDT users to find more bugs immediately while still typing. However, early feedback regarding these features shows that users are starting to expect miracles from the analysis

On the one hand seeing this is flattering, but on the other hand it makes me think we should perhaps explain what exactly the analysis can see and what is beyond its vision. If you take a few minutes learning about the concepts behind the analysis you’ll not only understand its limitations, but more importantly you will learn how to write code that’s better readable - in this case for reading by the compiler. Saying: with only slightly rephrasing your programs you can help the compiler to better understand what’s going on, to the effect that the compiler can answer with much more useful error and warning messages.

Since there’s a lot of analysis in this JDT compiler I will address just one topic per blog post. This post goes to improvements in the detection of resource leaks.

Resource leaks - the basics

Right when everybody believed that Eclipse Indigo RC 4 was ready for the great release, another blocker bug was detected: a simple resource leak basically prevented Eclipse from launching on a typical Linux box if more than 1000 bundles are installed. Coincidentally, at the same time the JDT team was finishing up work on the new try-with-resources statement introduced in Java 7. So I was thinking: shouldn’t the compiler help users to migrate from notoriously brittle handling of resources to the new construct that was designed specifically to facilitate a safe style of working with resources?

What’s a resource?

So, how can the compiler know about resources? Following the try-with-resources concept, any instance of type java.lang.AutoCloseable is a resource. Simple, huh? In order to extend the analysis also to pre Java 7 code, we also consider java.io.Closeable (available since 1.5).

Resource life cycle

The expected life cycle of any resource is : allocate—use—close. Simple again.

From this we conclude the code pattern we have to look for: where does the code allocate a closeable and no call to close() is seen afterwards. Or perhaps a call is seen but not all execution paths will reach that call, etc.

Basic warnings

With Juno M3 we released a first analysis that could now tell you things like:

• Resource leak: “input” is never closed
• Resource leak: “input” is never closed at this location (if a method exit happens before reaching close())

If the problem occurs only on some execution paths the warnings are softened (saying “potential leak” etc.).

Good, but…

Signal to noise - part 1

It turned out that the analysis was causing significant noise. How come? The concepts are so clear and all code that wouldn’t exhibit the simple allocate—use—close life cycle should indeed by revised, shouldn’t it?

In fact we found several patterns, where these warnings were indeed useless.

Resource-less resources

We learned that not every subtype of Closeable really represents a resource that needs leak prevention. How many times have you invoked close() on a StringWriter, e.g.? Just have a look at its implementation and you’ll see why this isn’t worth the effort. Are there more classes in this category?

Indeed we found a total of 7 classes in java.io that purely operate on Java objects without allocating any resources from the operating system:

• StringReader
• StringWriter
• ByteArrayInputStream
• ByteArrayOutputStream
• CharArrayReader
• CharArrayWriter
• StringBufferInputStream

For none of these does it make sense to warn about missing close().

To account for these classes we simply added a white list: if a class is in the list suppress any warnings/errors. This white list consists of exactly those 7 classes listed above. Sub-classes of these classes are not considered.

Wrapper resources

Another group of classes implementing Closeable showed up, that are not strictly resources themselves. Think of BufferedInputStream! Does it need to be closed?

Well? What’s your answer? The correct answer is: it depends. A few examples:

1
2
3
4
5
6
7
8

	void wrappers(String content) throws IOException {
		Reader r1, r2, r3, r4;
		r1 = new BufferedReader(new FileReader("someFile"));
		r2 = new BufferedReader(new StringReader(content));
		r3 = new FileReader("somefile");
		r4 = new BufferedReader(r3);
		r3.close();
	}

How many leaks? With same added smartness the compiler will signal only one resource leak: on r1. All others are safe:

• r2 is a wrapper for a resource-less closeable: no OS resources are ever allocated here.
• r3 is explicitly closed
• r4 is just a wrapper around r3 and since that is properly closed, r4 does not hold onto any OS resources at the end.
• returning to r1, why is that a leak? It’s a wrapper, too, but now the underlying resource (a FileReader) is not directly closed so it’s the responsibility of the wrapper and can only be triggered by calling close() on the wrapper r1.

Summarizing: wrappers don’t directly hold an OS resource, but delegate to a next closeable. Depending on the nature and state of the nested closeable the wrapper may or may not be responsible for closing. In arbitrary chains of wrappers with a relevant resource at the bottom, closing any closeable in the chain (including the bottom) will suffice to release the single resource. If a wrapper chain is not properly closed the problem will be flagged against the outer-most wrapper, since calling close() at the wrapper will be delegated along all elements of the chain, which is the cleanest way of closing.

Also for wrappers the question arises: how does the compiler know? Again we set up a white list with all wrapper classes we found in the JRE: 20 classes in java.io, 12 in java.util.zip and 5 in other packages (the full lists are in TypeConstants.java, search for “_CLOSEABLES”).

Status and outlook

Yes, a leak can be a stop-ship problem.

Starting with Juno M3 we have basic analysis of resource leaks; starting with Juno M5 the analysis uses the two white lists mentioned above: resource-less closeables and resource wrappers. In real code this significantly reduces the number of false positives, which means: for the remaining warnings the signal-to-noise ratio is significantly better.

M5 will actually bring more improvements in this analysis, but that will be subject of a next post.

My latest query in Eclipse’s bugzilla answered 189 bugs.

Out of these a certain TLA was mentioned just in the bug summaries 54 times.
Additionally, the long form of the same word occurred 5 times.

Those who have followed my previous posts, know which three letters I’m referring to. Looks like they’re even more relevant in some Eclipse components than I ever fancied (no - not telling, in which component I searched ).

The recent release of Juno M4 brought an interesting combination: The Object Teams Development Tooling now natively supports annotation-based null analysis for Object Teams (OT/J). How about that?

The path behind us

Annotation-based null analysis has been added to Eclipse in several stages:

Using OT/J for prototyping

As discussed in this post, OT/J excelled once more in a complex development challenge: it solved the conflict between extremely tight integration and separate development without double maintenance. That part was real fun.

Applying the prototype to numerous platforms

Next I reported that only one binary deployment of the OT/J-based prototype sufficed to upgrade any of 12 different versions of the JDT to support null annotations — looks like a cool product line

Pushing the prototype into the JDT/Core

Next all of the JDT team (Core and UI) invested efforts to make the new feature an integral part of the JDT. Thanks to all for this great collaboration!

Merging the changes into the OTDT

Now, that the new stuff was mixed back into the plain-Java implementation of the JDT, it was no longer applicable to other variants, but the routine merge between JDT/Core HEAD and Object Teams automatically brought it back for us. With the OTDT 2.1 M4, annotation-based null analysis is integral part of the OTDT.

Where we are now

Regarding the JDT, others like Andrey, Deepak and Aysush have beaten me in blogging about the new coolness. It seems the feature even made it to become a top mention of the Eclipse SDK Juno M4. Thanks for spreading the word!

Ah, and thanks to FOSSLC you can now watch my ECE 2011 presentation on this topic.

Two problems of OOP, and their solutions

Now, OT/J with null annotations is indeed an interesting mix, because it solves two inherent problems of object-oriented programming, which couldn’t differ more:

1.: NullPointerException is the most widespread and most embarrassing bug that we produce day after day, again and again. Pushing support for null annotations into the JDT has one major motivation: if you use the JDT but don’t use null annotations you’ll no longer have an excuse. For no good reasons your code will retain these miserable properties:

• It will throw those embarrassing NPEs.
• It doesn’t tell the reader about fundamental design decisions: which part of the code is responsible for handling which potential problems?

Why is this problem inherent to OOP? The dangerous operator that causes the exception is this:

2.: Objectivity seems to be a central property on any approach that is based just on Objects. While so many other activities in software engineering are based on the insight that complex problems with many stakeholders involved can best be addressed using perspectives and views etc., OOP forces you to abandon all that: an object is an object is an object. Think of a very simple object: a File. Some part of the application will be interested in the content so it can decode the bytes and do s.t. meaningful with it, another part of the application (maybe an underlying framework) will mainly be interested in the path in the filesystem and how it can be protected against concurrent writing, still other parts don’t care about either but only let you send the thing over the net. By representing the “File” as an object, that object must have all properties that are relevant to any part of the application. It must be openable, lockable and sendable and whatnot. This yields bloated objects and unnecessary, sometimes daunting dependencies. Inside the object all those different use cases it is involved in can not be separated!

With roles objectivity is replaced by a disciplined form of subjectivity: each part of the application will see the object with exactly those properties it needs, mediated by a specific role. New parts can add new properties to existing objects — but not in the unsafe style of dynamic languages, but strictly typed and checked. What does it mean for practical design challenges? E.g, direct support for feature oriented designs - the direct path to painless product lines etc.

Just like the dot, objectivity seems to be hardcoded into OOP. While null annotations make the dot safe(r), the roles and teams of OT/J add a new dimension to OOP where perspectives can be used directly in the implementation. Maybe it does make sense, to have both capabilities in one language although one of them cleans up what should have been sorted out many decades ago while the other opens new doors towards the future of sustainable software designs.

The road ahead

The work on null annotations goes on. What we have in M4 is usable and I can only encourage adopters to start using it right now, but we still have an ambitious goal: eventually, the null analysis shall not only find some NPEs in your program, but eventually the absense of null related errors and warnings shall give the developer the guarantee that this piece of code will never throw NPE at runtime.

What’s missing towards that goal:

1. Fields: we don’t yet support null annotations for fields. This is next on our plan, but one particular issue will require experimentation and feedback: how do we handle the initialization phase of an object, where fields start as being null? More on that soon.
2. Libraries: we want to support null specifications for libraries that have no null annotations in their source code.
3. JSR 308: only with JSR 308 will we be able to annotate all occurrences of types, like, e.g., the element type of a collection (think of List<@NonNull String>)

Please stay tuned as the feature evolves. Feedback including bug reports is very welcome!

Ah, and one more thing in the future: I finally have the opportunity to work out a cool tutorial with a fellow JDT committer: How To Train the JDT Dragon with Ayushman. Hope to see y’all in Reston!

Recent stress tests have revealed a few issues in the Eclipse platform where leaking file handles could cause “Too many open files”-style failures.

Until now that kind of issue was notoriously difficult to analyze, and a lot of luck was involved when I spotted one of those bugs by mere code inspection.

However, chances are that this particular kind of bug will soon be a curiosity from the past. How? Java 7 brings a new feature that should be used in exactly this kind of situation: by the new try-with-resources construct leaking of any resources can be safely avoided.

Now, current code certainly doesn’t use try-with-resources, how can we efficiently migrate existing code to using this new feature? As one step I just filed this RFE for a new warning where try-with-resources should probably be used but is not. At this point I can’t promise that we’ll be able to implement the new warning in a useful way, i.e., without too many false positives, but it sounds like a promising task, doesn’t it?

Here’s a mockup of what I’m thinking of:

And this is what the quickfix would create:

Sounds cool?

OK, I use Eclipse for getting some work done. Eclipse is software so we know it contains bugs. Given that Eclipse is open source software, we all can only expect it to run smoothly if we diligently report back all errors we encounter (and provide steps on how to reproduce etc.). I know all that and I really want to be a good bug reporter because I really want a good experience using Eclipse because I really want to get the work done (and I may even want to sustain the impression that I’ve got everything under control and thus working with Eclipse is a delight).

A task

The other day, I was a very bad bug reporter, and only today I find some time to reason about what happened. This was my task: In preparing the initial contribution for the Object Teams Project I had to rename a bunch of things from org.objectteams to org.eclipse.objectteams. Simply, huh? Back in the days of emacs+bash I might have chosen to just use one big

find . -exec /bin/sed -i -e "s/org.objectteams/org.eclipse.objectteams/g" {} \;

and voila, if fortuna was with me, I might have been done at the return of that single command. But things were actually just a little bit more challenging, like, a few occurrences would have to remain unchanged plus while touching about every single file in our software I was going to also do some clean-up: rename some packages to foo.internal.bar, fixing some copyright headers, etc. Also I preferred to change one plug-in at a time which would mean that all references to plug-ins not yet processed should stay unchanged, too. Plus a few more little deviations from the grand search-and-replace-globally.

OK, since I like to see myself an Eclipse-wizard this was a nice challenge for its refactoring support. Plug-in by plug-in I renamed, I renamed packages with / or without subpackages, and after each step I wanted to see that compiler and PDE agree with all changes and signal that everything is still (or again) consistent, ready to be built, actually. Perhaps things started the get wrong when I estimated the effort as one or two hours. So, after a day or so, I wasn’t perfectly relaxed any more. My fault, should’ve known better about that estimate. BTW, one of the reasons it took so long was simply the size of my workspace in comparison to the power of my computer / hard-disk: every time I performed a rename with updates in non-Java files, I was nervously looking at the screen: “should I sit and wait for the preview page, or should I go to the kitchen, get a chocolate, coffee, just something?“. I did some emailing in parallel, but let’s just keep this: due to those response times I wasn’t perfectly relaxed any more.

A story of bug reporting

What I was going to tell here is a story of bug reporting, because as a safe bet doing a real-life stress test to an Eclipse component should give you a good chance to discover and report a few bugs that have not yet been reported by others. And indeed, I was successful in discovering some bugs, in various components actually.

I think one of the first things that occurred was that the svn synchronize view would sometimes fail to open the compare editor, more precisely, I had to explicitly close the compare editor before comparing the next file. At first this really **** me off, because the error dialog was popping up in some kind of infinite loop. Fun!#$ Once I’d figure out how to work around this problem it soon became a habit to just close the compare editor before clicking the next. Next, the svn plugin made a refactoring fail, because it was trying to create a directory which the previous refactoring had already created. The most creative bug having to do with subversive was a chain-reaction of first failing to undo a refactoring and than during reporting this error blocking the UI of Eclipse so I could only kill the Eclipse process, insert a new coin and start a new game.

I don’t intend to blame a particular component. For clean-up of license headers I have a little home-grown plugin that I just wanted to quickly install into the running Eclipse instance, so I went for the cool new feature to export/install into the host. Oops, my plugin depends on another plugin that only exists in the workspace but not in the host, install failed for good reasons. I removed the dependency and tried again. Installation still failed for the same reason: the ghost of this removed dependency prevented installation into the host Eclipse. Oh, I should have incremented the version or let a version qualifier do this automatically, of course. Tried again, still failed. Tried something so slightly different I cannot recall, from there on it worked. Can I reproduce the two or three different levels of failure? I didn’t even take the time to think of it. Well I would’ve been disappointed without a bug from p2 in this list .

PDE did its share by reporting overlapping text edits in plugin.xml and therefore disabling its refactoring participant. What the **** caused those overlapping text edits, and how do I re-enable the refactoring participant to give it one more chance to behave well?

The list could go on if only I could remember. Instead I was happy to finish this 1.5 hours task after 2.7 days, ready to submit our initial code contribution, wow!

Looking back, I / we missed a great opportunity: we could have identified plenty of bugs in various components of Eclipse. With only a few more days of debugging I might have been able to present reproducing steps for all those bugs. And, if triaged and fixed by the corresponding devs, this might have contributed to M6 containing fewer of those bugs that just only occur in real world, never during testing. I failed, I managed only to submit two bug reports, with very little information on how to reproduce.

Lesson learned

Susan McCourt responded to an earlier bug report of mine in a very descriptive comment:

That is one of those things I’ve been meaning to fix forever, never wrote a
bug, and so keep forgetting to fix. And it seems like if I’m actually
[doing what triggers the bug], it’s because something is wrong, and so I again postpone
writing a bug.

Sure, when we hit a bug (or a bug hits us) we are always in some context of doing something challenging. Something that requires our mind to stay in focus. Something we want to get done.
Well, work isn’t perfectly linear, so we know how to react to interrupts. Bugs are such interrupts. Sometimes I like the challenge of isolating a bug etc. Sometimes I’m sufficiently relaxed when the bug occurs so I actually take the challenge. Sometimes the bug is sufficiently good-natured so making a small note and going back to the bug after the actual work is done is a perfect strategy. Some bugs, however, smell like depending on so many factors from your current context that reproduction an hour later seems extremely unlikely.

I think I have a solution to all this: given we don’t want to be distracted from our actual work, given also that some bugs need immediate care or they will escape our attempts to identify. Given some of the worst moments are when we start to isolate a bug and during that task a second bug stops us from working on the first bug etc. The only way to relentlessly follow all those tasks is to enable branching universes in your working environment. The frequent use of the work “task” may give a hint that I should finally start using Mylyn (I have no excuse for not doing so), but I would need a Mylyn that is able to capture full contexts: the complete state of my machine plus the full state of my brain. As a start I’m dreaming of always working in a virtual machine, and whenever something strange happens, I quickly (!!) create a snapshot of the virtual machine. Then I could first isolate (and fix ) the bug that just occurred, and then go back to the exact point where I interrupted my work and act as if nothing had happened. Branching universes with the ability of back porting fixes between branches is what I need. Of course the clock needs to be reset when returning from a bug reporting / fixing branch.

Yeah, that’s why I can’t quite live up to my dreams of good participation in open source development: I haven’t figured out how to enable branching universes for my character. If anybody has a hint on how to do this or any other strategy to not get overwhelmed between work and bug reporting, I’d really appreciate.

And if I don’t smile while writing a bug report, please excuse, I might just be terribly stressed because your bug interrupted my work on isolating another bug that stopped me from doing …

”Always look at the bright side of life…”