The Object Teams Blog

In a previous post I showed how the tycho-compiler-jdt Maven plug-in can be used for compiling OT/J code with Maven.

Recently, I was asked how the same can be done for OT/Equinox projects. Given that we were already using parts from Tycho, this shouldn’t be so difficult, right?

Once you know the solution, finding the solution is indeed easy, also in this case. Here it is:

We use almost the same declaration as for plain OT/J applications:

    <pluginManagement>
	<plugin>
	    <groupId>org.eclipse.tycho</groupId>
	    <artifactId>tycho-compiler-plugin</artifactId>
	    <version>${tycho.version}</version>
	    <extensions>true</extensions>
	    <dependencies>
		<dependency>
		    <groupId>org.eclipse.tycho</groupId>
		    <artifactId>tycho-compiler-jdt</artifactId>
		    <version>${tycho.version}</version>
		    <exclusions>
			<!-- Exclude the original JDT/Core to be replaced by the OT/J variant: -->
			<exclusion>
			    <groupId>org.eclipse.tycho</groupId>
			    <artifactId>org.eclipse.jdt.core</artifactId>
			</exclusion>
		    </exclusions>
		</dependency>
		<dependency>
		    <!-- plug the OT/J compiler into the tycho-compiler-jdt plug-in: -->
		    <groupId>org.eclipse</groupId>
		    <artifactId>objectteams-otj-compiler</artifactId>
		    <version>${otj.version}</version>
		</dependency>
	    </dependencies>
	</plugin>
    </pluginManagement>

So, what’s the difference? In both cases we need to adapt the tycho-compiler-jdt plug-in because that’s where we replace the normal JDT compiler with the OT/J variant. However, for plain OT/J applications tycho-compiler-jdt is pulled in as a dependency of maven-compiler-plugin and must be adapted on this path of dependencies, whereas in Tycho projects tycho-compiler-jdt is pulled in from tycho-compiler-plugin. Apparently, the exclusion mechanism is sensitive to how exactly a plug-in is pulled into the build. Interesting.

Once I figured this out, I created and published a new version of our Maven support for Object Teams: objectteams-parent-pom:2.1.1 — publishing Maven support for Object Teams 2.1.1 was overdue anyway

With the updated parent pom, a full OT/Equinox hello world pom now looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <!-- We use Object Teams: -->
        <groupId>org.eclipse</groupId>
        <artifactId>objectteams-parent-pom</artifactId>
        <version>2.1.1</version>
    </parent>
 
    <artifactId>OTEquinox-over-tycho</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <!-- We are building an Eclipse plug-in (or OSGi bundle): -->
    <packaging>eclipse-plugin</packaging>
 
    <repositories>
 
        <!-- This is where we get all Object Teams stuff from: -->
        <repository>
            <id>ObjectTeamsRepository</id>
            <name>Object Teams Repository</name>
            <url>http://download.eclipse.org/objectteams/maven/3/repository</url>
        </repository>
 
        <!-- Add any p2 repositories needed for your application, e.g.: -->
        <repository>
            <id>Juno</id>
            <name>Eclipse Juno Repository</name>
            <url>http://download.eclipse.org/releases/juno</url>
            <layout>p2</layout>
        </repository>
    </repositories>
 
    <build>
        <plugins>
            <plugin>
                <!-- We build this project using Tycho: -->
                <groupId>org.eclipse.tycho</groupId>
                <artifactId>tycho-maven-plugin</artifactId>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </build>
</project>

Looks pretty straight forward, right?

To see the full OT/Equinox Hello World Example configured for Maven/Tycho simply import OTEquiTycho.zip as a project into your workspace.

cheers,
Stephan

Being a part-time release engineer for the Object Teams project I can only agree with every word Kim writes about the job, I wish I could hire her for our project

She writes:

“Nobody in needs to understand how the build works, they just need to push a button. That’s great. Until the day before a release when your build fails with a cryptic message about unresolved dependencies. And you have no idea how to fix it. And neither does anyone else on the team.”

That puts a sad smile on my face and I’d like to add a little quality metric that seems cruel for today’s build systems, but might actually be useful for any software:

One extreme I experienced was in a PDE/Build-ant-build which I had to set to verbose to get any useful answer but then I had to find the relevant error message deeply buried in literally tens of megabytes of log output. Takes ages to browse that log file. Other tools rank towards the other end of the spectrum saying basically “it didn’t work”.

Why is the worst error message relevant? When you hit that worst message it’s close to saying “game over”. Especially when working on a build I’ve come to the point time and again where all my creativity and productivity came to a grinding halt and for days or weeks I simply made zero progress because I had no idea why that system didn’t work and what it expected me to do to fix the thing. Knock-out.

Obviously I hate that state when I make no progress towards my goal. And typically that state is reached by poor communication from some framework back to me.

Real coolness

I know people usually don’t like to work on improving error messages, but please, don’t think good error messages are any bit less cool than running your software on mars. On the one hand we try to build tools that improve developers’ productivity by a few percent and than the tool will give answers that bring that very productivity down to zero. That’s - inconsistent.

I’m tempted to repeat the p2 story here. Many will remember the merciless dump of data from the sat solver that p2 gave in its early days. Some will consider the problem solved by now. Judge for yourself: what’s the worst-case time a regular Eclipse user will need to understand what p2 is telling him/her by one of its error messages.

The intention of this post is not to blame any particular technology. The list would be quite long anyway. It’s about general awareness (big words, sorry ).

Consider the worst case

Again, why worst case? Because the worst case will happen. And it’s enough if it hits you once to easily compensate all the time savings the tool otherwise brought to you.

Communicate!

Framework developers, tool smiths: let your software communicate with the user and let it be especially helpful when the user is in dire need of help.

One small contribution in this field I’d like to share with you: in the OTDT every error/warning raised by the compiler not only tries to precisely describe what’s wrong but it is directly linked to the corresponding paragraph in the language definition that is violated by the current code. At least this should completely explain why the current code is wrong. It’s a small step, but I feel a strong need for linking specific help to every error message.

But first, the software has to anticipate every single error that will occur in order to produce useful messages. That’s the real reason why creating complex software is so challenging. Be it a build system or the “real” software.

Be cool, give superb error messages!

This week is (supposed to be) “quiet week” at Eclipse: Release Candidate 4 is in place and more testing is being done before the final Indigo Release. Time to speak of s.t. that the Object Teams project has neglected a bit in the past: compiling / building with Maven.

Compiling OT/J with Maven

In the days of Maven2, pluging-in a non-standard compiler required fiddling with the nexus compiler plugin, which was buggy and all that. With the recent development around Maven3 and Tycho things became easier. The following pom-snippet is all you need to tell Maven to use the OT/J compiler:

<build>
    <pluginManagement>
	<plugins>
	    <plugin>
		<!-- Use compiler plugin with tycho as the adapter to the OT/J compiler. -->
		<artifactId>maven-compiler-plugin</artifactId>
		<configuration>
		    <source>1.6</source>
		    <target>1.6</target>
		    <compilerId>jdt</compilerId>
		</configuration>
		<dependencies>
		    <!-- This dependency provides the implementation of compiler "jdt": -->
		    <dependency>
			<groupId>org.sonatype.tycho</groupId>
			<artifactId>tycho-compiler-jdt</artifactId>
			<version>${tycho.version}</version>
			<exclusions>
			    <!-- Exclude the original JDT/Core to be replaced by the OT/J variant: -->
			    <exclusion>
				<groupId>org.sonatype.tycho</groupId>
				<artifactId>org.eclipse.jdt.core</artifactId>
			    </exclusion>
			</exclusions>
		    </dependency>
		    <dependency>
			<!-- plug the OT/J compiler into the tycho-compiler-jdt plug-in: -->
			<groupId>org.eclipse</groupId>
			<artifactId>objectteams-otj-compiler</artifactId>
			<version>${otj.version}</version>
		    </dependency>
		</dependencies>
	    </plugin>
	</plugins>
    </pluginManagement>
</build>

This relies on the fact, that the OT/J compiler is compatible with the JDT compiler, good.

To make things go smooth in various build phases a few more bits are required:

• provide the OT/J runtime jar and its dependency (bcel)
• tell surefire about required command line arguments for running OT/J programs

These basics are collected in a parent pom so that all you need are these two snippets:

<repositories>
    <repository>
	<id>ObjectTeamsRepository</id>
	<name>Object Teams Repository</name>
	<url>http://download.eclipse.org/objectteams/maven/3/repository</url>
    </repository>
</repositories>
<parent>
    <groupId>org.eclipse</groupId>
    <artifactId>objectteams-parent-pom</artifactId>
    <version>2.0.0</version>
</parent>

This converts any Java project into an OT/J project, simply enough, huh?

At least compiling and testing works out-of-the box. Let me know what additional sophistication you need and if adjustments to other build phases are needed.

OT/J over m2e

Well, I’m not much of a Maven expert, but I’m an Eclipse enthusiast, so, how can we use the above inside the IDE?

The bad news: the m2e plug-in has hardcoded its support for the compiler plugin to use “javac” as the compiler. Naive attempts to use tycho-compiler-jdt have failed with the infamous "Plugin execution not covered by lifecycle configuration".

The good news: By writing a tiny OT/Equinox plug-in I was able to remedy those issues. If you have the m2e plug-in installed (Indigo version), the normal “Update Project Configuration” action will suffice:

After that, the project is recognized as an OT/J project, both by the IDE …

… and also by maven …

Where to get it?

The mentioned plug-in is brand-new and as such it is not part of the regular OTDT distribution. OTOH, the plug-in is so simple, I don’t expect a lot of changes needed. So, if you have an Indigo-ish Eclipse just add this update site:

• http://download.eclipse.org/objectteams/updates/contrib

Then install “Maven Integration for Object Teams (Early Access)”:

Don’t forget the <repository> and <parent> definitions from above and you should be ready to go! Yesss, sometimes it pays off that OT/J is a kind of Java, a lot of the tooling can just piggy-back on what’s already there for Java

Remember the last time you had to cram your whole hosehold into boxes, bags and cases? You may feel excited about your new home etc. but the whole boxing business is quite a drag, ain’t it? There’s of course at least two ways of approaching this:

1. Don’t look, just shovel everything randomly into boxes
2. Look at each single piece, indulge in memories associates with it and sort it to its likes

Obviously, (1) means that on the other side you will unpack a whole mess of junk. OTOH, (2) won’t be finished before the moving truck arives. Still deep inside there lives a bit of hope, that you’ll move to your new home with only that stuff you’ll actually want, and everything ready to be neatly deployed into its new destination. Moving could free you from all the junk you don’t want any more, right? And even more, after the move, you may want to know, where everything is, right?

When moving the Object Teams project to Eclipse I was in the lucky situation that I could indeed use the occasion to sort through some of our stuff. The software engineer might be tempted to even speak of some “quality assurance” along the way, but let’s be careful with our wording for now.

On January 26 this year, the Eclipse Object Teams Project was created and we started to put up signs “Object Teams is moving to Eclipse.org”. Recently, I changed that sign to “Object Teams has moved to Eclipse.org”. So, what exactly happened between then and now?

Learning the infrastructure

At first the newly appointed eclipse committer and project lead is overwhelmed with all the shiny technology: web server, wiki, version repositories, build servers, download servers, the portal, project metadata, accounts for this and that, bugzilla components, versions and milestones and what-not. “Alles so schön bunt hier!”

I won’t indulge in talking about the paper work needed at this stage, after some four days I had most my accounts and the project was registered as “incubation - conforming”, so we were ready to go into the parallel IP process.

Initial contribution

On project day #12 I submitted my first CQ and, yes, that submission was already the result of heavy refactoring: I had renamed most (not all!!) occurrences of org.objectteams to org.eclipse.objectteams. A piece of cake? Well not exactly if the code piles up to a zip of 35 MBytes (not including .class files and nested archives), and not if your team-support plug-in goes berserk on some of the renamings and if … (see also this post).

Parallel IP process - our version

In fact our version of the parallel IP process looked like this:

On one thread I was chasing after some people and institutions to just provide the necessary signed statements of code provenance. Wrt the individuals this was painless, however, the university and the research institute involved both had their very specific strategies for delaying the project. All-in-all it took them more than one week per sentence in the final document. Or would you prefer the words-per-day count?

The much feared IP analysis turned out to be a very constructive collaboration with Sharon Corbett. I was really amazed about the obscure pieces of code (and comments) she brought to light, things that I never knew where in there. So that was helpful information, actually . Most of all I was pleased by her quick responsiveness - quite in contrast to the other thread. Thanks Sharon! I should also thank Jan Wloka who from the outset of the project took care that we’d have copyright headers and that stuff.

The effect was: at the time I got the signatures that cleared us to check our sources into svn the IP analysis was already done and complete! And not only that, during that process we’ve done some significant cleanup.

Code cleanup triggered by the move:

• Proviously, Object Teams used the JMangler framework for launching with our bytecode transformers in place. This was a great thing to have back in the olden Java 1.3 times. But in 2010 our Java 5 based alternative had matured and we didn’t even have to put JMangler into any of our moving boxes
• We used to maintain a patched version of BCEL 4.4.1 (developed at Freie Universität Berlin, as the namespace de.fub still announced). I consulted the AspectJ folks who maintain a patched version, too. But their patch only vaguely resembles the original, and they clearly stated that they saw now chances of these changes being adopted upstream. So, I went back to our sources, checked the patches, checked the current version 5.2 of bcel and found that the remaining bugs could actually be easily worked around from the outside. That’s when I learned the details of Orbit: since our initial commits to Eclipse we never had to bother about the bcel version, it just comes right flying from the Orbit. So we got that legacy version cleared up.
• My heart skipped a tiny single beat, when I learned that one of our most central data structures could not be accepted at Eclipse: I had patched class WeakHashMap from the OpenJDK to create a DoublyWeakHashMap with quite unique characteristics concerning garbage collection. We need that! Yet, the license (GPL with “classpath exception”) was not accepted. I made a quick experiment with wrapping instead of patching and guess what I learned (again): while the patched version was created in the pursuit of performance, still, after changing the strategy (to what was destined to be slower) my measurements could not show any performance penalty. So, carve that in stone: never optimize without measuring. The new version has the same performance - and no license issues!

Of course, there were more issues like needing to file a new CQ just for using files like xhtml1-strict.dtd, but those caused no grief after all. Enter the next phase.

Getting everything to build and test on eclipse servers

OK, when we opened the boxes at our new home, some of the content was actually broken on the way.

Fixing broken builds

The ugliest part was getting an ancient set of PDE/Build scripts to run on build.eclipse.org. Digging through a 30MByte build-log looking for the cause of a build failure never was fun. The point that dissatisfies me with all the build technology I’ve seen so far: You have a build that works on one machine and with one version of the software. Then, one arbitrary piece of the setup changes, let’s take a big change as the example: moving your JDK from 5 to 6. It’s OK that things may break at this point, but the kind of breakage frequently seems to have nothing to do with what you’ve changed, e.g., after moving to a different JDK the compiler can no longer resolve java.long.Object, whereas everybody knows: that’s not the difference between the two JDKs. The problem is not broken builds, the problem is how little clues the logs give you for finding the root cause of the breakage, or even: telling you how to fix it. A technology that works when it works is one thing. A technology that helps you get it to work is another (and we’re working hard to make Object Teams fall into the second category).

Modernizing the build

OK, enough complained, the move to Eclipse again gave reason to cleanup that monster build and even update to using some of the automatic built-in p2 stuff (yes, finally we use p2.gathering=true), rather than manually invoking the various p2 applications (publisher, director). When it runs, you may even get the impression you know why it does.

The final round in improving our build was adding bundle signing, yeah! Of course, that’s when all the p2-metadata generated during the build don’t help you any more because those include the checksums of your unsigned jars. So I created a tiny little shell script to automate those steps required after a successful build&test. I ended up with 7 more transformations of our metadata needed at this stage. So we’re back at directly invoking p2 publisher, p2 director, do various XSL transformations etc. Most of these could actually be done by a PDE/Build - p2 integration, but let’s not expect too much, not now.

Did I mention the almost 50000 tests that successfully run during each build? Well, that’s what we owe our users, right?

It builds - let’s ship it!

OK, let’s move ahead to the success-story-part. Less than a month after the initial commit we had our first milestone. It’s so good to be back in business After fixing all those migration-induced regressions I’m sure our code has better quality than before.

User side migration

Only one burden we had to pass on to users: due to the changed namespace, the configuration files of existing OT/J projects have to be updated. Luckily, it wasn’t too difficult to add some specific build-path problems and quickfixes, which should make the migration for users pretty smooth.

Installing

Right while I was publishing our first milestones a new cool tool came around the corner: the Market Place Client. So, now if you download, e.g., the Helios Package “Eclipse IDE for Java Developers” you’ll get the OTDT installed without having to know the download address, just select Help > Eclipse Marketplace and search for “Object Teams”, and you’re ready to hit “Install“:

Interestingly, in order for this to work I had to ask for this feature, which later down the road triggered this blocker security issue. At the end of the day this made me ponder about generalizing various things that the user might want to know when installing software. And indeed, Object Teams should play an active role in this discussion: the whole business of OT/Equinox is based on the assumption that the user agrees with what we are doing. We already do our best in treating the users as grown-ups who can make their own decisions, if we provide sufficient information, like:

This little screenshot tells you a whole story about this version of the bundle org.eclipse.jdt.core (see last column):

• The icon in the 1st column says: this bundle is signed, but that signed content is going to be woven at load time before the JVM sees it
• Columns 2 & 3 give the obvious information that this is not the version provided by the JDT team but something from the Object Teams project, which BTW. is still in its incubation phase.
• Column 4 finally gives you all the gory details: a sophisticated version number plus the list of OT/Equinox plugins that have declared to adapt the current plugin.

That’s the kind of transparency we show upon request after the software is installed. The mentioned bug 316702 is about providing similar transparency already during install.

So, what’s the plan?

Given that all legal and technical matters have been sorted out to this point, and given that the tool is in an even better shape than the final OTDT 1.4.0 from objectteams.org, what’s our plan?

• Just recently I requested a Release Review, tentatively scheduled for July 7, so with only little delay after Helios we should have an actual, stable release.
• I decided to defer the project graduation some more months to give us time to define which parts of the software are actually API.

Where can I see it in action?

Well, given the current milestone releases (and the ease of installing) and all the documentation we have in the wiki nothing should stop you from running your first Object Teams demo in do-it-yourself mode

Otherwise, if you happen to be in Vienna on June 25, just come to this DemoCamp and I’ll help you to get started with Object Teams.

So, indeed for the Object Teams project that past 6 months were used to turn this:

into this: