Terracotta Killer App

In the comments to my post about what Terracotta is not, Jay Meyer wrote:

[Terracotta] is so disruptive that an application architect really needs to sit down and assume Terracotta is available and then design a real killer app.

That comment has been tumbling around in my head for the last month. I think Jay makes a good point. On the one hand, lots of people are using Terracotta for a bunch of different things. On the other hand, we do tend to struggle getting people to think in the Terracotta way and not slide back into bad old habits using the fine surgical instrument of Terracotta as a hammer (we have a running joke inside Terracotta that every use case looks like "distributed caching" because the EHCache integration module is so easy to use).

With that in mind, I've recently noticed a couple of interesting and creative uses of Terracotta. They may or may not be the "killer app", but they do really take advantage of Terracotta's unique features:

• Terracotta for configuration management.


• Terracotta for enterprise-wide control data.

In the first use case, JR Boyens describes how they used Terracotta for managing configuration across many JVMs. What I really like about this is how Terracotta takes a problem that should be simple (configuration) but is ridiculously hard in a distributed computing environment and reduces it back to its simplest essence.

There's a lot I like about the second use-case presented by Mark Turansky (and I'm really excited about the case study Mark is working on), but two things stand out in the particular blog post mentioned above. The first is using Terracotta to simplify making cluster-wide changes to data that affects enterprise-wide application behavior (this is starting to look like a theme):

For example, we offer a REST API to abort a pending message (such as http://ourendpoint/message/abort/abcdefg-the-guid-wxyz). The easiest way we found to tell the entire bus to disregard this message was to flip the bit on the message itself.

—Mark Turansky

Using Terracotta, to change the status on the message in response to a REST call is as simple as looking it up in a local data structure (like a Map), then flipping the status on the message object. Without using Terracotta to do this, you have to go through a ton of work (probably involving RMI or a database lookup or who knows what) to figure out in which nodes in your distributed computing environment the message is currently resident. Then, once you've found the message, you have to somehow deal with the concurrency issues of changing its status across multiple machines while it is potentially being processed.

Again, Terracotta takes a problem that should be simple but is hard in a distributed context and reduces it back to the familiar simplicity of a plain Java program.

The second thing I like about Mark's use-case is a side effect of Terracotta hiding the complexity of distributed computing: the architecture works just as well in a single JVM as it does across many JVMs. Removing infrastructure concerns from application code makes working on that application so much easier. Terracotta lets you work at the unit level without having to struggle with setting up a distributed computing environment.

If you were using a database, for example, to coordinate this message activity, you would be forced to run that database in order to work on a unit-level feature. Using Terracotta to take care of infrastructure concerns lets you work on a unit-level with Terracotta off so you can express the problem in pure Java. We've had users remove as much as 30% of their application code by switching to Terracotta.

This has a direct impact on code quality and developer productivity. Since there is a direct relationship between the number of lines of code and the number of bugs in an application, removing 30% of your application code should, at a stroke, remove a significant number of bugs. Developer productivity goes up because you don't have to spend time setting up and debugging interaction with databases and other fiddly parts of a distributed application environment just to work on a unit level feature. That gives you a lot more time to work on features.

By factoring out the infrastructure, Terracotta lets you truly work at the unit level and concentrate on getting the code logic right. To put it another (much more abstract) way, Terracotta lets you concentrate on semantics—on expressing what you mean—rather than on syntax—the mechanics underlying the expression.

Starting a Terracotta-Enabled JVM from a C Program

I just thought this thread was cool:

http://forums.terracotta.org/forums/posts/list/971.page

I wish I knew what the use-case was. It's very intruiging.

Terracotta Is Not an Object Cache

Understandably, Terracotta gets painted as an object cache sometimes. We certainly promote using Terracotta as a data cache and lots of other technologies that do data caching are object caches. But, Terracotta just isn't an object cache. Really.

To me, an object cache is a data cache that stores its data as objects and then keeps those objects around for a while so you don't have to go through all the work of building up those objects again every time you need access to the data.

Terracotta can be used like that, but it's sort of a shame if that's all you get out of it. Terracotta is waaaaaay more than that. Terracotta is:

1. Network-attached, virtual, persistent heap
2. Transparent, inter-JVM thread coordination

That's a lot of words all stuck together. What does that mean? First, the network attached, virtual, persistent heap part...

Network-attached, Virtual, Persistent Heap

Ok. This means, roughly, that Terracotta makes some of your JVM heap shared via a network so that a bunch of JVMs can all access the shared heap as if it were local heap. Like a network-attached filesystem, but for object data. (In the forthcoming Terracotta book from Apress, Ari makes a really great analogy between Terracotta and virtual filesystems. You should read it.)

It's "virtual" because each JVM accesses it as if it were actually part of the local heap. The JVM can't tell it's talking to network-attached memory. So, you don't have to write special code to talk to it. It's just there.

It's persistent because...well...it's persistent. The Terracotta virtual heap is managed by the Terracotta server (again, just like a fileserver). The server stores the object data on disk so your objects are durable. That means that you can make parts of your heap persistent just by declaring them to be so in a configuration file (or, via annotations, if you are nervous about XML file madness).

We say virtual *heap* to try to drive home the point that this is a service of the JVM, not a developer-facing API. There's nothing wrong with APIs for things that are within the problem domain of your code, but virtual heap isn't in the problem domain of your code. As such, your code shouldn't have anything in it about where your objects live.

(I can already tell this is going to rile some people up... virtualization always riles up some people who are sure that the sky is going to fall if they don't have direct and explicit control over everything. You should think about it in terms of the people paying you to write the software... did they ask you to write software about object data storage and network communications? If so, then it's in your problem domain. If not, then you're just wasting their money by wasting your time thinking and coding about it.)

Transparent Inter-JVM Thread Coordination

If the virtual heap stuff didn't make it clear that Terracotta isn't object caching, the next part about inter-JVM thread coordination ought to. Terracotta makes cross-thread communication and coordination (you know: synchronized, wait(), and notify()) available to threads on different JVMs.

This is HUGE. Do you know how hard it is to coordinate activity across JVMs? I mean, it's not impossible, but it's a LOT harder than it sounds.

At a stroke, Terracotta makes it plain simple. Terracotta gives you an inter-process communication mechanism for Java that works across JVMs and across machines, but is expressed in terms of the simple thread communication that you already know.

(If you don't already know this stuff, it might be a good idea to spend a little time looking at the java.util.concurrent package. There are a lot of nice, high-level tools in there for expressing inter-thread activity. If you're scared of concurrent programming, you shouldn't be. It's a skill that can be acquired just like any other aspect of programming. And, considering that computers are now getting faster by increasing parallelism in the form of multiple cores rather than just bluntly increasing clock speed, you're probably going to need to learn at least the basics of concurrent programming soon)

JVM Clustering
All of this stacks up to JVM clustering, not object caching. Terracotta is a very powerful tool that can reshape the way you think about distributed computing. This is a really big idea and about a thousand times more interesting than simple object caching.

Mac Leopard Crash

Twice this morning changing my location. Jeez.

Apparently a lot of people are having this problem:

http://discussions.apple.com/thread.jspa?threadID=1329965&start=0&tstart=0

Leopard sux. (Although, some people report it happening on Tiger—never happened to me on Tiger, though).

Robots!

Last week, I went to the San Francisco Java Meetup Group's March meetup on controlling robots with embedded Java. Shawn Silverman gave an awesome presentation/workshop where we all got to program a TrackBot first using a simulator, then using the real thing.

There's a callback API in Java for controlling the TrackBot that invokes specific methods when sensor data is emitted from the TrackBot's serial interface. Based on the sensor input, you can control each of four motors that control the robot's tracks.

Once you've written your controller program and tested it in the simulator, you have the option of plugging your computer directly into the TrackBot via a serial port (which can be made wireless using BlueTooth) or loading the program onto a very cool little embedded Java device that Sun has developed called the SunSPOT.

This was a really fun evening.

If you live in San Francisco, the San Francisco Java Meetup Group is really cool. Many thanks to the organizer, Aleksandar Gargenta, for putting together a fun and engaging Java Group in SF. The next meetup, on April 7th, is on Cuercas, Caucho's Java PHP implementation.

Mac Leopard Crash

Last login: Sat Mar 8 08:56:45 on console
macbook:~ orion$ uptime
8:58 up 2 mins, 2 users, load averages: 0.32 0.13 0.05
macbook:~ orion$


This time, I closed the lid on my laptop and, ten minutes later, I opened it again. The laptop wasn't exactly asleep (no pulsing glowy light and the disk was churning), but the display wouldn't turn on. I closed the lid again and waited for a little while to see if would actually go to sleep (sometimes it does, then it will wake up). No joy. Had to lean on the power button again.

To add insult to injury, every time the machine crashes, the stupid metadata crawler starts reindexing my drive from scratch and takes over the machine for a long time.

Leopard sux.

Mac Leopard Crash

macbook:~ orion$ uptime
9:40 up 6 mins, 2 users, load averages: 0.71 0.41 0.18
macbook:~ orion$


This time, I was changing my location and the whole machine locked up. This happens fairly often (1 in 10 times I change the location).

Pretty frustrating.

Leopard sux.

Mac Leopard Crash

host-144-0-0:~ orion$ uptime
9:08 up 3 mins, 2 users, load averages: 0.39 0.17 0.06

OS locked up while trying to change location. Last night it wouldn't wake up from sleep. Two crashes in two days. And, every time it crashes, it has to rebuild the spotlight indexes from scratch. Might have to turn off spotlight.

Leopard sux.

Words Spoken In Anger

My last post about OS X was unfair. I did to Apple what people do to our technology from time to time which is to vent a momentary frustration in an overly hostile way.

The truth is I looooooooove Apple. I'm an unabashed Apple fanboy. I'm pretty sure Apple doesn't give a flying f#%$ about me (or any of their customers, for that matter—I kind of think Apple hates their customers because they aren't as cool as Apple), but that doesn't mean I don't love them anyway. They've done wonders for product design in industries that have had sucky products for decades (computers, phones, the music industry, etc.) The world of gadgets and media will be a lot worse off when Steve Jobs retires.

But, I do fear that OS X is sliding into crappyness and that's really bothersome. I want OS X to be insanely great, not crappy.

OS X In Danger

... of not being my operating system anymore.

There are two things I love (or, rather, loved) about OS X:

1) It's actually UNIX underneath and not some amateur-hour piece of garbage like MacOS 9 and its predecessors or Windows.

2) It (used to) just work.  There were very few dumb impediments to getting things done.

Unfortunately, my experience with Leopard has changed the second.  My MacBook on Leopard  no longer behaves well at all and I'm starting to hate it.

Crappy Shutdown Behavior

The first thing that really annoys me is that it doesn't shut down.  I don't know what it's doing, but, when I tell it to shut down, rather than tearing everything down and dutifully turning off, instead it lets stuff time out for thirty seconds or so and then tells me it failed to shut down.  

Um... are you kidding me?  I'm just NOT going to wait for thirty seconds for a function to decide it's not going to work.  If I want it to shut down, I want it to shut down RIGHT NOW, no excuses.  I now give it ten seconds to shut down, then I hit the power button (or, yank the battery).  I'm very close to ignoring the shutdown process altogether.  Apple should just take the feature out if it's not going to work.

(This reminds me of another thing that I hate about OS X, although it's not new with Leopard: the force quit from the finder doesn't always work.  Force quit should be as effective as 'kill -9'.   If it's not going to actually kill the thing, then don't give me the damned option.  But, thankfully, since OS X is actually UNIX, I can use kill to get the behavior I want.  While I wish force quit actually worked the way it's supposed to, the fact that there's a workaround makes me hate the force quit feature, but not hate OS X as a whole.  But, for this other stuff I'm talking about there are no good workarounds.)

Crappy Wake Up and Login Behavior

The next thing that I hate about Leopard is that it has gotten VERY flaky about waking up properly when I open my laptop lid.  There are numerous manifestations of said flakiness, but the upshot is that I now dread closing the laptop lid because I know that there's about a 50% chance that I'll struggle to use my computer again when I open it.

This development really saddens me since elegant wake-up behavior was something I really loved about my Mac laptops.  My Windows laptops were always crappy in this regard and now that my Mac laptop behaves the same way as Windows, I feel betrayed.  Fighting with your laptop every time you open it up is a real usability problem and a big productivity drain.

Ridiculous Login Hooks

There are WAAAAAY to many things that have hooked themselves into my login process.  There are three or four things that check for updates and make my computer unusable for minutes after login.  You know what?  When I want to login, I want to login NOW.  I'll check for software updates later when I have a moment.  When I'm logging in, I REALLY want to use my computer.  Talk to me about upgrading in, like, ten minutes when I'm no longer frantically trying to find something out or get something done.

I know I can probably turn this stuff off, but I really don't want to be bothered.  I just want it to work.  Of course, this didn't used to be much of a problem when I could stay logged in for weeks at time.  But, now that Leopard forces me to hard reboot my laptop every day, it's a damned nuisance.

Frequent Lock-ups

This last thing is just really lame.  The perceived frequency of hard OS lockups has drastically increased.  The most predictable manifestation is when I'm switching my location from home to work.  Greater than 50% of the time, doing so causes the OS to lock up  (well, I don't actually know if the OS itself is locked up, but the display certainly is; I haven't yet tried to login through ssh to see if the OS is actually still responding, mostly because I'm trying to switch my network settings when this happens, so I don't have network when it does).

This is just crazy.  I thought the days of the OS just up and freezing were over.  But, I guess not.  And that's a shame.

Time to Switch?

I hope Apple gets its head out of its ass soon and fixes the OS so that I can get back to loving it.  I feel betrayed, but I'm still hoping it will come back.  However, if the next few releases don't fix these problems, I'm going to have to start looking around for another OS.  At least, I still love my MacBook.  It's still the best computer I've ever bought.