Ehcache 1.7.0 and Terracotta 3.1.1 GA

Ehcache 3.1.1 and Ehcache 1.7.0 went GA last week. The big news is that Terracotta now ships with Ehcache in the box and Ehcache gets a high-performance, high-scale, coherent distributed cache plugin backed by Terracotta that installs with two lines of configuration change.

You can try it out with a short getting started tutorial »

A Sample Application for Terracotta's Hibernate Cache

I just finished a screencast of a tour through a sample application demonstrating how to configure Terracotta for Hibernate. This one uses JBoss as a web container, but, apart from the steps to setting Java startup options, it's applicable for any container or for a standalone Java application.

Here's the screencast:

Terracotta for Hibernate sample screencast »

and here's a link to the sample code (which is, essentially, a web wrapper around the standard Hibernate tutorial):

Terracotta for Hibernate sample application »

Terracotta 3.1 Released!

Woohoo! We just released Terracotta 3.1. It has two awesome new features:

• Terracotta for Hibernate »


• Terracotta Distributed Cache »

I've also just posted a screencast tutorial showing how to get started with Terracotta for Hibernate:

Get Started With Terracotta for Hibernate Screencast »

Check it out!

Memory Fault

I just read an interesting blog post from the_codist() about the constraints of memory and what he calls developing with just-in-time information. I started to leave a comment, but it turned into an entire blog post, so I'll just write an actual blog post.

It had never occurred to me that what I have been ascribing to "memory loss" is really just a constrained window on a much larger and growing pool of information.

My wife and I, discussing the appalling failures of our once-spectacular respective memories over the years, have reached the conclusion that, when you are young, the most important thing you can do intellectually is memorize new information. But, as you age and the amount of information you've ingested piles up, it's less important to commit new information to memory and more important to fit new information into the larger conceptual framework you've built up over the years.

While not as shiny and sparkly as being able to remember obscure facts, it yields much more interesting thought. And, of course, in the context of programming, you are better equipped to avoid the pitfalls of a new programming language or technology having seen them in other, older environments.

As for the job interview where they ask you what's "the home interface of an EJB2.X Entity bean," you probably don't want to work with those people anyway because they don't know how to look for real skills in an interview. If they've hired only people who can answer questions like that, they more than likely have a crappy development team that you really want no part of. Unfortunately, if you're an independent consultant, you might not have the luxury of turning them down.

Two (Naive) Ideas About Cars And Oil

One: Let The Sick Companies Die

There is NO GOOD REASON TO SAVE THE US CAR INDUSTRY. They can't figure out how to make a product people want at a profit. Every company I've worked for that hasn't figured that out has gone out of business. Yes, it's painful, but a big, dumb bailout of big, dumb companies leaves crappy products on the market and prevents good products from being made.

I feel bad for the people who will lose their jobs, but they're not blame-free either: the auto workers unions have choked an already moribund industry to death.

Two: A Federal Reserve Board For Oil

US energy policy is fragmented and dumb. Energy policy should be like monetary policy: simple, with a few but very powerful levers.

Rather than mandating fuel efficiency, subsidizing alternative fuels, and otherwise trying to pick a winning technology, the government should appoint a peer of Ben Bernanke but for the price of oil. That person should then have total control over the wholesale price of oil in this country, much the way that the Fed gets to decide interest rates.

Then, an oil price target should be set at, say $500/barrel in 2020 with regular, step-wise increases from now until then. That would give everyone a clear signal that oil will not be a viable energy source in twelve years and give the energy and transportation industries time to react. This is the surest, simplest way to drive innovation and break the back of oil as the energy source for the world.

Scalability Is A War; And, Don't Pre-Optimize

Scalability Is A War

At last weekend's Silicon Valley Code Camp, I attended a panel discussion (right after and next door to my own presentation on Terracotta) called Scaling a Platform with Bebo, Yahoo (YOS), RockYou and Sun Microsystems. The issues and suggestions resonated very strongly with my own experience. My favorite phrase from the discussion came from Steve Cohen, head of engineering Bebo Platform, who said, "Scalability is a war that you are always fighting."

Very true. That notion is a consequence of his other point (also espoused by Terracotta's own Steve Harris): that you should never pre-optimize because what you think is the performance or scalability bottleneck is almost surely wrong. There is no architecture or implementation that is guaranteed to give you scalability from now until the end of time. The best thing you can do is to keep your architecture and implementation simple, elegant, and easy to change. Since there is no way you can truly anticipate bottlenecks, having a simple and flexible system will put you in the best position to relieve bottlenecks as they appear. Don't pre-optimize your code; don't pre-optimize your architecture; be prepared to re-enter the performance and scalability battle again and again as your capacity requirements increase.

I think Terracotta complements this approach very well, since it keeps your code clean and free of "scalability" cruft.

Other Interesting Things In No Particular Order

Bebo does ~400 million page views per day on 120 application servers (I'm assuming that most of those servers are dark during off-peak hours).

Cohen points out that Java is very fast. He told a story about a prominent site not running on Java that burgeoned with pride at having hacked on mysql to get a particular operation down to 45ms. Chuckling to himself, he pointed out that if an entire page rendering, much less a single operation within the page rendering cycle, took 45ms, they were sunk. They need their page renderings to be down near 10ms.

Cohen on memcached:

• memcached is very sensitive to rebalancing, since the hashing strategy is invalidated when new memcache servers are added. 
• The data in memcached must be very static in order to be useful.
  
• There are big differences among memcached clients
• It's very easy to saturate your network with memcached. I/O must be chunked up into large operations for memcached to not take over the network.  Bebo serializes and deserializes large HashMaps, not individual map entries. This sentiment was echoed by a member of the audience who had his network saturated by fine-grained access to memcached entries.

Some discussion on scalability and capacity testing:

In my experience, it's pretty much impossible to accurately test production load outside of a production environment. That isn't to say that you should ignore stress, load, and capacity testing, but you shouldn't assume that you've gotten it right until you are actually in production. At Walmart.com, we always tested new releases on small sub-clusters to see how they would react to real load. With that in mind, I asked the panel what they do.  They pretty much agreed with me and they all said that they do their final load testing in production, but there were some interesting strategies they brought up that deserve a mention:

• Use aspects to create instrumented builds with lots of logging and deploy it to a few servers.  Bebo does this and sends the logs to a syslog server that then dumps the aggregate logs to Hadoop.  Later, they parse those logs to find patterns.
• That led me to ask if they used Splunk.  The answer was no because Splunk is too expensive. That was somewhat disheartening to hear, since Splunk is sooo cool.
• Tom Hughes-Croucher, tech evangelist from Yahoo Developer Network (YDN), suggested recording real user sessions for playback in a test environment to test real user activity. 
  

Left Nav Is Dumb

I've built a few sites with fixed left-side navigation and I've stopped doing it because it sucks.  Here's a few reasons why:

• The whole left-side gutter is fixed, regardless of context.  It invades the semantic area of the main content in context insensitive ways.  For example, you can't put any lists on the left side of the main content area.
• It encourages the mindless proliferation of navigation elements.  If you can't get your navigational scheme to fit horizontally, you haven't thought hard enough about your navigational scheme.  Just adding more navigation elements just means you can't figure out a sensible way to organize your content.  You've left navigation as an exercise for the reader.

Stacked content elements are much more flexible, allowing your main content area to be wholly devoted to the content.  It also lets you do pyramid style presentation where the lead is most prominent and, as you continue to stack content, it can get progressively more detailed because nobody reads below the fold unless they are really interested.

Here's an interesting case study: CNN.com in Nov 2004 v. CNN.com in Nov 2008:

CNN.com Nov. 2004

Notice the intrusive left-nav. It's also a fixed width, which is also dumb.


Here's 2008's version:



Nice stacked nav. Much less of the page is devoted to navigation and the lead story is unencumbered by other goo.

So... WHY THE *$%#@ DID iGOOGLE MOVE FROM TOP-TABS TO LEFT TABS????

Here's my new and retrograde iGoogle page:

Gah! Look at all the empty space on the left that could be used for more content! And, the names of most of my tabs are elided. How is this better? When am I going to have the 50 tabs that would make this layout make any sense at all? Why can't I go back to the old way?

This single cack-handed change forced down my throat has made me go back to good old my.yahoo.com. Sheesh.

Terracotta Underware

Just read on Krzysztof Kliś' blog a description of Terracotta as "middleware between a JVM and an application". That's a pretty accurate and succinct description, although not the use of the term "middleware" that I'm used to.

I'm used to the term 'middleware' used to describe "a set of enabling services that allow multiple processes running on one or more machines to interact across a network" (Wikipedia)... or something like that. (Technology terms are notoriously squishy... just look at all the commotion over the terms "grid" and "cloud" computing.) Sometimes, I've heard middleware being used to describe the stuff between your application and your database. In any case, it basically boils down to the gunk you have to use to glue your distributed application together.

Since Terracotta:

• Sits below the JVMs, orchestrating their threads and persisting their memory
• Frees your architecture from the database

perhaps we can call Terracotta "Underware." As in, you finally graduated from clunky JEE diapers,  now you sport lightweight, adult Terracotta Underware:

 

Haruki Murakami

Seeing Haruki Murakami at Zellerbach Hall in Berkeley on Saturday was a treat. Somehow, Rosaclaire and I got ridiculously good seats and, at one point, we were no more than twelve feet from him while he read an early short story (The Rise and Fall of Sharpie Cakes) in Japanese. Not that I speak Japanese.  The story was later read in English (translated by Jay Rubin who was also in the audience) by his later interlocutor during the question and answer portion of the evening.

Process

The most interesting thing he talked about was his writing process which I have long wondered about. He doesn’t use any outlines; he just starts writing and ends up where he ends up. At one point, he likened his writing to simultaneously being the player and the programmer of a video game.  As the programmer part of him is working out what happens next, the player part of him waits poised to find out what happens and is then delighted when he does. It sounds really fun. To me, that seems like the only way to make writing bearable in the long term. And, I think it’s one of the main reasons why reading Murakami is so enjoyable. I always read his books urgently, always wondering what’s going to happen next.

Of course, if you’ve read Murakami, you know that what happens next is never predictable, is often fantastic or absurd, but always in a matter-of-fact way. The trick is that each new revelation always seems appropriate and somehow related to the whole, pushing the story forward towards its conclusion. But, how a sheep man or a character sitting at the bottom of a well and slipping into a phantasmic hotel room is related to anything is impossible to understand intellectually without a great deal of (irrelevant) critical wrangling. But, Murakami’s narrative always manages to pull it off as if it were the most natural thing in the world. What can I say? The man’s a genius.

Although he said he doesn’t write from an outline, he does go through an extensive revision process. It takes him, he said, between one to two years to complete a novel (one of which he said he had completed the week before (which is one of the reasons he was so relaxed and happy)). But, even though he revises a great deal, he said the initial draft was the most important. The first draft must get it mostly right; the revisions are done to smooth out the details. (As an interesting anecdote, he mentioned that what became The Wind Up Bird Chronicle and ??? started life as a single book; there was something wrong with it, though, he said, and spun them out into two separate books. Good thing. The Wind Up Bird Chronicle came out just right.)

A Bad Memory

He said he enjoys reading his own books in translation because the words aren’t his. He said that it’s embarrassing to read his own writing in Japanese, but in translation, he can just enjoy the story. And, he said, his memory is so bad that he never remembers what happens in his own books so, as he reads them anew, he himself is anxious to find out what happens.  I guess he gets to just play the video game instead of having to simultaneously program it as he goes along.

The Dark Place

He said he wakes up between four and five in the morning and writes for three or four hours. During that time he says his work as a writer becomes dangerous because he descends into a “dark place.” I gather this is a descent of fantasy into the dark depths of being human and alive. This sheds a great deal of light (excuse the pun) on all of that well-dwelling. 

He said that he must keep himself mentally and physically strong in order to endure the descent and to reliably make his ascent back from the dark place.  His avid running and swimming are hedges against the perils of writing.

Obsessions

The subject of obsessions came up, too. He admitted to having many obsessions: elephants, refrigerators, couches, cats, ears. I was thinking to myself: wells, Cutty Sark, mysterious, beautiful, but emotionally or psychically damaged women. Rosaclaire offered up pasta and shadowy right-wing conspiracies. And, of course, the sheep man.

This gave me a measure of relief about my own banal obsessions. Maybe I should try making them into art, also.

Murakami Rules

Already a slobbering fan, I’m now a better informed slobbering fan. As Rosaclaire pointed out, he fit in person exactly the image we had of him. Rosaclaire noted that she expected him to be wearing stylish but informal and loose-fitting clothes with top-siders. Being most curious about his writing process, I wasn’t surprised by it, but I’m very glad to know more about it. I now have an image of him writing as similar to John Hulce as Mozart crouched over his billiards table scrawling out the Requiem in Amadeus. Only not drunk and a lot healthier.  Oh--and, his head sort of bobbles as he talks.

Really, iTunes? You Can't Find the File You Just Downloaded? That I Paid For?

Where did it go? It's not in the iTunes library. It's not in the iTunes downloads folder. The iTunes store thinks it's already been downloaded, so it won't download it again.



As a friend of mine just pointed out, iTunes is like a vending machine. But, a broken vending machine.

(BTW, I'm downloading individual episodes because it wouldn't let me buy the entire season because it thought I had already downloaded the entire season. Gah. This seems like stuff that would be easy enough to test before releasing the shiny new software. Keeping track of downloaded files just isn't rocket science.)