The Wayback Machine - https://web.archive.org/web/20100125014440/http://planet.apache.org:80/committers/

Planet Apache

Ben Hyde—Sphere of Deviance

Bill Tozier drew my attention to a most excellent term of art, Daniel C. Hallin’s phrase Sphere of Deviance.  That posting has the nice insight that Jon Steward makes his living in the border lands were legitimate controversy meets the deviance.

Hillard was mapping the world of national discourse, i.e. old school journalism, during the period when the consensus about Vietnam war imploded.   But this framework is universally applicable.  For example Open Source, as a tool for enterprise software development has traveled over the last decade all the way into the center.   Freudian analysis has travel out.  Some ideas orbit the consensus like comets; i.e. the fetish for solving all problems with unregulated markets.

It’s fun to take a topic (peak oil, diets, business plans) and map the various ideas and players along with their trajectories onto that drawing.

For some reason this all reminds me of how the distant periphery of cities has always been where you find the real nut case cults hanging out.

Ben Laurie—Stupid: A Metalanguage For Cryptography

Various threads lately have got me thinking about implementing cryptography and cryptographic protocols. As I have mentioned before, this is hard. But obviously the task itself is the same every time, by its very nature – if I want to interoperate with others, then I must implement effectively the same algorithm as them. So why do we ever implement anything more than once? There are various reasons, varying as to their level of bogosity. Here’s a few

  • Trust: “I don’t want to trust anyone else’s code”. This, in my view is mostly bogus. If you don’t trust others to write your crypto, then you’ve got some pretty big problems on your hands…
    • You’re likely to be using some pretty heavyweight stuff like SSL and/or X.509, and reimplementing those is a seriously major job. Are you really going to do that?
    • Unless you are also a cryptographer, then you’re trusting the guys that designed the crypto you’re implementing anyway.
    • Ditto protocol desginer.
  • Languages: an implementation in Java isn’t going to work so well in Python. And although its true that you can plug C implementations into almost everything, there are legitimate and not-so-legitimate reasons for not wanting to do so…
    • You don’t trust native code: see above.
    • It makes your distribution hard to build and/or use and tricky to install.
    • You are running on a platform that doesn’t allow native code, for example, a web hosting company, or Google’s App Engine.
    • Native code has buffer overflows and MyFavouriteLanguage doesn’t: true, but probably the least of your worries, given all the other things that you can get wrong, at least if the native code is widely used and well tested.
  • Efficiency: you are not in the long tail of users who’s transactions per second is measured in fractions. In this case, you may well want specialised implementations that exploit every ounce of power in your platform.

Of these, reimplementation for efficiency clearly needs a completely hand-crafted effort. Trust issues are, in my view, largely bogus, but if you really want to go that way, be my guest. So what does that leave? People who want it in their chosen language, are quite happy to have someone else implement it and are not in need of the most efficient implementation ever. However, they would like correctness!

This line of thinking let me spend the weekend implementing a prototype of a language I call “Stupid”. The idea is to create a language that will permit the details of cryptography and cryptographic protocols to be specified unambiguously, down to the bits and bytes, and then compile that language into the language of your choice. Because we want absolute clarity, Stupid does not want to be like advanced languages, like OCaml and Haskell, or even C, where there’s all sorts of implicit type conversions and undefined behaviour going on – it wants it to be crystal clear to the programmer (or reviewer) exactly what is happening at every stage. This also aids the process of compiling into the target language, of course. So, the size of everything wants to be measured in bits, not vague things like “long” or “size_t”. Bits need to be in known places (for example, big-endian). Operations need to take known inputs and produce known outputs. Sign extension and the like do not want to happen magically. Overflow and underflow should be errors, unless you specifically stated that they were not, and so on.

To that end, I wrote just enough compiler to take as input a strawman Stupid grammar sufficient to do SHA-256, and produce various languages as output, in order to get a feel for what such a language might look like, and how hard it would be to implement.

The result is: you can do something rough in a weekend :-)

Very rough – but it seems clear to me that proceeding down this road with more care would be very useful indeed. We could write all the cryptographic primitives in Stupid, write relatively simple language plugins for each target language and we’d have substantially improved the crypto world. So, without further ado, what does my proto-Stupid look like? Well, here’s SHA-256, slightly simplified (it only processes one block, I was going cross-eyed before I got round to multiple blocks). Note, I haven’t tested this yet, but I am confident that it implements (or can be easily extended to implement) everything needed to make it work – and the C output the first language plugin produces builds just fine with gcc -Wall -Werror. I will test it soon, and generate another language, just to prove the point. In case the code makes your eyes glaze over, see below for some comments on it…

"This code adapted from Wikipedia pseudocode";

"Note 2: All constants in this pseudo code are in big endian";

"Initialize variables";
"(first 32 bits of the fractional parts of the square roots of the first 8 primes 2..19):";
uint32 h0 = 0x6a09e667;
uint32 h1 = 0xbb67ae85;
uint32 h2 = 0x3c6ef372;
uint32 h3 = 0xa54ff53a;
uint32 h4 = 0x510e527f;
uint32 h5 = 0x9b05688c;
uint32 h6 = 0x1f83d9ab;
uint32 h7 = 0x5be0cd19;

"Initialize table of round constants";
"(first 32 bits of the fractional parts of the cube roots of the first 64 primes 2..311):";
array(uint32, 64) k =
(0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5,
0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3,
0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc,
0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7,
0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13,
0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3,
0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5,
0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208,
0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2);

"For now, dummy in the message instead of declaring a function wrapper";
"Also, for now, allow enough room in the input for padding, etc, to simplify the loop";
uint32 message_bits = 123;
array(uint8, 64) message =
(0x12, 0x34, 0x56, 0x78, 0x9a, 0xbc, 0xde, 0xf0,
0x0f, 0xed, 0xcb, 0xa9, 0x87, 0x65, 0x43, 0x21);
uint32 pad_byte = 0;
uint32 pad_bit = 0;
uint32 tmp = 0;
uint32 tmp2 = 0;
array(uint32, 16) w = (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
uint32 i = 0;
uint32 s0 = 0;
uint32 s1 = 0;
uint32 a = 0;
uint32 b = 0;
uint32 c = 0;
uint32 d = 0;
uint32 e = 0;
uint32 f = 0;
uint32 g = 0;
uint32 h = 0;
uint32 maj = 0;
uint32 t1 = 0;
uint32 t2 = 0;
uint32 ch = 0;

"Pre-processing:";
"append the bit '1' to the message";

"note that we're using a 32-bit length for now";
"all the op32, op8 etc are _without_ wrap (where applicable) - i.e. wrap is an error";
"they also require left and right to both be the correct type and size";
"also, we have no precedence, it is up to you to bracket things";
"rshift is with zero padding";

pad_bit = 7 minus32 (message_bits mod32 8);
pad_byte = (message_bits plus32 1) rshift32 8;
message[pad_byte] = message[pad_byte] or8 (1 lshift8 pad_bit);

"append k bits '0', where k is the minimum number >= 0 such that the
resulting message length (in bits) is congruent to 448 (mod 512)";

"eq32 and friends return a boolean value (which is not even a bit)";

if (pad_bit eq32 0) {
pad_bit = 7;
pad_byte = pad_byte plus32 1;
} else {
pad_bit = pad_bit minus32 1;
}

"bor is like C || (i.e. RHS is only executed if LHS is false)";

"448/8 = 56";
while (((pad_byte mod32 512) ne32 56) bor (pad_bit ne32 7)) {
message[pad_byte] = message[pad_byte] and8 (not8 (1 lshift8 pad_bit));
if (pad_bit eq32 0) {
pad_bit = 7;
pad_byte = pad_byte plus32 1;
} else {
pad_bit = pad_bit minus32 1;
}
}

"append length of message (before pre-processing), in bits, as 64-bit big-endian integer";

message[pad_byte] = 0;
message[pad_byte plus32 1] = 0;
message[pad_byte plus32 2] = 0;
message[pad_byte plus32 3] = 0;

message[pad_byte plus32 7] = mask32to8 message_bits;
tmp = message_bits rshift32 8;
message[pad_byte plus32 6] = mask32to8 message_bits;
tmp = message_bits rshift32 8;
message[pad_byte plus32 5] = mask32to8 message_bits;
tmp = message_bits rshift32 8;
message[pad_byte plus32 4] = mask32to8 message_bits;

"for each chunk (we only have one, so don't bother with the loop for now)";

" break chunk into sixteen 32-bit big-endian words w[0..15]";
tmp = 0;
while(tmp ne32 16) {
tmp2 = tmp lshift32 2;
w[tmp] = ((widen8to32 message[tmp2]) lshift32 24)
plus32 ((widen8to32 message[tmp2 plus32 1]) lshift32 16)
plus32 ((widen8to32 message[tmp2 plus32 2]) lshift32 8)
plus32 (widen8to32 message[tmp2 plus32 3]);
tmp = tmp plus32 1;
}

" Extend the sixteen 32-bit words into sixty-four 32-bit words";
i = 16;
while(i ne32 64) {
s0 = (w[i minus32 15] rrotate32 7) xor32 (w[i minus32 15] rrotate32 18) xor32 (w[i minus32 15] rshift32 3);
s1 = (w[i minus32 2] rrotate32 17) xor32 (w[i minus32 2] rrotate32 19) xor32 (w[i minus32 2] rshift32 10);
w[i] = w[i minus32 16] plus32 s0 plus32 w[i minus32 7] plus32 s1;
}

" Initialize hash value for this chunk:";

a = h0;
b = h1;
c = h2;
d = h3;
e = h4;
f = h5;
g = h6;
h = h7;

" Main loop:";

i = 0;
while(i ne32 64) {
s0 = (a rrotate32 2) xor32 (a rrotate32 13) xor32 (a rrotate32 22);
maj = (a and32 b) xor32 (a and32 c) xor32 (b and32 c);
t2 = s0 plus32 maj;
s1 = (e rrotate32 6) xor32 (e rrotate32 11) xor32 (e rrotate32 25);
ch = (e and32 f) xor32 ((not32 e) and32 g);
t1 = h plus32 s1 plus32 ch plus32 k[i] plus32 w[i];
h = g;
g = f;
f = e;
e = d plus32 t1;
d = c;
c = b;
b = a;
a = t1 plus32 t2;
}

" Add this chunk's hash to result so far:";

h0 = h0 plus32 a;
h1 = h1 plus32 b;
h2 = h2 plus32 c;
h3 = h3 plus32 d;
h4 = h4 plus32 e;
h5 = h5 plus32 f;
h6 = h6 plus32 g;
h7 = h7 plus32 h;

"end of outer loop (when we do it)";

"Obviously I can also do this part, but right now I am going cross-eyed";
"Produce the final hash value (big-endian):
digest = hash = h0 append h1 append h2 append h3 append h4 append h5 append h6 append h7";

Notice that every operator specifies the input and output sizes. For example plus32 means add two 32-bit numbers to get a 32-bit result, with wrap being an error (this probably means, by the way, that the last few plus32s should be plus32_with_overflow, since SHA-256 actually expects overflow for these operations). So far we only deal with unsigned quantities; some “overflows” are actually expected when dealing with negative numbers, so that would have to be specified differently. Also, I didn’t deal with the size of constants, because I wasn’t sure about a good notation, though I am leaning towards 23_8 to mean an 8-bit representation of 23 (subscripted, like in TeX).

Because Stupid really is stupid, it should be very easy to write static analysis code for it, enabling checks to be omitted sometimes – for example, the fact that we only subtract 1 from pad_bit if pad_bit is non-zero means that we would not have to check for underflow in that case.

Anyway, I’m a bit bemused after writing a lot of rather repetitive code for the compiler, so I think I’ll wait for reactions before commenting further – but it does seem to me that this is a project worth pursuing. The compiler itself, whilst somewhat crude, particularly since it doesn’t yet do most of the checks I suggest should be there, is pretty small and easily understood: less than 1,500 lines of Perl and YAPP. I won’t bore you with the details, but if you want a peek, here’s a tarball.

Share This

Ben Hyde—What can’t go on won’t.

The New York Times has yet another piece on the question of walking away from underwater mortgages, this time an op-ed by Richard Thaler.  It finally references the excellent essay by Brent White on the current “norm asymmetry” between the mortgage holders and mortgage holders; i.e. one side ethics & roots while the other has only spreadsheets.  I wrote about that a while back.

These pieces all seem to presume that the underwater owners can continue to tread water.  That’s crazy talk.   These mortgages are fundamentally unsound.  A sound mortgage requires a few key elements.  It needs to be backed by appropriate collateral, and these aren’t.   It’s servicing must tap only a reasonable percentage of the holder’s income stream, and these don’t.  It needs to be reasonably liquid, i.e. that should the situation arise the mortgage holder can sell the instrument and/or the home owner can sell the house; and these aren’t liquid, not at all.

So what we have here is a standard bubble situation.  i.e. “What can’t go on won’t.”  Sooner or later these people will walk.  The only question is how much damage to their family’s economic status they take before they do.

I continue to think there is money to be made here.   An entrepreneurial opportunity: a business that facilitates the walk always.   At it’s core all it does is loan people money to fund their walking away; i.e. bit of legal cost, a lump sum to pay for rental deposits, maybe some moving expenses.  I wonder how many landlords are willing to let you put the 3 months rent worth of deposits on your credit card?

What I find fascinating about this idea is who it turns the question of borrower honor on it’s head.   For these walk-away enabling loans to work you need trust.  They are personal loans.  The business works because it accepts that you can trust somebody who walks away.  The business accepts that they are not dishonorable, but rather that they are pragmatic.  It works because by splitting the benefit of that pragmatism with them.  I love that.

Further I love that such a business would have to do what the bubble lenders failed to do.  It would actually need to know the customers.  If you wanted to set up such a business you’d need to have local knowledge of the customer’s actual situation.  You’d need to be able see through his lousy existing cash flow and recognize that if his income stream is stable and that as soon as his housing costs drop by a thousand dollars a month paying off this new loan is going to be straight forward.  Maybe you could hire all the loan officers who learned their trade in the years before the bubble.   Maybe this is business model to be sold to small banks where their local knowlege can be brought to bear.

I can’t quite capture it, but the key is in there someplace.  The original lenders didn’t bother to figure out who was trustworthy.  (They didn’t need to since they could offload the risk immediately.)   Now if somebody shows up willing to do that work they can profit from it.   Curiously, the longer an underwater home owner tread water the more you can trust him.

Shalin Shekhar Mangar—Saw a Kurkure advertisement on a website titled “No...



Saw a Kurkure advertisement on a website titled “No plastic in Kurkure”. ROTFL!

Nick Kew—Living with Maemo


OK, I’ve had the new pocket-puter a couple of weeks now, and apart from that keyboard I like it.  As predicted, I’ve come to terms with the touchscreen and find it easy to use (except for some web controls which can be hard to pick up: e.g. the volume control on the BBC iplayer).

Overall, I prefer the hardware on the old E71, with the obvious exceptions of the screen and camera where the N900 excels.  But the Maemo software is incomparably better.  Just to take one example, I want to connect to the ‘net using a wifi network where available but otherwise defaulting to the telephone network.  While Symbian requires a deal of faffing to do that, Maemo “just works”.

When I was contemplating the purchase, I asked on this blog what Maemo really is, and was assured that it’s a real Linux.  I can confirm that it is indeed that, and that I can install Linux packages through the Debian tools (apt-get et al).  I have yet to install gcc and a developer environment, but I don’t anticipate any difficulty with it.

Maemo is not stripped down to a toy: rather it takes a Debian base, and adds an alternative GUI, which is optimised for the small screen.  It’s intuitive and easy to use, and makes brilliant use of available screen space and the touchscreen.  Interactive applications toggle easily between fullscreen, fullscreen-with-toolbar, and thumbnail (minimised) with a consistent look-and-feel.  The web browser is a small-screen skin on gecko (firefox), and is not bad.  The mailer is positively nice, or will be when I figure out how to fix composition to get rid of pseudo-HTML: much better than some mainstream mailers I use, including thunderbird and to a lesser extent Mac mail.

One thing has me baffled: how do I bootstrap a password either for root or sudo?  After googling for a solution, I worked around it by installing a rootshell which gets me passwordless root powers (!), but that’s not the kind of hack to which I expect to have to resort.  /me shudders.

I’ve looked at Nokia’s OVI store, but I don’t see so much point to most of it when I have the whole repertoire of *X apps at my fingertips.  OK, having said that, I’m sure I’ll install some things: the radio player, for instance.  I installed a weather widget, but I don’t even recollect if that was from OVI or pre-loaded, and it’s only really a toy.  The only serious app I installed was the root shell, which seems to be a prerequisite for using apt!

One more slight niggle: on the E71, Nokia’s maps are nice, but Google’s are nicer.  On the N900 there’s no google maps: I can get them on the ‘net, but that loses the GPS functionality.  So it’s Nokia or nothing with the GPS.

But in a sense, all this is mere detail.  What I now have is connectivity from anywhere I can get the ‘phone network.  So I needn’t lose email, ssh, etc (and be fretting to get home) when I spend a day or two somewhere with no wifi available, whether it be in a technophobe house or up on the moors.  Yay!

Justin Mason—Links for 2010-01-23

Ben Hyde—Turning Off The Land Line

I switched my home’s phones to a cell phone back in August.  This post outlines what I did.

For the old phone line I had what I think was the cheapest service available.  The service was maybe 17$/month, but the confusopoly charges ran the bill up to about $34/month.  So a years service was costing me abour $400 dollars.  In addition to that I was buying my long distance service from yet another vendor so my total phone cost was about $450/year along with about 14 transaction events to handle the billing.

What I did was by a two pieces of capital equipment, a used cell phone and a clever widget that bridges from the cell phone to the existing in house phone wiring.  That capital equipment cost me about $100 total.  I then switched the phone number to Page Plus Cellular.   Page Plus Cellular is a Mobile Virtual Network Operator, or MVNO.  It stands on the physical network of Verizon Cellular.   While MVNO is the technical term for this, most people just say “Prepaid.”

Cell phone minutes are cheap on Page Plus.  When I set this up they were six cents each (if you buy them in $80 batches), now they are four cents each.  I estimate that the annual cost for the minutes we use is about $250/year, e.g. ~ $20/month.  I have to top up the account four times a year; so there are 4 transactions to handle/year.

So in the first year I’ll save about a hundred dollars have ten fewer transactions.  In next year I’ll save $200.

I think the quality is great, we occasionally get lousy connections but so far i haven’t had one were I couldn’t blame the counter party’s cell phone.  I guess it could count as a feature that I can take the home phone with us when we travel.

But over all this just is not that big a savings.

I appreciate at there are other schemes that use your internet connection.  I didn’t go down that route for three reasons.  I’ve had a hard time getting dependable quality with internet phone.  I didn’t want to get the two services entangled with each other.   It’s not clear if it would have been cheaper without taking on significantly more risk and support costs for me.

Because this cell phone doesn’t move I could presumably buy my minutes from one of the more marginal physical cell phone companies (say Metro PCS).  Suprisingly they don’t offer something cheaper.  But if something cheaper comes along I switch after spending down my current basket of minutes.

Yoav Shapira—Product review: Foursquare

Foursquare is a product / service that comes in the form of a web site and mobile applications. I've been using their web site and primarily their iPhone application for a few weeks now.

It's so much fun ;) It reminds me of Farmville in its addictiveness. It's, "Designing for Social Traction" (@bokardo's presentation, worth reading the whole thing) done well.

I've been using it every day, several times a day, since I got it. I probably use it as much as Facebook, or at least in the same neighborhood. It's just fun ;)

I love the badge aspect, just like I loved the various Farmville rewards. I even kind of like that they don't tell you how to get each badge: some of them are hard to discover. This can get frustrating or annoying, for sure, but it's a fine balance overall.

If you have an iPhone or other phone they support, try Foursquare out ;) It's cool.

Yoav Shapira—Product review: iPhone 3GS

Last month, after several years of sticking to my old trusty simple slow Nokia phone, it broke. I had to get a new phone, so I decided to finally get an Apple iPhone 3GS.

I am absolutely thrilled with it so far. It's been a pleasure to use, and it's changed my life in meaningful ways.

There have literally been millions of reviews. I'm not going to add much at all. If you haven't played with one, you should.

The speed difference over Alli's first-generation non-3G iPhone is significant, easily shown qualitatively.

These are the applications I use regularly:

- Tweetie2 for Twitter (I like it better than both Twitterific and TweetDeck) and TweetPhoto
- Facebook like everyone else
- Foursquare (separate blog post coming on this fun puppy soon)
- Sonos (to control our new speakers)
- Pandora
- Shazam
- OpenTable, Yelp
- Kayak
- DailyBurn
- The Weather Channel
- ESPN ScoreCenter
- Wine Guide

Some games: Paper Toss, Storm8's Racing Live, Tiger Woods golf, Sudoku, Tap Tap Revenge, Dark Nebula.

I'm too lazy to link to all of the above. They should be easy to Google, and all but one or two are free.

My only complaint about the iPhone is the battery life. I wish it were longer. It works OK for a day of normal use, but I do have to charge it every night.

Overall, I love this thing. It's just awesome.

Yoav Shapira—Restaurant review: Post 390

Last night Alli and I had dinner at Post 390, a newish bar / restaurant in the Back Bay. We had a blast overall and definitely plan to come back over there in the future, with friends.

Post 390 doesn't claim to be much: a new, stylish restaurant with "comfort" food in a convenient location. They deliver on all of that, and then some.

The place is good-looking in terms of decoration and layout. It's fun to walk around, check people out, enjoy the fireplaces and the bars. Really nice.

The atmosphere was positive and high-energy. A lot of people, a lot of talking, buzz, smiles, people having fun. Really nice.

The food was delicious. I had oysters grilled in their shells with linguica as my appetizer, while Alli skipped appetizers and waited for her filet mignon steak, which was one of the best we'd had in a while. My entree was the day's special, a baked cod fish with deviled crab meat and asparagus. It was perfectly executed, white, tender, succulent, delicious, and of course no need for a knife at all.

We paired these with a 2004 Cain Concept, one of my favorite recent vintage wines (as I've mentioned in the past: once, twice). This Napa red is amazing, and I'm going to try to buy another bottle or two.

Service was very good: attentive, friendly, polite, but not rushed for the most part. Nicely done, good attitude.

There was one major incident, however, which I want to share. When Alli got her steak, which came with watercress and onions, she started eating it. It was perfectly cooked, medium rare. When she dug into the pile of watercress on the side, not the steak, a bug came out! It was tiny, maybe 0.1 inches in length. It was alive and moving around a bit.

Naturally, we called the waiter, and gave the plate back. We looked around carefully at our other plates, the table, and the environment, and didn't see any other bugs. A total of four people from the restaurant came over the apologize and ask if we were OK, including the manager.

They fired up another steak right away, delivered it, and it too was perfectly cooked. (A nice achievement for a thick medium-rare filet mignon.) And they did not charge us for the steaks at all.

So they handled it very well. Both of us were shocked, given the quality of everything else in the restaurant. It seems like an unlikely event, and we still plan to return and enjoy Post 390 some more in the future.


Yoav Shapira—Book review: Silos, Politics, and Turf Wars

Last week, the same colleague who gave me Patrick Lencioni's other book, The Five Dysfunctions of a Team, gave me another book to read. It's called Silos, Politics and Turf Wars, and this one was also mildly interesting but not great.

This book talked about how when different parts of a company or organization do not march to the same beat, silos can form. More specifically, if not everyone is moving towards the same clear, shared, understood goal, and ideally using the same objectives and metrics, then silos will emerge eventually.

Because, according to the author, when people start working towards different goals, eventually they go into some form of competition with each other for more resources, higher priority, etc. Less collaboration happens, less work just done in the interest of the entire company, as opposed to the function / department / group the person is in, and his or her own career. And that's how silos emerge.

Like his other book, the author spends most of this one in a fictional scenario, illustrating silo-related issues and how to overcome them. And again, the last 20% or so of the book (just a ballpark estimate) is spent in a more conventional academic style, laying out the author's findings and recommendations.

I actually like some of the recommendations a lot. Pick an overall thematic goal for the whole company that is qualitative (we'll get to metrics later), time-bound (so people can see an end, don't despair, work towards the goal etc, but not start discarding it because it's always critical), shared by everyone (including the entire management team, so it's not just one person's problem) and serves as a good organizational rallying cry.


It's not always easy to come up with such a goal, nor with the 4-6 defining objectives that will let us get to that goal. But it's a worthy exercise.

Overall, I do like the message, but I thought that (again) the book was really verbose and repetitive, to the point where it got annoying to read.

Claus Ibsen—Camel in Action - Half way through

I thought it would be a good moment to sit down and write a little blog entry about how it goes with the Camel in Action book. What better moment to do that than sitting in Heathrow killing a couple of hours before before departing to the states for a business trip. I am exited as I get the chance to meet my fellow and talented co worker from the Fuse team.

It has been almost 7 months for me on the book in terms of writing. I started at full speed on the 4th of July which must be a date that our friends from the states can remember. A couple of months in the process Hadrian and I realized that we need to adjust the team. We invited Jonathan to join us which works great. He have already produced 2 full chapters and is up to speed on the third. This is impressive as he got the honor to rewrite the first chapter which is the hardest to write. All three of us have had a go at this chapter.

In terms on my end I have focused on some of the more technical heavier chapters which suits me fine. It's after all me who is responsible for some of the major overhaul, improvements and new features introduced in Camel 2.x. Yes I am sure to brain dump evert piece of information so you are sure to get the most up to date material in the book. For example I have amended the chapters several times during work on Camel 2.2 which have introduced some new features and improvements.

Two weeks ago we had our 2nd review completed. The review process is controlled by Manning and they do a great job finding a great bunch of people for the review panel. Its totally anonymous so we do get the word from the hearts. It's a mixed group of Camel experts, end users and authors whom have don't know Camel. I will get to what this 2nd review have had as impact on the book in a moment later.

We have also started the first copy editing process which means a professional editor from Manning is going over each and every chapter word by word, line by line, and checking that all add up. This is a remarkable work they do. We could start this work already since the 2 reviews have shown us that we are on the right track and the content so far in the book is great. Readers of my chapters will be glad to know that after the work of the copy editing the grammar and the flow in the language will be enjoyable to read. I am sorry but english is only my 2nd language.

Another impact the review and also you the readers of the book have had is that we have been in talks with Manning to extend the book with 2 new chapters. This is possible to do since the book have been received well so far. Thanks for all of you who have shown interest in the book. And especially you who have bought it. Its all thanks to you that we can include 2 new chapters to make the book cover Camel really really well.

We have already devoted one of the new chapters to cover concurrency, threading and the stuff associated with this. Its my next chapter when I have finished what is to become chapter 12 (monitoring and management).

As there are still one chapter to be decided what it should cover we have though to ask you. Of course we do have a couple of topics in mind but we would rather leave it totally open. So please if you have ideas then come along the Manning Author Online forum where we can discuss this.

At current time we have produced 7 chapters in a total of 250 pages which is available at the Manning MEAP program. That must be the half of 14 chapters, hence the block title.

Claus Ibsen
Heathrow, Terminal 5, B gates

Emmanuel Lecharny—Fixing performance issues in your application

I was lucky enough to assist to a presentation done by Kirk Pepperdine last Wednesday. I won't present Kirk (you can check his résumé here ), it's enough to say that he is well known as a java performance Guru.

The presentation was conducted in two parts, the first one was a Q & A session, the second part was about debugging live an application that was carefully slowed down by introducing bugs in it.

Preamble
Let me talk about the application first : the guys who invited Kirk (AFAIK, it was an 'extra', provided as a bonus following an internal presentation they paid for. Thank to Xebia for having shared this presentation with external people) have prepared the application (the well known and useless Pet Clinic) by adding some of the anti-pattern they have met when doing consulting for many of their clients. Kirk had no clue about the bugs that have been injected.


Q&A
It was asked us to provide some questions when we registered, and Kirk answered them extensively. Here are some of the Q and A I remember of :

Q : Which GC should we use ?
A : The one which works. Usually, just focus on your application, you'll not need to pick a specific GC .

Q : What do you think about other languages like Groovy, Scala, wrt performance ?
A : It's irrelevant. Picking a language to develop your application should not be a matter of performance only. 'Whatever works' is the way to go. If you want to build an application fast, and if it's not expected to be heavily loaded, then even php is a good choice.

Q : What tools to you use to check for performance bottlenecks
A : A few : a system monitor, HP-JMeter, an VisualVM

Q : How can you best write an application which depends heavily on concurrent code ?
A : Don't use any synchronization. There are ways to avoid synchronization, based on state machine theory. (pointers needed here ...)

Q : What is the ratio of GC problems you have to deal with when working for a client ?
A : Around 40%. Assuming that I'm the last hope for many of my clients, it's may be an irrelevant number. Usually, people successfully fix easier issues themselves.

Q : Do you check the code when you start tracking some performance issue ?
A : Never. I'm not a coder, I don't have time to go through thousands of line of code. I just spot the place in the code which has problem.

Q : Managers don't let me adding some traces in the application on production… What should I tell them ?
A : Managers know the difference between a slow application and a dead application. Do what you have to do, or find another client. (in other words : you don't cure cancer with aspirin...)

Q : Which profiler do you use, or prefer ?
A : YourKit: it's simple and efficient.

The most interesting presentation I have seen in years. I actually learn things in an area I thought I was efficient…

Live demo

What was the crux about this part was the processes Kirk adopted to point out the problems in the code.

Step 1

First, he asked for a baseline to work on. Namely, you should have a scenario which demonstrates the kind of real performances issues a real client perceives. Improving some application which is already perceived as working is a waste of time, energy and money. Without a base line, you also have no way to check that you have improved the application. Last, not least, define your expectations, otherwise, you won't meet them ! So here, the team has defined a JMeter test, and defined the expected response time for each page.

Step 2

Second, run the baseline scenario, and measure the response time, plus a few other counters :
- CPU (users and system)

That's it, nothing more. Here, the code has not been even checked. The only thing Kirk did was to remove all the tuning for the JVM, like the memory min and max size, and every other premature configurations.

The rational is that you have no idea at this point if those parameters have any effect, but they for sure have an impact, probably polluting the results.

Looking at the CPU consumption and response time (90%CPU, around 5% system), with an average of 10s per page, it was clear the application has a performance issue, but there was no clue about what's going on yet.

Step 3

Then he checked the way the GC was running. He added some instruction on the JVM setting to generate some GC traces, run the application for a few minutes, then checked the logs ("You have to be patient ! Memory leaks may take a while to be noticed.")

A quick look at the metrics shown that the GC was eating 13% of the whole CPU. Way too much.

Step 4

Kirk now decided to connect to the running application, using VisualVM. The idea was to check the way objects were allocated. After a few minutes of tests, the allocated objects graph shown that we have a linear increase over time, which means a memory leak.

Finding the memory leak was a matter of minutes : find an application object (no need to check a Java object like byte[] or String : "Java collection objects don't leak…"). What is the key for Kirk is the number of generations an object survived : the higher this number, the more likely this object is leaking. Very new to me.

As a side note, he also said that many of the existing tools don't provide this generation number. They base the detection of leaking object on delta between snapshots. Not convenient.

Then you can check where the object was allocated checking the stack trace, an now, look at the code.

At this point, the important lesson is : just look at the code when you know in which method you have a problem.

(the application had another memory leak he found too, using the very same approach)

Another lesson : he asked to remove the caches in the code, instead of blind-guessing what was wrong with those caches (they were leaking). His moto was : "Why would you optimize your code by adding cache when you have no idea about what's going wrong in your code ?"

Step 5

Once this initial problem was fixed, he re-runs the test, and he saw that the CPU was not going any upper than 50%. Very wrong when the response time was still awful. In this case, the System CPU was high (the ration between user and system should be around 5-10%/90-95%).

What does it mean ? Contention. How to find where we have contention ? Easy : generating a thread-dump.

No fancy profiler, no long source reading, just a thread-dump.

It immediately shown that only two threads were used to deal with 50 concurrent clients requesting the application.

A quick tuning on Tomcat (number of threads accepting requests), and we moved to the next step.

Step 6

One last measure shown now that we had much better performances, but with a very high CPU system usage : around 20%.

Same action here : thread dump, look at the blocking threads, go to the portion of code where the thread was waiting. A bad thread.sleep( 100 ) was found in the code.

And it was over for the demonstration : 2 hours to fix bugs that would have took days and days for most of us!

Conclusion

In two hours, he made the application running way faster, simply by using a couple of tools, and without reading the code.

Impressive.

Thanks to Kirk Pepperdine, Cyrille Le Clerc and Xebia !

Follow up
I have forgotten a few things :
  • at some point after stet 4, GC went up to 65%. Kirk suspected that some part of the code was calling the GC. You bet !
  • after the presentation, Kirk said that the very first step is really to catch all the GC problems first, as they will probably hide other problems.

Edward J. Yoon—BMW Z4 Wallpaper (1680x1050 size)


This is my car at Jungdongjin beach on the East Sea in Korea. It's made as a 1680x1050.

Edward J. Yoon—Lamborghini Muira car review – Top Gear – BBC


Edward J. Yoon—Google's Wonder Wheel (원더휠)


언제 추가된건지는 잘 모릅니다만, 구글 검색옵션, Show options 에 Wonder Wheel 이란게 추가되었더군요.
연관 검색어들의 관계를 시각화 해놨습니다. 검색 UI의 진화라고나 할까..

누구나 테스트할때 그렇듯이, 저도 제 이름을 검색했습니다. 제 이름에 연관 검색어로 Apache Hadoop, Apache Software Foundation, 유사이름 몇개가 나오더군요. 사용자 질의어가 무엇이며 그 주변에 연관 검색어들은 어떤것이 있는지 훌륭하게 보여줍니다.



Apache Hadoop을 클릭해봤더니 연관검색어로 Hadoop 설치/설정/다운로드 등등이 나오네요.



또, 이중에서 Hadoop 설치를 클릭했더니 ssh나 java homepath 관련 검색어들과 우측에 자동으로 검색된 내용들이 나옵니다.



그냥 현재의 질의어에 대한 연관검색어를 list로 보여주는것이 아니라,
각 연관검색어의 연결 고리를 시각적으로 보여주니 ..

게다가 보통 우리는 원하는 정보를 최정적으로 얻기위해 서너번의 재-검색은 기본입니다.
그걸 굳이 다시 입력창에 타이핑하고 검색하고 할 필요없이 이렇게 one-click으로 찾아보게 되었으니 ..

후우, 재밌는 형태인것 같고 좀 짱인것 같네요.
이런걸 사용하다 한국 포털사이트를 가면, ..
클릭하는 족족 '새 창'으로 계속 뜨니 뭐하자는 플레이인지..

Jeremy Quinn—Light at the End of the Spiral [Flickr]

sharkbait posted a photo:

Light at the End of the Spiral

It is a long way to the top of The Monument.
They even give you a certificate if you make it back down in one peice.

Justin Mason—Links for 2010-01-22

Ceki Gulcu—A poor man's conditional logging configuration

It is often desirable to keep the same logging configuration file during development and testing. (I use logback as my logging framework.)Ideally, one would check whether the application was running on the production environment and enable parts of the configuration file accordingly. Unfortunately, logback does not support conditional (if-then-else) configuration statements.However, it supports

Chris Pepper—Pen v keyboard v Newton v Graffiti v Treo v iPhone

A nice comparison of data entry performance. Typing on the iPhone is okay, but I much prefer a normal USB keyboard. I never did much entry using Rosetta or Grafitti. The Treo keyboard was decent, but not great. The BlackBerry disadvantage: noisy during meetings!

That said, my worst data entry method is definitely paper -- serious readability concerns! ;)

http://hardware.slashdot.org/story/10/01/22/0812222/Pen-vs-Keyboard-vs-Touch-vs-Everything-Else

Bryan Pendleton—Good Java trivia quiz

This Java trivia quiz is actually quite good.

The first pdf link is to the trivia questions.

The second pdf link is to the answers.

I got about 60% of the answers correct. I completely nailed the JDBC section and the history section, and did OK on the JDK 7 question and the concurrency, IO, and collections questions, but completely bombed the Swing questions.

Ben Hyde—Cost Advantage

I don’t like what I just figured out.

Today’s mail included a scary looking letter from my health insurance company demanding that I immediately call about a billing matter. So I called. I typed in numerous long strings of digits (dates, account numbers, event numbers, letter numbers, zipcode) and then spent an hour listening the Blue Danube waltz and numerous alternating assurances that my call was important and that if I pressed one I could leave a message (but that didn’t work). Finally the agent came on the line and, you knew this was coming, had me repeat all those long strings of digits again and a few other facts. Then she asked what my call was about. I read the the letter to her. She then asked what the event was. (i.e. broken arm) She then asked where it happened. (school) And finally if there was any other insurance carrier they who might be involved. (no).

Ok. So what did I figure out? Can you see it?

Their cost for that call was about 12$. So for them they might as well send out the scary letter for every single claim over say 120$. Maybe they can catch the doctor claiming something that didn’t happen, maybe they can catch the chance to shift the cost to another insurance company. My cost. Well that depends on what you think my time is worth. And, feel free to add a bit for my pain and suffering. But they don’t care about that; and the only feed back look that I might use to reduce this goes all the way to Washington.

This kind of robo-calling is only going to get worse. Most of their cost is the 3 minutes of labor their human agent expended – but really there was no need for a human on their end.

The insurance company will do this for every transaction. The credit card company will do it for every transaction that is the least bit interesting (large, out of town, etc.). The airline will do it on the off chance you might admit your not going to make the plane allowing them to resell the seat. etc. etc. In all these cases the cost for them is so very low and the cost to me … well who cares about that?

Shane Curcuru—Why don’t we switch to Dvorak?

I was following a @monkchips link to IgnoreTheCode about smartphone touch keyboard design, and thought about a bigger issue.

Why do we still use QWERTY?

Sure, it’s embedded in tons of hardware – both plastic/metal and muscle for many of us – but isn’t it a good time to switch? The immensely fast pace of learning for younger people, plus the soon-to-be outpouring of new keyboard and input styles seems like we could actually start the switch now.

Not only would touch-screen keyboards be simple to switch, but form factors and tactile feedback are changing for more and more of our data entry. This would be a good time for existing QWERTY folks to start switching their muscles to think AOEUI instead, since we’re already having to adjust to how glass-screen typing feels anyway.

Hmmm, that’d be a great geeky replacement for ROT13 too.

Share Shane's Sayings Technorati Google Bookmarks LinkedIn del.icio.us Facebook co.mments Live Reddit Slashdot Identi.ca Yahoo! Buzz

Ceki Gulcu—301 redirects with Wicket

Sometimes a web-page needs to redirect to a new address. A 301 redirect means that a page moved permanently, which in most cases is indistinguishable from a 302 "moved temporarily" redirect. Search engines however react differently to a 301 response than to a 302 response. If a page has moved permanently, sending a 301 will cause the search engine to update its index to the new page. For "

Howard M. Lewis Ship—In the Brain of Ben Gidley: Mar 23 in London

SkillsMatter Logo Along with my upcoming Tapestry training at SkillsMatter, there is now a session w/ Ben Gidley on March 23: In the Brain of Ben Gidley: Tapestry 5 In Action. Ben will be talking about project SeeSaw, a video-on-demand service built on top of Tapestry 5. He'll be going into depth about converting Struts developers to Tapestry 5, dealing with large volumes of users and large numbers of complex pages, and many of the other factors of bringing a premier web application to deployment.

Edward J. Yoon—가상-공간으로 출근하는 시대


최근 나는 개인 활동적인 '온라인 협업'이 예전보다 더 잦아지고 있다.
이때 보통 Google Docs, Gmail, IRC, GTalk를 사용하는데, ...
아주 환상적. Output이 쭉쭉 나오는 느낌. ㅋ

현재 회사에서 일하는 방식과 도구들.
예를 들면, JIRA, 사내 메신저, 전화기, MS outlook 메일 그리고,
답답한 의사결정과 협업부서/사원간의 뻔한 회의, ..
이 모든것들이 Google Tools에 기반한 On-line 협업으로도 충분히 커버가 된다는거.
아니 반대로, 이제 그런건 너무 구식으로 느껴지기도. ㅋ

아, 물론 모여서 움직이는게 훨씬 빠르겠지만,
한국 정서적으로 비대한 회사는 특성상 빠를수가 없으며,
뭔가 로봇처럼 형식적인 절차와 프로세스를 중시해야하는 시스템안에서는
도저히 Performance가 나올 수 없지 않겠나.

지구의 모든 생명체들을 자세히 보면 거대한 리듬과 싸이클에 맞춰 모두 열심히 움직인다.
뇌가 없는 생선들도 유유자적 헤엄만 치는것 같지만 그들 나름 출퇴근을 하며 생존을 위해 살아간다.
이제 곧 IT 기술이 세상 한가운데로 진입하면,
그 리듬과 싸이클에 걸맞는 신규 생명체들이 태어날걸로 본다.

이쯤되니, 나는 IT 산업 직장인들의 출퇴근은 그 자체가 부질없어보인다.
완전 기름낭비, traffic 혼란 가중만 시킬뿐..

결국 그렇다면, 서로게이트라는 영화처럼..
본체는 집에서 방콕하고 Cyber space에 나의 Avatar가 대신 출근하는 날도 머지않은것 아닌가. ㅋ

놀랄것도 없는게 지금도 존재한다. 전문블로거들이 일종에 그런 사람들이다.
App Store나 Indi-Software 같은 것들이 개발자에게도 직장을 만들어주고 있다... 훗

(적고보니 좀 .. ㅋㅋ)

Grant Ingersoll—Lucid Imagination » The Seven Deadly Sins of Solr

Props to Jay Hill on an excellent article on things to watch for when setting up Solr: Lucid Imagination » The Seven Deadly Sins of Solr.


Adrian Sutton—Three Types of Ant Scripts

Bryan comments on the three types of ant scripts:

In my experience, there are three types of Ant scripts that you encounter "in the wild":

  • Small Ant scripts, generally Java-only, which can use most of Ant's default behaviors and are clear and simple. A lot of open source build scripts are this way.
  • Serious commercial Ant scripts written before macrodef and import became available. These are generally impossible to understand and evolve, and the reality is that a small cadre of Build Wizards keep them running. Such systems often involve a substantial number of custom Ant tasks.
  • Serious commercial Ant scripts written to use macrodef and import. In my experience, the need for custom Ant tasks drops way off with Ant releases post-1.6.

This really does ring true to me. Ant scripts can fairly quickly become unwieldy and difficult to work with if you aren’t using import and macrodef, but with them you can achieve so much more without the complexity getting out of hand. They won’t absolve you of the need to properly understand ant and the declarative paradigm it wants you to work with, but it’s much more approachable.

If I can ever get someone to add the optional dependencies for scripting support on our build servers1 I may well find they help a lot too.

1 – which happen to be Windows boxes, behind a firewall on the other side of the world from me, so not easy to make remote changes on ↩

Matthias Wessendorf—Flexible ADS – Combining popups with ActiveDataService


In this post I showed how to create a simple counter, based on the Active Data Service from ADF Faces. This facility works out of the box with components like:

  • activeCommandToolbarButton
  • activeImage
  • activeOutputText
  • table
  • tree
  • All DVT components

However not always this is enough. There are a bunch use-cases where you want more!

One scenario could be that you have a long-running task in the background and when it is finished, you want to inform the user. Ideally you want to show him a popup which contains some information regarding the scheduled task. Let’s take a look how that is possible with ADF Faces.

As the user is most-likely not interested in watching a ticket/counter, we need to use a (hidden) trick. We use an invisible <af:activeOutputText>, which has one child element (<af:clientListener />):

...
<af:activeOutputText id="activecomp"
  value="#{counterBean.state}" visible="false">
    <af:clientListener type="propertyChange" method="activeDataCallback" />
</af:activeOutputText>
...

The <af:clientListener/> is interested in a “propertyChange” event, which get’s triggered when a property of the component changes. So every time an update is send to the client, the invisible component changes one property, here it is the “value” property. So we leverage that “trick” to call our custom JavaScript activeDataCallback() function when an update reaches the client ;-)

As you can tell from reading the JavaScript-Documentation the client-side event class has been modeled after the JSF ValueChangeEvent, but it is slightly different.

So let’s take a look at the required JavaScript:

...
<af:resource type="javascript">
 activeDataCallback = function(event)
 {
    showPopup();
 }
 showPopup = function()
 {
    var popup = AdfPage.PAGE.findComponentByAbsoluteId("demoPopup");
    popup.show();
 }
</af:resource>
...

In this example the callback is not interested in a property of the event argument. It just want’s to show a popup component, which is nested inside of the page:

...
<af:popup id="demoPopup" contentDelivery="immediate">
 <af:dialog closeIconVisible="false" title="Appointments!"
     visible="true" id="d2">

 <af:outputText value="Real content would go here" id="txtBox"/>

 </af:dialog>
</af:popup>
...

The result looks like this, once the active data arrives the client:

Ortwin Glück—[Code] Deutsches Gentoo Buch (PDF)

Soeben ist ein Buch über Gentoo frei im PDF Format erschienen.

Gunnar Wrobel
Gentoo Linux - Installation - Konfiguration - Administration
2008 Open Source Press

PDF Download

Christian Grobmeier—Rewire Propellerheads Reason 4.0 to Logic Pro 8

Rewiring Reason is a nice feature – you can use all the Reason synths, effects and instruments in Logic Pro (it should work even in Express Edition). Basis is the so called Rewire protocoll which has been invented by Propellerhead before a good while.

Basically its pretty forward to wire both application together. You just need to know that one of the apps should be the Master, while the other is the slave. Usually you have Logic Pro as the Master, from which you control everything and Reason only as some kind of big effect box.
To start with it, please open Logic Pro and afterwards Propellerhead Reason – NOT vice versa. The master needs to be opened first.

Then create a new track in Logic. The type is “External MIDI”.

In Reason, step up to highest hardware component, the hardware device. There is a button called “Adv. Midi Device”. Click it.
In Logic you are now able to choose Reason in the Library. Just select your “External MIDI” Track and open the Library. You’ll see a folder called “Reason” and then several instruments, depending of what you created. I have chosen one called “Synth”.

Now the signal should go into Reason and do things, but you need to wire it back into Logic to hear it. This can be done with a new track in Logic, an AUX track. You can create it in the Mixer view in “Options > Create new Aux”. Once done you are able to choose the channel which comes from Reason in Logics “I/O” section. I have chosen R/Cha 9/10. If you choose the stereo sum: this already works, but if you want to separate between different channels you need to patch the reason components. I wanted to use the single outputs, so, back to reason.
In Reason you can choose in Options that you would like to view the Rack from the back.

Choose the instrument you want and patch two cabels from the for example master out in the hardware controller on top. The hardware controller finally decides which channels you can hear in Logic. Because I have choosen 9 and 10, my stereo out from the instrument goes into the hardware device in slot 9 and 10 too.
Once done, this is finished and you have successfully rewired Logic to Reason and vice versa. Now your creativity can begin and learning start :-) Enjoy!

Bryan Pendleton—The Secretary of State on Internet Freedom

Ms. Clinton's speech on Internet Freedom is now online here.

Justin Mason—Links for 2010-01-21

Ross Gardler—Software Developers for Haiti

Some time ago I posted “Does it take a disaster to understand the power of open development?” Unfortunately it is now time to revisit that post with a call for software developers to help Haiti.

The Sahana Software Foundation are looking for additional developers to help with Sahana, a disaster management tool. They need to continue to build an information portal that is seeing increased interest and usage to assist organisations responding to events in Haiti. Information on the work underway can be found on the Sahana Haiti response Overview page.

The skills they are primarily looking for are (you don’t need them all, any will do):

  •  Python - the main development for SahanaPy is Python (we’re not using PHP for this instance)
  •  web2py - Sahana uses the web2py enterprise framework for SahanaPy (I’m told it is fairly easy to learn if you’re used to Django)
  •  OpenLayers
  •  jQuery

To find out more and offer your help please jump onto #sahana on freenode. Please give as much or as little as you want.Some of the core devs have been working really long hours the past 5-6 days, and any additional resource would assist us greatly, particularly as the effort is really starting to get some interest, traction and coverage, which leads to more and more feature requests.

Thanks for reading this far - please consider giving a few hours of your time to help out, if you can.

Bryan Pendleton—New releases in the web world

Seems like this has been a busy few weeks in the world of web development:



Meanwhile, although it's not really web-related, the proceedings are now online for last fall's LLVM/Clang developer's meeting. LLVM is now used for lots of things, with ActionScript/JavaScript being just one of many worlds that are involved with LLVM. Think you are doing some serious parallelism? Check out this release from a team that has used the LLVM architecture to build a program that runs on 120,000 processor cores! (Actually, according to these notes, they are now 50% higher, passing 180,000 processor cores!!!)

Luciano Resende—Articles: SCA Articles from IBM WebSphere Application Server Feature Pack for SCA

IBM DeveloperWorks site has published a series of article on WebSphere Application Server Feature Pack for SCA which is based on Tuscany. Most of the SCA related contents should apply to Tuscany too except some of the QoS and deployment supports.

Exploring the WebSphere Application Server Feature Pack for SCA: Part 1: An overview of the Service Component Architecture feature pack

Exploring the WebSphere Application Server Feature Pack for SCA: Part 2: Web services policy sets

Exploring the WebSphere Application Server Feature Pack for SCA: Part 3: Intents and policies available in the SCA feature pack

Exploring the WebSphere Application Server Feature Pack for SCA: Part 4: SCA Java annotations and component implementation

Exploring the WebSphere Application Server Feature Pack for SCA: Part 5: Protocol bindings for Service Component Architecture services

Exploring the WebSphere Application Server Feature Pack for SCA, Part 6: Using Spring with Service Component Architecture

Exploring the WebSphere Application Server Feature Pack for SCA, Part 7: Using Atom and JSON-RPC for Web 2.0 support

Aaron and Jenny Farr—Home birth 101

Because home birth has recently fallen out of fashion, most people are mystified by the whole idea. I’ve had mothers of several children ask me, “Well, what do you do at a home birth?”

Home birthing isn’t rocket science. After all, hospital birthing has become the norm only in the last 60 years when anesthesia became widespread practice. It’s fascinating to me that in a short 60 years, our society has lost touch with something that was considered “normal” since time immemorial.

Of course, home birthing isn’t for everyone. Only about 10% of US women home birth today. Most women feel more comfortable in a hospital, and so those women shouldn’t attempt a homebirth. Other women trust that birthing is a natural event and do not want modern intervention to stall or disrupt their empowering transition into motherhood. Hospitals have a lot of fancy tools, and they are encouraged to use them. Things like fetal monitoring, inducement drugs, and pain relief are all interventions that home birthing mothers are quite happy to go without, and in fact, home birthing mothers see those tools as a threat to having a healthy birth. And quite frankly, the evidence in on their side.

If a woman is planning a home birth, she has a lot more to prepare than if she were to have a hospital birth.

For starters, she has to more carefully consider who will be there. However, some mothers consider birthing completely alone. There is a new wave of women in the US who are having unassisted births. Many of these women have already had successful doctor or midwife attended home births, and so they feel confident in their ability to do it alone. Cultures from around the world, past and present, let women birth alone. This is not a recent trend just in the US. This is a movement for women who want to be completely in touch with their bodies, their inner strength and confidence to proceed with what they consider an act of normal human behavior. It is also completely legal. That aside, most home birthing women want a midwife or doctor in attendance. After choosing a qualified attendant, the mother can then choose birthing assistants. She should probably remember that home birthing isn’t a party, it’s a serious event. Limiting the amount of invited assistants is probably wise.

Once these people are chosen, it’s a good idea to give them all a set of expectations and jobs for the big event. Personally, I photocopied over a hundred pages from various birthing books and shared them with my assistants. After reading the material, all of them have thanked me for helping them more completely understand why I’m having a homebirth, and they now feel much more confident in my ability to have a baby at home. They also feel more confident in what they should and shouldn’t do when the time comes.

Preparing the birth space is also a big consideration. For example, a mother may ask things like, how private is the area? Are there enough electrical outlets? Will the room be warm enough? Is there enough room for the laboring mother to move about? Is there enough room for the attendant and assistants? Visualizing the event and adequately preparing the birthing space is crucial for a smooth delivery.

The most time consuming task is to gather all the necessary supplies. In this area, I think a home birth mother is much more prepared than a hospital birth mother. Gathering supplies ensures that the mother is considering her baby in the home space environment. She prepares every last detail so there is nothing to trouble her once the baby has passed through to our side. For the first few weeks after birth, the mother shouldn’t worry about getting diapers or extra washcloths, etc. The only thing the mother should be concerned with is who will bring her meals while she recovers, rests, and connects with her newborn.

Because my own list of supplies is exhaustive and may be different from another mother’s needs and wants, I will not include it here. Internet and book research will easily tune a home birthing mother to what she needs for the special event. And most importantly, midwives are acutely aware of what is required for a smooth home birth, and they will most definitely ensure that the mother is gathering all necessary items weeks before the due date. For example, my due date is in a little over three weeks, and I can easily have this baby tonight and feel wholly prepared and at ease. I’ve done my homework, and it feels good.

If you are considering a home birth, or would like to know more about this topic, I recommend studying the works of Ina May Gaskin. She wrote her first book in the 70’s called “Spiritual Midwifery”. If you are stranded at sea and about to give birth, this book will tell you absolutely everything you need to know. Every foreseeable complication is handled in layman’s terms. The other book I recommend is “Ina May’s Guide to Childbirth.” I’ve read many books on child birth and home birth, and these two were by far the most objective, researched, and complete.

Patiently waiting
Written by Jennifer Farr on Thursday, January 21, 2010. Permalink.

Bruce Snyder—How to Use Automatic Failover In an ActiveMQ Network of Brokers

Last week I tested a new feature in ActiveMQ 5.3.0 to support automatic failover/reconnect in a network of brokers. Besides adding this information to the ActiveMQ book, one person also suggested that I also post it on my blog for easier access, so here you go!

Folks familiar with ActiveMQ already know that a network of brokers allows many broker instances to be networked for massive scalability. Prior to the addition of this feature in ActiveMQ 5.3, if one of the brokers in the network went down, reestablishing a connection with that broker when it comes back up is a manual process wrought with difficulty. By adding support for failover to the network of brokers, any broker in the network can come and go at will without any manual intervention. A very powerful feature, indeed. Although this post is long, the outcome of the testing is well worth it.



The first thing to note is the topology for the network of brokers. I used a network of three brokers named amq1, amq2 and amq3. The attached diagram explains the topology, including the consumers and producers. amq1 and amq2 are stand alone with no network connector. amq3 defines a network connector with failover to amq1 and amq2. Consumers exist on amq1 and amq2. Producer will connect to amq3. To start with, I have only configured a uni-directional network connector in amq3. Later I will change the configuration for a bi-directional network connector.

Thanks to the ability to upload any file to Google Docs this week, you can download the configuration files for the three brokers.

The next thing to do is outline the steps I used to test out this feature. These steps were performed on Mac OS X (Unix) but could easily be adapted for Windoze. Below are those steps:

1) Open six terminal windows as defined below:
1a) Terminal 1 = cd into the amq1 dir
1b) Terminal 2 = cd into the amq2 dir
1c) Terminal 3 = cd into the amq3 dir
1d) Terminal 4 = cd into the amq1/example dir
1e) Terminal 5 = cd into the amq1/example dir
1f) Terminal 6 = cd into the amq1/example dir

2) Terminal 1: start up amq1 (./bin/activemq)
3) Terminal 2: start up amq2 (./bin/activemq)
4) Terminal 3: start up amq3 (./bin/activemq)

Thanks to the configuration of the ActiveMQ logging interceptor, you should see that amq3 makes a network connection to either amq1 or amq2. For the rest of these steps, let's assume that amq3 connected to amq1.

5) Terminal 4: start up a consumer on amq1 (ant consumer -Durl=tcp://0.0.0.0:61616)
6) Terminal 5: start up a consumer on amq2 (ant consumer -Durl=tcp://0.0.0.0:61617)
7) Terminal 6: start up a producer on amq3 (ant producer -Durl=tcp://0.0.0.0:61618)

You should see 2000 messages sent to amq3. The messages should be forwarded to either amq1. The consumer connected to amq1 should have received the 2000 messages and shut down.

8) Terminal 1: shut down amq1 (ctrl-c)

Note the logging that shows the failover taking place successfully. Let's test it to see if the demand forwarding bridge actually got started.

9) Terminal 6: start up a producer on amq3 (ant producer -Durl=tcp://0.0.0.0:61618)

You should see 2000 messages sent to amq3. The consumer connected to amq2 receives the 2000 messages and shuts down.

10) Terminal 1: start up amq1 (./bin/activemq)

11) Terminal 2: shut down amq2 (ctrl-c)

Again, the failover took place successfully. Let's continue just a bit further to see if it will continue to failover if I bring up amq1 again.

12) Terminal 4: start up a consumer on amq1 (ant consumer -Durl=tcp://0.0.0.0:61616)

13) Terminal 6: start up a producer on amq3 (ant producer -Durl=tcp://0.0.0.0:61618)

You should see 2000 messages sent to amq3. The consumer connected to amq1 receives the 2000 messages and shuts down.

This proves that the failover transport is supported in a network connector and it does work correctly with a uni-directional network connector. In addition to a uni-directional network connector, I also tested a bi-directional network connector. This only requires a slight change to the configuration of the network connector in amq3. In the amq3 XML configuration file, in the network connector element, add a duplex=true attribute. Below is the network connector element for amq3 with the change: 


<networkConnector name="amq3-nc" 
  uri="static:(failover:(tcp://0.0.0.0:61616,tcp://0.0.0.0:61617))" 
  dynamicOnly="true" 
  networkTTL="3" 
  duplex="true" />


With this minor change in configuration, the network connector is now bi-directional. I.e., communication between amq3 and whichever broker it connects to is two-way instead of just one-way. This means that messages can be sent in either direction, not just in one direction originating from amq3.

Below are the steps I used to test this specific change:

1) Open five terminal windows as defined below:
1a) Terminal 1 = cd into the amq1 dir
1b) Terminal 2 = cd into the amq2 dir
1c) Terminal 3 = cd into the amq3 dir
1d) Terminal 4 = cd into the amq1/example dir
1e) Terminal 5 = cd into the amq1/example dir

2) Terminal 1: start up amq1 (./bin/activemq)
3) Terminal 2: start up amq2 (./bin/activemq)
4) Terminal 3: start up amq3 (./bin/activemq)

You should see that amq3 makes a network connection to either amq1 or amq2. For the rest of these steps, let's assume that amq3 connected to amq1.

5) Terminal 4: start up a consumer on amq1 (ant consumer -Durl=tcp://0.0.0.0:61616)
6) Terminal 5: start up a producer on amq3 (ant producer -Durl=tcp://0.0.0.0:61618)

You should see 2000 messages sent to amq3. The messages should be forwarded to amq1. The consumer connected to amq1 should receive the 2000 messages and shut down.

Let's test the duplex capability of the network connector in amq3 now. To do this we'll send messages to amq1 and consume those messages from amq3.

7) Terminal 4: start up a consumer on amq3 (ant consumer -Durl=tcp://0.0.0.0:61618)
8) Terminal 5: start up a producer on amq1 (ant producer -Durl=tcp://0.0.0.0:61616)

You should see 2000 messages sent to amq1. The messages should be forwarded to amq3. The consumer connected to amq3 should receive the 2000 messages and shut down. This proves that the duplex feature is working. Now let's cause a failover/reconnect to take place and run through this same set of steps with amq3 and amq2.

9) Terminal 1: shut down amq1 (ctrl-c)

Notice the logging that shows the failover taking place successfully so that amq3 connects to amq2 now.

10) Terminal 4: start up a consumer on amq2 (ant consumer -Durl=tcp://0.0.0.0:61617)
11) Terminal 5: start up a producer on amq3 (ant producer -Durl=tcp://0.0.0.0:61618)

You should see 2000 messages sent to amq3. The messages should be forwarded to amq2. The consumer connected to amq2 should receive the 2000 messages and shut down.

Now let's test the duplex feature in the network connector.

12) Terminal 4: start up a producer on amq2 (ant consumer -Durl=tcp://0.0.0.0:61617)
13) Terminal 5: start up a consumer on amq3 (ant producer -Durl=tcp://0.0.0.0:61618)

You should see 2000 messages sent to amq2. The messages should be forwarded to amq3. The consumer connected to amq3 should receive the 2000 messages and shut down.

This proves that the duplex feature of the network connector works after a failover/reconnect to amq2.

This is a great addition to ActiveMQ that really improves the usability of a network of brokers. I already have some very large clients using this feature successfully, some of which are using a network of over 2000 brokers.

Hopefully these steps are clear enough to follow for your own use. If you need any clarifications, please contact me.

Sam Ruby—Content-Disposition

Yesterday, I collaborated with Joshua Peek on improving the parsing of the Content-Disposition header in Rack.  Content-Disposition is used on file upload scenarios.

The previous state was that rack used a simplistic regular expression that didn’t match either the RFC or what browsers actually sent.  What the new code does is first try to strictly follow RFC 2183.  When that fails, it tries to follow what browsers actually do.  And in this case, what browsers actually do is slap quotes around the file name, independent of whether the file name contains backslashes or quotes.  A notable exception to this is recent versions of Chrome.

Adrian Sutton—Apache Pivot

Thanks to a tweet from @bdelacretaz I discovered Apache Pivot today. It does indeed have a nice website, and the library itself looks great. It’s pitched as an RIA framework and mostly seems focussed on browser based deployment via applets. However, pivot apps can also be deployed as standalone applications which is where I think it’s most promising.

The killer feature as far as I can see, is the much richer set of components that are available, combined with a better set of customisation points such as effects, transitions, web queries and data binding. There are libraries to do most or all of this with Swing, but since it wasn’t designed for it up front, they tend to feel pretty clunky and “bolted-on”. Hopefully Pivot can avoid that.

Sadly, it does look a bit like you’re leaving Swing completely behind – if for no reason other than the look and feel is unlikely to match nicely with the Pivot themes without a lot of effort. While Pivot has a lot of components built-in, there are a vast world of custom Swing components around and it would be nice if you could leverage those.

Adrian Sutton—Better File System APIs

Dylan complained about the lack of testability with file system APIs, which reminded me of a neat looking library I stumbled across while looking at build systems: EntityFS. Among a bunch of other useful sounding things it provides:

File system APIs are backend-independent. File systems can be created in RAM memory, on Zip files, on file system directories, etcetera

Sadly it’s under the LGPL which makes it difficult if you happen to sell to big enterprises that are still somewhat scared of GPL type licenses, but they’re nowhere near as common as they used to be. I should note though that Holocene Software are offering commercial licenses for an undisclosed price.

Asankha Perera—HTTP client supporting REST, SOAP, Hessian etc for testing and Load testing!

The AdroitLogic UltraESB includes the ESB and SOA ToolBox - that can be used to test any payload over HTTP/S. Supporting all methods over HTTP such as GET, POST, PUT, DELETE, HEAD and OPTIONS, it allows a payload pasted into a text area, or read as a file to be sent to any HTTP/S endpoint, optionally authenticating the request with HTTP basic or digest authentication.


The user can select if the request should use HTTP 1.0 or 1.1, or send the entity as chunked or using 'Content-Length' encoding, if the '100-Continue' expect handshake should be used-or-not, if HTTP KeepAlive should be used or not, and if response  compression (i.e. GZip) is requested. If the request is over SSL, you could also turn off SSL Trust and Hostname verification for testing purposes - such as when using test certificates. Additionally, it can also use an identity certificate if 2-way SSL is requested.

To make this even better, the ToolBox also offers a HTTP/S Load test version of this same utility - that can be used to load test REST, SOAP, Hessian or any kind of payload over HTTP/S. This Load Test utility is a clone of the popular Apache Bench load test client - but much more advanced and stable!


The ToolBox also includes a TCP Dump utility - that can be used to view requests over the wire - in both HEX and Text formats. It additionally introduces support to capture a request into a file - which allows one to capture any kind of request message - even WS-Secured messages, Hessian binary messages etc, which can then be used with the JavaBench - to run a load test against your services.

For those who really knows what they are doing, and for example if you would like to test your ESB or SOA server for its support against malicious or malformed requests - there is a Raw socket client.


This allows one to hand-craft a HTTP request as you wish! The ToolBox also hosts a module that will start one or more Jetty servers with sample applications, which could then be used to test your ESB or SOA services. Starting multiple instances over different ports allows one to easily test load balancing and fail-over options of your ESB.

Finally the ToolBox allows you to start an instance of the UltraESB - the ultimate ESB that is simple to use, but supports ultra performance! Do read the full article "Getting started with the AdroitLogic ToolBox for the UltraESB" and follow through the end-to-end sample to get started.

Asankha Perera—Mock Services - be it either REST, SOAP or even any other!

The UltraESB allows the creation of Mock Services for almost any protocol over multiple transports. Thus you can mock REST, SOAP, Hessian, or even JMS, Email or File/FTP related services, or even other such services easily.

The article "Mock RESTfully with the UltraESB" shows how one may write Mock services for REST methods GET, POST, PUT and DELETE.. The mock service implementation can be written as Java as a class, code fragment or a Spring bean, or as any JSR 233 Scripting language code. However, you do NOT need to compile, bundle, deploy any of these - as the UltraESB does all that automatically! Just checkout the sample and see how a dozen lines of configuration can simplify your tasks!

The AdroitLogic ToolBox will allow one to test the mock services, as well as any other backend services. It includes sample RESTservices, a RFC 2616 compliant HTTP/S client that supports basic and digest authentication and 2-way SSL in addition to supporting GET, PUT, DELETE, POST, HEAD and OPTIONS for REST.

Bryan Pendleton—Checking items off the list

I've been spending a lot of time fixing bugs lately.

There's something very satisfying about fixing bugs. It really feels like you're getting something done, accomplishing something.

When I get into these moods, and feel like fixing a bunch of bugs, I get very methodical about it. I start compiling lists of bugs and reading through them. For each bug, I have a fairly rigid discipline:

  • Do I understand what the bug writer was describing?

  • Can I write a regression test which demonstrates this bug?

  • With the test in place, can I see where it's going wrong? Or step through it in the debugger?

  • Does my fix make the regression test pass?

  • Do the other regression tests reveal any downsides to my fix?

  • Is there anybody I should show this fix to?

  • Check in the test and fix, mark the bug resolved, and move on.



On the various bodies of software that I work with, there have accumulated a long list of potential bugs to fix. So I can afford to be rather choosy when I get into these moods. If a particular bug report starts to act up; for example, if it's hard to write a regression test, or if the obvious fix doesn't seem to work, or if other tests start to pitch a fit at my change, I just take a pass on this bug, put some notes in the bug report about what I tried and where I got stuck, and move on to the next.

In periods like this, my goal is to make a serious dent on the open bug list. I understand that I'm not necessarily fixing the most important bugs, or even the bugs which are most deserving of my time. I'm just trying to clear away the clutter and keep it from overwhelming me and the people around me.

So many bugs, so little time. It's just the way it is with software.

Asankha Perera—Proxy REST through the UltraESB

The UltraESB from AdroitLogic fully supports REST! Thus you can use it to proxy GET, POST, PUT, DELETE, HEAD, OPTIONS and TRACE compliant with the HTTP spec RFC 2616!

For an example on the configuration, refer to this article "RESTful Proxy Services with the UltraESB" which describes how a JBoss RESTEasy service can be proxied via the UltraESB - and demonstrates the above REST methods, and the automatic 'Location' header re-writing etc, which allows the Proxy service to fully hide the real service.

The UltraESB also includes a very useful utility - the ToolBox! It allows one to play with all the above HTTP methods with an easy to use and intiutive graphical user interface. Read this article on how to use the ToolBox as an ESB and SOA test toolkit - "Getting started with the AdroitLogic ToolBox for the UltraESB"

As the UltraESB ships ready-to-run samples, you do not need anything other than the 25MB download available from http://adroitlogic.org/download.html And this includes ALL thats required to test this sample, including a sample RESTful service, and the HTTP/S client from the ToolBox

Nóirín Shirley—Choosing Charities

Those of you who know me may have noticed that I don’t often respond to solicitations for charitable donations. Whether it’s a sponsored walk or a collection for malaria, I’m just not into “impulse buying”.

Those of you who know me better might know why this is. It’s not because I’m mean, honest :-) I’ve maxed out corporate Gift Matching programs with the employers who’ve had them (even when I was just an intern), and I hope to continue to do so. But I prefer to give in a “concentrated” fashion – rather than sprinkling my charitable donations across the vast spectrum of worthy causes, I choose a few each year that I really believe in, and do my best not to feel guilty that I can’t do everything!

When I lived in Ireland, particularly while I was still in college, I tried to “give global, act local”. I volunteered with various groups, from a local literacy program to the St John Ambulance. I even indulged in retail therapy for the St Vincent de Paul, both groceries and Christmas presents ;-)

Living in Switzerland, however, I’ve found that the attitude towards volunteer work is very different. Add my frequent travels (particularly in 2010) into the mix, and it’s just not a model that’s working for me any more. But my employment situation and the local tax regime mean that I have room to expand my financial giving – yay!

But I’m not sure where to put my money. We’re not talking millions, but I still think it’s worth spending time making sure it goes to something I believe in. That way, when I do have millions, I’ve already done the tough part :-)

Can you help?

To give you some background, I think if my giving had a “theme”, it would be this: Knowledge is Power.

I’m interested in improving access to knowledge, information, education. So one of my favourite charities is Literacy Bridge, which began with the idea “that the most effective approach towards ending global poverty requires empowering people with better access to knowledge”.

I’m also interested in preserving knowledge for future generations. Last year, for example, I sponsored the restoration and preservation of a collection of James Lind manuscripts, in celebration of dad’s birthday.

In general, I’m interested in charities serving those with the greatest need (not necessarily those who are easiest to reach), and I’m not looking for advocacy groups for one particular idea or cause.

What am I looking for?

  • Charities that understand the importance of inspiration. If I lived anywhere on the West Coast of the US, I’d already be a Friend of the California Academy of Sciences (and heck, I’m still considering it!). They understand that an interesting, engaging story is key to getting people to care. And whether the knowledge you want to impart is in science, the arts, or just basic literacy and numeracy, if you can’t get people to care about it, you’ll have a hard time achieving anything lasting.
  • Charities that engage in a personal connection. I prefer to share my donations among a smaller group of charities, which means each gets a larger share of the pot. In return, I’d like to hear what each charity is doing, and connect with more than just bank slips.
  • Charities that promote access to information over one particular message. Learning about family planning may be key to helping women in the developing world steer their fate, but if all you do is hand out contraceptives, they’re not going to learn how to run a small business that could give them a real degree of independence. Building people up, giving them the tools they need, is vital to sustainability.

Do you know a charity that fits the bill? Leave me a comment, or drop me an e-mail. Thanks!

Justin Mason—Links for 2010-01-20

Gary Gregory—JUnit Tip: Use rules to manage temporary files and folders


You can remove the burden of deleting temporary files and folders from your test code by using a JUnit TemporaryFolder Rule. Rules themselves are new in JUnit 4.7 and are used to change the behavior of tests. For example, the following test creates and deletes a temporary file and folder:

package test;

import java.io.File;
import java.io.IOException;

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;

public class TestTemporaryFolderRule {
    @Rule
    public TemporaryFolder testFolder = new TemporaryFolder();

    @Test
    public void testInTempFolder() throws IOException {
        File tempFile = testFolder.newFile("file.txt");
        File tempFolder = testFolder.newFolder("folder");
        System.out.println("Here I am, a tempoary test folder: " 
           + testFolder.getRoot());
        // test...
    }
}

In order to enable this feature, you must use the annotation @Rule on the declaration of the TemporaryFolder instance variable. The TemporaryFolder creates a folder in the default temporary file directory specified by the system property java.io.tmpdir. The method newFile creates a new file in the temporary directory and newFolder creates a new folder.
When the test method finishes, JUnit automatically deletes all files and directories in and including the TemporaryFolder. JUnit guarantees to delete the resources, whether the test passes or fails.

Grant Ingersoll—Just posted on: Apache Lucene Connector Framework now in Incubation at the ASF

I just put up some initial info on the new Apache Lucene Connector Framework project that is now in ASF Incubation.  See Lucid Imagination » Apache Lucene Connector Framework now in Incubation at the ASF.


Carlos Sanchez—Eclipse IAM 0.11.0, Archiva 1.3, Continuum 1.3.5

This is definitely release week! After Archiva 1.3 and Continuum 1.3.5 beta, I've just pushed the new release of Eclipse IAM 0.11.0:

This new version includes most notably

P2 Update site is published at http://q4e.googlecode.com/svn/trunk/updatesite-iam/

Ganymede users (Eclipse 3.4) should make sure they have added all the update sites listed in the installation instructions. If P2 complains about missing dependencies, check the update sites again.

Adopters of the latest and greatest Eclipse Galileo can install from the update site as usual.

If upgrading from Q4E 0.8.1 or earlier, some extra steps must be followed

The list of changes is available on the eclipse wiki.

Note that this is not an official Eclipse IAM release to allow our users to enjoy the progress made until we complete the move to the foundation and clear all the IP issues involving the maven embedder.

Ceki Gulcu—Marker based email notifications

Part of the SLF4J API, Markers allow developers to add meta-data to log statements. Not all errors are equal, with some errors being more unusual than others. If you wish to color log statement related to "unusual" errors you could write: Marker unusualMarker = MarkerFactory.getMarker("UNUSUAL"); Logger logger = LoggerFactory.getLogger("myPackage.myClass"); logger.error(unusualMarker, "An

Otis Gospodnetic—Sematext's Solr Digest, January 2010

 Sematext is reporting a new Solr Digest is ready.  Enjoy!

Steve Loughran—Datamining the bug database

My stance on developer surveys: go ask the tools is known.

I am pleased to see someone has done this with a lovely presentation on 10 years worth of Firefox bugzilla bugreps.

See that? Far more information than a survey provides, results of interest to developers, rather than just annoyance.

Rich Bowen—Evacuees

This morning, there was another so-called aftershock in Haiti. Any other time, it would be called a big earthquake. 6.1 is hardly an aftershock, but more of the main event. More buildings are down, the internet is out again, and it will be a while before we know what additional damage was done.

My sister and her kids are here in the US, and her husband is still at Quisqueya, helping run a field hospital and orphanage. The kids are in school, and generous friends and strangers have provided everything for them, from clothes to a place to sit at the lunchroom tables with friendly faces. Meanwhile, their dad, and many of their friends, are still in danger, and far away.

I'm feeling very sad this morning - sad for my sister and her scattered family, sad for the enormity of suffering of a people who have known little else, sad for the children who are wounded and hungry and frightened and lonely this morning in Haiti.

I wrote this poem over the last few days, after watching Ruth's kids and my kids playing, as though everything was no different from last summer. Right after I got done with it this morning, I found out about the new quake.

If you've been thinking about giving something to help folks in Haiti, but had let it slide by because it's not in the headlines any more, please consider giving to the Red Cross, or Doctors Without Borders, as they continue to alleviate the suffering of people who are utterly without resources.

Earthquake Evacuees

January 20, 2010

The boys are comparing loose teeth
The girls are somewhere
talking American Girls and shoes.

This week, they get to worry about
small things, like why
white people are driving the buses,
and why the electricity is on
all day every day,
and why nobody has walls around their houses

instead of when the ground will shake again
and why they have to sleep outside
and why so many people are laying so quietly
in front of their Escheresque homes.

Claus Ibsen—Apache Camel in the cloud (GAE)

Martin Krasser our new and talented committer have crafted the camel-gae component with a very cool tutorial and demo application as well.

Now you can try this out as the Camel tutorial demo application is running in the cloud at Google.
You can get to the demo from this link: http://camelcloud.appspot.com

You can read more about the tutorial here (has screenshots).

Steve Loughran—Making java server apps suspend/resume aware

I'm looking for an easy way for a server-side Java app to recognise that it has just resumed, and should handle that event (wait for the network to return, reconnect to things).

I don't see any easy way to do this, other than have a shell script to run on power resume which somehow notifies the app (touches a file, probably; queues something). The java app has to look for this file and react (polling, bad), or when it encounters network problems look at this file and say "did we just resume? Maybe the lan isn't live".

Life would be simpler if java apps could subscribe to power/LAN events

Matthias Wessendorf—combining CDI producer methods with EL


CDI’s Producer methods are nice to expose any sort of class (e.g. legacy or 3rd party (JDK)). Combining the @Produces with the @Named you can reuse the result in your XHTML JSF page.

Some Java code:

...
@Produces @Named public MyBean getMyBeanImpl()
{
....
return someBean;
}

Now the XHTML can have something like:

...
<h:outputText value="#{myBeanImpl.someOfItsProperties}" />
...

If you want, you can use the @Named with a named value (e.g. @Named(“fooBar”)). Now the fooBar also is required by the EL.

Pretty nice!

Chris Pepper—Flash Memory Performance

Update 2010/01/21: Thanks to @ceolaf for pointing out that Amazon has updated their page. They accepted my correction: "Fast read speeds of up to 15MB/second; write speed lower".

I have been using a Transcend 16gb Class 6 SDHC card Transcend 16gb Class 6 SDHC card in my Canon T1i, but needed another card. I found a good price on the somewhat confusingly labeled Sandisk Ultra 16gb SDHC card Sandisk Ultra 16gb SDHC card. As you can see, it prominently specifies "15MB/s*" on the label, and on Amazon's page. Additionally, it shows a smaller "C4" (Class 4) logo.

SDHC Class ratings are minimum write speeds in mbytes/sec, so the Transcend is guaranteed to write files at 6mbyte/sec or better, which is Canon's recommended minimum for shooting video or rapid stills on the T1i. I was a bit confused to see the C4 logo on the card, but I believed Amazon's "High speed card featuring fast 15MB/sec Read/Write speeds", which would be effectively Class 15. I couldn't find any explanation for that asterisk, but the price was decent, so I ordered a card.

I just got the card and did some simple tests -- copying a 1.1gb file to and from the card through my USB SDHC reader -- and discovered a few interesting things. The SanDisk wrote at 10,600,679 bits/sec and read at 21,621,758. The Transcend wrote at 9,399,092 bits/sec and read at 18,514,338 bits/sec. This means my SanDisk Ultra card qualifies as Class 10, and my Transcend is almost as fast (but there is no Class 7-9, so Class 6 is correct). The SanDisk is much faster than its marked Class 4, but nowhere near the 15mbyte/sec write speed Amazon promised.

Tonight I did some more poking, and found http://www.sandisk.com/products/imaging/sandisk-ultra-sdhc?tab=features, which explains that asterisk:

Fast read speeds of up to 15MB/second*; write speed lower

So Amazon's page is wrong, and I sent them a correction. I'm keeping the Ultra card because it's fast enough for the camera and probably whatever else I'll use it for, but disappointed it doesn't deliver on Amazon's specs.

Torsten Curdt—iProxy – poor man’s tethering

The iPhone – the always online device. What would be more natural than using it on the train as internet connection for your laptop!? Unfortunately that’s not really an option if you

  • don’t want to jailbreak your phone
  • aren’t using one of the “official” carriers
  • don’t want to pay extra
  • want to use iPhone OS 3.1 and up

The only viable solution: a proxy!

That’s exactly what the “Netshare” application was. But it has already been banned from the App Store for a long time. So I thought – I just write my own. A few hours later I was able to reach the internet through my iPhone via the “iProxy” app.

iProxy is not as convenient as the real tethering. The internet connection is a few clicks more away. But if you’ve got a developer certificate (or have a friend that has one) it certainly is cheaper than handing out the money to your favorite telco. Especially if you only need this connection only every now and then.

Now while I cannot make this app available to all the iPhone user out there through Apple, I do make it available as Open Source as of today. iProxy is released under the Apache license and freely available on github.

Enjoy!

Related posts:

Steve Loughran—Cloud MapReduce

Someone forwards me a link to an AWS Blog entry, Cloud MapReduce, which looks at Accenture's prototype MR engine built directly on top of the AWS stack: S3, simpledb, the message queue, etc

The paper is worth a read, they are very pleased about how few LOC it took to get working, and how it can be 60X faster than Hadoop.

My Initial thoughts

  1. A lot of the speedup comes from not shuffling; in Hadoop, shuffling/sorting stuff is optional. If you need to do it, and you do it in the right place, it pays off
  2. Physical Hadoop clusters normally bottleneck at the disk IO rates. Perhaps they are comparing Hadoop-on-EC2-VM performance with their Cloud MRs
  3. The LOC comparison is somewhat flawed as it doesn't include the lines of code needed for S3 (Java based, I believe), EC2 infrastructure, the message queue, database, etc
  4. Those LOC are the lines that Accenture get to maintain, from a maintenance cost perspective, the low #of lines is better.
  5. The SPOFs have not gone, just moved. No, the namenode may not fail, but all I have to do is report your credit card stolen to the bank and your cluster goes offline.

Its interesting in that it shows that a base cloud "stack" is important, and there is more than just VM hosting. You do benefit from a highly available , highly scalable filestore with direct (no need for a VM) remote access. Some kind of database is good, as is a message queue. Sometimes. And yes, you can write/rewrite applications that only work in this single environment. And once you do that, whoever owns the stack owns your code and your data

.

Footnotes