Liars & Outliers

Originally posted on Tue, 10/23/2012

I’ve been a Bruce Schneier fan for years. I read his blog often enough that I don’t feel the need to read his books. But then he offered a discount on a signed edition of his latest book– with the one stipulation that I write a review of it. So here’s the review.

A lot of brilliant thinkers tend to get stuck in their own perspective. There are plenty of mathematical geniuses who can’t contemplate the implications of their ideas. Plenty of programmers who can’t understand why users don’t recognize the brilliance of their user interfaces. Bruce Schneier isn’t one of them. His rose to fame with Applied Cryptography, a book of algorithms. But because great encryption doesn’t help when it’s part of weak security, he’s written written with increasing breadth about security. After 9/11, he wrote Beyond Fear about how we individually and as a society make poor security trade-offs. Now he’s written Liars & Outliers: Enabling the Trust that Society Needs to Thrive. This takes an even wider view, as he asks: how is it that people can trust each other at all? For example, a man gets into a taxi in a foreign city. The man and the driver will never see each other again. The taxi driver could easily rob him and get away, or he could ride without paying. And yet every day all over the world, drivers and passengers can trust each other.

When I got this book, my first thought was it’s too thick. After all, Schneier has said he wants this to be read by all sorts of decision makers. I’ve heard that if you want to be read by busy people, make your book just long enough to be read in a single long plane trip. Then again, I’m not a best-selling author. But it turns out this isn’t such a long read, it’s 250 pages, plus another 100 pages of notes and references. (This book-to-notes ratio is up there with Dr. Tatiana’s Sex Advice to All Creation, which has a remarkable amount of overlap with Liars & Outliers.) It’s really two books in one: a quicker, drier, less technical read without endnotes, or a more colorful read with endnotes. I kept one bookmark in the endnotes at all times.

Liars & Outliers is an overview of trust and cooperation. It draws primarily from sociology, economics, and psychology with a big helping of evolutionary biology, game theory, and security. Philosophy and theology also show up, to add a bit of color. It’s about the different strategies for enforcing conformity in a group, and when they do and don’t work. This is practical stuff for anyone who needs to manage a large group, whether it’s an online discussion group, a corporation, or a country of taxpayers. The book takes pains to discuss this from a neutral perspective: the forces are the same whether it’s a just society versus murderers or a murderous society against saints. To this end, Schneier uses the rather awkward term “defector” to refer to rule breakers.

The book is summed up quite well in the first chapter. In fact, if you’re in a hurry you could just read the first two and the last chapter. Occasionally the book starts to read like a taxonomy, as Schneier explains the breadth of, for example, security techniques. This is an academic book trying– and usually succeeding– in being a general-audience book. While it does drag a little at times, there are plenty of popular just-for-fun nonfiction books that drag more. (I’ll admit, though I read a lot, I’m not much of a book reader.) And with the whole history of deceit and treachery to draw upon, he has plenty of colorful examples.

I feel like I’ve said a lot about the book without saying much about the contents of the book. As someone who not only reads Schneier’s blog, but reads many of the same sources he draws upon, there weren’t many ah-ha moments for me. Perhaps the biggest one is this: trust is rare in nature. It’s hard to establish and easy to break, but once established it yields huge benefits. And humans are the most cooperative and trusting species on Earth. Why? In part because we have the greatest capacity to evaluate reputations, so we know when not to trust. In part because of a sense of morality which leads us to punish rule breakers.

I hear plenty of people argue that corporations are inherently inhuman and inhumane. While Schneier doesn’t say that exactly, he spells out each of the pressures society uses to enforce conformity to social norms, and how corporations respond to only two (reputational and institutional [legal.])

The overarching message of the book is that there are different ways of establishing a trusting relationship, and they work on different scales. Neighbors directly evaluate each other. People who know each other only by reputation can go by that. Less intimate groups use morals and institutions to maintain group norms. And when all else fails, security mechanisms can make non-compliance difficult. None of these completely eliminate non-conformity, which is good because even good rules can have bad consequences. The trick is using the right tools– and the right amount of pressure– under the right circumstances. It’s depressing to consider all the ways that things go wrong– from suicide bombers to self-destructive investment banks to a lack of a global response to global warming. But at the same time, humans are amazing creatures in that we even have a cooperative arrangement as abstract as the United Nations or a coalition of corporations.

Are green traffic lights really turquoise?

Originally posted on Wed, 10/17/2012

Yesterday I posted the following query to my friends:

Okay, everyone, here’s a quick homework assignment:

While you’re commuting tonight and tomorrow morning, look at the “go” stop lights. What color are they, REALLY? Yes, I know they are supposed to be green. But are they, really? And are different ones different colors? How many do you see that match the following colors:

1. Yellow-green
2. Pure green
3. Turquoise
4. Blue

No fair looking at photos of traffic lights, either. This is about color perception with your eyes.

(BTW, I’m partially color blind.)

So here’s what that’s all about. I’m red/green color blind, and green traffic lights have a blueish tint so that people like me can tell green from red. And yet people think of green lights as being a pure green. If you do a Google image search for green traffic light you get a mix of photographs– with blue-green lights– and clip art with pure green. Clearly illustrators aren’t looking at real lights.

This makes a difference to me because I see lots of user interfaces where they use red/green or red/amber/green to display key information: from traffic maps to MacOS window widgets. They never seem to use official traffic light colors. The worst, though, is red/green LEDs on appliances such as battery chargers. There’s absolutely no other context that can be used to indicate the status, since the brightness is identical.

Now, color perception is a tricky thing. It took me years to realize that I really am color blind, because I can nearly always figure out the color from context. (If that sounds weird, consider that you usually don’t notice your disabled color perception when you enter a low-light or colored light environment.) And color itself isn’t in nature– it’s a product of human perception. Consider that an apple looks to be the same color whether you see it under the cool light of noontime sun or the yellow cast of sunset. The light rays that hit your eyes are completely different, but your brain uses the background to compensate in order to construct a stable notion of color. That’s why I specified no photos– the background makes a big difference.

So the results? One person said yucca green, which she described as bluish. (Oddly, that swatch has slightly more red than blue.) Another said there’s variation, and the bluish ones are weird. And another said mostly pure green, with some yellow-green.

Oddly, I’d put the “yellow-green” option in just as a ringer; I never expected anyone to choose it. However, if the background is a bright blue sky, green will look more yellow.

So are green lights turquoise? I don’t trust my color vision enough to say. But you can compare. Blue-green traffic light, turquoise (equal parts blue and green light.)

Here are more details, for the pedantic.

Below is a picture of the official luminance ranges for traffic signals. Modern green traffic signals are allowed to be in the range from this:          to this:         . That’s assuming you’re looking at my color swaths on a pure white background. The ITE standard specifies colors relative to the range of human perception (in CIE 1931 color space)– that fingernail shape in the graph below. Computer displays, which shine only three colored lights, display a more limited, triangular color range with red, green, and blue corners. And you can’t get any of the legal traffic light colors from a computer screen!

ITE_Traffic_Light_Color

BTW, I got my CIE 1931 to RGB conversions here. You probably won’t find a better source, because any serious color expert would point out that you can’t accurately convert CIE 1931 to RGB.

And here’s a mind-blowing example of how language affects color perception.

Letter to great grandchildren

Originally posted on Fri, 10/12/2012

I haven’t restored my old blog entries yet, but if you want to read about my Seventh Generation letter writing project, here’s a snapshot, courtesy of the Internet Archive, which includes a few related posts. I’m relying on the Internet Archive to keep online copies of the letters I write to my descendants, in case something should happen to the paper copies.

I wrote another letter this summer, this time to my great grandkids, and I’ve now put it here. It’s scrambled, so please don’t unscramble it unless you are visiting from sometime between 2080 and 2100 or so and it’s addressed to you. Or if people in your time don’t use HTML with Javascript anymore, you may update it for the appropriate recipients.

One fun thing I’ve done this year is take pictures of the Boggle game that my wife, Jordan, was playing with my mom while I was writing the letter. I haven’t posted them yet; I haven’t figured out how to scramble them. The idea is for my great grandkids to play Boggle against my relatives.

How to get an iPhone 5

Originally published on Sat, 10/06/2012

Here’s how I’m doing it, not that I recommend this to anyone else.

  1. Pre-order online, on the first day. Get informed that you will need to wait 2-3 weeks.
  2. 3 weeks and 1 day later, get a call from Apple saying the order has been put on hold by Sprint.
  3. Call Sprint, wait on hold for 5 minutes. Tell them the name of your first pet.
  4. Get told warmly that they cannot help you– as much as she wants to– because the order was made through a third party (Apple) and Apple must call Sprint’s National Sales Support Desk. They will only talk to third party retailers, “I can’t even talk to them,” she says. Helpfully, she gives you a phone number.
  5. Call Apple, have an automated voice tell you it can help you with anything, just ask in a complete sentence. Humor the computer for a few moments before telling it you need to talk to a human being. It transfers immediately, and a woman answers on the fourth ring. Explain the situation, give her the number. She puts you on hold while she calls.
  6. A few minutes later, she says they gave her a different number to call. She also verifies that she has your wrong phone number (home number, presumably). She calls the next number, with you on hold.
  7. A few minutes later, the Apple rep introduces you to a Sprint rep, who wishes to ask a security question. In anticipation, you tell him your first cat’s name. After ascertaining the situation, the Sprint rep puts the two of you on hold while he talks to his supervisor. After a while, he comes back and says you must contact the National Sales Support Desk. Once informed that you’ve already done that, he talks to his other supervisor (more hold music) and he agrees to wait on hold while he places the call.
  8. All three of you talk to a man who inverstigates and finds no good reason for the delivery of your iPhone to be put on hold. He seems almost confident that the problem can be resolved. But first, he must ask for the Apple rep’s store ID number. She has no such number, since she is a customer service rep. The man replies, “we can only validate through a store” and gives some helpful hints as to where she might find an Apple department that he’s authorized to talk to.
  9. The Apple rep hangs up on Sprint and puts you on hold while she finds a way to communicate through proper channels. Eventually she brings you into a conference call. The other person transfers the two of you to Sprint, this time without hold music. At this point, the Apple rep explains that Apple cannot complete the delivery, so they are cancelling it, but will send you email so that when you place a new order you can respond to that email with your order number and Apple will expedite the delivery. They’ll try to get it to you as close to the original delivery date as possible. You can even order a different color or storage capacity.
  10. The latest Sprint rep assures you that there was no reason for the original hold on your order, claims it may have been a computer glitch– and offers to remove the hold. The Apple rep explains that the order has been cancelled. It will take 24 hours for the cancel to propagate. Do not order a new phone before then, or you will be renewing a contract that has just been renewed– and the order will be placed on hold as invalid.
  11. Wait 24 hours, or better yet, until Monday.
  12. Hope for the best.

That’s 1 hour, 20 minutes for the second call. Mostly on hold.

Making 3D pictures in The Gimp

For my 4Dth birthday, I’m designing a scavenger hunt which will include several 3D pictures which require red/blue 3D glasses to view. It’s easy to make them in The Gimp (or Photoshop, if you have a wad to blow on such things.) Figuring out exactly how to do it isn’t so easy. The basic idea is to merge two photos, one from the left eye’s perspective and the other from the right eye’s perspective. The final image should have only the red component of the left-eye picture and none of the red component of the right-eye picture. If you’re shooting a still life, use any camera and take one photo slightly to the left of the other. For close-ups, the eye separation should be very small: an inch at most. For a scene of a room, the eyes should be a natural distance apart, about 3 inches.

  1. For the left eye picture, add a layer above the photo filled with pure red ( RGB, a.k.a. #ff0000).
  2. Set the mode for that layer mode to “Addition.” You should see the photo in colors ranging from black to red. With your 3D glasses on, the red lens should show the picture as it appears on the screen, the blue lens should show a black picture.
  3. Under the Layer menu, select “Merge Down” to combine the two layers.
  4. For the right eye picture (a separate document), add a layer above the photo filled with pure blue-green ( RGB, a.k.a. #0000ff).
  5. Set the mode for that layer to “Addition”. With your 3D glasses on, you should see black through the left (red) eye.
  6. Copy (or “copy visible”) this picture.
  7. Paste into the other (left eye) picture. After pasting, hit the “new layer” button in the Layers dialog to turn the floating pasted image into a layer.
  8. Set the layer mode to “Multiply.” You should now have a different photo visible through each lens of your 3D glasses.

Scala emoticons

There are a lot of great things about Scala, but there are a few things where that programming language just goes overboard. Consider that there are several emoticons built into the standard library:

  • +:= (vampire pope) for adding items at the front of a sequence
  • <:< (sad witch) for comparing generic types
  • :/ (uneasy) fold right

And that’s just the ones I know about. Unfortunately, you can’t use a search engine to find any more, because none of the major search engines index emoticons. :/ (try it). Fortunately there is a well-hidden reference page.

Post a comment if you find any more.

Thoughts on recursion

I was poking through Code Complete, 2nd Edition by Steve McConnell recently, after I’d handed the book to a computer science student. The book is showing its age, as good object-oriented design is better understood than it used to be, but it remains the best book on how to program well.

One thing that struck me, though, was a particularly strong statement against recursion, on page 397 (2nd Edition):

If a programmer who worked for me used recursion to compute a factorial, I’d hire someone else.

To put this in context, McConnell had just presented a few appropriate uses for recursion (quicksort and maze solving [i.e. graph searching]), and is railing against the simple examples one typically learns in computer science classes. He sees recursion as an advanced technique, which should be avoided unless it provides a clear advantage.

McConnell says recursive algorithms are harder to read. I don’t buy that; it depends on your audience. But his biggest beef is with the stack. When a function calls itself over and over again, the stack grows (keeping track of all the variables in all those function calls) and you can run out of memory, the dreaded stack overflow.

In many computer science departments, the first language you learn is Scheme, which doesn’t support iteration, so you can’t compute a factorial (or do any other kind of looping) without recursion. Scheme handles this with tail recursion elimination, where if the last thing a function does is call itself (i.e. tail recursion), the Scheme interpreter overwrites the stack frame– essentially replacing recursion with a loop.

Since most languages don’t do tail recursion elimination, and since it can be harder to write a function as tail recursive, I mostly agree with McConnell– despite my initial reaction. However, I was thinking about what makes some algorithms amenable to recursion while others are a bad idea.

McConnell’s examples of good recursion– quicksort and graph searching– both involve branching. Most tree algorithms are recursive: one might say a tree is a recursive data structure. The recursive algorithm uses the stack to track the branching, which is cleaner than handling your own stack. (A good programmer can replace any recursive algorithm with a stack-based one.) What makes factorials different is that there’s no branching, so the stack is redundant. So we have two classes of algorithms:

  • Branching algorithms, which use recursion to track which branches have been visited.
  • Linear (non-branching) algorithms, which should be written in a tail-recursive manner to avoid stack overflows, or written with loops when the compiler/interpreter doesn’t support tail recursion elimination.

I should add that branching algorithms are also susceptible to stack overflows, but since the stack is providing a necessary service, it can’t be avoided with a non-recursive algorithm.

So here’s why I end up agreeing with McConnell: to properly write non-branching recursive algorithms, you must (1) write them in a tail-recursive manner, and (2) run them with tail call elimination. After you write a tail-recursive function, it’s easy to accidentally modify it to no longer be tail-recursive, and the difference can be subtle. That’s why Scala provides the tailrec annotation: so a function that’s supposed to be tail recursive won’t compile unless Scala can apply the tail call elimination. If you need the compiler’s help to safely apply a technique, it’s an advanced technique, and shouldn’t be done lightly.

UCU: better than MVC

I propose a replacement for the MVC (Model-View-Controller) model for software design. The problem with MVC is primarily that View and Controller both address the same concern: the user experience. After well over a decade, software developers now know better. My new model keeps the model, but all the words are changed to avoid confusion with MVC.

UCU: Utility — Connector — User Interface

  • A Utility provides a well-defined service. For example, accessing a file or storing a collection of items. It has a contract which defines what it does, and its correctness is measured against the contract. That contract is some combination of documentation, code (method signatures, interfaces, etc.), and a test suite.
  • A Connector is how components are discovered and constructed. Thus this category includes constructors, factory methods, and dependency injection. Connectors should be simple, since a program consists of a rat’s nest of interconnected components. Bad connections are hard to test, and even harder to debug, so the only recourse is to make them hard to be buggy. Constructors should do no extraneous work, except for checking for illegal arguments. When there’s an illegal connection, the program should fail immediately. For example, a constructor for a database utility which takes a database URL should check the syntax of the database URL, but it should not try to connect to the database. (If the database is inaccessible, that’s a problem with the database, not with the program trying to access it.)
  • The User Interface (UI) is the part of the program that relates to how the user perceives and interacts with the program. Its correctness is measured against the user’s experience. (Many practitioners prefer the term UX, User eXperience. In this case, because you can program an interface but not an experience, I am using UI to refer to the software.) Humans have different perspectives, and their expectations may change over time. As a result, UIs are developed using rapid prototyping techniques. Like a Broadway stage, the UI should be designed for frequent and rapid changes. UI components that get reused frequently are really utilities, thus classes tend to migrate from UI status to utility status.

For more reading, here are my earlier thoughts, before UCU gelled. And here’s a Google Tech Talk on writing clean, testable code which influenced UCU.

Education gaps

A NY Times story on education disparities explains that the gap between lower income and higher income families has been increasing at an alarming rate.

There are a lot of things going on here. Wealthier families are spending nine times as much on their kids as lower-income families, up from five times in 1972. They’re also spending more time with their kids, particularly before Kindergarten. And I suspect that’s the crucial factor.

There are a lot of single-parent families out there where the parent is working multiple part-time jobs just trying to pay the bills. Compare that with my family, where we have a stay-at-home mom and both parents read to kids every day. It’s just not fair. Don’t get me wrong: I don’t feel guilty that I don’t deserve it. I think everyone deserves to have the same opportunities.

The NY Times article says it’s not just about money, but I think that would be a fine place to start. Labor protections have been eroding as full time jobs have given way to less regulated part time jobs. And the free market isn’t going to fix that—not when the jobs can migrate to other countries.

The inequalities can be solved through services. My family has benefited from parent training and pre-K services (ECFE, for Early Childhood and Family Education), which the school district provides at little or no cost. (Not counting taxes.) Studies have found negligible difference in education between stay-at-home parenting and high quality day care. (What they’ve found is that good parents are better than bad day care, and good day care is better than bad parents.) If parents work, they need affordable, high-quality day care that will prepare kids for Kindergarten.

Some people will call this Socialism, and claim that it will loose to capitalism in the long run. Well, now we have long term data. If you look at countries that do take care of their kids—such as the Scandinavian countries—they’re doing fine. At best, they’re doing much better than the US; at worst, it’s hard to argue that they’re being held back.

There are those who say we can’t afford this. I say we can’t afford not to. For every Steve Jobs or Bill Gates, there are thousands of micro-entrepeneurs who have the drive to build businesses that employ a few people, but collectively do just as much innovation. We need both kinds of entrepreneur. Both exist right now because of quality, affordable, available education. We as a society need to stop penny-pinching away the investments that pay for themselves. Parents understand this for their own kids. Why can’t we agree on it for all kids?

Fire the job creators

As we’ve all heard, the job of those high-income earners who benefit from Bush-era tax cuts is to create jobs. But they seem to be slashing jobs more than creating them. So why do the job creators still have jobs?

In all seriousness, it’s disingenuous to expect businesses to create jobs. Whether a business sees itself as maximizing profit or maximizing value for a customer’s dollar, employees are expensive and therefore job creation needs to be avoided.

As a software developer, I’ve probably eliminated at least as many jobs as I’ve created. Mostly I’ve eliminated work, which allows people to work on other things and changes the nature of the job. That’s a productivity increase, but not one that gets measured in government statistics. I think I’ve generally made life better for those who have used my software, and the same is true for computer programmers in general. But there’s no guarantee that new jobs (e.g. webmaster) will outnumber the old ones (e.g. phone answering service.) My sense is that most job-creating innovation comes from individuals making incremental improvements to their own jobs, or even inventing their own jobs. And for that kind of innovation you need a middle class with enough of a safety net to take risks.

But it’s not just me. Recently NPR interviewed millionaires affected by the latest tax proposal, and they didn’t manage to find a single one whose hiring decisions would be significantly influenced by changes in the tax rate.