It is always amazing to hear such organizations admit that the comfort and service to the customer is simply not a goal.

1. Personalized Medicine
2. Personalized Wellness

Let’s talk first about Personalized Medicine. There is a lot of talk about the future of medicine being personalized medications. What is usually meant by this is that the blood and DNA of the patient is analyzed and then, using data gleaned from the EMR, a medicine precisely tailored to meet that patient’s need and their metabolism is prescribed. This is of course a wonderful vision—one that I would have loved to see realized a few years earlier. My mother was given several medicines for her recurrent ovarian cancer that were more or less ineffective.

Now it isn’t a pipe dream. There are blood and DNA tests run today for medicines like Warfarin or treatments for breast cancer. The best example is AIDS/HIV where DNA of the virus is used to determine which retrovirals will work. But in general, this is turning out to be very hard and very slow to do. It is hard just to figure out which medicines work for who based on their blood, DNA, and other phenotypic data. It works in some cases but fails in many. And even when a drug targeting a specific genetic profile is engineered, it is difficult and expensive to deliver to the right place in the body at the right time, in the right amount, and for the right duration. For example, we’ve known a lot about the genetics of cystic fibrosis —i.e., which proteins aren’t being generated properly in the lung cells due to mutations in a specific gene. Presently there are viruses that have been engineered that can generate the correct, functioning proteins, but the means to deploy an effective treatment has yet to be solved. Still there are clear examples of personalized therapies based on an individual’s DNA which help prolong life and have sufficient sales to warrant biotechnology/ pharmaceutical interest.  The clearest example of this is the drug from Genentech called trastuzumab (brand name Herceptin).  All in all, it is likely that it will be expensive and hard to change the DNA, but that the ability to produce solutions based on one’s DNA will be more viable.

Another issue is cost effectiveness in producing personalized medicines when such treatments serve a small market; the more specialized the medicine, the less likely it is to be developed. Thus, will we see a slew of highly personalized drugs targeting unique genomes or disease organisms? As was said in the movie “The Princess Bride,” when two magicians tried to bring the hero back to life and one magician asked another “think it will work?”, the reply was “it would take a miracle.” Of course in the movie he did come back to life, but life isn’t a movie.

Now let’s talk about Personalized Wellness. The leading causes of death relate to life style, lack of routine medical examination, and basic outages in care.  Put differently, it doesn’t require medical miracles to prevent far more disease and avoid far more suffering and deaths than all those caused by cancer (outside of lung cancer) each year. It requires personalized wellness and “good health incentives.” What is personalized wellness? It is personal advice to individuals about their health that takes into account their health data, their personalities, their goals, and their activities and what is the appropriate standard of care for them. It involves tracking their progress or lack thereof—what the Robert Wood Johnson Foundation has called ODLs or observations of daily living.

It’s possible for people who are at risk for diabetes or heart disease to avoid these diseases.  And for those who already suffer from them, it’s possible to cure them by clearing up their arteries or at least stop complications like blindness and renal failure. If they are living with asthma, get them the personalized help they need to minimize attacks and shorten episodes. If they are living with depression, give them support and tools like breathing calmly, meditation, regular exercise, and smart diets. This isn’t magic.  There is much scientific evidence about what works, and translations for healthy living are plentiful on the Web.  Think mint.com, a site that balances your budget, for health. The cost of building a site that empowers patients to manage their health is a tiny fraction of the cost of a single medicine being brought to market. Will DNA count in this space? Certainly. Some people have lower risks based on their genetic makeup, and others have higher risks. Certain nutritional interventions will benefit some people and may harm other.  But DNA testing can also inform intelligent prevention.

We want both personalized medicine and personalized wellness. But we can have the latter much sooner and it will probably do more good, at least in the next decade or two.

There is one thing making it very hard to deliver on this vision today. Much of personalized wellness advice depends on basic lab results like the lipid panel. The person with a total cholesterol of 150 may need different advice than the person with a total cholesterol of 250, for example. Today, if I go into a lab to get my blood drawn, say for my checkup, I cannot download the data into my personalized wellness tool of choice unless my doctor electronically approves it.  Not because the lab cannot support this—90% of labs performed outside hospitals are covered by Quest Diagnostics or LabCorp and both support electronic data transfer.  Rather, a doctor’s electronic approval is required to release the lab data to the patient, even when the patient wants this data. Well, most of the doctors aren’t using electronic systems and most of the ones who are don’t have the ability to approve these transfers, while some of the ones who do have the ability choose not to. The notable exception is Kaiser, which delivers labs to all of its patients online at the same time that the patients’ doctors get them. Three million patients use Kaiser’s PHR and the number one use is for viewing labs. Kudos, Kaiser!

But if you aren’t lucky enough to be a Kaiser member or want to use a different tool for this purpose, you are out of luck. (Actually, Kaiser may be integrating with Microsoft HealthVault and then one could use one’s own tools, but the timetable for rollout is unclear.) This is like not being able to use mint.com because your bank won’t allow the transfer of financial data to your account at that site.  It makes no sense, and is one more example of how the system foils patients’ attempts to take responsibility for their own health.  It clearly stifles innovation in an area that has the most potential to solve economic and personal health care issues in the U.S.

I call on DC and the State Legislatures to change these laws.  Learn from Kaiser.  Pass laws that specifically give the lab companies the obligation to deliver our data electronically directly to us – the people, if we want it. If you desire true health care reform that actually will lower costs and curb illness, unleash the power of the innovators to help consumers with personal wellness as mint.com does with financial wellness. Release our health data.

We are supposed to ask more lighthearted questions on this Thanksgiving weekend, but I’ve been unable to get WordPress to accept this questionnaire and I think it is a sign. We need to change the system profoundly to take human needs into account first. We need a system that works to meet these needs, not to try every possible futile procedure leaving those poor souls to suffer their last weeks or days in pain and indignity against their will. This is a serious business for those of us who have lived through this, seen the suffering first hand. We give thanks for many things this weekend but we look forward to the day when we can give thanks for a caring health care system.

To learn more please go to www.engagewithgrace.org.

We at Keas have been overwhelmed with astonishing potential partners in the health field who want to build great Keas Care Plans. We are humbled and gratified, but we are also urgently in need of someone to lead this effort for us. What sort of person do we need? We need someone with passion for the customer who will work with every partner to ensure that their care plans are engaging,  personalized, helpful and responsive and hire/manage the Keas producers we need to help the partners in this effort. We need someone who will be able to understand the health issues involved, but also the consumer passion and who can help our partners not just to deliver content personalized to the need, but video, twitter, great links, living discussions, polls, and everything else required to actually help the users of their Keas Care Plan to get the most out of it.

So, in short you need to be a leader, tireless,willing to get your fingernails dirty and lead by doing, passionate, unafraid of risk (this is a start up!), excited and knowledgeable about health, great at working with partners, with good business sense, experienced in building and leading teams that partner with others, and with an understanding of how the web is changing from a text world to an interactive and video world. If this is you and you want to help our partners produce the 100′s of care plans they now want to build, then let us know please at careers@keas.com.

I’ve was kindly asked to testify at a meeting in DC this week about standards at an hour when I’m normally not awake. But despite a deep aversion to not getting enough sleep, I was up and on the phone. What made me do such a thing? Well, the discussion was about what actually will work in terms of making health data liquid. What standards should be used for the integration of such data?

Somewhat to my surprise and usually to my pain, I’ve been involved in several successful standards. One was used to exchange data between databases and consumer applications like spreadsheets and Access. It was called ODBC and worked surprisingly well after some initial hiccups. Another was the standard for what today is called AJAX, namely building complex interactive web pages like gmail’s. Perhaps most importantly there was XML. These are the successes. There were also some failures. One that stands in my memory is one called OLE DB which was an attempt to supplant/replace ODBC. One that comes close to being a failure was/is the XML Schema specification. From all these efforts, there were a few lessons learned and it is these that I shared with DC this Thursday. What are they?

1. Keep the standard as simple and stupid as possible. The odds of failure are at least the square of the degrees of complexity of the standard. It may also be the square of the size of the committee writing the standard. Successful standards are generally simple and focused and easy to read. In the health care world, this means just focus first on that data which can be encoded unambiguously such as demographics, test results, medicines. Don’t focus on all types of health data for all types of health. Don’t focus on how to know if your partner should have access to what (see points 2,3, and 4 below).
2. The data being exchanged should be human readable and easy to understand. Standards are adopted by engineers building code to implement them. They can only build if they can easily understand the standard (see above) and easily test it. This is why, in the last 15 years, text standards like HTTP, HTML, XML, and so on have won. The developers can open any edit editor, look at the data being sent/received, and see if it looks right. When Tim Berners Lee first did this on the internet, most of the “serious” networking people out there thought using text for HTTP was crazy. But it worked incredibly well. Obviously this worked well for XML too. This has implications. It isn’t enough to just say XML. The average engineer (who has to implement these standards) should be able to eyeball the format and understand it. When you see XML grammars that only a computer can understand, they tend not to get widespread adoption. There are several so-called XML grammars that layer an abstract knowledge model on top of XML like RDF and in my experience, they are much harder to read/understand and they don’t get used much.  In my opinion Hl7 suffers from this.
3. Standards work best when they are focused. Don’t build an 18 wheeler to drive a city block. Standards often fail because committees with very different complex goals come together without actual working implementations to sanity check both the complexity (see point 1 above) and the intelligibility (see point 2 above). Part of the genius of the web was that Tim Berners-Lee correctly separated the protocol (HTTP) from the stuff the browser should display (HTML). It is like separating an envelope from the letter inside. It is basic. And necessary. Standards which include levels or layers all jammed into one big thing tend to fail because the poor engineers have to understand everything when all they need to understand is one thing. So they boycott it. In health care, this means don’t include in one standard how to encode health data and how to decide who gets it and how to manage security. If all I, as an engineer, want is to put together a list of medicines about a patient and send that to someone who needs it, then that’s all I should have to do. The resulting XML should look like a list of medicines to the me. Then, if it doesn’t work, I can get on the phone with my opposite number and usually figure out in 5 minutes what’s wrong. Also I can usually author this in a day or two because I don’t have to read/learn/understand a spec like a telephone book. I don’t have to have to understand the “abstract data model”. The heart of the initial XML spec was tiny. Intentionally so. I heard someone say indignantly about the push to simplify Health IT standards that we should be “raising the bar on standards” not lowering them. This is like arguing that we should insist that kids learn to drive an airplane to walk to the next door neighbor’s house. All successful standards are as simple as possible, not as hard as possible.
4. Standards should have precise encodings. ODBC was precise about data types. Basic XML is a tiny standard except for the precise encodings about the characters of the text, Unicode. That is most of the spec, properly so, because it ensures that the encodings are precise. In health care this means that the standard should be precise about the encodings for medicines, test results, demographics, and conditions and make sure that the encodings can be used legally and without royalties by all parties. The government could play a role here by requiring NPI’s for all doctor related activities, SNOMED CT for all conditions, LOINC for all labs, and some encoding for all medicines (be it NDC, rxNorm, or FDB) and guaranteeing that use of these encodings is free for all use.
5. Always have real implementations that are actually being used as part of design of any standard. It is hard to know whether something actually works or can be engineered in a practical sense until you actually do it. ODBC for example was built by many of us actually building it as we went along. In the health care world, a lot of us have built and used CCR as we go, learning what works and what doesn’t very practically and that has made it a good easy to use standard for bundling health data. And the real implementations should be supportable by a single engineer in a few weeks.
6. Put in hysteresis for the unexpected. This is something that the net formats do particularly well. If there is something in HTTP that the receiver doesn’t understand it ignores it. It doesn’t break. If there is something in HTML that the browser doesn’t understand, it ignores it. It doesn’t break. See Postel’s law.  Assume the unexpected. False precision is the graveyard of successful standards. XML Schema did very badly in this regard. Again, CCR does fairly well here.
7. Make the spec itself free, public on the web, and include lots of simple examples on the web site. Engineers are just humans. They learn best by example and if the standard adheres to the points above, then the examples will be clear and obvious. Usually you can tell if a standard is going to work if you go to a web site by the group and there is a clear definition and there are clear examples of the standard that anyone can understand. When you go to the HL7 site the generality and abstraction and complexity are totally daunting to the average joe. It certainly confuses me. And make no mistakes. Engineers are average joes with tight time deadlines. They are mostly not PhD’s.

Let’s be honest, a lot of standards are written for purposes other than promoting interoperability. Some exist to protect legacy advantages or to create an opportunity to profit from proprietary intellectual property. Others seem to take on a life of their own and seem to exist solely to justify the continued existence of the standards body itself or to create an opportunity for the authors to collect on juicy consultant fees explaining how the standard is meant to work to the poor saps who have to implement it. I think we can agree that,  whatever they are, those are usually not good standards. Health data interoperability is far too important an issue to let fall victim to such an approach.

For those of you who missed this news, Keas now has an open, free public beta at www.keas.com.

What is Keas? Keas brings you the best medical minds to deliver personalized help so that you can start to take charge of your health. These health experts build personalized expertise into a Keas Care Plan, based on the very same questions and feedback that occurs in person, during an office visit. In other words, these Care Plans look at or ask for your data just as health experts would. Given that data, Keas Care Plans can help you understand your health by charting the results that matter, indicating whether you are where you should be (in the green), have some risks (in the yellow), or clearly need serious attention (in the red). And because they are developed by health professionals who understand the nuances of health issues, Care Plans deliver “to-dos” for you to see at a glance what steps to take to get in the green and stay in the green.

We also announced a wonderful strategic alliance with Quest Diagnostics. If your doctor orders a blood test to be taken at a Quest Diagnostics Patient Service Center, when the results come in Quest and your doctor will help get your data into Keas. In addition, as part of the strategic alliance, Quest Diagnostics has worked with Keas to help interpret your data, based on your personal health status, as falling in the red, the yellow, or the green. It is another layer of expertise that offers you the best advice for taking charge of your health.

Thanks to the news coverage and our partnership with Quest Diagnostics, we are now getting large numbers of users each day. And that brings us yet another layer of expertise – you, the user.  As we develop communities based on individual Care Plans, your knowledge and wisdom will be invaluable to those who share your specific health concerns, and we’ll provide the tools for peer-to-peer support. In addition, we at Keas need your smart observations: we can only make our services great and truly useful with your help. We want to know from you what Keas Care Plans you need that we haven’t built. We want to know which Care Plans can be better and how. We want to know which “to-dos” need to be improved and expanded, and your preferred modes and frequency of messaging. We want to know what Keas should be doing for you that it isn’t already doing in order to provide the best personalized help from the best medical minds. So please keep your feedback coming.

Our commitment to you is that we will learn and work hard and steadily to fix the things that need fixing and add the things that need adding. Working together over the next few months, we can make Keas the tool you need to understand your health and take charge of it, with help from the best health experts and from each other. It is an exciting time.

If you are a health expert and want to join us in building Care Plans for your patients, please email us here.

What makes Google such a formidable competitor in the ads space is machine learning. What makes my bank now able to do such a good job of warning me about possible fraud is machine learning. What makes travel companies so good at pricing is data mining and machine learning. If I were giving any aspiring student going to a university to study computer engineering advice, it would be to focus on this area. It is almost like magic. We see it in subtle ways like NetFlix movie recommendations, but this is just the tip of the iceberg. Beneath the waves, almost every field is moving in this direction. And, these systems are dynamic and rapid. They are constantly learning and constantly improving.

There has been one notable exception. Health care. Machine learning and data mining do require a lot of data. Since you aren’t able to do controlled double blind randomized experiments, you need enough data to make the conclusions statistically significant in a messy data world. But given enough data, learning can and does happen. We are poised at the beginning of a similar sea-change in health care. As vast amounts of personal health care data start to get collected we will start to learn what is actually effective and what isn’t for whom. This is really a prerequisite for personalized health. The term is used loosely to mean giving people the personalized advice/treatment that they need based on their data. But the only way to personalize is to know what’s effective for whom. Some of this will doubtless be based on genomic information. But far more will just be based on looking at what is working for whom based on their conditions, ongoing test results, and treatments.  And this is key. The human body varies tremendously based both on environment and on inheritance. One size doesn’t fit all.

Until recently, a lot of machine learning from health data has been still-born for 3 reasons:

1. It has been too hard to translate what is known about personalized medicine from research into clinical practice. This is known as the “translation” problem. But online tools that do know these things are going to rapidly change this failure in translation in the decade to come.
2. There hasn’t been nearly enough data because almost no data was automated and, even when it was, it wasn’t tracking the data over the individual and their treatment plan. Instead, it was tracking the order over the insurance number and the practice because that’s where the money was. Between ARRA’s meaningful use mandates which are going to force tracking against the patient and the burgeoning consumer movement to take charge of their own health as the system increasingly limits their access to continuous care from physicians, this lack of data is going to change at least as profoundly in the decade to come.
3. There was no money in giving consumers personalized treatment and indeed movements against it, both the population studies (witness the debates right now about diabetics being told to lower their blood sugar) and because the doctor’s weren’t paid for outcomes. But consumers are going to demand the treatment for the best outcome. Also we’re learning that often, it will cost less. Often the standard care given is too much treatment, so brilliantly called out in the book “Overtreated” and, paradoxically, your outcomes are better as the cost goes down, not up.  Back surgery tends to be a post-child for this, also called out well in the book “Flatlined“. We are going to be forced to figure out how to be more cost-effective, and more effective in general in treating illness.

All the systems emerging to help consumers get personalized advice and information about their health are going to be incredible treasure troves of data about what works. And this will be a virtuous cycle. As the systems learn, they will encourage consumers to increasingly flow data into them for better more personalized advice and encourage physicians to do the same and then this data will help these systems to learn even more rapidly. I predict now that within a decade, no practicing physician will consider treating their patients without the support/advice of the expertise embodied in the machine learning that will have taken place. And finally, we will truly move to an evidence based health care system.

This is a personal issue to me and thus makes me particularly angry. When my poor mother finally turned the tide for the worse in her battle with Ovarian cancer, she was diagnosed at the Hutch in Seattle. I had to fight, almost physically, to get her out of a hospital 3,000 miles from her beloved apartment in NYC and back home to the apartment where she wanted to end her days. I almost had to medivac her before she recovered just enough to get her discharged and onto a wheelchair and thus onto a plane back to NYC. Once in NYC, I brought her into Sloan Kettering Memorial Hospital which had treated her well while treatment worked, but once it was clear that all options had ended and she only had a couple of months left, they and the current medical system left her and us terribly adrift. They basically sent an old lady away to die and confront fear and pain without any offer whatsoever of home help. They gave her a complicated regime for the painkillers and other meds that even the visiting nurses of NY (who were saints) couldn’t administer and Sloan offered no help in finding the “High Tech” nurses we required for my mothers care. We went through fear, pain and panic for the next couple of days trying to help her follow the meds regime they had given us on discharge which it turned out, even the visiting nurses of NY (saints!) couldn’t follow. I finally found some along with the help of an extraordinary friend of my mothers who had been an ICU nurse and we managed for the next 2 months until her last few days when she chose to go into a hospice. But the modern health care system tried hard to have her die in the hospital at a cost of 1,000′s of dollars a day when all she wanted was to be at home, pain-free, among friends. For that, not a penny. Her hospital never even called to see how she was doing.  And this is what the Republicans are trying to protect – this soulless cruel heartless vicious system. They should be ashamed. I am ashamed of them.

The theme of the conference was a more holistic one and included a truly wonderful talk by Mehmet Oz about overall health and wellness as well sessions on play, food, and sex. People forget how much of illness is caused by stress and how important it is to find ways in your life to just enjoy the world you live in.  None of us can completely avoid stress, but all of us should strive to balance our drive to succeed with the need to stop and smell the roses. Of course the conference on all this started every day at 7:45 AM and finished at 10:00 PM and this was a weekend, but the intentions were good.

If you want to know what I said there and would prefer to read than listen, here is the text of the talk I gave at the Aspen Health Forum.

__________________________________________________________________________________

There is a lot of talk about improving health care. And there is a lot to improve.

Inadequate Evidence: We don’t know enough about what works. We should require sharing of population statistics across practices and hospitals in order to better determine what works for whom. We should reward practices and hospitals that are delivering the best most cost-effective long-term outcomes and penalize those that deliver the worst.

Overtreated: Doctors often don’t know what the evidence suggests that they should do. Often even when they do they don’t do it not because of careful clinical thought but habit. The book Overtreated makes this painfully clear. We need to reward/encourage doctors based on long-term outcomes, not pills and procedures.

Flatlined: Payment is skewed away from the people who should be the coaches for consumers, the primary care physicians, because of a model that pays for the complexity of a procedure rather than for the cost-effectiveness of it. This model has encouraged countless unnecessary spinal fusions, stents, and other expensive procedures and discouraged any form of preventive medicine, patient jawboning or oversight. We need to reward/encourage doctors based on cost-effective long-term outcomes, not on difficultly of procedure. The book Flatlined makes this point incredibly clear.

No PCP <-> Patient time: Doctors can’t jawbone their patients anyway. In the old days, primary care physicians jaw-boned their patients. But that took time and energy, neither of which they have today when they have 30% increased patient loads. So they don’t. And people who should be healthy end up with joint replacements, depression, stress, heart disease, diabetes, gestational diabetes, and increased risks of cancer. And in so doing, the life-style related diseases cost us over $3 Billion a day.

Lifestyle: And the patients need jawboning badly. A key reason that our health care costs are out of control are consumer’s life styles and ignorance and lack of skin in the game. Patients literally do not know how to be healthy, have no sense of the implicit costs of being unhealthy, and have no incentives to be healthier short of chronic pain. up to 70% of our costs are due to life-style related diseases with obesity and inactivity and poor eating choices being key ones. 23 years ago, only 7 states had an obesity rate over 15%. Today only one state is under 20% and that majority are 25% or over. We have a wave of diabetics who don’t really take care of themselves leading to amputations, blindness, renal failure, heart disease, and other terrible consequences and costs.

They need more than jawboning. They need tools.

I dream of a day when everyone has online access to their health and wellness plan – Not their sickness plan, but their health & wellness Care Plan, personalized and tailored to their specific health data, their needs, their goals and  their realities.
I dream of a day when we don’t publish books about how to stay healthy, we publish personalized health & wellness Care Plans written by the best in the business, experts in their fields, but augmented by the feedback and realities of online engagement with their customers as the authors and experts determine what works for whom.
I dream of a day when we truly know which Care Plans work for whom because we have been able to measure which do.
I dream of a day when consumers talk about their health score as today, they might talk about their handicap or FICO score and work with trainers to lower it as they do their handicap
I dream of a day when their doctors are partners in these Care Plans, not people patients visit when they are sick

We can and will fuel the engine and drive the innovation and payments that will slowly but surely reverse most of these problems.
This dream requires only one thing to make it a reality. One thing that can unblock this logjam. Here is the “big idea”.

HealthDataRights.Org

Why the Declaration of Health Data Rights
Shockingly most doctors don’t even have the patient’s data electronically. But the organizations at the source do. Labs and the pharmacies and insurers and the imaging labs do. We empower the consumers to get their health data electronically, rapidly and online. This is the fuel that will fire the engine of consumer health-care because it enables the rise of online tools to help consumers manage their health and work in a participatory manner with others best equipped to help them driven by data and expertise specific to their needs. As human beings, we all have this right. To deny it is to deny healthy living to all. We are in the business of increasing health and this is the tap that must be turned on.

What will this enable.

Revolution Deferred: Next we move the routine and the information out of the doctors hands and into the consumers through these online Care Plans, whether it is getting ready for a visit, discharge, or managing a chronic disease. The internet has profoundly revolutionized every other business moving all routine work into the consumers hands, but giving them choices, flexibility, and transparent information. Travel. Banking. Books. Movies. Eating. Only in health care is this revolution deferred. We must move health care out of the 19th century and into the 21st. Then we will harness the creative talents of tens of thousands all across the world. What’s more, it will enable us to measure what works for whom in a computable fashion. It is unacceptable that this hasn’t already happened.

Health Coaches anywhere: We allow people anywhere to provide expertise online to consumers. What sense does it make that only a cardiologist in NY state who may well be out of date and has no incentives to help someone go on a diet/exercise regime is the only one who can treat a patient in NY state? Why can’t anyone with expertise in heart disease, anywhere in the world, help that patient? They will become the online partners in these health plans.

Patient Engagement: The most important of these tools will be Care Plans. They will be the online tools consumers use to know what it means for them to be healthy and how to be healthy. People need the tools and training and support to know what to do, why they should do it, and the encouragement to do it. This is called patient engagement. This is called participatory medicine. Once people have online control over their health data, their medicines and their labs and their images, a myriad of online Care Plans will emerge to help them understand how to be healthy

Incentives: Lastly, we provide some incentives, even small ones, in the form of lower health care costs, to individuals who are managing their health well as measured/defined by these computable protocols. To anyone who has tried it, a reward of a few $100 / year has a dramatic effect on compliance. Let’s say that 70MM people aren’t managing their health well or the health of their children. At $300 / year in incentives (e.g. decreased premiums) that’s $21 Billion a year or less than 1% of our health care costs IF they in fact become as healthy as they can in which case they save far more than that. We spend that on avoidable life-style related health-care each week right now. The return on that investment would be almost 100:1. It’s effect will be vastly greater than putting the current behemoth EMR’s into doctor’s offices which will often be like giving giant combines to the suburbanite who wants to mow the lawn.  Giving physicians the wrong tool and the wrong incentives will still lead to the wrong result. Giving the consumers the right incentives will drive consumerism throughout the healthcare system and drive the right results, by definition.

All this stems from an incredibly simple idea.

Give consumers the right to a copy of their own health data, without impediment or delay, online in the place of their choice.
We will see the rise of online Health Plans targeted at helping consumers to understand their health and to learn how to be healthy or stay healthy.
We will see the rise of money flowing to consumers, doctors, nutritionists, fitness experts, and health coaches to support the consumers in their efforts.
We will see the system evolve from a sick care system to a health care system, driven by consumer demand.

Why will this work? Because it harnesses the power of the world’s intelligence, the world’s online delivery and the worlds innovators – you – and rewards you for doing the right thing.
It isn’t bottle-necked by how many people in medical school become PCP’s because the online tools massively leverage expertise and remove rote work from physician’s lives and transcend national borders.

It funnels the money and the talent to what works.

When we all support the declaration of Health Data Rights in deed, not in words, we will change health-care.

Join me in making Health Data Rights a reality, not a dream

Best new post I’ve seen this year is about how we can really lower costs.If you read nothing else, read this please.

In their tirades against Obama’s attempts to fix a tragically broken health care system, the Wall Street Journal has descended to the ludicrous. See their article on Big Pharma gets played. They are shocked, shocked that US citizens may actually have the legal right to pay less for drugs by buying them from Canada.  The WSJ concludes by saying dramatically, that “the real victims of government health care will be American patients”. Because we pay less for drugs? When did the WSJ decide that price competition was bad for the consumer?

The WSJ editorials have consistantly ignored the elephant in the room, namely that US citizens are significantly sicker than most other civilized countries (we tend to rank between 25th and 37th) despite the US spending almost twice as much per capita  asthose countries and that our costs of health care – $2.5 Trillion on track to reach $4 Trillion within a decade as the baby boomers age – are unsupportable anyway. Something has to change. Also, consider that as the babyboomers age, the health care costs will be mainly in Medicare.

Strange things happen in Texas. Imagine if your broker was mismanaging your money and people in the organization spoke up and warned the SEC that this was happening. You might not know because you’re no expert on investing. Heck you might like the broker. But the people in the organization would know. They are the experts. What should happen? You would hope that the broker would be disciplined. But in Texas, when it is the vastly more important area of your health without which no real happiness is possible, if people speak up, they are indicted and threatened with 10 years in jail. Here’s hoping that the health care providers in Texas speak up loudly against  this  abuse of power.

A lot of doctors express skepticism out there about online support for health on the part of the doctors. A common comment is that my patients are older and aren’t on the net or aren’t willing to use the net to manage their health. Doctors should learn from Kaiser who reports that 87% of their Medicare patients are happy users of Kaiser’s My Health Manager.

It is interesting to watch some health care providers start to scream as it dawns on them that the meaningful use bill might actually require them to have proof that they are improving their patients health in some basic measurable ways in two and a half years. Incredibly, they claim that that isn’t enough time. In the world of online computing it’s an eternity.

In almost every field today, the power of online communities, online tools, and online data has been revolutionary, whether it be finance, travel, purchasing goods, books, getting the news, or entertainment. If, for example, we want to plan a trip, we can search online, book online, print out our boarding pass online, and see what others have thought online. Our schedule is clearly there and usually, if the site is well designed, it is there in a form where we can effortlessly digitally update our online calendars as well.

This revolution has led to far better information, far more freedom to plan and organize things and usually a far better experience. No more waiting in line at the counter at the airport or at the bank or at the store. No more wondering if we’re getting the right thing or going to the right place. No more (and you have to be my age to appreciate this) trying to get to the bank before 3:00 PM so that we’ll have money for the weekend or the vacation. Along with this convenience came the idea that the data is the consumer’s data and we deserve to know it. We have the right to it.

Yet, in the area where it matters most, our health, virtually none of this is true. It is as though the revolution never occurred.  We cannot easily see what our data is online (e.g. our labs, our prescriptions, our problem list, our images) unless we are lucky enough to be at Kaiser; let alone if, like almost all Americans, we have dealt with innumerable different clinics, hospitals, doctors, labs, and pharmacies during our life. Further, we cannot easily take the time online to understand what this data means for us and harness the online power of experts or the opinion of others to best understand our health and where we stand.

Let me give the simplest example. Many people in this country are given medicines by multiple doctors which may be dangerous in combination or have harmful effects if we have some specific condition. No fault of the doctors. They don’t have the holistic view of our health data either. They can’t even access our data online most of the time. Now the  American Recovery and Reinvestment Act of 2009 or ARRA aims to fix this for the physicians. But we, as the affected consumer, can’t easily go online, load in all our medicines and test results and problem lists in computable form and and with a mouse click learn if there’s an issue, as, for example, we might use an online flight tracker to see if there was a travel issue.  Thus our health and well-being take a back seat to our travel ease and comfort with potentially serious consequences.

Sometimes there are laws on the books that actually prevent us from getting our health data. If we have a risk of heart disease and really should keep an eye on our cholesterol, there is a simple lab test, a lipid panel, that we could use to make sure that our exercise, diet, and medicine programs were working. But in many states we cannot order this online. Even if it is ordered for us, in many states we cannot just ask the lab to directly download the results to us at the time the test is taken. It is as though they’ve made it illegal to find out how much we have in the bank or where our trip will take us.

Only our health is more important than anything else. Without good health, life is greatly compromised. If there is one place where we must have the right to the best information, advice, care, online support, and planning possible it is  our health.

Today I proudly join together with a group of  leaders across health care working to establish a Declaration of Health Data Rights. The text of the declaration is clear, appropriate and extraordinarily important to us all.

A Declaration of Health Data Rights

In an era when technology allows personal health information to be more easily stored, updated, accessed and exchanged, the following rights should be self-evident and inalienable. We the people:

1. Have the right to our own health data

2. Have the right to know the source of each health data element

3. Have the right to take possession of a complete copy of our individual health data, without delay, at minimal or no cost; If data exist in computable form, they must be made available in that form

4. Have the right to share our health data with others as we see fit

These principles express basic human rights as well as essential elements of health care that is participatory, appropriate and in the interests of each patient. No law or policy should abridge these rights.

My company Keas and I wholeheartedly endorse this declaration and are proud to be part of a growing community that recognizes its importance.

If you would like to add your voice to ours, you can get started by visiting http://www.HealthDataRights.org and see who else is endorsing this declaration and spread the word. Support and defend our rights to our health data and start the revolution.

1. Improve Quality, safety, efficiency, and reduce health disparities. – Read Overtreated to learn why this is key
2. Engage patients and families – Without this, we cannot solve the fundamental cost problems we face
3. Improve Care Coordination – It will be amazing to see care coordination come into the 21st century
4. Improve population and public health – They are correctly focusing on measuring what works and who is doing it
5. Ensure adequate privacy and security protections for personal health data – I believe consumers should control who sees what of their personal health data and am excited to see this supported.

I’m really delighted. This document isn’t perfect in my opinion. In particular, it calls for EHR’s to “provide patients with electronic copy of – or electronic access to – clinical information (including lab results, problem list, medication list, allergies) per patient preference (e.g through PHR). This allows the bad “old days” of tethered PHR’s to continue where as you switch doctors because of changes in your location, insurer, job, or just need a better doctor, or you want to include other doctors in complex care, or want outside or second opinions about your data, you cannot get it because not everyone is using the same EHR. It is key that the patients control their PHR’s and that they can transfer or download or copy this data in as computable a form as is available in the EHR into their PHR of choice.

But overall the committee that produced this document is to be congratulated. Kudo’s and congratulations to the members of the HIT Policy Committee and a special thanks to David Blumenthal. And while there are features in the matrix that aren’t called for until 2013 that I’d like to see in 2011, again, it is an excellent document.

The preamble does an excellent job calling out why we need this and what we need to improve. Don’t change a word of it please!!

Doctors are speaking up. Brian Klepper and David Kibbe just put up an interesting post. Bob Wachter, who I knew when at Google Health, referred to an excellent article by one of my heroes, Atul Gwande in his latest post pointing out that the current system has no discernible efforts to actually manage overall outcomes and costs.  Contrary to the inane platitudes one hears from certain Senators blandly assuring us that the health care system needs no repair, people across the board are pointing out that the system is broken. The center cannot hold. But we will not bow to the prediction in Yeat’s magnificent poem in which the best lack all conviction and the worst are filled with passionate intensity.The system can and must and will be fixed. It is time to speak up.

Watching legislature meet health care is like watching a slow motion train wreck. Witness the bill in contention in NJ right now. Essentially it will prohibit or drastically slow down the sale of innovative products that actually help consumers understand or manage their health. For example a product that prints out a medical history for the consumer to help him/her have a better doctor visit or a product that uses the FDA guidelines to warn about med/med interactions as Google does or a product that helps users collaborate on improving their health will all be prohibited if they actually used “health data” unless the CCHIT specifically licensed them. Imagine if every product to help you budget, plan a vacation and the finances, plan to save for your kids college education or retirement planning all had to be licensed by an agency let alone one that was never intended to do this. That is essentially what the NJ bill does. It is time to speak up.

Let me explain.

The simplest way to describe what the administration seems to want is to be able to insure more people (a LOT more) at a lower cost (since otherwise the total costs go up a lot). Many of us have a strong belief that we can only lower costs and/or improve health-care if we make the consumer part of the solution, what is often called “patient engagement”. At the end of the day, episodic care treating people only when they get seriously sick, enough to go to their doctor or the ER isn’t ideal. What we believe is ideal is teaching people to keep a constant eye on their health and keep them out of the doctor’s office and especially out of the ER.  One might think that the forthcoming funding for Electronic Health Records (EHR’s) would support systems that deliver this sort of patient engagement and long-term patient wellness and thus help lead to better support for patient engagement. But in fact, many of the  traditional EHR’s have not focused on this at all, hence our fear about giving them a de-facto monopoly.Secondly, most small practices can’t really afford to use big iron EHR’s. Even if it is free, they can’t really afford to do it because it will still require training, more time per patient potentially, and so on. Lastly, more EHR’s don’t work with other EHR’s so that coordinated care across practices isn’t supported and most people who are elderly or who have serious illnesses have more than one physician treating them.

The way around this is to build systems that don’t just duplicate what physicians do today during their face to face meetings with their patients, but rather provide new capabilities that will help with continuous and coordinated care and can generate additive revenues for physicians and then evolve by adding those features that automate the current physician activities as demanded by the physicians. What would such systems support? They would support having a way to chat with or exchange messages with a patient for a fee so that unnecessary office visits can be removed and the patient is more likely to reach out for help. Think eVisit-lite. They would support a simple way to monitor the health of a patient who either has a chronic disease or is on path to developing one again for a fee so that physicians are actually getting paid for keeping their patients healthier as opposed to being punished for it since, ideally, it will result in fewer visits/procedures over time. In short these systems will support physicians  managing an ongoing paid relationship with the patient rather than an episodic one measured only by in-office visits.  What should be done about helping physicians who are afraid of losing time to retraining? These systems should be as easy to use as a Southwest airlines reservation page. These systems should have a cost is so low that physicians don’t care. Most of these points aren’t typical of most of the big EHR’s currently being sold. Again, hence our fear that a de-facto monopoly of the incumbents will lose this opportunity to let 100 disruptive innovations flower.

It is my opinion that the bar for “meaningful use” and a “certified EHR” should be limited to the following:

• Easy way to share electronically computable data about medicines and labs with the patient’s URL’s of choice. These URL’s would point to the services that are helping/advising/monitoring the patient in a patient controlled way. This alone should be enough to declare a tool certified because it empowers consumers to take charge of their own health. This is also the backbone of cooperative care since then multiple physicians, regardless of vendor or practice, can exchange and share computable health data about a patient.
• Easy standardized way to support an inbox both between physicians using different EHR’s (think email today) and between patients and their physician/nurse/physician’s assistant. This should be optional, but Medicare and insurers should be encouraged to pay for such support. Kaiser has found that the burden isn’t high and it cuts in-office visits significantly. This should certainly be sufficient for meaningful use because this, in conjunction with the first point ensures that physicians can coordinate care for a patient. It also frees up the patient to pick the best other doctors who provide the best care, regardless of practice because the collaboration can occur across practices.
• Support for ePrescribing, largely, to be honest, so that the prescription information can flow to the patient.
• Easy way to put patients on ongoing fee-based computable care programs and monitor how they are doing sending alerts to the physician where necessary so that physicians know when their patients are trending in the wrong direction. Something as simple as monitoring blood pressure and weight and ankle swelling can prevent repeat heart attacks. Something as simple as monitoring total steps taken a day and blood sugar and meals can prevent serious diabetic complications. But physicians aren’t paid to put patients on such plans or for the time to monitor them. Instead we wait for catastrophe and pay for that.

Not one of these except for ePrescribe duplicates existing physician work flow . These are new services that should generate new revenues for physicians all focused on continuous and coordinated care. Most people don’t get these services from their doctors (Kaiser is always a notable exception precisely because, I believe,  they are actually paid to keep people well).  And these are the services that will truly drive patient engagement with their health and with their physicians. Ultimately, it is my premise that patient engagement with their physician is the key to unlocking our health system, driving true innovation and converting it from a sick care system to a true health care system. There is an excellent post by David Brailer, the former National Health Information Technology Coordinator, supporting the urgent need for innovation and patient engagement in health care.

I ask Dr Leavitt and CCHIT to help ensure that the funds unlocked by ARRA be used to support these capabilities in as open and easy a way as possible and to avoid, at all costs, a de-facto monopoly in the physicians’ offices by the current EHR vendors by defining meaningful use of certified EHR’s to meet only the few and simple requirements listed above.

Flatlined and Overtreated

Both make it so clear that we have to fix the system that you wonder how some Senators can continue to blithly insist that there is no problem and that we have a “pretty good system”.  We have good doctors and technology. The system itself, as these books make clear, isn’t good at all.

It is certainly a risk and some of the answers aren’t intuitive. The ARRA act calls for meaningful use of the EHR’s and the NY Times article assumes that meaningful use is determined by improving care and curbing costs. This is of course the goal and if the EHR does a good job of real-time clinical decision support, it will undoubtedly improve care.

But it is an error to conflate physicians using EHR’s with curbing costs. To curb costs it is essential to create more ways and more choices for patients to own their computable health data thus enabling patients to use their data to get help and advice. This is usually called Data Liquidity and it empowers patients by adding choice into the equation. Put simply, EHR’s should, at the patient’s request, send the patient’s data (and it is the patient’s data after all) to their PHR (personal health record) of choice. Then, instead of being reliant on a single overworked physician to understand and interpret their data, they can have access to many tools and many people to help them stay healthy and on their own dime. History has shown that choice and competition lead to far better and more cost-effective results.

Some object to this model of patient controlled PHR’s. In the current issue of NEJM there is an article by Drs Paul  Tang and Thomas Lees suggesting that the best PHR is one tethered to the EHR. They argue that only this way can there be a shared patient record and only this way is the patient’s security and privacy be assured thanks to HIPAA. The article further contrasts the untethered PHR to the one run/managed by the doctor by suggesting that only the latter will help the patient to easily manage and understand their blood pressure and glucometer readings with intelligent doctor oversight.,This is very much a false dichotomy.

First, HIPAA does not give patients the ability to control who can see what of their data. It simply makes sure that only a doctor treating them can have access to it. But the patient has no fine grained control. Microsoft’s HealthVault, by contrast, does provide such protection. Clearly no one would use a PHR that leaked their health data so the business imperative for PHR’s to be secure is actually far greater than provider imperative.

Secondly, as the article points out, many patients see many doctors. The odds that all one’s doctors work for the same institution with the same EMR are both slim and presuppose, again, a lack of choice for the patient.

There is a much simpler solution. First, require the newly “certified” EHR’s that ARRA will pay for to share the patients Labs and Med’s and Conditions in a standard computable way with the patient’s PHR on patient request. Many countries  have long since figured out how to do this using SNOMED codes. Surely we can do the same using standard encoding’s for the Meds, Conditions, and Labs and standard XML formats like CCR and CCD to describe the patient data. Indeed many health organizations have have already done just this with Google Health such as Beth Israel Deaconess, Cleveland Clinic, Quest Diagnostics, CVS and Walgreens.

Second, provide extra rewards to EHR’s that support interoperable secure messaging using a standard way to exchange messages. Then patients can own their data, have a wide range of choices and still have an efficient way to communicate with their doctors. And a huge side benefit will be that doctors can now see the bigger picture of what is happening with their patients outside of their own practice.

All this will lead to truly collaborative medicine with a payoff that can only come from informed decision making and feedom of choice by the patients.

Folks, keeping our costs under control and our people healthy will require a new era of collaborative medicine.  What is collaborative medicine? It is a health-care system in which consumers and their physicians work together on an ongoing basis to avoid unnecessary illness and suffering. Imagine doctors who had the time to really help you or those you care about to stay healthy and well.  Imagine if every person who needed help with their health had not only a dedicated online tool to help them manage their health, but also their primary care physician looking over their shoulder to help them follow their personal plan. In general, data shows that mindful and aware people manage their health better. The point of this post is that not only can we afford collaborative medicine — we can afford it with the funds already promised in the stimulus bill.  Best of all, it will save us far more than it will cost us in the long run.

This is a simple proposal for how to get us there.

First I want to make a startling point. As I pointed out in my TEPR talk, out of the $2.3 trillion we spend on health care, incredibly less than 1% of it actually goes to the  primary care physicians. Put differently folks, what we think of as medicine — which is seeing our doctor — leaves only about $20 billion in our doctor’s pocket (not counting specialists) out of the total of $2.3 trillion spent. Think what a small percentage of our total health care costs that means are going to our doctors. If you subtract $20 Billion from $2.3 trillion, you get $2.28 trillion. All the rest of that $2.28 trillion is going to insurance filing, insurance processing, hospitals, labs, medicines, imaging, and specialists.

If you want to think about this more personally, our national health care costs average about $7500 for each man, woman, and child and more than $15,000 for people who actually have, or are at risk of having, serious illnesses (probably only 100 million but I figured 150 million here just to be safe). Of that $15,000, less than $100 is going to the primary care physicians to treat you. Because of this, there is a shortage of primary care physicians which is why, when Massachusetts insured all of its citizens, the biggest problem became just getting an appointment.   And, even if you can get an appointment, most doctors have no tools or time to really help you manage your health on an ongoing basis.

Really it seems that there are about 100 million people who need much more active management and care. These are the people who will cost us or are costing us so much money today. How can we help them? We can offer each primary care physician $100 for each at-risk or ill patient each year to provide collaborative online support to their patients and offer a monetary award to each patient who actually reduces his/her risk factors by even one risk factor. What will this cost and could it come out of the $20 Billion just approved for Health IT in DC?

At $100 per at-risk or ill patient, primary care physicians will earn $10 billion more, out of the $20 billion allocated to online innovation. That’s a tiny fraction of the $2.3 Trillion we spend today but it is a windfall for these doctors, like getting a 50% raise This will help attract some desperately needed freshly-minted doctors to primary care medicine. And because at least half of that $2.3 trillion is spent on treating preventable lifestyle diseases, it will not take much for our much-happier primary care docs to have profound impact on their patients lives:  reducing just ONE one risk factor triggers enormous reductions in health care costs.

How much will it cost to deliver the required IT to support collaborative medicine between doctor and patient?  As I pointed out in my talk at TEPR , you will find that those of us creating health IT online would have no problem delivering the tools to the doctors and consumers alike for collaborative medicine for a tiny fraction of $20 billion currently proposed by the legislation. I suggest that $3.6 billion is plenty to pay for systems will enable 100 million people to be helped by doctors, therapists, and coaches alike.

Data consistently shows that offering people a reward, even a modest one, for improving their health increases compliance. We have money left over for this. If $20 billion is being spent, and $10 billion goes to the doctors and $3.6 billion goes to the IT systems to support this,  that leaves $6.4 billion for the 100 million Americans who have lifestyle issues that have led or are leading to disease. Figure that we’re shooting for 15% of them to remove a risk factor. That’s 15 million Americans. With $6.4 billion, we can reward each of them with several hundred dollars each. Even if, optimistically, 30% of them improve we can  still provide about $200 per person as a reward. And the savings to the US of 15 million people dropping even one risk factor will be vastly greater than this. Dropping even one risk factor typically will cut someone’s expected annual health costs by $1000-$2000 – more when they’re older. Not to mention the reward to them of actually feeling better, being more mobile, and the peace of mind to those who love them.  As tax payers we are going to be paying this bill anyway — we should try to cut it now.

My proposal benefits consumers and doctors alike. The doctors are getting paid more to deliver better care that is actually targeted, with the help of IT, at providing ongoing coaching and support for patients to encourage them to improve their lifestyles. Doctors are being paid more to do a better and enjoyable job. The consumers finally have tools to manage their health but with the added value of having the person they most trust in this matter, their primary care physician oversee their progress and plan.  More doctors will become primary care physicians because the pay is better and the job is more fun, which will address the shortage issue we’ve seen in Massachusetts. It will mean fewer patients going to specialists and having lots of expensive treatments and stays in hospitals as they start to make the changes that help them avoid diabetes, heart disease, and the complications thereof.

We’re prepared to spend $20 billion on trying to improve Health IT anyway. Let’s spend it correctly.

But I did agree to give a talk at TEPR and I gave it this week. I gave it because we’re at an extraordinary point in our history. We have a new administration who actually wants change and can see that the current health care system needs to change.  The basic thesis of the speech is that our health care system is broken and on track to get worse, and that the only way to fix this is to get truly participatory health care between the doctors and the patients and their other health coaches. Surprisingly, as I point out in the talk, we can make a major dent in our current $2.3 Trillion of costs by covering every single American who has health issues for $50 Billion or just over 2% of the cost. Of this money, only $3.6 billion would actually go to the people building the Health IT systems. The rest, as is called out in the talk, would go to provide incentives to consumers and health providers alike to work collaboratively on their health and, in particular, for consumers to be able to be the stewards of their own health with online and ongoing support from the health care community.

Other than that, this is an exciting moment at Keas. We raised a bunch of money in December so we’re well funded (although we’re carefully husbanding our cash). We are just about to launch our first pilots. We have an amazing team but we are still looking for 2 good programmers including one who is really gifted at user interface, Javascript, CSS, HTML, etc. In addition, we’re looking for one totally can-do doctor, passionate about improving the health system, collaborative health care, and online patient support and willing to do whatever it takes to make that happen at a startup’s salary.

What is the Keas vision? If you are one of the many at risk of losing your health, Keas will help you keep healthy. If you’re recovering from an illness Keas will help you to recover and stay well. If you suffer from a chronic disease Keas will help you be as well as you can be. Today no one helps you. You can’t assemble your health data to get the best care possible. Even if you can, your doctors rarely help because the system doesn’t pay them to keep you healthy. You don’t have tools that work online to help in these situations, partly because insurance doesn’t pay for them. Because of these problems people suffer both personal hardship and fear and economic deprivation, sometimes irreversibly. What is more we all pay enormous medical costs for this, and there are costs to society and to the competitiveness of our companies in lost productivity. It is our mission at Keas to fix this for you. Clearly it isn’t an easy mission or a short-term one. While we think we have some great ideas about how to make this possible, we have a lot to learn in the course of this adventure.

What is our culture? We are all focused on making a great service that the customers love and that truly helps them. We want to have fun and make a difference and get it right. We want to build the service with love and care. Every day, we want to make sure that our customer experience is as good as it can be. This doesn’t mean trying to get it perfect out of the gate. Heck you don’t know until people use it. See my talk on intelligent reaction. What this does mean is pouring resources into constant improvement once the service is out of the gate and steadily learning from the usage patterns to make sure that the service quickly and surely evolves in the right direction. This requires great listening skills and great empathy and great patience and data analysis skills and, oh yes, some creative design insight. It requires the humility to realize that it isn’t your vision for how the UI should look that matters, but what actually works for the customer.

What kind of people are we hiring? We are hiring people who are really good at what they do, relaxed, persistent, pragmatic and fearless. It is important that the people who come fit this profile because in a start up everything is uncertain. You have to roll with the punches. You have to expect that your plan will change as the data starts to come in. Success in most start-ups isn’t instant. If it were easy, it would already be done. You have to keep trying. Some ideas work. Some don’t. What we’re trying to do at Keas certainly is hard. We need great people who aren’t afraid to try hard things but are willing to also look at the facts, see when something isn’t working, chalk it up to experience, and try the next hard thing. If you want predictability go to a big company.

Who are we still looking for? Engineers. We intend to keep Keas small until the ideas are proven (we’re self-funded). There is room, however, for a few engineers and for a proven development lead. In general people need to be able to get to San Francisco (we’re by Mission street and 1st Street close to everything), but we could use an engineer or two in the east coast as well or one or two willing to be there for significant periods of time. What kind of an engineer do you need to be? Well, in a word, excellent. More generally, productive, fun to work with, willing to tackle any problem, willing to work in Java or PHP or RubyOnRails, use machine learning or whatever language makes sense for the job, careful to avoid Not-Invented-Here when possible, good-humored, and burning to do something that really matters. It will be a small elite engineering team and we need it to really rock.

Why would you join? If we get this right, literally 100’s of millions of people who currently are on track to get ill or have chronic diseases and are at risk of getting still sicker will live longer and better lives because of you. In addition the ideas are genuinely interesting, hard, require thought, insight, and creativity. There is almost nothing easy in what we need to do. That means almost all of it is challenging and fun. You get to come in at the beginning of this adventure and that is always a blast. You’re going to get to work with wonderful people who care and want to make a difference. Ever since I moved into helping out in the health arena, I’ve met people I’m in awe of, people who are brilliant, caring, somehow hold down 4 jobs at once each of which would be full time for many of us, and still have happy personal lives. These people are going to love you because if this works, you’re going to be letting them run like no one else has. All you’ll have to do is work with them, listen carefully to them, and then be brilliant and quick. What more could you ask?

.

The obvious focus at the conference was on the train wreck that is US health care today. It became clear at the conference that if things continue as they are right now, we will manage to be spending $4.4 trillion dollars by 2015 or almost the entire Federal budget and still be delivering less than high quality health care and probably not solving the new epidemics of obesity and diabetes in this country. Clearly, this can’t go on. We are currently spending over 16% of our GDP on healthcare whereas France is around 8% and our overall health is worse. To go to the projected 30% would be a disaster and it was telling that not one but two heads of the congressional budget office (one past and one present) were attending the conference. One of the moments I found most unintentionally ironic was Bill Frist saying that of course we needed universal insurance as an obviously frustrated Elizabeth Teisberg pointed out with her usual lapidary clarity that insuring all would actually be cheaper because we would have preventative medicine rather than disaster handling, clinic care rather than ER care, and much less money spent by insurers trying to duck covering people since all would be guaranteed coverage. Where were the Republicans on this during the last eight years? But there was a thoughtful discussion about how this, all by itself, isn’t a solution, and we need to alter the system to actually reward people for good overall care and wellness of patients and good outcomes rather than paying doctors for procedures. It short we need the system to help keep people well rather than only treat them (at best) when they are sick.

There was an interesting talk given at the conference where it was claimed that of the four things you want from insurance:

• Access for all
• Affordable by all
• Quality care
• Constant innovation

you can only get two. The European system was held up as one that delivers on access and price, but not quality. Frankly, while I’m aware of some of the long and even unacceptable waits in the UK and the limited access of cancer patients there to new drugs, it isn’t clear to me that Europe isn’t in general getting three out of four while we get only one (innovation).

The most striking and shocking graphs at the conference were about type II diabetes and its rise in this country from a rare occurrence to a national epidemic in just 16 years due to an epidemic of obesity. Watching the time lapse graphs of this spreading across the US is like watching some terrifying science fiction movie about aliens taking over the country except that this is real and has happened. The terrible cost of this disease will dwarf that of cigarette smoking which, even now, kills 440,000 US citizens a year or ten times as many as breast cancer for example and compared to all coronary disease killing about 2,400,000 a year including those due to cigarettes. As it is, we cannot afford Medicare and as I said above, within eight years the cost of the current health system is projected to approach the total cost of the entire Federal government and we haven’t saved any money for this. Furthermore, incredibly, despite these crushing expenditures we are getting worse outcomes and longevity than countries spending half of what we do such as France and Japan.

So the focus was largely on why do we have this catastrophe and how do we do better? I’ll talk more about this later. While sitting at this conference it was depressing to watch the Republicans fighting and Bush vetoing the proposed extensions to SCHIP where the states are trying to extend to just guarantee medical coverage to poor kids who otherwise aren’t getting it where again the alternatives are ER room, much more severe problems, years of illness (these are kids!), or families going deeply into debt to keep their kids healthy.

It was fascinating to learn about how much genetics is now able to play a role in diagnosing illnesses and in even predicting high risk, but my overall take was that while it is amazing what we do now know, it is still a very very long step from here to actual drugs which take advantage of what we know to cure the diseases in question. What genetics clearly can do for us, even today, is start to inform us about when we need to live our lives with particular care because of an unusually high risk of diabetes or breast cancer or ventricular fibrillation.

Overall, I was struck by two very basic points:

1. We need to work much harder to help people stay healthy. The epidemic we face is largely avoidable. If we could reach out to the huge number of people currently at risk of type II diabetes or in the early stages and just get them to eat better and slightly less and exercise 30 minutes a day, we’d literally save this country hundreds of billions of dollars or thousands of dollars for every man, woman, and child in this country not even counting the terrible costs in lost productivity and illness and poor quality of life for those who suffer.
2. If we can just agree that the job of medical care is to keep people well or get them better and reward people for doing this well rather than paying insurers and middlemen and doctors for procedures, we would save even more not even counting again the terrible costs in lost productivity and the damage to our overall competitiveness.

I was asked in some comments since I restarted blogging to discuss what I learned while running Google Health at Google. It is a delicate subject because I’ve publicly blogged about a fair amount of what I learned and some of the rest I think is now Google’s business. So, I’ve added a sidebar listing the talks I did give publicly on health while I was still at Google. This is a starting point for learning what I learned while working on health during the last two years.

Lastly, while I was at the conference, Microsoft launched HealthVault. I want to commend Microsoft for launching HealthVault in Beta. The web desperately needs an ATM networks for health records so that we can find and connect to the expertise we need online with our health data as context be it interpreting our labs or warning us about medicine issues or helping us recover from an illness. I read a snarky blog complaining that they were copying Google Health. I don’t see it that way. They launched. They are doing a good thing. Consumers need to be able to take charge or their health data and control it. Not the federal government. Not the hospitals. Not the insurers. Us. Even if Google does launch something similar, competition is a good thing for all of us. That being said I certainly hope that Microsoft follows the principle that our health data is our data for us to control and allows those of us who put data into the health vault to easily take it out of the health vault or copy from it electronically if we so choose and provides an open doorway to those who have programs to help us make sense of our health data. But I bet they will and if so, congratulations to them.

I’ve been dusting off extremely rusty engineering habits and writing code. Not elegant code to be frank. Just enough to think through my ideas. Some extremely clear-headed and smart people can work out everything abstractly in their heads and then just go and implement it. I’m not one of them. Watching me write code is like watching an indecisive sculptor work with clay. I shape it. I look. I wince. I reshape it. I play with it. I wince some more. I ask my friends, nurse my wounds, and then reshape it yet again. And so on. Constant iterative development. It takes three tries before it is even close to the way it should be, best case. I think it is totally worth it. The arguments and design decisions are just way more concrete and tested.

However, I don’t delude myself that the code I’m writing is anything but prototype code. Prototype code is really sneaky. It sort of works and it is easy to kid oneself and that it is just a step from this code to the working product. Especially today with Amazon’s EC2 and DreamHost and frameworks and LAMP and Ruby on Rails where it seems that as soon as it works, you can scale it. In point of fact, I think the usual facts apply and it is still a long hard slog to get from prototype to product, but it is useful to get agreement about what needs to be done when, which kinds of people are required and when, and as a tool to chat with partners and potential employees and potential customers before the real thing is done. All that being said, Smart engineers welcome!!

Oh yeah, what am I building? Actually, I’m going to keep that to myself for a bit. Come work with me and you can find out, but otherwise, you’ll need to wait.

And why am I blogging again? Well, when at Google I noticed a strange thing. If I wrote a controversial post (and if you look at my sidebar on old posts you’ll see a few) people assumed I spoke for Google and got really annoyed at Google which wasn’t fair and was embarrassing since Google was treating me really well. So I desisted. But now, it is my company and I’m willing to take some of those risks. It is the great thing about it being your company. I’m always fascinated by what I learn. I should say that not all my posts will be about XML and databases or even AJAX. I do still care about technology and will write about it when the mood hits me, but I’m equally likely to write a review of a great book I’ve read or a complaint about the way the health system in this country works and what problems we’re running into building this startup.

And why did I switch URL’s from www.adambosworth.net? Sheer laziness. WordPress just makes it so easy and I liked some of the features.

Glad to be back in more ways than one.

To you guys who built the Sidekick II, first thanks and kudos. I love this machine.

Secondly some suggestions. I should be able to use pictures anywhere meaning what if I receive one in email, I can add it to my gallery and I can send them over IM and I can set my status profile in IM to show them and make them a background to the main screen. Alt and the spinner for editing text within text needs to be more clearly explained since fixing typos is a common need. Blackberry’s editing has two features you really need, holding a key down should shift it, not repeat it, and the spinner should go through the international versions of it. Also two spaces should put a period after the prior word. You need to beef up your spell correction. “youl” ought to become you’ll without me having to tell the machine. When looking at long pages or messages, “t” should take me to the top and “b” to the bottom. There should be some trivial way to add entries to the address book over email. These days, storage space is really cheap. I have 1gb on my Canon camera for $80. Let me do this here and store ALL my mail and a lot more photos and some MP3′s please. 100 mail messages isn’t enough. Build a decent calculator for school kids with sin and cos and so on. This can be the killer phone for the IM generation. And again, take advantage of pictures. Let me put the pictures people mail me into my gallery and, with 2 clicks add to a person or to my IM status or as the background for the home screen or send to someone over IM. Give me a replaceable battery and a way to charge from Firewire or USB.

Unlike many of my peers in the computer industry, I was a history major in college and have loved and read history ever since. I studied, in particular, the progressive era in history, an era when the industrial revolution evolved from the grim satanic mills of England into the modern industrial world. But the understanding I always had was that none of this would or could have been possible without the renaisaance and without the slow but sure rise of secular humanism and the spirit of scientific and intellectual inquiry that started at that time. After the fall of the Roman empire, in many ways the lights went out and, in the 14th century particularly, life in Europe hit a new low stroke the the terrible plague, the start of the mini ice age, and the wars between France and England. In the 15th century we saw the Spanish inquisition and the reconquista, but really, it was the last gasp of intolerant religious fanaticism and the spirit of inquiry and discovery from art to music to science was everywhere. The lights had been turned back on. As a child, growing up in New York City, I took for granted that mankind had learned these lessons. I assumed that mankind understood that freedom to think, to reason, and to experiment were paramount and that any irrational intolerant irrational beliefs that threatened these freedoms or, even worse, abused or injured people in the name of some mystical or fanatic cause were horrific reminders of the past.

I fear now for my children growing up into a world where the leaders turn their backs on the spirit of reason and inquiry. Where the new cardinals of the church deny evolution not on any grounds of empirical reason or evidence, but rather like children having a temper tantrum because they want it not to be so. Where the leaders of this country try to take Terry Shiavo’s husband to court not because of any evidence, but because they are angry to have been proven wrong by science. Where cowardly murderers kill innocent men, women, and children and claim to do it in the name of a religion, meaning something that no one can possibly argue with from a rational point of view. Where the education board of Kansas makes the state a mockery by demanding that irrationality be held to be as valid as science. Where 1.2 billion people consider it acceptable for some man with a vision to utter a Fatwa ordering some person killed simply because he doesn’t like what the other person chooses to believe in or even just disapproves of his line of inquiry. Where political correctness means that if some lines of inquiry are pursued, others feel free to harass and abuse and even threaten the people trying to find out the facts. Where people believe that they have the right to tell others what to believe, what to wear, what to eat, what to say, and what to think.

I fear because today, so many seem to fear to speak out. So many seem to fear to say that any “faith” which presumes to dictate to others not because there is some clear fair process that led to the dictates (laws voted on by people whose individual rights are protected by a constitution) or because they are clearly preventing bodily harm to others (preventing rape, robbery, murder, or abuse), but rather simply because people of that faith believe that they have the “right” to dictate to others is wrong. Those who believe otherwise are trying to drag us back to the 14th century because they fear and hate a world in which the facts trump irrational belief and where, therefore, inquiry may always show that their “obvious truths” are just obsolete shiboleths.

It is wrong. The belief that some imam has that he has the right to snuff out someone’s life through a Fatwa because of their apostasy or heresy is wrong. The belief that terrorists have that they have the right to kill and maim and burn innocent people because they are angry at injustice is wrong. The belief that some governments have that they have the right to kill and wound their own citizens when they protest peaceably is wrong. The belief that some Catholic prelates have that they can dictate what people believe about science is wrong.

It is all akin to letting a child’s tantrum dictate the judgement of an adult. It promotes irrationality over reason and faith over facts. The reason that homicidal madmen are so frightening is that one cannot reason with them. No more can one reason with the people who rule on the school boards in Kansas, the people who bomb innocent people in buildings and subways, the people who shoot their own citizens for protesting, or the prelates who presume to tells others how to live rather than simply choosing that way for themselves and hoping it acts as an example for others.

It is time to speak up. It is time to say that facts are what matter, not faith, that human progress is accomplished through unfettered use of reason and inquiry and tolerance and discussion and debate, not through intolerant and irrational acts of terror or edicts. For all of our children and for the future, speak up against this wave of intolerance and irrationalism washing over the world.

One clear limitation of XML is that there isn’t an easy way to update it. If you already have some XML and want to alter it in some way, there is simply no standard right now for doing this and the DOM code is usally both hideous to write and relatively fragile since there are no guaranteed ID’s on elements or checksums. Another obvious limitation is moving binary data about. One of the XML founders tells a hair-raising story of a company telling him how they plan to move video around by encoding it and including it in the XML itself. Because of all this, I’ve recently been spending a lot more time working with ATOM and RSS 2.0. These are XML, but they are more. They have the idea of sets which means that one can understand how to insert or replace items and ATOM has a protocol for this. It also means that some very low tech ways can be described to get subsets out of them. They support the idea of LastUpdated and ID so that replacing an item within the document can make sense. And they have explicit and very sensible ways to point to binary data which describes where it is, what type it is, and how big it is and then leaves it to the client to use normal ways to subsequently fetch in this data.

Anyway, I don’t mean to argue to strongly with the thoughtful comments on the previous post, but to caution that this is all engineering, not religion, and pragmatism should rule.

First, the applications that are taking off today in Ajax aren’t customer support applications per se. They are more personal applications like mail or maps or schedules which often are used daily. Also people are a lot more familiar with the web and so slowly extending the idiom for things like expand/collapse is a lot less threatening than it was then. Google Maps for example uses panning to move around the map and people seem to love it.

Secondly, the physics didn’t work in 1997. A lot of Ajax applications have a lot of script (often 10 or 20,000 lines) and without broadband, the download of this can be extremely painful. With broadband and standard tricks for compressing the script, it is a breeze. Even if you could download this much script in 1997, it ran too slowly. Javascript wasn’t fast enough to respond in real time to user actions let alone to fetch some related data over HTTP. But Moore’s law has come to its rescue and what was sluggish in 1997 is often lightning quick today.

Finally, in 1997 or even in 1999 there wasn’t a practical way to write these applications to run on all browsers. Today, with work, this is doable. It would be nice if the same code ran identically on Firefox, IE, Opera, and Safari, and in fact, even when it does, it doesn’t run optimally on all of them requiring some custom coding for each one. This isn’t ideal, but it is manageable.

My son (Alex Bosworth) posted a popular post a week ago on the pitfalls of Ajax applications but he left out some of the features still missing from Ajax applications:

First, printing is still hard. The browser has never grown up to enable the page author to easily describe an alternate layout for printing which is a shame. Why isn’t there an “HTML” for printing which can describe rotation, freeze column or row headers, and so on?

Secondly, the browser isn’t a good listener to external events. If you want to build an application, for example, to show you instantly when someone bids or a price changes, it is hard. You can poll, but poll too frequently and the application starts to feel sluggish and it isn’t easy to do this. What you really want is an event driven model where in addition to the events like typing the page can describe events like an XMPP message or a VOIP request or a data-changed post for an ATOM feed.

Third, if you want the application to run offline, you are essentially out of luck. I’ve written about this at length before in this blog and don’t need to repeat what is required in detail. To summarize what I said earlier, a local cache, a smart template model, and a synchronization protocol are required to build applications that run equally well connected and disconnected and the way that the Blackberry works is a role model for all of us here.

In fact, I’ve written about all these outages before (see Evolution in Action), but in the context of the current excitement around Ajax, it seems reasonable to describe not only what is different and making it work, but what is still missing. Obviously, these things are fixable from a technical point of view. This isn’t rocket science. But if only one browser fixes them, it is unlikely to help at this point. We have a sort of deadly embrace. It is hard to predict how this will play out. History has shown that when innovation is stifled, sooner or later some one runs around it who has nothing to lose and changes the rules of the game completely. I’m confident that this will happen here as well. But I honestly don’t know when.

What I predict will drive this change is the advent of truly mobile computing on mobile devices. This is going to force the game to change. It is way too expensive to build solutions for mobile on J2ME and often too poor a customer experience when they are built using WAP (except for super simple things). I think that we’re going to rethink browsing around a model which has pub/sub, events, and caching built in and which doesn’t have the problems with re-layout. More on this in a subsequent post.

My son and I have actually stopped our normal give and take on RSS and open source (his blog) and hacking and DRM and instead been talking about baroque versus classic, Gianlorenzo Bernini and Francesco Borromini and Michaelangelo Buonarroti and how amazing it is that they could sustain the energy and vision to complete the Basilica de San Pietro over a 120 year period, ten generations or more. It makes the sustained 1-2 year efforts we sometimes put into software seem so insignificant and the the fact that we sometimes don’t put enough effort into making it really beautiful even more criminal. I return with rededicated desire and hope to work on things that last, that matter, and that are as well designed for humans as they can be.

I haven’t posted for quite a while because my last posts caused unfair attacks on Google by twisting the words I’d used in my posts and attributing my posts to Google. I want to be really clear about something. The opinions I express in this Blog are my own. They have nothing to do with Google’s opinions. Google only asks that I not leak information about future products. Period. But despite that, recent blog posts of mine were used to attack Google and this upset me deeply. Much to my surprise, Dare Obasanjo came up to me and told me, after some fairly vitriolic complaining from me to him about this earlier state of affairs, that he wished I’d continue to post. I thought about this over the weekend and decided that to some degree, you have to take your chances in this environment rather than just hide when you don’t like the behavior and that perhaps I was being over sensitive anyway. There are too many interesting things going on right now anyway.

I’ve been complaining about two things on the web for years. Think of the web as the worlds best communication machine. Then the promise should be that anyone can connect to any information or application or anyone else and that any application can connect to anyone or any application or any information. We got anyone to anyone early in the form of email and more recently in the form of IM and of Blogs. IM adds real time communication and presence and Blogs add broadcasting to the world along with a dialog with the world. We got anyone to any application from the esteemed Tim Berners Lee in the form of HTML, HTTP, and URL’s which changed our world. I say applications because there wasn’t any standard way to ask for information. We got, unfortunately, any application talking to anyone (we call this spam). Web services in one form or another are letting applications access other application although, as I’ve said elsewhere, I think that the standards are too prolix and that a lot of the action will come out of REST and RSS.

But we didn’t get two things. We didn’t get a standard way to get information (e.g. a standard query model for sites). And we didn’t get people working together in communities to create and construct things with one interesting exception, message boards/groups. Mail was the interface, not the web and not IM. I’ve been whining about this for about 5 years off and on and even started a company once to try and address this.

With Open Search the lack of standard ways to get information is, for the first time, beginning to change. There is now a simple but de-facto standard way to start querying sites for information. That’s hugely exciting. The current standard is limited, but a great start. And the web is now rapidly becoming the place for people to collaborate. Wiki’s are growing like wildfire. Folksonomies(tagging) are causing people to quickly and in an emergent bottoms up way, come together to build taxonomies that work for them and surprisingly rapidly become stable. Flickr which Yahoo just bought is a great example of this and Del.icio.us by Joshua Schachter pioneered this model and Wikipedia has picked it up. I’ve always been hugely suspicious of top down taxonomies and restrictive ones (e.g. if you’re a book, you’re not a newspaper) and confident that normal people would never bother to classify things according to someone else’s taxonomy. But I think that tagging has broken through that. It is sufficiently KISS (see my early talks on this for why I think this is good) and rewarding (you get attention if you pick popular tags) to have gained amazing momentum. The clever and audacious Dave Sifry of Technorati claims to have found 5MM tagged posts just in the last 2 and a half months (from del.icio.us and from Flickr and from various blogs). As long as we don’t let the ontologists take over and tell us why tags are all wrong, need to be classified into domains, and need to be systematized, this is going to work well albeit, sloppily. What it does is open up ways to find things related to anything interesting you’ve found and navigate not a web of links but a link of tags. At the same time Wikipedia has shown that a model in which content is contributed not just by a few employees, but by self-forming self-managing communities on the web can be amazingly detailed, complete, and robust. so now people are looking at ways in which the same emergent self-forming self-administering models of tagging and Wiki’s and moderation can be used for events (EVDB) and for music and for video and for medical information. It’s all very exciting. It is a true renaissance. I haven’t seen this much true innovation for quite a while. What I particularly like about all this is how human these innovations are. They are sloppy. To me Tags are sloppy practical de-facto ontologies. Wiki’s are sloppy about changes and version editing. It is accepted that we’re trying new things and that sometimes messes will occur. In short, it is unabashedly creative and imprecise. I’ve always believed in the twin values of rationalism and humanism, but humanism has often felt as though it got short shrift in our community. In this world, it’s all about people and belonging and working with others.

In this very triumph, comes the tension and the problems. Every one of these groups has to worry about spam. Wikipedia does occasionally get spammed, but their entries are long-lasting and resilient and usually get fixed. For information that is more time critical and evanescent however, this sort of vandalism can be much more harmful. While the commercial part of this is detestable, it is at least comprehensible. Often though, there are pointless attacks. Much as vicious people constantly invent viruses to destroy the existing web (amusingly now called the old web) and somehow tell themselves that this vandalism is acceptable, so they destroy the tenor of message boards, vandalize Wiki’s, screw up the tags just because they can and generally try to attack, smear, and destroy. The blogger world calls these people Trolls. Message boards end up needing moderation techniques because of these people. Bloggers learn to turn off comments. And Taggers will end up having to use reputation and other techniques to protect something hugely useful but potentially fragile, or to create gated communities like the old fortresses in history built to keep the vandals out. A great deal of the discussion at etech and even at PC Forum was how to keep the vandals from doing too much damage. Sadly, one reaction may be to curtail anonymity because it is so abused and with the loss of anonymity comes the loss of privacy.

Indeed the other concern is privacy. Presence is in the air. The web because of mobile and broadband and IM is becoming real-time. Real time presence changes everything and rapidly leads to thinking about much richer ways of communicating within communities. It highlights some of the, in my opinion, few limitations of the browser as a zero deployment user interface model. But it also risks us losing those last moments of privacy. Lufthansa has announced that it will support internet on planes. I will not fly on them. I need some periods in my life where I am unreachable. Indeed, every year in August, I vanish for a month from the web, turn off email, and deal with the withdrawal and suddenly I relearn how to think and concentrate. In a world where knowledge and thinking is everything, it is ironic that increasing availability had led to decreasing time in which to reflect, ponder, and just let the mind wander and yet these periods tend to be essential to truly thinking hard. If Nokia sold a phone that reported where I was at all times through presence (as some phone vendors actually already do) I wouldn’t buy it. We’re going to have to work out how to support all this in a manner in which the customers can effortlessly and intuitively opt in and out so that, when they want, they can be left alone and vanish from view and can control who can see them when.

It is going to be fascinating and exciting to watch how these tensions play out, namely the rising trend of people working together and collaborating and communicating over the web in increasingly real time ways contending with the human needs for privacy and reflection and with the unfortunate nature of some humans to vandalize rather than to construct.

Many years ago, Eric Michelman and Brad Silverberg and Ken Ong and I built a product, Reflex, before Windows, but with a GUI front end for a PC. In order to build it, we had to build fonts and event managers and heap managers and graphical routines and so on. Later Windows/Mac came along and made all this unnecessary, but then we still had to build huge amounts of code in Access for database and indexing and and much more code for rendering within Windows, brushes and XOR and line drawing and manual bitblitting. Still later we built a browser and we helped to build shared relational databases. As these became a central theme in most applications, we realized that it was now possible for anyone to build Reflex with infinitely less work because the database work was done as was the rendering (the browser) and so one could focus purely on the other issues. This is the nature of science, of learning, of education, of engineering and of software. We all benefit from those who came before us. We benefit most when the knowledge is free and generally accessible., but we benefit either way. It would seem that these cacophonous critics, yammering about giving back and sweepingly ignoring the 100′s of billions of times people use and appreciate what Google gives them for free every day from Search to Scholar to Blogger to gMail to Picasa, do not understand this basic fact.

Suggesting new lines of learning and research is no sin. It is how we grow and add value and has been throughout human history. Taking advantage of what has already been learned and taught is equally no sin. It is common sense and to do otherwise is usually a sign of hubris, arrogance, and immaturity.

Giving back is always done through what one is good at, be it making accessible the world’s literature and learning and knowledge online along with tools to search it, create it, and communicate about it, or through making the world’s goods available if that is one’s business. This is how we are all rewarded for casting our bread upon the waters. It is how economies grow and culture flourishes. And the fact that the critics of the earlier post seem to understand none of this suggests a world view so narrow minded as to make one gasp in wonder and horror.

It says that Google essentially has a parasitic relationship with the open source community. As it turns out, Google actually puts quite a lot of time and effort (and occasionally money) into supporting open source efforts.

It suggests that my post speaks for Google in what I ask for. Actually I don’t. Google is dong just fine with respect to storage and indexing. It has built what it needs to support its products. I really was speaking based far more on what I heard from many many large corporate customers and from almost all services.

Lastly, the counter post assumes that saying software is free is the same as saying that you can’t make money from it. In fact, customers are always willing to pay for support and service. Essentially, long ago, Microsoft turned itself largely into an annuity business where the licences companies hold with Microsoft are just that, payments for guaranteed support and ongoing upgrades. Many open source vendors are now doing the same. My point was not that people should never get paid, but rather that it makes more sense to pay for the ongoing support.

Nevertheless, it is a good and entertaining post and I recommend reading it for a different point of view.

It is this. Users of databases tend to ask for three very simple things:

1) Dynamic schema so that as the business model/description of goods or services changes and evolves, this evolution can be handled seamlessly in a system running 24 by 7, 365 days a year. This means that Amazon can track new things about new goods without changing the running system. It means that Federal Express can add Federal Express Ground seamlessly to their running tracking system and so on. In short, the database should handle unlimited change.

2) Dynamic partitioning of data across large dynamic numbers of machines. A lot people people track a lot of data these days. It is common to talk to customers tracking 100,000,000 items a day and having to maintain the information online for at least 180 days with 4K or more a pop and that adds (or multiplies) up to a 100 TB or so. Customers tell me that this is best served up to the 1MM users who may want it at any time by partioning the data because, in general, most of this data is highly partionable by customer or product or something. The only issue is that it needs to be dynamic so that as items are added or get “busy” the system dynamically load balances their data across the machines. In short, the database should handle unlimited scale with very low latency. It can do this because the vast majority of queries will be local to a product or a customer or something over which you can partion. It is, obviously, going to come at a cost for complex joins and predicates across entire data sets, but as it turns out, this isn’t that normative for these sorts of data bases and an be slower as long as point 3 below is handled well. And a lot of them can be solved with some giant indices that cover the datasets that are routinely scanned across customers or products.

3) Modern indexing. Google has spoiled the world. Everyone has learned that just typing in a few words should show the relevant results in a couple of hundred milliseconds. Everyone (whether an Amazon user or a customer looking up a check they wrote a month ago or a customer service rep looking up the history for someone calling in to complain) expects this. This indexing, of course, often has to include indexing through the “blobs” stored in the items such as PDF’s and Spreadsheets and Powerpoints. This is actually hard to do across all data, but much of the need is within a partioned data set (e.g. I want to and should only see my checks, not yours or my airbill status not yours) and then it should be trivial.

By the way, the inherent cost of the machines to do all this is relatively negligible. Assume 3 by 400GB cheap disks per machine mounted in racks of 60 and one rack would pretty much do it if there wasn’t a need for redundancy and logs, say two racks to cover that. Companies are already coming out this year with highly redundant disk arrays for $1 per GB or $1200 / machine for the ones above (not counting the $1000 for the machine and memory itself). In short, for 120 such machines, it will probably cost less than $500K and that’s less than 3-4 good programmers and it is one time a capital cost. But the cost to most people I’ve spoken to in terms of actual people to build and administer such systems is an order of magnitude more. For that matter, configure the 120 machines with 4GB each of memory and you could normally keep the current days work in memory and in many of these cases the data accessed will be the current days as people look for their waybills or flight statuses or check their Blog comments or whatever.

Users of databases don’t believe that they are getting any of these three. Salesforce, for example, has a lot of clever technology just to hack around the dynamic schema problem so that 13,000 customers can have 13,000 different views of what a prospect is.

If the database vendors ARE solving these problems, then they aren’t doing a good job of telling the rest of us. The customers I talk to who are using the traditional databases are esentially using them as very dumb row stores and trying very hard to move all the logic and searching out into arrays of machines with in memory caches. Oracle is doing some very clever high end things with streaming queries and the ability to see data as of some point in recent history (and even which updates affected the query within some date range) and with integrated pub/sub and queueing, but even Oracle seems to make systems too static and too ponderous to really meet the needs about and, oh yes, they seem to charge about ten times as much as one would expect for them.

Indeed, in these days of open source, I wonder if the software itself, should cost at all? Open Source solutions would undoubtedly get hacked more quickly to be robust and truly scalable across nice simple software. It wouldn’t be as pointwise fast, but the whole point is that these systems will scale linearly and are so cheap that it doesn’t matter. The advantage of Open Source is that those folks really understand how to build scalable clouds of machines with a default assumption of failure and load balancing. It’s called Apache. There are some other interesting problems that the database vendors are also ignoring but for now (like how do I ask for the set of complaints that are like the ones this customer has) but for now the three above seem like the big ones to me. My message is to the Open Source community that has, so ably, built LAMP (Linux, Apache and Tomcat and MySQL and PHP and PERL and Python). Please finish the job. Do for databases what you did for web servers. Give us dynamism and robustness. Give us systems that scale linearly, are flexible and dynamically reconfigurable and load balanced and easy to use.

Light that LAMP for us please.

I bring this up because these have been hard times for these beliefs. We are faced on the one hand with people who in the name of conservatism try their very best to equate morality with religion, equate religion with the right to kill, and then act on these beliefs to detroy people’s lives, their freedom to choose, and their trust in each other. I do not speak only of John Ashcroft and the Christian Right. I speak equally of the Islamic fanatics who seem to stand against everything I believe in and hold dear, namely the triumph of rationalism and humanism. Both sides seem to me to be sides of the same coin, quick to kill, quick to detest and fear and dislike those who would think for themselves and hate those who make their own moral choices and in a hard fought way find their own paths to a moral high ground. Both sides contend that one cannot be moral if one is not religious and both sides claim to be inspired by faith (which is of course unarguable). This is an abdication of man’s personal responsibility to figure out what it right and wrong, to behave with integrity and honor and kindness and justness in equal measure. It is the very antithesis of both humanism and rationalism.

For years I have been a conversative because I have believed that the only role of Government should be to protect the rights of people, not control their outcomes, leaving to ability and chance the right of people to grow and succeed or fail, and the left in the US seemed to have forgotten that. But this year, I had to switch. It was simply too appalling to stand with those who would detroy people’s freedom and detroy people’s lives in the name of their “faith”.

So let me suggest that Christmas be rededicated to a belief and faith in the human spirit, to a belief and a faith in the ability of humans of all beliefs and types to treat each other with dignity and respect and in the need to counter the inherent evil of treating people poorly simply because they do not share your irrational beliefs.

“Adam, have you ever considered that HTML and Javascript have almost wiped out software engineering from the face of this earth? Was it desirable to build (web) applications with such sloppy technologies which complexity is adapted to other classes of problems but which adoption has guaranteed hefty revenues to companies like BEA or Microsoft? IT doesn’t matter today, not because like machine tools or electricity everyone can acquire them, IT doesn’t matter today because sloppy technologies prevent companies to build mission critical systems at a reasonable cost and reasonable risk and most customers had to result to adopt SAP or PeopleSoft view of the world in attempt to diminish this cost and risk.”

In the face of such hyperbole, it is always debatable whether to respond or not. In some sense a response legitimizes what it should not. I was going to write a long article about real economics, about supply and demand and about how in a free market, like it or not, people get to choose what they want and what works for them even if IT doesn’t like it (as IT didn’t like spreadsheets). But I don’t need to. Paul Graham’s Hackers and Painters says it all, far better than I could ever hope to. The eponymous chapter alone, Hackers and Painters, is worth the read, but Paul’s intense desire to build a programming language for humans is what makes the book for me (even though I don’t really agree with his solution being a sort of PHP fan myself). I read a fair amount (son of a librarian, it sticks), and of the books I’ve read this year, this one will stick with me long after many others have faded. The book resonates with all the reasonableness and pragmatism and deep understanding of the human condition and of economics that is absent in the quoted paragraph above.

Mr Dubray, if you read nothing else, read Paul’s chapter on the other road ahead. It says what I said in my post in this blog on Evolution in Action, but Paul says it so much more eloquently and completely. It should help you to understand that the problem IT faces isn’t sloppy languages. It is irrelevance. For much (although certainly not all) of the work IT does, IT is like children building sand castles on the beach and watching the tide roll in. That tide is highly customizable web based solutions, Salesforce.com today, perhaps Talaris tomorrow. Ask the average Salesforce.com customer (meaning a salesrep) if he is happier with the solution he has now or the one he had back when IT was building a customer CRM for him. I think the answer will surprise you. Web Services have helped immensely here because it has made possible the integration of these solutions with internal logic for those things IT should still be working on. This is the promise and the future in my opinion.

Several people have complained that I was unfair to CSS. I didn’t mean to say that people should never use CSS. I use it in this Blog. It is a good thing. I like CSS. What I did mean is that when just trying to create a table with 2 cells on the left and one on the right, I don’t want to figure out the CSS for that. And, asking around, neither do a lot of other people. Mostly, I was just being amused that people try to be pixel precise in HTML when that wasn’t the original intent.

Still trying to be a 21st century kind of person.

I�m sandwiched by smarter and more august speakers. Don Ferguson of IBM builds edifices of such sophistication and elaboration as to daunt the designers of the extraordinary archways of the Alhambra. Tim Berners Lee created the web as we know it today and preaches a sort of religion about the semantic web from his eerie at MIT that is totally over my head. These are very smart gentlemen. One would be foolish to try to appear smart when speaking between them. Accordingly, I�m going to take the opposite tack. I�m going to talk about the virtues of KISS which I�ll conveniently describe as keeping it simple and sloppy and its effect on computing on the internet.

There has been, of course, an eternal tension between that part of humanity which celebrates our diversity, imperfectability, and faults, as part of the rich tapestry of the human condition and that part which seeks to perfect itself, to control, to build complex codes and rules for conduct which if zealously adhered to, guarantee an orderly process.

This talk is about this conflict as it relates to computing on the Internet. This talk is also a polemic in support of KISS. As such it is unfair, opinionated, and perhaps even unconscionable. Indeed, at times it will verge on a jeremiad.

It is an ironic truth that those who seek to create systems which most assume the perfectibility of humans end up building the systems which are most soul destroying and most rigid, systems that rot from within until like great creaking rotten oak trees they collapse on top of themselves leaving a sour smell and decay. We saw it happen in 1989 with the astonishing fall of the USSR. Conversely, those systems which best take into account the complex, frail, brilliance of human nature and build in flexibility, checks and balances, and tolerance tend to survive beyond all hopes.

So it goes with software. That software which is flexible, simple, sloppy, tolerant, and altogether forgiving of human foibles and weaknesses turns out to be actually the most steel cored, able to survive and grow while that software which is demanding, abstract, rich but systematized, turns out to collapse in on itself in a slow and grim implosion.

Consider the spreadsheet. It is a protean, sloppy, plastic, flexible medium that is, ironically, the despair of all accountants and auditors because it is virtually impossible to reliably understand a truly complex and rich spreadsheet. Lotus corporation (now IBM), filled with Harvard MBA�s and PhD�s in CS from MIT, built Improv. Improv set out “to fix all this”. It was an auditors dream. It provided rarified heights of abstraction, formalisms for rows and columns, and in short was truly comprehensible. It failed utterly, not because it failed in its ambitions but because it succeeded.

Consider search. I remember the first clunky demos that Microsoft presented when Bill Gates first started to talk about Information at your fingertips with their complex screens for entering search criteria and their ability to handle Boolean logic. One of my own products, Access had the seemingly easier Query by Example. Yet, today half a billion people search every day and what do they use? Not Query by Example. Not Boolean logic. They use a solution of staggering simplicity and ambiguity, namely free text search. The engineering is hard, but the user model is simple and sloppy.

Consider user interface. When HTML first came out it was unbelievably sloppy and forgiving, permissive and ambiguous. I remember listening many years ago to the head, then and now, of Microsoft Office, saying contemptuously in 1995 that HTML would never succeed because it was so primitive and that Word would win because Word documents were so rich and controlled in their layout. Of course, HTML is today the basic building block for huge swathes of human information. What is more, in one of the unintended ironies of software history, HTML was intended to be used as a way to provide a truly malleable plastic layout language which never would be bound by 2 dimensional limitations, ironic because hordes of CSS fanatics have been trying to bind it with straight jackets ever since, bad mouthing tables and generations of tools have been layering pixel precise 2 dimensional layout on top of it. And yet, ask any gifted web author, like Jon Udell, and they will tell you that they often use it in the lazy sloppy intuitive human way that it was designed to work. They just pour in content. In 1996 I was at some of the initial XML meetings. The participants� anger at HTML for �corrupting� content with layout was intense. Some of the initial backers of XML were frustrated SGML folks who wanted a better cleaner world in which data was pristinely separated from presentation. In short, they disliked one of the great success stories of software history, one that succeeded because of its limitations, not despite them. I very much doubt that an HTML that had initially shipped as a clean layered set of content (XML, Layout rules – XSLT, and Formatting- CSS) would have had anything like the explosive uptake.

Now as it turns out I backed XML back in 1996, but as it turns out, I backed it for exactly the opposite reason. I wanted a flexible relaxed sloppy human way to share data between programs and compared to the RPC’s and DCOM’s and IIOP’s of that day, XML was an incredibly flexible plastic easy going medium. It still is. And because it is, not despite it, it has rapidly become the most widely used way to exchange data between programs in the world. And slowly, but surely, we have seen the other older systems, collapse, crumple, and descend towards irrelevance.

Consider programming itself. There is an unacknowledged war that goes on every day in the world of programming. It is a war between the humans and the computer scientists. It is a war between those who want simple, sloppy, flexible, human ways to write code and those who want clean, crisp, clear, correct ways to write code. It is the war between PHP and C++/Java. It used to be the war between C and dBase. Programmers at the level of those who attend Columbia University, programmers at the level of those who have made it through the gauntlet that is Google recruiting, programmers at the level of this audience are all people who love precise tools, abstraction, serried ranks of orderly propositions, and deduction. But most people writing code are more like my son. Code is just a hammer they use to do the job. PHP is an ideal language for them. It is easy. It is productive. It is flexible. Associative arrays are the backbone of this language and, like XML, is therefore flexible and self describing. They can easily write code which dynamically adapts to the information passed in and easily produces XML or HTML. For them, the important issue is the content and the community, not the technology. How do they find the right RSS feeds? How do they enable a community to collaborate, appoint moderators, and dynamically decide whose posts can go through and whose should be reviewed? How do they filter information by reputation? These are the issues that they worry about, not the language.

In the same way, I see two diametrically opposed tendencies in the model for exchanging information between programs today:

On the one hand we have RSS 2.0 or Atom. The documents that are based on these formats are growing like a bay weed. Nobody really cares which one is used because they are largely interoperable. Both are essentially lists of links to content with interesting associated metadata. Both enable a model for capturing reputation, filtering, stand-off annotation, and so on. There was an abortive attempt to impose a rich abstract analytic formality on this community under the aegis of RDF and RSS 1.0. It failed. It failed because it was really too abstract, too formal, and altogether too hard to be useful to the shock troops just trying to get the job done. Instead RSS 2.0 and Atom have prevailed and are used these days to put together talk shows and play lists (podcasting) photo albums (Flickr), schedules for events, lists of interesting content, news, shopping specials, and so on. There is a killer app for it, Blogreaders/RSS Viewers. Anyone can play. It is becoming the easy sloppy lingua franca by which information flows over the web. As it flows, it is filtered, aggregated, extended, and even converted, like water flowing from streams to rivers down to great estuaries. It is something one can get directly using a URL over HTTP. It takes one line of code in most languages to fetch it. It is a world that Google and Yahoo are happily adjusting to, as media centric, as malleable, as flexible and chaotic, and as simple and consumer-focused as they are.

On the other hand we have the world of SOAP and WSDL and XML SCHEMA and WS_ROUTING and WS_POLICY and WS_SECURITY and WS_EVENTING and WS_ADDRESSING and WS_RELIABLEMESSAGING and attempts to formalize rich conversation models. Each spec is thicker and far more complex than the initial XML one. It is a world with which the IT departments of the corporations are profoundly comfortable. It appears to represent ironclad control. It appears to be auditable. It appears to be controllable. If the world of RSS is streams and rivers and estuaries, laden with silt picked up along the way, this is a world of Locks, Concrete Channels, Dams and Pure Water Filters. It is a world for experts, arcane, complex, and esoteric. The code written to process these messages is so early bound that it is precompiled from the WSDL�s and, as many have found, when it doesn’t work, no human can figure out why. The difference between HTTP, with its small number of simple verbs, and this world with its innumerable layers which must be composed together in Byzantine complexity cannot be overstated. It is, in short, a world only IBM and MSFT could love. And they do.

On the one hand we have Blogs and Photo Albums and Event Schedules and Favorites and Ratings and News Feeds. On the other we have CRM and ERP and BPO and all sorts of enterprise oriented 3 letter acronyms.

As I said earlier, I remember listening many years ago to someone saying contemptuously that HTML would never succeed because it was so primitive. It succeeded, of course, precisely because it was so primitive. Today, I listen to the same people at the same companies say that XML over HTTP can never succeed because it is so primitive. Only with SOAP and SCHEMA and so on can it succeed. But the real magic in XML is that it is self-describing. The RDF guys never got this because they were looking for something that has never been delivered, namely universal truth. Saying that XML couldn�t succeed because the semantics weren�t known is like saying that Relational Databases couldn�t succeed because the semantics weren�t known or Text Search cannot succeed for the same reason. But there is a germ of truth in this assertion. It was and is hard to tell anything about the XML in a universal way. It is why Infopath has had to jump through so many contorted hoops to enable easy editing. By contrast, the RSS model is easy with an almost arbitrary set of known properties for an item in a list such as the name, the description, the link, and mime type and size if it is an enclosure. As with HTML, there is just enough information to be useful. Like HTML, it can be extended when necessary, but most people do it judiciously. Thus Blogreaders and aggregators can effortlessly show the content and understanding that the value is in the information. Oh yes, there is one other difference between Blogreaders and Infopath. They are free. They understand that the value is in the content, not the device.

RSS embodies a very simple proposition that Tim Berners Lee has always held to be one of the most important and central tenets of his revolution, namely that every piece of content can be addressed by a URL. In the language of RSS we call these �PermaLinks�. This idea has profound value. Dave Sifry of Technorati pointed out to me recently that one of the most remarkable things about RSS and Web Logs (Blogs) is the manner in which they have started to solve one of the most tragic things about the web, namely the incivility of the discourse. The web, in many ways, today represents a textbook example of the tragedy of the commons. Because sending email is virtually free, we have spam. Because posting messages is virtually free and anonymous, we have groups where a small number of people can overwhelm the discussion with loud and senseless chatter. But one of the values of being able to reference every element is that now comments about elements can be distributed over the web. The web becomes something like a giant room in which people comment on other people�s thought via posts in their own Web Logs. In so doing they put their reputation on the line. These are hardly cheap and anonymous posts. They take up real estate in a place that is associated with your own point of view and reputation. And thus the comments tend to be measured, thoughtful, and judicious. Furthermore if they are not, either you can decide that it is OK or you can opt out. It is like dueling editorials in a pair of news papers.

By contrast, the rigid abstract layers of web service plumbing are all anonymous, endless messages flowing through under the rubric of the same URL. Unless they are logged, there is no accountability. Because they are all different and since the spec that defines their grammar, XML Schema, is the marriage of a camel to an elephant to a giraffe, only an African naturalist could love these messages. They are far better, mind you, than the MOM messages that preceded them. Since they are self describing, it is possible to put dynamic filters in to reroute or reshape them using XPATH and XSLT and XML Query and even other languages all of which can easily detect whether the messages are relevant and if so, where the interesting parts are. This is goodness. It is 21st century. But the origination and termination points, wrapped in the Byzantine complexity of JAX RPC or .NET are still frozen in the early bound rigidity of the 20th.

I would like to say that we are at a crossroads, but the truth is never so simple. The truth is that people use the tools that work for them. Just as for some programmers the right tool is PHP and for others the right tool is Java so it is true that for some programmers the right tool is RSS and for others it is WS-*. There is no clear �winner� here. What I am sure about is the volumes and the values. The value is in the information and its ability to be effortlessly aggregated, evaluated, filtered, and enhanced.

What does this mean to you? Think of the radio. When it was a novelty, the real value was in the radio itself. There was relatively little content, but lots of people wanted the radio. At a certain point, however, radios got good enough and transmission got good enough and the value ineluctably swung to the content. This is why the DRM fights are so bitter, why PodCasting is so revolutionary, why Howard Stern was paid so much to play on a private radio model. That�s where the value is. We have arrived at the same point for computing. The value is neither in the computers nor in the software that runs on them. It is in the content and the software�s ability to find and filter content and in the software�s ability to enable people to collaborate and communicate about content (and each other). Who here really cares if Excel adds a new menu item unless it is one that lets you more easily discover information on the web, possibly update and interact with it or with others about it.

What about mobile phones. What do they mean? Is it really interesting to have a spreadsheet or a power point on your mobile phone? Or is it more interesting to know where the nearest ATM is, where is the nearest Indian restaurant that your friends like, which are the CS books in the store for a given course, which course has the best section person and what its schedule is, or what the reviews of the books say. Is it really interesting to have an address book that is synced to your PC or is it more interesting to see the presence of the people who are involved in your class, your project, your party plans, and be able to coordinate and plan an event with them? And if it is the latter, then isn�t the value really coming from the knowledge of with whom you are working, socializing, and studying; what they think about things you care about such as movies, classes, restaurants and news articles rather than the software on the device itself? Isn�t the device really just a sort of n-way radio/classified? Soon as you deliver context and content and community and collaboration over the web, 2 billion people will be able to see and interact with your solutions.

There is a lot of talk about Web 2.0. Many seem to assume that the �second� web will be about rich intelligent clients who share information across the web and deal with richer media (photos, sound, video). There is no doubt that this is happening. Whether it is Skype or our product Hello, or iTunes, people are increasingly plugging into the web as a way to collaborate and share media. But I posit that this isn�t the important change. It is glitzy, fun, entertaining, useful, but at the end of the day, not profoundly new.

What has been new is information overload. Email long ago became a curse. Blogreaders only exacerbate the problem. I can�t even imagine the video or audio equivalent because it will be so much harder to filter through. What will be new is people coming together to rate, to review, to discuss, to analyze, and to provide 100,000 Zagat�s, models of trust for information, for goods, and for services. Who gives the best buzz cut in Flushing? We see it already in eBay. We see it in the importance of the number of deals and the ratings for people selling used books on Amazon. As I said in my blog,
�My mother never complains that she needs a better client for Amazon. Instead, her interest is in better community tools, better book lists, easier ways to see the book lists, more trust in the reviewers, librarian discussions since she is a librarian, and so on�.
This is what will be new. In fact it already is. You want to see the future. Don�t look at Longhorn. Look at Slashdot. 500,000 nerds coming together everyday just to manage information overload. Look at BlogLines. What will be the big enabler? Will it be Attention.XML as Steve Gillmor and Dave Sifry hope? Or something else less formal and more organic? It doesn�t matter. The currency of reputation and judgment is the answer to the tragedy of the commons and it will find a way. This is where the action will be. Learning Avalon or Swing isn�t going to matter. Machine learning and inference and data mining will. For the first time since computers came along, AI is the mainstream.

I find this deeply satisfying. It says that in the end the value is in our humanity, our diversity, our complexity, and our ability to learn to collaborate. It says that it is the human side, the flexible side, the organic side of the Web that is going to be important and not the dry and analytic and taxonomical side, not the systematized and rigid and stratified side that will matter.

In the end, I am profoundly encouraged and hopeful that the growth on the web is one which is slowly improving the civility and tenor of discourse. Just as Porn seems to be an unpleasant leading user of technology, so does crude and vituperative communication seem to be a pattern for early adopters and it is a relief to see that forms of governance, trust and deliberation are now emerging.

There are those who will say that all this is utopian. If Utopian means not being afraid to dream, then indeed it is. So was Tim�s initial vision of universal access to information. So is Google�s mission. T.E Lawrence wrote in the Seven Pillars of Wisdom,

�All men dream: but not equally. Those who dream by night in the dusty recesses of their minds wake in the day to find that it was vanity: but the dreamers of the day are dangerous men, for they may act their dream with open eyes, to make it possible�

I encourage all of you to act your dreams with open eyes. I encourage all of you to dream of an internet that enables people to work together, to communicate, to collaborate, and to discover. I encourage all of you to remember, that in the long run, we are all human and, as you add value, add it in ways that are simple, flexible, sloppy, and, in the end, everything that the Platonists in you abhor.

One concern expressed by several readers including Danny Ayers, Blaine Cook,Rimantas’ murmers, Alek Blog, and so on. A second point made is that one can have a rich client that dynamically reads content and then users can customize to their hearts content. Both points are valid. A third point, which I think is specious and I’m going to quickly dismiss, is that it is easier to run the service yourself. I think this is geek talk. For everyone who wants to run his or her own blog server, there are in my opinion 100 people like my mother or wife or daughter or brother in law or father who would never even consider it.

Someone called “Joe” sent in mail which said “I’ve done a lot of web development, DHTML, JSP, ASP. I think most developers would agree a lot of the time you spend trying to work around limitations of the platform rather than working on customer features”. Sure. Heck, in my prior life I designed and built DHTML and started ASP’s while I was at Microsoft. So guilty as charged. But programming in windows or Java is different? Come on.

More interestingly Joe, whoever you are, I think the best way to comment is the way that Blaine Cook did by posting on his blog. Dave Sifry of Technorati argues that the best comments are response posts (like this) because then you are putting your own reputation on the line in a sort of web wide extended conversation. I don’t think it is a coincidence that Blaine’s comment is one of the most penetrating and brings up the term “open data”. Notice, the value is coming from the community and the reputation and the content, not the tool used to author the post which is largely irrelevant.

Blaine, I think the issue of “open data” is a great one. We are going to need to see an “open data” model for people to want to put their data anywhere. Services will need to provide value to justify themselves, not data lock in. The value may be in the form of additional information, community discussion, ratings, search, data management, publishing features, relieving you of the tedium/cost of operations or monetization, but they will need to deliver value in a way where, if the value isn’t there, you can walk.

On the point of richer user interfaces I’m asked if it isn’t better to have rich clients (like Java or Macromedia) on top of open data rather than web based user interface. My short answer is that it depends. If I need offline access, at least today, then sure. If I need rich ways to manipulate media (photos, sound, video, voip) then sure. And I said so in the prior post I think. But in general, no. Because these apps don’t evolve as quickly. Even iTunes which is lauded as a sucess in the comments hasn’t evolved much for me and still, for example, has no community features. Even peer to peer ones. I can’t chat with my son and say listen to what I’m listening to. But as long as they evolve rapidly, great.

But I will still argue that in general, the value comes from the information, the community, the collaboration. The radio hit a certain point and then was good enough. It was a commodity. The content played over the radio, on the other hand, has continued to be of huge value. Sure new radios come out all the time with new bells and whistles. Compared to a new talk station (pleasing or infuriating or a great new song) how much do we really care? Sure iPod currently has made a business by tying itself to a service, but at some point, it will be like the radio, I predict and have gotten good enough.

About 9 years ago it became clear to me that, all things being equal, the current model of building software for a specific API to be deployed on someone’s desk was going to be far less sucessful than the model of deploying a service. Why so?

Well this is where the comment about evolution in action comes in. Things that breed rapidly more quickly adopt through natural selection to a changing environment. Services can typically deploy changes every month or even more rapidly because they only have one single configuration on a set of machines whose OS, storage and networking they totally control and which they manage in their data centers. These days Microsoft gives birth to new products at a pace that makes an elephant seem quick, about every 60 months, that means in the time that a service can make 60 adaptions to its customer’s needs, Microsoft makes one. It used to be that they shipped every 12 months. Then 18. Then 24. And so on. The creep is driven by the ever increasiongly complexity of features, hardware, os variations, and backward compatibility of the API’s so ably designed to lock developers in. They locked the developers in all right. The Microsoft ones. This alone to me has been a compelling argument that when a product can be delivered as a service, it should be.

Secondly there is the telling point of usability. In February 1997, fresh from having built/designed DHTML for MSFT and just before building XML, I made a month long pilgrimage to talk to users about the forthcoming IE 4.0 and what should go into IE 5.0. The results were eye opening to me. Customers, almost unanimously, told me that they didn’t want the rich user interface of GUI for most applications.For office worker Personal productivity applications, sure. But in general, they wanted the easiest most self evident user interface possible. Why? Customer support costs. They had found empirically that the training and support costs for web based applications were much lower than for custom built Windows applications. Donald Norman ought to have been delighted. Simplicity had won. The economist has a huge article this week about how IT needs to be simpler and somehow manages not to mention the browser much. One wonders if it is a coincidence that they frequently quote an old employee of mine, Chris Capossela, now apparently “the boss of productivity applications” so frequently or there is some other reason for this startling ommision. But, in short, services not only evolve far more rapidly to meet customer needs, they are much cheaper to manage for most customers.

Chris’s title “boss” is illustrative of the other major reason that services tend to more rapidly adapt to changing customer needs and to more rapidly fix problems. Services provide all actions as URL’s and Button clicks and these can be statistically analyzed by services to better understand what’s working for customers and what isn’t. Thus as thus statistical instrumentation highlights customers needs, the services can instantly and dynamically respond to these customer needs. In short there isn’t some person who is the “boss” of a service. The customers are, very literally, in control. Think of it as democracy versus totalitarianism. This last point showed up almost from day one in the web. The user interface customized itself to the users needs, location, and data in a dynamic way through the magic of dynamic page layout. Today, a full ten years later, most windows apps still don’t do that. But heck they are only 2 or 3 or 4 generations evolved. Services, in the last decade, may have evolved 600 times by now all in reaction to what they have learned directly from customer use.

Some may say this is idle chatter. They may challenge me to put my money where my mouth is. I do so. I left an extraordinarily lucrative job at MSFT 5 years ago to work for 24K a year to work on services. I have personally invested in a great services company, Salesforce.com, and another promising one, Talaris, and I’m working for the one I regard as the greatest of all, Google. Greatest because the idea that simplicity of user interface wins has always been a Google strength from the start. Yet, at the same time, Google has figured out how to use my old progeny, DHTML, to let services deliver personal information activites like email in richer and more interactive ways.

Is it right for everything to be a service? Certainly not. If you need offline access, if you’re manipulating rich media (photoshop), if you need to search those files customers choose to keep privately on their PC’s then client side code is required. Blog-Readers are the poster children for this especially now we have podcasting. But I have to say, I think Mozilla could easily build in such features into their awesome browser, Firefox. And keeping data on a PC is asking for trouble. It gets old. The machine is dropped. The disk breaks. Put it on a service and it is stored, reliably, even in the face of an earthquake these days. By the way, I write all these posts just using a browser to type them in.

Perhaps more importantly, as I said in a prior post, most of the value today is coming from the community, the reputation, the access to information and goods and services, and the media itself. This ineluctable fact coupled with the driving forces of much faster evolution in response to the natural selection of market needs, much cheaper and easier and more simple user interface, and much better ability to know what can be done better for the customer are all combining. Services will be the dominant model. Think of it as evolution in action.

Today, I wonder if this set of syllogisms about the platform is still true (if it ever was). Open Source has shown us that well understood software can and will be commoditized. The operating system has been. The Web server has been. The Applications Server (to the extent folks need it) has been and more message buses are being written in open source. The entire XML processing stack is open source. So the value in “well understood” software today is in the support, not the code. The community that forms around open source software seems quite up to the job of educating itself. The real value in my opinion has moved from the software to the information and the community. Amazon connects you to books, movies, and so on. eBay connects you to goodness knows how many willing sellers of specific goods. Google connects you to information and dispensers of goods and services. In every case, the push is for better and more timely access both to information and to people. I cannot, for the life of me, see how Longhorn or Avalon or even Indigo help one little bit in this value chain.

My mother never complains that she needs a better client for Amazon. Instead, her interest is in better community tools, better book lists, easier ways to see the book lists, more trust in the reviewers, librarian discussions since she is a librarian, and so on.

The platform of this decade isn’t going to be around controlling hardware resources and rich UI. Nor do I think you’re going to be able to charge for the platform per se. Instead, it is going to be around access to community, collaboration, and content. And it is going to be mass market in the way that the web is mass market, in the way that the iPod is mass market, in the way that a TV is mass market. Which means I think that it is going to be around services, not around boxes. I postulate, still, that 95% of the UI required for this world will be delivered over the browser for the same reason that we all still use a steering wheel in a car or have stayed with << >> for so long. Everybody gets it. But this will, by definition, be an open platform because the main value it has is in delivering information and communication. Notice that the big players, Amazon, eBay, and Google have already opened up their information through Web API’s. It is Open Data coupled with Open Communication built on top of Open Source that will drive the future, not Longhorn.

From my point of view, this is hard. I go into a PC store and they don’t understand amps or speakers and try to push little tiny tinny ones on me. They don’t understand wireless at all. I go into a stereo store and they think I still listen to sound from CD’s and video from cable or DVD’s when again, I typically want it on my disks. They will sell me a flatscreen for $5K or more without blinking, but ask for a 1 terabyte hard disk which should be <$1K and everyone just stares. One terabyte can store a LOT of movies and songs and shows. I also want great recording quality on my sound and the stereo stores seem to know nothing about this. I’m sure I’ll hear from Mac people about how this is all easy/possible on Mac’s, but I did go into an Apple store and wasn’t happy with their knowledge about amps and speakers and cabling and sound.

Apple does something that makes no sense to me at all. They think that the iPod is the sound source and I need a remote control for it thereforce. I want the iPod to be my remote control (when I’m in range of a sound system AND I have no headphones plugged in) and then talk to my “box” that is playing media. It has the UI I want. Lists the songs, albums, artists. etc. The clicker doesn’t. Even the iPod isn’t my favorite remote UI because I can’t type in the name of a song. Hence my desire to use my cell phone instead.

I have heard from Mac people since intially posting this telling me to use Airport Express and clicker. Also squeezebox. I’m going to look into squeezebox, but I’ve tried the Mac solutions (have an Airport Express at the apartment) and have yet to figure out how to get it to co-exist with my netgear wireless adaptor. I don’t want a clicker to be my remote control. I want a UI a lot like the iPod itself (screen, lists of albums, …) where I can pick and choose from the media using the remote. Not just pause, continue. I actually had planned to buy a Mac AS my “preamp” assuming it could do all these things (be my monitor, plug into my amp, be remote controlled by my iPod, talk to my terabyte hard disk using Rendevous, talk to my cable box). I asked the guys in the Apple Store for help on how to configure this and they were just clueless. So I didn’t.

In terms of why prior postings are closed to comments, I was the victim of some very unpleasant spam that turns into 1000′s of ugly comments with links to postings so these days I routinely close postings asap.

I have a posted comment about just using XML over HTTP. Yes. I’m trying, right now to figure out if there is any real justification for the WS-* standards and even SOAP in the face of the complexity when XML over HTTP works so well. Reliable messaging would be such a justification, but it isn’t there. Eventing might be such a justification, but it isn’t there either and both specs are tied up in others in a sort of spec spaghetti. So, I’m kind of a skeptic of the value apart from the toolkits. They do deliver some value, (get a WSDL, instant code to talk to service), but what I’m really thinking about is whether there can’t be a much simpler kindler way to do this.

And boy is it satisfying to do this when the people you are designing for are your friends, family, relatives, your smart alec son, and so on and when even your mother can use what you build. I call this the mom factor. It is corny but fun.

It is interesting to me how this focus around simplicity in the services world could carry through even to the plumbing people use. For example take so called web services. The original impetus behind XML, at least as far as I was concerned back in 1996, was a way to exchange data between programs so that a program could become a service for another program. I saw this as a very simple idea. Send me a message of type A and I’ll agree to send back messages of types B, C, or D depending on your A. If the message is a simple query, send it as a URL with a query string. In the services world, this has become XML over HTTP much more than so called “web services” with their huge and complex panoply of SOAP specs and standards. Why? Because it is easy and quick. Virtually anyone can build such requests. Heck, you can test them using a browser. That’s really the big thing. Anyone can play. You don’t have to worry about any of the complexity of WSDL or WS-TX or WS-CO. Since most users of SOAP today don’t actually use SOAP standards for reliability (too fragmented) or asynchrony (even more so) or even security (too complex), what are they getting from all this complex overhead. Well, for one, it is a lot slower. The machinery for cracking a query string in a URL is about as fast as one can imagine these days due to the need services have to be quick. The machinery for processing a SOAP request is probably over ten times as slow (that’s a guess). Formatting the response, of course, doesn’t actually require custom XML machinery. If you can return HTML, you can return XML. It is this sort of thinking that being at a service company engenders. How do you keep it really simple, really lightweight, and really fast. Sure, you can still support the more complex things, but the really useful things may turn out to be simplest ones.

You have to. The scale is orders of magnitudes more than is normally processed by a business process within even the largest corporation. It is hard enough to build these massively scalable services if you keep the moving parts simple, clear, and down to a small number. This is usually called the KISS principle as in keep it simple and stupid or, more rudely, keep it simple, stupid. It reflects the engineering realization that just delivering on the required speed and scale will require a lot of plumbing and monitoring as it is.

So, I’m having a lot of fun learning about a whole new world.

During the last year, I was writing primarily about what Web Services and mobile communications would mean to browsing. I postulated that web services meant that information rather than content could flow from the internet to the client to be formatted on the client side and that because of the intermittant nature of mobile communications and/or just plain latency and availability issues, this would be useful regardless as long as the data were to be cached on the client. I’m not going to write about this anymore. I was the victim of some rather unpleasant spam against the site, so I’ve closed the entries, deleted the ones where cleaning up the spam was more trouble than it was worth, and included, below a summary of most of the old entries.

Aaron Swartz points out to me that Tim Bray has argued that people don’t want rich clients

In general, I agree with Tim. Steve Jobs used to argue that a pc needed to be as easy to use as a radio or telephone (back when they were easy to use) and when the web came along I was surprised that Steve didn’t notice that the future was here. Indeed I said so at Microsoft in 1996/7 when saying this there was a hugely unpopular thing to say. Personally I like the web. It is easy, doesn’t require arcane knowledge to use, … So, in terms of user interface, I think it is generally the right solution. I’m always surprised when I hear people disparage the web as poor or limited or dumbed down UI as Macromedia does and Laszlo and Altio and Nexaweb and others have sometimes done.

Limitations:
But there is no doubt that there are two things it isn’t particularly good at:
Offline Access. IE’s attempt to make sites go on working offline is a joke. Offline requires a new paradigm.
Push. If Information needs to flow to the user more or less as it occurs then the browser paradigm isn’t always the best model, at least without help.

All by themselves, I’d argue that the gain outweights the pain. But add in occasionally or poorly connected which is the world of mobile devices and the more limited UI gestures that really work on mobile devices and I think the game gets more interesting.

More later.

Ok. when I say “Rich Client” from now on, I don’t mean a VB look and feel let alone a Flash look and feel. Macromedia and the inimitable Kevin Lynch do that really well (a lot better than Microsoft) and I’ll just stipulate that this can be the right thing. What I’m more interested in though is something that has the authoring simplicity and model of pages. If you can build HTML, you should be able to build this. You should be able to use your editor of choice. Indeed, as much as possible, this should be HTML. I mean 3 things:
1) The user can quickly and easily navigate through information in ways that the site providing the information didn’t expect
2) The user can run offline as well as online
3) If information changes and the user is connected, then the changes can be “pushed” into the UI if so desired

When I talk to customers about why on earth they are still using Java Applets or VB to build user interface, frequently the reasons are because they can’t do these three things otherwise.

Similarly, when I say “Web Services” I don’t mean that all of the SOAP crap is required. I mean that a service exposes some way to send/receive XML messages to/from it. REST is fine if it works (more on REST in another entry).

So, to summarize, a Web Services Browser is a browser that can access information published as XML messages by services, let the user interact in a rich and graceful way with this information or these services, but can run well in terms of interaction whether the user is online or offline.

Let me give an example. Suppose I’m managing a project and I want to be able to review the project status while on the road. I want to be able to sort by task priority, or employee or amount the task is late. I want to be able to update priorities, add comments, add tasks, and so on. If I am online, say at Starbucks or the airport or Hotel, then I want to be as up to date as possible and I want all the changes I’ve made offline to percolate back to the project. If I’m online and information changes in the service, I want to see the changes immediately flow into the page I’m viewing if they’re relevant to it so that if, for example, I’ve been viewing a summary dashboard style page and some tasks get updated I can see at a glance that they have changed. If I switch machines to my mobile PDA or just connect in through someone else’s computer or an Internet kiosk, I still want to see/update the information of course.

All this I want the authoring model to be more or less what I know. That’s the vision.

Reading all the comments on my last posting, I’m going to give up and detour for an entry about “Rich UI” before getting into the heart of how one might build a web services driven browser and why. It is clear that my comments on Rich UI were both overly facile and unclear.

Sean Neville of Macromedia has posted a really excellent comment to the last posting and Kevin Lynch has an excellent white paper which I encourage readers to look at.

One of Seans’s key points is that Macromedia focuses more on the richness of data and on the richness of the interaction with the user than of the widgets per se. Kevin Lynch has a great demo of the forthcoming central which makes the same point. When you see it, you are overwhelmed by how gracefully the media fit into the application and appear to be an integral part of it even as they are being dynamically fetched from the server. I’m excited about the work Macromedia is doing here. I think it is great work and can substantially enrich the web experience. I was also impressed with the Lazslo presentations (which at least when I saw them sat on top of Macromedia’s Flash engine) and with Altio which had its own Java rendering engine. In short, I’m not against “Rich UI”. Why would I be? I got into this field years and years ago when I fell in love with the Xerox Parc work and set out with partners like Eric Michelman and Andrew Layman to build the first project manager with a graphical user interface. Later Eric and I split off and teamed up with Brad Silverberg to build Reflex, one of the worlds first databases with a graphical user interface. None of this work, of course, used media itself as a type with the dazzling richness and aplomb that the new Macromedia Central demos do. But they shared the excitement and vision.

But what excites me about Central isn’t as much the media as other aspects of it. It has agents which intelligently look in the background for changes and let me know. It can run offline as well as online. It can have context which is so important for many mobile applications. It supports collaboration and direct access of information from the internet. It’s user interface makes use of the ability to ask for and send information to the internet dynamically to enable a much more interactive feel. So, it is reasonable to ask, in what possible way isn’t this already a complete realization of the vision I’ve been discussing?

In most ways it is. And of course, in many ways it is a superset. But there is one gotcha that causes me to think that this isn’t, strictly speaking, a browser. Browing is about building and traversing pages whose user interface palatte is an extraordinarily limited one. In a funny way, browsing is almost an anti-GUI revolution. It isn’t of course. Look at the rich use of layout, flow, media, and so on. But it is in one key sense. The set of user actions is tiny. Fill in fields. Click on URL’s or buttons. Lay things out using flow and/or Tables. Most of the other enhancements in HTML from one I helped build (DHTML) to Framesets to animation have been largely unused. Why is this? Why has the web kept its gesture of user interactions so small? I believe it is because most web applications are not intended for frequent/daily use. Instead of the old model in which you use 2 or 3 applications every day you tend to interact with hundreds or more every month or so. In this world, ease of use and discoverability are paramount, ease of authoring is key, and simplicity is all. Of course there are many applications that scream out for a product like Central. In general, however, I think they are applications that one expects to use over and over during the day in the ways many people used to use Office before email and the browser rendered it largely irrelevant.

This is not, to me, the ideal model for Amazon.com or BMW or my 401K application in my company portal or my broker. These are not applications I use everyday let alone frequently during a day. They are applications I run when I have a specific task in mind and I expect them to fit the usability rules of the browser. Mostly they behave just as I wish unless I go offline or I want to sort or filter in ways that they didn’t expect/predict.

In short, I think there is room for both. I think Macromedia is heading in exactly the right direction and have told Kevin Lynch so. At the same time, I think that there is a need for a plain old browser that can interact with the server at the information level and I think there is also a need for Central. We don’t use only one size and type of screwdriver. Why should this world be different?

Let’s make the web services part of the Calendar really simple. There is a service to get activities for a date range. In addition, there are operations to add an activity, to modify an activity, and to delete an activity. Each activity can have notes, one or more assigned people, phone number, title, priority, start time and duration, external visibility, reminder attributes, category, and repeating characteristics if any. In short, mostly what is in Yahoo’s calendar if they only surfaced it as a web service!

This information can be thought of as a local XML store with documents for each activity and web service operations used to read, write, delete, and query it. The “pages” that are designed to run offline together are bundled into a “Client Web”. These pages are standard web pages with three key extensions:

1) All HTML elements on the page can have a “selector” attribute that can select the requisite information from this local XML store and can repeat if the selector returns multiple items.

2) Any element can contain a macro that references the “current” selected XML element in {}’s. So, for example, a TABLE might have a TR with selector=”\activity” to list all activities and within it a TD might contain {\date}. This would mean that there would be a row in the table for each activity and that within each row, the TD should display the date for that particular activity.

3) URL’s and Forms can bind to “Actions” in a “local controller” script which then does whatever is necessary on the client and returns the next page to display. The script may just invoke a web service and return the same page. It may simply update some of the “XML store” and return the same page. It may simply switch pages. It is, in essence a super lightweight web server except that it doesn’t need to round-trip to the Internet if the pages being returned are all in the Client Web. There is a “begin” action in the controller that is run before the first invoked page is run (usually the entry point), and there is an “end” action that is invoked if the user leaves the “Client Web”. We call this the elementcontext in each case.

There a couple of other key points:

1) The macro language doesn’t only have access to the XML Store. There is a “magic” XML variable, $Form, which represents the values in the most recently submitted form. Thus the attribute
selector=”/activities[date >= $Form/startdate]” will select all activities with a date greater than the start date filled in in the form. Similarly, if a URL is selected the current XML path that was its context is available in $ElementContext

2) If the controller returns the page currently being displayed, it simply refreshes itself in a flickerfree manner. In short, the user experience is just as it would be in the online case, but with less obvious flicker.

Now, how does the Calendar work? it has a default page with the current month set up in the begin Action using the Selectors to pull in the items that make sense for each day in that month. Each day has URL’s that point to Actions for Add, Delete, Modify and each action in the controller script will have access to $ElementContext to see which day was in question. They will then invoke the requisite Web Service Operations to add/delete/modify.

The Delete Action is the simplest. It loads a Delete Confirmation Page from the ClientWeb, (details to follow in subsequent entries), sets it to filter on the Activity that is to be deletes and then returns this page. This page in turn has a URL pointing to the DeleteConfirmed Action which uses the web service to delete the Activity. The Update will basically do the same with an Update page and an UpdateConfirmed Activty pointed to by the Submit in the the Form. The insert will work the same way. In subsequent entries, I’ll imagine how the logic in the Controlller might look and discuss some choices for “parameterizing” the pages.

So, to summarize, a set of related pages designed to work with the XML information in quesiton called a ClientWeb, a set of information retrieved using Web Services, and a Controller to coordinate the actions that occur between pages and to invoke suitable web services when necessary. How does all this work offline? Also to be covered in subsequent entries. Lastly let me freely admit that this is a dream in progress, open to all, and sure to be wrong in some of its details.

There are basically three ways that information can turn into User Interface.

Given my desire to build a browser that interacts with the information being gleaned from web services one has to choose which of these three models to use. I chose the second in this blog. It is equally reasonable to choose the others and AboveAll seems to be building an interesting flavor of version one.

Once this is picked, then one has to ask where is the rendering engine and how does it talk to the Web Services. There are, roughly speaking, four ways I can think of:

Today, the strategy in the industry clearly is the last one. We render pages on the server and ship rendered user interface (HTML) up to the client. The industry does this because it makes it much easier to deploy changes to the logic. Change the web server and hey presto, every client sees the changes. People tend to forget that a huge benefit of the browser based model was simply to remove the deployment hell and risk that deploying EXE’s had meant. So, any change to this model should be considered judiciously. Nevertheless, I argue that the time has come for a richer browser model. It should still be essentially a no-deploy provision-less model, but the actual rendering needs to be able to take place on the client. Why?

Because of the increasingly mobile nature of computing. If everyone sat at a desk all desk connected to a broadband link I might argue otherwise although it is interesting to note that even as we all talk about Grid computing, we centralize the easiest thing to distribute out, namely the rendering. But we don’t all sit at desks. More laptops than PC’s sell these days and 80211b has liberated us all from the net tap. In addition, mobile devices are coming into their own as GPRS becomes ubiquitous. I just roamed all over the world with my Blackberry 6210 and had effortless synchronized instant access to my calender, contacts, and mail at all times. (i’m about to upgrade to the 7210 which is a killer device). In this world, the connection is intermittant and the bandwidth is unreliable. In Paris, for example, my GPRS went up and down like a Yo-Yo and in the US it is deeply unreliable. Just try using GPRS on the amtrak train from NYC to Boston! Many of us fly all the time and want to have access to our information on the plane, the car, the bus, and so on. Since many of us are mainly mobile, this is a severe problem. Mobile computing therefore, is the driver here.

It is worth noting that done correctly this will improve the customer experience even in the fully connected world. For example, my broker’s online web pages don’t let me filter, view and sort my assets in the way I would like. If I have my browser talking to my broker’s web services rather than to its rendered pages, then my pages can enable my view of the information, not just the one that my broker limits me to. Obviously the same is true for viewing news, mail, and so on. Similarly, I have to wait each time I want to view the next 23 messages in my mail browser (although I can configure it). In short, we may end up viewing the current model as a tyranny of the content provider in which we could view information only in the way they had in mind.

I am arguing that we need a model other than the fourth to meet the mobile users requirements and that it will ultimately benefit everyone. Which one? The first one (directly invoking web services from pages as the browser renders them) doesn’t really meet the bill because it still requires connectivity at rendering time and provides no mechanism to sort/filter the information. That leaves two models: Have a cache that talks to web services or have a cache that talks to another cache. I’m seriously torn here and I suspect that, in the long run, both will be required. It is much easier and lighterweight to just have the cache on the client synchronize with a cache on the server. Then the only thing required for communication to the Internet from the PC/Device is some sort of souped up SynchML. This may not be intuitive to the reader. It stems from my assumption (perhaps incorrect) that we can package the page itself as XML so that any change to the page can itself be delivered into the cache using the same SynchML protocol. On the other hand, this still limits the freedom of the mobile client to integrate information across web services since someone has to write a server somewhere which runs this cache that synchronizes. But in either case, one ends up assuming that there is a each page really is a cache of information on the client associated with the page plus the specific XML for the page, but separate from it, that in the background it can synchronize with information on the Internet.

I’m going to wave my hands for the time being about how this data gets synchronized between the client and the Internet and just assume that some sort of Synchronization Protocol exists.

But assuming this opens up some specific challenges. How does the page reference this information? If the user is clicking through links and is offline, how does this work? How do user interface gestures that are supposed to invoke processes on the Internet work when one is not connected (for example giving someone a raise, approving an expense report, moving a scheduled item, selling a stock, booking a hotel room, assigning a bug report to an engineer, and so on)?

Proposed answers to these questions shortly (a lot sooner than the last delay). I promise.

Well, if you cannot rely on the Internet under these circumstances, what do you do? The answer is fairly simple. You pre-fetch into a cache that which you’ll need to do the work. What will you need? Well, you’ll need a set of pages designed to work together. For example, if I’m looking at a project, I’ll want an overview, details by task, breakout by employee, late tasks, add and alter task pages, and so on. But what happens when you actually try to do work such as add a task and you’re not connected? And what does the user see.

To resolve this, I propose that we separate view from data. I propose that a “mobile page” consists both of a set of related ‘pages’ (like cards in WML), an associated set of cached information and a script/rules based “controller” which handles all user gestures. The controller gets all requests (clicks on Buttons/URL’s), does anything it has to do using a combination of rules and script to decide what it should do, and then returns the ‘page’ within the “mobile page” to be displayed next. The script and rules in the “controller” can read, write, update, and query the associated cache of information. The cache of information is synchronized, in the background, with the Internet (when connected) and the mobile page describes the URL of the web service to use to synchronize this data with the Internet. The pages themselves are bound to the cache of information. In essence they are templates to be filled in with this information. The mobile page itself is actually considered part of the data meaing that changes to it on the Internet can also be synchronized out to the client. Throw the page out of the cache and you also throw the associated data out of the cache.

Now every user action isn’t dependent on connectivity. Go to the next page. It is within the cached mobile page. Let the user filter or sort tasks. This is done by merging the result of a query on the cache into the target page using “binding” and controlled, where necessary, by the script in the controller. Change the priority of a task or create a new one or delete one or request that relevant users be notified about changes and all these changes are entered into the cache as “queued” requests. Once sent to the Internet as part of synchronization, their status is changed to “sent” and once processing on the Internet has actually handled these requests, their status is changed to “confirmed” or they are simply deleted. But the user actions that created these requests never was blocked. It will be important, of course, that the user interface make clear what is queued, what is sent, and what is confirmed.

Some actions may not return meaningful information until synchronization has occured. For example, ask for all news stories on a stock and they may not be local. In that case you see the request as Queued, Sent, and then finally resolved with a link to the now cached news stories and from then on news stories may synchronize to that link in the background.

This model is profoundly different than the normal browser. It assumes that connectivity may not be present. Accordingly, it makes sure that everything that the customer needs is present on the client. This means that not only does the client handle the formating of the information into pages, it also holds the intelligence about what happens between pages. Hence the script controller. Why script? Because it is small, lightweight, interpretive, easy to dynamically deploy and modify, and already used in pages so that a form of it is already part of the browser runtime. People forget how small script engines can be. The original dBase II ran on a 64K machine. I would expect that this particular script would be sandboxed. In my vision, all it could do is read and write the cache associated with the mobile page and decide which page should be displayed next. Since it is script, it provides a install free lightweight model for intelligence that runs as a result of user actions so that the model isn’t too constrained. Pure declarative systems always turn out to be too limited in my experience.

Of course, the simple model for all this is straight web services between the client and the Internet and I have no doubt that this will work for a huge amount of cases. The cache will just hold the XML returned from the web service or pushed into it (next entry).

So perhaps all this can be built on top of Luxor or Mozilla or .. If so, great! Having written one browser, I certainly don’t want to write another. I want to work with those who do to enable these capabilities.

Edwin Khodabakchian (who I consistently regard as one of the most interesting people working on the web today) says he is going to try and mock this up for me. Awesome!

Doğa Armangil, who I don’t know, says experience is showing her that a web browser + web services yields a better use experience than rich client without them. I hear similar things. A “very large” financial institution discovered that when they replaced their CICS->ShadowDirect->RS 6000 Protocol conversion->ODBC/OLEDB/ADO (MSFT Stack)->ASP to build pages with a native mainframe Web Services ->JSP, things got an order of magnitude faster. That’s right. Faster. They were stunned. But this doesn’t solve the intermittant connectivity issue per se.

Let’s start with a simple case that drives me crazy. As many know, I use and love a Blackberry 7210 for most of my work. I literally don’t use a PC sometimes for days on end right now. Mail is a joy. The phone works well for me (I carry a backup for the vast majority of the US that ATT Wireless seems unable to reach), my calender works well. Only my contacts don’t synch wirelessly (come on Blackberry!). But browsing isn’t fun at all. I like to keep an eye on google’s news. And navigating through it using the 7210 is really painful even when GPRS is humming and downright impossible on planes or, as it turns out, in Bedford NY when ATT wireless is your provider. So, I’d like my Blackberry to have google in the cache. Now if anything should be easy it is this. It is a nice simple hierarchy with subjects like Sports, Sci/Tech, … and a set of stories. But there are a couple of complications. The stories aren’t laid out at all well for a mobile device. Menus seem to go on forever. I don’t care about sports. Yes, I know, that’s un-American. Nor do I care about entertainment. What would the web services browser do?

As said earlier, it would traverse a cached data model. One can think of the data model as a simple tree. It would have a starting point, a trunk if you will, called google news. It would have a set of branches for each type of article which I’d mark as Business, Sci/Tech, and so on. Each branch could use a web service to fetch articles of that type. Each would return a set of articles, each of which would contain a one-line summary, a picture (if appr), the story, the byline, and a URL to the raw HTML. On my Blackberry I’d subscribe to the URL for google news. I’d immediately see my choices (e.g. the articles I could delve into) and as I scrolled the wheel over each, below, a short one-liner for each article. Scroll below to the one-liners for each article and click on one and, hey preto, details and story. Perhaps the article is abbreviated and I have to traverse through the URL to the HTML for the raw story. OK, then I can decide from the precis. Now my user experience will be much better. Today, I tend to turn on my Blackberry as the plane lands and then catch up on my mail on the rental bus. Tomorrow, I’d wait for the plane to take off and then catch up on the latest news. In essence, instead of going to a URL, I’m subscribing to one, but with navigation and presentation capabilities designed for me.

Want to reclaim the space? I’d always have a menu on any story OR on the main google news area that would let me DELETE as I can on my mail today. Or I age them.

I have the same problem with Yahoo Finance except that here the information is just a single portfolio I keep in Yahoo. I’d like it to trickle in in the background so that when/if I do check it, the experience is immediate and that the list is a list of holdings with the drill-in going to the short summary about the stock in question.

And so on.

All this should be a no-brainer so far. It is read-only. I don’t have write privileges. All I need is a way for each URL to specify a tree whose branches are words that make sense to me (like business, sci/tech, world or MSFT,ORCL,BEAS) and then tell me which web service will return relevant information and how to display both a summary and a detail view. (More on this in a later entry). When GPRS returns or I sit down with my Laptop in the airport/hotel/Starbucks, then I’ll catch up. But the key point is that I can access things I care about even when I’m not connected. The model has to know which branches I want it to be pre-emptive about fetching. Remember, I don’t want sports or entertainment cluttering up my machine. It should even know my priorities since I can lose connectiity at any time. And if the list of entries is too big I want what google news does today, namely to just give me a few and make me get the rest. But I’d like to control how many. So each branch should have prefetch meta-data. It should also have how stale the data can be before it should throw it out. I don’t want stories more than 2 days old, for example. So I should be able to mark each branch in terms of how old the data is before I throw it out.

Now even before we getting to sites where I can do things/change data (which is harder), there are some interesting problems.

Suppose google adds a new topic to its news. This is like a new branch. It lets me get to new types of stories, for example, China, which seems like it is getting big enough to deserve its own topic. How does the browser know that this new branch is there? Well if the meta-data about the tree is adequately self-describing, it could refresh the meta-data about the site and discover this. It would need to merge my annotations about aging, pre-fetching, and so on with this new tree. What if it isn’t google, but a set of employees in a company. How does the browser tell the difference between someone having updated the employee versus someone quitting and someone else joining. What if they have the same name? Clearly, you need some unique identity returned from each web service as part of each leaf so that the browser can uniquely identify the leaf returned and can detect that it needs the cache on the PC/PDA/.. to be updated.

Sometimes I would like the browser to do that and sometimes I’d want to control it. This is what I discussed, a couple of entries ago. Just use meta-data versus layout control. In the latter case, I’d have explicitly marked which branches I wanted to pursue and all others would be off limits. If you look at blogs for example, often they list in turn the blogs they follow and you can only go through them to the ones they list. Others do it more dynamically based on links/popularity and so on. You decide.

Blogs, of course, are an example of this sort of browser today, but why, oh why, are they a special case? Why doesn’t the browser work this way for any set of information?

How do I manage the amount of information on my PC/Blackberry/Phone? I discussed one model, namely describing how long to hang onto information. But it is important not to be wasteful. Think of Zagat’s online. I might have branches to search by cuisine, price-range, stars, and neighborhood. Each would list a set of values which would in turn be branches (twigs?) to a list of restaurants. Each restaurant would have a review, stars, description, price range, phone number, address. Some would let you book them using a different web service (we’ll talk about this in the next entry) or get a map. This can add up, at least in NYC or Seattle or SF (I divide my time almost equally between the three). But wait, it gets much worse! The same restaurant might be found (in fact certainly will) from all lists. We certainly don’t want it repeated 5 times. So in fact, all these twigs must point to the same leaf. (Yeah, for you CS types, that’s a graph, but I’m a history major and for me a graph is the population of the world growing over the last century). but even if we’re smart enough to just cache the restaurant information one time, it is a lot of space. So what to do? I’m a super impatient guy. I want access all the time with more or less instant response. Well, we can take the big stuff (pictures, maps, ..) and use a LRU cache for that which is fixed and then the rest really isn’t that big. The NYC Zagats has less than 1900 restaurants with about 300 bytes of info for each or 600KB. Teeny no. But if a PC, it is nothing. If a PDA, I could afford to subscribe to 3 or 4 of these. If a phone, I could not. So include some smarts on the cache. I mark a neighborhood and a cuisine or two. Then later I unsubscribe. Perhaps I limit the price-range. Now we’re down to 100 resturants max or about 30KB and I could have a bunch of these at any given time on my PDA. In short, when I “go to a URL”, I’m subscribing to it and it becomes automatic to subscribe/unsubscribe. Clearly the UI will need to give me a space feedback.

How does Zagat get paid? Same as today. To use their online site, I have to login. But here they could be smarter. Get a cut of the bookings if I use their web service to book. Give me a micro-bill choice for my PDA wireless provider.

In every case, the thing that has driven this is a tree whose branches are labeled so that I can see/subscribe/annotate them, perhaps with twigs and smaller twigs and so on, and finally leaves. Now this is a bit misleading. Let me give an example. Another area that I want access to offline is our company address book and I’d like it to include our company org chart. I can never figure out who works for who. (Perhaps this is deliberate). If I modeled it and then subscribed to it using this system (same deal in terms of size of information by the way and same filtering techniques work), then as I traverse through people, each person not only has interesting information about themselves, they have links to other people based on their relationships. Here leaves contain twigs and it is clear that the bough of the tree anology just broke under the weight of the – no I won’t go there. So, more generally, I can navigate to lists of information using meaningful names, drill into items in the list, and then from those items again navigate to interesting lists.

Lastly think about CRM or SFA (Customer Relations Management or Salesforce Automation). As I navigate through customers I branch to projects or contracts we have with them. Then I see the related support people and can drill into them and see the related other projects/contracts/opportunities and so on. Again, the same pruning techniques of limiting information by location, job function, project size/price range, and so on can quickly prune a lot of information down to a reasonable amount. Connectivity will always be better. Then all information can be reached. But this model lets the user control which information is available all the time.

This entry has only discussed what happens when you’re disconnected and the information you’re browsing is read-only. In the next entry, I’ll discuss what to do if you’re trying to reach a service (like booking a restaurant) or modify information and you’re disconnected and how, it seems to me, the browser should handle it.

I want to take time out to passionately support Jon Udell’s latest blog entry on Avalon. I’m not sure that Royale and Central are any more “open” than XAML, but I am darn sure that all of Microsoft’s latest efforts seem to be moving inoxerably to a closed client world and even a only-if-you-pay world. While I applaud Infopath for example, it will never be the force for change and the ubiquitous client that it could have been. It will be, if not still born, at least damaged by the lack of a free ubiquitous runtime. Foolish in my opinion because if the runtime had been free and ubiquitous I think this product would alone have sold the new office. While I don’t know if XAML’s client will be free or not, obviously tying the graphics to Windows says it all in terms of ubiquity if not iniquity. An XML grammar alone does not an open standard make. Thanks Jon for saying it so clearly.

On a related note since Jon cited Edwin, earlier I mentioned that I was reluctant to use BPEL for the controller in the web services browser I’ve been writing about. Edwin Khodabakchian of Collaxa nailed the issue that causes me to think it will not work. A ‘controller” for UI is really a misnomer. In a site filled with pages (or pages filled with rich UI interactive gestures) the user is in charge, not the controller. There is no directed flow. Occasionally we build directed flows and call them wizards, but in general sites are not written to move from task to task in some ponderous choreography. BPEL is intended to do just this and because of it, really isn’t suitable for playing the controller role even if it were versatile enough. So, oddly, this entry both agrees with Jon when he disagrees with Edwin and agrees with Edwin when he disagrees with me.

Changing Data Offline:james@doit.org writes that I should refrain from blogging because my blog “is a real slow” one. Perhaps this is true, but I shall persevere. In this entry, I’m going to discuss how I imagine a mobilized or web services browser handles changes and service requests when it isn’t connected. This is really where the peddle hits the metal. If you just read data and never ever alter it or invoke related services (such as approving an expense report or booking a restaurant) then perhaps you might not need a new browser. Perhaps just caching pages offline would be sufficient if one added some metadata about what to cache. Jean Paoli has pointed out to me that this would be even more likely if rather than authoring your site using HTML, you authored it as XML �pages� laid out by the included XSLT stylesheets used to render it because then you could even use the browser to sort/filter the information offline. A very long time ago when I was still at Microsoft (1997) we built such a demo using XSLT and tricky use of Javascript to let the user do local client side sorting and filtering. But if you start actually trying to update trip reports, approve requests, reserve rooms, buy stocks, and so on, then you have Forms of some sort, running offline, at least some of the time, and code has to handle the inputs to the �Forms� and you have to think through how they are handled.

XAML:First a digression. I promised I’d dig into this a bit more. At the end of the day, I think that thinking of XAML as an industry standard for UI is premature and does assume that Windows will have complete dominance. It is essentially an extremely rich XML grammar for describing UI and user interactions. It subsumes declaratively the types of things VB can do, the flow patterns in HTML or word, and the 2-D and time based layout one sees in Powerpoint or these days in Central and Royale from Macromedia. In short, it is a universal UI canvas, described in XML, targeting Avalon, Longhorn’s new graphics engine. That is the key point. It isn’t an industry standard unless you assume that Avalon’s graphics are pervasive which I think is a stretch. Also, people are talking about this as though it will be here next month. As far as I can determine, Microsoft’s next massive OS effort, Longhorn, will ship somewhere between 2005 and 2006. In short, it is probably 3 years away. 3 years from now my daughter will be grown up and in college and who knows what the world will look like. I have no doubt that Microsoft is salivating at the thought that this will subsume HTML (not to mention Flash and PDF) and thus put those pesky W3C folks out of business, but I can’t really worry about it. Kevin Lynch of Macromedia should be the pundit on this one. End of digression.

Browser Model so far:As discussed already, this new browser I’m imagining doesn’t navigate across pages found on the server addressed by URL’s. It navigates across cached data retrieved from Web Services. It separates the presentation – which consists of an XML document made up of a set of XHTML templates and metadata and signed script – from the content which is XML. You subscribe to a URL which points to the presentation. This causes the XML presentation document to be brought down, the UI to be rendered, and it starts the process of requesting data from the web services. As this data is fetched, it will be cached on the client. This fetching of the data normally will run in the background just as mail and calendar on the Blackberry fetch the latest changes to my mail and calendar in the background. The data the user initially sees will be the cached data. Other more recent or complete information, as it comes in from the Internet, will dynamically “refresh” the running page or, if the page is no longer visible, will refresh the cache. I’m deliberately waving my hands a bit about how the client decides what data to fetch when. I’m giving a keynote talk about this at XML 2003 and I want to save some of my thunder. So far, though, I�ve described a read only model, great for being able to access information warehouses and personal data and like clinical trial history or training materials or find good restaurants in the neighborhood or do project reviews all while offline, but not as good when used for actually updating the clinical trials or entering notes into them or building plans for a team or commenting on the training materials or booking the restaurants.

It�s a fake:It is very important to remember in this model that “reality” usually isn’t on the device, be it a PC or a Blackberry or a Good or a Nokia 6800. Because the information on the device is incomplete and may have been partially thrown out (it is a cache) you don’t know really which tasks are in a project or which patients are in a trial or which materials have been written for a section. You only know which ones you have cached. The world may have changed since then. Your client side data (when running offline) may be incomplete. So, if you modify data, you need to remember that you are modifying data that is potentially out of date.

Don�t change it. Request the change:Accordingly, I recommend that the model is that, in general, data isn’t directly modified. Instead, requests to modify it (or requests for a service) are created. For example, if you want to book a restaurant, create a booking request. If you want to remove a patient from a clinical trial, create a request to do so. If you want to approve an expense report, create a request to approve it. Then relate these requests to the item that they would modify (or create) and show, in some iconographical manner, one of 4 statuses:
1) A request has been made to alter the data but it hasn’t even been sent to the internet.
2) A request has been sent to the Internet, but no reply has come back yet.
3) The request has been approved
4) The request has been denied.

Expense Reports:Let me start with a simple example. While offline, the user sees a list of expense reports to approve. On the plane, he/she digs into them, checks out details, and then marks some for approval and adds a query to others. All these changes show up but with an iconic reminder to the status/query fields that these fields reflect changes not yet sent to the Internet. The user interface doesn’t stall out or block because the Internet isn’t available. It just queues up the requests to go out so that the user can continue working. The user lands and immediately the wireless LAN or GPRS starts talking to the internet. By the time the user is at the rental bus, the requests for approval or further detail have been sent and icons have changed to reflect that the requests have now been sent to the Internet. Some new data has come in with more expense reports to be approved and some explanations. By the time the user gets to his/her hotel, these requests on the Internet have been de-queued and processed invoking the appropriate back-end web services and responses have been queued up. By the time that the user connects in at the hotel or goes down to the Starbucks for coffee and reconnects there (or if the device is using GPRS much sooner) the responses have come in. If the requests are approved, then the icon just goes away since the changed data is now approved. If the requests are denied, then some intelligence will be required on the client, but in the simplest case, the icon shows a denied change request with something like a big red X, (this is what the Blackberry does if can’t send mail for some reason as I learned to my sorrow on Sunday). The user sees this and then looks at the rejection to see why.

Notice that all this requires some intelligence on the part of the web services browser and potentially some intelligence on receipt of the approvals or denials from the internet. In the model I’m imagining, the client side intelligence will be done in script that will be associated either with the user actions (pressing submit after approving or querying) or with the Internet actions (returning approval or rejection). The script will have access to the content and can modify it. For example, on receipt of a rejection, it might roll back the values to their prior ones. Server side intelligence will be handled using your web service server of choice.

Restaurant Reviews and booking:Let’s take a slightly more complicated example. I’m flying into Santa Fe and don’t know the town. Before I leave NYC for Santa Fe, I point at the mobilized URL for my favorite restaurant review and check off a price range and cuisine type (and/or stars) that I care about. By the time I get on the plane and disconnect from wireless or GPRS, the review has fetched all the restaurants and reviews for the restaurants I’ve checked off onto my PC or PDA. On the plane, I browse through this, pick a restaurant, and then, ask to “book it” since the user interface shows that it can be booked. A Booking request is then shown AND the script also modifies my calendar to add a tentative reservation. Both items clearly show that the requests have not yet left my computer. When I land, the requests go through to the Internet and on to the booking web service and to exchange. It turns out that the restaurant has a free table and I get back an approval with reservation number and time. But the service acting as a middle man on the Internet also updated my “real” calendar to reflect this change. Now I need to replace the tentative reservation in my calendar with the real one created in Exchange by the Internet and I might as well delete the booking request since my calendar now clearly shows the reservation. Script handles this automatically and I’m OK and a happy camper. But should I have even modified my local calendar? Probably not since the integration process on the Internet was going to do it anyway and it just makes it hard to synchronize. I should have waited for the change on the calendar to come back to my client.

In practice this tends to work:This all sounds quite tricky, but as someone who has been using a Blackberry for 3 years now, it really isn’t. You get very used to eye-balling your mail to see if it has actually been sent yet or not. You soon wish that the changes you make to your calendar had similar information since you’re never sure that your tireless assistant hasn’t made some changes to your calendar that conflict with your own and you want to know, are there changes approved or not. What it does require is a decision about where changes are made and how the user is made aware of them. If the user is connected, of course, and the web services are fast and the connection is quick, then all this will essentially be transparent. Almost before the user knows it, the changes will have been approved or rejected and so the tentative nature of some of the data will not be clear. In short, this system works better and provides a better user experience when connected at high speeds. Speed will still sell. But the important thing is that it works really well even when the connection is poor because all changes respond immediately by adding requests, thus letting the user continue working, browsing, or inspecting other related data. By turning all requests to alter data into data packets with the request, the user interface can also decide whether to show these overtly (as special outboxes for example or a unified outbox) or just to show them implicitly by showing that the altered data isn’t yet “final” or even not to alter any local data at all until the requests are approved. For example, an approvals system might only have buttons to create approval/denial requests and not enable the user to directly alter the items being approved (invoices, expenses, transfers) at all.

Answering comments is turning out to be more interesting than trying to explain my ideas which is, I suppose, part of the idea behind Blogs. In any case, I’ve had a slew of comments about REST. I admit to not being a REST expert and the people commenting, Mike Dierkin and Mark Baker in particular, certainly are. They tell me I’m all wet about the REST issues. I’ll dig into it. But I admit, I don’t get it. I need to be educated. I’m talking to our guy, Mark Nottingham, about this who does understand it all, but I also will be at XML 2003 in Philadelphia on Wednesday if people want to explain to me the error of my ways.

Let me explain why I’m confused. I don’t want to depend upon HTTP because I want to be able to use things like IM as a transport, particularly Jabber. So I want some protocol neutral way to move information back and forth. Maybe I’m wrong here, but I’m particularly wary of HTTP because in many cases I want the “sender” of the message to know reliably that the receiver has it. How does REST handle this? I’m going to try and find out. Secondly, whenever I send a set of messages to a set of services and responses start coming back, I want a standard protocol level way to correlate the responses with the sent messages. How do I do this without depending on the HTTP cookie and without the poor programmer having to worry about the specific headers and protocols? Again, I need to learn. Third, I DO believe in being able to describe a site so that programming tools can make it easy/transparent to read/write the requests and responses without the programmer thinking about the plumbing and formats. How is this done in the REST world? What’s the equivalent of WSDL. Sure WSDL is horribly complex and ugly. But what is the alternative here? I don’t mind, to tell the truth, if the description is URL centric rather than operation-centric, but I do want to know that the tooling “knows” how to build the appropriate Request. I care about programming models. Fourth, as I’ve said before, I think that the model for the moblized browser communicating with the internet is frequently going to be a fairly subtle query/context based synchronization model. For example, given the context of a specific trip and some cached details, update me with the latest details. Given the context of a specific project and some cached tasks and team members, update me with the latest tasks, team members, etc for that project. How do I do this in the REST model given that this isn’t trying to change the data on the server and that I need to pass in both the context, the implicit query, and the data I already know? Fifth, how do I “subscribe” to “Events” and then get “unsolicited messages” when someone publishes something I want to know about? KnowNow has been doing this for years, (Hi Rohit), but how do I do this in REST. So, REST folks, feel free to send me mail explaining all this to me. I’m trying to learn.

Google is clearly trying to think outside the box. Their use of DHTML is both audacious, astonishing, and sometimes almost annoying. It is incredibly quick and when entering a new contact name with just a key stroke or two, it is an almost visceral thrill. And it has a very untraditional outside of the box look and feel. But when Reply To requires an act of faith that a box will expand, I’m still not sure where I stand. And Google clearly doesn’t want me to delete. They want me to archive, file, label, and in general slowly build up a searchable volume of email whereas Mailblocks by default ages my mail.

I bring this up to point out that Web UI per se doesn’t mean an identical or even similar user experience. Both push, in their own way, the edge. In both cases, by the way, when I’m not connected, I can’t see my email as far as I know, which isn’t good. So, when people damn Web interfaces as being limiting, they should consider that it is in its own way a rich and malleable medium. It just isn’t mobile.

I’m getting some comments about Google’s mail product being withdrawn. To be honest, I think that’s ridiculous. It is an interesting and fun product and, frankly, it is voluntary. People need to get a life.

I’m getting mail/comments that interpret this as me being against Rich UI or Avalon. Nothing could be further from the truth. I think that, in its place, particularly personal productivity tools, Rich UI is great. What is more, I’m delighted to see Windows building a better model for authoring it. What I’m afraid of isn’t rich UI. I’m afraid of the browser being taken away. Many many customers love the browser precisely because sites tend to be easy to use. It always amazes me how little the people in our industry understand this. It is easy. It is as easy to use a well designed web site as using a radio, Job’s old vision. No GUI application is that easy. Period. They are richer. They are more interactive. They are great for productivity. But they aren’t as easy, or so customers tell me. And it would be a huge shame if we lost that.

Let me explain why I’m confused. I don’t want to depend upon HTTP because I want to be able to use things like IM as a transport, particularly Jabber. So I want some protocol neutral way to move information back and forth. Maybe I’m wrong here, but I’m particularly wary of HTTP because in many cases I want the “sender” of the message to know reliably that the receiver has it. How does REST handle this? I’m going to try and find out. Secondly, whenever I send a set of messages to a set of services and responses start coming back, I want a standard protocol level way to correlate the responses with the sent messages. How do I do this without depending on the HTTP cookie and without the poor programmer having to worry about the specific headers and protocols? Again, I need to learn. Third, I DO believe in being able to describe a site so that programming tools can make it easy/transparent to read/write the requests and responses without the programmer thinking about the plumbing and formats. How is this done in the REST world? What’s the equivalent of WSDL. Sure WSDL is horribly complex and ugly. But what is the alternative here? I don’t mind, to tell the truth, if the description is URL centric rather than operation-centric, but I do want to know that the tooling “knows” how to build the appropriate Request. I care about programming models. Fourth, as I’ve said before, I think that the model for the moblized browser communicating with the internet is frequently going to be a fairly subtle query/context based synchronization model. For example, given the context of a specific trip and some cached details, update me with the latest details. Given the context of a specific project and some cached tasks and team members, update me with the latest tasks, team members, etc for that project. How do I do this in the REST model given that this isn’t trying to change the data on the server and that I need to pass in both the context, the implicit query, and the data I already know? Fifth, how do I “subscribe” to “Events” and then get “unsolicited messages” when someone publishes something I want to know about? KnowNow has been doing this for years, (Hi Rohit), but how do I do this in REST. So, REST folks, feel free to send me mail explaining all this to me. I’m trying to learn.

XAML:First a digression. I promised I’d dig into this a bit more. At the end of the day, I think that thinking of XAML as an industry standard for UI is premature and does assume that Windows will have complete dominance. It is essentially an extremely rich XML grammar for describing UI and user interactions. It subsumes declaratively the types of things VB can do, the flow patterns in HTML or word, and the 2-D and time based layout one sees in Powerpoint or these days in Central and Royale from Macromedia. In short, it is a universal UI canvas, described in XML, targeting Avalon, Longhorn’s new graphics engine. That is the key point. It isn’t an industry standard unless you assume that Avalon’s graphics are pervasive which I think is a stretch. Also, people are talking about this as though it will be here next month. As far as I can determine, Microsoft’s next massive OS effort, Longhorn, will ship somewhere between 2005 and 2006. In short, it is probably 3 years away. 3 years from now my daughter will be grown up and in college and who knows what the world will look like. I have no doubt that Microsoft is salivating at the thought that this will subsume HTML (not to mention Flash and PDF) and thus put those pesky W3C folks out of business, but I can’t really worry about it. Kevin Lynch of Macromedia should be the pundit on this one. End of digression.

Browser Model so far:As discussed already, this new browser I’m imagining doesn’t navigate across pages found on the server addressed by URL’s. It navigates across cached data retrieved from Web Services. It separates the presentation – which consists of an XML document made up of a set of XHTML templates and metadata and signed script – from the content which is XML. You subscribe to a URL which points to the presentation. This causes the XML presentation document to be brought down, the UI to be rendered, and it starts the process of requesting data from the web services. As this data is fetched, it will be cached on the client. This fetching of the data normally will run in the background just as mail and calendar on the Blackberry fetch the latest changes to my mail and calendar in the background. The data the user initially sees will be the cached data. Other more recent or complete information, as it comes in from the Internet, will dynamically “refresh” the running page or, if the page is no longer visible, will refresh the cache. I’m deliberately waving my hands a bit about how the client decides what data to fetch when. I’m giving a keynote talk about this at XML 2003 and I want to save some of my thunder. So far, though, I�ve described a read only model, great for being able to access information warehouses and personal data and like clinical trial history or training materials or find good restaurants in the neighborhood or do project reviews all while offline, but not as good when used for actually updating the clinical trials or entering notes into them or building plans for a team or commenting on the training materials or booking the restaurants.

It�s a fake:It is very important to remember in this model that “reality” usually isn’t on the device, be it a PC or a Blackberry or a Good or a Nokia 6800. Because the information on the device is incomplete and may have been partially thrown out (it is a cache) you don’t know really which tasks are in a project or which patients are in a trial or which materials have been written for a section. You only know which ones you have cached. The world may have changed since then. Your client side data (when running offline) may be incomplete. So, if you modify data, you need to remember that you are modifying data that is potentially out of date.

Don�t change it. Request the change:Accordingly, I recommend that the model is that, in general, data isn’t directly modified. Instead, requests to modify it (or requests for a service) are created. For example, if you want to book a restaurant, create a booking request. If you want to remove a patient from a clinical trial, create a request to do so. If you want to approve an expense report, create a request to approve it. Then relate these requests to the item that they would modify (or create) and show, in some iconographical manner, one of 4 statuses:
1) A request has been made to alter the data but it hasn’t even been sent to the internet.
2) A request has been sent to the Internet, but no reply has come back yet.
3) The request has been approved
4) The request has been denied.

Expense Reports:Let me start with a simple example. While offline, the user sees a list of expense reports to approve. On the plane, he/she digs into them, checks out details, and then marks some for approval and adds a query to others. All these changes show up but with an iconic reminder to the status/query fields that these fields reflect changes not yet sent to the Internet. The user interface doesn’t stall out or block because the Internet isn’t available. It just queues up the requests to go out so that the user can continue working. The user lands and immediately the wireless LAN or GPRS starts talking to the internet. By the time the user is at the rental bus, the requests for approval or further detail have been sent and icons have changed to reflect that the requests have now been sent to the Internet. Some new data has come in with more expense reports to be approved and some explanations. By the time the user gets to his/her hotel, these requests on the Internet have been de-queued and processed invoking the appropriate back-end web services and responses have been queued up. By the time that the user connects in at the hotel or goes down to the Starbucks for coffee and reconnects there (or if the device is using GPRS much sooner) the responses have come in. If the requests are approved, then the icon just goes away since the changed data is now approved. If the requests are denied, then some intelligence will be required on the client, but in the simplest case, the icon shows a denied change request with something like a big red X, (this is what the Blackberry does if can’t send mail for some reason as I learned to my sorrow on Sunday). The user sees this and then looks at the rejection to see why.

Notice that all this requires some intelligence on the part of the web services browser and potentially some intelligence on receipt of the approvals or denials from the internet. In the model I’m imagining, the client side intelligence will be done in script that will be associated either with the user actions (pressing submit after approving or querying) or with the Internet actions (returning approval or rejection). The script will have access to the content and can modify it. For example, on receipt of a rejection, it might roll back the values to their prior ones. Server side intelligence will be handled using your web service server of choice.

Restaurant Reviews and booking:Let’s take a slightly more complicated example. I’m flying into Santa Fe and don’t know the town. Before I leave NYC for Santa Fe, I point at the mobilized URL for my favorite restaurant review and check off a price range and cuisine type (and/or stars) that I care about. By the time I get on the plane and disconnect from wireless or GPRS, the review has fetched all the restaurants and reviews for the restaurants I’ve checked off onto my PC or PDA. On the plane, I browse through this, pick a restaurant, and then, ask to “book it” since the user interface shows that it can be booked. A Booking request is then shown AND the script also modifies my calendar to add a tentative reservation. Both items clearly show that the requests have not yet left my computer. When I land, the requests go through to the Internet and on to the booking web service and to exchange. It turns out that the restaurant has a free table and I get back an approval with reservation number and time. But the service acting as a middle man on the Internet also updated my “real” calendar to reflect this change. Now I need to replace the tentative reservation in my calendar with the real one created in Exchange by the Internet and I might as well delete the booking request since my calendar now clearly shows the reservation. Script handles this automatically and I’m OK and a happy camper. But should I have even modified my local calendar? Probably not since the integration process on the Internet was going to do it anyway and it just makes it hard to synchronize. I should have waited for the change on the calendar to come back to my client.

In practice this tends to work:This all sounds quite tricky, but as someone who has been using a Blackberry for 3 years now, it really isn’t. You get very used to eye-balling your mail to see if it has actually been sent yet or not. You soon wish that the changes you make to your calendar had similar information since you’re never sure that your tireless assistant hasn’t made some changes to your calendar that conflict with your own and you want to know, are there changes approved or not. What it does require is a decision about where changes are made and how the user is made aware of them. If the user is connected, of course, and the web services are fast and the connection is quick, then all this will essentially be transparent. Almost before the user knows it, the changes will have been approved or rejected and so the tentative nature of some of the data will not be clear. In short, this system works better and provides a better user experience when connected at high speeds. Speed will still sell. But the important thing is that it works really well even when the connection is poor because all changes respond immediately by adding requests, thus letting the user continue working, browsing, or inspecting other related data. By turning all requests to alter data into data packets with the request, the user interface can also decide whether to show these overtly (as special outboxes for example or a unified outbox) or just to show them implicitly by showing that the altered data isn’t yet “final” or even not to alter any local data at all until the requests are approved. For example, an approvals system might only have buttons to create approval/denial requests and not enable the user to directly alter the items being approved (invoices, expenses, transfers) at all.