Over the past year or so I've encountered several products with embedded computers running open-source operating systems like Linux and BSD that have attempted, with varying degrees of success, to be (or to appear) ``developer friendly''. Some, notably the Agenda VR3 PDA, have been fairly successful at it. Some continue to muddle by; some have failed miserably. Some have yet to face the cold glare of reality -- this little essay is an attempt to advise these newcomers in hopes that they will be among the few that thrive in the often unruly but always interesting environment of open-source development.
In my opinion none of the products currently available are really developer-friendly. For the most part they simply fail to be sufficiently user-hostile -- developers are so starved for good, inexpensive hardware that they take such benign neglect as encouragement. The result is products like Tivo and ThinkNIC, which have developer communities that are barely tolerated -- certainly not acknowledged -- by their manufacturers.
At best, as with the Agenda, there is a link on the company's main website to the developers' site. A thriving developer community is one of the greatest assets a product can have -- every new application represents several new sales. Some may represent thousands. The developers are, almost invariably, users of the product -- the boldest, most imaginative, most expressive users, making their desires and requirements known in both the icily unambiguous language of source code and the occasionally overwrought rhetoric of their discussion lists.
A smart company listens to its developers if it wants its products to be successful. A wise company works with them. The recent phenomenal success of the Linux operating system (about 30% of internet-accessible servers) and the Apache web server (60% of the domains on the public web) are due entirely to their vibrant developer communities.
This essay is not about how to set up a developers' website. There are plenty of good examples for that -- and if a company doesn't do it the developers will happily join in. (Agenda, as always a source of good examples, has a mediocre developers' site themselves, with only the ``official'' code distribution, the faq, and the mailing lists. However, they also have a links page that points to the unofficial web sites, which are highly informative.)
On the contrary, this essay is about how a company can make the product itself more developer-friendly, and in fact can do so without any compromise to the user interface as seen by the ordinary consumer. It must be realized that this essay is just that -- an expression of personal opinion. On the other hand, the author has considerable experience to draw on, including several developer-friendly software products.
It's important to realize that ``developer'' usually does not mean a company that will write a shrink-wrapped application for your product and sell it to your customers. Far from it. In most cases, the average developer will be one of your customers with a bit -- or a lot -- of software experience, and a simple problem to solve.
More and more people -- not just developers -- are encountering Linux or some other open-source software system and learning how the open-source community works. Increasingly, the reaction of such people to a perceived deficiency in a product or an obviously desirable application is to go out on a newsgroup or mailing list and post a message:
Does anyone have a (program to do X / patch for this problem)? Can I do anything to help?
Often the answer is a a URL pointing at the home page for just such a project, or to a file that patches the problem. At least as often the answer is ``no, why don't you start one.'' Users are getting very self-reliant these days, as they discover that a simple shell or Perl script is no more difficult than a spreadsheet or a Word macro, and that browsing through well-written code, even in an unfamiliar programming language, to find the answer to an obscure question is usually faster than going through an endless chain of clueless phone-support technicians.
The traditional embedded computer is a ``black box'' hidden deep inside a product. The traditional desktop computer or server, in contrast, is a ``white box'' with most of its workings exposed to the user -- or rather to those users sophisiticated enough to be interested in such things. There are costs to both approaches, of course.
The most common manifestation of ``black box'' thinking in embedded systems is the fact that the user is given no access to the system's internal workings, and in particular is given no way to extend the system by installing new software. This adds considerably to the system's stability: there is no danger of the user intruducing defective or malicious applications into a smoothly-functioning system. Unfortunately, it also means that the user is stuck with whatever functionality the designers of the system chose to provide.
An increasing number of black box systems are based on embedded versions of Unix, including Linux and BSD. There are many good reasons for this: the free Unix systems are extremely reliable, their networking capability is superb, there are many device drivers available (particularly important for USB and PCMCIA peripherals), and of course there are no licensing costs. Developers are plentiful -- any experienced Linux or Unix developer can easily transfer their skills to an embedded project.
But black-box systems based on embedded Linux or Unix are intensely frustrating to any developers who happen to purchase one, because in almost all cases it's difficult or impossible to install software on what the sophisticated customer knows to be a supremely versatile and familiar platform. It's a little like buying a furnished apartment only to discover that, not only are there no bookshelves, but that the lease prevents you from installing any new furniture. This is why people buy their own homes -- and personal computers.
It's rare for a company to anticipate every possible need, desire, and whim of their customers, especially for something as complex as a computer-based, networked appliance. In fact, it's impossible. Without outside developers a product can only succeed if it fills a common need far better or at a lower price than the equivalent in software on a general-purpose computer. Even a successful product of this sort will only prosper until the general-purpose computing world catches up -- nobody buys dedicated word processors anymore. Even small size and portability isn't a guarantee of success: who carries both a calculator and a Palm Pilot?
By far the easiest way to make a programmable device more useful to more people is to actively encourage developers to write new applications for it. In the absence of good developer support, of course, a mere lack of discouragement will often suffice -- the NIC is surviving where other network computers failed largely, in my opinion, because it's relatively easy to replace the firmware (it's on CD-ROM in a standard format) and because there is no need to subscribe to an associated service: the device is sold on its own merits rather than as a loss-leader for a service that nobody really wants.
But wouldn't it be better to attract developers to your device? Of course it would.
The programmer is given a number of hooks on which to hang himself.
the MicroPlanner manual, ca. 1970
The simplest way to start making a device developer-friendly is to provide the ``hooks'' that make it easy for devopers to add their own enhancements to the product. It is not usually necessary to create an entire specialized application programming interface (API). Most operating systems have a well-defined set of interfaces to both IO devices and to other programs -- in most cases product-specific interfaces will fall into one category or the other.
The device driver interface is the most effective way to isolate
proprietary hardware and its associated low-level software. Unix devices
look like sequences of bytes -- all structure is provided by the
application. The IOCTL (I/O Control) operation is provided
for device control. It's rare that any specialized hardware cannot be
shoehorned into this simple framework.
By the way, if the hardware is patented or otherwise protected, there's no reason not to make the driver open source. Opening the source code for a driver provides a good backup in case the documentation is insufficient on some obscure point.
More complex subsystems are best exposed to the user as servers; the Web provides a ready-made framework on which to build. In fact, there is no better user interface for a server-based appliance than a web browser, and it's easy to make all of a device's functions available as web forms. If there are proprietary software components to a system, a server is an excellent way to isolate and protect them.
It is hardly ever necessary to provide a complete API in the form of a library. Some of the exceptional cases include special-purpose co-processors (such as encryption or compression chips), general-purpose co-processors such as DSP's, communication over unusual or proprietary protoccols, and integration with proprietary software. In many cases a little thought will find a standard interface protocol that can be adapted, and that will always be better for developers.
It is never necessary to provide a complete development
environment. Linux and Unix are superb software-development
environments, and most developers who use them have already settled on
their favorite tools and utilities. If your product is based on a
non-Intel processor, it is of course necessary to provide a suitable
cross-compilation environment (gcc, libraries, and
#include files), or at least a pointer to it. But it
shouldn't be necessary to go beyond that point.
Nothing is more important for your product than its developer community, and nothing is more important for the community than its website. Always bear in mind that the developers' website is a tool for community-building. It shouldn't look like an advertising brochure -- save that for your company's main site. It should be practical, easy to navigate, very light on the images, friendly, casual, and above all useful.
SourceForge, or rather any project site hosted on SourceForge, is a good example of a developers' website. Most of the SourceForge software is still available if you prefer to build your own. In fact, when the owners of the SourceForge announced that they were migrating parts of the system to proprietary software, a group of open-source developers brought the original code to a new site (Savannah) to continue development in an open environment. Another open-source package on which you can base a developers' site is Tigris. SourceCast, like SourceForge, is a hosted collection of sites that's mainly aimed at open-source projects by large companies, including HP, Sun, and Nokia.
Don't bother with a web-based discussion or forum package. Instead, set up some mailing lists (using, for example, MajorDomo or MailMan), and keep the archives on the website. The archives allow a new arrival to catch up, while the mailing lists allow each developer to use the mail software they prefer. Unlike a web forum, developers get the latest postings as soon as they arrive, without having to navigate through a web site to find them. A developer may be following several projects, after all.
A Wiki server, on the other hand, can be very useful as a repository of lore and documentation; a good example was set up for the Agenda VR3 at AgendaWiki.com/. A Wiki, unlike a mailing list, allows and in fact encourages online editing of its pages, which are automatically cross-linked. This allows developers to refine their own reference material. The biggest problem with a mailing list is searching the archives to find the few nuggets of information among reams of discussion.
Note that the Agenda Wiki, along with many other helpful sites, was set up not by the manufacturer but by an individual developer. Don't discourage such sites -- link to them, on a page labeled ``Resources'' or ``Links''.
A developers' site should also have a ``news'' section, but it needs to be news of interest to developers: new software releases, release notes, development roadmaps, project proposals, and the like. Put the consumer-oriented press releases on your main site.
The centerpieces of a developers' website are the CVS tree and the bug database.
The files associated with a software project form a directory tree; the
top levels usually contain such high-level items as src
(source code), doc (documentation), bin
(binaries -- programs, in other words), and lib (libraries
and other files shared by multiple programs). These are usually stored in
a version-control system; for open-source projects these days it's almost
invariably CVS. The whole thing is usually called the CVS tree.
CVS stands for ``Concurrent Version System'' -- the feature that makes it ideal for projects with a large number of geographically-dispersed developers is that each developer has a complete copy of the source tree and they can all work on it concurrently. Changes are merged automatically if possible, and the occasional conflicts are reported and have to be resolved manually by the developer attempting to check in conflicting changes.
The CVS repository is normally accessed through a password-controlled
server. Developers trusted to make changes directly have their own
usernames and passwords; all others use a username like
``anonymous'' and have read-only access. Updating the entire
working copy to the latest version is done with a single command:
``cvs update''.
There are several packages available that allow a CVS tree to be browsed over the Web. This allows a developer to get a quick overview of the product and perhaps decide which pieces of the tree they need for a particular task. Bonsai, developed at mozilla.org, is pretty good. I have yet to see a CVS tree browser that also provides cross-referencing; it would be very useful.
Also consider structuring your source tree as a web site. I always put a
HEADER.html file in every directory of my source tree; that
way making it the document root of a web server instantly turns it into a
browsable collection of documents. Such a site differs from a CVS tree
browser in that the header files allow documentation to be interspersed
with the code. See
pia.rii.ricoh.com/src/ for a simple example, and pia.rii.ricoh.com/.source/
for the same source tree combined with a cross-reference.
The ``bug database'' is, if anything, more important than the CVS tree. It contains more than bug reports, of course: it also contains requests for enhancements and perhaps other ``issues''. It also provides a way for users (in their additional role as product testers) to communicate with the developers, and for developers to communicate with one another.
The Mozilla project developed one of the better bug databases: Bugzilla. It's worth visiting the Bugzilla site just to see how it's used. Since Bugzilla allows developers to add comments and attach files to a bug, it can be used for discussion around a particular problem, and to attach test cases and proposed patches.
So far I haven't seen automatic cross-linking between a CVS tree browser and a bug database browser -- this would be a useful feature for someone to add.
Hooks are good, and a developer's website full of documentation and source code is good, but if nobody knows about them they don't help sell your product. The best way to let developers know what -- exactly what -- they can do with your product is to put as much of the development environment as possible inside it. This seems obvious, but for some reason nobody is doing it.
If a product is normally shipped with a CD-ROM of drivers and applications (as most -- perhaps all -- devices that are meant to connect to a Windows computer usually are), why not include a snapshot of the CVS tree and maintenance documentation on it as well? If a product has sufficient on-board disk (as server appliances and MP3 jukeboxes, and even the Tivo and Microsoft's X-Box have), put it inside the product! What better way to entice a developer?
Most users, of course, will simply discard the development package if it's onboard in the product -- in fact it can be done automatically as soon as the space is needed, requiring that a user take an explicit action to protect it. Developers won't discard it -- they'll thank you profusely for it.
If a product is based on Linux, put at least the base (including the
package manager -- e.g., apt for Debian or rpm
for RedHat) of a complete Linux distribution on any product with a hard
drive. Provide the rest of the distribution on CD's so that developers
can install the packages they need. For ordinary customers, provide a
simplified interface to the package manager so they can install updates
without having to know anything.
For products with no hard drive, it's usually possible to provide a way of mounting a more complete set of applications via either an external disk, a removable disk (e.g. a CD-ROM), or a network connection.