FeedMuncher and the Feed Aggregator API

FeedMuncher is a browser-based feed aggregator, built with a server-client architecture. Unlike other existing feed aggregators, all server-side aggregator functionality and user data is accessible through a newly designed API, the Feed Aggregator API (FAAPI). The current reference client is built solely through XHTML, CSS and JavaScript, while the server has been implemented in PHP with a Postgresql database backend. However, as long as the servers and clients communicate using FAAPI, there are no limitations to how servers and clients are implemented.

FeedMuncher's primary significance lies in the Feed Aggregator API, which has the potential to solve a real world problem today. There are a wide range of feed aggregators available today, ranging from desktop clients to web-based services. While this wide range of clients and services provide users with some freedom of choice, once the user has made a decision, he/she is subjected to a virtual lock-in.

The current situation is analogous to email in the early days of the consumer Internet, when services like CompuServe dominated the user experience. Such earlier services used proprietary software and protocols for the entire stack; users had to use proprietary software to connect to the network, and proprietary software to manage email. Even if users had the option of switching service providers, there were no standard protocols through which user data could be fetched or synchronized.

Today, most email accounts generally support standardized access or management protocols, like POP3 or IMAP. Through such protocols, users can freely choose and switch between clients without loss of relevant meta data. For example, a user accessing an IMAP account with Mozilla's Thunderbird can create folders, move messages, and mark messages as read/unreaded/deleted, etc. All changes made to IMAP accounts using such desktop clients are also visible and accessible through, for example, IMAP-compatible web interfaces when the user is on the road.

Unfortunately, feed aggregators today are still stuck in the paradigm of the early 90's email. If a user subscribes to a list of feeds and categorizes them, it is not trivial to export/import this information in other clients and services. If a user marks a post as read in one client, this information can not be retrieved by any other client. Because such meta-data can not be synchronized across multiple clients, users can not switch clients without suffering loss of data, or pay a high migration cost. Such barriers effectively empose a lock-in on users, while stifling innovation in the feed aggregator space.

The goal of FeedMuncher and FAAPI is to change all that.

Here are some basic features, currently implemented (or planned to be implemented):

• Subscribe to feeds and assign arbitrary tags (or categories)
• View recent items in a feed, set of feeds, or feed category
• Annotate feeds
• Mark items or feeds as read
• Tag items for archival purposes [not yet implemented]
• Search past items [not yet implemented]

The current FeedMuncher server is implemented in PHP with a Postgres backend. The server is actually merely a FAAPI server, and provides no other functionality. All FAAPI method invocations are accepted by a single handler script (faapi.xml.php), then passed on to the appropriate method handler. The server allows for different versions of the FAAPI to be supported in a single installation, and also allows individual methods to be easily overridden by adding extension files.

The current FeedMuncher client is implemented purely with XHTML, CSS and JavaScript. Once the "client" is downloaded to a browser, all functionality is provided without having to reload the page or access other html pages, and all interactive and dynamic aspects of the client are implemented purely with JavaScript. The client communicates with the server by making FAAPI calls through XMLHttpRequest, using a technique commonly referred to as AJAX (Asynchronous JavaScript and XML). XMLHttpRequest allows JavaScript to download data from the web server asynchronously, without reloading the page, and allows the script to access and manipulate returned data directly as native objects. In the case of FeedMuncher, the JavaScript code extracts necessary data from XML data returned as a result of FAAPI method invokations, and updates the display dynamically as required.

FAAPI is designed to be easy to use, light-weight, and flexible, and combines aspects of REST and XML-RPC. The API is REST-like in that methods are invoked as HTTP GET and POST queries, however, the response XML schema is modelled after XML-RPC. The API has built-in support for versioning, multilingual support, as well as authentication and session management.

When proposing/designing new protocols or public APIs, it is prudent to first examine what alternative options are available today. A few standards, protocols, formats and APIs that could conceivably be used are discussed below:

• Radio's XML-RPC Interface for the Aggregator is perhaps the most comprehensive API available. Unfortunately, the API seems to have not been updated in several years, and does not support common aggregator features like feed categories and item tags. Also, being XML-RPC, the API is slightly more cumbersome to use than REST.
• Bloglines Sync API is extremely limited in functionality, with only 2 methods supported.
• OPML is frequently used to store lists of feeds or bookmarks users have subscribed to. However, it can not be used to synchronize readership information (i.e. which posts were marked read, deleted, etc). Also, since OPML is a file format and not an API, it can only be used to export (or otherwise output) the information, and not for updating.
• Attention.xml is a proposed standard for storing a person's readership information, but, again, like OPML is only suitable for exporting/outputing such information.

As is hopefully evident, none of the above alone can fulfill what FAAPI aims to do.

For that matter, why a protocol on top of HTTP? The main reason is simplicity. A HTTP-based protocol allows server implementors to use the same ubiquitous and familiar languages and environments as those used to create dynamic web sites, without worrying about the complexities of a full-fledged TCP server. HTTP also offers some of the basic plumbing, without adding too much overhead.

Why REST and not XML-RPC or SOAP? Again, simplicity is the key. A REST-like API where methods are invoked as HTTP queries is much easier to work with than the likes of XML-RPC and SOAP. With REST, you can type queries into a web browser and get results. On the server side, queries can be accessed directly in environments like CGIs and PHP, without having to unpackage data. In short, REST gets the job done with minimal fuss.

FAAPI methods are invoked as HTTP queries. Methods that modify data on the server-side should use POST, while methods that merely retrieve data should use GET queries. A simple query might look like this:

faapi.xml.php?method=fetchRecent&tags=tech+blogs&limit=30&since=3+days+ago&showdeleted=0

In FAAPI, it is possible to "stack" multiple methods in a simple invokation, to minimize overhead. There are no limitations to the combinations of methods that can be invoked in a "stack", however, all parameters sent will be shared by all methods in the stack. A "stacked" query may look like:

faapi.xml.php?method=listFeedTags,listFeeds,listFeedTagAssoc,listTagFeedIDs,fetchRecent&tags=

When processing "stacks" of methods, the server should halt when an error is encountered. That is, if there are 3 methods in the "stack" and 2nd method returns an error, the 3rd method should never be executed.

Most methods in FAAPI require authentication, however, FAAPI's authentication mechanism is slightly more versatile than that in some XML-RPC based APIs. Clients may authenticate any meethod invokation using the following methods:

• Session cookies - A login method invokation will implicitly set a browser cookie containing a session ID. For browser-based clients, future method invokations will not require explicit authentication since the server will recognize the session cookie.
• Session ID - The login method returns a session ID, which can then be included in future method invokations.
• Username and password - In the worst case scenario, clients may chose to simply include the username and password parameters in a method invokation.

A FAAPI server may support multiple versions in a installation. This is particularly important for newly proposed APIs like this, where successive versions may not be backwards compatible. A client may request a particular version of a method be used by passing a version parameter. If a server does not support a requested version, it may return an error, or return the results of the same method from a different version (see more in Result Objects section).

An FAAPI method invokation returns an XML array of result objects. A result object currently contains the following fields:

• method - method name
• version - version of the method actually used (may be different to that requested)
• code - result code. 0 or above if successful, <0 on error
• data - result data. May contain error data, if error, otherwise requested data.

A sample (successful) result looks like this:

<results>
	<result>
		<method>listFeedTags</method>
		<version>0.1</version>
		<code>5</code>
		<data>
			<tags>
				<tag>friends blogs</tag>
				<tag>mac stuff</tag>
				<tag>news</tag>
				<tag>tech blogs</tag>
				<tag>tech news</tag>
			</tags>
		</data>
	</result>
</results>

A failed request may return something like this:

<results>
	<result>
		<method>login</method>
		<version>0.1</version>
		<code>-1</code>
		<data>
			<error>
				<errmsg>Authentication failed</errmsg>
				<errnum>-1</errnum>
			</error>
		</data>
	</result>
</results>

The following is a list of all supported FAAPI methods. More detailed information on each method will be available in the near future:

• login - Login/authenticate
• logout - Logout and end session
• register - Creates new user account
• listFeeds - Returns a list of feed objects
• subscribeFeed - Tag and add URL (or feedid) to user's subscription list
• unsubscribeFeedID - Unsubscribe user from given feed ID
• batchSubscribe - Subscribes URLs extracted from a block of text
• markReadUpto - Mark all posts in feeds as read, up to a given timestamp
• fetchRecent - Fetch recent posts
• listFeedTags - Returns list of unique tags (categories) user has assigned to feeds
• listFeedTagAssoc - Returns list of tuples mapping feed id to tag
• listFeedIDTags - Returns list of tags user has assigned to a given feed ID
• setFeedIDTags - Sets the complete set of tags assigned to a tag to the one given
• listTagFeedIDs - Returns list of feed IDs that have a given tag assigned to them
• setFeedComment - Sets a user entered comment to a feed
• flagItem - Flags an item as read, deleted, etc [not yet implemented]
• listItemTags - Returns list of tags assigned to an item [not yet implemented]
• tagItem - Assigns a user-defined tag to an item [not yet implemented]