Screaming Penguin - Source Does Matter

I am putting this in mostly for Atrox, but if anyone else is interested. Please, any feedback is great.

Last night I got the user security model and permissions filters in place for the new penguin aggregator. Later today, I am going to move a development build of the aggregator up to the prod server and let it run as a beta there.

The catchphase of the day is "many nodes, loosely coupled." I am shooting for extreme modularity with the new penguin. There will be several big components for the new penguin, all of which are tied together by a mix of Web Service, RSS and web service-ish systems. There there are 3 components currently running on the prod server in a limited basis: the cache server, the trackback/pingback manager and FeedPod. FeedPod is the TTS podcaster, and not part of the critical infrastructure, but a cool to have.

The cache server provides several basic services. First, it will handle the caching and automated time fetching of feeds. Second, it can provide feed type conversion. Everything in the new aggregator is built around RSS 2.0 right now because it supports the largest superset of functionality. I am monitoring and participating the ATOM IEFT protocol list right now, and once that comes to fruition, I might migrate the internals to run on ATOM since it will have the largest support. There are some hurdles with this right now, since ROME's RSS to ATOM converter is pretty limited (more on that in a minute). In the meantime, we can use the cache converter to publish ATOM feeds on the totspCode site. You can do this by giving it a feed URL of http://www.screaming-penguin.com/cache/convert/rss_2.0/http://server/ato....
(Valid options in the [rss_2.0] slot, BTW, are rss_0.91N, rss_0.91U, rss_1.0, rss_2.0 and atom_0.3.) Next, the cache server does Lucene indexing of content as it comes in over the wire. I am considering indexing all of the feeds and making them searchable, but for right now, it is limited to just the penguin. We need to get a (or several) historical RSS feeds of the penguin content generated for Lucene to index, then we can integrate the "new" search into the PHP site as well.

The atom converter to ROME needs a couple of things that I am going to implement and give back to the ROME project: 1. It needs to be aware of the TOTSP created Content Module support. Right now it will duplicate the description into the "summary" and "content" areas of the ATOM feed. It needs to know about the content. Next, it needs to know to put RSS Enclosures into the atom feed as "link rel='related'" links to the stories. There is also a little cleanup with the link typing.

The next part is the Trackback/Pingback service. There is a universal support system on the penguin now that is running. It can (a) monitor a feed and do a tb/pb autodiscover on any pages anyone links to and send the pings and (b) it has REST listener for trackbacks and an XML-RPC listener for pingbacks. We need to include the header information into the PHP code to let others autodiscover the service, and include the output. There are 2 options for outputting the information: you can get an XHTML fragment or a document.write based JavaScript file with either the number or content of the notifications. This is pretty easy to implement and we can provide this service to any friends with Blogger blogs or anything else that doesn't currently support it.

Next, as to stuff that ISN'T done...

The next part I am going to work on is what I call the "Publisher". This is going to be and X-TREME DHTML (I guess the new code word is AJAX) application. What I want is for users to be able to register, then use the publisher to publish to whatever blogs they have, even outside the penguin. It will then let you cross-post entries to multiple servers. I want this to integrate into the sidebar functionality to provide similar functionality to the Blogger "Blog This" plugins for MSIE and FF. I am going to be using the HTMLArea WYSIWIG HTML editor and some other external stuff, but I expect it will be cool. It will use it's own interface on the front end and will have a plug in system in the back that will support BloggerAPI and MetaWeblogAPI to start, then the ATOM protocol one it is stablized.

Then there is the CMS system. I haven't spent much time on this. I know Atrox wants to do a Spring thing, and I thought this would be a good place to start that effort. This really just needs to be a basic, headless CMS system that supports the MetaWeblogAPI for publishing and can render the pages. There are a lot of tools in the totsp-commons that should make this really, really easy. I will try and cover these at length later.

As to page layout in this system, there is currently the page layout system for the Aggregator which is really reall flexible. However, it can also broker XHTML fragments of the different non-content components (header, footer, left, right). There is also a new TOTSP JSP tag that will do a "virtual" HTTP include. We can stick this tag into the pages outside the aggregator that need a consistent LAF. (The search results page will soon provide an example of this.)

Another thing I would really like to see the CMS support is a "input/output plugin system". This would be a registered chain of filters that content would pass through as it is being inserted into or rendered out of the system. This will give us an extension point to do neat stuff with later. I have some ideas, but I will save that for later.

The last part is comments. I would really like to shop this out to a system like we used to have with the PHP system and Phorum, where posts are created as a thread in an external system. I really like JForum. It provides RSS feeds on forums and posts, which is pretty cool and handy. It is pretty simple from a DB standpoint, if not an API standpoint, so adding stuff into it shouldn't be a big deal -- this could actually be the first 2 plug-ins for the CMS, one to create a thread in JForum on the way in, one to get and display the posts on the way out.

JForum also has a pretty cool, user mamangement system. Right now the aggregator is using it's own stuff, but it is all Castor-mapped and structurally identical to the system that both JForum and Tomcat JDBC realms would use -- User ---- Role with MD5 hash passwords -- so change a couple of lines in the mappings and we are using the JForum users. When we go live I would like to do basic user managment with JForum since it has a good interface for that in place, then do an "extended" profile screen for all the penguin specific stuff.

That is it for right now as I need to leave. Any feedback on this would be keen.