Hobu, Inc.

Mac OSX (and all? BSDs) provide a kernel event notification mechanism called kqueue. Other operating systems provide a similar sort of notification system, but OSX’s is the only one I am interested at this time. I did find a rather interesting cross-platform notification library called libevent-python, which attempts to provide a unified API, but it appears to be geared toward network communication and not toward file system notification.

Here’s a BSD-licensed class that can be used to watch a directory. It requires the PyKQueue from hpio.

import osimport kqueueimport sets

class DirectoryChangedException(Exception):    pass

class Watch(object):    def __init__(self, directory):        self.directory = directory        self.dirfd = os.open(self.directory, os.O_RDONLY)        self.filter = kqueue.EV_ADD|kqueue.EV_CLEAR|kqueue.EV_ENABLE        self.event = kqueue.Event(self.dirfd,                                  kqueue.EVFILT_VNODE,                                  self.filter,                                  fflags=kqueue.NOTE_WRITE|\                                         kqueue.NOTE_DELETE|\                                         kqueue.NOTE_EXTEND,                                  data=self.dirfd)        self.kq = kqueue.kqueue()

    def watch(self):        self.files = sets.Set(os.listdir(self.directory))        self.adds = None; self.deletes=None; self.changed=None         while 1:            events = kqueue.kevent(self.kq, [self.event], 10, None)            if events:                new_files = sets.Set(os.listdir(self.directory))                self.deletes = list(self.files.difference(new_files))                self.adds = list(new_files.difference(self.files))                self.files = new_files                raise DirectoryChangedException, \                      “Directory %s changed…” % self.directory    def __del__(self):        os.close(self.dirfd)

One way to use it is as so:

import watchw = watch.Watch('/Users/hobu/foo')

while 1:    try:        w.watch()    except watch.DirectoryChangedException:        adds = w.adds        deletes = w.deletes        print "added files: ", adds        print "deleted files: ", deletes

It seems my post caused a bit of a stir. Andrew Hallam has the most complete linklog of the discussion. Here’s an update on what’s currently going on:

• The copyright/spec licensing issue is gone. This was an oversight, not malice.
• In private email, I was assured that by playing within the OGC, GeoRSS would not be “locked out” in the sense that individuals who cannot join OGC cannot participate in the standard development. We’ve yet to see details on how exactly this would work given that it is incongruent with the way OGC currently operates, but nonetheless this is encouraging.
• The mechanics and details of how exactly the “edge”, as Ed Parsons puts it, is supposed to interact and collaborate with OGC still make me a bit nervous.
• I still think that GeoRSS is/was being used as an OGC membership marketing opportunity for places like Where 2.0.

I can appreciate that GeoRSS would be somewhat new ground for OGC, especially given that it is one of a number of geo standards that was mostly developed outside the direct realm of influence of the OGC. I’ll also gladly give OGC credit for having a mostly positive influence on the industry, but I would state that the job isn’t done yet. Interoperating down in the trenches with member’s implementations is still a rough, unforgiving, and frequently frustrating experience.

Maybe whether or not OGC shepherds GeoRSS will have no impact on how it is implemented, but for me, as a developer, a member’s self-proclamation of “OGC support” can often mean an incomplete, inconsistent, and unfulfilling experience. The simplicity of GeoRSS makes it brain-dead simple to avoid repeating that experience. In that sense, GeoRSS going into the meat grinder of OGC still makes me a bit nervous that there is the potential for what goes in might not be the same thing as what comes out.

Errata

Ugh, Plone gave me fits this weekend as I was updating to a new version. I ended up re-publishing all of my entries again, causing a very public spamming of Planet GeoSpatial. I think things are mostly in order now. If you find something drop me an email.

Also, I do not allow comments and trackbacks on this weblog because I’ve no patience for dealing with the spam. Everyone has a weblog these days, right?

One area where the OGC model of standards development miserably fails is in the world of geohackers, enthusiasts, masher-uppers, and open source developers (see this Directions Magazine article for more background on OGC’s sometimes conflicting roles in the GIS industry). OGC is a membership organization, and even though they might have a lot to contribute to the discussion, individuals don’t generate any revenue for OGC and aren’t allowed to join. But the geohackers don’t really need the overhead of OGC either. They have the tools they need to develop a standard at their fingertips — an email list, a spirit of cooperation, a text editor, enthusiasm, and example applications. Raj Singh, Josh Lieberman and Allan Doyle demonstrated this by leading and coalescing the development of GeoRSS through EOGEO maillists.

The OGC is a faith-based initiative. The OGC is an industry-based organization that agrees upon GIS standards behind closed doors, releases (some of) those standards to the world, and hopes that members (and third parties) implement those standards to the specification. There is a certification process where an organization can buy a stamp of OGC approval for their application or implementation. Not every application or organization that participates in OGC has the official stamp, however. The implementation of OGC specs for the large GIS vendors historically has been uneven at best, with bits and bytes dangling about — giving the hope and appearance of interoperability but ultimately failing to deliver on the promise. In fact, in many ways it has been the small vendors and open source implementations like MapServer and GeoServer that have carried the torch of true OGC interoperability.

Yesterday, Carl Reed of OGC stated that he had posted the contents of most of the GeoRSS website as a white paper for specification consideration within the OGC. This is problematic for more than a couple of reasons, as Allan outlined in his response. First, I don’t think that GeoRSS ever asked to be an OGC standard. Second, the OGC white paper completely disregarded the Creative Commons license and claimed “Copyright © 2006 Open Geospatial Consortium, Inc. All Rights Reserved.” on a document where most of the contents were generated by the participants of the GeoRSS list and website. Third, even if there was popular interest by the members of the GeoRSS list to have GeoRSS be an OGC standard, a significant portion of them would be locked out of the development process because individuals can’t be members of OGC.

This move by OGC is very troubling. It appears the OGC leadership are trying to take the GeoRSS standard, which the members of georss.org developed out in the open, and stick it behind the OGC firewall in an attempt to gain membership revenues and derivative revenues (if there ever are any, ala Dave Winer and the sale weblogs.com to Network Solutions) from GeoRSS. Maybe they are trying to get on the Web 2.0/Where 2.0 bandwagon, but I don’t think shooting the driver (georss.org) and holding the wagon ransom (buy an OGC membership and you’ll get to participate) is an even remotely reasonable way to go about it.

Hopefully, I’ve got the story completely wrong and the old saw about “never attribute to malice that which can be explained by incompetence” applies in this case. Either way, the situation stinks.