Hobu, Inc.

In the midst of having a baby (well my wife, anyway ), I was diligently working on completing OGR ArcSDE write support for the upcoming GDAL 1.6 release. This post describes the state of that work and asks for testers to come in and start giving it a whirl.

MapServer has supported querying from versions for almost four years (I think this is something ArcIMS still doesn’t support, but ArcGIS server does), but OGR’s ArcSDE driver only supported querying from SDE.DEFAULT and basic query operations. I was contracted to flesh out the rest of the driver, including write support and participating in ArcSDE’s versioned editing machinery. In January, I received an excellent patch from Shawn Gervais of project10.net implementing geometry conversion and basic ArcSDE layer/attribute field creation that was a fantastic starting point for the work. With the hard bits done, I was able to jump into the miserable bits — implementing the versioned editing/query support.

To be an effective player with ArcSDE, client software really must participate in the versioned editing scheme. After the “base” tables are created and loaded in ArcSDE for a multiversioned layer (”registered with geodatabase” in Arc* parlance), ESRI clients typically edit layers by manipulating the adds/deletes/modifieds tables through ArcSDE’s edit state/versioning machinery. If you would want your edits to an ArcSDE layer from OGR to show up in ArcMap, for example, your changes must happen within this machinery, or they would not be available until the adds/deletes/modifieds tables were “compressed” down into the base tables — an operation that requires locking out all clients from the entire database. The simplistic description of these operations is:

1. The version client wishes to edit on is locked, preventing other clients from moving its state
2. A new edit state is created
3. ArcSDE is told the edits within this session are happening on the newly created state
4. Edits are made
5. The transaction is commtted and the version’s state is moved to the newly created state
6. The lock on the version is removed

OGR doesn’t provide a clean mapping of a number of the concepts of ArcSDE, however. A first hurdle is transaction support — OGR has rather anemic transaction support that happens at the Layer level and ArcSDE’s happens at the connection/DataSource level. To overcome this, OGR’s driver puts all of the operations within the opening/closing of the OGR DataSource. I’m not sure I like this compromise, but I can’t think of a cleaner way to do it without foisting the pain of the user having to know what stage in the editing scheme a particular operation is at and how to roll themselves back from it. The consequence of this choice is that the editing state machinery is switched on as soon as an ArcSDE DataSource is opened for update and switched off when the DataSource is Destroy()’d. Re-opening an ArcSDE DataSource connection is rather heavy (1-2 seconds), so it might be necessary to revisit this choice if people want high-throughput, multi-user operations.

Another challenge is ArcSDE’s coordinate spaces. ArcSDE has fixed coordinate spaces that are defined at layer creation time. This restriction has to do with the underlying implementation actually using integers with offsets to store coordinate info. Coming up with reasonable defaults for defining this space is challenging, especially if you don’t know the geographic extents to which someone might attempt to store data. This problem isn’t limited to just OGR, however, and many an Arc user has had trouble coming up with good X/Y domain values. For OGR, the values are set based on the coordinate system, and there currently isn’t a way to override that default, but if people have trouble with it, it shouldn’t be too difficult to allow a way to override it.

The final hitch is that there are so many options with ArcSDE that don’t map to OGR concepts. Beyond the X/Y domain stuff, there’s whether or not to use multiversion tables, DBTUNE keywords, and so on. Many of these are available as either layer creation options or general OGR environment variables. More will likely have to be added as actual users start attempting to do real work with the code. Leaky Abstractions, indeed.

Come help test

A final note for folks who might be interested in this stuff. Post a comment here or email me if you wish to get a Windows build (against 9.1, 9.2, or 9.3 (beta program FTW)) to test things out and see if they behave properly for you. I have an EDN license, which gets me ArcSDE, but it doesn’t get me ArcMap, so I have no way to really test if things are behaving exactly as they should. Testing help would be much appreciated.

The last two months have also been all-ArcSDE, all-the-time for me, as I’ve worked on three development efforts that have focused on integrating the ESRI database abstraction technology with Open Source technologies.

PySDE

In 2002-2003, I developed a SWIG’ified wrapper to the ESRI ArcSDE C API called PySDE that wrapped up the library and allowed its use via Python. I’ll be the first to admit that it was ugly as hell, but its development was my first introduction to the Open Source GIS community, and although there were some good ideas in it, to be truly useful and sustainable the thing needed to be rewritten.

I’ve been noticing more and more posts on the ArcSDE forums about folks wanting to script and program the ArcSDE SDK with C# rather than going through ArcObjects. I suppose they’re wanting to do this for more control and performance reasons. My experience with both MapServer and GDAL and their SWIG bindings has been very instructive, and I have started rewriting PySDE to follow GDAL’s approach and layout. This means that there is good potential to support more than just Python. I think that PySDE could still fill a useful niche for folks, and I’m looking for folks who wish to support its further development. Head to http://sde.hobu.biz if you’re interested in finding out more and looking at the current state of the code.

GDAL ArcSDE Driver

Project number two related to ArcSDE was the big one — development of a full GDAL raster driver. I had posted a little bit about this last fall, and I was able to secure some funding to make it happen. GDAL now has a driver checked into subversion that supports overviews, statistics, many different data types (1, 4, 8 bit and so on), coordinate systems, and colormaps. Head to http://www.gdal.org/frmt_sde.html to find out more information about the driver. If you’re interested in testing it out (I have been looking for more testers…), grab a copy of FWTools 1.2.2, check the box for the ArcSDE plugin, and after installing, head to http://hobu.stat.iastate.edu/mapserver/build_output/gdal_SDE.zip to get a new version of the plugin that has seen a number of improvements since the FWTools release. You’ll need the ArcSDE 9.1 SDK on your PATH to be able to use it.

Development of this driver was an excellent learning experience, and I would like to thank Frank Warmerdam for his guidance while I developed. I hope that the ArcSDE raster driver proves to be very useful, especially for government types hoping to wedge Open Source GIS’s foot in the door in their machine rooms. I think it (along with MapServer’s SDE support and OGR’s SDE vector support) has the potential to be a major glob of glue for organizations not looking to abandon their editing and analysis tools while still looking to things like MapServer and MapGuide to fulfill their web story.

MapServer ArcSDE Joins

The final ArcSDE item I worked on was small, but not so straightfoward. I added the ability for MapServer’s ArcSDE driver to one-to-one joins with another in-database table. Obviously this is very specific and only useful to a very small percentage of the userbase. One area where spatial databases like PostGIS and Oracle (and maybe MySQL if it ever supported geometric algebra and OGC Simple Features operations) is the construction of a query defines what the “layer” is as far as MapServer is concerned. ArcSDE doesn’t have that luxury, and some fire-burning hoop-jumping is required even to do a simple join.

Hobu Pro

With the upcoming move to the new city, I will also be making Hobu, Inc. a full-time endeavor. It will be an exciting and challenging step to take, but I think that now is as good as time as ever to try and build it into a sustainable business. The timeframe for going full-time is still in the air a bit, as we’ve not yet closed on our house in the old city, but once we know the house is sold (buy it here!), I will give my notice at Iowa State University and head off across the state to do GIS software development and consulting full-time. The hope is that we’re moved to the new city by the middle of April, but real estate markets as they are, it’s hard to know for sure.

I’m excited to bring it full-time, and your chance to hire Hobu Pro ™ to do your proprietary <-> open source GIS software bridging development dirty work is soon approaching