Hobu, Inc.

In the midst of having a baby (well my wife, anyway ), I was diligently working on completing OGR ArcSDE write support for the upcoming GDAL 1.6 release. This post describes the state of that work and asks for testers to come in and start giving it a whirl.

MapServer has supported querying from versions for almost four years (I think this is something ArcIMS still doesn’t support, but ArcGIS server does), but OGR’s ArcSDE driver only supported querying from SDE.DEFAULT and basic query operations. I was contracted to flesh out the rest of the driver, including write support and participating in ArcSDE’s versioned editing machinery. In January, I received an excellent patch from Shawn Gervais of project10.net implementing geometry conversion and basic ArcSDE layer/attribute field creation that was a fantastic starting point for the work. With the hard bits done, I was able to jump into the miserable bits — implementing the versioned editing/query support.

To be an effective player with ArcSDE, client software really must participate in the versioned editing scheme. After the “base” tables are created and loaded in ArcSDE for a multiversioned layer (”registered with geodatabase” in Arc* parlance), ESRI clients typically edit layers by manipulating the adds/deletes/modifieds tables through ArcSDE’s edit state/versioning machinery. If you would want your edits to an ArcSDE layer from OGR to show up in ArcMap, for example, your changes must happen within this machinery, or they would not be available until the adds/deletes/modifieds tables were “compressed” down into the base tables — an operation that requires locking out all clients from the entire database. The simplistic description of these operations is:

1. The version client wishes to edit on is locked, preventing other clients from moving its state
2. A new edit state is created
3. ArcSDE is told the edits within this session are happening on the newly created state
4. Edits are made
5. The transaction is commtted and the version’s state is moved to the newly created state
6. The lock on the version is removed

OGR doesn’t provide a clean mapping of a number of the concepts of ArcSDE, however. A first hurdle is transaction support — OGR has rather anemic transaction support that happens at the Layer level and ArcSDE’s happens at the connection/DataSource level. To overcome this, OGR’s driver puts all of the operations within the opening/closing of the OGR DataSource. I’m not sure I like this compromise, but I can’t think of a cleaner way to do it without foisting the pain of the user having to know what stage in the editing scheme a particular operation is at and how to roll themselves back from it. The consequence of this choice is that the editing state machinery is switched on as soon as an ArcSDE DataSource is opened for update and switched off when the DataSource is Destroy()’d. Re-opening an ArcSDE DataSource connection is rather heavy (1-2 seconds), so it might be necessary to revisit this choice if people want high-throughput, multi-user operations.

Another challenge is ArcSDE’s coordinate spaces. ArcSDE has fixed coordinate spaces that are defined at layer creation time. This restriction has to do with the underlying implementation actually using integers with offsets to store coordinate info. Coming up with reasonable defaults for defining this space is challenging, especially if you don’t know the geographic extents to which someone might attempt to store data. This problem isn’t limited to just OGR, however, and many an Arc user has had trouble coming up with good X/Y domain values. For OGR, the values are set based on the coordinate system, and there currently isn’t a way to override that default, but if people have trouble with it, it shouldn’t be too difficult to allow a way to override it.

The final hitch is that there are so many options with ArcSDE that don’t map to OGR concepts. Beyond the X/Y domain stuff, there’s whether or not to use multiversion tables, DBTUNE keywords, and so on. Many of these are available as either layer creation options or general OGR environment variables. More will likely have to be added as actual users start attempting to do real work with the code. Leaky Abstractions, indeed.

Come help test

A final note for folks who might be interested in this stuff. Post a comment here or email me if you wish to get a Windows build (against 9.1, 9.2, or 9.3 (beta program FTW)) to test things out and see if they behave properly for you. I have an EDN license, which gets me ArcSDE, but it doesn’t get me ArcMap, so I have no way to really test if things are behaving exactly as they should. Testing help would be much appreciated.

General Thoughts

OSGeo bootstrapped itself from the crumbled MapServer Foundation announcement at the end of 2005. Considerable energy and leadership from people like Frank Warmerdam, Jo Walsh, Gary Lang, and Tyler Mitchell have kept it moving forward and working to attain the goals which it set forth to accomplish. I expect that it will continue to gather momentum in 2007, despite Sean and other’s continued hating on it, because the benefits it provides are already starting to be realized in terms of visibility, cross-project collaboration, and financial stewardship.

ESRI had their Vista release. I’ve been mostly out of the ESRI loop the past couple of years, due to my involvement with open source stuff and the fact that the ESRI stack is too stovepipe-like for my development group’s taste. From my small business/local government perspective and GIS weblog perspective, it is clear that many are looking at open source/cheaper alternatives to accomplishing the same tasks. Integrated and polished open source solutions will have a real opportunity over the next 18 months (obviously in the web space with Mapguide, MapServer, GeoServer, and OpenLayers, but there are other areas too).

Apple continues to clean house. I so wish I would have bought stock after recovering buying my first Mac three years ago. 2007 should be exciting in the Apple world with a new OS release for Microsoft to attempt to copy and continued hardware releases from Apple’s wise decision to hop on the i386 bandwagon. Hobu, Inc. is completely an Apple shop now as far as workstations and laptops. Apple’s servers are interesting and nice from a management perspective, but their cost does not justify their existence when compared to commodity pizza boxes and a solid Linux distribution.

Review

February 4th

I was in an O’Hare hotel for the birthing of OSGeo.

February 12th

I completed the MySQL driver for OGR. This was my first experience developing for OGR, and it was straightforward. Maybe someday MySQL will realize that they need to beef up their spatial support. Currently for most, it is but a speedbump on the way to implementing PostGIS, I think.

Late February

I released the first version of the MapServer Buildkit. The Buildkit is a big ball of wax that has everything pre-configured for you to build MapServer on Windows with MSVC2003. The Buildkit has been the base from which MS4W has been built on, and its use has eliminated my need to release an independent set of MapServer windows binaries for ArcSDE and Oracle Spatial support.

March 5th

My wife accepted a job at the University of Iowa, and I found out I’ll be moving to a new town.

April

I was busy setting up Buildbots for GDAL, MapServer, and GEOS. They have been an excellent way to keep track of developments in these projects, and Mateusz has taken up the torch of helping to implement them for all OSGeo projects that want them.

June 1st

I was at MetroGIS explaining to Clint Brown of ESRI that it makes sense to release the ArcSDE C API because third party developers building software with it will *sell them more software.* It had no effect…

June 26th

LizardTech releases their latest SDK with input from Frank Warmerdam and myself about the licensing agreement. I also did some testing to make sure it worked as a universal binary. I earned a LizardTech polo shirt for my efforts.

August 29th

I headed off to FOSS4G in Lausanne, Switzerland after honeymooning in Italy for a couple of weeks.

Sept 18th

Returned home from the conference to a broken toilet, flooded house, and wet and trashed computers. Most of the damage was contained to my office, the bathroom above, and the basement below. One silver lining was a Mac Pro to replace my dual G5, but other than that, it’s been pretty miserable.

End of November and Early December

During this timeframe, I was working on numpy bindings for GDAL, I was closing bugs in preparation for GDAL’s release, and I received a commitment for funding to support the development of ArcSDE Raster support for GDAL.

December 10th

I created a new website to revive PySDE. I will provide more detail about that in a future post.

2007 and Beyond

As I noted in my review section, my wife has accepted a job at the other university in Iowa. Telecommuting back to my job at the Center for Survey Statistics and Methodology at Iowa State is not an option, and I will be striking out on my own to try my hand at consulting full time. The exact date has not been set yet, but it will coincide with the repairs to the house in Ames from the flood and our ability to sell it. So far, my prospects look fairly good, but in a couple of months I’ll be looking to take on more. Here’s your chance to get in line to hire me

See you at FOSS4G

January

In January, I was organizing the “lightning talks” for
OSGIS ‘05. The
plan was to have a bunch of speakers be given five minutes at the beginning of the
conference to build buzz, give people a sense of what all was going on in the Open
Source GIS world, and to provide a nice kickoff to the conference.

Also in January, I was doing the technical edits and review of
Beginning MapServer: Open Source GIS Development. The book provides a very
detailed exploration through CGI and MapScript MapServer development.

Throughout the beginning of the year, I was in the process of switching to
Mac and not looking back. I still have a
windows machine that I use from time to time for compiling things and playing a
video game or two, but most of my full-time development, content creation, and
everyday webbrowsing, email, and IRC at Hobu, Inc. is all Apple.

February 10th

My neighbor, with whom I share a driveway, backed his crusty old BMW through my
garage, knocking it off the foundation, blasting out one of the doors, and
doing $3000+ worth of damage.

ER Mapper announces
that they are open sourcing their ECW wavelet compression technology.

February 20th

ESRI notified me that they were publishing
A Guide to the Python Universe for ESRI Users in
ArcUser. I
was surprised that they were willing to publish an article in their glossy magazine
that actually covered very little ESRI software and focussed more on Open Source tools
that make GIS programming fun, enjoyable, and even tolerable.

March 2nd

A whacked-out screed by the president of the company that makes
Manifold in the GIS Monitor prompted me to write a good fisking of it.

March

March seemed to me to be the month where the GIS webloggers congealed into a critical
mass. Adena was blogging. Sean was blogging. James was
just about ready to start blogging. Many more started rolling out pretty quickly.
There were finally enough out there that you could put a category together into a news
aggregator and see enough new posts in a day to make it interesting.

June 16 – 18

The middle of June saw OSGIS ‘05 happen. It had a profound effect on the open source GIS community, just as OSGIS ‘04 and MUM 1 before it. It was an exhilarating and exhausting three days.

Sean and I gave our Hacks workshop, which failed and succeeded in equal parts in my mind, and the Lightning Talks were well received and went off without a hitch.

August 20th

I got married.

August 28th

Katrina hit. I spent some late nights over the next few months working on data dissemination efforts with a really smart mob.

October 6th

I signed an NDA with Autodesk to investigate a MapServer Foundation.

October and November

Throughout the fall, I continued talking with the Open Letter group about the foundation. Near the end of it, however, I was getting
really uneasy and nervous that we were forgetting a few important things.

November 28th

MapServer Foundation website lights up and Autodesk and DM Solutions make the announcement. Things started out a bit rocky, but we’re still learning and things are still moving.

December 16th

The new MapServer website lights up. This was my quixotic quest to have a good, through-the-web editable website for MapServer that was easy to navigate, easy to search, and generated massive amounts of google juice.

Conclusion

As you can see, it was a pretty busy year for the president of Hobu, Inc. Hopefully next year will see the MapServer Foundation light up, I will get to go to Switzerland, and I will actually get a chance to write some real software.

An editorial note. The opinions I am expressing here on this weblog are entirely my own. I do not claim to speak for any of the Open Letter signatories or anyone else but myself. My recollections and ideas are nothing but my own conjecture and thought process. Hopefully others who were part of this process will have stuff to say as well. Maybe their perceptions were quite different than mine.

Sssh!

AutoDesk and DM Solutions approached members of the MapServer Technical Steering Committee (MTSC) in late early October. We were told that it involved AutoDesk, it involved MapServer, it was going to be big, and it required that we sign a non-disclosure agreement (NDA). Questions started racing through my mind… What do they want with us? Have they been working on/with MapServer? Where will this lead? My curiosity was peaked, but I was torn. Is it right to for most of the major contributors of an open source project to sign an NDA with a single company? Why can’t this be done out in the open like everything else? What do they want with us? What do they want with me? I must admit, the idea of a big geospatial software giant being interested in our little web mapping software project was very alluring.

Curiosity wins, and I sign

I signed the NDA. I wanted to see where things lead. I knew that most everyone else in the MTSC had already signed. However, I felt uneasy about it. Who could I really represent but myself? Was I supposed to be representing anyone else? That most of the MTSC was part of the group implied that I was to do so. I contribute a lot to MapServer, and probably had as much right as anyone to be part of it, but I still felt very uneasy about it.

A furious pace

Quickly after I signed, I started to come up to speed with what was actually happening. AutoDesk was releasing their next-generation MapGuide under the LGPL. They were going to take a huge risk. They wanted work with MapServer. They needed an arms-length organization to put their software in. They wanted to investigate bootstrapping an open source web mapping software foundation ala Apache, Mozilla, et. al. to incorporate at least their new project and MapServer. They wanted co-brand MapGuide and call it MapServer Enterprise to build on the mojo that MapServer had been conjuring. They wanted us to take a huge risk. And they wanted to do it in an awful big hurry. November 28th, to be precise.

We were asked to make a hard and non-technical decision. A great opportunity had presented itself. It was an opportunity to move forward with some corporate backing to create a software foundation that has the potential to give MapServer a stronger future. There had been mumblings about a software foundation for quite a while. Some had even taken to posting some information about what it could do for the project and why it would be a benefit. Here was a chance to make that happen.

Choices

The choice was this. Work with AutoDesk and announce our intentions to work together, brand things together, and accept their support. Or, we could do nothing, wait, and watch another open source project with big corporate backing, a large installed base, and fresh technology come and disrupt the same space our project was in. Now, you could argue that the second option here happens with our without MapServer’s involvement quite successfully. The twist is that with MapServer’s involvement, this happens cooperatively rather than antagonistically.

Update: maybe the choice wasn’t as binary as I make it out to be, but in my mind it was. AutoDesk was coming into the open source web mapping arena with or without MapServer’s cooperation.

Buildup to announcement day

So we made our choice. Teleconferences were attended. IRC sessions were chatted. Emails were exchanged. A letter was drafted. Verbiage was parsed. Things were agreed upon. Signatories signed. Journalists were embargoed. Press
releases were drafted. Websites were built. Letters were translated. The entire thing blazed along, building up to November 28th.

Monday, November 28th

The first day after the announcement, things were actually pretty quiet in MapServer land. People seemed a bit bewildered, enamored, confused, and enthused. Most didn’t quite understand what had just happened. Instead of
one MapServer, all of a sudden there were now potentially three. The press picked up on things (even Slashdot had it, but they linked to a press release instead of the Open Letter, and the discussion was light and focused on AutoDesk), but it appeared as though people were still in digestion mode.

The real aftermath for the MapServer community

After some reflection, a prominent member of the community posted a long message to the list expressing how pissed off he was at how the thing went down. Hundreds of emails later, it was pretty clear that “MapServer Enterprise” the name was not going to work for MapServer folks. Neither was the attempt at rebranding MapServer to be “Cheetah.” After a week of tumultuous debate, AutoDesk agreed to rethink MapServer Enterprise as a name.

My Reflection

Indeed this episode provides many lessons to be learned, but that is true of all things that have never been attempted before. So here, in bullet form, are some things I felt we the Open Letter signers got right and the things we got wrong.

Right

• The arguments for a foundation are strong, and there is a strong desire to have a MapServer Foundation.
• This announcement means that more people than ever are curious about MapServer and
  MapGuide.
• Working with AutoDesk is the right thing to do.
• The spelling of my name on the Open Letter was correct.

Wrong

• The compressed timeframe between many Open Letter signatories’ disclosure and
  the announcement caused important and big things to be missed.
• More projects, companies, and individuals should have been disclosed about
  the announcement. More involvement would have prevented some of the larger
  missteps from happening.
• Names mean a lot, and the name MapServer Enterprise is perceived as “better”
  in any way you measure it, even if the software is not fully baked yet. If the names
  proposed would have been MapServer XXX and MapServer YYY, without setting one over
  the other in any way, things might have been different.
• Press releases designed to maximize the exposure of DM Solutions and AutoDesk
  colored the entire announcement. This misplaced the focus that should have
  been on the foundation and Open Letter.
• Press releases that were contrary to the Open Letter should have been carefully
  combed through. It should have been made more clear that this entire thing is
  proposed, not set in stone.
• Behind-the-scenes discussions of a self-appointed group was not the correct approach.

Non-technical MapServer leadership

You know that scene in Dances With Wolves where the indians are sitting around smoking the pipe in the teepee trying to decide whether or not to kill the white guy and the chief finally makes a decision and tells them what to do? Well, MapServer is nothing like that. MapServer’s assumed leadership is a number of individuals whose voices carry roughly equal weight, with one, Steve Lime, being more equal than the rest. MapServer’s leadership, the MTSC in particular, has almost exclusively only concerned itself with technical issues, and its benevolent dictator is an individual who rarely exercises that power.

In my opinion, the MapServer project needs to formalize leadership and decision-making for non-technical issues in the same way that it did for technical issues with the formation of the MTSC. Our reluctant dictator clearly doesn’t feel comfortable exercising a lot power on issues that are not technical. We might not need a teepee and a peace pipe, but
we clearly need a way to allow people to sit in the tent if they want to. Some may argue that MapServer is a meritocracy
and doesn’t need such a thing, but this episode demonstrates otherwise to me. Meritocratically speaking, most of the MapServer signatories of the Open Letter are viewed as leaders by our contribution and effort in the project. But does that merit give us fiat to proclaim stuff like this by decree?

Questions for the MapServer project

How do we procedurally get into the foundation?

Where we are and where we want to be are two different places right now. UMN and Steve Lime have a lot to say with respect to where MapServer goes, but so do a lot of other people. What are the steps required to actually make it happen?

What are the details of the proposed foundation?

Yes, we want a foundation. We also want to cooperate with AutoDesk and as many other groups as are interested. Now is the time to start putting out all of the nuts and bolts details of what the proposed foundation is to look like.

How do we bring more projects and groups into the tent?

In my mind it is very important to bring more groups in right now. Some have argued that this isn’t really possible given how the announcement happened, but I think it is imperative that we try.

Is it important that those in the MapServer project who care meet in person?

Maybe there should be an open invite to a MapServer project “summit” in Minnesota to go over some of the project’s governance issues. If so, I think it should happen this winter. The path of how we get from where we are to where we want to be (in the foundation) is not clear in my mind.