Hobu, Inc.

You know, like nunchuck skills, bowhunting skills, computer hacking skills… Girls only want boyfriends who have great skills.

Napoleon Dynamite has got it right… you gotta
have skills. But how do you get them? The trick to getting good at open source GIS hacking is to do it a lot (well, that’s
the trick to getting good at anything). There are many resources available out there on the net to help you
attain those skills.

Slashdot
The nerds congregate at Slashdot. They have congregated there for years. This is the place where you
will find out the latest happenings throughout the day, serendipitous stuff that you’ve never thought or
heard of before, and the full, if slanted, gamut of opinion about political, social, and technology issues
that affect the lives of geeks.

#mapserver on IRC
You may joke about mainlining the internet from time to time, but if you really want a whiff of
high-test ether, IRC is the best way to do it. The folks on the channel #mapserver on IRC chat
about everything. We work through technical issues on MapServer,
chat about happenings, pick each other’s brains, and learn from of some of the best GIS developers (open source
or otherwise) around.

The GIS Monitor is a weekly webzine that was a dependable source
of industry gossip, happenings, and technology. The verdict is still out on whether or not Adena’s
successor will be able to maintain the high level of quality that she kept. In either case, Adena still keeps
a weblog at Directions Magazine.

More later…

The decision of whether or not to respond to Dimitri Rotow’s (scroll down to Letters to the Editor) screed against open source GIS software
(among a bunch of other things) in this week’s edition of GIS Monitor is a difficult one. By
responding, do I validate the FUD (Fear, Uncertainty, and Doubt)? How do you respond to someone who
is clearly off his rocker with a severe case of foot-in-mouth disease?

Rather than go point by point through his nonsense, I think instead that I’ll go over some of the
ridiculous ones. First, a quote:

There are no open source GIS products out there of any merit.

This statement has a couple of problems. I don’t even know how to respond to the comment about
no open source GIS products having merit. Actually, what does he mean by merit anyway?

Saleable merit?
That’s not true. There are entire companies out there built around sales and support of open source
GIS products.

Technical merit?
Many of technical advances in GIS and computing come from academia and governments who
can support their development (it’s too hard to cost recover these kinds of things). Then they are picked
up by opportunistic companies (think of SAS or ESRI, which have made gonzo money doing this), folded
into software, and sold as “features.” ESRI, SAS, and yes, even Manifold have done this (I highly doubt
that their much touted support of PNG in their products is their own implementation).

Useablilty merit?
He may have a small point here, but his reasoning is all wrong. The reason why some
open source software is not as polished from a useability aspect is because the users of the software
haven’t demanded it yet. It is not because open source developers are poor developers or poorly organized.
On the other hand, if you take an open source project like Firefox, which has
millions of users, useability and polish start to become very prominent and important. Like a wise man
once said, “The last 10% takes 90% of the time.”

So, they create products for operating system environments that (statistically) no one uses. If they wanted lots of people to actually use their products, they’d create them for Windows.

I don’t know where Dmitri gets this idea. Almost all of the software on freegis.org
can be compiled and made run on Windows just as easily as any other operating system. As for the comment about
operating systems that no one (statistically) uses, let’s just say that 5% of x million is statistically significant
(disclosure: I work at the Statistics department at Iowa State University). Also, I’d like to note, that it’s the
operating system who’s marketshare is growing that is important, not the one with a stagnant/saturated market.

It’s not Internet that’s broadened usage of GIS, it is the ubiquitous availability of PC clones running Windows that anyone can afford coupled with the emergence of full-power, best-in-the-world GIS applications like Manifold that sell for under $250.

Dmitri’s reading of the tea leaves (or stems and seeds, rather) is dead wrong here. His complaint about “true” GIS
needing to be done on the desktop is wrong as well. MapQuest (and Google Maps) are the largest GIS applications
out there in terms of numbers of users. ESRI now touts ArcServer as their biggest growing product, and everyone
and their brother in the GIS software world is/has been jumping into the internet mapping space. The market
is clearly going this direction, whether or not folks like Dmitri want to believe it.

The great majority of GIS users are doing GIS in Microsoft environments on desktop boxes and they are completely ignoring both OGC and GML.

That may be true, but the governments and agencies that you yourself say provide all the data that you
need to use in GIS are not ignoring OGC developments. These organizations have larger goals, and getting
out of the data lock-in trap is clearly one of them. You, as a software provider that is completely dependent
on using data that is provided by these organizations, will have to live with it. And you already do. Your
latest service release of your product has OGC WMS listed as the first item.

Responding to trolls is tiring… more later. Btw, Manifold’s own webserver uses that “obsolete” operating system called Linux. See the Netcraft survey here.

Thanks to a patch from Pirmin Kalberer at Sourcepole.com, experimental SDE raster support has been added to MapServer for the 4.4 release. SDE raster support will not be enabled by default when SDE support is requested. To enable it, you will have to define USE_SDERASTER along with the normal USE_SDE compiler symbols.

While much more information can be found in the wiki, here is a sample LAYER block that outlines what will need to be done to define a SDE raster layer:

  LAYER
    NAME         rastertest
    TYPE         RASTER
    CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password"
    CONNECTIONTYPE sde
    DATA "RASTERTEST,FOOTPRINT,SDE.DEFAULT"
    STATUS       ON
    DEBUG ON
    PROCESSING "BANDS=1,2,3,4"
    PROCESSING "RASTERCOLUMN=RASTER"

  END

As you can see, the support is a little hackish at this point, but I expect that things will settle down as we move through the 4.4 release into the 4.5/4.6 series. There is also some talk of moving all of the SDE support (rasters and vectors) into the more general GDAL/OGR libraries, but nothing has been done on that front yet.

Why is this significant? If you are in an (almost) all-ESRI shop, you now have no excuse to not give MapServer a try. Data formats are no longer an impediment to using MapServer for your web rendering needs. Why struggle with servlets, AXL files, restarting your server to make simple changes, and limited output formats with ArcIMS? With MapServer 4.4, you will be able to drop a Windows CGI executable (or similar unix-like binary) into your Apache cgi-bin directory and quickly give it a try. I think you’ll find you like it

Using OSR, Proj.4,
py-Projection, and
Twisted Python, I made a web service that allows
you to project a coordinate in any projection system to any other projection system. See
the code below for examples on how to use it.

import xmlrpclib
p = xmlrpclib.ServerProxy('http://maps.hobu.net:7080/RPC2')

#Define the projection system in Proj.4
in_proj4 = '+proj=utm +zone=15 +ellps=GRS80 +datum=NAD83 +units=m +no_defs '

#Define the input projection system in EPSG
in_epsg = '26915'

#Define the input projection in ESRI WKT
in_esri = """PROJCS["NAD_1983_UTM_Zone_15N",
GEOGCS["GCS_North_American_1983",
DATUM["D_North_American_1983",
SPHEROID["GRS_1980",6378137,298.257222101]],
PRIMEM["Greenwich",0],UNIT["Degree",0.0174532925199433]],
PROJECTION["Transverse_Mercator"],
PARAMETER["False_Easting",500000],
PARAMETER["False_Northing",0],
PARAMETER["Central_Meridian",-93],
PARAMETER["Scale_Factor",0.9996],
PARAMETER["Latitude_Of_Origin",0],
UNIT["Meter",1]] “”"

#Define the input projection in OGC WKT
in_wkt = “”"PROJCS["NAD_1983_UTM_Zone_15N",
GEOGCS["GCS_North_American_1983",
DATUM["North_American_Datum_1983",
SPHEROID["GRS_1980",6378137,298.257222101]],
PRIMEM["Greenwich",0],
UNIT["Degree",0.0174532925199433]],
PROJECTION["Transverse_Mercator"],
PARAMETER["False_Easting",500000],
PARAMETER["False_Northing",0],
PARAMETER["Central_Meridian",-93],
PARAMETER["Scale_Factor",0.9996],
PARAMETER["Latitude_Of_Origin",0],
UNIT["Meter",1]]”"”

#Define the output projection in Proj.4
out_proj4 = “+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs ”

#Define the output projection in EPSG
out_epsg = ‘4326′

#Define the output projection in ESRI WKT
out_esri = “”"GEOGCS["GCS_North_American_1983",
DATUM["D_North_American_1983",
SPHEROID["GRS_1980",6378137,298.257222101]],
PRIMEM["Greenwich",0],
UNIT["Degree",0.0174532925199433]] “”"

#Define the output projection in OGC WKT
out_wkt = “”"GEOGCS["WGS 84",DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.01745329251994328,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4326"]]”"”

esri_albers = “”"PROJCS["USA_Contiguous_Albers_Equal_Area_Conic_USGS_version",
GEOGCS["GCS_North_American_1983",
DATUM["D_North_American_1983",
SPHEROID["GRS_1980",6378137.0,298.257222101]],
PRIMEM["Greenwich",0.0],
UNIT["Degree",0.0174532925199433]],
PROJECTION["Albers"],
PARAMETER["False_Easting",0.0],
PARAMETER["False_Northing",0.0],
PARAMETER["Central_Meridian",-96.0],
PARAMETER["Standard_Parallel_1",29.5],
PARAMETER["Standard_Parallel_2",45.5],
PARAMETER["Latitude_Of_Origin",23.0],
UNIT["Meter",1.0]]”"”

#Print the valid projection type strings
print ‘Valid projection type strings: ‘,p.getprojectiontypes()

#Project a point in UTM NAD 83 Zone 15 into Geographic using the
#various methods.  This point is located in a corn field.
x = 437142.35
y = 4658582.96
points = [[x, y]]
#Project using Proj.4
print ‘Proj.4 UTM 15 to Proj.4 DD –’,
print p.project(points, in_proj4, out_proj4, ‘proj4′, ‘proj4′)

print ‘Proj.4 UTM 15 to EPSG DD –’,
print p.project(points, in_proj4, out_epsg, ‘proj4′, ‘epsg’)

print ‘Proj.4 UTM 15 to ESRI DD –’,
print p.project(points, in_proj4, out_esri, ‘proj4′, ‘esriwkt’)

print ‘Proj.4 UTM 15 to OGC WKT –’,
print p.project(points, in_proj4, out_wkt, ‘proj4′, ‘wkt’)

#Project using EPSG
print ‘EPSG UTM 15 to Proj.4 DD –’,
print p.project(points, in_epsg, out_proj4, ‘epsg’, ‘proj4′)

print ‘EPSG UTM 15 to EPSG DD –’,
print p.project(points, in_epsg, out_epsg, ‘epsg’, ‘epsg’)

print ‘EPSG UTM 15 to ESRI DD –’,
print p.project(points, in_epsg, out_esri, ‘epsg’, ‘esriwkt’)

print ‘EPSG UTM 15 to OGC WKT DD –’,
print p.project(points, in_epsg, out_wkt, ‘epsg’, ‘wkt’)

#Project using ESRI
print ‘ESRI UTM 15 to Proj.4 DD –’,
print p.project(points, in_esri, out_proj4, ‘esriwkt’, ‘proj4′)

print ‘ESRI UTM 15 to EPSG DD –’,
print p.project(points, in_esri, out_epsg, ‘esriwkt’, ‘epsg’)

print ‘ESRI UTM 15 to ESRI DD –’,
print p.project(points, in_esri, out_esri, ‘esriwkt’, ‘esriwkt’)

print ‘ESRI UTM 15 to OGC WKT DD –’,
print p.project(points, in_esri, out_wkt, ‘esriwkt’, ‘wkt’)

#project using WKT
print ‘OGC WKT UTM 15 to Proj.4 DD –’,
print p.project(points, in_wkt, out_proj4, ‘wkt’, ‘proj4′)

print ‘OGC WKT UTM 15 to EPSG DD –’,
print p.project(points, in_wkt, out_epsg, ‘wkt’, ‘epsg’)

print ‘OGC WKT UTM 15 to ESRI DD –’,
print p.project(points, in_wkt, out_esri, ‘wkt’, ‘esriwkt’)

print ‘OGC WKT UTM 15 to OGC WKT DD –’,
print p.project(points, in_wkt, out_wkt, ‘wkt’, ‘wkt’)

#Project to ESRI Albers
print ‘ESRI UTM 15 to ESRI USGS Albers –’,
print p.project(points, in_esri, esri_albers, ‘esriwkt’, ‘esriwkt’)

print ‘Project array of points’
points = points*10
print p.project(points, in_esri, esri_albers, ‘esriwkt’, ‘esriwkt’)

Output:

Valid projection type strings:  [['wkt', 'epsg', 'esriwkt', 'proj4']]
Proj.4 UTM 15 to Proj.4 DD — [[-93.759900978958413, 42.076802903741402]]
Proj.4 UTM 15 to EPSG DD — [[-93.759900978958413, 42.076802903741402]]
Proj.4 UTM 15 to ESRI DD — [[-93.759900978958413, 42.076802903741402]]
Proj.4 UTM 15 to OGC WKT — [[-93.759900978958413, 42.076802903741402]]
EPSG UTM 15 to Proj.4 DD — [[-93.759900978958413, 42.076802903741402]]
EPSG UTM 15 to EPSG DD — [[-93.759900978958413, 42.076802903741402]]
EPSG UTM 15 to ESRI DD — [[-93.759900978958413, 42.076802903741402]]
EPSG UTM 15 to OGC WKT DD — [[-93.759900978958413, 42.076802903741402]]
ESRI UTM 15 to Proj.4 DD — [[-93.759900978958413, 42.076802903741402]]
ESRI UTM 15 to EPSG DD — [[-93.759900978958413, 42.076802903741402]]
ESRI UTM 15 to ESRI DD — [[-93.759900978958413, 42.076802903741402]]
ESRI UTM 15 to OGC WKT DD — [[-93.759900978958413, 42.076802903741402]]
OGC WKT UTM 15 to Proj.4 DD — [[-93.759900978958413, 42.076802903741402]]
OGC WKT UTM 15 to EPSG DD — [[-93.759900978958413, 42.076802903741402]]
OGC WKT UTM 15 to ESRI DD — [[-93.759900978958413, 42.076802903741402]]
OGC WKT UTM 15 to OGC WKT DD — [[-93.759900978958413, 42.076802903741402]]
ESRI UTM 15 to ESRI USGS Albers — [[184064.98645664263, 2121672.8254685458]]
Project array of points
[[184064.98645664263, 2121672.8254685458], [184064.98645664263, 2121672.82546854
58], [184064.98645664263, 2121672.8254685458], [184064.98645664263, 2121672.8254
685458], [184064.98645664263, 2121672.8254685458], [184064.98645664263, 2121672.
8254685458], [184064.98645664263, 2121672.8254685458], [184064.98645664263, 2121
672.8254685458], [184064.98645664263, 2121672.8254685458], [184064.98645664263,
2121672.8254685458]]

I spent plenty more time in the islands today, and I think the ESRI developers now are starting to look at the floor when they see me so they don’t have to talk to me anymore ;). I asked a question about whether or not ArcIMS supports query against specific versions in ArcSDE at version 9, but I didn’t get an answer. I might get an answer later, but for now, I can safely say that MapServer definitly does something that ArcIMS does with respect to ArcSDE.

I also found out today that ESRI will be end-of-lifing ArcView 3.x right after the conference. The specific term that the product specialist used was “extended support”, which means no updates and no certification with new operating systems (say, Windows XP SP2, for instance). This sucks, but it was coming. Too soon in my opinion, but there isn’t much we as Avenue users can do except lobby for them to open source ArcView 3.x so we can do what we need to with it.

I also made a contact with Wacom about getting a tablet/monitor for doing heads-up digitizing. This is something that could be very useful and much more efficient than doing the digitizing on-screen. I can’t wait to play with it.

I spent most of today in the developer islands talking to ESRI developers about many topics. I brought up the open sourcing of ArcView 3.x (it raised some eyebrows, but didn’t gain too much traction :), I talked to SDE developers about upcoming changes and additions to the SDE C API, and I chatted quite a bit with the Geoprocessing developers.

I finally finshed the development of my Powerpoint for my presentation on Thursday after seeing some of the Geoprocessing sessions and getting a chance to guage the level of expertise of the audience and who I should target it at. It’s ready to go for the most part, just got to get to Thursday now.

Patrick and I finished the night by going to Sushi Bar Nippon in the Gaslamp District. By far the best food of the trip, as we ate Canadian role, much tuna, and plenty of unagi (freshwater eel). We might try and get back there, I think…

After getting in just in time to see Jack Dangermond give the opening session, I sat down to a long morning of underwhelming product demos and much banter of the likes of “we are changing the world”, where everyone gives them a nice pat on the back and feels all nice and fuzzy.

In the afternoon, I was impressed by the upcoming addition in ArcSDE 9.1 of history tables in addition to the adds and deletes tables. This will give us the ability to have time-sensitive versions, much like I have been trying to implement (unsuccessfully, I might add) at work. Whether or not the actual in-the-field implementation actually works is yet to be seen, but the feauture looks promising.

Finally, them map gallery brought some refreshments and hundreds of maps. I joked that ESRI should develop a map beautification button that turns the crappy, poor colors, comic sans fonts, no legends, map titles or descriptive text maps into winners. It’s also funny that some of these same maps thought it necessary to enter themselves in the Best Cartography contest that goes on in the map gallery. The software might get better in the next years, but the fact is that most people never get beyond the defaults of the software. This is as clear as ever when it comes to the map gallery, where I can tell by the map which software product was used to draw it with.

This will be my third ESRI User Conference in the past six years, with my first conference being as a student assistent and my second bringing first place in the software integration category for avTerra. This year I will be presenting A Guide to the Python Universe for ESRI Users, but the presentation isn’t quite done yet

I came to San Diego a bit early to catch the last game of the Padres home stand with the Pirates. Petco park is a great baseball stadium, with nice views, interesting areas, and comfortable, old-style metal armrest seating. The Padres weren’t able to pull it out, giving up another run in the top of the ninth inning to lose 4-2. It was still nice to see the game and the park, though.

I was also able to grab some sushi with Patrick at Sushi Ito. I had my first chance to try uni, or sea urchin roe, which Patrick described as tasting like the tidal basin smells (to which I concurred). I don’t know that I would have it again, but it was an interesting experience nonetheless.

I have built a Windows installer for Python of the Thuban Project’s Proj.4 wrapper. It is available for download here

The article provides some interesting insight into Microsoft’s reaction to people rapidly moving away from their development platforms.

And here’s the clincher: I noticed (and confirmed this with a recruiter friend) that Windows API programmers here in New York City who know C++ and COM programming earn about $130,000 a year, while typical Web programmers using managed code languages (Java, PHP, Perl, even ASP.NET) earn about $80,000 a year. That’s a huge difference, and when I talked to some friends from Microsoft Consulting Services about this they admitted that Microsoft had lost a whole generation of developers. The reason it takes $130,000 to hire someone with COM experience is because nobody bothered learning COM programming in the last eight years or so, so you have to find somebody really senior, usually they’re already in management, and convince them to take a job as a grunt programmer, dealing with (God help me) marshalling and monikers and apartment threading and aggregates and tearoffs and a million other things that, basically, only Don Box ever understood, and even Don Box can’t bear to look at them any more.

Much as I hate to say it, a huge chunk of developers have long since moved to the web and refuse to move back.