An outsider overview of #sotm13

sotm_2013

Last weekend I was in Birmingham for the StateOfTheMap, to learn how we could be more involved in OSM in a number of projects we have down the line.

Although I’m a casual mapper and I did know some things about OSM and its core technologies, this was my first in-depth immersion into that world. Note also that during the conference I followed a specific path into the multiple choices we had so, do not expect me to write a complete summary of the conference neither a hands-on guide on “How Mapnick stylesheets were ported to CartoCSS” (enjoyed a lot that talk by the way!). I’ll focus on the community side of the conference.

Other that that, OSM is strugging with growth. For me, there is a subtle line which connects Alyssa Wright’s “Changing the Ratio of OSM communities“, Richard FairHurt’s “You are not the crowd“, the tools built by the Mapbox guys for the next generation of contributors, the world-class documentation the HOT team is creating and the multiple talks on gamification during the conference: they’re all talking about how should OSM growth. Being it the social side of it (how could we engage new contributors?) or the technical one (what tools do we need for people to find easy work with OSM?). That is a challenge, but a challenge that most of the communities I know would like to have.

As an outsider, I got the impression that OSM is like a teenager that still has to define itself in some aspects. And my belief is that it it manages to do it in a smoothly fashion, it will have an even brighter future ahead.

A new development era

Tarek Ziadé has posted a few months ago an interesting essay on his blog: A new development era. Summing up: web technologies (HTML5, JS) are gaining importance to build complex apps in the client (whatever it is: desktop, browser, phone, tablet) and the server side is becoming a proxy of lightweight services to interact with.

2000_2012 2015_

The post resonated with me due to the fact that my work during last months was to build a rich client app in Javascript with lightweight JSON services built in python. But, as far as I’ve seen it, this tendency is more spread than I thought: it’s not only happening within early adopters, but also within big players in the desktop realm (Windows, GNOME) and the old-school java server applications. Maybe is a new swing of the pendulum, or just that the promise of cross-platform apps that just work in multiple environments is appealing. What is certain, is that the next million apps seems to go towards that tendency.

Managers not MBAs

«If people want to be managers, there’s a better route to it: get into an industry, know it, prove yourself, get promoted into a managerial position and then, go to a program that uses managerial experience explicitly not other people’s cases, but your own experience.»

— Henry Mintzberg, on MBA education. In a debate with Ricardo Semler.

(Geo) Database evolution while developing

During last year, I followed with interest the different approaches on how to evolve the design of a database being discussed within the postgresql community. Following is my take on that one: how this year I developed a project with an intense evolving DB design using an agile approach.

The context

My requisites for this project were twofold:

  • An evolving DB design: at the beginning of the project I didn’t know how the DB design was to going to be. I had set to use some advanced techniques for data modeling which never had used in production (dynamic segmentation and linear referencing with PostgreSQL/PostGIS) and needed an approach which supported my evolving understanding of the domain.
  • Intense collaboration with analists: the project needed some intense work on data-processing to polish and create the data for the application. I knew this was to be an iterative process where both developers and analists would collaborate together to define and clarify the model we needed.

My approach

So, in the process of improving and automating my delivery pipeline, I set some rules for the project:

  • DB management through SQL and control versioning: the database was created from DDL scripts and data was stored as CSV (if alphanumeric) or SQL (generated from Shapefiles to store geographical information).
  • Application and database evolve together: so their code should too, which in practice means I put the app and DB directories/projects under the same git repo.
  • Test driven development: I needed to break the problem in small chunks I could deal with, while my understanding of the domain improved. Besides, when refactoring the DB (schemas, triggers, functions, etc) -which happened frequently- I needed to know all the pieces were working OK. I decided to use pgTap for that.

And how it turned out?

  • The pipeline worked smoothly: both the analists and developers were working in their confort zone with the proper tools; desktop GIS applications the formers, command-line and SQL the laters.
  • git provides an excelent mechanism for versioning text, so I had powerful tools at hand for versioning SQL structure and data (diff, cherry-pick, interative rebases, etc). Besides, see where the data was varying (name and type of fields, its values, etc) allowed us to early discovered some bugs and problems.
  • Database and application evolving to the same pace. By tagging the versions we can build in seconds the binaries needed for any version of the application with the proper DB.
  • Tests at DB level are a life-saver. pgTap allowed me to refactor the database whith no risk and a lot of confidence on what I was doing. I had all kind of tests: check if a trigger is launch if an UPDATE happens, a function is working, data integrity and model validation after the initial restore, etc.
  • Same process for deplying to developing, staging and production environments, which resulted in fewer errors and no panic-moments.
  • Having the data in the repo and regenerating BD from scracth was very comfy and quick (less than a minute in my laptop the whole DB: 100Mb of raw SQL) and similar numbers when deploying to stage through the wire. In a daily bases I only had to regenerate specific schemas of the DB, so waitings was an order of seconds.

Coda

We should consider the database as other deliverable to our clients and set the same quality, standards and methodology to develop it. In that sense, agile philosophy and practices match very well with the DB evolution.

At the tools level, I was reluctant to introduce new tools and steps I didn’t know very well in such a tight schedule, so I decided to stick to the basic and spartan (git repo, shell scripts, pgTap and SQL), then iterate and grow a solution for our specific case. Although I missed some refactoring tools, it turned out to be a good approach and now I´m in good position to know the tradeoffs of the process, which in next projects will help me to choose a specialized tool, if necessary.

Designing for growth

«Code should grow by addition rather than mutation.»

The best example of that axiom I ever found is the one in Beck’s Implementation patterns. What goes next is almost an exact reproduction of the book. After reading this post, if you liked, I’d strongly recommend you to buy a copy.

Imagine a graphic editor where one of the main metaphors is the Figure. It has methods like display(). So, if you want to support several figures in the editor, you can start by writing just a simple conditional:

public void display() {
    switch (getType()) {
        case RECTANGLE :
            //...
            break;
        case OVAL :
            //...
            break;
        case TEXT :
            //...
            break;
        default :
            break;
        }
}

This naïve example, shows early its cons: if you want to add a new kind of figure, you will need to modify the GraphicEditor class adding a new case statement in every switch you have written along the class. Not only it is tedious and error-prone, but it also requires modify a working functionality, putting it at risk. Besides, you should coordinate the changes with other developers using that class. At this point, likely you have thought on better ways to manage change: make the code growth by addition rather than mutation. How is that? Well, two of the tools you have available are inheritance (using classes) and delegation (using interfaces):

If you should choose one of another is a matter of context. It depends. For example, if you would want to rewrite the display method using delegation you would write:

public void display() {
    getType().display();
}

while let the rectangle and other figures to implement the logic of displaying themselves.

This change will allow us to add new kind of figures dinamically. We will only need to implement the Figure interface and use the mechanisms of GraphicEditor to select our new figure. No needs to change the existent code neither coordination costs or try to understand the whole picture before making any change. Just need to focus on implementing our tool. That’s the power of designing for growth: managing complexity.

Big transitions in small steps

Software G Forces: the effects of acceleration was one of my favorite talks this year, which together with the book Continuous delivery bring to life again the big theme of agile movement: how to better build software.

Today I ran into a video by Kent Beck, where he talks about the strategies to take into account when evolving big data structures, growing code or carrying architectural changes out: big transitions in small steps. This is complementary to the first in the sense that Beck drills down here and talks about specific practices on how to accelerate your development and deployment processes. Highly recommended:

 

Valve handbook on failure

«Screwing up is a great way to find out that your assumptions were wrong or that your model of the world was a little bit off. As long as you update your model and move forward with a better picture, you’re doing it right. […] There are still some bad ways to fail. Repeating the same mistake over and over is one. Not listening to customers or peers before or after a failure is another. Never ignore the evidence; particularly when it says you’re wrong.»

— Valve Handbook for new employees

A tour through Valve, a world-class video-game and distribution company with a flat structure.