Measuring the dependency of different variables

• Curiously, one of the effects of the wide readership that the post got was heightened interest in my django based package: django-gitcms.

  This is good. I still plan to put up a couple of websites based on that having spent a few hours two weeks ago making it more general (even as I was switching to Pylons for the other project).

• Some people pointed out that my criticisms were about the apps and not the full architecture. For example, someone said doing a photo app that handles twitter is easy with signals. RTFM.
  It’s actually a good idea. The only question is: where is that app?

  I think it’s fair to evaluate a platform based on the tools that are built around it as much as on the full potential. Django has a bit of a culture of releasing any piece of half-finished code that served its purpose once as an app.

• Let me reformulate a bit of the gist of what I wrote: Part of the django vs. Pylons trade-off as I understood it when I first chose django and still understand it now is that django constrains your website more, but gives you more for free, while Pylons is more build-your-own, but you have a lot more flexibility. When I realised that I was mostly building my own with django while still being constrained, I felt that I had been cheated on.

  Other websites might not like the flexibility or not mind the constraints. Fine.

• We should all unite against the common enemy: Ruby on Rails! I hear. No, PHP. Oh, yeah, PHP.

Follow me on twitter: @luispedrocoelho

Background

I started using Django in June for a side project. A friend told me about it, he wanted to use it for a project we were going to do and it seemed great. Unfortunately, none of those projects went live (for non-technical reasons). A few months later, I am helping another friend with django-based website, and I also redid my personal website in django (albeit a very peculiar use of django) and I have other plans for django websites based on the framework I built for myself.

Two months ago, I decided to rewrite a website I manage (PAPS) in django. It was drupal-based, which had served us well, but it was, by now too old and inflexible. One of the things that disturbed me about it was that I had started tweaking the code. Not much, but enough that I was afraid to upgrade and lose those changes.

Also, I wanted recaptcha on signups, openid logins, twitter integration and a bit of a feeling of modernity. I worked on it a few hours per week, in the evenings, but I didn’t make much progress until I switched to Pylons. By then, I had invested a month into the project, but kept getting stuck and moving forward only at immense effort.

Django

First, the good: its documentation is deservedly regarded as impeccable and an example to us all. Its really easy to set up a very simple website and have it just work fast.

Now, the bad: One of the major reasons why I thought that django was so great was the vibrant community of people writing apps. However, I think that this also exposes django’s basic architectural flaw: the app system is not the right system to break up functionality. This is why there is so much talk of reusable apps: most existing ones are hard to reuse.

I first tried pinax or one of the content management systems. As usual, you get a working website instantly, but then you hit a productivity barrier very fast. Then, I tried a bunch of apps to get the pieces of functionality I wanted and get them to play together. What I wanted to avoid was to fork the code and
adapt it to my project. Then, I would be forced to maintain it and diverging from the main tree.

I tried. I really did. But I could never really get open id to work. I gave up on almost all of the apps I tried too. They would have a structure that didn’t fit into the rest of my application, I would need to tweak the templates (most by reading the code to see what was going on, which names it exported). I would need to fix the urls and url names. I would basically be forking each app to work with my desires instead of re-using it.

Fundamentally, the app model is the wrong abstraction for reuse in web-applications. I don’t have an answer, but I know that the app model does not work. How does, say, recaptcha authentication fit into the app model? What are its models, its views, its templates? It’s better handled as a different sort of reuse. What if I want to have a photo gallery and have the insertion of new photos trigger a tweet? I always end up taking someone’s code and modifying when what I wanted was to take someone’s code and call it.

What finally broke the camels back was when I was working with a django model and thought Sqlalchemy would be so much better. So, I decided to try Pylons.

Later that evening, I had ported almost all my existing work to Pylons. The next day, I had made progress. Also, I finally stopped feeling frustrated.

Pylons

Pylons seems to get a bit less attention. The reasoning seems to be: django is better for simple sites, Pylons for complex sites. Most sites are simple. Therefore, django is more popular.

What I finally figured is that, if you’re writing most of the code yourself or editing other people’s stuff so badly that you might as well start from a couple of snippets, what is django buying you?

I can use sqlalchemy and I can use mako as a templating engine. I never really liked django’s ORM nor its templating engine. I know you can change the template language, but then other apps get very unhappy. Use best of breed for your tools instead of not-invented here is a good principle.

The debugger that comes with Pylons is awesome: if there is an error in your code, you can actually get an HTML+Ajax python shell on the webpage where you can inspect the state of the programme. SQLAlchemy’s error messages are beautiful. Here’s a recent one (from FormAlchemy): No session found. Either bind a session explicitly, or specify relation options manually so FormAlchemy doesn’t try to autoload them.

Here’s a screenshot of the debugger.

Pylons Error

The contrast with django is immense. After months of daily use, I still feel stumped trying to decipher its error messages. Not always, but enough times that its frustrating. Just yesterday, another friend called me up on a Django problem. manage.py was not finding settings.py anymore because django plays around with the paths and the imports in a weird way and there was an impossible to understand relationships to recent changes because it worked in previous versions.

There is nothing wrong with magic in a system (rarely do you hear someone complain that their framework is too magical for working with files, just let me see the disk sectors). But unfinished magic is very frustrating. Unfinished magic most often manifests when the error messages are not magical. So, it just works or it just doesn’t work. As if your operating system spat out a bunch of references to virtual file systems and inodes when you mistyped a file name instead of saying File mixpelled.txt not found. Django has too much unfinished magic: if you make a little typo, then you get an obscure error in a place unrelated to your typo. SQLAlchemy is finished magic: half the time it even suggests a solution to
your problem.

Then there are the little irritations in django: the Sites app is obviously something that solved a problem in django’s original environment, but is a nuisance for the other 99% of cases. The fact that the username cannot be an email: Such a limitation is out of place in a general purpose framework.

I do miss the admin interface.

Summary

Django is good if your website is simple and Pylons is better if it is complex. I still agree with this statement, but I have lowered the bar for simple dramatically to mean “almost static” webpages. Pylons has a slightly higher learning curve and takes a bit of time to get to something that works well, but, if you know django, you can start writing code very very fast because most of the concepts are so similar.

The debugger on Pylons is so nice it makes it worth the switch.

Right now, unless my site was braindead simple, I would never go back to Django.

Follow me on twitter: @luispedrocoelho

What’s wrong with people? The movie is just awful.

I hadn’t expected an excellent film, but it’s just boring.

As someone in my party commented they spent so much money on the visuals that they couldn’t afford creativity, a plot, or characters that weren’t paper thin.

Since Greece does not control its currency (which is controlled by a country [Germany], which is interested in keeping it strong), is it fair to say that it is on a sort of Gold Standard?

If so, what do its current budget troubles say about the ability of a gold standard to push governments towards balanced budgets?

For extra credit: Is California on a Gold Standard?

I read about this book in Less Wrong and immediately added it to my queue (a mental queue at this point).

The book is really good and its flaws are of the I wish he would tell me more type.

Most of the book is a recollection of experiments performed by the author or his associates, where people were asked to perform as if they were dictators of a small country/village/city/factory/… or simply having to set a thermostat in a room (many failed at this task!).

Some of the examples are eerie in their resemblance to real events, like the player who introduced worker’s ownership and later, when the factory wasn’t producing enough, proposed to shoot any worker who wasn’t producing his quota. In fact, this is one of those examples that seems to fit almost too perfectly. Other examples show the intellectual knots that people get themselves into:

In one of our planning games, people were asked to shape a country’s [policies]. [...] One participant found himself threatened on the foreign-policy front while at the same time he needed to cope with vast unemployment at home. The solution he hit on to deal with both these problems was to introduce universal military service. [...] However, he recalled that only a few hours earlier he had announced decisively that the government should do nothing to strengthen the military and should certainly not introduce any forced measures to that end. [...] What did he do? He introduced *voluntary conscription*, commenting as he did so, “Everybody will surely understand the need for this.”

Brilliant. As is the fact that many people seem to start distrusting the experiment when their hare-brained ideas cause problems.

Also interesting is the discussion on how teaching people some introduction into the problem field will make them feel much more confident while still performing poorly, while actual experience in the field helps much more (which should temper the hubris of the educated fool).

Many little examples such as this abound in the book. They are held together by the common themes of human folly so that it is not simply another management book of anecdotes. Also, unlike anecdote books, this one comes with numbers that show that not everyone does poorly and that there are significant differences between how those that do poorly and those that do smartly behave.

On the negative side, there is a tendency to spell out tables of numbers that is sort of annoying. I suppose someone told the author that (non-technical) people are too stupid to understand a table of numbers so the author sometimes seems to spell them out: “30% of users of type 1, fell into category 1, 40% into category 2, …” in a way that is tedious. Also, many of the references are just given in German, which makes them a bit harder to look for.

Predictioneer’s Game

Rating: 3/5

It’s a nice introduction to de Mesquita’s modeling approach to predicting the output of protracted human negotiations (be it negotiating with North Korea on nuclear weapons or in Congress about health-care reform or in the boardroom to prevent your rival from getting the CEO spot). de Mesquita claims to have a very high (90% is often mentioned) success rate.

It’s important to note that de Mesquita’s approach is immune to the Taleb critique of statistical model. Nassim Taleb’s argument works against phenomenological approaches which relate multiple variables statistically (such as if in the past, inflation has been correlated with low unemployment, then in the future it will be so or some similar rule, see also Goodhart’s Law). De Mesquita’s approach is mechanistics, it is based on the mechanics of the situation. If human nature were to change, then the model will be invalidated, but there is more permanence to the pursuit of self-interest, which is the basis of de Mesquita’s models, than to the statistical flukes that form the basis of the models that Taleb tears down. I have been looking for a good critique of de Mesquita, but am actually yet to find one which understands that saying Taleb doesn’t cut it. (For the record, I really like Nassim Taleb’s ideas. They just seem inapplicable to de Mesquita.) An attack on de Mesquita would have to go to his weak points: can his models really capture the essence of the problem? where are his high accuracy statements coming from? how often is the model completely side-tracked by an outside event (one of the major players dying or, in an example from the book, getting arrested on corruption charges)? This might be one of those cases where the weakness of the existing critique has taken away some of the skepticism I had after first reading the book.

It’s pop-science, so it doesn’t bring in the models, or the math, or the details, or the critique. It does make it want to go back and read the original papers though.

A couple of little details from the book: on and around page 40, you get a very informative discussion of why Herman van Rompuy is President of the European Council (the discussion is phrased as if it was about getting a guy named Curly nominated CEO). The interpretation of the Catholic prohibition of charging interest as a way to keep economic growth down was new to me (even if am I taking it as an interesting hypothesis until I have independent confirmation that it is a reading supported by the facts).

On the down-side, the prose is sometimes patronising. He doesn’t really need to repeat his arguments 3 times: once as if it was a intro undergraduate class, one as if it was a high-school class, and a third as if he was trying to explain it to his 4-year old daughter. The graphs out of Excel could have used some work too (even Excel does better graphs nowadays).

I have been thinking for months that Barack Obama will be a debt President (particularly if he gets a second term). By a debt president, I mean a president whose term is dominated by discussion of how to deal with the debt and interest payments. Not Iraq, not Afghanistan, not banks, not green energy, not health care, but debt (i.e., how to pay for Iraq, Afghanistan, the bailing out of every bank and its cousin, health care—green energy will never be a big item).

Maybe the fact that the NY Times put the topic on the front page is a sign that this discussion is moving to the fore. I predict it will becoming dominant in a couple of years.

In Norway, tax returns have been made public.

Alex Tabarrok asks:

Perhaps most interesting–does conspicuous consumption fall and efficiency increase in a society in which income is conspicuous?

An interesting variation on this is: Can we find evidence for people declaring more than what they truly earn so that the neighbours comment on it (reverse tax fraud).

He drives a really old car, but, in fact, he makes a ton on family income alone. I guess he just doesn’t want us to know. You know, old money doesn’t show off.

Probably this effect would be higher in the lower tax-brackets!

(I don’t know how to test for this based on that data, however.)

You know an idea is a crackpot idea when it tries to wrap itself in the veneer of respectability by borrowing the name of Economics.

On econtalk the person who headed the gmail team reveals that it was developed by all of a dozen people.