PyDev, Python and system default Unicode encoding problem

Posted on January 24, 2010  by Mikko Ohtamaa
Filed Under aptana studio, eclipse, plone, pydev, technology

Python 2 has a thing called “default encoding” to automagically encode Unicode strings when they are presented as byte strings. This is evil and has been discussed various times before.

What could be even more evil? Something in your development environment messes this setting set for you, without telling you that. This way you never encounter Unicode problems on your development computer and when you roll out your seemingly working code to production, the world goes haywire.

Evil. Evil. Evil. Thousands of curses and overworking hours to fix the problems.

I encountered this problem. And this is the code I used to track the problem down in site.py:

# Trap the bastard messing with the default encoding
# using a monkey patch
old_set_default_encoding = sys.setdefaultencoding

def aargh(x):
    import pdb ; pdb.set_trace()

sys.setdefaultencoding = aargh

--Return--
> /home/moo/py24/lib/python2.4/site.py(485)aargh()->None
-> import pdb ; pdb.set_trace()
(Pdb) bt
/home/moo/py24/lib/python2.4/site.py(613)?()
-> main()
/home/moo/py24/lib/python2.4/site.py(604)main()
-> execsitecustomize()
/home/moo/py24/lib/python2.4/site.py(514)execsitecustomize()
-> import sitecustomize
/home/moo/Desktop/Aptana Studio 2.0/plugins/org.python.pydev_1.5.3.1260479439/PySrc/pydev_sitecustomize/sitecustomize.py(99)?()
-> sys.setdefaultencoding(encoding) #@UndefinedVariable (it's deleted after the site.py is executed -- so, it's undefined for code-analysis)
> /home/moo/py24/lib/python2.4/site.py(485)aargh()->None
-> import pdb ; pdb.set_trace()

--Return--> /home/moo/py24/lib/python2.4/site.py(485)aargh()->None-> import pdb ; pdb.set_trace()(Pdb) bt  /home/moo/py24/lib/python2.4/site.py(613)?()-> main()  /home/moo/py24/lib/python2.4/site.py(604)main()-> execsitecustomize()  /home/moo/py24/lib/python2.4/site.py(514)execsitecustomize()-> import sitecustomize  /home/moo/Desktop/Aptana Studio 2.0/plugins/org.python.pydev_1.5.3.1260479439/PySrc/pydev_sitecustomize/sitecustomize.py(99)?()-> sys.setdefaultencoding(encoding) #@UndefinedVariable (it's deleted after the site.py is executed -- so, it's undefined for code-analysis)> /home/moo/py24/lib/python2.4/site.py(485)aargh()->None-> import pdb ; pdb.set_trace()

Looks like the culprint was PyDev (Eclipse Python plug-in).  The interfering source code is here. Looks like the reason was to co-operate with Eclipse console. However it has been done incorrectly. Instead of setting the console encoding, the encoding is set to whole Python run-time environment, messing up the target run-time where the development is being done.

There is a possible fix for this problem. In Eclipse Run… dialog settings you can choose Console Encoding on Common tab. There is a possible value US-ASCII. I am not sure what Python 2 thinks “US-ASCII” encoding name, since the default is “ascii”.

Creating a drag’n'drop basket with jQueryUI

Posted on January 5, 2010  by Mikko Ohtamaa
Filed Under jquery, jqueryui, technology
Tags: basket, cdn, google.code, shopping cart, travel planner

I have created an example how to create a “basked” with jQuery and jQueryUI with the following features

• The user can be pick items from the predefined set
• Items are dragged and dropped to the basket
• The basket value reflects a hidden input

This kind of user interface pattern is suitable for

• Shopping carts
• Travel planners

The example in fact bears the name “travel planner” but it is not tied to travel anyhow.

Note: this is just a screenshot - please see live example

The example code uses

• google.code() content delivery network to load jQuery and jQueryUI
• jQueryUI draggable and droppable features
• Well planned visual cues for the drag and drop operations: cursor changes, CSS hover classes
• <form> which <input> value is updated according to the basket content – all selected item ids form a comma separated list

The example code is well-documented with links to the further documentation.

• Live example
• View source code

Mobile browser wars: Nokia microB vs. Firefox Fennec

Posted on January 2, 2010  by Mikko Ohtamaa
Filed Under Business, browser, fennec, maemo, mobile, mozilla, n900, nokia
Tags: browser, comparison, fennec, microb, mozilla, n900, nokia, release, speed, usability, user interface, xul

Begun, this mobile browser war, has. When mobile internet is growing 8x faster than desktop internet everyone wants to have a share of it. In the core of this fight is the mobile browser – the doorway to the mobile internet.

Usually phone comes with a browser from the phone manufacturer: Safari ships with iPhone, Android ships with WebKit based browser and Maemo comes with Nokia microB. Besides the default browser, open platforms have seen third party browsers created for them: Opera Mini has 30 million users and several browsers have been created for Symbian platforms. (Note that iPhone is not really open platform regarding this as Apple developer terms specifically forbid creating alternative  browser engines for their Safari – all iPhone “browsers” are just the same Safari with new toppings).

Now Mozilla foundation is releasing Firefox Fennec (RC1 version is available  for Nokia N900), touted as the most innovative mobile browser this far. New user interface ideas, desktop syncronization and vibrant add-on community are something yet to be seen for mobile browsers. Mozilla did an amazing thing with Firefox when it actually managed to push Internet experience forward and compete against Microsoft’s bundled Internet Explorer with sheer quality. Can Mozilla repeat the same thing it did for desktop browsing for mobile browsing too?

Is Fennec good? I installed the release candidate and conducted some tests by visiting on popular sites. It is especially fruitful to compare Fennec against Nokia’s own microB browser as they both are based on the same Gecko rendering engine beneath the hood.

The differences of the browsers are, actually surprisingly, not limited to branding and user interface shell. Fennec is portable browser – Mozilla hopes to run Fennec on other mobile platforms beside Maemo in the future. Fennec user interface is based on Mozilla’s XUL library and you can actually run Fennec on your desktop computer too. Nokia’s interest, on the other hand, is have an optimized browser for their own mobile phones: microB user interface is using native Maemo user interface components.

Below are some aspects of the browsers compared against each other.

Start up time

• microB: instant
• Fennec: About ten seconds (warm start-up is little bit faster, but it is still slooooow….)

This is a pain for Fennec. Loading all that XUL Javascript  needed to run Fennec is just too much. You really don’t want to start Fennec for a quick browsing session, unless you have the patience of a cow. I am not sure whether N900 keeps microB loaded on the background all the time or what’s causing the difference.

User interface

This is really where Fennec shines. Nokia enjoys some reputation of being a boring engineer house with little innovation left to stir. After learning the trick of left and right sweep, which is cleverly demostrated on the start page, Fennec user interface instantly feels intuitive. microB, on the other hand, uses somehow clumsy “bottom right corner full-screen button” to access buttons and left-right sweep is not very well thought. For example, switching a tab/browser window takes three “clicks” on microB (show menu – switch application – choose next browser window) when Fennec does it with one sweep and click. Also, backward navigation is much more intuitive on Fennec and takes too many gestures on microB.

Both browsers have search integrated to the navigation bar. Fennec start screen is more clever, showing the history and shortcuts, while microB shows only the bookmarks.  Fennec navigation bar also is a combination of title and navigation bar, saving the precious screen estate on small physical form factor. Fennec zooms to text fields automatically when you start to input text into them and also have soft “tab keys” to navigate to next and previous input field.

Page reading and speed

On sites with above average layout complexity, Fennec is unbearable slow compared to microB, up to the point the browser is next to unusable in its current incarnation. As they both use the same rendering engine, I have hard time to understand how microB manages even the heaviest dynamic pages (Facebook profile page) when Fennec becomes unusable even on a moderate complex page (slashdot.org).

The thing with Fennec is that for some of the the time it does not register your interaction and does not have any indicator showing if it is responding – it has grinded to halt, little bit like desktop computer when swapping.  And even when Fennec is responding the scrolling of the page refreshment is sluggish compared to microB. This makes the page reading experience unusable. A normal user won’t stand 1-3 second frequent responsivity pauses or page movement which cannot be controlled.

microB must do the rendering somehow different  - is it hardware acceleration on font rendering, smarter management of images or some other trick?. However, until Fennec reaches the smoothiness of microB, there is no way I would switch to Fennec over microB.

(Note: You can press CTRL-Backspace from N900 keyboard to force application switch if you cannot exit from halted Fennec otherwise)

Mobile browsing

Though N900 has 800 pixel wide screen, it is still a mobile phone. Small physical size, low bandwidth with high latency and  limited CPU power might make you to pick a mobile internet version of the site when it is available. However, since the screen has exceptional high Dots-Per-Inch value, this poses a problem for rendering sites with the default font sizes.

Fennec does not seem to have a shortcut for setting a large text size. This is something one would hope to see on such high DPI device as the most of the time default web site fonts are too small to be usable. Also, Fennec does not use the shoulder plus and minus volume buttons for zooming – microB does it and it is very natural place for this function.

Fennec seems to have some difficulties with mobile site rendering: for example touch.facebook.com and yle.mobi are not scaled to full width. Instead a narrow colum of 1/3 screen width is displayed.

Bugginess

microB is very solid piece of software. It crashes more rarely than Safari on iPhone (might this be because of more memory – low memory conditions seem to be a normal crashing condition for Safari?).  Fennec is still in its first version and have some issues.

(Note: I managed to get Fennec to zombie state – I had to go to terminal and type killall fennec command to make the browser become launchable again).

Sites tested

Slashdot.org

Geek discussion site

microB: no problems

Fennec: slow, frequent pauses, not smooth scrolling

slashdot.org/palm

Very simple mobile version of the above.

microB:  Font too small

Fennec: Scales correctly

Facebook.com

High profile social networking site

microB: Sometimes little slow, but seems to work perfectly

Fennec: Unusable slow

touch.facebook.com

microB: Perfect (at least when scaling font up a little)

Fennec: Does not scale correctly (default scale uses only 1/3 of screen width, double click zooming scales too much)

yle.fi

Finnish national broadcasting company site

microB: Ok. Readable and usable with text size large.

Fennec: Ok. The default view is navigable, but not readable. You need to double-click zoom to read the text (Fennec doesn’t seem to have text size large option)-

yle.mobi

The mobile version of above.

microB: Perfect with text size large, ok otherwise (need to double click to zoom and then click to choose a link to follow).

Fennec: Ok – font size too small

GMail HTML version

The default Javascript version of GMail is too heavy for both the browsers. GMail still provides “Basic HTML” view as the fallback for devices with less CPU power and network bandwidth.

microB: Ok – you can do some basic emailing

Fennec:  Ok. Does not seem to be affected by as much of “slugginess” as other sites are (might the slugginess be a Javascript issue?)

Youtube.com

The web version of flash based video sharing site.

microB: Plays Flash movies ok – smooth scrolling even whilst a Flash movie is playing

Fennec: Frequent grinds to halt, sluggish, unusable. Manages to open Flash video, though.

m.google.com/youtube

The mobile version of above.

microB: Youtube claims the browser is unsuppoted

Fennec: Cannot enter the site – shows only the page of Youtube Mobile instructions

twitter.com  (web site)

microB: Perfect

Fennec: Ok. Sluggish when opening new pages, but still usable. Fennec start view ships with Twitter button, so one might assume this site is well tested for Fennec.

m.twitter.com

The mobile site of above.

microB: Ok – the default font size too small, but when settings text size large works well

Fennec: Ok – the default font size too small. Double click zoom does not work well on the twit feed, making reading difficult.

plone.org

A community site with relatively simple layout.

microB: Ok – minor rendering errors

Fennec: Ok – minor rendering errors

iltalehti.fi

Finnish tabloid web site with lots of images.

microB:  Ok

Fennec: Grinds to halt, unusable slow

Summary

Though having nice promise of innovation, the advise for Fennec development team would be “back to the basics”. The slugginess and response times of Fennec are such an issue that one would not yet consider it as an real alternative for Nokia’s default microB browser.

With Fennec’s user interface and microB’s speed one could have a near perfect mobile browser. Depending what kind of future co-operation Nokia and Mozilla foundation will have, we might live to see it.

Profiling PostgreSQL database

Posted on January 2, 2010  by Mikko Ohtamaa
Filed Under linux, postgresql, ubuntu

This blog post will have some short notes about monitoring and profiling PostgreSQL databases.

pgtop

pgtop provides UNIX top command like user interface for PostgreSQL. pgtop command is available as Perl CPAN module.

How to install Perl CPAN modules as non-root user on Ubuntu (note: when it prompts to run sudo, answer no).

To install pgtop install following CPAN modules first: Term::ANSIColor, Term::ReadKey, DBD::Pg

perl -MCPAN -Mlocal::lib -e 'CPAN::install(DBD::Pg)'
perl -MCPAN -Mlocal::lib -e 'CPAN::install(Term::ReadKey)'
perl -MCPAN -Mlocal::lib -e 'CPAN::install(Term::ANSIColor)'

pgtop install instructions

pgtop manul

Running pgtop:

perl pgtop -d databasename -u yourdbuser -p yourdbuserpassword

pgfouine

pgfouine is a log analyzer for PostgreSQL.

It will be damn big, and damn fast…

Posted on December 21, 2009  by kipi
Filed Under Business

Phew… been going thru hundreds of pages just last week released The Mobile Internet Report by Morgan Stanley. Interesting statistics and conclusions to read. I fully agree how those wise people at Morgan Stanley summarized the findings of their research:

Mobile Ramping Faster than Desktop Internet Did and Will
Be Bigger Than Most Think –
5 Trends Converging (3G + Social Networking + Video +
VoIP + Impressive Mobile Devices)

Regarding pace of change, we believe more users will
likely connect to the Internet via mobile devices than
desktop PCs within 5 years

And seems to happen as it did with that “Desktop Internet”… driving force is the users and many (most?) of the companies do not realize that until they do not have any possibilities than to move “into” mobile internet. Sweet spot has reached already in Europe, next year world wide but… yeah… companies will wake up, too.. yet again late….

Evangelism, we need a bit that… and that will be done!

Black Friday was not sad for mobile commerce

Posted on December 2, 2009  by kipi
Filed Under Business

650% growth in a year.. not bad, eh? Well, some might say that starting low makes it easy to grow a lot if measured by percentages. Yes, that’s true but wasn’t it so with “normal” web, too?

In a recent data reviewed by Dow Jones (via mocoNews), mobile phones are used increasingly to buy things. This is helped by higher-end phones as well as by retailers who are building their sites and services to serve mobile phone users, too.

Those retailers or web shops serving mobile phone users are still rare. Some are trying, some are getting there and the ones who have already taken some action, have not been disappointed. Like, Swiftcover in UK (swiftcover.mobi) combined the launch of their mobile site with outdoor marketing allowing people to get quotes for their car insurance straight away by using their mobile phones. Unfortunately can’t tell about exact numbers but, well, lets put it this way: it worked!

Nokia N900, sports tracking and geotagging

Posted on November 29, 2009  by Mikko Ohtamaa
Filed Under geotagging, mobile, n900, nokia, sports tracker, technology
Tags: ecoach, exif, geotagging, google maps, gpicsync, gps, gpx, kml, map my tracks, n900, nokia sports tracker, sports tracker, sports tracking

This blog contains some tips how to use your Nokia N900 smart phone as a “augmented reality” sports device.

Sports tracking

Sport tracking is about collecting your sports activity data using GPS and other equipment. After running/cycling/skiing/whatever you see where you have been, how much time it took and how fast you are. In same cases you are able to calculate burnt calories and estimated heart rate.

N900 has at least one sports tracking application out there, eCoach.  eCoach is also suitable for professionals as it has heart rate monitor integration.

eCoach allows you record and  store sport activities. During the activity it uses Open Street Map based map viewer to show your current location. At least Helsinki area has very detailed maps available there, showing even the smallest trails, so you can safely venture to unknown neighbourhoods.

eCoach exports its tracks as GPX gps data file format. eCoach does not have any service integration yet, but you can upload this file to Nokia Sports Tracker and Map My Tracks. The recommend the latter as it has better social media integration and seems to be under active development. On the otherhand I have been using Nokia Sportstracker since 2007 and it has not really development during the whole this time and seems to lack will to go forward. Also Nokia has disabled track profile for imported GPX files which gives a message “we really don’t care about this service”.

There is also a service called mapmyrun.com with various domain names like “maymysomething.com”. Steer away from this service as I tested it and it didn’t live up to my expections (too much advertising, horrible user interface).

Some sport tracks I have made

• Run in Map My Tracks
• Another run in Map My Tracks
• The same run in Nokia Sportstracker
• All my runs in Nokia Sporstracker

Geotagging

Geotagging is about having GPS  coordinates on your photos. This way photos can be put on the map autotically in photo sharing services like Yahoo Flick or Google Picasa. When you known location, capture time and sharing license of the photo, all kind of fantastic services can be created, like Microsoft Photosynth.

Technically geotagging works by embedded GPS coordiates into the EXIF metadata of JPEG files.

N900 has geotagging as out of the box feature – no additional software needed. Just turn on it on in Camera application settings.

Also, you can retrofit your photos with geotagging information afterwards. You can do this by hand using labels and drag and drop in the most of photo sharing applications, like Google Picasa. Also there exist automated tools if you have relates GPS records available as GPX or KML file: checkout GPicSync. This is handy if you record your sports in eCoach and forgot to turn on geotagging in N900 camera. GPicSync also has a Google Maps export feature if you want to create custom maps for your friends or customers.

• An example map created with N900, eCoach,  GPicSync and Google Maps from one of the runs above

Sports tracking + geotagging = ?

I am still trying to figure out how to combine sports tracking and geotagging to something cool. Maybe something along the lines of urban exploration.

But in any case here are some of cities I have “collected” from my travels

• Budapest
• London
• Copenhagen
• New York (GPS was not exactly working well…)
• Oulu

PhoneGap ported on N900 (Maemo)

Posted on November 24, 2009  by Mikko Ohtamaa
Filed Under maemo, mobile, n900, phonegap, technology

We have ported PhoneGap mobile application framework to the new Nokia N900 smartphone and its Maemo operating system. PhoneGap is a framework to build mobile applications easily with HTML and Javascript. With the new Maemo port PhoneGap platform support covers iPhone, Android, Nokia Series 60, Blackberry, Windows Mobile and Maemo, making it one of the most portable mobile application development solution available.

The demo application currently supports only Device Info and Accelerometer APIs. The porting work is still in its early stages and we suggest novice developers steer away from it. We’ll develop and maintain the work as long as we have client cases for mobile applications. If you are looking forward to port your commercial PhoneGap application to N900, please contact us.

PhoneGap port was done using QT and QWebView controller. The native shell source code is in C++. Build and packaging scripts are standard Makefiles. More technical information on the release notes page.

Code is available on GitHub.

So what’s cool about Maemo (compared to other PhoneGap platforms)?

Shortly: The openess of Maemo platforms enables developer innovation never seen before. There are zero artificial limitations chaining your imagination.

• No code signing whatsoever required
• No approval process to get your application distributed
• Very robust development tools and development environment. Hey, it’s Linux! The phone ships with X terminal built in.
• You have root access to the device if needed
• QWebView WebKit control which itself is open source – you can recompile from scratch and stick in the needed features
• The phone itself is rock solid. It beats iPhone 100-0.
• Maemo has very active open source community. You actually might get help when stuck, unlike with NDAs and other madness from some other vendors.

So what’s cool about PhoneGap (compared to other mobile application technologies)?

• Use low entry level HTML, CSS and Javascript technologies – even PHp coder can build his/her own mobile application! This is the most cost effective way to develop non-CPU-intensive applications.
• PhoneGap has the widest mobile platform support – the best medicine against fragmented mobile application markets
• You can always break out from the sandbox and use the native capabilities of the phone. This is something you cannot do with Flash Lite or Java ME.
• Very active community
• WebKit rendering engine enables CSS3 goodies and much more
• Easily convert your existing mobile site to an application

Do you really think 160 chars tells the story?

Posted on November 23, 2009  by kipi
Filed Under Business

Still receiving from time to time some marketing SMS messages from different “clubs” I have opted in. Been too lazy to opt-out but those messages also works as kind of reminder about the status and state of mobile marketing. Just few weeks ago got a marketing message from one of those clubs (just rough translation and all identifying info removed/changed):

Now all Red Ball Products are on sale! 20% off on all normal priced items, only this Saturday!

Eh, I have no clue what those Red Ball Products are… nor willing to visit the store just to see if those happens to be something interesting stuff I want to buy. So, how about if the message would been something like this:

Now all Red Ball Products are on sale! 20% off on all normal priced items, only this Saturday! More info about Red Ball Product’s: http://domain/rbpad

And just by clicking the link in received SMS message I could have easily learned about what the heck are Red Ball Products and other info, perhaps other offers too.

In November 17 Internet Advertising Bureau (IAB) released a research results done in UK amongst the senior (ad, digital marketing etc.) agency representatives. Whoa, 95% of digital (marketing) budgets include mobile, too!!! Gee, that’s great number… do we see even more great text messages?

Same research revealed that most of the mobile (marketing) campaigns have not been planned by mobile specialists (in 2008 only 32% of agencies had mobile specialists, in 2009 52%). So, no wonder that plain text message is still the thing, media and channel when agencies plan their mobile marketing activities…

But things are moving to right direction, even slowly. Even here in Finland we have seen with some companies and agencies that they do want to do more, use that personal channel of mobile phones much more effective ways. And those agencies will be front runners when mobile internet will became more mainstream media, also for marketing and interaction, not just for consuming content, like reading the news.

Installing Python Imaging Library (PIL) under virtualenv or buildout

Posted on November 19, 2009  by Mikko Ohtamaa
Filed Under plone, python, technology
Tags: buildout, pil, python, satchmo, site packages, virtualenv

I have greatly struggled to have PIL library support in isolated Python environments like virtualenv –no-site-packages.

For example, when installing Satchmo shop under virtualenv:

../bin/clonesatchmo.pyhe Python Imaging Library is not installed. Install from your distribution binaries.

../bin/clonesatchmo.py
The Python Imaging Library is not installed. Install from your distribution binaries.

Though it clearly is there, installed by easy_install PIL command:

ls ../lib/python2.5/site-packages/PIL-1.1.7-py2.5-linux-x86_64.egg
ArgImagePlugin.py	 ExifTags.py		  GimpGradientFile.pyc...

Does anyone know if this problem is with PIL itself, eggified PIL or something else?

In any case, there is an easy workaround: use system-wide PIL (sudo apt-get install python-imaging) and symlink PIL from your site-wide installation under the isolated Python environment:

(satchmo-py25)mulli% pwd
/srv/plone/mmaspecial/satchmo-py25/lib/python2.5/site-packages
(satchmo-py25)mulli% ln -s /usr/lib/python2.4/PIL .