Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Sunday, November 16, 2008

UPCScan 0.7: Where is my stuff?

UPCScan 0.7 is released. New features:

  • UPCScan can now find music CDs
  • If UPCScan can't find something on Amazon, it will still create an entry which you can then edit to fill in the details.
  • Entries can be deleted.
  • I've added lending information so you can quickly figure out who your new "ex-friends" should be.
  • I'm working on a series/issue information system to make it more simple to complete your collection. With this version, you'll need to edit the database directly to add series/issue information but the user interface can already display this data.
  • I'm working on a feature to create an OpenOffice document with the locations. This would allow you to print this out and then scan the locations in as you scan your collection to tell UPCScan under which location to file the items. If you can't wait, then you can use the barcode.py script to generate PNG images with barcodes which you can import in OpenOffice to achieve the same effect.

Download: upcscan-0.7.tar.gz (26,921 Bytes, MD5)

Thursday, October 09, 2008

Enthought Traits

I'm always looking for more simple ways to build applications. Let's face it, it's 2008 and after roughly 50 years, writing something that collects a few bits of data and presents them in a nice way is still several days of work. And that's without Undo/Redo, a way to persist the data, a way to evolve the storage format, etc.

Python was always promising and with the tkinter module, they set a rather high watermark on how you easily could build UIs ... alas Tk is not the most powerful UI framework out there and ... well ... let's just leave it at that.

With Traits, we have a new contender and I have to admit that I like it ... a lot. The traits framework solves a lot of the standard issues out of the box while leaving all the hooks and bolts available between a very thin polish so you can still get at them when you have to.

For example, you have a list of persons and you want to assign each person a gender. Here is the model:

class Gender(HasTraits):
    name = Str
    
    def __repr__(self):
        return 'Gender %s' % self.name

class Person(HasTraits):
    name = Str
    gender = Instance(Gender)
    
    def __repr__(self):
        return 'Person %s' % self.name

class Model(HasTraits):
    genderList = List(Gender)
    persons = List(Person)

Here is how you use this model:

female = Gender(name='Female')
male = Gender(name='Male')
undefined = Gender(name='Undefined')

aMale = Person(name='a male', gender=male)
aFemale = Person(name='a female', gender=female)

model = Model()
model.genderList.append(female)
model.genderList.append(male)
model.genderList.append(undefined)
model.persons.append(aFemale)
model.persons.append(aMale)

Nothing fancy so far. Unlike the rest of Python, with Traits, you can make sure that an attribute of an instance has the correct type. For example, "aMale.gender = aFemale" would throw an exception in the assignment.

The nice stuff is that the UI components honor the information you use to build your model. So if you want to show a tree with all persons and genders, you use code like this:

class Model(HasTraits):
    genderList = List(Gender)
    persons = List(Person)
    tree = Property
    
    def _get_tree(self):
        return self

class ModelView(View):
    def __init__(self):
        super(ModelView, self).__init__(
            Item('tree',
                editor=TreeEditor(
                    nodes = [
                       TreeNode(node_for = [ Model ],
                           children = 'persons',
                           label = '=Persons',
                           view = View(),
                       ),
                       TreeNode(node_for = [ Person ],
                           children = '',
                           label = 'name',
                           view = View(
                               Item('name'),
                               Item('gender',
                                  editor=EnumEditor(values=genderList,)
                               ),
                           ),
                       ),
                       TreeNode(node_for = [ Model ],
                           children = 'genderList',
                           label = '=Persons by Gender',
                           view = View(),
                       ),
                       TreeNode(node_for = [ Gender ],
                           children = '',
                           label = 'name',
                           view = View(),
                       ),
                    ],
                ),
            ),
            Item('genderList', style='custom'),
            title = 'Tree Test',
            resizable = True,
            width = .5,
            height = .5,
        )

model.configure_traits(view=ModelView())

First of all, I needed to add a property "tree" to my "Model" class. This is a calculated field which just returns "self" and I need this to be able to reference it in my tree editor. The tree editor defines nodes by defining their properties. So a "Model" node has "persons" and "genderList" as children. The tree view is smart enough to figure out that these are in fact lists of elements and it will try to turn each element into a node if it can find a definition for it.

That's it. Everything else has already been defined in your model and what would be the point in doing that again?

But there is more. With just a few more lines of code, we can get a list of all persons from a Gender instance and with just a single change in the tree view, we can see them in the view. If you select a person and change its name, all nodes in the tree will update. Without any additional wiring. Sounds too good to be true?

First, we must be able to find all persons with a certain sex in Gender. To do that, we add a property which gives us access to the model and then query the model for all persons, filter this list by gender and that's it. Sounds complex? Have a look:

class Gender(HasTraits):
    name = Str
    persons = Property
    
    def _get_persons(self):
        return [p for p in self.model.persons
                if p.gender == self]

But how do I define the attribute "model" in Gender? This is a hen-and-egg problem. Gender references Model and vice versa. Python to the rescue. Add this line after the definition of Model:

Gender.model = Instance(Model)

That's it. Now we need to assign this new field in Gender. We could do this manually but Traits offers a much better way: You can listen for changes on genderList!

    def _genderList_items_changed(self, new):
        for child in new.added:
            child.model = self

This code will be executed for every change to the list. I walk over the list of new children and assign "model".

Does that work? Let's check: Append this line at the end of the file:

assert male.persons == [aMale], male.persons

And the icing of the cake: The tree. Just change the argument "children=''" to "children = 'persons'" in the TreeNode for Gender. Run and enjoy!

One last polish: The editor for genders looks a bit ugly. To suppress the persons list, add this to the Gender class:

    traits_view = View(
        Item('name')
    )

There is one minor issue: You can't assign a type to the property "persons" in Gender. If you do, you'll get strange exceptions and bugs. Other than that, this is probably the most simple way to build a tree of objects in your model that I've seen so far.

To make things easier for you to try, here is the complete source again in one big block. You can download the Enthought Python Distribution which contains all and everything on the Enthought website.

from enthought.traits.api import \
        HasTraits, Str, Instance, List, Property, This

from enthought.traits.ui.api import \
        TreeEditor, TreeNode, View, Item, EnumEditor

class Gender(HasTraits):
    name = Str
    # Bug1: This works
    persons = Property
    # This corrupts the UI:
    # wx._core.PyDeadObjectError: The C++ part of the ScrolledPanel object has been 
    # deleted, attribute access no longer allowed.
    #persons = Property(List)
    
    traits_view = View(
        Item('name')
    )
    
    def _get_persons(self):
        return [p for p in self.model.persons if p.gender == self]
    
    def __repr__(self):
        return 'Gender %s' % self.name

class Person(HasTraits):
    name = Str
    gender = Instance(Gender)
    
    def __repr__(self):
        return 'Person %s' % self.name

# Bug1: This doesn't work; you'll get ForwardProperty instead of a list when
# you access the property "persons"!
#Gender.persons = Property(fget=Gender._get_persons, trait=List(Person),)
# Same
#Gender.persons = Property(trait=List(Person),)
# Same
#Gender.persons = Property()
# Same, except it's now a TraitFactory
#Gender.persons = Property

class Model(HasTraits):
    genderList = List(Gender)
    persons = List(Person)
    tree = Property
    
    def _get_tree(self):
        return self
    
    def _genderList_items_changed(self, new):
        for child in new.added:
            child.model = self

Person.model = Instance(Model)
Gender.model = Instance(Model)

female = Gender(name='Female')
male = Gender(name='Male')
undefined = Gender(name='Undefined')

aMale = Person(name='a male', gender=male)
aFemale = Person(name='a female', gender=female)

model = Model()
model.genderList.append(female)
model.genderList.append(male)
model.genderList.append(undefined)
model.persons.append(aFemale)
model.persons.append(aMale)

assert male.persons == [aMale], male.persons

# This must be extenal because it references "Model"
# Usually, you would define this in the class to edit
# as a class field called "traits_view".
class ModelView(View):
    def __init__(self):
        super(ModelView, self).__init__(
            Item('tree',
                editor=TreeEditor(
                    nodes = [
                       TreeNode(node_for = [ Model ],
                           children = 'persons',
                           label = '=Persons',
                           view = View(),
                       ),
                       TreeNode(node_for = [ Person ],
                           children = '',
                           label = 'name',
                           view = View(
                               Item('name'),
                               Item('gender',
                                  editor=EnumEditor(
                                      values=model.genderList,
                                  )
                               ),
                           ),
                       ),
                       TreeNode(node_for = [ Model ],
                           children = 'genderList',
                           label = '=Persons by Gender',
                           view = View(),
                       ),
                       TreeNode(node_for = [ Gender ],
                           children = 'persons',
                           label = 'name',
                           view = View(),
                       ),
                    ],
                ),
            ),
            Item('genderList', style='custom'),
            title = 'Tree Test',
            resizable = True,
            width = .5,
            height = .5,
        )

model.configure_traits(view=ModelView())

Wednesday, October 08, 2008

UPCScan 0.6: It's Qt, Man!

Update: Version 0.7 released.

Getting drowned in your ever growing CD, DVD, book or comic collection? Then UPCScan might be for you.

UPCScan 0.6 is ready for download. There are many fixed and improvements. The biggest one is probably the live PyQt4 user interface (live means that the UI saves all your changes instantly, so no data loss if your computer crashes because of some other program ;-)).

The search field accepts barcodes (from a barcode laser scanner) and ISBN numbers. There is a nice cover image dialog where you can download and assign images if Amazon doesn't have one. Note: Amazon sometimes has an image but it's marked as "customer image". Use the "Visit" button on the UI to check if an image is missing and click on the "No Cover" button to open the "Cover Image" dialog where you can download and assign images. I haven't checked if the result of the search query contains anything useful in this case.

UPCScan 0.6 - 24,055 bytes, MD5 Checksum. Needs Python 2.5. PyQt4 4.4.3 is optional.

Security notice: You need an Amazon Web Service Account (get one here). When you run the program for the first time, it will tell you what to do. This means two things:

  1. Your queries will be logged. So if you don't want Amazon to know what you own, this program is not very useful for you.
  2. Your account ID will be stored in the article database at various places. I'm working on an export function which filters all private data out. Until then, don't give this file to your friends unless you know what that means (and frankly, I don't). You have been warned.

Saturday, October 04, 2008

Scanning Your DVD, Book, Comic, ... Collection

Update: Version 0.6 released.

If you're like me, you have a lot of DVDs, books, comics, whatever ... and a few years ago, you kind of lost your grip on your collection. Whenever there is a DVD sale, you invariantly come home with a movie you already have.

After the German Linux Magazin published an article how to setup a laser scanner with Amazon, I decided to get me one and give it a try. Unfortunately, the Perl script has a few problems:

  • It's written in Perl.
  • It's written in Perl.
  • It's written in Perl.
  • There is no download link for the script without line numbers.
  • The DB setup script is missing.
  • The script uses POE.
  • It's hard to add new services.
  • Did I mention that it's written in Perl? Right.

So I wrote a new version in Python. You can find the docs how to use it in the header of each file. Additionally, I've included a file "Location codes.odt". You can edit it with OpenOffice and put the names of the places where you store your stuff in there. Before you start to scan in the EAN/UPC codes of the stuff in a new place, scan the location code and upcscan.py will make the link for you. It will also ask you for a nice name of the location when you scan a location code for the first time.

If you need more location codes, you can generate them yourself. The codes starting with "200" are for private use, so there is no risk of a collision. I'm using this Python script to generate the GIF images. Just put this at the end of the script:

if __name__=='__main__':
    import sys
    s = checksum(sys.argv[1])
    img = genbarcode(s, 1)
    img.save('EAN13-%s.gif' % s, 'GIF')
    print error

There is a primitive tool to generate a HTML page from your goods and a small tool to push your own cover images into the database if Amazon doesn't provide one.

Note: You'll need an AWS account for the script to work. The script will tell you where to get your account ID and where you need to put the ID when you start it for the first time.

Download upscan-0.1.tar.gz (54KB, MD5 Checksum)

Wednesday, July 16, 2008

Docs? Ask The Sphinx

If you need to generate docs for your Python projects, try Sphinx.

Tuesday, July 08, 2008

TurboGears 2.0 Is On Track

There are three things which hooked me to TurboGears:

  1. Every day stuff is simple, complex stuff is possible
  2. Automatic reload after code change (no need to restart)
  3. It's in Python

What I didn't like is that TG 2.0 has been so quiet for so long. I'm on the Planet Turbogears RSS feed and I wasn't sure whether 2.0 was alive or dead or whatever.

Well, it seems to be more alive than I expected and hopefully, we'll see a 2.0 soon. In "Doing the right thing should be easy" by Mark Ramm, you can find more details.

Friday, January 18, 2008

Portable UI

For many years, I've been looking for a way to write portable applications with a nice, responsive user interface. Many have tried and many have failed:

  • Python with tcl/tk - A nice experience from the developer side. The Python wrapper around the tk widget set shows how you can get compact, yet easy understandable code and write UI's in short time. If it just weren't that ugly ...
  • Java with Swing - Swing borrows a lot from X11, the grandfather of all graphical desktops. I have yet to see anyone managing to impress the world with their grandfather ...
  • Java with SWT - Now, here comes a contender. Java is pretty widely available (not quite as many platforms as Python, but still), it is pretty fast, okay, the download is a bit on the big side ... but no DLL hell, easy to setup (especially if you don't provide an installer and just push a ZIP out). SWT is nice, fast ... and bare bones. MFC? Well, they have JFace and in a few years, there might even be a text editing component that can do word wrap and still show line numbers. Oh, and SWT is available on even fewer platforms than Java. Palm, anyone?
  • HTML - Web based apps are all the hype. If you want to use your app on the run, it gets tricky. I don't know about the US, but here in Europe, going online with you mobile will ruin you. Literally. Also, I've had my struggles with HTML and CSS and I can do without. Either and both.

I've tried a few more but in the end, things never felt right. Until recently. I'm a big fan of treeline. Treeline uses Python and PyQt which wraps Qt (say: "cute"). Qt is a mature framework, currently at version 4.3.3, with 4.4 is around the corner. It doesn't have all the nifty stuff I can imagine (like an RTF editor; QTextEdit can only do a (big) fraction of that) but it gets closer to what I want than anything else.

In the past two weeks, I wrote a little clone of yWriter4. The little baby has currently about 8000 loc and about half of the functionality I want to give it (especially the text editing is still leaving a lot to be desired). Except for two bugs (signal names and GC issues), it's been a real pleasure to use. I managed to implement almost every feature within a few minutes or few hours (the storyboard took 6 hours, the scene chart view took two), also thanks to the good defaults of the framework. Here is an impression of v0.2:

So when you're considering to write a small to medium sized application which needs to run on Windows, Linux and MacOS, give PyQt a try.

Monday, January 14, 2008

Sorting Number Table Columns in PyQt4

Here is a simple trick to sort number columns in the QTableWidget of Qt4 and PyQt4: Format the number as a right aligned string:

for i range(12):
    item = QTableWidgetItem(u'%7d' random.randint(1, 10000))
    item.setTextAlignment(Qt.AlignRight)
    table.setItem (i, 1, item)

Saturday, July 21, 2007

What's Wrong With Java Part 2

OR Mapping With Hibernate

After the model, let's look at the implementation. The first candidate is the most successful OR mapper combination in the Java world: Hibernate.

Hibernate brings all the features we need: It can lazy-load ordered and unordered data sets from the DB, map all kinds of weird relations and it lets us use Java for the model in a very comfortable way: We just plain Java (POJO's actually) and Hibernate does some magic behind the scenes that connects the objects to the database. What could be more simple?

Well, an OO language which is more dynamic, for example. Let's start with a simple task: Create a standalone keyword and put that into the DB. This is simple enough:

   1:
   2:
   3:
   4:
   5:
Keyword kw = new Keyword();
kw.setType (Keyword.KEYWORD);
kw.setName ("test");

session.save (kw);

Saving Keyword in database

(Please ignore the session object for now.)

That was easy, wasn't it? If you look at the log, you'll see that Hibernate sent an INSERT statement to the DB. Cool. So ... how do we use this new object? The first, most natural idea, would be to use the object we just saved:

   1:
   2:
   3:
   4:
Knowledge k = new Knowledge ();
k.addKeyword (kw);

session.save (k);

Saving Knowledge with a keyword in the database

Unfortunately, this doesn't work. It does work in your test but in the final application, the Keyword is created in the first transaction and the Knowledge in the second one. So Hibernate will (rightfully) complain that you can't use that keyword anymore because someone else might have changed it.

Now, what? You have to ask Hibernate for a copy of every object after you closed the transaction in which you created it before you can use it anywhere else:

   1:
   2:
   3:
   4:
   5:
   6:
   7:
   8:
   9:
  10:
  11:
Keyword kw = new Keyword();
kw.setType (Keyword.KEYWORD);
kw.setName ("test");

session.save (kw);
kw = dao.loadById (kw.getId ());

Knowledge k = new Knowledge ();
k.addKeyword (kw);

session.save (k);

How to save Knowledge with a keyword in the database with transactions

Why do we have to load an object after just saving it? Well ... because of Java. Java has very strict rules what you can do with (or to) an object instance after it has been created. One of them is that you can't replace methods. So what, you'd think. In our case, things aren't that simple. In our model, the name of a Knowledge instance is a Keyword. When you look at the code, you'll see the standard setter. But when you run it, you'll see that someone loads the item from the KEYWORD table. What is going on?

   1:
   2:
   3:
public void setName (Keyword name) {
    this.name = name;
}

setName() method

Behind the scenes, Hibernate replaces this method by using a proxy object, so it can notice when you change the model (setting a new name). The most simple soltuion would be to replace the method setName() in session.save() with calls the original setter and notifies Hibernate about the modification. In Python, that's three lines of code. Unfortunately, this is impossible in Java.

So to get this proxy objects, you must show an object to Hibernate, let it make a copy (by calling save()) and then ask for the new copy which is in fact a wrapper object that behaves just like your original object but it also knows when to send commands to the database. Simple, eh?

Makes me wonder why session.save() doesn't simply return the new object when it is more safe to use it from now on ... especially when you have a model which is modified over several transactions. In this case, you can easily end up with a mix of native and proxy objects which will cause no end of headache.

Anyway. This approach has a few drawbacks:

  • If someone else creates the object, calls your code and then continues to do something with the original object (because people usually don't expect methods to replace objects with copies when they call them), you're in deep trouble. Usually, you can't change that other code. You loose. Go away.
  • The proxy object is very similar but not the same as the original object. The biggest difference is that it has a different class. This means, in equals(), you can't use this.getClass == other.getClass(). Instead, you have to use instanceof (the copy is derived from the original class). This breaks the contract of equals() which says that it must be symmetric.
  • If you have large, complex objects, copying them is expensive.
  • After a while, you will start to write factory methods that create the objects for you. The code is always the same: Create a simple object, save it, load it again and then return the copy. Apart from cut&paste, this means that you must not call new for some of your objects. Again, this breaks habits which leads to bugs.

All in all, the whole approach is clumsy. Really, it's not Hibernate's fault but the code is still ugly, hard to maintain (because it breaks the implicit rules we have become so used to). In Python, you just create the object and use it. The dynamic nature of Python allows the OR mapper to replace or wrap all the methods as it needs to and you never notice it. The code is clean, easy to understand and compact.

Another problem are the XML config files. Besides all the issues with Java XML parsers, it is always problematic to store the same information in two places. If you ever change your Java model, you better not forget to update the XML or you will get strange errors. You can't refactor the model classes anymore because there is code outside the scope of your refactoring tool. And let's not forget code completion which works pretty good for Java. Not so for XML files. If you're lucky, someone has written a code completion for your type of XML config. Still, there will be problems. If there is a new version, your code completion will lag behind.

It's like regexp: Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. -- Jamie Zawinski

Fortunately, Sun solved this problem with JPA (or at least eased the pain). JPA allows to use annotations to store the mapping configuration in the class file itself. Apart from a few small problems (like setting up everything), this works pretty well. Code completion works perfectly because any IDE which has code completion will be able to use the latest and greatest version of your helper JARs without any customization. Just drop the new JAR in your classpath and you're ready to do. Swell.

But there are more problems:

  • You must create a session object "somewhere" and hand it around. If you're writing a webapp, this better be thread-safe. Not to mention you must be able to override this for tests.
  • The session object must track if you have already started a transaction and nest them properly or you will have to duplicate code because you can't call existing methods if they use transactions.
  • Spring and AOP will help a lot but they also add another layer of complexity, you'll have to learn another API, another set of rules how to organize your code, etc.
  • JAR file-size. My code is 246KB. The JARs it depends on take ... 6'096KB, more than 40 times of my code. And I'm not even using Spring.
  • Even with JPA, Hibernate is not simple to use because Java itself is not simple to use.

In the end, the model was 5'400 LoC. A added a small UI to it using SWT/JFace which added 2'400 LoC.

If you look at the model in the previous installment, then the question is: Why do I need 5'000 LoC to write a program which implements an OR mapper for a model which has only three classes and 26 lines of code?

Granted, test cases and helper code take their toll. I could accept that this code needs four or five times the size of the model itself. Still, we have a gap.

The answer is that there are no or bad defaults. For our simple case, Hibernate could guess everything. Java could generate all the setters and getters, equals() and hashCode(). It's no black magic to figure out that Relation has a reference to Knowledge so there needs to be a database table which stores this information. Sadly, defaults in Java are always "safe" rather than "clever". This is the main difference to newer languages. They try to guess most of the stuff and then, you can fix those few exceptions that you always have. With Java, all the exceptions are handled but you have to do everyday stuff yourself.

The whole experience was frustrating, especially since I'm a seasoned Java developer. It took me almost two weeks to write the code for this small model mostly because because of a bug in Hibernate 3.1 and because I couldn't get my mind around the existing documentation. Also, parent-child relations were poorly documented in the first Hibernate book. The second book explains this much better.

Conclusion: Use it if you must. Today, there are better ways.

Next stop: TurboGears, a Python web framework using SQL Objects.