A Different View on Exceptions

The discussion about checked/unchecked exceptions is almost as old as Java. While we all have a point in your stance towards this, maybe we are looking at the problem from the wrong angle. Manuel Woelker wrote an article which concentrates on the receiver of the exception, the user, and how exceptions should behave to help the user: Exceptions From a User’s Perspective

How to Hide a Virus in Source Code

I've been looking for quite some time for this article: How can you hide a virus in the source code? Basically, you create a binary of a compiler which contains the virus and which is patched to infect other programs as it compiles them.

Reflections on Trusting Trust by Ken Thompson.

Another Lesson on Performance

Just another story you can tell someone who fears that "XYZ might be too slow":

I'm toying with the idea to write a new text editor. I mean, I've written my own OS, my own XML parser and I once maintained XDME, an editor written originally by Matthew Dillon. XDME has a couple of bugs and major design flaws that I always wanted to fix but never really got to it. Anyway.

So what is the best data structure for a text editor in 2008? List of lines? Gap-Buffer? Multi-Gap-Buffer?

XDME would keep the text in a list of lines and each line would point to a character array with the actual data. When editing, the characters would be copied into an edit buffer, the changes made and after the edit, the changed characters would be copied back, allocation a new character array if necessary.

This worked, it was a simple design but it had a flaw: it didn't scale. The length of a line was limited to the max size of the edit buffer and loading a huge file took ages because each line was read, analyzed, memory was allocated ... you get the idea.

So I wanted to make it better. Much better. I'd start with reading the file into a big buffer, chopped into evenly sized chunks to make reading both fast and memory efficient (imagine loading a 46MB file into a single memory chunk - after a couple of changes, I'd need to allocate a second 46MB chunk, copy the whole stuff over, etc, needing twice the amount of RAM for a short time).

During the weekend, I mulled the various ideas over, planned, started with a complex red-black tree structure for markers (positions in the text that move when you insert before them). It's huge, complex. It screams "wrong way!"

So today, I sat back and did what I should have done first: Get some figures. How much does it really cost to copy 4MB of RAM? Make a guess. Here is the code to check:

    public static void main (String[] args)
    {
        long start = System.currentTimeMillis ();
        
        int N = 10000;
        for (int i=0; i<N; i++)
        {
            int[] buffer = new int[1024*1024];
            System.arraycopy (buffer, 0, buffer, 1, buffer.length-1);
        }
        
        long duration = System.currentTimeMillis () - start;
        System.out.println (duration);
        System.out.println (duration / N);
    }

On my machine, this prints "135223" and "13". That's thirteen milliseconds to copy 4MB of RAM. Okay. It's obviously not worth to spend a second to think about the cost of moving data around in a big block of bytes.

That leaves the memory issue. I would really like to be able to load and edit a 40MB file in a VM which has 64MB heap. Also, I would like to be effective loading a file with 40MB worth of line-feeds as well as a file which contains just a single line with 40MB data in it.

But this simple test has solved one problem for me: I can keep the lines in an ArrayList for fast access and need not worry too much about performance. The actual character data needs to go into a chunked memory structure, though.

Morale: There is no way to tell the performance of a piece of code by looking at it.

25 Most Dangerous Programming Errors

If you want to improve your l33t [0d|ng skillz, especially keeping script kiddies off your back, here is a list of the 25 most common coding errors: http://www.sans.org/top25errors/

Holding a Program in One's Head

In my perpetual search for brain food for programmers, I've found this article: Holding a Program in One's Head

Stuck? Ask Stack Overflow

Stuck with a hard programming problem? Just solved an impossible problem and want to show the world your genius? Don't know how to solve a problem with your favorite OS or programming language? Check out stackoverflow.com.

How To Launch Software

If you're still wondering if "big bang" software releases are a good thing, read this.

The Space Between Two Characters

If you're claustrophobic, you're afraid of confined spaces. If you're a software developer, you can be afraid of non-existing space.

When it comes to editing text, we usually don't think about the space between the characters. There simply isn't any. When you write a text editor, things start to look different. Suddenly, you have a caret or cursor which goes between the characters and that space between two characters can suddenly become uncomfortably tight.

Fire up your favorite text editor, Word, Writer, whatever. It has to support character formatting, though. Now enter this:

Hello, world!

If in doubt, the bold text ends before the comma and the italic part starts with the "w" and ends with "!" including both.

Now move to the "e" and type "x". What do you get?

Hxello, world!

Piece of cake.

Now move to the "H" and type "x". What do you get? Is the new x bold or not? Do you get "xHxello" or "xHxello"? How about typing "x" after the "o" of our abused "Hello"? Is that new x bold or not? If it is, what is the most simple way to make it non-bold? Do you have to delete the comma, do you have to go through menus or toolbars or is there a simple, consistent way to add a character inside and outside of a formatted range of text?

Let's go one step further. Add a character after the "!". Is it italic? If not, you're lucky. If it is ... what's the most simple way to you get rid of the italic? If you press Return now, will the italic leak to the next line? If not, how can you make it leak? If that italic is the last thing in your text, can you add non-italic text beyond without fumbling with the formatting options?

There is no space between two characters and when you write a text editor, that non-existing space is biting you. Which is actually the problem: There is no consistent way to move in and out of a formatted range of characters.

The naive attempt would be to say "depending on the side you came from, you're inside or outside." So, if we have this (| is the cursor or caret): "Hello |world" and you type something, the question is: How did the caret end there? Did it come from "w|o" and moved one to the left? Or from "o| " and move one position right?

That works somewhat but it fails at the beginning and the end of the text plus you're in trouble during deleting text. What should happen after the last character of "Hello" has been deleted? Should that also delete the character range or should there be an empty, invisible bold range left and when you type something now, it should appear again? If you keep the empty invisible range, when do you drop it? Do you keep it as long as the user stays "in" it? Or until the document is saved? Loaded again from disk?

It's a mess and there is a reason why neither Word nor OpenOffice get it right: You can't. There is information in the head of the user (what she wants) but no way for her to tell the computer. Duh.

That is, unless you start to give the user a visual cue what is going on. The problems we have is that there is no simple, obvious way for the user to say "I want ..." because there is no space on the screen reserved for this. We barely manage to squeeze a caret between the characters. There is just not enough room.

Well, there could be. A simple solution might be to add a little hint to the cursor to show which way it is leaning right now. Right. How about "A|B"? Here, you have three options. Add bold, italic and normal.

In HTML, this is simple. I'm editing this text in Firefox using the standard text area. What looks fancy to you looks like this to me: "<b>A</b>|<i>B</i>"

And this is the solution: I need to add a visual cue for the start and end of the format ranges. Maybe a simple U-shape which underlines the text for which the character format applies. Or an image (> and < in this example): ">A<|>B<". And suddenly, it's completely obvious on which side of the range start and end you are and what you want. You can delete the text in the range without losing it or you can delete both and you can move in and out of the range at will.

The drawback is that you need to keep that information somewhere. It adds a pretty huge cost to the limits of a format range. I'll have to try and see how much that is and if I can get away with less by cleverly using the information I already have.

Also, it clearly violates WHYSIWYG. On the other hand, we get WYSIWYW which is probably better for the user.

Text Editor Component and JADS

While working on DecentXML (1.2 due this weekend), I've had those other two things that were bugging me. One is that there is no high-quality, open-source framework with algorithms and data structures. I'm not talking about java.lang.Collections, I'm talking about red-black trees, interval trees, gap buffers, things like that. Powerful data structures you need to build complex software.

Welcome the "Java Algorithm and Data Structure" project - jads. I haven't started opened a project page on SourceForge or Google Code, yet, but I'll probably do that this weekend.

Based on that, I'm working on a versatile text editor component for Java software. The final editor will work with user interfaces implemented in Swing, SWT and Qt. It's an extensible framework where you can easily replace parts with your own code to get the special features you need. I currently have a demo running which can display text, which allows scrolling and where you can do some basic editing. Nothing fancy but it's coming alone nicely.

If you want to hear more about these projects, post a comment or drop me a mail.

Another Lesson on Performance

Just another story you can tell someone who fears that "XYZ might be too slow":

I'm toying with the idea to write a new text editor. I mean, I've written my own OS, my own XML parser and I once maintained XDME, an editor written originally by Matthew Dillon. XDME has a couple of bugs and major design flaws that I always wanted to fix but never really got to it. Anyway.

There are various data structures which are suitable for a text editor and some of those depend on copying data around (see gap buffers). The question is: How effective is that? The first instinct of a developer is to avoid copying large amounts data and to optimize the data structure instead.

After years of training, I've yet to overcome this instinct and start to measure:

    public static void main (String[] args)
    {
        long start = System.currentTimeMillis ();
        
        int N = 10000;
        for (int i=0; i<N; i++)
        {
            int[] buffer = new int[1024*1024];
            System.arraycopy (buffer, 0, buffer, 1,
                buffer.length-1);
        }
        
        long duration = System.currentTimeMillis () - start;
        System.out.println (duration);
        System.out.println (duration / N);
    }

On my computer at work (which is pretty fast but not cutting edge), prints: "135223" and "13". That's thirteen milliseconds to copy 4MB of RAM. Okay. It's obviously not worth to spend a second to think about the cost of moving data around in a big block of bytes.

Lesson: If you're talking about performance and you didn't measure, you have no idea what you're talking about.

Still not convinced? Read this.

Quantity Always Trumps Quality

While I wouldn't completely subscribe to that without a grain of salt, the story is nice.

A Decent XML Parser

Since there isn't one, I've started writing one myself. Main features:

• Allows 100% round-tripping, even for weird whitespace between attributes in elements
• Suitable for building editors and filters which want to preserve the original file layout
• Error messages have line and column information
• Easy to reuse
• XML 1.0 compatible

You can download the latest sources here as a Maven 2 project.

DSLs: Introducing Slang

Did you ever ask for a more compact way to express something in your favorite programming language? Say hello to DSL (Domain Specific Language). A DSL is a slang, a compact way to say what you want. When two astronauts talk, they use slang. They need to get information across and presto. "Over" instead of "I'll now clear the frequency so you can start talking." And when these guys do it, there's no reason for us not to.

Here is an artical on Java World which gives some nice examples how to create a slang in Java and in Groovy. Pizza-lovers of the world, eat your heart out.

The Code Reuse Myth

The post "The Code Reuse Myth" by James Sugrue got me thinking.

The main problem with code reuse is that our programming languages don't support it. We sacrificed this to the gods of efficiency four decades ago and, while a few people dared to question the practice, all of them were struck down by unexpected lightning out of the blue, so far.

As James said, "context" is the keyword. What we'd need is a programming language where you can adapt concepts to a context that goes beyond "object instance" or "class" or "application". What we need is an efficient way to say "collect all objects and sort them by ID" where the context defines what "ID" is. What we need is an efficient way to describe a model (relations of basic data types) and then have some tool map that efficiently to reality so we can reuse parts or fragments of the model in different contexts.

OO can't do that because it's limited to factories and inheritance. Traditional preprocessors can't do it, because they can't see the AST. Having closures in a programming language is the first, tiny step in the direction to be able to push external context into existing code. This allows us to put the code to access the database into a library and influence what it does per row of the result with a closure.

But to be a real new paradigm, we need "closures" in data types as well. This means being able to reuse fragments of code and data structure definitions in a new context. These fragments need to pull along all the algorithms and structures they need without the developer having to pay close attention what is going on until the point where the result needs peep hole optimization because of performance issues.

Example:

fragment Named {
    String name

    String toString () {
        return "name=${name}"
    }
}

class File : Named {...}

class Directory : Named, File {...}

Looks simple but with OO, this will get you in a lot of trouble: Directory gets a name from the class File and from the fragment Named (this is an artificial example, bear with me). Which one should the compiler chose? In OO, I can't say "I don't care" or "Merge them".

With real fragments, you could say "Directory is a File in the sense that it supports a lot of the operations of a file (like rename, delete, get last modification time) but not all (you can't execute a directory or open it for reading)." So the example would look like this:

class File : Named { 
    void delete () {...}
    Reader openForReading () {...}
}

class Directory : Named, File.delete {...}

Now, a Directory has a name and it "copies" the method "delete" from File along with anything this method would need. Internally, the compiler could create an invisible File delegate or it could clone the source or byte code of the File class, or whatever. Or, even better, we could say "give me a copy of File.delete() but replace all references to the field File.path with Directory.path."

The main goal would be to allow to use the compiler as a cut'n'paste tool which checks the syntax and allows me to say "copy that method over there and replace xxx with yyy". Because that's why we think that code reuse could work: We see the same code over and over and over again and each time, the difference is just a tiny little bit of code which we can't "patch" because the language just doesn't allow it.

Are Bad Tests Worse Than No Tests At All?

In his article "Are Bad Tests Worse Than No Tests At All?", olivstor writes:

Are the drawbacks to bad tests worse than having no coverage at all? I think the answer is that in the short term, even bad tests are useful. Trying to squeeze a extra life out of them beyond that, however, pays diminishing returns. Just like other software, your tests should be built for maintenance, but in a crunch, you can punch something in that works. It's better to have bad tests than to have untested code.

Tests are like any other code: They can go bad.

In my career, I've found that it's surprisingly hard to write good tests if you have no experience in doing so. People starting to write tests make them too complex, too long, let them have too many dependencies and they take too long to run.

If you're in such a situation, you have to face the fact that you just programmed yourself in a corner and you must spent the effort to get out of there. Tests are no magic silver bullet. They are code and follow the usual rules of coding: When it hurts, something is broken and it won't stop hurting unless you fix it.

So in this sense, I agree that bad tests are better than no tests because they tell you early that you need to fix something. That's what their core purpose is.

Management might argue that you're spending too much time on testing. I've never had a problem to sell myself to them. I usually figure that I spend 50% of my time (or more!) writing tests and 50% actual coding - and I'm still much faster than those who write code 80% of the time or more. What's more: when my code goes into production, it's is rock solid or at least easy to fix when something comes up. In 99% of the cases, the things I need to fix were those which I didn't test.

This is a positive reinforcement loop which drives me to test more and more because it stops the hurting. If your tests cost more than they seem to return, you need to fix them until you get the same positive feeling when you think about them.

Starting Your Own OSS Project

If you're planning to roll your own little OSS toy project, you should read the article "Party of one: Surviving the solo open source project" by Kirill Grouchnikov. Very good points on what to do and what to avoid and why.

Collaborative Editing

Mustafa K. Isik has a video on vimeo demonstrating collaborative editing of a Java class in Eclipse. I'm not sure where this leads and my experience winces at the idea but it would sure help if the superman of your team could attach to your editing session, see the code you're working on and help you fix it (instead of having to get up and coming over).

How to Work Better

From the wall of a corporate building in Zurich:

1. Do One Thing at a Time
2. Know the Problem
3. Learn to Listen
4. Learn to Ask Questions
5. Distinguish Sense from Nonsense
6. Accept Change as Inevitable
7. Admit Mistakes
8. Say It Simple
9. Be Calm
10. Smile

Eye Candy For Developers

There are two new projects on the 'net which might influence the way we see software in the future (in the literal sense). code_swarm can visualize the CVS history of a project in a cunning way and Wordle creates tightly packed tag clouds.

What I find especially interesting about Wordle is that it creates the tag cloud in real time (as you watch) using a Java applet. If you happen to have Java 6 installed, the animations are very smooth, almost like Flash.

The Craft of Text Editing

While looking for a decent code base for a text editor on Google Code (tip: search for "label:editor"), I found a wiki page in the epytor project which links to "The Craft of Text Editing", by Craig A. Finseth. That document not only explains all the basic algorithms behind the scenes of a text editor but also compares different implementations and contains a wealth of useful general comments.