vdboor

Today I've been busy converting the KMess svn repository to git. [for the uninitiated: both svn and git are tools to track changes in the source code of an application. KMess is a MSN Messenger client for KDE ]. The reasons for git are twofold:

• KDE is moving to git.
  Instead of trying to move from sourceforge -> KDE extragear now, we can directly move to git.
  
• We seriously need better merge support now.
  As we're working on KMess 2.1 a lot of refactoring will happen; to create a library of our protocol code, and provide a plugin API. This requires branches for large changes, while continiously merging the mainline. Git can do this.
  

Which brings me back to today. Since our repository is also organized like KDE (/trunk/projectname/...) using svn-all-fast-export (made by Thiago) was the natural choice. This tool uses a "rules" file to specify which project a given repository paths should go to. It also offers ways to filter out commits.

Turns out that last feature is useful for us too. While the repository may look clean from the outside, it's history has some curlpits. A few times branches were created from the wrong path, deleted again, and recreated with the correct path. These can be filtered out.

Installing svn-all-fast-export

There are no binaries for the the application, so you'll have to run:

git clone git://repo.or.cz/svn-all-fast-export.git
cd svn-all-fast-export
qmake fast-export2.pro
make

The devel packages for Qt, subversion and apr need to be installed. For openSUSE 11.1 I had to correct the file src/Makefile afterwards because the path to apr include path wasn't /usr/include/apr-1.0 but/usr/include/apr-1 instead.

Using it

First, you'll have to create a rules file. There are plenty of examples in the "examples" folder for this.

The documentation of svn-all-fast-export is absent at the moment - except for a few sample rule files. Fortunately the source code was quite readable, and it's Qt 4 application. Looking at the source, I've learned:

• The option --identity-map=<file..> can be used to convert the svn accountnames to git's "full name <email>" format.
• The options --resume-from and --max-rev can be used to continue at a given part (e.g. after getting an error, you can write a rule for it, and continue with that revision).
• The output directory for every sub-project (repository with git) needs to exist already, and it should be initialized with git init. Not doing this gives an "broken pipe" abort, and you'll find yourself looking through the source code to find out why.
  

The result of my day includes of a large rules file, and shell scripts to automate everything. I've stored it in our subversion repository in a separate trunk/svn-to-git project, so feel free to look around. I hope these scripts can help you too. One thing.. there is again no documentation.... Read the source

Issues left open

There are a few issues I haven't been able to solve yet.

• A few sub projects were moved across the repository, which is split in separate git repositories (svn paths: /trunk/libisf -> /trunk/kmess/contrib/libisf). Is there a way to preserve the history for this?
• The network-library branch was branched off /trunk/ instead of /trunk/kmess/, and moved afterwards to a different location. I've tried to mark both folders with /kmess/ appended as being part of the branch. Now I'm stuck with a network-library branch which lacks the base revision. Only the changed files can be found in the branch.

Lately I got a request to blog more about the progress of KMess, and I've been scratching an itch on it. Since we're about to release a "KMess 2.0 beta 2" release I'll delay that for later today/tomorrow.

The train from work to home also started to get a bit boring so I've decided to install openSUSE on my MacBook. I really wanted to do this because KDE 4.2 is looking so sweet. Running OS X didn't feel right anymore at all.

The setup was a bit difficult due to EFI-partitioning being messed up, but the actual install was easy. In case you're looking for a HOWTO, the openSUSE wiki pages give you everything you need. Most hardware seams to be working, including: bluetooth, wifi, audio, compositing, video out, special keys, sensors, iSight webcam (requires firmware). There are some quirks remaining though: at the login I briefly see some video memory garbage and the synaptics touchpad sometimes gets locked after suspend to ram.

However, now I can have some fun in the train, show pictures to friends easier, potentially expose Linux/KDE4 to more people, read cached e-mail, etc... Perhaps I could code somewhat, but most of my energy is still spent on my daily job and the renovation of my apartment in the evenings/weekends. I'll blog a bit more about that later.

With this blog entry I'd like to inform you all about the amazing progress we're making with KMess 2.0. It's been a bit quiet on the blogging side, but development surely didn't stop. The development timeline at trac.kmess.org reveals there are 200 commits within the last 30 days.

Within a few days we hope to release a 2.0-alpha 2 release, which has the following new features:

• Everything is running quite stable under KDE 4.
• Tabbed chats are supported now. This is such a breeze, I wouldn't ever want to live without that anymore!
• Every dialog received a nice makeover.
• Connecting over HTTP is supported, making KMess usable for restricted LAN's.
• Ink sending works!
• Larger messages can be sent
• MSNplus formatting can be displayed.
• We've added an application-wide settings dialog for common settings.
• Lots of nifty improvements everywhere; choose your fire transfer ports, automatically download to a folder, copy contact 'now listening' information, etc..

We stil have our share of things we like to improve, implement and annoyances to fix, so the upcoming release will still be called an alpha. Most of the work on KMess 2 has been done by Valerio and Antonio, so I like to thank them for this as well!

This is how KMess 2 look like now:

In the screenshot you see the ink receiving and tabbed chat in action.

KMess on Windows

There is experimental support for Windows as well:

There is no "setup.exe" yet to double-click on, but a lengthy manual of steps to install. Hence the reason we still call it experimental. Everything seams to work though.

Personal stuff

I didn't always find time to work on KMess in the past few months. If you have a daily job as web developer, not all evenings are easily spent on coding as well. I still do a lot of things in the background, and I'm slowly starting to pick up coding again. I'm working through the first parts to receive webcam sessions, and you can expect this to be present in KMess too.

After the summer holidays I'll also be moving to to my very own appartment (maisonette actually). It has a small balcony, 3 bedrooms and a view to a lovely park in front, which I really love. Some work needs to be done before I can move in, like installing new kitchen and installing central heating. Guess you'll know what I'll be doing this autumn.

Oh and:

So if you see a guy coding for KMess on a white MacBook running KDE on Windows, it's probably me It's my first time on Akademy and I didn't make any plans yet, but love to join some BoF meetings and discuss innovation. Last time I checked, almost every pillar of KDE could find a place within KMess.

This is not meant as a sentimental entry. It's about marketing and communicating a message to your potential users. Too often I find myself at a website of an Open Source project and ask myself. "where am I now? what the -beep- is this? and what can I do with it?"

As personal example I take kmess.org as example. We tried really hard to fix this. The site is a lot of content nowadays, but we think there is a lot of room for improvement. While discussing this we Valerio came up with the following blog entry:

Why Ubuntu 8.04 needs better marketing

That article shows the big difference between a summary of technical details and something that appeals. Their revised announcement makes me all of a sudden excited about downloading Ubuntu 8.04. Less technical wording, clearly written sentenses and focus on what advantages does it has for me. Worth reading, this article is written really well!

In the comments there is also a nice observation:

The key however is to, like Apple, make a directly emotional appeal. Even Intel claims to be trying to take this route, inspired by Apple (successfully? Maybe). Take a look at Apple’s OS X page: http://www.apple.com/macosx/. The bold heading is very dominant. And it doesn’t really contain too much logic… again, it is an emotional appeal.

I really like Apple's site. It makes me enthusiastic about their products, and while browsing it just goes on. Yet I can also find a lot of technical details there. I couldn't describe why, but now I'm starting to understand the key aspect here: emotions.

Another example: Last week I was on a holiday/journey, and tried to explain a bit Aikido to the guy next to me. It's the martial art I practice and love. In the years I developed a few short phrases to explain it but somehow my description didn't get though at all.

Fortunately I got a little advise whispered in my other ear: "you are too technical". Dang! I'm currently inspired by "feeling first, mind later" theories and this is another eye opener for me. By using more vague descriptions adjusted to the receiver (communicating a feeling/emotion) the other guy managed to get it a lot better. Whoa. Using less strict descriptions actually makes people grasp something better?

This incident among others makes me realize a lot of people are probably wired this way (call them alpha's if you wish). As technicians we love to communicate details, and the receiver can reconstruct the same image in their mind. Most people are not like that, or can't manage to be so. I didn't expect this gap could be so big. I noticed how this guy next to me responds much better while communicating an emotion, feeling or vague description (which you can technically put down as inaccurate, misleading, etc..). It has a strong effect, as the message is received in a more powerful way.

Meanwhile I'm starting to get an itch to do something with this conclusion within the KMess website too. If a lot of people are wired this way, shouldn't our websites reflect that? I'd like to call it "communicating emotions".

<shameless-plug>We're looking for more ways to improve the KMess website. If you have suggestions, don't wait any longer and post them!</shameless-plug>

A while ago I mentioned briefly KMess would be ported to KDE 4. We weren't sure when to do it yet. Eventually we decided to port it as soon as possible before doing new changes to the codebase. That would only make the porting more difficult, or give SVN a vague status to new contributions. So at 15 January I started a branch for porting the code.

After all automated changes we went through all files one by one (resuling in ~190 commits) before things managed to compile again. For those that are interested, there is a screenshot of the first run. Things crashed at startup, crashed deep in QHttp, froze, and corrupted memory multiple times. Once you manage to open the settings panel again things look really funny.

We've managed to get these issues fixed in the last 2 weeks and things are starting to get back in shape. This is KMess after ~270 commits since the initial start of the 'kde4porting' branch:

Login dialog		Full desktop

Note that the application is still not usable. Now we've gotten past most crashes we can fix everything we've broken and make it pretty. Some things were broken on purpose to get past all compiling errors, like the contact list, now playing information and saving of settings. In fact, your settings will likely be eaten at this point.

There are some visible benefits for the KDE 4 porting already. Oxygen alone makes apps breathtaking beautiful. The networking code has less dependencies on KDE now, which helps to build a library from it later. The application startups almost instantly. Memory statistics also show some interesting effects (note the size is also affected by the shared libraries).

I'm not really sure what those numbers mean, but for the ignorant among us it makes KDE 4 applications look so much better.

I'm pleased to announce the release of KMess 1.5, a MSN Messenger client for KDE.

After a long time of development from version 1.4 first, and the 1.5-pre series after, we've been able to obtain a very stable and feature-filled version.

Changes

Users of KMess 1.5-pre2 will might notice that file transfers have become stable while operating full-speed. The ugly popup balloons are gone and you can send custom emoticons now. A lot of bugs are squashed in the process, and the overall application got the polish it deserves.

Users of the last official stable release (1.4.3) will notice the quite some new interesting features. Most parts of the user interface have been improved. Combined with rich colorful chat styles it's a refreshing breeze for the eyes.

This release brings the MSN support to a level we're happy with. During the development of KMess 1.5-pre1 and pre2, we've added support for the things you'd expect from a MSN Messenger client. That's nudges, now playing information, custom emoticons, personal messages, fast file transfers and automatic download of display pictures. To make sure you don't miss something in a conversation, Winks and offline messages can also be received, being able to send those as well is still a todo.

Looking back

Making the release of KMess 1.5 took a lot of time. Much more time then I anticipated. The introduction of a new developer (thanks Valerio!) helped a lot here. Looking at the changes I'm really stunned how much we've been able to polish.

It was quite difficult to get the direct connections stable. They've been a source of many crashes and lockups. Seeing it work out all smooth now is just unbelievable. It just works, like every user expects it to be. But after so much trouble I'm filled with excitement each time I see that progress bar rush to 100% in no-time. It amazes me every time.

What felt as a little project with the 1.4 / 1.5-pre1 release, seams different now. With the new website, trac installation for developers (tickets and wiki pages), announcement writing, and rock solid release it feels this little project has matured a lot. It gives me confidence we'll be able to pull up a lot more nowadays.

Future plans

KMess 1.5 will be the last version based on Qt 3 and KDE 3. The next major version will be based on Qt 4 / KDE 4. Originally I'd planned to get a release done before the KDE 4 beta's, but it took far more time to get the file transfers stable. It means there won't be a KMess 1.6, but a 2.0 version since KDE 4 will likely be adopted (e.g. with 4.1) by the time we're ready with 2.0.

I had some plans to collaborate with other projects before, but never really gotten to it yet. Since everyone is going for Decibel lately, I'm curious how we work towards that goal as well. I'm curious how it will work out, because I don't want to loose the specific MSN protocol features we have.

For now, enjoy the release!

Over the last few weeks I've been reading some intersting books, which are definitely worth to be read.

The first one is "Barbarians Led by Bill Gates". The book gives an intersting view point about Microsoft by an ex-programmer. How the work has been in those times, how projects were managed.

Interestingly to mention: it seams Windows was in fact seen as an unimportant project after it's initial failure with version 1.0. Management abandoned it, and some programmers kept poking arround to make it better - without all the management pressure. It appears the success of Windows 3.x was completely unexpected to Microsoft too!

The book also tells about the beginnings of the marketing machine - you can say the biggest product of all. However I missed details about Microsoft's (imho unethical) behavior towards the outside world. I guess that stems from the fact the author is an ex-programmer, and he mostly tells what he has seen from his position: how projects were managed. Still an intersting read though, because it gives quite a different view on Microsoft's own developments and successes.

The second book I'm reading right now is "Getting Things Done" by David Allen. And oh my this is real nice piece of work! I've gotten an introduction to GTD before, but with the book everything falls more in place. I'm at page 66 now, and it's already an revelling experience for me. Not only does David advocate a different model of tracking todo's and planning projects, he also explains how his methology actually co-operates with the natural, intuitive, way your mind operates all the time.

Additionally it's really geek friendly, addresses common pitfalls and yet I can manage my stuff with simple tools! I have got a pen+paper with me all times nowdays, I collect the todo-notes in a simple basket at home, and process those later into Kontact notes, Basket or KOrganizer (but even .txt files would do fine!).

Everyone who is doing "knowledge work" benefits from GTD. It works bottom-up, addressing the hectic atmosphere at your work/home first (which prevents you to move forward), and slowly helping you to get more overview of your projects, responsabilities and vision. It does this by helping you getting stuff out of your mind, so your mind can focus itself on more creative stuff, and do what's really good at (which isn't recalling todo's at the moment you actually needed them).

GTD promisis an interesting end result: to overcome stress, feel confident you won't miss something important, feel more relaxed and have more energy to actually do things. And while reading David's reasoning I believe this system will actually work for me, both personally and professionally.

Last week I've committed a major change to the KMess SVN. In this case "major" means the diffs were 100kB in size. It fixes the last two problems we had with MSN direct connections, and brings the next release a lot closer too.

The first problem was caused by our way of sending file data. KMess 1.5-pre2 pushes messages to aggressively to the socket, choking the buffers of KExtendedSocket. By only sending the few bytes it could, this broke all message-length fields of squential packets. Oops..! This is fixed thanks to the readyWrite() signal and a lot of internal API refactoring. Hence the 100kB diffs.

Another problem was caused by some unknown protocol goodies revealed by Windows Live Messenger (WLM). Half way the transfer progressbar stalled in WLM, while KMess still thought it was sending file data. wtf... A closer look revealed it did get some messages back (flag 0x01). Now what..? That flag is isn't known yet at msnpiki.

From the WLM logs I've learned this means "chunk out-of-order", and it happened after KMess received some kind of "application/x-msnmsgr-transudpswitch" or "application/x-msnmsgr-transdestaddrupdate" message. I still have to clue what that message means or what to do with it, except sending an ACK as usual. I did figure out this broke the transfer in SVN. KMess incremented the message-ID to send the ACK, and now the remaining file data was sent with the wrong message-ID. Whoa. Not good! Fortunately it was easy to fix, and learned about some interesting new protocol messages too.

Finally, these two problems also broke picture transfers, emoticons and winks. Why you might ask? Those are also sent in parallel over the same direct connection! Almost everything is sent over the direct connection! It's good for performance and server load, but hard to code right at once. We've got it all working now, which I'm really proud of.

I can finally say that KMess supports direct connections properly for MSN file and picture transfers. Yay!

The responses to by previous blog entry were more then I hoped for. Among them was an invitation to join the KDE-NL BBQ, which was helt last weekend. Whooha!

As newcomer, the experience was mindblowing. Everyone has the same passion and because KDE has so much to offer, everyone has something different to tell about too. So there is a lot to talk about: the state of development, Akademy, KDE4, marketing idea's, plans that are going, and all kinds of interesting details. There were developers from Amarok, KOffice, Mailody, the Music Notation Flake (Soc), but also translation, and marketing/promo teams.

Everyone was genuinely interested in each others stories, everyone clearly showed respect for each other, and we had some good laughs. If that wasn't cool enough, consider how conversations went full speed. Where you normally have to slow down because you're talking too fast, there was nothing to worry about among KDE enthusiasts. It's said that people keep spending soo much time on KDE because of the community. I definitely believe that's true. I enjoyed every piece of that day, and love to be part of such events more often.

I went homeward with Jos Poortvliet and Niels van Mourik. We had a some difficulties finding the the highway because we spent too much time talking. I had some good conversations with Jos back home, and thanks to him I didn't have to make the entire journey back by train. This would have taken more then 2 hours in total, now only 20 minutes.

At the train station I had an interesting experience as well. I sat waiting there with my notebook bag and a Baguette left from the BBQ. Which is quite odd at 23:00 to say the least. Some guys joined me to ask if I was from France, where I got that Baguette from. Oh BBQ? where? with friends? Well.. how do you say this.. er.. have you heard about Linux? yes? .. I had a BBQ with guys who work on KDE in The Netherlands.... er.. a set of graphical programs for Linux! Oh Cool! You work on Linux? It was pretty awesome to talk with those guys at a train station about Linux vs Windows vs Mac OS (due to my notebook), and how that guy didn't like Windows but still used it. Not that we could talk in detail - time was short and it mostly went about stereotypes - but every bit helps.

I spent the evening talking over MSN till about 2am, and was well awake before the alarm went off too. What a day!

My blog has just been added to PlanetKDE, so I'd like to say hi to everyone! ..and tell something about what I do for KDE development.

I'm one of the main developers of KMess, a MSN Messenger client for KDE. You can say KMess brings an "MSN Messenger like experience" to Linux without copying the annoying parts. I tried all clients when I started with Linux in 2003, and really loved the user interface of KMess. It felt much like MSN, but even better.

With my next entries I hope to get more technical, show some screenshots, or post idea's about Linux. I'm curious how this will all work out.

introducing...

I'm from the Netherlands, 23 years old, code webapplications for a living and have a passion for KDE (following the dot, blogs and akademy screencasts closely), and started contributing to KMess in 2004. This started because I liked every part of KMess; both UI design and code. It was very easy to jump in.

Currently we're finishing up KMess 1.5. It includes a lot of bugfixes and we got rid of more annoyances too. With the next release (either 1.6 or 2.0) we'll implement webcam and multimedia support. I actually hoped to have 1.5 out already to leave KDE 3 with a good 1.6 release, but it's likely we'll be porting to KDE 4 instead because of the multimedia requirements.

We all know Kopete is part of KDE, and KMess exists outside the KDE SVN. I don't actually mind this; it's good to have one standard client shipping, even posted some bugzilla comments for Kopete. The bottom line is that KMess got me started with KDE development. If you need more explanation then this, you're welcome to post a comment though!

something about msn

One of the most difficult parts of the MSN protocol is the client-to-client part (MSN P2P). It's used to exchange pictures, emoticons, files and all other kinds of invitations directly between two contacts. It's what makes most clients distinguish from each other. Not getting this part right means webcam sessions and file transfers break when someone sends an emoticon or changes their avatar. The reverse-engineered documentation help a lot, but I keep noticing they miss practical details you'll encounter as developer. In KMess, I actually had to refactor the MSN P2P code three times before I saw the whole overall design of it. I've been told Mercury's counter is at 6 now. That's something new MSN plugins can avoid, and I don't mind sharing my experience about that!

A lot of energy has been put in good support for MSN P2P in KMess, and the code works really well. It means we're ready to build stable webcam sessions in the next release. Combined with the APIDOX examples the P2P code is almost a reference guide to implement MSN P2P. Worth checking out!

Happy KMess'ing or .. Kopete'ing!

The KMess sourcecode has been migrated from CVS to subversion (SVN). (FYI: subversion is the software we use to store the in-development version of KMess). Subversion gives us some advantages over CVS. It's command line is easier to use, branches/tags are easier to mange, changes to many files - even moved files - are tracked properly with subversion. I actually hoped to migrate earlier, but I wanted to wait until 1.5-pre2 was released.

The CVS to SVN migration did took more time then I'd expected. The cvs2svn tool needs some help to convert things properly, and sourceforce's documentation leaves much ends open. And there was something I overlooked: the website documentation and forum howto's had to be updated as well... *oops*

An interesting side-effect of subversion are the statistics you can generate from it. If you like to see those statistics, download the repository first with: rsync -av rsync://kmess.svn.sourceforge.net/svn/kmess/ kmess-svnrepos. Then open it locally with kdesvn. The most simple statistic is the activity over the years. It's clearly visible that development slowed down during the 1.3 release, and basically stopped. I was also shocked to find out I'm responsable for ~1200 of the 1700 commits since 2004-07-12. It should be noted I had to commit directories and ChangeLog entries separately.

While updating the website, I also improved the documentation about the coding standards. With the additional examples it easier to follow the proper styles now.

Finally it's out! KMess 1.5-pre2 can be downloaded as of today. The current CVS code had many fixes and improvements, so I didn't want to delay a next release much more.

The most important bugfixes are better support for direct file transfer connections, and links in the chat window. There are also some nice improvements, like file transfer previews, now playing information, and an option to disable the nudge shaking effect. More information can be found at the front page and changelog.

My favorite features are previews for any file type (see previous entries), and clickable geek-style links. It happens all the time people write "dot.kde.org", "planetkde.org", or "slashdot.org". I wanted to make these links clickable as well, without getting too many false-positives. This worked out really well, so try it out!

Yesterday I was debugging some weird bugs. Actually spent several hours on it.

With the file transfer previews, Windows Live Messenger chops of some parts. This can be solved by sending the image as 96x96, with transparent parts added. QImage to the resque! Well.. sort of. I ran into a pretty annoying bug.

This is the code to generate a transparent image:

int size = QMAX( image.width(), image.height() );
QImage fixedImage( size, size, 32 );
fixedImage.setAlphaBuffer( true );
fixedImage.fill( qRgba( 0, 0, 0, 0 ) );  // black transparent.

Saved as PNG, it produces something like this:

Well, some parts of the image are transparent.. But the remaining of it.. juck! Something produces a fuzzy pattern here. Today it even seams to have more noise.

After trying the same code in a stand alone application, I got a perfectly transparent image. When the same code runs in KMess, it doesn't work anymore. That's the weirdest part of it. It can either be in Qt 3.3.8, KDE 3.5.6 or openSUSE 10.2, I really can't tell.

After some debugging, Google codesearch showed how to reset the individual image bits in a for-loop. A bit later I noticed how the image bits are actually one memory block (the Qt API is not really clear about that). So the fill() command can be replaced with memset(), hence the following code fixes the problem:

memset( fixedImage.bits(), 0, size * size * 4 );

This takes advantage of the image data/scanlines exposed by QImage. Pffew

With this fix in place, I'm looking forward to release 1.5-pre2. It should be out really soon, since CVS has some interesting improvements. I'll have to update the TODO and NEWS files, check for any blockers. Small bugs are not one of those, that's for the final 1.5. A next preview needs to be avoid really soon first!

There are some additional improvements to the file transfer previews. The previews are now generated with a more low-level KIO API. This no longer blocks the KMess interface to generate the preview.

As extra bonus, KMess generates previews of video files as well. When you send a movie to someone who uses the official client, they'll see something like this:


Pretty neat, isn't it?

KMess just got a new cool feature! I've received a patch for file transfer previews. The result now looks like this:

In the official client this is used to send previews of images. KMess also adds support for almost every imaginable file type, thanksverymuch KDE! Think PDF, OpenDocument, HTML, Qt designer files, really everything. Whooho!