CRiSP Weblog

This is driving me nuts. GCC does tail recursion optimisation. That is very nice, and means that if we have something like:

int func(...)
{
	do-stuff();

	func2(...);
}

The problem is that this means that if we put a breakpoint in gdb, or a printf, in the func2, we lose the stack frame for func(), and it appears that func2 is being called from the caller of func(), rather than func() itself.

I wish they wouldnt do these "nice" things. Makes debugging a pain, and am tempted to go back to earlier/very old versions of GCC to stop this warfare, where "what used to work", stops working and you have to fight the toolchain.

For example, the 'Scale' widget which is used in the color selector dialog needed to be implemented (from scratch). That took about a day (nice when things go according to plan).

Then I hit some issues with the default size of a combo field. That is fixed.

Next up was the protocol manager for a shell widget. What is that you say? Think of XmAddWMProtocolCallback(). Without this, if you click on the window manager "X" at the top right of a window, then your app is quickly terminated (the TCP connection to the X server is severed).

Took me a while to figure out / remember how this all worked. But, suffice to say, that unless you post an Atom/Property on the window (WM_DELETE_WINDOW), then the window manager will not ask politely, but just brute force the termination of the app.

But to put a property on the application shell window is not quite so easy, especially when we normally call the XmAddWMProtocolCallback() against a *widget* and not a *window*. A widget may not exist on screen at the time we call it, and that is why there is complexity in the Motif library - you are allowed to register interest in these window manager protocol messages, before the window is 'realised' (i.e. before being mapped to the screen). When the window is mapped, the appropriate property is posted on the desktop to allow the window manager to see what is going on.

Of course, if you try to do something "later" in an application means creating some form of data structure for later use, and then, making sure we dont suffer a memory leak.

My implementation of window manager protocols isnt perfect, but sufficient for what I need.

Why do I bother? I dont know, but what I know is that CRiSP with Motif, statically linked, occupies 3MB of code memory. With the new Motif replacement, it is about 2MB.

When CRiSP was first written this was about the size of a large floppy disk (and 40MB - thats megabytes, not gigabytes) was huge. Now the size of CRiSP fits in the L2/L3 cache of a cpu.

But the menuing code has taken about 3 months - I hope this is now complete. Why?

Any number of reasons:

• I am losing my 'touch'
• It is difficult
• There are lots of fiddly bits to get right

You choose. The issue with menus is the way input focus moves around from one widget to another. There are lots of scenarios to get right, and fixing one of them, would result in some existing feature suddenly breaking - like a see-saw as the code came together.

One problem I hit is that XtAddGrab/XtRemoveGrab doesnt handle double registration of a widget particularly well.

The code, whilst trying to be purist and object oriented, in the end had to be a little dirty - one class having too much intimate knowledge of what it is dealing with (a menu has menu items, which is mostly separators and buttons, for instance).

Heres some scenarios to consider:

• Click and reclick the menu bar button (should dismiss menu)
• Use keyboard to navigate a menu and popup sub menus, and then popdown submenus.
• Use ESC to popdown a menu
• Click outside the menu to dismiss it; click in another window to dismiss it too
• Click on a menu item, and have the menu popdown, and the callback invoked

May seem like simple stuff, but getting it all working is difficult, especially when grabs are put in place - suddenly input goes to the wrong widget, and everything you had working, stops working.

Why bother ? Why not just use Qt or Gtk ?

Because I dont want to use them. As good as those toolkits are, they are not available everywhere, and I dont want dependencies on other toolkits - toolkits which have a very active development life.

This is akin to the dtrace problem: do you develop software for the latest and greatest kernel/distro out there, or do you go back to old releases and ensure your software works with them?

Implementing menus is frustrating. (The input field widget took about 2 days to implement from start to finish, and the result is more functional than what it replaces). By contrast, menus have taken a couple of months. Much of the original code worked u front.

What makes menus tricky are a number of things but mainly related to input focus and 'appearing to do the right thing'. Menus can popup with a mouse, and then be navigated into sub menus with the mouse or keyboard. I delayed getting the input focus problems solved until late in the implementation, which, in turn, broke much of the existing logic. (Events firing to the menu button popping up the menu or the menu or menu items, etc). Just as bad was restoring the status quo when selecting a menu item or dismissing the menu.

To make life palatable, I have a nice suicide-timer - after 25s, a child does a "kill -9" on the parent, which avoids the tedium of trying to unwedge the X server.

Its all very close, and am just busy ensuring all combinations work properly. (Mainly nested menus as they popup/get dismised).

Many of the problems are my own misunderstandings: I spent a good few days trying to get XGrabPointer() to work for me, only to eventually realise what I needed was XtAddGrab(). [Someone on a forum mentioned that if you weren't confused by XGrabPointer/XGrabPointer, then you didnt know what you were doing!]

XGrabPointer is to do with freezing the X server, typically used for drawing type apps. For menus, you sort of need a mixture of the two. (You want to intercept a menu dismissal when you click outside of your own application).

(I was debating using the iPod Touch to fix my X server hanging problems, e.g. have a web server running, which responded to trivial HTTP requests, and on receipt of a request, could douse the rogue hung application).

Ok, here it is - an ipod touch web server to help debug X11 apps...

#! /usr/bin/env perl                                                          
                                                                              
use strict;                                                                   
use warnings;                                                                 
                                                                              
use IO::Socket;                                                               
                                                                              
sub main                                                                      
{                                                                             
   my $sock = IO::Socket::INET->new (                                    
      LocalPort => 8080,                                                 
      Type      => SOCK_STREAM,                                          
      ReuseAddr => 1,                                                    
      Listen    => 10);
   while (my $client = $sock->accept()) {                                
           print "Killing hung app...\n";                                
           my $str = `ps ax | grep /home/fox/crisp_v9.5/bin.linux-x86_32/crisp`;
           chomp($str);
           next if !$str;
           $str =~ m/^ *(\d+) /;
           next if !defined($1);
           kill(-9, $1);
   }
}
main();
0;

Some people are reporting issues on GCC 4.4 and later glibc's. If you get these issues, let me know. glibc seems determined to break existing apps (GCC isnt quite so bad).

User space stack tracing is probably hosed because getting the user space stack, depending on the trap context is fiddly and something I hadnt finished.

Need to catch up on my crisp Motif work (it all works, but its not quite perfect enough to release the crisp code yet).

As a BTW, I decided to implement 'inertial scrolling' on fcterm and crisp. So, you may find (eventually) a new release with the feature where the faster you scroll the wheel on the mouse, the faster it scrolls. (Works nicely on fcterm, but may need to do more surgery on crisp since it only works for X windows, and needs to work for Mac+Windows too).

Sometimes, my movie playlists will play back to back, but mostly not. Its totally erratic, and simply feels like a 1st grader programming error. Its almost like an uninitialised variable is causing it to work or not. Sometimes switching it on/off helps, or a resync, but who knows.

Then, the other area of total insanity is the app store. The app store integration is brilliant - ability to browse, and waste a few minutes seeing whats available is great. Then you can download free software or pay for software.

But...why on earth does the ipod go into a mode where *no* applications will run? The startup screens almost show but then the app aborts or exits. Who knows what - because Apple decided not to show you a reason why the app quit. Then, some time later, they will all work again. This is clearly a bug in Apples firmware. I have downloaded maybe 15-20 apps, and I can believe some have bugs, but not on the initialisation of the app. Why it should go into "not going to run anything" mode and then recover, I dont know.

I downloaded an RSS reader - great software, but I am fearful it creates a lot of little files for the unread news - maybe thats tickling a bug; maybe the filesystem is fragmented - you cannot see the filesystem so you are SOL to know whats going on.

What is annoying is that lots of people on the web report the same thing and as far as I can tell, noone has clued up on the real causes, and Apple doesnt admit to this issue. Their 'walled garden' is full of street-corner muggers.

At least it plays movies and music - my main desire, but the app crashing is unforgivable. At least, some form of diagnostic tool to figure out what would be nice. Maybe thats a tool that needs to be written, or maybe I will find out on google somewhere what is going on.

It was the equivalent of an iPod, in its day.

Now armed with an ipod touch, and visiting the ipod store and getting a feel for this new device, I came across an HP48 calculator emulator for the ipod.

Lets go back to 1992. In 1992, I wrote an emulator for the HP48 - made available over Usenet, and, it worked - it could do most HP things. It was based around X11, and the PCs of the day, were very lowly, but, still, the calculator emulator was fast enough.

I still have the source code to this. I believe this code was taken and used as the basis for other calc emulators, and there are now excellent emulators for Windows (havent checked Linux, but should be easy).

Now, lets go back to 2009. The App store version of the emulator is great - full power of an advanced RPN graphing and symbolic algebra calc. The pixels on the ipod are just enough to do just, but only just.

This app is available as source code. See here for the announcement of the i48 app Github entry for the source is http://github.com/dparnell/i48/tree

This is interesting for two reasons. One, this is a simple iPod app, and hence, is a useful example of how to create an app. Complete with XML files and bitmaps for the app store. (Interestingly, the Objective-C code is tiny, which is a testament to a good api on the ipod touch).

The other interesting point is that the emulator code bears the hallmark of my original donation to the community. I dont know if this code was based on my original or not - I started to lose interest in the HP as CRiSP took more of my time in the early 90's, and was vaguely aware of a Windows port (bear in mind this would be Windows 3.x or Windows 95, which I detested beyond belief - you needed to reboot the system if any app crashed if you wanted a nice life).

The attributions in the code date back to 1994, but there are signs of similarities. I cannot remember how I got started on the emulation. I know I picked up a possible HP28 (predecessor to the HP48) emulator and got some internal docs from HP at the time (I probably still have the emails dating back then), but its nice to know this excellent piece of hardware and the emulators live on - and now, I can carry on with carrying an HP + IPod together.

Ob complaint: why is the HP50 so expensive in the UK? Its more than 100 GBP vs about $90-100 in the US. Needless to say, HP havent received any money from me because of this obscene pricing, and I suspect, the number sold in the UK is pitiful, which is a shame.

So, what next? Who knows. I would like to write something for the ipod, but I dont have time, and theres a lot of ideas to wade thru from the existing app base on the app store. I just wish the ipod wasnt so locked down (see complaints in the prior blog).

(1) Nested folders (playlists) dont appear to work on the ipod touch.

(2) On the touch front screen, is a "Movie" button. Search the web to find out how hard it is to get anything to appear in here. On the touch, movies need to be specified as "Music Video" or "TV Show". What? Duh?

(3) If you make it a "Music Video": a. it appears in the music playlist, not the video one. Are they insane? Yes. I have 92 playlists for music and upwards of 100 playlists for films (can only fit 20-30) on the ipod. So, i have to scroll thru the music to find the films, unless i use an initially letter, e.g. "A" to group them together. Are they insane? Yes.

(3b) A music playlist can be played portrait or landscape. A TV Show can only be played portrait. This wouldnt normally matter except in portait mode, you can see the film title when the popup volume controls appear but not in landscape mode. Are they insane? Yes

(3c) A TV Show playlist will not continue from one part to the next. I record from the tv to DVD across to a PC in 5min fragments. A typical film is 20 5min fragments. So, after each 5min fragment, my tv show skips back to the contents screen instead of continuing on to the next episode/fragment. All the parts of the film are in a playlist. So, Apple, are you insane? Yes.

(4) I knew you could get an onscreen control on the ipod screen to see position/volume controls, and it has taken me absolutely ages to know what to do - tried tapping, double tapping all parts of the screen, but it was so unobvious. Are you insane Apple? Yes.

(5) ipod touch wont charge on some alarm clock devices which let you plug in an ipod. The new 3G touch wont use my expensive and overpriced remote control from the ipod classic. Apple can be so money grabbing - they are a business, after all, but now, with so many ipods out there, and so many 3rd party devices, it is totally unclear what can work with what.

(6) App store apps which crash. I can appreciate software has bugs in it, but when I run an app, which then crashes, I really would like to know this and not have the ipod return back to the menu screen with no clue about what happened. What happened to the "bomb"?

(7) The ipod touch screen gets too smudgy; I can live with that. The screen is too dark - an OLED would be nice, but, watch a film set in a dark room, its almost impossible to see what is happening. To change brightness requires too many actions. Without a fast-fwd or reverse button, skipping over adverts is painful - trying to get your fingers in the right place (assuming you can work out how to popup the on screen display).

(8) Why are the classic and ipod touch/phone so different - I mean the way iTunes treats them. This is insane. They had a very good GUI and linkage with iTunes, but on the touch, its "lets be as different as we can". I could go into more details but.

(9) My ipod touch is much louder than the classic, for which i have spent so much on trying different headphones so I could hear quiet passages of films in a noisy environment.

(10) iTunes (9.0 and 9.0.1) is *insane*. Plug in your ipod. With 2000+ 5min film fragments, visit the device 'Movies' folder. It wants to open every one to display a screen shot image so you can select what to sync. iTunes mushroomed to greater than 1GB of RAM, and spent cpu cycles like no tomorrow. No way to turn off the image display. Fortunately, the "TV Shows" tab doesnt do this. Insane.

(11) ipod classic shows up as a mountable filesystem. ipod touch doesnt. iTunes/Apple have pulled a fast one so you cannot see the device as a mountable filesystem. I presume this is where the touch/phone cracking utilities come in. They have made it hard to do certain things, and "dtrace" (yes, dtrace) isnt able to monitor some aspects of iTunes deliberately (Adam Leventhal reported on this a while back, but dtrace is still broken). This is easy enough to fix/work around, but am curious. This means fixing a broken or ipod touch requires the service of an even better expert than a normal ipod has, and 3rd party tools become fewer and further between.

(12) Microsoft released the Zune HD. Lets hope the Zune really competes with the ipod, because Apple need a kick to be innovative. Apple stopped being innovative when they opened the Apple Store.

Despite my rant above, I am happy with the touch, but it has a long way to go to ensure it works for what it was intended for (music and films).

BTW there is a big difference between a classic and a touch: if your library fits into a classic or touch, life can be easy. If it doesnt, and it wont on a touch, then micromanagement of files is insanely tedious and difficult with iTunes. They have made the classic mistake of creating a GUI and touting how easy it is to use. It isnt. You are simply mortal, Apple.

I've been on holiday (San Francisco, LA, Vegas...), and my first trial is the MGM Grand Hotel. They pulled a fast one. Their customer services leave a lot to be desired, and their web site is appallingly awful.

We booked a 3 day stay, and the price was very good. ... Til we checked out and then we found their prices had doubled and doubled again (we arrived Thu before Labor day and stayed the Fri + Sat night). Nowhere on the web booking did it say that each night was a different price. What is worse...when we arrived, they got me to sign the obligatory credit card form, and hiding (in plain site), was the room rate for each night, going up in exponential fashion. So, partially my fault for not noticing that, but the person (might not have been human, and cannot put down the word I want to describe) didnt point this out. (It would have been too late anyhow).

Other than that - holiday was great.

Trial

I upgraded my main server today to Ubuntu 9.04 - just a short time away from the 9.10 release, but I wasted much of the day getting VMWare Server 1.0.x working on the 2.6.31 kernel. (I gave up with the Ubuntu kernel after it wasnt installed properly after the upgrade). VMWare is a pain - at least there is source code to the drivers, and eventually I got it to compile, but generated a kernel panic when vmware was started (more than likely, my code changes were a little too dirty). Oh well....

Tribulations...

So, I decided to try switching back to VirtualBox (2.1). On startup, it told me 3.0 was available so I have upgraded to that. Interestingly, I noticed that 'rdesktop' works nicely for VirtualBox (my previous complaint was that the X GUI was horrible when dealing with high volume output across a low speed wifi network), and this may help enormously solve that problem, and get me away from VMWare. I really didnt want to suffer the VMWare Server 2.x release, and now I can live in the freeware world for virtualisation.

DTrace for 2.6.31 Kernel

As per normal, the new kernel doesnt compile the dtrace code due to the number of changes in the kernel. Fortunately, the changes look much easier than the ones VMWare had to contend with, so will try and fix this shortly.

CRiSP without Motif

What has been occupying me for the last few weeks was migrating CRiSP away from Motif - its nearly finished - just need to do some final touches to the menu system and menu bar, and it looks/feels much better.

iPod Touch

Had been eagerly waiting for the new ipods to come out (I own an iPod Classic, but not an iPhone - they are simply too horrendously expensive), and although the new iPod isnt technically much better than the older ones, have ordered one - so I can watch films on the way to work (the Classic screen size and volume has always caused me an issue). I am thinking of toying writing some apps for the iPod....maybe I could port CRiSP or DTrace to it :-) (But that might be pointless tho!)

So, if things go quiet for a while...I am busy getting a high score or just hacking on the ipod or crisp ... or dtrace...

Am on holiday from next weekend for a couple of weeks, and I want to do something more rewarding, so am switching back to CRiSP for a while to kick some tyres.

First up is more finer control of file auditing - you can tell CRiSP to keep track of files you edit in an audit trail; useful for those times when you forgot where you placed a file.

I've fixed some other customer reports.

I keep on staring at ribbon bars, and before I fully tackle this (theres some pre-alpha code in CRiSP to do this, but its not ready for primetime), I am revisiting the Motif factor. CRiSP is built on Motif and over the years, it has driven me insane. In recent weeks I have fixed some uninitialised memory refs in Motif which could cause core dumps, but I have always had a goal to remove it totally. Many of the widgets are native Xt widgets, and the few remaining just require a bit of debugging to get rid of it totally - thus making the code more supportable, and ready for other things. (And freeing up a fair amount of memory).

CRiSP has some theming support and in getting rid of Motif, it will be easier to complete that, and finally make menu items to have icons in them.

People have also asked for freetype font support (which exists in CRiSP in a semi undocumented fashion). So, if the Motif removal goes well, then freetype can be made available to most of the widgets.

Why?

Imagine the SYSCALL instruction fires. This is a special instruction in the amd/x86 cpus which moves from user mode to system mode, *without* pushing the return address on the stack. The Linux kernel, immediately after the transition (entry_64.S) puts the user space SP into the thread task area, but the PC is hiding. On entry to the kernel side of a syscall, it is in the RCX register, but by the time we hit a probe, e.g. sys_open(), we are miles away and the pt_regs array isnt accurate. At the point of probe, we force a breakpoint trap (luckily, only our code executes at this point, so we dont have to consider nested interrupts and blowing the state areas in the thread stack).

What makes this tricky is getting everything to work at once - anything even slightly wrong just gives bogus results -- stack traces which are not accurate or totally missing.

I am better now - I seem to get the first two stack frames, but the third one is elusive (I am either miscomputing the dwarf frame info or misapplying the result to find the next frame; for a third frame, its frustrating since we have gone thru the same looped code twice, so why the third is problematic is not clear).

The code so far is fairly horrid, with lots of experiments in their, and no 32-bit version yet done. My biggest fear is if any of this is subtly dependent on kernel releases (I think it is not), so that would be one weight off my chest.

(Kernel releases are subtly different in syscall/interrupt handling, and also structure layout for the user/process/thread, but I dont think we care too much, yet).

Alas, the current Windows CRiSP release has black arrows on the scrollbars... to be fixed this weekend. Nuts.

I am trying to get this to parse properly:

$ build/dwarf /lib/libpthread.so.0
....
CIE length=00000014
  Version:              01
  Augmentation:         "zRS"
  Code alignment factor: 1
  Data alignment factor: -8
  Return address reg:    0x10
  Augmentation Length:   len=0x01 1b
R encoding 1b (kernel)

2c38 FDE len=7c cie=001c pc=e0ff..e109 tpc=ffffffffffffffff
0000: dwarf.c: unsupported DW entry 0xf 12

libpthread.so.0 is where the open64 syscall is located when I do my ustack() test against the perl interpreter.

In theory the parsing shouldnt matter, as in the kernel, we skip over blocks of the dwarf instructions to find the matching block, but it helps me to relax a little and better understand this stuff so I can tackle why some SYSCALL instruction blocks arent being handled properly.

People are sending me bug reports on 2.6.30.* kernels (fixed an issue with 2.6.30.4, but now theres a 2.6.30.5 - I cannot keep up with these releases and the gratuitous kernel code changes on each release!). So, just trying to stay above water, but progress is slow.

If you see no response from me, then this could be the issue - just remail me; if you see dup emails from me, its me attempting to fix the issue.

A particular issue I am having at present is the sys_open syscall. gdb can show a stack trace but my kernel code cannot find the appropriate dwarf frames mirroring where we came from. So I need to put in more effort to work through the use case scenarios.

...
  0   3004              sys_nanosleep:entry
              0x7f76eab2e104: libc-2.6.1.so`sleep+0x94
              0x7f76eb55a576: libperl.so.5.8.8`Perl_pp_sleep+0x56
              0x7f76eb51d1ee: libperl.so.5.8.8`Perl_runops_standard+0xe
              0x7f76eb4c7f4a: libperl.so.5.8.8`perl_run+0x30a

  0   2482           sys_rt_sigaction:entry
              0x7f76eab2e17a: libc-2.6.1.so`sleep+0x10a
              0x7f76eb55a576: libperl.so.5.8.8`Perl_pp_sleep+0x56
              0x7f76eb51d1ee: libperl.so.5.8.8`Perl_runops_standard+0xe
              0x7f76eb4c7f4a: libperl.so.5.8.8`perl_run+0x30a
...

Its nearly there, but need to resolve some more issues, and then we should have a viable ustack() call even on omit-frame-pointers applications. (Still need to do the 32-bit equivalent of the above).