CRiSP Weblog

I keep on switching to VirtualBox only to find my one and only VM has died. Shame. I sob for the VM that died, and restart it.

I am wondering if hibernate causes it some grief and when the host hibernates, it leaves it in a state.

On the other hand...

I was using my machine, and noticed a horrendous paging slowdown. So bad it took me ages to see what was going on, and by then, it was game over. Firefox had died as had VirtualBox guest.

"dmesg" showed the out of memory killer had struck.

I have a 2GB RAM machine (no swap), with typically 400MB given to firefox, and 800+MB to VirtualBox and the rest for X, window manager and the other very heavyweight apps that make up Ubuntu and KDE.

But why? I dont know. I do know an XP guest, left unattended will drum into action, presumably as the windows updater fires at 24h intervals and probably caused the guest to suck up RAM. I think VirtualBox creates unreserved memory - on a good day, the RAM is unallocated, until the point it needs it. A common idiom for Java JVMs. However this is a very antisocial thing to do: if memory is constricted, then the kernel OOM killer can strike just when you dont want or expect it.

Maybe theres an option to stop this from happening or maybe I could just write a guest RAM hog app to force allocation up front. I dont mind firefox dying - its not a showstopper to restart it, but crashing the Windows XP guest *is*. Its very time consuming to restart it.

I've added 400MB swap to slow down this effect, and I do need a laptop which can take more than 2GB of RAM. I have my eyes set on an 8GB RAM baby (good price, but out of date processor - Dell Vostro 1720). No one is doing the 1920x1200 screens I want. C'est la vie.

I need to dive in VirtualBox internals a little more and/or update my version - the Ubuntu apt-get release is out of date, and maybe I should just download or compile my own - but I know that can be painful.

Maybe I need to track RAM better or run dtrace or ...

I had to go away for a week, got back to find my broadband wasnt working - waited three days for engineer to come out.

I recently switched to using VirtualBox on my main 17 inch laptop (1920x1200 Inspiron 9400). I had upgraded the Ubuntu distro and got fed up having to patch the vmware 1.0.8 source drivers to make it work and compile. VirtualBox has matured, and seems really good, now I am using it daily. I havent done any benchmarks - I *think* it feels more sluggish but I am not sure if this is the video rendering or the raw cpu. Its not enough to bother me.

What *is* very curious is that VMWare 1.0.x was hugely disk intensive. I dont know why - its an old release of vmware, and there are newer releases (VMWare server 2.x, VMWare workstation, etc). What vmware 1.x seemed to do was discard the page tables (an optimisation?) If the VM isnt used in a while (or, maybe after resuming from RAM suspend for the host), then theres a huge amount of paging to bring the VM back into memory - taking ages. My 2GB RAM laptop (maxed out), with about 400MB for firefox, hundreds of megabytes for X server + KDE desktop, and my fcterm's, and 700MB RAM allotment for XP guest, is just too much.

So, I am running VirtualBox - and the disk light hardly ever comes on - whether I am booting windows, or returning after a resume, or after doing lots of stuff outside the guest. I *think* a problem with VMware was mmapp()ing the RAM memory - good in case of crashes (vmware or host crashes), as you have an intact VM, but I think it stressed the kernel to ensure the dirty pages were written to disk. I havent checked if VirtualBox is doing this - but disk activity is on a par with what I am doing, rather than VirtualBox trying to be comfy on the sofa.

This is encouraging - I thought vmware 1.x was totally fine for me, but now I have learnt something.

....

Which brings me on to the 1920x1200 screens. Apple's new iMacs with the 2560x1440 27 inch screen are fabulous. As a developer, you cannot have enough pixels (despite my eyesight not being what it was). More pixels is better than a faster CPU or huge amounts of RAM.

My last laptop (Inspiron 9400) is maxed out - Dual Core (32-bit cpu) 1.6GHz, 2GB RAM, 500GB hard drive, 1920x1200. Some of DELLs machines are good value for money and the WUXGA screens are brilliant.

Now that we are being flooded with Intel i3/i5/i7's, everything new is 1080p screen size. Most laptops are fitted with pathetic screen resolutions (1600x900, I mean, come on! What drugs are people on !). 1600x900 or the other sizes are brilliant if you have a less than 12 inch screen, but on a 17 or larger screen?

Browsing google suggests we wont see 1920x1200 screens any more - the bang for buck in the market is home entertainment, HD video, etc and 1080p screens are "it". None of DELLs newest laptops have WUXGA support - only the older Vostro/Inspirons maxing out on dual core duos.

(I use DELL as a metric - there are so many competitors and keeping up with everything is time consuming, but, with DELL, you can, if you study there site and outlet page, work out where they are overpricing; their add-ons are extortionate, e.g. today, 60 GBP to go from a 320GB drive to a 500GB drive, when drives cost that on their own).

I am not meaning to be a DELL fanboy, but they are consistent in their delivery - in the area I care about, until now.

Anyway, so todays ideal machine is: 1920x1200, quad-core, 8GB RAM, 500+GB disk. Screen size is most important, and I waver between whether the cpu or the RAM is more important. To get that screen size from DELL means a trade off (DELL Vostro 1720) - vastly overpriced dual core duo's, but very reasonable 8GB price. Remember, today (Jan 2010) Core i3/i5/i7's are coming out very price competitive. Prices in the range 100GBP to 200GBP). So, for the Vostro above, I can upgrade from a 2.26GHz Core Duo P7570 (1066MHz bus, 3MB cache), to T9550 (2.66GHz, 1066MHz bus, 6MB cache), for 160 GBP. What is that - 30% performance enhancement, if I was lucky, for the same price as a quad core chip? Its even worse if you pick other starting points on the website, hitting 250+GBP for a tiny performance increase.

CPU speeds have flattened out, and its good *NOT* to be on the cutting edge. If you are, you have nowhere else to go. My 1.6GHz 32-bit chip will see a near doubling of performance when I upgrade, and by picking last years model, the cost-benefit is in my favor. (The new DELL Studio range are attactively priced, but that darn screen is limited to 1920x1080, and I miss my pixels!)

The issue is submitting a form. When I ran the form submits from 'elinks' I could see a "302" being returned on the form submission, yet, using WWW::Mechanize, I got a "200" response. Not sure I understand what WWW::Mechanize is doing on a form submission, and some strange cookie behavior.

Anyhow, I got it to work by rethinking the problem (using wget on appropriately formed URLs to the site; sort of "not ideal", but actually "pretty good", because now I can get all the channels - too many in fact, but thats better than coding up a form filler to pick the TV channels I am interested in).

You can see the results by clicking here:

http://www.crisp.demon.co.uk/tv/film.html

This is for my personal benefit and no guarantee this will be updated at all frequently, but I can use my iPod or iPhone to see whats on in one hit.

BTW am stilling griping about the ipod touch. My ipod touch freezes and needs a hard reboot maybe twice a day depending on what films I watch. I am certain its something in the film decoders which can crash the ipod or cause severe memory leaks. Even tilting the device stops, and I get the circulating "busy" icon (if I am lucky). Wish they would put better software fixes out.

The amount of info is quite staggering, and its useful to have custom views of their data.

I put together a script which can show upcoming new seasons of programs, and film summaries (along with ratings), sorted by rating.

I am just playing with WWW::Mechanize, in a perl script, to get the cookies set up so that the Sky Movies show up. (By default, the tv guide shows a reasonable selection of channels, but you need to customize the views to show what region and vendor you are using; of course, this uses cookies, so am experimenting with how to get the settings correct for the tv view).

Hopefully wont take too long. If anyone is interested in the script, I will post it up on my website.

I upgraded my laptop to Ubuntu 9.10; of course, vmware stopped working. I started building a new kernel and applying patches to vmware, but thought I would just switch to virtualbox. After a few hiccups, my Windows XP VMware image is running fine - screen and networking. (I had to install an Intel driver before the system would see it had a network). Its running nicely. The resizable video display is nice.

CRiSP contains facilities for viewing CSV type files - but now its time to ramp up the facility a bit. The feature in Excel which allows you to do pivotal views - is nicely done.

What is a Pivot? In Unix terminology, a pivot is typically done via a series of grep commands piped into grep commands. This works to find lines matching or not matching a criteria, but isnt ideal for selecting columns:

$ grep pattern1 file | grep -v pattern2 | grep pattern3

Once you start to take columns into account, the above command line gets more tricky, and "the optimal way" to do it, gets muddy. E.g. you can use the cut(1) command, or awk, or perl.

CRiSP contains the powerful g// command (at the Command: prompt, use g/pattern/ to show all lines matching a pattern). Once armed with this list, you can further refine by filtering (<F>) to include or exclude certain lines.

Recent releases of CRiSP have included the command prompt 'col' command (type "col help" to see the commands available).

But the next step is to provide an Excel-like view (which already exists in the Find/Data-Miner edit option (this needs to be renamed and moved to the View or Tools menu).

The next evolution of this facility is a dropdown on each column, which shows the unique values. This, in itself is useful and interesting, but, with the addition of being able to enable or disable rows which match the column values, means, you get a simple SQL like facility for drilling down and summarising data.

Lets see how it evolves, before making it visible in the finished product.

Anyway, in the spam is a bogus link to a banking website.

I am wondering, why we dont just bleed them dry. I know some people take a vigilante crusade on these people, but, if we all put in motion a "voting" scheme to vote down websites, e.g. a plugin to the browser/mailreader then you could just cause a mass blockage to a website by "clicking" on a link.

Maybe even the web browser should consult a white/black/dont-know list, and unless its been certified, it blocks access for you, on your behalf.

It could even look at the registration details of the site.

I am probably over simplifying the kinds of tactics one can do, or, have been done by people using honeypots etc.

$ strace <application>
[ Process PID=3D17199 runs in 32 bit mode. ]
...

switch (call & 0xffff) {
        /* x86-64: syscall = 0x0f 0x05 */
        case 0x050f: currpers = 0; break;

        /* i386: int 0x80 = 0xcd 0x80 */
        case 0x80cd: currpers = 1; break;

        default:
                currpers = current_personality;
                fprintf(stderr,
                        "Unknown syscall opcode (0x%04X) while "
                        "detecting personality of process "
                        "PID=%d\n", (int)call, pid);
                break;
}

What does this mean? Well, it agrees with my prior findings in dtrace, that on Linux, determining 32 or 64-bit ness is difficult because the kernel does not contain such an attribute. Strace is having to guess, and because it guesses, it can be lied to.

I havent tried this, but one could do an INT 0x80 from a 64-bit process (either we crash the kernel, or we succeed) but this would confuse strace. We could arrange to overwrite the instruction of the syscall (eg from another app or thread), and this too could cause strace to be defeated.

(You can make an application totally untracable, by forking your own tracer, which would preclude ptrace from another process from working).

In reality, the kernel knows what a binary type is via the 'personality' - effectively an object vmethod which is setup by virtue of execing the executable. (Shared libs are a problem, since they are just blobs of bytes, setup via an mmap arrangement, and could be anything - x86 32/64 bit, PPC or ARM, etc). So, an app can flip from 32 bit to 64 bit and back again, solely by some twiddling of what the PC points to.

(I am probably wrong on the latter points; you cannot suddenly flip the CPU from 32 to 64 and back again, without a protection level transition; but, a 64-bit app can execute a 32-bit instruction. Maybe I will have to try out this thought experiment and see what happens).

I have a pile of 'Dtrace wont compile' emails in my inbox. At least one of them looks to be due to the fact the sender hasnt got the headers installed on his system .. an enhancement that is needed to the build script.

Heres one email, which just surprised me and made my day: partially because of the kind complement, but moreso, because I have *nothing* to do or follow up on:

Thank you so very much for your DTrace porting efforts..

Its pretty darn awesome. :)

I dont mind being wrong, sometimes .... :-)

This is driving me nuts. GCC does tail recursion optimisation. That is very nice, and means that if we have something like:

int func(...)
{
	do-stuff();

	func2(...);
}

The problem is that this means that if we put a breakpoint in gdb, or a printf, in the func2, we lose the stack frame for func(), and it appears that func2 is being called from the caller of func(), rather than func() itself.

I wish they wouldnt do these "nice" things. Makes debugging a pain, and am tempted to go back to earlier/very old versions of GCC to stop this warfare, where "what used to work", stops working and you have to fight the toolchain.

For example, the 'Scale' widget which is used in the color selector dialog needed to be implemented (from scratch). That took about a day (nice when things go according to plan).

Then I hit some issues with the default size of a combo field. That is fixed.

Next up was the protocol manager for a shell widget. What is that you say? Think of XmAddWMProtocolCallback(). Without this, if you click on the window manager "X" at the top right of a window, then your app is quickly terminated (the TCP connection to the X server is severed).

Took me a while to figure out / remember how this all worked. But, suffice to say, that unless you post an Atom/Property on the window (WM_DELETE_WINDOW), then the window manager will not ask politely, but just brute force the termination of the app.

But to put a property on the application shell window is not quite so easy, especially when we normally call the XmAddWMProtocolCallback() against a *widget* and not a *window*. A widget may not exist on screen at the time we call it, and that is why there is complexity in the Motif library - you are allowed to register interest in these window manager protocol messages, before the window is 'realised' (i.e. before being mapped to the screen). When the window is mapped, the appropriate property is posted on the desktop to allow the window manager to see what is going on.

Of course, if you try to do something "later" in an application means creating some form of data structure for later use, and then, making sure we dont suffer a memory leak.

My implementation of window manager protocols isnt perfect, but sufficient for what I need.

Why do I bother? I dont know, but what I know is that CRiSP with Motif, statically linked, occupies 3MB of code memory. With the new Motif replacement, it is about 2MB.

When CRiSP was first written this was about the size of a large floppy disk (and 40MB - thats megabytes, not gigabytes) was huge. Now the size of CRiSP fits in the L2/L3 cache of a cpu.

But the menuing code has taken about 3 months - I hope this is now complete. Why?

Any number of reasons:

• I am losing my 'touch'
• It is difficult
• There are lots of fiddly bits to get right

You choose. The issue with menus is the way input focus moves around from one widget to another. There are lots of scenarios to get right, and fixing one of them, would result in some existing feature suddenly breaking - like a see-saw as the code came together.

One problem I hit is that XtAddGrab/XtRemoveGrab doesnt handle double registration of a widget particularly well.

The code, whilst trying to be purist and object oriented, in the end had to be a little dirty - one class having too much intimate knowledge of what it is dealing with (a menu has menu items, which is mostly separators and buttons, for instance).

Heres some scenarios to consider:

• Click and reclick the menu bar button (should dismiss menu)
• Use keyboard to navigate a menu and popup sub menus, and then popdown submenus.
• Use ESC to popdown a menu
• Click outside the menu to dismiss it; click in another window to dismiss it too
• Click on a menu item, and have the menu popdown, and the callback invoked

May seem like simple stuff, but getting it all working is difficult, especially when grabs are put in place - suddenly input goes to the wrong widget, and everything you had working, stops working.

Why bother ? Why not just use Qt or Gtk ?

Because I dont want to use them. As good as those toolkits are, they are not available everywhere, and I dont want dependencies on other toolkits - toolkits which have a very active development life.

This is akin to the dtrace problem: do you develop software for the latest and greatest kernel/distro out there, or do you go back to old releases and ensure your software works with them?

Implementing menus is frustrating. (The input field widget took about 2 days to implement from start to finish, and the result is more functional than what it replaces). By contrast, menus have taken a couple of months. Much of the original code worked u front.

What makes menus tricky are a number of things but mainly related to input focus and 'appearing to do the right thing'. Menus can popup with a mouse, and then be navigated into sub menus with the mouse or keyboard. I delayed getting the input focus problems solved until late in the implementation, which, in turn, broke much of the existing logic. (Events firing to the menu button popping up the menu or the menu or menu items, etc). Just as bad was restoring the status quo when selecting a menu item or dismissing the menu.

To make life palatable, I have a nice suicide-timer - after 25s, a child does a "kill -9" on the parent, which avoids the tedium of trying to unwedge the X server.

Its all very close, and am just busy ensuring all combinations work properly. (Mainly nested menus as they popup/get dismised).

Many of the problems are my own misunderstandings: I spent a good few days trying to get XGrabPointer() to work for me, only to eventually realise what I needed was XtAddGrab(). [Someone on a forum mentioned that if you weren't confused by XGrabPointer/XGrabPointer, then you didnt know what you were doing!]

XGrabPointer is to do with freezing the X server, typically used for drawing type apps. For menus, you sort of need a mixture of the two. (You want to intercept a menu dismissal when you click outside of your own application).

(I was debating using the iPod Touch to fix my X server hanging problems, e.g. have a web server running, which responded to trivial HTTP requests, and on receipt of a request, could douse the rogue hung application).

Ok, here it is - an ipod touch web server to help debug X11 apps...

#! /usr/bin/env perl                                                          
                                                                              
use strict;                                                                   
use warnings;                                                                 
                                                                              
use IO::Socket;                                                               
                                                                              
sub main                                                                      
{                                                                             
   my $sock = IO::Socket::INET->new (                                    
      LocalPort => 8080,                                                 
      Type      => SOCK_STREAM,                                          
      ReuseAddr => 1,                                                    
      Listen    => 10);
   while (my $client = $sock->accept()) {                                
           print "Killing hung app...\n";                                
           my $str = `ps ax | grep /home/fox/crisp_v9.5/bin.linux-x86_32/crisp`;
           chomp($str);
           next if !$str;
           $str =~ m/^ *(\d+) /;
           next if !defined($1);
           kill(-9, $1);
   }
}
main();
0;

Some people are reporting issues on GCC 4.4 and later glibc's. If you get these issues, let me know. glibc seems determined to break existing apps (GCC isnt quite so bad).

User space stack tracing is probably hosed because getting the user space stack, depending on the trap context is fiddly and something I hadnt finished.

Need to catch up on my crisp Motif work (it all works, but its not quite perfect enough to release the crisp code yet).

As a BTW, I decided to implement 'inertial scrolling' on fcterm and crisp. So, you may find (eventually) a new release with the feature where the faster you scroll the wheel on the mouse, the faster it scrolls. (Works nicely on fcterm, but may need to do more surgery on crisp since it only works for X windows, and needs to work for Mac+Windows too).

Sometimes, my movie playlists will play back to back, but mostly not. Its totally erratic, and simply feels like a 1st grader programming error. Its almost like an uninitialised variable is causing it to work or not. Sometimes switching it on/off helps, or a resync, but who knows.

Then, the other area of total insanity is the app store. The app store integration is brilliant - ability to browse, and waste a few minutes seeing whats available is great. Then you can download free software or pay for software.

But...why on earth does the ipod go into a mode where *no* applications will run? The startup screens almost show but then the app aborts or exits. Who knows what - because Apple decided not to show you a reason why the app quit. Then, some time later, they will all work again. This is clearly a bug in Apples firmware. I have downloaded maybe 15-20 apps, and I can believe some have bugs, but not on the initialisation of the app. Why it should go into "not going to run anything" mode and then recover, I dont know.

I downloaded an RSS reader - great software, but I am fearful it creates a lot of little files for the unread news - maybe thats tickling a bug; maybe the filesystem is fragmented - you cannot see the filesystem so you are SOL to know whats going on.

What is annoying is that lots of people on the web report the same thing and as far as I can tell, noone has clued up on the real causes, and Apple doesnt admit to this issue. Their 'walled garden' is full of street-corner muggers.

At least it plays movies and music - my main desire, but the app crashing is unforgivable. At least, some form of diagnostic tool to figure out what would be nice. Maybe thats a tool that needs to be written, or maybe I will find out on google somewhere what is going on.

It was the equivalent of an iPod, in its day.

Now armed with an ipod touch, and visiting the ipod store and getting a feel for this new device, I came across an HP48 calculator emulator for the ipod.

Lets go back to 1992. In 1992, I wrote an emulator for the HP48 - made available over Usenet, and, it worked - it could do most HP things. It was based around X11, and the PCs of the day, were very lowly, but, still, the calculator emulator was fast enough.

I still have the source code to this. I believe this code was taken and used as the basis for other calc emulators, and there are now excellent emulators for Windows (havent checked Linux, but should be easy).

Now, lets go back to 2009. The App store version of the emulator is great - full power of an advanced RPN graphing and symbolic algebra calc. The pixels on the ipod are just enough to do just, but only just.

This app is available as source code. See here for the announcement of the i48 app Github entry for the source is http://github.com/dparnell/i48/tree

This is interesting for two reasons. One, this is a simple iPod app, and hence, is a useful example of how to create an app. Complete with XML files and bitmaps for the app store. (Interestingly, the Objective-C code is tiny, which is a testament to a good api on the ipod touch).

The other interesting point is that the emulator code bears the hallmark of my original donation to the community. I dont know if this code was based on my original or not - I started to lose interest in the HP as CRiSP took more of my time in the early 90's, and was vaguely aware of a Windows port (bear in mind this would be Windows 3.x or Windows 95, which I detested beyond belief - you needed to reboot the system if any app crashed if you wanted a nice life).

The attributions in the code date back to 1994, but there are signs of similarities. I cannot remember how I got started on the emulation. I know I picked up a possible HP28 (predecessor to the HP48) emulator and got some internal docs from HP at the time (I probably still have the emails dating back then), but its nice to know this excellent piece of hardware and the emulators live on - and now, I can carry on with carrying an HP + IPod together.

Ob complaint: why is the HP50 so expensive in the UK? Its more than 100 GBP vs about $90-100 in the US. Needless to say, HP havent received any money from me because of this obscene pricing, and I suspect, the number sold in the UK is pitiful, which is a shame.

So, what next? Who knows. I would like to write something for the ipod, but I dont have time, and theres a lot of ideas to wade thru from the existing app base on the app store. I just wish the ipod wasnt so locked down (see complaints in the prior blog).