CRiSP Weblog

Ive had my eye on the DELL Inspiron 9300 for a good while now and the price of the machines are very good at keen - for a 1920x1200 display, but am fairly certain that come the new year, the line will be refreshed with a dual core Intel chip (since, alas they arent doing AMD .. yet?)

Of course - I don't need such a beast, except my current notebook has had a dodgy <F1> key for ages now and its pretty worn out.

I have had a dual cpu machine before, and its difficult to be able to use all the oomph in such a machine for the things I do, but a new upto date one may help with advancing the multithreaded version of CRiSP.

Current CRiSP releases have undocumented multithreaded macro support, but its only usable in a limited manner - more rearchitecting needs to be considered, and it will take a long time to be able to use it in a general manner whilst at the same time preserving reliability and dependability.

Having multiple threads can be helpful in some circumstances, e.g. for the File/Contents window view so that the user doesn't suffer and delays on slow networks or large folders.

I believe I am seeing much mail directed at me but cannot be certain.

But some of my mail replies are bouncing so cannot track who can or cannot see me.

I have been contemplated a move away from the crisp.demon.co.uk email address, since I am paying over the odds for a mail host service and ISP support, when I only use the mail feature to preserve the mail address. This would be painful to change, so am considering the options, but probably nothing will change for at least a year until I get my head around all the permutations needed to be reachable.

However, I hit a hiccup which was impacting the Windows release - namely missing compiled macros, so this feature had to be disabled.

CRiSP on Windows is distributed using the WISE install mechanism, but the config files are derived from the Unix version - and avoids having to keep two sets of configs in sync with each other.

Either I need to replace WISE with my own Win32 applet or figure out how to change the packagaing mechanism. (Getting rid of WISE is the best thing, but that means debugging a brand new Win32 app).

Oh well...

Just a quick update - Emacs has a macro which provides an advanced form of templating by using the file you are editing as the source of the template, by using certain comments to highlight how to expand features. Verilog requires a lot of duplication of definitions, and this is very tiring if done by hand.

In looking at the macro and trying to figure out what actually happens, showed up some issues, because I am not a Verilog expert and wasn't sure what i found was due to errors or misreadings. I am going to try again to move this forward.

So if anyone has an interest in Verilog and wants to know more or help out, then feel free to mail me.

First, it had Japanese Windows XP installed - kind of tricky to do anything with when you cannot even read the menus. They gave me an image of Windows XP - so I saved that off...

And installed Knoppix. Brilliant - it understands all the hardware - except for a few problems. The wireless ethernet even worked.

But the screen, which is 1280x768 was only being treated as 1024x768. After a bit of searching the web, and a lot of experimentation, I can now run in the full screen mode and the screen is beautiful.

Next - VMWare. Alas Workstation 4.5.2 failed to install, because the Linux kernel (2,6,11) has changed so VMWare breaks. This is a pain and one of the biggest annoyances of the kernel development process. Fortunately, google comes to the rescue and it didnt take long to get that going.

I installed kernel 2.6.12.5, but have experienced some problems with that - possibly due to KNOPPIX enabling too many drivers and options, and pruned them back. Looks like the SMP option must be turned off.

I can now run freebsd or Solaris X86 inside the VMWare session, and next is to try the original WindowsXP install so I regain some lost functionality *playing DVDs and CRiSP development).

The machine is lightweight enough to carry around and battery life seems good, although it wants to turn the fan on a lot if you do anything.

But then the CRiSP bug reports came in about menus not working, and no matter what I did, I couldnt reproduce them.

Eventually, one kind customer let me VNC into his system to see for myself, and after poking around a bit, found HE was running 10.4 but I wasnt!

So - that helped tremendously.

Unfortunately, Apple, in their great wisdom decided to make 10.4 incompatible with 10.3. At the .3/.4 part of an OS's life, you would think they would cherish application compatibility. Alas not.

I note in 10.4 there is yet another font/typography library to creep out of the woodwork. Maybe this one works and is usable.

Just one bug to go in CRiSP for Macs -- why the shift/ctrl/alt key in input fields moves the focus.

Recently, I have been doing more and more perl work, and find that the switch between CRiSP expressions and Perl ones a bit of a nuisance.

To rectify this, I have added support for pure Perl expressions - these mainly affect three aspects of expressions - (..), $1..$9.

The next release contains support for these expression types, by selecting the new Perl mode of syntax.

Dual cpus/64-bit cores will be the thing to buy in the future, especially when the price comes down. Its pretty amazing that Intel advertises the silly blue men and the Intel inside logo without ever stating what these things will achieve .. instead of a poultry 10-30% cpu enhancement, they will be upwards of 40-50%.

Anyone who doesnt really understand the technology will undoubtedly be better off with it, so long as the price differential is realistic.

With the world settling on Linux + Windows + MacOS as the preferred platforms, with the other BSDs and Solaris relegated to third world status, sanity is beginning to come to the platform world.

I have to say the quality of the BSDs and Solaris is impeccable but they are marketed badly. Solaris seems to be in tough world trying to justify its existence based on technology, when most people in the world want color, 3D, sound and hi-quality audio.

Off to fix more bugs...

The undo code is very old - dating back to the earliest versions of CRiSP (pre v1), and for over a decade has proved remarkably stable.

However, with disks getting larger, and 64-bit cpus becoming more common, the undo info might not fit into a 4GB file and changes were required to make the file supported more than 4GB's worth of data.

In expanding the file to handle more data, the overhead in the file increased, and forced some recoding to support a more modern style, which in turn caused a number of errors to be introduced. Not only that, I had forgotten some of the premises of the code, so silly things broke, and I hadnt realised the number of "key" features which needed to be verified.

Hopefully this is all working now. There remain more optimisations for pathological cases which could be done, but at present the code is dominated by the performance of very annoying pieces of code.

Maybe these will be addressed in the future.

Remember, CRiSP isn't just your product but mine also - its a test bed for software engineering solutions.

I hope to talk about more internal details shortly.

The problem is the OS's. Excluding Win32 apps, many software packages are 64-bit ready. CRiSP has been for around 10+ years starting with the DEC Alpha and SGI Irix.

But now that platforms are more prolific, its difficult to support these platforms.

Why?

Well consider Linux. We have linux/32 and linux/64. CRiSP runs fine on an 8MB machine and a 4GB/32-bit address space is oodles. So why would one want to run a 64-bit version of CRiSP?

Well for a start, it will be faster (more cpu registers, internal 64-bit arithmetic etc). But it may not be noticably faster (maybe 10-40% faster), so there is little merit in offering 64-bit CRiSP across every combination of platforms - at the moment.

As Windows 64 takes off and puts 32-bit cpus out for the trash, then we can live in a more pure 64-bit world. At the same time Linux/64 would be the preferred OS to multiboot or use inside a virtual machine (eg. VMWare, Xen, Virtual PC).

But as of this writing, Win64 is vapor-ware: you can get it, almost for free, but it voids the customers warranty. Who wants that?

If/when I buy PC's I buy them with the extremely horrible Microsoft $$$ license, and then install Linux on the box. The Windows comes in useful when the machine is eventually given away. I wouldn't want to buy a machine with Win/32 XP at this time because proper Win/64 is imminent, e.g. from suppliers like DELL.

Its a confusing time...

The trouble is, which machine and cpu to aim for? My current machines are not top of the line but more than sufficient. Having a dual-core chip is certainly desirable, but not if it generates too much heat and fan-noise.

64-bit is definitely desirable. I will never understand why, when the 80286 was introduced - a great CPU compared to the 8086, and then the 80386 came out, along with virtual memory, how they were so close and innovative. Yet it has taken 20 years for them to add 64-bit enhancements.

They are now talking about virtualisation technology in future CPUs. This was nearly there back in 1985, but they missed out a few key features which stopped it being done. The 80286 could support virtual 8086 DOS boxes and Windows and other "extenders" took advantage of that.

In recent years, it took VMware to bring the importance of this technology to the mass audience. A brilliant product, which everyone has fallen over themselves to catch up with.

Now we have XEN (Cambridge Labs) and a raft of open source cpu emulators letting you run multiple operating systems on one machine.

But virtualisation could be a pig. If the open source community or a company like VMware owns the inner ring technology, then you - the customer, can do what you want with your PC.

But say Microsoft decides virtualisation is key to some future feature in Windows, say, sandboxed Internet Explorer, or DRM. Then VMware or XEN wouldnt get a look in - they would need to create a hyper-hyper-visor to allow, say, Linux and MS Windows to run at the same time.

An OS vendor should have nothing to do with hyper-visors, as it spells lock in for the customer. A 3rd party should have control of this, and I fear the worst.

Hopefully AMD will track Intel and in a year or two, we will all be buying dual core 64-bit chips with hypervisor technology that doesnt use too much energy...

CRiSP works well for reading these files, but as soon as you do a "Save As" operation, then it was downhill. A patient user had let it run for more than 24 hours trying to save a 16GB file...when it finally ran out of memory.

This was under Windows XP. Of course, Linux is superior here :-) It ran out of memory in minutes instead of days :-)

The problem was to do with exponential behaviour in saving and buffering the files and the undo information recorded for the files. Having looked closely at the situation, some small tweaks and performance was much more linear.

Looking further, there were some pathological conditions where it would try and store too much in memory, causing heavy swapping or process termination. (Linux's out-of-memory killer is a severely misdesigned non-feature).

After further work, it looks like the situation is under control - attempting to edit and save, and resave large files works as expected - its still slow, but thats purely due to the amount of disk I/O and is a function of the file size and memory on your system.

I am hoping to experiment with some additional features to enhance performance and memory/disk usage next.

This is an interesting piece of work because its a kind of templating, but unlike the other types of templates in CRiSP, it is based on the buffer you are editing - you put keywords in the code (inside comments) and it can be used to replicate redundant grammatical information.

One of the things that is unclear is that Verilog, like VHDL, is designed for engineers, and they are both unforgiving languages, using strange constructs to represent natural parallelism.

I have look at this Emacs mode in the past and given up on it for a number of reasons: not understanding it, because I am not a Verilog user it is difficult to support and maintain, because the original Emacs code is GPL'ed. (I have nothing against GPL licenses but they are in another world when looking at commercial software).

I decided to bite the bullet again and look to understand this, and am now making headway into its implementation and what it does. Like all good macros, it is based on magic ... until you understand what it actually does and then you realise it is based on a series of tricks (that is not to undermine it in anyway). Once you understand the tricks, e.g. partial parsing of the code, then its a lot easier.

Most 'clever' algorithms are like this - complex until you understand them.

I might get to try it out one day if I didnt have to register with the Sun site for the billionth time in the last ten years.

Why on earth such a technology savvy company cannot figure out who you are without me having to remember yet another darn silly login.

I suspect Sun are aiming for more registrations than people on earth.

Oh well, maybe I'll try Solaris 11...

Grab yourself a mobile or an answering machine and reister with your companies equivalent of an 0900 number (you know, the ones where you pay $1 or 1 GBP or more minute).

Now sign up on as many web sites as you can with that number.

Now when cold calls come thru, YOU get paid and you employ delaying tactics to stop the callers.

Summary: End of cold calling...and you are rich...

Have a nice day

I knew about the new Apple announcements today, and didn't think they could impress or surprise me.

The Apple mini is a gem - small in size, quiet (I hope), and cheap/affordable, especially as an 'extra' machine.

When I have browsed the net for new interesting machines, I see lots of nice subnotebooks, and anything stamped with SONY seems to be overpriced - a conscious buying decision.

But Apple have done something to trounce SONY and Microsoft - a cheap, powerful, machine with no operating system tax. As a spare file server, Mac/Unix server, or however you look at it, its a treat.

Now I just need to decide if I really need one (yet)!

But Pocket IE is still pretty pathetic - its sluggish on a 624MHz ARM CPU. Like its Windows desktop cousin, its lacking in so many features its unbelievable. Things like being able to scale the fonts better than "Small, Smaller, Big, Bigger". Ability to turn off images would be good. Or the ability to switch landscape/portrait for this app only, rather than for all apps.

Thank heavens that Lister (see http://www.crisp.demon.co.uk for a copy) still works. It needs some slight reworking as I think of new features.

If you havent seen Lister, its a glorified "more" utilty for browsing text files, but it supports .html.gz files (although only a subset of HTML). I use this to download newsgroups and read on the move.

I need to get my scrabble game going for the odd journeys.

My AXIM came with two games - a car racing game which looks good but like all car games, I find incredibly boring. And a brain teaser app that looks nice but I havent gotten into it yet.

The wireless capabilities are very satisfactory at present - allowing me to browse the net from anywhere in the house.

Everything else looks typically Microsoft - looks nice but is definitely 'missing' something, (see IE above!).

The 640x480 screen means I can use smaller fonts and use the ClearType facility (which previously I would turn off), as it seems to work well on this screen.

I downloaded eVC++ 4.0 so I can rebuild Lister and the other tools, and it seems to work fine - especially as its free.

Alas, blew up my old 1GB Microdrive trying to extract it from the old PDA. I put a 64MB spare flash card in the new PDA and its significantly faster than the old microdrive. Hadnt realised how slow that was.

And no, CRiSP isnt coming to a PDA near you. Mind you it did use to run on an old HP100LX (512KB RAM!). Once, and it took 20seconds to startup on a 12MHz 80286 (I think thats what it was).

So far the challenges have been pretty straightforward - code cleanups, avoiding static/global vars so that the threads don't step on each others toes.

Mutual exclusion is a pain though. Linux/GCC provides recursive mutexes, which allow subroutines to apply locks and those routines to call sub-subroutines which increment the same locks. Unfortunately these locks are GNU extensions and not standard POSIX, which implies when the port is done to other OS's, chances are high I will be left high and dry.

Since I am aware of this I can cope/code for it, but I'm getting to the point where I need to track down more race conditions in the code.

I have a test macro which just runs a lot of primitives in the background. It misses a lot of issues which will need addressing, such as locks on buffers, GUI objects and other issues. For now I want to just concentrate on getting it running under duress.

Here's a pre-release of the thread mechanisms in case you want to see what it looks like:

# include	

void xx()
{	int	tid;
	int ret = pthread_create(tid, 0, "::thread", 0);
}
static void
thread(int arg)
{	int	i;
	int	zero;
	string	s;
	list	lst;

	for (i = 0; i < 10000000; i++) {
		message("%phello %d", i);
		int dh = opendir(".");
		remove("bye bye");

		while ((s = readdir(dh)) != "") {
			s += "hello-again";
			s = gsub("e", "", s);
//			re_search(SF_UNIX, "zzz\\z.*xxxxx.*");
			lst[0] = simplify_filename(s);
			open(s, O_RDONLY);
			switch (s) {
			  case 1:
			  case "x":
			  	break;
			  }
			}
		i += zero * sizeof(lst);
		closedir(dh);
		}
}

GNU C supports the "__thread" declaration specifier, which allows you to have a global variable which is unique on a per thread basis. This is nearly identical to Microsofts "declspec(thread)" syntax.

Except it doesn't work on earlier GCC releases. And requires an appropriate release of GLIBC. This means that with CRiSP support Linux GLIBC 2.1 and 2.2, we would have to potentially support 2.1+2.2+2.3 with/without proper working threads support.

Linux does have POSIX pthread support, so thats a start, but adapting a process to multithreadness using __thread is a very powerful option. The alternative is lots of source code modifications.

I took a look at Perl's source code to see how they do it, and it showed me the way to get maximum portability (passing a thread context structure from one function to the next). It too, does not rely on thread local storage. Its just messier and potentially a performance hit (size and speed).

I am disappointed that Linux thread local storage is so poor and cannot be relied on.

Happy new year