This showed up some issues, where symbols had gone private and inaccessible (or maybe this was due to cleanups in the compiler toolkit). My laptop is only 32bit, so, it concerns me that my 64-bit VMware Ubuntu wasnt showing the same issues.

This simply goes to show that even staying still is hard work - every time a new kernel comes out, working on the compilation issues (eg missing prototypes), or linker (undefined symbols) is a nuisance. With new kernel releases every 3 months or so (more frequently for RC releases), its a fair amount of work just playing catchup.

This goes to show that having dependencies in software, is a nuisance, and not a positive benefit. The only way around this is integration into the mainline kernel (I can't see that ever happening, because it would spoil peoples hacking fun, rather than a legal impossibility).

I've fixed a number of build issues, especially on 32-bit, and now the USDT simple examples build correctly.

I need to go back to the /proc/pid/ctl driver interface so we can see why dtracing a USDT executable doesnt work.

All was brilliant: mirrored the data to the new drive via a fast USB caddy (50GB in under an hour). Swapped drives over, booted Ubuntu 8.10 32-bit, installed and it just worked.

My screen was recognized, wifi, sound worked.

But I dont like GNOME, the menus are bizarre (that maybe inheritance from my old $HOME settings).

I tried KDE4 - pretty bug too power hungry and the huge fonts drove me mad. And it was downhill from here.

WIFI wouldnt work from KDE but would from GNOME. 2 days to fix that. So called 'improvements' mean you need a Ph.D in Linux hackery (thank heaven for Google and Ubuntu forums!).

Eventually its all working to my taste (KDE3 - not too intrusive), but the fonts are either ugly or huge, and a waste of space.

I need to get back to dtrace because I was making good progress (exaggeration!) on the /proc/pid/ctl driver interface, and I was worried I might need to do regression compiling against all recent Linux kernels (hence, why the disk upgrade is a good idea).

I just found that the /proc/pid driver has no 'write()' interface; you can read, but you cannot write to any of the entries in /proc/pid.

So nearly close. Looking at the code in fs/proc/base.c, we can see the structure which is missing the ".write" field, so we could patch it and handle the writes, but I suspect this could be a nasty thing to patch and rely on more and more kernel implementation aspects which may change in the future.

This means I need a better way to do process control -- ptrace() is not very good, it doesn't handle threads properly (which then complicates user space).

Now I am working on a 2.6.27 kernel, I could examine the utrace() kernel interface.

fs/proc isnt dead yet, but I will need to think about what I am actually trying to do.

I found, the other day, that I was spending a lot of time trying to get:

task->ptrace

So, looks like RH have pulled their usual stunt of updating kernel and causing me confusion with cross-version updates.

My kernels are looking stale, 6 months on, and I need to update to the later release, to avoid slugging it out with code which has already notably mutated.

I have a number of machines I test or compile on: 32-bit laptop, 64-bit server, VMware (Ubuntu 7 64bit), which is good from a diversity point of view, but bad because I may be tackling issues which are no longer relevant on later platforms.

C'est la vie when you are trying to fight against the GPL.

By the way, to prove I have nothing against the GPL, I did release blog.pl as GPL. Its only 500+ lines of perl, and if people can improve it, great. There are much better bloggers out there, but this fits me real cosy. (Next step, is to redo the whole crisp website with this, vs the current "C" code site generator which is so old, that its no where near as good as the Perl stuff).

Back to ctl.c....

Someone at google.com is playing with dtrace and addressing issues I havent had time for (newer kernel support), and this is really great. (I wont mention by name, since he may want to stay in the background, but I will mention him if he is happy for this to be known). (I am attributing the changes in the Changes file, so he is there).

The good thing is that we get two sets of eyeballs looking at what I am doing, finding issues where I am blind, or looking at things which are further down my priority list.

Also, this is good because the few patches so far are making me think how to manage a group effort.

He is looking at using kgdb to help debug dtrace; I am looking at the /proc/pid/ctl driver to figure out how to map procfs(4) primitives in libdtrace/libproc into the Linux kernel; this is a rough road - the Linux ptrace() interface is my guide book, but its a delicate area of race conditions and hidden issues which can only show up on a heavily loaded active box.

So, don't think of this project as *my* project, but *our* project, and I will try and co-ordinate and be insightful, whilst accepting donations of code or suggestions etc.

Already one person has asked about having a dtrace backend to generate SystemTap or Dtrace stuff, but I cannot really comment on the feasibility of this. (Sounds like an interesting subproject, and viable).

What does worry me is the ifdef-itis creeping into the code as new kernels come out; the Linux kernel owes us nothing, and can change the code and data structures to their hearts content. This means we have to worry that dtrace may break on each release, and the interesting areas will be how to protect ourelves against the unknown.

For example, the ctl.c code needs (does it?) to patch the kernel, and looks for ASM fingerprints to find what it needs. This needs to be refactored into a userland build-time tool, which can then generate nice messages like "This kernel is unsupported because ...", rather than a bizarre entry in /var/log/messages.

I am not a photographer, but they are my pictures, and I wanted to spruce up the blog a little.

I am still thinking about adding a backend to allow comments or voting, but this is side tracking me from dtrace....

This means it is fast - instantaneously fast. Its not quite as full featured as nanoblogger, but it suits my exact use cases.

Its available here, under the GPL.

http://www.crisp.demon.co.uk/tools.html

Commenting isnt supported as yet, but I may get around to providing a tunnel for commenting (without acting as a SPAM-bait).

But, I am making such a big jump that my old articles are invalid, so I had to do a lot of digging and write a convertor.

In the end, it might be more functional - I am not sure.

What I do know is that building the site is slower, and very much worse, blog articles are stored as a file (good), but all files stored in there own YYYY/MM/DD dir (very very bad).

So, now I waste more disk space on this, its lots of wasted dirs and now updating the ftp site is made very slow as my ftp sync script has to trawl/validate the local + remote directories.

Added to which, its all written in bash. That would be a good thing if the lots of dirs issue above hadnt just made it unmanagable.

Looks like I am going to have to nuke/rewrite it.

I do try hard to get on with other peoples software, but sometimes, its things like this that make me wish I hadnt relied on it.

(Added to which, my limited global ftp space is limited, so its wasted that too).

Why homebrew? Because, like everyone else, I dont want to be a spamtrap. I gave up on sendmail years ago due to complexity and the desire to stop reading the source code to figure out what all the magic $x incantations are.

Now, I need to create a daemon to check my mail is going and mail me !

So, to Adam, Paul and the other Dtrace contributors, apologies. But now, my month old outgoing mail has gone :-)

I have spent a lot of time over last few weeks plotting how to do this, and now we have something that works.

We have to patch a running kernel, which is a nuisance, as it creates more kernel dependencies that can break in future kernels, along with 64 vs 32 bit kernels (dont run the current release on a 32b kernel).

The code is in drivers/dtrace/ctl.c, and is quite ugly. It needs to be generated from a Perl script rather than hand crafted. (Even better: auto generation from the running kernel).

We may need to patch the kernel more to handle SDT type probes and tight interception of process creation/destruction, but at least I have something to play with now.

Of course, the whole lot may be pointless.

Someone from Sun (Adam) suggested looking at the Apple dtrace code, but that relies heavily on the MACH kernel interface for process control, and not the classic ptrace() syscall or anything resembling SVR4. So, thats out. FreeBSD has its own BSD derived and enhanced ptrace() layer, so we cannot use that as a guide either.

In order to get USDT to work, we need to get a decent mechanism to handle proc debugging (start/stop). I created /dev/dtrace_ctl, but wasnt happy that this is a global driver rather than a PID specific driver.

So I have been studying the /proc filesystem code to see how to get a new entry into /proc/pid/ctl.

If I was modifying kernel source, this would be easy. But I am not.

So, we have to take the point of view of being a root kit: we need to intercept key parts of the proc driver so we can install a new entry in the static data structure (tgid_base_stuff, in file fs/proc/base.c).

This is a counted array, not a null terminated one, and there are two references in the kernel to this, in base.c, so we need to intercept those and put in our own table.

The goal here is to have /proc/pid/ctl, so the libproc.a code can issue SVR4 style PIOC procfs ioctls. Theres no guarantee that the difficulty I am making for myself will work, or I wont hit a bigger issue having tackled this specific issue. (Nobody said I had to 'do this right' !)

Patching the kernel is nasty: the kernel will evolve and dtrace wont work, but we can live with that: lets make it work on kernels which exist, and see what happens in the future.

I am going to do a small amount of code reorg (moving internals in fbt_linux.c to dtrace_linux.c, where I am trying to keep most of the general code).

Does the content of /proc exist, if you are not looking at it ?!

Theres two answers to the question:

Yes. /proc entries come and go as processes get created and die, and this would add a lot of overhead to proc creation/death and be a problem.

No. Its an illusion: when you look, e.g. via opendir()/getdents()/open(), then the entries are constructed for you.

Why does this matter? Well, in dtrace we need to track process creation (I am concentrating on /proc/pid/ctl - how to create a sub-entry in /proc, which exists/disappears, with the process it is tied to).

I am being thick: I am studying the code, and am nearly there in totally understanding it, but missing some 'glue' logic.

What it does is succinct and clever.

The "Quantum" nature of /proc can be exposed. I conjured up a test to try it out.

Look at /proc/$$/fd/0 -- stdin for the current process. What would happen if we did something like this:

$ sleep 10000 < /proc/1234/fd/0 &

and killed the process (in this example, pid 1234). What would happen to our sleep's fd/0 stdin? It doesnt show as a deleted file entry.

Go on. Try it.

I then thought about what would happen if a new proc 1234 is created. (I had to create a fork bomb (by accident!) - to get the pid in question reused).

It opened my eyes to a potential security hole (which, fortunately) does not exist.

But it is amazing the cleverness and subtlety of something we take for granted on a day to day basis...

Things have been hectic in the day job, but thats a partial excuse. I have been sweeping around trying to find other things to do before continuing with DTrace.

These other things include some minor updates to CRiSP, and lots of enhancements/fixes to fcterm. fcterm, as I have written before, is the fast terminal emulator around, but has specific features, like multiple tabs, infinite scrollback, and an ability to not-die if the X server crashes (actually, it retries to connect within 10s, useful if you need to bounce the X server, eg when uing something like Hummingbird X, rather than a native X server).

Anyway, back to the plot. The next phase is to get dtrace to run an ELF executable, e.g. a USDT conforming app, and this requires access to /proc/pid/ctl, which does not exist on Linux (the SVR4 procfs(4) filesystem).

We can either use the ptrace() syscall, or write our own, and avoid lots of changes to the userland dtrace libs+bins.

ptrace() has survived well for decades, despite some shortcomings, but it misses a crucial feature which Solaris truss(1) has had for requires: run-on-last-close. If you kill -9 a truss app, the truss'ed app will continue to run, and this is not the case for a ptrace()'ed app - it will be stuck in the STOPped state.

Fixing this isnt difficult in the kernel source, but we are not going to modify the kernel source.

So, /dev/dtrace_ctl has been borne to try and implement the functionality. I would like to implement /proc/pid/ctl, but havent found out how to hook into /proc as yet from outside the kernel, without having concerns on PID death affecting us. (I may be worrying unnecessarily).

I will try the dtrace_ctl driver approach and may retry the /proc approach too.

Now that the kernel stands up fine to running user space probes, I now need to figure out where are they, or what to do with them.

They dont show up in 'dtrace -l', but I suspect thats more of a misunderstanding on my part about what/how.

The kernel does have the PID provider code linked in (fasttrap.c and related files), but some stuff is still commented out.

Perusing the solaris kernel shows I have a number of things missing, such as static probes on process creation/exit, and other subsystems. This will require some hacking to get working, but I *dont think* is important just now. (Hacking means I am going to have to disassemble these functions and patch in static probes, unless I can find good hooks within the kernel which I can daisy chain onto).

Theres complexity in handling user programs since the dtrace libraries rely on the procfs way of manipulating applications and Linux has ptrace() along with the new fprobe() (?) interface.

I am tempted to create a procfs() driver for Linux to hide this complexity. I have never found the ptrace() syscall interface friendly for multithreaded apps, but then its very easy to make mistakes and take a good while to debug them.

Am just poking around at the moment. This is the final lynchpin of dtrace -- if this can be made to work then the rest is just quality control.

I got my first quantize() graph out of dtrace today (its always worked, just I had never gotten around to trying it out).

Any form of programming results in bugs - unexpected behavior. A driver is no different - but the bugs are harder to fathom because many of your favorite debugging aids cannot work in kernel space.

My solution, is printk(). Its very low tech but it works...

I am a fan of gdb and have written my own x86 kernel debuggers (for x386 and x186 processors), but porting them is a pain. Now we have 64-bit chips, I have to decide if I want to port my ancient code. (The debugger is powerful but nothing grand).

In a driver you have to crawl - one atom at a time; enabling huge wads of code and expecting it to work, well, er, wont. Disabling lots of code and ensuring structure is there and then a sprinkling of print statements works well.

I try to use vmware whilst debugging since bad pointers can corrupt filesystems, and losing hundreds of gigs of filesystem is not nice, waiting to fsck or reformat/reinstall.

Linux has a nice feature: GPFs (bad pointers) are caught and logs written to /var/log/messages. If you are lucky no reboot is needed.

Having a stripped down startup is essential - being able to reboot in about 10s is ideal - no waiting for GUI startups. (I use rlogin or telnet or ssh into the vm session).

Mutex debugging is a pain, but I have found that Linux has a drop dead timer: if the kernel is unresponsive after 10+s a message is printed on the console. Next is a reboot (dont resume a VM snapshot, since you will not have access to what went wrong).

After reboot, /var/log/messages will have your printk() statements to help track down where you got stuck.

Next step is to avoid making bugs in the first place....

I have been working on the /dev/dtrace_helper driver code which is used by a user space app, to find where its not working. This has lead me into a corner of the Linux port, which had been stubbed out or #define'd to compile, saving for a rainy day, the work required to resolve.

That day arrived: much of the Linux kernel driver code is just plain ol' C with a few bits of assembler. However, the dtrace helper ioctl() code needs to be able to find and store attributes of real processes. So, my process-shadowing veneer needs to be stronger than it is. This is the key code which negates the need to change the GPL Linux kernel source; by keeping the dtrace driver pure of GPL, it needs a way to find stuff which is not normally of a concern to a driver.

This is not so much a GPL vs CDDL licensing issue, but more of an issue in that the Linux kernel changes, sometimes quite dramatically, from one release to another. If dtrace is not a part of the kernel, then it needs to be a good citizen and provide easy adaption to new kernels, or kernels compiled with differing compile time options.

Applications like VMware have a similar issue - many times a new kernel will come out and VMware wont work on it, since the compiled code doesnt conform to the new headers or functions.

Of the many thousands of lines of code in dtrace, only a very few (10-20) care about this aspect of the kernel, e.g. convert a PID to a process structure, store/retrieve attributes affecting scheduling. But these few lines are the most difficult...or not...

Other tasks have been taking my time this week, so I shall be going back into the water in a few days....

User Defined Tracing - How it works

USDT is a mechanism for user land applications to embed their own probes into executables. For example, a Perl or Python interpreter might use it to gain access to stack traces of applications which are already started.

The goal of the DTrace team was near zero overhead when not invoked. This works well - even commercial applications can embed probes and not worry about performance or run time dependencies.

There are a number of steps to make this work.

Heres an example: # include # include int main(int argc, char **argv) { while (1) { printf("here on line %d\n", __LINE__); DTRACE_PROBE1(simple, saw__line, 0x1234); printf("here on line %d\n", __LINE__); DTRACE_PROBE1(simple, saw__word, 0x87654321); printf("here on line %d\n", __LINE__); DTRACE_PROBE1(simple, saw__word, 0xdeadbeef); printf("here on line %d\n", __LINE__); sleep(1); } }

The DTRACE_PROBEx macros translate into a function call. To gain near-zero overhead, during linking, the function call is replaced by a series of NOP instructions. Heres an example disassembly of the above

0000000000400e7c <main>:
  400e7c:       55                      push   %rbp
  400e7d:       48 89 e5                mov    %rsp,%rbp
  400e80:       48 83 ec 10             sub    $0x10,%rsp
  400e84:       89 7d fc                mov    %edi,0xfffffffffffffffc(%rbp)
  400e87:       48 89 75 f0             mov    %rsi,0xfffffffffffffff0(%rbp)
  400e8b:       be 07 00 00 00          mov    $0x7,%esi
  400e90:       bf df 11 40 00          mov    $0x4011df,%edi
  400e95:       b8 00 00 00 00          mov    $0x0,%eax
  400e9a:       e8 01 f9 ff ff          callq  4007a0 <printf@plt>
  400e9f:       bf 34 12 00 00          mov    $0x1234,%edi
  400ea4:       90                      nop
  400ea5:       90                      nop
  400ea6:       90                      nop
  400ea7:       90                      nop
  400ea8:       90                      nop
  400ea9:       be 09 00 00 00          mov    $0x9,%esi
  400eae:       bf df 11 40 00          mov    $0x4011df,%edi
  400eb3:       b8 00 00 00 00          mov    $0x0,%eax
  400eb8:       e8 e3 f8 ff ff          callq  4007a0 <printf@plt>
  400ebd:       bf 21 43 65 87          mov    $0x87654321,%edi
  400ec2:       90                      nop
  400ec3:       90                      nop
  400ec4:       90                      nop
  400ec5:       90                      nop
  400ec6:       90                      nop
  400ec7:       be 0b 00 00 00          mov    $0xb,%esi
  ....

When an application is built, dtrace is run on the object files to rewrite the objects, stubbing out the calls for probes, and creating a table in the executable of the places where the stubs are located. (The code for this is located in libdtrace/dt_link.c).

When the application is started up, a piece of code is executed (before main() is called). [Code located in libdtrace/drti.c]. This code looks at the current system, to see if dtrace is loaded into the kernel and communicates with the /dev/dtrace/helper driver to inform it that new probes are available in this process.

Voila! We are done. Or nearly.

At this point, whilst the application is running, 'dtrace -l' should reveal your new probes.

The Kernel

When a user elects to monitor the probe, the patched (NOP-ed) code will be change into a call back into the kernel to notify the function/probe is being invoked. Cancellation of the probe will undo the patched code and we are done.

Next step is ensuring the driver helper functions are enabled and to test it out with a simple example in the release.

Its been a slow week trying to debug just a few lines of code in dt_link.c as GNU object files and Sun ones are subtly different in the way undef symbols are defined in object files.

Stay tuned...

Its not finished yet, but heres what I have found:

libdtrace/drti.c needs to be compiled to an object file, and linked in with target apps. (This now compiles).

"dtrace -G" is the magic used to covert a prototype file to the object file needed to link with the application to be probed.

We need the dtrace 'helper' device - something I had commented out early on, since I wasnt sure what it was. This is now enabled in /dev/dtrace_helper. (If I can work out how to create a /dev/dtrace/ dir, then I can more closely mimic Solaris; not a big deal whether this is done).

libdtrace/dt_link.c has needed a few minor mods to fit in with the new device name, but some workarounds but assumptions about /usr/ccs/ which is not the compiler directory under Linux.

A small complication is needed to store state on a per proc/task basis, but the shadow mechanism (par_alloc) is used for this. (I need to intercept process/task death to do the garbage collection; subject for another day).

This will be a major milestone to get this working; people have been asking about USDT for Perl/Ruby, and I want to put a probe into CRiSP - just so I can understand the in's and out's.

http://blogs.sun.com/barts/entry/putting_user_defined_dtrace_probe

which lets me get started. So, in theory, once I can compile with a decent header file, and reasonable portability, I can go digging in the kernel driver to see if we can invoke/intercept it.

Keep fingers crossed !

I had recently taken a short break from dtrace whilst enhancing fcterm to include the following features:

Infinite scrollback (sort of, now spills to files and can page in from the files, but you dont really want infinite scroll).

Performance: now faster than all the competition.

Auto-restart: if the X server crashes, fcterm will carry all of your pty state over to the newly started X server, without missing a heartbeat. Yes, it waits for a new X server to come along and continues from where you left off; no more lost editing sessions, or shell sessions.

Fixed a typo in dtrace today...time to go back in the water... stay tuned....

So, what is happening to dtrace? Its waiting for me to dive back in, having forgotten how it works....

Over the last couple of months I have been busy on other projects which had gathered dust.

proc

Firstly, 'proc' - a colored top, which does what I want, which 'top' doesnt. You can grab a binary from my downloads/tools section. I am still finding new stuff to add to it. Its ok - it tells me what I want to know about what a system is doing, just like top. It started as a project a long time ago, back on SunOS 4.x, and has been reworked for Linux. (Note, it doesnt play well with some terminal emulators, and I must document it and/or package up the source, but I am lazy !)....

fcterm

fcterm is my color terminal emulator. Written a long time ago as I was learning X windows programming, and the precursor to CRiSP - the editor. Recently I decided I wanted infinite scrollback and searching, having read about the 'Terminator' app on Elliotts Blog (http://elliotth.blogspot.com/). Written in java and its nice, but I wanted to go one better.

fcterm is faster than any other terminal emulator I have measured and now has near infinite scrollback (I will explain about this one day). Its taken a while to iron out the kinks, it works perfectly with 'cr' the editor, and is very useful.

Ok - back to the plot....

dtrace

I put a lot of effort into kernel and user space stack tracing parts of dtrace. It works, but Linux doesnt. Or rather, the way distros are compiled (gcc -fomit-frame-pointer) means you cannot get a good stack trace inside or outside the kernel. What you get depends on what a distro decided to do when they compiled code. Even on 64-bit, they use this flag when they dont need to. Sun doesnt do this, and so dtrace works beautifully. This put me off - acceptance of dtrace will be limited whilst it looks like *it* doesnt work, when its what distro makers do which dilutes the impact of dtrace.

lwn.net article on dtrace

Theres a good article on dtrace and SystemTap and the Linux traceing toolkit here: http://lwn.net/Articles/298685. DTrace for linux gets a mention.

(Aside: I was asked to contribute an article to lwn.net, but I just got too busy with other stuff to fit it in, and my grammar isn't too hot, but c'est la vie, maybe I will finish and/or distribute the partly finished detailed documents I have assembled).

The Big Deal

Heres the deal on DTrace for linux -- look at it, try it out. It may not work, but then its free, it costs nothing, and *you* are the one, yes *YOU* the reader, who can make it work. Me, I'm just a coding-donkey trying to hit the right keys on the keyboard, but whether DTrace ever lands in your distro is up to you.

The GPL vs CDDL debate is great for a bar/pub discussion, but everytime DTrace is mentioned, this gets dragged up.

The CPU in my machine is Intel (as I write). That CPU costs me money. Its full of patents, licenses and other commerical stuff. The CPU is not compiled into the kernel - its a blob of silicon. Add Linux to the CPU and hey presto ! They do something useful.

This is DTrace for Linux: no kernel source is needed (except compilation headers). No patches are made to the kernel source. Its a drop in driver. Once the driver is loaded, you can run dtrace. I am not even asking Linus or any other kernel maintainer to 'please add dtrace' to the kernel. I dont care - - thats just me ! Maybe, one day, when I have documented 'proc', and 'fcterm', and decided I want to enjoy the politics of kernel maintenance, I might ask.

But DTrace for Linux is just a bunch of software - you can read, modify, compile and use. Its 'free' as in 'beer'. I dont claim any rights over it, despite a lot of effort to get this far. If you are reading this far, you must be interested.

DTrace isnt for everybody - its technical, it requires understanding, but the technical audience can translate this into availability. Once dtrace is available, the dtrace scripts come into power.

Guess I am just ranting and finding an excuse not to add the next feature to "proc".

DTrace is on my agenda in the next few weeks to carry on from where I left off. I have not been brace enough to keep it loaded on a real kernel (I have only run it in VMware sacrificial lamb kernels, so I need to eat my own dog food).

Chat soon!

On holiday, with a 32-bit Linux laptop - so I managed to clean up and fix a number of 32-bit issues with dtrace. It now works quite well on Ubuntu 8.

I have just put out a new release - and fixed the accidentally broken 64-bit dtrace userland binary.

I have been working on getting the D functions: stack() and ustack() to work. For 32-bit kernels, ustack seems to work - but I have a lot of work to do to get the Psymtab.c code to lookup the symbol entries and print them symbollically. I hit issues with libelf.so not seeming to work properly for me, so need to investigate that issue.

The kernel stack is probably wrong and a pain due to -fomit-frame-pointer use in the Ubuntu kernel - the goal is to achieve what a kernel stack trace can do, along with symbols. (dtrace reads /proc/kallsyms so we should be able to see something).

The Sun code walks the stack looking at signals and interrupts - and I ripped out that code to get it to work at all. I will need to spend more effort here.

I need to get the 64-bit kernel and user stack dumps working, as that will give a huge amount of functionality to probe the system. If i can get the symtabs to work - that will be a great milestone.

Unfortunately, Unix has gotten itself in a mess with the ELF file format: the simple libelf library has difficulty handling 32 or 64 bit files given that the source process may be 32 or 64 bit itself, and so libraries such as <gelf.h> seem to exist to try and hide the word size issue.

Given that Solaris is a pure ELF system, and much of the symtab lookup on a Linux system is embedded in gdb (along with stack tracing), means that we end up with a bit of an emulation on top of an emulation in dtrace - but so far, no big issues (other than concern over the multitudes of libelf.so variants in the wild).

dtrace is still not ready for a prime time production system - it may work for you, it may not, but hopefully, over time, the port will become more robust, and work in most areas.

I hit some issues with accessing /proc/PID/mem where we can't access certain areas of memory - and I think this may be a bug in Linux kernels, but I need to work out if its me at fault or Linux.

I first noticed this in trying to get the:

ustack();

function to do something sane.

Fixed an issue with syscall tracing being too raw - the return of a syscall ideally needs to be -1 (arg0) and arg1 can be used to access the raw return value. This avoids differences among the platforms.

Trying to work through my own bug list and tests, so I can start looking at fasttrap.

A goal of fasttrap will be to put some dtrace probes into CRiSP (why? because I can!), which will let me better understand dtrace and userland probings. There may be a lot of things that could be done here, but I need to understand how it all fits together first.

There are still reliability issues with the port, but syscall tracing works - quite well, but havent validated every syscall. (We still know that tracing read may cause issues).

I guess I need to watch my inbox now :-)

A lot of debate surrounds GPL vs CDDL and why dtrace cannot be done. I dont believe thats the case - and I am not asserting anything good or bad about either license, just trying to live within the realms of what each group believes in.

Back to the plot...

Dtrace on 64 + 32 bit platforms works well for the basics. I fixed the cyclic timer stuff (sort of) - so that now we dont put undue stress on the system being monitored (e.g. /var/log/messages isnt filling up with debug statements quite so much).

I have slightly reorganised the tree and removed the need to manually locate libgcc.a (needed for the 64-bit mod/div stuff). A perl script will now locate it for you.

The profile (profile + ticks) probe code is now linked in but I need to work out what the omni timer stuff is doing.

Mojmir Svoboda gave a couple of better Linux calls to use - but kernel 2.6.24 doesnt compile whereas 2.6.24.4 and above does. (Strange because it compiles on 2.6.23.1). I need to clean up the calling sequence to kmem_cache_create() to resolve this.

Lex/yacc hit me again. I wrote a script (make test) which runs all the demo D scripts and validates for syntax errors, and thats kicking up lexer issues in the way predicates are parsed. My fight with flex should be over and if it happens again, out goes dt_lex.l and a plain portable C version will be written to get rid of the confusion of input(), unpuc() and YY_INPUT.

I found the PID provider (didnt notice it was even missing, since I am not a dtrace expert!). Its hiding in fasttrap (ah! now I know what it does). This wants hooks into the kernel. Watch this space for a solution which doesnt break the GPL or require a kernel recompile.

I looked at ON Solaris 20060823 and found a later version of dtrace. Not much changed (I think) in that release but have started upgrading the sources and comments to match. Sun will have made fixes for good reason, so better to grab them ASAP and get the diff/merge pain over with.

I am still trying to make daily updates or updates when I feel the release 'works' vs having uncovered issues in porting/compiling, so keep an eye on the download area and grab sources if you want to play.

It should be a matter of 'make all; make load'.

Heres some music for you to listen to ...

/home/fox/src/dtrace@vmubuntu: dtrace -f open
dtrace: description 'open' matched 2 probes
CPU     ID                    FUNCTION:NAME
  0     21                       open:entry
  0     22                      open:return
  0     21                       open:entry

Spent last couple of weeks trying to get 32-bit dtrace to work and after a lot of mishaps and sillynesses, it works on Ubuntu 8 kernel (untouched by human hand).

A new release is available and I can go back to validating the 64-bit port, but also testing/checking out functionality.

Still lots of things to do to dtrace - I fixed the eat cpu as fast as possible issue, but need to get the cyclic timers emulation working else dtrace will give up after 30s or so as it things the system will have lost responsiveness.

dtrace for 64b Linux works (sort of). Theres still some rough edges, and am trying to get the clock (cyclic.c) to work, otherwise dtrace will give up after 30s thinking we have hung the system.

I am taking a diversion to get the 32b version working - this has been a trying experience (deficiencies in Linux + VMware). E.g. at present 64-bit divide/module in the kernel is a pain, as the way the kernel is compiled precludes use of these functions in the kernel.

I can do dtrace -l on a 32b kernel, and off to fix the next set of 'things'.

The 32b port is delaying making further progress, but at least various portability issues have now been resolved.

Linux kernel 2.6 tries to protect the system call table from patching by marking the syscall table as read only and protected. I need to write the code to unprotect this (I have some sample code which modifies the CPUs CR3 register, but i would like to use the kernel routines to unprotect the page - when I find them).

I have added a new driver to the kit -- /dev/syscall, although I dont think its for userland consumption. This will drop the dtrace probes for all listed system calls, although I note the new kernels allow custom syscalls to be installed, so i will have to trawl these data structures.

Am just trying to debug some instability - even with the syscall driver disabled. Hopefully soon, we will have something decent to play with.

Stay tuned...

I had issues getting to the DTRACEIOC_ENABLE ioctl - after a bit of head scratching, this was resolved, and now we get to DTRACEIOC_GO.

Even better, we get past DTRACEIOC_GO.

What does this mean?

It means the dtrace binary talks to the kernel, can pass any D script in, and then wait for the buffered information to be made available.

I found out again, my dtrace.c kernel code was outdated, so spent some time merging the latest OpenSolaris code in - still some stuff to merge, but lots of useful things in there, such as 128-bit arithmetic for tracking big counters, and some validation checks on the way memory is accessed. (Validating memory doesnt help me, because i default to enabling probing to anywhere - need to fix this at some point so that hackers and accidents can't bring the system down).

Now we are past the DTRACEIOC_GO, i found some issues with userland dtrace - stubs i hadnt coded. Thats partially done (gethrtime() and a partial pthread cond wait function).

Heres an example 'session':

/home/fox/src/dtrace/drivers/dtrace@vmubuntu: dtrace -v -f journal_invalidatepage
dtrace: description 'journal_invalidatepage' matched 6 probes

Stability attributes for description journal_invalidatepage:

        Minimum Probe Description Attributes
                Identifier Names: Unstable
                Data Semantics:   Unstable
                Dependency Class: Common

        Minimum Statement Attributes
                Identifier Names: Stable
                Data Semantics:   Stable
                Dependency Class: Common

Not much to look at - i get no output, even after Ctrl-C. (Plus that arbitrary probe isnt interesting really).

I believe we are now firmly in phase 3 of the task: phase I was to build dtrace cmd + driver. Phase II was to get to a point where we dont crash the kernel and the userland command is functional.

Phase III is the point where the whole thing can actually start reporting something/anything.

Phase IV will be examination of SDT an PID providers so we can do really useful stuff.

Stay tuned.

PS I make releases each night if i feel there is progress or important bug fix.es

http://www.crisp.demon.co.uk/tools.html

Anyway, back at the dtrace camp - some progress. The startup code is wrong in the kernel - my driver was missing some of the subtlety of the dtrace_attach()/dtrace_open() code, so by the time cmd/dtrace tries to do an ioctl(DTRACE_ENABLE), we hit some null pointers.

I've now protected myself against this. (Such a kernel panic causes a reboot to be required); I know what i need to debug, just some more linux kernel searching to validate I am calling the correct api (dev_set_drvdata).

Had a near panic last night when my vmware/ubuntu refused to boot. I think ubuntu screwed up the /boot/grub/menu.lst - so i was booting a virgin kernel without an initrd ram disk. Fortunately, one of the many menu items was available, so i have been able to make more progress.

Am spending (too much) time investigating why Ubuntu bison/flex combination doesnt work compared to Fedora 8 bison/flex.

Unfortunately, these tools shoot themselves in the foot - they try to be compatible with old yacc/lex, but are just sufficiently different that a trivial issue becomes very difficult to debug.

Have always disliked lex since it provides so little utility, and debugging it - especially when the lex definition 'just works'.

I can see Apple, in the Darwin code, have hit the same issue, but somehow my issue is very subtle.

Oh well.

Once the portability issue is resolved, I can go back to the driver and just move things along, before I forget how this all works.

Finally able to access the modules and kallsyms to find /dev/fbt entry points to patch in the kernel. This is definitely a milestone - as now, in theory, dtrace can start patching the kernel to insert probes. I have yet to try this - next on my list, to see what actually happens.

Note that we only seem to have a subset of available kernel probes because -- I dont know! Maybe these are the only modules I am loading, and appear to be missing the kernel syms (maybe I need to modify the fbt driver to not just enumerate every module, but to enumerate every kernel/kallsyms entry).

But this gives a huge blast to move forward and start debugging D scripts.

I have truncated the output below - its showing 515 probe points in the kernel (a stripped down linux 2.6.24-4 kernel).

   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
    4        fbt         dtracedrv                         ctf_close entry
    5        fbt         dtracedrv                         ctf_close return
    6        fbt         dtracedrv                     ctf_func_args entry
    7        fbt         dtracedrv                     ctf_func_args return
    8        fbt        freq_table    cpufreq_frequency_table_target entry
    9        fbt        freq_table    cpufreq_frequency_table_target return
   10        fbt              dock      register_hotplug_dock_device entry
   11        fbt              dock      register_hotplug_dock_device return
   12        fbt              dock    unregister_hotplug_dock_device entry
   13        fbt              dock    unregister_hotplug_dock_device return
   14        fbt        parport_pc        parport_pc_unregister_port entry
   15        fbt        parport_pc             parport_pc_probe_port entry
   16        fbt        parport_pc             parport_pc_probe_port return
   17        fbt           parport    parport_ieee1284_ecp_read_data entry
   18        fbt           parport    parport_ieee1284_ecp_read_data return
   19        fbt           parport    parport_ieee1284_epp_read_data entry
   20        fbt           parport    parport_ieee1284_epp_read_data return
   21        fbt           parport      parport_ieee1284_read_nibble entry
   22        fbt           parport      parport_ieee1284_read_nibble return
   23        fbt           parport        parport_ieee1284_read_byte entry
   24        fbt           parport        parport_ieee1284_read_byte return
   25        fbt           parport                parport_wait_event entry
   26        fbt           parport                parport_wait_event return
   27        fbt           parport           parport_register_driver entry
   28        fbt           parport           parport_register_driver return
   29        fbt           parport    parport_ieee1284_epp_read_addr entry
   30        fbt           parport    parport_ieee1284_epp_read_addr return
   31        fbt           parport                   parport_release entry
   32        fbt           parport             parport_announce_port entry
   33        fbt           parport         parport_unregister_device entry
   34        fbt           parport     parport_ieee1284_write_compat entry
   35        fbt           parport     parport_ieee1284_write_compat return
   36        fbt           parport   parport_ieee1284_epp_write_data entry
   37        fbt           parport         parport_unregister_driver entry
   38        fbt           parport                  parport_put_port entry
   39        fbt           parport                  parport_put_port return
   40        fbt           parport   parport_ieee1284_epp_write_addr entry
   41        fbt           parport               parport_remove_port entry
   42        fbt           parport               parport_remove_port return
   43        fbt           parport             parport_register_port entry
   44        fbt           parport             parport_register_port return
   45        fbt           parport   parport_ieee1284_ecp_write_addr entry
   46        fbt           parport   parport_ieee1284_ecp_write_addr return
   47        fbt           parport   parport_ieee1284_ecp_write_data entry
   48        fbt           parport   parport_ieee1284_ecp_write_data return
   49        fbt          i2c_core                    i2c_new_device entry
   50        fbt          i2c_core                    i2c_new_device return
   51        fbt          i2c_core                         i2c_probe entry
   52        fbt          i2c_core                         i2c_probe return
   53        fbt          i2c_core          i2c_add_numbered_adapter entry
   54        fbt          i2c_core          i2c_add_numbered_adapter return
...

$ dtrace -l
   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR

Hooray! We went thru a ton of code in the kernel to dig that out! Now -- to find out what happened to my /dev/fbt entries...

I have decided to consolidate the four drivers into a single dtracedrv.ko driver, to avoid lots of fluff with inter-driver symbol resolution. Having separate drivers causes issues at link-time and leads to a hairy order of dependency as the modules are loaded.

Now we have a single dtracedrv.ko.

Any why is is dtracedrv.ko and not dtrace.ko ?!

Because i havent finished getting the makefiles to work. There is a file called dtrace.c which has most of the kernel guts in it - but not all of it, and the linux kbuild software gets confused if i want my driver to be called by the same name as a dependent source file.

I have also added changed-line support to CRiSP whilst I am at it. Someone had asked for the ability to add a comment in column 73 of a line that is modified (presumably COBOL or Fortran code), and this nearly works. Just need to add the setup menus.

Now...off to get 'dtrace -l' to give me something to probe !

/dev/fbt is used to allow monitoring around all functions in the kernel, by establishing probes on the entry and exit from a function. This, in theory, gives rise to thousands of probes. (/dev/sdt is needed for high level actions like process fork/death, etc - and will come later).

The GPL/CDDL issue in fbt is resolved by using a helper from user land to allow investigation of the running kernel.

The main thing to get to now is to see if the full dtrace userland path can be executed - eg, for 'dtrace -l' to work to see some probes, and then try some simple D scripts to see if the right things happen.

The pid provider will be needed to get access to current process properties, and hopefully wont be a big deal.

To date, the only kernel expectation is that we have reasonable compile defaults (eg modules, kallsyms, etc) and we havent had to touch or break the kernel.

dtrace wont compile without access to the running kernel sources, but its beginning to look good.

I have been enabling bits of ioctl() dtrace driver to try and get the cmd/dtrace binary to talk to the kernel. The next step is to find something to trace, so i have been following and understanding the 'providers' which are like 'device drivers' for dtrace. Eg fbt.c provides access to the symbol table in the kernel. This is the core mechanism for being able to monitor any part of the kernel.

Linux contains a symbol table (eg /proc/kallsyms).

But, its locked out for access by non-GPL drivers. I am studying what it does to work out a way to tunnel into it (alternatively, the symtab can be fed from user space into the driver -- more work, but may be a better alternative).

Enter the license wars. fbt.c can gain access to the public functions of kallsyms.c but only if it is declared to be a GPL driver. I am using Sun's implementation which is CDDL, so i need to find a way in. I have a way by declaring myself as GPL but that amounts to putting Sun's code under CDDL, which i do not have the right to do.

For the moment, I am being dirty and taking that route so I can experiment with the core technology and avoid too much licensing-legalese. I dont mind tackling that on its own basis but theres lots of bits to get together to make it all work.

There is a way to technically break the hurdle - which would involve yet another driver (up to three so far -- dtrace, fasttrap and fbt. More provider drivers are forthcoming, so adding another one will not be a big issue).

Its interesting to understand the legalese here - even if I am not a politician or a bigot (I think). I just want to solve the problem.

Once we have a symtab, I can work more on the dtrace driver to ensure 'dtrace -l' shows something useful, and then to try a real D script.

Still a long way to go. I am trying hard to avoid any changes to the kernel source - this will simplify deployment and installation.

One way is to split the main header file -- linux_types.h -- into a separate file for kernel side and user side stuff.

dtrace -l now simply segfaults - bad calling convention, so need to fix that and have started the switch over to make cmd/dtrace use the /dev/dtrace driver entrypoint.

Hope to make more progrss today and regularly update the website with a latest snapshot of the driver.

The last week was mostly wasted trying to get a VMware guest with Ubuntu 7 running so when i crash the kernel, I wont lose any work. This was incredibly painful due to the way the udev filesystem works, and the initram disk is setup. I would keep booting a 2.6.24.4 Linux kernel, and have the system not be useful because it couldnt find the hard drive.

Using google, i finally found the update-initramfs script (rather than mkinitramfs) and the kernel is bootable and good for recompiling the driver and testing.

The initial test is simply to load the driver, and do:

cat /dev/dtrace

And check /var/log/messages for the debug printks.

Now this is working, I can get back to user-to-kernel debugging and try and get cmd/dtrace to talk to drivers/dtrace.

You can watch out for daily source code snapshots on http://www.crisp.demon.co.uk/tools.html

If you grab the code, you are on your own for now - until I feel comfortable its ready for prime time.

If anyone wants to contribute - please feel free. There are going to be some fiddly bits later on to resolve (such as kernel hooking and requiring a custom dtrace kernel).

More later.

The current release builds on Ubuntu 7 releases (2.6.22) and Linux kernel 2.6.24.

More later

Anyone following this progress should be careful to avoid crashing a machine they care about.

back to the kernel driver trying to reduce compiler errors.

Trying to get a very simple command line program to parse and am hitting issues with flex vs Sun's lex code. (String parsing is going to the terminal and waiting for input).

This in turn is causing a parser error and dtrace aborts.

Still about 200 kernel driver for dtrace issues to resolve before getting close to a dtrace.o kernel module. Hopefully most of these are the same root cause compile issues.

Will try and blog each day on useful progress.

dtrace consists of some userland components and kernel drivers. I have a 'dtrace' binary - but its a bit of a sham - relying on Solaris headers and drivers, and quickly core dumps if you try and do anything with it. (I have about 50+ stub functions just so I can get to a binary and understand what it wants and how it works).

CTF - what is it? Well, it looks like a mostly optional component. It heavily relies on converting DWARF debug format to something the inkernel modules can deal with. The CTF code is fairly straightforward and relies on lots of solaris headers, and one has to be careful to filter out what is pure solaris and what is linux compliant.

Am going to tackle trying to build the dtrace driver, since this is the core of the whole system.

Linux's build environment is ok - its fine for me, but will present some difficulty when i give out the code.

I have been staring at the Apple dtrace port and the freebsd ones. Apple have done not a bad job - they have stripped out the stuff that is irrelevant, and FreeBSDs code is simply confusing because theres a lot of stuff left in without a clear indication of where to start and is the port complete.

More progress when theres something to shout about at a later date.

after waiting years for someone to port to linux, I am getting bored waiting.

Sun provide dtrace for Solaris, and its been ported to FreeBSD and MacOSX.

But the linux people ignore it.

So, time to have a go myself. At present progress isnt bad - most of the userland stuff compiles and i need to work on code changes to allow linking of the 'dtrace' binary.

After that - we go tackle the kernel part.

Suns /proc interface is good - very goo. Linux's is more obvious (since its all text files), but not as feature rich as Solaris.

So, lets see how it goes....

We welcome new ideas and competition.

Todays entrant is Komodo. I downloaded and installed it, and that is about as far as I can tell you.

CRiSP is huge - at 29MB of disk for the complete install, it contains a lot of functionality aimed at all sorts of people. A trimmed down CRiSP can probably fit into 5MB or so.

Komodo comes in at 129MB. Its based on the mozilla engine. At least its not Java - which would have seen any/all of my machines grind to a crawl. (Why is Java so slow? I know, but I'm not telling .. yet !)

Whats worse than the 129MB is that it wont work on my first target machine, so I will give up unless theres stuff to learn from them.

CRiSP supports a large variety of operating systems, many old and no longer supported in themselves (AIX, DEC Alpha, SGI), and far too many Linux variants as they have mutated binary compatibility issues.

Alas, the competition hasnt learnt yet how to support these legacy unix systems (and these systems are not that old even). Its all to do with too many library dependencies.

In the meantime, theres CRiSP...which just works (unless you know different!)

With HTML help, its easier to go through and give it consistency without worrying about Word crashing. Newer releases of Microsoft Office (Office 97, 2000, 2003, 2007) all break the older RTF file format used by CRiSP help.

Internally, the build of CRiSP compiles the HTML to compressed binary files, and from there, generates the character mode help from the compressed output file.

This has been lacking for many years - on the surface, it seemed reasonable, but on investigation, things broke, especially with the HTML conversion. For example, lists and tables.

This has undergone a variety of enhancements so that the help/txt files are readable and laid out sanely now.

Happy New Year