I have just put out a new release - and fixed the accidentally broken 64-bit dtrace userland binary.

I have been working on getting the D functions: stack() and ustack() to work. For 32-bit kernels, ustack seems to work - but I have a lot of work to do to get the Psymtab.c code to lookup the symbol entries and print them symbollically. I hit issues with libelf.so not seeming to work properly for me, so need to investigate that issue.

The kernel stack is probably wrong and a pain due to -fomit-frame-pointer use in the Ubuntu kernel - the goal is to achieve what a kernel stack trace can do, along with symbols. (dtrace reads /proc/kallsyms so we should be able to see something).

The Sun code walks the stack looking at signals and interrupts - and I ripped out that code to get it to work at all. I will need to spend more effort here.

I need to get the 64-bit kernel and user stack dumps working, as that will give a huge amount of functionality to probe the system. If i can get the symtabs to work - that will be a great milestone.

Unfortunately, Unix has gotten itself in a mess with the ELF file format: the simple libelf library has difficulty handling 32 or 64 bit files given that the source process may be 32 or 64 bit itself, and so libraries such as <gelf.h> seem to exist to try and hide the word size issue.

Given that Solaris is a pure ELF system, and much of the symtab lookup on a Linux system is embedded in gdb (along with stack tracing), means that we end up with a bit of an emulation on top of an emulation in dtrace - but so far, no big issues (other than concern over the multitudes of libelf.so variants in the wild).

dtrace is still not ready for a prime time production system - it may work for you, it may not, but hopefully, over time, the port will become more robust, and work in most areas.

I hit some issues with accessing /proc/PID/mem where we can't access certain areas of memory - and I think this may be a bug in Linux kernels, but I need to work out if its me at fault or Linux.]]> http://www.crisp.demon.co.uk/blog/archives/2008-07.html#e2008-07-13T16_59_54.txt 2008-07-13T16:59:54+01:00 Paul Fox I first noticed this in trying to get the:

ustack();

function to do something sane.

Fixed an issue with syscall tracing being too raw - the return of a syscall ideally needs to be -1 (arg0) and arg1 can be used to access the raw return value. This avoids differences among the platforms.

Trying to work through my own bug list and tests, so I can start looking at fasttrap.

A goal of fasttrap will be to put some dtrace probes into CRiSP (why? because I can!), which will let me better understand dtrace and userland probings. There may be a lot of things that could be done here, but I need to understand how it all fits together first.

There are still reliability issues with the port, but syscall tracing works - quite well, but havent validated every syscall. (We still know that tracing read may cause issues).]]> http://www.crisp.demon.co.uk/blog/archives/2008-07.html#e2008-07-05T09_27_41.txt 2008-07-05T09:27:41+01:00 Paul Fox here and here.

I guess I need to watch my inbox now :-)

A lot of debate surrounds GPL vs CDDL and why dtrace cannot be done. I dont believe thats the case - and I am not asserting anything good or bad about either license, just trying to live within the realms of what each group believes in.

Back to the plot...

Dtrace on 64 + 32 bit platforms works well for the basics. I fixed the cyclic timer stuff (sort of) - so that now we dont put undue stress on the system being monitored (e.g. /var/log/messages isnt filling up with debug statements quite so much).

I have slightly reorganised the tree and removed the need to manually locate libgcc.a (needed for the 64-bit mod/div stuff). A perl script will now locate it for you.

The profile (profile + ticks) probe code is now linked in but I need to work out what the omni timer stuff is doing.

Mojmir Svoboda gave a couple of better Linux calls to use - but kernel 2.6.24 doesnt compile whereas 2.6.24.4 and above does. (Strange because it compiles on 2.6.23.1). I need to clean up the calling sequence to kmem_cache_create() to resolve this.

Lex/yacc hit me again. I wrote a script (make test) which runs all the demo D scripts and validates for syntax errors, and thats kicking up lexer issues in the way predicates are parsed. My fight with flex should be over and if it happens again, out goes dt_lex.l and a plain portable C version will be written to get rid of the confusion of input(), unpuc() and YY_INPUT.

I found the PID provider (didnt notice it was even missing, since I am not a dtrace expert!). Its hiding in fasttrap (ah! now I know what it does). This wants hooks into the kernel. Watch this space for a solution which doesnt break the GPL or require a kernel recompile.

I looked at ON Solaris 20060823 and found a later version of dtrace. Not much changed (I think) in that release but have started upgrading the sources and comments to match. Sun will have made fixes for good reason, so better to grab them ASAP and get the diff/merge pain over with.

I am still trying to make daily updates or updates when I feel the release 'works' vs having uncovered issues in porting/compiling, so keep an eye on the download area and grab sources if you want to play.

It should be a matter of 'make all; make load'.

Heres some music for you to listen to ...

/home/fox/src/dtrace@vmubuntu: dtrace -f open
dtrace: description 'open' matched 2 probes
CPU     ID                    FUNCTION:NAME
  0     21                       open:entry
  0     22                      open:return
  0     21                       open:entry

A new release is available and I can go back to validating the 64-bit port, but also testing/checking out functionality.

Still lots of things to do to dtrace - I fixed the eat cpu as fast as possible issue, but need to get the cyclic timers emulation working else dtrace will give up after 30s or so as it things the system will have lost responsiveness.

]]> http://www.crisp.demon.co.uk/blog/archives/2008-06.html#e2008-06-22T21_57_04.txt 2008-06-22T21:57:04+01:00 dtrace progress 20080622 dtrace for 64b Linux works (sort of). Theres still some rough edges, and am trying to get the clock (cyclic.c) to work, otherwise dtrace will give up after 30s thinking we have hung the system.

I am taking a diversion to get the 32b version working - this has been a trying experience (deficiencies in Linux + VMware). E.g. at present 64-bit divide/module in the kernel is a pain, as the way the kernel is compiled precludes use of these functions in the kernel.

I can do dtrace -l on a 32b kernel, and off to fix the next set of 'things'.

The 32b port is delaying making further progress, but at least various portability issues have now been resolved.]]> http://www.crisp.demon.co.uk/blog/archives/2008-06.html#e2008-06-07T18_02_32.txt 2008-06-07T18:02:32+01:00 Paul Fox Linux kernel 2.6 tries to protect the system call table from patching by marking the syscall table as read only and protected. I need to write the code to unprotect this (I have some sample code which modifies the CPUs CR3 register, but i would like to use the kernel routines to unprotect the page - when I find them).

I have added a new driver to the kit -- /dev/syscall, although I dont think its for userland consumption. This will drop the dtrace probes for all listed system calls, although I note the new kernels allow custom syscalls to be installed, so i will have to trawl these data structures.

Am just trying to debug some instability - even with the syscall driver disabled. Hopefully soon, we will have something decent to play with.

Stay tuned...]]> http://www.crisp.demon.co.uk/blog/archives/2008-05.html#e2008-05-31T00_04_13.txt 2008-05-31T00:04:13+01:00 Paul Fox I had issues getting to the DTRACEIOC_ENABLE ioctl - after a bit of head scratching, this was resolved, and now we get to DTRACEIOC_GO.

Even better, we get past DTRACEIOC_GO.

What does this mean?

It means the dtrace binary talks to the kernel, can pass any D script in, and then wait for the buffered information to be made available.

I found out again, my dtrace.c kernel code was outdated, so spent some time merging the latest OpenSolaris code in - still some stuff to merge, but lots of useful things in there, such as 128-bit arithmetic for tracking big counters, and some validation checks on the way memory is accessed. (Validating memory doesnt help me, because i default to enabling probing to anywhere - need to fix this at some point so that hackers and accidents can't bring the system down).

Now we are past the DTRACEIOC_GO, i found some issues with userland dtrace - stubs i hadnt coded. Thats partially done (gethrtime() and a partial pthread cond wait function).

Heres an example 'session':

/home/fox/src/dtrace/drivers/dtrace@vmubuntu: dtrace -v -f journal_invalidatepage
dtrace: description 'journal_invalidatepage' matched 6 probes]]>


http://www.crisp.demon.co.uk/blog/archives/2008-05.html#e2008-05-22T22_18_11.txt

2008-05-22T22:18:11+01:00
Paul Fox


Anyway, back at the dtrace camp - some progress. The startup code
is wrong in the kernel - my driver was missing some of the subtlety of
the dtrace_attach()/dtrace_open() code, so by the time cmd/dtrace tries
to do an ioctl(DTRACE_ENABLE), we hit some null pointers.

I've now protected myself against this. (Such a kernel panic causes
a reboot to be required); I know what i need to debug, just some more
linux kernel searching to validate I am calling the correct api
(dev_set_drvdata).

Had a near panic last night when my vmware/ubuntu refused to boot.
I think ubuntu screwed up the /boot/grub/menu.lst - so i was booting
a virgin kernel without an initrd ram disk. Fortunately, one of the many
menu items was available, so i have been able to make more progress.]]>


http://www.crisp.demon.co.uk/blog/archives/2008-05.html#e2008-05-19T22_37_21.txt

2008-05-19T22:37:21+01:00
Paul Fox


Am spending (too much) time investigating why Ubuntu bison/flex
combination doesnt work compared to Fedora 8 bison/flex.

Unfortunately, these tools shoot themselves in the foot - they
try to be compatible with old yacc/lex, but are just sufficiently
different that a trivial issue becomes very difficult to debug.

Have always disliked lex since it provides so little utility,
and debugging it - especially when the lex definition 'just works'.

I can see Apple, in the Darwin code, have hit the same issue, but somehow
my issue is very subtle.

Oh well.

Once the portability issue is resolved, I can go back to the
driver and just move things along, before I forget how this all
works.]]>


http://www.crisp.demon.co.uk/blog/archives/2008-05.html#e2008-05-11T11_14_42.txt

2008-05-11T11:14:42+01:00
Paul Fox


Finally able to access the modules and kallsyms to find /dev/fbt
entry points to patch in the kernel. This is definitely a milestone -
as now, in theory, dtrace can start patching the kernel to insert
probes. I have yet to try this - next on my list, to see what actually
happens.

Note that we only seem to have a subset of available kernel probes
because -- I dont know! Maybe these are the only modules I am loading,
and appear to be missing the kernel syms (maybe I need to modify
the fbt driver to not just enumerate every module, but to enumerate
every kernel/kallsyms entry).

But this gives a huge blast to move forward and start debugging D scripts.

I have truncated the output below - its showing 515 probe points in the
kernel (a stripped down linux 2.6.24-4 kernel).

   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
    4        fbt         dtracedrv                         ctf_close entry
    5        fbt         dtracedrv                         ctf_close return
    6        fbt         dtracedrv                     ctf_func_args entry
    7        fbt         dtracedrv                     ctf_func_args return
    8        fbt        freq_table    cpufreq_frequency_table_target entry
    9        fbt        freq_table    cpufreq_frequency_table_target return
   10        fbt              dock      register_hotplug_dock_device entry
   11        fbt              dock      register_hotplug_dock_device return
   12        fbt              dock    unregister_hotplug_dock_device entry
   13        fbt              dock    unregister_hotplug_dock_device return
   14        fbt        parport_pc        parport_pc_unregister_port entry
   15        fbt        parport_pc             parport_pc_probe_port entry
   16        fbt        parport_pc             parport_pc_probe_port return
   17        fbt           parport    parport_ieee1284_ecp_read_data entry
   18        fbt           parport    parport_ieee1284_ecp_read_data return
   19        fbt           parport    parport_ieee1284_epp_read_data entry
   20        fbt           parport    parport_ieee1284_epp_read_data return
   21        fbt           parport      parport_ieee1284_read_nibble entry
   22        fbt           parport      parport_ieee1284_read_nibble return
   23        fbt           parport        parport_ieee1284_read_byte entry
   24        fbt           parport        parport_ieee1284_read_byte return
   25        fbt           parport                parport_wait_event entry
   26        fbt           parport                parport_wait_event return
   27        fbt           parport           parport_register_driver entry
   28        fbt           parport           parport_register_driver return
   29        fbt           parport    parport_ieee1284_epp_read_addr entry
   30        fbt           parport    parport_ieee1284_epp_read_addr return
   31        fbt           parport                   parport_release entry
   32        fbt           parport             parport_announce_port entry
   33        fbt           parport         parport_unregister_device entry
   34        fbt           parport     parport_ieee1284_write_compat entry
   35        fbt           parport     parport_ieee1284_write_compat return
   36        fbt           parport   parport_ieee1284_epp_write_data entry
   37        fbt           parport         parport_unregister_driver entry
   38        fbt           parport                  parport_put_port entry
   39        fbt           parport                  parport_put_port return
   40        fbt           parport   parport_ieee1284_epp_write_addr entry
   41        fbt           parport               parport_remove_port entry
   42        fbt           parport               parport_remove_port return
   43        fbt           parport             parport_register_port entry
   44        fbt           parport             parport_register_port return
   45        fbt           parport   parport_ieee1284_ecp_write_addr entry
   46        fbt           parport   parport_ieee1284_ecp_write_addr return
   47        fbt           parport   parport_ieee1284_ecp_write_data entry
   48        fbt           parport   parport_ieee1284_ecp_write_data return
   49        fbt          i2c_core                    i2c_new_device entry
   50        fbt          i2c_core                    i2c_new_device return
   51        fbt          i2c_core                         i2c_probe entry
   52        fbt          i2c_core                         i2c_probe return
   53        fbt          i2c_core          i2c_add_numbered_adapter entry
   54        fbt          i2c_core          i2c_add_numbered_adapter return
...
]]>