==============================================================
Linux Kernel State Tracer (LKST)
--------------------------------

Date:   Nov, 19, 2004

Version: 2.2.1

Authors: Yumiko Sugita      <sugita@sdl.hitachi.co.jp>
         Kazuyoshi Serizawa <serizawa@sdl.hitachi.co.jp>
         Tetsuhito Nakamura <tetsu@crl.hitachi.co.jp>
         Keisuke Hatasaki   <keisukeh@crl.hitachi.co.jp>
         Takahiro Tsurukubo <tsurukubo.t@jp.fujitsu.com>
         Yasunori Goto      <y-goto@jp.fujitsu.com>
         Satoshi Sakamoto   <s-sakamoto@jp.fujitsu.com>
         Takao Indoh        <indou.takao@jp.fujitsu.com>
         Masami Hiramatsu   <hiramatu@sdl.hitachi.co.jp>
	 Katsuhisa Ogasawara<ogasawar@sdl.hitachi.co.jp>

==============================================================
COPYRIGHT (C) HITACHI,LTD. 2001-2004 
COPYRIGHT (C) FUJITSU,LTD. 2001-2004 
==============================================================

INDEX
-----
1. Introduction
 1.1 Release Notes
2. Installation
3. How To Use
 3.1 How to insert new trace point (and new event)
 3.2 How to extract LKST trace data from LKCD dump file
 3.3 How to program an event handler
 3.4 How to port an event handler to LKST 2.0
 3.5 How to build new event handlers
 3.6 The Format of lkst_etypes
4. Known Issues
5. Future plans
6. Reporting Problems and Discussing LKST
7. License
8. Change History


1. Introduction
---------------
    Linux Kernel State Tracer(LKST) records information as trace data
    about events in the Linux Kernel. It records various events like 
    process context switch, send signal, exception, memory allocation,
    send packet, and so on.
    
    This program is a facility for analyzing trouble of the Linux Kernel.
    It becomes possible that developers investigate troubles by this facility
    without stopping important(used for Mission Critical purposes) server.
    
    Functions of present Linux could not offer enough information for 
    developers to solve the trouble of the system. 
    
       - Standard Kernel:When the kernel is panic, it displays only the
                         registers and stack information. If developers
                         want more information, it is necessary to insert 
                         printk() in the kernel and recompile it. 
       - LKCD           :LKCD store a dump when the system crash.
                         But, if developers use it only, they cannot 
                         investigate the sequence of events till kernel panic.
         (LKCD is a.k.a. Linux Kernel Crash Dumps)
    
    Event information of the kernel is always recorded as trace data 
    on the server under running by using LKST. So, these problems can be
    solved. 
        
    Moreover, LKST has some features. 
      - It is possible to change dynamically which events are recorded.
        Developers can obtain information about the events which they concern
        only. And, it reduces the overhead of components which is not related
        with trouble.
      - It is possible to change each handlers related each events.
        A default handler of all events is just recording the events.
        But, if it is necessary, default handler can be changed.
        This function can be used as following,
          +Notify user processes when the handler detects abnormal status.
          +Change a maskset,which controls what type of events should be
           recorded, dynamically by the handler.
           ->  The system can usually run with a few events trace for the cause
               of good performance. In addition when the handler detects 
	       abnormal status, it can change a maskset to get more detail
	       information.


1.1 Release Notes
-----------------
   - Based on linux-2.6.9
   - Support x86-64 architecture.
   - Change source tree,
   - Add a patchset for colinux (0.6.2-pre4)
   - add enpatch.sh script.
   - Fix some bugs.

2. Installation
---------------
   See file of "Install".


3. How To Use
-------------
   Please refer the online manual for details of each command.

  1) Display the present kernel trace data
      It is necessary to execute 2 commands.
      
      a. Get a log buffer from kernel
         Enter the command as follows. Then LKST saves the trace data into
          <logfile> file.
         
             % lkstbuf read -f logfile
         
      b. Display the trace data
         Enter the command as follows. This command displays the trace data.
         You can specify various options to filter the traced data.

             % lkstbuf print -f logfile

        <output example>
        -------------------------------------------------------------------
        event_type=context_switch
                cpu=02, pid=00001008
                time=Tue Mar 26 18:48:53.143274134 2002
                arg1=0xd68a4000 0x00000000 : pointer to task_struct(prev)
                arg2=0xf2dde000 0x00000000 : pointer to task_struct(next)
                arg3=0x00000001 0x0000000a : process state/process count

        event_type=mem_get_freepage
                cpu=02, pid=00001008
                time=Tue Mar 26 18:48:53.143270345 2002
                arg1=0xecc0f000 0x00000000 : pointer to page
                arg2=0x000001f0 0x00000000 : type of page(gfp_mask)
                arg3=0x00000000 0x00000000 : the number of page(order)
                arg4=0xc0168197 0x00000000 : call address

        event_type=mem_malloc
                cpu=02, pid=00001008
                time=Tue Mar 26 18:48:53.143268419 2002
                arg1=0x00000018 0x00000000 : size
                arg2=0x000001f0 0x00000000 : flags
                arg3=0xdfe3e5a0 0x00000000 : objp
                arg4=0xc0168578 0x00000000 : call address
        ...
        ...
        -------------------------------------------------------------------

             
  2) Change which events are recorded.
     You can change the traced event types by doing as follows.
     
     a.Get a maskset file.
         At first, please get a maskset file which control what event types
         should be recorded by LKST.
         
           % lkstm read -m id -d | grep 0x > maskset_file
         
         "id" is the decimal number which is used as an identifier of the
          maskset. Id 0, 1, and 2 are the reserved maskset; Id 0 does not
          record any event types, id 1 records all of the event types, and 
          id 2 records the default event types.
         
     b.Edit the maskset file.
         Next, edit the saved maskset_file.
         In default, event of spinlock() is not recorded, so id of
         event handler for event 0x080 is 0x0ff.
         (Event type number of spin_lock() is 0x080.)

            :               :
          0x07f           0x01
          0x080           0xff               <<<
          0x081           0xff               
            :               :

         To record the event, please change the value "0x0ff" to "0x001"
         at right side of the line which contains its event number.

            :               :
          0x07f           0x01
          0x080           0x01               <- change
          0x081           0xff               
            :               :
             
         Oppositely, If one event is not wanted to record, 
         please delete the line which is written it's event number.
         ex) event type of system call entry ( number = 0x030).
         
            :               :
          0x02f           0x01
          0x030           0xff               <- change
          0x031           0x01               
            :               :
         
         All the of Event type numbers are written in lkst-events-2.02.pdf
	 that is available from download page.
         
     c.Write the new maskset.
         The new maskset is written when the next command is executed.
         
           % lkstm write -m id -f maskset_file
         
         "id" can be specified from 4 to 254.
	 At the same time, you can specify the name of the new maskset with 
	 "-n" option. Instead of previous, do as below.
	 
	   % lkstm write -m id -n new_maskset_name -f maskset_file
	
       	 "new_maskset_name" must be different from the name of other masksets.
         Please repeat the operation from a. to c., if you want to make more 
         masksets. 
         
         If you want to confirm the state of the maskset, maskset_read 
         have to be executed alone. 
         
           % lkstm read -m id
         
     d.Select maskset
         In order to enable new maskset, execute the lkstm command.
         
           % lkstm set -m id 

     e.Confirm which maskset is currently selected.
         If you want to know which maskset has been selected,
         use lkst_status command. 

           % lkst status

          <output example>
        -------------------------------------------------------------------
        <Current status>
	version of LKST         : 2.2.1
        number of cpus          : 4
        number of masksets      : 4
        number of event-handlers: 3
        current_maskset_id      : 2
        current_buffer_id (cpu: 000): 0
        current_buffer_id (cpu: 001): 0
        current_buffer_id (cpu: 002): 0
        current_buffer_id (cpu: 003): 0
        -------------------------------------------------------------------

         ID of the selected maskset is shown to the right of 
         "current_maskset_id" in the forth line. In this example, it is 2.
         And the number of right of "number of masksets" is total number of
         the masksets. In this example, it is 4.
         cf. Other information means as follows.
            "number of event-handlers" :number of registered event handlers
            "number of cpus"           :number of system cpus
            "current_buffer_id"        :ID of currently selected buffers for
                                        all CPUs


  3) Invoke LKST logging daemon
     LKST logging daemon continues to output the contents of event
     buffer to log files.
     Please refer the online manual of lkstlogd(8) for details.
     Name of the files are /var/log/lkst/sebufCC.S, CC represents
     cpu id corresponds to the file and S represents a sequence
     number (described later).
     If directory /var/log/lkst does not exist, please make it as follows.

           % mkdir /var/log/lkst

     This is an example of command to invoke the daemon,

           % lkstlogd -a -b 2M -l 10M -n 2

     LKST logging daemon will stop and restart to write the file
     when the daemon receives a signal SIGUSR1.
     To send the signal, this is convenient,

           % kill -USR1 `cat /var/run/lkstlogd.pid`

     The size of these files is limited as specified by -l option,
     so old records would be overwritten. To avoid overwritten,
     LKST logging daemon can change files. Said sequence number
     will be increased on this change and to discriminate new 
     set of files and old files. To change file, 

           % kill -USR2 `cat /var/run/lkstlogd.pid`

     examples/lkst_watch_daemon is an example of this feature.
     This script watches unexpected termination of another daemon,
     then change the files.


  4) Dump logs in panic()
     Please do "3) Install addon tools" in "Install".
     After that, just do following,

                 % insmod lkst_mod_panicdump.ko maskset_id_base=2

     A parameter "maskset_id_base" specifies which maskset to
     be used as a template of the new maskset.
     For testing, this module can call panic(). See 6)-c below.


  5) How to display stack trace on each event
     a.Insmod the stack trace module.
	  Please do "3) Install addon tools" in "Install".
	  After that, do following,

                 % insmod lkst_mod_stacktrace.ko maskset_id_base=2

       A parameter "maskset_id_base" specifies a template.

     b.Execute special formatter
          Execute lkst_mod_stacktrace_read.pl. 
	  
                 % lkstbuf read > logfile
                 % perl lkst_mod_stacktrace_read.pl \
		     -s /boot/System.map-`uname -r` -k /proc/ksyms -d logfile

	  You can omit all of the options. The default value for -s and -k
	  are the same as those values of above command line. If option -d
	  is omitted, this formatter reads from current event buffers.

          <output example>
        -------------------------------------------------------------------
        0x00057f24: 0000269005494693 [mc], \
         type:        INT_HARDWARE_ENTRY(0x0010), pid: 0000000000, cpu: 000 \
          0x00000000 0x00000000 0x00000001 0x00000000 \
          0x00000000 0x00000000 0x00000000 0x00000000 \
            <LKST_ETYPE_INT_HARDWARE_ENTRY_HEADER+98> <default_idle+0> <defau
        lt_idle+0> <call_do_IRQ+5> <default_idle+0> <default_idle+0> <default
	_idle+44> <cpu_idle+82> 
        
        0x00057f26: 0000269005511255 [mc], \
         type:       INT_TASKLETHI_ENTRY(0x0012), pid: 0000000000, cpu: 000 \
          0xc01257d0 0x00000000 0x00000000 0x00000000 \
          0x00000000 0x00000000 0x00000000 0x00000000 \
            <LKST_ETYPE_INT_TASKLETHI_ENTRY_HEADER+60> <bh_action+0> <do_soft
        irq_Rsmp_f0a529b7+123> <LKST_ETYPE_INT_HARDWARE_ENTRY_HEADER+459> <de
        fault_idle+0> <default_idle+0> <call_do_IRQ+5> <default_idle+0> 
        
        0x00057f28: 0000269005513862 [mc], \
         type:              INT_BH_ENTRY(0x0016), pid: 0000000000, cpu: 000 \
          0x00000000 0x00000000 0xc012a390 0x00000000 \
          0x00000000 0x00000000 0x00000000 0x00000000 \
            <LKST_ETYPE_INT_BH_ENTRY_HEADER+71> <timer_bh+0> <LKST_ETYPE_INT_
	TASKLETHI_ENTRY_HEADER+74> <do_softirq_Rsmp_f0a529b7+123> <LKST_ETYPE
	_INT_HARDWARE_ENTRY_HEADER+459> <default_idle+0> <default_idle+0> <ca
	ll_do_IRQ+5> 
        -------------------------------------------------------------------

	NOTE: Contents of stack trace might include some unrelated address.


  6) Change event handlers
     As described on the "Introduction", users can register additional 
     event handler in order to add function more than recording the events.
     Please refer addons/lkst_mod_panicdump.c as an example.

     a.Register new event-handler.
     b.Change maskset.
          When lkst_mod_panicdump.o is loaded, the initializer in
	  the module will do them automatically.
	  The initializer will register lkst_evhandler_panicdump()
	  as a new eventhandler that ID is 128, and create a 
	  new maskset named "maskset_panicdump".

     c.Control event handler.
	  The control function of the event handler can invoke panic() 
	  for testing.

                 % echo panic | lksteh ctrl -e 128 -f -

	  The control function compares the argument and "panic",
	  if correspond, the call panic("LKST PANIC TEST");
                
  7) Add user buffer
     If a new user buffer is added, data can be recorded in the added 
     buffer.
     It becomes possible that a lot of data can be recorded without 
     overflowing the standard buffer.

     a.First, Create a buffer (or buffers if you run on SMP system).
       
          	% lkstbuf create -s <bytesize>
      
       <bytesize> is the size of the new buffer.
       
     b.Next, Select the new buffer to record.
     
     	        % lkstbuf jump -b <buffer_id>

     Please refer examples/lkst_test_run for more information.
     This is an example of switching the buffer where data is recorded
     when the specified command is executed. 

           % cd examples
           % sh lkst_test_run 65536 ps aux

     The first argument is the size of new buffer, and the latter arguments
     are command to execute.
     
  8) How to use LKST as an event counter
     a.Insmod the counter module.
          Make addons/lkst_mod_event_count.ko, and load it.
     b.Execute command.
          Execute addons/lkst_event_count, and then
          the total of each event counter is displayed.


3.1 How to insert a new trace point (and a new event)
-----------------------------------------------------
  1) Define a new event: If you want to use a new event, you should define it
     before use. LKST supports several ways to define a new event. There is a
     table to help determining how and where to define it.
     
  +----------+------------------------------+-----------------------------+
  |The event | Including the kernel binary  | Only kernel module(s)       |
  |is caused +---------+----------+---------+---------+---------+---------+
  |on ...    |One core |Functions |Inline   |One      |Several  |Inline   |
  |          |function |capable to|function |module   |modules  |function |
  |          |(one     |build as  |or       |(one     |         |or       |
  |          |location)|module(s) |several  |location)|         |several  |
  |          |         |          |locations|         |         |locations|
  +==========+=========+==========+=========+=========+=========+=========+
  |Definition|(Arch. depended)              |The same |The module loaded  |
  |place     | include/asm-*/lkst_etypes.h  |module   |first (the root    |
  |          |(Others/General)              |         |module of module's |
  |          | include/linux/lkst_etypes.h  |         |dependency tree)   |
  +----------+------------------------------+---------+-------------------+
  |Definition|                              |                             |
  |macro     |        LKST_ETYPE_DEF        |  LKST_ETYPE_DEF_MODULE (*)  |
  |          |                              |                             |
  +----------+---------+----------+---------+---------+---------+---------+
  |Hook type |         |          |         |         |         |         |
  |(**)      | NORMAL  |  MODULE  | INLINE  | NORMAL  | MODULE  | INLINE  |
  |          |         |          |         |         |         |         |
  +----------+---------+----------+---------+---------+---------+---------+
  |Hook macro|         |                    |         |                   |
  |          |LKST_HOOK|  LKST_HOOK_INLINE  |LKST_HOOK| LKST_HOOK_INLINE  |
  |          |         |                    |         |                   |
  +----------+---------+--------------------+---------+-------------------+
  |Event-type|         |                    |         |                   |
  | flag(***)| 0x0001  |       0x0003       | 0x0001  |      0x0003       |
  |          |         |                    |         |                   |
  +----------+---------+--------------------+---------+-------------------+
  
  (*) When using this macro, you should initialize and terminate new event 
      in proper functions. For more detail, see following 1.b) section.
      
  (**) If the event that never used by any LKST_HOOK* macro (for example, 
      the event is passed to lkst_evhandlerprim_entry_log() directory),
      define it as UNDEF hook-type.

  (***) The details about this flag are in 3.6

  For more detail, see following sections.
  
   1.a) If new trace points into kernel binary (vmlinux) with a new
       event, define new event type in linux/lkst_events.h as follows.

         LKST_ETYPE_DEF(event-ID, hook-type, event-symbol, "event-name", \
	 		"1st argument description", \
			"2nd argument description", \
			"3rd argument description", \
			"4th argument description")

           event-ID    --- Value of the event type(0x000..0xfff)

                              0x000-0x0ff: preset
                              0x100-0x1ff: user
                              0x200-0xeff: reserved
                              0xf00-0xfff: system

           hook-type   --- Type of hook header for Kernel Hooks
                           Specify either the following according to the
                           inserting location of the HOOK macro.

                           - NORMAL: If you insert HOOK macro in the kernel (
			     	     in other words, in vmlinux), use this 
				     type.

                           - MODULE: If you insert HOOK macro in a/some 
			     	     module(s) (or selectable to build as 
				     a module), use this type.

                           - INLINE: If you insert HOOK macro in the in-line
                                     function of the kernel, use this type.
                                     If you insert the same HOOK macro
                                     in the two or more places, use this.

			   - UNDEF:  If you do not use this event by HOOK 
			             macro, use this type.

                           NOTE: If you insert HOOK macro in the in-line
                                 function of the module, use MODULE type.
                                                 ~~~~~~

           event-symbol --- The mnemonic of the event type
           event-name --- The description of the event type
           1..4th arguments descriptions : The description for each arguments.

          (example)
              LKST_ETYPE_DEF(0x100, NORMAL, NEW_EVENT, "new event", \
	      		     "current task",
			     "target task",
			     NULL,
			     NULL)


   1.b) If the trace points are ONLY in the modules, the macro for declaration 
     of HOOK header can be put in the prime module (IOW, the root module of
     dependency tree). And the functions for the initialization and 
     termination of HOOK are inserted respectively in the functions of the 
     module-initialization and module-termination.

         Declaration macro for HOOK
           LKST_ETYPE_DEF_MODULE(event-ID, hook-type, event-symbol, \
	 		  	 "event-name", \
	 			 "1st argument description", \
				 "2nd argument description", \
				 "3rd argument description", \
				 "4th argument description")

	!NOTE!: you can specify NORMAL to hook-type if this event is ONLY 
	        caused in the same module and in one location.

         Initialization function
           lkst_hook_etype_register(&LKST_ETYPE_INFO(event-symbol));

         Termination function
           lkst_hook_etype_unregister(&LKST_ETYPE_INFO(event-symbol));

       See sample program "examples/lkst_mod_trc.c".

  2) Insert following HOOK macro where you want to add the trace point.

       Add following sentence to the top of the file in which the trace point
       is added.
       #include <linux/lkst.h>

       - LKST_HOOK_INLINE(etype, argument1, argument2, argument3, argument4)

             If you insert the same HOOK macros in the two or more places, 
	     or insert it in the in-line function or in the macro function,
	     use this HOOK.

       - LKST_HOOK(etype, argument1, argument2, argument3, argument4)

             The case excepts the above-mentioned. 

          * etype   : The value of etype is equal to event-ID. Also, 
	              You can use LKST_ET_SYM(event-symbol) or 
		      LKST_ETYPE_event-symbol.
          * argument1..4 : Arguments for HOOK.

  3) Use with user-space tools.
  
        Since version 2.0.1, lkstbuf and lkstm commands support '-E' option 
	with a CSV formatted file describing events to analyze. Also the 
	LKST module support /proc/lkst_etypes interface that provide
	information of events currently registered.
	 So, when using above two commands, you can specify /proc/lkst_etypes
	with -E option as following:
	
	  % lkstm read -m 3 -E /proc/lkst_etypes
	  
	  % lkstbuf print -E /proc/lkst_etypes -f lkstbuf.log
	
	This feature is also useful to analyze off-lined system's log. 
	First, store the system's event description with the system's log 
	as following:
	
	  % lkstbuf read -f lkstbuf.log
	  % cat /proc/lkst_etypes > etypes.csv
	  
        Then, you can analyze it with stored event description.
	
	  % lkstbuf print -E etypes.csv -f lkstbuf.log


3.2 How to extract LKST trace data from LKCD dump file
------------------------------------------------------
!!NOTE!! THIS FEATURE IS EXPERIMENTAL IN THIS VERSION(2.2.1)!!
         PLEASE USE AT YOUR OWN RISK!!

     a.Create dump file.
       See LKCD documents that can be downloaded at
       http://lkcd.sourceforge.net/.

       <example>
          (i) Save system memory to dump device.
              Enter the command as follows.

            % lkcd config
            % echo 1 > /proc/sys/kernel/sysrq

            Hit Sysrq+C, then start saving system memory to dump device,
            and rebooting automatically. 

            NOTE:
            At default, system memory is saved at swap disk.
            So if swapon runs in booting your system, system memory saved at
            swap disk is collapsed.
            Please save system memory at another disk except swap disk,
            or comment out the line of swapon in the rc.sysinit.

          (ii) Crate dump file.
              Enter the command as follows. Dump file is created at
              /var/log/dump/N (N is an integer of 0 or more.).

            % lkcd save

     b.Run lcrash command
          (i) Enter the command as follows. lcrash command starts and prompt is
              shown.

            % cd /var/log/dump/N
            (N is an integer of 0 or more.)
            % lcrash -n N
            map = map.N, dump = dump.N, outfile = stdout, kerntypes = kerntypes
            .N

            Please wait...
                    Initializing vmdump access ... Done.
                    Loading system map ................................. Done.
                    Loading type info (Kerntypes) ... Done.
                    Initializing arch specific data ... Done.
                    Loading ksyms from dump ....... Done.
            lkstdumplib being initialized
            >>

          (ii) Run lkstdump sub-command
               Enter the command as follows. Trace data is written to file
               <logfile>.

            >> lkstdump logfile
	    static buffer write(2048 records)
            cpu[0].buffer[0] write(1087 records)
            cpu[1].buffer[0] write(1087 records)

              <logfile>'s file format is the same as the file created by
              "lkstbuf read", so you can see <logfile> by using "lkstbuf print".


3.3 How to program an event handler
-----------------------------------
   You can make and add new event-handler as a kernel module.
   However, it requires high-level of programming ability and knowledge about
  linux kernel. Please do it at your own risk.

  1) Include following four files.
 
	#include<linux/version.h>
	#include<linux/module.h>
	#include<linux/init.h>
	#include<linux/lkst_private.h>

   The kernel module requires first three files. And LKST requires last one.

  2) Describe module information.

	MODULE_AUTHOR(/*write your name*/);
	MODULE_DESCRIPTION(/*write description of this module*/);
	MODULE_LICENSE("GPL");/*support GPL only*/

   Do *NOT* forget last description. Only GPL'ed modules can use kernel APIs
  of LKST.

  3) Write an event-handler function.
   You can use lkst_evhandlerprim_* functions in this function.

	static void lkst_evhandler_example(void *phookrec, int event_type,
					   lkst_arg_t arg1, lkst_arg_t arg2,
					   lkst_arg_t arg3, lkst_arg_t arg4)
	{
		preempt_disable();	/*IMPORTANT*/
		/*TODO: do something*/
		preempt_enable();	/*VERY IMPORTANT*/
	}
	
   !!!NOTE!!! : DO NOT forget to disable preemption in the handler, or the
                handler may log an entry to a buffer owned by another
		processor.

  4) Write an event-handler control function.(optional)
   You can do something to control handler's behavior.

	static int lkst_evhandler_example_ctrl(void *buf, size_t bufsize)
	{
		/*TODO: do something*/
	}

  5) Write sysfs interface functions.(optional)
	static ssize_t example_ctrl_store(struct lkst_eh_device *ed, 
					  const char *buf, size_t count)
	{
		/*TODO: do something*/
		return count; /* return 0 means sleep forever.*/
	}
	static ssize_t example_ctrl_show(struct lkst_eh_device *ed, char *buf)
	{
		/*TODO: do something*/
		return strlen(buf) + 1; /*you must return how many 
		                          characters you wrote.*/
	}

  6) Declare module initializer and terminator.

	static int mod_init(void);
	static void mod_cleanup(void);
	module_init(mod_init);		/*define as initializer*/
	module_exit(mod_cleanup);	/*define as terminator*/

  7) Define an lkst_eh_device for registering to lkst class.

	static LKST_EH_DEV_DEF(/*the name of evhandler*/, 
			       lkst_evhandler_example,
			       lkst_evhandler_example_ctrl,
			       example_ctrl_store,
			       example_ctrl_show);
     !! If the name of evhandler is HOGE, you just write HOGE in 1st argument,
        Do NOT write "HOGE" as a string!!

  8) Program module initializer.
    In initializer, you have to register new event-handler to LKST.
    Following is sample code.

	static int mod_init(void)
	{
		int	retval;
		/* register an event handler */
		retval = lkst_eh_device_register(&LKST_EH_DEV(/*the name of evhandler*/));
		if (retval < 0) 
			goto init_err;

		return 0;

	init_err:
		mod_cleanup(); /*If occurs an error, invokes terminator.*/
	       	return retval;
	}

  9) Program module terminator.
    In terminator, you have to deregister your event-handler from LKST.
    Following is sample code. (WARNING: *MUST* deregister it, or kernel might
    panic.)

	static void mod_cleanup(void)
	{
		if (LKST_EH_DEV(/*the name of evhandler*/).id != LKST_EVHANDLER_ID_VOID) {
			lkst_eh_device_unregister(&LKST_EH_DEV(/*the name of evhandler*/));
		}
	}


3.4 How to port an event handler to LKST 2.0
--------------------------------------------
* You can use old interface yet. However, it is not recommended. *

  1)  Define an lkst_eh_device data structure out of functions.

	static LKST_EH_DEV_DEF(example, example_handler, example_ctrl,
	       		       NULL, NULL);

    This code define a structure named 'lkst_eh_dev_example', initialize 
   that name member to "example", evhandler to example_handler, evhandler_ctrl
   to example_ctrl, and so on. In other words, this macro is expanded as 
   following:

	static struct lkst_eh_device lkst_eh_dev_example = {
		.name = "example",
		.id = LKST_EVHANDLER_ID_VOID,
		.evhandler = example_handler,
		.evhandler_ctrl = example_ctrl,
		.ctrl_store = NULL,
		.ctrl_show = NULL,
		.class_dev.class = &lkst_class,
		.class_dev.dev = NULL,
	};

    There are new members, ctrl_store, ctrl_show, and class_dev. class_dev is 
   a sysfs related data structure, so you should not touch that without
   thorough understanding. Remaining two new members are discussed 
   in step 4.

  2) Register the lkst_eh_device data structure in the module initializer.

    Instead of using lkst_evhandler_register(), you can use
   lkst_eh_device_register() to register an event handler.

	lkst_eh_device_register(&LKST_EH_DEV(example));

  LKST_EH_DEV() macro make the symbol (name) from the argument. for example,
  LKST_EH_DEV(example) is expanded to lkst_eh_dev_example.

    If succeed, this function returns zero. If fail, it returns an error value
   less than zero. This behavior different from lkst_evhandler_register().
    If you want to know what id number is assigned, please check 
   LKST_EH_DEV(example).id .

  3) Deregister the lkst_eh_device data structure in the module terminator.

    Instead of using lkst_evhandler_unregister(), you can use
   lkst_eh_device_unregister() to deregister an event handler.

	lkst_eh_device_unregister(&LKST_EH_DEV(example));

   !NOTE!: This function returns nothing (void).

  4) Optionally (But recommended), you can define ctrl_store and ctrl_show
     functions to be able to access from user-mode more easily.

    These two functions are new features. The ctrl_store function stores
   control data from user-mode via sysfs interface. And the ctrl_show shows
   module internal data to user-mode.

 	static ssize_t example_ctrl_store(struct lkst_eh_device *ed, 
					  const char *buf, size_t count)
	{
		/*TODO: do something*/
		return count; /* return 0 means sleep forever.*/
	}
	static ssize_t example_ctrl_show(struct lkst_eh_device *ed, char *buf)
	{
		/*TODO: do something*/
		return strlen(buf) + 1; /*you must return how many 
		                          characters you wrote.*/
	}

    Do NOT return zero from the ctrl_store() function. It should return the 
   number of read bytes. Also, the ctrl_show() should return the number of 
   written bytes.
    After defined these functions, rewrite the definition of the 
   lkst_eh_device data structure as following:

	static LKST_EH_DEV_DEF(example, example_handler, example_ctrl,
			       example_ctrl_store, example_ctrl_show);


3.5 How to build new event handlers
----------------------------------
   Suppose that you have two event handlers(lkst_mod_example1.c,
  lkst_mod_example2.c) in a directory (~/example_module/), and the package of 
  lkst is extracted into /usr/src. (there is /usr/src/lkst-2.2.1/addons).
  
  1) Copy and rename addons/Makefile.skel to Makefile in the directory.
    $ cp addons/Makefile.skel example_modules/Makefile

  2) Edit the Makefile according to comments written in it.
    for example:

--<start here>--    
KMODSRC := lkst_mod_example1.c lkst_mod_example2.c
KMODOBJ := $(KMODSRC:.c=.o)
KMOD_KO := $(KMODOBJ:.o=.ko)

PROGSRC := 
PROGOBJ := $(PROGSRC:.c=.o)
PROGBIN := $(PROGOBJ:.o=)

ifneq ($(KERNELRELEASE),)

obj-m	:= $(KMODOBJ)

else

TOPDIR := /usr/src/lkst-2.2.1/addons
SUB_DIRS := 
ADDCMD-clean := 
ADDCMD-distclean := 

include $(TOPDIR)/Makefile.base

endif
--<end here>--

  3) run make command.
    $ cd example_modules
    $ make


3.6 The Format of lkst_etypes
-----------------------------
You can get information of event-types from /proc/lkst_etypes as following.

0x001,PROCESS_CONTEXTSWITCH,0x0001,"context_switch","pointer to task_struct
prev ","pointer to task_struct next ","process state","process count"
0x002,PROCESS_WAKEUP,0x0001,"process_wakeup","pointer to wakeup-process p
","process state","synchronus",""
0x003,PROCESS_SIGSEND,0x0001,"process_sigsend","signal number sig ","pointer
to
process to which signal is sent","pointer to info structure info ",""
...

The format of this file is CSV(Comma Separated Value). 
Each columns mean event-ID, event-mnemonic, event-type flag, event-name, 
the description of 1st argument, the description of 2nd argument, 
the description of 3rd argument, and the description of 4th argument
respectively.
The "event-type flag" is the bitmap of flags. The least significant bit means
that the event is "maskable". You can set/unset event handlers to maskable
events. 2nd bit means that the event is recorded from multiple place in kernel.
This flag corresponds to hook-type of the event as following.
 UNDEF  = 0x0002 : An UNDEF hook-type event may be logged at multiple place.
                  And can't be changed whether it record or not by maskset.
 NORMAL = 0x0001 : A NORMAL hook-type event is logged at specified place.
                  And can be change the event-handler by maskset.
 INLINE = 0x0003 : An INLINE hook-type event may be logged at multiple place.
                  And can be change the event-handler by maskset.
 MODULE = 0x0003 : The MODULE hook-type is same as the INLINE.


4. Known Issues
---------------
     - The time data in the event records written by 
       IOCTL(LKST_IOC_BUFFER_READ) and lkst_buffer_read do not conform to the
       draft of POSIX standard 1003.25. 


5. Future plans
---------------
     - Support LKCD again. 
       (make lkstdump.sial working without lkstdumplib.so)
     - Support raw buffer read mode.
     - Usability improvement.
     - Performance improvement.
     - Add more extensions for analyzing various faults.
     - Add connectivity to Event Logging (http://evlog.sourceforge.net/).
     - Cleanup sources and documents.


6. Reporting Problems and Discussing LKST
-----------------------------------------
email:  lkst-develop@lists.sourceforge.net
        or lkst-develop@lists.sourceforge.jp
visit:  https://sourceforge.net/projects/lkst/
        or https://sourceforge.jp/projects/lkst/
        or http://oss.hitachi.co.jp/sdl/english/lkst.html


7. License
----------
COPYRIGHT (C) HITACHI,LTD. 2001-2004 
COPYRIGHT (C) FUJITSU,LTD. 2001-2004 

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.


8. Change History
-----------------
2.2.1    19th of November, 2004  Release v2.2.1
				  Fix a bug to read buffers correctly.
2.2      18th of November, 2004  Release v2.2
			          Update for Linux 2.6.9
				  Support x86-64 architecture.
				  Change source tree,
				  add enpatch.sh script.
				  Fix some bugs.
2.1.1     9th of August, 2004   Release v2.1.1
				  Update for Linux 2.6.7
				  Add three workqueue-related events.
				  Drop two obsolate config options.
				  Cleanup codes.
				  Fix some bugs.
2.1      16th of June, 2004     Release v2.1
				  Update for Linux 2.6.6
				  Add the security logging module.
				  Display masksets appropriately.
				  Add the event-type flag.
				  Update Documents.
				  Fix some bugs.
2.0.1    14th of January, 2004  Release v2.0.1
				  Update for Linux 2.6.0
				  Fix some scripts.
				  Update Documents.
				  Remove unused events.
				  Add -E option to lkstbuf and lkstm.
				  Fix some bugs.
2.0	29th of October, 2003	Release v2.0
				  Update for Linux 2.6
				  Support sysfs.
				  Support procfs.
				  Support new kernelhooks.
				  Add new APIs and macros.
				  Cleanup and split sources.
				  Add some scripts.
				  Don't support LKCD (currently).
				  Fix some bugs.
1.5      5th of August, 2003	Release v1.5
				  Add watchdog tools.
				  Make makefile more useful.
				  Add overwritten detection event.
				  Add -S option to lkstbuf print.
				  Modify default maskset.
				  Fix some bugs.
1.5-rc2	 8th of May, 2003       Trial release 2
				  Fix some bugs.
1.5-rc1	 6th of May, 2003       Trial release 1
				  Merge patches for IPF(IA64).
				  Change bitwidth and dword order of arguments.
1.4	13th of March, 2003	Release v1.4
				  Fix the display format of time.
				  Change the results of LKST_IOC_BUFFER_LIST to
				  enable to get the number of buffers.
1.4-rc2  18th of February, 2003  Trial release 2
				  Fix same bugs.
				  Add some configurable kernel options.
				  Add lkst_mod_pmc.c .
				  Add LKST_ETYPE_OOPS_PGFAULT event.
				  Add LKST_ETYPE_OOPS_NMIWDOG event.
				  Simplify format of the list of buffers.
				  Disable logging while kernel dumping.
				  Add lkstbuf print -V option.
1.4-rc1  24th of January, 2003   Trial release
				  Rewrite most of codes associated with event-
				  buffers.
				  Separate resources per processor.
				  Awake reading process in schedule().
				  Allow to log lock event with lkstlogd.
				  Support inter-buffer jump feature.
				  Update commands.
				  Update addons.
				  Add light-weight handler as an addon module.
				  Modify kernel APIs and ioctl interfaces.
				  Add maskset configuration feature.
				  Add the version of LKST in lkst.h.
				  Fix some bugs.
1.3.1	25th of November, 2002	Release v1.3.1
				  Update for kernel 2.4.19.
				  Separate Kernel Hooks from LKST patch.
				  Add module that record only specified process.
				  Make sure lkstlogd release own resources.
				  lkstlogd report warning to syslog.
				  Add and modify APIs for event-handler and maskset.
				  Fix some bugs.
1.3	30th of August, 2002	Release v1.3
                                  Performance improvement of Kernel Hooks.
				  Modify default maskset.
				  Support CSV format by lkstbuf command.
				  Add and modify APIs for event-handler.
				  Add config to delete hooks on inline func.
				  Add script to make /dev/lkst.
				  Fix some bugs.
1.2	28th of June, 2002	Release v1.2
                                  Performance improvement of default handler.
1.1	14th of May, 2002	Release v1.1
				  Replace LKCD patch with the official one.
1.01	26th of April, 2002	Release v1.01
				  Update lkstlogd command.
				  Add trace data of exception events.
				  Fix some bugs.
1.0	12th of April, 2002	Release v1.0
				  Add kernel crash dump function by using LKCD.
				  Update some command.
				  Add log header to log file.
				  Fix some bugs.
0.9	31st of March, 2002	Release v0.9 for some enhancements.
				  Add LKST logging daemon,
				  Replace trace point hook to use Kernel Hooks,
				  Add some add-ons and examples.
0.1a	18th of January, 2002	Fix bug of the racing between 
				  etrc_evhandler_default() and 
				  etrc_buffer_{delete,create}().
				  Add some minor modification, 
				  Clear some comments.
0.1	14th of December, 2001	First public drop

See changelog to know more details.
