nabou - a system integrity monitor
nabou [options]
Nabou is a system integrity monitor. That means, it runs every night and watches for changes on files and directories. If a file has changed in any way, it will inform you by email(if you prefer that). Beside of this nabou can also look for changed or added user accounts, cronjobs, weird processes and suid files. And you can define your own checks using inline scriptlets.
It stores the properties for each file in a dbm database and will warn you if something has been changed on a file. The most important thing to check for, is the MD5-checksum. This checksum will never be the same if the file content has changed even if only one letter has changed. But you can also look for some other properties, like ownership or filemode. See the section Configuration for more details.
You can use nabou as an Intrusion Detection System or simply as a system monitor.
See more detailed description in the section OPERATE NABOU later in this manual page.
entry(s) for file, ... or for all files of no
files are specified.
Nabou uses a config file format similar to the well known apache-configuration format. Lines starting with a # and empty lines will be ignored. Additional, you can place a comment after an option, i.e.:
useshadow 1 # we are using shadow support
You can comment out large blocks using a C-style comment:
/* passwd /etc/passwd shadow /etc/shadow shells /etc/shells */
The configuration is devided into several sections. There are some main-options and some blocks for a special purpose. Most of the options are simply switches, where a value of 1 turns it on and 0 turns it off. If you omit an option then this has the same effect as setting it to 0 (zero).
You can omit an option if nabou provides a buildin default value, which described for every option.
I suggest you, not to use this options unless you are unable to get nabou running properly.
HINT: You can also configure multiple instances of nabou, where every instance checks other directories!
The db block starts with <db> and ends with </db>. An example:
<db>
basedir /home/thomas/nabou/db
protected 1
cipher IDEA
pwdDB thorpwd
sugidDB thorsugid
csumDB thorcsum
cronDB thorcron
miscDB thormisc
diskusageDB thordu
</db>
The following attributes must exist:
The mail block starts with <mail> and ends with </mail>. An example:
<mail>
rcpt root
from root
subject report from nabou
</mail>
The mailblock is required if you turned usemail on. The following attributes must exist:
addresses(comma-separated) of
persons who will receive a carbon copy of
the email reports.
The binary block starts with <bin> and ends with </bin>. An example:
<bin>
sendmail /usr/sbin/sendmail
crontab /usr/bin/crontab
</bin>
The suidblock starts with <suid> and ends with </suid>. It will only be used by nabou if you turned on suid-checking by setting chaeck_suid to 1.
The suid block defines what attributes should be checked for suid sgid files. If there is no suid block configuration or if the suid block contains no appropriate attributes, then nabou will use defaults, which is MD5 check and file-mode check. You can use all chk_* attributes described below in the directory block. You can set chk_all to 1 if you want to perform all available checks. The suid block does not support the exclude, include, inherit or recursive attributes. An example:
<suid>
chk_mode 1
chk_size 1
chk_uid 1
chk_gid 1
</suid>
Finaly, the directory blocks. These are the most important things in the configuration. Each directory block defines various options for one directory. Thus , you are able to define different attributes to be checked for each directory. In general, a directory block stars with:
<directory /some/dir>
and ends with
</directory>
You may define as many directory blocks as you need. Previous versions of nabou had some predefined default checks turned on in case your directory block looked like this one:
<directory /etc> </directory>
Today it will not do any checks on a dir unless you tell it what to check. it is also possible to define ``recursive 1'' and nothing more, then it will tell you about added or removed files recursively under that dir, but it will not do any check on any file.
These are the options you may define:
<directory /dev> exclude tt* exclude pt* </directory>
If you want to ignore files under a subdirectory, then you are required to specify this file/dir relative to the directoy block. An example:
<dir /usr/local> exclude share/blackbox/* </directory>
In the example below, /usr/local/share/blackbox/* will be ignored. The exclude attribute cannot used together with the include attriute (see above).
If you need more than a few files to be excluded you might consider using an exclude block as the better way of doing such. An example:
<directory chk_logdir>
recursive 1
chk_mode 1
chk_uid 1
chk_gid 1
du_increase 30
<exclude>
/var/log/XFree86.0.log
/var/log/boot.log
/var/log/dmesg
/var/log/xdm-errors
/var/log/xfs.errors
</exclude>
</directory>
file(s) specified by include. It
will ignore all other files under that directory!
You cannot use shell-wildcards with the include-
attribute! An example:
<directory /home/vasall> include .history include todo </directory>
The include attribute cannot be used together with the exclude attribute.
The include statement does not a block notation as the exclude statement does!
Excluded files will not counted. If you use one or more include statements, then only the diskusage of these files will be counted.
<directory /bin> chk_md5 1 recursive 1 </directory> <directory /sbin> inherit /bin </directory>
In this example, nabou will do an md5-checksum lookup and do a recursive lookup for the directory /sbin. If the inherit attribute refers to a directory, which is not defined within the configuration, then nabou will warn you about this and use the default attributes chk_md5=1 and recursive=0.
You can specify one or more file attributes nabou shall look for:
The attributes above will be used, if you turn on chk_all! The followng attributes can also be checked:
The following file attributes cannot be checked with nabou:
access time: that's senceless, because, the accesstime changes
when nabou calls stat(2) on a file! And the attributes rdev
and blcksize. See stat(2) for details or look
at the file dbformat.txt shipped with the nabou tarball.
If you need to define many different files/directories in your config you will soon find it boring to write a complete directory block for each single directory/file.
For that reason a new feature exists, which allows you to define check templates. Once you have defined such a template you can apply it to many directories/files within one check block.
DEFINE BLOCKS
Sample:
<define chk_logfile>
chk_shrink 1
chk_mode 1
chk_uid 1
chk_gid 1
</define>
This defines a new check template called chk_logfile (you can choose any name you like). This particular template is used to check mode/uid/gid bits and to check if a file shrinks, which is generally not good with logfiles.
You are allowed to use any possible check statement and option which is available for directory blocks too.
CHECK BLOCKS
Now that we have defined our own check template, we want to apply it to some file within one block:
<check chk_logfile>
/var/log/messages
/var/log/wtmp
/var/log/faillog
/var/log/lastlog
/var/log/kernel
/var/log/firewall
/var/log/secure
</check>
That's it.
One thing is important: if you want to use the exclude statement inside a check block, then you must specify the absolute pathname, relative pathnames are not allowed (or: they will not work as you expect!) inside check blocks.
There is a very special and useful parameter which allows you include another file:
<<include /root/anotherfile>>
nabou reads its contents on this position as if is was written overthere. An example:
--- mail.rc ---
<mail>
rcpt tom@daemon.de
from root@localhost
subject nabou report
</mail>
--- end ---
--- nabou.rc --- <<include mail.rc>> ... --- end ---
If you run nabou in multiple instances, say one checks only for suid files once a week and another one checks only for files every day, but both instances shall to use identical mailoptions, you can use the construction above.
Nabou does not follow symlinks (at the moment, this may change in a future release), because of some program's tendencies to have circular symlinks which will run you out of memory pretty fast. And that's evil.
You can see a complete configuration example in the file nabourc shipped with the nabou tarball.
This is the really funny part :-)
You can extend the functionality of nabou by writing ``inline scripts'', (I name them ``scriptlets'' from now on). All of this stuff is done inside the nabou configuration file. You can define any number of scriptlets. After defining a scriptlet you can refer to it by using the directory block option
chk_custom name
The parameter to chk_custom must be the name of a defined scriptlet. nabou will call this scriptlet on every file within a directory. That means, you need to refer to a scriptlet on a per-directory basis. See the example below.
The scriptlet itself must be written in valid perl5. nabou will save this code in an anonymous sub and call it using a closure. This sub will get 3 arguments from nabou, a File object, the current directory and a string containing the current message for the current file.
If the message is empty no other check found something, if is is not empty it contains the explanation of what nabou already found about this file.
The File object needs some further explanation: You can access every file attribute by using the -> notation. Possible attributes are size, uid, atime and so on (see dbformat.txt for more available attributes).
Beside of these, there are two more attributes: filename and md5. filename returns the file name of the current file (the absolute file name!). md5 returns the checksum of the current file. This maybe a MD5 checksum but can also be SHA1 or MD2, which depends on your configuration(see use_algo above!).
If you want complete integration of the scriptlet into the nabou workflow, then you need to return a message, you don't have to print it just out to STDOUT!
One or more script(s) must be defined using the <script> block.
This block starts with <script> and ends with </script>. Inside the
block you need to define the code using a so called here-document.
The name of this here document must be the script-name followed
by an end identifier after the magical << characters.
Nabou will read everything in until the end identifier occurs. There is no space, tabulator or anything else allowed after this identifier! The name that you give the scriptlet will be used later for referencing to, by using the chk_custom attribute in the directory block.
An example says it all:
--- snip ---
<script>
test <<EOF
my($file, $dir) = @_;
return "SIZE of " . $file->filename . ": " .
$file->size . " bytes\n";
EOF
</script>
<directory /tmp/test>
chk_md5 1
chk_mode 1
chk_custom test
</directory>
--- snap ---
In this example nabou will run the following sub:
sub {
my($file, $dir) = @_;
return "SIZE of " . $file->filename . ": " .
$file->size . " bytes\n";
}
on every file it finds in /tmp/test.
Additional you access the complete namespace of nabou. This is powerful, but also danger. But a really good use of this case is that you can access the %config hash, which contains a data structure of the whole configuration. You could create your own configuration block, which will be used by your scriptlet! An example custom config block (which will simply ignored by nabou itself, but it will be loaded into %config!):
<test>
user max
pass nim9w
port 31733
</test>
Inside your scriptlet you can access to the values inside <test> using the following method:
$user = $config{test}->{user};
or
$pass = $config{test}->{pass};
There are two sciptlet names which have a special purpose: BEGIN
and END. The BEGIN scriptlet will be executed only once at startup
IF it exists, and the END scriptlet will be executed only once at
the end respectively IF it exists. This is very usefull if you
want to open e file or database, which your scriptlet(s) later can
use.
An example:
<script>
test <<EOF
my($file, $dir) = @_;
print TEST " DIR: $dir\n";
print TEST "FILE: " . $file->filename . "\n\n";
return "";
EOF
BEGIN <<EOF
open TEST, ">/home/thomas/nabou/test" or die $!;
EOF
END <<EOF
close TEST;
EOF
</script>
<directory /usr>
chk_custom test
</directory>
As you can see, the ``test'' scriptlet, which will be executed for every file under /usr prints a line to a filehandle ``TE''. Under normal circum- stances this wouldn't be possible, because it were not opened somewhere before. But if you define a BEGIN scriptlet as in the example above, which opens this filehandle, then the ``test'' scriptlet can access this handle.
You need to know, that both the BEGIN and the END scriptlets are not evaluated as a anonymous sub-routine. They are evaluated inside the main namespace, in other words, variables which you define here are accesible globally!
A note about return values of scriptlets: If you don't want the scriptlet to cause a match, then you need to return an empty string:
return "";
otherwise it would be a match and appear in the nabou report!
Another feature of nabou is to monitor running processes and to send out a report in case it finds something weird. But how does nabou know, if a process is weird? Good question. It uses the /proc filesystem. Because of this fact you cannot use this feature of nabou if you are running a system without a /proc filesystem enabled. Nabou cannot use the /dev/kmem device!
Nabou gathers a list of all running processes with all properties it can get from /proc. Most properties are those, which you can see when you use ps. But there are also some other, very interesting properties. Currently it uses 4 different methods to find out if a process could be dangerous:
If you use nabou for process monitoring, you can use the -D option, which causes nabou to turn itself into the background (using fork(2). You can also define exceptions for processes which should be ignored furing the normal run, like xterm's or so.
See the provided example config file psrc for a demonstration.
A proc block starts with <proc> and ends with </proc>. It is used by the process monitoring routines of nabou. It may contain one or more of the following options:
But be warned: This directory may become very huge!
Additional you can define exclude blocks, which starts with <exclude /path/to/program> and ends with </exclude>. Every exclude block may contain the following options:
cmdline /root/bin/checkusers.sh cmdline /root/bin/checksnort.sh
You can also use perl regular expressions here, but be very careful!
An example:
<exclude /bin/bash>
# login shells
cmdline -bash
md5 c36b467680f96a6c63053df2c0df379e
</exclude>
As I mentioned above you can use custom perl scriptlets with the process monitoring mode of nabou too, and it works similar to the scriptlet engine in the file-system mode. But a scriptlet gets other parameters from nabou: a Process object and a message about the checks already done.
The Process object contains all available information about the current checked process. If the message is empty no other check found something, if is is not empty it contains the explanation of what nabou has already found about this file.
An example:
<script>
test <<EOF
my($prc, $lastmatch) = @_;
if($prc->exe =~ /^\/tmp\/.*sh.*$/) {
return "shell running from /tmp!";
}
else {
return "";
}
EOF
</script>
As you can see, you can access every property using the arrow notation: cmdline is: $prc->cmdline, the pid is: $prc->pid.
Here is a list of all available properties, which can also be used for reporting (see above about the report option!):
see a more detailed description of the other properties in the proc(5)
manpage: ``man proc'', look for ``stat Status information about the process'':
state ppid pgrp session tty tpgid flags minflt cminflt majflt cmajflt utime stime cutime cstime counter priority timeout itrealvalue starttime vsize rss rlim startcode endcode startstack kstkesp kstkeip signal blocked sigignore sigcatch wchan nswap cnswap exit_signal
There is a complete (tested) sample configuration supplied with nabou called psrc, which you can use as a starting point. Addional, there is a script called ps supplied in the sub directory ``supplement'', which you can use to see a complete ps-like listing with the following informations: PID RUID EUID RGID EGID FH TTY EXE CMD.
Before you can install nabou as a daily cronjob, you need to run it once with the -i or --init commandline flag. This causes nabou to initialize it's databases based on your configuration.
An example:
nabou --init --config /root/.nabourc
Please note, that the output of the initial run of nabou can become very large! You may redirect this output to a file:
nabou --init --config /root/.nabourc > init-log
nabou will ask you for a passphrase if the <db> block option protected is turned on and save the encrypted passprase in the file ``keydb'' in your naboudb basedirectory. The phrase will be encrypted using crypt(1), which is a one-way algorithm. This passphrase will be used later if you want to use the --update or --reset option.
If you are using a nabou instance for process-monitoring only as demonstrated by the sample ``psrc'' supplied with the package, then the options -i or -r are redundant.
If the --reset or --init(or -i/-r) flag is supplied, nabou will not mail out a report even if usemail is turned on.
Now you are ready to install it as a daily cronjob. An example:
30 0 * * * /root/bin/nabou --config /root/.nabourc > /dev/null 2>&1
This crontab entry runs nabou every day at 00:30.
There is another commandline option which you can use to re- initialize it's database: -r or --reset. But be very careful with this option.
If you only want to update the database entry for one file, you can use the option -u or --update, which requires a filename as argument. If the <db> block option protected is turned on, then nabou will ask you for a passphrase. If the environment variable $NABOU_PASSWD is set, this value will be used as passphrase.
If you omit a filename for --update then nabou will perform a normal run based on the current config, which is the same as running nabou without any options (beside --config).
You can also run multiple instances of nabou, but better use different databases for every instance. If every instance should use some identical config options, you might make use of the include statement mentioned earlier in the CONFIGURATION section.
You can run nabou with the -q or --quiet flag, which causes it only to report changes. In other words, if nothing changed, no report will be sent/printed.
You can view the contents of a nabou database by using the command line flag -d or --dump. This flag requires the database name as argument and dumps out a comma separated list, one line for one file. The meanings of each field are described in the file dbformat.txt. The dump will be a little bit formatted, time values will be converted to human-readable values (just as the date command tells you), uid and gid values will be converted to their representations (i.e. 0 = root and so on).
If you prefer to get the raw contents unformatted then you can use the flag --raw in addition to the --dump or -d flag.
Nabou behaves very, very paranoid if the <db> block option protected is turned on! In fact, it uses the key, which you supplied the first time (--init) for an additional data- base field. This field contains an encrypted copy of the record for that file.
If you run nabou from crontab (without any options but
--config), then it will NOT update the databases! Instead it
will only warn you about changes. The next time it runs, it
will warn you again, and so forth, until you log on and update
the database yourself using the commandline-flag --update
without any parameters. Nabou will ask you for the passphrase
and use this passphrase to decrypt the encrypted record-field.
If the result is identical with the exiting(old) record, then
it will update that record and create a new encrypted field
based on that new record.
If the result does NOT match, then something weird has happend with the database. In this case, it will not update the record and send out an email to the receipient specified in <mail> - alert or to ``root'' if not specified.
If you turn on readonly or protected(which automatically turns on readonly too), then I recommend you to protect your databases.
There are several ways for protecting files from being written. You can use a read-only mounted medium, or you can protect the files using chattr(1). The very best and most secure way is to use LIDS (http://www.lids.org). You can secure the complete nabou database directory and you might protect nabou and you config-file(s) as well:
lidsadm -A -o /root/db -j READ lidsadm -A -o /root/bin/nabou -j READ lidsadm -A -o /etc/nabourc -j READ
If LIDS is well configured, then only root logged in from a local console is able to turn off LIDS and thus to perform a nabou update. Remote users, and even root, are not able to turn it off, and because of this - not able to perform an update of nabou.
If you use such a setup, then you can be sure you will be informed, if someone nasty got root and installed a trojan horse or added a new UID 0 user account or something else.
/etc/nabourc /sbin/nabou /usr/doc/nabou-VERSION/*
ps, ls, find, md5sum, perl, http://www.nabou.org