Topics: AIX, Monitoring, Networking, Red Hat, Security, System Administration

Determining type of system remotely

If you run into a system that you can't access, but is available on the network, and have no idea what type of system that is, then there are few tricks you can use to determine the type of system remotely.

The first one, is by looking at the TTL (Time To Live), when doing a ping to the system's IP address. For example, a ping to an AIX system may look like this:

# ping 10.11.12.82
PING 10.11.12.82 (10.11.12.82) 56(84) bytes of data.
64 bytes from 10.11.12.82 (10.11.12.82): icmp_seq=1 ttl=253 time=0.394 ms
...
TTL (Time To Live) is a timer value included in packets sent over networks that tells the recipient how long to hold or use the packet before discarding and expiring the data (packet). TTL values are different for different Operating Systems. So, you can determine the OS based on the TTL value. A detailed list of operating systems and their TTL values can be found here. Basically, a UNIX/Linux system has a TTL of 64. Windows uses 128, and AIX/Solaris uses 254.

Now, in the example above, you can see "ttl=253". It's still an AIX system, but there's most likely a router in between, decreasing the TTL with one.

Another good method is by using nmap. The nmap utility has a -O option that allows for OS detection:
# nmap -O -v 10.11.12.82 | grep OS
Initiating OS detection (try #1) against 10.11.12.82 (10.11.12.82)
OS details: IBM AIX 5.3
OS detection performed.
Okay, so it isn't a perfect method either. We ran the nmap command above against an AIX 7.1 system, and it came back as AIX 5.3 instead. And sometimes, you'll have to run nmap a couple of times, before it successfully discovers the OS type. But still, we now know it's an AIX system behind that IP.

Another option you may use, is to query SNMP information. If the device is SNMP enabled (it is running a SNMP daemon and it allows you to query SNMP information), then you may be able to run a command like this:
# snmpinfo -h 10.11.12.82 -m get -v sysDescr.0
sysDescr.0 = "IBM PowerPC CHRP Computer
Machine Type: 0x0800004c Processor id: 0000962CG400
Base Operating System Runtime AIX version: 06.01.0008.0015
TCP/IP Client Support  version: 06.01.0008.0015"
By the way, the example for SNMP above is exactly why AIX Health Check generally recommends to disable SNMP, or at least to dis-allow providing such system information trough SNMP by updating the /etc/snmpdv3.conf file appropriately, because this information can be really useful to hackers. On the other hand, your organization may use monitoring that relies of SNMP, in which case it needs to be enabled. But then you stil have the opportunity of changing the SNMP community name to something else (the default is "public"), which also limits the remote information gathering possibilities.

Topics: AIX, Red Hat, Security, System Administration

System-wide separated shell history files for each user and session

Here's how you can set up your /etc/profile in order to create a separate shell history file for each user and each login session. This is very useful when you need to know who exactly ran a specific command at a point in time. For Red Hat Linux, put the updates in either /etc/profile or /etc/bashrc.

Put this in /etc/profile on all servers:

# HISTFILE
# execute only if interactive
if [ -t 0 -a "${SHELL}" != "/bin/bsh" ]
then
 d=`date "+%H%M.%m%d%y"`
 t=`tty | cut -c6-`
 u=`who am i | awk '{print $1}'`
 w=`who -ms | awk '{print $NF}' | sed "s/(//g" | sed "s/)//g"`
 y=`tty | cut -c6- | sed "s/\//-/g"`
 mkdir $HOME/.history.$USER 2>/dev/null
 export HISTFILE=$HOME/.history.$USER/.sh_history.$USER.$u.$w.$y.$d
 find $HOME/.history.$USER/.s* -type f -ctime +91 -exec rm {} \; \
  2>/dev/null

 H=`uname -n | cut -f1 -d'.'`
 mywhoami=`whoami`
 if [ ${mywhoami} = "root" ] ; then
  PS1='${USER}@(${H}) ${PWD##/*/} # '
 else
  PS1='${USER}@(${H}) ${PWD##/*/} $ '
 fi
fi

# Time out after 60 minutes
# Use readonly if you don't want users to be able to change it.
# readonly TMOUT=3600
TMOUT=3600
export TMOUT
For AIX, put this in /etc/environment, to turn on time stamped history files:
# Added for extended shell history
EXTENDED_HISTORY=ON
For Red Hat, put this in /etc/bashrc, to enable time-stamped output when running the "history" command:
HISTTIMEFORMAT='%F %T '; export HISTTIMEFORMAT
This way, *every* user on the system will have a separate shell history in the .history directory of their home directory. Each shell history file name shows you which account was used to login, which account was switched to, on which tty this happened, and at what date and time this happened.

Shell history files are also time-stamped internally. For AIX, you can run "fc -t" to show the shell history time-stamped. For Red Hat, you can run: "history". Old shell history files are cleaned up after 3 months, because of the find command in the example above. Plus, user accounts will log out automatically after 60 minutes (3600 seconds) of inactivity, by setting the TMOUT variable to 3600. You can avoid running into a time-out by simply typing "read" or "\" followed by ENTER on the command line, or by adding "TMOUT=0" to a user's .profile, which essentially disables the time-out for that particular user.

One issue that you now may run into on AIX, is that because a separate history file is created for each login session, that it will become difficult to run "fc -t", because the fc command will only list the commands from the current session, and not those written to a different history file. To overcome this issue, you can set the HISTFILE variable to the file you want to run "fc -t" for:
# export HISTFILE=.sh_history.root.user.10.190.41.116.pts-4.1706.120210
Then, to list all the commands for this history file, make sure you start a new shell and run the "fc -t" command:
# ksh "fc -t -10"
This will list the last 10 commands for that history file.

Topics: AIX, Security, System Administration

Avoid using env_reset in sudoers file

By default, when using sudo, the env_reset sudo option is enabled.

From the sudoers manual, about the env_reset sudo option:

This causes commands to be executed with a new, minimal environment. On AIX the environment is initialized with the contents of the /etc/environment file. The new environment contains the TERM, PATH, HOME, MAIL, SHELL, LOGNAME, USER, USERNAME and SUDO_* variables in addition to variables from the invoking process permitted by the env_check and env_keep options. This is effectively a whitelist for environment variables.

If, however, the env_reset option is disabled, any variables not explicitly denied by the env_check and env_delete options are inherited from the invoking process. In this case, env_check and env_delete behave like a blacklist. Since it is not possible to blacklist all potentially dangerous environment variables, use of the default env_reset behavior is encouraged.

In all cases, environment variables with a value beginning with () are removed as they could be interpreted as bash functions. The list of environment variables that sudo allows or denies is contained in the output of "sudo -V" when run as root.


So, what does this all mean? Well, it means that you should not use env_reset in the /etc/sudoers file.

First of all, if you would use:

Defaults env_reset
Then that would do you no good, because the default is already to reset the environment variables.

If you would use (notice the exclamation mark before env_reset):
Defaults !env_reset
Then it means you don't reset any environment variables from the invoking process, for ALL users. That is a security risk, as sudo will preserve variables such as PATH or LD_LIBRARY, and these variables can be configured with values such as "." or "/home/username", or they can be utilized by malicious software.

With the default env_reset all sudo sessions will invoke a shell with minimum shell variables, including those set in /etc/profile and some others if specified in sudoers file (using the env_keep option). So this will make a more controlled sudo access without bypassing sudo security restrictions.

Okay, so what if you need to run a command through sudo that requires a certain environment variable? A good example is the tcpdump command. When running tcpdump via sudo, you may encounter the following error message:
$ sudo tcpdump -i en12
tcpdump: bpf_load: genmajor failed: A file or directory in the path name does not exist.
In this case, tcpdump is known to require the ODMDIR environment variable to be set. One way is to use "Defaults !env_reset" in /etc/sudoers, but the sudoers manual above explains that this is discouraged. Another method is to allow only specific users in /etc/sudoers, by disabling env_reset, such as:
User_Alias           UTCPDUMP = tim, john
Defaults:UTCPDUMP    !env_reset
But this still allows specific users to "play" with all environment variables. So unless you trust these users very much, an even better way is to use the env_keep sudo option, to specify the environment variables that need not be reset (that is, if you know the correct environment variables that are required). In the case of the tcpdump command, we will want to retain the ODMDIR environment variable:
Defaults env_keep += ODMDIR
With the above line in /etc/sudoers, you will notice that running the tcpdump command via sudo will now work properly.

So, the bottom line is: Don't use env_reset at all in /etc/sudoers. If really necessary, use env_reset for only specific users, or even better, specify the required environment variables using env_keep.

Of course, the AIX Health Check software will check if env_reset is used in /etc/sudoers, and if so, warn about this potential security risk.

Topics: AIX, Security, System Administration

Difference between sticky bit and SUID/GUID

This is probably one of things that people mess up all the time. They both have to do with permissions on a file, but the SUID/GUID (or SETUID short for set-user-id/SETGID short for set-group-id) bit and the sticky-bit are 2 completely different things.

The SUID/GUID

The letters rwxXst select file mode bits for users:

  • read (r)
  • write (w)
  • execute (or search for directories) (x)
  • execute/search only if the file is a directory or already has execute permission for some user (X)
  • set user or group ID on execution (s)
  • restricted deletion flag or sticky bit (t)
The position that the x bit takes in rwxrwxrwx for the user octet (1st group of rwx) and the group octet (2nd group of rwx) can take an additional state where the x becomes an s. When this file when executed (if it's a program and not just a shell script), it will run with the permissions of the owner or the group of the file. That is called the SUID, when set for the user octet, and GUID, when set for the group octet.

So if the file is owned by root and the SUID bit is turned on, the program will run as root. Even if you execute it. The same thing applies to the GUID bit. You can set or clear the bits with symbolic modes like u+s and g-s, and you can set (but not clear) the bits with a numeric mode.

SUID/GUID examples

No SUID/GUID: Just the bits rwxr-xr-x are set:
# ls -lt test.pl -rwxr-xr-x 1 root root 179 Jan 9 01:01 test.pl
SUID and user's executable bit enabled (lowercase s): The bits rwsr-x-r-x are set.
# chmod u+s test.pl
# ls -lt test.pl
-rwsr-xr-x 1 root root 179 Jan  9 01:01 test.pl
SUID enabled and executable bit disabled (uppercase S): The bits rwSr-xr-x are set.
# chmod u-x test.pl
# ls -lt test.pl 
-rwSr-xr-x 1 root root 179 Jan  9 01:01 test.pl
GUID and group's executable bit enabled (lowercase s): The bits rwxr-sr-x are set.
# chmod g+s test.pl
# ls -lt test.pl 
-rwxr-sr-x 1 root root 179 Jan  9 01:01 test.pl
GUID enabled and executable bit disabled (uppercase S): The bits rwxr-Sr-x are set.
# chmod g-x test.pl
# ls -lt test.pl 
-rwxr-Sr-x 1 root root 179 Jan  9 01:01 test.pl
The sticky bit

The sticky bit on the other hand is denoted as a t, such as with the /tmp or /var/tmp directories:
# ls -ald /tmp
drwxrwxrwt 36 bin bin 8192 Nov 27 08:40 /tmp
# ls -ald /var/tmp
drwxrwxrwt  3 bin bin  256 Nov 27 08:28 /var/tmp
This bit should have always been called the "restricted deletion bit" given that's what it really denotes. When this mode bit is enabled, it makes a directory such that users can only delete files and directories within it that they are the owners of. For regular files the bit was used to save the program in swap device so that the program would load more quickly when run; this is called the sticky bit, but it's not used anymore in AIX.

More information can be found in the manual page of the chmod command or on http://en.wikipedia.org/wiki/Sticky_bit.

Topics: AIX, Security, System Administration

Generating random passwords

When you set up a new user account, and assign a password to that account, you'll want to make sure that it is a password that can not be easily guessed. Setting the initial password to something easy like "changeme", only allows hackers easy access to your system.

So the best way you can do this, is by generating a fully random password. That can easily be achieved by using the /dev/urandom device.

Here's an easy command to generate a random password:

# dd if=/dev/urandom bs=16 count=1 2>/dev/null | openssl base64 | sed "s/[=O/\]//g" | cut -b1-8
This will create passwords like:
ej9yTaaD
Ux9FYusx
QR0TSAZC
...

Topics: AIX, Security, System Administration

Fix user accounts

Security guidelines nowadays can be annoying. Within many companies people have to comply with strict security in regards to password expiration settings, password complexity and system security settings. All these settings and regulations more than often result in people getting locked out from their accounts on AIX systems, and also getting frustrated at the same time.

To help your users, you can't go change default security settings on the AIX systems. Your auditor will make sure you won't do that. But instead, there are some "tricks" you can use, to ensure that a user account is (and stays) available to your end user. We've put all those tricks together in one simple script, that can fix a user account, and we called it fixuser.ksh. It will fix 99% of all user related login issues.

You can run this script as often as you like and for any user that you like. It will help you to ensure that a user account is not locked, that AIX won't bug the user to change their password, that the user doesn't have a failed login count (from typing too many passwords), and a bunch of other stuff that usually will keep your users from logging in and getting pesky "Access Denied" messages.

The script will not alter any default security settings, and it can easily be adjusted to run for several user accounts, or can be run from a crontab so user accounts stay enabled for your users. The script is a win-win situation for everyone: Your auditor is happy, because security settings are strict on your system; Your users are happy for being able to just login without any hassle; And the sys admin will be happy for not having to resolve login issues manually anymore.

The script can be run by entering a specific user account:

# fixuser.ksh username
The script:
#!/usr/bin/ksh

fixit()
{

  myid=${1}

  # Unlock account
  printf "Unlocking account for ${user}..."
  chuser account_locked=false ${user}
  echo " Done."

  # Reset failed login count
  printf "Reset failed login count for ${user}..."
  chuser unsuccessful_login_count=0 ${user}
  echo " Done."

  # Reset expiration date
  printf "Reset expiration date for ${user}..."
  chuser expires=0 ${user}
  echo " Done."

  # Allow the user to login
  printf "Enable login for ${user}..."
  chuser login=true ${user}
  echo " Done."

  # Allow the user to login remotely
  printf "Enable remote login for ${user}..."
  chuser rlogin=true ${user}
  echo " Done."

  # Reset maxage
  printf "Reset the maxage for ${user}..."
  m=`lssec -f /etc/security/user -s default -a maxage | cut -f2 -d=`
  chuser maxage=${m} ${user}
  echo " Done."

  # Clear password change requirement
  printf "Clear password change requirement for ${user}..."
  pwdadm -c ${user}
  echo " Done."

  # Reset password last update
  printf "Reset the password last update for ${user}..."
  let sinceepoch=`perl -e 'printf(time)' | awk '{print $1}'`
  n=`lssec -f /etc/security/user -s default -a minage | cut -f2 -d=`
  let myminsecs="${n}*7*24*60*60"
  let myminsecs="${myminsecs}+1000"
  let newdate="${sinceepoch}-${myminsecs}"
  chsec -f /etc/security/passwd -s ${user} -a lastupdate=${newdate}
  echo " Done."
}

unset user

if [ ! -z "${1}" ] ; then
  user=${1}
fi

# If a username is provided, fix that user account

unset myid
myid=`id ${user} 2>/dev/null`
if [ ! -z "${myid}" ] ; then
  echo "Fixing account ${user}..."
  fixit ${user}
  printf "Remove password history..."
  cp /dev/null /etc/security/pwdhist.pag 2>/dev/null
  cp /dev/null /etc/security/pwdhist.dir 2>/dev/null
  echo " Done."
else
  echo "User ${user} does not exist."
fi

Topics: AIX, Security, System Administration

Clearing password history

Sometimes when password rules are very strict, a user may have problems creating a new password that is both easy to remember, and still adheres to the password rules. To aid the user, it could be useful to clear the password history for his or her account, so he or she can re-use a certain password that has been used in the past. The password history is stored in /etc/security/pwdhist.pag and /etc/security/pwdhist.dir. The command you can use to disable the password history for a user is:

# chuser histsize=0 username
Actually, this command does not the password history in /etc/security/pwdhist.dir and /etc/security/pwdhist.pag, but only changes the setting of histsize for the account to zero, meaning, that a user is not checked again on re-using old passwords. After the user has changed his or her password, you may want to set it back again to the default value:
# grep -p ^default /etc/security/user | grep histsize
        histsize = 20
# chuser histsize=20 username
In older AIX levels, this functionality (to use chuser histsize=0) would actually have cleared out the password history of the user. In later AIX levels, this functionality has vanished.

So, if you truely wish to delete the password history for a user, here's another way to clear the password history on a system: It is accomplished by zeroing out the pwdhist.pag and pwdhist.dir files. However, this results in the deletion of all password history for all users on the system:
# cp /dev/null /etc/security/pwdhist.pag
# cp /dev/null /etc/security/pwdhist.dir
Please note that his is a temporary measure. Once these files are zeroed out, as soon as a user changes his or her password again, the old password is stored again in these files and it can't be reused (unless the histsize attribute for a user is set to 0).

Topics: AIX, Security

Heartbleed bug

OpenSSL on AIX can be impacted by the Heartbleed bug. Only OpenSSL 1.0.1e (IBM AIX VRMFs - 1.0.1.500 & 1.0.1.501) is vulnerable to the Heartbleed bug (CVE-2014-010). All OpenSSL v0.9.8.xxxx and v12.9.8.xxxx are NOT vulnerable to this CVE.

IBM released OpenSSL 1.0.1g by the end of April 2014, which is the official fix.

The following is information about an ifix that was made available by IBM. The ifix is just a workaround, and currently IBM recommends upgrading to OpenSSL 1.0.1.511 instead (see below).

  • This is a workaround compiled with the feature turned off.
  • This is not OS dependent. It only depends on the OpenSSL level.
Below are the download and install/uninstall instructions.

The OpenSSL ifix doesn't require a reboot. However... It's a shared library update, so any daemons that use it will need to be restarted such as sshd. If you aren't sure what applications running on your machine use OpenSSL, it's recommended to reboot.

To download it, go to: https://testcase.software.ibm.com/ and log in as "Anonymous" (no password needed). Click on the "fromibm" folder, and then click on the "aix" folder. Scroll down the list until you find the following file and click on it to download:
0160_ifix.140409.epkg.Z
Once the download is complete, transfer the file to your AIX system. Log on to your AIX system, go to the directory where you put the file, and run the following command as the root user.

To preview the installation of 0160_ifix.140409.epkg.Z, please do the following:
# emgr -p -e 0160_ifix.140409.epkg.Z
To install the ifix, run the following:
# emgr -X -e 0160_ifix.140409.epkg.Z
If you need to uninstall the iFix for some reason, run the following command as root:
# emgr -r -L 0160_ifix.140409.epkg.Z
The following is more information, updated on June 13, 2014:

IBM has released several new levels for OpenSSL that address both the Heartbleed bug, as well as several other security vulnerabilities that have been identified recently.

We currently recommend downloading OpenSSL 1.0.1.511. This level can be used on AIX 5.3, 6.1 and 7.1. You can find OpenSSL in the IBM Web Download Pack at:

http://www-03.ibm.com/systems/power/software/aix/expansionpack/

Click on Downloads (on the right), log in with your IBM user ID (or register for one, if you don't already have an IBM user ID). Select openssl on the next page, and click on Continue at the bottom. Click Submit to accept IBM's privacy statement on the next page, and you'll be forwarded to a list of possible downloads. Here, click on "Download using http", and select the OpenSSL images for openssl-1.0.1.511.tar.Z. You probably also want to review the Readme beneath it as well.

You will download the openssl-1.0.1.511.tar.Z file. Transfer that onto your AIX systems into a separate folder.

Uncompress the file:
# gzip -d openssl-1.0.1.511.tar.Z
Now you will have a tar file.

Un-tar it:
# tar xf openssl-1.0.1.511.tar
That will give you folder openssl-1.0.1.511 within your current folder. Go into that folder:
# cd openssl-1.0.1.511
Here you can find 3 filesets; run inutoc to generate the .toc file:
# ls
openssl.base       openssl.license    openssl.man.en_US
# inutoc .
Then install the filesets:
# update_all -d . -cY
Now, it should be installed. Before logging out, make sure you can access your system through ssh using a separate window.

For more information, see http://heartblead.com. Please ensure your AIX Health Check level is up to date. Version 14.04.10 and up includes a check for your AIX systems to see if any are impacted by the Heartbleed bug.

Topics: Red Hat, Security, System Administration

Linux Extended File System Attributes

On Linux, you sometimes may run into an issue where you can't change permissions of a file, even though you're root, and you have access. For example:

# ls -asl authorized_keys
8 -rw------- 1 root root 6325 Sep 17 02:48 authorized_keys
# chmod 700 authorized_keys
chmod: changing permissions of `authorized_keys': Operation not 
permitted
# whoami
root
This is usually caused by the Extendef File System Attributes, especially if package e2fsprogs is installed. Two commands that will come in handy here are /usr/bin/chattr and /usr/bin/lsattr.

The most common attributes are:
  • A - When the file is accessed the atime record is not modified. This avoids a certain amount of disk I/O.
  • a - When this file is opened, it is opened in append only mode for writing.
  • i - This file cannot be modified, renamed or deleted.
For example:
# lsattr authorized_keys
----i-------- authorized_keys
This shows that the immutable flag (i) is in place on the file, and thus the reason why the file can't be modified. To remove it, use chattr:
# chattr -i authorized_keys
# lsattr authorized_keys
------------- authorized_keys
Now any commands to modify the file, will work:
# chmod 700 authorized_keys

Topics: AIX, Security, System Administration

mkpasswd

An interesting open source project is Expect. It's a tool that can be used to automate interactive applications.

You can download the RPM for Expect can be downloaded from http://www.perzl.org/aix/index.php?n=Main.Expect, and the home page for Expect is http://www.nist.gov/el/msid/expect.cfm.

A very interesting tool that is part of the Expect RPM is "mkpasswd". It is a little Tcl script that uses Expect to work with the passwd program to generate a random password and set it immediately. A somewhat adjusted version of "mkpasswd" can be downloaded here. The adjusted version of mkpasswd will generate a random password for a user, with a length of 8 characters (the maximum password length by default for AIX), if you run for example:

# /usr/local/bin/mkpasswd username
sXRk1wd3
To see the interactive work performed by Expect for mkpasswd, use the -v option:
# /usr/local/bin/mkpasswd -v username
spawn /bin/passwd username
Changing password for "username"
username's New password:
Enter the new password again:
password for username is s8qh1qWZ
By using mkpasswd, you'll never have to come up with a random password yourself again, and it will prevent Unix system admins from assigning new passwords to accounts that are easily guessible, such as "changeme", or "abc1234".

Now, what if you would want to let "other" users (non-root users) to run this utility, and at the same time prevent them from resetting the password of user root?

Let's say you want user pete to be able to reset other user's passwords. Add the following entries to the /etc/sudoers file by running visudo:
# visudo

Cmnd_Alias MKPASSWD = /usr/local/bin/mkpasswd, \
                      ! /usr/local/bin/mkpasswd root
pete ALL=(ALL) NOPASSWD:MKPASSWD
This will allow pete to run the /usr/local/bin/mkpasswd utility, which he can use to reset passwords.

First, to check what he can run, use the "sudo -l" command:
# su - pete
$ sudo -l
User pete may run the following commands on this host:
(ALL) NOPASSWD: /usr/local/bin/mkpasswd, !/usr/local/bin/mkpasswd root
Then, an attempt, using pete's account, to reset another user's password (which is successful):
$ sudo /usr/local/bin/mkpasswd mark
oe09'ySMj
Then another attempt, to reset the root password (which fails):
$ sudo /usr/local/bin/mkpasswd root
Sorry, user pete is not allowed to execute 
'/usr/local/bin/mkpasswd root' as root.

Number of results found for topic Security: 28.
Displaying results: 1 - 10.