Solaris/OpenIndiana NFS server to Ubuntu NFS host - 4294967294 problem

Error keywords seen:  User and Group set to 4294967294 instead of UID/GID expected

Operating System: Open Indiana oi_148 (Solaris) and Ubuntu 11.04 (Natty Narwhal)

Software: NFS v4


When mounting an nfs shared directory from an OpenIndiana home server, the directory ownership was set to 4294967294:4294967294, despite the ownership on the server being 1000:1000, and the equivalent UID / GID being set up on the client machine.


The solution is to edit the config file /etc/default/nfs-common - the two lines required are:


NEED_STATD="no"
NEED_IDMAPD="yes"



This is enough to change the reported ownership from 4294967294 to 'nobody:nogroup' - which is progress, of a sort.



Our next requirement is to make sure that the nfs client and the nfs server are both using the same domain name.  On the client, change /etc/idmapd.conf so that the domain parameter is correct - in my case, 'homenetwork'.

Domain = homenetwork

Secondly, on the server, make sure that the domainname is correctly set.  As this is running a Solaris(-based) OS, it's very easy - just create or edit the contents of /etc/defaultdomain so that it contains (nothing more than) the correct domain:

homenetwork

And you're done - reboot both sides for luck, and everything should now appear as you expect.

File in wrong format error: compiling Apache on Redhat 5

Apache version: 2.0.63
Operating System: Redhat 5 (2.6.18-53.el5)
Error keywords seen:
/usr/lib/libexpat.so: could not read symbols: File in wrong format
collect2: ld returned 1 exit status
(in make step)


A few people seem to have experienced this problem when compiling Apache on Redhat, so I thought I'd share my fix.

The problem is that /usr/lib/libexpat.so links to a 32-bit binary library, where a 64-bit library is expected:
# ls -l /usr/lib/libexpat.so
lrwxrwxrwx 1 root root 27 May 8 2008 /usr/lib/libexpat.so -> ../../lib/libexpat.so.0.5.0
# file /lib/libexpat.so.0.5.0
/lib/libexpat.so.0.5.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), stripped

The correct library can be found at /lib64/libexpat.so.0.5.0:
# file /lib64/libexpat.so.0.5.0
/lib64/libexpat.so.0.5.0: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), stripped

We can easily see that this /lib64 directory is not referenced during the compile:

#gcc -print-search-dirs
install: /usr/lib/gcc/x86_64-redhat-linux/4.1.2/
programs: =/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/bin/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/bin/
libraries: =/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/lib/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/lib/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../:/lib/x86_64-redhat-linux/4.1.2/:/lib/:/usr/lib/x86_64-redhat-linux/4.1.2/:/usr/lib/


The question of course, is how to ensure that the correct library is picked up. You could recreate the link from /usr/lib to the 64-bit library, but a better way is to set the library in the LIBRARY_PATH variable:

# export LIBRARY_PATH=/lib64

Now we can see that the /lib64 directory will be searched:

# gcc -print-search-dirs
install: /usr/lib/gcc/x86_64-redhat-linux/4.1.2/
programs: =/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/bin/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/bin/
libraries: =/lib64/x86_64-redhat-linux/4.1.2/:/lib64/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/lib/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/lib/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../:/lib/x86_64-redhat-linux/4.1.2/:/lib/:/usr/lib/x86_64-redhat-linux/4.1.2/:/usr/lib/

When we rerun configure and make, the compile now completes as it should.

dup2: Bad file descriptor

Operating System: Redhat Enterprise Linux 4
Linux Kernel: 2.4.21-37.ELsmp
Error keywords seen:
dup2: Bad file descriptor
/dev/null: Read-only filesystem


This is one of the scariest errors I've ever seen. Mainly because it appeared on boot-up after reracking one of our more important Oracle database servers. One of those 'blood runs cold' moments when you realise that the simple, 'turn off, move, turn on' plan has gone wrong, and you're looking at a halted system with an error message you've never seen before.

Left with a prompt inviting me to log in as root, and fix the problem, the first, obvious, thing to do is fsck the drive. fsck (-f) uncovered and repaired a handful of unreferenced inodes and a few missing blocks. But this didn't work - the same error message appeared after a reboot.

Google's great though, and thanks to this page I was able to fix the error quite quickly.

I'm not convinced the error is caused by device inode permission problems, but certainly some form of corruption was affecting /dev/null

The root filesystem had been mounted as read-only, so the first task is to remount this with a write option:

mount -n -o remount,rw /

Next we remove the offending /dev/null entry:

rm -f /dev/null

Now we create a new /dev/null

mknod -m 666 /dev/null c 1 3


Happily, the system now rebooted without a problem.

Recovering a box without the root password or a user account

How to rescue a Linux box where you've lost the root password, and you don't have an account. (This is equivalent to saying, how to take control of a Unix system)

We'll do this in 4 stages:

1) prepare a temporary account to use to log into the system
2) boot the target system using an alternative boot media
3) remove the root password, and set up the temporary account
4) boot the system normally, and log in via the temporary account, then switch to the root user, and assume control of the system

1) prepare a temporary account to use to log into the account

On a separate Unix system, add a temporary user via your preferred method, and give it a password. Now extract the hashed password from /etc/shadow on that system. Assuming that you have called the account 'temp', this command will extract it in one line:

grep ^temp: /etc/shadow|awk -F: '{print $2}'

It'll look something horribly unfriendly like $1$ndbiqpq5$4DVeD3KsBmlHZv8jre3S31 - make a careful note of it. There's no escaping this bit.

2) Set up your boot media - be it a CD, a DVD, a PXE kernel, a bootable data stick - whatever you want to use, insert it as appropriate in the machine and turn it on. So long as you can get a terminal session, it'll do. The Redhat install CDs even mount the system for you to /mnt/sysimage - but otherwise do this manually.

3) On the target system, find and edit the /etc/passwd file, you're looking for the root entry, which should be right at the top. It'll look something like this:

root:x:0:0:root:/root:/bin/ksh

Delete out the 'x' in the second column, so it now reads:

root::0:0:root:/root:/bin/ksh

You'll also need to add in a line for a temporary user:

temp:x:9999:10:Temporary - remove after use!:/home/temp:/bin/ksh

Now find and edit the /etc/shadow file. Again, you'll need to edit the root entry, which will look something like:

root:$1$ndbiqpq5$4DVeD3KsBmlHZv8jre3S31:13244:0:99999:7:::

Replace this completely with the line:

root::::

Also add a line for the temporary user such as:

temp:$1$ndbiqpq5$4DVeD3KsBmlHZv8jre3S31:14243:0:99999:7:::

Be sure to replace the second column with the password hash that you know, rather than the example one I use here.

4) Boot the system normally. Log in using the temporary account and password, and acquire a terminal session. Switch to the root user (su -) - you should not be prompted for a password and should simply become the root user. With this power, you should now edit the /etc/passwd file once more, and add back the x you removed earlier, so that:

root::0:0:root:/root:/bin/ksh

becomes:

root:x:0:0:root:/root:/bin/ksh

Now set a new root password - 'passwd' or 'passwd root', and remove the temporary account 'userdel temp'

You now have root access to the system, and can add new accounts as required.