File in wrong format error: compiling Apache on Redhat 5

Apache version: 2.0.63
Operating System: Redhat 5 (2.6.18-53.el5)
Error keywords seen:
/usr/lib/libexpat.so: could not read symbols: File in wrong format
collect2: ld returned 1 exit status
(in make step)


A few people seem to have experienced this problem when compiling Apache on Redhat, so I thought I'd share my fix.

The problem is that /usr/lib/libexpat.so links to a 32-bit binary library, where a 64-bit library is expected:
# ls -l /usr/lib/libexpat.so
lrwxrwxrwx 1 root root 27 May 8 2008 /usr/lib/libexpat.so -> ../../lib/libexpat.so.0.5.0
# file /lib/libexpat.so.0.5.0
/lib/libexpat.so.0.5.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), stripped

The correct library can be found at /lib64/libexpat.so.0.5.0:
# file /lib64/libexpat.so.0.5.0
/lib64/libexpat.so.0.5.0: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), stripped

We can easily see that this /lib64 directory is not referenced during the compile:

#gcc -print-search-dirs
install: /usr/lib/gcc/x86_64-redhat-linux/4.1.2/
programs: =/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/bin/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/bin/
libraries: =/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/lib/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/lib/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../:/lib/x86_64-redhat-linux/4.1.2/:/lib/:/usr/lib/x86_64-redhat-linux/4.1.2/:/usr/lib/


The question of course, is how to ensure that the correct library is picked up. You could recreate the link from /usr/lib to the 64-bit library, but a better way is to set the library in the LIBRARY_PATH variable:

# export LIBRARY_PATH=/lib64

Now we can see that the /lib64 directory will be searched:

# gcc -print-search-dirs
install: /usr/lib/gcc/x86_64-redhat-linux/4.1.2/
programs: =/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/bin/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/bin/
libraries: =/lib64/x86_64-redhat-linux/4.1.2/:/lib64/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/lib/x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/lib/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../x86_64-redhat-linux/4.1.2/:/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../:/lib/x86_64-redhat-linux/4.1.2/:/lib/:/usr/lib/x86_64-redhat-linux/4.1.2/:/usr/lib/

When we rerun configure and make, the compile now completes as it should.

dup2: Bad file descriptor

Operating System: Redhat Enterprise Linux 4
Linux Kernel: 2.4.21-37.ELsmp
Error keywords seen:
dup2: Bad file descriptor
/dev/null: Read-only filesystem


This is one of the scariest errors I've ever seen. Mainly because it appeared on boot-up after reracking one of our more important Oracle database servers. One of those 'blood runs cold' moments when you realise that the simple, 'turn off, move, turn on' plan has gone wrong, and you're looking at a halted system with an error message you've never seen before.

Left with a prompt inviting me to log in as root, and fix the problem, the first, obvious, thing to do is fsck the drive. fsck (-f) uncovered and repaired a handful of unreferenced inodes and a few missing blocks. But this didn't work - the same error message appeared after a reboot.

Google's great though, and thanks to this page I was able to fix the error quite quickly.

I'm not convinced the error is caused by device inode permission problems, but certainly some form of corruption was affecting /dev/null

The root filesystem had been mounted as read-only, so the first task is to remount this with a write option:

mount -n -o remount,rw /

Next we remove the offending /dev/null entry:

rm -f /dev/null

Now we create a new /dev/null

mknod -m 666 /dev/null c 1 3


Happily, the system now rebooted without a problem.