The Linux Experiment

I’ve had a few nasty experiences this week with Linux and figured I’d vent here. Unlike my previous efforts with Linux From Scratch and Gentoo, my complaints this time around are related to upgrading Ubuntu.

Ubuntu 10.04 to 12.04: Save yourself the trouble

At this point the current Ubuntu LTS release (12.04) is my preferred distribution to work with: it has become widespread enough that troubleshooting and previous solutions online are easy to locate. In a professional capacity, I also maintain systems that are still on 8.04 LTS (supported until April 2013, so we have to be pretty aggressive about replacing them) or 10.04 LTS (good until April 2015).

I attempted to complete two upgrades from the 10.04 release this week to 12.04 – one 10.04 LTS “desktop” installation, and one 10.04 LTS headless server installation. Both were virtual machines running under VMWare ESXi, but neither had given me any trouble during normal use.

Canonical’s updater process (the wrapper around dist-upgrade) appears to be pretty slick; it gives you appropriate warnings, attempts to start a SSH daemon as a fallback mechanism and starts on its merry way to download the necessary packages to bring your system completely up to date. On my 10.04 desktop VM, the installer fell apart completely during the package replacement/removal/installation sequence. I was left with two nasty message boxes: one advising that my system was now in a broken state, and another that completely contained rectangular, unprintable characters.

To put it bluntly, I was not amused, but it wasn’t a critical system and I was content to replace it with a fresh 12.04 installation rather than waste additional time troubleshooting with apt or dpkg. Strike one for the upgrader.

At least the server came back up!

Next on the upgrade schedule was the 12.04 server VM. Install, package replacement and reboot went fine, but I had several custom PPAs installed to support development of XenonMKV (Github page) – specifically ppa:krull/deadsnakes to add Python 2.7 to Ubuntu 10.04.

Python 2.7 still worked when the server came back up, and all my usual tools of choice like SABnzbd+, SickBeard and CouchPotato were still functional.

For some reason, though, I’d gotten it into my head this evening to check out Mezzanine as a potential WordPress replacement. Mezzanine uses Django, a Python Web framework, and the list of supported features is pretty encompassing.

Sidebar: Django and mod_wsgi – complicated enough?

One of the most irritating things from a system administration point of view is getting Web applications to run in a standard server environment – typically a Linux base system and Apache or nginx to serve content. I suppose I’ve been spoiled with how easy it is to get PHP-based sites up and running these days in that configuration by adding an Apache module through apt. A lot of new Web app frameworks come with their own small webservers for development and testing, but generally their creators recommend that when you’re ready to put your site live, that the product run under a well-known Web or application server.

The Django folks recommend using mod_wsgi in their documentation, which in and of itself really just says “RTFM for mod_wsgi and then you’ll have a much better idea of how to do this.” I had to go poking around on Google for the installation article since there are some broken links, but okay, it’s an Apache module with a small bit of configuration (even though a simple walkthrough in the Django documentation would go a long way to making deployment easier.) This is where I ran into my dependency/PPA problem on Ubuntu 10.04.

I’ve suppose I’ve screwed the pooch…

Running the suggested command, I tried: sudo apt-get install libapache2-mod-wsgi and got the following

The following packages have unmet dependencies:
libapache2-mod-wsgi : Depends: libpython2.7 (>= 2.7) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.


Backtracking, I then found out why the library wasn’t going to get installed:


The following packages have unmet dependencies:
libpython2.7 : Depends: python2.7 (= 2.7.3-0ubuntu3.1) but 2.7.3-2+lucid1 is to be installed


Aha! The Python installation from the PPA for Lucid – 10.04 – was installed and acting as the 2.7 package. Since the newly-upgraded Ubuntu 12.04 uses Python 2.7 as a dependency for a good portion of the default applications, I couldn’t just purge or uninstall it, and my attempts to force a reinstallation all ended in:


Reinstallation of python2.7 is not possible, since it cannot be downloaded.


Rebuild?

At this point it looks like I’ll have to rebuild the server VM as well, but if any readers have any bright ideas on fixing this dependency hell – please comment with your suggestions!

As you may be able to tell from my recent, snooze-worthy technical posts about compilers and makefiles and other assorted garbage, my experience with Linux from Scratch has been equally educational and enraging. Like Dave, I’ve had the pleasure of trying to compile various desktop environments and software packages from scratch, into some god-awful contraption that will let me check my damn email and look at the Twitters.

To be clear, when anyone says I have nobody to blame but myself, that’s complete hokum. From the beginning, this entire process was flawed. The last official LFS LiveCD has a kernel that’s enough revisions behind to cause grief during the setup process. But I really can’t blame the guys behind LFS for all my woes; their documentation is really well-written and explains why you have to pass fifty --do-not-compile-this-obscure-component-or-your-cat-will-crap-on-the-rug arguments.

Patch Your Cares Away

Read more…

If you run into the following error trying to build LFS from a Ubuntu installation:


make[1]: *** No rule to make target `/mnt/lfs/sources/glibc-build/Versions.all', needed by `/mnt/lfs/sources/glibc-build/abi-versions.h'. Stop.


The mawk utility installed with Ubuntu, and symlinked to /usr/bin/awk by default does not properly handle the regular expressions in this package. Perform the following commands:


# apt-get install gawk
# rm -rf /usr/bin/{m}awk
# ln -snf /usr/bin/gawk /usr/bin/awk


Then you’re just a make clean; ./configure –obnoxious-dash-commands; make; make install away from success.

I went to play around with my Linux from Scratch installation after getting a working version of KDE 4.7.3 up and running. For a few days now my system has been running stood up to light web browsing use and SSH shenanigans, and hasn’t even dropped a remote connection.

This was until this evening, when I decided to reboot to try and fix a number of init scripts that were throwing some terrible error about problems in lsb_base under /lib/ somewhere. The system came back up properly, but when I startx‘d, I was missing borders for most of my windows. Appearance Preferences under KDE wouldn’t even lanch, claiming a segmentation fault.

There were no logs available to easily peruse, but after a few false starts I decided to check the filesystem with fsck from a bootable Ubuntu 11.04 USB stick. The results were not pretty:


root@ubuntu:~# fsck -a /dev/sdb3
fsck from util-linux-ng 2.17.2
/dev/sdb3 contains a file system with errors, check forced.
/dev/sdb3: Inode 1466546 has illegal block(s).

/dev/sdb3: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)


Running fsck without the -a option forced me into a nasty scenario, where like a certain Homer Simpson working from his home office, I repeatedly had to press “Y”:

At the end of it, I’d run through the terminal’s entire scroll buffer and continued to get errors like:


Inode 7060472 (/src/kde-workspace-4.7.3/kdm/kcm/main.cpp) has invalid mode (06400).
Clear? yes

i_file_acl for inode 7060473 (/src/kde-workspace-4.7.3/kdm/kcm/kdm-dlg.cpp) is 33554432, should be zero.
Clear? yes

Inode 7060473 (/src/kde-workspace-4.7.3/kdm/kcm/kdm-dlg.cpp) has invalid mode (00).
Clear? yes

i_file_acl for inode 7060474 (/src/kde-workspace-4.7.3/kdm/kcm/CMakeLists.txt) is 3835562035, should be zero.
Clear? yes

Inode 7060474 (/src/kde-workspace-4.7.3/kdm/kcm/CMakeLists.txt) has invalid mode (0167010).
Clear? yes


I actually gave up after after seeing several thousand of these inodes experiencing problems (later I learned that fsck -y will automatically answer yes, which means I’ve improved my productivity several thousand times!)

I was pretty quick to assess the problem: the OCZ Vertex solid state drive where I’d installed Linux has been silently corrupting data as I’ve written to it. Most of the problem sectors are in my source directories, but a few happened to be in my KDE installation on disk. This caused oddities such as power management not loading and the absence of window borders.

So what goes on from here? I plan to replace the OCZ drive under warranty and rebuild LFS on my spinning disk drive, but this time I’ll take my own advice and start building from this LiveUSB Ubuntu install, with an up-to-date kernel and where .tar.xz files are recognized. Onward goes the adventure!

I’ve decided to try and upgrade my LFS system to the latest version of KDE (4.7.3 as of the time of this writing) and correspondingly needed to upgrade phonon-backend-gstreamer. Unfortunately, following the previous version’s compilation instructions provided this nasty message:

[ 4%] Building CXX object gstreamer/CMakeFiles/phonon_gstreamer.dir/audiooutput.cpp.o
In file included from /sources/phonon-backend-gstreamer-4.5.1/gstreamer/audiooutput.cpp:22:0:
/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:200:38: error: ‘NavigationMenu’ is not a member of ‘Phonon::MediaController’
/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:200:38: error: ‘NavigationMenu’ is not a member of ‘Phonon::MediaController’
/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:200:69: error: template argument 1 is invalid/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:262:11: error: ‘NavigationMenu’ is not a member of ‘Phonon::MediaController’
/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:262:11: error: ‘NavigationMenu’ is not a member of ‘Phonon::MediaController’/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:262:42: error: template argument 1 is invalid
/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:263:45: error: ‘Phonon::MediaController::NavigationMenu’ has not been declared
/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:317:11: error: ‘NavigationMenu’ is not a member of ‘Phonon::MediaController’
/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:317:11: error: ‘NavigationMenu’ is not a member of ‘Phonon::MediaController’/sources/phonon-backend-gstreamer-4.5.1/gstreamer/mediaobject.h:317:42: error: template argument 1 is invalid
make[2]: *** [gstreamer/CMakeFiles/phonon_gstreamer.dir/audiooutput.cpp.o] Error 1make[1]: *** [gstreamer/CMakeFiles/phonon_gstreamer.dir/all] Error 2make: *** [all] Error 2

To fix this issue, make sure you have the latest GStreamer and phonon-backend-xine installed. Then I followed some of the advice from this KDE forum topic.

If, like me, you installed Qt into /opt/qt, create a symbolic link into the qt directory pointing to your system’s latest version of phonon. For later success with kde-runtime, create links to the libphonon libraries in /opt/qt-4.7.1/lib to your recently compiled /usr/lib64 versions (adjust paths to /usr/lib on 32-bit systems):

# mv /opt/qt-4.7.1/include/phonon /tmp
# ln -snf /usr/include/phonon /opt/qt-4.7.1/include/phonon
# cd /opt/qt-4.7.1/lib
# rm -rf libphonon*
# ln -snf /usr/lib64/libphonon.so libphonon.so
# ln -snf /usr/lib64/libphonon.so.4 libphonon.so.4
# ln -snf /usr/lib64/libphonon.so.4.5.1 libphonon.so.4.5.1
# ln -snf /usr/lib64/libphononexperimental.so libphononexperimental.so
# ln -snf /usr/lib64/libphononexperimental.so.4 libphononexperimental.so.4
# ln -snf /usr/lib64/libphononexperimental.so.4.5.1 libphononexperimental.so.4.5.1

Then rerun the compilation process for phonon-backend-gstreamer and voila, no more errors. (You’ll probably still have more issues to work out, but this gets past the phonon-backend-gstreamer blockade.)

Seriously, I’d like to know, because this is a bit ridiculous.

For all the heavily encouraged coding styles out there, nearly all the open source software packages I’ve had to compile for Linux from Scratch have either

1. Insanely chatty defaults for compilation; that is, GCC provides ‘notices’ about seemingly minor points, or
2. A large number of warnings when compiling – unused variables, overloaded virtual functions, and deprecated features soon to disappear.

In the worst case, some of these warnings appear to be potential problems with the source. Leaving potentially uninitialized variables around seems to be a great way to run into runtime crashes if someone decides to use them. Overloading virtual functions with a different method signature has the same possible impact. And comparing signed and unsigned numbers is just a recipe for a crash or unpredictable behaviour down the line.

I just don’t get it. In my former development experiences, any compiler notifications were something to pay attention to. Usually when first developing an application, they were indicative of a typo, a forgotten variable or just general stupidity with the language. If a warning was absolutely unavoidable, it was specifically ignored in the build scripts with a clear explanation as to why.

So what’s the case with your programs? Have you noticed any stupid or insightful compiler messages scrolling past?

I’ve now been actively using my (Beyond) Linux from Scratch installation for about a week now, and it’s actually pretty neat to have something working that I built with just a general outline. Granted, the LFS guide is very well put together, but going beyond the basic console of a system requires a bit of time and effort.

In really any other distro, the package manager should really be your best friend (except when it breaks.) Even in a source-based Linux like Gentoo, Portage gives you a pretty decent idea of what’s installed and is able to keep track of dependencies. With LFS, there are really some times where I don’t want to have to locate and download seventeen .tar.bz2 files, and ./configure –prefix=/usr; make; make install to each one in sequence. What’s worse is when you run into three dependencies for a particular piece of software, and the first two install properly, but the third one depends on ten additional packages.

This is what building software in LFS looks like.

There are also some libraries that despite being built on an x86_64 system will come out as 32-bit, and require special compiler or configure flags in order to build a pure 64-bit version. LFS x86_64 does not really have patience for anything 32-bit. This is generally fine because you’re building most of the applications yourself, but you can’t “just run” any typical application unless it’s taken the architecture into account.

In summary, while it’s awesome to go to SourceForge and have the very latest version of a package, sometimes I just don’t feel like going through all those hoops and satisfying twenty conditions for a compile to take place. Perhaps I’m OK if your application uses a built-in library rather than relying on whatever happens to be installed in /usr/lib.

The takeaway from this is that besides providing the source, considerate developers should try and build an i686 and x86_64 binary from that same source. If your build system has issues or you find it painful to produce binary releases, remember that anyone attempting to follow the INSTALL file will run into the same pain points. Firefox, for example, has both i686 and x86_64 release tarchives. The 64-bit version works quite well on my LFS installation and it’s how I’m writing this post.

Using the instructions from the BLFS book with the latest available 3.6 build of Firefox, I was able to achieve success. I figured I’d try out 3.6 before going onto something with a terribly inflated version number, and as per usual, ran into some problems:

• Rebuild libpng-1.5.5 with APNG support. This is actually optional as I ended up commenting out the –with-system-png option in mozconfig.
• In the suggested mozconfig, comment out the last two lines:
  

  #ac_add_options --with-system-libxul
  #ac_add_options --with-libxul-sdk=/usr/lib/xulrunner-devel-1.9.2.13

  to create a standalone build.

• Apply the GCC patch from this Bugzilla report (direct download).
• Apply a partial patch from the Chromium project of all places. I’ve customized it here:
  

  
  # TLE Patch for Firefox/LFS

  diff -u a/gfx/ots/src/os2.cc b/gfs/ots/src/os2.cc
  — a/gfx/ots/src/os2.cc 2011-11-02 07:10:17.000000000 -0400
  +++ b/gfx/ots/src/os2.cc 2011-11-02 07:10:30.000000000 -0400
  @@ -5,6 +5,7 @@
  #include “os2.h”

  #include “head.h”
  +#include <cstddef>

  // OS/2 – OS/2 and Windows Metrics
  // http://www.microsoft.com/opentype/otspec/os2.htm

• Apply a GCC4.6-specific patch to fix various .cpp files. Some parts of the patch will fail; that’s expected.
• Manually edit layout/style/nsCSSRuleProcessor.cpp and go to line 1199. Change the source code as follows:
  
  const nsCaseInsensitiveStringComparator ciComparator;
  should become
  
  const nsCaseInsensitiveStringComparator ciComparator = nsCaseInsensitiveStringComparator();
  
• For the toolkit/components/places/src/SQLFunctions.cpp file, change line 126 to:
  const nsCaseInsensitiveStringComparator caseInsensitiveCompare = nsCaseInsensitiveStringComparator();
• In toolkit/crashreporter/google-breakpad/src/common/linux/language.cc, make sure line 51 is changed to:
  const CPPLanguage CPPLanguageSingleton = CPPLanguage();
• In toolkit/xre/nsAppRunner.cpp, line 990:
  
  static const nsXULAppInfo kAppInfo = nsXULAppInfo();
  
• While this is resolved in newer Firefox versions, copy security/coreconf/Linux2.6.mk to security/coreconf/Linux3.1.mk to add support for the 3.1 kernel.

Your reward will be a working Firefox installation:

So I wanted to install XMMS in Linux From Scratch, as it’s one of the more reliable MP3 players and one of the first multimedia Linux apps I’ve used. It’s very reminiscent of Winamp 2:

If you would also like to get it installed, you’ll need the source and glib-1.2.10. Then, check out a common problem when installing glib, and a patch to fix the ./configure step.

kdebindings

Are you running into the following problem when compiling kdebindings? Well, you’re probably not, because you picked a saner distribution than LFS, but here goes anyway!

ASSERT failure in QList::at: “index out of range”, file /qt/trunk/include/QtCore/qlist.h, line 456
/bin/sh: line 1: 7841 Aborted (core dumped)

From http://old.nabble.com/Smokegen-core-dump-td30797484.html, you can fix this with a patch to indexedstring.cpp:

--- generator/parser/indexedstring.cpp.orig 2011-02-23 22:12:38.695255708 +0100
+++ generator/parser/indexedstring.cpp 2011-02-24 02:36:09.035361151 +0100
@@ -195,12 +195,15 @@
}

QByteArray IndexedString::byteArray() const {
+  qDebug() << "strings()->size():" << strings()->size() << ", m_index:" << m_index;
if(!m_index)
return QByteArray();
else if((m_index & 0xffff0000) == 0xffff0000)
return QString(QChar((char)m_index & 0xff)).toUtf8();
-  else
+  else if (m_index < strings()->size())
return strings()->at(m_index).toUtf8(); /*arrayFromItem(globalIndexedStringRepository->itemFromIndex(m_index));*/
+  else
+    return QByteArray();
}

unsigned int IndexedString::hashString(const char* str, unsigned short length) {

I ended up removing the first qDebug() line before the if statement as I don’t need my compiler to be that chatty – I just need this package to compile properly. Reconfigure and attempt to make with:

cmake -DCMAKE_INSTALL_PREFIX=$KDE4_PREFIX \
    -DKDE_DEFAULT_HOME=.kde4 \
    -DSYSCONF_INSTALL_DIR=/etc/kde4 \
    .. &&
make

kdebase-runtime

You can patch away your problems if you run into the following message:

[ 39%] Building CXX object kioslave/nfs/CMakeFiles/kio_nfs.dir/kio_nfs.o
In file included from /sources/kdebase-runtime-4.6.0/kioslave/nfs/kio_nfs.cpp:21:0:
/sources/kdebase-runtime-4.6.0/kioslave/nfs/kio_nfs.h:33:21: fatal error: rpc/rpc.h: No such file or directory
compilation terminated.

First, get libtirpc installed to make this work, but then again, you could have just guessed that you needed it, right?

Used under Creative Commons NC license from zhenech

There are some LFS-specific instructions to follow before libtirpc will compile:

• Unpack glibc-2.14.1
• In its directory, execute:

  mkdir -p /usr/include/rpc{,svc}
  cp sunrpc/rpc/*.h /usr/include/rpc
  cp nis/rpcsvc/*.h /usr/include/rpcsvc

• Compile libtirpc with ./configure --prefix=/usr && make && make install

Then from Sourcemage, linking to an old Bugzilla installation:

diff --git a/kioslave/nfs/CMakeLists.txt b/kioslave/nfs/CMakeLists.txt
index b973a73..6556769 100644
--- a/kioslave/nfs/CMakeLists.txt
+++ b/kioslave/nfs/CMakeLists.txt
@@ -3,8 +3,8 @@ set(kio_nfs_PART_SRCS kio_nfs.cpp mount_xdr.c nfs_prot_xdr.c )

 kde4_add_plugin(kio_nfs ${kio_nfs_PART_SRCS})

-
-target_link_libraries(kio_nfs   ${KDE4_KIO_LIBS})
+include_directories(/usr/include/tirpc)
+target_link_libraries(kio_nfs   ${KDE4_KIO_LIBS} tirpc)

 install(TARGETS kio_nfs  DESTINATION ${PLUGIN_INSTALL_DIR} )

Once this is complete you should be able to get kdebase-runtime compiled.