Archive

Posts Tagged ‘NTFS’

TrueCrypt, kernel compilation and where’d /boot go?

October 5th, 2009 1 comment

Since I’ve installed Gentoo, I haven’t had access to my other drives. One of them is an NTFS-formatted WD Raptor, and the other is a generic Seagate 300GB drive that contains my documents, pictures and Communist propaganda inside a TrueCrypt partition. Getting the Raptor to work was fairly simple (as far as Gentoo goes) – I added the entry to my /etc/fstab file and then manually mounted the partition:

/dev/sdb1               /mnt/raptor     ntfs-3g         noatime         0 0

The TrueCrypt drive proved to be more of an issue. After installing the software and attempting to mount the partition, I encountered an error:

device-mapper: reload ioctl failed: Invalid argument 
Command failed

A quick Bing and the Gentoo Wiki described this problem exactly, with the caveat that I had to recompile my kernel to add support for LRW and XTS support. Into the kernel configuration I went – the only difference I noticed is that LRW and XTS are considered “EXPERIMENTAL” but aren’t noted as such on the requirements page:

Kernel configuration options

(This may come down to an x64 vs. x86 issue, but I haven’t run into any issues with these options enabled (yet!))

Of course, then came the make && make modules_install commands, which didn’t take too long to complete. The question then became, how do I install the new kernel? Looking in my /boot partition, I only had a few template files – and not the kernel itself or any grub settings. Essentially, /boot had nothing in it but the system still launches properly!

I then tried mounting /dev/sda1 manually, and the kernel and grub.conf showed up properly in the mountpoint. Something is obviously wrong with the way my system remounts /boot during the startup process, but at least now I’m able to install the new kernel. After copying /arch/x86_64/boot/bzImage to the newly available directory, I rebooted and the new kernel was picked up properly. TrueCrypt now lets me open, create and delete files from /media/truecrypt1, and automatically uses ntfs-3g support to accomplish this.

Overall, I’m pretty pleased at how easily I can recompile a kernel, and installation was seamless once I figured out that /boot wasn’t pointing to the right location. I expect I’ll try and manually remove the directory from /dev/sda3 and see if that makes a difference.




I am currently running various *BSD variants for this Experiment.
I currently run a mix of Windows, OS X and Linux systems for both work and personal use.
For Linux, I prefer Ubuntu LTS releases without Unity and still keep Windows 7 around for gaming.
Check out my profile for more information.

Mounting an NTFS-formatted External Drive

September 20th, 2009 7 comments

I have a Western Digital 250GB NTFS-formatted external hard drive that I use primarily to store backups of my Windows machine. Since I’m away from my house for a couple of days, I used the drive to bring along some entertainment, but encountered some troubles getting Debian Lenny to play nice with it:

mount-errorAfter searching around for a bit, I found a helpful thread on the Ubuntu forums that explained that this problem could be caused by a few different things. First, with the drive plugged in, I ran

sudo fdisk -l

from the terminal, which brought up a summary of all disks currently recognized by the machine:

jon@debtop:/$ sudo fdisk -l
Disk /dev/sda: 40.0 GB, 40007761920 bytes
255 heads, 63 sectors/track, 4864 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xcccdcccd
 Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          31      248976   83  Linux
/dev/sda2              32        4864    38821072+  83  Linux

Disk /dev/dm-0: 39.7 GB, 39751725568 bytes
255 heads, 63 sectors/track, 4832 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1: 38.0 GB, 38067503104 bytes
255 heads, 63 sectors/track, 4628 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table
Disk /dev/dm-2: 1681 MB, 1681915904 bytes
255 heads, 63 sectors/track, 204 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Disk /dev/dm-2 doesn't contain a valid partition table
Disk /dev/sdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x5b6ac646

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       30401   244196001    7  HPFS/NTFS

Judging by the size of the drives, I figured out that the OS saw my drive at the location /dev/sdb, and the partition that I wanted to mount (the only partition on the drive) at the location /dev/sdb1.

Now, to determine why Linux wasn’t mounting the drive, I checked the fstab file at /etc/fstab to see if there was some other entry for sdb that was preventing it from mounting correctly:

# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
/dev/mapper/debtop-root /               ext3    errors=remount-ro 0       1
/dev/sda1       /boot           ext2    defaults        0       2
/dev/mapper/debtop-swap_1 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

Since there was no entry there that should have overwritten sdb, I gave up on that line of inquiry, and decided to try manually mounting the drive. I know that Debian can read ntfs drives using the -t ntfs argument for the mount command, so I navigated over to the /media/ directory and created a folder to mount the drive in:

jon@debtop:/$ cd /media/
jon@debtop:/media$ sudo mkdir WesternDigital
jon@debtop:/media$ ls
cdrom  cdrom0  floppy  floppy0  WesternDigital
jon@debtop:/media$ sudo mount -t ntfs /dev/sdb1 /media/WesternDigital/
jon@debtop:/media$ sudo -s
root@debtop:/media# cd WesternDigital
root@debtop:/media/WesternDigital# ls
KeePass.kdbx  nws  $RECYCLE.BIN  System Volume Information  workspace.tc

As you can see, the contents of my external drive were now accessible in the location where they ought to have been if Debian had correctly mounted the drive when it was plugged in. The only caveat to the process is that the mount function is available only to root users, meaning that the mountpoint was created by root, and my user account lacks the necessary permissions to read or write to the external drive:

no-permissions

I figured that this issue could be solved by using chmod to grant all users read and write permissions to the mountpoint:

root@debtop:/media# chmod +rw WesternDigital
chmod: changing permissions of `WesternDigital': Read-only file system

Well what the hell does that mean? According to this post (again on the Ubuntu forums), the ntfs support in Linux is experimental, and as such, all ntfs drives are mounted as read only. Specifically, this drive is owned by the root user, and has only read and execute permisions, but lacks write permissions.

According to this thread on the slax.org forums, there is another ntfs driver for Linux called ntfs-3g that will allow me full access to my ntfs-formatted drive. After sucessfully adding the ntfs-3g drivers to my system, I dismounted the drive, and attempted to re-mount it with the following command:

mount -t ntfs-3g /dev/sdb1 /media/WesternDigital

This time, the mount command appeared to almost work, but I got an error message along the way, indicating that the drive had not been properly dismounted the last time it was used on Windows, and giving me the option to force the mount:

Mount is denied because NTFS is marked to be in use. Choose one action:

Choice 1: If you have Windows then disconnect the external devices by
 clicking on the 'Safely Remove Hardware' icon in the Windows
 taskbar then shutdown Windows cleanly.

Choice 2: If you don't have Windows then you can use the 'force' option for
 your own responsibility. For example type on the command line:

 mount -t ntfs-3g /dev/sdb1 /media/WesternDigital -o force

Well, since I didn”t have a Windows box lying about that I can use to dismount the drive properly, I’ll took a shot at using the force option. After warning me again that it was resetting the log file and forcing the mount, the machine finally mounted my drive with full permissions for the owner, group, and other users!

drwxrwxrwx 1 root root  4096 2009-09-18 15:40 WesternDigital

After a couple of manual tests, I confirmed that both my user account and the root user had full read/write/execute access to this drive, and that I could use it like any other drive that the system has access to. Further, thanks to the painful XBMC install process, I already had the codecs required to play all of the TV shows that I brought along.

Of filesystems and partitions

August 25th, 2009 1 comment

Following from my last post about finalizing all of those small little choices I will now continue along that line but discuss the merits of the various filesystems that Linux allows me to choose from, as well as discuss how I am going to partition my drive.

Filesystem?

For a Windows or Mac user the filesystem is something they will probably never think about in their daily computing adventures. That is mostly because there really isn’t a choice in the matter. As a Windows user the only time I actually have to worry about the filesystem is when I’m formatting a USB drive. For my hard drives the choices are NTFS, NTFS, and.. oh yeah NTFS. My earliest recollection of what a filesystem is happened when my Windows 98 machine had crashed and I had to wait while the machine forced a filesystem check on the next start up. More recently FAT32 has gotten in my way with it’s 4GB file size limitation.

You mean we get a choice?

Linux seems to be all about choice so why would it be surprising that you don’t get to pick your own filesystem? The main contenders for this choice are ext2, ext3, ext4, ReiserFS, JFS, XFS, Btrfs and in special places Fat16, Fat32, NTFS, and swap.

Ext2

According to the great internet bible, ext2 stands for the second extended filesystem. It was designed as a practical replacement for the original, but very old, Linux filesystem. If I may make an analogy for Windows users, ext2 seems to be the Linux equivalent to Fat32, only much better. This filesystem is now considered mostly outdated and only really still used in places where journaling is not always appropriate; for example on USB drives. Ext2 can be used on the /boot partition and is supported by GRUB.

Ext2 Features

  • Introduced: January 1993
  • File allocation: bitmap (free space), table (metadata)
  • Max file size: 16 GiB – 64 TiB
  • Max number of files: 10^18
  • Max filename length: 255 characters
  • Max volume size: 2 TiB – 32 TiB
  • Date range: December 14, 1901 – January 18, 2038

Ext 3

Ext3 is the successor to ext2 and removed quite a few of the limitations and also added a number of new features, most important of which was journaling. As you might have guessed it’s full name is the third extended filesystem. While ext3 is generally considered to be much better than ext2 there are a couple of problems with it. While ext3 does not have to scan itself after a crash, something that ext2 did have to do, it also does not have a an online defragmenter. Also because ext3 was primarily designed to shore up some of ext2’s faults, it is not the cleanest implementation and can actually have worse performance than ext2 in some situations. Ext3 is still the most popular Linux filesystem and is only now slowly being replaced by its own successor ext4. Ext3 can be used on the /boot partition and is fully supported by GRUB.

Ext3 Features

  • Introduced: November 2001
  • Directory contents: Table, hashed B-tree with dir_index enabled
  • File allocation: bitmap (free space), table (metadata)
  • Max file size: 16 GiB – 2 TiB
  • Max number of files: Variable, allocated at creation time
  • Max filename length: 255 characters
  • Max volume size: 2 TiB – 16 TiB
  • Date range: December 14, 1901 – January 18, 2038

Ext4

Ext4 is the next in the extended filesystem line and the successor to ext3. This addition proved to be quite controversial initially due to its implementation of delayed allocation which resulted in a very long time before writes. However ext4 achieves very fast read time thanks to this delayed allocation and overall it performs very well when compared to ext3. Ext4 is slowly taking over as the defacto filesystem and is actually already the default in many distributions (Fedora included). Ext4 cannot be used on the /boot partition because of GRUB, meaning a separate /boot partition with a different filesystem must be made.

Ext4 Features

  • Introduced: October 21, 2008
  • Directory contents: Linked list, hashed B-tree
  • File allocation: Extents/Bitmap
  • Max file size: 16 TiB
  • Max number of files: 4 billion
  • Max filename length: 256 characters
  • Max volume size: 1 EiB
  • Date range: December 14, 1901 – April 25, 2514

ReiserFS

Created by Hans ‘I didn’t murder my wife’ Reiser, in 2001 this filesystem was very promising for its performance but has since been mostly abandoned  by the Linux community. It’s initial claim to fame was as the first journaling filesystem to be included within the Linux kernel. Carefully configured, ReiserFS can achieve 10 to 15x the performance of ext2 and ext3. ReiserFS can be used on the /boot partition and is supported by GRUB.

ReiserFS Features

  • Introduced: 2001
  • Directory contents: B+ tree
  • File allocation: Bitmap
  • Max file size: 8 TiB
  • Max number of files: ~4 billion
  • Max filename length: 4032 characters theoretically, 255 in practice
  • Max volume size: 16 TiB
  • Date range: December 14, 1901 – January 18, 2038

Journaled File System (JFS)

Developed by IBM, JFS sports many features and is very advanced for its time of release. Among these features are extents and compression. Though not as widely used as other filesystems, JFS is very stable, reliable and fast with low CPU overhead. JFS can be used on the /boot partition and is supported by GRUB.

JFS Features

  • Introduced: 1990 and 1999
  • Directory contents: B+ tree
  • File allocation: Bitmap/extents
  • Max file size: 4 PiB
  • Max number of files: no limit
  • Max filename length: 255 characters
  • Max volume size: 32 PiB

XFS

Like JFS, XFS is one of the oldest and most refined journaling filesystems available on Linux. Unlike JFS, XFS supports many additional advanced features such as striped allocation to optimize RAID setups, delayed allocation to optimize disk data placement, sparse files, extended attributes, advanced I/O features, volume snapshots, online defragmentation, online resizing, native backup/restore and disk quotas. The only real downsides XFS suffers from are its inability to shrink partitions, a difficult to implement un-delete, and quite a bit of overhead when new directories are created and directories are deleted. XFS is supported by GRUB, and thus can be used as the /boot partition, but there are reports that it is not very stable.

XFS Features

  • Introduced: 1994
  • Directory contents: B+ tree
  • File allocation: B+ tree
  • Max file size: 8 EiB
  • Max filename length: 255 characters
  • Max volume size: 16 EiB

Btrfs

Btrfs, or “B-tree FS” or “Butter FS”, is a next generation filesystem will all of the bells and whistles. It is meant to fill the gap of lacking enterprise filesystems on Linux and is being spearheaded by Oracle. Wikipedia lists its new promised features as online balancing, subvolumes (separately-mountable filesystem roots), object-level (RAID-like) functionality, and user-defined transactions among other things. It’s stable version is currently being incorporated into mainstream Linux kernels.

Btrfs Features

  • Introduced: 20xx
  • Directory contents: B+ tree
  • File allocation: extents
  • Max file size: 16 EiB
  • Max number of files: 2^64
  • Max filename length: 255 characters
  • Max volume size: 16 EiB

So what’s it all mean?

Well there you have it, a quick and concise rundown of the filesystem options for your mainstream Linux install. But what exactly does all of this mean? Well, as they say, a picture speaks a thousand words. Many people have done performance tests against the mainstream filesystems and many conclusions have been drawn as to what is the best in many different circumstances. As I assume most people would chose either XFS, ext3, ext4 or maybe even Btrs if they were a glutton for punishment I just happen to have found some interesting pictures to show off the comparison!

Rather than tell you which filesystem to pick I will simply point out a couple of links and tell you that while I think XFS is a very underrated filesystem I, like most people, will be going with ext4 simply because it is currently the best supported.

Links (some have pictures!):

EXT4, Btrfs, NILFS2 Performance Benchmarks

Filesystems (ext3, reiser, xfs, jfs) comparison on Debian Etch

Linux Filesystem Performance Comparison for OLTP with Ext2, Ext3, Raw, and OCFS on Direct-Attached Disks using Oracle 9i Release 2

Hey! You forgot about partitions!

No, I didn’t.

Yes you did!

OK, fine… So as Jon had pointed out in a previous post the Linux filesystem is broken down into a series of more or less standard mount points. The only requirements for Fedora, my distribution of choice, and many others are that at least these three partitions exist: /boot for holding the bootable kernels, / (root) for everything else, and a swap partition to move things in and out of RAM. I was thinking about creating a fourth /home partition but I gave up when I realized I didn’t know enough about Linux to determine a good partition size for that.

OK, so break it down

/boot

Fedora recommends that this partition is a minimum of 100MB in size. Even though kernels are each roughly 6MB in size it is better to be safe than sorry! Also because ext4 is not supported by GRUB I will be making this partition ext3.

LVM

I know what you’re thinking, what the hell is LVM? LVM stands for Logical Volume Manager and allows a single physical partition to hold many virtual partitions. I will be using LVM to store the remainder of my partitions wrapped inside of a physical encrypted partition. At least that’s the plan.

swap

Fedora recommends using the following formula to calculate how much swap space you need.

If M < 2
S = M *2
Else
S = M + 2

Where M is the amount of memory you have and S is the swap partition size in GiB. So for example the machine I am using for this experiment has 4 GiB of RAM. That translates to a swap partition of 6 GiB. If your machine only has 1 GiB of RAM then the formula would translate to 2 GiB worth of swap space. 6 GiB seems a bit overkill for a swap partition but what do I know?

/ (root)

And last but not least the most important part, the root partition. This partition will hold everything else and as such will be taking up the rest of my drive. On the advice of Fedora I am going to leave 10 GiB of LVM disk space unallocated for future use should the need arise. This translates to a root partition of about ~300 GiB, plenty of space. Again I will be formatting this partition using ext4.

Well there you go

Are you still with me? You certainly are a trooper! If you have any suggestions as to different disk configurations please let me know. I understand a lot of this in theory but if you have actual experience with this stuff I’d love to hear from you!