Archive

Posts Tagged ‘ext3’

Create a virtual hard drive volume within a file in Linux

June 15th, 2014 No comments

If you are not familiar with the concept of virtual hard drive volumes, sometimes called file containers, they are basically regular looking files that can be used by your computer as if they were real hard drives. So for example you could have a file called MyDrive.img on your computer and with a few quick actions it would appear as though you had just plugged in an external USB stick or hard drive into your computer. It acts just like a normal, physical, drive but whenever you copy anything to that location the copied files are actually being written to the MyDrive.img file behind the scenes. This is not unlike the dmg files you would find on a Mac or even something akin to TrueCrypt file containers.

Why would I want this?

There are a number of reasons why you may be interested in creating virtual volumes. From adding additional swap space to your computer (i.e. something similar to a page file on Windows without needing to create a new hard drive partition) to creating portable virtual disk drives to back up files to, or even just doing it because this is Linux and it’s kind of a neat thing to do.

What are the steps to creating a file container?

The process seems a bit strange but it’s actually really straight forward.

  1. Create a new file to hold the virtual drive volume
      (Optional) Initialize it by filling it with data
  2. Format the volume
  3. Mount the volume and use it

Create a new file to hold the virtual drive volume

There are probably a million different ways to do this but I think the most simple way is to run the following command from a terminal:

fallocate -l <size> <file to create>

So let’s say you wanted to create a virtual volume in a file called MyDrive.img in the current directory with a size of 500MiB. You would simply run the following command:

fallocate -l 500M MyDrive.img

You may notice that this command finishes almost instantly. That’s because while the system created a 500MiB file it didn’t actually write 500MiB worth of data to the file.

This is where the optional step of ‘initializing’ the file comes into play. To be clear you do not need to do this step at all but it can be good practice if you want to clean out the contents of the allocated space. For instance if you wanted to prevent someone from easily noticing when you write data to that file you may pre-fill the space with random data to make it more difficult to see or you may simply want to zero out that part of the hard drive first.

Anyway if you choose to pre-fill the file with data the easiest method is to use the dd command. PLEASE BE CAREFUL – dd is often nicknamed disk destroyer because it will happily overwrite any data you tell it to, including the stuff you wanted to keep if you make a mistake typing the command!

To fill the file with all zeros simply run this command:

dd if=/dev/zero of=<your file> bs=1M count=<your file size in MiB>

So for the above file you would run:

dd if=/dev/zero of=MyDrive.img bs=1M count=500

If you want to fill it with random data instead just swap /dev/zero for /dev/urandom or /dev/random in the command:

dd if=/dev/urandom of=MyDrive.img bs=1M count=500

Format and mount the virtual volume

Next up we need to give the volume a filesystem. You can either do this via the command line or using a graphical tool. I’ll show you an example of both.

From the terminal you would run the appropriate mkfs command on the file. As an example this will format the file above using the ext3 filesystem:

mkfs -t ext3 MyDrive.img

You may get a warning that looks like this

MyDrive.img is not a block special device.
Proceed anyway? (y,n)

Simply type the letter ‘y’ and press Enter. With any luck you’ll see a bunch of text telling you exactly what happened and you now have a file that is formatted with ext3!

If you would rather do things the graphical way you could use a tool like Disks (gnome-disk-utility) to format the file.

From the menu in Disks, select Disks -> Attach Disk Image and browse to your newly created file (i.e. MyDrive.img).

Don't forget to uncheck the box!

Don’t forget to uncheck the box!

Be sure to uncheck “Set up read-only loop device”. If you leave this checked you won’t be able to format or write anything to the volume. Select the file and click Attach.

This will attach the file, as if it were a real hard drive, to your computer:

MyDriveAttached

Next we need to format the volume. Press the little button with two gears right below the attached volume and click Format. Make sure you do this for the correct ‘drive’ so that you don’t accidentally format your real hard drive!

Make sure you're formatting the correct drive!

Make sure you’re formatting the correct drive!

From this popup you can select the filesystem type and even name the drive. You may also use the “Erase” option to write zeros to the file if you wanted to do it here instead of via the terminal as shown previously. In the image above the settings will format the drive using the ext4 filesystem. Click Format, confirm the action and you’re done. Disks will format the file and even auto-mount it for you. You can now copy files to your mounted virtual drive. When you’re done simply eject the drive like normal or press the square Stop button in Disks. To use that same volume again in the future just re-attach the disk image using the steps above.

To mount the formatted file from the terminal you will need to first create a folder to mount it to. Let’s say we wanted to mount it to the folder /media/MyDrive. First create the folder there:

sudo mkdir /media/MyDrive

Next mount the file to the folder:

sudo mount -t auto -o loop MyDrive.img /media/MyDrive/

Now you can copy files to the drive just like before. When you’re finished unmount the volume by running this command:

sudo umount /media/MyDrive/

And there you have it. Now you know how to create virtual volume files that you can use for just about anything and easily move from computer to computer.

This post originally appeared on my personal website here.




I am currently running a variety of distributions, primarily Linux Mint 17.
Previously I was running KDE 4.3.3 on top of Fedora 11 (for the first experiment) and KDE 4.6.5 on top of Gentoo (for the second experiment).
Check out my profile for more information.

Of filesystems and partitions

August 25th, 2009 1 comment

Following from my last post about finalizing all of those small little choices I will now continue along that line but discuss the merits of the various filesystems that Linux allows me to choose from, as well as discuss how I am going to partition my drive.

Filesystem?

For a Windows or Mac user the filesystem is something they will probably never think about in their daily computing adventures. That is mostly because there really isn’t a choice in the matter. As a Windows user the only time I actually have to worry about the filesystem is when I’m formatting a USB drive. For my hard drives the choices are NTFS, NTFS, and.. oh yeah NTFS. My earliest recollection of what a filesystem is happened when my Windows 98 machine had crashed and I had to wait while the machine forced a filesystem check on the next start up. More recently FAT32 has gotten in my way with it’s 4GB file size limitation.

You mean we get a choice?

Linux seems to be all about choice so why would it be surprising that you don’t get to pick your own filesystem? The main contenders for this choice are ext2, ext3, ext4, ReiserFS, JFS, XFS, Btrfs and in special places Fat16, Fat32, NTFS, and swap.

Ext2

According to the great internet bible, ext2 stands for the second extended filesystem. It was designed as a practical replacement for the original, but very old, Linux filesystem. If I may make an analogy for Windows users, ext2 seems to be the Linux equivalent to Fat32, only much better. This filesystem is now considered mostly outdated and only really still used in places where journaling is not always appropriate; for example on USB drives. Ext2 can be used on the /boot partition and is supported by GRUB.

Ext2 Features

  • Introduced: January 1993
  • File allocation: bitmap (free space), table (metadata)
  • Max file size: 16 GiB – 64 TiB
  • Max number of files: 10^18
  • Max filename length: 255 characters
  • Max volume size: 2 TiB – 32 TiB
  • Date range: December 14, 1901 – January 18, 2038

Ext 3

Ext3 is the successor to ext2 and removed quite a few of the limitations and also added a number of new features, most important of which was journaling. As you might have guessed it’s full name is the third extended filesystem. While ext3 is generally considered to be much better than ext2 there are a couple of problems with it. While ext3 does not have to scan itself after a crash, something that ext2 did have to do, it also does not have a an online defragmenter. Also because ext3 was primarily designed to shore up some of ext2′s faults, it is not the cleanest implementation and can actually have worse performance than ext2 in some situations. Ext3 is still the most popular Linux filesystem and is only now slowly being replaced by its own successor ext4. Ext3 can be used on the /boot partition and is fully supported by GRUB.

Ext3 Features

  • Introduced: November 2001
  • Directory contents: Table, hashed B-tree with dir_index enabled
  • File allocation: bitmap (free space), table (metadata)
  • Max file size: 16 GiB – 2 TiB
  • Max number of files: Variable, allocated at creation time
  • Max filename length: 255 characters
  • Max volume size: 2 TiB – 16 TiB
  • Date range: December 14, 1901 – January 18, 2038

Ext4

Ext4 is the next in the extended filesystem line and the successor to ext3. This addition proved to be quite controversial initially due to its implementation of delayed allocation which resulted in a very long time before writes. However ext4 achieves very fast read time thanks to this delayed allocation and overall it performs very well when compared to ext3. Ext4 is slowly taking over as the defacto filesystem and is actually already the default in many distributions (Fedora included). Ext4 cannot be used on the /boot partition because of GRUB, meaning a separate /boot partition with a different filesystem must be made.

Ext4 Features

  • Introduced: October 21, 2008
  • Directory contents: Linked list, hashed B-tree
  • File allocation: Extents/Bitmap
  • Max file size: 16 TiB
  • Max number of files: 4 billion
  • Max filename length: 256 characters
  • Max volume size: 1 EiB
  • Date range: December 14, 1901 – April 25, 2514

ReiserFS

Created by Hans ‘I didn’t murder my wife’ Reiser, in 2001 this filesystem was very promising for its performance but has since been mostly abandoned  by the Linux community. It’s initial claim to fame was as the first journaling filesystem to be included within the Linux kernel. Carefully configured, ReiserFS can achieve 10 to 15x the performance of ext2 and ext3. ReiserFS can be used on the /boot partition and is supported by GRUB.

ReiserFS Features

  • Introduced: 2001
  • Directory contents: B+ tree
  • File allocation: Bitmap
  • Max file size: 8 TiB
  • Max number of files: ~4 billion
  • Max filename length: 4032 characters theoretically, 255 in practice
  • Max volume size: 16 TiB
  • Date range: December 14, 1901 – January 18, 2038

Journaled File System (JFS)

Developed by IBM, JFS sports many features and is very advanced for its time of release. Among these features are extents and compression. Though not as widely used as other filesystems, JFS is very stable, reliable and fast with low CPU overhead. JFS can be used on the /boot partition and is supported by GRUB.

JFS Features

  • Introduced: 1990 and 1999
  • Directory contents: B+ tree
  • File allocation: Bitmap/extents
  • Max file size: 4 PiB
  • Max number of files: no limit
  • Max filename length: 255 characters
  • Max volume size: 32 PiB

XFS

Like JFS, XFS is one of the oldest and most refined journaling filesystems available on Linux. Unlike JFS, XFS supports many additional advanced features such as striped allocation to optimize RAID setups, delayed allocation to optimize disk data placement, sparse files, extended attributes, advanced I/O features, volume snapshots, online defragmentation, online resizing, native backup/restore and disk quotas. The only real downsides XFS suffers from are its inability to shrink partitions, a difficult to implement un-delete, and quite a bit of overhead when new directories are created and directories are deleted. XFS is supported by GRUB, and thus can be used as the /boot partition, but there are reports that it is not very stable.

XFS Features

  • Introduced: 1994
  • Directory contents: B+ tree
  • File allocation: B+ tree
  • Max file size: 8 EiB
  • Max filename length: 255 characters
  • Max volume size: 16 EiB

Btrfs

Btrfs, or “B-tree FS” or “Butter FS”, is a next generation filesystem will all of the bells and whistles. It is meant to fill the gap of lacking enterprise filesystems on Linux and is being spearheaded by Oracle. Wikipedia lists its new promised features as online balancing, subvolumes (separately-mountable filesystem roots), object-level (RAID-like) functionality, and user-defined transactions among other things. It’s stable version is currently being incorporated into mainstream Linux kernels.

Btrfs Features

  • Introduced: 20xx
  • Directory contents: B+ tree
  • File allocation: extents
  • Max file size: 16 EiB
  • Max number of files: 2^64
  • Max filename length: 255 characters
  • Max volume size: 16 EiB

So what’s it all mean?

Well there you have it, a quick and concise rundown of the filesystem options for your mainstream Linux install. But what exactly does all of this mean? Well, as they say, a picture speaks a thousand words. Many people have done performance tests against the mainstream filesystems and many conclusions have been drawn as to what is the best in many different circumstances. As I assume most people would chose either XFS, ext3, ext4 or maybe even Btrs if they were a glutton for punishment I just happen to have found some interesting pictures to show off the comparison!

Rather than tell you which filesystem to pick I will simply point out a couple of links and tell you that while I think XFS is a very underrated filesystem I, like most people, will be going with ext4 simply because it is currently the best supported.

Links (some have pictures!):

EXT4, Btrfs, NILFS2 Performance Benchmarks

Filesystems (ext3, reiser, xfs, jfs) comparison on Debian Etch

Linux Filesystem Performance Comparison for OLTP with Ext2, Ext3, Raw, and OCFS on Direct-Attached Disks using Oracle 9i Release 2

Hey! You forgot about partitions!

No, I didn’t.

Yes you did!

OK, fine… So as Jon had pointed out in a previous post the Linux filesystem is broken down into a series of more or less standard mount points. The only requirements for Fedora, my distribution of choice, and many others are that at least these three partitions exist: /boot for holding the bootable kernels, / (root) for everything else, and a swap partition to move things in and out of RAM. I was thinking about creating a fourth /home partition but I gave up when I realized I didn’t know enough about Linux to determine a good partition size for that.

OK, so break it down

/boot

Fedora recommends that this partition is a minimum of 100MB in size. Even though kernels are each roughly 6MB in size it is better to be safe than sorry! Also because ext4 is not supported by GRUB I will be making this partition ext3.

LVM

I know what you’re thinking, what the hell is LVM? LVM stands for Logical Volume Manager and allows a single physical partition to hold many virtual partitions. I will be using LVM to store the remainder of my partitions wrapped inside of a physical encrypted partition. At least that’s the plan.

swap

Fedora recommends using the following formula to calculate how much swap space you need.

If M < 2
S = M *2
Else
S = M + 2

Where M is the amount of memory you have and S is the swap partition size in GiB. So for example the machine I am using for this experiment has 4 GiB of RAM. That translates to a swap partition of 6 GiB. If your machine only has 1 GiB of RAM then the formula would translate to 2 GiB worth of swap space. 6 GiB seems a bit overkill for a swap partition but what do I know?

/ (root)

And last but not least the most important part, the root partition. This partition will hold everything else and as such will be taking up the rest of my drive. On the advice of Fedora I am going to leave 10 GiB of LVM disk space unallocated for future use should the need arise. This translates to a root partition of about ~300 GiB, plenty of space. Again I will be formatting this partition using ext4.

Well there you go

Are you still with me? You certainly are a trooper! If you have any suggestions as to different disk configurations please let me know. I understand a lot of this in theory but if you have actual experience with this stuff I’d love to hear from you!