Tuesday, April 1, 2014

terminal-colors.d

I have already talked about it at g+. For util-linux we don't want to follow the current strange behavior when command-line utils have disabled colors by default, but distributions use shell aliases in profile files to revert the default.

It does not mean that we want ignore color blind people, obscure terminals or people who just don't like colors. The solution is to provide simple, elegant and package independent way how to control (disable) colorized output. And yes, aliases are not elegant.

The next util-linux release is going to introduce /etc/terminal-colors.d directory
to control colors for terminal utils. It's pretty simple:

Synopsis:
/etc/terminal-colors.d/[name.]disable
/etc/terminal-colors.d/name.enable
Examples:
  •  disable colors for all utils:
    # touch /etc/terminal-colors.d/disable 
    
  • disable colors for dmesg(1):
    # touch /etc/terminal-colors.d/dmesg.disable
  • disable colors for all utils except dmesg(1):
    # touch /etc/terminal-colors.d/disable
    # touch /etc/terminal-colors.d/dmesg.enable
For utils from util-linux the command line option --color[={auto,never,always}] overrides the setting from terminal-colors.d (so if you for some reason love shell aliases then it's still possible...).

The terminal-colors.d concept is completely open, it would be nice to see it supported in another projects (hint: gcc, grep, coreutils:-) too. It's so trivial that all you need is a few access(F_OK) calls. The thing we maintain in util-linux is terminal-colors.d.5 man page only.

Maybe later for util-linux we will also use the directory (or/and copy in $HOME) to store color schemes.

Keep your Bike-sheds colorized! :-)

Wednesday, February 5, 2014

stable mutt-kz 1.5.22.1

The first official mutt-kz release with tarballs is available now. If you have no idea about mutt-kz: it is mutt fork with virtual folders, tags and fulltext search (based on notmuch), supports INBOX->archive use case, integrated index-color and sidebar patches. All changes to the original mutt are backported to mutt-kz.

I'm really happy that there is a community around the project with testers and contributors. The mutt-kz mailing list has 80 members now. Thanks!

Features:

  • virtual folder (based on notmuch queries) - defined in .muttrc
  • virtual folders on-the-fly generated from queries
  • command to see entire thread for specific message (including already archived messages or messages excluded from the current virtual folder)
  • UI to modify notmuch labels 
  • labels transformation to special chars (e.g. funny unicode text icons) in mutt index
  • new %g and %Gx mutt index and pager format sequences to display notmuch labels 
  • special labels for sent emails 
  • really smart fulltext search (thanks to notmuch and Xapian), for example: "from:kzak tag:ul date:1year.." to search for my posts to util-linux (ul tag) mailing list in last year. 
  • nested if-else conditional and date format expressions (for index format)
  • new mutt pattern modifier for labels 
  • all implemented by regular mutt commands (no hacks) so usable in mutt hooks and macros 
  • you really don't have to delete messages, just move things to archive, notmuch is fast enough to work with large sets of messages 
  • sidebar to show mailboxes or virtual folders 
  • and more... 
my .muttrc: http://people.redhat.com/kzak/mutt-kz/muttrc

Tuesday, October 15, 2013

util-linux v2.24 - fdisk(8)

This release is mostly about small incremental changes, a small exception is fdisk(8).

I have finished fdisk(8) refactoring, this task we started with Davidlohr Bueso and Petr Uzel two releases ago. The result is a code that is easy to extend and all disklabel specific code is no more maintained together as a ball of spaghetti.

The goal is to have one library (libfdisk) shared between all the fdisks (sfdisk, cfdisk, ...). Now it's used in fdisk only. I hope that one day we're going to have shared library with a public stable API.

 The most visible changes:
  • fdisk dialogs and output unification, more verbose messages
  • colors for warnings
  • new (but backwardly compatible) menus
  • all operations for all disklabels based on sectors by default
  • man page clean up
  • improved GPT disklabel support (rename partition, partition and disk UUID modification)

    The last missing thing is support for manual modification of hybrid GPT. This is planned for the next release v2.25.

    My laptop with GPT:
# fdisk -l /dev/sda

Disk /dev/sda: 149.1 GiB, 160041885696 bytes, 312581808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3549290F-417C-4941-8503-F7835109B821

Device           Start          End   Size Type
/dev/sda1         2048      2050047  1000M EFI System
/dev/sda2      2050048      6146047     2G Microsoft basic data
/dev/sda3      6146048     26462207   9.7G Linux swap
/dev/sda4     26462208     98142207  34.2G Microsoft basic data
/dev/sda5     98142208    230662143  63.2G Microsoft basic data
/dev/sda6    230662144    312580095  39.1G Microsoft basic data
  • disklabel hex dump (command 'D')

    This is nice feature for advanced users. You don't have to fight with hexdump, search the right offsets etc. fdisk(8) will locate all relevant disklabel parts on the device and print all in hex format. (Don't forget that many things don't use static offsets -- for example GPT array with partitions, MBR extended partitions, etc).

  • For example (disk with two logical partitoins):
# echo -e 'x\nD\nq\n' | fdisk  /dev/sdb
 
Welcome to fdisk (util-linux 2.24.rc2-22-1d8c-dirty).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
Command (m for help): 
Expert command (m for help): 
MBR: offset = 0, size = 512 bytes. 
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
*
000001b0  00 00 00 00 00 00 00 00  14 92 2b 9e 00 00 00 20
000001c0  21 00 83 df 13 0c 00 08  00 00 00 20 03 00 00 df
000001d0  14 0c 05 7c 30 4c 00 28  03 00 00 98 0f 00 00 00
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa
 
EBR: offset = 105906176, size = 512 bytes.
06500000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
*
065001c0  34 0d 83 bf 26 19 00 08  00 00 00 20 03 00 00 bf
065001d0  27 19 05 9f 39 26 00 28  03 00 00 28 03 00 00 00
065001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
065001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa
 
EBR: offset = 211812352, size = 512 bytes.
0ca00000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
*
0ca001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 e0
0ca001c0  08 19 83 9f 39 26 00 08  00 00 00 20 03 00 00 00
0ca001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
*
0ca001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa

Thursday, August 22, 2013

libmount Python binding

This week I merged Python libmount binding into util-linux. The original prototype was from Ondrej Oprala (Thanks!).

The binding provides the same functionality as libmount in C -- for now supported objects are:
  • Fs - filesystem description
  • Table - tab files parser and container for filesystems
  • Context - high-level API to mount/umount.   

Examples:
$ python                                                        
>>> import libmount                                             
>>> tb = libmount.Table('/proc/self/mountinfo')                 
>>> fs = tb.find_target('/home/kzak')                           
>>> print fs.source                                             
/dev/mapper/luks-10d813de-fa82-4f67-a86c-23d5d0e7c30e  
 
# python                                                        
>>> import libmount                                             
>>> cx = libmount.Context()                                     
>>> cx.target = '/mnt/backup'                                   
>>> cx.mount()                                                   
>>> fs = cx.mtab.find_target('/mnt/backup')                     
>>> print fs.fstype
nfs4
# python -c "import libmount; libmount.Context(target='/mnt/backup').umount()"

The side effect of the work on the Python binding is that libmount uses reference counting for libmnt_{fs,table,cache} objects now. So we're ready for more complicated scenarios.

.. and --without-python if you don't like it ;-)

Monday, April 29, 2013

dmesg -H is sexy

dmesg(1) from util-linux 2.23 provides some small usability improvements and all the improvements is possible to enable by -H,--human command line option.
  • enable pager (you know this for example from 'git log')
  • enable relative timestamps to display local time and delta in human readable format
  • enable colors for timestamp prefix, subsystem prefix and message body (different colors for different log levels)
dmesg -H util-linux 2.23

Hint: add

    alias dmesg="dmesg --human" 

somewhere to your /etc/profile.d/ directory.

Tuesday, April 23, 2013

umount(8), mount(8) and nsenter(1)

umount(8) from util-linux 2.23 (now -rc2) supports new command line options --recursive and --all-targets. The new command nsenter(1) opens doors to namespaces.

 # umount /mnt/A
 umount: /mnt/A: target is busy.

This is pretty common situation, the problem is obvious:

 # findmnt -R /mnt/A
 TARGET     SOURCE    FSTYPE OPTIONS
 /mnt/A     /dev/sdb1 ext2   rw,relatime,stripe=32
 └─/mnt/A/B /dev/sdb2 ext2   rw,relatime,stripe=32

so you have to unmount /mnt/A/B before /mnt/A. In some cases especially in scripts it could be a little bit tricky to umount all in the right order. The user friendly solution is --recursive:

 # umount --recursive /mnt/A

Note that this solution is not atomic and possible umount options (like --lazy) are applied to all umount(2) calls.

The another improvement is the option --all-targets. It umounts all mountpoints for the given filesystem (device) in the current namespace. This options is usable in situations when the same device is mounted on more places. The option --all-targets could be used together with the option --recursive.

 # findmnt -R /dev/sdb1
 TARGET     SOURCE    FSTYPE OPTIONS
 /mnt/A     /dev/sdb1 ext2   rw,relatime,stripe=32
 └─/mnt/A/B /dev/sdb2 ext2   rw,relatime,stripe=32
 /mnt/B     /dev/sdb1 ext2   rw,relatime,stripe=32
 /mnt/C     /dev/sdb1 ext2   rw,relatime,stripe=32

verbose mode provides more details about the umount order:
 
 # umount --recursive --all-targets --verbose /dev/sdb1
 umount: /mnt/C (/dev/sdb1) unmounted
 umount: /mnt/B (/dev/sdb1) unmounted
 umount: /mnt/A/B (/dev/sdb2) unmounted
 umount: /mnt/A (/dev/sdb1) unmounted

Note that /proc/self/mountinfo contains information about mountpoints hierarchy as well as chronological order.

All these umount(8) improvements have a small limitation -- umount(8) works with the current namespace only. It means if you want to be really paranoid than you should not expect that after --all-targets is the devices completely unmounted.

Fortunately we have a new command nsenter(1) to enter the namespaces of the other processes. Let's create a session with unshared mount namespace:
 
 # mount --bind --make-private /mnt/test /mnt/test
 
 # unshare --mount
 
 # mkdir /mnt/test/foo
 # mount /dev/sdb1 /mnt/test/foo
 
 # findmnt -R -o +PROPAGATION /mnt/test
 /mnt/test       /dev/sda4[/mnt/test] ext4   rw,relatime,data=ordered private
 └─/mnt/test/foo /dev/sdb1            ext2   rw,relatime,stripe=32    private
 
 # echo $$
 28008

Note that --make-private is necessary if the parent is mounted as shared (this is default for example on Fedora). Another session (shell):

 # findmnt -R /mnt/test
 TARGET    SOURCE               FSTYPE OPTIONS
 /mnt/test /dev/sda4[/mnt/test] ext4   rw,relatime,data=ordered

The /mnt/test/foo is invisible in this namespace, but we can enter the namespace by nsenter(8) command and we can make changes in the namespace:
 
 # nsenter --mount --target 28008
 
 # mkdir /mnt/test/bar
 # mount /dev/sdb2 /mnt/test/bar
 
 # findmnt -R /mnt/test
 TARGET          SOURCE               FSTYPE OPTIONS
 /mnt/test       /dev/sda4[/mnt/test] ext4   rw,relatime,data=ordered
 ├─/mnt/test/foo /dev/sdb1            ext2   rw,relatime,stripe=32
 └─/mnt/test/bar /dev/sdb2            ext2   rw,relatime,stripe=32
 
 # echo $$
 29886

It means we have two sessions (shells with PIDs 28008 and 29886) that share the same mount namespace.

The another example is to umount a directory in all namespaces:
 
 for p in $(pidof bash); do
  nsenter --mount --target $p -- umount --recursive /mnt/test
 done

The command nsenter(1) (as well as unshare(1)) supports mount, uts, IPC, net, PID and user namespaces. Note that you can use
 
  findmnt --task $pid

to list another mount namespaces. nsenter(1) is necessary only if you want to do changes to the namespaces.

2.23 highlights:
  • mount(8) allows to use propagation flags together with another mount operations (e.g. mount --make-private /dev/sda1 /mnt)
  • mount(8) allows to specify propagation flags in /etc/fstab by mount options (private,slave, ...)
  • mount(8) supports new option x-mount.mkdir to create mountpoint directories
  • findmnt(8) lists propagation flags (e.g. findmnt -o +PROPAGATION)
  • unshare(1) and nsenter(1) execute shell if no program specified
Note that Linux kernel still does not allow to use propagation flags together with another mount operations. All is implemented in userspace by additional mount(2) calls -- one call for one propagation flag, see strace output:

 # strace -e mount  mount --bind  --make-private /mnt/test /mnt/test
 mount("/mnt/test", "/mnt/test", 0x1886f10, MS_MGC_VAL|MS_BIND, NULL) = 0
 mount("none", "/mnt/test", NULL, MS_PRIVATE, NULL) = 0


Wednesday, February 20, 2013

local yum repository for koji builds

It takes days to see new packages on the official Fedora update-testing mirrors, and I'm so impatient... And sometimes it's necessary to test unofficial builds. The solution is pretty simple with "koji download-build" command.

Let's create a /usr/local/bin/local-repo script:
  #!/bin/bash
  REPODIR=/tmp/local-repo

  mkdir -p $REPODIR
  koji download-build $1
  createrepo -d .
And local repository config file /etc/yum.repos.d/local-updates-testing.repo
  [local-updates-testing]
  name=Local $releasever - $basearch - Test Updates
  failovermethod=priority
  baseurl=file:///tmp/local-repo/
  enabled=0
  gpgcheck=0
  metadata_expire=10
Now if you want to download the latest package:
  # local-repo pkgname-ver-rel
  # yum --enablerepo=local-updates-testing update
Note that if you want to use the repository for something large than /tmp is probably not the best place :-)