wiki:Internal/OpenFlow/miscUnix

Version 28 (modified by akoshibe, 12 years ago) ( diff )

—

*Nix related tidbits.

This is a page on various tools and techniques related to administration, programming, disaster recovery - basically, odds and ends that come in handy once every so often on a UNIX-like system. Anything based on a link is reiterated just for completeness, and just in case a link dies.

This page is a compilation of things done with Ubuntu and FreeBSD, the OSes that the author of this page use/break on a regular basis1. Although syntax will inevitably be different amongst various systems, the general practices and ideas should apply to any UNIX-like OS. Additions/corrections to the page are very much welcome.

Shell-based Command-line tools. So far, this section includes:

  • extundelete - Linux file system recovery
  • cdrecord - burning bootable ISO's
  • process I/O redirection (gdb)
  • fixing garbled text (gcc)

Development-related Tools and workarounds

  • git server setup

Network Various networking-related things.

  • Disabling SSH timeout
  • NAT boxes
    • with ufw (Linux)
    • with pf (*BSD)

Quick Printing Printing under *nix, relatively quickly

  • CUPS
  • lpd

one-liners miscellaneous single-sentence tips.

Shell-based tools and techniques

File recovery with Extundelete

Extundelete lets you recover files (directories) that you've accidentally deleted on an ext file system. The best way to use the tool is to have a live CD with the tool on it, although it does not seem to come packaged with Linux by default.

  1. Boot the machine off of live CD.
  2. Install extundelete. On Ubuntu, you need to edit sources.list, since it is not part of the standard repositories.
  3. Restore the files on the partition: 
    # sudo extundelete --restore-all /dev/sda1

This restores everything that has been deleted in /dev/sda1, given the tool finds them. All recovered files are placed in a recovery directory that you can package and back up before rebooting the machine.

Command-line CD burner

source : ​http://www.yolinux.com/TUTORIALS/LinuxTutorialCDBurn.html

Many CD burners are fairly dependency-heavy, mostly thanks to their GUI. cdrecord is a command-line based CD/DVD burner with just two dependencies at ~2.3kB: 

$ sudo apt-get install cdrecord
...
The following NEW packages will be installed:
  genisoimage wodim
...
After this operation, 2,306 kB of additional disk space will be used.

Now, to use:

  1. Check for supported devices. You'll need to specify what to use when burning the image. cdrecord seems to just be a wrapper plus then some for woodim, and the two can be used interchangeably for this step. 
    $ wodim --devices
wodim: Overview of accessible drives (2 found) :
-------------------------------------------------------------------------
 0  dev='/dev/scd0'     rwrw-- : 'SONY' 'DVD-ROM DDU1615'
 1  dev='/dev/scd1'     rwrw-- : 'TSSTcorp' 'CDDVDW SE-208AB'
-------------------------------------------------------------------------
  2. Burn. The meaning of the options can be listed with 'cdrecord —help'. 
    $ sudo cdrecord -v -eject speed=16 dev='/dev/scd1' Downloads/ubuntu-12.04-server-amd64.iso

This will spew a bunch of output, but complete with something like below: 

Starting new track at sector: 0
Track 01:  684 of  684 MB written (fifo 100%) [buf  99%]  16.8x.
Track 01: Total bytes read/written: 717533184/717533184 (350358 sectors).
Writing  time:  327.327s
Average write speed  14.6x.
Min drive buffer fill was 99%
Fixating...
Fixating time:   24.715s
BURN-Free was never needed.
wodim: fifo had 11302 puts and 11302 gets.
wodim: fifo was 0 times empty and 11100 times full, min fill was 97%.

And eject the CD.

Redirecting the output of a live process.

source: ​http://etbe.coker.com.au/2008/02/27/redirecting-output-from-a-running-process/

Say that you have a running process whose outputs are either being piped to /dev/null or a terminal that you can't get a hold of, but you'd like to be able to see its output. Conversely, you may want to redirect something's output to /dev/null. You can force the process to change its output using gdb, the GNU debugger. This is a powerful tool that allows you to attach to a live process and inspect and manipulate the contents of its address space.

Here I have a script that I had backgrounded but is still outputting to my terminal (annoying). I will redirect the program's output to a file.

  1. Find the PID of your process. Mine is called arrival_collector.rb, so I look for it using ps. 
    # ps -ef | grep collector
root     17793 11276  0 18:39 pts/1    00:00:00 ruby arrival_collector.rb -r a,h,ee,f,wknd1,wknd2 -s scott,busch_a,pubsafs,foodsci,scott,scott -v

/proc/ shows that its STDOUT is my terminal, /dev/pts/1: 

# ls -l /proc/17793/fd
total 0
lrwx------ 1 root root 64 2012-02-19 18:45 0 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:45 1 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:40 2 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:45 3 -> socket:[1566783742]
  1. Attach to the process with GDB. 
    # gdb -p [PID] [path/to/executable]

It wil spew a bunch of output but eventually give you a prompt: 

# gdb -p 17793 /opt/grailrtls/grail3_ruby/grail3protocols/arrival_collector.rb
GNU gdb (GDB) 7.2-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
...
Loaded symbols for /lib/libnss_files.so.2
0xb78c7424 in __kernel_vsyscall ()
(gdb)
  1. Close STDOUT: 
    (gdb) p close(1)
$1 = 0
  1. Point STDOUT to a file. Like in the link, I will use creat() to point STDOUT to a log file in /tmp/. creat() takes the path to the file and its permissions as the two arguments. 
    (gdb) p creat("/tmp/output.log", 0600)
$2 = 1
  1. Exit gdb. Choose yes when asked if you want to quit. 
    (gdb) quit
A debugging session is active.

        Inferior 1 [process 17793] will be detached.

Quit anyway? (y or n) y
Detaching from program: /usr/bin/ruby1.8, process 17793

Now when you check /proc/ you should see that your STDOUT is directed att the file you created: 

# ls -l /proc/17793/fd
total 0
lrwx------ 1 root root 64 2012-02-19 18:45 0 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:45 1 -> /tmp/output.log
lrwx------ 1 root root 64 2012-02-19 18:40 2 -> /dev/pts/1
lrwx------ 1 root root 64 2012-02-19 18:45 3 -> socket:[1566783742]

you can confirm this with tail -f (or the fact that your program has stopped outputting to terminal).

Fixing garbled gcc and man page output.

On some machines, gcc and man pages might produce garbled text. This is usually caused by xterm not supporting UTF-8, or from mismatch in locale information if the garbling is happening when you are working on a remote machine. In either case (for people working with US English), setting LANG to en_US or C fixes things: 

$ export LANG=en_US

Set it the same for both local and remote machines if it's happening over SSH.

Development-related

Various odds-and-ends related to tools and workarounds in coding sessions.

git server setup

A central git repository which things can be pushed to/pulled from, but no changes are directly made to, is a bare repository. These are good as central coordination/synchronization points for larger projects and/or if you work on a project from many locations.

Here we assume that you have a git repo that you wish to make a bare repo from, and a host for your bare repo. The host should be running git, and you should have SSH access to it.

  1. Copy the .git directory from your working git repo, and rename it. If your repo is named "project", the practice is to rename the .git taken from it to "project.git": 
     $ cp -Rf project/.git project.git
  2. change "bare" to "true" in file config in project.git: 
     [core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        logallrefupdates = true
 ...
  3. copy the new bare repo to your server. 
     $ scp -r project.git [bare repo host]:[path to destination]
    You should now be able to pull from/push to the host with the bare repository as usual. For example, with git clone: 
     $ git clone ssh+git://[bare repo host]/path/to/project.git
    You will be asked to authenticate with your SSH password.

If you already have a local working directory, and you wish to start using your bare repo, just edit .git/config in your local git repo to point it to your server: 

 ...

 [remote "origin"]
        fetch = +refs/heads/*:refs/remotes/origin/*
        url = ssh+git://[bare repo host]/path/to/project.git
 ...

Your changes to config will be applied starting with the next the git operation you do.

Alternatively, the --bare flag for git init lets you if you start a bare repository from scratch: 

 $ git init --bare my-repo.git

Networking-related odds and ends

Various non-experimental network setups, usually done for convenience.

Disabling SSH timeout

source: ​http://docs.oseems.com/application/ssh/disable-timeout

You can essentially prevent SSH from timing out after a long idle period by adding the following to /etc/ssh/ssh_config : 

ServerAliveInterval 100

The above will make the client send a keepalive signal once every 100 seconds. Alternatively, you can modify server-side configs by adding the following to /etc/ssh/sshd_config : 

ClientAliveInterval 30
TCPKeepAlive yes 
ClientAliveCountMax 99999

And restarting sshd: 

/etc/init.d/ssh restart

NAT boxes.

NAT boxes are handy if you want to build a gateway. First and foremost, you need to enable IP packet forwarding on your to-be-NAT box.
On Linux, the command 

sudo sysctl -e net.ipv4.ip_forward=1

at the shell enables it. For persistence, add the following line to /etc/default/ufw: 

net.ipv4.ip_forward=1


Similarly for FreeBSD (assuming you have sudo installed), 

sudo sysctl net.inet.ip.forwarding=1

enables it, and the following line in /etc/sysctl.conf makes it persistent: 

net.inet.ip.forwarding=1

with ufw

source: ​https://nowhere.dk/articles/tip_nat_with_ubuntus_ufw_firewall

ufw is your standard Linux firewall, and comes with Ubuntu server edition. Turning a multi-interface Linux box into a router is a matter of the following steps:

  1. configure IP forwarding

edit /etc/default/ufw : 

DEFAULT_FORWARD_POLICY="ACCEPT"
  1. set up IP masquerading in ufw

edit /etc/ufw/before.rules, just after the header : 

# nat Table rules
 *nat
 :POSTROUTING ACCEPT [0:0]

 # Forward traffic through ppp0 - Change to match you out-interface
 -A POSTROUTING -s 192.168.1.0/24 -o ppp0 -j MASQUERADE

 # don't delete the 'COMMIT' line or these nat table rules won't
 # be processed
 COMMIT

The address block after -s should match the address block behind the NAT firewall.

  1. restart ufw: 
    sudo ufw disable && sudo ufw enable

with pf

pf is the OpenBSD packet filter, a piece of software intended for heavy-duty packet filtering/firewalls and comes with some Berkeley UNIX derivatives. Assuming you have IP forwarding enabled, the following configuration in /etc/pf.conf should give you a NAT firewall: 

ext_if="bge0"
int_if="em0"
external_addr="192.168.203.155"
internal_net="192.168.1.0/24"
nat on $ext_if from $internal_net to any -> ($ext_if)
pass in all 
pass out all

ext_if is the interface facing the external network, and int_if is the interface connected to your NATed net. Once saved, start pf: 

sudo pfctl -e -f /etc/pf.conf

If it throws errors, make sure that the kernel module (pf.ko or something similar) is loaded.

Quick Printing Setup.

This section contains info that should let you get up and printing from a network printer relatively quickly.

with CUPS

CUPS stands for Common UNIX Printing System, and is fairly standard as a means to print from UNIX and UNIX-like things. We'll describe the steps needed for quick CUPS printing setup (bare-bones) in Ubuntu 11.04

  1. install packages. 
    cups cupsys-driver-gutenprint libcupsys2 libcupsimage2
  2. The default port that CUPS serves its admin GUI on is tcp:631: 
    $ netstat -na -4 | grep 631
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN
    If you see it, point your browser to localhost:631. You should get a shiny-white welcome page. Yes, it was made by Apple.
  1. Add your printer from the Administration tab → Add Printer. It will ask for the root user and password (OK if you are on sudoers). You will be asked to provide info on the printer, and choose the drivers.


If all goes well, you will be able to see the printer's status with lpstat. 

$ lpstat -p -d
printer Phaser_5500DT is idle.  enabled since Sat 28 Apr 2012 01:14:59 AM EDT
no system default destination

If you want to print, you can do so by invoking lp: 

$ lp -o sides=two-sided-long-edge -d Phaser_5500DT [filename]

with lpd

lpd comes packaged with FreeBSD, and is therefore extensively covered in its handbook. The above link lets you get going relatively quickly; here parts of the link are repeated.

  1. (1) set up the spool directory (where printed files go temporarily), with (2) modified perms to keep jobs private: 
    mkdir /var/spool/lpd/lp                      (1)
chown daemon:daemon /var/spool/lpd/lp         
chmod 770 /var/spool/lpd/lp                  (2)
  2. enable lpd. Add the following to /etc/rc.conf: 
    lpd_enable="YES"
    to make lpd start at boot-up. "lpd" at the shell as root will also start it.
  1. edit /etc/printcap : 
    lp:\
        :lp=9100@192.168.200.32:\
        :sh:\
        :mx#0:\
        :sd=/var/spool/lpd/lp:\
        :lf=/var/log/lpd-errs:
  • lp : the printer (port@machine IP), specifically Phaser_5500DT (the WINLAB printer in the large cubicle room)
  • sh : suppress page header
  • mx : maximum file size set to 0 (unlimited)
  • sd : spool directory
  • lf : file for error logging


You can test if your setup works by piping a text string to lpr: 

printf "printer test page\r\n\f" | lpr

You also may have to do some extra work to print certain file formats. The WINLAB printer doesn't support direct printing of PDFs, so a quick work-around is to (1) convert, and (2) print it as a Postscript file: 

pdf2ps /tmp/myfile.pdf /tmp/myfile.ps       (1)
lpr /tmp/myfile.ps                          (2)

one-liners.

  • in bash, if seq doesn't exist, try gseq.
  • options for mounting a *BSD filesystem in Linux: -t ufs -o ufstype=ufs2 (not '44bsd' like the man pages say)
  • a dependency fix not requiring '-f install' with apt-get: 'build-dep [package]', then install as usual.
  • [sudo] ip addr flush [iface] resets the IP configurations for an interface on Ubuntu.
  • [sudo] ip route flush resets the route table on Ubuntu; [sudo] route flush does the same on FreeBSD.
  • tar -xvf [archive.tgz] [filename] lets you extract a single file from an archive and -tf lists contents.
  • the —exclude flag for tar can be used to ignore certain files when packing whole directories into tarballs.
  • find -x [dir] -size +10000 -exec du -h {} \; finds files over 10kB, including hidden files.
  • common native serial devices: /dev/ttyS0 for Linux, /dev/cuau0 for FreeBSD.
  • ctrl-l clears a terminal and ctrl-u clears the current line, including password prompts.


1."Use" here is more close to "break" in this situation.

Note: See TracWiki for help on using the wiki.