Occasionally useful Linux tricks

List the contents of all files in a folder

# Lists contents of all files in current directory, with filename and line-number prepended to each line
grep -n . *

# Recursively list contents of all files in current directory and subdirectories, with filename and line-number prepended to each line
grep -Rn .

You’ve been added to more groups, but don’t want to log off and back on again to use the new privileges:

sudo sudo -u mark bash

The first sudo gives us root access which is necessary for the second sudo, which logs us back in as ourself and starts a bash shell. This shell has the privileges of the new groups you were added to.

Transferring data over a slow network:

# Both of these are too slow due to our crappy internet connection
ssh user@server 'producer' | consumer
producer | ssh user@server 'consumer'

# If our CPU is sitting idle while waiting for the data to transfer, let's give it some work to do!
ssh user@server 'producer | pbzip2 -c9' | pbzip2 -d | consumer
producer | pbzip2 -c9 | ssh user@server 'pbzip2 -d | consumer'

These use the multithreaded implementation of bzip2 to achieve fairly fast and powerful compression to squeeze information along the wire faster.

If pbzip2 leaves you CPU-bottlenecked, you can reduce the compression level (e.g. -c5 instead of -c9) or use gzip which is faster but won’t compress as well. To use parallel gzip, you’ll want to replace pbzip2 with pigz.

If you still have plenty of CPU and also RAM to spare when using pbzip2 and the transfer is taking too long, try parallel lzma instead with pxz in place of pbzip2.

Monitoring the progress of transfers / measuring speed of devices

# Test sequential speed of disk sda by reading first 4GB of the disk (sudo needed for raw disk access)
sudo pv -bartpSs 4G /dev/sda > /dev/null

# File archiving over SSH (with pbzip2 as shown previously)
size="$(ssh user@server 'du -csB1 /path/to/files')"
ssh -T user@server "tar -c /path/to/files | pv -cbarteps ${size} --force | pbzip2 -c9" > /path/to/archives/name.tar.bz2

Running the above operations without making the machines grind to a halt

# CPU-heavy workloads can be told to play nicely by prepending them with "nice"
... | nice pbzip2 -c9 | ...

# IO-heavy workloads can be told to play nicely by giving them idle priority with "ionice"
ionice -c3 tar -c /path/to/files | ... | ionice -c3 > /path/to/archives/name.tar.bz2

# Example from (3) with progress:
size="$(ssh user@server 'du -csB1 /path/to/files')"
ssh -T user@server "ionice -c3 tar -c /path/to/files | pv -cbarteps ${size} --force | nice pbzip2 -c9" | ionice -c3 cat > /path/to/archives/name.tar.bz2

Firing up Eclipse CDT in a temporary Ubuntu environment with Docker

# Download an Ubuntu image
docker pull ubuntu

# Install eclipse-cdt
docker run -i ubuntu apt-get -y install eclipse-cdt

# Get ID of that container (which is now dead)
docker ps -a

# Snapshot that container
docker commit [ID] eclipse-cdt

# Run eclipse with workspace directory mapped to the host (select /root/workspace when Eclipse asks for workspace path)
docker run -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY -v ~/eclipse-workspace:/root/workspace eclipse-cdt eclipse

Easy sandboxing for improved privacy and security

I have created a script to allow easy sandboxing of applications, by running them in a separate temporary single-use user account. By controlling which groups the temporary user is added to, one can control hardware access (e.g. audio, webcam, networking, etc). With permissions set correctly, one can protect their home folder from the prying eyes of dodgy closed-source software.

This script is not intended to be a *perfect* security solution – it is far from perfect, but will suffice in most situations where a little extra isolation is required.

There are two scripts:

The first script (sandbox) is used to launch a program in a sandbox. The second script (sandchrome) demonstrates this by launching an incognito chrome session in a sandbox, with access to audio explicitly granted.

Laser cutting and constructive solid geometry (CSG)

My brother is an artist – he has business cards that reflect this quite nicely, using his artwork as the background behind his typography. I decided that I could also do with some “domain specific” business cards. (Micro-)miniature electronics such as tiny LEDs are a common gimmick nowadays, so I took a more mechanical route with my designs. A key criterion was that the design should incorporate sufficient complexity and domain knowledge that the average Google-graduate and script-kiddy couldn’t replicate it easily.

While 3D printing is nice, it isn’t great for producing precise planar objects, and can also be quite expensive and slow. Forms of CNC cutting (e.g. plotters, laser cutters) are more suitable, although getting a decent quote is proving to be a challenge.

There were several mechanisms that interested me and which would look “elegant” if embedded in a business card. These included:

The first would be difficult to get into an ultra-thin form-factor, and would have issues with friction. The third would be very complex, requiring a thin hairspring and repeatable manufacturing precision. So I’m left with some form of geartrain. The geartrain must work in any orientation, so some of the cool designs in that video which require gravity are not suitable. Epicyclic geartrains are generally simple to manufacture and look quite cool, so I settled for a simple planetary configuration.

I was relieved to find that I can use OpenSCAD and FreeCAD to produce designs for laser cutting, so I can use my existing software and acquired skills from designing 3D-printable objects for this laser-cut project.

The design will be cut from thin sheets of aluminium or steel, and perspex or polycarbonate. My name and contact details will be engraved into the underside of the transparent plastic, so the external face remains smooth.

A rendering is shown in the video below:

(Rendered with OpenSCAD, Imagemagick, FFMPEG, Bash)

OpenSCAD was never intended to produce high-quality renderings (Blender and POV-RAY already meet that goal very nicely), but it has done a respectable decent job nonetheless.

I designed another card, on the theme of “cloud computing”: