Recently I was debugging some problem with deployment in Syncano (where I work) and I wanted to catch an exact moment when the problem occurred. Docker was dying on a host without any good reason.

Let's assume that I wanted to observe what Docker containers are running at a given moment.

Here is my snippet, we will later describe it in detail:

$ watch -n 3 'date >> dump_ps && docker ps | tee -a dump_ps'

It uses:

  • watch
  • date
  • docker ps (but can be replaced by any command that you're interested in)
  • tee
  • tmux (later in the blog post)

What it does?

It executes a command in single quotes 'date >> dump_ps && docker ps | tee -a dump_ps' every three seconds, displays it on the screen and appends to file.

The command consists of two parts date >> dump_ps and docker ps | tee -a dump_ps, both of them append to dump_ps file.

date >> ps is pretty self-explanatory, because it just appends a current date to a file. You can pass arguments to the date program to adjust the format - date man page.

If we typed:

$ watch -n 3 'date >> time_is_passing'

and interrupted after few seconds, we could see the time written to time_is_passing file:

$ cat time_is_passing 
nie, 22 mar 2015, 03:57:08 PDT
nie, 22 mar 2015, 03:57:11 PDT
nie, 22 mar 2015, 03:57:14 PDT
nie, 22 mar 2015, 03:57:17 PDT

Output may differ if you have different locale / timezone settings.

The second part of the command: docker ps | tee -a dump_ps is a little more complicated.

If you have docker installed, docker ps will show you info about the containers running on your host. More info about Docker command line commands may be found here.

If you don't want to play with docker, you can replace this command with top - it will show you your processes instead.

Ok, so the first command would normally display an output on the screen or pass it through the pipe to another program. What we want to do here is to both show and append the text to the file. There is a program just for that in every Linux distro. It's called tee.

tee from wikipedia

Image from wikipedia article about tee.

$ echo "I love command line" | tee interests
I love command line

$ cat interests 
I love command line

However if we wrote another interest to the file in a similar way:

$ echo "I love climbing" | tee interests
I love climbing

$ cat interests                         
I love climbing

We would replace the whole file! It's because tee without arguments is like using >. To append to a file pass the -a flag to tee.

$ echo "I love command line" | tee -a interests
I love command line

$ echo "I love Linux" | tee -a interests
I love Linux

$ cat interests                         
I love climbing
I love command line
I love Linux

Cool! We can now build an ad hoc dashboard that refreshes every three seconds and dumps all the info to the file.

But what if we wanted to monitor more than one thing at once on the same screen? Let's enter the world of tmux.

Tmux is a terminal multiplexer - if you haven't heard of it, grab a tutorial.

It can be installed with apt-get or yum:

sudo apt-get install tmux

or

sudo yum install tmux

Ok, so if you have tmux installed, type:

$ tmux
$ Ctrl-B+"

Your screen should split vertically. Then run the command in the first pane and switch to the other one with Ctrl-B+Up Arrow and run the next command.

Here is a result from my laptop:

Imgur

I hope that you've learned something new. Shell is full of small gems that really shine when used together.

Happy hacking!