dtee (double tee)¶
Description¶
Run a program with standard output and standard error copied to files while maintaining the original standard output and standard error as normal.
Purpose¶
To run programs from cron, suppressing all output unless the process outputs an error message or has a non-zero exit status whereupon the original output will be written as normal and the exit code will be appended to standard error.
It will do this by default when invoked as cronty, providing an alternative
to cronic but without splitting up the output.
(Also, to do tee(1) with standard output and standard error at the same
time.)
Contents¶
Architecture¶
Background¶
Commands on Unix systems output messages to two logical streams: standard output (for normal messages) and standard error (for error messages). In the normal execution of the command there may be messages of both types being output and they are usually related so it’s important that they appear in the correct order. This is normally achieved by having the file descriptors of both streams be the same underlying destination (a terminal, pipe to another process, a file). There is no distinction between streams by the operating system when the message is written.
Problem¶
Commands are not always run separately by a user on a terminal. They may be run
from a script or unattended from cron(8).
When this happens it can be useful to know if the process wrote any error messages, so typically the file descriptors for standard output and standard error would be different destinations (two separate log files). The script can then easily distinguish between normal messages and error messages.
This works well to determine the outcome of the command and examine any messages it outputs. A problem arises when it is necessary to provide the original output of the command to a user. It is not possible to guarantee the reading of messages from two file descriptors in the correct order and there is no assistance provided by the operating system for doing this.
The output from commands can be confusing if the messages are no longer in the original order. Splitting the output up into two blocks (normal and error) is not helpful.
Workarounds¶
One option has been to use LD_PRELOAD to modify the behaviour of the process
and identify the destination stream as the messages are being written. This is
error-prone because there are lots of different library functions for outputting
to standard streams as well as functions within the C library that may bypass
their external API and output messages directly.
Processes may also write directly to the file descriptors using system calls
or their executables may be statically linked, preventing preloading from
working. They may have multiple threads to complicate the manipulation of
messages. For security reasons it’s not possible to preload libraries into
setuid executables using LD_PRELOAD so this option doesn’t work for those
commands.
Solution¶
Splitting of standard output and standard error while retaining the order of
output can be performed using three unix(7) datagram sockets. A single input
socket is used (so that messages can be read in order) and two output sockets
are connected to the same input socket (so that they share the same reliable
ordered buffer).
The source address of each message is provided by the operating system on every read so it is possible to identify which output stream was used by binding to different paths for each stream.
It would be preferable to use sequenced-packet sockets instead but that would
require two pairs of sockets because it is not possible to have one socket
connected to two peers. Perhaps there will be a sockettriple function in the
future that can do this.
Dependencies¶
The following tools and libraries are required as part of the build process, to run dtee or to produce documentation.
Limitations¶
Datagram sockets can only process writes as individual packets with a maximum
packet size. Therefore, if the program being run attempts to write(2)
more than this size in one call the write will fail and that part of the output
will be lost.
This is not usually a problem because the default socket buffer size is usually
much higher than the size programs typically write with. For safety, the socket
buffer size will be increased to at least PIPE_BUF and BUFSIZ if the
default is smaller than these values.
Writes to the socket (on Linux or GNU Hurd) will block until there is capacity
available in the socket buffer. If the process uses sendfile(2) then (on
Linux) the writes occur in PIPE_BUF sized chunks so it works as normal, but
why are you using an interactive program that outputs such large quantities of
data?
For more details read the architecture document.
FreeBSD/OpenBSD¶
Writes to the socket do not block when the receive buffer of the peer socket is full. The default socket receive buffer is quite small so it will be raised to 512KB (for the send buffer, 256KB). This avoids problems most of the time.
Messages are likely to be lost from programs that write large amounts of data (more than 128..256KB) very quickly or do so inefficiently (1 byte at a time).
NetBSD¶
Like FreeBSD/OpenBSD but the socket buffer can only be raised to 128KB so messages are very likely to be lost if data is written quickly. Unlike the other BSDs, this will result in an error on the receive call so it will not go unreported.
DragonFlyBSD¶
Like FreeBSD/OpenBSD but even with a 512KB socket buffer it loses messages of
PIPE_BUF size that are written very quickly. Writes of BUFSIZ size are ok
because they result in fewer messages.
GNU Hurd¶
Does not currently have support for returning addresses of Unix sockets, so none of the output works. It may be possible to implement custom pipe-like objects with three file descriptors in user space.
Writes larger than the page size (4KB) are truncated and there’s no way to increase the size of the socket buffer.
Build and install¶
The meson build system is used to build and install dtee:
meson build/release # configure dtee build
ninja -C build/release # compile
ninja -C build/release test # run the tests
ninja -C build/release install # install to default locations
A Makefile that calls meson and ninja is provided for convenience.
See the list of dependencies for more information on build, test and runtime requirements.
Manual page¶
Description¶
Run command with standard output and standard error copied to files while maintaining the original standard output and standard error as normal.
Options¶
Output files¶
Standard streams can be written to any number of specified files, in addition to normal output. Output is not line buffered.
| -o <filename>, --out-append=<filename> | |
| Append standard output to filename, creating the file if it does not exist. | |
| -O <filename>, --out-overwrite=<filename> | |
| Copy standard output to filename, truncating and overwriting existing content. | |
| -e <filename>, --err-append=<filename> | |
| Append standard error to filename, creating the file if it does not exist. | |
| -E <filename>, --err-overwrite=<filename> | |
| Copy standard error to filename, truncating and overwriting existing content. | |
| -c <filename>, --combined-append=<filename> | |
| Append standard output and standard error to filename, creating the file if it does not exist. | |
| -C <filename>, --combined-overwrite=<filename> | |
| Copy standard output and standard error to filename, truncating and overwriting existing content. | |
General options¶
| -i, --ignore-interrupts | |
| Ignore keyboard interrupt signals (SIGINT). This does not prevent the command being executed (and other processes in the same progress group) from receiving the signal. | |
| -q, --cron | Operate in cron mode (this is implied when invoked as cronty). Suppresses all output unless the process outputs an error message or has a non-zero exit status whereupon the original output will be written as normal and the exit code will be appended to standard error. |
Miscellaneous¶
| -h, --help | Display usage information and exit. |
| -V, --version | Output version information and exit. |
See also¶
Resources¶
Change log¶
0.0.1 – 2018-11-11¶
Update to allow improvements in packaging.
Fixed¶
- Infinite loop in the test scripts if check variables are undefined (this is unlikely).
- Support for unity builds when
-Wshadowis used.
Packages¶
Source packages for Linux distributions are kept in the dtee-package repository.
Debian binary packages¶
Packages are available on Bintray for Debian and Ubuntu.
Install the Bintray public key:
wget https://bintray.com/user/downloadSubjectPublicKey?username=bintray -O - | apt-key add -
Follow the instructions for your specific distribution. If you are using a newer distribution than the ones listed then use the builds for the most recent prior version.
Debian 8 (jessie)¶
Add the following APT data source to /etc/apt/sources.list.d/dtee.list:
deb https://dl.bintray.com/dtee/debian/ jessie main
- Run the following commands:
apt install apt-transport-httpsapt updateapt install dtee
Debian 9 (stretch)¶
Add the following APT data source to /etc/apt/sources.list.d/dtee.list:
deb https://dl.bintray.com/dtee/debian/ stretch main
- Run the following commands:
apt install apt-transport-httpsapt updateapt install dtee
Pronunciation¶
- dtee
- /diː.ˈtiː/
- cronty
- /ˈkrɒn.tiː/