diff --git a/README.md b/README.md index 1621c5ad..2766d38c 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,8 @@ * [Quick start for developers](#quick-start-for-developers) * [Community](#community) * [Configuration](#configuration) +* [Logging](#logging) * [Debugging](#debugging) - * [Enabling debug for various components](#enabling-debug-for-various-components) * [Limitations](#limitations) * [Home Page](#home-page) @@ -79,26 +79,51 @@ To see details of your systems runtime environment (including the location of th $ cc-runtime cc-env ``` -## Debugging +## Logging -The runtime provides `--log=` and `--log-format=` options. However, it can -also be configured to log to the system log (syslog or `journald`) such that +The runtime provides `--log=` and `--log-format=` options. However, you can +also configure it to log to the system log (syslog or `journald`) such that all log data is sent to both the specified logfile and the system log. The latter is useful as it is independent of the lifecycle of each container. -To view runtime log output, +To view runtime log output: ```bash $ sudo journalctl -t cc-runtime ``` -### Enabling debug for various components +To view shim log output: + +```bash +$ sudo journalctl -t cc-shim +``` + +To view proxy log output: + +```bash +$ sudo journalctl -t cc-proxy +``` + +Note: + +The proxy log entries also include output from the agent (`cc-agent`) and the +hypervisor, which includes the guest kernel boot-time messages. + +## Debugging + +The runtime, the shim (`cc-shim`), the proxy (`cc-proxy`), +and the hypervisor all have separate `enable_debug=` debug +options in the [configuration file](#Configuration). All of these debug +options are disabled by default. See the comments in the installed +configuration file for further details. + +If you want to enable debug for all components, assuming a standard configuration file path, run: -The runtime, the shim (`cc-shim`), and the hypervisor all have separate debug -options in the [configuration file](#Configuration). +```bash +$ sudo sed -i -e 's/^#\(enable_debug\).*=.*$/\1 = true/g' /usr/share/defaults/clear-containers/configuration.toml +``` -The proxy (`cc-proxy`) has a command-line option to enable debug output. See -the [proxy documentation](https://github.com/clearcontainers/proxy#debugging) for further details. +See the [agent debug document](docs/debug-agent.md) and the [kernel debug document](docs/debug-kernel.md) for further details. ## Limitations diff --git a/docs/debug-agent.md b/docs/debug-agent.md index 1b041235..5a4b0d39 100644 --- a/docs/debug-agent.md +++ b/docs/debug-agent.md @@ -4,44 +4,49 @@ Clear Containers runtime launch a special virtual machine when a new pod is created. This VM uses a VM-agent to spawn processes on behalf of the pod/container(s) running inside the VM. -From Clear Containers 3.0-beta release a new agent is used, -[Clear Containers Agent](https://github.com/clearcontainers/agent): +The Clear Containers 3.0-beta release uses a new agent (`cc-agent`), +https://github.com/clearcontainers/agent. -The Clear Containers Agent relies heavily on libcontainer used by runc -(standard program when running containers on bare metal). +The Clear Containers agent relies heavily on [`libcontainer`](https://github.com/opencontainers/runc/tree/master/libcontainer) +used by [`runc`](https://github.com/opencontainers/runc/), which is the +standard program when running containers on bare metal. To provide a debug log of any agent activity on a guest, the Clear Containers -agent sends logs through a QEMU serial console that are collected by [cc-proxy](https://github.com/clearcontainers/proxy) -and shown in its logs. By default, the Clear Containers agent logs are not collected by -`cc-proxy` but can be enabled by adding the `-log debug` option to `cc-proxy`. - -1- Add the `-log debug` option to `cc-proxy` - -``` -mkdir -p /etc/systemd/system/cc-proxy.service.d/ -cat << EOT > /etc/systemd/system/cc-proxy.service.d/proxy-debug.conf -[Service] -ExecStart= -ExecStart=/usr/libexec/clear-containers/cc-proxy -log debug -EOT -# Restart cc-proxy to provide debug logs. -systemctl daemon-reload -systemctl restart cc-proxy -``` -2- Run a container to generate the agent debug logs - -``` -sudo docker run -ti busybox true -``` -3- Filter the agent debug logs from the `cc-proxy` logs -The `cc-proxy` logs show the sources of its collated information. To only see -the agent debug logs, filter `cc-proxy` logs by the QEMU serial console (the -agent logs were sent through it). The QEMU serial console is represented by -`source=qemu`. - -``` -journalctl -u cc-proxy | grep source=qemu -``` - -The debug log format is: -DEBU[0019] *AGENT MESSAGE* source=qemu vm=*CONTAINER_ID* +agent sends log data through a QEMU serial console, which is collected by +[cc-proxy](https://github.com/clearcontainers/proxy) and shown in the proxy's +logs. + +By default, the Clear Containers agent logs are not collected by `cc-proxy` +but can be enabled by enabling the proxy debug option: + +1. Enable proxy debug + + Set the `enable_debug=` option in the `[proxy.cc]` section to `true`, which + assumes a standard configuration file path: + + ``` + $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml + ``` + +1. Run a container to generate the logs: + + ``` + $ sudo docker run -ti busybox true + ``` + +1. Filter the agent debug logs from the `cc-proxy` logs + + The `cc-proxy` logs show the sources of its collated information. To only + see the agent debug logs, filter `cc-proxy` logs by the QEMU serial console + that sent the agent logs. The QEMU serial console is represented by + `source=qemu`. + + ``` + $ sudo journalctl -t cc-proxy | grep source=qemu | egrep '\' + ``` + + To extract all logs entries for a particular container: + + ``` + $ sudo sudo journalctl -t cc-proxy | grep source=qemu | grep vm=CONTAINER_ID | egrep '\' + ``` diff --git a/docs/debug-kernel.md b/docs/debug-kernel.md index ab26b4f6..b5dcc54a 100644 --- a/docs/debug-kernel.md +++ b/docs/debug-kernel.md @@ -10,37 +10,34 @@ fails due to the kernel. To provide a debug log of the kernel boot, Clear Containers supports an option to enable kernel boot logs using the configuration file. -1. Enable debug in `configuration.toml` +1. Enable kernel boot messages + + Set the `enable_debug=` option in the `[proxy.cc]` section to `true`, which + assumes a standard configuration file path: ``` - $ enable_debug = true + $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml ``` -2. Run a container to generate the logs +1. Run a container to generate the logs ``` $ sudo docker run -ti busybox true ``` -3. Filter the kernel boot logs from the `cc-proxy` logs +1. Filter the kernel boot logs from the `cc-proxy` logs The `cc-proxy` logs show the sources of its collated information. To see - the kernel boot logs, filter `cc-proxy` logs by the QEMU serial console. + the kernel boot logs, filter `cc-proxy` logs by the QEMU serial console, + excluding the agent messages. The QEMU serial console is represented by `source=qemu`. ``` - $ journalctl -u cc-proxy | grep source=qemu + $ sudo journalctl -t cc-proxy | grep source=qemu | egrep -v '\' ``` -4. Obtain logs using standalone `cc-proxy` - - In some cases it may be easier to run the `cc-proxy` standalone by first stopping - the `cc-proxy` service and running it standalone to capture the logs. + To extract all logs entries for a particular container: ``` - $ sudo systemctl stop cc-proxy - $ sudo /usr/libexec/clear-containers/cc-proxy -log debug + $ sudo sudo journalctl -t cc-proxy | grep source=qemu | grep vm=CONTAINER_ID | egrep -v '\' ``` - - This will result in the `cc-proxy` log being printed on the terminal. This method - can also be used to capture the logs in a specific `cc-proxy` file.