From f4e26f9918924cb603955c58f2a3881229960681 Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Wed, 6 Dec 2017 09:11:30 +0000 Subject: [PATCH 1/4] doc: Use colon for consistency All commands should be introduced with a sentence that ends with a colon. Signed-off-by: James O. D. Hunt --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1621c5ad..87ff3f5c 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ also be configured to log to the system log (syslog or `journald`) such that all log data is sent to both the specified logfile and the system log. The latter is useful as it is independent of the lifecycle of each container. -To view runtime log output, +To view runtime log output: ```bash $ sudo journalctl -t cc-runtime From ef02de7932bc41dfb90d5a9a732b104107d2cf2d Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Wed, 6 Dec 2017 09:13:17 +0000 Subject: [PATCH 2/4] doc: Improve debug and logging details Renamed "Debugging" section to "Logging" and explained how to view log output for all system components. Also created a top-level "Debugging" section which explains how to enable full debug. Fixes #845. Signed-off-by: James O. D. Hunt --- README.md | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 87ff3f5c..ef2f755c 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,8 @@ * [Quick start for developers](#quick-start-for-developers) * [Community](#community) * [Configuration](#configuration) +* [Logging](#logging) * [Debugging](#debugging) - * [Enabling debug for various components](#enabling-debug-for-various-components) * [Limitations](#limitations) * [Home Page](#home-page) @@ -79,7 +79,7 @@ To see details of your systems runtime environment (including the location of th $ cc-runtime cc-env ``` -## Debugging +## Logging The runtime provides `--log=` and `--log-format=` options. However, it can also be configured to log to the system log (syslog or `journald`) such that @@ -92,13 +92,35 @@ To view runtime log output: $ sudo journalctl -t cc-runtime ``` -### Enabling debug for various components +To view shim log output: + +```bash +$ sudo journalctl -t cc-shim +``` + +To view proxy log output: + +```bash +$ sudo journalctl -t cc-proxy +``` -The runtime, the shim (`cc-shim`), and the hypervisor all have separate debug -options in the [configuration file](#Configuration). +Note that the proxy log entries also include output from the agent (`cc-agent`) and the +hypervisor (which includes the guest kernel boot-time messages). + +## Debugging + +The runtime, the shim (`cc-shim`), the proxy (`cc-proxy`) and the hypervisor all have separate `enable_debug=` debug +options in the [configuration file](#Configuration). By default, all these +debug options are disabled. Look at the comments in the installed +configuration file for further details. + +If you wish to enable debug for all components, assuming a standard configuration file path, run: + +```bash +$ sudo sed -i -e 's/^#\(enable_debug\).*=.*$/\1 = true/g' /usr/share/defaults/clear-containers/configuration.toml +``` -The proxy (`cc-proxy`) has a command-line option to enable debug output. See -the [proxy documentation](https://github.com/clearcontainers/proxy#debugging) for further details. +For further details, see the [agent debug document](docs/debug-agent.md) and the [kernel debug document](docs/debug-kernel.md). ## Limitations From 0bedb711fe54e857ef7d6fc6508313ad3f3659ca Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Wed, 6 Dec 2017 12:05:13 +0000 Subject: [PATCH 3/4] doc: Update agent and kernel debug documents The agent and kernel debug documents had deviated from reality. Signed-off-by: James O. D. Hunt --- docs/debug-agent.md | 71 ++++++++++++++++++++++---------------------- docs/debug-kernel.md | 26 +++++++--------- 2 files changed, 46 insertions(+), 51 deletions(-) diff --git a/docs/debug-agent.md b/docs/debug-agent.md index 1b041235..518dcd30 100644 --- a/docs/debug-agent.md +++ b/docs/debug-agent.md @@ -4,44 +4,43 @@ Clear Containers runtime launch a special virtual machine when a new pod is created. This VM uses a VM-agent to spawn processes on behalf of the pod/container(s) running inside the VM. -From Clear Containers 3.0-beta release a new agent is used, -[Clear Containers Agent](https://github.com/clearcontainers/agent): +From Clear Containers 3.0-beta release a new agent (`cc-agent`) is used, +[Clear Containers Agent](https://github.com/clearcontainers/agent). -The Clear Containers Agent relies heavily on libcontainer used by runc -(standard program when running containers on bare metal). +The Clear Containers agent relies heavily on [`libcontainer`](https://github.com/opencontainers/runc/tree/master/libcontainer) used by [`runc`](https://github.com/opencontainers/runc/) (standard program when running containers on bare metal). To provide a debug log of any agent activity on a guest, the Clear Containers agent sends logs through a QEMU serial console that are collected by [cc-proxy](https://github.com/clearcontainers/proxy) and shown in its logs. By default, the Clear Containers agent logs are not collected by -`cc-proxy` but can be enabled by adding the `-log debug` option to `cc-proxy`. - -1- Add the `-log debug` option to `cc-proxy` - -``` -mkdir -p /etc/systemd/system/cc-proxy.service.d/ -cat << EOT > /etc/systemd/system/cc-proxy.service.d/proxy-debug.conf -[Service] -ExecStart= -ExecStart=/usr/libexec/clear-containers/cc-proxy -log debug -EOT -# Restart cc-proxy to provide debug logs. -systemctl daemon-reload -systemctl restart cc-proxy -``` -2- Run a container to generate the agent debug logs - -``` -sudo docker run -ti busybox true -``` -3- Filter the agent debug logs from the `cc-proxy` logs -The `cc-proxy` logs show the sources of its collated information. To only see -the agent debug logs, filter `cc-proxy` logs by the QEMU serial console (the -agent logs were sent through it). The QEMU serial console is represented by -`source=qemu`. - -``` -journalctl -u cc-proxy | grep source=qemu -``` - -The debug log format is: -DEBU[0019] *AGENT MESSAGE* source=qemu vm=*CONTAINER_ID* +`cc-proxy` but can be enabled by enabling the proxy debug option. + +1. Enable proxy debug + + Set the `enable_debug=` option in the `[proxy.cc]` section to `true` (assumes a standard configuration file path): + + ``` + $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml + ``` + +1. Run a container to generate the logs: + + ``` + $ sudo docker run -ti busybox true + ``` + +1. Filter the agent debug logs from the `cc-proxy` logs + + The `cc-proxy` logs show the sources of its collated information. To only see + the agent debug logs, filter `cc-proxy` logs by the QEMU serial console (the + agent logs were sent through it). The QEMU serial console is represented by + `source=qemu`. + + ``` + $ sudo journalctl -t cc-proxy | grep source=qemu | egrep '\' + ``` + + To extract all logs entries for a particular container: + + ``` + $ sudo sudo journalctl -t cc-proxy | grep source=qemu | grep vm=CONTAINER_ID | egrep '\' + ``` diff --git a/docs/debug-kernel.md b/docs/debug-kernel.md index ab26b4f6..5d83455e 100644 --- a/docs/debug-kernel.md +++ b/docs/debug-kernel.md @@ -10,37 +10,33 @@ fails due to the kernel. To provide a debug log of the kernel boot, Clear Containers supports an option to enable kernel boot logs using the configuration file. -1. Enable debug in `configuration.toml` +1. Enable kernel boot messages + + Set the `enable_debug=` option in the `[proxy.cc]` section to `true` (assumes a standard configuration file path): ``` - $ enable_debug = true + $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml ``` -2. Run a container to generate the logs +1. Run a container to generate the logs ``` $ sudo docker run -ti busybox true ``` -3. Filter the kernel boot logs from the `cc-proxy` logs +1. Filter the kernel boot logs from the `cc-proxy` logs The `cc-proxy` logs show the sources of its collated information. To see - the kernel boot logs, filter `cc-proxy` logs by the QEMU serial console. + the kernel boot logs, filter `cc-proxy` logs by the QEMU serial console, + excluding the agent messages. The QEMU serial console is represented by `source=qemu`. ``` - $ journalctl -u cc-proxy | grep source=qemu + $ sudo journalctl -t cc-proxy | grep source=qemu | egrep -v '\' ``` -4. Obtain logs using standalone `cc-proxy` - - In some cases it may be easier to run the `cc-proxy` standalone by first stopping - the `cc-proxy` service and running it standalone to capture the logs. + To extract all logs entries for a particular container: ``` - $ sudo systemctl stop cc-proxy - $ sudo /usr/libexec/clear-containers/cc-proxy -log debug + $ sudo sudo journalctl -t cc-proxy | grep source=qemu | grep vm=CONTAINER_ID | egrep -v '\' ``` - - This will result in the `cc-proxy` log being printed on the terminal. This method - can also be used to capture the logs in a specific `cc-proxy` file. From 56aa8749d3db0fd36b9b7f2e546f56c0119c6865 Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Mon, 11 Dec 2017 08:54:45 +0000 Subject: [PATCH 4/4] doc: Update logging and debug docs based on review Apply further changes to the debug and logging documents based on review feedback. Signed-off-by: James O. D. Hunt --- README.md | 21 ++++++++++++--------- docs/debug-agent.md | 26 ++++++++++++++++---------- docs/debug-kernel.md | 3 ++- 3 files changed, 30 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index ef2f755c..2766d38c 100644 --- a/README.md +++ b/README.md @@ -81,8 +81,8 @@ $ cc-runtime cc-env ## Logging -The runtime provides `--log=` and `--log-format=` options. However, it can -also be configured to log to the system log (syslog or `journald`) such that +The runtime provides `--log=` and `--log-format=` options. However, you can +also configure it to log to the system log (syslog or `journald`) such that all log data is sent to both the specified logfile and the system log. The latter is useful as it is independent of the lifecycle of each container. @@ -104,23 +104,26 @@ To view proxy log output: $ sudo journalctl -t cc-proxy ``` -Note that the proxy log entries also include output from the agent (`cc-agent`) and the -hypervisor (which includes the guest kernel boot-time messages). +Note: + +The proxy log entries also include output from the agent (`cc-agent`) and the +hypervisor, which includes the guest kernel boot-time messages. ## Debugging -The runtime, the shim (`cc-shim`), the proxy (`cc-proxy`) and the hypervisor all have separate `enable_debug=` debug -options in the [configuration file](#Configuration). By default, all these -debug options are disabled. Look at the comments in the installed +The runtime, the shim (`cc-shim`), the proxy (`cc-proxy`), +and the hypervisor all have separate `enable_debug=` debug +options in the [configuration file](#Configuration). All of these debug +options are disabled by default. See the comments in the installed configuration file for further details. -If you wish to enable debug for all components, assuming a standard configuration file path, run: +If you want to enable debug for all components, assuming a standard configuration file path, run: ```bash $ sudo sed -i -e 's/^#\(enable_debug\).*=.*$/\1 = true/g' /usr/share/defaults/clear-containers/configuration.toml ``` -For further details, see the [agent debug document](docs/debug-agent.md) and the [kernel debug document](docs/debug-kernel.md). +See the [agent debug document](docs/debug-agent.md) and the [kernel debug document](docs/debug-kernel.md) for further details. ## Limitations diff --git a/docs/debug-agent.md b/docs/debug-agent.md index 518dcd30..5a4b0d39 100644 --- a/docs/debug-agent.md +++ b/docs/debug-agent.md @@ -4,19 +4,25 @@ Clear Containers runtime launch a special virtual machine when a new pod is created. This VM uses a VM-agent to spawn processes on behalf of the pod/container(s) running inside the VM. -From Clear Containers 3.0-beta release a new agent (`cc-agent`) is used, -[Clear Containers Agent](https://github.com/clearcontainers/agent). +The Clear Containers 3.0-beta release uses a new agent (`cc-agent`), +https://github.com/clearcontainers/agent. -The Clear Containers agent relies heavily on [`libcontainer`](https://github.com/opencontainers/runc/tree/master/libcontainer) used by [`runc`](https://github.com/opencontainers/runc/) (standard program when running containers on bare metal). +The Clear Containers agent relies heavily on [`libcontainer`](https://github.com/opencontainers/runc/tree/master/libcontainer) +used by [`runc`](https://github.com/opencontainers/runc/), which is the +standard program when running containers on bare metal. To provide a debug log of any agent activity on a guest, the Clear Containers -agent sends logs through a QEMU serial console that are collected by [cc-proxy](https://github.com/clearcontainers/proxy) -and shown in its logs. By default, the Clear Containers agent logs are not collected by -`cc-proxy` but can be enabled by enabling the proxy debug option. +agent sends log data through a QEMU serial console, which is collected by +[cc-proxy](https://github.com/clearcontainers/proxy) and shown in the proxy's +logs. + +By default, the Clear Containers agent logs are not collected by `cc-proxy` +but can be enabled by enabling the proxy debug option: 1. Enable proxy debug - Set the `enable_debug=` option in the `[proxy.cc]` section to `true` (assumes a standard configuration file path): + Set the `enable_debug=` option in the `[proxy.cc]` section to `true`, which + assumes a standard configuration file path: ``` $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml @@ -30,9 +36,9 @@ and shown in its logs. By default, the Clear Containers agent logs are not colle 1. Filter the agent debug logs from the `cc-proxy` logs - The `cc-proxy` logs show the sources of its collated information. To only see - the agent debug logs, filter `cc-proxy` logs by the QEMU serial console (the - agent logs were sent through it). The QEMU serial console is represented by + The `cc-proxy` logs show the sources of its collated information. To only + see the agent debug logs, filter `cc-proxy` logs by the QEMU serial console + that sent the agent logs. The QEMU serial console is represented by `source=qemu`. ``` diff --git a/docs/debug-kernel.md b/docs/debug-kernel.md index 5d83455e..b5dcc54a 100644 --- a/docs/debug-kernel.md +++ b/docs/debug-kernel.md @@ -12,7 +12,8 @@ to enable kernel boot logs using the configuration file. 1. Enable kernel boot messages - Set the `enable_debug=` option in the `[proxy.cc]` section to `true` (assumes a standard configuration file path): + Set the `enable_debug=` option in the `[proxy.cc]` section to `true`, which + assumes a standard configuration file path: ``` $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml