Improve doc for debug

README.md

@@ -13,8 +13,8 @@
 * [Quick start for developers](#quick-start-for-developers)
 * [Community](#community)
 * [Configuration](#configuration)
+* [Logging](#logging)
 * [Debugging](#debugging)
-    * [Enabling debug for various components](#enabling-debug-for-various-components)
 * [Limitations](#limitations)
 * [Home Page](#home-page)
 
@@ -79,26 +79,51 @@ To see details of your systems runtime environment (including the location of th
 $ cc-runtime cc-env
 ```
 
-## Debugging
+## Logging
 
-The runtime provides `--log=` and `--log-format=` options. However, it can
-also be configured to log to the system log (syslog or `journald`) such that
+The runtime provides `--log=` and `--log-format=` options. However, you can
+also configure it to log to the system log (syslog or `journald`) such that
 all log data is sent to both the specified logfile and the system log. The
 latter is useful as it is independent of the lifecycle of each container.
 
-To view runtime log output,
+To view runtime log output:
 
 ```bash
 $ sudo journalctl -t cc-runtime
 ```
 
-### Enabling debug for various components
+To view shim log output:
+
+```bash
+$ sudo journalctl -t cc-shim
+```
+
+To view proxy log output:
+
+```bash
+$ sudo journalctl -t cc-proxy
+```
+
+Note:
+
+The proxy log entries also include output from the agent (`cc-agent`) and the
+hypervisor, which includes the guest kernel boot-time messages.
+
+## Debugging
+
+The runtime, the shim (`cc-shim`), the proxy (`cc-proxy`),
+and the hypervisor all have separate `enable_debug=` debug
+options in the [configuration file](#Configuration). All of these debug
+options are disabled by default. See the comments in the installed
+configuration file for further details.
+
+If you want to enable debug for all components, assuming a standard configuration file path, run:
 
-The runtime, the shim (`cc-shim`), and the hypervisor all have separate debug
-options in the [configuration file](#Configuration).
+```bash
+$ sudo sed -i -e 's/^#\(enable_debug\).*=.*$/\1 = true/g' /usr/share/defaults/clear-containers/configuration.toml
+```
 
-The proxy (`cc-proxy`) has a command-line option to enable debug output. See
-the [proxy documentation](https://github.com/clearcontainers/proxy#debugging) for further details.
+See the [agent debug document](docs/debug-agent.md) and the [kernel debug document](docs/debug-kernel.md) for further details.
 
 ## Limitations
 

docs/debug-agent.md

@@ -4,44 +4,49 @@ Clear Containers runtime launch a special virtual machine when a new pod is
 created. This VM uses a VM-agent to spawn processes on behalf of the
 pod/container(s) running inside the VM.
 
-From Clear Containers 3.0-beta release a new agent is used,
-[Clear Containers Agent](https://github.com/clearcontainers/agent):
+The Clear Containers 3.0-beta release uses a new agent (`cc-agent`),
+https://github.com/clearcontainers/agent.
 
-The Clear Containers Agent relies heavily on libcontainer used by runc 
-(standard program when running containers on bare metal).
+The Clear Containers agent relies heavily on [`libcontainer`](https://github.com/opencontainers/runc/tree/master/libcontainer)
+used by [`runc`](https://github.com/opencontainers/runc/), which is the
+standard program when running containers on bare metal.
 
 To provide a debug log of any agent activity on a guest, the Clear Containers
-agent sends logs through a QEMU serial console that are collected by [cc-proxy](https://github.com/clearcontainers/proxy)
-and shown in its logs. By default, the Clear Containers agent logs are not collected by 
-`cc-proxy` but can be enabled by adding the `-log debug` option to `cc-proxy`.
-
-1- Add the `-log debug` option to  `cc-proxy`  
-
-```
-mkdir -p /etc/systemd/system/cc-proxy.service.d/
-cat << EOT >  /etc/systemd/system/cc-proxy.service.d/proxy-debug.conf
-[Service]
-ExecStart=
-ExecStart=/usr/libexec/clear-containers/cc-proxy -log debug
-EOT
-# Restart cc-proxy to provide debug logs.
-systemctl daemon-reload
-systemctl restart cc-proxy
-```
-2- Run a container to generate the agent debug logs
-
-```
-sudo docker run -ti busybox true
-```
-3- Filter the agent debug logs from the `cc-proxy` logs
-The `cc-proxy` logs show the sources of its collated information. To only see
-the agent debug logs, filter `cc-proxy` logs by the QEMU serial console (the
-agent logs were sent through it). The QEMU serial console is represented by
-`source=qemu`.
-
-```
-journalctl -u cc-proxy | grep source=qemu
-```
-
-The debug log format is:
-DEBU[0019] *AGENT MESSAGE* source=qemu vm=*CONTAINER_ID*
+agent sends log data through a QEMU serial console, which is collected by
+[cc-proxy](https://github.com/clearcontainers/proxy) and shown in the proxy's
+logs.
+
+By default, the Clear Containers agent logs are not collected by `cc-proxy`
+but can be enabled by enabling the proxy debug option:
+
+1. Enable proxy debug
+
+   Set the `enable_debug=` option in the `[proxy.cc]` section to `true`, which
+   assumes a standard configuration file path:
+
+   ```
+   $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml
+   ```
+
+1. Run a container to generate the logs:
+
+   ```
+   $ sudo docker run -ti busybox true
+   ```
+
+1. Filter the agent debug logs from the `cc-proxy` logs
+
+   The `cc-proxy` logs show the sources of its collated information. To only
+   see the agent debug logs, filter `cc-proxy` logs by the QEMU serial console
+   that sent the agent logs.  The QEMU serial console is represented by
+   `source=qemu`.
+
+   ```
+   $ sudo journalctl -t cc-proxy | grep source=qemu | egrep '\<cc-agent\>'
+   ```
+
+   To extract all logs entries for a particular container:
+
+   ```
+   $ sudo sudo journalctl -t cc-proxy | grep source=qemu | grep vm=CONTAINER_ID | egrep '\<cc-agent\>'
+   ```

docs/debug-kernel.md

@@ -10,37 +10,34 @@ fails due to the kernel.
 To provide a debug log of the kernel boot, Clear Containers supports an option
 to enable kernel boot logs using the configuration file.
 
-1. Enable debug in `configuration.toml`
+1. Enable kernel boot messages
+
+   Set the `enable_debug=` option in the `[proxy.cc]` section to `true`, which
+   assumes a standard configuration file path:
 
    ```
-   $ enable_debug = true
+   $ sudo awk '{if (/^\[proxy\.cc\]/) {got=1}; if (got == 1 && /^#enable_debug/) {print "enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/clear-containers/configuration.toml
    ```
 
-2. Run a container to generate the logs
+1. Run a container to generate the logs
 
    ```
    $ sudo docker run -ti busybox true
    ```
 
-3. Filter the kernel boot logs from the `cc-proxy` logs
+1. Filter the kernel boot logs from the `cc-proxy` logs
 
    The `cc-proxy` logs show the sources of its collated information. To see
-   the kernel boot logs, filter `cc-proxy` logs by the QEMU serial console.
+   the kernel boot logs, filter `cc-proxy` logs by the QEMU serial console,
+   excluding the agent messages.
    The QEMU serial console is represented by `source=qemu`.
 
    ```
-   $ journalctl -u cc-proxy | grep source=qemu
+   $ sudo journalctl -t cc-proxy | grep source=qemu | egrep -v '\<cc-agent\>'
    ```
 
-4. Obtain logs using standalone `cc-proxy`
-
-   In some cases it may be easier to run the `cc-proxy` standalone by first stopping
-   the `cc-proxy` service and running it standalone to capture the logs.
+   To extract all logs entries for a particular container:
 
    ```
-   $ sudo systemctl stop cc-proxy
-   $ sudo /usr/libexec/clear-containers/cc-proxy -log debug
+   $ sudo sudo journalctl -t cc-proxy | grep source=qemu | grep vm=CONTAINER_ID | egrep -v '\<cc-agent\>'
    ```
-
-   This will result in the `cc-proxy` log being printed on the terminal. This method
-   can also be used to capture the logs in a specific `cc-proxy` file.