Collecting Serial Log Data
For particularly difficult errors to capture, serial logging can be a low level way to record data. This project offers an always-on logging device for your PC.
Problem Statement
The issue that drove me to design a serial logging device was a intermittent issue with my desktop system. The system would periodically fail to wake from sleep. Although I was able to narrow the issue down to one with the graphics drivers, I could not get further. The machine did not appear to rejoin the network, so I could not access it via ssh. Stuck, I decided to leverage the Linux Kernel's ability to open a serial console.
In order to leverage the kernels' serial console fully, I did not want to connect another desktop via a null modem cable and talk to the desktop's login console directly. Rather, I wanted to use the console's ability to output kernel logs, and capture the kernel logs as they were output. In order to do this, I decided to design an embedded device to collect and write these logs to an SD card. This allowed me to avoid using an entire computer solely to collect logs, and therefore enabled me to keep the logging device running at all times.
Development Process
The first steps for me were selecting a development platform. I went with a Ti Tiva C Launchpad, with an ARM Cortex-M4F MCU. This relatively powerful MCU enabled me to develop using an RTOS, and allowed for fast operation. I selected TI-RTOS as my real time operation system to develop for, due to TI's readily available examples and drivers from the RTOS.
Once I had selected a device to develop for and set up my toolchain, I began designing my application. I opted to create an embedded console, complete with command history and editing, which allowed me to easily issue commands to the MCU to check the application's status and manipulate the logger. Once I had created a console. I connected an SD card to my development board, and then used the FatFS drivers to create and write to a log file on the SD card. Following this, I enabled a second serial connection to read data from the device I wanted to log. As a final step, I enabled the ability to "connect" the commandline console to the serial logging device.
Enabled Features
The basic feature of the logger works as expected. Once the system being logged (the target machine) powers on, it will provide power to the serial logger, and the now powered serial logger will mount the SD card, and write any data the target machine outputs over the UART serial connection to a log file on the SD card.
Beyond the core functionality of the logger, I have enabled several features accessible via the serial commandline. First, I have enabled the user to unmount and remount the SD card, so that they can remove the SD card and view all past logs. Furthermore, I have added a "real time terminal". When requested, the serial logger will relay all characters written to the serial commandline to the target machine, and write the target's output back to the commandline, in effect creating a null modem connection. This can be useful to connect to the target machine machine's serial console if the target machine fails to wake from sleep fully.
Challenges and Future Development
One challenge within the development of this device was unmounting and remounting the SD card. This centers around the fact that the SD card is using the SPI bus, which is not intended to be hotplugged. One fix I have worked on is using a transistor to switch power off to the SD card when it is unplugged, and this has improved the issue somewhat. There is still more debugging work to be done to understand the exact resolution for this issue.
Future development for the serial logged mainly includes quality of life improvements. For example, using the USB connection to expose the SD card would prevent the requirement to unmount the card, and simply connecting to the device. Additionally, I would like to expose the serial commandline over a USB connection. This should be feasible as well, since the commandline is designed to be interface agnostic.