Exploring the Operating System of Reolink RLC-810A

In Part 4 of our series, we focused on the file system and before concluding, we were able to extract files from two UBIFS images contained within the firmware. Here in Part 5, we will try to better understand the structure of the root file system by introducing some basic concepts of the operating system used by Reolink RLC-810A, namely Linux.

Why Linux?

Linux kernel is one of the most widely used operating systems in the world, if not the first by popularity. You might think that commercial operating systems such as Windows or macOS are the most widely used, but on the other hand a whole army of embedded devices (such as the camera in this series) have based their firmware on Linux.

Created in the early 1990s by Linus Torvalds, the Linux kernel is a monolithic, modular, multitasking kernel primarily developed in response to the commercialization of the Unix operating system maintained by AT&T. The Linux kernel and the entire ecosystem above it is largely open-source — every person on the planet is free to be able to consult the sources and modify them.

The choice of Linux as the basis for embedded device firmware stems from a number of features that are essential for development. First among them is the flexibility of Linux — i.e, through being modular, it is possible to load additional components at runtime that enrich Linux with new features. The second is given by the stability of the kernel, thanks to Linux’s dedicated developer community and the wide dissemination of the kernel that has allowed more eyes on code development. The third reason is perhaps one of the most important — it is free. When you market a device, there are a number of licenses that the company has to pay for (hardware, various devices, etc.). For companies that use Linux in their products, there is no cost.

Before we begin to analyze the various folders, it is important to note that almost all operating systems used for embedded devices are a distribution based on the Linux kernel. The characteristics of being open-source, having extensive online support, and being a fairly stable kernel have made the entire family of Linux/Unix-based operating systems the ideal operating system for this type of device.

The Folders of the File System.

Let’s continue our analysis and explore in detail the structure of the operating system. First, we change the directory to rootfs. Then, we run ls to get a more complete overview of the directories.

bin dev etc init lib linuxrc mnt proc root sbin sys tmp usr var

As we might expect, this is the FileSystem Hierarchy Standard feature used for file systems in Unix-like operating systems. In fact, the standard defines which folders and virtual file systems are to be mounted at boot time by the operating system and what each folder contains. This standard is shared among all Unix-like operating systems to ensure compatibility between programs on different platforms.

Let’s explore each folder and file in more detail:

• bin: contains binary files shared by all users. In this folder we find the shell, cat, ls, cp, and many others.
• dev: contains files for peripherals (devices). Most Unix-like operating systems adopt the “Everything as a file” approach where each peripheral is described by some special files that allow manipulating the state of a device. It also contains other pseudo-virtual devices such as /dev/null which produces no output or /dev/random which generates pseudo-random numbers.
• etc: contains system configuration files. It is a very interesting folder that allows you to understand how the manufacturer configured the camera.
• init: the executable that U-Boot loads and executes at the beginning of booting.
• lib: contains essential libraries for binaries in /bin/ and /sbin/.
• linuxrc: file that is executed by init to start populating and mounting some virtual file systems (such as dev, proc).
• mnt: destination used to mount additional file systems (temporarily).
• proc: virtual file system that contains information about active processes (mainly readable as text files).
• root: the root folder for the “root” or “supervisor” user.
• sbin: folder similar to sbin, differing only in the binaries it contains. In this case, they are binaries that must be run by root for system administration.
• sys: file system that allows interfacing directly with hardware devices.
• tmp: temporary folder. The files inside are generally deleted when the system is rebooted.
• usr: contains applications and files used by users. Applications such as browsers, instant messaging, and anything not used by the system are put in this folder.
• var: files with variable data. In particular, we have /var/log which contains the logs produced by the various system components and applications.

Now that we have introduced the different folders, we can begin to explore their contents one by one. Over the course of the next few articles, we will be looking at getting to the bottom of some of the folders to better understand their meaning.

Bin Folder

The bin folder contains the essential binary files for the operating system. Inside we find executables such as ls (to get a list of files), cp (to copy a file), echo (to print a string to the screen), etc.

In our case, we can proceed to see the various binaries in more detail using ls.

ash      chmod     date           echo     fgrep   hostname  linux64  mkdir   netstat       ping           rc_profile    setarch    su        umount      zcat
busybox  chown     dd             ed       fsync   hush      ln       mknod   nice          ping6          rev           setserial  sync      uname
cat      conspy    df             egrep    getopt  iostat    login    more    nvtrtspd      pq_video_rtsp  rm            sh         tar       uncompress
catv     cp        dmesg          false    grep    ipcalc    ls       mount   nvtrtspd_2ch  printenv       rmdir         sleep      test_vos  usleep
chattr   cpio      dnsdomainname  fatattr  gunzip  kill      lsattr   mpstat  nvtrtspd_ipc  ps             scriptreplay  stat       touch     vi
chgrp    cttyhack  dumpkmap       fdflush  gzip    linux32   lzop     mv      pidof         pwd            sed           stty       true      watch

Many of these are commands routinely found in any operating system. The mkdir (make directory) command in Linux/Unix allows users to create or make new directories. With mkdir, you can also set permissions, create multiple directories (folders) at once, etc. What is immediately eye-catching, however, are some particular binary files that are not customarily found in Unix-like distributions, such as nvtrtspd or pq_video_rtsp. We will return to these because they seem to be interesting binaries.

Let’s then proceed to identify the shell. The shell is a command-line interface used to interface with the operating system — you can execute commands, request programs to start, and change settings. Along with the kernel, it is a fundamental component of an operating system.

There are several popular textual shells — with Bash being the most famous. Bash is used for GNU/Linux systems, but also cmd.exe for Windows, Z Shell for macOS. For Reolink RLC-810A, the shell is contained within a binary file called busybox. Depending on how the application has been configured, there can be either bash or zshell.

BusyBox

BusyBox is a tool that allows many standard Unix applications to be encapsulated in one small executable that can be released within the firmware without too much complication. The main advantage is a reduced operating system size, since only one binary is needed to execute multiple commands. Calling different commands with a single executable works by symbolic links that invoke the single binary.

Considered the Swiss Army Knife of Embedded Linux, Busybox is primarily written in the C language and is currently used by a variety of embedded device distributions, including BuildRoot and OpenWRT.

Every command in the system is symbolically linked to the BusyBox binary. When any command is executed, busybot reads from argv[0] the name of the command to be executed and in argv[i] all the parameters of that command. The link between command name and binary on BusyBox is confirmed by a simple file ls in the /bin folder.

ls: symbolic link to busybox

By exclusion, we find the only files that are custom binaries, not BusyBox’s own: nvtrtspd, nvtrtspd_2ch, nvtrtspd_ipc, pq_video_rtsp, rc_profile, test_vos. We can move some binaries to an ad-hoc folder for the future, at which time we will delve by reverse engineering into the functionality and features of each program.

The Configuration Files (etc)

Once we have concluded the somewhat tedious binaries part, we can delve into another folder that will prove very interesting: etc. The etc folder contains all the configuration files for system applications. These configuration files are shared among all users accessing the system.

Originally the etc folder was to contain all files that did not belong to any particular category (hence the use of “et cetera” from Latin), but it quickly became the de facto file configuration folder. Some modern interpretations refer “etc” to “Editable Text Configuration” or “Extended Tool Chest.”

Resuming with our analysis, we continue by focusing on each file found. For the sake of simplicity and breadth, I preferred to include all relevant files within the /etc/ folder in alphabetical order.

Device Tree of Sensors (application.dtb)

The first file we analyze within the /etc/ folder is called application.dtb and is a device tree — a file that allows us to describe a particular hardware configuration. We run the command used during Part 3 to view its contents:

dtc -I dtb -O dts -o - application.dtb

and here is the result:

/dts-v1/;
{

        sensor@1 {
                dev_name = "nvt_sen_imx291";
                cfg = "sen_imx291_cfg1\0sen_imx291_cfg2";
                sie = <0x00 0x01>;
        };

        sensor@2 {
                dev_name = "nvt_sen_imx323";
                cfg = "sen_imx323_cfg1";
                sie = <0x03>;
        };
};

The device tree is used to describe two optical sensors, apparently not used in this camera. It is used by a testing program included in the firmware, but almost never used by the end user.

bootchartd.conf

The bootchard.conf file allows you to import some settings to the famous bootchart tool that is used to measure Linux boot performance. In normal contexts, whether it is a computer or a cellular device, we do not have much interest in spending a few extra seconds during boot up.

However, when it comes to embedded devices, every second is crucial because the concept of “waiting” does not exist for the end user. For example, to start recording any video, we just turn on a camera and press a button. The user does not know that a series of operations are being performed at that time and therefore expects to be able to use the tool right away!

The tool is also useful in embedded contexts to try to detect dangerous bottlenecks. In the case of Reolink, there are two settings given to the kernel: sample_period (i.e., the time interval to measure performance) and process_accounting, which allows tracking all processes executed at startup.

firmware.info

The firmware.info file contains some useful information about the firmware and the Software Development Kit used to generate the embedded distribution. It is not very useful for operating system purposes, but it allows you to extrapolate additional pieces of information.

SDK_VER="NVT_NT96660_Linux_V0.4.8"
BUILDDATE="Tue Mar 1 18:25:28 CST 2016"

fstab, group, and hostname

The fstab file indicates which virtual file systems to mount, in what location, and possibly other parameters. The fstab file is read by the mount command that occurs automatically at startup to determine the overall file system structure.

proc            /proc                   proc    defaults        0       0
sysfs           /sys                    sysfs   defaults        0       0
tmpfs           /dev                    tmpfs   defaults        0       0
tmpfs           /tmp                    tmpfs   defaults,noatime,nosuid,nodev,exec,mode=1777,size=5M 0 0
tmpfs           /var/run                tmpfs   defaults,rw,nosuid,mode=0755    0       0
tmpfs           /mnt/tmp                tmpfs   defaults,noatime,nosuid,nodev,exec,mode=1777,size=20M 0 0 
debugfs         /sys/kernel/debug       debugfs defaults        0       0

The group file, on the other hand, reports the division of users into different groups. In itself, it is not very interesting for our investigation. root:x:0:.

The hostname file reports the name of the machine — BAICHUAN, a clear reference to the company behind Reolink.

inetd.conf

The inetd.conf file is the configuration file for the inetd daemon and each line represents a service that is managed by this daemon. The inetd daemon allows you to control the Internet services that are made available by the machine. The daemon starts services based on the configuration file. Specifically we find ftpd which is started on port 21 by root and makes file uploading/downloading available from sd and tftpd on port 69.

[..redacted..]
21 stream tcp nowait root ftpd ftpd -w /mnt/sd
69 dgram udp nowait root tftpd tftpd -l -c 

inittab

The inittab file specifies the initial configuration to be performed by the init process for BusyBox. It consists of entries that follow the format:

Identifier:Level:Action:Command

where:

• identifier: a string that uniquely identifies an entry;
• level: level associated with the entry, represented by a number (0 to 9). Each entry is executed at the time the system is in the level equal to the entry;
• action: specifies to the process init how to treat the process; and
• command: the command to be executed.

::sysinit:sh /etc/init.d/rcS

# Stuff to do when restarting the init process
::restart:/sbin/init

# Stuff to do before rebooting
::ctrlaltdel:/sbin/reboot
::shutdown:/etc/init.d/rcK
::shutdown:/bin/umount -a -r
::shutdown:/sbin/swapoff -a

The following actions are specified by the inittab:

• at system startup, start the initial scripts (/etc/init.d/rcS);
• if the system is rebooted, then it also restarts the init process; and
• when any action between reboot or shutdown is recorded, then run the /etc/init.d/rcK script and unmount the filesystem.

mdev.conf

Mdev is the tool developed ad-hoc by BusyBox to replace udev. Udev is the device manager for Linux-like operating systems and runs as a daemon. Unlike Unix-based operating systems that rely solely on device management via static text files on /dev, udev allows dynamic management of the creation or removal of these special files.

The syntax that follows mdev.conf is quite arty:

[-]devicename_regex user:group mode [=path]|[>path]|[!] [@|$|*cmd args...]

In our case, we find: mmcblk[0-2] root:root 660 */etc/mdev-script/autosd.sh. It translates to “If you find devices that are mmcblk, then run the autosd.sh script”. Mmcblk is the name of the subsystem that SD/MMC devices use when they are connected to the board. When a new microSD is detected, the autosd.sh script checks a number of parameters at the hardware level (microSD name, type) and if it matches, then it tries to create a new hard link to the contents of the microSD. Since a very thorough check is done on the type of microSD card, it explains why some microSDs may not work.

passwd

The passwd file is used to configure which users have access to the login system and with what combination of username and password they can log in. The syntax for understanding a passwd file is as follows:

username:password:userid:groupid:useridinfo:homedirectory:shell

where:

• username: the username that is used during login.
• password: if there is only one x character, then the password is encrypted and its contents are available in the file /etc/shadow; most of the times if the password is not encrypted, it is hashed with DES.
• userid: is the ID of the user.
• groupid: is the group to which the user belongs.
• useridinfo: additional field that allows storing additional information about the user.
• homedirectory: absolute path where the user will be located once logged in.
• shell: absolute path to the shell to be started.

For Reolink RLC-810A, in particular, we find this file:

root:XF4sg5T82tV4k:0:0:root:/root:/bin/sh

It means that the default user for the webcam is the following username: root, and the hashed password is XF4sg5T82tV4k. With hashcat, we can retrieve original password: bc2020. This password does not allow us to enter exposed interfaces on the Network such as via browser. Rather, if we attached the board via UART to a terminal, we could get inside the system using that particular username/password combination.

Including a passwd file and leaving the password visible within a firmware poses no additional risk if the login requires physical, local access to the board. However, I cannot comment with respect to the security of this login since I have not found any information regarding this password. If anyone finds out more, they can write me an e-mail.

profile and profile_prjcfg

The profile file is used to initialize the default system-wide shell. In fact, when the shell is started as an interactive type shell, it reads the /etc/profile file, executes the commands within it, and reads another file ~/.bash_profile which is the equivalent of the profile file, however specific to a user.

# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

# source profile_prjcfg on /etc/init.d/rcS (init script cycle) and /etc/profile (after startup cycle)
source /etc/profile_prjcfg
export PATH="/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/bin:/sbin"
export LD_LIBRARY_PATH="/lib:/usr/local/lib:/usr/lib:/mnt/app"
export TERMINFO=/usr/share/terminfo
#export LD_PRELOAD="libnvtlibc.so"

if [ -f /etc/hostname ]; then
        /bin/hostname -F /etc/hostname
fi

# coredump setting
echo 1 > /proc/sys/kernel/core_uses_pid
ulimit -c unlimited
echo "/var/log/core-%e-%p-%t" > /proc/sys/kernel/core_pattern

export HOSTNAME=`/bin/hostname`
export PS1='[\u@\h:\w]\$ '

echo "$HOSTNAME Linux shell..."

Some interesting commands in this file are the following:

• source /etc/profile_prjcfg: run the profile_prjcfg file, setting some environment variables.

• echo 1 > /proc/sys/kernel/core_uses_pid and echo "/var/log/core-%e-%p-%t" > /proc/sys/kernel/core_pattern: these two options are used to configure the core tool to generate core dumps when an application fails.

• ulimit -c unlimited: there is no memory limit for the shell. The ulimit command allows you to set a memory limit that the shell or one of the shell-started programs must adhere to, or else be “decapitated” (i.e., killed) by the operating system. Using this command poses serious problems since a process is free to be able to allocate as much memory as it requires. Sometimes it becomes necessary to increase the memory available to a process (and thus bypass the limit) when a program has some strange memory leak. The workaround, however, should be temporary and not permanent in production. This suggests a low-quality development practice.

As for the profile_prjcfg file, it is mainly a set of environment variables that are set. For now, their meaning is unknown since most of the environment variable names are based on the Novatek SDK. We will try to find some clues that can help clarify the ideas.

export MODEL=/home/zfj/520_project/sdk/v2.02/NT9852x_linux_sdk_v2.02.000/na51055_linux_sdk_bc/configs/Linux/cfg_IPCAM1_523_EVB/ModelConfig.mk
export BOARD_DRAM_ADDR="0x00000000"
export BOARD_DRAM_SIZE="0x08000000"
export BOARD_FDT_ADDR="0x00100000"
export BOARD_FDT_SIZE="0x00100000"
export BOARD_SHMEM_ADDR="0x00200000"
export BOARD_SHMEM_SIZE="0x00100000"
[.... continues ....]

Again, we can see how the building phase of a firmware leaves some metadata behind. The developer who built the embedded distribution included within the file in production some references, which are obviously nonexistent. This is not the first time we will notice this and during the course of our series we will find many more.

Other Configuration Files

Within the /etc folder are other configuration files such as services, udhcpdw.conf, wifiap_wpa2.conf, and wpa_supplicant.conf. Interestingly enough, some services (e.g., configuration files for Wi-Fi) that are not minimally used by this camcorder are nevertheless present within the firmware. In fact, the RLC-810A camera has only one way to be connected to the Internet: Ethernet, which it also uses as an electrical resource thanks to PoE (Power Over Ethernet).

Here we conclude our analysis of the configuration and binary files. It was interesting to see how each configuration file gave us some clues about the machine’s enabled services and how it operated. Next week, in Part 6 of our series, we will focus on the /lib folder and the subsequent folders to be analyzed.