Technical Notes

Short Overview

Symptom: Inability to login after the first reboot when MySQL package is added to a custom image based on "systemd-image" recipe provided by Angstrom-2012.05 distribution. The second login fails with the following message:

"Cannot make/remove an entry for the specified session"

Problem: MySQL package uses legacy SystemV initialisation scripts while Angstrom-2012.05 uses more modern SystemD one. There is obviously a conflict and some existing services do not get started.

Cure: Delete "start" and "kill" links to /etc/init.d/mysqld from /etc/rcX.d directories and configure SystemD to start MySQL service properly (see below for details).

Debugging

In order to identify what's wrong with an attempt to login it is necessary to see more details from the log file. To achieve this we will forward logging to console. Update the "/etc/systemd/journald.conf" file on the SD card with the following line and try to boot again:

    ForwardToConsole=yes

The attempt to login will fail again but with more output. Something like:

    login[191]: pam_env(login:session): Unable to open env file: /etc/default/locale: No
such file or directory
    login[191]: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
    login[191]: pam_systemd(login:session): Failed to connect to system bus:
    Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or
    directory

login[191]: Cannot make/remove an entry for the specified session
systemd[1]: serial-getty@ttyO0.service holdoff time over, scheduling restart.

It seems that D-Bus service hasn't been started. So, what's the difference between the first boot and the second?

My investigation showed that after the image is created and extracted to SD card there is "S98run-postinsts" script in /etc/rcS.d that has the following content:

#!/bin/bash
    opkg-cl configure
    rm -f //etc/rcS.d/S98run-postinsts

It performs some extra configuration and deletes itself. As a result, after the first boot new links are added to /etc/rcX.d directories that didn't exist before:

    # find /etc -name *mysqld
    /etc/rc0.d/K45mysqld
    /etc/rcS.d/S45mysqld
    /etc/rc6.d/K45mysqld
    /etc/init.d/mysqld
    /etc/rc1.d/K45mysqld

Resolution

If we remove all the "start" and "kill" links to /etc/init.d/mysqld the system will boot and login will be performed without any errors:

    rm -f /etc/rc0.d/K45mysqld
    rm -f /etc/rcS.d/S45mysqld
    rm -f /etc/rc6.d/K45mysqld
    rm -f /etc/rc1.d/K45mysqld

However MySQL daemon is not started during boot. To rectify this we will have to create proper SystemD configuration.

Let's check how MySQL is supposed to get started using legacy SystemV initialisation scripts:

    # cat /etc/init.d/mysqld

    ...
    start)
        /usr/bin/mysqld_safe &
    ;;
    ...

Now it's time to create proper MySQL configuration for SystemD, create "mysql.service" file in "/lib/systemd/system" directory with the following contents:

    [Unit]
    Description=MySQL database server
    After=syslog.target
    After=network.target

    [Service]
    ExecStart=/usr/bin/mysqld_safe
    ExecStop=/bin/kill -15 $MAINPID
    PIDFile=/var/lib/mysql/mysqld.pid

    # We rely on systemd, not mysqld_safe, to restart mysqld if it dies
    Restart=always

    [Install]
    WantedBy=multi-user.target

Then try to activate mysql service to check whether the configuration is created properly:

# systemctl start mysql.service

The output is supposed to be something like:

Starting MySQL database server...
Started MySQL database server [ OK ]

To check the status of the service use:

# systemctl status mysql.service

mysql.service - MySQL database server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled)

          Active: active (running) since Sat, 01 Jan 2000 00:06:33 +0000; 16s ago
        Main PID: 233 (mysqld_safe)
          CGroup: name=systemd:/system/mysql.service
                  â 233 /bin/sh /usr/bin/mysqld_safe
                  â 319 /usr/libexec/mysqld --basedir=/usr --datadir=/var/...

or use "ps" to check that mysql is running:

    # ps | grep mysql
113 root      2724 S    {mysqld_safe} /bin/sh /usr/bin/mysqld_safe
    290 mysql    24240 S    /usr/libexec/mysqld --basedir=/usr --datadir=/var/my
    312 root      2116 S    grep mysql

The last stage is to enable the mysql service to be started during boot:

# systemctl daemon-reload
# systemctl enable mysql.service

Update: It seems that similar conflict is created by "/etc/rcS.d/S39alsa-state" link when "console-image" recipe is used to create rootfs image. Once deleted, the system boots properly and login becomes possible.