Issues when Migrating Home Assistant to USB SSD

SD card to USB SSD Migration Procedure

Migrate Your Home Assistant Server From microSD Card to SSD ja sam to izveo kroz proces Backup, Reinstall, and Restore Home Assistant, ali ništa baš nije prošlo glatko.

SSH to host to port 22222 as root

vi /mnt/boot/config.txt

append line “program_usb_boot_mode=1”

reboot

raspberrypi-enable-usb-boot.md Oficijelno objašnjenje procedura je isto, ali se teško nalazi:Raspberry Pi Documentation - Raspberry Pi hardware

Najzad sam rešio napajanje

Od početka ikada dobijao sam Under-voltage detected! greške odnsono ikonu munje u gornjem desnom uglu, kada je Rpi3 povezan na monitor.

How Much Power Does Raspberry Pi 3B+ Use?

I najzad sam uspeo to da rešim tako što koristim 65W GAN trafo ali baš na USB Type-C portu. Oni originalni PS ne vrede ništa, ali mogu da ih koristim za druge urešaje bez problema.

Database is switched from Sqlite to MariaDb

Explain Recorder vs History vs Logbook

And some nice config here Fixing Home Assistant DataBase Problems With MariaDB Addon | What Are We Fixing Today?

mysql://user:password@SERVER_IP/DB_NAME?charset=utf8mb4

Sa povezanim monitorom preko HDMI porta

Iz nekoga razloga, nenormalna količina grešaka vezanih za bluetooth. Rešio sam potpunim isključenjem Bluetooth pomoću:

Add the following to disable Bluetooth vi /mnt/boot/config.txt

# Disable Bluetooth
dtoverlay=pi3-disable-bt

Disable Bluetooth - Installation / Home Assistant OS - Home Assistant Community Tips - Disabling Bluetooth on Raspberry Pi

Low memory

Iz nekog razloga sada odjednom nema dovoljno memorije pa se zato dešava

Out of memory: Killed process XXXX (node.js) ...

Odavde sam pročitao Home Assistant - High Memory Usage - Installation / Home Assistant OS - Home Assistant Community

i to sam uradio

ha os update --version 9.5

odnosno downgrade-ovao host na prethodne verzije. Osim toga, isključio sam boot za sve nepotrebne add-inove, i tako je rešen memorijski problem. Ne zaboravi da moj Rpi3 ima samo 1GB memorije.

About memory usage of add-ins

ESPHome add-on could use 1GB of ram on it’s own (source)

Monitor memory usage

One of the more likely causes is out-of-memory situations so make sure to monitor memory usage e.g. using the system monitor integration.

System Monitor - Home Assistant

Primer configuration.yaml i lovelace panel za monitoring Rpi3 resursa: homeassistant crashing every day since an update in december · Issue #1232 · home-assistant/operating-system

Postoji caka za još optimizacije RPi3 kada se bootoje sa USB SSD

The Raspberry Pi polls endlessly for a SD card to be inserted if there is not a SD card in the slot. It polls so quickly that it actually registers as CPU usage on your system.

In short, add dtparam=sd_poll_once to /mnt/boot/config.txt

Raspberry Pi: Reclaim ~1% CPU Usage when SSD/USB Booting Pyssel: Raspberry Pi 2, 3, 4 USB SSD or USB drive boot

Enumeracija USB Portova

Default raspored je sa ove slike: Usb Port Numbering - Raspberry Pi, HD Png Download - 660x768(#750132) - PngFind

levo-gore: USB1, levo-dole: USB2 desn-gore: USB3, desn-dole: USB4

Ja ipak nisam imao ovaj problem, ali ponekada je neophodno assign fixed USB port names

Assign fixed USB port names to your Raspberry Pi • freva.com

Hassio Addons

List of addons at hassio/alexbelgium-addons Interesantan: Draggon/hassio-hdd-tools

Various SSD and Hass.io Issues

Docker usage on Hass.io RPi3, most of this should be executed via SSH on the host, specifically through port 22222.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# radi i iz ssh on port 22
docker run -it --pid=host jess/htop

# list containers
docker ps -a

# containers usage statistics, like top
# https://docs.docker.com/engine/reference/commandline/stats/
docker stats

# display the running processes of a container
# https://docs.docker.com/engine/reference/commandline/top/
docker top mycontainerid o pid cmd

A great project is ctop, which is similar to htop but for containers, with the repository located at bcicen/ctop: Top-like interface for container metrics.

Cleaning up Docker space is done using the purge commands, but there are many of them, so caution is needed to avoid deleting something that shouldn’t be deleted.

1
2

# Refer to the Docker documentation for more information on pruning
docker system prune --volumes --force

zigbee2mqtt port 8485 in use

Having trouble again? This time it’s because the port ‘8485’ is already in use by something else on the host, and the solution is simply to completely delete the location where the port number is entered in the configuration.

Check the Speed of the SSD USB

All of this needs to be done through SSH on port 22222, and it would be preferable if there is no background load.

I have recently migrated to an SSD and I would like to determine if there has been any improvement in performance. I want to test the speed of the SSD using the following commands. To make a comparison, another user has provided the following results.

dd if=/dev/sda of=/mnt/data/supervisor/tmp/temp.img bs=8k count=256k
rm /mnt/data/supervisor/tmp/temp.img

The output for that user is:

2147483648 bytes (2.0GB) copied, 20.917460 seconds, 97.9MB/s

Good backup service

Two excellent addons that enhance the built-in backup system are jcwillox/hass-auto-backup, as well as its companion jcwillox/hassio-rclone-backup which utilizes rclone.

Configuring Event History

When doing this, we need to be careful not to quickly fill up the disk, so one of the better configurations is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# https://www.home-assistant.io/integrations/recorder/
#
recorder:
  purge_keep_days: 5 # default is 10 days
  db_url: mysql://desetar:pu5ikaric@core-mariadb/homeassistant?charset=utf8mb4
  exclude:
    domains:
      - automation
      - update
    entity_globs:
      - sensor.sun*
      - weather.*
    entities:
      - sensor.date
      - sensor.last_boot # Comes from 'systemmonitor' sensor platform
      - sun.sun # Don't record sun data
    event_types:
      - call_service # Don't record service calls

Monitoring resources

If we are interested in hardware health, we should first use the official and lightweight integration System Monitor. However, if we want more than that, something like htop but in an application, then we should use the integration Glances.

Proper monitoring of Docker containers must be done using SSH. The following projects are best for this purpose, all accessible through SSH on port 22 or port 22222.

1
2
3
4
5

# ctop image
docker run -it --rm --name=ctop --volume /var/run/docker.sock:/var/run/docker.sock:ro alcapone1933/ctop:latest 

# htop image
docker run -it --rm --name=htop --net host --pid host --ipc host --log-driver none defnotgustavom/htop 

Here are some images for testing speed:

1
2
3
4

docker run -it --rm --privileged --name=hdd-hdparm tinyserve/hdparm -tT /dev/sda 
docker run -it --rm --privileged --name=hdd-diskmark e7db/diskmark

dd if=/dev/zero of=testfile bs=1M count=1000 conv=fdatasync

When I try to perform the test directly from SSH, it doesn’t work:

1
2
3

dd if=/dev/zero of=testfile bs=1M count=100

docker run -it --rm --privileged --net host --pid host --ipc host --name=alpine alpine

Which USB port is which?

The RPi3 only has USB2.0 ports, so the speed is limited compared to the USB3 port on the RPi4. Please refer to the article What USB Ports are on the Raspberry Pi? for detailed information.

I tested the SSD speed myself and my speed is terrible:

1
2
3
4
5

# dd if=/dev/sda of=/mnt/data/supervisor/tmp/temp.img bs=8k count=256k
262144+0 records in
262144+0 records out
2147483648 bytes (2.0GB) copied, 379.623865 seconds, 5.4MB/s
# rm /mnt/data/supervisor/tmp/temp.img

or 5.4MB/s is the speed on that small adapter without power supply.

Now I’m testing it on an adapter that has its own power supply, but I still got an equally bad or even worse result.

# dd if=/dev/sda of=/mnt/data/supervisor/tmp/temp.img bs=8k count=256k
262144+0 records in
262144+0 records out
2147483648 bytes (2.0GB) copied, 595.021445 seconds, 3.4MB/s

That’s around 3.4MB/s, which is really terrible because I should be getting around 20-25MB/s for USB2.0, presumably. Theoretically, the Raspberry Pi 3B+ supports USB 2.0 specification, which allows for transfer speeds up to 480Mbit/s or 60MByte/s.

Is the slow SSD over USB due to the SSD-USB controller?

Damn, what a hassle. Now I have to look for fucking firmware for a cable? Refer to Fixing Storage Adapters for Raspberry Pi via Firmware Updates for more information.

First, let’s check which controller is on the USB-SSD, and on the RPi3, after the lsusb command, it shows the following devices:

LAN Microchip Technology SMSC9512/9514 Fast Ethernet Adapter
JMicron Technology JMS56x USB to ATA Bridge (probably JMS567)

This is expected according to Murphy, as this is THE worst chip for RPi3. Apparently, any from the following is excellent:

ASMedia Technology Inc, or any ASMXXXX is apparently excellent.

That’s why I ordered one from AliExpress because I really don’t want to flash firmware for a damn cable.

My attempts to maximize RPi3 speed

To achieve maximum speed on my RPi3, I have taken the following steps:

Disable unnecessary features such as audio, Bluetooth, and Wi-Fi. I also want to prevent the RPi3 from constantly checking if the SD card is inserted, which it does by default. All these changes can be made in the config file:

/mnt/boot/config.txt

I can edit this file only when SSH is enabled on port 22222, using the built-in vi editor.

For detailed instructions, refer to the excellent Raspberry Pi Documentation on the config.txt file. However, the specific information I need is located in this extensive document: raw.githubusercontent.com/raspberrypi/firmware/master/boot/overlays/README

Here are the steps to perform the desired changes:

• Disable audio: Comment out the line dtparam=audio=on, as the default value is dtparam=audio=off.
• Restrict SD card polling to once after booting: Add the line sd_poll_once=on.
• Disable Bluetooth: Add the line dtoverlay=disable-bt. Note that pi3-disable-bt is only an alias for backwards compatibility.
• Disable Wi-Fi: Add the line dtoverlay=disable-wifi.

Additionally, there are known issues with JMicron controllers, which are described in detail here. I have even attempted to resolve these issues by disabling the “UASP” protocol on the controller, but it doesn’t seem to have helped.

UASP (USB Attached SCSI Protocol) supposedly accelerates the use of large capacity SCSI protocol in server environments, resulting in faster read and write speeds. However, since RPi3 only supports USB 2.0, UASP does not make a difference for me.

Therefore, to disable UASP on JMicron controllers, I can use quirks. Here’s how:

Add usb-storage.quirks=152d:9561:u to /mnt/boot/config.txt. The ID of the controller can be obtained from lsusb, as explained in USB3.0 to SATA adapter causes problems.

After implementing all these changes, the system performance has slightly improved, but occasional freezes still occur.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Enable audio (loads snd_bcm2835)
dtparam=audio=off

# Disable Bluetooth
dtoverlay=disable-bt

# Disable WiFi
dtoverlay=disable-wifi

# Disable UAS on USB3 that we don't have, on JMicron adapters
usb-storage.quirks=152d:9561:u