-
Notifications
You must be signed in to change notification settings - Fork 196
Embedded Filesystems
Sometimes in an embedded application you might like to be able to read/write from/to some non-volatile memory. It is common for microcontrollers to be paired with SPI flash devices for this purpose, and while it is certainly possible to use such a device without a filesystem, files are a useful and familiar abstraction that can help free up a program from having to manage things like wear leveling, block/sector level erasure, etc.
Request for SPI flash and filesystem support was mentioned in this issue: https://github.com/tinygo-org/drivers/issues/45
Low level I/O driver for SPI NOR flash is available in TinyGo drivers since v0.12.0: https://github.com/tinygo-org/drivers/pull/124
The tinygo.org/x/drivers/flash package provides a driver for initializing and performing read-write operations on a SPI flash device. In general, most applications should not need to use this package directly except to initialize a device and pass it on to a more high-level API such as a filesystem. This packages supports using either SPI (uses a single data line and is available on most/all targets) or QSPI (uses 4 data lines - "quad" - currently only supported on specific pins on ATSAMD51 targets). The package also has functionality to auto-detect the features of some SPI flash parts based on the JEDEC ID (see devices.go), such as maximum speed and whether or not QSPI is supported. If the part you're trying to use is not one that is known to the package, it is also possible to pass your own function to the library for provide details about the part's features. See the DeviceIdentifier interface and the DefaultDeviceIdentifier variable in flash/devices.go for more information.
The flash package can be used with SPI ("Serial Peripheral Interface") on nearly any target that has a flash chip wired up to the right pins. Many development boards have an integrated flash chip; for example any of the Adafruit "Express" M0 or nRF52 "Express" boards should be able to support this, as well as various Particle boards, and others. Interfacing with the chip via SPI takes the following steps:
import "tinygo.org/x/drivers/flash"
// SPI1 pins below should work for Itsy Bitsy M0 board; may need to adjust for other targets
var (
// constructor for *flash.Device
flashdev = flash.NewSPI(
&machine.SPI1,
machine.SPI1_MOSI_PIN,
machine.SPI1_MISO_PIN,
machine.SPI1_SCK_PIN,
machine.SPI1_CS_PIN,
)
)
func main() {
// Initializes the flash chip and reads the JEDEC ID to determine its capabilities.
// We didn't have to call machine.SPI1.Configure() because it is invoked internally
// by flashdev.Configure(). Initially the package with configure the SPI interface
// to run at 5 MHz to get the JEDEC ID; and once the max speed of the chip is known
// it will be reconfigured to run at the maximum speed or 24 MHz, whichever is less.
flashdev.Configure(&flash.DeviceConfig{
Identifier: flash.DefaultDeviceIdentifier,
})
}QSPI ("Quad Serial Peripheral Interface") is very similar in concept to regular SPI, except that it utilizes up to 4 data lines instead of just one. QSPI generally requires special hardware support from a microcontroller, and microcontrollers with this feature often pair it with XIP ("Execute In Place") support for executing code from flash memory. On both ATSAMD51 and nRF52, the QSPI peripheral allows for "memory mapping" the flash memory, so that instead of reading in pages of memory and copying it in and out buffers, we can get a pointer to some offset in the QSPI address space and treat it as if was regular program memory. Behind the scenes the microcontroller takes care of issuing commands to the flash chip and translating memory addresses to pages that need to be fetched from the flash device. Aside from being a foundation to implement filesystems on top of, this feature could also be useful to TinyGo programs for storing things like graphics or fonts, as an alternative to formatting them as byte arrays in the source code and compiling them into the program itself (this could both save flash memory on the microcontroller and also speed up compile times considerably).
Currently the flash package only supports QSPI for ATSAMD51 chips. In the future it should be possible to support the QSPI peripheral on nRF52 as well, but it is not implemented yet. Development boards that support this mode of operation include the Adafruit "Express" M4 boards (Feather, Itsy Bitsy, Metro), PyPortal, PyBadge, etc. On ATSAMD51 chips, QSPI is only available on a fixed set of pins. In TinyGo these pins are specified in src/machine_atsamd51.go (https://github.com/tinygo-org/tinygo/blob/efdb2e852ec79486cf8556c8b53e389f1f66a0f2/src/machine/machine_atsamd51.go#L1205), so the following code should work to initialize a QSPI flash device on any of the aforementioned boards, or on any SAMD51 target that has a QSPI-enabled flash chip hooked up to the right pins:
import (
"machine"
"tinygo.org/x/drivers/flash"
)
var (
// constructor for *flash.Device
flashdev = flash.NewQSPI(
machine.QSPI_CS,
machine.QSPI_SCK,
machine.QSPI_DATA0,
machine.QSPI_DATA1,
machine.QSPI_DATA2,
machine.QSPI_DATA3,
)
)
func main() {
// Initializes the flash chip and reads the JEDEC ID to determine its capabilities.
flashdev.Configure(&flash.DeviceConfig{
Identifier: flash.DefaultDeviceIdentifier,
})
}For the ATSAMD51 implementation, QSPI is initialized to run at 4MHz so that it will work with most chips to read the JEDEC ID. Once the chip is recognized and the max speed is known, the interface will speed up to either the maximum speed for the QSPI peripheral or the maximum speed of the flash device, whichever is less. In practice for the Adafruit boards mentioned above this will probably be around 60MHz (the QSPI speed is controlled by a divider, so unless the flash device can operate at 120MHz, the next slowest speed on ATSAMD51 will be 60MHz). This is obviously significantly faster than the 24MHz maximum when using SPI, especially considering each cycle transfers 4 bits instead of 1.
With low-level IO functions available, it is possible to begin experimenting with more high-level filesystem support. So far I've experimented with 2 filesystem implementations on top of the SPI flash driver, both in the https://github.com/bgould/tinyfs repository:
- github.com/bgould/tinyfs/fatfs
- github.com/bgould/tinyfs/littlefs
Of the two, LittleFS is more full-featured and is what I would recommended for use with SPI flash, as it is far superior with respect to wear leveling and resiliency in case of power loss. FAT is not a great filesystem for use with flash memory, especially if you will be constantly be writing/overwriting files; in that case you would be constantly erasing the same blocks on the device which will eventually destroy them. That said, FAT is kind of the "lowest common denominator" when it comes to compatibility - Windows, OS X, and Linux all support creating/editing FAT filesystems without special drivers for instance, and nearly all SD cards are formatted this way, so it is useful to have support for FAT, especially for read-only or low-write use cases for which wear leveling is little or no concern.
LittleFS (https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md) is a filesystem designed and optimized for embedded applications and flash memory, and is maintained as part of the ARM Mbed OS. The package reference above has CGo wrappers around most of the important functions in the C implementation provided by Mbed to provide a os package style interface from Go. The library is designed to be usable from both from TinyGo as well as standard Go, so the use of some CGo features that are not yet supported in TinyGo needed to be worked around. The status of this library to be "beta" at the moment (could be some bugs and/or may rework some of the API at some point) but it works and should be usable for common use cases like reading/writing config files, graphics/fonts, or data logging. Following is a short example of how to initialize a LittleFS filesystem on a flash chip connected over QSPI:
package main
import (
"machine"
"time"
"github.com/bgould/tinyfs/littlefs"
"tinygo.org/x/drivers/flash"
)
var (
blockDevice = flash.NewQSPI(
machine.QSPI_CS,
machine.QSPI_SCK,
machine.QSPI_DATA0,
machine.QSPI_DATA1,
machine.QSPI_DATA2,
machine.QSPI_DATA3,
)
filesystem = littlefs.New(blockDevice)
)
func main() {
// Configure the flash device using the default auto-identifier function
config := &flash.DeviceConfig{Identifier: flash.DefaultDeviceIdentifier}
if err := blockDevice.Configure(config); err != nil {
for {
time.Sleep(5 * time.Second)
println("Config was not valid: "+err.Error(), "\r")
}
}
// Configure littlefs with parameters for caches and wear levelling
filesystem.Configure(&littlefs.Config{
CacheSize: 512,
LookaheadSize: 512,
BlockCycles: 100,
})
// The filesystem needs to be mounted before it can be used. Below is
// an example of trying to mount the filesystem and then automatically
// formatting and mounting if there is not a valid filesystem to start.
// We'll try three times to mount before formatting just to be safe.
mounted := false
for i := 0; !mounted && i < 3; i++ {
if err := filesystem.Mount(); err != nil {
println("could not mount:", err)
} else {
mounted = true
}
}
if !mounted {
println("formatting flash with new LFS")
formatted := false
for i := 0; !mounted && i < 3; i++ {
if err := filesystem.Format(); err != nil {
println("could not format:", err)
} else {
formatted = true
}
}
if !formatted {
panic("fatal error trying to format flash")
}
}
// Now you can use the filesystem similar to how you might use
// the Go `os` package. Note that to make it easier to switch to
// other tinyfs filesystems, it is recommended to use only the methods
// on the `tinyfs.Filesystem` interface
var fs tinyfs.Filesystem = filesystem
f, err := fs.Open("config.txt")
if err != nil {
// ...
}
defer f.Close()
// etc...
}FAT (File Allocation Table) is a simple type of filesystem originally developed for floppy disks, and even though it has been superseded by more advanced systems on modern computers, it is still supported natively by all major operating systems and is the defacto standard format for SD cards, so it is a good choice is maximum compatibility is a concern. FAT does not have any notion of a journal, so it is not very robust in case of power or hardware failure. Additionally, it is not designed with wear-leveling in mind, so it is easy to wear out sectors on a flash devices if the same files/content are being overwritten all the time.
The implementation of FAT used in the above package is the oofatfs from Micropython (https://github.com/micropython/oofatfs), which is a modified version of the FatFs library found at http://elm-chan.org/fsw/ff/00index_e.html. At the present time, the library is configured as read-only, and thus requires a valid FAT filesystem already flashed to the device. One option for doing this if you have a board that supports CircuitPython is to download get the latest version for your board from https://circuitpython.org/downloads and flash it. A USB drive should show up when connected to your computer, and you can then add and delete whatever files you like. You should be able to then read this files from your TinyGo program, so this could certainly be useful for config files, graphics, etc. An example of how to read a FAT filesystem connected over SPI might look like this (should work on Itsy Bitsy M0):
package main
import (
"machine"
"time"
"github.com/bgould/tinyfs/fatfs"
"tinygo.org/x/drivers/flash"
)
var (
blockDevice = flash.NewSPI(
&machine.SPI1,
machine.SPI1_MOSI_PIN,
machine.SPI1_MISO_PIN,
machine.SPI1_SCK_PIN,
machine.SPI1_CS_PIN,
)
filesystem = fatfs.New(blockDevice)
)
func main() {
// Configure the flash device using the default auto-identifier function
config := &flash.DeviceConfig{Identifier: flash.DefaultDeviceIdentifier}
if err := blockDevice.Configure(config); err != nil {
for {
time.Sleep(5 * time.Second)
println("Config was not valid: "+err.Error(), "\r")
}
}
// Configure FATFS with sector size (must match value in ff.h - use 512)
filesystem.Configure(&fatfs.Config{
SectorSize: 512,
})
// FAT is read-only for the time being; so just try to mount it
// and panic if it fails
if err := filesystem.Mount(); err != nil {
panic("could not mount:", err)
}
// Then we can use it like the Go `os` package... Note that to make
// it easier to switch to other tinyfs filesystems, it is recommended
// to use only the methods on the `tinyfs.Filesystem` interface
var fs tinyfs.Filesystem = filesystem
f, err := fs.Open("config.txt")
if err != nil {
// ...
}
defer f.Close()
// etc...
}