Uploading base files for littlefs file system use

[thatguy-jaysenodell]

I've recently moved "back" to not using the Arduino IDE after years of not using ecplise. I'm using ESP8266 to provide web connectivity to my project. In working to get things "figured out" quickly the Arduino IDE has the ability to upload a littlefs directly to the ESPs. I can not figure out how to get this to work directly from eclipse from any online sources. I suspect I need to add an external tool to the eclipse tool menu, but the build chain is completely obscured.

If somone can point to an existing documentation for this it would be appreciated. Everything I'm seeing is all Arduino IDEv1 with no real applicability to eclipse.

Thanks.

My system is
OSX 14.5
Eclipse Version: 2024-06 (4.32.0) Build id: 20240606-1231
Sloeber (from Eclipse plugin pannel): Sloeber.io Arduino eclipse plugin 4.4.3.202402080117 io.sloeber.core

[jantje]

the Arduino IDE has the ability to upload a littlefs directly to the ESPs.

I guess you refer to upload over the air (abriviated as OTA)
In that case the esp should be visible in the upoload ports if not find out the ip adress and enter the ip adress 2 times in the uploadport (IP adresses seperated with a space)

[thatguy-jaysenodell]

No, I have OTA working. Here is a screen cap from Aruino IDE v2 with the Upload LittleFS feature as the second opion in the command picker. I have ported this project to Sloeber succefully and use the IDE to upload the LittleFS file. I would prefer to move entirely to Sloeber.

Way back when I used eclipse without the Sloeber or Arduino (and strictly with AVR) I had to make a pile of tweaks for fuse setting tools as external tools. I'm assuming there is something I need to do here to push the files with esp8266fs.py (which is what is used in the Arduino v1 days) but am unsure where we are hiding the esp toolchain (stupid Mac).

[jantje]

Seems like you should be able to mount the filesystem on the esp and then dump your files
https://github.com/littlefs-project/littlefs

[thatguy-jaysenodell]

I think that will lead me in the right direction... specifically the mklittlefs project and maybe the littlefs-python project. That is effectively what the arduino tool is doing. Let me work on getting that functional then I'll report back.

I decided to run an upload and grab the console display for commands and such. Odd that the Arduino IDE v2 is using the v1.5 uploader but... might just be because I upgraded vs starting from scratch. Here's the scrubbed-for-user-dirs command into from the arduino IDE.

LittleFS Filesystem Uploader v1.1.8 -- https://github.com/earlephilhower/arduino-littlefs-upload

 Sketch Path: ${project_dir}/LightingController
 Data Path: ${project_dir}/LightingController/data
 Device: ESP8266 series

Building LittleFS filesystem
Command Line: ${HOME}Library/Arduino15/packages/esp8266/tools/mklittlefs/3.0.4-gcc10.3-1757bed/mklittlefs -c ${project_dir}/LightingController/data -p 256 -b 8192 -s 2072576 /var/folders/sp/ln7_tw115_50r97xdtv_clb40000gn/T/tmp-34170-yc9N1tOGA8AF-.littlefs.bin
/wifi.dat
/info.txt

Uploading LittleFS filesystem
Command Line: ${HOME}Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3 ${HOME}Library/Arduino15/packages/esp8266/hardware/esp8266/3.0.2/tools/upload.py --chip esp8266 --port /dev/cu.usbserial-14110 --baud 115200 write_flash 2097152 /var/folders/sp/ln7_tw115_50r97xdtv_clb40000gn/T/tmp-34170-yc9N1tOGA8AF-.littlefs.bin

[jantje]

Running commands like

${HOME}Library/Arduino15/packages/esp8266/tools/mklittlefs/3.0.4-gcc10.3-1757bed/mklittlefs -c ${project_dir}/LightingController/data -p 256 -b 8192 -s 2072576 /var/folders/sp/ln7_tw115_50r97xdtv_clb40000gn/T/tmp-34170-yc9N1tOGA8AF-.littlefs.bin
${HOME}Library/Arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3 ${HOME}Library/Arduino15/packages/esp8266/hardware/esp8266/3.0.2/tools/upload.py --chip esp8266 --port /dev/cu.usbserial-14110 --baud 115200 write_flash 2097152 /var/folders/sp/ln7_tw115_50r97xdtv_clb40000gn/T/tmp-34170-yc9N1tOGA8AF-.littlefs.bin

Should be easy in Sloeber

[thatguy-jaysenodell]

Excellent news... from a "general" perspective I have this working from shell command line use the arduino ide v1.5 tool chain. The ESPtools work with no issues once the syntax is understood. I have a couple of best practices questions for you

1. Should I use the tool chain that is part of Sloeber vs the Arduino IDE? My gut says yes but I would prefer informed input.

jaysen@mba13 cpp-2024-06 % find . -name upload.py
./Eclipse.app/Contents/Eclipse/arduinoPlugin/packages/esp8266/tools/python3/3.7.2-post1/lib/python3.8/distutils/command/upload.py
./Eclipse.app/Contents/Eclipse/arduinoPlugin/packages/esp8266/tools/python3/3.7.2-post1/lib/python3.8/site-packages/setuptools/command/upload.py
./Eclipse.app/Contents/Eclipse/arduinoPlugin/packages/esp8266/hardware/esp8266/3.0.2/tools/upload.py
jaysen@mba13 cpp-2024-06 % find . -name mklittlefs
./Eclipse.app/Contents/Eclipse/arduinoPlugin/packages/esp8266/tools/mklittlefs
./Eclipse.app/Contents/Eclipse/arduinoPlugin/packages/esp8266/tools/mklittlefs/3.1.0-gcc10.3-e5f9fec/mklittlefs
./Eclipse.app/Contents/Eclipse/arduinoPlugin/packages/esp8266/tools/mklittlefs/3.0.4-gcc10.3-1757bed/mklittlefs
jaysen@mba13 cpp-2024-06 % 

2. While this is a one step process with Arduino IDE, I feel that this should be a separate process with Sloeber: a) build LittleFS b) upload LittleFS. Would this align with the general vision for Sloeber (eg more control of the stages)? The segragated build stage would put the fs.bin in the release directory.
3. The standard for Arduino IDE is using a directory "data" within the project to contain the files for littlefs. My thought is that data is used as the default but an option be provided to use any identified directory as the source.

Thoughts?

[jantje]

1. I'd go with sloeber as Sloeber does not know about Arduino IDE (and can run without it being installed
2. If this is a 1 step process in Arduino IDE I do not understand why the ESP people didn't opt for a "programmer". But I'm no longer surprised about the software choices the ESP organization makes.
3. I assume the name "data" for the data folder is just a name "hard/default(?)coded" somewhere. You'll find out.

Thoughts?
Where do I begin ....
You seem to find solutions so I'll give a general overview and 2 possible routes and hope this will get you started.
Read this is sequence as the second route assumes you understood the first route.

The expand the makefile route

Sloeber uses make and make builds targets ("all" and "clean" are the most known but you can have any target name you like)
Sloeber already adds following targets (you can find this in the file called "makefile" in the release folder)

BurnBootLoader: 
	@echo trying to burn bootloader ${bootloader.tool}
	${tools.${bootloader.tool}.erase.pattern}
	${tools.${bootloader.tool}.bootloader.pattern}

uploadWithBuild: all
	@echo trying to build and upload with upload tool ${upload.tool}
	${tools.${upload.tool}.upload.pattern}

uploadWithoutBuild: 
	@echo trying to upload without build with upload tool ${upload.tool}
	${tools.${upload.tool}.upload.pattern}
    
uploadWithProgrammerWithBuild: all
	@echo trying to build and upload with programmer ${program.tool}
	${tools.${program.tool}.program.pattern}

uploadWithProgrammerWithoutBuild: 
	@echo trying to upload with programmer ${program.tool} without build
	${tools.${program.tool}.program.pattern}

CDT allows you to build any target and Sloeber allows you to "extend" the "makefile" to add targets.
By simply defining targets in a file named "makefile.targets" in the root of your project you have added targets to the Sloeber/CDT build system and can "build" these targets.
As you can see the build recipes (that is the make terminology for the actual commands run to make a target) provided above are quite cryptic.
Lets analyze

uploadWithBuild: all
	@echo trying to build and upload with upload tool ${upload.tool}
	${tools.${upload.tool}.upload.pattern}

uploadWithBuild: all

This line creates the target uploadWithBuild and makes it dependent (":") on the target all.
The uploadWithBuild is the build target you should use in sloeber/CDT.
Because uploadWithBuild is dependent on all (which is the normal build target and as such builds the project) the project will be build before the "recipes" are being executed.
The next line is
@echo trying to build and upload with upload tool ${upload.tool}
This is a recipe. You can tell the difference between a recipe and a target because the recipe starts with a tab character.
Targets can be executed in any order but recipes are always executed in sequence.
This recipe will feedback the command and tool that is being run. The ${upload.tool} means "expand the environment variable upload.tool.
The environment variable upload.tool is provided by Sloeber based on the information provided by the user in the project properties->sloeber
There you select a board.txt file and a board ID in this boards.txt file.
FI
yun.upload.tool=avrdude
If you would have selected yun as the board Sloeber creates the environment variable upload.tool and assigns it the value "avrdude"
Note that -as a general rule- all "boardID. values" in the boards.sloeber.txt file are converted to environment variables as well as all the lines in the platform.sloeber.txt and all the lines in the programmers.sloeber.txt. (more on the sloeber.txt later)
On top of that Arduino specifies a whole bunch of environment variables.
That explains why there are so many environment variables and why all actual recipes are environment variables.

The next line is

	${tools.${upload.tool}.upload.pattern}

Again (due to the tab) a recipe. The word "pattern" is a giving away that the environment variable is a command (in Arduino Speak)
It is a concatenated(?) environment variable. As we know ${upload.tool} equals "avrdude" we are looking for the environment variable tools.avrdude.upload.pattern. In the platform.txt that goes with the yun (arduino avr) you will find:
tools.avrdude.upload.pattern="{cmd.path}" "-C{config.path}" {upload.verbose} {upload.verify} -p{build.mcu} -c{upload.protocol} "-P{serial.port}" -b{upload.speed} -D "-Uflash:w:{build.path}/{build.project_name}.hex:i"
This command is in "Arduino format" no $ for the environment variables and platform variants ... Sloeber translated this in the platform.sloeber.txt file. platform.sloeber.tx states
tools.avrdude.upload.pattern="${tools.avrdude.cmd.path}" "-C${tools.avrdude.config.path}" ${upload.verbose} ${upload.verify} -p${build.mcu} -c${upload.protocol} "-P${serial.port}" -b${upload.speed} -D "-Uflash:w:${build.path}/${build.project_name}.hex:i"
As you can see: a command filled with environment variables which (if all goes well) expand to the correct values.
I think it is very likely that it is possible to make a makefile.targets file that contains all the extra commands you need and enabling this functionality is only a matter of copying the makefile.targets to the root of the project.

Using a programmer

In the very same way as described above I think it is possible to create a "programmer".
However the programmer is not directly accessible in Sloeber (I men it is multiple clicks) so changing the programmer may not be the best way forward for your case.

[thatguy-jaysenodell]

Thank you for the feedback.

I have figured out shell commands for USB and OTA uploads using the Sloeber tool chain. No issues. I don't really ISP load the ESPs I have. There are significant differences in the methods for USB/OTA but it isn't significantly different than uploading the built project. I think this lends to your suggestion of a makefile target based solution.

1. As noted, the Sloeber toolchain works well. I'm using that to keep in all known entity.
2. My thought for the programmer was actually one of two options: a) use a series of external tools and config files in the project b) ask you about adding data to the project info to support variables to the build/upload tool. I think your makefile option in conjunction with complile options (in project info or in a .md) should get me there for now.
3. Yes, "data" is hardcoded into the make/upload method in the IDE. I think it would be simpelest, especially for the initial go and to align with existing examples/docs, to stick with the data convention. At least for now.

I have a few things to work on so it may be a bit before I can get anything tested on this. I'll let you know what I come up with.

Thanks for the great tool.

[thatguy-jaysenodell]

And we are good for LittleFS and OTA delivery. I have ... issues that could be part of my eclipse/sloeber setup so feel free to tell me I need to change my install.

makefile.targets.txt

mklittlefs target path is a mess
The platforms.txt is incorract OR the esp boards v3.0.2 package is incomplete. Under Arduino IDE the mklittlefs binary exists in the proper location to allow the variable tools.mklittlefs.path to work vs the nonsense I had to put in there.

mklittlefs really needs variable input
The more I thought about it the more this eventually needs to pull some data from project fields. As it sits right now, I'm eating 2m of flash. That's stupid if you don't need it. If this was something we could pick from the "Flash FS size" field in the project settings that would allow tailoring this per project with little need to mess up a make file. There are routines in the platform.txt that I beleive could be used to calculate the correct value but those seem to take physical input. This just need to convert a conventional size to bytes.
fsSize = reqSize.pos(M) ? reqSize*1024*1020 : reqSize*1024; //assume m or k byte
The would then be passed as the -s parameter to mklittlefs.

why am I only looking at LittleFS?
Because that's what the sketches used. SPIFFS is out of the box? Add a bool in the properties for which FS to use then we just swap out the "little" for "spif" in the targets (assumeing we figure out why I had to mangle the path to mk*fs).

no other tools are doing this... why?
With this being nearly 100% out of the box, why are no IDE supporting FS OTA delivery out of the box? Seems like if delivering code over OTA is of value, the FS over OTA should be of value as well. What am I missing?

[jantje]

mklittlefs target path is a mess

As mklittlefs is a tool; the path to mklittlefs should be defined by the environment variable tools.mklittlefs.path is only true if the tool is part of the version of the platform in the json file used to install mklittlefs.
If I remember correctly you installed mklittlefs manually so there is no reason to assume the environment variable tools.mklittlefs.path even exists.
This is all defined by Arduino here https://arduino.github.io/arduino-cli/0.34/platform-specification/
If you would use runtime.hardware.path you could get rid of 1 ..
Note that I watched the wrong platform-specification page so Sloeber is behind and misses interesting things like build.source.path

mklittlefs really needs variable input

You can do math in make if you want (and also in py but that would mean 2 files instead of one)
https://stackoverflow.com/questions/1926063/how-do-i-perform-arithmetic-in-a-makefile
Contains ways for which the following seem the mos OS independent
https://www.cmcrossroads.com/article/learning-gnu-make-functions-arithmetic

why am I only looking at LittleFS?

I'm not asking these questions anymore. Whatever floats your boat.

no other tools are doing this... why?

There are so many boards; so many solutions; it is simply impossible to stay up to date.
And I thought it is implemented in Arduino IDE so someone is doing it. Och and I'm currently the only one developing Sloeber ... do you think it would be wise of me to start doubting the usefulness just because of that?
So what are you missing? Confidence

[thatguy-jaysenodell]

Because I can't leave well enough alone...

I installed the esp8266 board vs 3.1.0. We have the same issue with the tools paths in platform.txt. Demonstration pruned for brevity:

using env vars ${tools.mklittlefs.path}/${tools.mklittlefs.cmd}
arduinoPlugin/packages/esp8266/hardware/esp8266/3.1.2/tools/mklittlefs/mklittlefs
actual fs location using find
arduinoPlugin/packages/esp8266/tools/mklittlefs/3.1.0-gcc10.3-e5f9fec/mklittlefs

To me this is a problem with the platform definitions. I see the same mess up when I look at the definitions under the Arduino IDE locations (~/Libraries/Arduino/packages/esp8266/...)

Adding all this to suggest "fixing the path" isn't a Sloeber problem. Seems to me this is somewhat intentional though as it exists over all the versions and everytime you search for fs upload over OTA you get "impossible, don't ask".

[jantje]

I did some more digging and came up with this

result := ${shell ${runtime.tools.python3.path}/python3 -c print(\"${build.flash_size}\".replace(\"M\",\"*1024*1020\"))}
result := ${shell ${runtime.tools.python3.path}/python3 -c print(${result})}
test:
	echo ${result}

This should be extendable

[thatguy-jaysenodell]

OK. So there's a whole mess of stuff here that one needs to be careful of that I was intentionally ignoring until now.

• The upload has to actually use the proper flash partition start address for non-OTA uplaods (OTA too, but OTA manages that) .
• Partions really set the size of the FS and need to be considered more so than the file size.
• This is all already covered in board defs and selectable under the project properties, Sloeber, Flash Size options (that's the location for my D1 clone -- see screen shot).
• The math needs to be taking build.spiffs_end - build.spiffs_start (the physical addressess of the fash FS partition ends) and converting that into the size.
• To that end, move all the hardcoded references for block and page sized to use the board def variables.

This will be my fun for this afternoon...

Thank you again for the help in understanding how this works in eclipse/sloeber.

[thatguy-jaysenodell]

Here is the fully functional makefile.targets that is entirely based on variables with the exception of:

• there are references to esp8266 in the upload rules. this is because i copied them from the platform.txt for esp8266.
• i had to copy the upload rules as there is no way that I could see to just override the file name to be uploaded in the recipies already present.
• the use of "data" is the convention for the FS source dir. I figured we should stick with it.
• I'm happy to contribute this to the community (feel free to pack it into the Sloeber source).

fs_size := "${build.spiffs_end} - ${build.spiffs_start}"
fs_size := ${shell ${runtime.tools.python3.path}/python3 -c print\(${fs_size}\)}

mklittlefs:
        @echo Making the LittleFS binary using "${tools.mklittlefs.cmd}"
        ${runtime.tools.mklittlefs.path}/${tools.mklittlefs.cmd} -c ${build.source.path}/data -p ${build.spiffs_pagesize} -b ${build.spiffs_blocksize} -s ${fs_size} ${build.path}/${build.project_name}FS.bin

mkspiffs:
        @echo Making the SPIFFS binary using "${tools.mkspiffs.cmd}"
        ${tools.mkspiffs.path}/${tools.mkspiffs.cmd} -c ${build.source.path}/data -p ${build.siffs_pagesize} -b ${build.spiffs_blocksize} -s ${fs_size} ${build.path}/${build.project_name}FS.bin

uploadFS-OTA: 
        @echo Uploading FS via OTA 
        "${tools.esptool.network_cmd}" -I "${runtime.platform.path}/tools/espota.py" -i "${serial.port}" -p "8266" "--auth=${network.password}" -f "${build.path}/${build.project_name}FS.bin" -s

uploadFS:
        @echo Uploading FS via upload.py connection
        "${tools.esptool.cmd}" -I "${runtime.platform.path}/tools/upload.py" --chip esp8266 --port "${serial.port}" --baud "${upload.speed}" "${upload.verbose}" ${upload.erase_cmd} ${upload.resetmethod} write_flash ${build.spiffs_start} "${build.path}/${build.project_name}FS.bin"

makefile.targets.txt

[jantje]

Congratulations.

and everytime you search for fs upload over OTA you get "impossible, don't ask".

Am I right that this is a nice example of

People Who Say It Cannot Be Done Should Not Interrupt Those Who Are Doing It

[thatguy-jaysenodell]

Yes. especially when I wanted to just get an unnecessarily complicated LED lighting system for my 3d printer enclosure set up across the room without needing to run a cable to it.

That said, it looks like platformio may have fs over ota built in. Too bad I needed something that would be less restrictive of direct access to hardware as well as being a proper IDE. I had started using Sloeber/eclipse way back but stopped working AVR stuff. Seemed like the right place to start when I decided to get back into it. I appreciate your patience. Hopefully I can contribute something useful in the future!