Webserver on NodeMCU
This project is about building a webserver using NodeMCU and a MircoSD card that could host my website.
My website now contains a little less than 3000 files with a total size of
about 100 Mbytes. The NodeMCU has 64Kbyte of data RAM and has 1 MB of built-in
flash. The flash is too small to hold all the data, which means that the files
need to be stored on an external device, such as a MicroSD card. The problem
is that reading data from a SD card is rather slow. So, it would be nice if
the directory information could be stored in the internal memory of the
NodeMCU, preferable in RAM. When using a FAT16 file system the file allocation table does not fit in RAM, but it
could be placed in the flash memory, but it would mean that the flash memory
needs to be written almost every time when the contents of the SD card
changes, which could limit the life time of the flash memory. To retrieve a
file from a FAT file system requires quite a number of read operations to
read the directory and the the file allocation table to know where the file
is stored. Another problem with the FAT16 is that it requires
8.3 filenames, while
the website does not limits itself to this format. A solution could be to
rename all the files to the proper format by processing the HTML and replacing
the names by shorter names before placing them on the SD card.
The idea is to develop a file system that requires less read operations to
retrieve a file and that support arbitrary file names. The main idea is to
store all the data of a file as a consecutive piece of data on the SD card,
such that only the starting sector is needed, making use of the fact that the
available space is much larger than the required space. The directory
information (one flat directory would be sufficient) could be stored at the
start of the SD card, or could be included with the data itself. I will first
attempt the later idea. This would mean that during start-up of webserver, it
would read SD card (about one sector for each file) and build up a tree in RAM
with information of the names of the files and their starting locations.
The sources for this project will be maintained at the
ESP8266WebServer git repository.
File as a block device
By February 21, 2017, I finished
a first implementation of the custom file
system where a file is used as a block device, in order to check the basic
algorithms of allocating space for the files in the block device.
Accessing SD card as block device
The command: fdisk /dev/mmcblk0 -l returned the following information
about a 8GB microSDHC card that I had bought to experiment with:
Disk /dev/mmcblk0: 7.4 GiB, 7948206080 bytes, 15523840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 8192 15523839 15515648 7.4G b W95 FAT32
Under Linux the partition type can be changed using the following command:
# sudo fdisk /dev/mmcblk0
Assuming that the SD card is available under /dev/mmcblk0. Within
fdisk the command p can be used to display the partition
table, which is stored in the Master boot record. This usually will contain one partition of the type
W95 FAT32. Because we want to fill this partition with for an
experimental file system, the partition type 7Fh for "individual or local use and temporary or
experimental projects" should be used. This can be set the t command
and entering 7f. This partition type now displays as unknown.
The w command can be used to write the changes to the SD card.
Please note that any contents on the SD card will become inaccessible, so use
this command at your own risk.
On April 14, 2017, I finished a first impelementation of the
SDfs.cpp program, which can be used to store files in a file
(including a partition, when using a device descriptor such as
/dev/mmcblk0p1). You also have to provide a path that contains the
files and the file sd.log, which constains information about the
files to be added or removed. The program writes to the file sd2.log
an updated version of sd.log with information of the date and time
the files were added. The chkhtml.c
program (which I used for site
maintenance) can read and write to the sd.log file.
Integrated SDK for ESP8266/ESP8285 chips and FreeRTOS
On April 25, 2017, I started installing
the integrated SDK for ESP8266/ESP8285 chips. When I ran make, it reported
errors on gdb.
On April 29, 2017, I spend some
time searching for a solution for the reported errors, but did not find
anything useful. I also looked if I could remove gdb from the make, as
I had understood that it was not really useful for debugging. When I executed
make again, it returned with no errors and all the essential tools being present
to compile FreeRTOS. I
cloned the FreeRTOS github tree, but did not compile it yet.
Links
For accessing SD:
For accessing SD from Linux:
For accessing SD from Windows:
For accessing SD from NodeMCU:
For developing webserver:
Home and email
| Being a hacker