Webserver on NodeMCU

This project is about building a webserver using NodeMCU and a MircoSD card that could host my website. My website now contains a little less than 3000 files with a total size of about 100 Mbytes. The NodeMCU has 64Kbyte of data RAM and has 1 MB of built-in flash. The flash is too small to hold all the data, which means that the files need to be stored on an external device, such as a MicroSD card. The problem is that reading data from a SD card is rather slow. So, it would be nice if the directory information could be stored in the internal memory of the NodeMCU, preferable in RAM. When using a FAT16 file system the file allocation table does not fit in RAM, but it could be placed in the flash memory, but it would mean that the flash memory needs to be written almost every time when the contents of the SD card changes, which could limit the life time of the flash memory. To retrieve a file from a FAT file system requires quite a number of read operations to read the directory and the the file allocation table to know where the file is stored. Another problem with the FAT16 is that it requires 8.3 filenames, while the website does not limits itself to this format. A solution could be to rename all the files to the proper format by processing the HTML and replacing the names by shorter names before placing them on the SD card.

The idea is to develop a file system that requires less read operations to retrieve a file and that support arbitrary file names. The main idea is to store all the data of a file as a consecutive piece of data on the SD card, such that only the starting sector is needed, making use of the fact that the available space is much larger than the required space. The directory information (one flat directory would be sufficient) could be stored at the start of the SD card, or could be included with the data itself. I will first attempt the later idea. This would mean that during start-up of webserver, it would read SD card (about one sector for each file) and build up a tree in RAM with information of the names of the files and their starting locations.

The sources for this project will be maintained at the ESP8266WebServer git repository.

File as a block device

By February 21, 2017, I finished a first implementation of the custom file system where a file is used as a block device, in order to check the basic algorithms of allocating space for the files in the block device.

Accessing SD card as block device

The command: fdisk /dev/mmcblk0 -l returned the following information about a 8GB microSDHC card that I had bought to experiment with:
Disk /dev/mmcblk0: 7.4 GiB, 7948206080 bytes, 15523840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1       8192 15523839 15515648  7.4G  b W95 FAT32

Under Linux the partition type can be changed using the following command:

# sudo fdisk /dev/mmcblk0
Assuming that the SD card is available under /dev/mmcblk0. Within fdisk the command p can be used to display the partition table, which is stored in the Master boot record. This usually will contain one partition of the type W95 FAT32. Because we want to fill this partition with for an experimental file system, the partition type 7Fh for "individual or local use and temporary or experimental projects" should be used. This can be set the t command and entering 7f. This partition type now displays as unknown. The w command can be used to write the changes to the SD card. Please note that any contents on the SD card will become inaccessible, so use this command at your own risk.

SDfs program

On April 14, 2017, I finished a first impelementation of the SDfs.cpp program, which can be used to store files in a file (including a partition, when using a device descriptor such as /dev/mmcblk0p1). You also have to provide a path that contains the files and the file sd.log, which constains information about the files to be added or removed. The program writes to the file sd2.log an updated version of sd.log with information of the date and time the files were added. The chkhtml.c program (which I used for site maintenance) can read and write to the sd.log file.

Integrated SDK for ESP8266/ESP8285 chips and FreeRTOS

On April 25, 2017, I started installing the integrated SDK for ESP8266/ESP8285 chips. When I ran make, it reported errors on gdb.

On April 29, 2017, I spend some time searching for a solution for the reported errors, but did not find anything useful. I also looked if I could remove gdb from the make, as I had understood that it was not really useful for debugging. When I executed make again, it returned with no errors and all the essential tools being present to compile FreeRTOS. I cloned the FreeRTOS github tree, but did not compile it yet.


For accessing SD: For accessing SD from Linux: For accessing SD from Windows: For accessing SD from NodeMCU: For developing webserver:

Home and email | Being a hacker